Streampunk

Why new software?

A few reasons to build this software:

• educational purpose (coordination primitives on etcd, traffic control)
• scaling / management issues with existing radios (Ondarossa)

Radios often need to deal with large traffic spikes, for example when covering
large demonstrations, and their utilization varies wildly.

• keeping many machines always on is just too expensive; we want it to be easy to add new machines when we really need it
• shit happens: the day of the big demo is the day your main Icecast server will go down

Alternatives

What to do when one machine isn't enough anymore:

1) standard Icecast relay setup

• no mechanism to route clients
• master is SPOF

2) giss.tv Icecast patches (single traffic-controlling master)

• master redirects clients to least-loaded relay
• master is SPOF

Single points of failure are annoying because they add maintenance overhead. Prolonged master failures are bad because that's where all the sources are connected, so they make the rest of the network useless.

Distributed systems that scale down

We think that there is a somewhat unexplored niche in distributed systems in these times of big data: systems that are meant to scale down easily.

• small, efficient (simplistic) codebase
• designed to take advantage of tiny, low-power machines
• all components of a "real" distributed system

This design can produce reliable services that are extremely cheap to run, both in terms of cost and administrator time.

Autoradio

autoradio can coordinate a bunch of machines into a fault-tolerant Icecast cluster. It removes the single point of failure by running a master-election protocol on top of a distributed database (etcd).

We use a few simple distributed primitives:

• presence broadcasts (with utilization stats)
• master-election (using CAS and heartbeats)

The cluster tolerates failure of N/2-1 nodes. It can guarantee recovery from node failures (including the master) within the time constant of the master-election protocol, usually a few seconds.

The service can easily scale horizontally by adding new machines with an automated procedure.

Autoradio : traffic flow

Traffic is controlled at the DNS and HTTP level. Sources are reverse proxied to the master, while clients are redirected to the appropriate relay. In more detail:

1) Client finds a server by using DNS. There is little control over this stage, client hopefully will get a working server.

2) Client makes a HTTP request for the stream URL. Autoradio picks a server for the new connection, and sends back a redirect to a special "passthrough" URL on the chosen server.

3) The client makes the final HTTP request on the desired backend, autoradio on that machine simply proxies the connection to the local Icecast daemon.

Problems: some very old clients do not understand HTTP redirects.

Autoradio : traffic control

Autoradio implements a tiny programmable traffic control engine, with predictive estimates of backend utilization. What this means is that it can:

• prevent individual servers from overloading
• mitigates thundering herd effects in case of high connection rates
• distribute load somewhat equally to minimize impact of single server loss

The server selection algorithm can be configured, for example:

listeners_available,listeners_score,weighted

will distribute requests randomly, weighted by listener utilization, and will block requests to servers with utilization greater than 100%.

Autoradio : transcoding

Operators can create transcoded streams, providing a version of a stream automatically re-encoded with different parameters (codec, bitrate).

Autoradio uses liquidsoap to do this, running an instance for each transcoded stream, wrapped within a master-election protocol to guarantee global uniqueness.

Meanwhile, clients

we discovered that many clients don't really know HTTP: they work with Icecast,
but not with a real HTTP server.

butt, we're looking at you

Icecast doesn't really speak HTTP, either.

Questions?

Thank you