Streampunk.cc is a small, self-sustained organization providing Icecast streaming services to free radios.
It is based on Autoradio, a software dedicated to these goals.
A few reasons to build this software:
Radios often need to deal with large traffic spikes, for example when covering
large demonstrations, and their utilization varies wildly.
What to do when one machine isn't enough anymore:
1) standard Icecast relay setup
2) giss.tv Icecast patches (single traffic-controlling master)
Single points of failure are annoying because they add maintenance overhead. Prolonged master failures are bad because that's where all the sources are connected, so they make the rest of the network useless.
We think that there is a somewhat unexplored niche in distributed systems in these times of big data: systems that are meant to scale down easily.
This design can produce reliable services that are extremely cheap to run, both in terms of cost and administrator time.
autoradio can coordinate a bunch of machines into a fault-tolerant Icecast cluster. It removes the single point of failure by running a master-election protocol on top of a distributed database (etcd).
We use a few simple distributed primitives:
The cluster tolerates failure of N/2-1 nodes. It can guarantee recovery from node failures (including the master) within the time constant of the master-election protocol, usually a few seconds.
The service can easily scale horizontally by adding new machines with an automated procedure.
Traffic is controlled at the DNS and HTTP level. Sources are reverse proxied to the master, while clients are redirected to the appropriate relay. In more detail:
1) Client finds a server by using DNS. There is little control over this stage, client hopefully will get a working server.
2) Client makes a HTTP request for the stream URL. Autoradio picks a server for the new connection, and sends back a redirect to a special "passthrough" URL on the chosen server.
3) The client makes the final HTTP request on the desired backend, autoradio on that machine simply proxies the connection to the local Icecast daemon.
Problems: some very old clients do not understand HTTP redirects.
Autoradio implements a tiny programmable traffic control engine, with predictive estimates of backend utilization. What this means is that it can:
The server selection algorithm can be configured, for example:
will distribute requests randomly, weighted by listener utilization, and will block requests to servers with utilization greater than 100%.
Operators can create transcoded streams, providing a version of a stream automatically re-encoded with different parameters (codec, bitrate).
Autoradio uses liquidsoap to do this, running an instance for each transcoded stream, wrapped within a master-election protocol to guarantee global uniqueness.
we discovered that many clients don't really know HTTP: they work with Icecast,
but not with a real HTTP server.
butt, we're looking at you
Icecast doesn't really speak HTTP, either.