Now the Red5 Media Server is used for live streaming beyond Flash, including HLS, WebSockets, and RTSP. We’ve since expanded into other open source developer tools and projects ranging from live-stream load testing to client-side streaming libraries, mobile SDK wrappers, extensions, and tutorials. This is going to be difficult to simulate using tools like JMeter or any other generic load testing tool that typically favours HTTP. The biggest drawback is that RTMP only works in Flash and not in HTML5.
There are two types of test I’ll describe below. First of all using Apple HLS streams, which is HTTP Live Streaming via port 80, supported by iOS and Safari, and also by Android (apps and browser). Then we have Adobe’s RTMP over port 1935, mostly used by Flash players on desktop, this covers browsers like Internet Explorer and Chrome on desktop. These tests apply to Wowza server but I think it’ll also cover Adobe Media Server.
All links to files and software mentioned are duplicated at the end of this post.
- RTMP Load Testing Tools Resources All the load testing tools that can help a website or intranet application developer to ensure good performance and no crash under stress.
- Small Projects targeted load testing tools: Apache Bench - tool provided in the distribution of Apache. It makes it possible to test the performances of any server by specifying a number of requests to send to the web server with a concurrency level (simultaneity of the requests). Httpperf - tool.
It’s worth noting that you can stick to HLS entirely by using an HLS plugin for Flash video players such as this one, and that is what we’re doing in order to make good use of Amazon’s CloudFront CDN.
For the purpose of testing you may also wish to simulate some live camera streams from static video files, see further down this post for info on how to do that on your computer, server or EC2.
Testing RTMP Live Streaming with Flazr
In this test we want to load test a Wowza origin server itself to see the direct effect of a lot of users on CPU load and RAM usage. This test is performed with Flazr, via RTMP on port 1935.
Assuming you’ve set up your Wowza or Adobe Media server already, for example by using a pre-built Wowza Amazon EC2 AMI. We’re using an m3.xlarge instance for this test as it has high network availabilty and a tonne of RAM – and we’re streaming 4 unique 720p ~4Mbit streams to it, transcoded to multiple SD and HD outputs (CPU use from this alone is up to 80%).
Installing flazr
First up, for the instance size and test configuration I’m using I modified flazr’s client.sh to increase the Java heap size to 8GB, otherwise you run out of RAM. Next up FTP and upload (or wget) flazr to a directory on your server/EC2 instance. Then SSH in and:
The order of parameters does seem to matter in later versions of flazr, but either way this test runs for 60 seconds, with a load of 1000 viewers. Given all the transcoding our CPU was already feeling the pain, but there was no sign of trouble. We managed 4500 before anything started to stutter in our test player from another m3.xlarge instance.
Of course this only matters if you are not using a CDN, but it’s good to know this EC2 instance can handle a lot of HD viewers.
Testing HLS Live Streaming (or a CDN such as Amazon CloudFront) with hlsprobe
Onto HLS streaming, the standard for mobile apps and sites. We have used Wowza CloudFront Formations to set up HLS caching for content delivery, so that we can handle a very large number of viewers without impacting on the CPU load or network throughput of the origin server, and to giver us greater redundancy. Given CloudFront works with HLS streams we are not using RTMP for this test, so we cannot use Flazr again. To test HLS consumption –that being the continuous download of .m3u8 files and their linked .ts video chunks– we can use a tool called hlsprobe, which written in python.
If you’re on a Mac and don’t have python I recommend you install it via brew to get up and running quickly. If you don’t have brew, get it here.
hlsprobe also relies on there being an SMTP server running, not that you need a fully functional one but:
Then to install hlsprobe’s dependencies and hlsprobe itself:
A sample config is linked at the end of the post.
Running hls probe is as simple as this (note the -v verbose mode, you can turn that off once you have it working).
Now if you fire up the Wowza Engine Manager admin interface, you should still see the connection count and network traffic, but the traffic. If you’re testing your CDN such as with CloudFront, you should note that your CPU usage does not increase substantially as you add thousands of clients.
Simulating cameras to Wowza via nodeJS
Rtmp Load Testing Tool Harbor Freight
It’s good to be able to simulate live streams at any time, either from your computer or in my case, from some EC2 instances. To do this I’ve written a simple nodejs script which loops a video, optionally transcoding as you go. I recommend against that due to high CPU use and therefore frame-loss; in my sample script I am passing through video and audio directly, the video is already using the correct codecs, frame size and bitrate via Handbrake.
The script runs ffmpeg, so you’ll need to install that first:
Edit the js script to point to your server, port, and video file, the run the script with:
If the video completes, it’ll restart the stream but there will be a second of downtime, some video players automatically retry, but make sure your video is long enough for the test to be safe.
These are just a couple of ways of load testing a live streaming server, there are 3rd parties out there but we’ve not had great success so far, and this way you have a lot more control over the test environments.
Rtmp Streaming Software
Links
Rtmp Stream
fakestream.js – NodeJS script to simulate live streams
config.yaml – Sample config for hlsprobe
hlsprobe – Tool for testing HLS streams
Flazr – Tool for testing RTMP streams
OSMF-HLS – OSMF HLS Plugin to support HLS in Flash video players