Automated Testing for OpenTok Applications in the Browser

tokbox-inc_markWeb Application Developers are used to being able to write automated tests for their applications and have them run with every PR and before deploying to production to give a level of confidence that things are not broken. OpenTok and real-time applications in general present new challenges when it comes to writing and running automated tests. There are challenges when it comes to getting access to microphones and cameras, testing multiple participants and installing the plugin for Internet Explorer among others.

There has been lots of work around WebRTC testing automation and our friends at and &yet have written some great articles on the subject. However these articles don’t cover some of the specifics of testing OpenTok applications for example testing Internet Explorer and installing the OpenTok plugin for Internet Explorer. If you haven’t already I would recommend taking some time to read the articles by the folks at and &yet before coming back to this. Also if you’re not familiar with Travis and Selenium WebDriver you might want to check those out too.

This article details the automated testing setup for opentok-meet which includes automated tests for Chrome and Firefox as well as Internet Explorer using Travis, Karma and Selenium WebDriver. With every PR to opentok-meet the whole suite of unit and integration tests run in IE 10 & 11 and the latest versions of Chrome and Firefox to make sure that everything is working before it is merged in. We also use a similar workflow for testing opentok.js itself and the WebRTC team use similar tools and techniques with the webrtc repo.

The scripts used to run these tests can be found in the opentok-test-scripts repo.

Travis Workflow

The Travis setup for opentok-meet is built on top of travis-multirunner. It uses the same Travis Build Matrix setup with `BROWSER` and `BVER` environment variables to specify which browser and version. In addition there is a `SAUCELABS` and a `BROWSERSTACK` optional environment variables that if set will start up Sauce Connect and BrowserStackLocal respectively. You can also specify `ie` as the `BROWSER` with `10` or `11` as the `BVER`.

Then the test script is run for every BROWSER/BVER combination. It uses travis-multirunner to install the browser if it can (not IE). It then starts Sauce Connect or BrowserStackLocal if they’re needed. Finally it runs the unit tests followed by the integration tests and if there are any failures it fails the build.

Running Tests in Chrome and Firefox

We’re using travis-multirunner to run the tests in Chrome and Firefox. Travis multi-runner handles installing the right version of Chrome on the virtual machine so that we can run the tests right there on the Travis machine. This means that the way you run the tests on Travis is just the same as how you would run them on your own machine locally.

In Chrome and Firefox there are some flags that you need to set to bypass the allow access dialog box that pops up when you use your camera. Also, the testing box that we’re using does not have a microphone or camera attached and so we want to use fake devices. We’re using karma to run our unit tests and protractor to run our integration tests and they have different ways of setting these flags. I’ll show you the way we set these values in protractor, the test runner that you’re using will have it’s own way to but the same concept applies.

To do this in Chrome we set use-fake-device-for-media-stream and use-fake-ui-for-media-stream. We also set the path to where the Chrome binary is like so:

For Firefox we set media.navigator.permission.disabled and media.navigator.streams.fake like so:

And we set the binary path like so:

Running Tests in IE

Travis is running on Linux and there is no Linux version of Internet Explorer so we have to run the tests somewhere else. There are a few services out there that support this. We tried using Sauce Labs and Browserstack. These services all basically work the same way. The idea is that you run your webserver on Travis and then you tell Browserstack or Sauce Labs to launch a browser pointing at the URL of your WebServer. Because the URL usually isn’t publicly accessible and you probably don’t want it to be, you setup an SSH proxy connection to allow access to it. This is called Sauce Connect for Sauce Labs and BrowserstackLocal for Browserstack. The opentok-test-scripts repo includes some scripts to help with setting up these proxys.

We decided to go with Sauce Labs for IE support because we need to be able to install the OpenTok plugin for Internet Explorer and Sauce Labs had a way for us to do this using pre-run executables but Browserstack did not. The pre-run executables feature allows you to specify a URL to an exe file that you want to run before the tests run. It will let you specify an `exe` or a `bat` file on Windows, a shell script or Apple script on OSX and a shell script on Linux. So we can bundle the OpenTok Plugin installer which comes as an `msi` file into an `exe` and pass the link to Sauce Labs. We also need the IE plugin to do what Chrome and Firefox do with the fake devices and not requiring permission to access devices.

We wrote a script as part of the opentok-test-scripts repo which does this for you. It downloads the right version of the plugin depending on the path you specify. Then it generates an install script that modifies the configuration file to tell the IE plugin to use fake devices and bypass the allow dialog. Finally it packages this all up as a SauceLabsInstaller.exe file which you can pass to Sauce Labs. To use this script we simply add a line to our before_script in the .travis.yml which generates the SauceLabsInstaller.exe file and then puts it in the public folder so we can serve it to Sauce Labs.

Then we modify the protractor configuration to pass the prerun executable to Sauce Labs.

Multi-party tests

A large part of what you want to test with your integration tests in an OpenTok application is the interaction between multiple participants. With Protractor or Selenium WebDriver you can achieve this by using the forkNewDriverInstance() method to fork a new browser. This works great on Chrome and Firefox running on Travis. The issue on IE is that this actually spins up a whole new virtual machine in Sauce Labs. This means that you have to wait for the whole pre-run executable to install again which can take a long time.

So instead I decided to just use JavaScript to open a new browser window and test between windows. This seems to work well so far. You can also control the second browser window using getAllWindowHandles and switchTo. Similar to this Stack Overflow post.