How to run Karma tests in browsers in Docker

17 January 2018

Written by Jon Bevan

There are a couple of benefits to running karma tests inside of browsers that are running inside of Docker:

  • its easy to test against multiple versions of a browser
  • you can test against different browser versions than your build server has installed
  • you can test against browsers that aren't even installed on your build server!

In my scenario I wanted to achieve two things:

  • testing against a more recent version of Firefox than our build servers had installed
  • making the tests more reliable.

Often our Karma tests would fail because Firefox didn't respond within 10s. By "dockerizing" Firefox I hoped to gain some reliability.

Firstly some things to avoid:

Don't build your own Docker containers

You'll need to solve problems like how to install the various browsers latest releases on whatever base OS your container use; you'll either need to maintain these containers or extend your automated builds to include generating the container for each run, which could be quite slow.

In the end I used cypress/browsers:chrome56-ff47 but I probably could have used the selenium images instead.

Don't move your entire Karma build/preprocessing to run inside the Docker container

We have what feels like a relatively large Webpack build pipeline and shifting that from running on the host machine to running in the Docker container was a mistake:

  • each container would have an empty yarn cache without some non-trivial config
  • introducing Docker adds several layers of additional software complexity to the procedure unnecessarily
  • the Docker container would run out of memory while bundling all our dependencies code.

The success story:

All that Karma needs to do is launch a new Docker container pointing the relevant browser at the right URL.

NB: for automated builds you'll want to have pulling the Docker image as a prerequisite to the Karma test run as it'll take a while... Alternatively you can guess how long it will take and bump the connection timeouts in your Karma config. We happen to run our tests via gradle so I added a task dependency on a new Gradle Docker Plugin task that pulls the image.

Thankfully Karma can launch a browser via a shell script. Just install the karma-script-launcher plugin and provide an ABSOLUTE path to the script in your browsers array. At the time of writing no errors/warnings are provided by Karma if it can't find the script.

That script should not terminate once the browser has started.

The script receives a single argument that is the URL of the page Karma wants the browser to use. Because we're running our browsers in Docker we need to modify the hostname.

Additionally we also need to shutdown the Docker container when we're done.

So here's what our Firefox shell script looks like:

#!/bin/bash

set -u
set -e
set -o pipefail

URL="$1"
OSNAME=$(uname)

# Fix the URL so the container can access the host
if [[ "$OSNAME" == "Darwin" ]]; then
    DOCKER_URL=$( echo "$URL" | sed s/localhost/docker.for.mac.localhost/g | sed s/0.0.0.0/docker.for.mac.localhost/g )
else
    DOCKER_URL=$( echo "$URL" | sed s/localhost/172.17.0.1/g | sed s/0.0.0.0/172.17.0.1/g )
fi

echo "Launching firefox: $DOCKER_URL"

# Setup the shutdown/cleanup function
killFirefox() {
  docker rm -fv firefox-karma
}

trap "killFirefox; exit 0" SIGKILL EXIT

# Launch!
docker run --rm --name firefox-karma cypress/browsers:chrome63-ff57 firefox -headless "$DOCKER_URL"

That shell script just sits in the same directory as our karma.conf.js which has this snippet in it:

browsers: [
    __dirname + '/docker-browser-firefox.sh',
    // __dirname + '/docker-browser-chrome.sh'
],

I tried getting this working with Chrome headless but only got the following message libudev: udev_has_devtmpfs: name_to_handle_at on /dev: Operation not permitted even when using -v /dev/shm:/dev/shm as part of the docker run command and --no-sandbox --disable-gpu as part of the browser args.

Summary

Voila! Karma tests running in browsers running in Docker :)

This could probably be packaged up in to a nice NPM module, but I figured that all that it would do was replace the shell script so I wasn't entirely convinced the effort was worth it.

blog comments powered by Disqus