Today I released Tox-Docker to GitHub and the Python Package Index. Tox-Docker is a plugin for Tox, which, you guessed it, manages one or more Docker containers during test runs.

Why Tox-Docker?

Tox-Docker began its life because I needed to test some code that uses the PostgreSQL JSONB column type. In another life, I might have done this by instructing Jenkins to first install Postgres, start the server, create a user, create a database, and so on. There's even a small chance that this would work well, some of the time -- so long as tests didn't fail, the build didn't die unexpectedly without cleaning up, multiple tests didn't run at once, and so on. In fact, I've done exactly this sort of hackery a few times in the past already. It is dirty, and often requires manual cleanup after failures.

So, when confronted with the need to write tests that talk to a real Postgres instance, rather than reaching into my old toolbox I was determined to find a better solution. Docker can run multiple instances of Postgres at the same time in isolation from one another, which obviates the need for mutual exclusion of builds. Docker containers are lightweight to start, and easy to clean up (you can delete them all at once with a single command), so when the tests are done, we can simply remove it and move on.

There was still the question of how to manage the lifecycle of the container, though, which is where Tox comes in. Tox is a test automation tool that standardizes an interface between your machine (or your continuous integration environment) and your test code. Like Docker, Tox encourages isolation, by creating a clean virtualenv for each test run, free of old package installs, custom hacks, and so on. Tox already has a well-defined set of steps it runs to build your package, install dependencies, start tests, and gather results. Happily, it allows plugins to hook into this sequence to add custom behavior.

How Tox-Docker Works

Tox's plugins implement callback hooks to participate in the test workflow. For Tox-Docker, we use the pre-test and post-test hooks, which set up and tear down our Docker environment, respectively. Importantly, the post-test hook runs regardless of whether the tests passed, failed, or errored, ensuring that we'll have an opportunity to clean up any Docker containers we started during the pre-test hook. Finally, Tox plugins can also hook into the configuration system, so that projects using Tox-Docker can specify what Docker containers they require.

The simplest use of Tox-Docker is to specify the Docker image or images, including version tags, that are required during test runs. For instance, if your project requires Postgres, you might add this to your tox.ini:

[testenv]
docker = postgres:9.6

With Tox-Docker installed, the next time you run tox, you will see something like the following:

py27 docker: pull 'postgres:9.6'
py27 docker: run 'postgres:9.6'
py27 runtests: PYTHONHASHSEED='944551639'
py27 runtests: commands[0] | py.test test_your_project.py
============================= test session starts ==============================
platform linux2 -- Python 2.7.12, pytest-3.0.7, py-1.4.33, pluggy-0.4.0
rootdir: /home/travis/build/you/your-project, inifile:
collected 3 items
test_your_project.py ...
=========================== 3 passed in 0.02 seconds ===========================
py27 docker: remove '72e2ffea02' (forced)

Tox-Docker has picked up your configuration, and pulled and started a PostgreSQL container, which it shuts down after the tests finish. This is equivalent to running docker pull, docker run, and docker rm yourself, but without the manual hassle.

Challenges and Helpers

Not every Dockerized component can be started and be expected to "just work". Most services will require or allow for some amount of configuration, and your tests will need some information back out of Docker to know how to use the services. In particular, we need:

  1. A way to pass settings into Docker containers, in cases where the defaults are not sufficient
  2. A way to inform the tests how to communicate with the service inside the container, specifically, what ports are exposed
  3. A way to delay beginning the test run until the container has started and the application within it is ready to work

Tox-Docker lets you specify environment variables with the dockerenv setting in the tox.ini file:

[testenv]
docker = postgres:9.6
dockerenv =
    POSTGRES_USER=user_name
    POSTGRES_DB=database_name

Tox-Docker takes these variables and passes them to Docker as it launches the container, just as you might do manually with the --env flag. These variables are also made available in the environment that tests run in, so they can be used to construct a connection string, for instance.

Additionally, Tox-Docker interrogates the container just after it's started, to get the list of exposed TCP or UDP ports. For each port, Tox-Docker constructs an environment variable named after the container and exposed port, whose value is the host-side port number that Docker has mapped the exposed port to. Postgres listens on TCP port 5432 within the container, which might be mapped to port 32187 on your host system. In this case, an environment variable POSTGRES_5432_TCP will be set with value "32187".

Tests can use these environment variables to parameterize connections to the Dockerized services, rather than having to hard-code knowledge of the environment.

Finally, in order to avoid false-negative test failures or errors, Tox-Docker waits until it can connect to each of the ports exposed by the Docker container. This is not a perfect way to determine that the service inside is actually ready, but Docker provides no way for a service inside the container to signal to the outside world that it's finished starting up. In practice, I hope that this heuristic is good enough.

Future Work

The most obvious next step for Tox-Docker is to support Docker Compose, the Docker tool that lets you launch a cluster of interconnected containers in a single command. For the projects I am working with, I haven't yet had need of Docker Compose, but for projects of a certain level of complexity this will be preferable to attempting to manually manage this in tox.ini.

Installation and Feedback

Tox-Docker is available in the Python Package Index for installation via pip install tox-docker. Contributions, suggestions, questions, and feedback are welcome via GitHub.