Tox-Docker began its life because I needed to test some code that uses the PostgreSQL JSONB column type. In another life, I might have done this by instructing Jenkins to first install Postgres, start the server, create a user, create a database, and so on. There's even a small chance that this would work well, some of the time -- so long as tests didn't fail, the build didn't die unexpectedly without cleaning up, multiple tests didn't run at once, and so on. In fact, I've done exactly this sort of hackery a few times in the past already. It is dirty, and often requires manual cleanup after failures.
So, when confronted with the need to write tests that talk to a real Postgres instance, rather than reaching into my old toolbox I was determined to find a better solution. Docker can run multiple instances of Postgres at the same time in isolation from one another, which obviates the need for mutual exclusion of builds. Docker containers are lightweight to start, and easy to clean up (you can delete them all at once with a single command), so when the tests are done, we can simply remove it and move on.
There was still the question of how to manage the lifecycle of the container, though, which is where Tox comes in. Tox is a test automation tool that standardizes an interface between your machine (or your continuous integration environment) and your test code. Like Docker, Tox encourages isolation, by creating a clean virtualenv for each test run, free of old package installs, custom hacks, and so on. Tox already has a well-defined set of steps it runs to build your package, install dependencies, start tests, and gather results. Happily, it allows plugins to hook into this sequence to add custom behavior.
How Tox-Docker Works
Tox's plugins implement callback hooks to participate in the test workflow. For Tox-Docker, we use the pre-test and post-test hooks, which set up and tear down our Docker environment, respectively. Importantly, the post-test hook runs regardless of whether the tests passed, failed, or errored, ensuring that we'll have an opportunity to clean up any Docker containers we started during the pre-test hook. Finally, Tox plugins can also hook into the configuration system, so that projects using Tox-Docker can specify what Docker containers they require.
The simplest use of Tox-Docker is to specify the Docker image or images,
including version tags, that are required during test runs. For instance, if
your project requires Postgres, you might add this to your
[testenv] docker = postgres:9.6
With Tox-Docker installed, the next time you run
tox, you will see
something like the following:
py27 docker: pull 'postgres:9.6' py27 docker: run 'postgres:9.6' py27 runtests: PYTHONHASHSEED='944551639' py27 runtests: commands | py.test test_your_project.py ============================= test session starts ============================== platform linux2 -- Python 2.7.12, pytest-3.0.7, py-1.4.33, pluggy-0.4.0 rootdir: /home/travis/build/you/your-project, inifile: collected 3 items test_your_project.py ... =========================== 3 passed in 0.02 seconds =========================== py27 docker: remove '72e2ffea02' (forced)
Tox-Docker has picked up your configuration, and pulled and started a
PostgreSQL container, which it shuts down after the tests finish. This is
equivalent to running
docker run, and
yourself, but without the manual hassle.
Challenges and Helpers
Not every Dockerized component can be started and be expected to "just work". Most services will require or allow for some amount of configuration, and your tests will need some information back out of Docker to know how to use the services. In particular, we need:
- A way to pass settings into Docker containers, in cases where the defaults are not sufficient
- A way to inform the tests how to communicate with the service inside the container, specifically, what ports are exposed
- A way to delay beginning the test run until the container has started and the application within it is ready to work
Tox-Docker lets you specify environment variables with the
setting in the
[testenv] docker = postgres:9.6 dockerenv = POSTGRES_USER=user_name POSTGRES_DB=database_name
Tox-Docker takes these variables and passes them to Docker as it launches
the container, just as you might do manually with the
--env flag. These
variables are also made available in the environment that tests run in, so
they can be used to construct a connection string, for instance.
Additionally, Tox-Docker interrogates the container just after it's started,
to get the list of exposed TCP or UDP ports. For each port, Tox-Docker
constructs an environment variable named after the container and exposed
port, whose value is the host-side port number that Docker has mapped the
exposed port to. Postgres listens on TCP port 5432 within the container,
which might be mapped to port 32187 on your host system. In this case, an
POSTGRES_5432_TCP will be set with value "32187".
Tests can use these environment variables to parameterize connections to the Dockerized services, rather than having to hard-code knowledge of the environment.
Finally, in order to avoid false-negative test failures or errors, Tox-Docker waits until it can connect to each of the ports exposed by the Docker container. This is not a perfect way to determine that the service inside is actually ready, but Docker provides no way for a service inside the container to signal to the outside world that it's finished starting up. In practice, I hope that this heuristic is good enough.
The most obvious next step for Tox-Docker is to support Docker
Compose, the Docker tool that lets you
launch a cluster of interconnected containers in a single command. For the
projects I am working with, I haven't yet had need of Docker Compose, but
for projects of a certain level of complexity this will be preferable to
attempting to manually manage this in