Sophisticated Google container structure tests

Last week we did an innovation week at our company, crowding up together and trying to figure out what can be done to improve our systems. Our group chose to setup a private docker registry and to automate the creation of docker images for our test-system. After some research we came up with Google's framework named container structure tests, to verify that the automatically created containers are actually working.

The Container Structure Tests provide a powerful framework to validate the structure of a container image. These tests can be used to check the output of commands in an image, as well as verify metadata and contents of the filesystem.

GoogleContainerTools @ Github

The way it is always, you can create very simple test scenarios very fast, but there is few documentation when it comes to more complicated stuff. With this post I want to sum up the pitfalls you might encounter and offer solutions, so you can get the most out of the framework. If you need to know the basic stuff first jump over to the read-me and come back later ;)

Pitfalls and solutions

Image entry-points are removed by default

Every docker container comes with an entry-point defining what it should do on startup. These can influence the structure of the container or consume a lot of time, so they are removed by default. In our case we needed the entry-point, since we wanted to test whether our PostgreSQL container is working properly. What you should do (according to the docs) is using the setup section of the test like so:

commandTests:
  - name: "postgres starts without errors"
    setup:
      - ["./docker-entrypoint.sh", "postgres"]
    command: "psql"
    args: ["-d", "db-name", "-U", "db-user", "-c", "\q"]

This small test should start a new container, run the entrypoint script for postgres and finally check that we can connect to a database without any error. The exit code is expected to be zero by default. Sadly, this is not how it actually works as you will see in the next section.

Every command runs in a separate container instance

The setup section and the teardown section as well, are a list of commands, whereas the command section is just a single command. All of these commands run in their own separate container and then commit a modified image to be the new base image for the next command in the list. Since in our postgres example the entrypoint is starting a database in a setup command, this database will be running in this command's container only. This leads to the need of multiple commands in the same container, which we can't accomplish using the setup section.

Multi-line commands

We can trick the framework to run multiple commands in the same container using bash -c <list of commands>. Since this can get convoluted pretty fast, we can make use of YAML's "literal style" option (the | sign) to preserve newlines.

  - name: "postgres starts without errors"
    command: "bash"
    args:
      - -c
      - |
          bash -c './docker-entrypoint.sh postgres &' &&
          sleep 10 &&
          psql -d "db-name" -U "db-user" -c '\q'

This is the final (and actually working) version of the same test. As you can see, we are now running the docker-entrypoint script in the same container like the psql command. But since the script is starting a database instance we had to wrap it up in a second bash -c command so we could detach it from the console output with the ampersand (&) at the end. Furthermore we had to add some sleep time to give the database a chance to come up before we check if it is working.

Switching user profiles

Sometimes it might be necessary to run a command as a different user than root. As user postgres for example :D Fortunately, this can be accomplished similar to bash -c using su -c like that:

  - name: "run a command as a different user"
    command: "su"
    args:
      - postgres
      - -c
      - whoami
    expectedOutput: ["postgres"]

Alrighty, that's all I wanted to share for now. I hope the post will spare some time of yours. Please keep in mind that Google's framework container structure tests has been made to verify the structure and general setup of your container. It is not meant to be used for something like integration tests. Thank's for reading and have a nice day :)

Greets, Domi