tldr In this article I show how to use Docker Compose to pick and choose services and branches for any local or QA environment. I also showcase the branch aware build pipeline introduced in Jenkins 2.0.
In the first part of the series I addressed three out of four requirements
The last outstanding requirement makes the workflow so exciting to me, the ability to provision an environment with a handpicked set of components and branches unlocks productivity to a great extent.
But first we have to make sure that all branches get built automatically.
Besides the Jenkinsfile, Jenkins has another new feature in 2.0, it is now aware of branches in the source code repository. This nifty feature spares us a ton of boilerplate work as we don’t have to set up new instances of the build pipeline for every branch ourselves.
By picking Multibranch Pipeline on the job creation page we get the same behavior as the Pipeline job, plus Jenkins automatically creates a sub-project for each branch with a Jenkinsfile that it finds in the repository.
We also got a new item on the side bar to reindex branches on demand. It is able to detect new branches and delete pipelines that belong to deleted ones.
I made one more modification to the pipeline to make it practically useful. In this new final step I push the built Docker container image to a registry. With this step the container becomes available to other processes for verification and deployment.
At this point components are built continuously on every branch. As a next step I describe their interrelatedness with Docker Compose and handle them as a logical unit to bring up local and later remote environments.
Docker Compose has a simple yaml syntax, and while it has powerful options, it is very easy to define simple stacks.
version: '2' services: web-python: image: laszlocph/composetest ports: - "5000:5000" depends_on: - redis boot: image: laszlocph/spring-boot-dummy redis: image: redis
Above I defined three services,
Running this stack takes nothing more than executing the docker-compose up command. We don’t have to fiddle with individual docker run commands as it will start all containers in the right order, with the right volumes and exposed ports just as they are described in the docker-compose file.
Furthermore it defines a Docker network, so the services are able to communicate with each other in separation to other network traffic. They can do that by simply mentioning each other’s name as Compose puts an entry in the containers host file with all available services and their IPs.
Once the services are up you can verify the containers with the docker ps command as they are completely regular containers. Docker Compose seamlessly integrates with other Docker tools.
laszlo@~/multi-env: docker-compose up Creating network "multienv_default" with the default driver Creating multienv_boot_1 Creating multienv_redis_1 Creating multienv_web-python_1 Attaching to multienv_boot_1, multienv_redis_1, multienv_web-python_1 boot_1 | boot_1 | . ____ _ __ _ _ boot_1 | /\\ / ___'_ __ _ _(_)_ __ __ _ \ \ \ \ boot_1 | ( ( )\___ | '_ | '_| | '_ \/ _` | \ \ \ \ boot_1 | \\/ ___)| |_)| | | | | || (_| | ) ) ) ) boot_1 | ' |____| .__|_| |_|_| |_\__, | / / / / boot_1 | =========|_|==============|___/=/_/_/_/ boot_1 | :: Spring Boot :: (v1.4.0.RELEASE) boot_1 | redis_1 | 1:C 12 Oct 09:33:23.093 # Warning: no config file specified, using the default config. In order to specify a config file use redis-server /path/to/redis.conf redis_1 | _._ redis_1 | _.-``__ ''-._ redis_1 | _.-`` `. `_. ''-._ Redis 3.2.4 (00000000/0) 64 bit redis_1 | .-`` .-```. ```\/ _.,_ ''-._ redis_1 | ( ' , .-` | `, ) Running in standalone mode redis_1 | |`-._`-...-` __...-.``-._|'` _.-'| Port: 6379 redis_1 | | `-._ `._ / _.-' | PID: 1 redis_1 | `-._ `-._ `-./ _.-' _.-' redis_1 | |`-._`-._ `-.__.-' _.-'_.-'| redis_1 | | `-._`-._ _.-'_.-' | http://redis.io redis_1 | `-._ `-._`-.__.-'_.-' _.-' redis_1 | |`-._`-._ `-.__.-' _.-'_.-'| redis_1 | | `-._`-._ _.-'_.-' | redis_1 | `-._ `-._`-.__.-'_.-' _.-' redis_1 | `-._ `-.__.-' _.-' redis_1 | `-._ _.-' redis_1 | `-.__.-' redis_1 | ... web-python_1 | * Running on http://0.0.0.0:5000/ (Press CTRL+C to quit) web-python_1 | * Restarting with stat web-python_1 | * Debugger is active! web-python_1 | * Debugger pin code: 238-470-859
You can get the true power of Docker Compose from its CLI reference, I only want to highlight a few commands that can get you very far:
If you version your compose file together with the source code it guarantees that anyone in the team can run the stack locally, but because Docker Compose integrates with other Docker tools you can also point it at a remote machine, or a cluster of Docker Engines. You can start up a QA environment or even a production stack just as easy as running one locally. And that has huge value for me.
Imagine when a new colleague joins the team and she has a local environment up and running within fifteen minutes. Or when she has her first feature ready to showcase, she can share the QA environment with the team with a single command. Now this is what I call proper onboarding.
First, let’s create a remote environment with an other Docker tool. Docker Machine allows me to provision easily a VM with Docker Engine installed, be that locally with VirtualBox or remotely on Amazon AWS or Azure.
docker-machine create --driver virtualbox composeHost
Docker Machine - just like Vagrant - pulls in an image and launches a VM in Virtualbox and also provides convenience methods to operate the VM.
Notable commands are:
Once the new node is running I can point my docker executable to it by running
eval $(docker-machine env composeHost)
From this moment on, every docker command I run is not executed locally, but on the remote host. This may look magical, but what happens in the background is that it sets a few environment variables that is regarded by the docker executable.
laszlo@~: docker-machine env composeHost export DOCKER_TLS_VERIFY="1" export DOCKER_HOST="tcp://192.168.99.100:2376" export DOCKER_CERT_PATH="/home/laszlo/.docker/machine/machines/composeHost" export DOCKER_MACHINE_NAME="composeHost" # Run this command to configure your shell: # eval $(docker-machine env composeHost)
While the above clarifies what happens, the outcome is still magical as the docker-compose command will also operate on the remote machine as it relies on standard docker commands. Providing the same interface for local and remote work for no additional cost.
Once I’m done with the remote work, or got confused which server I’m on, I can easily set the environment back to local by calling
eval $(docker-machine env -u)
At this point I can run my stack locally or remotely, but the static port binding prevents me from running multiple instances at the same time.
This is a real bummer since I’m working on multiple feature branches at the same time and I want to expose them in various stages of development. Furthermore there are typically not three but thirty components in my stack, making it a bit slow and resource intensive to spin up new stacks.
For this reason I built a little tool to handpick what services I need and what feature branch to include in an environment. It simply reads the compose file, then the component set can be defined interactively. It writes a new compose file what I can use later to spin up the environment.
The script also fetches the available branches for services where the image element is annotated with the Githup repository URL.
As shown on the above screenshot, I selected all services and picked the new-endpoint branch for the boot component. Then I took the prepared compose file and started a new environment on a remote host by using this script.
I was inspired by the Docker Compose and Machine commands so I also introduced a few convenience scripts myself.
I hope you are as pumped about the screen above as I am. While the script can be made web based, more robust, etc, I achieved what I set out to do.
For me Docker lived up to the promise. Especially as the rabbit hole was never too deep. During my research, problems were easy to resolve after half an hour of searching and making sensible choices. Docker is here and can provide real productivity improvements.
I also reached the bleeding edge of the ecosystem. While Docker Compose integrates with all tools, this does not stand for the clustering improvements introduced in version 1.12. More specifically, Compose can be pointed to a cluster and run distributed, but only with Docker Swarm, not with the newly introduced Docker Engine in swarm mode.
Important to highlight that the two are not the same.
There are plenty of next steps ahead of me, clustering is a big topic. My goal is to find a nice way to run production stacks with similar ease as the workflow above. I will also peak into competing platforms and investigate connecting topics.
NEW Part three is out Docker cluster management on AWS