Docker Compose with Remote Containers

My last post detailed developing inside containers with Visual Studio, making it possible to use existing docker images as a fully fledged development environment.

This post will dive a bit deeper, looking at how we can use Docker Compose to spin up multiple containers to support scenarios where we might want to make use of additional services or APIs.

Step 1: Move to Docker Compose

First, we will update our .devcontainer configuration to use docker-compose, instead of just a straight DockerFile. There are three components to this:

DockerFile

In this case the docker file defines the container inside which we will do our development; this remains unchanged in this simple example:

ARG VARIANT="14-buster"
FROM mcr.microsoft.com/vscode/devcontainers/javascript-node:0-${VARIANT}
devcontainer.json

This file controls how Visual Studio Code will handle remote containers for development. It is slightly different to the previous example as it will refer to a docker-compose file:

{
	"name": "Node.js",
	"dockerComposeFile": "./docker-compose.yml",
	"service": "app",
	"workspaceFolder": "/workspace",

	// Set *default* container specific settings.json values on container create.
	"settings": { 
		"terminal.integrated.shell.linux": "/bin/bash"
	},

	// Add the IDs of extensions you want installed when the container is created.
	"extensions": [
		"dbaeumer.vscode-eslint"
	],
	"remoteUser": "node"
}

Some of the important items to note:

  • dockerComposeFile: path to the docker-compose file
  • service: name of the container from the docker-compose file which will be used as the dev container
  • remoteUser: required when container is configured with non-root user
docker-compose.yaml

Finally, the docker-compose.yaml file defines the containers and services we want to spin up. To start this is just replicating the dev container:

version: '3'

services:
  app:
    build: 
      context: .
      dockerfile: Dockerfile
      args:
        VARIANT: 14

    volumes:
      - ..:/workspace:cached

    # Overrides default command so things don't shut down after the process ends.
    command: sleep infinity

    # Use a non-root user for all processes.
    user: node

With these elements in place, it should be possible to execute a container rebuild:

Step 2: Add Additional Services

Now that we are using docker compose, we can simply update docker-compose.yaml to include the additional containers we want, like this:

version: '3'

services:
  app:
    build: 
      context: .
      dockerfile: Dockerfile
      args:
        VARIANT: 14

    volumes:
      - ..:/workspace:cached

    # Overrides default command so things don't shut down after the process ends.
    command: sleep infinity

    # Runs app on the same network as the database container, allows "forwardPorts" in devcontainer.json function.
    network_mode: service:deepstack-ai

    # Use a non-root user for all processes.
    user: node

  deepstack-ai:
    image: deepquestai/deepstack:latest
    volumes:
      - localstorage:/datastore
    environment:
      - VISION-DETECTION=True

volumes:
  localstorage:

After executing another rebuild, you should see VS Code pull down the appropriate images, and spin everything up, leaving you with both containers running and ready to use:

Summary

Docker-compose can be used with Visual Studio Code’s remote containers to make it possible to spin up multiple containers as needed. This is useful when want to build on existing services, such as a database or AI API etc.

Developing in Containers with Visual Studio Code

Visual Studio Code now offers the ability to use a docker container as a fully fledged development environment with the introduction of the Remote Containers extension.

Workspace files are made accessible from inside a container which can also host the tools relevant to the development environment, leaving VS Code acting as a remote UI to enable a ‘local quality’ development experience:

Container Architecture

The obvious benefit here is the ability to very rapidly spin up a development environment through the use of pre-existing containers which already provide all required components.

Starting Up

First thing to do is create the config files that will tell VS Code how to configure the environment; this can be done by executing ‘Add Development Container Configuration Files’ (Ctrl + Shift + P):

This will create devcontainer.json and Dockerfile files under .devcontainer within the workspace.

The Dockerfile defines the container that Code will create and then connect to for use as a development environment. A bare bones Dockerfile for use with a Node app may look like this:

FROM node:slim
USER node

devcontainer.json defines how VS Code should work with a remote container. A simple example below shows how to reference the Dockerfile:

// For format details, see https://aka.ms/devcontainer.json. For config options, see the README at:
// https://github.com/microsoft/vscode-dev-containers/tree/v0.140.1/containers/typescript-node
{
	"name": "TriggerService",
	"build": {
		"dockerfile": "Dockerfile",
	},

	"settings": { 
		"terminal.integrated.shell.linux": "/bin/bash"
	},

	"extensions": [
		"dbaeumer.vscode-eslint",
		"ms-vscode.vscode-typescript-tslint-plugin"
	],

	"remoteUser": "node"
}

With both of these files in place, VS Code will prompt to re-open in the container environment (or use the command palette to execute ‘Reopen in Container’):

Dev Container Progress Notification

Once started up, an indicator in the bottom left shows that VS Code is currently connected to a container:

Create a Simple App

At this point VS Code is now connected to the node:slim container as configured in the Dockerfile.

Because this image provides everything needed to start developing a Node application, we can start by using npm to install Express:

npm init -y
npm install express

Then create index.js under the src folder:

const express = require( "express" );
const app = express();
const port = 8080;

// define a route handler for the default home page
app.get( "/", ( req, res ) => {
    res.send( "Hello world!" );
} );

// start the Express server
app.listen( port, () => {
    console.log( `server started at http://localhost:${ port }` );
} );

Next we need to update the package.json file to set the main entry point and start command:

{
  "name": "test-app",
  "version": "1.0.0",
  "description": "",
  "main": "src/index.js",
  "scripts": {
    "start": "node .",
    "test": "echo \"Error: no test specified\" && exit 1"
  },
  "keywords": [],
  "author": "",
  "license": "ISC",
  "dependencies": {
    "express": "^4.17.1"
  }
}

Now executing the following command from the terminal will start up the application inside the container:

npm run start

The key thing to note here is that we stood up this simple Node app without ever having to actually install Node on our host system; everything was pulled down via the node:slim docker image.

At this point the application is exposed on port 8080, so can be accessed at http://localhost:8080.

What’s Next?

We have only covered enough here to get up and running, barely scratching the surface of what can be done with remote containers.

Next up, debugging from inside a container, and using docker compose to handle spinning up multiple containers.

Vagrant Provisioning with Ansible

Messing around with Vagrant again, this time using Ansible to automate configuration post deployment.

Ansible is billed as an automation platform which makes it easier to deploy systems and applications. It does this through a scripting framework which supports a wide range of functionality covering deployment and configuration.

Vagrant Config

To define which Ansible playbooks should be run, the vm.provision config can be used in a Vagrantfile:

  config.vm.define "vm1" do |vm1|  
vm1.vm.box = "centos/7"
vm1.vm.hostname = "vm1"
vm1.vm.network "private_network", ip: "192.168.10.10"

vm1.vm.provision "docker", type:"ansible" do |ansible|
ansible.playbook = "docker-playbook.yml"
end

vm1.vm.provision "kubernetes", type: "ansible" do |ansible|
ansible.playbook = "kube-playbook.yml"
end
end

A Simple Playbook

Ansible uses the concept of playbooks to define a set of repeatable deployment steps.

This example deploys Docker on a CentOS system (should be saved as docker-playbook.yml in same directory as the Vagrantfile):

- hosts: all
become: yes
tasks:
- name: install docker dependancies
yum:
name: "{{ packages }}"
vars:
packages:
- yum-utils
- device-mapper-persistent-data
- lvm2

- name: Add Docker repo
get_url:
url: https://download.docker.com/linux/centos/docker-ce.repo
dest: /etc/yum.repos.d/docker-ce.repo
become: yes

- name: install docker
yum:
name: "{{ packages }}"
vars:
packages:
- docker-ce
- docker-ce-cli
- containerd.io

- name: Start Docker service
service:
name: docker
state: started
enabled: yes

- name: Add user vagrant to docker group
user:
name: vagrant
groups: docker
append: yes

Running

Running ‘vagrant up’ will cause all configured provision entries to run:

$ vagrant up
Bringing machine 'vm1' up with 'virtualbox' provider…
<snip>
==> vm1: Configuring and enabling network interfaces…
==> vm1: Rsyncing folder: /home/rich/vagrant_storm/ => /vagrant
==> vm1: Running provisioner: file…
==> vm1: Running provisioner: shell…
vm1: Running: inline script
==> vm1: Running provisioner: docker (ansible)…
vm1: Running ansible-playbook…
PLAY [all] *
TASK [setup] ***
ok: [vm1]
TASK [install docker dependancies] ***
changed: [vm1]
TASK [Add Docker repo] *
changed: [vm1]
<etc...>

Provisioning Again

Provisioning can also be run independently once the VM(s) are up

vagrant provision

By naming each provision entry, it’s also possible to run a specific item:

$ vagrant provision --provision-with kubernetes
==> vm1: Running provisioner: kubernetes (ansible)…
vm1: Running ansible-playbook…
PLAY [all] *
TASK [setup] ***
ok: [vm1]
TASK [add Kubernetes' YUM repository]
ok: [vm1]
TASK [install kubernetes]
ok: [vm1]
TASK [start kubelet] ***
changed: [vm1]
PLAY RECAP *
vm1 : ok=4 changed=1 unreachable=0 failed=0

Summary

Making use of Ansible along with Vagrant provides a super-fast, repeatable way to bring up a system with a defined configuration.

This will form the basis of an automation system for end-to-end testing, which future posts will build on.

Vagrant Networking Basics

Vagrant supports three basic types of networking:

  • Port Forwarding / NAT
  • Private Network
  • Public Network

By default, no networking is enabled (outside of Vagrant’s internal management mechanism), so one of these must be configured in the Vagrantfile to make the VM accessible by network.

Port Forwarding / NAT

The most basic network configuration forwards traffic from the host machine to the guest VM only on specific ports. By default only TCP is forwarded; config looks like this:

Vagrant.configure("2") do |config|
  config.vm.network "forwarded_port", guest: 80, host: 8080
end

Vagrant will also detect configuration conflicts where the same port is in use multiple times, and will prevent deployment of such a config.

Private Network

Private networks provide host only access to the guest VM; that is the networking is not bridged, and will not be accessible outside of the host VM.

Config looks like this when assigning an address via DHCP:

Vagrant.configure("2") do |config|
  config.vm.network "private_network", type: "dhcp"
end

Public Network

Public networks provide access to the guest VM which is available externally to the host system. Depending on provider, this is achieved through bridging, making the guest VM as public as the host machine is.

By default, DHCP is used for assigning addresses; config looks like this:

Vagrant.configure("2") do |config|
  config.vm.network "public_network"
end

Static IPs

The ‘ip’ config setting can be used to assign specific IPs for both private and public networks:

config.vm.network "private_network", ip: "10.0.0.100"

Note that there appears to be a bug in Vagrant 1.9.1 that prevents static IPs being applied properly in some RHEL based images. A workaround is to force the interface to come up by adding an additional provisioning line in the Vargrantfile:

config.vm.provision "shell", inline: "ifup eth1", run: "always

More Reading

Vagrant provides many more options for networking, with some features varying by provider.

Networking docs can be found here for more detail.

Running a Battlesnake Competition

Battlesnake is a community run open source AI competition, pitting player implemented bots against each other in a multiplayer snake arena.

Bots are simple web-servers which must respond to a defined API, and so can be implemented in any language. A yearly competition brings people together from all over the world to compete in Victoria, Canada.

In my role as a development manager, we try to run events that push the team outside of their comfort zone a bit, whilst having a little fun. Previously I have successfully run a Vindinium day (a similar AI competition which sadly seems to have disappeared), and discovered Battlesnake whilst looking for a replacement.

Battlesnake is a great project, but has a couple of shortcomings when it comes to my particular use case:

  1. Bots are implemented as web servers, which would require opening ports in the company firewall in order to use the public server. Big no no from the IT department.
  2. As a team building excercise, it’s desirable to run a private competition so that the devs are competing which each other, instead of internet strangers.

Battlesnake_ui

To meet the above needs, I created Battlesnake_ui; a light-weight Battlesnake game server aimed at internal competitions of limited size.

It can be found on Github, instructions for use will follow in another post shortly.


Features:

  • Auto match start mode for match making during development / free play
  • Manual start mode to allow full control during competition
  • TV page for showing games on big screens
  • Admin UI for configuration
  • Simple single package deployment

The Plan

Plan for the day will be as follows:

  • Intro session to describe the game
  • Forming of teams
  • Free time to develop a bot. Free access to match making during this time for testing
  • Pizza
  • Competition time
    • League
    • Knockout
    • Longest snake
  • Prizes and close

This format has worked well in the past; expect a post-mortem post once the event finishes.


Todo Lists Are Evil; You Should Definitely Use Them

Todo lists are great! You write down everything you need to do, split it into sub sections / sub-tasks, make it look pretty, and gain a massive sense of achievement.

Look at how productive I’ve been! I’m so organised!

Except all that you have really achieved is putting some stuff in a list.

But You Should Really Use Them

Having said all that, I’m still a big fan of having an up to date todo list:

A Starting Point for the day

Assuming your todo list is up to date, it acts as a great reminder of ‘where was I?’. This makes it much faster to get back into the flow of things.

Categorise Priorities

The modern workplace is a constant battle against interruptions: email, instant messaging, desk drive-bys; they are all source of more work to do, so it’s important to categorise what’s important, rather than just jumping to the most recent request.

Keeping a categorised list (I use ‘now’, ‘next’ and ‘future’) means new requests can be slotted in as appropriate, thus maintaining some semblance of flow.

Know what’s next

I find a key part of remaining productive is lowering the barrier to getting started on something. Having the decision made by a well defined list of ‘what’s next’ keeps thinking to a minimum.

Review what’s been achieved

Motivation ebs and flows. A great motivator is being able to review just how much has been achieved (assuming that is you’ve managed to get something done).

Taking a moment to review both the breadth of achievement, as well as key milestones is a great way to remind yourself that you’re on the right track.

Just Do It

So yes, Todo lists can be evil as a source of false sense of achievement, but as an organisational tool I find them to be one of the easiest ways to improve my own productivity.

You don’t need any fancy tools to get started, just open notepad++ and start writing stuff down!