Consistent Development environments using devcontainers

Background

Recently I was working on a python project with a team of 20-30 people. From past experience of working on python, I knew that I could use python virtual environments to avoid issues in local development. I created few scripts to ease the setup process which would create the python virtual environment and install all the other tools required for local development. Everything looked good in theory, but soon I realized that I was spending more and more time in help debugging and fixing issues in most of people’s local environments. The immediate next step I did was to create a documentation for the steup process. It helped but not too much, I was still spending time on fixing setup related issues.

One day, I was on call with another team and I noticed that they had a .devcontainer folder in 2 of their repositories. They told that its what they use to develop the project. Out of curiosity, I started reading about it and found that it could be the solution to all my setup related problems.

What are DevContainers?

DevContainer is a specification for using docker containers for development. Docker containers provide an isolated and consistent environment to run an application. In case of DevContainers, it allows you to define a consistent development environment across all your developers.

From their website:

A development container (or dev container for short) allows you to use a container as a full-featured development environment. It can be used to run an application, to separate tools, libraries, or runtimes needed for working with a codebase, and to aid in continuous integration and testing. Dev containers can be run locally or remotely, in a private or public cloud, in a variety of supporting tools and editors.

How it works behind the scenes ?

You start with defining few configs required to setup the development environment. These are stored in a .devcontainer folder at root of your project. devcontainer cli or any other implementation will read these config files and create a docker container as per the config/specification given. A small IDE connector service is installed onto this container which allows you to interact with devcontainer from your IDE. In case of VSCode, a VSCode server is installed on the container and your local VSCode instance connects to this server via Remote Connection feature.

Prerequisites for setting up DevContainers

To begin using devcontainers you need to have a docker engine running on your machine. I prefer to use Rancher Desktop because of its permissible license and compatibility with docker cli. You can install Docker Desktop or Colima as well.

The next thing which you need is a compatible IDE. I’ll be using VSCode for this example as devcontainers feature is available for free on VSCode.

On VSCode you will need following extensions to interact with devcontainers:

• Dev Containers: ms-vscode-remote.remote-containers
• Remote Explorer: ms-vscode.remote-explorer
• Remote - SSH: ms-vscode-remote.remote-ssh
• Remote - Tunnels: ms-vscode.remote-server

Setting up a DevContainer

Create the DevContainer

• Open the project or folder where you want to setup devcontainer in VSCode.
• Cmd + Shift + P to open the command palette and type Dev Containers: Add Dev Container Configuration Files
• Now VSCode will ask if you want to use a template to create your devcontainer or create a custom one. For my need I select Python3
• VSCode will ask if you want to install any features like docker-in-docker etc into your devcontainer
• Now VSCode asks if you want to configure additional options like dependabot.
• VSCode will create a .devcontainer folder and prompt you to start the devcontainer.
• Click on Reopen in Container to start the devcontainer.

Congratulations! You have successfully created a devcontainer for your project.

Customizing DevContainer for your needs

Above steps create a basic bare minimum devcontainer. You can customize it with various options. You can find the full list of supported options in devcontainer.json schema. We will be updating .devcontainer/devcontainer.json file to customize the behaviour of the container.

Installing Additional features

My python application was using testcontainers for runing the functional tests. For this reason I wanted to install docker inside my devcontainer. There are multiple ways via which I could get docker working inside my container.

I found that DevContainers allows you to install additional features like docker-in-docker or docker-outside-of-docker or Ngrok etc in a declarative fashion. You can find a list of available features here, additionally you can check GitHub for more such devcontainer features.

devcontainer features are like docker images which can provide additional scripts/configs to configure your devcontainer.

You need to add the feature to devcontainer.json file to install them. For example following snippet installs docker-outside-of-docker and postman:

"features": {
    "ghcr.io/devcontainers/features/docker-outside-of-docker:1": {},
    "ghcr.io/frntn/devcontainers-features/newman:latest": {
        "version": "5.3.2"
    }
}

Configuring VSCode

For my python application I need to install few extensions and configure some settings. There are multiple ways to configure your editor. After discovering features option in devcontainer, I configured the devcontainer.json file to install the required extensions and apply the standard settings. I added following customizations section to install few extensions for my python project and configure vscode settings to use black as the code formatter:

"customizations": {
  "vscode": {
    "extensions": [
      "ms-python.python",
      "ms-python.pylint",
      "ms-python.debugpy",
      "ms-python.vscode-pylance",
      "ms-python.black-formatter"
    ],
    "settings": {
      "[python]": {
        "editor.defaultFormatter": "ms-python.black-formatter",
        "editor.formatOnSave": true,
        "editor.codeActionsOnSave": {
          "source.organizeImports": "explicit"
        },
      },
      "black-formatter.args": [
        "--line-length",
        "100"
      ]
    }
  }
}

Port Forwarding

I needed to connect to my application running inside the devcontainer from my local machine. For this devcontainer also provides port forwarding. You can specify which ports you want to forward in devcontainer.json file.

"forwardPorts": [8000]

Above snippet will forward port 8000 from the container to 8000 of the host. BTW, VSCode provides a temporary port forwarding feature with devcontainer. Which can come handy if you forgot to forward ports.

Using docker compose files to run multiple services

My python project needed to connect to a Database service. To provide a consistent development environment, I again used another feature of devcontainer which is to use docker compose files to setup the devcontainer.

You can define either a single docker compose file or multiple docker compose files in dockerComposeFile section of the devcontainer.json file.

I created 2 files for ease of maintainance. One for the database service and another for the devcontainer.

"dockerComposeFile": ["docker-compose.yaml", "opensearch-compose.yaml"],
"service": "app", // the name of the service inside the docker compose file where you want to develop your application

Following is my application docker compose file:

services:
  app:
    container_name: my-project
    restart: unless-stopped
    image: mcr.microsoft.com/devcontainers/python:0-3.11
    volumes:
      - ../..:/opt:cached

    # Overrides default command so things don't shut down after the process ends.
    command: sleep infinity
    depends_on:
      opensearch:
        condition: service_healthy
    networks:
      my-project-net: # All of the containers will join the same Docker bridge network
        aliases:
          - my-app
    environment:
      - PYTHONUNBUFFERED=1

    # Use "forwardPorts" in **devcontainer.json** to forward an app port locally.
    # (Adding the "ports" property to this file will not forward from a Codespace.)
networks:
  my-project-net:
    name: my-project-net

Running custom scripts on various events

As you can see in above sections that my application needs OpenSearch but running OpenSearch on my local machine created many issues like it needed me to increase vm.max_map_count on the VM created by Rancher Desktop.

DevContainers has an option to run scripts on various events like initialize, postCreate, postStart, etc. I used initializeCommand to solve this issue on everybody’s workstation.

Also, I wanted to install the python dependencies and few other tools like iputils-ping, bind9-utils and dnsutils to on these containers. So I used postCreateCommand to install these tools.

Following snippet configures the VM of Rancher Desktop to increase the vm.max_map_count before creating the devcontainer and installs the python dependencies from requirements.txt and then runs a custom script located at ./.devcontainer/post-create.sh to configure the devcontainer:

"initializeCommand": "~/.rd/bin/rdctl shell sudo sysctl -w vm.max_map_count=262144",
"postCreateCommand": "pip install --user -r requirements.txt && ./.devcontainer/post-create.sh",

Reducing setup time

Everything was going good till One day I was having a bad network and rebuilding devcontainer was taking a lot of time. So I decided to check how I can optimize the setup process and cache few things during the process. So far I was using a docker image provided by Microsoft for VS Code Dev Containers. I knew that I can use it as a base image and then install my other tools on top of it. So I created a Dockerfile and pointed my docker-compose.yaml to it.

FROM mcr.microsoft.com/devcontainers/python:0-3.11-bullseye

ENV PYTHONUNBUFFERED 1

RUN sudo apt-get update && \
    sudo apt install vim iputils-ping bind9-utils dnsutils -y

This reduced the setup time by caching the OS image I was using. But still there was room to optimize it further. So I decided to use uv for my python project which helped me in caching the dependencies it has downloaded into a local cache. You can read about caching in uv here.

Troubleshooting

docker command not found

• Please make sure that you have a docker engine installed on your machine.
• Please make sure that you start the docker engine before you start the VSCode.
• In case you are using Rancher Desktop
  • Make sure Prefrence -> Application -> Environment is set to Automatic.
  • Make usre ~/.rd/bin is added to your PATH environment variable.

Docker login issue

unauthorized: your account must log in with a Personal Access Token (PAT) - learn more at docs.docker.com/go/access-tokens"

• You should either logout of public DockerHub account on your local machine or you should login back to public DockerHub to fix this error.
• You may see this kind of error for other docker registeries as well, you can follow similar steps for those as well.

Failed to create devcontainer

• Not enough space
  1. Make sure the VM started by your docker engine has enough space left.
  2. Make sure your host machine has enought space left.
• name conflicts - containers/networks/volumes etc
  1. Make sure there is no conflict between the container started by your devcontainer and other containers started via other processes
     • Run docker ps -a or docker network ls or docker volume ls command and make sure there are no conflicts.
     • If there are conflicts then either modify your devcontainer or remove the conflicting containers/networks/volumes.
• .devcontainer/post-create.sh: Permission denied
  1. Make sure ls and rdctl shell ls command gives you same output. If not then follow below steps:
     • Open System Settings
     • Navigate to Privacy & Security
       • Navigate to Files and Folders
       • Make sure you have given access to Rancher Desktop
     • Navigate to Privacy & Security
       • Navigate to Full Disk Access
       • Make sure you have given access to Rancher Desktop
  2. Make sure .devcontainer/post-create.sh has execute permissions.
  3. Make sure Owner is set correctly.

Frequent reconnects on macOS/Windows

You may face slowness while using devcontainers on macOS/Windows. This is because of the way docker works on these platforms. These platforms do not support docker natively, so the docker engine has to create a Linux VM for running the containers.

Also, the code is mounted as a volume which becomes slower due to the overheads of 2 layers of virtualization happening on these platforms.

The best solution is to use cached volumes or create volumes for code inside the VM created by docker engine.

You can read more on improving performance here.

It could be because of Extension host terminated unexpectedly as well. This could be caused by VSCode server and the VSCode app may experience compatibility issues. In that case you can try downgrading VSCode or the DevContainer extension. You can also try DevContainers: Rebuild Container Without Cache option it will fix any issue with VSCode server cached by docker.

Conclusion

It is a great tool to simplify the development process for bigger teams. It provides a consistent environment for developers to work on.

But DevContainers are not a perfect solution for everyone. For example few of the team members had old workstations which lead to slow down of development process due to the overhead added by devcontainers and docker. For them I still had to fall back to traditional way of setting up the project and I used settings.json and extensions.json file in .vscode folder to make sure that everyone has the same extensions.

Few times, We had to clean up all whole docker engine to fix the name conflicts.

Both Apple and Microsoft are working towards supporting docker natively. I would recommend using devcontainers if you have good workstations.

The devcontainer I use for python development is available for download from GitHub.