Introduction

In this section, I will describe how I manage environment variables. I will explain this for a particular use-case: a deploy environment based on docker-compose. However, the same ideas apply when you are using dev or prod, or when you are not using docker-compose.

We will jump straight in and look at the compile-env tool that can generate .env files from a single source of truth. Then, we will look how the output of compile-env is used in the dynamic use-case (where variables are injected into a system) and static use-case (where variables are read from a file by the software that is run on the system).

Generating .env files using compile-env

The compile-env tool generates .env files from a single source of truth. There are different reasons for generating .env files, rather them writing them by hand:

• the same values may appear in multiple .env files. Ideally though, we'd define these values just once.
• the .env file for a service will usually contain a mixture of variables that are related to different concerns (e.g. postgres, django, aws). To keep things organized, it helps to group all values related to a concern in a single file (e.g. postgres.env, django.env, aws.env), and to distribute these values to the service-specific .env files that need them.
• .env files are easier to grok if they have concrete values, rather than expressions. For example, for understanding the system it's more useful to see SERVICE_VERSION=backend-2.1 than to see SERVICE_VERSION=${SERVICE_NAME}-${VERSION}. A compilation step can take care of this type of value interpolation. The generated .env files can be used by the system and read by humans who are trying to understand the system.

Installation

The compile-env tool can be installed with pip install compile-env.

The compile-env config file

For our use-case, the compile-env configuration file looks like this:

global_dependencies: [django.env, postgres.env]

outputs:
  ../../backend/.env/deploy.injected.env:
    targets: [../../backend/.env/deploy.injected.env.in]
    dependencies: [secrets.env]
  ../../backend/.env/deploy.env:
    targets: [../../backend/.env/deploy.env.in]
  ../../postgres/.env/deploy.injected.env:
    targets: [../../postgres/.env/deploy.injected.env.in]

To run the tool we can call cd ./env/deploy && compile-env compile-env.config.yaml. This will generate three output .env files. All of them make use of the environment variables in django.env and postgres.env. The deploy.injected.env output file also depends on secrets.env.

The .env template file

Every output file has a related template. For deploy.injected.env the template is deploy.injected.env.in. Templates look like normal .env files, except that they interpolate any values that appear on the right hand.

# This file is automatically generated from src/env/deploy/compile-env.config.yaml

#

# secrets: allowed

# git : no

DJANGO_DATABASE_PASSWORD=${DJANGO_DATABASE_PASSWORD}

Note that in this snippet, the DJANGO_DATABASE_PASSWORD value comes from the django.env file that was specified in the global_dependencies key of compile-env.config.yaml.

Managing secrets

In compile-env.config.yaml we've specified that the deploy.injected.env output file uses secrets.env. I recommend to always include the word "secret" in any file that contains secrets. To protect this file, one can use git-secret (a great tool) or keep it out of git altogether.

The dynamic and static use-cases

From the perspective of a dev-ops person, the variables that are injected into the system are dynamic (the dev-ops can change them), and the variables that are read from source files are static (dev-ops will not change these). In our use-case, both the dynamic and the static variables are stored in files that are generated with compile-env. When the dynamic variables are not stored in a file, some adjustments will be needed (so that the output of compile-env can be used as dynamic variables), but this is outside of the scope of this article.

The dynamic case: injecting variables into the deploy environment

The dynamic case in our example is based on docker-compose. Variables are injected into the system by setting the env_file property of a docker-compose file:

version: '3.7'
services:
  backend:
    env_file: [./backend/.env/deploy.injected.env]
    image: todoapp_backend_deploy

As you can see, the variables are injected by reading from the deploy.injected.env file. Don't be confused by the fact that we are reading from a file, which gives the impression that we are dealing with static data. This is a technicality. When a dev-ops person looks at docker-compose.deploy.yml then they will understand that the variables in deploy.injected.env are injected into the container, and they will determine a mechanism to do the same in production (and probably this mechanism is not based on reading a file).

The deploy.injected.env file (which was generated from the backend/.env/deploy.injected.env.in template) looks like this:

# This file is automatically generated from src/env/deploy/compile-env.config.yaml
#
# secrets: allowed
# git    : no

DJANGO_DATABASE_PASSWORD=variety-native-enter-boat-loss

The static use-case: reading from a .env file

The static use-case - where variables are read from a file - looks like this:

# This file is automatically generated from src/env/deploy/compile-env.config.yaml
#
# secrets: not allowed
# git    : yes

DJANGO_SETTINGS_MODULE=app.settings.deploy

In the static use-case, the .env file is just another configuration file. It cannot be tweaked (by dev-ops) like the injected variables can, but it offers a central place where some of the key parameters of the system are determined. The fact that variables are read from a file (rather than being injected) makes life easier for dev-ops people, because it means that they don't have to manage these variables, as long as there is never any need to tune them later. Note that the backend/.env/deploy.env file documents the fact that this file may not contain secrets and therefore can be added to git.