In this section, I will describe how I manage environment variables. I will explain this for a particular use-case: a deploy environment based on docker-compose. However, the same ideas apply when you are using dev or prod, or when you are not using docker-compose.
We will jump straight in and look at the compile-env tool that can generate .env files from a single source of truth. Then, we will look how the output of compile-env is used in the dynamic use-case (where variables are injected into a system) and static use-case (where variables are read from a file by the software that is run on the system).
The compile-env
tool generates .env files from a single source of truth.
There are different reasons for generating .env files, rather them writing them by hand:
postgres.env
, django.env
, aws.env
), and to distribute these values to the service-specific .env files that need them.SERVICE_VERSION=backend-2.1
than to see SERVICE_VERSION=${SERVICE_NAME}-${VERSION}
.
A compilation step can take care of this type of value interpolation. The generated .env files can be used by the system and read by humans
who are trying to understand the system.The compile-env
tool can be installed with pip install compile-env
.
For our use-case, the compile-env configuration file looks like this:
global_dependencies: [django.env, postgres.env]
outputs:
../../backend/.env/deploy.injected.env:
targets: [../../backend/.env/deploy.injected.env.in]
dependencies: [secrets.env]
../../backend/.env/deploy.env:
targets: [../../backend/.env/deploy.env.in]
../../postgres/.env/deploy.injected.env:
targets: [../../postgres/.env/deploy.injected.env.in]
To run the tool we can call cd ./env/deploy && compile-env compile-env.config.yaml
. This will generate three output .env files.
All of them make use of the environment variables in django.env
and postgres.env
. The deploy.injected.env
output file also depends on secrets.env
.
Every output file has a related template. For deploy.injected.env
the template is deploy.injected.env.in
.
Templates look like normal .env files, except that they interpolate any values that appear on the right hand.
# This file is automatically generated from src/env/deploy/compile-env.config.yaml
#
# secrets: allowed
# git : no
DJANGO_DATABASE_PASSWORD=${DJANGO_DATABASE_PASSWORD}
Note that in this snippet, the DJANGO_DATABASE_PASSWORD
value comes from the django.env
file
that was specified in the global_dependencies
key of compile-env.config.yaml
.
In compile-env.config.yaml
we've specified that the deploy.injected.env
output file
uses secrets.env
. I recommend to always include the word "secret" in any file that contains secrets.
To protect this file, one can use git-secret
(a great tool) or keep it out of git
altogether.
From the perspective of a dev-ops person, the variables that are injected into the system are dynamic
(the dev-ops can change them), and the variables that are read from source files are static
(dev-ops will not change these). In our use-case, both the dynamic and the static variables
are stored in files that are generated with compile-env
. When the dynamic variables are not stored in a file,
some adjustments will be needed (so that the output of compile-env
can be used as dynamic variables),
but this is outside of the scope of this article.
The dynamic case in our example is based on docker-compose. Variables are injected into the system
by setting the env_file
property of a docker-compose file:
version: '3.7'
services:
backend:
env_file: [./backend/.env/deploy.injected.env]
image: todoapp_backend_deploy
As you can see, the variables are injected by reading from the deploy.injected.env
file.
Don't be confused by the fact that we are reading from a file, which gives the impression that we are dealing with static data.
This is a technicality. When a dev-ops person looks at docker-compose.deploy.yml
then they will understand that the variables in deploy.injected.env
are injected into the container, and they will determine a mechanism to do the
same in production (and probably this mechanism is not based on reading a file).
The deploy.injected.env
file (which was generated from the backend/.env/deploy.injected.env.in
template) looks like this:
# This file is automatically generated from src/env/deploy/compile-env.config.yaml
#
# secrets: allowed
# git : no
DJANGO_DATABASE_PASSWORD=variety-native-enter-boat-loss
The static use-case - where variables are read from a file - looks like this:
# This file is automatically generated from src/env/deploy/compile-env.config.yaml
#
# secrets: not allowed
# git : yes
DJANGO_SETTINGS_MODULE=app.settings.deploy
In the static use-case, the .env file is just another configuration file. It cannot be tweaked (by dev-ops) like the injected variables can,
but it offers a central place where some of the key parameters of the system are determined. The fact that variables are
read from a file (rather than being injected) makes life easier for dev-ops people, because it means that they don't have to manage these
variables, as long as there is never any need to tune them later.
Note that the backend/.env/deploy.env
file documents the fact that this file may not contain secrets and therefore can be added to git.