Buy me a coffeeBuy me a coffee

Configure a Django project to handle multiple environments, creating different settings and handling packages for each one.

Process

Current
graph TB subgraph Django Project conf["settings.py"] pack[packages] end conf==>dev[Development Environment] pack==>dev conf==>prod[Production Server] pack==>prod
Goal
graph LR subgraph Production Server prod_conf["production_settings.py"] prod_pack["Production Server Packages"] end subgraph Development Environment dev_conf["development_settings.py"] dev_pack["Developer Environment Packages"] end

Overview

Django projects should have several environments each with its own peculiarities. Each environment is optimized for a specific task, e.g.: develop your app, having packages that helps you test it, another with all the security measures for production, and so on.

In this guide we will split the standard settings file into specific settings optimized for each environment.

A common deployment architecture consists of a 4-tier architecture, having different settings for one of these tiers:

  1. development (DEV)
  2. testing (TEST)
  3. staging (STAGE)
  4. production (PROD)

A default Django app just starts with a single configuration file located in DJANGO_PROJECT/settings.py. This approach is fine for small projects but to fit in the above 4-tier architecture, the project needs to be changed to address two main problems:

  • each environment should have a specific settings file
  • each environment should have its own packages

The configuration file should be tracked in version control, even developers local configuration file, all the developers of a project should use the same development configuration.

However there are special config keys that should be left out of versioning, like the SECRET_KEY setting (used for cryptographic signing functionalities).

Settings

There are two main approaches to work with settings in different environments:

  1. Use environment variables
  2. split settings.py file into multiple files and use each one in each different environment

I prefer the first approach whenever possible because it leaves secret keys outside config files versioning using environment variables as recommended by http://12factor.net/config.

Splitting settings description

The default Config file that comes shipped with Django should be pulled apart into several settings for each environment: local, staging, test, production.

This can be done easily inheriting from a base config file, changing what the specific environment needs

Using different settings

To set SECRET_KEY and DJANGO_DEBUG variables in each environment add this to settings.py.

# Read SECRET_KEY from an environment variable
import os
SECRET_KEY = os.environ.get('DJANGO_SECRET_KEY', '[email protected];sljfk=$adsk;lhPUO*R*P(@HBBGG)g')
DEBUG = bool( os.environ.get('DJANGO_DEBUG', True) )

This will set a default value for the SECRET_KEY variable at low security environments like development and testing, then you will have to set DJANGO_SECRET_EKY and DJANGO_DEBUG variables in production.

Variables can be set with environment variables e.g.: export A_SECRET_KEY=foobar1234 $ export A_SECRET_KEY=shhh1234, and placed in:

  • .bashrc or .profile
  • virtualenvwrapper’s bin/activate hook file

If virtualenvwrapper is being used, the default development settings parameter to work with manage.py can be specified in the postactivate hook: echo "export DJANGO_SETTINGS_MODULE=settings.local" >> $VIRTUAL_ENV/bin/postactivate

When using manage.py many commands accepts the parameter to specify a specific settings file: python manage.py runserver --settings=myproject.settings.local

There is a handy manage.py parameter to compare the current settings file with the one that installs Django by default: $ manage.py diffsettings so you can see what you have changed from the original settings file so far.

Checking settings environment variables

To check that the secret environment key is being loaded, it is possible to check it from a python shell:


$ ./manage.py shell
Python 3.6.4 (default, Feb  5 2018, 16:52:44) 
[GCC 7.3.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
(InteractiveConsole)
>>> import os
>>> os.environ["A_SECRET_KEY"]
"shhh1234"

Then to get the value for a specific environment, the production config file in version control only needs to get this environment variable value: A_SECRET_KEY = os.environ["A_SECRET_KEY"]

In a production environment like Heroku, this can be done with: $ heroku config:set A_SECRET_KEY=shhh1234

Creating different settings files

When building a project, Django automatically creates a configuration file in <project_name>/settings.py.

To split this file into multiple settings files (one for each environment local, testing, staging and production), the best way is to create a base.py config with common settings across all of them and create specific config files for each environment in specific files having a structure like:

.
└── REPO-ROOT `git repo`
    β”œβ”€β”€ ...
    └── PROJECT-ROOT
        β”œβ”€β”€ settings
        |   β”œβ”€β”€ __init__.py
        |   β”œβ”€β”€ base.py
        |   └── local.py
        β”œβ”€β”€ manage.py
        └── ...

To build this structure:

  1. Create the settings directory

$ mkdir PROJECT_NAME/settings


  1. Add __init__.py file to make this directory a Python package

$ touch PROJECT_NAME/settings/__init__.py

  1. Move the default settings file into the new settings directory and change its name

$ mv PROJECT_NAME/settings.py PROJECT_NAME/settings/base.py


  1. Create all the configuration files (local.py, testing.py, staging.py, production.py ) and specify to inherit base.py configurations adding from .base import * to each file:

$ for name in local.py testing.py staging.py production.py; do echo "from .base import *" >> PROJECT_NAME/settings/${name} ; done

  1. Use the new settings file in one of previously described ways:
    • set an environment variable and call scripts normally
    • call scripts specifying a settings file

Set environment variables

Configure the current environment to use the appropriate settings file, using PYTHONPATH and DJANGO_SETTINGS_MODULE:

 
  
  $ export DJANGO_SETTINGS_MODULE=mysite.settings.local
  $ export PYTHONPATH=~/path/to/my/project
  #django-admin will use the above settings by default
  $ django-admin runserver
  
  

Use the --settings parameter with manage.py or django-admin:

 

$ django-admin runserver --settings=mysite.settings.local --pythonpath=/path/to/my/project
Performing system checks...

System check identified no issues (0 silenced).

July 23, 2016 - 22:43:48
Django version 1.9.6, using settings 'mysite.settings.local'
Starting development server at http://127.0.0.1:8000/
Quit the server with CONTROL-C.

Differences between manage.py and django-admin

Django has two administrative scripts: django_admin.py and manage.py (in the root of each Django project).

manage.py is a wrapper of django-admin, the only difference between them is that manage.py sets the DJANGO_SETTINGS_MODULE environment variable to settings.py by default, if not changed.

So with the about settings scheme, it is better to use django-admin and choose the proper settings file.

Generally, when working on a single Django project, it’s easier to use manage.py than django-admin. If you need to switch between multiple Django settings files, use django-admin with DJANGO_SETTINGS_MODULE or the –settings command line option.

Packages for each environment

Now a similar approach for handling package dependencies.

Each environment has to have a set of packages that fits its purpose and operating system requirements. We have to configure a python virtual environment so it is possible to install packages in development, that are not needed in production and viceversa, or that can be installed in different Operating Systems.

Each environment needs a specific file, having a base.txt requirement file with common packages across environments and then adding the needed packages for each environment.

.
└── REPO-ROOT `git repo`
    β”œβ”€β”€ .gitignore
    β”œβ”€β”€ requirements
    |   β”œβ”€β”€ base.txt
    |   β”œβ”€β”€ local.txt
    |   β”œβ”€β”€ production.txt
    |   └── test.txt
    β”œβ”€β”€ ...
    └── PROJECT-ROOT
        β”œβ”€β”€ manage.py
        └── ...

Using pip it is possible to specify which file has the list of packages you want:

pip install [options] -r <requirements file> [package-index-options] ...
  -r, --requirement <file>    Install from the given requirements file. This option can be used multiple times.

So to make it possible for each environment to inherit the packages from the base.txt requirement file using pip, each new file should begin with: -r base.txt:

/requirements/base.txt:

# list of packages present in all environments

/requirements/local.txt:

-r base.txt

/requirements/test.txt:

-r base.txt

/requirements/production.txt:

-r base.txt
  • To generate a requirements file: $ pip freeze $ pip freeze --local > requirements/base.txt
  • To install the requirements file
    • in a local/development environment: $ pip install -r requirements/local.txt
    • in a testing environment: $ pip install -r requirements/testing.txt

Conclusion

We started having a settings.py single file and break it up into a new directory with specific environment settings.

As with any major change to the default installation, after generating these directories, it is a good practice to describe them in /docs/architecture.rst and what are the commands used to get them running in /docs/installation.rst for other developers or just for oneself when reviewing the project in the future.

References

Summary

graph TB env["Set environment variables"] env==>split["Split settings.py into separated files
with custom settings for each environment"] split==>pack["Configure different Packages for each environment"]
Uruguay
Marcelo Canina
I'm Marcelo Canina, a developer from Uruguay. I build websites and web-based applications from the ground up and share what I learn here.