5.3. Django

  • Code versions using Git and uploaded to Bitbucket.

  • Requirements.txt created

  • Postgresql DB dump (not covered by Git)

  • Copy of local_settings.py because it is not covered by Git

  • Copy of database settings/password from the main server (probably in local_settings.py

  • Any user uploaded files

  • Conf files - Gunicorn, Nginx and Supervisor

  • SSL private key and ssl-bundle and ssl confirmation of ownership file

5.3.1. Existing Project

These instructions are for an existing project. If you have not started yet you need to follow New Project instructions.

Ensure you have followed either the Development Machine or Production Server instructions for creating the server and setting up Virtualenv, pip and Git.

To use an existing project:

cd XX_proj
source bin/activate
git clone git@bitbucket.org:USER/proj_name.git
cd XX_proj
pip install -r requirements.txt
# Set up local settings
python manage.py migrate
python manage.py createsuperuser
python manage.py collectstatic  # For production

5.3.2. New Project

These instructions are for creating a new Django project. Follow the Existing Project instructions if you already have one.

Start with a fresh Django project if you do not have one to download using git:

cd XX_proj
source bin/activate
pip install django gunicorn  # Confirm which one: psycopg2  psycopg2-binary
django-admin startproject XX_proj
cd XX_proj
pip freeze > requirements.txt
python manage.py startapp base

5.3.3. Project Settings

Warning

The settings section needs re-writing because my approach has changed.

A lot of the default project settings are okay but some tweaks need making to make version control and deployment easier and more secure.

5.3.3.1. Local Settings

Note

For Django, remove the secret key from settings.py and put it into local_settings.py. Just in case you make your project public in the future you need to minimise security risks.

Create a local_settings.py file for the secret key and database settings. Add code to the bottom of settings.py for importing the local settings:

try:
    from .local_settings import * # the fullstop is for Python 3
except ImportError:
    pass

Copy the secret key from settings to local settings then delete from settings. This is because settings will be committed to git but local settings won’t. It’s more secure this way.

5.3.3.2. Settings

In settings leave TZ as UTC for now. In the past, changing it to London caused some issues. It might be because the server sorts it but I’m not sure.

ALLOWED_HOSTS = ['www.DOMAIN.co.uk', 'localhost', '127.0.0.1']

Check installed apps are correct.

Check WSGI paths etc. are correct if copied from a previous project:

SITE_ID = 1

LOGIN_URL = '/accounts/login/'

CRISPY_TEMPLATE_PACK = 'bootstrap3'

AUTHENTICATION_BACKENDS = (
    # Needed to login by username in Django admin, regardless of `allauth`
    'django.contrib.auth.backends.ModelBackend',

    # `allauth` specific authentication methods, such as login by e-mail
    'allauth.account.auth_backends.AuthenticationBackend',
)

ACCOUNT_AUTHENTICATION_METHOD = 'email'
ACCOUNT_EMAIL_REQUIRED = True
ACCOUNT_UNIQUE_EMAIL = True
ACCOUNT_USERNAME_REQUIRED = False
ACCOUNT_EMAIL_CONFIRMATION_AUTHENTICATED_REDIRECT_URL = '/accounts/profile/'

NOTE: ADD OTHER SETTINGS WHEN I CAN COPY AND PASTE ACROSS

DEFAULT_FROM_EMAIL='no-reply@secretgifter.co.uk'

FILEBROWSER_VERSIONS_BASEDIR = '_fb-img-versions'

Database settings in local_settings:

DATABASES = {
    "default": {
        # Ends with "postgresql_psycopg2", "mysql", "sqlite3" or "oracle".
        "ENGINE": "django.db.backends.sqlite3",
        # DB name or path to database file if using sqlite3.
        "NAME": os.path.join(BASE_DIR, 'db.sqlite3'),
        # Not used with sqlite3.
        "USER": "",
        # Not used with sqlite3.
        "PASSWORD": "",
        # Set to empty string for localhost. Not used with sqlite3.
        "HOST": "",
        # Set to empty string for default. Not used with sqlite3.
        "PORT": "",
    }
}

File paths in local_settings.py for development:

STATIC_URL = '/static/'
STATIC_ROOT = '/home/stewart/proj_name/project/static'
MEDIA_URL = '/media/'
MEDIA_ROOT = '/home/stewart/proj_name/project/static/media'

5.3.3.3. Context Processors

This code is used by a context processor to show the code only in production so testing during the development phase does not affect the statistics:

GOOGLE_ANALYTICS_PROPERTY_ID = 'UA-XXXXX-X'
GOOGLE_ANALYTICS_DOMAIN = 'westraven.co.uk'

GOOGLE_SITE_VERIFICATION = 'XXXXXX'
BING_SITE_VERIFICATION = 'XXXXXXX'
FACEBOOK_APP_ID = ''
FACEBOOK_ADMIN_ID = ''

Template settings NOTE DIRS:

TEMPLATES = [
    {
        'BACKEND': 'django.template.backends.django.DjangoTemplates',
        'DIRS': [os.path.join(BASE_DIR, 'templates')],
        'APP_DIRS': True,
        'OPTIONS': {
            'context_processors': [
                'django.template.context_processors.debug',
                'django.template.context_processors.request',
                'django.contrib.auth.context_processors.auth',
                'django.contrib.messages.context_processors.messages',
                "wr_main.context_processors.google_analytics",
                "wr_main.context_processors.site_ownership_verification"
            ],
        },
    },
]

5.3.4. Update Django

https://docs.djangoproject.com/en/2.0/howto/upgrade-version/

It’s important to stay on top of upgrades so issues can be fixed gradually. Some installed apps do not work with the latest version straightaway:

pip install -U django==2.2.11

5.3.5. Django Tests

Creating and running tests for Django projects.

Test settings need creating and an import condition needs adding to the bottom of the settings:

if 'test' in sys.argv:
    try:
        from wr_proj.test_settings import *
    except ImportError:
        pass

You also need to create a test_settings.py file in the same place as the settings file giving the standard database code from the settings template so a sqlite3 database can be created and destroyed for the testing process.

5.3.5.1. General Notes

Django uses a standard Python library module called unittest. unittest is a testing framework which has elements such as TestCase that sub class it and is used to create an individual unit of testing which might be referred to as a scenario.

setUp() and tearDown() are functions that run at the beginning and end of the tests to create environment variables such as self.selenium = webdriver.Firefox() which can then be used in the tests e.g. self.selenium.get('%s%s' % (self.live_server_url, '/dashboard/account/login/')).

Warning

If your tests rely on database access such as creating or querying models, be sure to create your test classes as subclasses of django.test.TestCase rather than unittest.TestCase. https://docs.djangoproject.com/en/1.8/topics/testing/overview/

Note

Note that the test client is not intended to be a replacement for Selenium or other “in-browser” frameworks. Django’s test client has a different focus. In short:

  • Use Django’s test client to establish that the correct template is being rendered and that the template is passed the correct context data.

  • Use in-browser frameworks like Selenium to test rendered HTML and the behavior of Web pages, namely JavaScript functionality. Django also provides special support for those frameworks; see the section on LiveServerTestCase for more details.

A comprehensive test suite should use a combination of both test types.

https://docs.djangoproject.com/en/1.8/topics/testing/tools/

Once selenium is installed, it can be used to open a browser for testing. LiveServerTestCase will run runserver so you do not need two separate terminals open. It won’t however load static files which is why you use:

from django.contrib.staticfiles.testing import StaticLiveServerTestCase
from selenium.webdriver.firefox.webdriver import WebDriver

5.3.5.2. Testing

Testing kept giving a database creation error. It is because the user was created with no permission for creating databases. This was a problem specifically using postgres, not sqlite so I tried a few things to fix it:

sudo -u postgres psql -c 'ALTER USER "th-thinkingcomputingDB3" CREATEDB;'

Note

That code still did not fix the problem

As a temporary work around, I’ve got a check for ‘test’ being passed through and if it is, switching the settings to use an sqlite3 database because it runs in memory and more importantly it does not give an error. A lot of people say it is better to use sqlite for testing because it is quicker but some say it is better to test against postgresql because that is used in production and there might be some constraints or errors specific to it.

I reckon that for now the workaround is much better than no testing in place at all:

import sys

if 'test' in sys.argv:
    try:
        from test_settings import *
    except ImportError:
        pass

That code goes at the bottom of settings.py and imports from test_settings.py:

DATABASES = {
    "default": {
        # Ends with "postgresql_psycopg2", "mysql", "sqlite3" or "oracle".
        "ENGINE": "django.db.backends.sqlite3",
        # DB name or path to database file if using sqlite3.
        "NAME": 'db.sqlite3',
        # Not used with sqlite3.
        "USER": "",
        # Not used with sqlite3.
        "PASSWORD": "",
        # Set to empty string for localhost. Not used with sqlite3.
        "HOST": "localhost",
        # Set to empty string for default. Not used with sqlite3.
        "PORT": "",
    }
}

Note

A book I was following recommended having functional_test.py although others said if all my tests begin with test they will be auto discovered. Other places recommend having a separate folder for tests. For now, I just need to get some tests in place and worry about structuring them in the future.

I’m still not fully set regarding my testing structure but there seems to be a distinction between unit tests and functional tests. Unit tests are what should accompany each app because they test whether each of the parts of the code for that app work as expected. Functional tests written from the point of view of the user so there could be some functional tests associated with each specific app but there could also be tests that follow a particular path or scenario for different users interacting with the site.

Note

All tests need to begin with test_

5.3.5.3. Coverage

pip install coverage will install a tool supported by Django for checking whether all your code is covered by tests:

coverage run manage.py test myapp -v 2  # OR
coverage run --source='.' manage.py test secretgifter --settings secretgifter.settings.base
# Then:
coverage report # OR
coverage html

5.3.6. Celery and Redis

<https://realpython.com/asynchronous-tasks-with-django-and-celery/>

For starting Celery workers in production:

https://sodocumentation.net/django/topic/7091/running-celery-with-supervisor

5.3.7. Backup and Restore

Django offers a way of backing up the project via dumpdata. I use it in conjuction with the postgres solution.

5.3.7.1. Dumpdata

Standard dumpdata commands work if you’re backing up to the same database, otherwise you need to exclude a few things that will cause an integrity error:

python manage.py dumpdata --exclude auth.permission --exclude contenttypes > db.json
mv db.json /home/account/directory

Try these excludes instead for a wagtail site:

python manage.py dumpdata --natural-foreign --indent 2 \
    -e contenttypes -e auth.permission \
    -e wagtailcore.groupcollectionpermission \
    -e wagtailcore.grouppagepermission -e wagtailimages.rendition \
    -e sessions > data.json

https://saashammer.com/blog/how-export-restore-wagtail-site/

Use dump/load instead of pg_restore for wagtail.

Warning

This doesn’t back up the media files for anything uploaded. You need to run collectstatic then copy the media file in.

5.3.7.2. Media

Copy either manually or using rsync the media files from the server. These are files uploaded by users. Static files should already be stored as part of the project and can be reinstated using collectstatic.

5.3.7.3. Local

local.py needs backing up from the server and keeping very safe because it contains the password for connecting to the database etc. It should be on the gitignore list already. Do not store the backup in the main project structure to avoid it being committed to version control.

5.3.7.4. Loaddata

On the new server, ensure the postgres database is created using Restore and connection details are in local.py.

Run migrations to create the database then load the data:

python manage.py migrate
python manage.py loaddata db.json
python manage.py changepassword <superuser_name>

https://coderwall.com/p/mvsoyg/django-dumpdata-and-loaddata