Cache Django Website Using Redis

The performance of your application is critical to the success of your product. The repercussions of a slow application can be measured in dollars and cents in an environment where users anticipate website response times of less than a second. Even if you aren’t selling anything, quick page loading improve the visitor experience.

Everything that happens on the server between the time it gets a request and the time it provides a response adds to the time it takes for a page to load. As a general rule, the more server-side operations you can reduce, the faster your application will run. One technique to ease server stress is to cache data after it has been processed and then serve it from the cache the next time it is requested. In this lesson, we’ll look at some of the variables that slow down your application and show you how to use Redis caching to mitigate their effects.

What is Caching?

The technique of keeping copies of files in a cache, or temporary storage area, so that they can be retrieved more rapidly, is known as caching. A cache is a temporary storage facility for copies of files or data, however the phrase is most commonly associated with Internet technologies.

What is Redis?

Redis is an in-memory data structure store with caching capabilities. Redis can provide data quickly since it keeps data in RAM. We don’t have to limit ourselves to Redis for caching. Another popular in-memory caching system is Memcached, but many people believe that Redis is preferable to Memcached in most situations. I like how simple it is to set up and use Redis for other things, like Redis Queue.

Getting Started

We have created an example application to introduce you to the concept of caching. Our application uses:

• Django
• Django Debug Toolbar
• django-redis
• Redis

Install the App

To clone the repository, install virtualenvwrapper, if you don’t already have it. This is a tool that lets you install the specific Python dependencies that your project needs, allowing you to target the versions and libraries required by your app in isolation.

Next, change directories to where you keep projects and clone the example app repository. Once done, change directories to the cloned repository, and then make a new virtual environment for the example app using the mkvirtualenv command:

Copied!$ mkvirtualenv django-redis
(django-redis)$
$ mkvirtualenv django-redis
(django-redis)$

NOTE: Creating a virtual environment with mkvirtualenv also activates it.

Install all of the required Python dependencies with pip, and then checkout the following tag:

Copied!(django-redis)$ git checkout tags/1
(django-redis)$ git checkout tags/1

Build the database and populate it with sample data to complete the example app setup. Make a superuser account so you can access the admin site. To verify sure the app is operating properly, follow the code samples below and then try executing it. Check that the data has been correctly loaded by going to the admin page in the browser.

Copied!(django-redis)$ python manage.py makemigrations cookbook
(django-redis)$ python manage.py migrate
(django-redis)$ python manage.py createsuperuser
(django-redis)$ python manage.py loaddata cookbook/fixtures/cookbook.json
(django-redis)$ python manage.py runserver
(django-redis)$ python manage.py makemigrations cookbook
(django-redis)$ python manage.py migrate
(django-redis)$ python manage.py createsuperuser
(django-redis)$ python manage.py loaddata cookbook/fixtures/cookbook.json
(django-redis)$ python manage.py runserver

Once you have the Django app running, move onto the Redis installation.

Install Redis

Download and install Redis using the instructions provided in the documentation. Alternatively, you can install Redis using a package manager such as apt-get or homebrew depending on your OS.

Run the Redis server from a new terminal window.

Copied!$ redis-server
$ redis-server

Next, start up the Redis command-line interface (CLI) in a different terminal window and test that it connects to the Redis server. We will be using the Redis CLI to inspect the keys that we add to the cache.

Copied!$ redis-cli ping
PONG
$ redis-cli ping
PONG

Redis provides an API with various commands that a developer can use to act on the data store. Django uses django-redis to execute commands in Redis.

Looking at our example app in a text editor, we can see the Redis configuration in the settings.py file. We define a default cache with the CACHES setting, using a built-in django-redis cache as our backend. Redis runs on port 6379 by default, and we point to that location in our setting. One last thing to mention is that django-redis appends key names with a prefix and a version to help distinguish similar keys. In this case, we have defined the prefix to be “example”.

Copied!CACHES = {
    "default": {
        "BACKEND": "django_redis.cache.RedisCache",
        "LOCATION": "redis://127.0.0.1:6379/1",
        "OPTIONS": {
            "CLIENT_CLASS": "django_redis.client.DefaultClient"
        },
        "KEY_PREFIX": "example"
    }
}
CACHES = {
    "default": {
        "BACKEND": "django_redis.cache.RedisCache",
        "LOCATION": "redis://127.0.0.1:6379/1",
        "OPTIONS": {
            "CLIENT_CLASS": "django_redis.client.DefaultClient"
        },
        "KEY_PREFIX": "example"
    }
}

NOTE: Although we have configured the cache backend, none of the view functions have implemented caching.

App Performance

Everything the server does to process a request slows the application load time, as we discussed at the start of this lesson. Running business logic and rendering templates has a considerable computational overhead. The time it takes to query a database is affected by network latency. When a client submits an HTTP request to the server, several factors come into play. When users make a large number of requests per second, the server’s performance suffers as it tries to process them all.

When we use caching, we let the server process a request once before saving it to our cache. When our application receives many requests for the same URL, the server gets the results from the cache rather than reprocessing them each time. We usually place a time limit on the cached results so that the data may be refreshed on a regular basis, which is a crucial step to take to prevent presenting stale data.

You should consider caching the result of a request when the following conditions are met for the application your developing:

• rendering the page involves a lot of database queries and/or business logic,
• the page is visited frequently by your users,
• the data is the same for every user,
• and the data does not change often.

Start by Measuring Performance

Begin by testing the speed of each page in your application by benchmarking how quickly your application returns a response after receiving a request.

To achieve this, we’ll be blasting each page with a burst of requests using loadtest, an HTTP load generator, and then paying close attention to the request rate. Visit the link above to install. Once installed, test the results against the /cookbook/ URL path:

Copied!$ loadtest -n 100 -k  http://localhost:8000/cookbook/
$ loadtest -n 100 -k  http://localhost:8000/cookbook/

Notice that we are processing about 16 requests per second:

Copied!Requests per second: 16
Requests per second: 16

When we look at what the code is doing, we can make decisions on how to make changes to improve the performance. The application makes 3 network calls to a database with each request to /cookbook/, and it takes time for each call to open a connection and execute a query. Visit the /cookbook/ URL in your browser and expand the Django Debug Toolbar tab to confirm this behavior. Find the menu labeled “SQL” and read the number of queries:

cookbook/services.py

Copied!from cookbook.models import Recipe


def get_recipes():
    # Queries 3 tables: cookbook_recipe, cookbook_ingredient,
    # and cookbook_food.
    return list(Recipe.objects.prefetch_related('ingredient_set__food'))
from cookbook.models import Recipe


def get_recipes():
    # Queries 3 tables: cookbook_recipe, cookbook_ingredient,
    # and cookbook_food.
    return list(Recipe.objects.prefetch_related('ingredient_set__food'))

cookbook/views.py

Copied!from django.shortcuts import render
from cookbook.services import get_recipes


def recipes_view(request):
    return render(request, 'cookbook/recipes.html', {
        'recipes': get_recipes()
    })
from django.shortcuts import render
from cookbook.services import get_recipes


def recipes_view(request):
    return render(request, 'cookbook/recipes.html', {
        'recipes': get_recipes()
    })

The application also renders a template with some potentially expensive logic.

Copied!
  <title>Recipes</title>

<body>
{% for recipe in recipes %}
  <h1>{{ recipe.name }}</h1>
    <p>{{ recipe.desc }}</p>
  <h2>Ingredients</h2>
  <ul>
    {% for ingredient in recipe.ingredient_set.all %}
    <li>{{ ingredient.desc }}</li>
    {% endfor %}
  </ul>
  <h2>Instructions</h2>
    <p>{{ recipe.instructions }}</p>
{% endfor %}

</body>
<html>
<head>
  <title>Recipes</title>
</head>
<body>
{% for recipe in recipes %}
  <h1>{{ recipe.name }}</h1>
    <p>{{ recipe.desc }}</p>
  <h2>Ingredients</h2>
  <ul>
    {% for ingredient in recipe.ingredient_set.all %}
    <li>{{ ingredient.desc }}</li>
    {% endfor %}
  </ul>
  <h2>Instructions</h2>
    <p>{{ recipe.instructions }}</p>
{% endfor %}
</body>
</html>

Implement Caching

Consider how many network calls our program will make as users begin to visit our website. If 1,000 people use the API to get cookbook recipes, our application will query the database 3,000 times, with each request generating a new template. As our application develops in size, this figure will only increase. This view, fortunately, is a good candidate for caching. A cookbook’s recipes rarely, if ever, change. Furthermore, because perusing cookbooks is the app’s main feature, the API for retrieving recipes is almost certain to be used frequently.

The view function is modified to employ caching in the example below. When the function executes, it looks in the cache for the view key. If the key exists, the app retrieves and returns the data from the cache. If this is not the case, Django will query the database and save the results in the cache with the view key. Django will query the database and render the template the first time this function is called, as well as perform a network call to Redis to save the data in the cache. Each subsequent call to the function will query the Redis cache instead of the database and business logic.

example/settings.py

Copied!# Cache time to live is 15 minutes.
CACHE_TTL = 60 * 15
# Cache time to live is 15 minutes.
CACHE_TTL = 60 * 15

cookbook/views.py

Copied!from django.conf import settings
from django.core.cache.backends.base import DEFAULT_TIMEOUT
from django.shortcuts import render
from django.views.decorators.cache import cache_page
from cookbook.services import get_recipes

CACHE_TTL = getattr(settings, 'CACHE_TTL', DEFAULT_TIMEOUT)


@cache_page(CACHE_TTL)
def recipes_view(request):
    return render(request, 'cookbook/recipes.html', {
        'recipes': get_recipes()
    })
from django.conf import settings
from django.core.cache.backends.base import DEFAULT_TIMEOUT
from django.shortcuts import render
from django.views.decorators.cache import cache_page
from cookbook.services import get_recipes

CACHE_TTL = getattr(settings, 'CACHE_TTL', DEFAULT_TIMEOUT)


@cache_page(CACHE_TTL)
def recipes_view(request):
    return render(request, 'cookbook/recipes.html', {
        'recipes': get_recipes()
    })

Notice that we have added the @cache_page() decorator to the view function, along with a time to live. Visit the /cookbook/ URL again and examine the Django Debug Toolbar. We see that 3 database queries are made and 3 calls are made to the cache in order to check for the key and then to save it. Django saves two keys (1 key for the header and 1 key for the rendered page content). Reload the page and observe how the page activity changes. The second time around, 0 calls are made to the database and 2 calls are made to the cache. Our page is now being served from the cache!

When we re-run our performance tests, we see that our application is loading faster.

Copied!$ loadtest -n 100 -k  http://localhost:8000/cookbook/
$ loadtest -n 100 -k  http://localhost:8000/cookbook/

Caching improved the total load, and we are now resolving 21 requests per second, which is 5 more than our baseline:

Copied!Requests per second: 21
Requests per second: 21

Inspecting Redis with the CLI

At this point we can use the Redis CLI to look at what gets stored on the Redis server. In the Redis command-line, enter the keys * command, which returns all keys matching any pattern. You should see a key called “example:1:views.decorators.cache.cache_page”. Remember, “example” is our key prefix, “1” is the version, and “views.decorators.cache.cache_page” is the name that Django gives the key. Copy the key name and enter it with the get command. You should see the rendered HTML string.

Copied!$ redis-cli -n 1
127.0.0.1:6379[1]> keys *
1) "example:1:views.decorators.cache.cache_header"
2) "example:1:views.decorators.cache.cache_page"
127.0.0.1:6379[1]> get "example:1:views.decorators.cache.cache_page"
$ redis-cli -n 1
127.0.0.1:6379[1]> keys *
1) "example:1:views.decorators.cache.cache_header"
2) "example:1:views.decorators.cache.cache_page"
127.0.0.1:6379[1]> get "example:1:views.decorators.cache.cache_page"

NOTE: Run the flushall command on the Redis CLI to clear all of the keys from the data store. Then, you can run through the steps in this tutorial again without having to wait for the cache to expire.

Wrap-up

Processing HTTP requests is expensive, and the cost climbs as your program becomes more popular. Implementing caching can help you reduce the amount of processing your server does in some cases. This tutorial covered the fundamentals of Django caching with Redis, however it just scratched the surface of a complex subject.

There are numerous problems and traps to be aware of when using caching in a complex application. It’s difficult to keep track of what gets cached and for how long. In Computer Science, invalidating caches is one of the most difficult tasks. Therefore is a security concern to ensure that confidential data may only be accessed by its intended users, and it must be treated with extreme caution when caching.

Play around with the source code in the example application, and remember to keep performance in mind as you continue to build with Django.