Duel of the Djangos

You’ve spotted that last piece of commodity that you love and someone else is up for grabs for it. Will you let them have it, snatch it off them or resolve through fair means !?! Who gets the say? Well in our case, it’s…

The Story….

We are running our entire application on Django 1.7. Like any other legacy system( let’s call it OLD_PROJ) it was a labyrinth of complex code. To make life easier we decided to move on to a better approach with an API based system, micro services powered by Django Rest Framework, Django 1.10 (our test drive on it) and other such fancy solutions. We called it HULK !

But any oldie can vouch for it that you can’t get rid of them so easily. So we had to maintain a huge part of the older system and expand the newer one. But these two projects had something in common! The database. Now arises the actual problem. 2 projects, 2 django versions, one shared database! Its Django1.7 vs Django1.10…

This would be an easy deal if we were to use raw SQL queries to connect to multiple databases instead of leveraging Django ORM. But we love experimenting and whats the point, if we can’t utilise the framework to its fullest!!

So the problem in hand is that we need to reuse an existing database. How hard can it be? How about just add it to the Databases config in settings.py?

Oops…Trouble!

The Mystery…

Let’s call the database for HULK (NEW_PROJECT) as HULK_DB and the one for OLD_PROJ as OLD_DB. Lets say there is an app A in HULK and it needs to use models m1, m2, m3 from the OLD_PROJ. If we were to copy the model definitions from the OLD_PROJ to HULK into A/models.py , then, on running migrate django would just recreate these tables in the HULK_DB, which is not what we want as we want to reuse the existing data in OLD_DB.

Note: You can create the usual A/models.py and define all the native models for HULK and use them as you would normally. These tables would be registered in HULK_DB.

Coming back to reusing old data, the first step was to identify all the models that need to be reused and avoid django migrate from registering these models in HULK. For that we followed the steps below:

1. Created a django app in HULK called goibibo_models (to store all model definitions).
2. Created a models file, lets say common_models.py. (This can be named anything except models.py). All model definitions required from OLD_PROJ were copied into this.
3. By default, django generates <appname_modelname> as the table_name to be accessed for read/write operations, hence we have to modify the Meta of each model class to access the actual table names:
   * changing the ‘db_table’ property to actual name of the table in OLD_DB
   * setting a common app_label to all models copied from OLD_PROJ (explained later)

# Example:

class Notes(models.Model):
    notes = models.TextField()
    createdon = models.DateTimeField(auto_now_add=True)
    modifiedon = models.DateTimeField(auto_now=True)

    class Meta:
        app_label = 'goibibo_models'
        db_table = 'common_itemnotes'

Simple yet, but then when a project is created and you setup the ‘DATABASES’ property in settings.py and run migrations, Django internally creates all the basic tables for auth (eg: User), contentypes(for storing generic relations) and loads its registry based on the Django installed_apps for that project. In this case we wanted to reuse these inbuilt django models too from an already existing database. Oops again !!!

The Eureka Moment…

To solve this we had to override the built in types from Django source code! This was a pretty scary thing to do, but the efforts were fruitful. For all these models we named the app_label as ‘goibibo_inbuilt’ to differentiate them from other app models.

Below is the snippet of code overriding the django User model.

Similarly the code overriding other django inbuilt types are as follows.
1. custom_contenttypes.py
2. custom_genericcontenttypes.py

Now that we have overridden all inbuilt types, we can make use of them in our models definition to utilise, CustomUser and Generic Foreign Key types. This is illustrated below by adding few more attributes to the Notes model mentioned earlier.

1. The user field points to a user in the OLD_DB
2. The content_object uses content_type and object_id to build the custom GenericForeignKey field that can refer to multiple other models instead of a single one (like ForeignKey type).

The Finale

So now we have reached the final step. At this point we have 2 different sets of codes:
1. one native to hulk using the default inbuilt types, models.py
2. another, a set of files with custom code to access the old database.

The only thing left is to redirect the requests accordingly to the databases. For that we used the database routing feature of Django. We do this by ensuring that any model with app_label as ‘goibibo_models’ or ‘goibibo_inbuilt’ (as changed earlier) should access the old database.

# settings.py

DATABASE_ROUTERS = ['multiple_db_routers.HulkDataRouter']

DATABASES = {
    'hulk_db': {
        <configure HULK database>
    },
    'old_db': {
        <configure OLD_APP database>
    }
}

Below is the code for multiple_db_routers.py

And the winner is…

This completes our setup and changes and now we can effectively utilise the django ORM for different django projects with different versions and different databases.

# To access table specific to hulk from HULK_DB(Django1.10)
from models.py import HulkModel

# To access table from OLD_PROJ from OLD_DB(Django1.7)
from common_models import Notes

Time now to announce the winner. And the winner is…..
DJANGO!! Which one, you say..?

Duel of the Djangos was originally published in Backstage on Medium, where people are continuing the conversation by highlighting and responding to this story.