Migrate MongoDB Data Seamlessly with mongo-migrate and Python!

Migrate MongoDB Data Seamlessly with mongo-migrate and Python!

Play this article

The first time I used MongoDB was when we were building an internal ERP system. The use case for MongoDB was not to store any data but to store some schema and workflow data. This data stayed more or less the same once the application is deployed. While this project gave me a peek into the MongoDB ecosystem, the actual learning about its capabilities came in the next project where we used MongoDB to store the data for a configuration management system.

After the excitement of using a new system died out, we identified there are issues. I want to talk about one such issue we faced and discuss a solution.

The Problem Statement

We were facing two main issues during the development phase.

  1. Managing the database schema:

    We were unable to test out feature branches independently when the feature being implemented contained database schema changes. When such changes came up, we had to serialize the implementation so that other teams are unaffected by this change.

  2. Migrating the database into production:

    When we rolled out the changes into production, it was a cumbersome process. Creating a lot of custom scripts to migrate the user database which was part of version A to the target version. These became largely invalid and not maintained, adding to the technical debt.

The solution that I am going to discuss will solve the below problems:

  1. Bring up the database to the same schema as it was during the development of that version (or commit).

  2. Both upgrades and downgrades can be performed gracefully by the developer, making the process uniform.

  3. Contains incremental changes and not the whole schema.

  4. Versioning schema changes along with the code.

How do you migrate MongoDB?

I knew Django had a feature where it generated the migration files using declarative commands and I searched if MongoDB library Pymongo provided any such feature built-in, but it did not. I checked if other libraries that provided the migrations existed. There were a few, but when I inspected the code, most of them did provide a declarative interface like Django. Thus I decided to write my own. Also, I wanted to develop something open-source (MIT licensed) for a long time and this tool felt like the right one.

Mongo-Migrate

Over the weekend, I created mongo-migrate, a Python library capable of giving the developer the power to quickly create migrations, and upgrade or downgrade schema based on these incremental migration files. For most of the scenarios, you can use the command line tool mongo-migrate after you pip install, but for advanced use cases, you can import the MigrationManager class and the already created migrations.

mongo-migrate gives you three commands - create, upgrade, and downgrade. If you are eager to try it out without going through the commands, go ahead and run this command on the command line to install the package.

pip install mongo-migrate

The Create Command

It accepts a migration folder path and creates empty migration files in this folder.

usage: mongo-migrate create [-h] [--host HOST] [--port PORT] 
            [--database DATABASE] [--migrations MIGRATIONS] 
            [--title TITLE] --message MESSAGE

optional arguments:
  -h, --help            show this help message and exit
  --host HOST           the database host
  --port PORT           the database port
  --database DATABASE   provide the database name
  --migrations MIGRATIONS
                        provide the folder to store migrations. 
                        By default creates migrations/
  --title TITLE         title will be used in the file name
  --message MESSAGE     message will be recorded as a comment 
                        inside the migration file

The template migration file (sample below) has three public methods:

  1. upgrade(): Add code to upgrade the database from the current version to the target version should go here. For example, adding a collection or adding a field.

  2. downgrade(): Add code relevant for downgrading the database from the current version to the target version should go here. For example, removing a collection or removing a field.

  3. comment(): Adds a user-given message as a comment. Currently stored for documentation purposes.

You should add your code to perform the schema migration. (See sample below)

The Upgrade Command

The Upgrade command accepts a migration folder path and target version to go to. It systematically performs the upgrade of the database from the current version.

usage: mongo-migrate upgrade [-h] [--host HOST] [--port PORT] 
                [--database DATABASE] [--migrations MIGRATIONS] 
                [--upto UPTO]

optional arguments:
  -h, --help            show this help message and exit
  --host HOST           the database host
  --port PORT           the database port
  --database DATABASE   provide the database name
  --migrations MIGRATIONS
                        provide the folder where the migrations are stored 
                        By default looks for migrations/
  --upto UPTO           target migration timestamp

The Downgrade Command

The Downgrade command accepts a migration folder path and target version to go to. It systematically performs the rollback of the database from the current version until reaches the target version.

usage: mongo-migrate downgrade [-h] [--host HOST] [--port PORT] 
                [--database DATABASE] [--migrations MIGRATIONS] 
                [--upto UPTO]

optional arguments:
  -h, --help            show this help message and exit
  --host HOST           the database host
  --port PORT           the database port
  --database DATABASE   provide the database name
  --migrations MIGRATIONS
                        provide the folder where the migrations are stored. 
                        By default looks for migrations/
  --upto UPTO           target migration timestamp

How do I use mongo-migrate in a project?

Let me walk you through the steps quickly without making it sound complicated.

Step 1: Set up your virtual environment and install mongo-migrate.

Step 2: Use the mongo-migrate create command to create the first migration file template. You should provide the MongoDB host, port, and database.

Step 3: Modify the migration file created and add the code necessary to upgrade the database. Add this code to the upgrade() method.

Step 4: Also, add the steps to revert the above change in the downgrade() method.

Step 5: Run the mongo-migrate upgrade command to implement these changes onto the database.

Step 6: In case you need to revert the changes, run the mongo-migrate downgrade command.

Package Dependencies

This library was written with minimal dependency to keep it clean. The only library it uses additionally is the Pymongo library.

Conclusion

Remember, MongoDB schema migrations don't have to be complex. With mongo-migrate, managing schema changes, version control, and performing migrations is a breeze. Explore our tool, suggest ideas to improve this tool further, and simplify schema migrations!