MongoDB: How to move a collection to a different database (4 ways)

Introduction

This tutorial covers the process of moving a collection from one MongoDB database to another. It is fairly common to encounter situations where data grows and the need to reorganize your database arises. Whether it is due to refactoring, scaling requirements, or organizational changes, the ability to move collections around can be a vital skill for any developer, database administrator, or DevOps professional who deals with MongoDB.

We will assume that you are familiar with the basics of MongoDB and have MongoDB installed along with the MongoDB Shell (mongosh) for interfacing with the database. Let’s start with the simplest method and incrementally dive into more complex tasks.

Method 1: Using mongodump and mongorestore

One of the easiest ways to move a collection between databases is to use the mongodump and mongorestore utilities. These tools allow you to export a collection to a BSON file and then import it into a different database.

mongodump --db old_database --collection your_collection
mongorestore --nsFrom 'old_database.your_collection' --nsTo 'new_database.your_collection' --drop dump/old_database/your_collection.bson

This operation is safe and can be reverted by simply restoring the dumped BSON file into the original database in case something goes wrong. Remember that --drop will drop the collection before doing the restore if it already exists, thereby preventing errors due to duplicate data.

Method 2: Using MongoDB Aggregation Pipeline

If you want to avoid the overhead of creating physical files on disk, you can use the MongoDB Aggregation Pipeline to copy a collection directly from one database to another. Connect to your MongoDB instance using the shell:

mongosh

In the mongo shell, switch to the source database:

use old_database

Next, leverage the $out or $merge operators in the aggregation framework to output documents to a collection in a different database:

db.your_collection.aggregate([
    {
        $merge: {
            into: { db: 'new_database', coll: 'your_collection' },
            whenMatched: 'replace',
            whenNotMatched: 'insert'
        }
    }
])

Note that $merge operation will replace or insert documents if they already exist, unlike $out that replaces the entire collection. This approach is great for continuous data replication or for migrations where downtime is a constraint.

Method 3: Copying Collections with a Script

If you have programming experience, specifically with JavaScript, you can write a script to handle the migration using the MongoDB driver. Here is an example using Node.js:

const { MongoClient } = require('mongodb');

async function copyCollection(sourceUri, targetUri, sourceCollection, targetCollection) {
    const sourceClient = new MongoClient(sourceUri);
    const targetClient = new MongoClient(targetUri);

    try {
        await sourceClient.connect();
        await targetClient.connect();

        const sourceDb = sourceClient.db();
        const targetDb = targetClient.db();

        const documents = await sourceDb.collection(sourceCollection).find({}).toArray();
        if (documents.length) {
            await targetDb.collection(targetCollection).insertMany(documents);
            console.log('Collection copied successfully');
        } else {
            console.log('No documents to copy');
        }
    } catch (err) {
        console.error('An error occurred:', err);
    } finally {
        await sourceClient.close();
        await targetClient.close();
    }
}

copyCollection('mongodb://localhost:27017/old_database', 'mongodb://localhost:27017/new_database', 'your_collection', 'your_collection');

The console log will indicate success or errors.

This script can be adapted for complex scenarios, such as transforming documents during copying or copying to a different MongoDB cluster.

Method 4: MongoDB Atlas

If you operate in the cloud, specifically with MongoDB Atlas, the process may differ. MongoDB Atlas provides built-in tools to handle data migration across clusters which can be leveraged through the UI or using Atlas’s API.

Use the Atlas Data Lake service to move collections between different projects or clusters. Mapping of databases and collections can be articulated via the Atlas UI, which under the hood uses a similar approach to mongodump and mongorestore but is more integrated and streamlined.

Best Practices and Process Integrity

Regardless of the method used, it’s crucial to ensure that the process is safe and consistent, meaning that:

• Ensure robust error handling and rollback capabilities.
• Confirm read and write access to the source and target databases.
• Maintain consistency by locking the collection or having the application in maintenance mode during the transfer.
• Test the process thoroughly in a staging environment before proceeding with the production.

Conclusion

In this guide, we discussed several methods for moving a MongoDB collection to a different database. From basic tools such as mongodump and mongorestore, to scripting with intervals, MongoDB provides a robust set of approaches for data migration. Be sure to understand the pros and cons of each method and choose the one that best fits your needs, infrastructure, and business requirements. Good luck with your data re-architecture!