Abstract

This post will guide you through setting up a locally MongoDB replica set using Docker for testing and learning purposes. Once you’ve successfully set up the replica set on your local machine, it’s easy to implement the same setup in production.

This setup features auto primary switching using HAProxy, which is useful when the framework you use does not come with built-in support for Connect to a Replica Set. We’ll go through the necessary steps and provide explanations along the way. By the end of this tutorial, you’ll have a local MongoDB replica set with HAProxy that can be used for testing and learning. Additionally, we’ll discuss implementing this setup in production by adding isolation, such as separated Docker virtual machines.

Introduction

A MongoDB replica set is a group of MongoDB instances that maintain the same data set, providing redundancy and increasing data availability. This tutorial will show you how to set up a MongoDB replica set using Docker and HAProxy for auto primary switching.

Attention, if you can use a framework that supports connecting to a replica set, you don’t need HAProxy. Refer to the MongoDB documentation (Archive) for more information.

Prerequisites

• Docker installed on your machine
• Basic knowledge of Docker and MongoDB

Step-by-Step Guide

Step 1: Generate Replica Key

First, we need to generate a replica key for authentication. Run the following commands to create a key file:

openssl rand -base64 741 > /path/to/your/replica_key/replica.key
chmod 600 /path/to/your/replica_key/replica.key
chown 999:999 /path/to/your/replica_key/replica.key

Replace /path/to/your/replica_key with the path where you’d like to store the key file.

Step 2: Create Network

Create a Docker network for the MongoDB containers using the following command:

docker network create mongo_net --driver bridge

Step 3: Create MongoDB Containers

Create three MongoDB containers using the following commands:

docker run -d --name mongo1 --hostname mongo1 --network mongo_net -p 27011:27017 -v "/path/to/your/mongo/db1:/data/db" -v "/path/to/your/replica_key/replica.key:/data/key/replica.key" -e MONGO_INITDB_ROOT_USERNAME=root -e MONGO_INITDB_ROOT_PASSWORD=password123 mongo:6 mongod --replSet myReplicaSet --bind_ip localhost,mongo1 --keyFile "/data/key/replica.key"
docker run -d --name mongo2 --hostname mongo2 --network mongo_net -p 27012:27017 -v "/path/to/your/mongo/db2:/data/db" -v "/path/to/your/replica_key/replica.key:/data/key/replica.key" -e MONGO_INITDB_ROOT_USERNAME=root -e MONGO_INITDB_ROOT_PASSWORD=password123 mongo:6 mongod --replSet myReplicaSet --bind_ip localhost,mongo2 --keyFile "/data/key/replica.key"
docker run -d --name mongo3 --hostname mongo3 --network mongo_net -p 27013:27017 -v "/path/to/your/mongo/db3:/data/db" -v "/path/to/your/replica_key/replica.key:/data/key/replica.key" -e MONGO_INITDB_ROOT_USERNAME=root -e MONGO_INITDB_ROOT_PASSWORD=password123 mongo:6 mongod --replSet myReplicaSet --bind_ip localhost,mongo3 --keyFile "/data/key/replica.key"

Replace /path/to/your/mongo/ and /path/to/your/replica_key with the appropriate paths. These commands create three MongoDB containers, each with a replica key and a data directory mounted.

Step 4: Initiate Replica Set

Run the following command to initiate the replica set:

docker exec mongo1 mongosh --username root --password password123 --eval 'rs.initiate({
  _id: "myReplicaSet",
  members: [
    { _id: 1, host: "mongo1:27017" },
    { _id: 2, host: "mongo2:27017" },
    { _id: 3, host: "mongo3:27017" },
  ]
})'

You should see the output { "ok" : 1 } indicating that the replica set has been successfully initiated.

Step 5: Check Replica Set Status

Check the status of the replica set with the following command:

docker exec -it mongo1 mongosh --username root --password password123 --eval "rs.status()"

The output should show three members with stateStr of PRIMARY or SECONDARY and health of 1. If you see infoMessage: 'Could not find member to sync from', wait a few seconds and check again.

Step 6: Create HAProxy Config

Create an HAProxy configuration file named haproxy.cfg.

global
    log /dev/log local0
    log /dev/log local1 notice
    user haproxy
    group haproxy
    daemon
    maxconn 4096

defaults
    log global
    mode tcp
    option tcplog
    option dontlognull
    timeout connect 5000
    timeout client 50000
    timeout server 50000

frontend mongodb_frontend
    bind *:27017
    mode tcp
    option tcplog
    default_backend mongodb_backend

backend mongodb_backend
    option tcp-check

    # Credit: https://blog.danman.eu/mongodb-haproxy/
    # Wire protocol
    tcp-check send-binary 3a000000 # Message Length (58)
    tcp-check send-binary EEEEEEEE # Request ID (random value)
    tcp-check send-binary 00000000 # Response To (nothing)
    tcp-check send-binary d4070000 # OpCode (Query)
    tcp-check send-binary 00000000 # Query Flags
    tcp-check send-binary 61646d696e2e # fullCollectionName (admin.$cmd)
    tcp-check send-binary 24636d6400 # continued
    tcp-check send-binary 00000000 # NumToSkip
    tcp-check send-binary FFFFFFFF # NumToReturn
    # Start of Document
    tcp-check send-binary 13000000 # Document Length (19)
    tcp-check send-binary 10 # Type (Int32)
    tcp-check send-binary 69736d617374657200 # ismaster:
    tcp-check send-binary 01000000 # Value : 1
    tcp-check send-binary 00 # Term
    tcp-check expect binary 69736d61737465720001 #ismaster True
    
    # Severs. Note that the port is the internal port of the MongoDB container.
    server mongo1 mongo1:27017 check inter 3s fall 3 rise 2
    server mongo2 mongo2:27017 check inter 3s fall 3 rise 2
    server mongo3 mongo3:27017 check inter 3s fall 3 rise 2

The configuration above will check the status of the MongoDB containers every 3 seconds and switch the primary if needed.

Step 7: Create HAProxy Container

Create and run the HAProxy container with the following command:

docker run -d --name haproxy --hostname haproxy --network mongo_net -p 27017:27017 -v "/path/to/haproxy:/path/to/haproxy" haproxy:lts-alpine3.17

Note that if you check its logs now, you’ll see something like this:

[WARNING]  (8) : Server mongodb_backend/mongo3 is DOWN, reason: Layer7 timeout, info: " at step 15 of tcp-check (expect binary 'ismaster')", check duration: 3004ms. 1 active and 0 backup servers left. 0 sessions active, 0 requeued, 0 remaining in queue.

This is quite normal, as we mentioned earlier that the config will check for primary and point to it. In one replica set, there can only be one primary, so the other two will be shown as down.

If you made any changes to the haproxy.cfg file, restart the HAProxy container:
docker restart haproxy

Step 9: Test the Replica Set

Stop one of the MongoDB containers and restart all containers to test the replica set:

# Stop one of the MongoDB containers, then see if the auto primary switching works
docker stop mongo1

Step 10: Connect to the Proxied MongoDB

Finally, connect to the reverse-proxied MongoDB using a client such as MongoDB Compass. Make sure to check the “Direct Connection” checkbox, or you may encounter an error similar to getaddrinfo ENOTFOUND mongo1.

Conclusion

By following these steps, you have successfully set up a MongoDB replica with HAProxy for auto primary switching. You can now connect to the MongoDB replica set using the HAProxy container’s address and port 27017.

In production, you may not want to set up all MongoDB instances on the same machine. In that case, configure each MongoDB instance on separate physical or virtual machines and adjust the addresses in the HAProxy configuration accordingly.