I'm going to create a three-member replica set using a cluster of Docker with 1 manager and 3 workers. Each MongoDB replica member will be placed at each Docker work node.

Overview

A 3-member replica set provides enough redundancy to survive most network partitions and other system failures. It also has sufficient capacity for many distributed read operations. The standard replica set deployment for the production system is a three-member replica set.

Requirements

• 4 servers (1 manager, 3 workers), if you do not have, you can get small free credit at Vultr, UpCloud or DigitalOcean for creating your servers.
• 4 hostname for your servers, for example, manager.example.com, worker0.example.com, worker1.example.com, worker2.example.com.
• Valid SSLs for your hostnames. 
• Basic knowledge of SSH
• Basic knowledge of MongoDB and Docker

Deploy New Replica Set

For better performance, use XFS filesystem for all your servers. I going to use Ubuntu 18.04 in this article. For SSL, I will use letsencrypt.org and I assume that you have your domain in cloudflare.com.

Install Docker on all servers:

ssh into all your servers and run the following commands:

# Update server
apt-get update && apt-get upgrade -y

# Install Docker
curl https://raw.githubusercontent.com/harrytang/linux-cmds/master/install-docker-ubuntu.sh | sh

Config manager/workers:

ssh into the manager server and run the the command belows (replace YOUR_SERVER_IP with your real IP)

docker swarm init --advertise-addr YOUR_SERVER_IP

Then, ssh into 3 workers servers and run this command (again, replace TOKEN and YOUR_MANAGER_IP with your real value)

docker swarm join --token TOKEN YOUR_MANAGER_IP:2377

We want each MongoDB of replica set runs at separate Docker node, the simple way to define this is using Docker label. First, list all nodes by run this command on the Manager Server:

docker node ls

ID HOSTNAME STATUS AVAILABILITY MANAGER STATUS ENGINE VERSION
ahen750s5130miryjyn2fv7b2 * manager Ready Drain Leader 19.03.2
j7ahrccg7pmq5xtw7hrj4a781   worker0 Ready Active 19.03.2
kh9uktekpsuu9btj1pjuv3hkm    worker1 Ready Active 19.03.2
vhw3wy9js9wkxqyedylnjkhc2   worker2 Ready Active 19.03.2

Now you can see your node IDs, let apple the lables

docker node update --label-add mongo.rs=0 j7a
docker node update --label-add mongo.rs=1 kh9
docker node update --label-add mongo.rs=2 vhw

Config MongoDB on workers:

You need 3 hostnames with valid certificates, for example worker0.example.com, worker1.example.com, worker2.example.com, with the help of acme.sh and letsencrypt.org you can easily get the certificates. Full description of getting the certificates is beyond the scope of this blog. I assume you already have valid certificates for your all your hostnames.

We also need to create some directories to store the data and configure file, just run the command below to get things done:

curl https://raw.githubusercontent.com/powerkernel/docker-mongodb-rs/master/mongodb-rs-init.sh | sh

You will see a directory in /data/mongodb. Since we use Keyfile Access Control for our replica set, so we need copy and overwrite the /data/mongodb/keyfile from worker0 to worker1 and worker2. 

We also need to tell acme.sh to reload mongodb everytime the certificates renewed. So run the command below on every workers:

acme.sh --install-cert -d hostname.example.com \
--key-file        /root/certs/hostname/key.pem \
--cert-file       /root/certs/hostname/cert.pem \
--fullchain-file  /root/certs/hostname/fullchain.pem \
--ca-file         /root/certs/hostname/ca.pem \
--reloadcmd       "cat /root/certs/hostname/key.pem /root/certs/hostname/fullchain.pem > /root/certs/hostname/mongodb.pem && cat /root/certs/hostname/key.pem /root/certs/hostname/fullchain.pem > /root/certs/all/hostname.pem && /root/scripts/reload-hostname-cert.sh"

Note: replacehostname.example.com with your hostname, for example worker0.example.com.

Deploy MongoBD on manager node

Now login to the manager node, download this docker-compose.yml file and run the command below:

docker stack deploy -c docker-compose.yml mongodb

Initiates a replica set:

ssh to one of your container on the workers (use docker ps to get the CONTAINER_ID):

docker exec -it CONTAINER_ID /bin/bash

Then run the commands below:

mongo --tls --tlsAllowInvalidCertificates

rs.initiate( {
  _id : "mongo-rs",
  members: [
    { _id: 0, host: "worker0.example.com:27017" },
    { _id: 1, host: "worker1.example.com:27017" },
    { _id: 2, host: "worker2.example.com:27017" }
  ]
})

Next, add root user for MongoDB

admin = db.getSiblingDB("admin")
admin.createUser(
  {
    user: "root",
    pwd: "YOUR_MONGO_ROOT_PASS",
    roles: [ { role: "root", db: "admin" } ]
  }
)

Connect to your MongoDB replica set

mongodb://root:[email protected]:27017,worker1.example.com:27017,worker2.example.com/admin?replicaSet=mongo-rs&tls=true

Or you can set up DNS Seedlist Connection Format:

mongodb+srv://mongo.example.com

Tip: If you want to add more users, use the following commands

use admin
db.createUser(
  {
    user: "YOUR_USER_NAME",
    pwd: "YOUR_LONG_LONG_PW",
    roles:
    [
      { role: "readWrite", db: "YOUR_DB_NAME" }
    ],
    mechanisms: [ "SCRAM-SHA-1" ] 
  }
)

Set up high availability

We now just have 1 manager node, you can join multiple manager nodes to the cluster, so that if one manager node fails, another can automatically take its place without impact to the cluster.

Congratulations, now you have a production MongoDB replica set!

ref:
- https://docs.mongodb.com/manual/tutorial/deploy-replica-set-with-keyfile-access-control/#deploy-repl-set-with-auth
- https://docs.docker.com/engine/swarm/