Deploy MongoDB replica set on Docker swarm Nov 3, 2018

I'm going to create a three-member replica set using a cluster of Docker with 1 manager and 2 workers. Each MongoDB replica member will be placed at each Docker node.

MongoDB replica set

Overview

A 3-member replica set provides enough redundancy to survive most network partitions and other system failures. It also has sufficient capacity for many distributed read operations. The standard replica set deployment for the production system is a three-member replica set.

Requirements

  • 3 servers, if you do not have, you can get small free credit at Vultr or DigitalOcean for creating your servers.
  • 3 hostname for your servers, for example, mongo0.example.com, mongo1.example.com, mongo2.example.com.
  • SSL for your hostnames. 
  • Basic knowledge of SSH
  • Basic knowledge of MongoDB and Docker

Deploy New Replica Set

For better performance, user XFS filesystem for all your servers. I going to use Ubuntu 18.04 in this article. For SSL, I will use letsencrypt.org and I assume that you have your domain in cloudflare.com.

Config Docker manager:

ssh to one your server and run the following commands (remember to replace your_cloudflare_api_key, your_cloudflare_email, YOUR_SERVER_IP, example.com):

# Update server
apt-get update && apt-get upgrade -y

# Install Docker
apt-get install \
apt-transport-https \
ca-certificates \
curl \
software-properties-common -y

curl -fsSL https://download.docker.com/linux/ubuntu/gpg | sudo apt-key add -

add-apt-repository \
"deb [arch=amd64] https://download.docker.com/linux/ubuntu \
$(lsb_release -cs) \
stable" -y

apt-get update -y

apt-get install docker-ce -y

# Install SSL
curl https://get.acme.sh | sh
export CF_Key="your_cloudflare_api_key"
export CF_Email="your_cloudflare_email"
acme.sh --dns dns_cf --issue -d example.com -d *.example.com

# swarm mode
docker swarm init --advertise-addr YOUR_SERVER_IP

Config workers:

repeat this with your other 2 servers

# Update server
apt-get update && apt-get upgrade -y

# Install Docker
apt-get install \
apt-transport-https \
ca-certificates \
curl \
software-properties-common -y

curl -fsSL https://download.docker.com/linux/ubuntu/gpg | sudo apt-key add -

add-apt-repository \
"deb [arch=amd64] https://download.docker.com/linux/ubuntu \
$(lsb_release -cs) \
stable" -y

apt-get update -y

apt-get install docker-ce -y

# Install SSL
curl https://get.acme.sh | sh
export CF_Key="your_cloudflare_api_key"
export CF_Email="your_cloudflare_email"
acme.sh --dns dns_cf --issue -d example.com -d *.example.com

# swarm mode
docker swarm join --token TOKEN YOUR_MANAGER_IP:2377

Note: again, replace TOKEN and YOUR_MANAGER_IP with your real value.

Setting things up:

We want each MongoDB of replica set runs at separate Docker node, the simple way to define this is using Docker label. First, list all nodes by run this command on the Manager Server:

docker node ls

ID HOSTNAME STATUS AVAILABILITY MANAGER STATUS ENGINE VERSION
r46sbw1wtq7y3wf7bkem0t0g7 * mongo0.example.com Ready Active Leader 18.06.1-ce
kws3jgzt2lf8ymds838q7ubzu mongo1.example.com Ready Active 18.06.1-ce
fzicah0uabizjhruwm363rih3 mongo2.example.com Ready Active 18.06.1-ce

Now you can see your node IDs, let apple the lables

docker node update --label-add mongo.rs=0 r46
docker node update --label-add mongo.rs=1 kws
docker node update --label-add mongo.rs=2 fzi

As we will use Keyfile Access Control for our replica set, we need to generate a key file, then copy this key file to other servers.

openssl rand -base64 756 > /root/.acme.sh/example.com/mongodb-keyfile
chmod 400 /root/.acme.sh/example.com/mongodb-keyfile
chown 999 /root/.acme.sh/example.com/mongodb-keyfile

I will name the key file mongodb-keyfile and save in the acme.sh ssl folder: /root/.acme.sh/example.com.

Now for the MongoDB SSL, we can just combine example.com.key and fullchain.cer in /root/.acme.sh/example.com to mongodb.pem. Note that everytime the SSL get renewed by letsencrypt.org, you need to redo the combination above. So let add a monthly crontab script for that:

#!/bin/bash
cd /root/.acme.sh/example.com
cat example.com.key fullchain.cer > mongodb.pem

Now the main job, deploy the MongoDB replica set. Create a mongodb-rs.yml file for this:

version: '3.7'

services:
mongo-0:
image: mongo:4
ports:
- target: 27017
published: 27017
protocol: tcp
mode: host
volumes:
- /home/db:/data/db
- /root/.acme.sh/example.com:/home/ssl
deploy:
mode: replicated
replicas: 1
restart_policy:
condition: on-failure
placement:
constraints:
- node.labels.mongo.rs == 0
command: ["mongod", "--replSet", "mongo-rs", "--keyFile", "/home/ssl/mongodb-keyfile", "--bind_ip_all", "--ipv6", "--sslMode", "requireSSL", "--sslAllowConnectionsWithoutCertificates", "--sslPEMKeyFile", "/home/ssl/mongodb.pem"]

mongo-1:
image: mongo:4
ports:
- target: 27017
published: 27017
protocol: tcp
mode: host
volumes:
- /home/db:/data/db
- /root/.acme.sh/example.com:/home/ssl
deploy:
mode: replicated
replicas: 1
restart_policy:
condition: on-failure
placement:
constraints:
- node.labels.mongo.rs == 1
command: ["mongod", "--replSet", "mongo-rs", "--keyFile", "/home/ssl/mongodb-keyfile", "--bind_ip_all", "--ipv6", "--sslMode", "requireSSL", "--sslAllowConnectionsWithoutCertificates", "--sslPEMKeyFile", "/home/ssl/mongodb.pem"]

mongo-2:
image: mongo:4
ports:
- target: 27017
published: 27017
protocol: tcp
mode: host
volumes:
- /home/db:/data/db
- /root/.acme.sh/example.com:/home/ssl
deploy:
mode: replicated
replicas: 1
restart_policy:
condition: on-failure
placement:
constraints:
- node.labels.mongo.rs == 2
command: ["mongod", "--replSet", "mongo-rs", "--keyFile", "/home/ssl/mongodb-keyfile", "--bind_ip_all", "--ipv6", "--sslMode", "requireSSL", "--sslAllowConnectionsWithoutCertificates", "--sslPEMKeyFile", "/home/ssl/mongodb.pem"]

Before running the deploy command below, you need to create a folder /home/db on all servers for storing you MongoDB data:

docker stack deploy -c mongodb-rs.yml mongodb

Initiates a replica set:

ssh to one of your Docker container (use docker ps to get the CONTAINER_ID):

docker exec -it CONTAINER_ID /bin/bash

Then run the commands below:

mongo --ssl mongo0.example.com

rs.initiate( {
_id : "mongo-rs",
members: [
{ _id: 0, host: "mongo0.example.com:27017" },
{ _id: 1, host: "mongo1.example.com:27017" },
{ _id: 2, host: "mongo2.example.com:27017" }
]
})

Next, add root user for MongoDB

admin = db.getSiblingDB("admin")
admin.createUser(
{
user: "root",
pwd: "YOUR_MONGO_ROOT_PASS",
roles: [ { role: "root", db: "admin" } ]
}
)

Connect to your MongoDB replica set

mongodb://root:YOUR_MONGO_ROOT_PASS@mongo0.example.com:27017,mongo1.example.com:27017,mongo2.example.com/admin?replicaSet=mongo-rs&ssl=true

Or you can set up DNS Seedlist Connection Format:

mongodb+srv://mongo.example.com

Congratulations, now you have a production MongoDB replica set!

ref:
- https://docs.mongodb.com/manual/tutorial/deploy-replica-set-with-keyfile-access-control/#deploy-repl-set-with-auth
- https://docs.docker.com/engine/swarm/

Advertisement

Latest Updates