What happens when a Primary fails in a MongoDB replica set? Automatic failovers?

In distributed systems, high availability is not a luxury—it's a necessity. And one of the very important parts of that is automatic failovers. What are automatic failovers and how do they work? Let's see today!

These concepts are actually similar in every database, but as MongoDB is natively built for replica-sets, we will talk about that here, whereas other DB systems need 3rd-party or managed systems to implement the same.

Let’s dive into how database leadership changes hands in a replica set system.

First, what is a replica set?

A replica set in MongoDB is a group of mongod instances that maintain the same data set. They are basically copies of the same data kept in multiple independent running MongoDB servers (mongod).

For example, in a five-node replica set:

• One node is the primary–the only one accepting writes.
• Four nodes are secondaries–they replicate the primary's oplog and stay in sync.

The main picture -- primary fails 🚨

Let’s say the primary node suddenly becomes unreachable (network issue, crash, machine went down, etc.). What happens next?

How a new primary takes over: A step by step process 🧠

1. Heartbeat detection (within ~10 seconds)

Each node in the replica set sends heartbeats to all other nodes every two seconds. If a node misses multiple heartbeats (typically for 10 seconds), it's marked as down. This is when the system knows that something is off, and if the primary is available or not.

2. Election time—yes, it is a democracy!

If secondaries can no longer reach the primary, an election process is triggered.

MongoDB uses a Raft-inspired consensus protocol to choose the next leader (or primary). The steps (or the process) for the election are: 

• Eligible secondaries step up as candidates.
• Each candidate requests votes from other nodes.
• Nodes vote based on criteria like who has the most recent data, or if the DevOps/DBA team has set some replica priority settings.
• A majority (in this case, three out of five nodes) must agree on the new primary.

MongoDB follows a fully connected mesh topology. That means that each node is connected to each node, and that everyone is talking to each other. This way, everyone knows who has the most votes, and who is taking over.

3. New primary announced

Once elected, the new primary starts accepting writes. The replica set reconfigures itself, and secondaries begin syncing from the new leader.

One new machine or node also pops up as the original primary went down, making the count of nodes four, so now the count can actually get back to five.

4. Clients reconnect automatically

Drivers like the MongoDB Node.js or Python driver are built to handle this. They connect not to a single node but to the entire replica set (MongoDB URI). They detect the change and route requests to the new primary with minimal interruption to your application.

How MongoDB prevents split-brain (inconsistency)

A primary is only elected if a majority of nodes can communicate. This ensures that two different primaries aren't elected in partitioned networks (two different groups of nodes which can talk to each other, two disjoint sets). If only two out of five nodes can talk to each other, no election happens, avoiding data inconsistencies.

Why does automatic failover matter?

This automatic failover mechanism ensures:

• Zero manual intervention during outages.
• Minimal downtime for applications.
• Data consistency and safety.

In mission-critical systems, these features are invaluable.

Conclusion

Whether you're designing scalable backend systems or just dividing your database into distributed databases, understanding how automatic failover works is key to building fail-proof architectures (almost no system is 100% fail-proof).

Follow Shrey’s Newsletter: https://www.linkedin.com/newsletters/system-design-architecture-6871521381876584448/

• Databases
• Mongo

Shrey Batra

Author

Head of Eng @ HROne | Founder @ Cosmocloud (Acquired) | Ex-LinkedIn | MongoDB Champion | Book Author | Patent Holder