Ensure Data Consistency using MongoDB write concern.

Imagine a situation where you are placing an order on an E-commerce website for a highly sought-after product that’s in limited stock. Once you clear the payment and place the order, the E-commerce server will have to update the quantity of the product, user transaction details, and order details in its database.

Consider the E-commerce database has three Mongodb database nodes. One primary node and two secondary replica nodes. The primary node handles write and the secondary node handles read operations. When the E-commerce server sends the request to update the product details to the database, the database server writes the product details in three databases in order to maintain data consistency. Without write concern, the database server sends back an acknowledgment once the data is written in the primary node. For the secondary nodes, the server follows the fire-and-forget approach where the E-commerce server sends back acknowledgment without getting confirmation from the secondary nodes.

Now, due to some network glitch or unforeseen technical complication, all writes to the secondary node fail, and the product and user details are updated only on the primary node and not on the secondary nodes. So what happens now? When the user places the order and goes to the order details page, the user will not see the placed order details. Also, the product quantity will not be updated. This leads to a potential risk of data inconsistency which can lead to inaccurate information being presented to users, causing confusion and disappointment. Here's where we need write concern.

Write Concern

When write concern is enabled, before acknowledging, the database server ensures the product quantity and order details are securely written in all three nodes to ensure consistency of information thus preventing discrepancies.

Table of Content

1. How does Write Concern work in MongoDB?
2. What are the options in Write Concern and How to specify Write Concern in MongoDB?
3. Improve performance and latency while using the Write Concern strategy.

How does Write Concern work in MongoDB?

As explained above, Write concern describes the level of acknowledgment or confirmation that MongoDB provides for each and every write operation.

MongoDB read and write

Before getting into the working flow of write concern let's first understand how a write operation happens in MongoDB. When you perform a write operation in MongoDB, your MongoDB driver on the client side will send a write request to the Primary node. After the write operation is performed in the Primary node, the MongoDB server records the data modification in a file called OPLOG ( Operational Log ). The purpose of this file is to record any kind of data modification. All replica sets will have this OPLOG file so that they can maintain the current state of the database.

After recording the modification request in the OPLOG file, the MongoDB server triggers write calls to the secondary nodes. But, before even getting back the acknowledgment from secondary nodes, MongoDB sends back a successful acknowledgment to the client driver. In the background, the secondary nodes use the OPLOG file of the primary node to copy the latest operations in an asynchronous manner.

Now what is the problem here? Let’s assume to have a network glitch / some technical problem while writing records to the secondary nodes, what happens? Writes on the secondary node will not happen. Because the writes on the primary node are successful, MongoDB acknowledges that the write is successful. But when we query, we will face discrepancies with the data presented when MongoDB load balances the queries between primary and secondary nodes.

MongoDB write concern

Here is where Write concern comes into place. When write concern is enabled, MongoDB waits till the writes on secondary nodes are also successful and then provides acknowledgment.

What are the options in Write Concern and How to specify Write Concern in MongoDB?

MongoDB lets you set write concerns for write operations by specifying certain options. You can add write concern using the option writeConcern with values w ( level of acknowledgment required for a write operation ), j ( specifies whether the write operation should be committed to the journal ) and wtimeout ( defines the maximum amount of time, in milliseconds, that a MongoDB write operation should wait )

db.collection.insertOne(
  { key: "value" },
  { writeConcern: { w: "majority", j: true, wtimeout: 10000 } }
);

Write concern can also be specified globally on the connection string as a query param if you do not want to do it in each and every query.

mongodb://localhost:27017/mydatabase?w=majority&j=true

Talking about the options, write concern accepts three options.

1. w Option.
   The w option specifies the level of acknowledgment required for a write operation. It determines how many replicate sets must acknowledge the write.
   a.w=majority: The operation is acknowledged by a majority of the replica sets, including the primary. This level provides stronger data consistency. This might impact on query performance.
   b.w=0: No acknowledgment is requested. The query is sent to the server, but the client does not wait for any acknowledgment.
   c.w=1: The operation is acknowledged by the primary node. This is the default and provides basic confirmation that the write succeeded on the primary node.
   d. (w=N) Custom values: You can specify a custom number N to indicate the number of nodes that need to acknowledge the write operation for it to be considered successful.

2. j option
The j option specifies whether the write operation should be committed to the journal before acknowledgment. This provides an additional layer of durability. When the option is set to true, the records on the collections can be retrieved from the journal even if the MongoDB server crashes after acknowledgment.
3. wtimeout option
The wtimeout option defines the maximum number of times that a MongoDB write operation should wait. This parameter is used to set a timeout for write operations that are waiting for acknowledgment. The wtimeout parameter is required when you're using a write concern that requires acknowledgments from many nodes and you want to ensure that the write operation doesn't loop indefinitely if acknowledgments are not received within a specified time window.

Improve performance and latency while using the Write Concern strategy.

a. Make sure to use the correct write concern level based on your application and the number of replica sets. Having a lower level of acknowledgment will also affect the data's durability. So, always choose the correct level of acknowledgment your application needs to maintain data consistency.

b. Always ensure to check your write operation performance regularly and use profiling and benchmarking to identify any kind of bottlenecks. This will help you to understand and choose the correct write concern level required for your application.

Contact me for any queries.
E-Mail: sjlouji10@gmail.com
Linkedin: https://www.linkedin.com/in/sjlouji/

sjlouji - Overview

Happy Coding!