MongoDB – Implement Batch for CRUD operations on Mongo Collections
Today this article will see how to Implement Batch Updates to collections of MongoDB documents programmatically using C# MongoDB driver in a .NET or .NET Core application.
Sometimes if dealing with millions of records, it is not feasible to load them in application memory to perform any CRUD operations.
In such cases, we can implement Batching or Chunk the size of the records in order and perform various CRUD database operations on MongoDB.The idea is to iterate over all data but in small iterations.
Today in this article, we will cover below aspects,
We shall also see the use of the BulkWriteAsync an asynchronous extension method from MongoDriver.
Getting started
Please create any .NET/.NET Core application.
Use MongoDB Driver Nuget package
Please add the MongoDB driver NuGet package using the Nuget Package manager.
PM> Install-Package MongoDB.Driver -Version <>
I shall keep the MongoDB driver interface used here simple enough to concentrate on how to add/update new fields to existing documents in batches (Example – 10k documents).
Please visit for better approaches like using DI or IOC here,
Establish the connection to the Database using the MongoDB driver,
_mongoClient = new MongoClient(connectionString);
var db = _mongoClient.GetDatabase("Book");
collection = db.GetCollection<BsonDocument>("Book");
var bulkUpdatModel = new List<WriteModel<BsonDocument>>();
Implement Batch Logic
Get all the Records in Batches from the given collection.
You can define a batch of any count as required. Example 1k or 10k or 100k etc.
foreach (var records in Batches())
{
foreach (var record in records)
{
var update = Builders<BsonDocument>.Update.Set("DateLastUpdated", DateTime.Now);
var filter = Builders<BsonDocument>.Filter.Eq("_id", (ObjectId)record.GetElement("_id").Value);
var upsertOne = new UpdateOneModel<BsonDocument>(filter, update) { IsUpsert = true };
bulkUpdatModel.Add(upsertOne);
}
await collection.BulkWriteAsync(bulkUpdatModel);
bulkUpdatModel.Clear();
}
Batch logic to fetch the actual records,
Example
I have defined batchCount = 10k
Above logic when executed, 10k records at a time will be updated using BulkWriteAsync asynchronously.
That means if you are dealing with millions of records with sizes in GB or TB such batch or chunk will load batches and perform the operation asynchronously.
For long-running operations, you can very much implement multithreading using the Task library and combine the results for each chunk.
That way the operation will get performed in a quick time and in an efficient way.
That’s all! Happy Coding!
References:
Do you have any comments or ideas or any better suggestions to share?
Please sound off your comments below.
Happy Coding !!
Please bookmark this page and share it with your friends. Please Subscribe to the blog to receive notifications on freshly published(2024) best practices and guidelines for software design and development.
