Today in this article we shall see an approach to Split large File in Chunks using File Enumerator.

We will read a large-size file by breaking a file into small chunks of files using a connected approach i.e file enumeration.

This approach can be used in the below scenarios,

• Dealing with big-size files with more than 1 GB.

• The file is readily accessible to Enumerate line by line.

• You know the number of lines, you want to process in each chunk.

In our previous article, we came up with unique logic based on the size of the data*bytes we want to deal with.

Today in this article, we will see a simple approach using a number of Lines.

Let’s use the same sample discussed in the previous article.

We have a Text file with a total of ~150 k records.

I would like to create a file with chunks of 15 K records possibly in each file.

Expected result

Below each file approximately contain 15K records.

We can achieve the above result by using the below set of codes.

static void Main(string[] args)
        {

            int fileSplitSize = 15000;

            using (var fileLineReader = File.ReadLines("file-input-thecodebuzz.txt").GetEnumerator())
            {
                bool IsFileReadOn = true;
                for (int chunk = 0; IsFileReadOn; chunk++)
                {
                    IsFileReadOn = WriteChunkToFile(fileLineReader, fileSplitSize, chunk);
                }
            }
        }

In the above logic File.ReadLines(“file-input-thecodebuzz.txt”).GetEnumerator()) read the lines of the file line by line using an enumerator until the end of the file.

It also lets you control the flow and you can perform a write-to file once the chunk size matches with the current line while file processing

Below is the chunking logic,

So overall when we break the file into multiple files based on a number of records, we want to make sure data is not truncated.

Once after successful reading, You shall see a total of 10 files of 15k records generated in the selected output folder, and more importantly, you don’t lose any line of records or data from the file.

If you are interested to learn another approach using Chunk sizing, please see the below article,

References:

• Read a Large File in Chunks in C# -Approach II

Do you have any comments or ideas or any better suggestions to share?

Please sound off your comments below.

Happy Coding !!

Please bookmark this page and share it with your friends. Please Subscribe to the blog to receive notifications on freshly published (2025) best practices and guidelines for software design and development.

The post Split large File in Chunks using File Enumerator- Approach 3 first appeared on TheCodeBuzz.

Today in this article, we will see how to Merge multiple files into one using C# .NET and create a full file appended with content from all files.

We will cover below a few approaches to achieve file merge functionality.

This could be a legitimate need for many scenarios.

We will see a very generic approach that can be used for any file type like TEXT or DAT file etc.

We will see the basic approach and also use the FileStreams approach to deal with a stream of bytes to read and write the content.

Let’s get all files from a given Directory using the below code.

  var inputFiles = Directory.GetFiles(@"C:\Test\Chunks");

We have a total of 10 files in the example path

Approach 1 – Using File.ReadAlltext and File.AppendAllText

This approach is pretty simple and very handy if you are dealing with a limited size of files (maybe a few MBs).

var inputFiles = Directory.GetFiles(@"C:\Test\Chunks");
                foreach (var file in inputFiles)
                {
                    //Read all the lines from file
                    var contents = File.ReadAllText(filePath);

                    //Open file and Append to the end of file 
                    File.AppendAllText("FullFile.txt", contents, Encoding.Default);
                }


            

In the above approach, we are copying all the content of the file and append it to a new file.

Approach 2 – Using FileStreams and bytes

The below approach is also useful if you are dealing with a merge of big files (size in GBs) into a single file.

You may have bigger files containing GB of data like 1GB, 2GB, or more.

We will define chunk size so that we can operate on any big-size file in chunks.

It will let you process the specified amount of data at a time and lets you create a single file with combined data.

This approach also provides you with better control over file processing and access.

Example:

Chunk size is defined as 1 MB

const int chunkSize = 1024 * 1024; // 1MB

 


var inputFiles = Directory.GetFiles(@"C:\Test\Chunks");
            using (var output = File.Create("FullFile.txt"))
            {
                foreach (var file in inputFiles)
                {
                    using (var input = File.OpenRead(file))
                    {
                        var buffer = new byte[chunkSize];
                        int bytesRead;
                        while ((bytesRead = input.Read(buffer, 0, buffer.Length)) > 0)
                        {
                            output.Write(buffer, 0, bytesRead);
                        }
                    }
                }
            }

Above we are taking the highest care to read files with no sharing of file access while reading the content of the file.

File property i.e FileShare.None is useful to avoid accidental changes to input files while files is being read at the same time.

File property FileAccess.Read – provides the Read access to the file. Data can be read from the file.

Please see below the full file generated with a total of approximately 10.3 MB after the merge as below,

Please see below best practices while performing read and write on Files,

• How to use Files- Coding Best practices

Do you have any comments or ideas or any better suggestions to share?

Please sound off your comments below.

Happy Coding !!

Please bookmark this page and share it with your friends. Please Subscribe to the blog to receive notifications on freshly published (2025) best practices and guidelines for software design and development.

The post How to Merge Multiple Files into one using C# .NET first appeared on TheCodeBuzz.

Today in this article, we will learn How to use Files- Coding Best practices to be followed.

Handling files properly (being an unmanaged resource) is important and since all file operation uses OS-level operation behind the picture, it’s important to take care of the file handles including read operation or file create or delete operation.

Today in this article, we will cover below aspects,

We will use generic guidelines and will see a few examples using C# code. However, you can use the discussed approaches for any coding language.

Understand File Enumeration – File Best practices

While using FileStream class, you get the ability to specify the FileShare access type another process that can have while dealing with File Processing.

Example of FileShare Enumeration,

Example

     using (FileStream stream = File.Open(filePath,FileMode.OpenOrCreate,               FileAccess.ReadWrite, FileShare.ReadWrite))
            {


                /// You logic here////
            }

There are 5 enumerations that can be used which define the file access level.

FileShare.None

• Allows subsequent opening of the file for reading. If this flag is not specified,
• Any request to open the file for reading (by this process or another process) will fail until the file is closed. However, even if this flag is specified,
• Additional permissions might still be needed to access the file.

FileShare.Read

• Allows subsequent opening of the file for Reading. If this flag is not specified.

• Any request to open the file for writing (by this process or another process will fail until the file is closed. However, even if this flag is specified

• Additional permissions might still be needed to access the file.

FileShare .Write

• Allows subsequent opening of the file for reading or writing.

• If this flag is not specified, any request to open the file for reading or writing (by this processor or another process) will fail until the file is closed.

• However, even if this flag is specified, additional permissions might still be needed to access the file

FileShare .Delete

Allows subsequent deleting of a file.

Proper use of FileShare enumeration will help you resolve the most common issues while dealing with files.

Implements IDisposable for Files

Dealing with File instances or handle is important.

Most file methods are actually core OS-level methods and their associated behavior is determined in the run time.

These file-level classes come under unmanaged resources and hence implement an IDisposable interface in C# language.

Other languages will also have their own way of managing file objects

It’s always safe to use the “using” statement while dealing with files instead of creating and managing the instance ourselves.

Wrap all your code inside “using” statements. it opens the file for reading and writes and closes the file handle once the code scope goes out of using statement.

Implement a Retry pattern for File processing

This is not mandatory but always come in handy while dealing with file exceptions or temporary glitch accessing the file.

The retry mechanism is a very basic and common pattern for I/O operations.

Example :

Retry(() =>PerformFileProcessing(filePath), 3);

Here method PerformFileProcessing performs all file read or write operations.

Example – Simple retry pattern

Implement Thread Synchronization for File -Lock and Unlock

The file can be accessed by multiple threads within the application or by external process threads.

If you have multiple threads attempting to access the same file, consider using a Synchronization mechanism using Lock or another threading synchronization mechanism.

Example

 public class WiteToFileUsingLock
    {
        public string Filepath { get; set; }
        private static object lockerFile = new Object();

        public void WriteToFile(StringBuilder text)
        {
            lock (lockerFile)
            {
                using (FileStream file = new FileStream(Filepath, FileMode.Append, FileAccess.Write, FileShare.Read))
                using (StreamWriter writer = new StreamWriter(file, Encoding.Unicode))
                {
                    writer.Write(text.ToString());
                }
            }

        }
    }

Watch if the File is in use by another process

Please make sure to verify the file is not opened by another process. If Yes make sure to handle the scenario gracefully. You could retry or exit the operation based on the need.

This could be any specific process on the system or program opening the file at the time when your program or application is already using it. Closing or stopping the other process may resolve the access issue.

There are multiple ways to deal with File Open, Creating, Reading, or writing operations using C# code. Its important to how different methods are being used in the code.

A few examples of File related methods but not limited are as below,

File.Create

File.Open

File.ReadAllText()

File.WriteAllText()

File.ReadAllLines()

File.WriteAllLines()

The above method allows performing Read or writing conveniently.

Example

          
            // Create the file, or overwrite if the file exists.

            using (FileStream fs = File.Create(path))
            {
                byte[] info = new UTF8Encoding(true).GetBytes("This is test file.");

                // Add details to the file.
                fs.Write(info, 0, info.Length);
            }

          

Note: When using File.Create, please note FileShare value of None is used as default behavior i.e no other process or code can access the file until the original file handle is closed.

Do you have any comments or ideas or any better suggestions to share?

Please sound off your comments below.

Happy Coding !!

Please bookmark this page and share it with your friends. Please Subscribe to the blog to receive notifications on freshly published (2025) best practices and guidelines for software design and development.

The post How to use Files- Coding Best practices first appeared on TheCodeBuzz.