Click here to Skip to main content
15,881,248 members
Please Sign up or sign in to vote.
2.50/5 (2 votes)
See more:
Hi everyone,

to this day i never quite questioned System.IO performance, but i have done some testing and realized the how badly and frustrating is to write a simple text file.

I am using this simple code to check if file is available for use and then just append a line to a text file.

C#
public void _AppendData(string _path, string _text)
{
     lock (locker)
     {
          /* Append request log to master log file */
          FileInfo fi = new FileInfo(_path);

          using (FileStream file = new FileStream(fi.FullName, FileMode.Append, FileAccess.Write, FileShare.Read))
          {
              using (StreamWriter writer = new StreamWriter(file))
                  {
                      writer.Write(_text);
                  }
          }
      }
}


I have created a simple loop to write 10000 lines of text "This is line x." to a single file which took 00:02:36 in total to do. Mind you i have a capable machine here with 32 gigs of RAM and Intel's I7 chip. Now i know that IO depends on the disk drive it self therefore a Solid State Drive would perform differently here.

As this code works quite well for what i need it to do i am happy to leave it as it is, however i am very curious to know if there is a faster way to do the same job for under a minute on standard drive?
Posted
Comments
[no name] 28-Jun-15 20:57pm    
Why not post the actual code you used?
Silver13 28-Jun-15 21:35pm    
Ok, but there is not much to it.
The call is made by pressing the button on the form.

string _path = @"C:\Output.txt";
string _text = string.Empty;

for (int x = 0; x < 10000; x++)
{
_text = "This is line " + x.ToString();
_AppendDataFile(_path, _text);
_text = string.Empty;
}

[no name] 28-Jun-15 21:38pm    
Think about it. That is not "I have created a simple loop to write 10000 lines of text". Look at what your 10000 iterations is doing then you will have your answer.
Silver13 28-Jun-15 22:05pm    
You are right pwasser! It was simpler to have one string _text and append my data to it in a format i want then just do a one call to create and write to file at the end. It went from over 2 minutes to just 2 seconds. Thanks for the nudge, i needed it. Cheers.
[no name] 28-Jun-15 22:14pm    
Very good.

If you must write records as you go, I can think of two changes to your original loop that should significantly improve its performance.

1) Keep the file open, and either pass the StreamWriter into your _AppendData method.

2) Set the buffer property on the StreamWriter to something on the order of 8192 bytes.

Over many years, I have found that 8KB is the sweet spot for buffer size, regardless of programming language or platform. Though it has been several years since I did so, I have performed timed tests, using identical code, varying only the size of the buffer. In every case, performance improved significantly as I increased the buffer from 256 bytes to 8192 bytes. Beyond that, the effect on performance was negligible.
 
Share this answer
 
The problem with performance was in handling the data in the loop. This code has managed to reduce processing from over 2 minutes to just 2 seconds.

C#
string _path = @"C:\Output.txt";
string _text = string.Empty;

for (int x = 0; x < 10000; x++)
{
    _text = _text + "This is line " + x.ToString() + Environment.NewLine;
}

_AppendDataFile(_path, _text);
 
Share this answer
 
Comments
BillWoodruff 13-Oct-15 1:15am    
Use a StringBuilder and it will be faster, and use less memory.

This content, along with any associated source code and files, is licensed under The Code Project Open License (CPOL)



CodeProject, 20 Bay Street, 11th Floor Toronto, Ontario, Canada M5J 2N8 +1 (416) 849-8900