Fastest System.IO method?

Question

2.50/5 (2 votes)

See more:

Hi everyone,

to this day i never quite questioned System.IO performance, but i have done some testing and realized the how badly and frustrating is to write a simple text file.

I am using this simple code to check if file is available for use and then just append a line to a text file.

C#

public void _AppendData(string _path, string _text)
{
     lock (locker)
     {
          /* Append request log to master log file */
          FileInfo fi = new FileInfo(_path);

          using (FileStream file = new FileStream(fi.FullName, FileMode.Append, FileAccess.Write, FileShare.Read))
          {
              using (StreamWriter writer = new StreamWriter(file))
                  {
                      writer.Write(_text);
                  }
          }
      }
}

I have created a simple loop to write 10000 lines of text "This is line x." to a single file which took 00:02:36 in total to do. Mind you i have a capable machine here with 32 gigs of RAM and Intel's I7 chip. Now i know that IO depends on the disk drive it self therefore a Solid State Drive would perform differently here.

As this code works quite well for what i need it to do i am happy to leave it as it is, however i am very curious to know if there is a faster way to do the same job for under a minute on standard drive?

Posted 28-Jun-15 14:42pm

Silver13

Add a Solution

Comments

[no name] 28-Jun-15 20:57pm

Why not post the actual code you used?

Silver13 28-Jun-15 21:35pm

Ok, but there is not much to it.
The call is made by pressing the button on the form.

string _path = @"C:\Output.txt";
string _text = string.Empty;

for (int x = 0; x < 10000; x++)
{
_text = "This is line " + x.ToString();
_AppendDataFile(_path, _text);
_text = string.Empty;
}

[no name] 28-Jun-15 21:38pm

Think about it. That is not "I have created a simple loop to write 10000 lines of text". Look at what your 10000 iterations is doing then you will have your answer.

Silver13 28-Jun-15 22:05pm

You are right pwasser! It was simpler to have one string _text and append my data to it in a format i want then just do a one call to create and write to file at the end. It went from over 2 minutes to just 2 seconds. Thanks for the nudge, i needed it. Cheers.

[no name] 28-Jun-15 22:14pm

Very good.

2 solutions

Add a Solution

Add your solution here

Treat my content as plain text, not as HTML

Preview 0

…

Existing Members

Sign in to your account

...or Join us

Download, Vote, Comment, Publish.

Your Email
Password
Forgot your password?

Your Email
This email is in use. Do you need your password?
Optional Password

I have read and agree to the Terms of Service and Privacy Policy
Please subscribe me to the CodeProject newsletters

When answering a question please:

Read the question carefully.
Understand that English isn't everyone's first language so be lenient of bad spelling and grammar.
If a question is poorly phrased then either ask for clarification, ignore it, or edit the question and fix the problem. Insults are not welcome.
Don't tell someone to read the manual. Chances are they have and don't get it. Provide an answer or move on to the next question.

Let's work to help developers, not make them feel stupid.

This content, along with any associated source code and files, is licensed under The Code Project Open License (CPOL)

David A. Gray · Accepted Answer · 2015-06-28T16:38:00

If you must write records as you go, I can think of two changes to your original loop that should significantly improve its performance.

1) Keep the file open, and either pass the StreamWriter into your _AppendData method.

2) Set the buffer property on the StreamWriter to something on the order of 8192 bytes.

Over many years, I have found that 8KB is the sweet spot for buffer size, regardless of programming language or platform. Though it has been several years since I did so, I have performed timed tests, using identical code, varying only the size of the buffer. In every case, performance improved significantly as I increased the buffer from 256 bytes to 8192 bytes. Beyond that, the effect on performance was negligible.

Silver13 · Accepted Answer · 2015-06-28T16:21:00

Solution 1

The problem with performance was in handling the data in the loop. This code has managed to reduce processing from over 2 minutes to just 2 seconds.

C#

string _path = @"C:\Output.txt";
string _text = string.Empty;

for (int x = 0; x < 10000; x++)
{
    _text = _text + "This is line " + x.ToString() + Environment.NewLine;
}

_AppendDataFile(_path, _text);

Posted 28-Jun-15 16:21pm

Silver13

Comments

BillWoodruff 13-Oct-15 1:15am

Use a StringBuilder and it will be faster, and use less memory.