text file logging in multi threaded application

Question

4.00/5 (1 vote)

See more:

I’m working towards changing a vb.net program from standard to a multi threaded model.

It runs as a windows service and has no GUI and performs a number of complex reporting and file generation functions on demand for the multiple users simultaneously when requested by the user.
For me to see what the program is doing it’s essential to have good logging which currently logs to a text file when “trace” is turned on. It app captures a few hundred lines in the log for each run.

I see some issues with using a single text file under multi threading.
I thought a database would make it easy to query and follow each thread despite the processing chronology of other threads in-between. I tried MySQL database for storing logs, the problem was logging slowed the app from 3 seconds for a standard run to 34 seconds. That’s a big performance hit.

Now I’m back to the idea of using multiple text files but then uploading it to MySQL afterwards and then dealing with logical presentation later.

I read about people using the below method
TextWriter tw = TextWriter.Synchronized(File.AppendText(filePath));

I’m thinking would that slow it down juggling between threads?
At the moment I’m leaning towards the idea of generating a text file name for each thread based on the GUID assigned for that thread and then later uploading the contents of all the disparate files into MySQL afterwards.

The thinking behind this is that if each thread has its own StreamWriter running it wont be juggling concurrency with any of the other threads wanting to write to a text file.

The objective is keep everything running as fast as possible the whole way.
Does anyone have any input or had experience around this with regard to speed of logging in a multi threading environment?

Posted 30-Mar-11 22:18pm

Martin Stevens

Add a Solution

3 solutions

Add a Solution

Add your solution here

Treat my content as plain text, not as HTML

Preview 0

…

Existing Members

Sign in to your account

...or Join us

Download, Vote, Comment, Publish.

Your Email
Password
Forgot your password?

Your Email
This email is in use. Do you need your password?
Optional Password

I have read and agree to the Terms of Service and Privacy Policy
Please subscribe me to the CodeProject newsletters

When answering a question please:

Read the question carefully.
Understand that English isn't everyone's first language so be lenient of bad spelling and grammar.
If a question is poorly phrased then either ask for clarification, ignore it, or edit the question and fix the problem. Insults are not welcome.
Don't tell someone to read the manual. Chances are they have and don't get it. Provide an answer or move on to the next question.

Let's work to help developers, not make them feel stupid.

This content, along with any associated source code and files, is licensed under The Code Project Open License (CPOL)

Dylan Morley · Accepted Answer · 2011-03-30T22:23:00

I appreciate you've probably already got a codebase, but possibly check out log4net[^]

I use this this very successfully in a high volume, mutil-threaded environment.

http://stackoverflow.com/questions/50213/are-there-any-issues-with-using-log4net-in-a-multi-threaded-environment[^]

http://stackoverflow.com/questions/1519211/multithread-safe-logging[^]

If you're not familiar with it, I suggest you give it a go. No point reinventing the wheel :)

OriginalGriff · Accepted Answer · 2011-03-30T22:24:00

Solution 2

If you are going to hold a stream for each thread anyway, why not use a MemoryStream? Why back it with a text file unless you expect the app to crash out?

Posted 30-Mar-11 22:24pm

OriginalGriff

Comments

Dave Kreskowiak 31-Mar-11 12:19pm

One downside of this would be if the logs generated are extensive. You could run the machine out of RAM. Yes, I speak from experience. no, I didn't come up with that idea - I just witnessed the effects! ;)

OriginalGriff 31-Mar-11 15:03pm

This is true, but since the OP said that a standard run took three seconds, I wasn't too worried!

Sergey Alexandrovich Kryukov · Accepted Answer · 2011-03-31T09:09:00

If you need logging, you better use logging, not home-based techniques.

See my post in response to these Questions:

This is how to make a custom logger (to log to text, to UI or whatever):
MsBuild OutPut to the TextBox on the fly in Windows Application[^]

This is how to log to the System log, but with structure:
How to create event log under a folder[^]
I offered a whole complete wrapper code here.

—SA

text file logging in multi threaded application

3 solutions

Solution 1

Solution 2

Solution 3

Add your solution here

Preview 0