What to ask: Memory (private working set)

Question

1.00/5 (1 vote)

See more:

Hi guys,
I was wondering if there is any impact of the Memory (private working set) towards the performance of the windows services.

I developed a Windows Service program in C#.
Basically what it does is, reading an unstructured text file and then converting it to a structured text file.
File size ranging 1GB to 2GB per file, in multi-threaded.

As what I noted, the higher the Memory (private working set) from the Task manager; it goes, the poorer the program to read and write file.

I already troubleshooting and recode the program in any ways. Still no luck
Any comments to the above, very much appreciated.

Thanks

Posted 21-May-15 1:52am

Mercurius84

Add a Solution

Comments

Tomas Takac 21-May-15 8:12am

Are you reading the whole file into memory for processing?

Mercurius84 23-May-15 3:59am

nope. it is
chunk by chunk
I noticed i have no issue if reading one file.
but if on multi threads my window service program screw. poor performance.
I tried writing similar code on vbs
mean while my window service executive these vbs in multi thread.
suprisingly result is so way greater than before

im looking into that solution

3 solutions

Add a Solution

Add your solution here

Treat my content as plain text, not as HTML

Preview 0

…

Existing Members

Sign in to your account

...or Join us

Download, Vote, Comment, Publish.

Your Email
Password
Forgot your password?

Your Email
This email is in use. Do you need your password?
Optional Password

I have read and agree to the Terms of Service and Privacy Policy
Please subscribe me to the CodeProject newsletters

When answering a question please:

Read the question carefully.
Understand that English isn't everyone's first language so be lenient of bad spelling and grammar.
If a question is poorly phrased then either ask for clarification, ignore it, or edit the question and fix the problem. Insults are not welcome.
Don't tell someone to read the manual. Chances are they have and don't get it. Provide an answer or move on to the next question.

Let's work to help developers, not make them feel stupid.

This content, along with any associated source code and files, is licensed under The Code Project Open License (CPOL)

Abhipal Singh · Answer 1 · 2015-05-21T02:29:00

Solution 2

I would suggest you to use StreamReader and StreamWriter provided by .NET

All you need to do is read your input file line by line, format it and redirect it to the output file.

This will enable you to use your memory efficiently and also, it will provide you better performance as you are dealing with large file sizes.

However, you might need to change your algorithm of formatting to work on line-by-line scenario.

Posted 21-May-15 2:29am

Abhipal Singh

Comments

Mercurius84 21-May-15 8:35am

My machine is have 150GB of memory it should sufficient enough to perform heavy duty tasks.

However, my application already using the said function

anyway to increase the upper limit or any suggested solution?
Thanks

Richard Deeming 21-May-15 9:11am

Is your service running as 64-bit or 32-bit?

Mercurius84 21-May-15 9:33am

it is running on 64 bit
windows server 2012

Richard Deeming 21-May-15 9:38am

But is the process running as a 64-bit process?

Check in the Task Manager - does it have "(32 bit)" after the name in the "Processes" view?

Mercurius84 21-May-15 10:29am

nope at all

Abhipal Singh 21-May-15 8:45am

Wow, Does your machine have 150 GB of RAM?!!

Are you using Streaming to read files?

Mercurius84 21-May-15 8:53am

Yeah...she is a monster. Haha

This code:

using (FileStream fs = File.Open(p_strFilePath, FileMode.Open, FileAccess.Read, FileShare.ReadWrite))
using (BufferedStream bs = new BufferedStream(fs))
using (StreamReader sr = new StreamReader(bs))
{
string m_strLine = "";
int cur_count = 0;
string secondLine = "";
while ((m_strLine = sr.ReadLine()) != null)
{
secondLine = (cur_count < allLines.Count) ? allLines[cur_count] : "";
}
}

what we did here is
a file (unstructure) -> process -> into (structure) file as in pdf.
we have pdf merger if the content of file belongs to the same location.

Summary:
Inside the file, consists pages of customer information from different location.
we split this information to its' own entity.
Then we group them by location as in pdf

Abhipal Singh 21-May-15 9:18am

Haha. Nice! and thanks for explaining things clearly:)

Even if you have that much RAM, I don't think the .NET framework will be able to use it efficiently. However, that is a different story.

I am trying to understand where are you getting the bottleneck. The code you gave should work fine as you are reading lines and not the entire file in one go.

However, I think you are not writing the files in line-by-line fashion, which will ultimatly create the dreaded bottleneck.

Correct me if I am understanding the issue wrong?

Mercurius84 21-May-15 9:37am

actually the file generate by our backend system. It has formating code on it. example:
carriage control characters
http://en.m.wikipedia.org/wiki/ASA_carriage_control_characters

im afraid i have to answer you yes. reading line by line.

Abhipal Singh 21-May-15 9:41am

I can understand you are reading it line by line..
However, I am asking if you are 'writing' your output file line by line or not?

Also, have you checked for memory leaks?

Mercurius84 21-May-15 10:28am

sorry, it was writing
line by line
not yet
i need support ppl to install memory checker on that server.

Abhipal Singh 21-May-15 13:52pm

Nice!

Did you checked the memory growth on Perfmon counters..?
If you see it as a growing trend and the memory is not getting reclaimed even after your processing is complete. Then it is definatly Memory Leak.

Do let us know the root cause of the issue when you find it :)

Mercurius84 22-May-15 1:12am

Done the checking. All memory reclaimed after each task completed.
It seems, once running the read file program. It allocates approximate amount of memory size.
for example: file size 2 gb, It allocates approximate 2gb.

I tried with a restart the service reading that 2gb file.
It went fast with the process that I mentioned earlier.
e.g. at first I see the counter from 1...in a blink it went to 15
then once the private memory reaches about 1gb or above...getting slower.
e.g. counter number 1....a blink...number 2...another blink number 3.

Hope that you get what I mean..
Thanks

Mehdi Gholam · Answer 2 · 2015-05-21T02:16:00

You are limited to the amount of available physical memory of your system and OS, if you reach the upper limit the OS will swap out unused memory to disk which will make your app and system slow down.

If you can chunk load and process your files then do that instead of loading everything into memory first.

Mercurius84 · Answer 3 · 2015-05-25T19:50:00

Solution 3

I have found the problem.
It is due to StringBuilder that is collection the strings as the bottom of the streamreader code.

After removing it, the speed is superb...

Many thanks guy for helping :)

Posted 25-May-15 19:50pm