Click here to Skip to main content
15,886,873 members
Please Sign up or sign in to vote.
1.33/5 (3 votes)
See more:
Hello there,

I have developed an app that does process strings. The only thing it does - takes the string from database processes it and submits the results back. So it processes 1 job (1 string) in about 200ms if we count CPU time and I am counting the cpu time like this:
1. get processor time before job starts
Process.GetCurrentProcess().TotalProcessorTime.TotalMilliseconds

2. I get the same processor time after job is complete and subtract the values.

Now - everything is fine if I start 100 of these executables on a single server. The total cpu consumption of a server (48 core machine) is about 10% and the average CPU time for single job to be processed (average through all 100 processes) is about 200-300ms.

The oddness start to happen when I start 200 of these executables (2 times more so would expect the overall CPU usage of the server to be about 20%). But... The CPU usage starts to ramp up to 100% on the server and the average CPU time for single job to be processed becomes ~2000-4000 ms (10-20 times more).
Any idea what is happening here? can it be that there is flaw in the code? Or is it something with server process management.

The server is Windows Server 2008 R2 standard. But it also seems to be happening on srv 2012 too.
The app is written with .net 4.5 (but I've build it with 4.6.1 to test too - still a prolem with 200 processes)

Thanks in advance for any hints or suggestions.

br,
gii

What I have tried:

checked the c# code of the app there seems to be nothing that would result in CPU leaking. tested on win srv 2008 and 2012.
prayed to god... but did not answer...
Posted
Updated 25-Feb-17 9:50am
Comments
Jon McKee 25-Feb-17 3:56am    
Quick question: What's the CPU usage like when you start 50 compared to 100 compared to 200? Having that low-end 50 could be useful in determining if it's a scaling issue with the code.
MK-Gii 25-Feb-17 8:43am    
the cpu ussage at 50 is pretty low. bumping from 5 to 10 with peaks. overall this processing is low cpu intensive, but for some reasons when I start multiple processes it goes greatly up.
The break point is something about 120- 125. After having that much the cpu starts peaking from ~10-15% to 100 and after some time it's like half of the time stays at 100% CPU ussage.

1 solution

I don't have your code or your server but here are some things to check based on your post and response to my comment:

1) You have quite a lot of (presumably active) processes running. A process is an expensive thing compared to an AppDomain or a thread. When switching between threads the kernel also has to switch address spaces if the new thread is from a different process (highly likely with 200 active processes). In the Task Manager check "View > Show Kernel Times" and see how much of that processor % is being used for the kernel. Consider using AppDomains to see if your problem is resolved. If not, consider a threading (pool) approach. Generally I use the rule "processes < AppDomains < threads". Choose the highest on the list that meets your requirements because the lower you go the more overhead is incurred.

2) Reducing the number of processes would also allow you to bundle up SQL statements and batch execute them. Probably not your issue unless you're running the 200 processes and the DB on the same machine but a nice bonus nonetheless.

3) Debug and determine memory usage, ensure different processes aren't fighting over resources (DB access for example), and just give your code a once-over for good measure. From 5-10% at 50 to 10% at 100 with a threshold of 120-125 until you jump to 100% sounds like you're dealing with overloading some part of the system itself but you never know.

4) Strings can be ugly things to handle en masse. It may simply be that your program is handling them inefficiently and chewing up memory or forcing the GC to run frequently. This[^] is a good basic primer on how strings are handled in C#.
 
Share this answer
 
v2
Comments
MK-Gii 26-Feb-17 2:37am    
Hey.
I've tried the thread approach long time ago but did not quiet worked. I had the following setups: one app with thread pool set to 100 threads and one app with 1 thread. Starting 200 new processes where each has a single thread was way much more faster than having one with 100 threads. The cpu ussage was lower for threaded, but data processing speed was like 10 times slower than the one with 200 processes (1 thread per app)
Jon McKee 26-Feb-17 3:10am    
You probably don't need to set the threads manually. It defaults to the number of cores and ramps up from there. The default ThreadPool though isn't really meant for long-running or heavily-blocking tasks. My suggestion would be something simple and lightweight that spins up your own Threads or you may want to use Tasks. That being said I still think analyzing the system and determining what exactly is causing the problem is the best idea before any major code changes.
MK-Gii 1-Mar-17 1:39am    
By the way- I've been trying to solve but cant seem to find out why the heck thiskernel time is so high (80%).
I've dones this:
1. starter 100 apps - all good. cpu ussage about 5%.
2. started 20 more - all good. cpu ussage about 5-6%. jumpy
3. started 10 more - after 10 secondskernel time peeked to 80% and kept there.

I used performance monitor to watch DPC time, interupts time and context switches. None of these figures changed. Context switches are at about 20.000/s as it was before when having 100 apps running.
Any thoughts where to look more?
Jon McKee 1-Mar-17 2:11am    
Unfortunately that's a tough question to answer without your code and hardware. I don't have experience with a program with that many processes since I avoid that in design.

That being said here's what I'm thinking: your processes rarely block. That's an assumption I'm making based on what I've read. You have at minimum 120+ processes all clawing for processor time. CPUs are better designed for blocking tasks. Running >2x the available cores in constantly active processes is going to guarantee context-switches at every opportunity (probably time-slice timeout since I assume your processes are on the same priority). This is gonna eat up processor time with kernel tasks since it needs to both switch the thread context and process context.

Again, I don't have your setup, but from my point of view the program needs a redesign focusing on minimizing processes using AppDomains and threads while possibly (bonus points) minimizing the need on these with the async-await pattern (TAP) where applicable.

EDIT: Of course all this could be moot. It could be an issue somewhere else like the SQL server. It could be a lot of things. I'd still focus on tackling the scaling issue from a design point of view. Even if you fix this specific issue now, what happens if you need to scale to 300 or 400? Cheers :thumbsup:
MK-Gii 26-Feb-17 3:45am    
Just tested this = you are right. Kernel takes up 80% + of cpu power....

This content, along with any associated source code and files, is licensed under The Code Project Open License (CPOL)



CodeProject, 20 Bay Street, 11th Floor Toronto, Ontario, Canada M5J 2N8 +1 (416) 849-8900