Click here to Skip to main content
15,921,169 members
Please Sign up or sign in to vote.
1.00/5 (2 votes)
See more:
The crash happens randomly. sometime exe runs for a day and sometimes for few minutes only. Application generates a mini crash dump, I tried to understand it with WinDbg and it gives me a call stack:
ChildEBP RetAddr Args to Child
082fc084 6f7953a2 082fc0ec 76ed6ff0 c17d5131 ntdll!ZwWaitForSingleObject+0x15
WARNING: Stack unwind information not available. Following frames may be wrong.
082fc09c 76f11194 000022a8 00004e20 00000000 mscorwks+0x53a2
082fc0b4 76f11148 000022a8 00004e20 00000000 kernel32!WaitForSingleObjectExImplementation+0x75
082fc0c8 6f99e37d 000022a8 00004e20 bf428143 kernel32!WaitForSingleObject+0x12
082fc0f8 6f99e8c7 000022a8 00004e20 bf42840f mscorwks!LogHelp_LogAssert+0x2ba6e
082fc5b4 6f9a1064 0000229c 000022a8 000022a0 mscorwks!LogHelp_LogAssert+0x2bfb8
082fccf8 6f9a16a3 082fce78 00000004 00000000 mscorwks!LogHelp_LogAssert+0x2e755
082fcd38 6f9df5f2 082fce78 00000004 082fce78 mscorwks!LogHelp_LogAssert+0x2ed94
082fcd5c 6f9e4267 00000000 082fce78 00000004 mscorwks!GetAddrOfContractShutoffFlag+0xa296
082fcdb4 6f9e4313 082fce78 082fce48 76f50303 mscorwks!GetAddrOfContractShutoffFlag+0xef0b
082fcdc0 76f50303 082fce78 bfbf17f2 00000000 mscorwks!GetAddrOfContractShutoffFlag+0xefb7
082fce48 779274ff 082fce78 779273dc 00000000 kernel32!UnhandledExceptionFilter+0x127
082fce4c 082fce78 779273dc 00000000 082ffa04 ntdll!__RtlUserThreadStart+0x62
082ffa04 778e9f45 000f3b30 00c7a4e0 ffffffff 0x82fce78
082ffa1c 00000000 000f3b30 00c7a4e0 00000000 ntdll!_RtlUserThreadStart+0x1b

I'm new to WinDbg, can anyone help me out with this crash, or guide me how to use this call stack addresses to figure out problem areas?
Posted
Updated 20-Oct-14 4:23am
v2
Comments
Sergey Alexandrovich Kryukov 20-Oct-14 10:24am    
Not enough information...
—SA
Rahul Patnaik 21-Oct-14 3:46am    
Application Details:
Application is a COM server component built with multi-threaded apartment.
At a time multiple COM clients can connect and fetch data to/from this application.
It has 100s of thread running/waiting simultaneously.
Application has all needed thread synchronisation and was working with no issues on 32 bit environment
Application built with ATL_NO_MP_HEAP MACRO.
Now as the production environment upgraded to 64 bit.this makes us rebuild the application as 32 bit running on 64 bit(WOW64 application) Now after making it WOW64,we have instability in our application..like it works for a day and crashes suddenly.

Development Environment :
VS2008 3.5 SP1

The application crashing on server having following configuration:
Operation System : Windows Server 2008 R2 SP1
System Manufacturer: Dell Inc.
System Model: PowerEdge R720
System Type: x64-based PC
Processor(s): 2 Processor(s) Installed.
[01]: Intel64 Family 6 Model 45 Stepping 7 GenuineIntel ~3300 Mhz
[02]: Intel64 Family 6 Model 45 Stepping 7 GenuineIntel ~3300 Mhz
Total Physical Memory: 32,723 MB
Available Physical Memory: 27,099 MB
Virtual Memory: Max Size: 65,743 MB
Virtual Memory: Available: 59,872 MB
Virtual Memory: In Use: 5,871 MB
[no name] 20-Oct-14 12:05pm    
When run on 32 bit windows - is that a single core or multi-core processor? The 64 bit machine is likely a multicore processor. Some multi-threaded programs don't work well on multi-core processors. Try setting the process's affinity to a specific core.

http://msdn.microsoft.com/en-us/library/windows/desktop/ms686223%28v=vs.85%29.aspx
Rahul Patnaik 21-Oct-14 3:49am    
Thanks for your reply.

Application Details:
Application is a COM server component built with multi-threaded apartment.
At a time multiple COM clients can connect and fetch data to/from this application.
It has 100s of thread running/waiting simultaneously.
Application has all needed thread synchronisation and was working with no issues on 32 bit environment
Application built with ATL_NO_MP_HEAP MACRO.
Now as the production environment upgraded to 64 bit.this makes us rebuild the application as 32 bit running on 64 bit(WOW64 application) Now after making it WOW64,we have instability in our application..like it works for a day and crashes suddenly.

Development Environment :
VS2008 3.5 SP1

The application crashing on server having following configuration:
Operation System : Windows Server 2008 R2 SP1
System Manufacturer: Dell Inc.
System Model: PowerEdge R720
System Type: x64-based PC
Processor(s): 2 Processor(s) Installed.
[01]: Intel64 Family 6 Model 45 Stepping 7 GenuineIntel ~3300 Mhz
[02]: Intel64 Family 6 Model 45 Stepping 7 GenuineIntel ~3300 Mhz
Total Physical Memory: 32,723 MB
Available Physical Memory: 27,099 MB
Virtual Memory: Max Size: 65,743 MB
Virtual Memory: Available: 59,872 MB
Virtual Memory: In Use: 5,871 MB

With all the above information can you suggest me anything else also, which can help me to solve this crash. And I want to test all suggestion in one go only because it is a production server, and I've limited access to it.

Thanks
Stefan_Lang 22-Oct-14 10:58am    
@access to production server: if the only way to find and fix the bug is using and modifiying the production server, then you should better get a test server instead! You could try a virtual machine with the same environmant as your production server, but if this is related to parallel execution strategy as some are suggesting, then you may not be able to reproduce it on a VM. It might still be a good idea to install such a VM for testing purposes, and much cheaper than an actual server.

1 solution

odds are the crash IS there in 32bit, you just haven't seen it ... the x64 presentation is probably caused by different heap frag/return/initialisation methods or "parallel execution strategy" changes in that OS - i speak from a lot 32->64 migration experience

It's down to debugging - either walk the stack further down (there's none of your code on it) in WinDbg or use Visual Studio remotely to catch the crash - in both situations, ensure you've got MS Symbols loaded - makes the stack trace more useable - Symbol Server[^]

Try turning optimisation off too, specifically for debug, that helps to make the debugging easier
 
Share this answer
 
v2
Comments
Rahul Patnaik 22-Oct-14 2:20am    
Thanks for your reply Barney.
I've tried to debug the application with Symbol server. However, I always get random locations. I even tried tools like gflag and application verifier also, but could not find any help.
As you said "parallel execution strategy" could be the reason for this, is it possible to control its behavior with "SetProcessAffinityMask" function ?
Your valuable inputs will help me a lot, thanks in advance.
[no name] 22-Oct-14 11:45am    
Rahul - If your old 32 bit production server was single core, setting processor affinity worth a try. It is low effort - about as much effort as this reply.
barneyman 23-Oct-14 16:26pm    
as bling says, setting affinity is worth a pop, but it's only obscuring the real issue - 'random crashes' normally relates to bad/wrong/nonexistent thread sychronisations (using pointers [memory/COMitf] after they're deleted etc) - for COM, check your com objects are FreeThreaded properly, for memory, try overriding delete to poison the pointer it gets given - also, understand the lifecycle of memory in the winheap (http://www.codeguru.com/cpp/w-p/win32/tutorials/article.php/c9535/Inside-CRT-Debug-Heap-Management.htm)
Rahul Patnaik 29-Oct-14 2:05am    
Thanks for your replies.
I tried SetProcessAffinityMask to control the crash, and it worked for some extend.
As Barney suggested, I'm revisiting all thread synchronization part again. Right now I'm doing it manually, is there any better way to do it... ?
barneyman 29-Oct-14 19:41pm    
not really - you can try something like intel's Parallel Studio (cost's a lot, but very good) or just prod away at it ... a good method to find race/sync issues is to insert sleeps in code you think is creating issues and force timing - also, remember that in a multicore system things ARE happening concurrently, unlike a single core system, where they PRETEND to be, so guard vector add/dels/iters, flag get/sets etc - you can try CriticalSections as a 'bludgeon' to find roughly where the code is failing drill in with mutex objects

you either love or hate this stuff ... there is no third way - sorry - as an example, i once spent 3 weeks chasing an extremely rare MT-race condition with SQL Server :)

This content, along with any associated source code and files, is licensed under The Code Project Open License (CPOL)



CodeProject, 20 Bay Street, 11th Floor Toronto, Ontario, Canada M5J 2N8 +1 (416) 849-8900