I had the same issue, especially when debugging apps with many threads.
It was caused by the feature "Show threads in source".
See the following link for details:
[Code Project: Show threads in source][1]
[Visual Studio Single Step Performance Fixes][2]
After disabling this feature, problem has been fixed.
[1]: http:
[2]: http: