Confirm quit dialog on WM_CLOSE but not for shutdown etc.

If you really want to do it through direct handling of WM_CLOSE and then cancel closing, look at the message description:
http://msdn.microsoft.com/en-us/library/windows/desktop/ms632617%28v=vs.85%29.aspx[^].

As you can see, the message does not have parameters except message ID, so you don't have any information on the cause of the message.

If you wanted to prevent closing unconditionally, you would need in WM_CLOSE message handler, you would need to return TRUE from WinProc, without calling the DefWindowProc function, but if you need to do it on the condition related to the source of the message, you don't receive anything which gives you a clue.

You should do some work-around. Insider the application, devise a custom message like "WM_USER_CLOSING", send it inside the application process, handle it, and actually close the window or not, depending on the user's decision. It won't affect a handling of the message WM_CLOSE sent from external application, just as you wanted. But — please see my comment to the question; probably you can create better application design instead.

—SA