The singular non-modality of MFC modal dialogs

• Download source - 19.4 Kb

Introduction

For the non-programmer, modal dialog boxes are those that refuse to go away till you dismiss them, which is usually done by clicking on the OK or Cancel buttons. For the programmer (the non-VB ones anyway), modal dialog boxes are those that disable their immediate parent window before they are created, and enable their parent window when they are closed. Thus the essence of a modal dialog is that you cannot do anything to the parent window till you close the modal dialog. Modeless dialogs are comparatively more polite and less fussy in that they do not insist on being dismissed to allow you to access their parent windows. I trust that by now, any infinitesimal doubts any of you might have had regarding the modality of a dialog has been absolutely eradicated.

MFC has a class called CDialog which is a CWnd derived class that is specifically used for the creation and display of dialog boxes on screen - both modal and modeless dialogs are supported. Modeless dialogs are created using Create() and you have to destroy them on your own using

DestroyWindow()

and they behave just like any normal modeless window. Modal dialogs are created and shown using the formidable DoModal() method of the CDialog class. And you close a modal dialog by calling EndDialog() or alternatively by calling OnOK() or OnCancel() both of which internally call EndDialog(). The CDialog::EndDialog method will call the EndDialog Win32 API function which is defined in user32.dll. EndDialog (the API function) will not immediately close the dialog, rather it will set a flag which will instruct the message queue to exit the loop, destroy the dialog window and enable the parent window. So far so good; everything seems so peaceful and serene, and the little birds in the tree across the yard are tweeting in a beautiful voice.

The non-modality factor

The Win32 API supports various dialog box related functions which include functions for creating both modeless and modal dialogs. There are the

CreateDialogXXX

set of functions like CreateDialog,

CreateDialogIndirect

etc. which are used to create modeless dialog boxes and the DialogBoxXXX set of functions like DialogBox, DialogBoxIndirect etc. which create modal dialogs. And as mentioned in the previous section there is also the EndDialog function which is used to end modal dialogs and only modal dialogs. Modeless dialogs must be terminated by calling DestroyWindow directly. The basic reason why you should not attempt to use EndDialog on a modeless dialog is that modeless dialogs do not have their own modal message loop nor do they disable their parent window, which basically nullifies the utility value of using EndDialog on them.

Alright, now this is what will give you a solid kick - MFC CDialog based modal dialogs are not modal dialogs. Just in case you did not read that right the first time, here it goes again - MFC CDialog based modal dialogs are not modal dialogs. I would have said it a third time but the fear of being termed a parrot thwarts me from doing so. Open dlgcore.cpp and examine the source code and you'll see that this astonishing statement is true. The MFC CDialog class uses the

CreateDialogIndirect

API function to create a pseudo-modal dialog and if you look up CreateDialogIndirect on MSDN you'll see that it is used to create a modeless dialog. The MFC command routing mechanism uses a combination of message maps and virtual functions to achieve what it does and a true modal dialog will totally wreck this mechanism because then the modal message loop is controlled outside the scope of the MFC command routing machinery. Thus the developers had no choice but to creat a pseudo-modal modeless dialog and then document it very intensely as a modal dialog.

Basically these are the summarized steps that are done when a pseudo-modal dialog is created via CDialog::DoModal -

• A boolean flag bEnableParent is set to FALSE
• If the parent window is enabled, then it is disabled and

  bEnableParent
  

  is set to TRUE
• The dialog box is created using CreateDialogIndirect
• A message pump is maintained using CWnd::RunModalLoop
• Once the modal loop exits (when CDialog::EndDialog is called) the parent window is enabled if bEnableParent is TRUE
• The dialog window is destroyed by calling DestroyWindow
• And DoModal returns the argument passed to

  EndDialog

As you can see the developers have taken great pains to ensure that the pseudo-modal dialogs behave exactly as modal dialogs are supposed to. They have even tried to accommodate for unusual scenarios where there might be two modal dialogs both having the same parent - this is where the bEnableParent comes into play. Thus when the second modal dialog comes up, it will not set bEnableParent to TRUE because the parent window is already disabled. Thus when it is dismissed it will not enable the parent window which is exactly what is needed because another dialog modal to the same parent window is still active on screen.

EndDialog - a tiny flaw

As mentioned a couple of times already in this chapter, modal dialogs are dismissed using EndDialog. Let's see what EndDialog looks like (for those interested, it is defined in dlgcore.cpp)

void CDialog::EndDialog(int nResult)
{
    ASSERT(::IsWindow(m_hWnd));

    if (m_nFlags & (WF_MODALLOOP|WF_CONTINUEMODAL))
        EndModalLoop(nResult);

    ::EndDialog(m_hWnd, nResult);
}

CWnd::EndModalLoop is called to exit the message loop maintained by CWnd::RunModalLoop which is quite fine and then the EndDialog API function is called to terminate the modal dialog. This is also fine and Hey, hey, hey, just wait a cotton picking minute there!!!! The EndDialog API function is to be used only to close modal dialogs, but here we are attempting to use it to close a pseudo-modal dialog which is actually nothing but a modeless dialog disguised to reflect a modality it does not truly possess. [stunned silence...] Well, after that initial shock, let's all relax a bit. After all it's not the end of the world and EndDialog is not really a very harmful function call; it will only attempt to end a non-existent modal message loop and will enable the parent window of the dialog window. The former attempt will fail obviously and the latter attempt will just do something which we would have done anyway on our own, because the moment the pseudo-modal pump exits, DoModal will re-enable the parent window. So, is everything well and good, are the birds chirping peacefully once again?

The bug

Remember all that bEnableParent stuff we discussed a couple of paragraphs earlier, and how I mentioned how this flag is used to make sure that when multiple modal dialogs exist together that have the same parent window, this flag prevents the premature enabling of the parent window when one of the modal dialog siblings are dismissed? Well guess what? The careless programmer at Microsoft who coded this particular function had just rendered all those precautionary measures totally null and void. Because now any MFC modal dialog that is closed using EndDialog (which also means OnOK and OnCancel because those functions will internally call EndDialog) will enable it's parent window irrespective of what value bEnableParent holds. This basically means if you have two or more MFC modal dialogs which have the same parent window, then the moment you dismiss any one of those modal dialogs, all the other modal dialogs will lose their modality because now the parent window has been re-enabled.

Steps to reproduce this bug

• Create an MFC dialog based application
• Add a new dialog to it and associate a class with it called CChildDialog
• Set two timers in the OnInitDialog method of the main dialog :-

SetTimer(1000,1000,NULL);	
SetTimer(2000,2000,NULL);

• Bring up a pseudo-modal dialog each, in the timer handler :-

void CModalDemoDlg::OnTimer(UINT nIDEvent)
{
    KillTimer(nIDEvent);
    CChildDialog dlg(this);
    dlg.DoModal();

    CDialog::OnTimer(nIDEvent);
}

• Run the program and wait 2 seconds by which time you will have the main dialog and two modal child dialogs
• Try to access the main dialog and you will be unable to do so because of the presence of the two modal dialogs
• Now dismiss one of the child dialogs using the OK or the Cancel button
• Now try to access the main dialog and you'll see to your astonishment that you will be able to do so, despite the presence of a modal dialog on screen

The project I have attached was created using VC++.NET 2003 Final Beta and I apologize to all those of you who do not have that version. But following the above mentioned steps shouldn't take you more than a few minutes at most. What is so amazing is that this bug has gone unnoticed through several versions of MFC. I checked as far back as VC++ 6 and that has the exact same problem too. As far as I see it, all someone's got to do is to comment out or delete the call to the EndDialog API call. I am guessing that what happened was this - the MFC developers were so much used to calling the native API equivalents from their wrapper functions (for example they would call the MessageBox API from inside CWnd::Messagebox) that someone must have automatically typed in that line without thinking and the QA guys must also have overlooked this error. And this problem remained mostly unknown because it was a very rare situation for a program to have a multi-modal window architecture where several modal dialog windows have the same parent window.

Work-around

Okay, so until Microsoft corrects this bug, what we could do (other than having to correct the MFC code and recompile MFC) is to override the OK and Cancel button handlers and to use this code instead of the default :-

{
    if (m_nFlags & (WF_MODALLOOP|WF_CONTINUEMODAL))
        EndModalLoop(IDOK); // or IDCANCEL
}

Notice how we are not calling the base class (if we do then ::EndDialog(...) gets called and all our efforts are wasted). If you want to exit the modal dialog in a place outside the OK/Cancel button handlers you should use the same code except that you might want to return a different value (like perhaps IDYES for example).

Conclusion

For all I know I might be the biggest jack-ass in town and there might be a perfectly legitimate reason for this matter, but something tells me that, that is a very remote contingency. By the way I'd like to thank Shog9 for sending me the VC++ 6 version of dlgcore.cpp at a rather late hour of the night (which to him is the equivalent of noon for most of us). I'd also like to explicitly mention here that this article is by no means an attempt to mock the amazing set of Microsoft programmers who developed the MFC library.

Version History

• April 5 2003 - Article first published
• April 7 2003 - Article updated with Work-around to the problem