How can you create a crossbrowser modal dialog that returns values

The whole idea is wrong. I mean, it's not just possible or impossible; there is no such concept and cannot be. This is not how things work. A window or a modal dialog is not a function; you have nothing to return from. Moreover, you don't need anything to return from, and you don't have a call.

It's hard to explain, but perhaps you will understand it when you understand the basics better. For this understanding, you don't even need or understand ASP.NET; it's enough to understand just the client-side part of the life cycle, in a single session; how a browser works with HTML and JavaScript. You also need to understand the event-oriented paradigm.

And of course you should never use deprecated showModalDialog. Why do you think it was deprecated? Because the idea was wrong.

Nothing pauses nothing, in contrast to confirm (which is a pretty bad thing; I would not use it in a production-grade product). You have a big event cycle; you manipulate the page, it responds to event. Then you create another window, it has its own cycle, so both windows work in their cycles. You can work with both at the same time. You don't even have modal behavior at all. You can only mimic it (pale imitation, anyway) by disabling all controls on the original page; but why? You really need to clear your understanding of things.

But now, you should better receive the whole idea of window.open. Not only it is inconvenient for the users in most cases, but you also can have two kinds of troubles 1) many users will block this behavior in browser, 2) in general case, you cannot programmatically close this window; you will face this problem if it is opened as a browser's tab when the same browser window already have tabs.

I would strongly advice using a different concept, so called modal popup. Of course, not returns as well. I tried to explain everything in my article Modal Popup From Scratch.

In all cases, even if you use window.open (again, don't do it; you can possibly do it in some special cases, but its not a good idea to use it as a dialog) you idea of pausing, or blocking call, or "return" does not work. The real event-oriented behavior is actually much better than your imaginary behavior.

Here is what you basically do: let's say, you have two states of your HTML/JavaScript application: a "modal" state, or a state with two windows, and a "normal" state. When you make a transition back to "normal" state, you handle some events causing it. For example, the user clicks some button. It can be a special event invoked in all cases when the "modal state" ends. You handle it, and this is the point where you immediately do some action.

See also:
Inversion of control — Wikipedia, the free encyclopedia,
Event-driven programming — Wikipedia, the free encyclopedia.

Very briefly, this is about inversion of control. In your case, let's say you have a function doSomething(yesOrNo). This is wrong way of doing thing:

var yesOrNo = showDialog(); // there is no such thing
doSomething(yesOrNo);
// this is wrong, because the code immediately calls some function;
// but you cannot have the point in your code
// where you can execute this fragment

and write way should be something like

myModalPopup.onModalStateEnds = function() {
   var yesOrNo = myModalPopup.getState();
   doSomething(yesOrNo);
}

In the second fragment, your function doSomething is not called. Instead, this code changes the event handling of the object myModalPopup. Without calling the function immediately, it says: "when you go out of 'modal state', call this function…" Something like that can be done for modal popup, but also in case of a separate browser window (I hate to repeat it third time, but I'll say again: better don't do it.) You would have to create some property in a new window to pass some knowledge on what to do at the end, possibly the event property (a function object).

—SA