I also encountered the same issue while developing using JQuery dialog.
what i done is
First i mentioned the button with client id as button in a script tag in the page ,
then i used Jquery ajax call back in client click of this button to do the server operations.
$('#<%=btnClose.ClientID%>').button().click(function () {
//Do ajax call back here
});
by doing this you can see that the style of button will automatically change(depends on your css)
OR else place another button outside the dialog and then invoke the button click or__dopostback from the modal dialog button client click.(not a right practice)
As per my experience you can only do client operations with buttons placed Jquery dialog