Hi,
In your question i see two parts,
1. the validator for password shows error message whenever postback happens from modal popup.
Ans. Give Validation group to the login button.
Specifying Validation Groups[
^]
2. You have given Target Control of modal popup to login button, because of this, the popup is opening even if the valid password.
Ans. remove target control of modal popup from login button. let the login button fire the server side event, and if you find that, password is expiring then show the modal popup from server side with hidden target control ID.
check below links to show modal popup from server side with hidden target control id:-
Modal Popup Extender server side Show()[
^]
ASP.NET AJAX Control Toolkit: Show a ModalPopup and then do PostBack[
^]
hope it helps