According to
MSDN[
^] this error means "Dialog View did not find a record for the dialog.".
I would therefore question the accuracy of your statement that you "successfully added the dialog".
Visual Studio includes the functionality to add limited custom dialogs to setup projects, and I have found them to work well. I would expect them to be sufficient for most applications.