Installshield is awful. And the setup project type bundled with many versions Visual Studio is not good, too; Microsoft is getting rid of it without replacing it with anything else. So, what to do?
My solution is using open-source WiX, the open-source installation toolset:
I recently made a migration to it and am very glad I did. It's not perfect too (and the whole Microsoft Installer and its API is well behind other technologies), but it is much better than anything I ever heard of so far. Actually, if you use WiX, you will understand that the tools mentioned above are actually nearly illegal (technically), they are using their proprietary project files. Only WiX provide excellent compliance with the contemporary MSBuild and related XML-based project standard, so you can use WiX projects using standard MSBuild techniques: customize it, integrate with bigger master projects or solutions, write your own extensions, if you need some, and so on.
Also, it has very good and smooth integration with Visual Studio of different versions.
Finally, one big benefit is: WiX code is non-visual. Everything is explicitly declared in XML code using some official documented XML scheme. Nothing is hidden behind the XML code files.
As to the behavior and features, many advanced features are not comprehensively documented, unfortunately. But communities already collected considerable experience and provide a lot of useful advice. So far, I was able to find solutions to all chores I practically needed. And the set of ready-to-use solutions is wider that that I found in a legacy Setup project. Some actions which required development of custom steps are already embedded in WiX. It includes adjusting Windows Firewall to your application, setting permissions of files and directories, different action depending on the project platform and target platform, and a lot more.
The WiX source code is available, but, in the version I used, it did not compile. This problem was documented. I hope the problem of this product will be fixed with time.