Click here to Skip to main content
16,021,765 members
Articles / Programming Languages / C#

Create custom dialogs for use in your Visual Studio Setup projects

Rate me:
Please Sign up or sign in to vote.
4.90/5 (138 votes)
20 Jun 2007CPOL21 min read 1.1M   10.1K   337   248
Create custom dialogs for use in your Visual Studio Setup projects

STOP (12/6/2016)

This is a plea from the author to stop trying to do this. This article is now over 9 years old and nowadays there are far, far, far better ways to create a customised windows installer. Even if your back is against the wall, like mine was 9 years ago, just say no to whoever suggested following this article and tell them even the author of the article thinks that this way only leads to madness.

May I instead recommend the WIX Toolkit. It integrates with Visual Studio and your build environment and frankly I wouldn't use anything else if I needed to create a windows installer.

Now if you want to read some crazy stuff that I did in 2007 and that really you can be too clever for your own good, please continue and read the article below but only for the purpose of your own enjoyment.

PS. In case you haven't worked it out I do not intend on supporting this way of customising installers anymore.

Introduction

The Visual Studio setup projects that come installed as part of Visual Studio are handy for knocking out quick-and-easy installers, but they are a bit limited in scope. Two main drawbacks are:

  • You have only a limited number of dialogs to choose from, which can easily run out on a relatively simple install. Say, for example, that you have a need for a further set of textboxes for a piece of information that contextually belongs on its own and does not look correct if tagged onto one of the other dialogs.
  • The information you can enter is limited. Have you ever needed a dialog with 5 radio buttons? What about a pick-list of options?

Background

<ramble>A bit of a ramble going on here so feel free to skip this bit.</ramble>

Over the years, I've started to become a bit of a build monkey, i.e. taking all the code produced by the developers and getting it to build using automated builds, running unit tests as part of the build and finally getting it to deploy and run on a machine that has not had a developer (or Visual Studio) near it. Plus, doing whatever development work comes my way as well. One of the things I have learned over time is that the person who knows most about how the code should work when running on another machine is the developer that wrote the code in the first place. So, how do we get the developers to tell us what we need to know? Let's examine the behaviour of 90% of developers and the environment that they work in.

1) Developers hate writing documentation. They will dodge the topic when raised and even, on some occasions, fix bugs in old VB code which have suddenly become urgent after many years of being blissfully ignored.

2) Buying new software on top of the initial Visual Studio purchase is a nightmare of justification and licensing. Justifying the purchase of installer software in case the setup needs something that can't be met with the built-in setup projects takes time and paperwork. Uh-oh, looks a bit like documentation and so it is avoided.

3) Some installer software tools resort to scripting languages to customise the install and do other actions. Most developers don't like script-based languages and, because some installer tools use a custom script language, it also means a new learning curve. So, calls for a specialist are made from developers and that new individual then has to deal with point 1 in order to get their work done. You could use free installer tools e.g. the WIX Toolset, but there are not many people out there that know this tool. Also, the learning curve is fairly steep, as you have to know a lot about installers to understand what is happening when it all goes wrong.

4) They have Visual Studio and it comes with the ability to do setup projects. They come out of the tin and they can use .NET Installer classes to customise their install.

Because of the last item, point 2 is overcome due to the ability of developers to build and test the installer locally. They can also integrate it into their solution and use drag and drop to put files in all the right places. Should the install need a bit of customisation, we can use .NET Installer-based classes to modify our configuration files with information gathered from the simple dialogs. This solves the issues raised by points 1 and 3.

Okay, some installer zealots would say that the use of scripts and .NET Installer-classes is bad ju-ju and should be avoided. Instead we should all use proper custom actions, i.e. native DLLs. However, we live in the real world, folks, and we do what we can in the limited time given to us. The problem in going with point 4 is that it is usually at the end of the project when we discover we need that extra customisation. By then, it is too late to buy new tools and get to grips with them before the product is shipped.

That leaves us with a problem: how do we create that extra dialog?

Visual studio dialogs

There is very limited documentation about the installer dialogs. What does exist seems to be related to language globalization/customisation. However, what we learn is that the installer dialogs are in the folder %ProgramFiles%\Microsoft Visual Studio 8\Common7\Tools\Deployment\VsdDialogs and that these files can be edited using the Orca tool that ships as part of the Windows Installer SDK. By viewing the dialogs in those folders using the Orca tool, it is easy to determine which .wid file belongs to which entry in the Add Dialog screen in Visual Studio. Before we start editing these files it is recommended that a backup is made of these files.

Things that are noticed straightaway:

  • There are lots of well-known tables that are documented on the MSDN website and there are some that are custom, i.e. those beginning with Module.
  • All of the dialogs have a unique name. See the tables Control, ControlCondition, ControlEvent and Dialog, amongst others.
  • All of the custom actions have a unique name. See the CustomAction table.
  • All of the properties have a unique name. See the Property table.

This unique naming makes sense when you think that we could be adding them all into one installer or, as we can suspect, merged into one installer. This is why there may be a limited number of pre-made dialogs.

The custom tables

ModuleComponents

Purpose unknown.

ModuleConfiguration

Looks like what is read by Visual Studio and presented in the property window.

ModuleDialog

This looks like the information that is displayed in the Add Dialog section of Visual Studio. There is also an Attributes field here that I think is used to control when the dialogs appear. For example, all of the Admin dialogs have an attribute that is greater than or equal to 8192 -- i.e. the 13th bit is set -- whereas for normal dialogs it is the 12th bit. Certain dialogs only show up in web setup projects; 5th bit (or 7th bit in admin dialogs) is set. Other dialogs also have some unique setting. For instance, for both admin and normal dialogs the finished dialog has the 3rd bit set. We can therefore infer that the Attributes field is used to both control when the dialogs show up and perhaps control positioning in the UI. It would therefore make sense that if you make a new custom dialog for use in a similar situation as another dialog that already exists, then using the same attributes as that dialog would be the approach to take.

Screenshot - customdialogsvsd1.jpg

ModuleIgnoreTable

Purpose unknown.

ModuleInstallExecuteSequence

These actions look like they would be merged into the InstallExecuteSequence of the main installer.

ModuleInstallUISequence

These actions look they would be merged into the InstallUISequence of the main installer.

ModuleSignature

Contains an internal name perhaps; uses a GUID-like string as part of a name and a language code.

ModuleSubstitution

Looks like a mapping table of values from the Visual Studio environment; defined in ModuleConfiguration to the standard tables within the installer before or during the moment they are merged into one installer.

First custom dialog

Thus armed, we are ready to create our first custom dialog Textboxes (D) which are amazingly like Textboxes (A/B/C).

1) First, we make a copy of Textboxes (A) or, as it is known on the disk, VsdCustomText1Dlg.wid in the VsdDialogs/0 folder.

2) Open in Orca and change all occurrences of CustomTextA to CustomTextD in the Dialog, Control, ControlCondition and ControlEvent tables. See the Dialog, Condition, and Argument columns where applicable.

3) Make the custom action names unique in the CustomAction table.

4) Make the property names unique in the Property table.

5) Change the property names in the DefaultValue column in the ModuleConfiguration table to match those of our new properties.

6) Change the display name in ModuleDialog to match what we want to present in the UI, should we succeed in getting it to appear. Don't forget to change the name of our dialog as well.

7) Change the names of the custom actions to match the new names of our custom actions in the ModuleInstallExecuteSequence and ModuleInstallUISequence tables.

8) Generate a new GUID (I use the guidgen tool) and format it so that dashes become underscores. Also modify the ModuleID table.

9) Finally, modify the Row column in the ModuleSubstitution table to refer to our new dialog and associated custom actions.

Start Visual Studio and see if it appears in the Add Dialog window of a set-up project. Nope. Hang on, maybe it is a bit more advanced than that. Perhaps it is trying to present the one in my default language in Visual Studio, which happens to be English (United States) and has a LCID of 1033. Make a copy in the VsdDialogs/1033 folder, changing the Language field in the ModuleSignature table to 1033 for good measure. Restart VisualStudio and have another look. Success!

Screenshot - customdialogsvsd2.jpg

But does it work and will it play nicely with the other dialogs? We can only hope that the developers of this bit of Visual Studio do not care what dialogs are available so long as we keep to that same pattern (naming conventions and suchlike). Let's create a test project, add all four of our text box dialogs, play with a few settings, shuffle the order, compile and test. It's looking good. Add a quick and dirty CA to display a few properties. Another success. So now we know how to make a copy of an existing dialog and we can easily repeat that process should the need arise.

Five radio buttons

Is it possible to extend the dialogs to present more information than they have before? Let's try making a 5-button dialog.

1) Create a copy of VsdCustom4ButtonDlg.wid and do the modifications that we did before to make it unique.

2) Adding a new radio button is relatively simple in that we only need to update the RadioButton table. Using Orca, we can copy and paste a row and then edit that row such that its order is 5 and it has a Y-value of 96. NOTE: we need to make sure that the containing control in the Control table is also big enough.

3) We can check the spacing of controls and general layout of our dialog by using the Dialog Preview feature that can be found in the Tools menu.

4) Using Orca, edit the ModuleConfiguration table so that we can manage the Label and Value of our new button in Visual Studio. Let's duplicate rows Button4Label and Button4Value to make Button5Label and Button5Value, respectively. The fields with the hashes (#) look like they may be referring to resource text strings; we can hopefully ignore them for now.

5) Again using Orca, edit the ModuleSubstitution table to add 3 new rows that will set the Property, Text and Value properties of our new radio button. You will have to change the Row field to BUTTON5;5 before you can change the Table field back to RadioButton.

6) Save to disk and make a copy to put into the VsdDialogs/1033 folder.

Restart Visual Studio and open up the test set-up project. A partial success, it seems we can add the new dialog and we can see 5 radio buttons when we run the installer. However, the properties for the fifth button don't seem to be appearing in the Properties window of Visual Studio. What about the # values? Could the problem be that the code uses them to generate the descriptors when building the collection for the property grid and thus may filter out duplicates? Let's edit the ModuleConfiguration table and the changed the fields with # values in them to what we want to display. Hopefully the developers at Microsoft have written the code such that if it did not find the resource it would fall back and display what it sees instead. Save to disk, make a copy in the VsdDialogs/1033 folder, restart Visual Studio and try again. Success!

Screenshot - customdialogsvsd3.jpg

New 5 radio button dialog: notice that the description is for the 4 radio button dialog. Originally I didn't know where this description entry was, but thanks to a tip from Grump we now know that the information can be edited using Orca under the View | Summary Information menu option.

Screenshot - custom_description.jpg

We can also see that this dialog has a GUID as a package code and that when we investigate the original dialogs, the value of this GUID matches the GUID in the ModuleSignature table. It does not appear that this GUID as edited by this dialog has any effect on the creation of installers, but we will maintain the mapping just in case.

Screenshot - customdialogsvsd4.jpg

The figure above displays the property grid, showing the extra properties and the hard-coded labels. Now we know how to add new controls and extend the configuration of the dialogs that we can add to our installers.

Controlling the dialog with custom actions

A few of the controls on the dialog can be managed via the existing tables, e.g. enabling buttons. But if we want a finer control, we may need to resort to a custom action (CA). For this example, we'll create an update/upgrade to the License Agreement dialog screen. We'll incorporate a custom action that will check that the user has made an attempt to read the licence by checking that the window has been scrolled to the bottom. When this has happened, a property called LicenseViewed will be created and set to "1." NOTE: I am using a version of the IsLicenceViewed CA by Kallely L. Sajan, found here. I have only made a slight modification to the CA such that it gets the name of the required window from a property within the installer, i.e. make it more generic.

1) Compile, if you can, the LicenseViewed C++ project. Otherwise, you may use the pre-built assembly. The project requires that you have the Platform SDK installed.

2) Make a copy of VsdLicenseDlg.wid that is in the VsdDialogs/0 folder. Name it VsdLicenceViewedDlg.wid.

3) Open the new file with the Orca tool and do the modifications that we did before to make it unique. In this example, I renamed the dialog to EulaFormA and used a name of Licence Agreement (Viewed).

4) In order to add our new DLL to the installer, we need to add 2 tables: Binary and CustomAction. Luckily, these are well known (standard) tables and Orca can create them without issue. Using the menu, select Tables->Add Tables. Select Binary and CustomAction and finally click OK.

5) Select the Binary table and add a new row. Enter LicenseCA for the name and browse to LicenseViewed.dll for the data. Our CA assembly is now stored.

6) Select the CustomAction table and add a new row. Enter the following information, noting that the CA is a type-1 custom action and is one of the most common types:

 

Action CheckLicenseViewed
Type 1
Source LicenseCA
Target CheckLicenseViewed

The source is the name of the assembly as stored in the Binary table. The Target is the exported function that is within that assembly.

7) Select the ControlEvent table and add a new row. Fill it with the following information:

Dialog EulaFormA (or whatever you have renamed it as)
Control LicenseText (this is the control that displays the license)
Event DoAction
Argument CheckLicenseViewed (the CA)
Condition 1

This will force our CA to be executed when the scroll bar is moved. Not very elegant, but it is simple.

8) Select the ControlCondition table and change the following conditions:

EulaFormA_Property<>"Yes" becomes EulaFormA_Property<>"Yes" or LicenseViewed=""

and

EulaFormA_Property="Yes" becomes EulaFormA_Property="Yes" and LicenseViewed="1"

This makes sure that the Next button only becomes available when both criteria are met.

9) Finally, save the dialog and create a copy in the VsdDialogs/1033 folder. Make sure to update the Language code in the ModuleSignature table.

Screenshot - customdialogsvsd5.jpg

Screenshot - customdialogsvsd6.jpg

As you can see in the above example, even with "I Agree" selected the Next button is not active until the viewer has scrolled to the bottom.

Screenshot - customdialogsvsd7.jpg

What we learn from this exercise is that when our dialog is built into the installer, all of the tables that exist -- even ones that were unknown to it, e.g. the Binary table is not used as a standard in any of the other WID files -- will be added to the final installer as long as we preserve uniqueness when these tables are merged.

Unattended installs

One of the ways of deploying installers -- especially in an automated build environment -- is by unattended installs, i.e. running the installer via the msiexec command line tool and passing parameters. However, the way the dialogs have been created and the use of custom actions to override our properties with the preset values that we supply via Visual Studio means that any parameters that we supply on the command line are overridden with our presets. We can, however, use custom actions to allow a parameter to be passed via the command line and show this in the UI or alternatively be used to preset a value during an unattended install. I have supplied an example TextBox dialog that allows all 4 edit boxes to be populated via the command line.

Screenshot - textboxes_unattended.jpg

Rather than just setting the property directly to that of the value supplied via Visual Studio, the property instead is set using a temporary property, i.e.

  1. Set temp property with value from command line
  2. If temp property is empty, then set temp property with preset value
  3. Set actual property with value from command line

I suggest that you look at the tables CustomAction, ModuleInstallExecuteSequence, ModuleInstallUISequence and ModuleSubstitution in the supplied WID file, as these were the tables that had the changes made after making a clone of the Textboxes (A) dialog.

Credentials dialog

One of the things that has not been investigated in-depth is the Visual Studio interaction that takes place and which utilises the ModuleConfiguration and ModuleSubstitution tables. We can see from the earlier examination when creating Textboxes (D) that there appears to be some support for lists in the ModuleConfiguration table; e.g.

Screenshot - customdialogsvsd8.jpg

It would be nice if we could use this ability to perhaps toggle the state of the edit control from being a normal text box to a password box, such as:

0;Hidden=0;Normal=7;Password=2097159

However, quick experimentation shows that this does not work when the Password selection is chosen to be applied to the edit box. It is suspected that the Attributes column is a special case, as Labels have a default attribute of 3 and Text boxes have one of 7. These are controlled with a single true/false selection, though.

If we do need a password box, we could either create a special dialog that has the correct attributes set or we could modify the installer post-build and set the attributes directly, e.g. setupprjpwd.asp. If a transform was created instead, this could be applied automatically to the installer after each build. As such, it could automate the process of setting the attributes. NOTE: MsiTrans.exe is a tool you can use for this.

It is possible to set a property or the default contents of a control with a list, but it is the name that is presented in the UI and the value that is applied. It also seems that only numbers are applicable for values, which somewhat limits the options available to us when using this list control in our property grid.

Let's try making a custom dialog that we can use for setting the credentials of something, e.g. a windows service or an application pool. What we need is a dialog that will let us choose between the various machine accounts LocalSystem, LocalService and NetworkService and will also allow us to select a Custom Account and provide credentials for that account. Something along the lines of an application pool identity set would be ideal. That dialog has 2 radio buttons, a list box and 2 edit boxes, but we could add a third to receive a confirmation password for good measure.

Creating the dialog

The intial steps involved have been covered before so we will quickly summarise:

1) Make a copy of the Textboxes (A) dialog, since we have 3 edit boxes.

2) Edit the Control table and remove the first edit box and label, replacing them with a RadioButtonGroup and a ComboBox control.

3) Add a ComboBox table and included the required entries.

4) Add a RadioButton table and included the entries.

5) Modify the ControlCondition table so that we can control the enable/disable state of the control based on the radio button selection, e.g.

Screenshot - customdialogsvsd9.jpg

Screenshot - customdialogsvsd10.jpg

Updating the tables

Now that we have our dialog, we can update the tables that allow us to manage it in Visual Studio. Here are the next steps:

1) Modify the ModuleConfiguration table so that we are presented with the following in Visual Studio, where AccountTypeValue and PreconfiguredAccountValue are combo box controls with preset values:

Screenshot - customdialogsvsd11.jpg

2) Update the ModuleSubstitution table to pass the values from the configuration into the installer. It is at this point that we find it is not possible to complex substitutions and that we could not update the ControlCondition table with the names of the new properties which were used in the Condition field. So,

3) Rework the Control, ControlCondition, ComboBox and RadioButton tables to use a private property. In MSIs, public properties are properties whose names are in UPPERCASE.

4) Add custom actions to map the required properties relating to the radio button and combo box. Control these custom actions in the InstallExecuteSequence and the InstallUISequence tables based on conditions that are also dependent on private properties.

5) Again update the ModuleSubstitution table using less complex substitutions against the Property and CustomAction table.

6) Finally, update the ControlEvent table to copy the values from the private properties that are used to control the dialog to our named public properties.

We can now correctly set our values based on our own presets. We get the resulting values being set in our installer properties.

Extending the dialog

A few readers have enquired if it is possible to control the Next button based on whether the passwords match. It would be nice to think that we could do this using the current tables, as we know that Edit fields fire action events. Although it would seem that this is not documented or supported and we can compare properties (conditions) and control the enabled state of buttons via the ControlEvent table, what we have found however is that properties attached to edit fields only get updated when the edit field loses focus.

Instead, we can use a custom action hosted in a DLL and use that to compare the 2 password entry controls and enable/disable the Next button as appropriate. We can then trigger this CA whenever we get an event from either of the edit fields and when the radio button changes. We also added a small condition, in that we only fired the CA if both password fields were set to visible in the configuration.

Finally, we can apply the property setting technique that we used for unattended installs to create a fully fledged Credential acquiring dialog.

Conclusion

It would seem that with a little effort we can add new dialogs to existing setup projects. If anyone has come across similar instructions before, please let me know so that I can update/correct this article.

Samples

The downloadable samples must be installed in the correct folder; see the ReadMe.txt file contained in the ZIP. There is also a modified version of the CA originally provided by Kallely L. Sajan.

References

Feedback and voting

If you have read this far, then please remember to vote. If you do or don't like it, or if you agree or disagree with what you have read, then please say so in the forum below. Your feedback is important!

History

  • 16 May, 2007 - Article submitted
  • 20 May, 2007 - Minor wording changes
  • 14 June, 2007 - Minor wording changes, unattended installs section
  • 17 June, 2007 - Extended password dialog update

License

This article, along with any associated source code and files, is licensed under The Code Project Open License (CPOL)


Written By
Employed (other) Purplebricks
Australia Australia
All articles are supplied as-is, as a howto on a particular task that worked for me in the past. None of the articles are supposed to be out-of-the-box freeware controls and nor should they be treated as such. Caveat emptor.

Now living and working in Australia, trying to be involved in the local .NET and Agile communities when I can.

I spend a good chunk of my spare time building OpenCover and maintaining PartCover both of which are Code Coverage utilities for .NET.

Comments and Discussions

 
GeneralMy vote of 5 Pin
Member 150263652-Feb-21 16:59
Member 150263652-Feb-21 16:59 
QuestionText Boxes don't accept spaces Pin
Member 881056726-Oct-19 3:33
Member 881056726-Oct-19 3:33 
AnswerRe: Text Boxes don't accept spaces Pin
Member 881056726-Oct-19 6:08
Member 881056726-Oct-19 6:08 
BugLittle Typo in "Controlling the dialog with custom actions" Pin
hi reddit1-Jul-16 8:41
professionalhi reddit1-Jul-16 8:41 
GeneralRe: Little Typo in "Controlling the dialog with custom actions" Pin
Shaun Wilde11-Jul-16 17:29
Shaun Wilde11-Jul-16 17:29 
QuestionVsdCustomText4Dlg.wid Pin
jdc78-Feb-16 5:12
jdc78-Feb-16 5:12 
AnswerRe: VsdCustomText4Dlg.wid Pin
Shaun Wilde11-Jul-16 17:27
Shaun Wilde11-Jul-16 17:27 
QuestionUnattended Install from cmd Pin
Member 1185771324-Nov-15 6:34
Member 1185771324-Nov-15 6:34 
AnswerRe: Unattended Install from cmd Pin
Shaun Wilde11-Jul-16 17:51
Shaun Wilde11-Jul-16 17:51 
GeneralVS 2013 vsdDialogs Folder Pin
Jan.Asmus29-Aug-14 1:21
Jan.Asmus29-Aug-14 1:21 
QuestionHow to Print the License Agreement in the LA Dialog in a msi setup project ? Pin
Member 109184658-Jul-14 1:01
Member 109184658-Jul-14 1:01 
AnswerRe: How to Print the License Agreement in the LA Dialog in a msi setup project ? Pin
Shaun Wilde1-Oct-14 21:20
Shaun Wilde1-Oct-14 21:20 
QuestionСustom tables Pin
Adamanteus23-May-14 12:59
Adamanteus23-May-14 12:59 
Thanks, I found in your article, much useful information - but one problem appeared, when I tried to add custom table ModuleInstallExecuteSequence, using Orca that's possible to do if manually repeat fields in
InstallExecuteSequence table.
But after it VS not merged this tables, with error of presence some unknown variables, with empty values.

So, questions is - how to add custom tables to VS dialog modules ?
AnswerRe: Сustom tables Pin
Shaun Wilde28-May-14 0:23
Shaun Wilde28-May-14 0:23 
QuestionComboBox Property Pin
Member 1013166028-Apr-14 7:35
Member 1013166028-Apr-14 7:35 
AnswerRe: ComboBox Property Pin
Shaun Wilde20-May-14 15:38
Shaun Wilde20-May-14 15:38 
QuestionPre-populate Serial Number field Pin
Microdev20-Apr-14 13:52
Microdev20-Apr-14 13:52 
AnswerRe: Pre-populate Serial Number field Pin
Shaun Wilde27-Apr-14 17:17
Shaun Wilde27-Apr-14 17:17 
GeneralMy vote of 5 Pin
Reza Ahmadi5-Mar-14 22:02
Reza Ahmadi5-Mar-14 22:02 
GeneralRe: My vote of 5 Pin
Shaun Wilde5-Mar-14 23:28
Shaun Wilde5-Mar-14 23:28 
GeneralRe: My vote of 5 Pin
Reza Ahmadi6-Mar-14 2:36
Reza Ahmadi6-Mar-14 2:36 
GeneralMy vote of 5 Pin
Sharath Kumar K16-Sep-13 2:07
professionalSharath Kumar K16-Sep-13 2:07 
GeneralMy vote of 3 Pin
Mohit Choksi10-May-13 9:48
Mohit Choksi10-May-13 9:48 
QuestionUnicode support? Pin
supernorb6-Apr-13 13:17
supernorb6-Apr-13 13:17 
AnswerRe: Unicode support? Pin
Shaun Wilde11-May-13 22:26
Shaun Wilde11-May-13 22:26 

General General    News News    Suggestion Suggestion    Question Question    Bug Bug    Answer Answer    Joke Joke    Praise Praise    Rant Rant    Admin Admin   

Use Ctrl+Left/Right to switch messages, Ctrl+Up/Down to switch threads, Ctrl+Shift+Left/Right to switch pages.