Create custom dialogs for use in your Visual Studio Setup projects

• Download source - 136.2 Kb

Introduction

The Visual Studio setup projects that come installed as part of Visual Studio are handy for knocking out quick-and-easy installers, but they are a bit limited in scope. Two main drawbacks are:

• You have only a limited number of dialogs to choose from, which can easily run out on a relatively simple install. Say, for example, that you have a need for a further set of textboxes for a piece of information that contextually belongs on its own and does not look correct if tagged onto one of the other dialogs.
• The information you can enter is limited. Have you ever needed a dialog with 5 radio buttons? What about a pick-list of options?

Background

<ramble>A bit of a ramble going on here so feel free to skip this bit.</ramble>

Over the years, I've started to become a bit of a build monkey, i.e. taking all the code produced by the developers and getting it to build using automated builds, running unit tests as part of the build and finally getting it to deploy and run on a machine that has not had a developer (or Visual Studio) near it. Plus, doing whatever development work comes my way as well. One of the things I have learned over time is that the person who knows most about how the code should work when running on another machine is the developer that wrote the code in the first place. So, how do we get the developers to tell us what we need to know? Let's examine the behaviour of 90% of developers and the environment that they work in.

1) Developers hate writing documentation. They will dodge the topic when raised and even, on some occasions, fix bugs in old VB code which have suddenly become urgent after many years of being blissfully ignored.

2) Buying new software on top of the initial Visual Studio purchase is a nightmare of justification and licensing. Justifying the purchase of installer software in case the setup needs something that can't be met with the built-in setup projects takes time and paperwork. Uh-oh, looks a bit like documentation and so it is avoided.

3) Some installer software tools resort to scripting languages to customise the install and do other actions. Most developers don't like script-based languages and, because some installer tools use a custom script language, it also means a new learning curve. So, calls for a specialist are made from developers and that new individual then has to deal with point 1 in order to get their work done. You could use free installer tools -- e.g. WiX -- but there are not many people out there that know this tool. Also, the learning curve is fairly steep, as you have to know a lot about installers to understand what is happening when it all goes wrong.

4) They have Visual Studio and it comes with the ability to do setup projects. They come out of the tin and they can use .NET Installer classes to customise their install.

Because of the last item, point 2 is overcome due to the ability of developers to build and test the installer locally. They can also integrate it into their solution and use drag and drop to put files in all the right places. Should the install need a bit of customisation, we can use .NET Installer-based classes to modify our configuration files with information gathered from the simple dialogs. This solves the issues raised by points 1 and 3.

Okay, some installer zealots would say that the use of scripts and .NET Installer-classes is bad ju-ju and should be avoided. Instead we should all use proper custom actions, i.e. native DLLs. However, we live in the real world, folks, and we do what we can in the limited time given to us. The problem in going with point 4 is that it is usually at the end of the project when we discover we need that extra customisation. By then, it is too late to buy new tools and get to grips with them before the product is shipped.

That leaves us with a problem: how do we create that extra dialog?

Visual studio dialogs

There is very limited documentation about the installer dialogs. What does exist seems to be related to language globalization/customisation. However, what we learn is that the installer dialogs are in the folder %ProgramFiles%\Microsoft Visual Studio 8\Common7\Tools\Deployment\VsdDialogs and that these files can be edited using the Orca tool that ships as part of the Windows Installer SDK. By viewing the dialogs in those folders using the Orca tool, it is easy to determine which .wid file belongs to which entry in the Add Dialog screen in Visual Studio. Before we start editing these files it is recommended that a backup is made of these files.

Things that are noticed straightaway:

• There are lots of well-known tables that are documented on the MSDN website and there are some that are custom, i.e. those beginning with Module.
• All of the dialogs have a unique name. See the tables Control, ControlCondition, ControlEvent and Dialog, amongst others.
• All of the custom actions have a unique name. See the CustomAction table.
• All of the properties have a unique name. See the Property table.

This unique naming makes sense when you think that we could be adding them all into one installer or, as we can suspect, merged into one installer. This is why there may be a limited number of pre-made dialogs.

The custom tables

ModuleComponents

Purpose unknown.

ModuleConfiguration

Looks like what is read by Visual Studio and presented in the property window.

ModuleDialog

This looks like the information that is displayed in the Add Dialog section of Visual Studio. There is also an Attributes field here that I think is used to control when the dialogs appear. For example, all of the Admin dialogs have an attribute that is greater than or equal to 8192 -- i.e. the 13th bit is set -- whereas for normal dialogs it is the 12th bit. Certain dialogs only show up in web setup projects; 5th bit (or 7th bit in admin dialogs) is set. Other dialogs also have some unique setting. For instance, for both admin and normal dialogs the finished dialog has the 3rd bit set. We can therefore infer that the Attributes field is used to both control when the dialogs show up and perhaps control positioning in the UI. It would therefore make sense that if you make a new custom dialog for use in a similar situation as another dialog that already exists, then using the same attributes as that dialog would be the approach to take.

ModuleIgnoreTable

Purpose unknown.

ModuleInstallExecuteSequence

These actions look like they would be merged into the InstallExecuteSequence of the main installer.

ModuleInstallUISequence

These actions look they would be merged into the InstallUISequence of the main installer.

ModuleSignature

Contains an internal name perhaps; uses a GUID-like string as part of a name and a language code.

ModuleSubstitution

Looks like a mapping table of values from the Visual Studio environment; defined in ModuleConfiguration to the standard tables within the installer before or during the moment they are merged into one installer.

First custom dialog

Thus armed, we are ready to create our first custom dialog Textboxes (D) which are amazingly like Textboxes (A/B/C).

1) First, we make a copy of Textboxes (A) or, as it is known on the disk, VsdCustomText1Dlg.wid in the VsdDialogs/0 folder.

2) Open in Orca and change all occurrences of CustomTextA to CustomTextD in the Dialog, Control, ControlCondition and ControlEvent tables. See the Dialog, Condition, and Argument columns where applicable.

3) Make the custom action names unique in the CustomAction table.

4) Make the property names unique in the Property table.

5) Change the property names in the DefaultValue column in the ModuleConfiguration table to match those of our new properties.

6) Change the display name in ModuleDialog to match what we want to present in the UI, should we succeed in getting it to appear. Don't forget to change the name of our dialog as well.

7) Change the names of the custom actions to match the new names of our custom actions in the ModuleInstallExecuteSequence and ModuleInstallUISequence tables.

8) Generate a new GUID (I use the guidgen tool) and format it so that dashes become underscores. Also modify the ModuleID table.

9) Finally, modify the Row column in the ModuleSubstitution table to refer to our new dialog and associated custom actions.

Start Visual Studio and see if it appears in the Add Dialog window of a set-up project. Nope. Hang on, maybe it is a bit more advanced than that. Perhaps it is trying to present the one in my default language in Visual Studio, which happens to be English (United States) and has a LCID of 1033. Make a copy in the VsdDialogs/1033 folder, changing the Language field in the ModuleSignature table to 1033 for good measure. Restart VisualStudio and have another look. Success!

But does it work and will it play nicely with the other dialogs? We can only hope that the developers of this bit of Visual Studio do not care what dialogs are available so long as we keep to that same pattern (naming conventions and suchlike). Let's create a test project, add all four of our text box dialogs, play with a few settings, shuffle the order, compile and test. It's looking good. Add a quick and dirty CA to display a few properties. Another success. So now we know how to make a copy of an existing dialog and we can easily repeat that process should the need arise.

Five radio buttons

Is it possible to extend the dialogs to present more information than they have before? Let's try making a 5-button dialog.

1) Create a copy of VsdCustom4ButtonDlg.wid and do the modifications that we did before to make it unique.

2) Adding a new radio button is relatively simple in that we only need to update the RadioButton table. Using Orca, we can copy and paste a row and then edit that row such that its order is 5 and it has a Y-value of 96. NOTE: we need to make sure that the containing control in the Control table is also big enough.

3) We can check the spacing of controls and general layout of our dialog by using the Dialog Preview feature that can be found in the Tools menu.

4) Using Orca, edit the ModuleConfiguration table so that we can manage the Label and Value of our new button in Visual Studio. Let's duplicate rows Button4Label and Button4Value to make Button5Label and Button5Value, respectively. The fields with the hashes (#) look like they may be referring to resource text strings; we can hopefully ignore them for now.

5) Again using Orca, edit the ModuleSubstitution table to add 3 new rows that will set the Property, Text and Value properties of our new radio button. You will have to change the Row field to BUTTON5;5 before you can change the Table field back to RadioButton.

6) Save to disk and make a copy to put into the VsdDialogs/1033 folder.

Restart Visual Studio and open up the test set-up project. A partial success, it seems we can add the new dialog and we can see 5 radio buttons when we run the installer. However, the properties for the fifth button don't seem to be appearing in the Properties window of Visual Studio. What about the # values? Could the problem be that the code uses them to generate the descriptors when building the collection for the property grid and thus may filter out duplicates? Let's edit the ModuleConfiguration table and the changed the fields with # values in them to what we want to display. Hopefully the developers at Microsoft have written the code such that if it did not find the resource it would fall back and display what it sees instead. Save to disk, make a copy in the VsdDialogs/1033 folder, restart Visual Studio and try again. Success!

New 5 radio button dialog: notice that the description is for the 4 radio button dialog. Originally I didn't know where this description entry was, but thanks to a tip from Grump we now know that the information can be edited using Orca under the View | Summary Information menu option.

We can also see that this dialog has a GUID as a package code and that when we investigate the original dialogs, the value of this GUID matches the GUID in the ModuleSignature table. It does not appear that this GUID as edited by this dialog has any effect on the creation of installers, but we will maintain the mapping just in case.

The figure above displays the property grid, showing the extra properties and the hard-coded labels. Now we know how to add new controls and extend the configuration of the dialogs that we can add to our installers.

Controlling the dialog with custom actions

A few of the controls on the dialog can be managed via the existing tables, e.g. enabling buttons. But if we want a finer control, we may need to resort to a custom action (CA). For this example, we'll create an update/upgrade to the License Agreement dialog screen. We'll incorporate a custom action that will check that the user has made an attempt to read the licence by checking that the window has been scrolled to the bottom. When this has happened, a property called LicenseViewed will be created and set to "1." NOTE: I am using a version of the IsLicenceViewed CA by Kallely L. Sajan, found here. I have only made a slight modification to the CA such that it gets the name of the required window from a property within the installer, i.e. make it more generic.

1) Compile, if you can, the LicenseViewed C++ project. Otherwise, you may use the pre-built assembly. The project requires that you have the Platform SDK installed.

2) Make a copy of VsdLicenseDlg.wid that is in the VsdDialogs/0 folder. Name it VsdLicenceViewedDlg.wid.

3) Open the new file with the Orca tool and do the modifications that we did before to make it unique. In this example, I renamed the dialog to EulaFormA and used a name of Licence Agreement (Viewed).

4) In order to add our new DLL to the installer, we need to add 2 tables: Binary and CustomAction. Luckily, these are well known (standard) tables and Orca can create them without issue. Using the menu, select Tables->Add Tables. Select Binary and CustomAction and finally click OK.

5) Select the Binary table and add a new row. Enter LicenseCA for the name and browse to LicenseViewed.dll for the data. Our CA assembly is now stored.

6) Select the CustomAction table and add a new row. Enter the following information, noting that the CA is a type-1 custom action and is one of the most common types:

Action	CheckLicenseViewed
Type	1
Source	LicenseCA
Target	CheckLicenseViewed

The source is the name of the assembly as stored in the Binary table. The Target is the exported function that is within that assembly.

7) Select the ControlEvent table and add a new row. Fill it with the following information:

Dialog	EulaFormA (or whatever you have renamed it as)
Control	LicenseText (this is the control that displays the license)
Event	DoAction Argument CheckLicenseViewed (the CA)
Condition	1

This will force our CA to be executed when the scroll bar is moved. Not very elegant, but it is simple.

8) Select the ControlCondition table and change the following conditions:

EulaFormA_Property<>"Yes" becomes EulaFormA_Property<>"Yes" or LicenseViewed=""

and

EulaFormA_Property="Yes" becomes EulaFormA_Property="Yes" and LicenseViewed="1"

This makes sure that the Next button only becomes available when both criteria are met.

9) Finally, save the dialog and create a copy in the VsdDialogs/1033 folder. Make sure to update the Language code in the ModuleSignature table.

As you can see in the above example, even with "I Agree" selected the Next button is not active until the viewer has scrolled to the bottom.

What we learn from this exercise is that when our dialog is built into the installer, all of the tables that exist -- even ones that were unknown to it, e.g. the Binary table is not used as a standard in any of the other WID files -- will be added to the final installer as long as we preserve uniqueness when these tables are merged.

Unattended installs

One of the ways of deploying installers -- especially in an automated build environment -- is by unattended installs, i.e. running the installer via the msiexec command line tool and passing parameters. However, the way the dialogs have been created and the use of custom actions to override our properties with the preset values that we supply via Visual Studio means that any parameters that we supply on the command line are overridden with our presets. We can, however, use custom actions to allow a parameter to be passed via the command line and show this in the UI or alternatively be used to preset a value during an unattended install. I have supplied an example TextBox dialog that allows all 4 edit boxes to be populated via the command line.

Rather than just setting the property directly to that of the value supplied via Visual Studio, the property instead is set using a temporary property, i.e.

1. Set temp property with value from command line
2. If temp property is empty, then set temp property with preset value
3. Set actual property with value from command line

I suggest that you look at the tables CustomAction, ModuleInstallExecuteSequence, ModuleInstallUISequence and ModuleSubstitution in the supplied WID file, as these were the tables that had the changes made after making a clone of the Textboxes (A) dialog.

Credentials dialog

One of the things that has not been investigated in-depth is the Visual Studio interaction that takes place and which utilises the ModuleConfiguration and ModuleSubstitution tables. We can see from the earlier examination when creating Textboxes (D) that there appears to be some support for lists in the ModuleConfiguration table; e.g.

It would be nice if we could use this ability to perhaps toggle the state of the edit control from being a normal text box to a password box, such as:

0;Hidden=0;Normal=7;Password=2097159

However, quick experimentation shows that this does not work when the Password selection is chosen to be applied to the edit box. It is suspected that the Attributes column is a special case, as Labels have a default attribute of 3 and Text boxes have one of 7. These are controlled with a single true/false selection, though.

If we do need a password box, we could either create a special dialog that has the correct attributes set or we could modify the installer post-build and set the attributes directly, e.g. setupprjpwd.asp. If a transform was created instead, this could be applied automatically to the installer after each build. As such, it could automate the process of setting the attributes. NOTE: MsiTrans.exe is a tool you can use for this.

It is possible to set a property or the default contents of a control with a list, but it is the name that is presented in the UI and the value that is applied. It also seems that only numbers are applicable for values, which somewhat limits the options available to us when using this list control in our property grid.

Let's try making a custom dialog that we can use for setting the credentials of something, e.g. a windows service or an application pool. What we need is a dialog that will let us choose between the various machine accounts LocalSystem, LocalService and NetworkService and will also allow us to select a Custom Account and provide credentials for that account. Something along the lines of an application pool identity set would be ideal. That dialog has 2 radio buttons, a list box and 2 edit boxes, but we could add a third to receive a confirmation password for good measure.

Creating the dialog

The intial steps involved have been covered before so we will quickly summarise:

1) Make a copy of the Textboxes (A) dialog, since we have 3 edit boxes.

2) Edit the Control table and remove the first edit box and label, replacing them with a RadioButtonGroup and a ComboBox control.

3) Add a ComboBox table and included the required entries.

4) Add a RadioButton table and included the entries.

5) Modify the ControlCondition table so that we can control the enable/disable state of the control based on the radio button selection, e.g.

Updating the tables

Now that we have our dialog, we can update the tables that allow us to manage it in Visual Studio. Here are the next steps:

1) Modify the ModuleConfiguration table so that we are presented with the following in Visual Studio, where AccountTypeValue and PreconfiguredAccountValue are combo box controls with preset values:

2) Update the ModuleSubstitution table to pass the values from the configuration into the installer. It is at this point that we find it is not possible to complex substitutions and that we could not update the ControlCondition table with the names of the new properties which were used in the Condition field. So,

3) Rework the Control, ControlCondition, ComboBox and RadioButton tables to use a private property. In MSIs, public properties are properties whose names are in UPPERCASE.

4) Add custom actions to map the required properties relating to the radio button and combo box. Control these custom actions in the InstallExecuteSequence and the InstallUISequence tables based on conditions that are also dependent on private properties.

5) Again update the ModuleSubstitution table using less complex substitutions against the Property and CustomAction table.

6) Finally, update the ControlEvent table to copy the values from the private properties that are used to control the dialog to our named public properties.

We can now correctly set our values based on our own presets. We get the resulting values being set in our installer properties.

Extending the dialog

A few readers have enquired if it is possible to control the Next button based on whether the passwords match. It would be nice to think that we could do this using the current tables, as we know that Edit fields fire action events. Although it would seem that this is not documented or supported and we can compare properties (conditions) and control the enabled state of buttons via the ControlEvent table, what we have found however is that properties attached to edit fields only get updated when the edit field loses focus.

Instead, we can use a custom action hosted in a DLL and use that to compare the 2 password entry controls and enable/disable the Next button as appropriate. We can then trigger this CA whenever we get an event from either of the edit fields and when the radio button changes. We also added a small condition, in that we only fired the CA if both password fields were set to visible in the configuration.

Finally, we can apply the property setting technique that we used for unattended installs to create a fully fledged Credential acquiring dialog.

Conclusion

It would seem that with a little effort we can add new dialogs to existing setup projects. If anyone has come across similar instructions before, please let me know so that I can update/correct this article.

Samples

The downloadable samples must be installed in the correct folder; see the ReadMe.txt file contained in the ZIP. There is also a modified version of the CA originally provided by Kallely L. Sajan.

References

• Orca
• Windows Installer SDK
• WiX

Feedback and voting

If you have read this far, then please remember to vote. If you do or don't like it, or if you agree or disagree with what you have read, then please say so in the forum below. Your feedback is important!

History

• 16 May, 2007 - Article submitted
• 20 May, 2007 - Minor wording changes
• 14 June, 2007 - Minor wording changes, unattended installs section
• 17 June, 2007 - Extended password dialog update