No... This is not the right way to do it (although you can delete all of the files of your application to remove it from client's machine but) you should try using the
Installer
[
^] class of
System.Configration.Install
namespace. It allows you to have a function to Install and Uninstall the application.
You can run the Install() function to run the installation, and then you can run the Uninstall function to delete the files from client's machine. If you go to the Installer MSDN document (I have attached) you will find an example of this also. Then you can provide the files to save and also files to remove when user triggers both of the functions. Read the documentation for more on this procedure.