I don't think that this is unfortunate. It's just that you haven't handled it.
You have to use Custom Actions for that:
1.Add a new library ("CustomActions") to the setup project
2.Add => New Item => Installer class
3.Switch to code view and override the Uninstall method
public override void Uninstall(IDictionary savedState)
1.Add the project output (Primary Output) of the CustomActions project to the setup project.
2.Right click your setup project and click View => Custom Actions
3.Rigth click uninstall => Add Custom Action => Application Folder => Primary Output of CustomActions