For anyone who is intersted I found out the the problem seems to be with having the toolbar and the the Provider, remove one and the problem is fine, add one and the issue is there. I don't know why.
However I found a workaround for the issue. You can get around it by doing the following.
var portableSettingsProvider = new PortableSettingsProvider();
foreach (SettingsProperty property in Properties.Settings.Default.Properties)
property.Provider = portableSettingsProvider;
In your Main() before you application is ran.
If anyone know WHY those two have a problem with each other I would love to know.