I haven't got Expresso loaded, (or VS - don't ask) so this is untested, but I suggest that you remove the leading '^', the trailing '$', and the repeat count '*'.
Regex.IsMatch(@"[\~|\`|\^|\<|\>]")
May work better - at the moment you are looking for a string which consists entirely of zero or more repetitions of your illegal characters only.