c# .net tilde (ascii 126) sorting a string

C# is .NET, so it defaults to Unicode characters, not ASCII - however, this makes little different in unaccented English.

The unicode character order is exactly as you would expect: http://unicode-table.com/en/[^] and a quick check:

string[] data = {"I", "~J", "123", "K", "~E", "A", "~G", "U"};
Array.Sort(data);
foreach (string s in data)
    {
    Console.WriteLine(s);
    }

shows exactly what you see!

But that's because it's using the default comparer, which is not an ordinal compare - it's a linguistic comparison!
Try using an Ordinal comparison:

string[] data = {"I", "~J", "123", "K", "~E", "A", "~G", "U"};
Array.Sort(data, StringComparer.Ordinal);
foreach (string s in data)
    {
    Console.WriteLine(s);
    }

And you will get the result you want:

123
A
I
K
U
~E
~G
~J