It's because the cast operator can do two things, and you're confusing them here.
The first example is a
converting cast; the value was an int, and you are casting it to a uint. This is defined to be a valid converting cast. If it were a user defined type, there would be an operator of the form
public static explicit operator uint(int value){ ... }
There are also implicit converting casts, for example
int i = 2;
double d = i;
Within the numeric types, all converting casts are (I think, anyway) valid; those which could possibly lose data or change its interpretation (narrowing and signed-unsigned conversions) are explicit, those which can't (widening) are implicit.
The second type of cast is a type reassignment. This only works if the thing you're trying to cast actually
is of the type you're trying to cast it to (or has an explicit cast operator from that type). There is no explicit conversion cast defined from object to uint, and o is actually an int, not a uint, so the cast fails.
What you have to do in this situation is first get the int using a type reassignment cast (which will work because o actually is an int), and then use a converting cast on that:
uint u = (uint)(int)o;