Seems like most of the answers overlook the obvious.
You want your key to be an int and the value to be multiple values of any type. Why not use the boxing feature of .NET?
var map = new Dictionary<int, List<object> >();
List<object> list;
if ( map.TryGetValue(5, out list) )
{
foreach (var item in list)
Console.WriteLine(item);
}
</object>