No, there is no better way than using two loops, except if you accept unrolling the loop on the first index.
Anyway, you'd better avoid indexing the array inside the inner loop as this would cost you the overhead of the map lookup repeated 40000 times !
It is much better to keep an array of pointers to the four buffers and index them.
unsigned char* Buffer;
for (int i= 0; i < 10000; i++)
int Avg= 0;
By the way, you could also do without the map container...
You didn't specify if you want the row or column averages. The other case is even simpler (a single pointer suffices).