It probably comes from your implementation rather than unsafe/safe, or the size of the mask you use.
800*600 for an image is not that big.
And if you use a big mask size, it won't change much.
You can try this anyway if you want:
http://ndevilla.free.fr/median/median.pdf[
^]