This is the usual challenge in writing shared memory code. In some cases, you can force the shared memory to map to some high, unused address. In general, this may not be practical or convenient.
Since the addresses are different, you can't store pointers there. Instead - you can store byte offsets. That's worked well for me.
typedef struct msgbuf1
static char *as_pointer(char *base, message_buf *desc)
This also works for memory mapped files.