Not sure you mean this, but you could try handling WM_NCCALCSIZE
] and WM_NCPAINT
] and draw your own border around your dialog window.
In case you should need it, you can do
SetWindowPos(NULL, 0, 0, 0, 0, SWP_NOZORDER | SWP_NOMOVE | SWP_NOSIZE | SWP_DRAWFRAME);
to make your dialog redraw its frame.