Not sure you mean this, but you could try handling
WM_NCCALCSIZE[
^] and
WM_NCPAINT[
^] and draw your own border around your dialog window.
In case you should need it, you can do
...
SetWindowPos(NULL, 0, 0, 0, 0, SWP_NOZORDER | SWP_NOMOVE | SWP_NOSIZE | SWP_DRAWFRAME);
...
to make your dialog redraw its frame.