GlobalAllocPtr() return NULL When run it many times, although I had used GlobalFreePtr()

GlobalAllocPtr() is a macro: GlobalLock(GlobalAlloc((flags), (cb)))
When there is not enough memory, GlobalAlloc() will fail and return NULL. The return type of GlobalAlloc() is a HGLOBAL: A handle to a global memory object. When passing such a NULL handle to GlobalLock(), it will return NULL and set the last error to 6 (ERROR_INVALID_HANDLE).

So it seems that you are using up all your memory. A possible reason is that you are not freeing your memory. When using GlobalAllocPtr(), the memory must be freed using GlobalFreePtr(). When passing the memory object to the system, it must be unlocked before. Otherwise the system can't free it because there is still a lock count:

pDIB = (BYTE*)GlobalAllocPtr( GHND, DIBsize );
// Copy data to memory here
// Option1: Memory no longer used: Free it using GlobalFreePtr(pDIB)
// Option2: Unlock the memory before passing it to
// somewhere else that takes ownership of the memory:
HGLOBAL hGlobal = GlobalPtrHandle(pDIB);
GlobalUnlock(hGlobal);
// Pass global memory to function that takes ownership of the memory
FuncExpectingAHGLOBAL(hGlobal);

Thanks for your help.
In Option 1: I had used and this error still have.
In Option 2: I will try it and report to you later.

I have a question: When size need to allocate larger than 512KB, GlobalAllocPtr() does not using Heap memory any more, in this case GlobalAllocPtr() using direct Virtual Memory to allocate memory. And then when free memory by GlobalFreePtr(), does it work? because I think GlobalFreePtr() work in Heap memory, not in Virtual Memory
Any help from you?
Thank you so much.

GlobalAlloc() memory is on the heap. See http://msdn.microsoft.com/en-us/library/windows/desktop/aa366574%28v=vs.85%29.aspx.

In general you should use global memory only when it is required (passed as parameter to some function, e.g. clipboard and drag & drop operations).
In all other cases you should use new / delete.

I would also suggest to not use the macros. Use the pure functions. So you have a better overview of the usage and locking and can check for errors when they occur and not later (if GlobalAlloc() fails it will set an error code which is overwritten by GlobalLock() when using the GlobalAllocPtr() macro):

HGLOBAL hGlobal = ::GlobalAlloc(nSize);
if (NULL == hGlobal)
{
// out of memory; show error message and/or return failure state
return; // out of memory
}
// Lock the memory and get a pointer to it
pDIB = (LPBYTE)::GlobalLock(hGlobal);
// Do something with the memory
::CopyMemory(pDIB, lpSrc, nSize);
// Pass memory to other function to do somethin with it
SomeFunc(pDIB);
// Unlock memory so that it can be accessed by others and freed
::GlobalUnlock(hGlobal);
// Pass global memory to other function
SomeFunc(hGlobal);
// If function does not take ownership of memory, free it
::GlobalFree(hGlobal);

I also make size lager when it small. above 512* 1024 bytes, but it's still happen.
It occur when run >1500 times. Do you give me a hint?