Not really familiar with VB, but the
IntPtr[
^] class should allow you to interact with pointers. You might want to overload the C++ functions to accept those as well (if I understand correctly, and the C++ is C++/CLI).
And worse case, if you can get it to work in C#, write a C# wrapper for the C++ functions that are giving you trouble.