Click here to Skip to main content
15,861,168 members
Articles / Programming Languages / C#
Article

Handling Unmanaged Memory Pointers in Managed Memory

Rate me:
Please Sign up or sign in to vote.
4.25/5 (19 votes)
19 Dec 2005CPOL6 min read 120K   1K   43   11
This article shows some examples of using pointers from unmanaged-memory libraries in managed code (C#)

Abstract

This article explores some of the issues in using libraries of unmanaged memory in the new managed memory environment (.NET). We take a look directly at the memory address (pointer) interfaces provided by these libraries and methodologies are presented in C# for manipulating this memory in .NET.

Introduction

Managed memory and garbage collection is a godsend in .NET, but as new development teams are moving to this environment a large number of legacy (unmanaged memory) libraries still exist that provide needed functionality right now. There is too large an investment in these libraries, and too large an investment needed to rewrite them, to just discard them. The good news is .NET provides some tools to access these libraries even in the “keep your hands off” managed memory environment. These tools are not without their caveats though, and this article demonstrates some techniques for accessing and using the libraries where the interfaces are unmanaged memory, specifically pointers. This article is not intended to be a complete primer or tutorial on data marshaling, so a basic understanding of what the Marshal class (System.Runtime.InteropServices.Marshal) does is assumed. What we are dealing with here is specifically memory or pointer interfaces.

Emulating Pointers

IntPtr is the type that is used to represent system pointers in the managed memory environment. So if we were to call a function that passed back a pointer to a data structure, we could import the function as shown here:

C#
[DllImport("Legacy.dll")] 
public static extern void GetData(IntPtr pDataRecord);

After an invocation of this function, we now have a pointer to our data structure, so how do we access this information? In C++, we may have used a memcpy to copy this data into our data structures, or we may just use pointer math (offsets) to copy individual variables. But accessing the offset of an IntPtr is not so straightforward in C#. The compiler does not allow two IntPtrs to be added, as illustrated by the following compiler error message:

Operator '+' cannot be applied to operands of type 'System.IntPtr' and 'System.IntPtr'

Suppose you needed to read an integer (32-bit) from an offset of 6 bytes from this pointer. The marshal class provides a function ReadInt32 that allows this:

C#
Int32 I = Marshal.ReadInt32(pDataRecord,4);

This works and there are many more Marshal methods which will allow you to marshal data from memory offsets. You can get around the compiler error and still do your pointer math by doing the following:

C#
IntPtr ip = new IntPtr(pDataRecord.ToInt32()+4);

This is probably not what was intended by the compiler developers, but does work. It does, however, expose your code to the same risks and maintenance issues as C++ style pointers and errors. You would be well-served to try to utilize the built in functionality of the Marshal class if possible. Marshalling data structures: Since data is passed to us as data structures, we should set up compatible data structures in our code, allowing the use of the Marshal class and methods to simplify accessing this data. Given the data structure as defined in the unmanaged code is this:

C#
typedef struct DataRecord
{
    int RecordBufferSize;
    int NumberOfFields;
    int NumberOfElements;
    int RecordType;
    UINT RecordInfoFlags;
    char* FieldDescription;
} DataRecord, *PDataRecord;

Our data structure in C# would be defined like this:

C#
[StructLayout(LayoutKind.Sequential,CharSet=CharSet.Ansi,Pack=2)]
    public struct DataRecord
    {
       [MarshalAs(UnmanagedType.I4)] public int RecordBufferSize;
       [MarshalAs(UnmanagedType.I4)] public int NumberOfFields;
       [MarshalAs(UnmanagedType.I4)] public int NumberOfElements;
       [MarshalAs(UnmanagedType.I4)] public int RecordType;
       [MarshalAs(UnmanagedType.U4)] public uint RecordInfoFlags;
    public string            FieldDescription;
    }

Notice the StructLayout. There are three options for defining the type of layout structure to be used:

  • Auto – Chosen by runtime, but objects cannot be exposed outside of managed memory.
  • Explicit – The offset of the variable is defined by using the FieldOffsetAttribute.
  • Sequential – The variables are laid out sequentially as they appear, with the packing specified by StructLayoutAttribute.Pack.

Here we have used Sequential as the order of variables specified is the order we will receive it. An example will be given later illustrating the need for Explicit. Additionally the string is marshaled as an ANSI character set, which is the default. The Pack=2 tells the compiler to align the variables along a 2-byte alignment. The default is 8, and in this illustration setting it to 2 has no effect. In this example, the MarshalAs attribute is used specifying the type of data that will be marshaled. It is not necessary in this example because the marshal object is capable of discerning the marshaling needed based on the data type. This would be useful in the event the data needs to be marshaled as a differing data type. There are two ways to create the marshaling for this function. The first is to define the import function here as shown earlier:

C#
[DllImport("Legacy.dll")] 
public static extern void GetData(IntPtr pDataRecord);

We would invoke this function, receive a pointer to the data structure, and then use the Marshal.PtrToStructure method to allocate memory for the structure, copy the data from the memory pointed to by the IntPtr into the structure, and return a reference to the object as shown here:

C#
IntPtr pDataRecord;
UnManagedLib.GetData(pDataRecord);
UnManagedLib.DataRecord ds = (UnManagedLib.DataRecord) Marshal.PtrToStructure 
				(pDataRecord , typeof(UnManagedLib.DataRecord));

Or alternatively, we can define the function import to allow the marshaling to handle the data transfer as shown here:

C#
[DllImport("Legacy.dll")] 
public static extern void GetData([MarshalAs(typeof(DataRecord))] pDataRecord);

UnManagedLib.DataRecord ds;
UnManagedLib.GetData(ds);

Suppose we have a union inside of the structure. How will that affect our marshaling? This can be accomplished by changing the StructLayout mentioned above to Explicit. Assume the following C++ struct definition:

C++
typedef struct DataVariable
{
    char InternalName[16];
    char Mnemonic[4];
    char CurveLabel[26];
    USHORT Format;
    union
    {
        USHORT NumberOfBytes;
        USHORT NumberOfElements;
    };
    USHORT UnitType;
    USHORT SpecialHandling;
    SHORT NumberOfDecimalPlaces;
    USHORT OffsetInRecord;
} DataVariable, *PDataVariable;

C# does not support the union definition, but by using the Explicit StructLayout we can create the same effect if we define the structure like this:

C#
[StructLayout(LayoutKind.Explicit,CharSet=CharSet.Ansi)]
public struct DataVariable
{
    [FieldOffset(0), MarshalAs( UnmanagedType.ByValTStr , SizeConst=16)] 
	public string InternalName;
    [FieldOffset(16), MarshalAs( UnmanagedType.ByValTStr, SizeConst=4)] 
	public string Mnemonic;
    [FieldOffset(20), MarshalAs( UnmanagedType.ByValTStr, SizeConst=26)] 
	public string CurveLabel;
    [FieldOffset(46), MarshalAs( UnmanagedType.U2)] public ushort Format;
    [FieldOffset(48), MarshalAs( UnmanagedType.U2)] public ushort NumberOfBytes;
    [FieldOffset(48), MarshalAs( UnmanagedType.U2)] 

	public ushort NumberOfElements; // Union with above
    [FieldOffset(50), MarshalAs( UnmanagedType.U2)] public ushort UnitType;
    [FieldOffset(52), MarshalAs( UnmanagedType.U2)] public ushort SpecialHandling;
    [FieldOffset(54), MarshalAs( UnmanagedType.I2)] public short NumberOfDecimalPlaces;
    [FieldOffset(56), MarshalAs( UnmanagedType.U2)] public ushort OffsetInRecord;
}

The location or offset of each field in the structure is explicitly defined. Note that the fields NumberOfBytes and NumberOfElements have the same offset, thus emulating the C++ union. Some other things to note here are the use of the unmanaged type ByValTStr. This is used on traditional C-style fixed size strings used in structures. The marshaling of this data structure is similar to the above example and won't be repeated here.

Memory Management

Now that we've looked at the movement of data back and forth between managed and unmanaged memory, what about the management of that memory? If the unmanaged memory library you called allocated memory for the structure it passed you, then it will remain in place (assuming the library itself does not move it). The lifetime and freeing of the memory will depend on the implementation of the library. In the managed memory environment it is a good assumption that the memory will not remain at the same address throughout the life of the program. Fortunately the Marshal class provides some methods for overcoming this. If the unmanaged memory library expects you to allocate the memory for the structure prior to passing it as a parameter, you will need to use the Marshal.AllocHGlobal function. Using the example from above, see the following:

C#
IntPtr pDataRecord = Marshal.AllocHGlobal(4);
UnManagedLib.GetData(pDataRecord);
UnManagedLib.DataRecord ds = (UnManagedLib.DataRecord) Marshal.PtrToStructure 
				(pDataRecord , typeof(UnManagedLib.DataRecord));
Marshal.FreeHGlobal(4);

Memory is allocated for the IntPtr (we could've used the Marshal.SizeOf method also to get 4), the pointer is passed in, used, and then the memory is freed. You must always be aware, however, of the expected lifetime of the memory in the library you are calling. The FreeHGlobal cannot be called until the library will not use the memory again. Again, this is determined by the implementation of the library you are interfacing to.

Conclusion

The Marshal class provides a rich set of functionality that allows the use of legacy unmanaged-memory libraries in our new managed-memory applications. The use of pointers is obscured as much as is possible while still allowing the access and manipulation of specific memory addresses, and the Marshal memory methods must be used to insure the runtime memory management does not conflict with the unmanaged-memory libraries.

History

  • 19th December, 2005: Initial post

License

This article, along with any associated source code and files, is licensed under The Code Project Open License (CPOL)


Written By
Web Developer
United States United States
This member has not yet provided a Biography. Assume it's interesting and varied, and probably something to do with programming.

Comments and Discussions

 
GeneralMy vote of 4 Pin
kurniawan Indra28-Jul-11 14:33
kurniawan Indra28-Jul-11 14:33 
QuestionWhat about a SetData? Pin
ungureanub12-Jul-07 3:58
ungureanub12-Jul-07 3:58 
AnswerRe: What about a SetData? [modified] Pin
mekanoo6-Sep-07 9:37
mekanoo6-Sep-07 9:37 
QuestionHow to translate a union within a structure that also contains another union Pin
kosmas13-Feb-06 23:14
kosmas13-Feb-06 23:14 
Laugh | :laugh: Hello,

I got a problem to implement the structure for C#. The C structure I tried to translate into c# is as follows:

/*****************************************************************************/
/* Structure for common verb header */
/*****************************************************************************/
struct LUA_COMMON
{
unsigned short lua_verb; /* Verb Code */
unsigned short lua_verb_length; /* Length of Verb Record */
unsigned short lua_prim_rc; /* Primary Return Code */
unsigned long lua_sec_rc; /* Secondary Return Code */
unsigned short lua_opcode; /* Verb Operation Code */
unsigned long lua_correlator; /* User Correlation Field */
unsigned char lua_luname[8]; /* Local LU Name */
unsigned short lua_extension_list_offset; /* Offset of DLL Extention Lis*/
unsigned short lua_cobol_offset; /* Offset of Cobol Extension */
unsigned long lua_sid; /* Session ID */
unsigned short lua_max_length; /* Receive Buffer Length */
unsigned short lua_data_length; /* Data Length */
char far *lua_data_ptr; /* Data Buffer Pointer */
unsigned long lua_post_handle; /* Posting handle */

struct LUA_TH lua_th; /* LUA TH Fields */

struct LUA_RH lua_rh; /* LUA RH Fields */

struct LUA_FLAG1 lua_flag1; /* LUA_FLAG1 */

unsigned char lua_message_type; /* sna message command type */

struct LUA_FLAG2 lua_flag2; /* LUA_FLAG2 */

unsigned char lua_resv56[7]; /* Reserved Field */
unsigned char lua_encr_decr_option; /* Cryptography Option */
} ;

struct SLI_SEND_EX_SPECIFIC
{
unsigned char lua_sequence_number[2]; /* This field **MUST** be first */
unsigned long lua_data_length_ex; /* data length field for SLI_SEND_EX */
} ;

struct SLI_RECEIVE_EX_SPECIFIC
{
unsigned long lua_max_length_ex; /* max buffer for SLI_RECEIVE_EX */
unsigned long lua_data_length_ex; /* data length for SLI_RECEIVE_EX */
} ;


union LUA_SPECIFIC
{
struct SLI_OPEN open;
unsigned char lua_sequence_number[2]; /* sequence number */
struct SLI_SEND_EX_SPECIFIC send; /* extra length fields */
struct SLI_RECEIVE_EX_SPECIFIC receive; /* extra length fields */
unsigned char lua_peek_data[12]; /* Data Pending */
} ;

/*****************************************************************************/
/* GENERIC LUA VERB RECORD TYPEDEF DECLARATION */
/*****************************************************************************/
typedef struct
{
struct LUA_COMMON common; /* common verb header command */
union LUA_SPECIFIC specific; /* command specific portion of record */
} LUA_VERB_RECORD;

I had translate into the following:

///
/// LUA Specific structure.
///

[StructLayout(LayoutKind.Explicit, CharSet=CharSet.Ansi)]
internal struct LUA_SPECIFIC
{
[FieldOffset(0)]
public SLI_OPEN open;
///
/// sequence number
///

[FieldOffset(0), MarshalAs(UnmanagedType.ByValArray, SizeConst=2)]
public Byte[] lua_sequence_number;
///
/// extra length fields
///

[FieldOffset(0)]
public SLI_SEND_EX_SPECIFIC send;
///
/// extra length fields
///

[FieldOffset(0)]
public SLI_RECEIVE_EX_SPECIFIC receive;
///
/// Data Pending
///

[FieldOffset(0), MarshalAs(UnmanagedType.ByValArray, SizeConst=12)]
public Byte[] lua_peek_data;
}

///
/// LUA Verb record structure.
///

[StructLayout(LayoutKind.Sequential, CharSet=CharSet.Ansi)]
internal struct LUA_VERB_RECORD
{
///
/// common verb header command
///

internal LUA_COMMON common;
///
/// command specific portion of record
///

// [FieldOffset(68)]
internal LUA_SPECIFIC specific;
}
After the program running, it caused an exception with the message as following:

Could not load type LUA_SPECIFIC from assembly 0 because it contains an object field at offset 0 that is incorrectly aligned or overlapped by a non-object field.

Is there anyone who can convert the LUA_VERB_RECORD correctly?
Thank you very much.
AnswerRe: How to translate a union within a structure that also contains another union Pin
Magallo1-Jun-06 5:58
Magallo1-Jun-06 5:58 
GeneralRe: How to translate a union within a structure that also contains another union Pin
kosmas4-Jun-06 15:56
kosmas4-Jun-06 15:56 
GeneralRe: How to translate a union within a structure that also contains another union Pin
salilak11-Jul-07 0:01
salilak11-Jul-07 0:01 
GeneralGetData() needs a reference Pin
jeffb4220-Dec-05 11:09
jeffb4220-Dec-05 11:09 
GeneralRe: GetData() needs a reference Pin
rwilly649-Nov-07 11:24
rwilly649-Nov-07 11:24 
GeneralMarshalAsAttribute constructor Pin
Alexey A. Popov20-Dec-05 6:27
Alexey A. Popov20-Dec-05 6:27 
GeneralRe: MarshalAsAttribute constructor Pin
rwilly649-Nov-07 10:48
rwilly649-Nov-07 10:48 

General General    News News    Suggestion Suggestion    Question Question    Bug Bug    Answer Answer    Joke Joke    Praise Praise    Rant Rant    Admin Admin   

Use Ctrl+Left/Right to switch messages, Ctrl+Up/Down to switch threads, Ctrl+Shift+Left/Right to switch pages.