Fast Conversions between tightly packed Structures and Arrays

Introduction

Unions would be the simplest solution for interpreting arrays as structures, and structures as arrays, making conversions unneeded. However, while simple unions are possible in .NET, it is not possible to overlap value types with arrays, because these are reference types: only the array's address would overlap the start region of the sequence intended to be overlapped.

Interpreting the same data differently does not require copy operations, though. This section shows two fast-performing methods doing such conversions.

1.1 Array to Structure

Imagine a situation in which you retrieve some dozen bytes from persistent memory and want to fill the fields of a structure record with those. The namespace Runtime.InteropServices has appropriate methods to deal with such situations. However, they need to be used sensibly.

For instance, an often seen suggestion is to use the Marshal.SizeOf method on the structure to be filled, then to allocate an appropriately dimensioned buffer in a bytes array, proceed to use one of a variety of Copy methods (Array.Copy is also mentioned frequently) to fill this buffer, obtain a pinned handle with the GCHandle.Alloc method, then use Marshal.PtrToStructure with the obtained handle on the structure type, which after casting to the structure type will contain the data. (Don't forget to release the handle with GCHandle.Free.)

This is overly complicated, as the involved slow copy operation is unnecessary in most cases (as is the call to the SizeOf method). A possible function to convert a byte array to a record of a hypothetical structure STarget without copying anything would be:

private STarget ArrayToStructure(byte[] abSource)
{
    GCHandle iHandle;
    STarget rTarget;
    try
    {
        iHandle = GCHandle.Alloc(abSource, GCHandleType.Pinned);
        rTarget = (STarget)Marshal.PtrToStructure(iHandle.AddrOfPinnedObject(), 
            typeof(STarget));
    }
    finally
    {
        iHandle.Free();
    }
    return rTarget;
}

Private Function ArrayToStructure(ByVal abSource As Byte()) As STarget
    Dim iHandle As GCHandle
    Dim rTarget As STarget
    Try
        iHandle = GCHandle.Alloc(abSource, GCHandleType.Pinned)
        rTarget = CType(Marshal.PtrToStructure(iHandle.AddrOfPinnedObject(), 
            GetType(STarget)), STarget)
    Finally
        iHandle.Free()
    End Try
    Return rTarget
End Function

While this does fulfill the request to give access to the bytes array via a structure record's members, it is quite a specific solution targeted at just the structure type STarget.

The next code snippet addresses this shortcoming. (The modifications are highlighted.) The .NET Framework version 4.5.1 introduced a generic version for Marshal.PtrToStructure, which opens up the possibility to act not only on a specific structure, but on all structures, thus eliminating the need for an explicit cast. (The generic name S was chosen over the traditional T to represent a structure. The reason will be apparent shortly.)

private S ArrayToStructure<S>(byte[] abSource) where S : struct
{
    GCHandle iHandle;
    S rTarget;
    try
    {
        iHandle = GCHandle.Alloc(abSource, GCHandleType.Pinned);
        rTarget = Marshal.PtrToStructure<S>(iHandle.AddrOfPinnedObject());
    }
    ...
}

Private Function ArrayToStructure(Of S As Structure)(ByVal abSource As Byte()) As S
    Dim iHandle As GCHandle
    Dim rTarget As S
    Try
        iHandle = GCHandle.Alloc(abSource, GCHandleType.Pinned)
        rTarget = Marshal.PtrToStructure(Of S)(iHandle.AddrOfPinnedObject())
    ...
End Function

With minimal changes, the method has evolved into a generic version now, which makes accessible about any structure on a bytes array. And markedly, on a bytes array only, so the method still is of somewhat limited use: what if the array is to represent Unicode characters, or 3-dimensional locations in double precision?

To make the method truly versatile, the array should be generic itself as well. And this is indeed possible, requiring just minimal changes once again. (A represents a generic array, in contrast to S representing a generic structure.)

private S ArrayToStructure<A, S>(A aoArray) where S : struct
{
    GCHandle iHandle;
    S rTarget;
    try
    {
        iHandle = GCHandle.Alloc(aoArray, GCHandleType.Pinned);
        ...
    }
}

Private Function ArrayToStructure(Of A, S As Structure)(ByVal aoArray As A) As S
    Dim iHandle As GCHandle
    Dim rTarget As S
    Try
        iHandle = GCHandle.Alloc(aoArray, GCHandleType.Pinned)
        ...
End Function

The above method now allows any array of primitive datatypes to be interpreted as any structure record, but it's still not flawless. Before proceeding with incorporating the method, it might be a good idea to check out what could go wrong.

S is constrained to be a non-nullable structure, so Nothing can not be provided by the caller. It would be nice, if a constraint existed also to accept only arrays, but MS laconically states:

The following types may not be used as constraints: Object, Array, or ValueType.

For Object, this obviously makes sense, as ultimately everything is an object, so there's no need in specifying such a constraint. I'm not entirely sure, however, why Array was excluded from the constraints list; it might be for historic reasons. (Apparently, the ones listed in the quote belong to so-called special classes.) Fortunately, the constraint can easily be achieved by changing the method's signature to accept arrays only instead of specifying a generic:

private S ArrayToStructure<S>(System.Array aData) where S : struct
{
    GCHandle iHandle;
    S rTarget;
    try
    {
        iHandle = GCHandle.Alloc(aData, GCHandleType.Pinned);
        ...
    }
}

Private Function ArrayToStructure(Of S As Structure)(ByVal aData As System.Array) As S
    Dim iHandle As GCHandle 
    Dim rTarget As S
    Try
        iHandle = GCHandle.Alloc(aData, GCHandleType.Pinned)
        ...
End Function

Now only arrays can be fed to the method, returning the result to any structure record.

Of course, an array is a reference type, thus the caller can pass Nothing as the array. In doing so, iHandle.AddrOfPinnedObject will yield 0, and Marshal.PtrToStructure with 0 will throw the exception System.NullReferenceException. For efficiency reasons, however, no error handling is implemented on this method, which is supposed to be fast. If the caller can not ensure non-null input, this must be caught there.

This is the whole method.

public S ArrayToStructure<S>(System.Array aData) where S : struct
{
    GCHandle iHandle;
    S sRecord;
    try
    {
        iHandle = GCHandle.Alloc(aData, GCHandleType.Pinned);
        sRecord = Marshal.PtrToStructure<S>(iHandle.AddrOfPinnedObject());
    }
    finally
    {
        iHandle.Free();
    }
    return sRecord;
}

    Public Function ArrayToStructure(Of S As Structure)(
        ByVal aData As System.Array) As S

        Dim iHandle As GCHandle
        Dim sRecord As S
        Try
            iHandle = GCHandle.Alloc(aData, GCHandleType.Pinned)
            sRecord = Marshal.PtrToStructure(Of S)(iHandle.AddrOfPinnedObject())
        Finally
            iHandle.Free()
        End Try
        Return sRecord
    End Function

Assume the existence of a tightly packed structure like this:

[StructLayout(LayoutKind.Explicit)]
public struct STest
{
    [FieldOffset(0)]
    public UInt32 Field1;
    [FieldOffset(4)]
    public UInt32 Field2;
    [FieldOffset(8)]
    public UInt64 Field3;
}

<StructLayout(LayoutKind.Explicit)>
Public Structure STest
    <FieldOffset(0)> Public Field1 As UInt32
    <FieldOffset(4)> Public Field2 As UInt32
    <FieldOffset(8)> Public Field3 As UInt64
End Structure

and a bytes array initialized as follows:

byte[] abArray = new byte[16];
for (byte i = 0; i <= 15; i++)
    abArray[i] = i;

Dim abArray(0 To 15) As Byte
For i As Byte = 0 To 15
    abArray(i) = i
Next

The caller would then use the above method thus:

C#

STest rTest = ArrayToStructure<STest>(abArray);

VB

Dim rTest As STest = ArrayToStructure(Of STest)(abArray)

Care must be taken on the interpretation of the retrieved data. If the bytes array is to be converted into a record of the above structure, the result depends on the array's layout in RAM and the endianness of the involved architecture. On little-Endian machines like Intel's, this might produce unintuitive results. The contents in the following table are shown in hex.

This can easily be visualized with:

Console.WriteLine("{0} {1} {2}", Hex(rTest.Field1), Hex(rTest.Field2), 
    Hex(rTest.Field3))

The console will then show (without the leading zeroes):

03020100 07060504 0F0E0D0C0B0A0908

1.2 Structure to Array

On the way back, from a structure record to an array, a complication needs to be considered: it is not possible to instantiate an abstract System.Array type in order to return a such: the compiler cannot infer what the caller expects the array type to be. Thus, the type must be communicated to the method. There are at least two possibilities to do so:

Following the same reasonings as the other way round, a quite concise method can be written to achieve the desired array-wise interpretation, both the structure type S as well as the array element type E being generics. The array needs to be instantiated appropriately by the caller. Once again, no size determination nor expensive copying is required.

public void StructureToArray<S, E>(S rStruct, ref E[] aArray)
     where S : struct
     where E : struct
{
    GCHandle iHandle;
    try
    {
        iHandle = GCHandle.Alloc(aArray, GCHandleType.Pinned);
        Marshal.StructureToPtr<S>(rStruct, iHandle.AddrOfPinnedObject(), false);
    }
    finally
    {
        iHandle.Free();
    }
}

    Public Sub StructureToArray(Of S As Structure, E As Structure)(
        ByVal rStruct As S, ByRef aArray() As E)

        Dim iHandle As GCHandle
        Try
            iHandle = GCHandle.Alloc(aArray, GCHandleType.Pinned)
            Marshal.StructureToPtr(Of S)(
                rStruct, iHandle.AddrOfPinnedObject(), False)
        Finally
            iHandle.Free()
        End Try
    End Sub

Also, this method is very flexible. Thus, care must be taken in interpreting the result, especially in little-endian systems like Intel's.

Assuming the existence of the same structure, record and array as in the previous subsection, with the record being initialized as follows (leading zeroes for illustration purposes only, they will be removed in actual code):

rTest.Field1 = 0x00010203;
rTest.Field2 = 0x04050607;
rTest.Field3 = 0x08090A0B0C0D0E0F;

With rTest
    .Field1 = &H00010203
    .Field2 = &H04050607
    .Field3 = &H08090A0B0C0D0E0F
End With

If users require this record for whatever reason being returned as an array of UInt32 elements, they would use the method thus:

UInt32[] aiArray = new UInt32[4];
StructureToArray<STest, UInt32>(rTest, aiArray);

Dim aiArray(0 To 3) As UInt32
StructureToArray(Of STest, UInt32)(rTest, aiArray)

Providing this record with the instruction to return an array of four UInt32 elements will cause a re-interpretation to happen, from the actual layout present in RAM.

To visualize:

Console.WriteLine("{0} {1} {2} {3}", Hex(aiArray(0)), Hex(aiArray(1)), 
    Hex(aiArray(2)), Hex(aiArray(3)))

in order to display the array's elements on the console (again with leading zeroes for illustration purposes only):

00010203 04050607 0C0D0E0F 08090A0B

History

• 7^th March, 2021: Initial version