Click here to Skip to main content
14,835,749 members
Articles / Programming Languages / Visual Basic
Article
Posted 8 Mar 2021

Tagged as

Stats

7.7K views
12 bookmarked

Fast Conversions between tightly packed Structures and Arrays

Rate me:
Please Sign up or sign in to vote.
4.76/5 (10 votes)
15 Mar 2021CPOL6 min read
Interpreting tightly packed structures as 1-dimensional arrays and vice-versa.
In this article, two fast-performing methods are presented to interpret arbitrary arrays of primitive datatypes as structure records, and vice-versa.

Introduction

Unions would be the simplest solution for interpreting arrays as structures, and structures as arrays, making conversions unneeded. However, while simple unions are possible in .NET, it is not possible to overlap value types with arrays, because these are reference types: only the array's address would overlap the start region of the sequence intended to be overlapped.

Interpreting the same data differently does not require copy operations, though. This section shows two fast-performing methods doing such conversions.

1.1 Array to Structure

Imagine a situation in which you retrieve some dozen bytes from persistent memory and want to fill the fields of a structure record with those. The namespace Runtime.InteropServices has appropriate methods to deal with such situations. However, they need to be used sensibly.

For instance, an often seen suggestion is to use the Marshal.SizeOf method on the structure to be filled, then to allocate an appropriately dimensioned buffer in a bytes array, proceed to use one of a variety of Copy methods (Array.Copy is also mentioned frequently) to fill this buffer, obtain a pinned handle with the GCHandle.Alloc method, then use Marshal.PtrToStructure with the obtained handle on the structure type, which after casting to the structure type will contain the data. (Don't forget to release the handle with GCHandle.Free.)

This is overly complicated, as the involved slow copy operation is unnecessary in most cases (as is the call to the SizeOf method). A possible function to convert a byte array to a record of a hypothetical structure STarget without copying anything would be:

C#

C#
private STarget ArrayToStructure(byte[] abSource)
{
    GCHandle iHandle;
    STarget rTarget;
    try
    {
        iHandle = GCHandle.Alloc(abSource, GCHandleType.Pinned);
        rTarget = (STarget)Marshal.PtrToStructure(iHandle.AddrOfPinnedObject(), 
            typeof(STarget));
    }
    finally
    {
        iHandle.Free();
    }
    return rTarget;
}

VB

vb.net
Private Function ArrayToStructure(ByVal abSource As Byte()) As STarget
    Dim iHandle As GCHandle
    Dim rTarget As STarget
    Try
        iHandle = GCHandle.Alloc(abSource, GCHandleType.Pinned)
        rTarget = CType(Marshal.PtrToStructure(iHandle.AddrOfPinnedObject(), 
            GetType(STarget)), STarget)
    Finally
        iHandle.Free()
    End Try
    Return rTarget
End Function

While this does fulfill the request to give access to the bytes array via a structure record's members, it is quite a specific solution targeted at just the structure type STarget.

The next code snippet addresses this shortcoming. (The modifications are highlighted.) The .NET Framework version 4.5.1 introduced a generic version for Marshal.PtrToStructure, which opens up the possibility to act not only on a specific structure, but on all structures, thus eliminating the need for an explicit cast. (The generic name S was chosen over the traditional T to represent a structure. The reason will be apparent shortly.)

C#

C#
private S ArrayToStructure<S>(byte[] abSource) where S : struct
{
    GCHandle iHandle;
    S rTarget;
    try
    {
        iHandle = GCHandle.Alloc(abSource, GCHandleType.Pinned);
        rTarget = Marshal.PtrToStructure<S>(iHandle.AddrOfPinnedObject());
    }
    ...
}

VB

vb.net
Private Function ArrayToStructure(Of S As Structure)(ByVal abSource As Byte()) As S
    Dim iHandle As GCHandle
    Dim rTarget As S
    Try
        iHandle = GCHandle.Alloc(abSource, GCHandleType.Pinned)
        rTarget = Marshal.PtrToStructure(Of S)(iHandle.AddrOfPinnedObject())
    ...
End Function

With minimal changes, the method has evolved into a generic version now, which makes accessible about any structure on a bytes array. And markedly, on a bytes array only, so the method still is of somewhat limited use: what if the array is to represent Unicode characters, or 3-dimensional locations in double precision?

To make the method truly versatile, the array should be generic itself as well. And this is indeed possible, requiring just minimal changes once again. (A represents a generic array, in contrast to S representing a generic structure.)

C#

C#
private S ArrayToStructure<A, S>(A aoArray) where S : struct
{
    GCHandle iHandle;
    S rTarget;
    try
    {
        iHandle = GCHandle.Alloc(aoArray, GCHandleType.Pinned);
        ...
    }
}

VB

vb.net
Private Function ArrayToStructure(Of A, S As Structure)(ByVal aoArray As A) As S
    Dim iHandle As GCHandle
    Dim rTarget As S
    Try
        iHandle = GCHandle.Alloc(aoArray, GCHandleType.Pinned)
        ...
End Function

The above method now allows any array of primitive datatypes to be interpreted as any structure record, but it's still not flawless. Before proceeding with incorporating the method, it might be a good idea to check out what could go wrong.

S is constrained to be a non-nullable structure, so Nothing can not be provided by the caller. It would be nice, if a constraint existed also to accept only arrays, but MS laconically states:

The following types may not be used as constraints: Object, Array, or ValueType.

For Object, this obviously makes sense, as ultimately everything is an object, so there's no need in specifying such a constraint. I'm not entirely sure, however, why Array was excluded from the constraints list; it might be for historic reasons. (Apparently, the ones listed in the quote belong to so-called special classes.) Fortunately, the constraint can easily be achieved by changing the method's signature to accept arrays only instead of specifying a generic:

C#

C#
private S ArrayToStructure<S>(System.Array aData) where S : struct
{
    GCHandle iHandle;
    S rTarget;
    try
    {
        iHandle = GCHandle.Alloc(aData, GCHandleType.Pinned);
        ...
    }
}

VB

vb.net
Private Function ArrayToStructure(Of S As Structure)(ByVal aData As System.Array) As S
    Dim iHandle As GCHandle 
    Dim rTarget As S
    Try
        iHandle = GCHandle.Alloc(aData, GCHandleType.Pinned)
        ...
End Function

Now only arrays can be fed to the method, returning the result to any structure record.

Of course, an array is a reference type, thus the caller can pass Nothing as the array. In doing so, iHandle.AddrOfPinnedObject will yield 0, and Marshal.PtrToStructure with 0 will throw the exception System.NullReferenceException. For efficiency reasons, however, no error handling is implemented on this method, which is supposed to be fast. If the caller can not ensure non-null input, this must be caught there.

This is the whole method.

C#

C#
public S ArrayToStructure<S>(System.Array aData) where S : struct
{
    GCHandle iHandle;
    S sRecord;
    try
    {
        iHandle = GCHandle.Alloc(aData, GCHandleType.Pinned);
        sRecord = Marshal.PtrToStructure<S>(iHandle.AddrOfPinnedObject());
    }
    finally
    {
        iHandle.Free();
    }
    return sRecord;
}

VB

vb.net
Public Function ArrayToStructure(Of S As Structure)(
    ByVal aData As System.Array) As S

    Dim iHandle As GCHandle
    Dim sRecord As S
    Try
        iHandle = GCHandle.Alloc(aData, GCHandleType.Pinned)
        sRecord = Marshal.PtrToStructure(Of S)(iHandle.AddrOfPinnedObject())
    Finally
        iHandle.Free()
    End Try
    Return sRecord
End Function

Assume the existence of a tightly packed structure like this:

C#

C#
[StructLayout(LayoutKind.Explicit)]
public struct STest
{
    [FieldOffset(0)]
    public UInt32 Field1;
    [FieldOffset(4)]
    public UInt32 Field2;
    [FieldOffset(8)]
    public UInt64 Field3;
}

VB

vb.net
<StructLayout(LayoutKind.Explicit)>
Public Structure STest
    <FieldOffset(0)> Public Field1 As UInt32
    <FieldOffset(4)> Public Field2 As UInt32
    <FieldOffset(8)> Public Field3 As UInt64
End Structure

and a bytes array initialized as follows:

C#

C#
byte[] abArray = new byte[16];
for (byte i = 0; i <= 15; i++)
    abArray[i] = i;

VB

vb.net
Dim abArray(0 To 15) As Byte
For i As Byte = 0 To 15
    abArray(i) = i
Next

The caller would then use the above method thus:

C#

C#
STest rTest = ArrayToStructure<STest>(abArray);

VB

vb.net
Dim rTest As STest = ArrayToStructure(Of STest)(abArray)

Care must be taken on the interpretation of the retrieved data. If the bytes array is to be converted into a record of the above structure, the result depends on the array's layout in RAM and the endianness of the involved architecture. On little-Endian machines like Intel's, this might produce unintuitive results. The contents in the following table are shown in hex.

Array Element 00 01 02 03 04 05 06 07 08 09 10 11 12 13 14 15
Element Value 00 01 02 03 04 05 06 07 08 09 0A 0B 0C 0D 0E 0F
RAM Layout 00 01 02 03 04 05 06 07 08 09 0A 0B 0C 0D 0E 0F
Field Value 0x03020100 0x07060504 0x0F0E0D0C0B0A0908
Structure Record Field1 Field2 Field3

This can easily be visualized with:

C#
Console.WriteLine("{0} {1} {2}", Hex(rTest.Field1), Hex(rTest.Field2), 
    Hex(rTest.Field3))

The console will then show (without the leading zeroes):

03020100 07060504 0F0E0D0C0B0A0908

1.2 Structure to Array

On the way back, from a structure record to an array, a complication needs to be considered: it is not possible to instantiate an abstract System.Array type in order to return a such: the compiler cannot infer what the caller expects the array type to be. Thus, the type must be communicated to the method. There are at least two possibilities to do so:

Result Generics Arguments Comments
TypedArray() Of S As Struct Struct, DesiredArrayType Most .NET-conforming signature, but building and returning the array is very expensive.
[None] Of S As Struct,
E As Struct
RecordToInterpret,
[Out]InitializedArray()
Not really the most intuitive .NET style, but array is readily available, thus it's very fast.

Following the same reasonings as the other way round, a quite concise method can be written to achieve the desired array-wise interpretation, both the structure type S as well as the array element type E being generics. The array needs to be instantiated appropriately by the caller. Once again, no size determination nor expensive copying is required.

C#

C#
public void StructureToArray<S, E>(S rStruct, ref E[] aArray)
     where S : struct
     where E : struct
{
    GCHandle iHandle;
    try
    {
        iHandle = GCHandle.Alloc(aArray, GCHandleType.Pinned);
        Marshal.StructureToPtr<S>(rStruct, iHandle.AddrOfPinnedObject(), false);
    }
    finally
    {
        iHandle.Free();
    }
}

VB

vb.net
Public Sub StructureToArray(Of S As Structure, E As Structure)(
    ByVal rStruct As S, ByRef aArray() As E)

    Dim iHandle As GCHandle
    Try
        iHandle = GCHandle.Alloc(aArray, GCHandleType.Pinned)
        Marshal.StructureToPtr(Of S)(
            rStruct, iHandle.AddrOfPinnedObject(), False)
    Finally
        iHandle.Free()
    End Try
End Sub

Also, this method is very flexible. Thus, care must be taken in interpreting the result, especially in little-endian systems like Intel's.

Assuming the existence of the same structure, record and array as in the previous subsection, with the record being initialized as follows (leading zeroes for illustration purposes only, they will be removed in actual code):

C#

C#
rTest.Field1 = 0x00010203;
rTest.Field2 = 0x04050607;
rTest.Field3 = 0x08090A0B0C0D0E0F;

VB

vb.net
With rTest
    .Field1 = &H00010203
    .Field2 = &H04050607
    .Field3 = &H08090A0B0C0D0E0F
End With

If users require this record for whatever reason being returned as an array of UInt32 elements, they would use the method thus:

C#

C#
UInt32[] aiArray = new UInt32[4];
StructureToArray<STest, UInt32>(rTest, aiArray);

VB

vb.net
Dim aiArray(0 To 3) As UInt32
StructureToArray(Of STest, UInt32)(rTest, aiArray)

Providing this record with the instruction to return an array of four UInt32 elements will cause a re-interpretation to happen, from the actual layout present in RAM.

Structure Record Field1 Field2 Field3
Field Value 0x00010203 0x04050607 0x08090A0B0C0D0E0F
RAM Layout 03 02 01 00 07 06 05 04 0F 0E 0D 0C 0B 0A 09 08
Element Value 0x00010203 0x04050607 0x0C0D0E0F 0x08090A0B
Array Element Element 0 Element 1 Element 2 Element 3

To visualize:

C#
Console.WriteLine("{0} {1} {2} {3}", Hex(aiArray(0)), Hex(aiArray(1)), 
    Hex(aiArray(2)), Hex(aiArray(3)))

in order to display the array's elements on the console (again with leading zeroes for illustration purposes only):

00010203 04050607 0C0D0E0F 08090A0B

History

  • 7th March, 2021: Initial version

License

This article, along with any associated source code and files, is licensed under The Code Project Open License (CPOL)

Share

About the Author

Visual Herbert
Switzerland Switzerland
No Biography provided

Comments and Discussions

 
QuestionSpan and ReadOnlySpan? Pin
Andreas Saurwein23-Mar-21 1:44
MemberAndreas Saurwein23-Mar-21 1:44 
QuestionHow does this benchmark? Pin
ShawnVN9-Mar-21 21:02
MemberShawnVN9-Mar-21 21:02 
AnswerRe: How does this benchmark? Pin
Member 792845817-Mar-21 7:15
MemberMember 792845817-Mar-21 7:15 
GeneralRe: How does this benchmark? Pin
Visual Herbert18-Mar-21 6:21
MemberVisual Herbert18-Mar-21 6:21 
GeneralMy vote of 5 Pin
DaMi VietNam9-Mar-21 14:05
MemberDaMi VietNam9-Mar-21 14:05 
GeneralRe: My vote of 5 Pin
Visual Herbert9-Mar-21 14:18
MemberVisual Herbert9-Mar-21 14:18 
QuestionRecommend changing term "primitive datatype" to "blittable datatype" Pin
TnTinMn9-Mar-21 12:11
MemberTnTinMn9-Mar-21 12:11 
Questionplease show the missing C# code for: Pin
BillWoodruff9-Mar-21 2:56
mveBillWoodruff9-Mar-21 2:56 
AnswerRe: please show the missing C# code for: Pin
Visual Herbert9-Mar-21 4:29
MemberVisual Herbert9-Mar-21 4:29 
Suggestioninstructive Pin
Mr.PoorEnglish8-Mar-21 19:12
MemberMr.PoorEnglish8-Mar-21 19:12 
GeneralRe: instructive Pin
Visual Herbert9-Mar-21 2:38
MemberVisual Herbert9-Mar-21 2:38 
GeneralRe: instructive Pin
Mr.PoorEnglish9-Mar-21 5:54
MemberMr.PoorEnglish9-Mar-21 5:54 
GeneralRe: instructive Pin
Visual Herbert9-Mar-21 6:22
MemberVisual Herbert9-Mar-21 6:22 
QuestionFast Conversions between Structures and Arrays Pin
FalitPlasto8-Mar-21 5:56
MemberFalitPlasto8-Mar-21 5:56 
GeneralMy vote of 5 Pin
FalitPlasto8-Mar-21 5:41
MemberFalitPlasto8-Mar-21 5:41 
GeneralRe: My vote of 5 Pin
Visual Herbert8-Mar-21 18:39
MemberVisual Herbert8-Mar-21 18:39 

General General    News News    Suggestion Suggestion    Question Question    Bug Bug    Answer Answer    Joke Joke    Praise Praise    Rant Rant    Admin Admin   

Use Ctrl+Left/Right to switch messages, Ctrl+Up/Down to switch threads, Ctrl+Shift+Left/Right to switch pages.