Click here to Skip to main content
Click here to Skip to main content

Generic BinaryReader and BinaryWriter Extensions

, 26 Feb 2009 CPOL
Rate this:
Please Sign up or sign in to vote.
Generics and extension methods in C++/CLI
A view from .NET Reflector

Introduction

A while ago, I was working on an application that needed to convert between structs and byte[]. I had many different kinds of structs that I needed to work with, and at first, I tried to maintain a large number of methods for each struct type. You could imagine how bloated the code was.

Eventually, I decided I would give a crack at making all of that conversion code generic, and this miniature project was born. The project provides a static class (like BitConverter) and extension methods for both BinaryReader and BinaryWriter to enable conversion between structs (of any kind) and a series of bytes as an Array or Stream.

Getting Started: The Impossible Way

Since I was somewhat familiar with C# already, naturally I would start the project in C# as well. To start it off, I wrote some code for one of the two conversion methods (shown below) and hit compile:

public static unsafe byte[] GetBytes<T>(T value)
{
    // Check to see if the input is null.
    if (value == null)
    {
        throw new ArgumentNullException();
    }
 
    // Check to see if the input has value type semantics.
    if (!(value is ValueType))
    {
        throw new ArgumentException();
    }
 
    // Create an array of bytes the same size as the input.
    byte[] bytes = new byte[sizeof(T)];
 
    // Pin the byte array down so the garbage collector can't move it.
    fixed (byte* p1 = &bytes[0])
    {
        // Cast the pointer type from byte to T.
        T* p2 = (T*)p1;
 
        // Assign the value to the byte array.
        *p2 = value;
    }
 
    // Return the byte array.
    return bytes;
}

Instead of compiling successfully, Visual Studio gave me an error: “Cannot take the address of, get the size of, or declare a pointer to a managed type ('T').” This was logical. After all, T was not guaranteed to have value type semantics; you can't take the size of something that doesn't have a definite size, and a class can have any size during runtime unless it happens to derive from System.ValueType (e.g. Byte, Int32, UInt64, Single, Decimal, DateTime…). The next logical step would be to force the input to be a ValueType. I removed the error checking code and constrained the type parameter to ValueType:

public static unsafe byte[] GetBytes<T>(T value) where T : ValueType
{
    // Create an array of bytes the same size as the input.
    byte[] bytes = new byte[sizeof(T)];
 
    // Pin the byte array down so the garbage collector can't move it.
    fixed (byte* p1 = &bytes[0])
    {
        // Cast the pointer type from byte to T.
        T* p2 = (T*)p1;
 
        // Assign the value to the byte array.
        *p2 = value;
    }
 
    // Return the byte array.
    return bytes;
}

After hitting the compile key, the error that Visual Studio gave me this time baffled me: “Constraint cannot be special class 'System.ValueType'.” Upon further research, I came upon an MSDN article that suggested constraining the type parameter to a struct. The constraint was accepted by the compiler, but again, it complained that taking the address, pointer, or size of a generic type was not allowed. (Smart compiler, eh?) This, to me, seemed like a dead end. How else was I supposed to write this method?

The C++/CLI Way

I also happened to be playing around with C++/CLI in Visual Studio at around the same time, when an idea hit me. If C++/CLI could do so many things that C# could not (like mixing managed code with unmanaged code), could it let me get around both restrictions in C#? There was only one way to find out, and after an hour of reading C++/CLI tutorials, I came up with this:

generic <typename T>
where T : System::ValueType
array<unsigned char> ^GenericBitConverter::GetBytes(T value)
{
       array<unsigned char> ^bytes = gcnew array<unsigned char>(sizeof(T));
       *reinterpret_cast<interior_ptr<T>>(&bytes[0]) = value;
       return bytes;
}

The method actually worked, and in just 3 lines of code!

Here is the expanded, more readable version (equivalent to the above, but split up to make the steps more obvious):

generic <typename T>
where T : System::ValueType
array<unsigned char> ^Iku::GenericBitConverter::GetBytes(T value)
{
       // Create an array of bytes the same size as the input.
       array<unsigned char> ^bytes = gcnew array<unsigned char>(sizeof(T));
 
       // Create a pointer to the byte array.
       interior_ptr<unsigned char> ptr1 = &bytes[0];
 
       // Cast the pointer to the same type as the input.
       interior_ptr<T> ptr2 = reinterpret_cast<interior_ptr<T>>(ptr1);
 
       // Assign the value to the byte array.
       *ptr2 = value;
 
       // Return the byte array.
       return bytes;
}

GenericBitConverter

… and that is the GenericBitConverter. It works with any type that is derived from ValueType (a struct), and thus will work with all the basic .NET data types like double and decimal (except string, which is really an object) as well as enumerations and other complex structures.

Making an Extension Method in C++/CLI

Now that the GenericBitConverter was in place, it would be nice if the same functionality were available to the BinaryReader and BinaryWriter classes, so I decided to extend those classes with the same functionality as well. This was not any easier than getting the GenericBitConverter to work properly. Writing the method was easy at first:

generic <typename T>
where T : System::ValueType
T Iku::IO::BinaryReaderExtension::ReadValue(BinaryReader ^binaryReader)
{
       return GenericBitConverter::ToValue<T>
		(binaryReader->ReadBytes(sizeof(T)), 0);
}

It just would not show up as an extension method in the C# code editor.

It was time to investigate why, so I wrote one in C# and I disassembled the assembly. While inspecting the output, I found the ExtensionAttribute attribute applied to three places: the assembly, the static class containing the extension method, and the extension method itself. Applying the attributes to the proper place in the C++/CLI project, I was able to get the extension method to show up:

Extension method for BinaryWriter

… and there are the generic BinaryReader and BinaryWriter extensions.

Now you can do stuff like this:

DateTime timeStamp = binaryReader.ReadValue<DateTime>();
static void Main(string[] args)
{
    SampleEnumeration sampEnum = SampleEnumeration.All ^ SampleEnumeration.Four;

    byte[] enumBytes = GenericBitConverter.GetBytes<SampleEnumeration>(sampEnum);

    // Writes "f0 ff ff ff ff ff ff ff"
    foreach (byte bite in enumBytes)
    {
        Console.Write("{0:x2} ", bite);
    }

    Console.WriteLine();
}

enum SampleEnumeration : long
{
    None = 0x0000000000000000,
    All = -1L,
    One = 0x0000000000000001,
    Four = 0x000000000000000f
}

Neat? Feel free to use this within your own C# or Visual Basic projects (by referencing the output of this project). It works perfectly!

Points of Interest

  • C++/CLI also lets you constrain generic parameters to the ValueType type; it does not complain. C# just demands struct in place of ValueType.
  • The C++/CLI unsigned char type is actually a synonym for the Byte type and not the Char type. The C++/CLI Char type is actually wchar_t.
  • Interior pointers (interior_ptr) point to their objects, but do not fix them in place. This is markedly different from C#’s fixed behavior, which fixes the object in memory and prevents the garbage collector from moving it (as long as the fixed statement is in scope).
  • Although C++/CLI does not offer any syntactic sugar for creating extension methods, it can be done by placing an ExtensionAttribute attribute on the method, its containing class, and its containing assembly. (ExtensionAttribute can be found under the namespace System::Runtime::CompilerServices) This nature of extension methods means it can be made from any language that supports attributes, static classes, and static methods.
  • The C# compiler will assume a generic type can be any regular object even if you constrain it to be a struct (or a ValueType). I have not found a way around it in C# (and thus this project was born).

Updates

  • Feb. 27, 2009: Added a download for the compiled assembly; not everyone has the ability to compile a C++/CLI project (users of the Express Editions come to mind). Now everyone can use it!

License

This article, along with any associated source code and files, is licensed under The Code Project Open License (CPOL)

Share

About the Author

Kevin Li (Li, Ken-un)

United States United States
No Biography provided

Comments and Discussions

 
BugCLI runtime may rearrange fields. PinmemberZergusik5-Feb-13 10:32 
QuestionIs there a way to achieve this generic byte converter in VB.NET? PinmemberShimmy Weitzhandler17-Jul-10 14:33 
GeneralMy vote of 5 PinmemberShimmy Weitzhandler17-Jul-10 14:31 
Questionassembly attribute for c++/cli assembly PinmemberMember 127498925-Feb-10 3:59 
GeneralC# version (not exact, but equivalent) [modified] Pinmembergunters1-Jun-09 5:29 
GeneralRe: C# version (not exact, but equivalent) PinmemberDaniel Grunwald31-Dec-09 10:49 
GeneralPure C# Version? [modified] PinmemberRichard Deeming5-Mar-09 6:53 
GeneralRe: Pure C# Version? PinmemberLee, Gun-Woon5-Mar-09 18:42 
GeneralEfficiency (VM and usage) [modified] PinmemberUser of Users Group1-Mar-09 4:17 
GeneralRe: Efficiency (VM and usage) PinmemberLee, Gun-Woon2-Mar-09 11:00 
GeneralRe: Efficiency (VM and usage) [modified] PinmemberUser of Users Group3-Mar-09 12:56 
It is just an overload, so your interface remains more natural but you provide the user with an option (less natural, but even better performance). But I agree with you too. Btw, framework design guidelines from Brad and co has this type of empty discussion we need to read and then unlearn everything they preach if one is to move out of OO-chains and into HPC. Their assumptions are on avoiding interfaces, etc and plenty of other advice, which is clearly essential in any useful C# generics and communication scenarios; one where this and all other generic techniques start to make a bit of sense. Doh!
 
Also another advantage is at least a reduction of one type specification for the case of ToValues if IIRC.
 
> Visual Studio C++/CLI automatically target's the x86 platform
 
Not necessarily, try compiling while running on x64 OS and you could easily hit an exception immediatelly against a specific x86 build. But that is more to do with image format though. The default is 'Any CPU' config, and LLVM or whatever the model makes it safer.. but go across implementations it is an unknown land.
 
You can force these corflags in PE header however. Anyway, there are many scenarios where it won't work cross platform without some manual verification and tweaking, including being very careful with types and its members and attributes etc. On Itanium this code in native form would crash it outright but for CLI I am not sure..
 
I was more aiming at native interop, more later.
 
> We're still talking about C# as it is right now. By the way, I think the univoter (the guy who votes 1 on > people's posts) is stalking you. Your posts seem to turn out grey when I encounter them.
 
[Yeah, we are friends really. He is the guy that does what 99% of other dogs on the street do, chase trains, cars and walks proud every day at how advanced cee-sharper SmallTalk/Java lego toy is (35 years later and the handicap compiler still around and VM and framework more bloated than ever; not to mention a crap JIT and 3.5 memory lang.object-everything eater).]
 
There is more to it though. If you do interop with native C++, you better know what you are doing TM. Including understanding why your compiler pads in thousands of scenarios, alignment, sizeof assumptions, inheritance, you name it. Then there is a VM doing its own thing, and reserves all rights to change.. So even for interop using interop, MS makes sure you stay alert and locked-in.. Too many things to mention, but I agree it is controllable.. and doing this is about as safe as you can be.
 
Multi-dispatch is another story.. (something C# will see in v48.0, done efficiently that is).
 
Btw, I didn't vote on the article..
 
modified on Tuesday, March 3, 2009 7:02 PM

GeneralNote PinmemberUser of Users Group1-Mar-09 3:59 
GeneralRe: Note PinmemberLee, Gun-Woon2-Mar-09 11:03 
GeneralRe: Note PinmemberUser of Users Group3-Mar-09 12:48 
JokeGood job. Pinmemberdevace1-Mar-09 2:49 
GeneralRe: Good job. PinmemberLee, Gun-Woon2-Mar-09 11:08 
Questionbroken link? PinmemberKenan E.K.27-Feb-09 7:25 
AnswerRe: broken link? PinmemberLee, Gun-Woon27-Feb-09 7:58 
GeneralRe: broken link? PinmemberKenan E.K.27-Feb-09 12:47 
GeneralRe: broken link? PinmemberLee, Gun-Woon27-Feb-09 17:01 
GeneralA very good one PinmemberAnandChavali27-Feb-09 1:03 
AnswerRe: A very good one PinmemberLee, Gun-Woon27-Feb-09 4:28 
GeneralExcellent PinmemberGSerjo26-Feb-09 22:00 
AnswerRe: Excellent PinmemberLee, Gun-Woon27-Feb-09 4:33 
GeneralRe: Excellent Pinmemberapynes27-Feb-09 7:43 
AnswerRe: Excellent PinmemberLee, Gun-Woon27-Feb-09 8:09 
GeneralRe: Excellent Pinmemberapynes27-Feb-09 9:21 
AnswerRe: Excellent PinmemberLee, Gun-Woon27-Feb-09 12:32 
GeneralRe: Excellent PinmemberUser of Users Group28-Feb-09 1:16 
JokeRe: Excellent PinmemberShimmy Weitzhandler17-Jul-10 14:32 

General General    News News    Suggestion Suggestion    Question Question    Bug Bug    Answer Answer    Joke Joke    Rant Rant    Admin Admin   

Use Ctrl+Left/Right to switch messages, Ctrl+Up/Down to switch threads, Ctrl+Shift+Left/Right to switch pages.

| Advertise | Privacy | Terms of Use | Mobile
Web03 | 2.8.141030.1 | Last Updated 27 Feb 2009
Article Copyright 2009 by Kevin Li (Li, Ken-un)
Everything else Copyright © CodeProject, 1999-2014
Layout: fixed | fluid