Click here to Skip to main content
Click here to Skip to main content
Go to top

Programming Memory-Mapped Files with the .NET Framework

, 4 Jan 2011
Rate this:
Please Sign up or sign in to vote.
An introduction to MMF and shared memory in .NET applications.

Introduction

First of all, what is a memory-mapped file or MMF? MMF is a kernel object that maps a disk file to a region of memory address space as the committed physical storage. In plain English, MMF allows you to reserve a range of addresses and use a disk file as the physical storage for the reserved addresses. When a MMF is created, you access the mapped disk file as if you were accessing memory. MMF makes accessing files easy. For example, Windows loads EXEs or DLLs using MMF. MMF is a corner stone for almost all modern forms of inter-process communication (or IPC). I felt inconvenient when early .NET Framework releases did not support MMF. The good news is that starting version 4, .NET Framework has built-in support for MMF!

This article has two objectives:

  • The tutorial will focus on sample MMF applications. For theoretical discussions, refer to Jeffery Richter's classic text "Programming Applications for Microsoft Windows - 4th Ed.", Chapter 17. For .NET Framework 4 MMF specifications, check out this MSDN page.
  • The tutorial is written for programmers who are familiar with the C# language. General knowledge of C++ shall help you to understand the discussion of shared memory transparency. In order to build and run these samples, you should have Visual Studio 2010 installed on your machine.

Imagine you have got some modules or applications written in native C/C++ and you want to use them. The old modules use MMF to exchange data. Without MMF support in earlier .NET, you can do two things among others:

  • Platform Invoke Win32 MMF support. You need to handle lots of marshalling between managed and unmanaged worlds.
  • You can always use Windows debug symbol support, which allows to access variables as symbols. However, this mechanism is rather involving and beyond beginners' comfort level. You have to fiddle with marshalling as well.

With .NET Framework 4, you just write an MMF-enabled application to exchange data with old C/C++ applications directly. This article will demonstrate how to write MMF-enabled .NET applications. We start from basic examples. Then we'll move on to more advanced use of MMF in the shared memory design.

Simple MMF Applications

We demonstrate three sample MMF applications:

  • File Copy
  • Simple IPC
  • Single Instance

File Copy

The first sample program copies contents from an existing file to a new file. The source code is self-explaining, and listed as follows:

using System;
using System.IO;
using System.IO.MemoryMappedFiles;

namespace NetMMFCopyFile
{
    class Program
    {
        static void Main(string[] args)
        {
            int offset = 0;
            int length = 0;
            byte[] buffer;

            if (args.GetLength(0) != 2)
            {
                Console.WriteLine("Usage: NetMMFCopyFile.exe file1 file2");
                return;
            }

            FileInfo fi = new FileInfo(args[0]);
            length = (int)fi.Length;

            // Create unnamed MMF
            using (var mmf1 = MemoryMappedFile.CreateFromFile(args[0], 
                FileMode.OpenOrCreate, null, offset + length))
            {
                // Create reader to MMF
                using (var reader = mmf1.CreateViewAccessor(offset, 
                length, MemoryMappedFileAccess.Read))
                {
                    // Read from MMF
                    buffer = new byte[length];
                    reader.ReadArray<byte>(0, buffer, 0, length);
                }
            }

            // Create disk file
            using (FileStream fs = File.Create(args[1]))
            {
                fs.Close();
            }
 
            // Create unnamed MMF
            using (var mmf2 = MemoryMappedFile.CreateFromFile(args[1], 
                FileMode.Create, null, offset + length))
            {
                // Create writer to MMF
                using (var writer = mmf2.CreateViewAccessor(offset, 
                length, MemoryMappedFileAccess.Write))
                {
                    // Write to MMF
                    writer.WriteArray<byte>(0, buffer, 0, length);
                }
            }
        }
    }
}

A number of things are noticeable. We first call the static method CreateFromFile() to create an MMF object. The MemoryMappedFile class provides several overloaded static methods:

  • CreateFromFile() creates an MMF from the existing disk file.
  • CreateNew() creates an MMF based on the system page file.
  • CreateOrOpen() creates a new MMF or opens an existing MMF.
  • OpenExisting() opens an existing MMF.

The second and third methods create non-persisted MMF. In Windows implementation, non-persisted MMF is backed by the system page file. It only exists during the scope of the MMF.

We called CreatedViewAccessor() to create a view to the MMF we created. The MMF view can be read-only, or read-and-write, depending on the accessibility parameter we passed to the call. The view can map all or portions of the MMF. As you access the data through a view, you can only access the data that is mapped in the view. There are also two overloaded methods to create a view for the MMF:

  • CreateViewAccessor() creates an access object from the MMF.
  • CreateViewStream() creates a stream from the MMF.

Obviously, you can use CreateViewStream() to modify the above sample application, which I leave to you as an exercise. The checks for the existence of copy-to file are omitted for brevity. You should add logic to handle such cases as that the copy-to file already exists.

Simple IPC

In the second example, we'll discuss how two applications exchange data via MMF. Obviously, we could extend the copy file example in such a way that two applications can exchange data through the disk file. Basically, one application writes data to the file and the other reads data from the same file.

A better mechanism is to exchange data without depending on the presence of disk files. MMF can support exactly that. Here is the complete code to create an MMF and write some bytes to it:

using System;
using System.IO.MemoryMappedFiles;
using System.Threading;

namespace NetMMF_Provider
{
    class Program
    {
        static void Main(string[] args)
        {
            int offset = 0;
            int length = 32;
            byte[] buffer = new byte[length];

            if (args.GetLength(0) != 1)
            {
                Console.WriteLine("Usage: NetMMF_Provider.exe name");
                return;
            }
   
            // Fill buffer with some data
            for (int i = 0; i < buffer.Length; i++)
            {
                buffer[i] = (byte)('0' + i); 
            }

            // Create named MMF
            using (var mmf = MemoryMappedFile.CreateNew(args[0], offset + buffer.Length))
            {
                // Lock
                bool mutexCreated;
                Mutex mutex = new Mutex(true, "MMF_IPC", out mutexCreated);

                // Create accessor to MMF
                using (var accessor = mmf.CreateViewAccessor(offset, buffer.Length))
                {
                    // Write to MMF
                    accessor.WriteArray<byte>(0, buffer, offset, buffer.Length);
                }

                mutex.ReleaseMutex();

                // Press any key to exit...
                Console.ReadKey();
            }
        }
    }
}

Apparently, we created an MMF differently this time. Firstly, we called CreateNew() which has been discussed earlier. Secondly, the first parameter of CreateNew() is the name of the MMF we create. We call this MMF a named MMF. The purpose of naming is to identify the MMF among foreign processes.

Here is the code segment for accessing the MMF and reading data from it:

// Create named MMF
using (var mmf = MemoryMappedFile.OpenExisting(args[0]))
{
    // Create accessor to MMF
    using (var accessor = mmf.CreateViewAccessor(offset, buffer.Length))
    {
        // Wait for the lock
        Mutex mutex = Mutex.OpenExisting("MMF_IPC");
        mutex.WaitOne();

        // Read from MMF
        accessor.ReadArray<byte>(0, buffer, 0, buffer.Length);
    }
}

In the reader code, we call the CreateOrOpen() method because the MMF may have already been created. Basically, if it was created, we open it. If not, we create it. It is critical that you pass the correct name to the call. Otherwise, you're creating a new MMF, instead of sharing the existing one as you thought.

You may be wondering why we create a mutex in both the writer and reader code. It is for synchronizing data access between the two processes. The writer locks the mutex, writes to the MMF, and then releases the lock. The reader waits on the lock until the writer finishes writing and releases it. Then the reader reads the data from the MMF. That is how we ensure data integrity, which is no different from other forms of IPC.

Single Instance

The so called "Single Instance" means that you only want one instance of your program running at a given time. Since MMF can be uniquely named, you can use MMF as a token to indicate whether your application is already running. The idea is simple: one cannot create the same named MMF twice. Here is the code segment:

static void Main(string[] args)
{
    string name = "NetMMFSingle";

    // Create named MMF
    try
    {
        var mmf = MemoryMappedFile.CreateNew(name, 1);
    } catch
    {
        Console.WriteLine("Instance already running...");
        return;
    }

    ConsoleKeyInfo key = Console.ReadKey();

    // Your real code follows...
}

Hope you have found MMF useful. We are ready to explore more of its power.

Shared Memory

You have probably heard of "Shared Memory" now and then. This terminology usually refers to a data block that is shared by more than one application that is running on one or more computer stations. There are several different implementations of shared memory solutions, hardware, software, or combined. This article is focused on MMF based shared memory design, which is an extension of what you've seen in the Simple IPC sample.

When you design a shared memory solution, some basic features are commonly required:

  • Shared memory shall be data oriented, i.e., all the data to be exchanged should be wrapped in a structure for easy access.
  • Access to the shared memory is transparent, i.e., using the shared memory as if you were using the data structure directly.
  • Access to the shared memory shall be protected to ensure data integrity.

The following code represents a limited implementation of a shared memory class in .NET 4:

using System;
using System.IO;
using System.Collections.Generic;
using System.IO.MemoryMappedFiles;
using System.Threading;

namespace NetSharedMemory
{
    public class SharedMemory<T> where T: struct
    {
        // Constructor
        public SharedMemory(string name, int size)
        {
            smName = name;
            smSize = size;
        }

        // Methods
        public bool Open()
        {
            try
            {
                // Create named MMF
                mmf = MemoryMappedFile.CreateOrOpen(smName, smSize);
                
                // Create accessors to MMF
                accessor = mmf.CreateViewAccessor(0, smSize, 
                               MemoryMappedFileAccess.ReadWrite);

                // Create lock
                smLock = new Mutex(true, "SM_LOCK", out locked);
            }
            catch
            {
                return false;
            }

            return true;
        }

        public void Close()
        {
            accessor.Dispose();
            mmf.Dispose();
            smLock.Close();
        }

        public T Data
        {
            get 
            {
                T dataStruct;
                accessor.Read<T>(0, out dataStruct);
                return dataStruct; 
            }
            set 
            {
                smLock.WaitOne();
                accessor.Write<T>(0, ref value);
                smLock.ReleaseMutex();
            }
        }

        // Data
        private string smName;
        private Mutex smLock;
        private int smSize;
        private bool locked;
        private MemoryMappedFile mmf;
        private MemoryMappedViewAccessor accessor;
    }
}

Now we write a sample application to use this SharedMemory class. First we create some data.

public struct Point 
{
    public int x;
    public int y;
}

public struct MyData
{
    public int myInt;
    public Point myPt;
}

To keep the code simple, I declared all attributes in the structures public. In the real world, you should protect the data and expose them through properties. Anyway, now we write a class to use SharedMemory. The class shall read the data from the shared memory block, modify the data, and finally write the changes back to the shared memory.

class Program
{
    static void Main(string[] args)
    {
        SharedMemory<MyData> shmem = 
             new SharedMemory<MyData>("ShmemTest", 32);
        
        if(!shmem.Open()) return;

        MyData data = new MyData();
        
        // Read from shared memory
        data = shmem.Data;
        Console.WriteLine("{0}, {1}, {2}", 
            data.myInt, data.myPt.x, data.myPt.y);
       
        // Change some data
        data.myInt = 1;
        data.myPt.x = 2;
        data.myPt.y = 3;

        // Write back to shared memory
        shmem.Data = data;

        // Press any key to exit
        Console.ReadKey();

        // Close shared memory
        shmem.Close();        
    }
}

The code is by all means very simple. Build the project "SharedMemoryClient". Run two instances of it. You will notice that the data the second instance reads contains some changes made by the first instance.

There are a number of places you should pay attention to:

  • ShareMemory is a generic (parameterized) class that takes a structure as the parameter to the class. That structure parameter is a place holder for the data to be shared.
  • When protecting the shared memory, we only lock writing to, not reading from, the data block. This is purely a practical consideration for the sake of performance. You may protect both reading and writing depending on applications.
  • The structure copy we performed is a shallow, not deep one. In other worlds, the data we exchange should not contain reference types. This makes sense as we have no knowledge of what the data block actually contains. Shallow copy is usually enough as shared memory is a data oriented, not object oriented design.

Points of Interest

The implementation of Shared Memory presented above does not fully meet the requirements of transparency, mainly due to the lack of support for pointer operations in C#. In native C++, overloading the operator "->" would provide a better solution. Unfortunately, we couldn't overload the C# operator "." as we do with the C++ operator "->". As such, the combined use of property and structure copy is a somehow compromised solution.

As we've seen, there could be different implementations of MMF. C++ and C# implement MMF differently. Managed and unmanaged platforms also implement MMF differently. One question we haven't addressed is: can .NET 4 applications exchange data with applications that were written and built in VC++ 6.0? The answer is yes. In Windows internals, MMF is a named kernel object, like a mutex. What really matters with MMF is its name, size of the memory it reserves, and offset of the memory address. Note that the size and offset are important because you want to ensure the data you are accessing is not truncated or otherwise corrupted!

History

This is the first revision of the article and the source code.

License

This article, along with any associated source code and files, is licensed under The Code Project Open License (CPOL)

Share

About the Author

Jun Du
Architect GuestLogix Inc.
Canada Canada
Jun is an experienced software architect. He wrote his first computer code on the tape machine for a "super computer". The tape machine reads holes on the black pape tape as source code. When manually fixing code, you need a punch and tranparent tape. To delete code, you block holes or cut off a segment and glue two ends together. To change code, you block old holes and punch new holes. You already know how to add new code, don't you? Anyway, that was his programming story in early 1980's.
 
Jun completed university with the specialty in oceanography, and graduate study in meteorology. He obtained his Ph.D. in physics. Jun has worked in a number of different areas. Since mid-90's, he has been working as a software professional in both military & commercial industries, including Visual Defence, Atlantis Systems International and Array Systems Computing.
 
Currently, Jun is an architect at GuestLogix, the global leader in providing onboard retail solutions for airlines and other travel industries. He is also the founder of Intribute Dynamics, a consulting firm specialized in software development. He has a personal blog site, although he is hardly able to keep it up to date.
 
In his spare time, Jun loves classic music, table tennis, and NBA games. During the summer, he enjoyes camping out to the north and fishing on wild lakes.

You may also be interested in...

Comments and Discussions

 
QuestionNice article PinmemberKountree11-Apr-14 3:25 
QuestionSo where is that shared memory??? PinmemberMember 966248323-Sep-13 23:58 
QuestionA slightly revised approach PinmemberJerry Evans24-Nov-12 6:06 
QuestionCasting MMF data to a Class PinmemberCrooked path27-Sep-12 7:43 
AnswerRe: Casting MMF data to a Class PinmemberPatilvaishalimangesh10-Jun-13 18:37 
AnswerRe: Casting MMF data to a Class Pinmemberharveyt30-Mar-14 12:23 
GeneralRe: Casting MMF data to a Class Pinmemberfredthered24-Jul-14 13:50 
QuestionYou should implement destructor and IDispose PinmemberYaron Shkop7-Sep-12 1:43 
QuestionWhen should I call Close() PinmembertheCPkid20-Jun-12 4:47 
SuggestionMake a series of these Pinmembertorial18-May-12 16:58 
Hi Jun,
 
I'd love to see you make a series of articles on Memory Mapped Files -- it is an area that I think isn't discussed much. Possible topics:
a) How to use MemoryMappedFiles for IPC (which you've started covering, but you also hinted about sharing memory across nodes -- examples in an article would be awesome -- especially compared to sharing info via a DB)
b) How to use MemoryMappedFiles for bypassing RAM constraints (this is my most pressing need, but I've never quite figured out how to approach this w/ Memory Mapped Files)
 
Perhaps other could chime in-- but if this is an area that you are strong in, I think those articles would be well received.
Questionprivilege problems PinmemberMember 86176402-Feb-12 20:16 
QuestionBug in Mutex Usage PinmemberRed Rover10-Jan-12 4:37 
AnswerRe: Bug in Mutex Usage PinmemberJohn J Jesus25-Aug-14 9:28 
QuestionIs MMF automatically released? PinmemberHoang Cuong12-Jan-11 23:15 
AnswerRe: Is MMF automatically released? PinmemberJun Du13-Jan-11 3:15 
GeneralMy vote of 5 PinmemberEddy Vluggen4-Jan-11 22:36 
GeneralRe: My vote of 5 PinmemberJun Du5-Jan-11 6:07 
General"due to the lack of support for pointer operations in C#" Pinmemberspringy7621-Dec-10 2:17 
GeneralNice, but... PinmemberJohn Simmons / outlaw programmer20-Dec-10 23:31 
GeneralI would really like to give this article a 5 PinmemberKenJohnson20-Dec-10 23:13 
GeneralRe: I would really like to give this article a 5 PinmemberDewey4-Jan-11 20:04 

General General    News News    Suggestion Suggestion    Question Question    Bug Bug    Answer Answer    Joke Joke    Rant Rant    Admin Admin   

Use Ctrl+Left/Right to switch messages, Ctrl+Up/Down to switch threads, Ctrl+Shift+Left/Right to switch pages.

| Advertise | Privacy | Mobile
Web01 | 2.8.140926.1 | Last Updated 4 Jan 2011
Article Copyright 2010 by Jun Du
Everything else Copyright © CodeProject, 1999-2014
Terms of Service
Layout: fixed | fluid