Rendering PDF Documents with Mupdf and P/Invoke in C#

• Download MupdfSharp.zip - 2.8 MB

Notice:

This article was left here for historical reasons. For P/Invoking the most recent version of MuPDF, please read this article:

Compiling MuPDF DLL to Render and Edit PDF Documents

Introduction

I've been looking for a PDF rendering engine on .NET Framework for quite a few years. However, there is NO free, native .NET based one available on this planet yet. At the end, I was attracted by a GNU licensed, fast and slim PDF rendering engine--Mupdf, and wrote my own C# wrapper for it. 

This article will introduce the first steps on writing such a P/Invoke based C# wrapper for the Mupdf engine and render PDF pages into picture files.  

Background and relative projects  

During the search for PDF render engines, I found two projects providing C# interfaces nicely. The libraries provided by these two projects can be used to render PDF pages using C#.

The first project was initially posted here on CodeProject.com, titled View PDF files in C# using the Xpdf and muPDF and finally settled down on GoogleCode, named pdfviewer-win32. It used C++/CLI to bridge C# and the Mupdf. The positive side of the C++/CLI approach is that it allows developers to access every most internal parts of the Mupdf library. The down side is that developers have to write many C++/CLI wrapper classes. Writing and debugging such kind of classes isn't a piece of cake, and many C# developers even don't know about C++.

The second project was hosted on GoogleCode, named mupdf-converter. It has 2 layers: the first one introduces some C++ classes to encapsulate the library; the second one provides some C functions as static methods exposed by those classes. Finally, those C functions are exported in a DLL for P/Invoke. In short, the flow of the wrapper is MuPDF -> C++ Wrapper -> C (exported as DLL functions) -> .NET P/Invoke.

Since there're existing projects such as pdfviewer-win32 or mupdf-converter, why not just use them? It was fine unless you want to keep up with the most recent development or use all functionalities provided by the Mupdf library, when you have to eventually dig into the Mupdf yourself. The pdfviewer-win32 project provides a lot of functionalities, but it had not been updated for more than one year when I wrote this article. The mupdf-converter library provides merely very few functionalities--You could not do other things with it except rendering PDF pages--and it works only on .NET 4.0 or above.

I studied the source code of mupdf-converter and found that it was quite simple and its two-layer-wrapper implementation was quite bloated. Actually it is unnecessary to use a C++ class to wrap functions from Mupdf. As Mupdf is written in C, it is possible to make a DLL out of it and call its export functions via P/Invoke directly. That is, the calling path can be shorten to MuPDF -> .NET P/Invoke. Inspired by this library, I began to make my own and hence wrote this article.

Warning: The following part of the article assumes that you know how to write P/Invoke functions in C#. If you don't know a bit about it, you can learn it from MSDN. If you don't want to take that trouble. Please use the existing library provided by pdfviewer-win32 or mupdf-converter.

The 'cooking' procedures

The procedure of making our P/Invoke-only MupdfSharp library includes the following steps.

1. Obtaining the Mupdf DLL library.
2. Learning the essential concepts and export functions in Mupdf. 
3. Writing the P/Invoke functions.

Obtaining the Mupdf DLL

If you have once downloaded and compiled the source code of Mupdf, you would probably find that the compiled output are several EXE files. There is no DLL library at all. So if you are not familiar with MAKE files, it can take you quite some hours to learn and modify the MAKE file in order to get the DLL library.

Fortunately there are always nice guys in this world. The developers of SumatraPDF, a slim PDF viewer program utilizing the power of Mupdf, are our savers. They have released their code files on Github and the compile result of their project does lay down a DLL library for you to reuse. So the steps can be quite simple. 

1. Go to the project host of SumatraPDF: https://github.com/sumatrapdfreader/sumatrapdf
2. Download the source code package (or use an SVN tool to synchronize with their latest work). 
3. Open Visual C++ (you can use the free Express version here) and load the project.
4. Select "Release" as the build configuration and compile the project. 
5. Find out the "libmupdf.dll" file out of the release folder.
6. You have the Mupdf DLL now.

The developers of SumatraPDF are very diligent programmers. They keep quit a closed track with the latest development of Mupdf and update their code frequently. Therefore, you can quite well trust them and use their library instead of trying to compile your own Mupdf DLL out of the official code.

Notice:

At the spring of this year, I found that the SumatraPDF developers seemed to have ceased the synchronization with the MuPDF library, quite some scores of months before.

As they said, I checked the API of MuPDF and understood their reason: "The issue, as usually, is that it's non-trivial amount of work. Sumatra has a fairly large amount of changes to mupdf, mupdf changed a lot, upgrading requires lots of work that isn't priority (at least not to me)."

I hence wrote an article on how to compile our own version of MuPDF.

2017-11-18

Learning the essential concepts and functions

Once you have the libmupdf.dll in your hands. You can begin to study the functions provided by the Mupdf library.

The library is written in C. In the C programming world, the function definitions are placed in header files, which have the file extension ".h".

In the SumatraPDF project, the header files of the MuPDF library are placed in the mupdf\include\mupdf folder.

From the documentation section of the MuPDF official website, five header files are listed--"fitz.h", "pdf.h", "html.h", "svg.h" and "xps.h". We only need to get started with the first two of them. Then we will follow them and find out the needed functions and structures. 

The following lines are part of the "fitz.h" file. When we are exploring the functionalities of the MuPDF library, we will follow the #include instruction, which indicates that those referred header files are used in the library as well, and we will examine them one by one.

#ifndef MUDPF_FITZ_H
#define MUDPF_FITZ_H

#include "mupdf/fitz/version.h"
#include "mupdf/fitz/system.h"
#include "mupdf/fitz/context.h"

// the rest of the file is omitted here

As we open the files listed in the fitz.h file or pdf.h file, for instance, context.h, we will see some definitions for functions and structures. The definitions of functions and structures will be essential for writing P/Invoke funtions later.

To learn about the working mechanism of MuPDF, we can start with the example file "source/tools/mudraw.c" listed in the MuPDF documentation web site. By studying the code, we can learn how MuPDF works.

Structures that hold the document content

Five key structures in fitz.h are used in the code, listed below. 

1. fz_context structure (a.k.a. fz_context_s defined in context.h): Used to hold information when working with PDF files (or other supported document formats). You have to prepare the context before working.
2. fz_document structure (a.k.a. fz_document_s defined in document.h): Used to hold the opened PDF document.
3. fz_page structure (a.k.a. fz_page_s defined in document.h): Used to work with PDF pages. Once you opened a document, you can load its pages and work with them.
4. fz_pixmap structure (a.k.a. fz_pixmap_s defined in pixmap.h): Represents the rendered visual results of a page. If you want to render the PDF document, you have to setup a pixmap and draw on it.
5. fz_device structure (a.k.a. fz_device_s defined in device.h): Represents the device on which the render result is drawn. Typically, we will prepare a the device from the pixmap to draw on. 

You don't have to care about the internal layout of those structures. It means that when we write the P/Invoke code, we don't have to setup corresponding structs in our C# code. Using IntPtrs to hold references to them will do. Knowing this can considerably simplify our works in the P/Invoke part later.

Functions that operate with the structures

With the knowledge of the above key structures, we find out the following useful functions in the example file.

1. fz_new_context_imp:  Creates the fz_context.
2. fz_free_context: Frees the resources used by the fz_context. 
3. fz_open_document_with_stream:  Creates a fz_document instance out of a given fz_stream.
4. fz_open_file_w: Opens a fz_stream with a Unicode encoded file name. Notice: If you don't work with documents which have non ASCII characters in their file names, you can just use the fz_open_document function to open documents, instead of using the combination of fz_open_file_w and fz_open_document_with_stream. 
5. fz_close_document:  Closes the fz_document.
6. fz_close: Closes the fz_stream.
7. fz_count_pages:  Gets the page numbers in a document.
8. fz_load_page:  Creates a fz_page instance for a given page number.
9. fz_free_page:  Frees the resources used by the fz_page.
10. fz_bound_page:  Gets the dimension of a page.
11. fz_new_pixmap:  Creates a fz_pixmap instance to hold the rendered visual result.
12. fz_clear_pixmap_with_value:  Fills the fz_pixmap with a color (usually white).
13. fz_new_draw_device:  Creates a fz_device to draw the rendered result.
14. fz_lookup_device_colorspace:  Gets a fz_colorspace structure, used in the fz_run_page function.
15. fz_run_page:  Renders the page to the specific fz_device.
16. fz_free_device:  Frees the resources used by the fz_device.
17. fz_drop_pixmap:  Frees the resources used by the fz_pixmap.
18. fz_pixmap_samples: Gets the data of the rendered fz_pixmap, used to render the Bitmap.

Notice: if you use the DLL compiled from SumatraPDF

In the recent release of the SumatraPDF, in order to downsize the DLL library file, the developers decided to strip some functions when compiling the libmupdf.dll file. Consequently some functions in the fitz.h file are not compiled into the DLL file and we have to use the equivalent exported functions from the pdf.h file instead.

Writing the P/Invoke functions

Note: The chapter will show you how the P/Invoke functions are composed. If you don't care about how they came from, just skip this chapter.

Once we know about what functions we are going to use, we can start writing P/Invoke functions in C#. Firstly we begin with the following code in fitz\context.h for the fz_new_context_imp function, since it creates the fz_context instance and the following of the code will rely on that object.

fz_context *fz_new_context_imp(fz_alloc_context *alloc, fz_locks_context *locks, unsigned int max_store, const char *version);

1. The fz_context * is the return value of the function, a pointer to a fz_context structure.  In our P/Invoke function, IntPtr will be used since we don't care about the internal structure of fz_context.
2. fz_new_context_imp is, of course, the function name.
3. fz_alloc_context * and fz_locks_context * are the types of the first two parameters of the function. When initializing the fz_context with the fz_new_context function, we can simply pass IntPtr.Zero for those two parameters, hereby we will use IntPtr in the corresponding positions of the P/Invoke function.
4. max_store determines the size of resources cached by the engine. The recommended size (256 << 20) provided by the header file equals 256 megabytes.
5. version is a string passed to the function.

Here is the P/Invoke function code for the fz_new_context_imp function. 

[DllImport (DLL, EntryPoint="fz_new_context_imp")]
static extern IntPtr NewContext (IntPtr alloc, IntPtr locks, uint max_store, string version); 

Since the caller actually doesn't have to know about alloc, locks and max_store, we can add an overload function to make the method look neat.

const uint FZ_STORE_DEFAULT = 256 << 20;
const string MuPDFVersion = "1.6";

public static IntPtr NewContext () {
    return NewContext (IntPtr.Zero, IntPtr.Zero, FZ_STORE_DEFAULT, MuPDFVersion);
}

But... you may ask, "Wait a minute. How do you know that we can pass two IntPtr.Zeros and a FZ_STORE_DEFAULT to the NewContext function? And what the heck is the magic string '1.6'?" Good questions. The answer for the first question is written in fitz\context.h--"when you don't need to preallocate some memory nor set locks, you can simply use NULL". In the world of P/Invoke, NULL means IntPtr.Zero, for the first two parameters. The value of FZ_STORE_DEFAULT was also mentioned there.

The answer to the second question can be found in fitz\context.h and fitz\version.h. At first, we see the following line in context.h:

#define fz_new_context(alloc, locks, max_store) fz_new_context_imp(alloc, locks, max_store, FZ_VERSION)

If you study the example code provided by MuPDF or SumatraPDF, you can see that they used fz_new_context instead of fz_new_context_imp. However, as the above line of code shows, the fz_new_context is just a defined macro in C, which was not exposed to the C# world out of the DLL. We have to call fz_new_context_imp instead when we P/Invoke the library.

And next, we can find out the definition to FZ_VERSION in version.h.

Warning: If you pass a version number which mismatches the version of the engine, the above function will return IntPtr.Zero instead of a fz_context instance. When you are compiling your own libmupdf.dll library, you shall at least take a look at the fitz\version.h file and check whether the FZ_VERSION definition matches the version parameter we pass into the function here.

Please refer to those *.h files in the fitz folder and the pdf folder when you have further questions.

Let's do the same thing to the rest functions and write P/Invoke functions accordingly. During the course we will inevitably encounter three more new structures, fz_bbox, fz_rectangle and fz_matrix, referenced by those functions listed above. They are quite simple ones. We just define them as structs in our code, according to their definitions in fitz.h. Eventually we will reach something similar to the following code ready to be used by P/Invoke. 

public struct BBox
{
    public int Left, Top, Right, Bottom;
}
public struct Rectangle
{
    public float Left, Top, Right, Bottom;
}
public struct Matrix
{
    public float A, B, C, D, E, F;
}
class NativeMethods {

    const uint FZ_STORE_DEFAULT = 256 << 20;
    const string DLL = "libmupdf.dll";
    const string MuPDFVersion = "1.6";
 
    [DllImport (DLL, EntryPoint="fz_new_context")]
    static extern IntPtr NewContext (IntPtr alloc, IntPtr locks, uint max_store, string version);
    public static IntPtr NewContext () {
        return NewContext (IntPtr.Zero, IntPtr.Zero, FZ_STORE_DEFAULT, MuPDFVersion);
    }

    [DllImport (DLL, EntryPoint = "fz_free_context")]
    public static extern IntPtr FreeContext (IntPtr ctx);
 
    [DllImport (DLL, EntryPoint = "fz_open_file_w", CharSet = CharSet.Unicode)]
    public static extern IntPtr OpenFile (IntPtr ctx, string fileName);

    [DllImport (DLL, EntryPoint = "pdf_open_document_with_stream")]
    public static extern IntPtr OpenDocumentStream (IntPtr ctx, IntPtr stm);

    [DllImport (DLL, EntryPoint = "fz_close")]
    public static extern IntPtr CloseStream (IntPtr stm);
 
    [DllImport (DLL, EntryPoint = "pdf_close_document")]
    public static extern IntPtr CloseDocument (IntPtr doc);
 
    [DllImport (DLL, EntryPoint = "pdf_count_pages")]
    public static extern int CountPages (IntPtr doc);
 
    [DllImport (DLL, EntryPoint = "pdf_bound_page")]
    public static extern void BoundPage (IntPtr doc, IntPtr page, ref Rectangle bound);

    [DllImport (DLL, EntryPoint = "fz_clear_pixmap_with_value")]
    public static extern void ClearPixmap (IntPtr ctx, IntPtr pix, int byteValue);
 
    [DllImport (DLL, EntryPoint = "fz_lookup_device_colorspace")]
    public static extern IntPtr LookupDeviceColorSpace (IntPtr ctx, string colorspace);
 
    [DllImport (DLL, EntryPoint = "fz_free_device")]
    public static extern void FreeDevice (IntPtr dev);
 
    [DllImport (DLL, EntryPoint = "pdf_free_page")]
    public static extern void FreePage (IntPtr doc, IntPtr page);
 
    [DllImport (DLL, EntryPoint = "pdf_load_page")]
    public static extern IntPtr LoadPage (IntPtr doc, int pageNumber);
 
    [DllImport (DLL, EntryPoint = "fz_new_draw_device")]
    public static extern IntPtr NewDrawDevice (IntPtr ctx, IntPtr pix);
 
    [DllImport (DLL, EntryPoint = "fz_new_pixmap")]
    public static extern IntPtr NewPixmap (IntPtr ctx, IntPtr colorspace, int width, int height);
 
    [DllImport (DLL, EntryPoint = "pdf_run_page")]
    public static extern void RunPage (IntPtr doc, IntPtr page, IntPtr dev, ref Matrix transform, IntPtr cookie);
 
    [DllImport (DLL, EntryPoint = "fz_drop_pixmap")]
    public static extern void DropPixmap (IntPtr ctx, IntPtr pix);
 
    [DllImport (DLL, EntryPoint = "fz_pixmap_samples")]
    public static extern IntPtr GetSamples (IntPtr ctx, IntPtr pix);
 
}
  

Using the code - the flow of the program 

The goal of this article is to render PDF documents into pictures. The procedure is quite straightforward. 

1. Loads the document. 
2. Iterates each page in the document.  
3. Renders each page to Bitmap and saves them to the disk. 
4. Releases allocated resources during the operation. 

The skeleton of the code flow  

The code is listed below.

static void Main (string[] args) {
    IntPtr ctx = NativeMethods.NewContext (); // Creates the context
    IntPtr stm = NativeMethods.OpenFile (ctx, "test.pdf"); // opens file test.pdf as a stream
    IntPtr doc = NativeMethods.OpenDocumentStream (ctx, ".pdf", stm); // opens the document
    int pn = NativeMethods.CountPages (doc); // gets the number of pages in the document
    for (int i = 0; i < pn; i++) { // iterate through each pages
        IntPtr p = NativeMethods.LoadPage (doc, i); // loads the page (first page number is 0)
        Rectangle b = new Rectangle ();
        b = NativeMethods.BoundPage (doc, p, ref b); // gets the page size
        using (var bmp = RenderPage (ctx, doc, p, b)) { // renders the page and converts the result to Bitmap
            bmp.Save ((i+1) + ".png"); // saves the bitmap to a file
        }
        NativeMethods.FreePage (doc, p); // releases the resources consumed by the page
    }
    NativeMethods.CloseDocument (doc); // releases the resources
    NativeMethods.CloseStream (stm);
    NativeMethods.FreeContext (ctx);
}

You can see that the flow of the above code is quite clean.

The rendition of the page

What is left undone is that we have not yet written the code for the RenderPage function. We will finish it with the following lines of code.  

static Bitmap RenderPage (IntPtr context, IntPtr document, IntPtr page, Rectangle pageBound) {
    Matrix ctm = new Matrix ();
    IntPtr pix = IntPtr.Zero;
    IntPtr dev = IntPtr.Zero;
 
    int width = (int)(pageBound.Right - pageBound.Left); // gets the size of the page
    int height = (int)(pageBound.Bottom - pageBound.Top);
    ctm.A = ctm.D = 1; // sets the matrix as the identity matrix (1,0,0,1,0,0)

    // creates a pixmap the same size as the width and height of the page
    pix = NativeMethods.NewPixmap (context, 
      NativeMethods.LookupDeviceColorSpace (context, "DeviceRGB"), width, height);
    // sets white color as the background color of the pixmap
    NativeMethods.ClearPixmap (context, pix, 0xFF);
 
    // creates a drawing device
    dev = NativeMethods.NewDrawDevice (context, pix);
    // draws the page on the device created from the pixmap
    NativeMethods.RunPage (document, page, dev, ctm, IntPtr.Zero);
 
    NativeMethods.FreeDevice (dev); // frees the resources consumed by the device
    dev = IntPtr.Zero;
 
    // creates a colorful bitmap of the same size of the pixmap
    Bitmap bmp = new Bitmap (width, height, PixelFormat.Format24bppRgb); 
    var imageData = bmp.LockBits (new System.Drawing.Rectangle (0, 0, 
                      width, height), ImageLockMode.ReadWrite, bmp.PixelFormat);
    unsafe { // converts the pixmap data to Bitmap data
        // gets the rendered data from the pixmap
        byte* ptrSrc = (byte*)NativeMethods.GetSamples (context, pix);
        byte* ptrDest = (byte*)imageData.Scan0;
        for (int y = 0; y < height; y++) {
            byte* pl = ptrDest;
            byte* sl = ptrSrc;
            for (int x = 0; x < width; x++) {
                //Swap these here instead of in MuPDF because most pdf images will be rgb or cmyk.
                //Since we are going through the pixels one by one
                //anyway swap here to save a conversion from rgb to bgr.
                pl[2] = sl[0]; //b-r
                pl[1] = sl[1]; //g-g
                pl[0] = sl[2]; //r-b
                //sl[3] is the alpha channel, we will skip it here
                pl += 3;
                sl += 4;
            }
            ptrDest += imageData.Stride;
            ptrSrc += width * 4;
        }
    }
    // free bitmap in memory
    bmp.UnlockBits (imageData);
    NativeMethods.DropPixmap (context, pix);
    return bmp;
}

OK, everything is ALMOST done. Just run the program and see each PDF pages in test.pdf converted into PNG files.

Points of Interest

There are quite a few problems you need to notice during practical development.

The first thing to notice is exceptions: documents may be corrupted, engaged, etc. Mupdf library does throw exceptions when such kinds of problems occur. However, since we are doing P/Invokes. All we can do is to catch the AccessViolationException, and rethrow specific, redefined exceptions, for example, PdfDocumentException can be retrown when we catch an exception while loading a document. The key is to find out when the exceptions will be thrown. You can find out information in the *.h files. 

The second thing to notice is to remember releasing the resources. You may develop classes to encapsulate the creation and destroy of objects. For example, you may write a MupdfPage class to handle stuff about fz_page. Inside the constructor of MupdfPage, it creates the fz_page instance. The class should implement the IDisposable interface and put the P/Invoke code for fz_free_page in the Dispose method to release the resources.

The third thing to think about is to expand the functionalities of your wrapper. Currently the wrapper introduced by this article allows you to convert PDF pages into pictures. You may want to do more things and have more fun out of Mupdf. There're several ways:

1. Dig more out of the pdf.h and pdf\*.h.  There are a lot more functions not covered by this article in that file.  For example, you can find functions to open password protected documents (search for pdf_needs_password and pdf_authenticate_password).
2. Learn from examples and existing open-source projects. The official Mupdf website has provided several examples and the above C header files online. The three projects mentioned in this article are also good references for you to go further. 
3. Notice again: The libmupdf.dll file compiled by the SumatraPDF project does not contain all "fz_" functions in the fitz.h file, but you can find the actual implementations from pdf.h. To find out what functions are missing, you can lookup the function names in the libmupdf.def file of the SumatraPDF source code. If a function name is listed there, it may be used in your P/Invoke code; otherwise, use the functions in pdf.h instead. 

The fourth thing to consider is running on 64-bit machines. This is one of the most common issues when using P/Invoke. The DLL provided in the download of this article, compiled from the source code of SumatraPDF is 32-bit. It must be called from the 32-bit .NET Framework. On the 64-bit machines, the default .NET Framework will be the 64-bit one, which will fail when P/Invoking the 32-bit Mupdf DLL. Rather than recompiling the Mupdf DLL as a 64-bit DLL, you can instead change the CPU platform of the C# project from Any CPU to x86. Therefore, the compiled .NET program will be forced to run on the 32-bit .NET Framework and work well with the 32-bit Mupdf DLL.

The fifth thing to think about may be the basic of image processing. In our example, we render the PDF page with the default resolution. But what if we want it to be smaller (as page thumbnails) or bigger (for easier reading)?  The "zoom" factor of the image lies in the Matrix structure that we pass into the RunPage function. If we modify its A and D number, the image will be scaled horizontally and vertically respectively (modifying other values will cause the image to be rotated, sheared or translated). If we set A or D to negative values, the rendered image will be flipped horizontally or vertically. By the way, don't forget to resize the dimension of the rendered pixmap and Bitmap to hold the resized image. Here's the code for your reference. You may probably program the zoomX and zoomY as parameters of the RenderPage function and replace the second code block in the original function with the rest lines of code below.

float zoomX = 1.0, zoomY = 1.0;

int width = (int)(zoomX * (pageBound.Right - pageBound.Left)); // gets the size of the scaled page
int height = (int)(zoomY * (pageBound.Bottom - pageBound.Top));
ctm.A = zoomX;
ctm.D = zoomY; // sets the matrix as (zoomX,0,0,zoomY,0,0) 

Acknowledgements  

• The pdfviewer-win32 project has introduced me to XPDF and Mupdf.
• The mupdf-converter project has inspired me to write this wrapper without using C++/CLI.
• The SumatraPDF project has helped a lot on compiling the DLL file.  

History 

• Added a note for the discontinued synchronization between SumatraPDF and MuPDF. 2017-9-15
• Updated to reflect the changes in the MuPDF websites. 2017-1-24
• Updated location of the SumatraPDF project. 2014-12-30
• Once again, revised to reflect the recent changes in SumatraPDF and MuPDF. 2014-10-9
• Revised to reflect the recent changes in SumatraPDF and MuPDF. 2013-9-29
• License changed to GPL3 (compatible to MuPDF's). 2013-3-28.