Click here to Skip to main content
Click here to Skip to main content

Typesafe wrapping of a C++ Quant library

, 3 May 2014 CPOL
Rate this:
Please Sign up or sign in to vote.
Use advanced techniques to provide a high performance, concurrent, typesafe API around a non-threadsafe typeless one

Introduction

Using a native C++ library from C# is never straightforward but ones involving Quantative Finance present their own unique set of challenges in part due to the mathematical focus of those that work in this field - aka the "Quants".

This article uses some advanced techniques (in the C# and C++ languages, in the Windows O/S and even in Visual Studio itself) to provide a high performance, concurrent, typesafe C# API:

  • Presents a Variant-centric design for a typical native, non-threadsafe, C-like Quant library including analytic and numeric pricing of basic ("vanilla") options (via Financial Numerical Recipes);
  • A single C++ Translator DLL to wrap this library in a generic way and to translate exceptions;
  • And a very typesafe, Quant-centric, C# API allowing for concurrency.

Some of the techniques and technologies demonstrated include:

  • Generated API via XML markup using inbuilt Visual Studio support for XSDs and T4 templates;
  • Support for skipping or defaulting arguments, mapping "magic" strings to enumerations and handling C++ unions in C#;
  • Use of generic disposable structs to eliminate all boxing and minimise heap (GC) use;
  • Direct run time generation of P-Invoke functors including manual DLL loading and decoding;
  • Native Variant memory handling in C# including typesafe reuse between native calls;
  • High performant C# Variant wrapper using unsafe C# code for direct BSTR and SAFEARRAY en-/de-coding;
  • Use of C++ macros, variadic templates and runtime walking of the Export Address Table to create the single native Translator DLL;
  • Advanced use of Windows Side by Side manifests to add concurrency to a non-threadsafe native library.

Motivation

This section shows a simple example followed by real Quant library-style calls:

The First Example

Given a C++ signature for Function4 within the NativeAPI.DLL like this:

//_Function4@16
extern "C" VARIANT __declspec(dllexport) __stdcall
Function4(VARIANT* _index, VARIANT* _arg1, VARIANT* _arg2, VARIANT* _arg3)

This is often deliberately done to provide an intuitive VBA declaration like this:

Declare Function Function4 Lib "NativeAPI" Alias "_Function4@16" _
(Index, Arg1, Optional Arg2, Optional Arg3)

Unfortunately the obvious P-Invoke signature is not supported (as of .NET v4.5):

//1. This fails with a not supported error at run time
[DllImport("NativeAPI.DLL")]
extern static object Function4(
    ref object index, ref object first, 
    ref object second, ref object third);

So the ugly mangled name must be used with all arguments shifted to expose the hidden return argument that is part of the stdcall calling convention being used:

//2. This works due to the stack layout of stdcall
[DllImport("NativeAPI.DLL", EntryPoint = "_Function4@16")]
extern static void Function4(
    out object result, 
    ref object index, ref object first, 
    ref object second, ref object third);

The use of objects by reference makes everything non-typesafe, inefficient and painful because .NET will marshal structs, interfaces and classes into their COM equivalents causing runtime errors that are hard to troubleshoot:

try
{
    object index = 1;
    object first = "first";
    object second = null;
    object third = "third";

    //This works: both inputs and output are untyped
    object res;
    Function4(out res, ref index, ref first, ref second, ref third);

The bigger problem is that any C++ exceptions are swallowed by .NET and replaced with the same generic message, losing the essential error information provided by the Quant library:

     //This works but leaks the native C++ exception
    index = null;
    Function4_Works(out res, ref index, ref first, ref second, ref third);
}
catch (Exception ex)
{
    var msg = ex.Message;
    //C# only ever shows this message:
    //"System.Runtime.InteropServices.SEHException (0x80004005): 
    //External component has thrown an exception"

    //But the C++ error is really:
    //"Function4: index must be an integer"
}

The First Solution

The function is marked up using Visual Studio's inbuilt support for XML editing constrained to an XSD:

<function id="Function4" type="?Any">
   <arg id="Indexer" type="Integer"/>
   <arg id="Choice1" type="?Any"/>
   <arg id="Choice2" type="?Any"/>
   <arg id="Choice3" type="?Any"/>
</function>

Then the function can be efficiently called and the result retrieved with the native C++ exception being optionally rethrown as a C# one with full fidelity:

using (var fn = NativeFunctions.Function4.New())
{
    //Set arguments
    fn.Indexer.Set(1);
    fn.Choice1.Set(1.23);
    fn.Choice2.Set("String");
    //Choice3 is optional!

    //And call
    NativeFunctions.Result choice = fn.Invoke();

The debugger view at this point is instructive, accurately showing the native Variant type and value as interpreted by .NET itself:

Function4 in Debugger

The code continues, showing efficient re-calling and exception translation:

     //Efficiently deal with 'variant' results
    if (choice.Native.IsDouble)
    {
        double dbl = choice.GetDouble();
    }

    //Change one argument's value
    fn.Indexer.Set(3);
    //And call again
    choice = fn.Invoke();

    if (choice.Native.IsEmpty)
    {
        //Should be empty
    }

    //Force an error
    fn.Indexer.Set(-1);
    try
    {
        //And call again
        choice = fn.Invoke();
    } 
    catch (NativeFunctionException ex)
    {
        var msg = ex.Message;
        //Function4: index must be an integer
    }
}

The Complex Example

Given C++ signatures for functions within the NativeAPI.DLL as follows:

//namespace Native
#define API(FUNCTION) VARIANT __declspec(dllexport) __stdcall FUNCTION

API(CreateList)(VARIANT* _name, VARIANT* _list);

API(FunctionComplex)(VARIANT* _payOff, VARIANT* _frq, VARIANT* _overrides, 
VARIANT* _endDates, VARIANT* _roll, VARIANT* _ccy, VARIANT* _curve, 
VARIANT* _options, VARIANT* _schedule, VARIANT* _barrier);

Then the ugly mangled name must be used with all arguments shifted as before:

[DllImport("NativeAPI.DLL", EntryPoint = "?CreateList@Native@@YG?AUtagVARIANT@@PAU2@0@Z")]
extern static void CreateList(out object result, ref object name, ref object values); 

[DllImport("NativeAPI.DLL", EntryPoint = "?FunctionComplex@Native@@YG?AUtagVARIANT@@PAU2@000000000@Z")]
extern static void FunctionComplex(out object result, ref object payoff, ref object frq, ref object overrides, 
    ref object endDates, ref object roll, ref object ccy, ref object curve, ref object options, 
    ref object schedule, ref object barrier);

Firstly the overrides must be created as a named list within the Quant library:

object dummy;
object name = "Overrides";
//Note the shape of this array
object values = new object[,] 
{ 
    {"Amount", "PayDelay"},
    {0.9, 0.5}, //Amount
    {2, 2}      //PaymentLag
};
            
CreateList(out dummy, ref name, ref values);

Assume the schedule is stored in a typesafe structure somewhere:

var schedule = new KeyStruct<DateTime, DateTime, decimal>[]
{
    KeyStruct.Create(DateTime.Now, DateTime.Now.AddMonths(1), 1.0m),
    KeyStruct.Create(DateTime.Now.AddMonths(2), DateTime.Now.AddMonths(3), 1.1m)
};

The call setup is now quite tricky and requires intimate knowledge of the array shapes and magic strings as defined by different Quants working on the library often at different times:

object result;
//Magic strings
object payoff = "D";
object frq = "FourWeekly";
//Name of previously CreatedList prepended with '!'
object overrides = "!" + name;
//Note the array shape required; must remove times
object endDates = new object[,] { { DateTime.Now.Date, DateTime.Now.AddMonths(1).Date } };

//Can also be a string rule
object roll = 7;
object ccy = "AUD";
//Can also be the name of a pre-created curve
object curve = 0.1;

//More magic strings and fixed array shapes
object options = new object[,] { { "Extrapolate", true } };
//Note the array shape required; must remove times
object schedule2 = new object[,] { { DateTime.Now.Date, DateTime.Now.AddMonths(1).Date } };

//Must be very careful with shape (extra column) and casting here
var _barrier = new object[schedule.Length, 3 + 1];
for (int i=0; i<schedule.Length; i++)
{
    //Must remove any time component
    _barrier[i, 0] = schedule[i].Key.Date;
    _barrier[i, 1] = schedule[i].Value1.Date;
    //Must carefully cast decimal types
    _barrier[i, 2] = (double)schedule[i].Value2;
}
//Must always use local 'object' type for ref args
object barrier = _barrier;

FunctionComplex(out result, ref payoff, ref frq, ref overrides, ref endDates,
    ref roll, ref ccy, ref curve, ref options, ref schedule2, ref barrier);

Processing the result is equally painful especially if the type is not fixed:

//And now cast result
double res2 = (double)result;

The Complex Solution

The functions are again marked up but with a richer syntax:

<create id="CreateList" type="List">
   <name/>
   <arg id="Data" type="Any" isArray="2d"/>
</create>

<function id="FunctionComplex" type="Double">
   <enum id="Payoff" type="QuantBarrierType"/>
   <argT id="PaymentFrequency" type="?EnumOrString" T="QuantFrequency"/>
   <argT id="Overrides" type="?NamedList" T="QuantOverrides"/>
   <arg id="EndDates" type="?Date" isArray="1d"/>
   <arg id="Roll" type="?StringOrNumber"/>
   <arg id="SettlementCcy" type="?String"/>
   <arg id="SettlementCurve" type="?Curve"/>
   <argT id="Options" type="?Map" T="QuantCurveOptions"/>
   <argT id="Schedule" type="Matrix" T="QuantAsianAveragingSchedule"/>
   <argT id="LowBarrier" type="?Matrix" T="QuantContinuousBarrierSchedule,
                                           QuantDiscreteBarrierSchedule"/>
</function>

Again the overrides are created first but this time using enumerations and without any boxing:

var overrides = QuantNamedList<QuantOverrides>.New("Overrides");
using (var bldr = overrides.NewBuilder(2, 2))
{
    bldr.Add(QuantOverrides.Amount).SetDoubleColumn(new[] { 0.9, 0.5 });
    bldr.Add(QuantOverrides.PaymentLag).SetIntegerColumn(new[] { 2, 2 });

    bldr.Invoke();
}

Assuming the same typesafe structure for the schedule exists, then the setup is far easier and more efficient than before. Observe how an array of structures can be set directly without any boxing or (unbounded) heap usage:

using (var fn = NativeFunctions.FunctionComplex.New())
{
    //Easy way if sourcing a simple collection
    fn.EndDates.Set(new[] { DateTime.Now, DateTime.Now.AddMonths(1) });

    var bldBarrier = fn.LowBarrier.SetWithBuilder1(2);
    //Optimal approach when translating an array of structures into individual arrays
    bldBarrier.StartDates.Set(schedule, t => t.Key);
    bldBarrier.EndDates.Set(schedule, t => t.Value1);
    bldBarrier.Levels.Set(schedule, t => (double)t.Value2);

Support for typed matrices via IDisposable structs and custom mapping of enumerations to strings:

     using (var bldOptions = fn.Options.SetWithBuilder(1))
    {
        bldOptions.Add(QuantCurveOptions.Extrapolate, true);
    }
    fn.Payoff.Set(QuantBarrierType.Discrete);

Full typesafety for the string names used including in unions with numbers:

     //Use pre-set named map
    fn.Overrides.Set(overrides);
    fn.SettlementCurve.Set(0.1);

More unions including those mixing enumerations, strings and numbers:

     fn.PaymentFrequency.Set(QuantFrequency.FourWeekly);
    fn.Roll.Set(7);

Choice between simpler or more efficient but complex approaches:

     var bldAsian = fn.Schedule.SetWithBuilder(2);
    //Usual easiest approach:
    //bldAsian.Dates.Set(new[] { DateTime.Now, DateTime.Now.AddMonths(1) });
    //Optimal approach:
    bldAsian.Dates.SetAt(0, DateTime.Now);
    bldAsian.Dates.SetAt(1, DateTime.Now.AddMonths(1));

Choice of exception re-throwing or manual error handling and full typesafety of the result without any unnecessary heap usage (no use of 'object' even for nested arrays):

     fn.SettlementCcy.Set("AUD");

    double res = fn.Invoke();
}

The debugger view at this point is again instructive, accurately showing the native Variant type and value as interpreted by .NET itself. Most importantly, you see the actual values passed to the function, such as "D" for the Payoff even though an enumeration was used to set it:

FunctionComplex in Debugger

Classic Option Pricing with simple Volatility

Start with a local scope to show how to reuse a VARIANT value across function calls: in this case, the name of the fixings/observations for the underlying:

 using (var scope = NativeFunctions.Scope())
{
    //Instead of local scoped variables can use a TypedVariantCache
    //for the long term storage of typed native values.
    var fixings = VariantFixings.New(scope);

Now load the fixings/observations into the Quant libary, assign an arbitrary name, and enable reuse of the BSTR. Note how the library is expecting a typed matrix: the first column containing the dates and the last column the corresponding values:

     using (var fn = NativeFunctions.CreateFixings.New())
    {
        fn.AsOf.Set(DateTime.Now);
        fn.QuantName.Set("USDFixings");

        var builder = fn.DatesAndValues.SetWithBuilder(3);
        builder.Dates.Set(new[] { DateTime.Now, DateTime.Now.AddMonths(1), DateTime.Now.AddMonths(3) });
        builder.Numbers.Set(new[] { 1.1, 1.5, 1.9 });

        fn.Invoke();

        //Store for later - maintains typesafety
        fn.QuantName.Save(fixings);
    }

First price the option three months before expiry with flat risk free rate and volatility. Also calculate one risk measure (delta):

     double pvAnalytic;
    using (var fn = NativeFunctions.ModelOptionBlackScholes.New(QuantMeasure.PV, QuantMeasure.Delta))
    {
        fn.AsOf.Set(DateTime.Now);
        fn.ExpiryDate.Set(DateTime.Now.AddMonths(3));
        fn.Payoff.Set(QuantCallOrPut.Call);
        fn.Rate.Set(0.15);  //15%
        fn.Spot.Set(1.1);
        fn.Strike.Set(1.0);
        fn.Vol.Set(0.20);   //20%

        var res = fn.Invoke();

The result is a vector of VARIANTs, one for each measure. By knowing the measure types this can be efficiently decoded without any boxing:

         //1. PV = scalar
        pvAnalytic = res.GetResult().GetDouble();
        //2. Delta = scalar
        var delta = res.GetResult().GetDouble();

It is now straightforward to re-price the option at expiry, at which point the fixings/observations must be supplied. This is done below in a fully type safe manner that efficiently reuses the previously allocated BSTR for the name:

         //On expiry day and fixed
        fn.AsOf.Set(DateTime.Now.AddMonths(3));
        //The cast must be explicit as this approach is not typesafe
        //fn.Fixings.Set((QuantFixings)"USDFixings");
        //The typesafe way is:
        fn.Fixings.SetFromShared(fixings);

        res = fn.Invoke();

Results are again easy and efficient to access including handling of the empty case:

         //Fixed at 1.9, less 1.0 strike, gives 0.9 payoff
        var pvExpired = res.GetResult().GetDouble();
        //No delta once fixed
        var bIsEmpty = res.IsEmpty; //True
        delta = res.GetResult().GetDouble(); //NaN
    }
}

Numerical Option Pricing with complex Volatility

A complex volatility process is represented here by an unnamed (or named) map containing a named one, something like this:

 "VolProcess" -> { 
                  { "VolatilityProcess", 0.20 }, 
                  /* other options */ 
                }; 
{
 {"Vol",      "!VolProcess" },
 { "VolType", "Flat"}
}

This tricky layout (including the named map) can be created as follows:

var vp = QuantVolProcess<QuantVolOptions>.New("VolProcess", 0.20); //20%
//Not using any additional options
vp.SetNoOptions();

The pricing then proceeds as before:

 double pvNumerical;
using (var fn = NativeFunctions.ModelOptionFD.New(QuantMeasure.PV, QuantMeasure.Delta))
{
    fn.AsOf.Set(DateTime.Now);
    fn.ExpiryDate.Set(DateTime.Now.AddMonths(3));
    fn.Payoff.Set(QuantCallOrPut.Call);
    fn.Rate.Set(0.15);  //15%
    fn.Spot.Set(1.1);
    fn.Strike.Set(1.0);
    fn.VolProcess.Set(vp);

    var res = fn.Invoke();

    //1. PV = scalar
    pvNumerical = res.GetResult().GetDouble();

    //2. Delta = scalar (not calculated by this model)
    var bIsEmpty = res.IsEmpty; //True
    var delta = res.GetResult().GetDouble(); //NaN

Options to control the numeric algorithm can now be set before recalculating:

     //Now increase the steps
    var options = fn.Options.SetWithBuilder(2);
    options.Add(QuantFDModelOptions.SpotSteps, 100);
    options.Add(QuantFDModelOptions.TimeSteps, 100);
    //Alternative approach to a 'using' block:
    options.ThrowIfIncomplete();

    res = fn.Invoke();
    pvNumerical = res.GetResult().GetDouble();

}

var pvDiff = pvNumerical - pvAnalytic;

Solution Architecture

The example C++ Quant library design is divided up into a main NativeAPI.DLL and a dependent Utils.DLL. It is deliberately not thread safe in any way: specifically, static variables are used so only one instance of these DLLs is expected to be loaded into a single process and all calls to the library must occur from a single thread.

This kind of setup is common in Quant libraries that were primarily written for Excel 2003 and earlier; i.e. before Microsoft introduced concurrency support for XLLs. It is also still common if the library was primarily for use with VBA which does not support multiple threads.

By introducing a single C++ Translator.DLL, it is possible to automatically enumerate the public exports of the NativeAPI.DLL and decode the number of arguments at runtime. This then allows for the right wrapper function to be chained around the original to support API argument count changes between versions and exception translation.

Concurrency can then be achieved by introducing a Windows SxS manifest file and duplicating this along with the binaries, once for each thread desired.

Overview Diagram

Managed Topics

Discussion of the pure C# oriented design and implementation (the advanced interop is in the following section).

XSD Constrained XML Editing

Visual Studio has a good XML editor that provides suggestions, constraints and error checking in the presence of an XSD. It's XSD editor, however, really works best in pure XML mode. The first sections of the NativeFunctions.XSD follow:

<?xml version="1.0" encoding="utf-8"?>
<xs:schema id="NativeFunctions"
  targetNamespace="http://www.tovica.com/schemas/cp3/nativefunctions/1"
  elementFormDefault="qualified"
  xmlns="http://www.tovica.com/schemas/cp3/nativefunctions/1"
  xmlns:xs="http://www.w3.org/2001/XMLSchema"
>

<xs:simpleType name="PascalCaseString">
  <xs:restriction base="xs:string">
  <xs:pattern value="[A-Z][a-zA-Z0-9]*"/>
  </xs:restriction>
</xs:simpleType>

...

The supported data types are:

 <xs:simpleType name="Types">
<xs:restriction base="xs:string">
  <xs:enumeration value="Double"/>
  <xs:enumeration value="Date"/>
  <xs:enumeration value="DateTime"/>
  <xs:enumeration value="String"/>
  <xs:enumeration value="Integer"/>
  <xs:enumeration value="Boolean"/>
  <xs:enumeration value="Any"/>
<a name="900"></a>  <xs:enumeration value="Curve"/>
  <xs:enumeration value="Fixings"/>
  <xs:enumeration value="Surface"/>
  <xs:enumeration value="StringOrNumber"/>

  <xs:enumeration value="?Double"/>
  <xs:enumeration value="?Date"/>
  <xs:enumeration value="?DateTime"/>
  <xs:enumeration value="?String"/>
  <xs:enumeration value="?Integer"/>
  <xs:enumeration value="?Boolean"/>
  <xs:enumeration value="?Any"/>
  <xs:enumeration value="?Curve"/>
  <xs:enumeration value="?Fixings"/>
  <xs:enumeration value="?StringOrNumber"/>
</xs:restriction>
</xs:simpleType>

Generics are also supported, normally for enumerations or sharing/reusing VARIANTs:

 <xs:simpleType name="Generics">
<xs:restriction base="xs:string">
  <xs:enumeration value="NamedMap"/>
  <xs:enumeration value="NamedList"/>
  <xs:enumeration value="Map"/>
  <xs:enumeration value="List"/>
  <xs:enumeration value="Matrix"/>
  <xs:enumeration value="EnumOrString"/>
  <xs:enumeration value="EnumOrNumber"/>
  <xs:enumeration value="EnumBool"/>
  <xs:enumeration value="?NamedMap"/>
  <xs:enumeration value="?NamedList"/>
  <xs:enumeration value="?Map"/>
  <xs:enumeration value="?List"/>
  <xs:enumeration value="?Matrix"/>
  <xs:enumeration value="?EnumOrString"/>
  <xs:enumeration value="?EnumOrNumber"/>
</xs:restriction>
</xs:simpleType>

The root element shows the overall design of the XML:

 <xs:element name="root">
<xs:complexType>
<xs:choice maxOccurs="unbounded">
  <xs:element name="templated" type="Templated"/>
  <xs:element name="function" type="Function"/>
  <xs:element name="create">
<xs:complexType>
<xs:sequence>
  <xs:element name="measures" type="Measures" minOccurs="0"/>
  <xs:element name="name"/>
  <xs:group ref="ArgumentsGroup" maxOccurs="unbounded"/>
</xs:sequence>
  <xs:attribute name="id" type="PascalCaseString" use="required"/>
  <xs:attribute name="type" type="NameTypes" use="required"/>
</xs:complexType>
</xs:element>
</xs:choice>
</xs:complexType>
</xs:element>

The corresponding NativeFunctions.XML just needs to have the right header in order for Visual Studio to pick up the XSD:

 <?xml version="1.0" encoding="utf-8"?>
<root xmlns="http://www.tovica.com/schemas/cp3/nativefunctions/1"
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
xsi:schemaLocation="http://www.tovica.com/schemas/cp3/nativefunctions/1 NativeFunctions.xsd">

<!-- Introductory examples -->
<function id="Function3" type="String">
<arg id="FirstArg" type="String"/>
<arg id="SecondArg" type="?String"/>
<arg id="ThirdArg" type="?String"/>
</function>

...

The following example shows how arguments can be skipped or default values assigned:

 <function id="FunctionWithSkip" type="Date">
<arg id="AsOf" type="Date"/>
<arg id="Days" type="Integer">0</arg>
<arg id="Months" type="Integer">0</arg>
<arg id="Years" type="Integer">0</arg>
<skip id="BusDayConv"/>
<skip id="Calendars"/>
</function>

And these final ones show the more advanced abilities to add the type safety to the Quant library, including how Measures are handled for the model calls:

 <create id="CreateMap" type="Map">
  <name/>
  <arg id="Data" type="Any" isArray="2d"/>
  </create>

<create id="CreateFixings" type="Fixings">
  <name/>
  <arg id="AsOf" type="Date"/>
  <argT id="DatesAndValues" type="Matrix" T="QuantDatesAndNumbers"/>
</create>

<templated id="ModelOptionBlackScholes">
  <measures useLegacyMethod="yes"/>
  <arg id="AsOf" type="Date"/>
  <bool id="Payoff" type="QuantCallOrPut"/>
  <arg id="Spot" type="Double"/>
  <arg id="Strike" type="Double"/>
  <arg id="ExpiryDate" type="Date"/>
  <arg id="Rate" type="Curve"/>
  <arg id="Vol" type="Surface"/>
  <arg id="Fixings" type="?Fixings"/>
</templated>

<templated id="ModelOptionFD">
  <measures/>
  <arg id="AsOf" type="Date"/>
  <bool id="Payoff" type="QuantCallOrPut"/>
  <arg id="Spot" type="Double"/>
  <arg id="Strike" type="Double"/>
  <arg id="ExpiryDate" type="Date"/>
  <arg id="Rate" type="Curve"/>
  <process type="QuantVolOptions"/>
  <arg id="Fixings" type="?Fixings"/>
  <argT id="Options" type="?Map" T="QuantFDModelOptions"/>
</templated>

Code Generation using T4

Next to the NativeFunctions.XML and XSD files the project contains a TT file. This is an ASP.NET-like language that Visual Studio uses to allow code generation. Editing of the TT file (or choosing "Run Custom Tool" off the right-click menu on it) causes Visual Studio to compile and run the code. The top of the file is pretty standard:

<#@ template debug="true" hostspecific="true" language="C#" #>
<#@ output extension=".cs" #>
<#@ assembly name="System.Xml" #>
<#@ assembly name="System.Core" #>
<#@ assembly name="System.Xml.Linq" #>
<#@ import namespace="System.Collections.Generic" #>
<#@ import namespace="System.Linq" #>
<#@ import namespace="System.Xml.Linq" #>
<#@ import namespace="System.IO" #>
<#@ import namespace="System" #>
// <#= new string('*', 78) #>
//GENERATED FILE!
//Do not edit this file manually; modify the associated NativeFunctions.xml file instead
// <#= new string('*', 78) #>

<#
Dictionary<string, Wrapper> wrappers;
var count = Initialise(out wrappers);
this.WriteLine("//Generated {0:N0} function wrappers", count);
#>
using ManagedAPI.Functions.Arguments;

...

The output file will be NativeFunctions.CS. The Initialise() call is responsible for parsing the NativeFunctions.XML file into a reasonable data structure, "wrappers". This is then repeatedly looped over to create the code:

namespace ManagedAPI.Functions
{
  partial class NativeFunctions
  {
    /// <summary>
    /// Available quant functions - can be used directly to query existence.
    /// Alternatively, just use the Type property of the function argument wrappers.
    /// </summary>
    public enum Function
    {
      None = 0,
      <#
      this.PushIndent(new string(' ', 4 * 3));
      foreach (var p in wrappers)
      {
        this.WriteLine("{0},", p.Key);
      }
      this.PopIndent();
      #>
    }

See the NativeFunctions.TT file for all the gory details.

Function Arguments

Generation is also used here (Arguments.TT) to avoid the large amount of repetitive tedious code required to support all the argument types in their basic scalar, vector and shared modes. Some code reduction was achieved by reusing these structs for the function result as well. For example, the generated bit of a basic required string argument is:

 namespace Arguments
{
    [DebuggerDisplay("{Native.Type}: {Native.Boxed}")]
    public partial struct RequiredStringArg
    {
        readonly NativeArguments Buffer;
        public Variant Native { get { return Buffer[Index]; } }
        public readonly string Name;
        public readonly int Index;

        static void ThrowIfEmpty(Variant v, string name)
        {
            if (v.IsEmpty) throw new NativeMissingValueException(name);
        }
        internal void ThrowIfEmpty()
        {
            ThrowIfEmpty(Native, Name);
        }
        internal static String Get(string name, IntPtr r)
        {
            var v = new Variant(r);
            ThrowIfEmpty(v, name);
            return v.GetString(name);
        }
        internal void ThrowIfBadType()
        {
            ThrowIfEmpty();
            if (!Native.IsString) throw new NativeTypeMismatchException(Name);
        }

        internal RequiredStringArg(string name, NativeArguments buffer, int idx)
        {
            Name = name;
            Buffer = buffer;
            Index = idx;
        }
        public void Set(String value)
        {
            Native.SetString(value);
        }

        /// <summary>
        /// Given a shared variant (one allocated elsewhere and stored for reuse: e.g. TypedVariantCache)
        /// set the value of the given function argument field.
        /// This can be repeated on any number of arguments.
        /// The lifecycle of this argument now matches the shared variant, not the enclosing function.
        /// </summary>
        public void SetFromShared<SHARED>(SHARED sharedVariant)
            where SHARED : Interfaces.IVariantString
        {
            Buffer.UseVariantAgain(sharedVariant.Native, Index);
        }

        /// <summary>
        /// Given a shared variant (one allocated elsewhere and stored for reuse: e.g. TypedVariantCache), 
        /// transfer the given function argument into it for later reuse.
        /// This value can then be reused for any number of arguments on this function or any other.
        /// The lifecycle of this argument now matches the shared variant, not the enclosing function.
        /// </summary>
        public void Save(Variants.VariantString sharedVariant)
        {
            Buffer.StoreArgumentForReuse(sharedVariant.Native, Index);
        }

        /// <summary>
        /// Advanced: first use SaveResults, then pass that variant into this routine
        /// to reuse the result for one or more arguments.
        /// </summary>
        /// <param name="v">A subset of the variant that was used with SaveResults</param>
        public void SetFromResult<R>(R v) 
            where R : NativeFunctions.IResult
        {
            Buffer.UseVariantAgain(v.Native, Index);
        }
    }
}

Leaving only the following to be typed in manually:

     partial struct RequiredStringArg
    {
        public void Set(StringBuilder value)
        {
            Native.SetString(value);
        }
    }

Additional functionality around buffer handling including scoped buffers and shared native arguments or results of different scopes can be found in:

  • NativeBuffer - the native VARIANT array for arguments and the result;
  • NativeArguments - passing the NativeBuffer to the function and dealing with argument scope;
  • TypedVariantCache - global storage of native arguments for reuse;
  • Builders and NamedBuilders - helper structs for efficiently setting typed arrays or matrices including Maps and Lists;
  • QuantVolProcess, QuantFixings, QuantEnumOrNumber, etc - many helper structs for setting arguments and holding typed results and unions.

Functions as Disposable structs

The general pattern for the functions themselves is to generate them totally from the NativeFunctions.XML file markup as structures that implement IDisposable - i.e. are normally used exclusively within 'using' statements.

The first bits generated are the constants for convenience and the essential field for NativeArguments which itself has the link back to the NativeInstance via Thread Local Storage:

/// <summary>
/// Quant FunctionWithSkip function: must be created with any overload of New() and Disposed following the IDispose/using pattern
/// </summary>
/// <example>
/// using (var fn = FunctionWithSkip.New())
/// {
/// fn.AsOf.Set(DateTime.Now);
/// var res = fn.Invoke();
/// //Use res here - see Invoke() overloads for more details
/// }
/// </example>
public partial struct FunctionWithSkip : IDisposable
{
    public const NativeFunctions.Function Type = NativeFunctions.Function.FunctionWithSkip;
    public readonly static string Name = "FunctionWithSkip";
    public readonly static string Result = "FunctionWithSkip.Result";
    /// <summary>
    /// The buffer to use for this function call and it's results
    /// </summary>
    public NativeArguments Native;

Each argument follows as a field. Note that any field can be skipped or defaulted, not just the ones at the end of the call:

     //Number of used arguments: 4
    //SKIPPED: 2
    const int HighestArgCount = 4;

    /// <summary>
    /// Required: Date
    /// </summary>
    public RequiredDateArg AsOf;
    /// <summary>
    /// Required: Integer (DEFAULT = 0)
    /// </summary>
    public RequiredIntegerArg Days;
    /// <summary>
    /// Required: Integer (DEFAULT = 0)
    /// </summary>
    public RequiredIntegerArg Months;
    /// <summary>
    /// Required: Integer (DEFAULT = 0)
    /// </summary>
    public RequiredIntegerArg Years;
    //SKIP: public RequiredNoneArg BusDayConv;
    //SKIP: public RequiredNoneArg Calendars;

This approach relies on 'New()' to create the structure and in there the argument slot number is explicitly passed to each argument, allowing for any of them to be skipped or even the order changed without altering any calling code. Note also that a different buffer can be provided for advanced usage such as one buffer per model, or nested calls:

     /// <summary>
    /// Quant FunctionWithSkip function: must be Disposed - e.g. within a 'using' block
    /// </summary>
    public static FunctionWithSkip New(NativeArguments buffer = null)
    {
        if (buffer == null) buffer = NativeInstance.Current.Functions.DefaultBuffer;
        var res = new FunctionWithSkip
        {
            Native = buffer,
            AsOf = new RequiredDateArg("FunctionWithSkip.AsOf", buffer, 0),
            Days = new RequiredIntegerArg("FunctionWithSkip.Days", buffer, 1),
            Months = new RequiredIntegerArg("FunctionWithSkip.Months", buffer, 2),
            Years = new RequiredIntegerArg("FunctionWithSkip.Years", buffer, 3),
        };
        res.SetDefaults();
        return res;
    }

Basic argument type checks and scalar default settings follow with extension points via partial functions:

     [Conditional("DEBUG")]
    void VerifyArguments()
    {
        AsOf.ThrowIfBadType();
        Days.ThrowIfBadType();
        Months.ThrowIfBadType();
        Years.ThrowIfBadType();
        CustomVerifyArguments();
    }
    partial void CustomVerifyArguments();

    void SetDefaults()
    {
        Days.Set(0);
        Months.Set(0);
        Years.Set(0);
        CustomSetDefaults();
    }
    partial void CustomSetDefaults();

Since DLL exports are being used via a generic Translator the Quant library native binaries can be changed without recompiling (or re-releasing) the managed application that uses them. This means it is possible that not only may function argument counts change but the availability of functions may also. The following allows for programmatical checking so that the managed application can degrade functionality cleanly if it is not available:

     /// <summary>
    /// Whether this function is available in the currently loaded Quant version
    /// </summary>
    public bool IsSupported { get { return NativeInstance.Current.Functions.IsSupported(Type); } }

    /// <summary>
    /// Throw if this function is not available in the currently loaded Quant version
    /// </summary>
    public void ThrowIfNotSupported()
    {
        NativeInstance.Current.Functions.ThrowIfNotSupported(Type);
    }

Once all the arguments have been supplied the function can be invoked, any exceptions propagated and the result type-checked all in one go. Note how the argument structure is re-used to perform this latter check:

     /// <summary>
    /// Result is of type: Date
    /// </summary>
    public Date Invoke()
    {
        VerifyArguments();
        var raw = Native.Call(Type);
        return RequiredDateArg.Get(Result, raw);
    }

Alternatively, any exception can be avoided by splitting the call up. Between or after the second call, methods are available off NativeArguments to access the raw result directly, the raw error code or its decoded value:

     /// <summary>
    /// Advanced: use to avoid the Quant exception on an error - if return true, 
    /// call InvokeEnd() to obtain the result. See Native for accessing the error code or message.
    /// Result is of type: Date
    /// </summary>
    public bool InvokeBegin()
    {
        VerifyArguments();
        var code = Native.CallRaw(Type);
        return code == 0;
    }

    /// <summary>
    /// Advanced: use only if InvokeBegin() return true to access the result in a typesafe manner.
    /// See Native for accessing the error code or message.
    /// Result is of type: Date
    /// </summary>
    public Date InvokeEnd()
    {
        var raw = Native.RawResult;
        return RequiredDateArg.Get(Result, raw);
    }

The lifetime of the native result can be transferred to a different scope - i.e. this stops the 'using' block from freeing the result along with the arguments:

     /// <summary>
    /// Advanced: given shared variant (allocated elsewhere e.g. via a TypedVariantCache), 
    /// store the result of this function in it.
    /// This result can then be reused on any number of arguments.
    /// The lifecycle of the result is now external to this enclosing function.
    /// </summary>
    public void SaveResults(ref VariantResult shared)
    {
        Native.StoreResultForReuse(ref shared, Result);
    }

Finally the end of the function including the freeing of the native values (but not the NativeBuffer itself) of the arguments and result is generated:

     /// <summary>
    /// Advanced: Use this before using this structure again, either in a loop or when stored as a variable
    /// </summary>
    public void ResetToDefaults()
    {
        Free();
        SetDefaults();
    }

    internal void Free()
    {
        Native.FreeForReuse(HighestArgCount);
    }

    void IDisposable.Dispose()
    {
        Free();
    }
}

Additional options for accessing the results are required for when doing so can be expensive. For example, the following is provided for a vector of strings:

 /// <summary>
/// Result is of type: Vector of Strings
/// USAGE: 
/// 1. This overload uses a default Adaptor to produce appropriately typed arrays
/// 2. Alternatives: See other overloads
/// 3. Advanced: See InvokeAsVariant()
/// </summary>
public String[] Invoke()
{
    var a = new Adaptors.ArrayOutput<String>();
    Invoke(ref a);
    return a;
}

/// <summary>
/// Result is of type: Vector of Strings
/// USAGE: 
/// 1. Use this overload and an appropriate implementation of the IOutput callback
///    to fill in a result (see Adaptors namespace);
/// 2. Alternatives: See other overloads
/// 3. Advanced: See InvokeAsVariant()
/// The return result is the number of elements in the vector.
/// </summary>
public int Invoke<T>(ref T output) where T : Variant.IOutput<String>
{
    VerifyArguments();
    var raw = Native.Call(Type);
    return RequiredStringVectorArg.Get(Result, raw, ref output);
}

/// <summary>
/// Result is of type: Vector of Strings - the result will be verified as a vector 
/// but the element type is NOT verified.
/// Advanced: Use this overload to access the raw results.
/// </summary>
public Results InvokeAsVariant()
{
    VerifyArguments();
    var raw = Native.Call(Type);
    return RequiredStringVectorArg.Get(Result, raw);
}

Finally functions that accept Measure lists also have additional support such as multiple ways to set the Measures themselves. This trades efficiency for convenience:

/// <summary>
/// Quant ModelOptionFD function: must be Disposed - e.g. within a 'using' block
/// NOTE: This is a convenience overload
/// For more advanced scenarios set the measures directly on the function itself
/// </summary>
public static ModelOptionFD New(ICollection<QuantMeasure> measures)
{
    var buffer = NativeInstance.Current.Functions.DefaultBuffer;
    buffer.Functions.SetMeasures(buffer[0], measures, measures.Count);
    return New(buffer);
}

/// <summary>
/// Advanced: Sets the measures 
/// The usual easier approach is to use the New() overloads when creating the function itself
/// </summary>
public void SetMeasures(IEnumerable<QuantMeasure> measures, int count)
{
    Native.Functions.SetMeasures(Measures.Native, measures, count);
}

Typed Matrices

The reliance on untyped arrays is particularly painful especially when the required shape and content change depending on some other argument. For example, FunctionComplex() has an argument that is marked up as:

 <argT id="LowBarrier" type="?Matrix" T="QuantContinuousBarrierSchedule,
                                        QuantDiscreteBarrierSchedule"/>

This generates the following union of those two types:

public OptionalMatrixArg<QuantContinuousBarrierSchedule,QuantDiscreteBarrierSchedule> LowBarrier;

Taken from earlier, the following shows how a QuantContinousBarrierSchedule can be set using the first builder:

     var bldBarrier = fn.LowBarrier.SetWithBuilder1(2);
    //Optimal approach when translating an array of structures into individual arrays
    bldBarrier.StartDates.Set(schedule, t => t.Key);

The argument is manually coded as a generic accepting the list of types for the union, in this case, two:

 public struct OptionalMatrixArg<T, U>
where T : struct, IMatrixBuilder<T>
where U : struct, IMatrixBuilder<U>
{
    readonly NativeArguments Buffer;
    public Variant Native { get { return Buffer[Index]; } }
    public readonly string Name;
    public readonly int Index;
    
    ....

Each builder can then be exposed by deferring to the appropriate generic type's implemented interface:

     /// <summary>
    /// Set the native variant using a typed builder.
    /// </summary>
    public T SetWithBuilder1(int rows, int cols = -1)
    {
        var m = new T();
        return m.New(Native, rows, cols);
    }

Exposing something like StartDates above involves making a series of argument-like structures that operate instead of pre-allocated mixed matrices, column-by-column as follows:

public partial struct RequiredDateColumn
{
    readonly Variant.Vector Native;

    public void Set(IList<Date> value)
    {
        Native.SetDateColumn(value);
    }
    public void SetAt(int idx, Date value)
    {
        Native[idx].SetDate(value);
    }
    
    ....
}

Finally the QuantContinuousBarrierSchedule can be manually coded using the above. Note how in this case the Quant library accepts zero rebates but not empty pay dates, making things more complex:

 public struct QuantContinuousBarrierSchedule : IMatrixBuilder<QuantContinuousBarrierSchedule>
{
    public static readonly string Name = "ContinuousSchedule";
    public const int MaxColumns = 5;

    public readonly RequiredDateColumn StartDates, EndDates;
    public readonly RequiredDoubleColumn Levels;
    public readonly OptionalDoubleColumn Rebates;
    public readonly OptionalDateColumn PayDates;

    public bool HasPayDates { get { return Columns == MaxColumns; } }
    public readonly int Columns;

    internal QuantContinuousBarrierSchedule(Variant v, int rows, int cols = -1)
        : this()
    {
        Columns = cols <= 4 ? 4 : MaxColumns;
        var vector = v.SetNewMatrix(Columns, rows);
        StartDates = new RequiredDateColumn(vector); vector++;
        EndDates = new RequiredDateColumn(vector); vector++;
        Levels = new RequiredDoubleColumn(vector); vector++;
        Rebates = new OptionalDoubleColumn(vector); vector++;
        if (Columns == MaxColumns) PayDates = new OptionalDateColumn(vector); vector++;
    }

The QuantDiscreteBarrierSchedule has a different number of columns and types. For more schedules, see QuantAsianAveragingSchedule and QuantFaderBarrierSchedule.

Mapping with Enumerations

Enumerations with attributes is a one way to declaratively support the replacement of "magic" strings by clearer enumerations. For example, the following would ensure that "C" was passed to the Quant library if the user specified Continuous for the argument:

 public enum QuantBarrierType
{
    [EnumName("C")]
    Continuous,
    [EnumName("D")]
    Discrete,
}

Similarly, the replacement of annoying booleans with clear two-value enumerations makes for a much better API. For example, the following would pass FALSE to the Quant library if the user specified Put for the argument or TRUE if Call:

 public enum QuantCallOrPut
{
    Put = 0,
    Call,
}

The Quant library may support many alternatives and those values may need to be normalised to a single one when writing XML compliant with an XSD for example. In the following all the alternatives map to the primary name (the first one):

 /// <summary>
/// The available frequencies with primary and alternative names
/// </summary>
public enum QuantFrequency
{
    [EnumName("Daily", "1D")]
    Daily,
    [EnumName("Weekly", "1W")]
    Weekly,
    [EnumName("TwoWeekly", "BiWeekly", "2W")]
    TwoWeekly,
    [EnumName("FourWeekly", "28D", "4W")]
    FourWeekly,
    [EnumName("Monthly", "1M")]
    Monthly,
    [EnumName("Quarterly", "3M")]
    Quarterly,
    [EnumName("SemiAnnual", "Semi", "6M")]
    SemiAnnual,
    [EnumName("Annual", "1Y", "12M")]
    Annual,
    Term,
}

Finally, some of the names chosen by the Quant library developers may be just too obscure or ambiguous:

 public enum QuantMeasure
{
    AccrualEndDates = 0,
    AccrualStartDates,
    [EnumName("FullDiscountRho")]
    BucketedDiscountRho,
    Cashflows,
    Coupons,
    Dates,
    [EnumName("DCFs")]
    DayCountFractions,
    Delta,
    [EnumName("DFs")]
    DiscountFactors,
    ....

Additionally the well known problem with converting enumerations to strings comes into play here - i.e. repeated calls to ToString() on an enum are not only slow but result in a duplicate (non-interned) string and allow for numbers to be specified.

The following shows the first part of the implementation of a class that resolves these issues reasonably efficiently:

 [DebuggerStepThrough]
public static class Enum<T> where T : struct, IConvertible
{
    #region Accessors
    /// <summary>
    /// Short name of the enumeration
    /// </summary>
    public readonly static string Name;

    /// <summary>
    /// String names in sorted ascending order
    /// These strings are all interned
    /// See GetValueByIndex for way to get value given index in this array
    /// </summary>
    public readonly static string[] SortedNames;

    /// <summary>
    /// Values in numerical sorted ascending order
    /// </summary>
    public readonly static T[] SortedValues;

    /// <summary>
    /// Number of items
    /// </summary>
    public static int Count { get { return SortedNames.Length; } }

    /// <summary>
    /// Alternative string mappings
    /// NOTE: May be null if none present
    /// </summary>
    public readonly static SortedList<string, string> Alternatives;

    /// <summary>
    /// Given the index of a name in the SortedNames collection, return the corresponding value
    /// See SortedNames and SortedValues.
    /// </summary>
    public static T GetValueByIndex(int idx)
    {
        return SortedValues[_name2value[idx]];
    }

    /// <summary>
    /// Given the index of a value in the SortedValues collection, return the corresponding name
    /// See SortedNames and SortedValues.
    /// </summary>
    public static string GetNameByIndex(int idx)
    {
        return SortedNames[_value2name[idx]];
    }

    #endregion

There are many methods available including extension methods - see EnumConverter.cs for details. The initalisation is done via the static constructor as shown:

     #region One time initialisation
    readonly static int[] _value2name, _name2value;
    static Enum()
    {
        var type = typeof(T);
        if (!type.IsEnum) throw new InvalidOperationException("Type is not an enum: " + type.FullName);
        Name = type.Name;

        //Store stored names with corresponding values
        //SortedNames = Enum.GetNames(type);
        SortedValues = (T[])Enum.GetValues(type);

        //Allow for name to be overridden
        var fields = typeof(T).GetFields(BindingFlags.Public | BindingFlags.Static);
        //XmlNameAttribute
        SortedNames = new string[fields.Length];
        for (int i = 0; i < fields.Length; i++)
        {
            var field = fields[i];
            var name = field.Name;
            var attributes = (EnumNameAttribute[])field.GetCustomAttributes(typeof(EnumNameAttribute), false);
            if (attributes.Length == 1)
            {
                name = attributes[0].Name;
                var alternatives = attributes[0].Alternatives;
                if (alternatives != null)
                {
                    Alternatives = Alternatives ?? new SortedList<string, string>(fields.Length, StringComparer.Ordinal);
                    foreach (var a in alternatives) Alternatives.Add(a, name);
                }
            }
            SortedNames[i] = name;
        }

        //And sort
        Array.Sort(SortedNames, SortedValues, StringComparer.Ordinal);

        //Now sort by value for reverse lookup
        _value2name = new int[SortedNames.Length];
        for (int i = 0; i < _value2name.Length; i++) _value2name[i] = i;
        Array.Sort(SortedValues, _value2name, Comparer);

        //And store name to value lookup
        _name2value = new int[SortedNames.Length];
        for (int i = 0; i < _name2value.Length; i++) _name2value[_value2name[i]] = i;
    }
    #endregion

Native Topics

Discussion of the advanced C# interop topics and the Windows C++ implementation.

The Translator

The C# code needs a P-Invoke friendly API to call and that is what the Translator.DLL provides, starting with the walking of the Export Address Table of the Quant library (NativeAPI.DLL) to report on all exported functions at run time:

//_GetExportedFunctions@12
extern "C" HRESULT __declspec(dllexport) __stdcall
GetExportedFunctions(HMODULE module, LPSAFEARRAY* pDemangled, LPSAFEARRAY* pAddresses)
{
    DWORD cNames = 0;
    *pDemangled = nullptr;
    *pAddresses = nullptr;

    auto pHeaders = ImageNtHeader(module);
    if (pHeaders)
    {
        auto pTable = (PIMAGE_EXPORT_DIRECTORY)::ImageDirectoryEntryToData(module, TRUE, 
                                                 IMAGE_DIRECTORY_ENTRY_EXPORT, &cNames);
        cNames = 0;
        if (pTable)
        {
            cNames = pTable->NumberOfNames;
            auto pNames = (DWORD*)(pTable->AddressOfNames + (DWORD)module);
            auto pFunctions = (DWORD*)(pTable->AddressOfFunctions + (DWORD)module);
            if (cNames > 0 && pNames && pFunctions)
            {
                //Assume the full size - will use empties to indicate failures
                *pDemangled = ::SafeArrayCreateVector(VT_BSTR, 0, cNames);
                *pAddresses = ::SafeArrayCreateVector(VT_I4, 0, cNames);
                
                auto pCurrD = (BSTR*)(*pDemangled)->pvData;
                auto pCurrA = (LPVOID*)(*pAddresses)->pvData;

                auto buffer = new char[MaxUndecoratedNameLength + 1];
                auto i = cNames;
                do
                {
                    auto pName = (LPCH)(*pNames + (DWORD)module);
                    auto pFunction = (PROC)(*pFunctions + (DWORD)module);
                    if (pName && pName[0] && pFunction)
                    {
                        auto cbSymbol = ::UnDecorateSymbolName(pName, buffer, 
                                          MaxUndecoratedNameLength + 1, UNDNAME_COMPLETE);
                        if (cbSymbol > 0 && cbSymbol < MaxUndecoratedNameLength
                                                           - 4 /* Misbehaving function */)
                        {
                            *pCurrA = pFunction;
                            *pCurrD = (BSTR)::SysAllocStringLen(nullptr, cbSymbol);
                            ::MultiByteToWideChar(CP_UTF7, 0, buffer, cbSymbol + 1, *pCurrD, cbSymbol);
                        }
                    }
                    ++pCurrA;
                    ++pCurrD;
                    ++pFunctions;
                    ++pNames;
                    --i;
                } while (i > 0);
                delete buffer;
            }
        }
    }
    return cNames > 0 ? S_OK : E_FAIL;
}

The corresponding P-Invoke signature in NativeFactory is then:

delegate int _GetExportedFunctions(IntPtr module,
    [MarshalAs(UnmanagedType.SafeArray, SafeArraySubType = VarEnum.VT_BSTR)] 
    out string[] pDemangled,

    [MarshalAs(UnmanagedType.SafeArray, SafeArraySubType = VarEnum.VT_I4)]
    out IntPtr[] pAddresses
);

The Translator itself has to have thread-affinity so that the same instance can be easily shared across multiple copies of the Quant library while still providing a place to store the native C++ exception message, one per copy:

static __declspec(thread) char TLS_Error[255];

//_GetThreadLastError@0
extern "C" LPSTR __declspec(dllexport) __stdcall
GetThreadLastError(void)
{
    return TLS_Error;
}

Macros and a variadic template are used to more easily create identical wrapper functions differing only by the number of arguments expected:

//1. Arguments - note that args[0] is for the result
#define F_ARGS(NUMBER, ...) &args[NUMBER]
//2. Invoker
#define DECLARE_FUNCTION(COUNT) \
template<typename... Params> \
static HRESULT __stdcall \
MACRO_CAT(Invoke, COUNT)(LPVOID fn, VARIANT* args) \
{ \
  try \
  { \
    args[0] = ((VARIANT(__stdcall *)(Params*...))fn)(MACRO_ARGS(COUNT,F_ARGS)); \
    return S_OK; \
  } \
  catch (std::exception& ex) \
  { \
    ::strncpy_s(TLS_Error, ex.what(), _TRUNCATE); \
    return E_FAIL; \
  } \
}

Taking the case when COUNT is 3 this expands out into something roughly equivalent to this:

HRESULT Invoke3(FUNCTION2 fn, VARIANT* args)
{
  try
  {
    args[0] = FUNCTION2(&args[1], &args[2]);
    return S_OK;
  }
  catch (std::exception& ex)
  {
    TLS_Error = ex.what();
    return E_FAIL;
  }
}

That is, given the address of any Quant library function that takes two arguments, call it with those arguments, storing the result and recording any exception message in the TLS buffer.

Crucially this implies the same P-Invoke signature can be used for all Quant library functions and so it does not matter if the number of arguments changes between Quant library versions. This allows for the Quant library to be changed without recompiling the Translator or, indeed, the C# application using it:

delegate int _FunctionWrapper(IntPtr function, IntPtr buffer);

//Max arguments - must be at most equal to the equivalent in Translator.dll
internal const int MaxFunctionArguments = 12;
readonly static _FunctionWrapper[] _FunctionWrappers = 
                         new _FunctionWrapper[MaxFunctionArguments];

As referenced in the above code, the number of arguments supported has to be fixed at compile time. For simplicity, this is done in the C# as well, rather than just exporting another function from the Translator. The macros are then used to "stamp out" the code as follows:

//The wrappers: append more as needed
DECLARE_FUNCTION(0)
DECLARE_FUNCTION(1)
...
DECLARE_FUNCTION(12)

//Table of wrappers: append more as needed
static const PROC Invokers[] = 
{
    ADDRESS_FUNCTION(0),
    ADDRESS_FUNCTION(1),
    ...
    ADDRESS_FUNCTION(12),
};

//_GetFunctionWrapper@4
extern "C" LPVOID __declspec(dllexport) __stdcall
GetFunctionWrapper(int argc)
{
    //Args is number of arguments excluding the return result
    if (argc >= 0 && argc < _countof(Invokers)) return Invokers[argc];
    return nullptr;
}

Finally, the NativeFactory in the Managed.DLL is responsible for locating the Translator.DLL, loading it and fixing up all entry points as shown below:

public static bool SearchForNativeBinariesAndSetupTranslator()
{
  if (!SearchForNativeBinaries()) return false;

  //Expected to be in assembly location
  var check = Path.Combine(Path.GetDirectoryName(Assembly.GetExecutingAssembly().Location), TRANSLATOR);
  if (!File.Exists(check)) return false;

  var hTranslator = NativeHelpers.NativeLoadLibrary(TRANSLATOR);
  _ExportedFunctions = NativeHelpers.GetDelegate<_GetExportedFunctions>(hTranslator, "_GetExportedFunctions@12");
  _GetLastError = NativeHelpers.GetDelegate<GetLastErrorFunctionType>(hTranslator, "_GetThreadLastError@0");

  //Hook up the faster helpers
  Variant.NativeFree = NativeHelpers.GetDelegate<Variant.VariantHelperType>(hTranslator, "_VariantsFree@8");
  Variant.NativeInit = NativeHelpers.GetDelegate<Variant.VariantHelperType>(hTranslator, "_VariantsInit@8");

  //Get the wrapper accessor
  var getWrapper = NativeHelpers.GetDelegate<_GetFunctionWrapper>(hTranslator, "_GetFunctionWrapper@4");
  //And create the wrappers
  for (int i = 0; i < MaxFunctionArguments; i++)
  {
    var wrapper = getWrapper(i);
    Debug.Assert(wrapper != null, "MaxArgument count needs to be increased in Translator");
    _FunctionWrappers[i] = NativeHelpers.GetDelegate<_FunctionWrapper>(hTranslator, wrapper);
  }
  return true;
}

The Translator exposes both the addresses of the function wrappers and the full mangled names of all the functions exported by the NativeAPI. For performance reasons, it is better to "bake in" the addresses directly into the IL - i.e. given a function in NativeAPI and it's corresponding wrapper in the Translator, create a single C# delegate as follows:

internal delegate int NativeFunctionType(IntPtr buffer);

/// <summary>
/// Create a delegate around a compiled expression that calls the given NativeAPI
/// function with the given 'argc' argument count via the Translator
/// </summary>
/// <param name="address">Native address of function within NativeAPI</param>
/// <param name="argc">Number of VARIANT* arguments; function must return a VARIANT</param>
/// <returns></returns>
internal static NativeFunctionType CreateFunction(IntPtr address, int argc)
{
  if (_SxSPathsByVersion == null) throw new InvalidOperationException("Call SearchForNativeBinariesAndSetupTranslator first");
  if (argc >= _FunctionWrappers.Length) throw new IndexOutOfRangeException("GetFunction: max function arguments exceeded");

  var wrapper = _FunctionWrappers[argc];
  var arg1 = Expression.Parameter(typeof(IntPtr), "buffer");

  //var res = wrapper(addresses, buffer)
  var exp = Expression.Invoke(Expression.Constant(wrapper), Expression.Constant(address), arg1);
  var lambda = Expression.Lambda<NativeFunctionType>(exp, arg1);
  return lambda.Compile();
}

Concurrency and Manifests

The Translator.DLL is loaded in the traditional manner: via the Windows LoadLibrary function as follows:

internal static IntPtr NativeLoadLibrary(string fullFileName)
{
    var res = LoadLibraryW(fullFileName);
    if (res != IntPtr.Zero) return res;
    var err = Marshal.GetLastWin32Error();
    throw new FileLoadException(string.Format("Unable to load library (0x{0:X8}) '{1}'", err, fullFileName));
}

However in order to load more than one instance of the same physical group of interdependent DLLs, the following is required:

  • Each physical group must have a unique path - e.g. be in a different subfolder like "Build\Debug\1\*" and "Build\Debug\2\*";
  • Every DLL must have any embedded manifests removed - e.g. this will be present if linking dynamically with the CRT etc;
  • A single manifest must be created that lists all the DLLs in the physical group (and the CRT etc if necessary);
  • LoadLibrary is used but within an Activation Context - i.e. with that manifest active.

So in each subfolder "1" and "2" there is a copy of the NativeAPI and it's dependent Utils - and for simplicity both were linked statically with the CRT so they have no embedded manifest. Now an XML file such as "NativeAPI.Manifest" can be created in both locations with identical content:

<?xml version='1.0' encoding='UTF-8' standalone='yes'?>
<assembly xmlns='urn:schemas-microsoft-com:asm.v1' manifestVersion='1.0'>
  <assemblyIdentity type='win32' name='NativeAPI' version='1.0.0.0' processorArchitecture='x86'/>
  <description>Native API</description>
  
  <file name='NativeAPI.DLL'/>
  <file name='Utils.DLL'/>
</assembly>

The manifest is then used to create an Activation Context only for the duration of the LoadLibrary call. This happens concurrently on different threads, one per physical group or instance, and C#'s version of TLS is used to "assign" the NativeInstance class to the current thread:

public class NativeInstance
{
    public readonly int Id;
    public readonly Version Version;
    public readonly string EntryPoint;
    public readonly string ManifestName;

    public readonly NativeFunctions Functions;

    /// <summary>
    /// TLS: Provides access to the current native instance assigned to this thread, or null
    /// </summary>
    public static NativeInstance Current { [DebuggerStepThrough] get { return _currentInstance; } }

    #region Native loading
    [ThreadStatic]
    static NativeInstance _currentInstance;

    internal NativeInstance(int instanceId, Version version, string targetDLL, string manifest = null)
    {
        if (_currentInstance != null) 
            throw new InvalidOperationException("An NativeInstance is already assigned to this thread: "
             + _currentInstance.Id);

        Id = instanceId;
        Version = version;
        EntryPoint = targetDLL;

        uint? cookie = null;
        if (manifest != null)
        {
            ManifestName = Path.GetFileName(manifest);
            cookie = NativeHelpers.NativePushActivationContext(Path.GetDirectoryName(manifest), manifest);
        }
        try
        {
            //Load specific instance DLLs
            var hModule = NativeHelpers.NativeLoadLibrary(targetDLL);
            //Resolve all exported functions
            Functions = new NativeFunctions(this, hModule, version);
        }
        finally
        {
            if (cookie != null) NativeHelpers.NativePopActivationContext(cookie.Value);
        }

        //If we get here all is well
        _currentInstance = this;

        //Safe to map all measures
        Functions.MapAllUsedMeasures();
    }
    #endregion

Decoding Variants

The NativeAPI is based solely on VARIANTs which, given .NET's originally incarnation as "Com+ v2", unsurprisingly can be mapped almost perfectly to an Object, however:

  • The commonly used VT_ERROR type is indistinguishable from an integer (like VT_I4);
  • All the Marshal APIs relating to this were designed before Generics in .NET v2.0 so everything has to be boxed both ways;
  • The array mapping requires each element having to be boxed as well.

This latter problem is particularly bad for Quant libraries since pricing of Financial Instruments (e.g. Options) often involves asking also for quite a few risk and informational "Measures" at the same time - i.e. the result is an array of arrays of different shapes.

On the positive side, the possible input and output types and array shapes are normally quite small. For example, passing a vector (a 1D array) is converted across to a 2D array - so only this latter type needs to be handled.

Manual decoding is used including unsafe C# code to avoid the issues with .NET's inbuilt support, all based around the address of a single native block of memory (16 bytes) holding the VARIANT as follows:

public struct Variant
{
    readonly IntPtr _variant;

    #region Helpers

    #region Supported variant types
    readonly static VarEnum[] _supported = new[] 
    {
        VarEnum.VT_EMPTY,
        VarEnum.VT_I4,
        VarEnum.VT_R8,
        VarEnum.VT_DATE,
        VarEnum.VT_BSTR,
        VarEnum.VT_BOOL,
        VarEnum.VT_VARIANT|VarEnum.VT_ARRAY,

    };

    static bool VarTypeIsSupported(VarEnum type)
    {
        return Array.BinarySearch(_supported, type, Enum<VarEnum>.Comparer) >= 0;
    }

    static Variant()
    {
        Array.Sort(_supported);
    }
    #endregion

Since the API exposed by the Translator involves a contiguous block of VARIANTs, the decoder can attach to any one of these as follows:

     internal Variant(IntPtr ptr, int index)
    {
        _variant = ptr + index * 16;
    }

A VARIANT is a classic C union with a 2-byte discriminator which can be safely cleared to zero (VT_EMPTY) provided that it doesn't contain a BSTR or a SAFEARRAY:

     #region Destruction
    //Free variant, resetting to VT_EMPTY
    [SuppressUnmanagedCodeSecurity]
    internal void Free()
    {
        if (IsReferenceType) NativeHelpers.NativeNativeFree(_variant);
        else Reset();
    }

    //Force type to VT_EMPTY without freeing first
    [SuppressUnmanagedCodeSecurity]
    internal void Reset()
    {
        Marshal.WriteInt16(_variant, (short)VarEnum.VT_EMPTY);
    }
    #endregion

To facilitate debugging, especially in the presence of inconsistent Quant library use of VARIANT types, it is useful to expose both the raw type without any flags (like VT_ARRAY) and the .NET's interpretation of the raw data:

     public VarEnum Type { get { return (VarEnum)((byte)VarType); } }
    internal VarEnum VarType { get { return (VarEnum)Marshal.ReadInt16(_variant); } }

    public object Boxed { get { return !IsEmpty ? Marshal.GetObjectForNativeVariant(_variant) : null; } }

A nested type is used to handle vectors - i.e. it represents the data payload of a SAFEARRAY with element type VARIANT:

    //Wrapper for native contiguous array of VARIANTs
    public struct Vector
    {
        readonly IntPtr _variant;
        readonly int _count;

        public static readonly Vector None;

        #region Construction

        //Attach to given variant
        internal Vector(IntPtr ptr, int count)
        {
            _variant = ptr;
            _count = count;
        }

        //Attach to variant at index in given variant array
        private Vector(IntPtr ptr, int index, int count)
        {
            _variant = ptr + index * 16;
            _count = count;
        }
        #endregion

Again to facilitate .NET debugging, the raw type and .NET's interpretation of the raw data are exposed:

         public object Boxed { get { return Marshal.GetObjectsForNativeVariants(_variant, _count); } }

For style reasons all unsafe (static) C# usage is scoped. Direct decoding relies on looking at the native memory layout of a VARIANT:

     [SuppressUnmanagedCodeSecurity]
    internal unsafe class Unsafe
    {
        static VarEnum GetType(byte* pVariant)
        {
            return (VarEnum)(*(ushort*)pVariant);
        }

        //Prototype for the following unchecked accessors
        delegate T VariantGet<T>(byte* pVariant);

        static DateTime UncheckedDate(byte* pVariant)
        {
            return DateTime.FromOADate(*(double*)(pVariant + 8));
        }
        static double UncheckedDouble(byte* pVariant)
        {
            return *(double*)(pVariant + 8);
        }
        static int UncheckedInteger(byte* pVariant)
        {
            return *(int*)(pVariant + 8);
        }
        static bool UncheckedBoolean(byte* pVariant)
        {
            return *(int*)(pVariant + 8) != 0;
        }
        static string UncheckedString(byte* pVariant)
        {
            var ptr = *(int*)(pVariant + 8);
            return ptr != 0 ? Marshal.PtrToStringBSTR(new IntPtr(ptr)) : null;
        }

Note the way the boolean mapping is defined as "not 0" to avoid subtle issues where VBA strictly compares only against "-1". Similarly, zero length BSTRs are not expected to be employed by the Quant library, something carefully checked for in the reverse case.

The following routine is used to manually decode a BSTR directly into a StringBuilder to avoid the use of immutable .NET strings entirely where possible. It relies on the native memory layout having the length before the start of the (UTF16) string data itself:

             internal static int GetBSTR(IntPtr pBStr, StringBuilder sb)
            {
                if (pBStr != IntPtr.Zero)
                {
                    var len = Marshal.ReadInt32(pBStr - 4) / 2;
                    if (len > 0)
                    {
                        sb.EnsureCapacity(len + sb.Length);
                        for (int i = 0; i < len; i++)
                        {
                            sb.Append((char)Marshal.ReadInt16(pBStr, i * 2));
                        }
                    }
                }
                return sb.Length;
            }

The general approach taken is then to call a series of routines for each type as shown for the "double" case below which uses NaN to represent the VT_EMPTY case:

         static T GenericGetElement<T>(string field, VarEnum check, byte* pv, 
                                      VariantGet<T> uncheckedFunction, T @default = default(T))
        {
            VarEnum type;
            if ((type = GetType(pv)) == check)
            {
                return uncheckedFunction(pv);
            }
            if (type == VarEnum.VT_EMPTY || type == VarEnum.VT_ERROR) return @default;
            throw ErrorCOM(field, "Type mismatch - expected {0} but got {1}", check.ToString(), type.ToString());
        }

        internal static double GetDouble(string field, IntPtr v)
        {
            return GenericGetElement<double>(field, VarEnum.VT_R8, (byte*)v.ToPointer(), UncheckedDouble, double.NaN);
        }

    internal double GetDouble(string field)
    {
        return Unsafe.GetDouble(field, _variant);
    }

For array decoding, a common anti-boxing pattern is employed: using a generic constrained to an interface directly as an argument. Additionally, 'ref' is used to ensure only the address of the struct implementing the interface is passed around and also for any state to be updated. Examination of the IL byte code for this approach shows that "constrained calls" are used and no boxing occurs:

     public interface IOutput<T>
    {
        void OnMatrix(int row, int col, T data);
        void OnColumn(int row, T data);
    }

The actual decoding of the SAFEARRAY involves some tedious unsafe C# code so is not shown here but the routines around it are shown to illustrate the anti-boxing pattern:

         static void UncheckedGetColumn<T, O>(string field, VarEnum check, int rows, ref byte* pvData, 
                  VariantGet<T> f, ref O output, T @default = default(T))
            where O : IOutput<T>
            where T : IEquatable<T>
        {
            if (pvData == null || rows <= 0) return;
            output.OnColumn(~rows, @default);
            for (int i = 0; i < rows; i++)
            {
                var raw = GenericGetElement(field, check, pvData, f, @default);
                if (!EqualityComparer<T>.Default.Equals(raw, @default)) output.OnColumn(i, raw);
                pvData += 16;
            }
        }
        static int UncheckedGetVector<T, O>(string field, VarEnum check, byte* pVariant, 
                                                  VariantGet<T> f, ref O output, T @default = default(T))
            where O : IOutput<T>
            where T : IEquatable<T>
        {
            int cDim, len, dummy;
            var pvData = UncheckedArray(field, pVariant, out cDim, out len, out dummy);
            if (pvData == null) return 0;
            if (cDim == 2) throw ErrorCOM(field, "Internal vector problem - vector expected but got matrix");
            UncheckedGetColumn(field, check, len, ref pvData, f, ref output, @default);
            return len;
        }

        static int GenericGetVector<T, O>(string field, VarEnum check, byte* pv, 
                                     VariantGet<T> uncheckedFunction, ref O output, T @default = default(T))
            where O : IOutput<T>
            where T : IEquatable<T>
        {
            VarEnum type;
            if ((type = GetType(pv)) == (VarEnum.VT_ARRAY | VarEnum.VT_VARIANT))
            {
                return UncheckedGetVector(field, check, pv, uncheckedFunction, ref output, @default);
            }
            if (type == VarEnum.VT_EMPTY || type == VarEnum.VT_ERROR) return 0;
            throw ErrorCOM(field, "Type mismatch - expected {0} but got {1}", check.ToString(), type.ToString());
        }

There are many routines provided covering all the types including specific support for matrices of the same type and also the more common mixed type - i.e. where each column is a different type.

The corresponding setting routines are slightly more tricky as they must normalise the inputs. For example, the following shows how P-Invoke's special handling of StringBuilders can be used to avoid duplication when creating a BSTR:

         static void UncheckedNewString(byte* pVariant, StringBuilder value)
        {
            if (value.Length > 0)
            {
                *(int*)(pVariant + 8) = NativeHelpers.NativeAllocBSTR(value).ToInt32();
                return;
            }
            //NativeAPI disallows zero-length strings
            *(int*)(pVariant + 8) = 0;
        }
        
        [DllImport(OLEAUT32_DLL, ...)]
        //The first argument will be marshalled by pinning the managed buffer
        private static extern IntPtr SysAllocStringLen([In] StringBuilder sb, int cch);

The anti-boxing pattern is used along with a complex delegate-based signature for the key function for streaming data out of a collection of structures into a single column of the one type without any boxing:

 static class Singleton<T>
where T : IEquatable<T>
{
  public static readonly Func<T, T> Identity = x => x;
}

static byte* UncheckedSetColumn<TCollection, TElement, TEnumerable>(VarEnum type, int skip, int take, byte* pvData,
VariantSet<TElement> setter, TEnumerable input, Func<TCollection, TElement> selector, TElement @default)
where TElement : IEquatable<TElement>
where TEnumerable : IEnumerable<TCollection>
...

Running the Examples

There are three mutually exclusive code branches - i.e. two of them commented out:

  • The default one loads the two copies of the Quant binaries in side-by-side mode and runs them in two separate threads:
//Multi-instance threading
var t1 = new Thread(x => Run((bool)x));
t1.Start(false);
var t2 = new Thread(x => Run((bool)x));
t2.Start(false);

//Wait
t1.Join();
t2.Join();
  • Uncomment the following to run the standard P-Invoke single-instance approach:
//For comparative purposes
PInvoke();
return;
  • And uncomment the following to run the singleton approach for easier debugging:
//Singleton for easy development
Run(bSingleInstance: true);
return;

Note that this latter mode is best when making changes since the side-by-side mode requires any changes to the NativeAPI.DLL or Utils.DLL to be manually duplicated into the 1 and 2 subfolders of Debug.

References

Financial Numerical Recipes

SAFEARRAY native struct

VARIANT native struct

BSTR native type

History

  • 1.0 - Initial write up.

License

This article, along with any associated source code and files, is licensed under The Code Project Open License (CPOL)

Share

About the Author

tcassisi
Tovica Services Pty Ltd
Australia Australia
Toni Cassisi

Comments and Discussions

 
QuestionExtremely good article PinprofessionalCatchExAs7-Jun-14 0:25 
QuestionA very nice article about how to provide a high performance, concurrent, typesafe API around a non-thread safe typeless one PinprofessionalVolynsky Alex3-May-14 22:33 

General General    News News    Suggestion Suggestion    Question Question    Bug Bug    Answer Answer    Joke Joke    Rant Rant    Admin Admin   

Use Ctrl+Left/Right to switch messages, Ctrl+Up/Down to switch threads, Ctrl+Shift+Left/Right to switch pages.

| Advertise | Privacy | Terms of Use | Mobile
Web02 | 2.8.141220.1 | Last Updated 4 May 2014
Article Copyright 2014 by tcassisi
Everything else Copyright © CodeProject, 1999-2014
Layout: fixed | fluid