Write your own Code Generator or Template Engine in .NET

• Download source files - 506 Kb

Abstract

This paper demonstrates building a code generator, template engine, template parser, or template processor in .NET. The demo implementation uses cutting edge .NET technologies available today such as C# .NET 2.0, MS Provider Pattern, Enterprise Library January 2006, CodeDom etc.

Introduction

If you have used Code Smith or similar tools, you may be wondering how this type of tools work. I am going to reveal in this article how easy it is to write a code generator or template engine in .NET. A template engine is a software or a software component that is designed to transform pre-formatted input into various kinds of text output. One of the major benefits of using a template engine is that it enhances productivity by reducing unnecessary reproduction of effort. For example, while designing an application, whether we implement famous design patterns (like MVC) or invent some of our own, we tend to come up with a generic structure of code throughout the application. Later on, we can write templates for the generic parts, and use template engines to quickly generate the code for us.

Template Engines makes a huge difference in development productivity. For example, if you have a database with 300 tables, and each table having at least 5 columns, imagine how long it will take to hand write all the Business Objects for all those tables. Not only this, when you complete that, you have to write your data access layer, business logic layer, and finally, the application code, and this adds up to a lot of work. However, with the help of a template engine, you can generate generic pieces of code (code that follows a similar design pattern) in minutes. You only need to write a single template, and then you simply ask the template engine to loop through each of the tables and their columns in the database, and it will generate your code for you.

So, we can all agree on how template engines can be a very helpful tool for a developer and save heaps of time. Without spending any more time on an introduction to template engines, we will jump straight into writing one.

Technologies Used in Smart Template Engine (STE)

For a refresher on the Microsoft� Provider Design Pattern, Enterprise Library you might want to have a look at my previous articles �Flexible and Plug-in-based .NET Application using Provider Pattern� and Ready-to-use Mass Emailing Functionality with C#, .NET 2.0, and Microsoft� SQL Server 2005 Service Broker. And for a quick starter on CodeDom, you can have a look at the CodeDom quick reference.

Template Engine Workflow

To make this all happen, we need to write a parser to turn a Template into valid .NET code, and later use the CodeDom to run the parsed code and produce the desired output.

Take this example:

Say we want to generate an output like this

public class Test
{
    public Test()
    {
    }
    // Do Something

    // Do Something

}

Where we want a template to replace the class name and constructor, the template might look like the following. [Note: I am using ASP / ASP.NET scripting style for this example.]

<%private string classname = �Test�;%>
public class <%=classname%>
{
   public <%=classname%> ()
   {
   }
   <% for (int i=0; i< 2 ; i++)
   {%>
      // Do Something
   <%}%>
}

We have to write a parser to turn the Template into the following piece of code and later execute the code using CodeDom:

string classname = �Test�;
MemoryStream mStream = new MemoryStream();

StreamWriter writer = new
StreamWriter(mStream,System.Text.Encoding.UTF8);

writer.Write(@"public class �);
writer.Write(classname);      
writer.Write(@"{
    public �);
    
    writer.Write(classname);
    
    writer.Write(@"()
    {
    }
    }�);
for (int i=0; i< 10 ; i++)
{  
   writer.Write(@"   // Do Something" );

}
StreamReader sr = new StreamReader(mStream); 
writer.Flush();
mStream.Position = 0;
string code = sr.ReadToEnd();

Notice what the parser did. It considered the tags <%= %> and <% %> and added some predefined code around the content of the template. Next, we will look at the flow from the client's point of view.

Client => Template => Turned into a valid .NET class by the parser => which is then executed via CodeDom => Generates the desired output

STE is a full implementation of the above concept. STE uses the MS Provider Pattern, so it�s extensible and flexible. The demo application comes with a parser and an output generator.

STE Framework

From the diagram, you will notice that multiple providers are used to accomplish this task. To understand the STE Framework, we need to understand the Template and TemplateEngineSetting objects and the two providers in detail.

Fig: Business Objects: Template, TemplateEngineSetting

Template Object in the STE Framework

The Template object is simply a business object which can hold the input, output, and the generated code using libraries, references, and exceptions. The Template object is passed among the different layers.

TemplateEngineSetting Object in the STE Framework

The TemplateEngineSetting object is simply another business object which holds the global settings for the template engine.

An Implementation of the Provider Pattern in STE

Fig Smart Template Engine Providers

STE Providers

Parse Template Provider: This provider provides parsing functionality.

public abstract class ParseTemplateProvider : ProviderBase  {     
   public abstract Template Parse(Template template);
}

Generate Output Provider: This provider generates the desired output.

public abstract class GenerateOutputProvider : ProviderBase  {
   public abstract Template Generate(Template template);
}

Configuration information: As described in the Microsoft� Provider specification, we describe the concrete provider that is implemented, in the configuration section. The beauty of the provider pattern is that the appropriate provider is instantiated at runtime from information contained in the configuration file, and you may define an unlimited number of providers.

<smartTemplateEngine.providers>
 <parseTemplate defaultProvider="AspStyleParseTemplateProvider">
  <providers>
   <add name="AspStyleParseTemplateProvider" 
    type="SmartTemplateEngine.ImplementedProviders.AspStyleParseTemplateProvider,
          SmartTemplateEngine.ImplementedProviders" />
  </providers>
 </parseTemplate>
 <generateOutput defaultProvider="CodeDomGenerateOutputProvider">
  <providers>
   <add name="CodeDomGenerateOutputProvider" 
    type="SmartTemplateEngine.ImplementedProviders.CodeDomGenerateOutputProvider,
          SmartTemplateEngine.ImplementedProviders" />
  </providers>
 </generateOutput>
</smartTemplateEngine.providers>

In the above section, I have added the AspStyleParseTemplateProvider and CodeDomGenerateOutputProvider to our providers list.

The use of Enterprise Library in the STE Framework: SME uses Enterprise Library January 2006 for caching, logging, and exception handling.

Parse Template

The demo application contains an ASP/ASP.NET style parser which considers ASP/ASP.NET tags, specifically <%%> and <%=%>, and produces C# code. The following piece of code does all the magic.

public override Template Parse(Template template)
{            
   ParseManager manager = new ParseManager(template.Input, 
                          template.Setting.Using_Libraries);
   template.GeneratedCode = manager.GetParsedResult();            
   template.IsSuccessful = true;
   return template;
}

Let�s dig deeper and look at the GetParsedResult method. A number of Regular Expressions have been used to parse the template. In this demo, I am only using the <% %> and <%= %> tags. However, you can feel free to implement tags starting with <%@ etc.

public string GetParsedResult()
{
   string modifiedcode = 
          GetCodeWithoutAssemblyNameSpacePropertyAndOther();
   string finalcode = references +
   @"
   namespace
   SmartTemplateEngine.ImplementedProviders
   {

      /// <summary>
      /// Summary description for ParseManager.
      /// </summary>
      public class Parser
      {
         �     
         public static string Render()
         {
            �        
            #:::#RenderMethod#:::#
               �. 
            return returndata;
         }
      }
   }";
   finalcode = Regex.Replace(finalcode,
               "#:::#RenderMethod#:::#",modifiedcode);
   return finalcode;
}

private string GetCodeWithoutAssemblyNameSpacePropertyAndOther()
{
   string tempcode = code;
   //Modify this part if you want to read the 

   //<%@ tags also you can implement your own tags here.

   tempcode = Regex.Replace(tempcode, 
              "(?i)<%@\\s*Property.*?%>",string.Empty);
   tempcode = Regex.Replace(tempcode,
              "(?i)<%@\\s*Assembly.*?%>",string.Empty);
   tempcode = Regex.Replace(tempcode,
              "(?i)<%@\\s*Import.*?%>",string.Empty);
   tempcode = Regex.Replace(tempcode,
              "(?i)<%@\\s*CodeTemplate.*?%>",string.Empty);
   //For the demo I am only dealing with the <%= and <% tags

   tempcode = ParseScript(tempcode);
   tempcode = Regex.Replace(tempcode,@"<%=.*?%>",
              new MatchEvaluator(this.RefineCalls),
              RegexOptions.Singleline);
   tempcode = Regex.Replace(tempcode,@"<%%>",
              string.Empty,RegexOptions.Singleline);
   tempcode = Regex.Replace(tempcode,@"<%[^=|@].*?%>",
              new MatchEvaluator(this.RefineCodeTag),
              RegexOptions.Singleline);
   return tempcode;
}

The above piece of code is self explanatory. It handles the ASP/ASP.NET tags that are found in the templates, and after dealing with the ASP style tags in the GetCodeWithoutAssemblyNameSpacePropertyAndOther method, it then replaces the #:::#RenderMethod#:::# (an example only) with the refined string returned from the method.

Generate Output

This provider receives valid .NET code produced by the parser, and uses CodeDom to compile and produce the desired output.

public override Template Generate(Template template)
{
   try
   {
     template.IsSuccessful = true;
     LanguageType language = LanguageType.CSharp;
     string entry = "Render";
     string code = template.GeneratedCode.Trim();
     �                
     CompileEngine engine = new CompileEngine(code, 
                                language, entry);
     engine.References = template.Setting.References;
     engine.Run();
     template.Output = engine.OutputText;
     �. 
   }
   catch (Exception x)
   {
     template.InnerException = x;
    �
   }
   return template;
}

I have used the compilation engine from the article: A tool for dynamic compile and run of C# or VB.NET code. I then modified the code partially according to my needs, and then used the C# code compilation section.

Assembly assembly = CreateAssembly( SourceCode );
CallEntry( assembly, EntryPoint );

The heart of the CompileEngine lies in the following two methods:

• CreateAssembly compiles the source code and makes the assembly in memory, and
• CallEntry(Assembly...) calls the entry point by reflection.

private Assembly CreateAssembly(string strRealSourceCode)
{
         �
         //Create an instance whichever code provider that is needed

         CodeDomProvider codeProvider = null;
         if (Language == LanguageType.CSharp)
            codeProvider = new CSharpCodeProvider();
         //create the language specific code compiler

         ICodeCompiler compiler = codeProvider.CreateCompiler();

         //add compiler parameters

         CompilerParameters compilerParams = new CompilerParameters();
         // you can add /optimize

         compilerParams.CompilerOptions = "/target:library";
         compilerParams.GenerateExecutable = false;
         compilerParams.GenerateInMemory = true;
         �

         // add some basic references

         compilerParams.ReferencedAssemblies.Add("mscorlib.dll");

         //add any aditional references needed


         foreach (string refAssembly in References)
         {
             �
             compilerParams.ReferencedAssemblies.Add(refAssembly);
             �
         }

         //actually compile the code

         CompilerResults results = compiler.CompileAssemblyFromSource(
                                   compilerParams, strRealSourceCode );
         //Do we have any compiler errors

         if (results.Errors.Count > 0)
         {
            foreach (CompilerError error in results.Errors)
               LogErrMsgs("Compile Error:  " + error.ErrorText );
            return null;
         }

         //get a hold of the actual assembly that was generated


         Assembly generatedAssembly = results.CompiledAssembly;
         //return the assembly

         return generatedAssembly;
}

private void CallEntry(Assembly assembly, string entryPoint)
{
         try
         {
            //Use reflection to call the static Main function

            Module[] mods = assembly.GetModules(false);
            Type[] types = mods[0].GetTypes();

            foreach (Type type in types)
            {
               MethodInfo mi = type.GetMethod(entryPoint, BindingFlags.Public | 
                               BindingFlags.NonPublic | BindingFlags.Static);  
               if (mi != null)
               {
                  if( mi.GetParameters().Length == 1 )
                  {
                     if( mi.GetParameters()[0].ParameterType.IsArray )
                     {
                        // if Main has string [] arguments

                        string [] par = new string[1];
                        mi.Invoke(null,par);
                     }
                  }
                  else
                  {
                     OutputText = (string)mi.Invoke(null, null);
                  }
                  return;
               }
            }
            �
         }
         �
}

Future Enhancements

Application Domains in the .NET Framework load assemblies, but they cannot explicitly unload them. When you run dynamic pieces of code, you need to run and compile them independently, then essentially discard them. Once loaded, none of that application space can be recovered if the assembly is loaded into the current main application's AppDomain. Unfortunately, the CompileEngine that I used in this demo suffers from this problem. To avoid this situation, I would suggest reading this article Dynamically executing code in .NET by Rick Strahl, where he shows techniques to load assemblies in a remote application domain. Using that knowledge, you might want to write a different provider using the technique described in his article.

Also, feel free to extend it and make it fancy like many commercial template engines, by adding a nice template editor with code completion and code highlighting. You can also write a better parser which takes care of other tags, line, character positioning etc. The framework could also be extended in other directions by introducing additional providers, i.e., a provider to deal with the generated output such as to display output on the screen or to save as file etc.

Demo Application

The demo application has five tabs:

Template Tab: Here you write your template.

Generated Code Tab: Here, you can see the generated C# code after parsing a template by the default Parse Template Provider.

Output Tab: Here you can see the output generated by the default Generate Template Output Provider.

Using Libraries Tab: Here, you can specify the �using� statements for the generated class. This gives you the flexibility to use any number of custom .NET libraries. �using� statements will be concatenated at the beginning of the predefined class structure that is used in the Parser Template Provider.

References: Here you define the DLLs that will be loaded in the dynamic assembly.

If you use any custom libraries, just make sure it is available in the bin folder, or has been registered in the GAC. You can also use the full path of the custom DLL, i.e., C:\DotNetWorkFolder2005\SmartTemplateEngineInDotNet\Src\ SmartTemplateEngine.WindowsApplication\bin\Debug\SmartTemplateEngine.Util.dll.

Other Functionalities: The demo comes with a Save Template feature. It serializes the input, using libraries, and the reference part of a template into a file.

Demo Templates: The demo comes with demo templates to get you up and running. One of the included templates demonstrates the generation of a C# class by accessing a database table and by iterating through its columns. All columns of the table turn into properties of the C# class as you may have expected.

Conclusion

STE is a demonstration of a concept only. To use it as a real world template engine, it needs to be drastically improved. Some directions of potential improvement have been discussed in the Future Enhancements section.

This demo also shows implementation of the MS Provider Pattern, and can also be used as a reference application for learning parsing and using CodeDom technology.

Special thanks to Christopher Heale for proof reading this article.