Click here to Skip to main content
Click here to Skip to main content

Writing Your First Visual Studio Language Service

, 11 Dec 2009
Rate this:
Please Sign up or sign in to vote.
A guide to writing a language service for Visual Studio using Irony.

Introduction

This is part 1 in a multi-article series on building your own IDE. The series will show how to extend the Visual Studio Isolated Shell to create an IDE. Each article in the series will build on the previous article to add new features to the environment. This article will create the My C language service in C# using the Irony Compiler Construction Kit. The language was initially created in the ManagedMyC example that is included with the Visual Studio SDK. The SDK example uses the Managed Babel System.

Using the Code in the Download

The first download supplied with this article contains the completed My C language service sample discussed in this article. It requires Visual Studio 2008 Standard edition or higher, and the Microsoft Visual Studio 2008 SDK.

Visual Studio - My C Programming Language

In order to run the solution:

  1. Download and unzip the source files.
  2. Open MyCLanguageService.sln in Visual Studio 2008.
  3. Build and run the project (F5).
  4. When the Visual Studio experimental hive opens, open a My C source file (.myc) in the code editor. Two test files are included in the download: short.myc and thing.myc.

Background

A language service in Visual Studio provides language-specific support (syntax coloring and highlighting, statement completion, brace matching, parameter information tooltips, etc.). A language service is implemented by creating a Visual Studio Integration Package. A project template is provided when the Visual Studio SDK is installed.

Managed language services are created using the Managed Package Framework. In addition, the Managed Babel System uses MPF, and provides a scanner and parser, MPLex, and MPPG. MPLex and MPPG are a variation of the Garden Point Scanner Generator (GPLex) and the Garden Point Parser Generator (GPPG) created by the Queensland University of Technology. These tools take a LEX-like and YACC-like specification and generate a scanner and parser in C#.

The Irony Compiler Construction Kit combines the lexical and parser specification in one C# file. Also, since the file is already in C#, no code generation takes place. This creates a solution that is easier to debug and maintain.

Creating the Language Service Package

The second download is a compressed file containing a template to generate a Visual Studio Integration Package for an Irony Language Service. In order to use the template, download and unzip the contents to a temporary location. The two DLL files, Wizard.dll and IronyLanguageServiceWizard.dll, should be placed in the Global Assembly Cache (C:\Windows\assembly). The third file, IronyLanguageServicePackage.zip, should be placed in the Visual Studio user templates directory, typically "My Documents\Visual Studio 2008\Templates\ProjectTemplates". This will install a Visual Studio template that can be accessed in the New Project dialog. It will appear in the My Templates section when C# is selected.

Add New Project Dialog

When creating a new Irony Language Service Package, a wizard is run to gather information specific to the language that is being created.

Irony Language Service Wizard

The first page requests basic information about the language. The company name will be used as the namespace for all the classes. Also, if your language supports multiple file extensions, separate each extension with a semicolon.

Irony Language Service Wizard - Language Information

Next, the wizard requests the features your language will support in Visual Studio. Leaving the options checked does not create any additional work, because the necessary integration is already included in the package.

Irony Language Service Wizard - Language Options

Package Load Keys (PLK) are required for VSPackages to load successfully. The Visual Studio SDK provides a Developer Load Key (DLK) that makes it possible for VSPackages to load without a PLK during development. If the language being developed will be used on machines without the Visual Studio SDK, a PLK is a must. A PLK can be generated here.

Irony Language Service Wizard - PLK

Defining Grammar in Irony

After running the Irony Language Service Wizard, there should be a Grammar.cs file open in the editor. This is where the Irony Grammar will be defined. The LEX and YACC files included with the ManagedMyC sample are being used for the language specification.

First, declare the terminals. Terminals include keywords, operators, comments, variables, etc.

CommentTerminal blockComment = new CommentTerminal("block-comment", "/*", "*/");
CommentTerminal lineComment = new CommentTerminal("line-comment", "//",
    "\r", "\n", "\u2085", "\u2028", "\u2029");
NonGrammarTerminals.Add(blockComment);
NonGrammarTerminals.Add(lineComment);

NumberLiteral number = new NumberLiteral("number");
IdentifierTerminal identifier = new IdentifierTerminal("identifier");

Next, declare the non-terminals:

NonTerminal program = new NonTerminal("program");
NonTerminal declarations = new NonTerminal("declaration");
NonTerminal declaration = new NonTerminal("declaration");
NonTerminal simpleDeclarations = new NonTerminal("simple-declarations");
NonTerminal simpleDeclaration = new NonTerminal("simple-declaration");
NonTerminal semiDeclaration = new NonTerminal("semi-declaration");
NonTerminal parenParameters = new NonTerminal("paren-parameters");
NonTerminal parameters = new NonTerminal("parameters");
NonTerminal classOption = new NonTerminal("class-option");
NonTerminal variableType = new NonTerminal("variable-type");
NonTerminal block = new NonTerminal("block");
NonTerminal blockContent = new NonTerminal("block-content");
NonTerminal statements = new NonTerminal("statements");
NonTerminal statement = new NonTerminal("statement");
NonTerminal parenExpressionAlways = new NonTerminal("paren-expression-always");
NonTerminal parenExpression = new NonTerminal("paren-expression");
NonTerminal forHeader = new NonTerminal("for-header");
NonTerminal forBlock = new NonTerminal("for-block");
NonTerminal semiStatement = new NonTerminal("semi-statement");
NonTerminal arguments = new NonTerminal("arguments");
NonTerminal parenArguments = new NonTerminal("paren-arguments");
NonTerminal assignExpression = new NonTerminal("assign-expression");
NonTerminal expression = new NonTerminal("expression");
NonTerminal booleanOperator = new NonTerminal("boolean-operator");
NonTerminal relationalExpression = new NonTerminal("relational-expression");
NonTerminal relationalOperator = new NonTerminal("relational-operator");
NonTerminal bitExpression = new NonTerminal("bit-expression");
NonTerminal bitOperator = new NonTerminal("bit-operator");
NonTerminal addExpression = new NonTerminal("add-expression");
NonTerminal addOperator = new NonTerminal("add-operator");
NonTerminal multiplyExpression = new NonTerminal("multiply");
NonTerminal multiplyOperator = new NonTerminal("multiply-operator");
NonTerminal prefixExpression = new NonTerminal("prefix-expression");
NonTerminal prefixOperator = new NonTerminal("prefix-operator");
NonTerminal factor = new NonTerminal("factor");
NonTerminal identifierExpression = new NonTerminal("identifier-expression");

Then, define the rules for the non-terminals. Also, in Irony, the root non-terminal must be specified.

this.Root = program;

program.Rule = declarations;

declarations.Rule = MakeStarRule(declarations, declaration);

declaration.Rule
    = classOption + variableType + identifier + parameters + block
    | classOption + identifier + parenParameters + block
    | variableType + identifier + parenParameters + block
    | identifier + parenParameters + block
    | simpleDeclaration;

simpleDeclarations.Rule = MakePlusRule(simpleDeclarations, simpleDeclaration);

simpleDeclaration.Rule = semiDeclaration + ";";

semiDeclaration.Rule
    = semiDeclaration + "," + identifier
    | classOption + variableType + identifier
    | variableType + identifier;

parameters.Rule
    = parameters + "," + variableType + identifier
    | variableType + identifier;

parenParameters.Rule
    = ToTerm("(") + ")"
    | "(" + parameters + ")";

classOption.Rule
    = ToTerm("static")
    | "auto"
    | "extern";

variableType.Rule
    = ToTerm("int")
    | "void";

block.Rule
    = ToTerm("{") + "}"
    | "{" + blockContent + "}";

blockContent.Rule
    = simpleDeclarations + statements
    | simpleDeclarations
    | statements;

statements.Rule = MakePlusRule(statements, statement);

statement.Rule
    = semiStatement + ";"
    | "while" + parenExpressionAlways + statement
    | "for" + forHeader + statement
    | "if" + parenExpressionAlways + statement
    | "if" + parenExpressionAlways + statement + "else" + statement;

parenExpressionAlways.Rule = parenExpression;

parenExpression.Rule = ToTerm("(") + expression + ")";

forHeader.Rule = "(" + forBlock + ")";

forBlock.Rule = assignExpression + ";" + expression + ";" + assignExpression;

semiStatement.Rule
    = assignExpression
    | "return" + expression
    | "break"
    | "continue";

arguments.Rule
    = expression + "," + arguments
    | expression;

parenArguments.Rule
    = ToTerm("(") + ")"
    | "(" + arguments + ")";

assignExpression.Rule
    = identifier + "=" + expression
    | expression;

expression.Rule
    = relationalExpression + booleanOperator + expression
    | relationalExpression;

booleanOperator.Rule
    = ToTerm("&&")
    | "||";

relationalExpression.Rule
    = bitExpression + relationalOperator + bitExpression
    | bitExpression;

relationalOperator.Rule
    = ToTerm(">")
    | ">="
    | "<"
    | "<="
    | "=="
    | "!=";

bitExpression.Rule
    = addExpression + bitOperator + bitExpression
    | addExpression;

bitOperator.Rule
    = ToTerm("|")
    | "&"
    | "^";

addExpression.Rule
    = multiplyExpression + addOperator + addExpression
    | prefixExpression;

addOperator.Rule
    = ToTerm("+") | "-";

multiplyExpression.Rule
    = prefixExpression + multiplyOperator + multiplyExpression
    | prefixExpression;

multiplyOperator.Rule
    = ToTerm("*")
    | "/";

prefixExpression.Rule
    = prefixOperator + factor
    | factor;

prefixOperator.Rule = ToTerm("!");

factor.Rule
    = identifierExpression + parenArguments
    | identifierExpression
    | number
    | parenExpression;

identifierExpression.Rule
    = identifier
    | identifierExpression + "." + identifier;

Last, the keywords are defined for the My C language. This is necessary for the syntax coloring. All terminals in the keyword list will be displayed as keywords in the code editor for Visual Studio.

this.MarkReservedWords("break", "continue", "else", "extern", "for", 
                       "if", "int", "return", "static", "void", "while");

Testing the My C Grammar

There are two options for testing the grammar: Grammar Explorer and Visual Studio Experimental Hive. The next two sections are a walkthrough in setting up the environment for either option.

Grammar Explorer

One of the many nice things about Irony is that it includes a tool for testing Irony grammars, called Grammar Explorer.

To setup the environment, right-click the language service project and select Properties. Select the Debug tab and choose Start External Program, and point it to the Irony.GrammarExplorer executable in the Resources directory.

Visual Studio - Project Properties

The first time Grammar Explorer is run, the grammar will need to be added. To add a grammar into Grammar Explorer, click the button next to the drop down and click Add Grammar from the menu.

Grammar Explorer - Add Grammar

Find the language service DLL and click Open.

Grammar Explorer - Open Grammar

The grammar should appear in the list and be selected, so the only thing to do here is click OK.

Grammar Explorer - Select Grammar

Once the grammar is added, it can be selected from the drop down. Now, it will display the grammar errors.

Grammar Explorer - Grammar Errors

The shift-reduce conflict on the else is in the original YACC grammar, and is caught by MPPG when the ManagedMyC example is compiled. This is a common problem with C grammars. Here is an example demonstrating the conflict:

if (x>y) if (x<z) foo(); else bar();

The compiler faces the problem of deciding whether to reduce "if (x<z) foo();" to an if_statement and then shift the "else", or if the "else" should be shifted first: both would be equally valid under the grammar as stated, but reduction would result in this:

if (x>y)
{
  if (x<z)
  {
    foo();
  }
}
else
{
  bar();
}

... associating the "else" with the first "if", whereas shifting the "else" produces...

if (x>y)
{
  if (x<z)
  {
    foo();
  }
  else
  {
    bar();
  }
}

... with the "else" associated with the second "if".

Irony provides a way to resolve this issue, which is presented below. However, if left alone, the grammar will still perform correctly. This is because the default behaviour of Irony is to shift, which produces the correct result in C and similar languages.

Visual Studio Experimental Hive

To setup the environment, right-click the language service project and select Properties. Select the Debug tab and choose Start External Program, and point it to the Visual Studio 2008 executable (default: C:\Program Files\Microsoft Visual Studio 9.0\Common7\IDE\devenv.exe). Also, add /ranu /rootsuffix Exp to the command line arguments.

Visual Studio - Project Properties

Build and run the project in Visual Studio (F5). This will launch the experimental hive. To test the language service, open a file with the .myc extension, or create a text file and save it with the .myc extension. Two files were included in the demo project: short.myc and thing.myc.

Visual Studio - My C Programming Language

Grammar Additions\Changes

This section covers additions or changes made to the My C sample grammar to demonstrate additional features of Irony.

Possible Non-terminal Duplication Warning

This warning is displayed anytime a non-terminal's rule contains a single non-terminal. There are several ways to resolve this issue. First, consider whether the non-terminal is really necessary. If the non-terminal should be there, the issue can be resolved by merging the two non-terminals.

The My C grammar has a program non-terminal that is causing this warning. This non-terminal was included from the original YACC file, and was probably placed there for expansion. To resolve the warning, the program and the declarations non-terminals are merged. Note that the program non-terminal must follow declarations so that the assignment is made after its rule is initialized.

this.Root = program;

declarations.Rule = MakeStarRule(declarations, declaration);

//Must follow declarations so that assignment 
//is made after rule is initialized.
program.Rule = declarations.Rule; 

Shift-reduce Conflicts

Irony includes a method for resolving shift-reduce conflicts, PreferShiftHere(). It allows you to tell the compiler where to shift. So, the error from earlier can be resolved by the following:

statement.Rule
    = semiStatement + ";"
    | "while" + parenExpressionAlways + statement
    | "for" + forHeader + statement
    | "if" + parenExpressionAlways + statement
    | "if" + parenExpressionAlways + statement + 
      PreferShiftHere() + "else" + statement;

Number Prefixes and Suffixes

Irony also supports number prefixes and suffixes. For example, the C language uses the prefix 0x to specify hexadecimal numbers. It also uses the f suffix to designate a value as a float. This can be defined in Irony with the following:

NumberLiteral number = new NumberLiteral("number");
number.AddPrefix("0x", NumberFlags.Hex);
number.AddSuffixCodes("f", TypeCode.Single);

Registering Brace Pairs

In YACC, brace pairs are matched for each rule.

Block
    : OpenBlock CloseBlock              {Match(@1, @2); }
    | OpenBlock BlockContent CloseBlock {Match(@1, @3); }
    ;

In Irony, it is a little different. Any terminal that should be matched in Visual Studio is registered in the grammar with the function RegisterBracePair. The My C grammar registers parenthesis and curly braces.

this.RegisterBracePair("{", "}");
this.RegisterBracePair("(", ")");

Operator Precedence

Another strength of Irony is the ability to define operator precedence. The My C language gives a higher precedence to the multiply and divide operators.

this.RegisterOperators(1, "+", "-");
this.RegisterOperators(2, "*", "/");

Points of Interest

When I first discovered Visual Studio language services, I used the Managed Babel System. After working with a YACC grammar, I quickly found that it was difficult to debug and would be hard to maintain in the future. Irony seems to solve both of these issues, and performs just as well. Irony also uses BNF expressions, so it was a very easy transition from YACC.

Articles to Come

  • Part 2 - Building your First Isolated Shell
  • Part 3 - Building your First Custom Project System
  • Part 4 - Building your First Advanced Language Service
  • Part 5 - Building your First Microsoft Build Task
  • Part 6 - Building your First Custom Debug Engine

History

  • December 10, 2009
    • Updated article and download content to work with the alpha release of Irony (October 13, 2009). Changes to the source code were made by Thaddeus Ryker.
  • February 19, 2009
    • Rewrote article to take more of a step-by-step approach.
    • Included the Irony Language Service Wizard.
    • Used the My C language as the sample, instead of My C#.
    • Included Irony.GrammarExplorer.exe.
  • February 10, 2009
    • Initial version.

License

This article, along with any associated source code and files, is licensed under The Code Project Open License (CPOL)

Share

About the Author

Ben Morrison
Software Developer
United States United States
No Biography provided

Comments and Discussions

 
GeneralRe: Error recovery PinmemberTakuan.Daikon3-Jan-10 8:27 
QuestionCan not find Wizard.dll and IronyLanguageServiceWizard.dll Pinmemberthiennhan13-Dec-09 23:33 
AnswerRe: Can not find Wizard.dll and IronyLanguageServiceWizard.dll PinmemberBen Morrison14-Dec-09 5:36 
GeneralRe: Can not find Wizard.dll and IronyLanguageServiceWizard.dll Pinmemberthiennhan14-Dec-09 16:08 
GeneralExcellent post PingroupMd. Marufuzzaman13-Dec-09 4:28 
GeneralVery interesting! PinmemberMarcelo Ricardo de Oliveira12-Dec-09 4:36 
GeneralRun AuthoringScope.GetMethods PinmembermykhayloZober22-Sep-09 7:52 
GeneralRe: Run AuthoringScope.GetMethods PinmemberBen Morrison23-Sep-09 4:30 
GeneralRe: Run AuthoringScope.GetMethods Pinmembersmbika21-Jan-10 9:47 
GeneralRe: Run AuthoringScope.GetMethods PinmemberBen Morrison21-Jan-10 10:03 

General General    News News    Suggestion Suggestion    Question Question    Bug Bug    Answer Answer    Joke Joke    Rant Rant    Admin Admin   

Use Ctrl+Left/Right to switch messages, Ctrl+Up/Down to switch threads, Ctrl+Shift+Left/Right to switch pages.

| Advertise | Privacy | Mobile
Web03 | 2.8.140821.2 | Last Updated 11 Dec 2009
Article Copyright 2009 by Ben Morrison
Everything else Copyright © CodeProject, 1999-2014
Terms of Service
Layout: fixed | fluid