Introduction
This is part 1 in a multi-article series on building your own IDE. The series will show how to extend the Visual Studio Isolated Shell to create an IDE. Each article in the series will build on the previous article to add new features to the environment. This article will create the My C language service in C# using the Irony Compiler Construction Kit. The language was initially created in the ManagedMyC example that is included with the Visual Studio SDK. The SDK example uses the Managed Babel System.
Using the Code in the Download
The first download supplied with this article contains the completed My C language service sample discussed in this article. It requires Visual Studio 2008 Standard edition or higher, and the Microsoft Visual Studio 2008 SDK.
In order to run the solution:
- Download and unzip the source files.
- Open MyCLanguageService.sln in Visual Studio 2008.
- Build and run the project (F5).
- When the Visual Studio experimental hive opens, open a My C source file (.myc) in the code editor. Two test files are included in the download: short.myc and thing.myc.
Background
A language service in Visual Studio provides language-specific support (syntax coloring and highlighting, statement completion, brace matching, parameter information tooltips, etc.). A language service is implemented by creating a Visual Studio Integration Package. A project template is provided when the Visual Studio SDK is installed.
Managed language services are created using the Managed Package Framework. In addition, the Managed Babel System uses MPF, and provides a scanner and parser, MPLex, and MPPG. MPLex and MPPG are a variation of the Garden Point Scanner Generator (GPLex) and the Garden Point Parser Generator (GPPG) created by the Queensland University of Technology. These tools take a LEX-like and YACC-like specification and generate a scanner and parser in C#.
The Irony Compiler Construction Kit combines the lexical and parser specification in one C# file. Also, since the file is already in C#, no code generation takes place. This creates a solution that is easier to debug and maintain.
Creating the Language Service Package
The second download is a compressed file containing a template to generate a Visual Studio Integration Package for an Irony Language Service. In order to use the template, download and unzip the contents to a temporary location. The two DLL files, Wizard.dll and IronyLanguageServiceWizard.dll, should be placed in the Global Assembly Cache (C:\Windows\assembly). The third file, IronyLanguageServicePackage.zip, should be placed in the Visual Studio user templates directory, typically "My Documents\Visual Studio 2008\Templates\ProjectTemplates". This will install a Visual Studio template that can be accessed in the New Project dialog. It will appear in the My Templates section when C# is selected.
When creating a new Irony Language Service Package, a wizard is run to gather information specific to the language that is being created.
The first page requests basic information about the language. The company name will be used as the namespace for all the classes. Also, if your language supports multiple file extensions, separate each extension with a semicolon.
Next, the wizard requests the features your language will support in Visual Studio. Leaving the options checked does not create any additional work, because the necessary integration is already included in the package.
Package Load Keys (PLK) are required for VSPackages to load successfully. The Visual Studio SDK provides a Developer Load Key (DLK) that makes it possible for VSPackages to load without a PLK during development. If the language being developed will be used on machines without the Visual Studio SDK, a PLK is a must. A PLK can be generated here.
Defining Grammar in Irony
After running the Irony Language Service Wizard, there should be a Grammar.cs file open in the editor. This is where the Irony Grammar will be defined. The LEX and YACC files included with the ManagedMyC sample are being used for the language specification.
First, declare the terminals. Terminals include keywords, operators, comments, variables, etc.
CommentTerminal blockComment = new CommentTerminal("block-comment", "/*", "*/");
CommentTerminal lineComment = new CommentTerminal("line-comment", "//",
"\r", "\n", "\u2085", "\u2028", "\u2029");
NonGrammarTerminals.Add(blockComment);
NonGrammarTerminals.Add(lineComment);
NumberLiteral number = new NumberLiteral("number");
IdentifierTerminal identifier = new IdentifierTerminal("identifier");
Next, declare the non-terminals:
NonTerminal program = new NonTerminal("program");
NonTerminal declarations = new NonTerminal("declaration");
NonTerminal declaration = new NonTerminal("declaration");
NonTerminal simpleDeclarations = new NonTerminal("simple-declarations");
NonTerminal simpleDeclaration = new NonTerminal("simple-declaration");
NonTerminal semiDeclaration = new NonTerminal("semi-declaration");
NonTerminal parenParameters = new NonTerminal("paren-parameters");
NonTerminal parameters = new NonTerminal("parameters");
NonTerminal classOption = new NonTerminal("class-option");
NonTerminal variableType = new NonTerminal("variable-type");
NonTerminal block = new NonTerminal("block");
NonTerminal blockContent = new NonTerminal("block-content");
NonTerminal statements = new NonTerminal("statements");
NonTerminal statement = new NonTerminal("statement");
NonTerminal parenExpressionAlways = new NonTerminal("paren-expression-always");
NonTerminal parenExpression = new NonTerminal("paren-expression");
NonTerminal forHeader = new NonTerminal("for-header");
NonTerminal forBlock = new NonTerminal("for-block");
NonTerminal semiStatement = new NonTerminal("semi-statement");
NonTerminal arguments = new NonTerminal("arguments");
NonTerminal parenArguments = new NonTerminal("paren-arguments");
NonTerminal assignExpression = new NonTerminal("assign-expression");
NonTerminal expression = new NonTerminal("expression");
NonTerminal booleanOperator = new NonTerminal("boolean-operator");
NonTerminal relationalExpression = new NonTerminal("relational-expression");
NonTerminal relationalOperator = new NonTerminal("relational-operator");
NonTerminal bitExpression = new NonTerminal("bit-expression");
NonTerminal bitOperator = new NonTerminal("bit-operator");
NonTerminal addExpression = new NonTerminal("add-expression");
NonTerminal addOperator = new NonTerminal("add-operator");
NonTerminal multiplyExpression = new NonTerminal("multiply");
NonTerminal multiplyOperator = new NonTerminal("multiply-operator");
NonTerminal prefixExpression = new NonTerminal("prefix-expression");
NonTerminal prefixOperator = new NonTerminal("prefix-operator");
NonTerminal factor = new NonTerminal("factor");
NonTerminal identifierExpression = new NonTerminal("identifier-expression");
Then, define the rules for the non-terminals. Also, in Irony, the root non-terminal must be specified.
this.Root = program;
program.Rule = declarations;
declarations.Rule = MakeStarRule(declarations, declaration);
declaration.Rule
= classOption + variableType + identifier + parameters + block
| classOption + identifier + parenParameters + block
| variableType + identifier + parenParameters + block
| identifier + parenParameters + block
| simpleDeclaration;
simpleDeclarations.Rule = MakePlusRule(simpleDeclarations, simpleDeclaration);
simpleDeclaration.Rule = semiDeclaration + ";";
semiDeclaration.Rule
= semiDeclaration + "," + identifier
| classOption + variableType + identifier
| variableType + identifier;
parameters.Rule
= parameters + "," + variableType + identifier
| variableType + identifier;
parenParameters.Rule
= ToTerm("(") + ")"
| "(" + parameters + ")";
classOption.Rule
= ToTerm("static")
| "auto"
| "extern";
variableType.Rule
= ToTerm("int")
| "void";
block.Rule
= ToTerm("{") + "}"
| "{" + blockContent + "}";
blockContent.Rule
= simpleDeclarations + statements
| simpleDeclarations
| statements;
statements.Rule = MakePlusRule(statements, statement);
statement.Rule
= semiStatement + ";"
| "while" + parenExpressionAlways + statement
| "for" + forHeader + statement
| "if" + parenExpressionAlways + statement
| "if" + parenExpressionAlways + statement + "else" + statement;
parenExpressionAlways.Rule = parenExpression;
parenExpression.Rule = ToTerm("(") + expression + ")";
forHeader.Rule = "(" + forBlock + ")";
forBlock.Rule = assignExpression + ";" + expression + ";" + assignExpression;
semiStatement.Rule
= assignExpression
| "return" + expression
| "break"
| "continue";
arguments.Rule
= expression + "," + arguments
| expression;
parenArguments.Rule
= ToTerm("(") + ")"
| "(" + arguments + ")";
assignExpression.Rule
= identifier + "=" + expression
| expression;
expression.Rule
= relationalExpression + booleanOperator + expression
| relationalExpression;
booleanOperator.Rule
= ToTerm("&&")
| "||";
relationalExpression.Rule
= bitExpression + relationalOperator + bitExpression
| bitExpression;
relationalOperator.Rule
= ToTerm(">")
| ">="
| "<"
| "<="
| "=="
| "!=";
bitExpression.Rule
= addExpression + bitOperator + bitExpression
| addExpression;
bitOperator.Rule
= ToTerm("|")
| "&"
| "^";
addExpression.Rule
= multiplyExpression + addOperator + addExpression
| prefixExpression;
addOperator.Rule
= ToTerm("+") | "-";
multiplyExpression.Rule
= prefixExpression + multiplyOperator + multiplyExpression
| prefixExpression;
multiplyOperator.Rule
= ToTerm("*")
| "/";
prefixExpression.Rule
= prefixOperator + factor
| factor;
prefixOperator.Rule = ToTerm("!");
factor.Rule
= identifierExpression + parenArguments
| identifierExpression
| number
| parenExpression;
identifierExpression.Rule
= identifier
| identifierExpression + "." + identifier;
Last, the keywords are defined for the My C language. This is necessary for the syntax coloring. All terminals in the keyword list will be displayed as keywords in the code editor for Visual Studio.
this.MarkReservedWords("break", "continue", "else", "extern", "for",
"if", "int", "return", "static", "void", "while");
Testing the My C Grammar
There are two options for testing the grammar: Grammar Explorer and Visual Studio Experimental Hive. The next two sections are a walkthrough in setting up the environment for either option.
Grammar Explorer
One of the many nice things about Irony is that it includes a tool for testing Irony grammars, called Grammar Explorer.
To setup the environment, right-click the language service project and select Properties. Select the Debug tab and choose Start External Program, and point it to the Irony.GrammarExplorer executable in the Resources directory.
The first time Grammar Explorer is run, the grammar will need to be added. To add a grammar into Grammar Explorer, click the button next to the drop down and click Add Grammar from the menu.
Find the language service DLL and click Open.
The grammar should appear in the list and be selected, so the only thing to do here is click OK.
Once the grammar is added, it can be selected from the drop down. Now, it will display the grammar errors.
The shift-reduce conflict on the else is in the original YACC grammar, and is caught by MPPG when the ManagedMyC example is compiled. This is a common problem with C grammars. Here is an example demonstrating the conflict:
if (x>y) if (x<z) foo(); else bar();
The compiler faces the problem of deciding whether to reduce "if (x<z) foo();" to an if_statement and then shift the "else", or if the "else" should be shifted first: both would be equally valid under the grammar as stated, but reduction would result in this:
if (x>y)
{
if (x<z)
{
foo();
}
}
else
{
bar();
}
... associating the "else" with the first "if", whereas shifting the "else" produces...
if (x>y)
{
if (x<z)
{
foo();
}
else
{
bar();
}
}
... with the "else" associated with the second "if".
Irony provides a way to resolve this issue, which is presented below. However, if left alone, the grammar will still perform correctly. This is because the default behaviour of Irony is to shift, which produces the correct result in C and similar languages.
Visual Studio Experimental Hive
To setup the environment, right-click the language service project and select Properties. Select the Debug tab and choose Start External Program, and point it to the Visual Studio 2008 executable (default: C:\Program Files\Microsoft Visual Studio 9.0\Common7\IDE\devenv.exe). Also, add /ranu /rootsuffix Exp to the command line arguments.
Build and run the project in Visual Studio (F5). This will launch the experimental hive. To test the language service, open a file with the .myc extension, or create a text file and save it with the .myc extension. Two files were included in the demo project: short.myc and thing.myc.
Grammar Additions\Changes
This section covers additions or changes made to the My C sample grammar to demonstrate additional features of Irony.
Possible Non-terminal Duplication Warning
This warning is displayed anytime a non-terminal's rule contains a single non-terminal. There are several ways to resolve this issue. First, consider whether the non-terminal is really necessary. If the non-terminal should be there, the issue can be resolved by merging the two non-terminals.
The My C grammar has a program non-terminal that is causing this warning. This non-terminal was included from the original YACC file, and was probably placed there for expansion. To resolve the warning, the program and the declarations non-terminals are merged. Note that the program non-terminal must follow declarations so that the assignment is made after its rule is initialized.
this.Root = program;
declarations.Rule = MakeStarRule(declarations, declaration);
program.Rule = declarations.Rule;
Shift-reduce Conflicts
Irony includes a method for resolving shift-reduce conflicts, PreferShiftHere(). It allows you to tell the compiler where to shift. So, the error from earlier can be resolved by the following:
statement.Rule
= semiStatement + ";"
| "while" + parenExpressionAlways + statement
| "for" + forHeader + statement
| "if" + parenExpressionAlways + statement
| "if" + parenExpressionAlways + statement +
PreferShiftHere() + "else" + statement;
Number Prefixes and Suffixes
Irony also supports number prefixes and suffixes. For example, the C language uses the prefix 0x to specify hexadecimal numbers. It also uses the f suffix to designate a value as a float. This can be defined in Irony with the following:
NumberLiteral number = new NumberLiteral("number");
number.AddPrefix("0x", NumberFlags.Hex);
number.AddSuffixCodes("f", TypeCode.Single);
Registering Brace Pairs
In YACC, brace pairs are matched for each rule.
Block
: OpenBlock CloseBlock {Match(@1, @2); }
| OpenBlock BlockContent CloseBlock {Match(@1, @3); }
;
In Irony, it is a little different. Any terminal that should be matched in Visual Studio is registered in the grammar with the function RegisterBracePair. The My C grammar registers parenthesis and curly braces.
this.RegisterBracePair("{", "}");
this.RegisterBracePair("(", ")");
Operator Precedence
Another strength of Irony is the ability to define operator precedence. The My C language gives a higher precedence to the multiply and divide operators.
this.RegisterOperators(1, "+", "-");
this.RegisterOperators(2, "*", "/");
Points of Interest
When I first discovered Visual Studio language services, I used the Managed Babel System. After working with a YACC grammar, I quickly found that it was difficult to debug and would be hard to maintain in the future. Irony seems to solve both of these issues, and performs just as well. Irony also uses BNF expressions, so it was a very easy transition from YACC.
Articles to Come
- Part 2 - Building your First Isolated Shell
- Part 3 - Building your First Custom Project System
- Part 4 - Building your First Advanced Language Service
- Part 5 - Building your First Microsoft Build Task
- Part 6 - Building your First Custom Debug Engine
History
- December 10, 2009
- Updated article and download content to work with the alpha release of Irony (October 13, 2009). Changes to the source code were made by Thaddeus Ryker.
- February 19, 2009
- Rewrote article to take more of a step-by-step approach.
- Included the Irony Language Service Wizard.
- Used the My C language as the sample, instead of My C#.
- Included Irony.GrammarExplorer.exe.
- February 10, 2009
Submit an article or tip
Post your Blog
News 
Suggestion 
Bug 
Joke 
Rant 
Admin 
