Click here to Skip to main content
11,796,631 members (74,190 online)
Click here to Skip to main content

Tokenizer and analyzer package supporting precedence prioritized rules

, 1 Jan 2002 147.8K 2.5K 54
Rate this:
Please Sign up or sign in to vote.
A library allowing you to conveniently build a custom tokenizer and analyzer supporting precedence priorized rules

grammarIDE   Evaluation


This library has its origin in some study project I am currently working on. All source code is covered by the LGPL - this means you can freely use the code in any project (commercial or non-commercial).

How it works

The tokenizer uses a hierarchical map structure to store the tokens, and should be as fast as the paragon lexx (at least n*O(lexx) Smile | :) . The analyzer does not have the restriction of being a LALR(1) analyzer (if used correctly), has some caching capabilities and allows resolution of literal tokens. It also supports a precedence prioritized rule set.

All projects make heavy use of the STL. Because the STL implementation shipped with MSVC6 is not the best option, you should download the STL Port 4.5.1 implementation freely available at I have now managed to get rid most of the C4786 warnings (hint: switch /FI).

How to setup the analyzer/tokenizer

A typical rule definition for a simple expression evaluator looks like this:

std::tstringstream init(
    "200:+\n"       "201:-\n"
    "202:*\n"       "203:/\n"
    "204:^\n"       "205:;\n"
    "206:(\n"       "207:)\n"
    "' Whitespace tokens:\n"
    "0: \n"         "0:\t\n"
    "0:\\n\n"       "0:\\r\n"

This initializes the tokenizer to recognize the separators '+', '-', '*', '/', '^', ';', '(', ')' and to skip the usual white space characters, and initializes the analyzer to recognize the math rules '+,*,-,/,^' and declares that '*' has a higher priority than '+', for example.

The interface class

Usually you have to deal only with one class, cxtPackage. This class exports all methods needed to access the library. Some of them are:

  • vSetInputStream() - sets the input stream Smile | :)
  • vSetDelimeterIDs() - can be used to set the end-of-statement tokens, in C++ this could be ';'
  • nReadUntilDelimeter() - parses the input stream until the next delimiting token is found or the end of the input stream is reached
  • papbCheckForRule() - analyzes the token stream for a given rule and returns a parse tree (if successful)
  • vRebalance() - rearranges the parse tree according to the precedence priority rules

Those are the most important ones. For more details please see the sample project or check the page which will have some minimal documentation on the classes available for download soon.

New: The grammar IDE

The new grammar IDE included in the complete download provides a environment to develop and test analyzer rulesets. It has some syntax-highlighting features, shows errors by marking the lines in the editor and has an integrated test environment to live-check the results of the ruleset. I have no documentation yet and the IDE is still early beta and it has some cosmetic bugs (for example, it is possible to insert via clipboard RTF-formatted text into the editor Smile | :) , but most of it is already usable.

The sample projects

There are two sample projects included in the complete download. One is an almost-empty sample application you can easily use to explore the library, and the other project, simpleCalc, shows how to use the library to build a simple expression evaluator in 200 lines. A step-by-step explanation on how the sample works is here.

How to use it in own projects

Make sure to insert the projects cxTokenizer, cxAnalyzer and cxtPackage in the workspace of your project. Adjust the dependencies of your project to depend on cxtPackage (which itself depends on cxAnalyzer, which in turn depends on cxTokenizer). If your project doesn't use the MFC, you have to include the file common.cpp which is located in the base directory of the project files in your project. If you have problems or questions, feel free to mail them to For the most recent updates, see also


  • 2002-01-02
    C++ (w/o templates and pre-processor) grammar is now included. See also here.
  • 2001-12-29
    Fixed small bug in analyzer. IDE download now includes a not yet complete C grammar.
  • 2001-12-27
    Updated Homepage-URL to ad-free host.
  • 2001-12-26
    Uploaded new release including the grammarIDE and some enhancements in the analyzer.


This article has no explicit license attached to it but may contain usage terms in the article text or the download files themselves. If in doubt please contact the author via the discussion board below.

A list of licenses authors might use can be found here


About the Author

Alexander Berthold
Web Developer
Germany Germany
No Biography provided

You may also be interested in...

Comments and Discussions

Generalbroken link Pin
Ali Khanlarkhani17-Dec-02 0:52
memberAli Khanlarkhani17-Dec-02 0:52 
Generalrepresneting grammars and parse trees Pin
Steven Corkey2-Dec-02 8:14
sussSteven Corkey2-Dec-02 8:14 
GeneralParse Tree Pin
Ed Deighton4-Aug-02 3:17
sussEd Deighton4-Aug-02 3:17 
QuestionWebsite links broken ? Pin
Jerry Evans20-May-02 8:12
memberJerry Evans20-May-02 8:12 
I keep getting a 403 error
AnswerRe: Website links broken ? Pin
realfly28-Jul-02 20:36
memberrealfly28-Jul-02 20:36 
GeneralResults of comparison vs bison/flex parser Pin
Matt Wigdahl8-Jan-02 12:10
memberMatt Wigdahl8-Jan-02 12:10 
GeneralRe: Results of comparison vs bison/flex parser Pin
Martin.Holzherr18-May-05 5:04
memberMartin.Holzherr18-May-05 5:04 
GeneralRe: Results of comparison vs bison/flex parser Pin
Matt Wigdahl18-May-05 7:29
memberMatt Wigdahl18-May-05 7:29 
GeneralDefining grammar Pin
Ray Hayes2-Jan-02 5:27
memberRay Hayes2-Jan-02 5:27 
QuestionSeriously, why not just use lex??? Pin
Matt Wigdahl28-Dec-01 9:50
memberMatt Wigdahl28-Dec-01 9:50 
AnswerRe: Seriously, why not just use lex??? Pin
Alexander Berthold28-Dec-01 11:32
memberAlexander Berthold28-Dec-01 11:32 
GeneralRe: Seriously, why not just use lex??? Pin
Matt Wigdahl28-Dec-01 11:50
memberMatt Wigdahl28-Dec-01 11:50 
GeneralRe: Seriously, why not just use lex??? Pin
Alexander Berthold28-Dec-01 23:02
memberAlexander Berthold28-Dec-01 23:02 
GeneralJust a correction Pin
Gabriel Praino28-Dec-01 2:48
memberGabriel Praino28-Dec-01 2:48 
GeneralRe: Just a correction Pin
Alexander Berthold28-Dec-01 3:32
memberAlexander Berthold28-Dec-01 3:32 
GeneralRe: Just a correction Pin
kuhx19808-Dec-03 1:36
memberkuhx19808-Dec-03 1:36 
GeneralRe: Just a correction Pin
Tim Smith29-Dec-01 2:20
memberTim Smith29-Dec-01 2:20 
General'lexx' and 'yacc' and Antlr Pin
Paul Selormey25-Dec-01 20:09
memberPaul Selormey25-Dec-01 20:09 
GeneralRe: 'lexx' and 'yacc' and Antlr Pin
Alexander Berthold25-Dec-01 23:52
memberAlexander Berthold25-Dec-01 23:52 
GeneralSuggestion Pin
Shark19-Jun-01 2:57
memberShark19-Jun-01 2:57 
GeneralRe: Suggestion Pin
Alexander Berthold19-Jun-01 3:02
memberAlexander Berthold19-Jun-01 3:02 
GeneralSuggestion Pin
Shark19-Jun-01 3:02
memberShark19-Jun-01 3:02 
GeneralWarning c4768 Pin
Paul Martinsen18-Jun-01 13:19
memberPaul Martinsen18-Jun-01 13:19 
GeneralHungarian notation Pin
Sven Axelsson14-Jun-01 23:17
memberSven Axelsson14-Jun-01 23:17 
GeneralRe: Hungarian notation Pin
William Bartholomew18-Jun-01 12:21
memberWilliam Bartholomew18-Jun-01 12:21 
GeneralRe: Hungarian notation Pin
Sven Axelsson18-Jun-01 21:56
memberSven Axelsson18-Jun-01 21:56 
GeneralRe: Hungarian notation Pin
Alexander Berthold19-Jun-01 1:45
memberAlexander Berthold19-Jun-01 1:45 
GeneralRe: Hungarian notation Pin
Alexander Berthold19-Jun-01 3:15
memberAlexander Berthold19-Jun-01 3:15 
GeneralRe: Hungarian notation Pin
Sven Axelsson19-Jun-01 3:34
memberSven Axelsson19-Jun-01 3:34 
GeneralRe: Hungarian notation Pin
Tim Lesher27-Dec-01 6:49
memberTim Lesher27-Dec-01 6:49 
GeneralRe: Hungarian notation Pin
Alexander Berthold27-Dec-01 8:29
memberAlexander Berthold27-Dec-01 8:29 
GeneralRe: Hungarian notation Pin
Jean-Marc Molina22-Feb-02 9:45
memberJean-Marc Molina22-Feb-02 9:45 

General General    News News    Suggestion Suggestion    Question Question    Bug Bug    Answer Answer    Joke Joke    Rant Rant    Admin Admin   

Use Ctrl+Left/Right to switch messages, Ctrl+Up/Down to switch threads, Ctrl+Shift+Left/Right to switch pages.

| Advertise | Privacy | Terms of Use | Mobile
Web03 | 2.8.151002.1 | Last Updated 2 Jan 2002
Article Copyright 2001 by Alexander Berthold
Everything else Copyright © CodeProject, 1999-2015
Layout: fixed | fluid