Click here to Skip to main content
Click here to Skip to main content
Search within:



 
Comments & Discussions

Tokenizer and analyzer package supporting precedence prioritized rules

By , 1 Jan 2002
 
cxtpackagetut_win32vc.zip
common.cpp
common.h
common_error.h
common_win32.h
COPYING
cpAbsd.dsw
cxAnalyzer
analyzer.h
cxAnalyzer.dsp
cxAnalyzer.plg
cxAnalyzerException.h
cxAnalyzerExpression.cpp
cxAnalyzerExpression.h
cxAnalyzerMain.cpp
cxAnalyzerMain.h
cxAnalyzerTree.cpp
cxAnalyzerTree.h
cxAnalyzerTypeMap.cpp
cxAnalyzerTypeMap.h
cxaParseTree.cpp
cxaParseTree.h
cxaToken.cpp
cxaToken.h
cxaTokenStream.cpp
Readme.txt
StdAfx.cpp
StdAfx.h
cxTokenizer
cxTokenizer.cpp
cxTokenizer.dsp
cxTokenizer.h
cxTokenizer.plg
cxTokenizerCharTokenRule.cpp
cxTokenizerCharTokenRule.h
cxTokenizerContext.cpp
cxTokenizerContext.h
cxTokenizerContextCookie.h
cxTokenizerContextDiags.cpp
cxTokenizerDiags.cpp
cxTokenizerException.cpp
cxTokenizerException.h
cxTokenizerInputStream.h
cxTokenizerMap.cpp
cxTokenizerMap.h
cxTokenizerMapData.cpp
cxTokenizerMapData.h
cxTokenizerMapDataDiags.cpp
cxTokenizerMapDiags.cpp
cxTokenizerNumberTokenRule.cpp
cxTokenizerNumberTokenRule.h
cxTokenizerSTLInputStream.cpp
cxTokenizerSTLInputStream.h
cxTokenizerStringTokenRule.cpp
cxTokenizerStringTokenRule.h
cxTokenizerTextInputStream.cpp
cxTokenizerTextInputStream.h
cxTokenizerTokenRule.cpp
cxTokenizerTokenRule.h
Readme.txt
StdAfx.cpp
StdAfx.h
tokenizer.h
tokenizer_error.h
tokenizer_flags.h
tokenizerBase.h
cxtPackage
cxtPackage.cpp
cxtPackage.dsp
cxtPackage.h
cxtPackage.plg
package.h
Readme.txt
StdAfx.cpp
StdAfx.h
mathTok
mathTok.cpp
mathTok.dsp
mathTok.plg
ReadMe.txt
StdAfx.cpp
StdAfx.h
simpleCalc
ReadMe.txt
simpleCalc.bmp
simpleCalc.cpp
simpleCalc.dsp
simpleCalc.plg
StdAfx.cpp
StdAfx.h
tkCommon
ctkCheckValid.h
ctkExternalObjectPointer.h
ctkFlagsMixin.h
ctkHLinkedList.cpp
ctkHLinkedList.h
ctkMisc.h
ctkSerializable.h
tkCommon.h
cxtpackage_win32vc.zip
common.cpp
common.h
common_error.h
common_win32.h
COPYING
cpAbsd.dsw
analyzer.h
cxAnalyzer.dsp
cxAnalyzer.plg
cxAnalyzerException.h
cxAnalyzerExpression.cpp
cxAnalyzerExpression.h
cxAnalyzerMain.cpp
cxAnalyzerMain.h
cxAnalyzerTree.cpp
cxAnalyzerTree.h
cxAnalyzerTypeMap.cpp
cxAnalyzerTypeMap.h
cxaParseTree.cpp
cxaParseTree.h
cxaToken.cpp
cxaToken.h
cxaTokenStream.cpp
Debug
Readme.txt
StdAfx.cpp
StdAfx.h
cxTokenizer.cpp
cxTokenizer.dsp
cxTokenizer.h
cxTokenizer.plg
cxTokenizerCharTokenRule.cpp
cxTokenizerCharTokenRule.h
cxTokenizerContext.cpp
cxTokenizerContext.h
cxTokenizerContextCookie.h
cxTokenizerContextDiags.cpp
cxTokenizerDiags.cpp
cxTokenizerException.cpp
cxTokenizerException.h
cxTokenizerInputStream.h
cxTokenizerMap.cpp
cxTokenizerMap.h
cxTokenizerMapData.cpp
cxTokenizerMapData.h
cxTokenizerMapDataDiags.cpp
cxTokenizerMapDiags.cpp
cxTokenizerNumberTokenRule.cpp
cxTokenizerNumberTokenRule.h
cxTokenizerSTLInputStream.cpp
cxTokenizerSTLInputStream.h
cxTokenizerStringTokenRule.cpp
cxTokenizerStringTokenRule.h
cxTokenizerTextInputStream.cpp
cxTokenizerTextInputStream.h
cxTokenizerTokenRule.cpp
cxTokenizerTokenRule.h
Readme.txt
StdAfx.cpp
StdAfx.h
tokenizer.h
tokenizer_error.h
tokenizer_flags.h
tokenizerBase.h
cxtPackage.cpp
cxtPackage.dsp
cxtPackage.h
cxtPackage.plg
package.h
Readme.txt
StdAfx.cpp
StdAfx.h
ctkCheckValid.h
ctkExternalObjectPointer.h
ctkFlagsMixin.h
ctkHLinkedList.cpp
ctkHLinkedList.h
ctkMisc.h
ctkSerializable.h
tkCommon.h
grammaride.zip
cpp-grammar.txt
grammarIDE.exe
readme.txt
sample-grammar.txt
stlport_vc645.dll
pkgcomplete.zip
COMMON.CPP
COMMON.H
common_error.h
common_win32.h
COPYING
cpAbsd.dsw
cpp-grammar.txt
ANALYZER.H
cxAnalyzer.dsp
cxAnalyzerException.h
cxAnalyzerExpression.cpp
cxAnalyzerExpression.h
cxAnalyzerMain.cpp
cxAnalyzerMain.h
cxAnalyzerTree.cpp
cxAnalyzerTree.h
cxAnalyzerTypeMap.cpp
cxAnalyzerTypeMap.h
cxaParseTree.cpp
cxaParseTree.h
cxaRuleCache.cpp
cxaRuleCache.h
cxaRuleCacheElement.cpp
cxaRuleCacheElement.h
cxaToken.cpp
cxaToken.h
cxaToken.inl
cxaTokenStream.cpp
Readme.txt
StdAfx.cpp
StdAfx.h
cxTokenizer.cpp
cxTokenizer.dsp
cxTokenizer.h
cxTokenizerCharTokenRule.cpp
cxTokenizerCharTokenRule.h
cxTokenizerCommentTokenRule.cpp
cxTokenizerCommentTokenRule.h
cxTokenizerContext.cpp
cxTokenizerContext.h
cxTokenizerContextCookie.h
cxTokenizerContextDiags.cpp
cxTokenizerDiags.cpp
cxTokenizerException.cpp
cxTokenizerException.h
cxTokenizerInputStream.h
cxTokenizerMap.cpp
cxTokenizerMap.h
cxTokenizerMapData.cpp
cxTokenizerMapData.h
cxTokenizerMapDataDiags.cpp
cxTokenizerMapDiags.cpp
cxTokenizerMatchTokenRule.h
cxTokenizerMatchTokenRule.inl
cxTokenizerNumberTokenRule.cpp
cxTokenizerNumberTokenRule.h
cxTokenizerSTLInputStream.cpp
cxTokenizerSTLInputStream.h
cxTokenizerStringTokenRule.cpp
cxTokenizerStringTokenRule.h
cxTokenizerTextInputStream.cpp
cxTokenizerTextInputStream.h
cxTokenizerTokenRule.cpp
cxTokenizerTokenRule.h
Readme.txt
StdAfx.cpp
StdAfx.h
tokenizer.h
tokenizer_error.h
tokenizer_flags.h
tokenizerBase.h
cxtPackage.cpp
cxtPackage.dsp
cxtPackage.h
PACKAGE.H
Readme.txt
StdAfx.cpp
StdAfx.h
todo.txt
DEBUG.CPP
DEBUG.H
emptyTestApp
emptyTestApp.clw
emptyTestApp.cpp
emptyTestApp.dsp
emptyTestApp.h
emptyTestApp.rc
emptyTestAppDlg.cpp
emptyTestAppDlg.h
ReadMe.txt
res
emptyTestApp.ico
emptyTestApp.rc2
Resource.h
StdAfx.cpp
StdAfx.h
grammarIDE
DlgEvaluate.cpp
DlgEvaluate.h
DlgProperties.cpp
DlgProperties.h
grammarIDE.clw
grammarIDE.cpp
grammarIDE.dsp
grammarIDE.h
grammarIDE.rc
grammarIDEDoc.cpp
grammarIDEDoc.h
grammarIDEView.cpp
grammarIDEView.h
LeftView.cpp
LeftView.h
ltrItemTreeWnd.cpp
ltrItemTreeWnd.h
MainFrm.cpp
MainFrm.h
PaneEditorWnd.cpp
PaneEditorWnd.h
propsItemGrammar.cpp
propsItemGrammar.h
ReadMe.txt
res
grammarIDE.ico
grammarIDE.rc2
grammarIDEDoc.ico
icon1.ico
Toolbar.bmp
vssver.scc
zoomable.ico
Resource.h
StdAfx.cpp
StdAfx.h
readme.txt
sample-grammar.txt
ReadMe.txt
simpleCalc.bmp
simpleCalc.cpp
simpleCalc.dsp
StdAfx.cpp
StdAfx.h
ctkCheckValid.h
ctkEnumerator.h
ctkExternalObjectPointer.h
ctkFlagsMixin.h
ctkHLinkedList.cpp
ctkHLinkedList.h
ctkMisc.h
ctkSerializable.h
tkCommon.h
vssver.scc
vcstl_nowarnings.h
pkgsrconly.zip
COMMON.CPP
COMMON.H
common_error.h
common_win32.h
COPYING
cpAbsd.dsw
cpp-grammar.txt
ANALYZER.H
cxAnalyzer.dsp
cxAnalyzerException.h
cxAnalyzerExpression.cpp
cxAnalyzerExpression.h
cxAnalyzerMain.cpp
cxAnalyzerMain.h
cxAnalyzerTree.cpp
cxAnalyzerTree.h
cxAnalyzerTypeMap.cpp
cxAnalyzerTypeMap.h
cxaParseTree.cpp
cxaParseTree.h
cxaRuleCache.cpp
cxaRuleCache.h
cxaRuleCacheElement.cpp
cxaRuleCacheElement.h
cxaToken.cpp
cxaToken.h
cxaToken.inl
cxaTokenStream.cpp
Readme.txt
StdAfx.cpp
StdAfx.h
cxTokenizer.cpp
cxTokenizer.dsp
cxTokenizer.h
cxTokenizerCharTokenRule.cpp
cxTokenizerCharTokenRule.h
cxTokenizerCommentTokenRule.cpp
cxTokenizerCommentTokenRule.h
cxTokenizerContext.cpp
cxTokenizerContext.h
cxTokenizerContextCookie.h
cxTokenizerContextDiags.cpp
cxTokenizerDiags.cpp
cxTokenizerException.cpp
cxTokenizerException.h
cxTokenizerInputStream.h
cxTokenizerMap.cpp
cxTokenizerMap.h
cxTokenizerMapData.cpp
cxTokenizerMapData.h
cxTokenizerMapDataDiags.cpp
cxTokenizerMapDiags.cpp
cxTokenizerMatchTokenRule.h
cxTokenizerMatchTokenRule.inl
cxTokenizerNumberTokenRule.cpp
cxTokenizerNumberTokenRule.h
cxTokenizerSTLInputStream.cpp
cxTokenizerSTLInputStream.h
cxTokenizerStringTokenRule.cpp
cxTokenizerStringTokenRule.h
cxTokenizerTextInputStream.cpp
cxTokenizerTextInputStream.h
cxTokenizerTokenRule.cpp
cxTokenizerTokenRule.h
Readme.txt
StdAfx.cpp
StdAfx.h
tokenizer.h
tokenizer_error.h
tokenizer_flags.h
tokenizerBase.h
cxtPackage.cpp
cxtPackage.dsp
cxtPackage.h
PACKAGE.H
Readme.txt
StdAfx.cpp
StdAfx.h
todo.txt
DEBUG.CPP
DEBUG.H
emptyTestApp.clw
emptyTestApp.cpp
emptyTestApp.dsp
emptyTestApp.h
emptyTestApp.rc
emptyTestAppDlg.cpp
emptyTestAppDlg.h
ReadMe.txt
emptyTestApp.ico
emptyTestApp.rc2
Resource.h
StdAfx.cpp
StdAfx.h
readme.txt
sample-grammar.txt
ctkCheckValid.h
ctkEnumerator.h
ctkExternalObjectPointer.h
ctkFlagsMixin.h
ctkHLinkedList.cpp
ctkHLinkedList.h
ctkMisc.h
ctkSerializable.h
tkCommon.h
vssver.scc
vcstl_nowarnings.h
 
#ifdef _DEBUG
bool	cxTokenizerMap::fCheckValid() const
	{
	if(!m_tmdFPRules.fCheckValid())
		return false;
	if(!m_tmdWIRules.fCheckValid())
		return false;

	return true;
	}

bool	cxTokenizerMap::fRunDiagnostics()
	{
	std::tstringstream	strm(
		_T("[seperators]\n")
		_T("100:-\n")
		_T("101:--\n")
		_T("102:-+\n")
		_T("103:-*\n")
		_T("[tokens]\n")
		_T("102:test\n")
		_T("[rules]\n")
		_T("103:strings\n")
		_T("104:numbers\n") );

	cxTokenizerMap					mapTest;
	cxTokenizerMapData				*pos;
	cxTokenizerMapData::iterator	it;

	mapTest.vLoadFromStream(strm);

	if(mapTest.m_vecRulesToDelete.size()!=7)
		return false;

	it			=mapTest.m_tmdWIRules.find('\0');
	if(it==mapTest.m_tmdWIRules.end())
		return false;

	it++;
	if(it->first!='\0')
		return false;

	it			=mapTest.m_tmdWIRules.find('-');
	if(it==mapTest.m_tmdWIRules.end())
		return false;
	if(it->first!='-')
		return false;

	pos			=(*it).second;
	if(pos->m_pRule==NULL)
		return false;

	it			=pos->find('-');
	if(it==pos->end())
		return false;

	it			=pos->find('+');
	if(it==pos->end())
		return false;

	it			=pos->find('*');
	if(it==pos->end())
		return false;

	pos			=(*it).second;
	if(pos->m_pRule==NULL)
		return false;
	
	if(!mapTest.fCheckValid())
		return false;

	return true;
	}
#endif

By viewing downloads associated with this article you agree to the Terms of use and the article's licence.

If a file you wish to view isn't highlighted, and is a text file (not binary), please let us know and we'll add colourisation support for it.

License

This article has no explicit license attached to it but may contain usage terms in the article text or the download files themselves. If in doubt please contact the author via the discussion board below.

A list of licenses authors might use can be found here

About the Author

 Alexander Berthold
Web Developer
Germany Germany
Member
No Biography provided

About Article
A library allowing you to conveniently build a custom tokenizer and analyzer supporting precedence priorized rules
Type Article
Licence 
First Posted 13 Jun 2001
Views 136,284
Downloads 2,281
Bookmarked 54 times
Top News

Old Japanese Man Creates Amazing Art Using Excel (Wait, Excel?)

Get the Insider News free each morning.
Related Videos
C++ Programming - 0602 Order Of Precedence
Supporting Multiple Screen Sizes
Fast tokenizer
Inside the Mathematical Expressions Evaluator
Irony - .NET Compiler Construction Kit
JavaScript Mathematical Expression Evaluator
Evaluation Engine
An Introduction to JavaCC
Expression evaluator : using RPN
Simple CSS Parser
Token Manager
a Tiny Parser Generator v1.2
The Whole Shebang: Building Your Own General Purpose Language Interpreter (Who Needs A Reason?!)
Implementing Programming Languages Using C# 4.0
A Math Expression Evaluator
How to write a simple interpreter in JavaScript
Math Parser .NET
Crafting an interpreter Part 1 - Parsing and Grammars
State of the Art Expression Evaluation
MinosseCC: a lexer/parser generator for C#
Token Replacement in Office Documents
How to use the Token Manager plugin for Wordpress
Permalink | Advertise | Privacy | Mobile
Web04 | 2.6.130523.1 | Last Updated 2 Jan 2002
Article Copyright 2001 by Alexander Berthold
Everything else Copyright © CodeProject, 1999-2013
Terms of Use
Layout: fixed | fluid