Click here to Skip to main content
13,152,454 members (37,695 online)
Click here to Skip to main content


43 bookmarked
Posted 18 Apr 2001

Fast tokenizer

, 5 Sep 2001
Fast tokenizer for C++ - like 'lexx'
// cooLexxer.h: interface for the cooLexxer class.

Class:      cooLexxer
Author:     Alexander Berthold
Copyright:  Alexander Berthold
Date:       19.4.2001
Purpose:    - Main class for the lexical analyzer.
            - Scans the input character stream for tokens.
            - Calls a call-back class ('cooLexxerListener') for each
              recognized token.
            - Uses an incremental scan; should be pretty fast.
            - Can also parse computed tokens, like "text" and 1234.

    You may freely use this code in your projects. If you find
    any bugs or if you make other corrections/enhancements, i'd
    appreciate if you'd let me know about that. My email is

    If you share this code, please do not remove this text.

#if !defined(AFX_COOLEXXER_H__D79EE6C7_40E0_4452_A7A4_20278169F438__INCLUDED_)
#define AFX_COOLEXXER_H__D79EE6C7_40E0_4452_A7A4_20278169F438__INCLUDED_

#if _MSC_VER > 1000
#pragma once
#endif // _MSC_VER > 1000

class cooLexxerInputStream;

class cooLexxerListener :
	public ctkCheckValid,
	public ctkExternalObjectPointer
// Operations
	virtual	void			vRegisterToken(const std::tstring& strTokenText, const cooLexxerTokenRule* pltrRule) = 0;

class cooLexxer
// Construction/Destruction
	cooLexxer(cooLexxerInputStream* plisData,
			  cooLexxerMap* plmLexxerMap,
			  cooLexxerListener* pllReceiver);
	virtual ~cooLexxer();

// Attributes
	// Contains the state of the lexxer
	cooLexxerContext		m_lcContextInfo;
	// The 'tree' containing the tokens
	cooLexxerMap			*m_plmLexxerMap;
	// The input stream
	cooLexxerInputStream	*m_plisData;
	// The listener to the token output
	cooLexxerListener		*m_pllReceiver;

// ctkCheckValid operations
#ifdef _DEBUG
	virtual bool			fCheckValid() const;
	static bool				fRunDiagnostics();
	virtual bool			fCheckValid() const { return true; };

// Protected operations
	/*** Internal token recognition methods ***/

	// A delimeting token has been recognized
	void					vDelimTokenRecognized();
	// A non-delimeting token has been recognized
	void					vRecognizeNonDelimTokens(std::tstring strTokenText);

	/*** Post token recognition methods ***/

	// Is called by the lexxer after a token has been recognized
	void					vRegisterToken(const std::tstring& strTokenText, cooLexxerTokenRule* pptrRule);

	/*** Helper functions ***/

	// Find the longest currently active (completed) rule

	// Find the longest applied rule, not yet invalidated rule

	// Mark all active rules except the given one for deletion
	void					vMarkAllForDeletionExcept(const cooLexxerContext::cooListEntry* pEntry);

// Operations
	// Parse the next character, 'fOverrideIsLastChar' tells if the next
	// char is to be treated as it was the last of the stream.
	void					vParseCharacter(bool fOverrideIsLastChar = false);

#endif // !defined(AFX_COOLEXXER_H__D79EE6C7_40E0_4452_A7A4_20278169F438__INCLUDED_)

By viewing downloads associated with this article you agree to the Terms of Service and the article's licence.

If a file you wish to view isn't highlighted, and is a text file (not binary), please let us know and we'll add colourisation support for it.


This article has no explicit license attached to it but may contain usage terms in the article text or the download files themselves. If in doubt please contact the author via the discussion board below.

A list of licenses authors might use can be found here


About the Author

Alexander Berthold
Web Developer
Germany Germany
No Biography provided

You may also be interested in...

Permalink | Advertise | Privacy | Terms of Use | Mobile
Web03 | 2.8.170924.2 | Last Updated 6 Sep 2001
Article Copyright 2001 by Alexander Berthold
Everything else Copyright © CodeProject, 1999-2017
Layout: fixed | fluid