Click here to Skip to main content
Click here to Skip to main content
Go to top

The Token Iterator

, 3 May 2000
Rate this:
Please Sign up or sign in to vote.
Token Iterator provides an easy to use, familiar, and customizable way in which to go through the tokens contained in a string
  • Download source files - 6 Kb
  • The purpose of Token Iterator is to provide an easy to use, familiar, and customizable way in which to go through the tokens contained in a string. Our hypothetical example will be breaking a string into words, and outputting the words as a list. Here is the code.

    #include <iostream>
    #include <iterator>
    #include <string>
    #include <algorithm>
    #include "tokenizer.h"
    
    
    int main(){
    	
    	using namespace std;
    	using namespace jrb_stl_extensions;
    	
    	// Separate a string into words
    	string s;
    	cout << "Please enter a string with punctuation\n";
    	getline(cin,s);
    	TokenIterator<string> begin(s), end;
    	copy(begin,end,ostream_iterator<string>(cout,"\n"));
    
    }
    

    A few words of explanation. All this class and the supporting classes are packaged in jrb_stl_extensions namespace.

    Easy to use. There are only 2 lines that do the actual work.

    Let's analyze this a bit more. TokenIterator has a default template parameter that specifies the TokenizerFunc which is an STL style functor. The default is the PunctSpaceTokenizer which will tokenize separating based on whitespace and punctuation. The constructor has this signature

    PunctSpaceTokenizer(bool returnPunct = false, StringType p = WT_Punctuation1, 
                        StringType w = WT_Whitespace)

    returnPunct - Tells whether we want to return the punctuation. Returning the punctuation can be important say when building a mathematical expression parser, where while we want to skip whitespace, we do not want to skip the +'s and -'s.

    p - Punction. There are two constants called WT_Punctuation1, which is all the punctuation on a standard American keyboard. WT_Punctuation2, is the same as WT_Punctuation1, except that it does not have -(hyphen/dash) or '(apostrophe/single quote). The reason for this is that some words are hyphenated or have apostrophes (like can't) and we want to keep them as one token.

    w - Whitespace. The difference between whitespace and punctuation, is that punctuation can be returned using he returnPunct flag. WT_Whitespace is a constant that has the standard whitespace.

    Now, lets look at the familiar part. TokenIterator is an STL forward iterator and can be used with any STL algorithm that can accept it such as copy. In addition, copying the TokenIterator will NOT result in the whole string being copied. Since the string is NEVER modified, a reference counted pointer is shared among all TokenIterator's referring to a particular string.

    On to customizability. The PunctSpaceTokenizer might not suffice for all your needs. Not to worry, TokenIterator is easily customizable for the TokenizerFunc.

    Here are the requirements for TokenizerFunc.

    1. Typedefs - TokenType
      This will refer to the type of the token. For our examples this is string. This will be the return type of operator*(). An example where more than a string token would be needed might be a parser that returns an object that contains the string, and other identifying information.
    2. operator()(...)
      This has the following prototype
      iter operator()(iter* pTokEnd,iter end,TokenType& tToken)
      Return value - This returns the start of the next token in the string.
      pTokEnd - This should be set to the STL style end position (ie past the end) of the token in the string
      end - This is the end possition of the string, and is passed into the functor
      tToken - This should be set to current token. TokenType is string is will be [retval,pTokEnd)

    If you want an example, study the CSVTokenizer functor.

    We will examine using CSVTokenizer. CSVTokenizer breaks a string into C(ie comma)-separated fields. The comma is a template parameter, and can be any character. Assuming that a comma is the parameter, The string will be broken into fields separated by commas, unless the commas are inside quotes. In addition, the constructor takes a character that is defaulted to \ ('\\' in C syntax). That character acts like the same character in C, namely an escape character. For example, \" means a literal " and \\ means a literal \

    Perhaps an example will help:

    John \"Big John\" Doe,"1111 Anytown, USA 12345" will be broken into
    John "Big John" Doe
    1111 Anytown, USA 12345
    

    An example of using it follows. Add the following lines below our previous sample.

    // Here is some code that will break up a entries formatted like this
    // "field1","field3","field5,5"
    // it uses c-like escape codes for quotes namely \" for " and \\ for \ 
    // a comma is the field separator unless it is embedded inside quotes
    cout << "Please enter a comma separated line of fields\n";
    getline(cin,s);
    TokenIterator<string,CSVTokenizer<string> > begin2(s),end2;
    copy(begin2,end2,ostream_iterator<string>(cout,"\n"));
    

    This will output the fields.

    Well, there is my overview of TokenIterator. I hope you enjoy using this class.

    Note: When the sample is compiled with MSVC 6, the warnings that result are one talking about not having a return in main, and that the template resulted in an identifier that was truncated to 255 characters in debug.

    John Bandela
    Copyright 2000 John R. Bandela

    License

    This article has no explicit license attached to it but may contain usage terms in the article text or the download files themselves. If in doubt please contact the author via the discussion board below.

    A list of licenses authors might use can be found here

    Share

    About the Author

    John R. Bandela

    United States United States
    No Biography provided

    Comments and Discussions

     
    Ranturgent!!! Pinmemberanna cepe21-Aug-09 4:21 
    GeneralDouble quotes PinsussAlexander_Vi20-Nov-03 5:04 
    How can I parse strings with double quotes inside ?
    For example
    ABC, "name ""XYZ""", 123
    GeneralDouble quotes PinsussAlexander_Vi20-Nov-03 5:04 
    GeneralBoost PinsussAnonymous1-Nov-03 18:16 
    GeneralRe: Boost PinmemberJohn R. Bandela3-Nov-03 17:09 
    GeneralLink Error PinmemberBernhard22-Jun-03 23:37 
    GeneralRe: Link Error PinmemberBernhard23-Jun-03 4:43 
    QuestionCan not compile with VS2003 Final Beta PinmemberJochen Kalmbach22-Nov-02 3:30 
    AnswerRe: Can not compile with VS2003 Final Beta Pinsusskovey1-Jun-04 16:26 
    QuestionHow to return only alphanumerics ? PinmemberAnonymous10-May-01 5:35 
    Questionwstring? PinmemberEd Bianchi25-Mar-01 6:57 
    AnswerRe: wstring? PinmemberJohn Bandela26-Mar-01 18:16 
    GeneralRe: wstring? Pinmemberluthe13-Nov-03 16:09 
    GeneralCorrection to code PinsussJohn R. Bandela8-May-00 16:39 
    GeneralRe: Correction to code PinsussWilka8-May-00 20:28 
    Generalproblem... PinsussWilka8-May-00 11:24 
    GeneralRe: problem... PinsussJohn R. Bandela8-May-00 16:33 
    GeneralFix for problem - See my post Correction to Code PinsussJohn R. Bandela8-May-00 16:41 
    GeneralString Tokenizer PinsussDave lorde8-May-00 0:20 
    GeneralRe: String Tokenizer PinsussAJ8-May-00 4:38 
    GeneralRe: String Tokenizer PinsussAnonymous27-Feb-05 22:57 
    GeneralRe: String Tokenizer PinsussJohn R. Bandela8-May-00 5:40 
    GeneralRe: String Tokenizer PinsussDave Lorde8-May-00 5:50 
    GeneralRe: String Tokenizer PinmemberPhil Nash25-Mar-01 20:09 
    GeneralRe: String Tokenizer PinsussDave Lorde8-May-00 23:36 
    GeneralRe: String Tokenizer PinmemberAnonymous10-May-01 5:11 

    General General    News News    Suggestion Suggestion    Question Question    Bug Bug    Answer Answer    Joke Joke    Rant Rant    Admin Admin   

    Use Ctrl+Left/Right to switch messages, Ctrl+Up/Down to switch threads, Ctrl+Shift+Left/Right to switch pages.

    | Advertise | Privacy | Mobile
    Web03 | 2.8.140916.1 | Last Updated 4 May 2000
    Article Copyright 2000 by John R. Bandela
    Everything else Copyright © CodeProject, 1999-2014
    Terms of Service
    Layout: fixed | fluid