Click here to Skip to main content
15,889,693 members
Articles / Mobile Apps / Windows Mobile
Article

CTokenizer class

Rate me:
Please Sign up or sign in to vote.
3.82/5 (10 votes)
21 Oct 2001CPOL 162K   1.4K   38   32
A simple tokenizer class that can be used on CStrings

Introduction

Sometimes you need to use strtok in MFC projects, but it has two big limitations:

  • Can't use it to tokenize more than one string at the time.
  • It's not easy to use safely with CStrings

To overcome these limitations I wrote CTokenizer.

Usage

It's quite easy to use as the following code shows:

CTokenizer tok(_T("A-B+C*D-E"), _T("-+"));
CString cs;

while(tok.Next(cs))
    TRACE2("Token: '%s', Tail: '%s'\n", (LPCTSTR)cs, (LPCTSTR)tok.Tail());

The preceding code produces the following output:

Token: 'A', Tail: 'B+C*D-E'
Token: 'B', Tail: 'C*D-E'
Token: 'C*D', Tail: 'E'
Token: 'E', Tail: ''

As you can see, it's no rocket science! It's a very simple and handy class

To summarize:

  • Create an object of type CTokenizer, passing as arguments the string to tokenize and the delimiters
  • Call Next() until it returns false
  • If you need to change the delimiters, call SetDelimiters()
  • If you need the un-tokenized part of the string, call Tail()

Updates

  • 10/22/2001 Using std::bitset instead of std::vector. Reduces memory requirements and increases performance.
  • 10/22/2001 Fixed bug related to ASCII characters above 127. Reported by John Simpsons

License

This article, along with any associated source code and files, is licensed under The Code Project Open License (CPOL)


Written By
Software Developer (Senior)
United States United States
This member has not yet provided a Biography. Assume it's interesting and varied, and probably something to do with programming.

Comments and Discussions

 
GeneralTail didn't work with '\' Pin
trey24-Jul-00 5:13
trey24-Jul-00 5:13 
GeneralRe: Tail didn't work with '\' Pin
Eddie Velasquez24-Jul-00 11:50
Eddie Velasquez24-Jul-00 11:50 
GeneralRe: Tail didn't work with '\' Pin
trey25-Jul-00 6:39
trey25-Jul-00 6:39 
GeneralRe: Tail didn't work with '\' Pin
#realJSOP15-Oct-01 3:56
mve#realJSOP15-Oct-01 3:56 
GeneralNot really the comment script (was: Including vector) Pin
Uwe Keim13-Jan-00 9:54
sitebuilderUwe Keim13-Jan-00 9:54 

General General    News News    Suggestion Suggestion    Question Question    Bug Bug    Answer Answer    Joke Joke    Praise Praise    Rant Rant    Admin Admin   

Use Ctrl+Left/Right to switch messages, Ctrl+Up/Down to switch threads, Ctrl+Shift+Left/Right to switch pages.