Click here to Skip to main content
11,643,493 members (73,162 online)
Click here to Skip to main content

CTokenizer class

, 21 Oct 2001 CPOL 136.9K 1.3K 38
Rate this:
Please Sign up or sign in to vote.
A simple tokenizer class that can be used on CStrings


Sometimes you need to use strtok in MFC projects, but it has two big limitations:

  • Can't use it to tokenize more than one string at the time.
  • It's not easy to use safely with CStrings

To overcome these limitations I wrote CTokenizer.


It's quite easy to use as the following code shows:

CTokenizer tok(_T("A-B+C*D-E"), _T("-+"));
CString cs;

    TRACE2("Token: '%s', Tail: '%s'\n", (LPCTSTR)cs, (LPCTSTR)tok.Tail());

The preceding code produces the following output:

Token: 'A', Tail: 'B+C*D-E'
Token: 'B', Tail: 'C*D-E'
Token: 'C*D', Tail: 'E'
Token: 'E', Tail: ''

As you can see, it's no rocket science! It's a very simple and handy class

To summarize:

  • Create an object of type CTokenizer, passing as arguments the string to tokenize and the delimiters
  • Call Next() until it returns false
  • If you need to change the delimiters, call SetDelimiters()
  • If you need the un-tokenized part of the string, call Tail()


  • 10/22/2001 Using std::bitset instead of std::vector. Reduces memory requirements and increases performance.
  • 10/22/2001 Fixed bug related to ASCII characters above 127. Reported by John Simpsons


This article, along with any associated source code and files, is licensed under The Code Project Open License (CPOL)


About the Author

Eddie Velasquez
Software Developer (Senior)
United States United States
No Biography provided

You may also be interested in...

Comments and Discussions

GeneralRuntime error wiwhen compiled with visual 2005 Pin
zzdomi26-Feb-09 2:08
memberzzdomi26-Feb-09 2:08 
Question"more than one string at the time"? Pin
l_d_allan28-Feb-06 5:24
memberl_d_allan28-Feb-06 5:24 
AnswerRe: "more than one string at the time"? Pin
DViggiano10-Jul-07 5:51
memberDViggiano10-Jul-07 5:51 
GeneralPls provide test program Pin
l_d_allan28-Feb-06 5:17
memberl_d_allan28-Feb-06 5:17 
GeneralMemleakage Pin
spinoza11-May-05 4:51
memberspinoza11-May-05 4:51 
GeneralRe: Memleakage Pin
Eddie Velasquez11-May-05 5:21
memberEddie Velasquez11-May-05 5:21 
GeneralNull values Pin
hitnrun16-Dec-03 4:59
memberhitnrun16-Dec-03 4:59 
GeneralRe: Null values Pin
Eddie Velasquez16-Dec-03 5:30
memberEddie Velasquez16-Dec-03 5:30 
GeneralRe: Null values Pin
hitnrun16-Dec-03 6:02
memberhitnrun16-Dec-03 6:02 
GeneralRe: Null values Pin
Eddie Velasquez16-Dec-03 7:49
memberEddie Velasquez16-Dec-03 7:49 
GeneralRe: Null values Pin
hitnrun17-Dec-03 4:42
memberhitnrun17-Dec-03 4:42 
GeneralRe: Null values Pin
Eddie Velasquez17-Dec-03 4:53
memberEddie Velasquez17-Dec-03 4:53 
GeneralThis won't work for UNICODE! Pin
Mustafa Demirhan21-Apr-03 21:29
memberMustafa Demirhan21-Apr-03 21:29 
GeneralRe: This won't work for UNICODE! Pin
Eddie Velasquez22-Apr-03 7:19
memberEddie Velasquez22-Apr-03 7:19 
GeneralBug! Pin
Anonymous23-Sep-01 23:51
memberAnonymous23-Sep-01 23:51 
GeneralRe: Bug! Pin
Anonymous25-Sep-01 2:49
memberAnonymous25-Sep-01 2:49 
GeneralRe: Bug! - Solution ? Pin
John Simpsons22-Oct-01 2:24
memberJohn Simpsons22-Oct-01 2:24 
GeneralRe: Bug! Pin
Eddie Velasquez22-Oct-01 3:28
memberEddie Velasquez22-Oct-01 3:28 
GeneralRe: Bug! Pin
John Simpsons22-Oct-01 4:00
memberJohn Simpsons22-Oct-01 4:00 
GeneralRe: Bug! Pin
Eddie Velasquez22-Oct-01 4:11
memberEddie Velasquez22-Oct-01 4:11 
GeneralRe: Bug! Pin
Eddie Velasquez22-Oct-01 5:54
memberEddie Velasquez22-Oct-01 5:54 
GeneralRe: Bug! Pin
Eddie Velasquez22-Oct-01 6:36
memberEddie Velasquez22-Oct-01 6:36 
GeneralRe: Bug! Pin
dead_alive8-Dec-03 4:51
memberdead_alive8-Dec-03 4:51 
GeneralThanks! Pin
Daniel Turini20-Sep-01 3:24
memberDaniel Turini20-Sep-01 3:24 
GeneralCan't download Pin
Bruce Hearder25-Aug-01 4:08
memberBruce Hearder25-Aug-01 4:08 
GeneralTail didn't work with '\' Pin
Trey24-Jul-00 5:13
sussTrey24-Jul-00 5:13 
GeneralRe: Tail didn't work with '\' Pin
Eddie Velasquez24-Jul-00 11:50
sussEddie Velasquez24-Jul-00 11:50 
GeneralRe: Tail didn't work with '\' Pin
Trey25-Jul-00 6:39
sussTrey25-Jul-00 6:39 
GeneralRe: Tail didn't work with '\' Pin
John Simmons / outlaw programmer15-Oct-01 3:56
memberJohn Simmons / outlaw programmer15-Oct-01 3:56 
GeneralIncluding vector Pin
Eddie Velasquez13-Jan-00 3:06
sussEddie Velasquez13-Jan-00 3:06 
GeneralRe: Including vector Pin
Eddie Velasquez13-Jan-00 3:13
sussEddie Velasquez13-Jan-00 3:13 
GeneralNot really the comment script (was: Including vector) Pin
Uwe Keim13-Jan-00 9:54
sussUwe Keim13-Jan-00 9:54 

General General    News News    Suggestion Suggestion    Question Question    Bug Bug    Answer Answer    Joke Joke    Rant Rant    Admin Admin   

Use Ctrl+Left/Right to switch messages, Ctrl+Up/Down to switch threads, Ctrl+Shift+Left/Right to switch pages.

| Advertise | Privacy | Terms of Use | Mobile
Web01 | 2.8.150731.1 | Last Updated 22 Oct 2001
Article Copyright 1999 by Eddie Velasquez
Everything else Copyright © CodeProject, 1999-2015
Layout: fixed | fluid