Click here to Skip to main content
13,353,462 members (65,564 online)
Click here to Skip to main content
Add your own
alternative version


65 bookmarked
Posted 7 Jul 2004


, 7 Jul 2004
Rate this:
Please Sign up or sign in to vote.
StringTokenizer class that can be used for breaking up a string (or stream) into smaller strings.


This is yet another implementation of string tokenizer. This tokenizer allows you to break strings to tokens. The following tokens are recognized:

  • WORD - series of alpha characters or _
  • NUMBER - decimal number
  • QUOTEDSTRING - string that starts with " and ends with " and uses "" as escape character
  • WHITESPACE - space or tab
  • EOL - end of line. Recognized Windows \r\n, Unix \n, or Mac \r
  • SYMBOL - any symbol character (customizable)

Each token contains line #, column #, kind, and string data.

Here is a small example of how it works:

string input = "hello \"cool programmer\", your number: 3.45!";

StringTokenizer tok = new StringTokenizer(input);
tok.IgnoreWhiteSpace = true;
tok.SymbolChars      = new char[]{',', ':'};

Token token;
    token = tok.Next();
    Console.WriteLine(token.Kind.ToString() + ": " + token.Value);
} while (token.Kind != TokenKind.EOF);

and the output will be:

Word: hello
QuotedString: "cool programmer"
Symbol: ,
Word: your
Word: number
Symbol: :
Number: 3.45
Unknown: !

Note that ! is returned as Unknown, because it wasn't defined as symbol. You can specify which characters are symbols by: tok.SymbolChars. You can also specify whether whitespace is ignored.

All the source code is included so you can customize and modify the tokenizer. This little section will explain on how to extend the tokenizer, so that you can parse your own custom tokens.

Suppose you want to read $string as special token called Variable, where Variable is $ followed by variable name: (ALPHANUMERIC | _)*. What you would do is add new value to TokenKind enum (in Token.cs):

enum TokenKind

and in StringTokenizer inside of Next method, add new case right before default:

switch (ch)
    case '$':
        return ReadVariable();


then, you just need to write ReadVariable method.

protected Token ReadWord()
    StartRead();    // this marks the position of the beginning of the token

    // consume first character which is $. If you don't want $ to be returned
    // as part of Value of the token, just calls StartRead() after Consume

    while (true)
        char ch = LA(0);    // look at next available character
        // if it's letter or underscore, we just
        // consume it and continue reading
        if (Char.IsLetter(ch) || ch == '_')
        else        // if not we break the loop

    // CreateToken creates the token with line and
    // column positions and the value of the token
    // is going to be from the string that was started
    // when StartRead was called, until current position
    return CreateToken(TokenKind.Variable);

That's it!

Of course, this tokenizer is very simple right out of the box and easy to modify, but for more complicated parsing, it's much better to use lexer/parser generator tools like antlr ( This tokenizer is really easy to use and does not have big overhead.


This article has no explicit license attached to it but may contain usage terms in the article text or the download files themselves. If in doubt please contact the author via the discussion board below.

A list of licenses authors might use can be found here


About the Author

Chief Technology Officer
United States United States
No Biography provided

You may also be interested in...

Comments and Discussions

GeneralDifferent quote character for strings... Pin
Nathan Baulch5-Jul-06 22:57
memberNathan Baulch5-Jul-06 22:57 
Generalnice... Pin
falfan26-Jun-06 13:37
memberfalfan26-Jun-06 13:37 
GeneralRe: nice... Pin
Werdna26-Jun-06 14:00
memberWerdna26-Jun-06 14:00 
Generalcannot strip whitespaces Pin
Tim Julian15-Jun-06 6:41
memberTim Julian15-Jun-06 6:41 
GeneralRe: cannot strip whitespaces Pin
Ravi Bhavnani15-Jun-06 7:50
memberRavi Bhavnani15-Jun-06 7:50 
GeneralRe: Alternative Pin
Werdna28-Sep-05 7:49
memberWerdna28-Sep-05 7:49 
GeneralRe: Alternative Pin
dave.dolan1-Oct-06 17:34
memberdave.dolan1-Oct-06 17:34 
GeneralQuestion about string tokenizer Pin
James Brannan9-Aug-05 23:39
memberJames Brannan9-Aug-05 23:39 
GeneralRe: Question about string tokenizer Pin
Werdna10-Aug-05 4:31
memberWerdna10-Aug-05 4:31 
GeneralRe: Question about string tokenizer Pin
James Brannan10-Aug-05 8:50
memberJames Brannan10-Aug-05 8:50 
GeneralGreat Pin
reinux16-Jul-04 17:11
memberreinux16-Jul-04 17:11 
GeneralOoh! Pin
dzCepheus14-Jul-04 17:57
memberdzCepheus14-Jul-04 17:57 

General General    News News    Suggestion Suggestion    Question Question    Bug Bug    Answer Answer    Joke Joke    Praise Praise    Rant Rant    Admin Admin   

Use Ctrl+Left/Right to switch messages, Ctrl+Up/Down to switch threads, Ctrl+Shift+Left/Right to switch pages.

Permalink | Advertise | Privacy | Terms of Use | Mobile
Web01 | 2.8.180111.1 | Last Updated 8 Jul 2004
Article Copyright 2004 by Werdna
Everything else Copyright © CodeProject, 1999-2018
Layout: fixed | fluid