Click here to Skip to main content
Click here to Skip to main content

StringTokenizer

, 7 Jul 2004
Rate this:
Please Sign up or sign in to vote.
StringTokenizer class that can be used for breaking up a string (or stream) into smaller strings.

Introduction

This is yet another implementation of string tokenizer. This tokenizer allows you to break strings to tokens. The following tokens are recognized:

  • WORD - series of alpha characters or _
  • NUMBER - decimal number
  • QUOTEDSTRING - string that starts with " and ends with " and uses "" as escape character
  • WHITESPACE - space or tab
  • EOL - end of line. Recognized Windows \r\n, Unix \n, or Mac \r
  • SYMBOL - any symbol character (customizable)

Each token contains line #, column #, kind, and string data.

Here is a small example of how it works:

string input = "hello \"cool programmer\", your number: 3.45!";

StringTokenizer tok = new StringTokenizer(input);
tok.IgnoreWhiteSpace = true;
tok.SymbolChars      = new char[]{',', ':'};

Token token;
do
{
    token = tok.Next();
    Console.WriteLine(token.Kind.ToString() + ": " + token.Value);
        
} while (token.Kind != TokenKind.EOF);

and the output will be:

Word: hello
QuotedString: "cool programmer"
Symbol: ,
Word: your
Word: number
Symbol: :
Number: 3.45
Unknown: !

Note that ! is returned as Unknown, because it wasn't defined as symbol. You can specify which characters are symbols by: tok.SymbolChars. You can also specify whether whitespace is ignored.

All the source code is included so you can customize and modify the tokenizer. This little section will explain on how to extend the tokenizer, so that you can parse your own custom tokens.

Suppose you want to read $string as special token called Variable, where Variable is $ followed by variable name: (ALPHANUMERIC | _)*. What you would do is add new value to TokenKind enum (in Token.cs):

enum TokenKind
{
...,
Variable
}

and in StringTokenizer inside of Next method, add new case right before default:

switch (ch)
{
    ...
    case '$':
    {
        return ReadVariable();
    }

    default:
    ...
}

then, you just need to write ReadVariable method.

protected Token ReadWord()
{
    StartRead();    // this marks the position of the beginning of the token

    Consume(); 
    // consume first character which is $. If you don't want $ to be returned
    // as part of Value of the token, just calls StartRead() after Consume

    while (true)
    {
        char ch = LA(0);    // look at next available character
        // if it's letter or underscore, we just
        // consume it and continue reading
        if (Char.IsLetter(ch) || ch == '_')
            Consume();
        else        // if not we break the loop
            break;
    }

    // CreateToken creates the token with line and
    // column positions and the value of the token
    // is going to be from the string that was started
    // when StartRead was called, until current position
    return CreateToken(TokenKind.Variable);
}

That's it!

Of course, this tokenizer is very simple right out of the box and easy to modify, but for more complicated parsing, it's much better to use lexer/parser generator tools like antlr (http://www.antlr.org/). This tokenizer is really easy to use and does not have big overhead.

License

This article has no explicit license attached to it but may contain usage terms in the article text or the download files themselves. If in doubt please contact the author via the discussion board below.

A list of licenses authors might use can be found here

Share

About the Author

Werdna
Chief Technology Officer
United States United States
No Biography provided

Comments and Discussions

 
QuestionKeeps Crashing with VS2010 PinmemberBiedermannS5-Nov-12 0:00 
QuestionWhy why why? Pinmembersteveski7425-Feb-10 12:42 
AnswerRe: Why why why? [modified] PinmemberJonathan Wood8-May-11 8:50 
GeneralThat was useful Pinmember Muammar© 13-Apr-08 4:51 
GeneralThank you! Thank you! Thank you! Pinmember982731982739817239871928371298376-Dec-06 7:27 
GeneralDifferent quote character for strings... PinmemberNathan Baulch5-Jul-06 21:57 
Generalnice... Pinmemberfalfan26-Jun-06 12:37 
GeneralRe: nice... PinmemberWerdna26-Jun-06 13:00 
Generalcannot strip whitespaces PinmemberTim Julian15-Jun-06 5:41 
GeneralRe: cannot strip whitespaces PinmemberRavi Bhavnani15-Jun-06 6:50 
Odd. Inspection of StringTokenizer.cs in the src package reveals the public bool property IgnoreWhiteSpace.
 
/ravi
 
My new year's resolution: 2048 x 1536
Home | Music | Articles | Freeware | Trips
ravib(at)ravib(dot)com
GeneralRe: Alternative PinmemberWerdna28-Sep-05 6:49 
GeneralRe: Alternative Pinmemberdave.dolan1-Oct-06 16:34 
GeneralQuestion about string tokenizer PinmemberJames Brannan9-Aug-05 22:39 
GeneralRe: Question about string tokenizer PinmemberWerdna10-Aug-05 3:31 
GeneralRe: Question about string tokenizer PinmemberJames Brannan10-Aug-05 7:50 
GeneralGreat Pinmemberreinux16-Jul-04 16:11 
GeneralOoh! PinmemberdzCepheus14-Jul-04 16:57 

General General    News News    Suggestion Suggestion    Question Question    Bug Bug    Answer Answer    Joke Joke    Rant Rant    Admin Admin   

Use Ctrl+Left/Right to switch messages, Ctrl+Up/Down to switch threads, Ctrl+Shift+Left/Right to switch pages.

| Advertise | Privacy | Mobile
Web02 | 2.8.141022.2 | Last Updated 8 Jul 2004
Article Copyright 2004 by Werdna
Everything else Copyright © CodeProject, 1999-2014
Terms of Service
Layout: fixed | fluid