Click here to Skip to main content
11,578,241 members (65,022 online)
Click here to Skip to main content

String Tokenizer Iterator Class

, 26 Jun 2002 Public Domain 96.1K 22
Rate this:
Please Sign up or sign in to vote.
A string tokenizer iterator class that works with std::string
<!-- Add the rest of your HTML here -->


As a part of a larger project I had to write some basic string utility functions and classes. One of the things needed was a flexible way of splitting strings into separate tokens.

As is often the case when it comes to programming, there are different ways to handle a problem like this. After reviewing my options I decided that an iterator based solution would be flexible enough for my needs.

Non-iterator based solutions to this particular problem often have the disadvantage of tying the user to a certain container type. With an iterator based tokenizer the programmer is free to chose any type of container (or no container at all). Many STL containers such as std::list and std::vector offer constructors that can populate the container from a set of iterators. This feature makes it very easy to use the tokenizer.

Example usage

std::vector<std::string> s(string_token_iterator("one two three"),
// output:
// one
// two
// three

// same output as above

The code has been tested with Visual C++.NET and GCC 3.

The Code

#include <string>
#include <iterator>

struct string_token_iterator 
  : public std::iterator<std::input_iterator_tag, std::string>
  string_token_iterator() : str(0), start(0), end(0) {}
  string_token_iterator(const std::string & str_, const char * separator_ = " ") :
  string_token_iterator(const string_token_iterator & rhs) :

  string_token_iterator & operator++()
    return *this;

  string_token_iterator operator++(int)
    string_token_iterator temp(*this);
    return temp;

  std::string operator*() const
    return std::string(*str, start, end - start);

  bool operator==(const string_token_iterator & rhs) const
    return (rhs.str == str && rhs.start == start && rhs.end == end);

  bool operator!=(const string_token_iterator & rhs) const
    return !(rhs == *this);


  void find_next(void)
    start = str->find_first_not_of(separator, end);
    if(start == std::string::npos)
      start = end = 0;
      str = 0;

    end = str->find_first_of(separator, start);

  const char * separator;
  const std::string * str;
  std::string::size_type start;
  std::string::size_type end;


This article, along with any associated source code and files, is licensed under A Public Domain dedication


About the Author

Daniel Andersson
Web Developer
Sweden Sweden
No Biography provided

You may also be interested in...

Comments and Discussions

GeneralRe: A little trouble with the basics? Pin
Daniel Andersson5-Jul-04 21:32
memberDaniel Andersson5-Jul-04 21:32 
QuestionTwo bugs - or features? Pin
RealSkydiver31-Mar-04 23:18
memberRealSkydiver31-Mar-04 23:18 
AnswerRe: Two bugs - or features? Pin
Daniel Andersson31-Mar-04 23:56
memberDaniel Andersson31-Mar-04 23:56 
GeneralRe: Two bugs - or features? Pin
RealSkydiver1-Apr-04 21:15
memberRealSkydiver1-Apr-04 21:15 


>The last parameter in the std::string constructor call in operator* is the size of the
>resulting string. You are correct in that if end is npos it will take on a "weird" value.
>However, the effective length of the string that is created is the smallest
>of (str.size()->offset) and the length parameter.

A few more words on this issue: the behavior of the std::string constructor ( in this case 'basic_string(const_iterator first, const_iterator last, const A& al = A());' ) seams to vary for different STL's. While it works under VC6.0/7.x and gcc, it coredumps under Solaris/CC.

GeneralRe: Two bugs - or features? Pin
Daniel Andersson1-Apr-04 21:23
memberDaniel Andersson1-Apr-04 21:23 
GeneralRe: Two bugs - or features? Pin
tidi23-Sep-04 23:02
membertidi23-Sep-04 23:02 
GeneralRe: Two bugs - or features? Pin
tidi23-Sep-04 23:50
membertidi23-Sep-04 23:50 
GeneralNice Pin
Giles23-Sep-03 6:28
memberGiles23-Sep-03 6:28 
GeneralRe: Nice Pin
mouratos14-Oct-06 14:30
membermouratos14-Oct-06 14:30 
GeneralProblems with vector Pin
bss25-Feb-03 22:33
sussbss25-Feb-03 22:33 

General General    News News    Suggestion Suggestion    Question Question    Bug Bug    Answer Answer    Joke Joke    Rant Rant    Admin Admin   

Use Ctrl+Left/Right to switch messages, Ctrl+Up/Down to switch threads, Ctrl+Shift+Left/Right to switch pages.

| Advertise | Privacy | Terms of Use | Mobile
Web04 | 2.8.150603.1 | Last Updated 27 Jun 2002
Article Copyright 2002 by Daniel Andersson
Everything else Copyright © CodeProject, 1999-2015
Layout: fixed | fluid