Click here to Skip to main content
13,295,953 members (51,045 online)
Click here to Skip to main content
Add your own
alternative version


23 bookmarked
Posted 26 Jun 2002

String Tokenizer Iterator Class

, 26 Jun 2002
Rate this:
Please Sign up or sign in to vote.
A string tokenizer iterator class that works with std::string
<!-- Add the rest of your HTML here -->


As a part of a larger project I had to write some basic string utility functions and classes. One of the things needed was a flexible way of splitting strings into separate tokens.

As is often the case when it comes to programming, there are different ways to handle a problem like this. After reviewing my options I decided that an iterator based solution would be flexible enough for my needs.

Non-iterator based solutions to this particular problem often have the disadvantage of tying the user to a certain container type. With an iterator based tokenizer the programmer is free to chose any type of container (or no container at all). Many STL containers such as std::list and std::vector offer constructors that can populate the container from a set of iterators. This feature makes it very easy to use the tokenizer.

Example usage

std::vector<std::string> s(string_token_iterator("one two three"),
// output:
// one
// two
// three

// same output as above

The code has been tested with Visual C++.NET and GCC 3.

The Code

#include <string>
#include <iterator>

struct string_token_iterator 
  : public std::iterator<std::input_iterator_tag, std::string>
  string_token_iterator() : str(0), start(0), end(0) {}
  string_token_iterator(const std::string & str_, const char * separator_ = " ") :
  string_token_iterator(const string_token_iterator & rhs) :

  string_token_iterator & operator++()
    return *this;

  string_token_iterator operator++(int)
    string_token_iterator temp(*this);
    return temp;

  std::string operator*() const
    return std::string(*str, start, end - start);

  bool operator==(const string_token_iterator & rhs) const
    return (rhs.str == str && rhs.start == start && rhs.end == end);

  bool operator!=(const string_token_iterator & rhs) const
    return !(rhs == *this);


  void find_next(void)
    start = str->find_first_not_of(separator, end);
    if(start == std::string::npos)
      start = end = 0;
      str = 0;

    end = str->find_first_of(separator, start);

  const char * separator;
  const std::string * str;
  std::string::size_type start;
  std::string::size_type end;


This article, along with any associated source code and files, is licensed under A Public Domain dedication


About the Author

Daniel Andersson
Web Developer
Sweden Sweden
No Biography provided

You may also be interested in...

Comments and Discussions

QuestionWhat license? Pin
Michael Broida5-Oct-12 5:57
memberMichael Broida5-Oct-12 5:57 
I like this code and would like to use it.
But I need to know what license (+version) applies to use of the code.
Is the author (Daniel Andersson) still around?

Thank you!
AnswerRe: What license? Pin
Daniel Andersson5-Oct-12 6:02
memberDaniel Andersson5-Oct-12 6:02 
GeneralRe: What license? Pin
Michael Broida5-Oct-12 7:11
memberMichael Broida5-Oct-12 7:11 
GeneralMy vote of 3 Pin
cccfff77730-Jun-10 5:49
membercccfff77730-Jun-10 5:49 
GeneralSubtle Bug Pin
mollevp31-Jul-06 5:59
membermollevp31-Jul-06 5:59 
AnswerRe: Subtle Bug Pin
Daniel Andersson13-Aug-06 23:37
memberDaniel Andersson13-Aug-06 23:37 
QuestionA little trouble with the basics? Pin
mtwombley4-Jul-04 18:49
membermtwombley4-Jul-04 18:49 
AnswerRe: A little trouble with the basics? Pin
Daniel Andersson4-Jul-04 22:53
memberDaniel Andersson4-Jul-04 22:53 
GeneralRe: A little trouble with the basics? Pin
mtwombley5-Jul-04 10:39
membermtwombley5-Jul-04 10:39 
GeneralRe: A little trouble with the basics? Pin
Daniel Andersson5-Jul-04 22:32
memberDaniel Andersson5-Jul-04 22:32 
QuestionTwo bugs - or features? Pin
RealSkydiver1-Apr-04 0:18
memberRealSkydiver1-Apr-04 0:18 
AnswerRe: Two bugs - or features? Pin
Daniel Andersson1-Apr-04 0:56
memberDaniel Andersson1-Apr-04 0:56 
GeneralRe: Two bugs - or features? Pin
RealSkydiver1-Apr-04 22:15
memberRealSkydiver1-Apr-04 22:15 
GeneralRe: Two bugs - or features? Pin
Daniel Andersson1-Apr-04 22:23
memberDaniel Andersson1-Apr-04 22:23 
GeneralRe: Two bugs - or features? Pin
tidi24-Sep-04 0:02
membertidi24-Sep-04 0:02 
GeneralRe: Two bugs - or features? Pin
tidi24-Sep-04 0:50
membertidi24-Sep-04 0:50 
GeneralNice Pin
Giles23-Sep-03 7:28
memberGiles23-Sep-03 7:28 
GeneralRe: Nice Pin
mouratos14-Oct-06 15:30
membermouratos14-Oct-06 15:30 
GeneralProblems with vector Pin
bss25-Feb-03 23:33
sussbss25-Feb-03 23:33 
GeneralRe: Problems with vector Pin
Daniel Andersson26-Feb-03 5:11
memberDaniel Andersson26-Feb-03 5:11 
GeneralRe: Problems with vector Pin
SaurweinAndreas15-Sep-03 19:16
memberSaurweinAndreas15-Sep-03 19:16 
Generalmake it unicode aware Pin
Anonymous27-Jun-02 19:25
memberAnonymous27-Jun-02 19:25 
GeneralRe: make it unicode aware Pin
Anonymous23-Oct-02 8:56
sussAnonymous23-Oct-02 8:56 
GeneralVery good article (and a pointer to some more stuff) Pin
Joaquín M López Muñoz27-Jun-02 10:22
memberJoaquín M López Muñoz27-Jun-02 10:22 

General General    News News    Suggestion Suggestion    Question Question    Bug Bug    Answer Answer    Joke Joke    Praise Praise    Rant Rant    Admin Admin   

Use Ctrl+Left/Right to switch messages, Ctrl+Up/Down to switch threads, Ctrl+Shift+Left/Right to switch pages.

Permalink | Advertise | Privacy | Terms of Use | Mobile
Web04 | 2.8.171207.1 | Last Updated 27 Jun 2002
Article Copyright 2002 by Daniel Andersson
Everything else Copyright © CodeProject, 1999-2017
Layout: fixed | fluid