Click here to Skip to main content
13,506,329 members
Click here to Skip to main content
Add your own
alternative version

Tagged as


20 bookmarked
Posted 16 May 2000

Tokenizing strings in VBScript

, 16 May 2000
Rate this:
Please Sign up or sign in to vote.
A simple function that allows you to tokenize a string sing multiple token separators

This is an extremely simple function for tokenizing strings. You supply the function with a string that you wish to tokenize, and an array of tokens the delimit the tokens.

For example, suppose you have the string "Tom, Dick and Harry" and you'd like to break it up into "Tom", "Dick", "Harry". Your string is thus "Tom, Dick and Harry" and your array contains the "," and "and" separators:

Dim Str, Seps(2)
Str     = "Tom, Dick and Harry"
Seps(0) = ","
Seps(1) = "and"

Dim i, a
a = Tokenize(Str, Seps)

Response.Write "<p>Found " & UBound(a) & " tokens</p>"
Response.Write "<ol>"
For i=1 to UBound(a)
	Response.Write "<li>Keyword " & i & " = " & a(i-1) & "</li>"
Response.Write "</ol>"

The results will be

Found 3 tokens
  1. Keyword 1 = Tom</tt>
  2. Keyword 2 = Dick</tt>
  3. Keyword 3 = Harry</tt>

The function is as follows:

Function Tokenize(byVal TokenString, byRef TokenSeparators())

	Dim NumWords, a()
	NumWords = 0
	Dim NumSeps
	NumSeps = UBound(TokenSeparators)
		Dim SepIndex, SepPosition
		SepPosition = 0
		SepIndex    = -1
		for i = 0 to NumSeps-1
			' Find location of separator in the string
			Dim pos
			pos = InStr(TokenString, TokenSeparators(i))
			' Is the separator present, and is it closest to the beginning of the string?
			If pos > 0 and ( (SepPosition = 0) or (pos < SepPosition) ) Then
				SepPosition = pos
				SepIndex    = i
			End If

		' Did we find any separators?	
		If SepIndex < 0 Then

			' None found - so the token is the remaining string
			redim preserve a(NumWords+1)
			a(NumWords) = TokenString

			' Found a token - pull out the substring		
			Dim substr
			substr = Trim(Left(TokenString, SepPosition-1))
			' Add the token to the list
			redim preserve a(NumWords+1)
			a(NumWords) = substr
			' Cutoff the token we just found
			Dim TrimPosition
			TrimPosition = SepPosition+Len(TokenSeparators(SepIndex))
			TokenString = Trim(Mid(TokenString, TrimPosition))
		End If	
		NumWords = NumWords + 1
	loop while (SepIndex >= 0)
	Tokenize = a
End Function


This article, along with any associated source code and files, is licensed under The Code Project Open License (CPOL)


About the Author

Chris Maunder
Founder CodeProject
Canada Canada
Chris is the Co-founder, Administrator, Architect, Chief Editor and Shameless Hack who wrote and runs The Code Project. He's been programming since 1988 while pretending to be, in various guises, an astrophysicist, mathematician, physicist, hydrologist, geomorphologist, defence intelligence researcher and then, when all that got a bit rough on the nerves, a web developer. He is a Microsoft Visual C++ MVP both globally and for Canada locally.

His programming experience includes C/C++, C#, SQL, MFC, ASP, ASP.NET, and far, far too much FORTRAN. He has worked on PocketPCs, AIX mainframes, Sun workstations, and a CRAY YMP C90 behemoth but finds notebooks take up less desk space.

He dodges, he weaves, and he never gets enough sleep. He is kind to small animals.

Chris was born and bred in Australia but splits his time between Toronto and Melbourne, depending on the weather. For relaxation he is into road cycling, snowboarding, rock climbing, and storm chasing.

You may also be interested in...

Comments and Discussions

GeneralNested token's Pin
deadtroll13-Oct-06 9:17
memberdeadtroll13-Oct-06 9:17 
GeneralNested Tokenize not working VBScript/ASP Pin
Ditta17-Jul-03 5:54
sussDitta17-Jul-03 5:54 
GeneralYet another version Pin
Slash17-May-00 3:55
sussSlash17-May-00 3:55 
GeneralRe: Yet another version Pin
Anonymous9-Mar-05 13:06
sussAnonymous9-Mar-05 13:06 
QuestionSo you extended the VBScript Split function? Pin
Uwe Keim17-May-00 1:08
sussUwe Keim17-May-00 1:08 
AnswerRe: So you extended the VBScript Split function? Pin
Chris Maunder17-May-00 12:47
sussChris Maunder17-May-00 12:47 
GeneralRe: So you extended the VBScript Split function? Pin
Klong6-Jul-01 7:55
memberKlong6-Jul-01 7:55 
GeneralRe: So you extended the VBScript Split function? Pin
Ali Issa19-Jul-01 1:51
memberAli Issa19-Jul-01 1:51 
GeneralRe: So you extended the VBScript Split function? Pin
Gareth Watson10-Jul-02 0:10
sussGareth Watson10-Jul-02 0:10 
AnswerRe: So you extended the VBScript Split function? Pin
DDarling11-Jul-02 9:26
sussDDarling11-Jul-02 9:26 
GeneralRe: So you extended the VBScript Split function? Pin
Anonymous17-Nov-03 5:21
sussAnonymous17-Nov-03 5:21 

General General    News News    Suggestion Suggestion    Question Question    Bug Bug    Answer Answer    Joke Joke    Praise Praise    Rant Rant    Admin Admin   

Use Ctrl+Left/Right to switch messages, Ctrl+Up/Down to switch threads, Ctrl+Shift+Left/Right to switch pages.

Permalink | Advertise | Privacy | Terms of Use | Mobile
Web01 | 2.8.180417.1 | Last Updated 17 May 2000
Article Copyright 2000 by Chris Maunder
Everything else Copyright © CodeProject, 1999-2018
Layout: fixed | fluid