This is an extremely simple function for tokenizing strings. You supply the function with
a string that you wish to tokenize, and an array of tokens the delimit the tokens.
For example, suppose you have the string "Tom, Dick and Harry" and you'd like to break it
up into "Tom", "Dick", "Harry". Your string is thus "Tom, Dick and Harry" and your array
contains the "," and "and" separators:
Dim Str, Seps(2)
Str = "Tom, Dick and Harry"
Seps(0) = ","
Seps(1) = "and"
Dim i, a
a = Tokenize(Str, Seps)
Response.Write "<p>Found " & UBound(a) & " tokens</p>"
Response.Write "<ol>"
For i=1 to UBound(a)
Response.Write "<li>Keyword " & i & " = " & a(i-1) & "</li>"
next
Response.Write "</ol>"
The results will be
Found 3 tokens
1. Keyword 1 = Tom</tt>
2. Keyword 2 = Dick</tt>
3. Keyword 3 = Harry</tt>
The function is as follows:
Function Tokenize(byVal TokenString, byRef TokenSeparators())
Dim NumWords, a()
NumWords = 0
Dim NumSeps
NumSeps = UBound(TokenSeparators)
Do
Dim SepIndex, SepPosition
SepPosition = 0
SepIndex = -1
for i = 0 to NumSeps-1
Dim pos
pos = InStr(TokenString, TokenSeparators(i))
If pos > 0 and ( (SepPosition = 0) or (pos < SepPosition) ) Then
SepPosition = pos
SepIndex = i
End If
Next
If SepIndex < 0 Then
redim preserve a(NumWords+1)
a(NumWords) = TokenString
Else
Dim substr
substr = Trim(Left(TokenString, SepPosition-1))
redim preserve a(NumWords+1)
a(NumWords) = substr
Dim TrimPosition
TrimPosition = SepPosition+Len(TokenSeparators(SepIndex))
TokenString = Trim(Mid(TokenString, TrimPosition))
End If
NumWords = NumWords + 1
loop while (SepIndex >= 0)
Tokenize = a
End Function
Chris Maunder is the co-founder of
CodeProject, DeveloperMedia and ContentLab, and has been a prominent figure in the software development community for nearly 30 years. Hailing from Australia, Chris has a background in Mathematics, Astrophysics, Environmental Engineering and Defence Research. His programming endeavours span everything from FORTRAN on Super Computers, C++/MFC on Windows, through to to high-load .NET web applications and Python AI applications on everything from macOS to a Raspberry Pi. Chris is a full-stack developer who is as comfortable with SQL as he is with CSS.
In the late 1990s, he and his business partner David Cunningham recognized the need for a platform that would facilitate knowledge-sharing among developers, leading to the establishment of CodeProject.com in 1999. Chris's expertise in programming and his passion for fostering a collaborative environment have played a pivotal role in the success of CodeProject.com. Over the years, the website has grown into a vibrant community where programmers worldwide can connect, exchange ideas, and find solutions to coding challenges. Chris is a prolific contributor to the developer community through his articles and tutorials, and his latest passion project,
CodeProject.AI.
In addition to his work with CodeProject.com, Chris co-founded ContentLab and DeveloperMedia, two projects focussed on helping companies make their Software Projects a success. While at CodeProject, Chris' roles included Architecture and coding, Product Development, Content Creation, Community Growth, Client Satisfaction and Systems Automation, and many, many sales meetings. All while keeping his sense of humour.