15,881,424 members
Sign in
Sign in
Email
Password
Forgot your password?
Sign in with
home
articles
Browse Topics
>
Latest Articles
Top Articles
Posting/Update Guidelines
Article Help Forum
Submit an article or tip
Import GitHub Project
Import your Blog
quick answers
Q&A
Ask a Question
View Unanswered Questions
View All Questions
View C# questions
View C++ questions
View Javascript questions
View Visual Basic questions
View Python questions
discussions
forums
CodeProject.AI Server
All Message Boards...
Application Lifecycle
>
Running a Business
Sales / Marketing
Collaboration / Beta Testing
Work Issues
Design and Architecture
Artificial Intelligence
ASP.NET
JavaScript
Internet of Things
C / C++ / MFC
>
ATL / WTL / STL
Managed C++/CLI
C#
Free Tools
Objective-C and Swift
Database
Hardware & Devices
>
System Admin
Hosting and Servers
Java
Linux Programming
Python
.NET (Core and Framework)
Android
iOS
Mobile
WPF
Visual Basic
Web Development
Site Bugs / Suggestions
Spam and Abuse Watch
features
features
Competitions
News
The Insider Newsletter
The Daily Build Newsletter
Newsletter archive
Surveys
CodeProject Stuff
community
lounge
Who's Who
Most Valuable Professionals
The Lounge
The CodeProject Blog
Where I Am: Member Photos
The Insider News
The Weird & The Wonderful
help
?
What is 'CodeProject'?
General FAQ
Ask a Question
Bugs and Suggestions
Article Help Forum
About Us
Search within:
Articles
Quick Answers
Messages
Artificial Intelligence
Artificial Intelligence
Neural Networks
Machine Learning
Deep Learning
ML.Net
Big Data
Data Science
Tensorflow
ChatGPT
Internet of Things
Internet of Things
Arduino
Edge Device
Raspberry-Pi
Wearables
IoT Edge
Industrial IoT
DevOps
DevOps
Agile
Automation
Continuous Build
Continuous Delivery
Continuous Deployment
Continuous Integration
Deployment
Git
Installation
Integration Testing
Load Testing
Quality Assurance
TFS
Unit Testing
Testing
Containers
Containers
Docker
Kubernetes
Virtual Machine
High Performance Computing
High Performance Computing
GPU
Parallel Processing
Parallelization
Vectorization
Hosted Services
Hosted Services
AWS
Cordova
ExtJS
Google Cloud
Microservices
PhoneGap
Serverless
Storage
Web Hosting
WordPress
Azure
Security
Security
Blockchain
Cryptography
Encryption
Identity
Identity Server
Web Development
Web Development
Blazor
CSS3
Flask
HTML5
Kestrel
LESS
Nginx
Node.js
React
SCSS
Spring
Vue.js
ASP.NET
CSS
Apache
HTML
IIS
XHTML
Mobile Apps
Mobile Apps
Flutter
Ionic
iOS
Nativescript
React Native
Xamarin
Windows Mobile
Android
Desktop Programming
Desktop Programming
Cocoa
Electron
QT
Universal Windows Platform
X11
Windows Forms
ATL
MFC
Swing
Win32
WTL
WPF
XAML
System
Database Development
Database Development
Data Visualization
Elasticsearch
Lucene
MariaDB
MongoDB
NoSQL
PostgreSQL
Redis
MySQL
SQL Server
Productivity Apps and Services
Productivity Apps and Services
LibreOffice
OpenOffice
Sharepoint
Team Communication Tools
Biztalk
Microsoft Office
SAP
Game Development
Game Development
Doom
Kinect
Playstation 4
Playstation 5
Unity
Unreal
XBox
Multimedia
Multimedia
Audio
Image Processing
Video
Audio Video
DirectX
GDI
GDI+
OpenGL
General Programming
General Programming
Algorithms
Compression
Computational Geometry
Emulation
Exceptions
File
Internet
Localization
Memory Management
Optimization
Parser
Performance
Regular Expressions
Sorting
State Machines
Architecture
Design / Graphics
Printing
String
Threads
Usability
Programming Languages
Programming Languages
ASM
Bash
Basic
COBOL
Dart
Go
Haskell
Kotlin
Lua
Pascal
R
Razor
Rust
Scala
Swift
Typescript
Objective C
Visual Basic
VBScript
C++
C#
F#
FORTRAN
Java
Javascript
Perl
Python
Ruby
SQL
PHP
PowerShell
XML
Articles by Alexander Berthold (Articles: 2)
Articles: 2
Articles
Technical Blogs
Tips and Tricks
Reference Articles
Projects
Average article rating: 4.85
Programming Languages
C++
Fast tokenizer
18 Apr 2001 Updated:
5 Sep 2001
Rating: 4.71/5 Votes: 11 Popularity: 4.49
Licence: Not specified Views: 79,634
Bookmarked: 44
Downloaded: 1,450
Please
Sign up or sign in
to vote.
Fast tokenizer for C++ - like 'lexx'
Tokenizer and analyzer package supporting precedence prioritized rules
13 Jun 2001 Updated:
1 Jan 2002
Rating: 5.00/5 Votes: 4 Popularity: 3.01
Licence: Not specified Views: 180,757
Bookmarked: 54
Downloaded: 2,782
Please
Sign up or sign in
to vote.
A library allowing you to conveniently build a custom tokenizer and analyzer supporting precedence priorized rules
Average blogs rating:
No blogs have been submitted.
Average tips rating:
No tips have been posted.
Average reference rating:
No reference articles have been posted.
Average project rating:
No projects have been posted.
Alexander Berthold
Web Developer
Germany
This member has not yet provided a Biography. Assume it's interesting and varied, and probably something to do with programming.
Submit an article or tip