How do I parse a line statement into tokens in c++?

Question

0.00/5 (No votes)

See more:

I want to create a parser program in c++ so i can split each word / symbol into tokens. Am trying to split it by space, later will be split by '(',')','*','/', etc. I got a problem when input "Hello World", it just "d" which is saved. Any help, Thanks

C++

void main(){
	clrscr();
	char input[100], token[100], *temp[100];
	int lengthLine, i, pos, posTok;

	do{
		cout << ">>> "; cin.getline(input, 256);
		lengthLine = strlen(input);

		posTok = 0;

		for (pos=0;pos<lengthLine;pos++){
			if (input[pos] != ' '){
				strcpy(temp[posTok],&input[pos]);
			}else{
				posTok++;
			}
		}

		for (i=0;i<=posTok;i++){
			cout << temp[posTok] << endl;
		}
	}while(strcmp(input,"exit") != 0);
	getch();
}

[edit]Code block added - OriginalGriff[/edit]

Posted 13-Jun-15 21:52pm

rudy-peto

Updated 13-Jun-15 22:05pm

OriginalGriff

v2

Add a Solution

1 solution

Add a Solution

Add your solution here

Treat my content as plain text, not as HTML

Preview 0

…

Existing Members

Sign in to your account

...or Join us

Download, Vote, Comment, Publish.

Your Email
Password
Forgot your password?

Your Email
This email is in use. Do you need your password?
Optional Password

I have read and agree to the Terms of Service and Privacy Policy
Please subscribe me to the CodeProject newsletters

When answering a question please:

Read the question carefully.
Understand that English isn't everyone's first language so be lenient of bad spelling and grammar.
If a question is poorly phrased then either ask for clarification, ignore it, or edit the question and fix the problem. Insults are not welcome.
Don't tell someone to read the manual. Chances are they have and don't get it. Provide an answer or move on to the next question.

Let's work to help developers, not make them feel stupid.

This content, along with any associated source code and files, is licensed under The Code Project Open License (CPOL)

OriginalGriff · Answer 1 · 2015-06-13T22:14:00

The problem is that you aren't breaking it very well - look at your code.
You declare temp and an array of 100 "pointer-to-char" values, and you use strcpy to copy the whole remaining line into it:

C++

strcpy(temp[posTok],&input[pos]);

But...you never assign any values to the pointers, so they are all a bit random.
And you never terminate the "tokens" either.
What you need to do it set up temp, but then use malloc to allocate space for each token as you go.
What I would do is:
1) Create an array of token pointers.
2) Use malloc to allocate a "token string space", and set the first char to '\0'. Set it into the next free token array slot.
3) Loop thorugh each character in the input string
3.1) If it's a space copy a '\0' to the token, and allocate a new token space as in (2).
3.2) If it isn't, copy the character to the token.
4) Don't forget to use free to release the memory you allocated!

Make sense?