Click here to Skip to main content
15,906,628 members
Please Sign up or sign in to vote.
1.00/5 (3 votes)
See more:
C++
#include <iostream>
#include <fstream>
#include <cstdlib>
#include <unordered_map>
#include <unordered_set>
#include <vector>
#include <string>
#include <locale>

using namespace std;

unordered_set<string> keywords = {"if","then","else","begin","end"};
unordered_set<string> special = {"(",")","[","]","+","-","=",",",";"};
unordered_set<string> digit_set = {"0","1","2","3","4","5","6","7","8","9"};

bool isKeyword(string key) {
return keywords.find(key) != keywords.end();
}

bool isSpecial(string key) {
return special.find(key) != special.end();
}

bool isDigit(string key) {
return digit_set.find(key) != digit_set.end();
}

bool isChar(string key) {
return (key.length() == 1 && (key.at(0) >= 'A' && key.at(0) <= 'Z' && key.at(0) >= 'a' && key.at(0) <= 'z'));
}

bool isReal(string key) {
int found = key.find(".");
if(found != string::npos) {
string part1 = key.substr(0,found-1);
string part2 = key.substr(found+1,key.length());
if(isDigit(part1) && isDigit(part2))
return true;
}
return false;
}

int main(int argc, char *argv[]) {
if(argc < 2) {
cout << "Incorrect usage! use: ./tokenize filename.ext";
exit(-1);
}

ifstream fin;
fin.open(argv[1]);
unordered_map<string,vector<string>> symbol_table;
vector<string> keyword_vec, real_vec, special_vec, character_vec, digit_vec;
symbol_table["keyword"] = keyword_vec;
symbol_table["real"] = real_vec;
symbol_table["special"] = special_vec;
symbol_table["character"] = character_vec;
symbol_table["digit"] = digit_vec;

string temp;
while(fin) {
getline(fin,temp,' ');
if(isKeyword(temp))
symbol_table["keyword"].push_back(temp);
else if(isReal(temp))
symbol_table["real"].push_back(temp);
else if(isSpecial(temp))
symbol_table["special"].push_back(temp);
else if(isChar(temp))
symbol_table["character"].push_back(temp);
else if(isDigit(temp))
symbol_table["digit"].push_back(temp);
}
fin.close();
cout << "\n-----File Summary------";
cout << "\n-----------------------";
cout << "\nKeywords : \t" << symbol_table["keyword"].size();
cout << "\nReal Values : \t" << symbol_table["real"].size();
cout << "\nSpecials : \t" << symbol_table["special"].size();
cout << "\nCharacters : " << symbol_table["character"].size();
cout << "\nDigits : " << symbol_table["digit"].size();
return 0;
}


What I have tried:

Is there a way to convert this into Python language
Posted
Updated 3-Nov-20 20:03pm
v2
Comments
Rick York 3-Nov-20 18:32pm    
Yes. Asking us to do it for you is not it.
Dana Gorsuch 3-Nov-20 18:43pm    
Wasn't asking anyone to do it for me. Was looking for resources to help convert it myself. Thanks anyway for the help. Just an intro student here trying to learn something.
Rick York 3-Nov-20 20:29pm    
Google and DuckDuckGo are both better resources for searching than asking people in a forum. Asking in a forum implies either A:) you don't know how to search because else why would you wait over an hour or longer for a response instead of getting one in five seconds? or B:) you are actually asking for someone to do it for you.

I honestly can not fathom how or why it would be a better to ask this question in a forum instead of a search engine. Maybe that's just me. I can be obtuse on occasion.

However, the fact that you included the code indicates your response is disingenuous. If you were just asking for resources the code is irrelevant. In other words - I don't buy it for a second.

1 solution

Translating such code to Python it is not very difficult, you can do it yourself, gaining insight.
Try and post here specific questions when you are stuck.
 
Share this answer
 

This content, along with any associated source code and files, is licensed under The Code Project Open License (CPOL)



CodeProject, 20 Bay Street, 11th Floor Toronto, Ontario, Canada M5J 2N8 +1 (416) 849-8900