StringParser - CodeProject

Introduction

StringParser is an object that helps you extract information from a string. The class is perhaps best suited to parse HTML pages downloaded from the web (see my WebResourceProvider class that helps you do this). You use StringParser by constructing it with some content (i.e. a string) and using its navigational and extraction methods to extract substrings from the content. StringParser also provides some static methods designed specifically for parsing HTML.

API

Here are some of the methods provided by StringParser. Please see the accompanying documentation for an exhaustive list.

Navigational API
resetPosition()
skipToEndOf()
skipToEndOfNoCase()
skipToStartOf()
skipToStartOfNoCase() Extraction API
extractTo()
extractToNoCase()
extractUntil()
extractUntilNoCase()
extractToEnd() Position query API
at()
atNoCase() HTML parsing API
getLinks()
removeComments()
removeEnclosingAnchorTag()
removeEnclosingQuotes()
removeHtml()
removeScripts()

Example 1 - Extracting delimited text

This example shows how to extract text contained between two delimiters.

  // Extract text between the comma and question mark
  string strExtract = "";
  string str = "Hello Sally, how are you?";
  StringParser p = new StringParser (str);
  if (p.skipToStartOf (",") && p.extractTo ("?", ref strExtract))
     Console.Writeln ("Extracted text = {0}", strExtract);
  else
     Console.Writeln ("No text extracted.");

Example 2 - Extracting the nth occurence of a delimited string

This example shows how to obtain the href attribute of the third anchor tag (<a>) in an HTML string. The example assumes the string contains valid HTML.

  // Get href attribute of 3rd <a> tag
  string strExtract = "";
  string str = "..."; // HTML
  StringParser p = new StringParser (str);
  if (p.skipToStartOfNoCase ("<a") &&
      p.skipToStartOfNoCase ("<a") &&
      p.skipToStartOfNoCase ("<a") &&
      p.skipToStartOfNoCase ("href=\"") &&
      p.extractTo ("\"", ref strExtract))
     Console.Writeln ("Extracted text = {0}", strExtract);
  else
     Console.Writeln ("No text extracted.");

Example 3 - Global case-insensitive replacement

This example shows how to case-insensitively replace a string in the parser's content..

  // Replace every occurence of <td> with <td class="foo">
  string str = "..."; // HTML
  StringParser p = new StringParser (str);
  p.replaceEvery ("<td>", "<td class=\"foo\">");

Example 4 - Poor man's web scraping

This example shows how to obtain a stock's quote from the content downloaded from Yahoo Finance (MSFT). The example makes assumptions about the format of the web page.

  // Scrape http://finance.yahoo.com/q?s=msft
  string strQuote = "";
  string str = "..."; // HTML downloaded from http://finance.yahoo.com/q?s=msft
  StringParser p = new StringParser (str);
  if (p.skipToEndOfNoCase ("Last Trade:</td><td class="yfnc_tabledata1"><big><b>") &&
      p.extractTo ("</b>", ref strQuote))
     Console.Writeln ("MSFT (delayed) = {0}", strQuote);

Example 5 - Get list of hyperlinked phrases

This example shows how to obtain the list of hyperlinked phrases in HTML content.

  ArrayList phrases = new ArrayList();
  string str = "..."; // HTML content
  StringParser p = new StringParser (str);
  while (p.skipToStartOfNoCase ("<a")) {
    string strPhrase = "";
    if (p.skipToEndOf (">") && p.extractTo ("<a>", ref strPhrase))
       phrases.Add (strPhrase);
  }

Demo applications

C# applications (with full source code) that use StringParser can be found here:

DomainWalker - a web topology analyzer
GoogleTranslator - an object that uses Google to translate natural language
SimpleRSS - an RSS channel reader

Revision History

15 Jan 2006
Initial version.

License

This article, along with any associated source code and files, is licensed under The Code Project Open License (CPOL)

About the Author

Ravi Bhavnani

Canada

Member

Ravi Bhavnani is an ardent fan of Microsoft technologies who loves building Windows apps, especially PIMs, system utilities, and things that go bump on the Internet. During his career, Ravi has developed expert systems, desktop imaging apps, marketing automation software, EDA tools, a platform to help people find, analyze and understand information, trading software for institutional investors and advanced data visualization solutions. He currently works for a company that provides enterprise workforce management solutions to large clients.

His interests include the .NET framework, reasoning systems, financial analysis and algorithmic trading, NLP, CHI and UI design. Ravi holds a BS in Physics and Math and an MS in Computer Science and was a Microsoft MVP (C++ and C# in 2006 and 2007). He is also the co-inventor of 2 patents on software security and generating data visualization dashboards. His claim to fame is that he crafted CodeProject's "joke" forum post icon.

Ravi's biggest fear is that one day he might actually get a life, although the chances of that happening seem extremely remote.

Article Top

Poor

Excellent

Add a reason or comment to your vote: x
Votes of 3 or less require a comment

Comments and Discussions

You must Sign In to use this message board. (secure sign-in)

Search this forum
	Profile popups Noise Layout Per page

My vote of 5

samiDiab

3:33 29 Feb '12

skipTo and extractTo methods fit my case very well
Sign In·View Thread·Permalink

Extracting Meta Keywords and Descriptions

keith_fra

9:48 26 Jul '07

Hi, this is really good stuff! Any thoughts on the best way to extract out just the meta keywords and descriptions? I have stuck on this for about a week and then I came across your site. I already have the HTML String I was just wondering the best way to get those elements out. Thanks!
Sign In·View Thread·Permalink	2.00/5 (1 vote)

RewindTo?

krn_2k

8:23 19 Jun '07

Hi, great helpful class. A couple of questions: Is it possible to search/extract backwards? Many it is required to search for some unchanging string to locate information, then read backwards to get the variable (unsearchable) data located just in front of it. Also is it possible to record a Position, then move back to that position if a search fails: MoveTo(Position)? Thanks!
Sign In·View Thread·Permalink	5.00/5 (1 vote)

Re: RewindTo?

Ravi Bhavnani

8:37 19 Jun '07

krn_2k wrote:
Is it possible to search/extract backwards?

This functionality is present in the MFC version[^] of the WebResourceProvider class. I'll add them to StringParser.

krn_2k wrote:
is it possible to record a Position, then move back to that position if a search fails: MoveTo(Position)?

I'll add that property.

Thanks for your comments!

/ravi

This is your brain on Celcius
Home | Music | Articles | Freeware | Trips
ravib(at)ravib(dot)com

Re: RewindTo?

krn_2k

8:44 19 Jun '07

Great thank you, got my vote!
Sign In·View Thread·Permalink

extract tags

rama jayapal

21:50 29 Mar '07

i tried to extract all the <input> tags from my webpage
i used the following code but i didnt get the desired result to extract the tags
can u verify is my code correct using Stringparser

string html = "URL";
System.Net.HttpWebRequest webrequest = (HttpWebRequest)System.Net.WebRequest.Create(html);
System.Net.HttpWebResponse webresponse = (HttpWebResponse)webrequest.GetResponse();
StreamReader webstream = new StreamReader(webresponse.GetResponseStream(), Encoding.ASCII);
webrequest.Method = "GET";
string strml = webstream.ReadToEnd();
ArrayList phrases = new ArrayList();
string str = strml; // HTML content
StringParser p = new StringParser(str);
while (p.skipToStartOfNoCase("<input"))
{
string strPhrase = "";
if (p.skipToEndOfNoCase(">") && p.extractTo("<input>", ref strPhrase))
phrases.Add(strPhrase);
}

webstream.Close();
webresponse.Close();
webrequest.Abort();

Re: extract tags

Ravi Bhavnani

2:58 30 Mar '07

It's hard to tell if your logic is correct without knowing the downloaded content and what you're looking to extract. I could try to help if you posted a sample url and the desired extract target. /ravi This is your brain on Celcius Home \| Music \| Articles \| Freeware \| Trips ravib(at)ravib(dot)com
Sign In·View Thread·Permalink

good stuff

tonyc2a

6:34 25 Feb '07

This is good stuff, thanks! I can't wait to see future enhancements.
Sign In·View Thread·Permalink

Re: good stuff

Ravi Bhavnani

6:42 25 Feb '07

Thanks, Tony! /ravi This is your brain on Celcius Home \| Music \| Articles \| Freeware \| Trips ravib(at)ravib(dot)com
Sign In·View Thread·Permalink

String parser for Client server

venkiiz

3:23 23 Jan '07

Can I use the stringparser and googlw translator to convert the text in a client server applications? Help please... Venkatesh V Consultant
Sign In·View Thread·Permalink

Re: String parser for Client server

Ravi Bhavnani

4:40 23 Jan '07

Technically, "yes". See this[^] article for a self-contained translator class.

But be careful - because natural language translation is rarely 100% accurate, you should have a native speaker check the results. IOW, it's prudent to not completely rely on machine translation. Further, usage of Google's translation tools for commercial use is almost certainly a violation of their TOS, so be sure to check this before deploying your application.

/ravi

This is your brain on Celcius
Home | Music | Articles | Freeware | Trips
ravib(at)ravib(dot)com

C++ Version

Imtiaz Murtaza

20:18 23 Nov '06

Is there any C++ version of this class ? Imtiaz
Sign In·View Thread·Permalink

Re: C++ Version

Ravi Bhavnani

1:57 24 Nov '06

Yes. Most of `StringParser`'s logic can be found in my C++ article WebResourceProvider[^]. /ravi My new year's resolution: 2048 x 1536 Home \| Music \| Articles \| Freeware \| Trips ravib(at)ravib(dot)com
Sign In·View Thread·Permalink

the best articale!!!

ronicohen

7:51 17 Nov '06

very helpful articale and it the best i read abiut this subject thanks man
Sign In·View Thread·Permalink

Re: the best articale!!!

Ravi Bhavnani

7:59 17 Nov '06

Thanks! /ravi My new year's resolution: 2048 x 1536 Home \| Music \| Articles \| Freeware \| Trips ravib(at)ravib(dot)com
Sign In·View Thread·Permalink

This is awesome!

zythra

17:49 15 Apr '06

Oh man, this is too cool. So simple but extremely powerful. I do a lot of screen scraping and this is just awesome. Well done Ravi! Well written and easy to use. I thank you for posting such a fine piece of code.
Sign In·View Thread·Permalink

Re: This is awesome!

Ravi Bhavnani

4:56 16 Apr '06

Thanks! /ravi My new year's resolution: 2048 x 1536 Home \| Music \| Articles \| Freeware \| Trips ravib(at)ravib(dot)com
Sign In·View Thread·Permalink

How to keep session on

sumoncsekugmail

0:57 19 Feb '06

In some site (href="http://www.fsbo.com/search.phtml" for example)for every HTTPWebRequest call will create an seassion. can u pls tell me me how to continue the same seassion for the next HTTPWebRequest Call
Sign In·View Thread·Permalink

Re: How to keep session on

Ravi Bhavnani

2:28 19 Feb '06

Perhaps the `KeepAlive`[^] property may help? /ravi My new year's resolution: 2048 x 1536 Home \| Music \| Articles \| Freeware \| Trips ravib(at)ravib(dot)com
Sign In·View Thread·Permalink

Reading Meta tags

rizwan_rashid

18:26 14 Feb '06

Re: Reading Meta tags

Ravi Bhavnani

2:44 15 Feb '06

rizwan_rashid wrote:
plz tell me general method

I'm afraid there isn't any. Because HTML syntax is very forgiving, the best you could do is obtain the content of each meta tag (by extracting the text between <meta and >, then parse the content by successively extracting the value of the (optional) description, keywords, etc. tags.

Remember to use the case-insensitive forms of the navigational APIs.

/ravi

My new year's resolution: 2048 x 1536
Home | Music | Articles | Freeware | Trips
ravib(at)ravib(dot)com

HELP!!

rizwan_rashid

6:15 14 Feb '06

<META content=10.1045/november2004-hodge name=DOI>
<META http-equiv=content-type content="text/html; CHARSET=iso-8859-1">
<META content="D-Lib Magazine" name=description>
<META content="D-Lib Magazine, Digital Libraries, Digital Library Research"
name=keywords>
<META content="MSHTML 6.00.2900.2802" name=GENERATOR></HEAD>
<BODY bgColor=#ffffff>
---------------------
Above is the html content to be parsed.i want to get values of contents given in META tag.e.g i want to get content which has name=description in META tag.i tried a lot to reach particular tag.(order of tags can be different).plz tell me general method so i can read META tag contents value which have name=description,name=keywords,name=Author...
Thank You

HELP!!!!

rizwan_rashid

0:44 12 Feb '06

i want to extract title and some lines of text from a webpage.
Sign In·View Thread·Permalink

Re: HELP!!!!

Ravi Bhavnani

1:14 12 Feb '06

rizwan_rashid wrote:
i want to extract title and some lines of text from a webpage.

Here's how you'd extract the title:

  string strTitle = "";
  string strHtml = "..."; // fill string with HTML
 
  StringParser p = new StringParser (strHtml);
  if (p.skipToEndOfNoCase ("<title>")) {
     if (p.extractToNoCase ("</title>", ref strTitle)) {
         Console.Writeln ("Title = " + strTitle);
     }
  }

/ravi

My new year's resolution: 2048 x 1536
Home | Music | Articles | Freeware | Trips
ravib(at)ravib(dot)com

Re: HELP!!!!

rizwan_rashid

1:38 12 Feb '06

thanks!! but actually i m confuesed of this " stringHtml="..."; "should i write here path where html file is locating or write just html code as string. and plz tell me if i want to read html file.
Sign In·View Thread·Permalink