AfterWork HTML Parser in C#

Download source code - 21.8 KB

Introduction

This is a tribute to MIL HTML Parser which I used couple of times and which turned out to be not capable of reading some HTMLs around.

Background

This is an HTML lexical analyzer, which is one step away from a decent HTML parser. Basically, the only difference is that this analyzer produces a sequence of HTML tokens and doesn't build an HTML tree-structure.

This thing is well-trained to handle many situations of reading loosely formatted HTML pages (which are pretty common in the Internet).

Using the code

Here are a couple of examples to get a quick introduction of how this thing works.

In the following example, we take a page from eBay and see the sequence of HTML tokens produced by the lexer:

public void DemoExampleTest()
{
    HttpWebRequest request = (HttpWebRequest)WebRequest.Create("http://www.ebay.com/");
    HttpWebResponse response = (HttpWebResponse)request.GetResponse();
    string html;
    using (StreamReader streamReader = new StreamReader(response.GetResponseStream()))
    {
        html = streamReader.ReadToEnd();
    }
    HtmlReader reader = new HtmlReader(html);
    IndentBuilder tracker = new IndentBuilder();
    HtmlReader.Read(reader, tracker);
    Trace.WriteLine(reader.Builder.ToString());
    // Trace.WriteLine(tracker.ToString());  // << == UNCOMMENT WITH CAUTION!!!
}

/* -- This is what you are likely to see  ---
[TAG_STARTS:<][DTD:!DOCTYPE][WHITESPACE: ][DTD_TOP:html][WHITESPACE: ][DTD_AVAIL:PUBLIC]
[WHITESPACE: ][DTD_FPI:"-//W3C//DTD HTML 4.01 Transitional//EN"][WHITESPACE: ]
[DTD_URL:"http://www.w3.org/TR/html4/loose.dtd"][TAG_ENDS:>][TAG_STARTS:<][NAME:html]
[TAG_ENDS:>][TAG_STARTS:<][NAME:head][TAG_ENDS:>][TAG_STARTS:<][NAME:meta][WHITESPACE: ]
[ATTR:http-equiv][ASSIGN:=][QUOTED_VALUE:"Content-Type"][WHITESPACE: ][ATTR:content]
[ASSIGN:=][QUOTED_VALUE:"text/html; charset=UTF-8"][TAG_ENDS:>][TAG_STARTS:<][NAME:link] ...
--------------------------------------------*/

In the next example, we work with the same page with some adjustments to the HTML grammar being used. Take a look at the handler of the TokenChaning event, this is a very good place to put your code for building an HTML tree-structure.

public void PracticalExampleTest()
{
    HttpWebRequest request = (HttpWebRequest)WebRequest.Create("http://www.ebay.com/");
    HttpWebResponse response = (HttpWebResponse)request.GetResponse();
    string html;
    using (StreamReader streamReader = new StreamReader(response.GetResponseStream()))
    {
        html = streamReader.ReadToEnd();
    }
    HtmlGrammarOptions options = new HtmlGrammarOptions();
    options.HandleCharacterReferences = true;
    options.DecomposeCharacterReference = true;
    options.HandleUnfinishedTags = true;
    HtmlGrammar grammar = new HtmlGrammar(options);
    HtmlReader reader = new HtmlReader(html, grammar);
    reader.Builder.TokenChaning += delegate(TokenChangingArgs args)
    {
        if (args.HasBefore)
        {
            Trace.WriteLine(args.Before.Id + ": " + args.Before.Value);
        }
    };
    HtmlReader.Read(reader, null);
}

Points of interest

I personally believe that there must be a free decent HTML parser in the public domain. It took me several days to realize that there is nothing in the Internet that can be used out of the box in a C# .NET project for this purpose. Well, let's try to make a difference.

History

This is actually a prototype of a parser I've been working on lately. Which means this thing needs some additional work to be finished. But it's somewhat already functional, and reads 98% of HTMLs you would ever see. There are a few cases where it can suddenly stop, but just have a quick look at the HtmlGrammar class. All missing links and states can be easily added there.