Click here to Skip to main content
15,861,125 members
Articles / Web Development / HTML

AfterWork HTML Parser in C#

Rate me:
Please Sign up or sign in to vote.
4.10/5 (6 votes)
11 Jun 2009Public Domain2 min read 57.1K   2.1K   26   6
Actually, this is more of a lexical analyzer, but still very applicable for reading HTML and building a DOM tree.

Introduction

This is a tribute to MIL HTML Parser which I used couple of times and which turned out to be not capable of reading some HTMLs around.

Background

This is an HTML lexical analyzer, which is one step away from a decent HTML parser. Basically, the only difference is that this analyzer produces a sequence of HTML tokens and doesn't build an HTML tree-structure.

This thing is well-trained to handle many situations of reading loosely formatted HTML pages (which are pretty common in the Internet).

Using the code

Here are a couple of examples to get a quick introduction of how this thing works.

In the following example, we take a page from eBay and see the sequence of HTML tokens produced by the lexer:

C#
public void DemoExampleTest()
{
    HttpWebRequest request = (HttpWebRequest)WebRequest.Create("http://www.ebay.com/");
    HttpWebResponse response = (HttpWebResponse)request.GetResponse();
    string html;
    using (StreamReader streamReader = new StreamReader(response.GetResponseStream()))
    {
        html = streamReader.ReadToEnd();
    }
    HtmlReader reader = new HtmlReader(html);
    IndentBuilder tracker = new IndentBuilder();
    HtmlReader.Read(reader, tracker);
    Trace.WriteLine(reader.Builder.ToString());
    // Trace.WriteLine(tracker.ToString());  // << == UNCOMMENT WITH CAUTION!!!
}

/* -- This is what you are likely to see  ---
[TAG_STARTS:<][DTD:!DOCTYPE][WHITESPACE: ][DTD_TOP:html][WHITESPACE: ][DTD_AVAIL:PUBLIC]
[WHITESPACE: ][DTD_FPI:"-//W3C//DTD HTML 4.01 Transitional//EN"][WHITESPACE: ]
[DTD_URL:"http://www.w3.org/TR/html4/loose.dtd"][TAG_ENDS:>][TAG_STARTS:<][NAME:html]
[TAG_ENDS:>][TAG_STARTS:<][NAME:head][TAG_ENDS:>][TAG_STARTS:<][NAME:meta][WHITESPACE: ]
[ATTR:http-equiv][ASSIGN:=][QUOTED_VALUE:"Content-Type"][WHITESPACE: ][ATTR:content]
[ASSIGN:=][QUOTED_VALUE:"text/html; charset=UTF-8"][TAG_ENDS:>][TAG_STARTS:<][NAME:link] ...
--------------------------------------------*/

In the next example, we work with the same page with some adjustments to the HTML grammar being used. Take a look at the handler of the TokenChaning event, this is a very good place to put your code for building an HTML tree-structure.

C#
public void PracticalExampleTest()
{
    HttpWebRequest request = (HttpWebRequest)WebRequest.Create("http://www.ebay.com/");
    HttpWebResponse response = (HttpWebResponse)request.GetResponse();
    string html;
    using (StreamReader streamReader = new StreamReader(response.GetResponseStream()))
    {
        html = streamReader.ReadToEnd();
    }
    HtmlGrammarOptions options = new HtmlGrammarOptions();
    options.HandleCharacterReferences = true;
    options.DecomposeCharacterReference = true;
    options.HandleUnfinishedTags = true;
    HtmlGrammar grammar = new HtmlGrammar(options);
    HtmlReader reader = new HtmlReader(html, grammar);
    reader.Builder.TokenChaning += delegate(TokenChangingArgs args)
    {
        if (args.HasBefore)
        {
            Trace.WriteLine(args.Before.Id + ": " + args.Before.Value);
        }
    };
    HtmlReader.Read(reader, null);
}

Points of interest

I personally believe that there must be a free decent HTML parser in the public domain. It took me several days to realize that there is nothing in the Internet that can be used out of the box in a C# .NET project for this purpose. Well, let's try to make a difference.

History

This is actually a prototype of a parser I've been working on lately. Which means this thing needs some additional work to be finished. But it's somewhat already functional, and reads 98% of HTMLs you would ever see. There are a few cases where it can suddenly stop, but just have a quick look at the HtmlGrammar class. All missing links and states can be easily added there.

See also

There are also some useful stuff you might be interested in:

License

This article, along with any associated source code and files, is licensed under A Public Domain dedication


Written By
United States United States
C# .NET developer since 2003

Comments and Discussions

 
Question不能下载,网页打开非常缓慢 Pin
Member 126762848-Aug-16 20:03
Member 126762848-Aug-16 20:03 
QuestionHow to run this program Pin
shiblybcc16-Sep-12 22:51
shiblybcc16-Sep-12 22:51 
AnswerRe: How to run this program Pin
Aleksey Bykov17-Sep-12 1:55
Aleksey Bykov17-Sep-12 1:55 
GeneralHTML Agilty kit Pin
Ismail Mayat11-Jun-09 5:57
Ismail Mayat11-Jun-09 5:57 
QuestionDid you look at SGMLReader? Pin
HightechRider11-Jun-09 5:43
HightechRider11-Jun-09 5:43 

General General    News News    Suggestion Suggestion    Question Question    Bug Bug    Answer Answer    Joke Joke    Praise Praise    Rant Rant    Admin Admin   

Use Ctrl+Left/Right to switch messages, Ctrl+Up/Down to switch threads, Ctrl+Shift+Left/Right to switch pages.