CodeProject: MIL HTML Parser. Free source code and programming help

Report Article

Discuss

Send to a friend

58 votes for this Article.

Popularity: 8.25 Rating: 4.68 out of 5

Library and demonstration application - 31.3 Kb

Introduction

This library produces a domain tree of a given HTML document, allowing the developer to navigate and change the document in an methodical way. In addition to the basic HTML production, this library can also be used to produce XHTML documents, as it includes an HTML 4 entity encoder. Included in this release is a demonstration application in VB.NET showing how to use the library. I hope that it is all fairly self-explanatory.

Background

This library was written to avoid having to convert a document into XML prior to reading, whilst preserving the distinct HTML qualities. This gets round some deployment issues I had with different platforms.

Using the code

The simplest way to use the code is to add it into your solution as a C# class library. There are no third-party dependencies so it is just a matter of adding the source files in. Alternatively, you can build the DLL and add it as a reference.

Points of Interest

The XHTML production is fairly basic - there is no built-in DTD checking. So far, I have had no problems in the generation, but I'm keen on getting that sorted.

History

1.4
- Bugfix: XHTML elements are in the proper case
- Bugfix: XHTML <html> will default to xmlns="http://www.w3.org/1999/xhtml"
1.3
- Bugfix: <!DOCTYPE...> and <!...> now treated as comments
- Bugfix: Malformed or incomplete attribute values causing infinite loop fixed
1.2
- Bugfix: <tag/> now handled properly
- Bugfix: Parse errors of scripts
- Bugfix: Parse errors of styles
- HTML 4 entity encoding
- DOM tree navigation
- Basic node searching
- HTML production
- XHTML production (as per http://www.w3.org/TR/xhtml1/DTD/xhtml1-strict.dtd)
- Added some component model stuff & comments
- Hid the parser
1.1
- Initial release

License

This article has no explicit license attached to it but may contain usage terms in the article text or the download files themselves. If in doubt please contact the author via the discussion board below.

A list of licenses authors might use can be found here

About the Author

Member 987427

Location:

United Kingdom

Other popular .NET Framework articles:

The 30 Minute Regex Tutorial
Learn how to use regular expressions in 30 minutes with Expresso.
Strings UNDOCUMENTED
Detailed looked at the implementation of strings in .NET
Expresso - A Tool for Building and Testing Regular Expressions
For learning, building, and debugging .NET Framework regular expressions
Unraveling the Mysteries of .NET 2.0 Configuration
Learn how to utilize the powerful new .NET 2.0 configuration features to simplify and centralize your configuration code.
Simple Windows Service Sample
A simple application to show how to create a Windows service.

Article Top

You must Sign In to use this message board.

Msgs 1 to 25 of 57 (Total in Forum: 57) (Refresh)

FirstPrevNext

Lowercased href

exxellence

2:04 12 Nov '08

In version 1.4 all XHTML tags are lowercase, however all attributes are also lowercased, including the HREF attribute of the A tag. This shouldn't be done because URL's and querystrings are casesensitive, for instance YouTube URL's. I changed this in HTMLattribute.cs in the XHTML property, so that HREF attributevalues aren't lowercased. Just to let you know.
Sign In·View Thread·PermaLink

feature missing

zeltera

5:34 17 Aug '08

I cannot select for a node it's attributes... Is this a missing feature or I just didn't see it?? thanks for this great parser. zeltera's blog
Sign In·View Thread·PermaLink	2.00/5 (1 vote)

Re: feature missing

smitsc

9:11 7 Oct '08

since you're a recent user maybe you could show me a simple example of how to make this work ??
Sign In·View Thread·PermaLink

DOCTYPE breaks the parser

benblo

6:18 14 May '08

I have a document that starts with a comment and doctype :

<!-- #BeginTemplate "/Templates/manual-scriptref-page.dwt" --><!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Transitional//EN" "http://www.w3.org/TR/xhtml1/DTD/xhtml1-transitional.dtd">
<html xmlns="http://www.w3.org/1999/xhtml">
<body>bla</body>
</html><!-- #EndTemplate -->

The parser breaks... renders 3 roots: the html tag as a text element, then the head and body.
After a few trials, turns out a doc starting with a comment gets parsed ok, but starting with a doctype doesn't.

... just wanted to let the author know, seems odd since the history mentions a doctype bugfix: is the zip up-to-date?

Is it a bug?

huyhk

21:57 27 Feb '08

I parse this HTML string

<a target='_blank' class=team href=/data/league/team.php?t=610&s=413 class=team>Mariehamn</a>

And got an HTMLElement object with 8 attributes


target="_blank"
class="team"
href=""
null
data=null
league=null
team.php?t="t=610&s=413"
class="team"

I think the problem is from href attribute.

Re: Is it a bug?

Natural Cause

23:34 26 Mar '08

It could be your poorly contructed HTML... You don't wrap any of the values in double quotes, you have class specified twice...
Sign In·View Thread·PermaLink

Suggestions for new interface methods

Berend Engelbrecht

10:10 26 Feb '08

I have succesfully used your library in a project to query books by isbn and collect catalogue data on them from various sites. Your html parser was by far the best of the five or so parser libraries that I tried, but still I missed some features in the API. I made some changes to my copy of the source, perhaps you would be willing to consider them?

My changes were:
1- change abstract class HtmlEncoder from internal to public, so that I can decode any html text fragment myself.
2- bugfix in decoding &xNN; hexadecimal html escape in HtmlEncoder.cs (see message earlier today)
3- Introduce extended matching for methods for attribute values:
      public enum SearchMethod
      {
         ExactMatch, // default
         ValueBeginsWith, // uses .StartsWith to match beginning of attribute value
         ValueContains // uses .IndexOf to match any part of a value
      }

I made an extra overload to FindByAttributeNameValue that has a searchMethod parameter to incorporate this.

Usage example: Consider an amazon.com "product overview" for a book. Authors are contained in A elements where the href attribute contains the substring "&field-author=". Having the SearchMethod parameter allows me to directly find only the nodes that I need:

HtmlNodeCollection nc = htmlDoc.FindByAttributeNameValue("href", "&field-author=", true, SearchMethod.ValueContains);

4- added an extra method FindByNameAttributeNameValue to match both node name and an attribute name/value pair. The example above can be made more efficient by also specifying the node name a:

HtmlNodeCollection nc = htmlDoc.FindByNameAttributeNameValue("a", "href", "&field-author=", true, SearchMethod.ValueContains);

This will return the same collection, but significantly faster because it no longer has iterate through every attribute of each node in the html document, but only through the small subset of a nodes.

Best regards,

Berend Engelbrecht

Found a bug

stavinski

11:09 14 Jan '08

I was using the HtmlDocument.Create(...) against HTML returned from the msn search site, and kept getting a FormatException, i managed to trace it to this call:

int v = int.Parse( token.ToString().Substring(2,token.Length-3) );

line 831 in the HtmlEncoder, the token.ToString().Substring(2, token.Length-3) resulted in the following value "xB7" as it is using a hex base character entity "·", think some logic needs to be added to check for hex entity as opposed to dec.

Thanks,
Mike

Re: Found a bug - me too, and the solution

Berend Engelbrecht

22:04 25 Feb '08

Since I had to parse a web site that used A0; for nonbreaking spaces everywhere, I took the liberty of fixing it in my copy. I would welcome that my fix (or similar code) is included in the standard version:

            if (token[1] == '#')
            {
              // Berend: also support hex notation
              try
              {
                if (token[2] == 'x')
                {
                  int v = int.Parse(token.ToString().Substring(3).Split(';')[0], System.Globalization.NumberStyles.HexNumber);
                  output.Append((char)v);
                }
                else
                {
                  int v = int.Parse(token.ToString().Substring(2, token.Length - 3));
                  output.Append((char)v);
                }
              }
              catch (Exception ex)
              {
                Trace.Write(ex);
              }
            }

HTML marked

maingaosuong

17:19 25 Sep '07

Hi all
I'm implementing a Winform app about 'HTML parser'.
In my app, the users input an URL (such as: www.amazon.com) and my app will show the expected page in a web browser control.
I want to let users can choose an area on that page and a label control will show all texts in that selected area. How can I do that???
I mean that: how can I determine the HTML tags (in that page) which enclose all selected texts ???
EX:
HTML:
<html>
<body>

selected text

none selected text

</body>
</html>

Page:
selected text
none selected text

When I drag the mouse to enclose "selected text", I want to determine that table with id=1 is selected and "selected text" will be showed in a label control.

Please show me your ideas.
Thank in advance.

mns

regarding extracting tags

rama jayapal

4:49 29 Mar '07

can anyone solve my problem

i have developed a webapplication where i have parsed the contents of the webpage using

MILHTML parser

i have the document now in html format

i need to use the parser's attributes like

htmldocument

htmlelement

htmlnode

htmlattributes

am really new to this Dotnet environment and now i need to know

how to find the the tags with<input type=hidden....">

i need to seperate the input tags first and then find their attributes like type="submit,hidden" name="" etc....

have anybody done this before or can anybody give me an idea abt how to write the recursive function to seperate the input tags from the document

plz help am running short of time

thanks

Rama

Re: regarding extracting tags

James S.F. Hsieh

23:09 29 Mar '07

Maybe the following program can match your requirement
in DOL HTML Parser (http://www.codeproject.com/useritems/DOL_HTML_Parser.asp[^]).
Good Luck

// Open HTML file "xxx.htm"
DHtmlGeneralParser parser = DHtmlGeneralParser();
DHtmlDocument htmlDoc = new DHtmlDocument(parser);
htmlDoc.Load(@"..\xxx.htm");

DHtmlNodeCollection result = new DHtmlNodeCollection();

// Find all tag of this pattern <input type="oooo"> in all html document
// function: void FindByNameAttribute
// (
// DHtmlNodeCollection result, // a collection to collect result
// string name, // tag name which you want to find
// string attributeName, // attribute name which you want to find
// bool searchChildren // whether it searchs child with recursive
// )
htmlDoc.Nodes.FindByNameAttribute(result, "input" "type", true);

You can try it. :)

James S.F. Hsieh

23:36 27 Mar '07

The MIL HTML Parser is an useful library for me, but the project has stopped to maintain. I created a project "DOLS HTML Parser" based on MIL HTML Parser in codeproject and wish it can help everyone. A non-well-formed HTML Parser and CSS Resolver, The URL:http://www.codeproject.com/useritems/DOL_HTML_Parser.asp[^]
Sign In·View Thread·PermaLink

misses some IMG nodes

encapsul

8:53 10 Mar '07

Great code! For the most part it works, but more often than note it does not pick up on an IMG node I am looking for. I have tweaked the source HTML docs a bit and usually get it to work, but havn't nailed down the cause. Is there a requirement or restrictions for the source HTML/ XHTML? thanks!
Sign In·View Thread·PermaLink

Strings instead of Streams?

Yuvi Panda

7:15 26 Sep '06

I just thought of asking, why use a String instead of a Stream? Yuvi Panda T 15 Year old Microsoft Student Partner Blogs at : http://yuvipanda.blogspot.com
Sign In·View Thread·PermaLink	1.00/5 (1 vote)

Re: Strings instead of Streams?

Lennard Fonteijn

14:16 1 Dec '06

Ì dont understand why you find that so hard, while its soo easy!

Try this:

Dim mDocument As MIL.Html.HtmlDocument
Dim html As String = "Your HTML thingies here, instead of a StreamReading result)
mDocument = HtmlDocument.Create(html, False)

Then just do whatever yo want with mDocument, just a bit of hushling with the demo project...

Though, I find it a rather stupid question for a Microsoft Partner, since the readed streams are actually Strings...

SO instead of:
Dim html As String = Stream.ReadToEnd

you just change Stream.ReadToEnd to your string...

Or am I misunderstanding a question here?

You mess with the best, you die like the rest... well... kinda???

2.00/5 (1 vote)

Re: Strings instead of Streams?

Yuvi Panda

15:32 1 Dec '06

Read Streams are *not* strings. The ReadToEnd method is not the best way to use streams, especially if you are parsing...

Strings are immutable, meaning that everytime I do an operation on them, a new one is created. So, if I am doing 25 operations on a 60 kb string, that'll allocate some 1.2MB! And, that's just garbage, waiting to be GCed...

So, in my app, I process about an MB of HTML a second(since It's a Scrapper), using a string based solution would just not scale...

Streams Scale: If you do a Read with a 25 byte buffer, you only allocate 25 bytes, so it scales...

Yes, building a Stream based parser is harder, but I've already found one: HTMLAgilityPack[^], which is quite fast, and stream based...

Thanks anyway...

Yuvi Panda T
15 Year old Microsoft Student Partner
Blogs at : http://yuvipanda.blogspot.com

Move the project elsewhere

KCorax2

10:55 21 Aug '06

I strongly suggest moving your project somewhere with more exposure. I googled hard to find it and at this point it's much more usefull than microsoft's wrapper of the IE control. Such libraries are important just because the cost of keeping them up to date for a small project or for research purposes (my case) is prohibitive.

I suggest GotDotNet workspaces or the shiny new Codeplex.

Btw great work Big Grin

4.25/5 (4 votes)

Can not fix some error of HTML code .

dinhdv2001

15:36 24 Jul '06

Does it fix this error like ie : Text "Hello" is subnode of

I think it is very good but it can't fix some error of html code like IE. For example, I have a html code here:
Hi
Hello
? Who has fix this error? Please help me. sfdafsda
Sign In·View Thread·PermaLink

Spaces Removed Between Tags

Michael Babb

10:50 6 Jun '06

Spaces between words get removed if a formatting tag is contained between the words. For example: Dear Ally gets displayed as: DearAlly
Sign In·View Thread·PermaLink	2.00/5 (1 vote)

a suggestion

XiongHarry

23:09 14 Apr '06

I found that the eats some CJK(Chinese,Japanese,Korean) words.
It took me a long time to find the reason.

Finally I found that it just because of line 315:
Dim sr As System.IO.StreamReader = System.IO.File.OpenText(OpenHtmlFileDialog.FileName)

It will be ok when making a chang to :
Dim sr As System.IO.StreamReader = New System.IO.StreamReader(OpenHtmlFileDialog.FileName, System.Text.Encoding.Default)

Thanks a lot for your work.
It makes a good help to me.

Minor error in parsing <!DOCTYPE ..>

jmas8109

4:27 16 Mar '06

Hello!
I have found a minor error during weeping out of the <!DOCTYPE ...> element. If there is something meaningfull immediatelly behind <!DOCTYPE ...> section, then this first two characters of it are removed.
This error lies in the file HtmlParser.cs in RemoveSGMLComments function. The repaired version of the while cycle (at the beginning of this function) is at the end of this letter.

Greetings,
jmas8109

Source code section:
--------------------

while( i < input.Length )
{
if( i + 2 < input.Length && input.Substring( i , 2 ).Equals( "<!" ) )
{
i += 2;
i = input.IndexOf( ">" , i );
if( i == -1 )
{
break;
}
i += 1; // originally there was i += 3 (which is a bug)

jmas8109

This rocks.

rhino-x

12:43 29 Jan '06

This has made my day. There are 5 billion articles and samples about translating XML->HTML, etc. but no one wants to touch HTML->something freaking usable. Thanks!
Sign In·View Thread·PermaLink	4.00/5 (1 vote)

Good job!

Bobga

12:43 4 Nov '05

This is good article. I will say 5 star. We can't stop asking "WHY!!"
Sign In·View Thread·PermaLink

bug with filterindex in vb demo

mgambrell

14:37 17 Oct '05

the filterindex being returned by the common file dialog is 1-based, yielding indices 1 and 2. the code using the filterindex is checking for 0 or 1. the result is that xhtml is always exported.
Sign In·View Thread·PermaLink	2.00/5 (1 vote)

Last Visit: Monday, December 1, 2008 6:34 AM Last Update:Monday, December 1, 2008 6:35 AM

12 Next »

General News Question Answer Joke Rant Admin