A Web Developer’s Guide to Parsing HTML with C#

What methods should devs use to parse HTML? Don't say regex! Instead, we present two options for parsing HTML using C#.

Dec 2nd, 2023 5:00am by David Eastman

Featued image for: A Web Developer’s Guide to Parsing HTML with C#

Image via Unsplash

A question that appeared in my last post within a set of interview topics was: “Why is regex bad at parsing HTML?” So, what methods should one use to parse HTML?

Developers use regular expressions (regex) to extract information from text strings; for example, a well-formed regex can grab a valid email address from within a form response. HTML is a presentation format that helps a web browser create a web page from text strings and images. The problem with parsing (that is, reading) HTML is that it uses opening and closing tags for context. This is hardly a problematic concept for humans, but computers work less well when the meaning of something depends on cues elsewhere.

In the HTML below, each reference to “Hello” has to be treated entirely differently by the parser:

Hello<div class="Hello">Hello<b>Hello</b></div>

1	Hello<div class="Hello">Hello<b>Hello</b></div>

Regex could find all the hellos, but would struggle a bit with finding just the bold hello. So we need other methods.

This post, then, is a gentle look at parsing snippets of HTML using elementary C# code with available package options.

C# Packages for Parsing HTML

I’ll look at two C# packages whose API styles will be similar to parallel packages for other languages. Keeping the problem in code means you can be flexible with how you interpret restraints and error conditions.

Imagine that you have asked your housing residents to contribute to a yearly maintenance report that you will eventually represent as a web page. This year, you have asked for each contributing resident to present their report as a very simple HTML snippet. The styling will all be done later. The residents all happen to be geeks, and they agree.

To restrain the exuberant residents, you devise a number of simple rules:

All the paragraphs must appear within a <div> with a class called “YearReport”.
The first paragraph must have a title attribute, from which we will extract the title’s text.
There should no more than three paragraphs in total.

Remember, this is intended to be content within a main report — so that will control styling. These reports are just content.

Here is a good example of a resident’s snippet:

<!DOCTYPE html> 
<html> 
  Here is my report: 
  <div class="YearReport"> 
    <p title="Report on Drains">Bad year for drains</p> 
    <p>This is a paragraph about drains.</p> 
    <p>This is another paragraph about drains.</p> 
  </div> 
</html>

<!DOCTYPE html>

<html>

Here is my report:

<p title="Report on Drains">Bad year for drains</p>

<p>This is a paragraph about drains.</p>

<p>This is another paragraph about drains.</p>

</div>

</html>

Let’s now check out some C# HTML parsing packages. Note that we aren’t interested in modifying code, just parsing it.

HTML Agility Pack (HAP)

I’ll start with HTML Agility Pack (or HAP), as that appears to be the most popular and is available as a NuGet package within Visual Studio.

Our first thing will be to find the YearlyReport div that all content must be within. This uses XPath, which is unsurprising but not something I want to examine. We just want to check, first, that there exists a <div> that has the attributes class=”YearlyReport”:

var node = htmlDoc.DocumentNode
                        .SelectSingleNode("//div[@class="YearReport"]");

1 2	var node = htmlDoc.DocumentNode .SelectSingleNode("//div[@class="YearReport"]");

The mysterious string is XPath, whose job is to describe positions within an HTML/XML document. The one above says simply “find a <div> anywhere in this document, that has a class attribute with the value ‘YearReport’”.

Any well-documented technology will already be within the purview of generative AI, making the XPath extra step much less of a barrier. Asking ChatGPT to create the required XPath statement — if not the whole HTML parsing example — will certainly work.

Here is the code needed to test the constraints to a reasonable degree:

using HtmlAgilityPack; 

var htmlDoc = new HtmlDocument(); 
htmlDoc.Load("//Users/eastmad/Projects/TheNewStack/HtmlParser/GoodReport.html"); 
var node = htmlDoc.DocumentNode .SelectSingleNode("//div[@class="YearReport"]"); 
if (node != null) 
{ 
  HtmlNodeCollection paragraphnodes = node.SelectNodes("p"); 
  int numberofparas = 0; if (paragraphnodes != null) 
  { 
     string title = paragraphnodes.First().GetAttributeValue("title", "No title node found"); 
     Console.WriteLine($"The report title is: {title}"); 
     numberofparas = paragraphnodes.Count; 
  } 
  Console.WriteLine($"There are {numberofparas} paragraphs. The suggested number is between 1 and 3."); 
} 
else Console.WriteLine($"This report is invalid: it needs a correctly defined <div>");

using HtmlAgilityPack;

var htmlDoc = new HtmlDocument();

htmlDoc.Load("//Users/eastmad/Projects/TheNewStack/HtmlParser/GoodReport.html");

var node = htmlDoc.DocumentNode .SelectSingleNode("//div[@class="YearReport"]");

if (node != null)

{

HtmlNodeCollection paragraphnodes = node.SelectNodes("p");

int numberofparas = 0; if (paragraphnodes != null)

{

string title = paragraphnodes.First().GetAttributeValue("title", "No title node found");

Console.WriteLine($"The report title is: {title}");

numberofparas = paragraphnodes.Count;

}

Console.WriteLine($"There are {numberofparas} paragraphs. The suggested number is between 1 and 3.");

}

else Console.WriteLine($"This report is invalid: it needs a correctly defined <div>");

Note that the path to the file works for me, and will be different for you.

If we run this while it is looking at our example HTML, the result as expected is:

The report title is: Report on Drains 
There are 3 paragraphs. The suggested number is between 1 and 3.

1 2	The report title is: Report on Drains There are 3 paragraphs. The suggested number is between 1 and 3.

If I use any adulterated HTML that trivially break the restrictions, this code will find it. But we are clearly able to improve on it, using proper Exceptions, etc.

AngleSharp: A Non-XPath Option

However, I am now keen to see if I can use a library that doesn’t need XPath.

AngleSharp originally didn’t support XPath, but it has lots of NuGet package love now. It uses CSS selectors that for many will be a simpler choice. Using the same examples and restrictions, let’s rewrite the above code for that library:

using AngleSharp.Html.Parser; 
var htmlParser = new HtmlParser(); 
Stream file = File.OpenRead("//Users/eastmad/Projects/TheNewStack/HtmlParser/GoodREport.html"); 
var htmlDoc = htmlParser.ParseDocument(file); 
var node = htmlDoc.QuerySelector("div.YearReport"); 
if (node != null) 
{ 

  var paragraphnodes = node.QuerySelectorAll("p"); 
  int numberofparas = 0; if (paragraphnodes != null) 
  { 
    string? title = paragraphnodes.First().GetAttribute("title"); 
    Console.WriteLine($"The report title is: {title}"); 
    numberofparas = paragraphnodes.Count(); 
  } 
  Console.WriteLine($"There are {numberofparas} paragraphs. The suggested number is between 1 and 3."); 
} 
else Console.WriteLine($"This report is invalid: it needs a correctly defined <div>");

using AngleSharp.Html.Parser;

var htmlParser = new HtmlParser();

Stream file = File.OpenRead("//Users/eastmad/Projects/TheNewStack/HtmlParser/GoodREport.html");

var htmlDoc = htmlParser.ParseDocument(file);

var node = htmlDoc.QuerySelector("div.YearReport");

if (node != null)

{

var paragraphnodes = node.QuerySelectorAll("p");

int numberofparas = 0; if (paragraphnodes != null)

{

string? title = paragraphnodes.First().GetAttribute("title");

Console.WriteLine($"The report title is: {title}");

numberofparas = paragraphnodes.Count();

}

Console.WriteLine($"There are {numberofparas} paragraphs. The suggested number is between 1 and 3.");

}

else Console.WriteLine($"This report is invalid: it needs a correctly defined <div>");

I use the var keyword here to infer the type (when the types would be packaged defined) in order to show that the structures in the two examples are similar.

Other than a bit more coaxing to get the HTML file, this code looks very similar to the HAP version. The CSS selector to find the correct <div> is easier to guess at, but must still be considered as extra technology to learn.

Conclusion

I hope this goes to show that HTML can be used in a pipeline, without too much fear that it will be difficult to handle without tricky code. The kicker is that I still needed either XPath or CSS selectors to quickly hone in on the right <div>, which only adds to the allure of AI tool help.

David has been a London-based professional software developer with Oracle Corp. and British Telecom, and a consultant helping teams work in a more agile fashion. He wrote a book on UI design and has been writing technical articles ever since....