Parse HTML of a webpage in which Ajax requests are present on document load

Ajax calls require a javascript VM to run. You will not be able, using .Net WebRequest objects, to get the Ajax result as you will not be able to execute the Javascript for the document without linking a Javascript VM to a DOM management framework. A.k.a a Browser.

You have three options.

Try and write code which looks at the pages javascript and allows you to emulate the AJAX call an integrate the content. But this is not a nice way of doing it and any change to a page could complete throw off your code.

Fiddler is a tool which implements a man-in-the-middle process and logs all requests and responses, including AJAX. You could browser the site with this and recode the results. I'm not sure how Fiddler achieves what it does but you could aim to write a similar process.

The better option is to use the .Net WebBrowser control. You can use this control to access the current page text. Load the page wait long enough for the AJAX to complete and then get the content from the browser control.