Click here to Skip to main content
15,884,099 members
Please Sign up or sign in to vote.
0.00/5 (No votes)
See more:
Hi,
I have been working on a macro for scraping webpages.
The earlier version of the tool used "msxml2.xmlhttp" to load pages

VB
Set html_Page = CreateObject("htmlFile")
With CreateObject("msxml2.xmlhttp")
.Open "GET", page_url, False
.send
 html_Page.body.innerHTML = .responseText
 End With

I use this html_Page object to parse through tags on the page.

When I took a dump of all the text on the page("html_Page") , I found only static content was available.

The target Page may have been static earlier ,hence the macro worked fine for a while.
Since, the website update ,the page now has some data this seems to be fetched after page load through JS/Ajax. I confirmed this by loading the page in Firefox with JS disabled.

So is there a way to run the JS on this page so that all elements are loaded with "msxml2.xmlhttp" or something similar?
Posted

This content, along with any associated source code and files, is licensed under The Code Project Open License (CPOL)



CodeProject, 20 Bay Street, 11th Floor Toronto, Ontario, Canada M5J 2N8 +1 (416) 849-8900