Click here to Skip to main content
16,017,351 members
Please Sign up or sign in to vote.
1.00/5 (1 vote)
See more:
Hi to all,
I want to extract dynamic data from two different web pages (different providers) to process they in a web page (which should be my application).
What procedures and/or tools would be the most appropriate for the extraction?

What I have tried:

I have tried to find the solution on Google but I have not seen anything that allows me to even start tests.
Posted
Updated 9-Feb-23 14:54pm

That'll depend a lot on the framework you are using for your app - if it's a .NET C++ program, then I'd recommend the Html Agility Pack[^] which helps with scraping a heck of a lot.

However ... there is a problem with any scraping tech: websites change, and often. And as soon as they do, most scraping ceases to work until you rework your app to support the new data format. And then again in two weeks time you are doing it again. YOu can end up with a perpetual cycles of mods just to keep up - particularly if you are processing multiple sites to collect your data.
I'd suggest that instead you look to see if the site producers have an API you can use to access the data in a consistent manner (even if you need to use multiple APIs to cover each site). That way, you get a more reliable end product!
 
Share this answer
 
Comments
Richard Deeming 8-Feb-23 9:35am    
And if they don't provide an API, then there's a pretty good chance that they don't want you using the data on your own site, and may end up instigating legal proceedings to stop you from doing it. :)
OriginalGriff 8-Feb-23 10:51am    
SEP field engaged. :D
Member 2023687 8-Feb-23 13:41pm    
None of the web pages offer APIs but the most important thing is that, in effect, both have updates (1 to 3 per year), a serious problem that I, in my inexperience, had not foreseen.
Thank you very much, OriginalGriff
OriginalGriff 8-Feb-23 14:14pm    
You're welcome!
Member 2023687 9-Feb-23 6:44am    
Dear OriginalGriff, I modify my question in the following sense, the two web pages that I want to capture their dynamic data in real time, both receive the data from two devices installed in my home, (2 inverters + 1 Huawei battery and a car charger electrical Wallbox) and that communicate with my PC and my mobile via my own WiFi. Is there a way to capture the data through my WiFi?
C++
https://www.google.com/search?client=firefox-b-d&q=c%2B%2B+get+html+from+url


Simple HTTPS GET
C++
https ://curl.se/libcurl/c/https.html
 
Share this answer
 

This content, along with any associated source code and files, is licensed under The Code Project Open License (CPOL)



CodeProject, 20 Bay Street, 11th Floor Toronto, Ontario, Canada M5J 2N8 +1 (416) 849-8900