manipulating a web page from c#

Question

0.00/5 (No votes)

See more:

Hello. I've just spent an extremely frustrating Memorial Day holiday trying to find ways to select data on a web page via drop down boxes (date, report type) from a console app. I've tried HTMLDoc and HtmlAgilityPack with meager results. While scouring the web I discovered WebBrowser but found that this won't work through a console app. What I want to do is retrieve exchange settlement data on a daily basis from a website and convert it to .csv format and process it in my application. If anybody out there's done something similar and could give advice I'd greatly appreciate it. Thanks.

Posted 27-May-13 9:42am

Tom Paronis

Add a Solution

1 solution

Add a Solution

Add your solution here

Treat my content as plain text, not as HTML

Preview 0

…

Existing Members

Sign in to your account

...or Join us

Download, Vote, Comment, Publish.

Your Email
Password
Forgot your password?

Your Email
This email is in use. Do you need your password?
Optional Password

I have read and agree to the Terms of Service and Privacy Policy
Please subscribe me to the CodeProject newsletters

When answering a question please:

Read the question carefully.
Understand that English isn't everyone's first language so be lenient of bad spelling and grammar.
If a question is poorly phrased then either ask for clarification, ignore it, or edit the question and fix the problem. Insults are not welcome.
Don't tell someone to read the manual. Chances are they have and don't get it. Provide an answer or move on to the next question.

Let's work to help developers, not make them feel stupid.

This content, along with any associated source code and files, is licensed under The Code Project Open License (CPOL)

**Zoltán Zörgő** · Answer 1 · 2013-05-27T10:02:00

Solution 1

Forget automating any web browser. You should know already that behind the scenes there is a communication between the browser and the web server conform a protocol named Hypertext Transfer Protocol (yes, that is http). Everything you do in the browser ends up in such a data exchange.
There is a class in the framework that you can use to simulate the browser activity directly on http level: HttpClient[^].
Simple samples: http://www.jayway.com/2012/03/13/httpclient-makes-get-and-post-very-simple/[^]
Complex samples http://blogs.msdn.com/b/webdev/archive/2012/08/26/asp-net-web-api-and-httpclient-samples.aspx[^]

Well, it depend on the web application you are trying to fetch, how complex this interaction will be. To get better knowing the process behind, I suggest you use Fiddler[^]. You will have to record your steps and examine the requests sent by the browser and the answers you get. If you are lucky, there is an ajax call for the content you are looking for. If not, you will need the Agility Pack you have already tried to process the result - when, and only when you have stepped trough the logic of the application.

Posted 27-May-13 10:02am

Zoltán Zörgő

Comments

Tom Paronis 27-May-13 16:59pm

Zoltan; Thanks for your response. I've already mucked about a bit w HttpClient but didn't get too far. This seems simple in concept (selecting a date and getting data for it) and I'll definitely look through the samples and examine Fiddler. Right now I need a beer! Been a long day.

Zoltán Zörgő 27-May-13 17:09pm

Can you give me the url?

Tom Paronis 27-May-13 17:48pm

Sure. I want to get settlement data for certain dates. I'm a C++ financial apps guy who's not well versed in data scraping but I have to learn ASAP.

https://www.theice.com/marketdata/reports/ReportCenter.shtml#report/10

Getting settlements for certain contract/date combinations and converting them to .csv format is the ultimate goal here. Show me how and I'll buy you a beer when you come to New York. Here's how I have fun when I'm not wrestling with this stuff!
http://www.youtube.com/watch?v=tHTfv6_UP3w&list=PLRhrH4q6OOVYwJNct9uHWsn631mHNCx9r

Zoltán Zörgő 28-May-13 2:52am

You said yourself, that you need to learn - I won't hinder you :). I examined the url you had given. This is an easy task. Follow the steps I suggested, and you will get your result. If you get stuck somewhere we are happy to help you, but we won't do the job for you. And the beer too will be yours.

Sergey Alexandrovich Kryukov 28-May-13 0:24am

Good points, a 5.
—SA

Tom Paronis 29-May-13 9:58am

This does seem like the way to go; I've been poring over some of the articles and this is what I've been looking for. Unfortunately this means I now have to get VS 2012 and port everything over. Nothing good in this life is free!

Zoltán Zörgő 29-May-13 10:13am

There is no actual difference with .net 4.5 in this topic. We are still here to help you - but not doing your job :)

Tom Paronis 29-May-13 10:32am

I certainly wouldn't want you to do my job (although I could use some extra hours in the day). What I meant to say was that all our work was done in Visual Studio 2010 and .Net 4.5 requires Visual Studio 2012. Another forced upgrade! :)

Zoltán Zörgő 29-May-13 15:25pm

I understand. But this upgrade will most likely not be so difficult. I would experiment with your url, if you would tell me what exactly do you want to fetch? The page has several sub-reports with several parameters. You wrote about a console app, thus I suppose you need a specific report and not some interactive tool.

Tom Paronis 29-May-13 17:05pm

I greatly appreciate your help but this will be put on hold until we migrate our system (financial risk management) to Visual Studio 2012 sometime in the next month or so. I'd like to get this done soon but we have our hands full with other pressing issues. As you probably know, the work never ends. In a nutshell, I want to download settlement data for certain contract/date combinations, convert to .csv and use in our application. One more item for the endless queue! :)

Zoltán Zörgő 29-May-13 17:16pm

Ok. But how do you want to tell that application which one is that certain combination?

Tom Paronis 29-May-13 17:27pm

That will depend on the user's data. Right now we have users who want the data for Brent Crude and Russell 2000 contracts. There will be some kind of crude query made based on the customer's data files. Haven't quite figured that one out yet.