Click here to Skip to main content
15,747,637 members
Please Sign up or sign in to vote.
5.00/5 (1 vote)
I use WebRequest() & WebResponse() methods from C# for scraping data from sites. It works well for unauthenticated sites. Now I want to scrape details from an authenticated sites which prompts for login details for getting into the site. The site is ''. How do I pass the credentials while requesting the page?

I used the below code,but it doesn't work:

NetworkCredential credential = new NetworkCredential("XXXX", "XXXX"); //USERNAME,PWD
         HttpWebRequest wr = (HttpWebRequest)WebRequest.Create(target_url);
wr.Credentials = credential;
wr.Method = "POST";

HttpWebResponse wrs = (HttpWebResponse)wr.GetResponse();
StreamReader sr = new StreamReader(wrs.GetResponseStream(), Encoding.GetEncoding("ISO-8859-1"), false);
         source = sr.ReadToEnd();

I also tried setting 'DefaultCredentials=true' and keeps the site opened in my firefox browser.

Please help me to get my problem solved.

Thanks & Regards,
Updated 4-Oct-10 23:49pm
E.F. Nijboer 5-Oct-10 14:48pm    
Couldn't you simply check the "auto-login" checkbox? You would otherwise have to simulate the login procedure (the part within the loginform div)

1 solution

I was working with this issue some period of time. So, first of all you should analize all kind of requests in some http analyzer and track the behaviour of required. The most difficult level is to implement an authorization because it can be work differently for every site. As usual it's connected with cookies where the login data is stored after logging in. You need to repeat this technique in your code and then you will manage to access the page.

Sending login data can be implemented in different ways too. They are query string and post parameters not just using ready-to-use NetworkCredential object.

Before to implement scraping try to look for developer API for site cause it pretty simplifies the developing.
Share this answer

This content, along with any associated source code and files, is licensed under The Code Project Open License (CPOL)

CodeProject, 20 Bay Street, 11th Floor Toronto, Ontario, Canada M5J 2N8 +1 (416) 849-8900