Click here to Skip to main content
15,877,051 members
Please Sign up or sign in to vote.
2.00/5 (2 votes)
help me to crawl website ,scarpping website in asp.net c#,
i am unable to get data while i am trying to scrap website

HttpWebRequest newRequest ;

newRequest = (HttpWebRequest)WebRequest.Create("http://flights.makemytrip.com/makemytrip/search/O/O/E/1/0/0/S/V0/DEL_BOM_14-08-2015?intid=homepage_Widget_Search_New-Delhi_Mumbai");
newRequest.Method = WebRequestMethods.Http.Get;
newRequest.ContentType = "text/html;charset=ISO-8859-1;application/json";

newRequest.UserAgent = "Mozilla/5.0 (Windows NT 6.1; WOW64; rv:36.0) Gecko/20100101 Firefox/36.0";
newRequest.Referer = finalUrlRef;
response = (HttpWebResponse)newRequest.GetResponse();
string readerNewRequest = new StreamReader(response.GetResponseStream()).ReadToEnd();
Posted
Updated 2-Apr-15 9:04am
v3
Comments
Sergey Alexandrovich Kryukov 2-Apr-15 16:49pm    
And what's the problem? One thing: you are pretending that your request content is JSON, but sending any JSON.
—SA
George Swan 2-Apr-15 17:07pm    
The URL string looks incorrect, try :-
"http://flights.makemytrip.com/makemytrip/search/O/O/E/1/0/0/S/V0/DEL_BOM_14-08-2015?intid=homepage_Widget_Search_New-Delhi_Mumbai"
Purushottam Kumar 4-Apr-15 10:11am    
text/html;charset=ISO-8859-1;application/json means you are sending any of this type of information to the request and you want any of them. but when i send it i was unable to get page load data in response.

and this url is ok

1 solution

Look (or look again) at the code sample doing the same thing: https://msdn.microsoft.com/en-us/library/system.net.webrequest%28v=vs.110%29.aspx.

Find 10 differences. :-)

Or, look at my comment to the question. Why "application/json"? You are not sending any JSON, you are not sending any content at all. Maybe you mix up this content type and content type of expected HttpWebResponse, but why? It makes no sense. If you need that, use response.ContentType:
https://msdn.microsoft.com/en-us/library/system.net.httpwebresponse.contenttype(v=vs.110).aspx,
https://msdn.microsoft.com/en-us/library/system.net.httpwebresponse%28v=vs.110%29.aspx.

—SA
 
Share this answer
 
Comments
Purushottam Kumar 4-Apr-15 10:11am    
text/html;charset=ISO-8859-1;application/json means you are sending any of this type of information to the request and you want any of them. but when i send it i was unable to get page load data in response.
Sergey Alexandrovich Kryukov 4-Apr-15 17:19pm    
You are not sending it, according to your code...
—SA

This content, along with any associated source code and files, is licensed under The Code Project Open License (CPOL)



CodeProject, 20 Bay Street, 11th Floor Toronto, Ontario, Canada M5J 2N8 +1 (416) 849-8900