Click here to Skip to main content
15,899,026 members
Please Sign up or sign in to vote.
1.00/5 (1 vote)
See more:
I keep using this piece of code many, many times when downloading a webpage. I am not sure how to make this into a method or class which would shorten the lines of redundant code. Appreciate any help.

C#
HtmlAgilityPack.HtmlDocument doc = new HtmlAgilityPack.HtmlDocument();
HttpWebRequest request = (HttpWebRequest)WebRequest.Create(url);
request.AutomaticDecompression = DecompressionMethods.Deflate | DecompressionMethods.GZip;
request.UserAgent = @"Mozilla/5.0 (Windows NT 6.3; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/42.0.2311.135 Safari/537.36";
request.Accept = @"text/html";
HttpWebResponse response = (HttpWebResponse)request.GetResponse();
if (response.StatusCode == HttpStatusCode.OK)
{
    Stream dataStream = response.GetResponseStream();
    using (StreamReader streamread = new StreamReader(dataStream))
    doc.Load(streamread);
    if (doc != null)
    {
Posted
Updated 14-Jun-15 16:19pm
v2
Comments
Sergey Alexandrovich Kryukov 14-Jun-15 22:18pm    
The question does not seem to make any sense. If you really use "this piece of code" anyhow, it's already a part of some method of some class or struct.
—SA
theadmin 15-Jun-15 8:41am    
Let me try my question in a different way. I am connecting to about 50-60 webpages, using the above code I am calling every webpage like

ProcessPage1()
{
HtmlAgilityPack.HtmlDocument doc = new HtmlAgilityPack.HtmlDocument();
HttpWebRequest request = (HttpWebRequest)WebRequest.Create(url);
request.AutomaticDecompression = DecompressionMethods.Deflate | DecompressionMethods.GZip;
request.UserAgent = @"Mozilla/5.0 (Windows NT 6.3; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/42.0.2311.135 Safari/537.36";
request.Accept = @"text/html";
HttpWebResponse response = (HttpWebResponse)request.GetResponse();
if (response.StatusCode == HttpStatusCode.OK)
{
Stream dataStream = response.GetResponseStream();
using (StreamReader streamread = new StreamReader(dataStream))
doc.Load(streamread);
if (doc != null)
{
}
.............
}

ProcessPage2()
{
HtmlAgilityPack.HtmlDocument doc = new HtmlAgilityPack.HtmlDocument();
HttpWebRequest request = (HttpWebRequest)WebRequest.Create(url);
request.AutomaticDecompression = DecompressionMethods.Deflate | DecompressionMethods.GZip;
request.UserAgent = @"Mozilla/5.0 (Windows NT 6.3; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/42.0.2311.135 Safari/537.36";
request.Accept = @"text/html";
HttpWebResponse response = (HttpWebResponse)request.GetResponse();
if (response.StatusCode == HttpStatusCode.OK)
{
Stream dataStream = response.GetResponseStream();
using (StreamReader streamread = new StreamReader(dataStream))
doc.Load(streamread);
if (doc != null)
{
}
.............
}

ProcessPage3()
{
HtmlAgilityPack.HtmlDocument doc = new HtmlAgilityPack.HtmlDocument();
HttpWebRequest request = (HttpWebRequest)WebRequest.Create(url);
request.AutomaticDecompression = DecompressionMethods.Deflate | DecompressionMethods.GZip;
request.UserAgent = @"Mozilla/5.0 (Windows NT 6.3; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/42.0.2311.135 Safari/537.36";
request.Accept = @"text/html";
HttpWebResponse response = (HttpWebResponse)request.GetResponse();
if (response.StatusCode == HttpStatusCode.OK)
{
Stream dataStream = response.GetResponseStream();
using (StreamReader streamread = new StreamReader(dataStream))
doc.Load(streamread);
if (doc != null)
{
}
.............
}

Hopefully what I am asking makes sense now.

This content, along with any associated source code and files, is licensed under The Code Project Open License (CPOL)



CodeProject, 20 Bay Street, 11th Floor Toronto, Ontario, Canada M5J 2N8 +1 (416) 849-8900