Programmatically make(fake ;)) HTTP Requests using WebRequest, WebResponse and WebClient

There is no doubt that Microsoft .NET provides us with wonderful APIs to interact with a variety of interfaces/entities.
One such API/library exists in the System.Net namespace. WebRequest and WebResponse are two such libraries which help us to programmatically make HTTP Requests and analyse the Responses in the code itself.

It's actually pretty straightforward making GET and POST Requests and analysing the corresponding responses from C# code.

I’ll try to discuss the most common methods to accomplish the same.

HTTP GET Request

An HTTP Get Request is simple to make using WebRequest class. An example of one of the requests is as under:

public static string HttpGet(string URI)
{
System.Net.WebRequest req = System.Net.WebRequest.Create(URI);
//req.Proxy = new System.Net.WebProxy(ProxyString, true); //true means no proxy
System.Net.WebResponse resp = req.GetResponse();
System.IO.StreamReadersr = new System.IO.StreamReader(resp.GetResponseStream());
return sr.ReadToEnd().Trim();
}

Just passing the URL for the HTTP Request returns the response in string format. Ok, that was simple; now let's make an HTTP Post call from C#.

HTTP POST Request

Making an HTTPPost request is pretty straight forward, if you know what an “HTTP Post” is. An HTTP Post is basically any request which posts some form data (aggregated from User input).

Usually, this form data is sent as a concatenated string of name=value& pairs.

Apart from the Form Data which is sent as part of the HTTP request body, there are other parameters as well which make up a POST request, such as cookies and HTTP header data.

The components which make an HTTP request can be easily identified using tools such as Fiddler. In case you don’t know, Fiddler is a web debugging software which can be used to intercept and analyze HTTP traffic originating from our system.

I’ll cover a few details on how to use Fiddler in the last section of this post. But I guess the below function would hold you in good stead to get you started.

public static string HttpPost(string URI, string Parameters)
{
System.Net.WebRequest req = System.Net.WebRequest.Create(URI);
  //req.Proxy = new System.Net.WebProxy(ProxyString, true);
//Add these, as we’re doing a POST
req.ContentType = "application/x-www-form-urlencoded";
req.Method = "POST";
//req.Timeout = 200000;

/*This is optional*/

(req as HttpWebRequest).Referer = @"https://someURL/";
(req as HttpWebRequest).UserAgent = @"Mozilla/5.0 (Windows NT 6.2; WOW64) 
     AppleWebKit/537.36 (KHTML, like Gecko) Chrome/29.0.1547.76 Safari/537.36″;
(req as HttpWebRequest).Headers.Add("Origin", @"https://someURL/");

/*This is optional*/

//We need to count how many bytes we’re sending. Post’ed Faked Forms should be name=value&
byte[] bytes = System.Text.Encoding.ASCII.GetBytes(Parameters);
req.ContentLength = bytes.Length;
System.IO.Stream os = req.GetRequestStream();
os.Write(bytes, 0, bytes.Length); //Push it out there
os.Close();
System.Net.WebResponse resp;
resp = req.GetResponse();
System.IO.StreamReader sr = new System.IO.StreamReader(resp.GetResponseStream());
return sr.ReadToEnd().Trim();
}

Download File Request

Sometimes, we might just want to download a file from a URL. This usually a simple GET request and involves use of another function of the library System.Net. Below is a simple implementation of the same.

WebClientclient = new WebClient();

client.DownloadFile(URL, Path.Combine("<directory Path>", "FileName.<extension>"));

The above is a very simple implementation of a file download scenario. We can also make an asynchronous file download request using the above method, which might be a more practical scenario.

HTTP Response Analysis

Fiddler is an excellent utility for analysis HTTP traffic originating from your system. As I have always been saying, it's always a good idea to get under the layer of abstraction Microsoft has blinded us with and actually be able to see how the “web” works. You’ll be able to fully exploit the libraries provided by Microsoft in System.Net only if you have a clear understanding of the HTTP traffic.

I’ll try to introduce (strictly for newbies :P) you to a very handy tool (fiddler) for accomplishing the same. (I know it may sound like Hand-holding to some, BUT hey, who doesn’t need it every now and then.)

First, download and install Fiddler from the link Fiddler.

Open up Fiddler. You’ll be presented with a screen as below:

The Left pane is where you’ll be able to see the overview all the HTTP Traffic. The Right pane shows the details.

Let's open up http://chinmoymohanty.com on our browser. On inspecting in Fiddler, you’ll find the below output:

The Left pane shows all the HTTP Requests which might have been made when we hit the website URL. Clicking on one of the rows brings out the Request (top-right pane) and Response (bottom-right pane). Clicking on the “Raw” tab would show the actual HTTP Request/Response (You might need to decode the response if it was encoded; don’t worry, Fiddler will alert you about that).

As is quite easy to understand here, using data from the Fiddler analysis, you could easily construct/fake and HTTP Post/Get call. Beware, in some cases, the server might be expecting encoded data, so it might not work in some rare scenarios.

Hope this post was informative enough and would provide the needed kick for beginner programmers to code/automate HTTP calls.

Please shout out your views in the comments section.