How to use HttpWebRequest and HttpWebResponse in .NET

Introduction

Internet applications can be classified broadly into two kinds: client applications that request information, and server applications that respond to information requests from clients. The classic Internet client-server application is the World Wide Web, where people use browsers to access documents and other data stored on Web servers worldwide.

Applications are not limited to just one of these roles; for instance, the familiar middle-tier application server responds to requests from clients by requesting data from another server, in which case it is acting as both a server and a client.

The client application makes a request by identifying the requested Internet resource and the communication protocol to use for the request and response. If necessary, the client also provides any additional data required to complete the request, such as proxy location or authentication information (user name, password, and so on). Once the request is formed, the request can be sent to the server.

Identifying Resources

The .NET Framework uses a Uniform Resource Identifier (URI) to identify the requested Internet resource and communication protocol. The URI consists of atleast three, and possibly four, fragments: the scheme identifier, which identifies the communications protocol for the request and response; the server identifier, which consists of either a Domain Name System (DNS) host name or a TCP address that uniquely identifies the server on the Internet; the path identifier, which locates the requested information on the server; and an optional query string, which passes information from the client to the server. For example, the URI "http://www.MyWebSite.com/whatsnew.aspx?date=today" consists of the scheme identifier "http", the server identifier "www. MyWebSite.com", the path "/whatsnew.aspx", and the query string "?date=today".

After the server has received the request and processed the response, it returns the response to the client application. The response includes supplemental information, such as the type of the content (raw text or XML data, for example).

Developing applications that run in the distributed operating environment of today's Internet requires an efficient, easy-to-use method for retrieving data from resources of all types. Pluggable protocols let you develop applications that use a single interface to retrieve data from multiple Internet protocols.

The .NET Framework uses specific classes to provide the three pieces of information required to access Internet resources through a request/response model: the Uri class, which contains the URI of the Internet resource you are seeking; the HttpWebRequest class, which contains a request for the resource; and the HttpWebResponse class, which provides a container for the incoming response.

Here, I am going to show with practical code, how to send request and receive response to Internet/intranet sites. I am assuming that the reader possesses basic knowledge of C# and Visual Studio.

Open a Visual studio C# class library project, copy and paste the following code up to the part where it says “End of Base Class”:

using System;
using System.Collections.Specialized;
using System.Net;
using System.Text;
using System.IO;
 
namespace BaseClassNameSpace.Web.BaseServices
{
/// <summary>
///This base class provides implementation of request 
///and response methods during Http Calls.
/// </summary>
  
      public class HttpBaseClass
      {
 
            private string UserName;
            private string UserPwd;
            private string ProxyServer;
            private int ProxyPort;
            private string Request;
            
            public HttpBaseClass(string HttpUserName, 
              string HttpUserPwd, string HttpProxyServer, 
              int HttpProxyPort, string HttpRequest)
            {
                  UserName = HttpUserName;
                  UserPwd = HttpUserPwd;
                  ProxyServer = HttpProxyServer;
                  ProxyPort = HttpProxyPort;
                  Request = HttpRequest;
            }
 
/// <summary>
/// This method creates secure/non secure web
/// request based on the parameters passed.
/// </summary>
/// <param name="uri"></param>
/// <param name="collHeader">This parameter of type
///    NameValueCollection may contain any extra header
///    elements to be included in this request      </param>
/// <param name="RequestMethod">Value can POST OR GET</param>
/// <param name="NwCred">In case of secure request this would be true</param>
/// <returns></returns>
            public virtual HttpWebRequest CreateWebRequest(string uri, 
              NameValueCollection collHeader, 
              string RequestMethod, bool NwCred)
            {
                  HttpWebRequest webrequest = 
                   (HttpWebRequest) WebRequest.Create(uri);
                  webrequest.KeepAlive = false;
                  webrequest.Method = RequestMethod;
 
                  int iCount = collHeader.Count;
                  string key;
                  string keyvalue;
 
                  for (int i=0; i < iCount; i++)
                  {
                        key = collHeader.Keys[i];
                        keyvalue = collHeader[i];
                        webrequest.Headers.Add(key, keyvalue);
                  }
                  
                  webrequest.ContentType = "text/html"; 
                  //"application/x-www-form-urlencoded";
 
                  if (ProxyServer.Length > 0)
                  {
                     webrequest.Proxy = new 
                      WebProxy(ProxyServer,ProxyPort);
                  }
                  webrequest.AllowAutoRedirect = false;
 
                  if (NwCred)
                  {
                      CredentialCache wrCache = 
                              new CredentialCache();
                      wrCache.Add(new Uri(uri),"Basic",
                        new NetworkCredential(UserName,UserPwd));
                      webrequest.Credentials = wrCache;
                  }
                  //Remove collection elements
                  collHeader.Clear();
                  return webrequest;
            }//End of secure CreateWebRequest
            
            /// <summary>
            /// This method retreives redirected URL from
            /// response header and also passes back
            /// any cookie (if there is any)
            /// </summary>
            /// <param name="webresponse"></param>
            /// <param name="Cookie"></param>
            /// <returns></returns>
            public virtual string GetRedirectURL(HttpWebResponse 
                 webresponse, ref string Cookie)
            {
                  string uri="";
 
                  WebHeaderCollection headers = webresponse.Headers;
                  
                  if ((webresponse.StatusCode == HttpStatusCode.Found) || 
                    (webresponse.StatusCode == HttpStatusCode.Redirect) ||
                    (webresponse.StatusCode == HttpStatusCode.Moved) ||
                    (webresponse.StatusCode == HttpStatusCode.MovedPermanently))
                  {
                        // Get redirected uri
                        uri = headers["Location"] ;
                        uri = uri.Trim();
                  }
 
                  //Check for any cookies
                  if (headers["Set-Cookie"] != null)
                  {
                        Cookie = headers["Set-Cookie"];  
                  }
                  //                string StartURI = "http:/";
//                if (uri.Length > 0 && uri.StartsWith(StartURI)==false)
//                {
//                      uri = StartURI + uri;
//                }
                  return uri;
            }//End of GetRedirectURL method
 
 
            public virtual string GetFinalResponse(string ReUri, 
             string Cookie, string RequestMethod, bool NwCred)
            {
                  NameValueCollection collHeader = 
                        new NameValueCollection();
 
                  if (Cookie.Length > 0)
                  {
                        collHeader.Add("Cookie",Cookie);
                  }
 
                  HttpWebRequest webrequest = 
                    CreateWebRequest(ReUri,collHeader, 
                    RequestMethod, NwCred);
                        
                  BuildReqStream(ref webrequest);
 
                  HttpWebResponse webresponse;
 
                  webresponse = (HttpWebResponse)webrequest.GetResponse();
 
                  Encoding enc = System.Text.Encoding.GetEncoding(1252);
                  StreamReader loResponseStream = new 
                    StreamReader(webresponse.GetResponseStream(),enc);
 
                  string Response = loResponseStream.ReadToEnd();
 
                  loResponseStream.Close();
                  webresponse.Close();
                  
                  return Response;
            }
 
            private void BuildReqStream(ref HttpWebRequest webrequest)
                  //This method build the request stream for WebRequest
            {
                  byte[] bytes = Encoding.ASCII.GetBytes(Request);
                  webrequest.ContentLength=bytes.Length;
 
                  Stream oStreamOut = webrequest.GetRequestStream();
                  oStreamOut.Write(bytes,0,bytes.Length);
                  oStreamOut.Close();
            }
      }
}//End of HttpBaseClass class
 
//"End of Base Class"

Save the above file with the name, say, HttpBaseClass.cs.

Let's go through the various pieces of the above program.

System.Net namespace provides HttpWebRequest and HttpWebResponse classes. I already explained about these classes in the beginning.

Sytem.IO namespace provides classes to send request and receive response in streams.

This is the base class, which provides methods to request data, parse any redirection URL, receive response and convert response (since it's coming in streams) into meaningful data. From this class, you will create a derived class (wait, that would be next step until we finish this class) and you can override the public methods (that’s why they are defined with virtual keyword). Let's go to the actual working of the base class code.

When you create an instance of this class, you will need to provide certain parameters to the constructor. Parameters are:

• string HttpUserName: User name
• string HttpUserPwd: User password

Above two variables are required for secure sites, which require User ID and password to be sent along with the request header. Usually, for secure sites, request header is like this:

Authorization: Basic NNNNNNNNNNNNNNNNNN==

Authorization is the property, Basic is the encoding type and NNNNNNNNNNNNNNNNNN== is the actual user ID and password encoded in Base64 format.

• string HttpProxyServer, int HttpProxyPort: If your request is going to an external site of your corporate networks, then probably you would require proxy server name and proxy port number (usually 8080). Check your browser settings for these properties and pass them accordingly to the constructor. You can pass empty string and 0 if you are connecting to an intranet site.
• String HttpRequest: Request Body. It can be an XML or other request text.

Let's see what’s happening under the hood of CreateWebRequest method. This method is the first step in creating request with the target URL. Parameters to be passed in this method are:

• String uri: Target URI (E.g.: Google.com, Yahoo.com, yourownintranetsite.com)
• NameValueCollection collHeader: A variable of type namevaluecollection. If you see in the method at one line (in the for loop), if additional headers are required, they are being extracted from this variable and inserted into request header. For example, your requested site may require additional cookies to be sent in the header.
• String RequestMethod: It can be “POST” or “GET”.
• bool NwCred: This variable value is set based on if the site requires userID/password (if it's secure site) or not. See in the code if its true, then using CredentialCache and NetworkCredential classes, user ID and password are passed in Base64 format. Right now, I have hard coded security type as “Basic”, but you can set other security settings such as Digest, Kerberos, NTLM, depending on requirements.

Line HttpWebRequest webrequest = (HttpWebRequest) WebRequest.Create(uri)

Creates a HttpWebRequest instance.

If you are hitting external site, outside of your corporate intranet, then this line would be executed based on if ProxyServer variable contains server value or is being set to empty string in the constructor at the time of creation of an instance of this class.

webrequest.Proxy = new WebProxy(ProxyServer,ProxyPort);

Setting webrequest.AllowAutoRedirect = false is for handling of redirection(s) by yourself and that’s why it's set to false.

I have a question here for any geeks who read this article. When I was setting it to true, I was getting error message, but if I used ServerXMLHttp (COM way), every redirection was handled automatically (mystery to me)??????

Let's see the other method GetRedirectURL: this method is invoked to check if any redirection or cookies are coming back after target URI is hit. There may be cases when there is no redirection but still next request to URI expects some cookies back. For example, the site where my application hits now passes back redirection URI and cookies (which contains session ID). I hit redirected URI again with cookie in the header and then I get the final response. This is quite evident in GetFinalResponse method in line:

if (Cookie.Length > 0)
{
  collHeader.Add("Cookie",Cookie);
}

This collHeader namevalue collection is again passed to CreateWebRequest method. Let me explain the parameters passed in GetFinalResponse method:

• string ReUri: Redirected URI. If there is no redirection, original URI is passed.
• string Cookie: If there is any cookie from method GetRedirectURL.
• string RequestMethod: “POST” or “GET”. If you think there will be some redirected URI, then initially (when CreateWebRequest method is called) you get GET as request method, and in final response, you can say “POST” method, depending on your requirements.
• bool NwCred: As explained in method CreateWebRequest.

Within this method, BuildReqStream method is called which builds the request stream for WebRequest. In Internet communication, request text is built in the form of streams and so is response. When response comes back in the stream, you convert it back into original format. See the part:

Encoding enc = System.Text.Encoding.GetEncoding(1252);

1252 is encoding for Windows format. Please read more about encoding and streams in MSDN.

StreamReader loResponseStream = 
   new StreamReader(webresponse.GetResponseStream(),enc);
string Response = loResponseStream.ReadToEnd();

Before this part, you see we are declaring:

HttpWebResponse webresponse;
webresponse = (HttpWebResponse)webrequest.GetResponse();

In these 2 lines, we are actually getting the response after we have created request in CreateWebRequest method. Likewise, HttpWebRequest class is there for creating request, so is the HttpWebResponse for getting response.

Now, let's see how we actually use the above class.

Open a new class library file in the same project, copy and paste the following code:

using System;
using System.Collections.Specialized;
using System.Net;
using System.Text;
using System.IO;
using BaseClassNameSpace.Web.BaseServices;
 
namespace DeriveClassNameSpace.Services.Web
{
 
      public class HttpRequestResponse
      {
            private string URI;
            private string Request;
            private string UserName;
            private string UserPwd;
            private string ProxyServer;
            private int ProxyPort;
            private string RequestMethod = "GET";
 
            public HttpRequestResponse(string pRequest, 
                                 string pURI)//Constructor
            {
                  Request = pRequest;
                  URI = pURI;
            }
            
            public string HTTP_USER_NAME
            {
                  get 
                  {
                        return UserName; 
                  }
                  set 
                  {
                        UserName = value; 
                  }
            }

            public string HTTP_USER_PASSWORD
            {
                  get 
                  {
                        return UserPwd; 
                  }
                  set 
                  {
                        UserPwd = value; 
                  }
            }
 
            public string PROXY_SERVER
            {
                  get 
                  {
                        return ProxyServer; 
                  }
                  set 
                  {
                        ProxyServer = value; 
                  }
            }
 
            public int PROXY_PORT
            {
                  get 
                  {
                        return ProxyPort; 
                  }
                  set 
                  {
                        ProxyPort = value; 
                  }
            }
 
            public string SendRequest()
            /*This public interface receives the request 
            and send the response of type string. */
            {
              string FinalResponse="";
              string Cookie="";
 
              NameValueCollection collHeader = new NameValueCollection();

              HttpWebResponse webresponse;
 
              HttpBaseClass BaseHttp =  new 
                HttpBaseClass(UserName,UserPwd,
                ProxyServer,ProxyPort,Request);
              try
              {
                HttpWebRequest webrequest = 
                  BaseHttp.CreateWebRequest(URI, 
                  collHeader, RequestMethod, true);
                webresponse = 
                 (HttpWebResponse)webrequest.GetResponse();
 
                string ReUri = 
                  BaseHttp.GetRedirectURL(webresponse,
                  ref Cookie);
                //Check if there is any redirected URI.
                webresponse.Close();
                ReUri = ReUri.Trim();
                if (ReUri.Length == 0) //No redirection URI
                {
                  ReUri = URI;
                }
                RequestMethod="POST";
                FinalResponse = BaseHttp.GetFinalResponse(ReUri, 
                                   Cookie, RequestMethod, true);
 
              }//End of Try Block
 
              catch (WebException e)
              {
                 throw CatchHttpExceptions(FinalResponse = e.Message);
              }
              catch (System.Exception e)
              {
                 throw new Exception(FinalResponse = e.Message);
              }
              finally
              {
                 BaseHttp=null;
              }
              return FinalResponse;
            } //End of SendRequestTo method
 
            private WebException CatchHttpExceptions(string ErrMsg)
            {
              ErrMsg = "Error During Web Interface. Error is: "+ErrMsg ;
              return new WebException(ErrMsg);
            }
      }//End of RequestResponse Class
}

We will see the implementation of the above class in a later stage, but before that, let's consider what’s happening inside this class. In the HttpRequestResponse constructor, the 2 parameters are:

• string pRequest: Request text which would go with request body. See BuildReqStream method in HttpBaseClass.
• string pURI: Target URI (external or intranet site URI).

The public property methods are:

• HTTP_USER_NAME: UserID for secure web site.
• HTTP_USER_PASSWORD: Password for secure web site.
• PROXY_SERVER: Proxy server name required when you are trying to hit external site from your corporate network.
• PROXY_PORT: Proxy port required when you are trying to hit external site from your corporate network.

Let's see the method SendRequest: this public interface will get back the desired response you are expecting after parameters in the constructor of this class are passed. Let's see what's happening under its hood:

HttpBaseClass BaseHttp = new HttpBaseClass(UserName, 
                    UserPwd,ProxyServer,ProxyPort,Request);

The above line creates an instance of BaseHttp class (the very first class you copied and pasted).

Pass the desired parameters in BaseHttp class constructor.

Inside of try-catch block: HttpWebRequest webrequest = BaseHttp.CreateWebRequest(URI, collHeader, RequestMethod, true) creates the web request.

This part webresponse = (HttpWebResponse)webrequest.GetResponse() creates the web response.

If there is any redirection from BaseHttp, GetRedirectURL gets stored in ReUri, and also if there is any cookie that gets stored in Cookie variable.

After all validations, BaseHttp.GetFinalResponse is called by passing in the following variables:

• ReUri: Redirected or original URI.
• Cookie: If there is any.
• RequestMethod = “POST”. Remember, we discussed above that till the time we are checking for redirection, we can set request method as Get, and when we are about to call BaseHttp.GetFinalResponse method, we can set the request method to POST.
• true/false: This value entirely depends on your requirements. If redirected URI still wants secure access, then pass as true, else false. In one requirement which I had, first time URI was requiring secure hit, but in second URI, no secure request was required because it already passed me the required cookie value.

Build the assembly for the above class. Open new project and set the reference to this new DLL.

Now, you can call the method SendRequest after passing required values in the above class constructor.

Let me know if this works for you. Let me know any further queries you have. Also some sites requires SSL certificate. I have that code also and if you need them, let me know.

Thanks.