Click here to Skip to main content
15,938,848 members
Please Sign up or sign in to vote.
0.00/5 (No votes)
See more:
I have a code like this:

C#
public class WebDataDownloader
{
    public string GetJSONFromURL(string url)
    {
         string file = string.empty;
         //*********get the json file using httpRequest ***********

         try
         {
            HttpWebRequest httpWebRequest = (HttpWebRequest)WebRequest.Create(url);
            httpWebRequest.Method = WebRequestMethods.Http.Get;
            httpWebRequest.Accept = "application/json; charset=utf-8";
            //httpWebRequest.UserAgent = @"Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/80.0.3987.132 Safari/537.36";
            //ServicePointManager.SecurityProtocol = SecurityProtocolType.Ssl3 | SecurityProtocolType.Tls12 | SecurityProtocolType.Tls11 | SecurityProtocolType.Tls;

            using (var response = (HttpWebResponse)httpWebRequest.GetResponse())
            {
                using (var sr = new StreamReader(response.GetResponseStream()))
                {
                    file = sr.ReadToEnd();
                }
            }
        }
        catch(Exception exp)
        {
            Debug.WriteLine(exp.Message);
        }

        return file;
    }
}


Now I call this method multiple times in my main thread to get JSON and parse it. Something as given below:

public class MyClass
{
    WebDataDownloader myWebClient = new WebDataDownloader();
    string Json1 = myWebClient.GetJSONFromURL(@"https://www.nseindia.com/api/market-data-pre-open?key=ALL");

    //Do some action on this data

    string Json2 = myWebClient.GetJSONFromURL(@"https://www.nseindia.com/api/equity-stock?index=fu_nifty50");

    //Do some action on this data

    string Json3 = myWebClient.GetJSONFromURL(@"https://www.nseindia.com/api/liveEquity-derivatives?index=nifty_bank_fut");

    //Do some action on this data
}


But, the problem is, the first call for (Json1) using

response = (HttpWebResponse)httpWebRequest.GetResponse()


works fine, but 2nd and 3rd calls does not work in the same instance.

What I have tried:

I have gone through many posts in different forums but all those posts are quite old and I guess i have tried every trick. Tried the same with WebClient also but no success. The code that I have mentioned is a code that used to work perfectly fine till 7-8 months ago with User Agent property uncommented. It is recently, that it has started failing for multiple urls.

Can anybody please help me with this...

Thanks in advance.
Posted
Updated 27-Apr-20 4:47am

I wonder if it is because you're (re)using (bad pun) an old instance of HttpWebRequest - what would happen if you did this (may need massaging) :-

using HttpWebRequest httpWebRequest = (HttpWebRequest)WebRequest.Create(url);
{
    httpWebRequest.Method = WebRequestMethods.Http.Get;
    httpWebRequest.Accept = "application/json; charset=utf-8";

    using (var response = (HttpWebResponse)httpWebRequest.GetResponse())
    {
        using (var sr = new StreamReader(response.GetResponseStream()))
        {
            file = sr.ReadToEnd();
        }
    }
}


I was tempted to roll up the 'usings' into as few blocks as possible, but this is likely more understandable from your POV
 
Share this answer
 
You mean putting HTTPWebRequest also in the using statement?
But we cannot do it because it does not implements IDisposable.
 
Share this answer
 
To be honest, if you have the chance, I would ditch the use of HttpWebResponse altogether and go with the newer .NET approach of using HttpClient. Your code may end up looking something like this:
C#
public class HttpRequestManager
{
  private readonly IHttpClientFactory httpClientFactory;
  public HttpRequestManager(IHttpClientFactory httpClientFactory)
  {
    this.httpClientFactory = httpClientFactory ?? throw new ArgumentNullException(nameof(httpClientFactory));
  }

  public async Task<string> GetJSONFromUrlAsync(string url, CancellationToken token = CancellationToken.None)
  {
    using (var client = httpClientFactory.CreateClient());
    using (HttpRequestMessage request = new HttpRequestMessage(HttpMethod.Get, url));
    using (var response = await client.SendAsync(request, token));
    response.EnsureSuccessStatusCode();
    string rawData = await response.Content.ReadAsStringAsync();
    return rawData;
  }
}
Note that there may be some minor syntax mistakes in this as I just knocked this up in notepad to demonstrate the technique. Get this working and you have none-blocking download capacity.
 
Share this answer
 
Comments
Harisri 27-Apr-20 14:04pm    
Hi, I could get your code refined and corrected a bit some what like this:

public class HttpRequestManager
{
private readonly IHttpClientFactory httpClientFactory;

public HttpRequestManager(IHttpClientFactory httpClientFactory)
{
this.httpClientFactory = httpClientFactory ?? throw new ArgumentNullException(nameof(httpClientFactory));
}

public async Task<string> GetJSONFromUrlAsync(string url, CancellationToken cancellationToken = default(CancellationToken))
{
string rawData = string.Empty;

using (var client = httpClientFactory.CreateClient())
{
using (HttpRequestMessage request = new HttpRequestMessage(HttpMethod.Get, url))
{
using (var response = await client.SendAsync(request, cancellationToken))
{
response.EnsureSuccessStatusCode();
rawData = await response.Content.ReadAsStringAsync();
}
}
}
return rawData;
}
}

But when I have to call the method to return the data, I could not figure out how to instantiate the class you specified.

HttpRequestManager myHTTPManager = new HttpRequestManager((IHttpClientFactory)DefaultHttpClientFactory);
Task<string> jsonData = myHTTPManager.GetJSONFromUrlAsync(myUrl);

I mean how to pass arguments to the HttpRequestManager constructor. Can you please guide...
Pete O'Hanlon 29-Apr-20 5:53am    
Sorry I haven't had a chance to respond before; IHttpClientFactory is normally injected into your code if you are using .NET Core and uses the DefaultHttpClientFactory. If you aren't using dependency injection, you can remove the constructor and replace the first using statement with (using HttpClient client = new HttpClient())

By the way, the using statement format you saw in my example was the new way of adding using blocks (introduced in C# 8). You can find details here: https://docs.microsoft.com/en-us/dotnet/csharp/whats-new/csharp-8#using-declarations

This content, along with any associated source code and files, is licensed under The Code Project Open License (CPOL)



CodeProject, 20 Bay Street, 11th Floor Toronto, Ontario, Canada M5J 2N8 +1 (416) 849-8900