Httpwebresponse giving error "operation timed out"

Question

0.00/5 (No votes)

See more:

I have a code like this:

C#

public class WebDataDownloader
{
    public string GetJSONFromURL(string url)
    {
         string file = string.empty;
         //*********get the json file using httpRequest ***********

         try
         {
            HttpWebRequest httpWebRequest = (HttpWebRequest)WebRequest.Create(url);
            httpWebRequest.Method = WebRequestMethods.Http.Get;
            httpWebRequest.Accept = "application/json; charset=utf-8";
            //httpWebRequest.UserAgent = @"Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/80.0.3987.132 Safari/537.36";
            //ServicePointManager.SecurityProtocol = SecurityProtocolType.Ssl3 | SecurityProtocolType.Tls12 | SecurityProtocolType.Tls11 | SecurityProtocolType.Tls;

            using (var response = (HttpWebResponse)httpWebRequest.GetResponse())
            {
                using (var sr = new StreamReader(response.GetResponseStream()))
                {
                    file = sr.ReadToEnd();
                }
            }
        }
        catch(Exception exp)
        {
            Debug.WriteLine(exp.Message);
        }

        return file;
    }
}

Now I call this method multiple times in my main thread to get JSON and parse it. Something as given below:

public class MyClass
{
    WebDataDownloader myWebClient = new WebDataDownloader();
    string Json1 = myWebClient.GetJSONFromURL(@"https://www.nseindia.com/api/market-data-pre-open?key=ALL");

    //Do some action on this data

    string Json2 = myWebClient.GetJSONFromURL(@"https://www.nseindia.com/api/equity-stock?index=fu_nifty50");

    //Do some action on this data

    string Json3 = myWebClient.GetJSONFromURL(@"https://www.nseindia.com/api/liveEquity-derivatives?index=nifty_bank_fut");

    //Do some action on this data
}

But, the problem is, the first call for (Json1) using

response = (HttpWebResponse)httpWebRequest.GetResponse()

works fine, but 2nd and 3rd calls does not work in the same instance.

What I have tried:

I have gone through many posts in different forums but all those posts are quite old and I guess i have tried every trick. Tried the same with WebClient also but no success. The code that I have mentioned is a code that used to work perfectly fine till 7-8 months ago with User Agent property uncommented. It is recently, that it has started failing for multiple urls.

Can anybody please help me with this...

Thanks in advance.

Posted 26-Apr-20 15:15pm

Harisri

Updated 27-Apr-20 4:47am

Add a Solution

3 solutions

Add a Solution

Add your solution here

Treat my content as plain text, not as HTML

Preview 0

…

Existing Members

Sign in to your account

...or Join us

Download, Vote, Comment, Publish.

Your Email
Password
Forgot your password?

Your Email
This email is in use. Do you need your password?
Optional Password

I have read and agree to the Terms of Service and Privacy Policy
Please subscribe me to the CodeProject newsletters

When answering a question please:

Read the question carefully.
Understand that English isn't everyone's first language so be lenient of bad spelling and grammar.
If a question is poorly phrased then either ask for clarification, ignore it, or edit the question and fix the problem. Insults are not welcome.
Don't tell someone to read the manual. Chances are they have and don't get it. Provide an answer or move on to the next question.

Let's work to help developers, not make them feel stupid.

This content, along with any associated source code and files, is licensed under The Code Project Open License (CPOL)

Garth J Lancaster · Answer 1 · 2020-04-26T16:11:00

I wonder if it is because you're (re)using (bad pun) an old instance of HttpWebRequest - what would happen if you did this (may need massaging) :-

using HttpWebRequest httpWebRequest = (HttpWebRequest)WebRequest.Create(url);
{
    httpWebRequest.Method = WebRequestMethods.Http.Get;
    httpWebRequest.Accept = "application/json; charset=utf-8";

    using (var response = (HttpWebResponse)httpWebRequest.GetResponse())
    {
        using (var sr = new StreamReader(response.GetResponseStream()))
        {
            file = sr.ReadToEnd();
        }
    }
}

I was tempted to roll up the 'usings' into as few blocks as possible, but this is likely more understandable from your POV

Harisri · Answer 2 · 2020-04-26T23:20:00

Solution 2

You mean putting HTTPWebRequest also in the using statement?
But we cannot do it because it does not implements IDisposable.

Posted 26-Apr-20 23:20pm

Harisri

Pete O'Hanlon · Answer 3 · 2020-04-27T04:48:00

Solution 3

To be honest, if you have the chance, I would ditch the use of HttpWebResponse altogether and go with the newer .NET approach of using HttpClient. Your code may end up looking something like this:

C#

public class HttpRequestManager
{
  private readonly IHttpClientFactory httpClientFactory;
  public HttpRequestManager(IHttpClientFactory httpClientFactory)
  {
    this.httpClientFactory = httpClientFactory ?? throw new ArgumentNullException(nameof(httpClientFactory));
  }

  public async Task<string> GetJSONFromUrlAsync(string url, CancellationToken token = CancellationToken.None)
  {
    using (var client = httpClientFactory.CreateClient());
    using (HttpRequestMessage request = new HttpRequestMessage(HttpMethod.Get, url));
    using (var response = await client.SendAsync(request, token));
    response.EnsureSuccessStatusCode();
    string rawData = await response.Content.ReadAsStringAsync();
    return rawData;
  }
}

Note that there may be some minor syntax mistakes in this as I just knocked this up in notepad to demonstrate the technique. Get this working and you have none-blocking download capacity.

Posted 27-Apr-20 4:48am

Pete O'Hanlon

Comments

Harisri 27-Apr-20 14:04pm

Hi, I could get your code refined and corrected a bit some what like this:

public class HttpRequestManager
{
private readonly IHttpClientFactory httpClientFactory;

public HttpRequestManager(IHttpClientFactory httpClientFactory)
{
this.httpClientFactory = httpClientFactory ?? throw new ArgumentNullException(nameof(httpClientFactory));
}

public async Task<string> GetJSONFromUrlAsync(string url, CancellationToken cancellationToken = default(CancellationToken))
{
string rawData = string.Empty;

using (var client = httpClientFactory.CreateClient())
{
using (HttpRequestMessage request = new HttpRequestMessage(HttpMethod.Get, url))
{
using (var response = await client.SendAsync(request, cancellationToken))
{
response.EnsureSuccessStatusCode();
rawData = await response.Content.ReadAsStringAsync();
}
}
}
return rawData;
}
}

But when I have to call the method to return the data, I could not figure out how to instantiate the class you specified.

HttpRequestManager myHTTPManager = new HttpRequestManager((IHttpClientFactory)DefaultHttpClientFactory);
Task<string> jsonData = myHTTPManager.GetJSONFromUrlAsync(myUrl);

I mean how to pass arguments to the HttpRequestManager constructor. Can you please guide...

Pete O'Hanlon 29-Apr-20 5:53am

Sorry I haven't had a chance to respond before; IHttpClientFactory is normally injected into your code if you are using .NET Core and uses the DefaultHttpClientFactory. If you aren't using dependency injection, you can remove the constructor and replace the first using statement with (using HttpClient client = new HttpClient())

By the way, the using statement format you saw in my example was the new way of adding using blocks (introduced in C# 8). You can find details here: https://docs.microsoft.com/en-us/dotnet/csharp/whats-new/csharp-8#using-declarations