Click here to Skip to main content
15,568,681 members
Please Sign up or sign in to vote.
0.00/5 (No votes)
See more:
hello and good day programmers,
i have a problem that is kind of difficult to solve on my own. what i want to achieve is getting news from different source and then updating the observableCollection with the returned json. the probem now is that i am getting the request one by one and then updating the observableCollection which i used as binding to my listview but it makes the program do heavy works and i dont like the way it updates in the listview.
so please what is the best way to get the news from different sources and added to the listview at same time or what is the best way to achieve this..
i am making more than one httprequest which i guess is not the right step to take. please help

What I have tried:

using (var httpClient = new HttpClient())
                    {
                        httpClient.DefaultRequestHeaders.TryAddWithoutValidation("Accept", "application/json");
                        httpClient.DefaultRequestHeaders.TryAddWithoutValidation("User-Agent", "Mozilla/5.0 (Windows NT 6.2; WOW64; 
                        rv:19.0) Gecko/20100101 Firefox/19.0");
                        response2 = await httpClient.GetAsync("https://website.com/feed/");
                    }
                        string stud6 = await response2.Content.ReadAsStringAsync().ConfigureAwait(false);
                        Article obj6 = JsonConvert.DeserializeObject<Article>(stud6);
                    // _Feeds.Clear();
                     foreach (var kvp in obj6.items)
                    {
                       
                        var feed = new ArticleModel()
                        {
                            Title = kvp.title,
                            pubDate = kvp.pubDate.Replace("+0000", ""),
                            Link = kvp.link,
                            Description = kvp.description,
                            Content = kvp.content,
                            Author = kvp.author,
                            Thumbnail = kvp.thumbnail,
                        };
                        _Feeds.Add(feed);
                        //await RefreshDB();
                    }

<pre>using (var httpClient = new HttpClient())
                    {
                        httpClient.DefaultRequestHeaders.TryAddWithoutValidation("Accept", "application/json");
                        httpClient.DefaultRequestHeaders.TryAddWithoutValidation("User-Agent", "Mozilla/5.0 (Windows NT 6.2; WOW64; 
                        rv:19.0) Gecko/20100101 Firefox/19.0");
                        response2 = await httpClient.GetAsync("https://website.com/feed/");
                    }
                        string stud6 = await response2.Content.ReadAsStringAsync().ConfigureAwait(false);
                        Article obj6 = JsonConvert.DeserializeObject<Article>(stud6);
                    // _Feeds.Clear();
                     foreach (var kvp in obj6.items)
                    {
                       
                        var feed = new ArticleModel()
                        {
                            Title = kvp.title,
                            pubDate = kvp.pubDate.Replace("+0000", ""),
                            Link = kvp.link,
                            Description = kvp.description,
                            Content = kvp.content,
                            Author = kvp.author,
                            Thumbnail = kvp.thumbnail,
                        };
                        _Feeds.Add(feed);
                        //await RefreshDB();
                    }

<pre>using (var httpClient = new HttpClient())
                    {
                        httpClient.DefaultRequestHeaders.TryAddWithoutValidation("Accept", "application/json");
                        httpClient.DefaultRequestHeaders.TryAddWithoutValidation("User-Agent", "Mozilla/5.0 (Windows NT 6.2; WOW64; 
                        rv:19.0) Gecko/20100101 Firefox/19.0");
                        response2 = await httpClient.GetAsync("https://website.com/feed/");
                    }
                        string stud6 = await response2.Content.ReadAsStringAsync().ConfigureAwait(false);
                        Article obj6 = JsonConvert.DeserializeObject<Article>(stud6);
                    // _Feeds.Clear();
                     foreach (var kvp in obj6.items)
                    {
                       
                        var feed = new ArticleModel()
                        {
                            Title = kvp.title,
                            pubDate = kvp.pubDate.Replace("+0000", ""),
                            Link = kvp.link,
                            Description = kvp.description,
                            Content = kvp.content,
                            Author = kvp.author,
                            Thumbnail = kvp.thumbnail,
                        };
                        _Feeds.Add(feed);
                        //await RefreshDB();
                    }
Posted
Updated 7-Nov-22 18:52pm

1 solution

Here is a DownloadService with a batching mode (10 at a time) for grabbing files, like images.

C#
public class DownloadService
{
    public string SavePath { get; set; }
    private IHttpClientFactory _httpFactory { get; }

    public DownloadService(IHttpClientFactory httpFactory)
        => _httpFactory = httpFactory;

    public async Task<bool> DownloadToFileAsync(string url, string filePath)
    {
        Console.WriteLine($"Downloading: {url} as {filePath}");
        filePath = Path.Combine(SavePath, filePath);

        var request = new HttpRequestMessage(HttpMethod.Get, url);
 
        var client = _httpFactory.CreateClient();
        var response = await client.SendAsync(request);

        if (response.IsSuccessStatusCode)
        {
            try
            {
                await using Stream webStream = await response.Content.ReadAsStreamAsync();
                await using FileStream fileStream = new FileStream(filePath, FileMode.Create, FileAccess.Write,
                    FileShare.None,
                    bufferSize: 64 * 1024, useAsync: true);

                await webStream.CopyToAsync(fileStream).ConfigureAwait(false);

            }
            catch (Exception ex)
            {

                throw;
            }
            
            return true;
        }

        Console.WriteLine($"StatusCode: {response.StatusCode} for {url}");
        return false;
    }

    public async Task<bool> BulkDownloadToFileAsync(Dictionary<string, string> jobs)
    {
        // note: this is considered spamming requests, not a good idea if a single website/server...
        //    return (await Task.WhenAll(jobs.Select(job
        //                => DownloadToFileAsync(job.Value, job.Key)))
        //            .ConfigureAwait(false))
        //        .All(x => x);

        // note: a better option is to batch requests
        var batches = jobs.Chunk(10);

        foreach (var batch in batches)
        {
            bool success = (await Task.WhenAll(batch.Select(job => DownloadToFileAsync(job.Value, job.Key)))
                    .ConfigureAwait(false))
                .All(x => x);

            if (!success)
                Debugger.Break();

            // pause before continuing
            await Task.Delay(200).ConfigureAwait(false);
        }

        return true;
    }
}

To use:
C#
var builder = new HostBuilder()
    .ConfigureServices((hostContext, services) =>
    {
        services.AddHttpClient();
        services.AddTransient<DownloadService>();
    }).UseConsoleLifetime();

var host = builder.Build();

using var serviceScope = host.Services.CreateScope();

var services = serviceScope.ServiceProvider;

var jobs = new Dictionary<string, string>
{
    ["local_save_filename"] = "remote_url_to_file",
    ["local_save_filename"] = "remote_url_to_file",
    ["local_save_filename"] = "remote_url_to_file",
    // etc...
};

var downloadService = services.GetRequiredService<DownloadService>();
downloadService.SavePath = @"path_to_save_downloads";

var success = await downloadService
    .BulkDownloadToFileAsync(jobs)
    .ConfigureAwait(false);

Notes:
* Best practice is to use the HttpClientFactory when working with the HttpClient - ref: HttpClient guidelines for .NET - .NET | Microsoft Learn[^]
* The above code is for demonstration purposes only and is not production ready.

I am sure that you get the idea and can adapt to your needs.

Hope this helps!
 
Share this answer
 
v3
Comments
sergeibystrov 8-Nov-22 3:14am    
using the "jobs.Chunk" it has a red underlining but i don't know how to resolve the namespace. for the IHttpClientFactory i installed Microsoft.Extension.Http
Graeme_Grant 8-Nov-22 6:15am    
It is a working solution. I am using DotNet 6.0 & you will need the following Nuget packages: Microsoft.Extensions.Hosting & Microsoft.Extensions.Http
sergeibystrov 13-Nov-22 13:24pm    
Can't find microsoft.extension.hosting I need something neat and not complicated as this as with this I will still have to reshape code to suite my case which is not downloading but grabbing Jason strings
Graeme_Grant 13-Nov-22 15:51pm    
It is a core library for Dot Net. Nuget will serve it up for you.

This content, along with any associated source code and files, is licensed under The Code Project Open License (CPOL)



CodeProject, 20 Bay Street, 11th Floor Toronto, Ontario, Canada M5J 2N8 +1 (416) 849-8900