Click here to Skip to main content
Rate this: bad
good
Please Sign up or sign in to vote.
See more: C# VB.NET
I have to open a web page (https), enter username and password, click some links to get to a desired page and then download all files (pdf/txt) to a shared folder on our internal network.
 
I'd like to do this with windows application/Service which would run once in a day. What is the best way to do this?
Posted 20-Mar-13 1:09am
Comments
MuhammadUsman1 at 20-Mar-13 23:42pm
   
Hello Developers on(CP)
Any idea?

1 solution

Rate this: bad
good
Please Sign up or sign in to vote.

Solution 1

See the example below, this will download any sourcecode attachwed to any article from CodeProject by logging in.
 
private static void Download(string email, string password, string destination, string downloadUri)
{
 
    using (CookiesAwareWebClient client = new CookiesAwareWebClient())
    {
        NameValueCollection values = new NameValueCollection();
        values.Add("Email",email );
        values.Add("Password", password);
        values.Add("x", "10"); 
        values.Add("y", "10"); 
        values.Add("login", "login");
 
        // We authenticate first
        client.UploadValues("https://www.codeproject.com/script/Membership/LogOn.aspx?rp=%2f", values);
 
        // Now we can download

        string modifieduri = "http://www.codeproject.com/script/articles/download.aspx?file=" + downloadUri.Replace("http://www.codeproject.com","") + "&rp=";
 

        string filename = System.IO.Path.GetFileName(downloadUri);
 
        string filepathname = System.IO.Path.Combine(destination, filename);
 
        client.DownloadFile(modifieduri, filepathname);
    }
}
 

The class used(CookiesAwareWebClient) in the code above:
using System;
using System.Collections.Generic;
using System.Linq;
using System.Text;
using System.Net;
 
namespace CodeProjectDownload
{
    public class CookiesAwareWebClient : WebClient
    {
        public CookieContainer CookieContainer;
        public CookieContainer MyProperty
        {
            get { return CookieContainer; }
            set { CookieContainer = value; }
        }
        public CookiesAwareWebClient()
        {
            CookieContainer = new CookieContainer();
        }
 
        protected override WebRequest GetWebRequest(Uri address)
        {
            WebRequest request = base.GetWebRequest(address);
            ((HttpWebRequest)request).CookieContainer = CookieContainer;
            return request;
        }
    } 
}
 
Another Option(Generic):
using System;
using System.Collections.Generic;
using System.Linq;
using System.Text;
using System.Net;
using System.IO;
using System.Text.RegularExpressions;
 
namespace IndianaMedicAidAccess
{
    public class HTTHelper
    {
        private static HttpWebResponse POST(string postData, string url, string referer, string cookie)
        {
            try
            {
                byte[] byteArray = Encoding.UTF8.GetBytes(postData);
                HttpWebRequest request = (HttpWebRequest)HttpWebRequest.Create(url);
                request.Method = "POST";
                request.KeepAlive = true;
                request.AllowAutoRedirect = false;
                request.Accept = "*/*";
                request.ContentType = "application/x-www-form-urlencoded";
                if (!string.IsNullOrEmpty(cookie))
                    request.Headers.Add(HttpRequestHeader.Cookie, cookie);
                if (!string.IsNullOrEmpty(referer))
                    request.Referer = referer;
                request.ContentLength = byteArray.Length;
                request.UserAgent = "Mozilla/5.0 (Windows; U; Windows NT 6.0; en-US; rv:1.9.0.5) Gecko/2008120122 Firefox/3.0.5";
                //   request.Proxy = null;
                Stream dataStream = request.GetRequestStream();
                dataStream.Write(byteArray, 0, byteArray.Length);
                dataStream.Close();
                try
                {
                    return (HttpWebResponse)request.GetResponse();
                }
                catch
                {
                    return (HttpWebResponse)null;
                }
            }
            catch
            {
                return (HttpWebResponse)null;
            }
        }
        public static string ProcessLogin(string postData, string LoginUrl, string Referer)
        {
            string LoginStatus = "Unsucessful";
            string uri = string.Empty;
            string cookie = string.Empty;
            HttpWebResponse response = POST(postData, LoginUrl, Referer, "");
            if (response == null)
            {
                return LoginStatus;
            }
 
            WebHeaderCollection headers = response.Headers;
 
            if ((response.StatusCode == HttpStatusCode.Found) ||
                    (response.StatusCode == HttpStatusCode.Redirect) ||
                    (response.StatusCode == HttpStatusCode.Moved) ||
                    (response.StatusCode == HttpStatusCode.MovedPermanently))
            {
 

                if (headers["Set-Cookie"] != null)
                {
 
                    string cookies = headers["Set-Cookie"];
                    List<string> fieldList = cookies.Split(',').ToList();
                    response.Close();
                    foreach (string str in fieldList)
                    {
                        try
                        {
                            if (str.IndexOf("=") > -1)
                            {
                                string mystr = str.Split(new string[] { ";" }, StringSplitOptions.RemoveEmptyEntries)[0];
                                cookie += mystr + ";";
                            }
                        }
                        catch { }
                    }
                    cookie = cookie.LastIndexOf(";") == cookie.Length - 1 ? cookie.Substring(0, cookie.Length - 1) : cookie;
                    if (string.IsNullOrEmpty(cookie))
                    {
                        String[] fields = Regex.Split(cookies, ";\\s*");
                        cookie = fields[0];
                       
                    }
                }
            }
            return cookie;
 
        }
    }
    class Program
    {
        static void Main(string[] args)
        {
            string LoginUrl = "https://interchange.indianamedicaid.com/Administrative/LogonValidate.aspx";
            string Referer="https://interchange.indianamedicaid.com/Administrative/logon.aspx?Logon=Y";
            string _userName = "youruserId"; // please change
            string _password = "passwd"; // please change
            string downloadUri = "https://xfile.indianamedicaid.com/......."; // please change
            string destination = Path.Combine("c:\\Somefolder", "FileName.ext");  //please change
            string postData = String.Format("logonid={1}&logonpswd={2}&Log On=Log On", _userName, _password);
            
            string cookie = HTTHelper.ProcessLogin(postData, LoginUrl, Referer);
 
            WebClient wb = new WebClient();
            wb.Headers.Add(HttpRequestHeader.Cookie, cookie);
            wb.DownloadFile(downloadUri, destination);
        }
    }
}
  Permalink  
v3
Comments
aspnet_regiis -i at 21-Mar-13 0:48am
   
Good job +5.This should help
MuhammadUsman1 at 21-Mar-13 1:27am
   
Thanks Dear.
it is working for code project. but on which website i am working there is no ID for UserName and Password and the Sign On i an image i am not logging in there. What i try there?
Kuthuparakkal at 21-Mar-13 1:48am
   
Please post the url you are trying to login/download
MuhammadUsman1 at 21-Mar-13 4:09am
   
Using code project when i log in. I want to download latest 3 Articles on daily basis. tell how to get links and download selected Articles.
 
And thanks again Dear. when feel free please reply i am waiting for your Reply.
Kuthuparakkal at 21-Mar-13 4:45am
   
Updated solution with more generic idea, please check!
MuhammadUsman1 at 22-Mar-13 0:33am
   
Thanks Dear.
I am not good in english and i hope you understand my scenario.
In my case when i login on website there are some links available there. i want to download pdf/txt formate file by click on those links. but those links are changed on daily basis and the files name also changed on daily basis.
After completion of downloading of any one file the link will be removed/hide from website.
Kuthuparakkal at 22-Mar-13 0:39am
   
I do not find any issue with your english... it's good!
So you do this:
1. First Login(Process Login) and get cookie.
2. Pass the cookie to make "GET" request of the page you see the links.
3. Take the source of it and search for links using known pattern and store it as a List.
4. Enumerate links and pass it download process(webclient used here in the example) and get it downloaded.
 
If time permitting I may post some example codes
Renju Vinod at 21-Mar-13 4:50am
   
+5
Kuthuparakkal at 21-Mar-13 4:57am
   
Thanks Renju

This content, along with any associated source code and files, is licensed under The Code Project Open License (CPOL)

  Print Answers RSS
0 OriginalGriff 275
1 DamithSL 265
2 CPallini 235
3 Maciej Los 185
4 George Jonsson 170
0 OriginalGriff 5,305
1 DamithSL 4,382
2 Maciej Los 3,760
3 Kornfeld Eliyahu Peter 3,470
4 Sergey Alexandrovich Kryukov 2,901


Advertise | Privacy | Mobile
Web03 | 2.8.141216.1 | Last Updated 21 Mar 2013
Copyright © CodeProject, 1999-2014
All Rights Reserved. Terms of Service
Layout: fixed | fluid

CodeProject, 503-250 Ferrand Drive Toronto Ontario, M3C 3G8 Canada +1 416-849-8900 x 100