Click here to Skip to main content
Click here to Skip to main content

Tagged as

Efficient data entry through browser automation

, 29 Feb 2012 Ms-PL
Rate this:
Please Sign up or sign in to vote.
You will learn how to create a semi automated crawler, and automate browsing

Efficient data entry through browser automation

Table of content

After writing this article, I found a way to work on all computer and not just IE : using Selenium. Selenium is an interoperability layer common to several browsers. I don't like the design that is over engineered with lots of interface, features are very hard to find through intellisense, but it does a very good job ! The only limitation compared with WebBrowser is that you can't subscribe to javascript events with C# code ! (but you can execute javacript)

Introduction

I am a .NET trainer and 80 students attended my course...

Do you imagine ? I had to give marks on an exam to 80 students, BY HAND AND KEYBOARD ?

I have 3 fears in the world : doing nothing, doing a repetitive task, and spiders.

When I can, I prefer to create exams myself, so I just instruct to my students that they need to do whatever they can to make unit tests pass : 1 question, 1 unit test, 1 point. Simple, effective, pragmatic, scalable, and I teach them the real use case of unit tests.

"Don't waste your (my) time twice (80 times) with the same bug, find it once, write a test, let it pass and move on."

"I'm paid to make you a great developer, you will be paid to deliver value in softwares, we will not be paid to fix bugs." (Ok... in reality we are often paid to fix bugs)

But this time... this time someone else have created the exam... and without unit testing ! 80 exams !

So, as a conscious developer, I made my computer works for me... with 2 visual studio projects :

  • The first was a XAML parser/code parser to mark exams automatically. (Have you created a Button in a Grid in your xaml file ? 1 point)
  • The second was a web crawler that enter all marks automatically in the website of the school.

It was funnier than doing all by hand... and guess what ? Students never knew it until I told them ! Smile | :)

"Why I got this mark ??"
"Let me check my notes (logs)... you forgot to use a Pivot, - 1, you did not use transitions -2, your button has no event handler -1..."
"Ok stop I understand ! Wow you took note on each of us and all of that in one day !"
"You bet ! I worked very hard this night ! Exhausting !... I release it under MS-PL, do you want the sources ?"
"...What ?"

But what interest us today is the second project... the browser automation crawler !

This crawler passed the relevance test : it has already two customers, me and another trainer too lazy to copy marks from excel to school's website.

You may ask : "Why would you browser automation ? why don't you use a simple, classic HTTP crawler and emit HTTP requests ?"

As a teacher/trainer, I need to enter marks in a website written in ASP.NET/AJAX, so HTTP requests were very complicated to create.

Even with a tool like Fiddler and the fantastic Request to Code plugin, I didn't managed to hack the right HTTP requests to send marks after 20 min.

So I took another approach : browser automation.

With HTTP crawler you say :

  • Make a POST to LOGIN URL, with parameter Login="Nicolas" and Password="Password" and ViewState="....."
  • Save the cookie
  • Generate other HTTP requests with this cookie...

With browser automation you say :

  • Click on the "loginBox" enter Nicolas
  • Click on the "password" enter Password
  • Click on the submitForm button
  • Click on next
  • Click on the ddl dropdownlist and wait I make a choice
  • Click on continue...

In summary, you use the browser to make requests for you.

Use case : Vote 5 for this article automatically

Disclaimer to CodeProject admins : I never used such shameful method to artificially have more vote... You can check by yourself with the pitifull vote number I get for my articles... That's the first time !

I know that all of you are very busy BUT very curious. So this use case is about using your curiosity to give me a 5.

So with a classic crawler you might say :

  • Get request to http://www.codeproject.com/Articles/338036/BrowserAutomationCrawler
  • POST request with login parameter and password parameter to action "submitLogin"
  • Save the cookie
  • POST vote=5, articleId=MyArticleId to action=Vote, with the cookie

It might work, except that I completely make up the parameters and action name and so you would need fiddler to fine tunes the requests correctly. (And depending on the website it can be very very hard, especially with AJAX stuff)

Another way to do the same thing is to say :

  • Go to http://www.codeproject.com/Articles/338036/BrowserAutomationCrawler
  • If "logout" is present then you are already logged,
    else
    Wait that I click on sign in (so I can manually fill email and password)
  • Then click the option vote 5
  • Fill the comment textbox with "5 for me, great Nicolas, thanks for you work ! Smile | :) "
  • Click on vote button

Go ahead download the sources and try yourself !!!

Let's check the code

First I have to instantiate a WebBrowser (the control I use to automate the web browser), and setup the url of my article, the vote you give me, the comment, and maybe your credentials for CodeProject (optional, since you will be able to enter them manually).

[STAThread]
static void Main(string[] args)
{
    Form form = new Form();
    form.Width = 1024;
    form.Height = 780;
    WebBrowser browser = new WebBrowser();
    browser.Dock = DockStyle.Fill;
    form.Controls.Add(browser);

    new VoteCrawler()
    {
        WebBrowser = browser,
        ArticleUrl = "http://www.codeproject.com/Articles/338036/BrowserAutomationCrawler",
        Rating = 5,
        Comment = "5 for me, great Nicolas, thanks for you work ! :)"
        //You can choose to specify email and password here on enter them directly on the website (or if IE has your cookie, it will vote anyway)
        //Email = "email",
        //Password = "password"
    }.Crawl();

    form.ShowDialog();
}

Then here is the code to vote automatically:

public class VoteCrawler : Crawler
{
    public string ArticleUrl
    {
        get;
        set;
    }
    public string Comment
    {
        get;
        set;
    }
    public int Rating
    {
        get;
        set;
    }
    public string Email
    {
        get;
        set;
    }
    public string Password
    {
        get;
        set;
    }

    protected override void Automate()
    {
        GoTo(ArticleUrl);
        EnsureIsSignedOn();
        Click("ctl00_RateArticle_VoteRBL_" + (Rating - 1).ToString());
        Fill(new ClassSelector("RateComment"), Comment);
        Click("ctl00_RateArticle_SubmitRateBtn");
    }

    void EnsureIsSignedOn()
    {
        var isLogged = Actions.Ask(() => WebBrowser.Document.GetElementById("ctl00_MemberMenu_Signout") != null);
        if(!isLogged)
        {
            if(Email != null)
                Fill("Email", Email);
            if(Password != null)
                Fill("Password", Password);

            var submit = new IdSelector("subForm").SelectChildren(e => e.GetAttribute("type") == "submit");
            if(Email == null || Password == null)
                WaitClickOn(submit);
            else
                Click(submit);
            WhenLoaded();
            EnsureIsSignedOn();
        }
    }
}

If IE has saved your cookie, you don't need to enter login/password, it will just vote for you.

I just inherit from the Crawler class and override Automate, the code is self descriptive.

var isLogged = Actions.Ask(()=> WebBrowser.Document.GetElementById("ctl00_MemberMenu_Signout") != null);

You can see that I use the WebBrowser class to do my stuff, with Actions.Ask. As I will explain in under the hood, I use a winform component. Automate does not run in the UI thread, Actions will invoke the action inside the UI thread.

The question to easily find id's of HTML elements like ctl00_RateArticle_VoteRBL_4, and what if there is no id ?

Finding id's is easy, with chrome right click on the element, inspect element and it brings you where you need to go.

For more complex request you can use custom or built-in Selectors (string are implicitely converted to IdSelector)

For example, to fill the comment box once you click on the 5 option, here is the code to select and fill it : Fill the textbox with class RateComment.

Fill(new ClassSelector("RateComment"), "5 for me, great Nicolas, thanks for you work ! :)");

Here is the class Crawler and selectors :

ok it takes more words of article to describe than words of code, since the code itself is only approximately 300 lines.

How it works ?

Under the hood

System.Windows.Forms.WebBrowser is a class to embbed browser inside winform application.

The interesting part is that it is a wrapper around COM interface of IE7. (or IE6 I don't remember)

So you can run javascript with C# code, or invoke events you want on HtmlElement.

For exemple, in Crawler.Click I call the click event in javascript in the browser.

public Crawler Click(Selector selector)
{
    Actions.Do((end) =>
    {
        selector.Select(WebBrowser.Document).First().InvokeMember("click");
        end();
    });
    return this;
}

Or I modify the DOM directly :

public Crawler Fill(Selector selector, string value)
{
    Actions.Do((end) =>
    {
        selector.ForEach(WebBrowser.Document, e => e.InnerText = value);
        end();
    });
    return this;
}

WebBrowser is a winform component, so it run on the UI thread. Since Automate should run sequentially but without blocking the running thread, I run it on a separate thread.

public void Crawl()
{
    if(WebBrowser == null)
        throw new InvalidOperationException("WebBrowser should be affected");
    ThreadPool.QueueUserWorkItem(s =>
    {
        Automate();
    });
}

AutomationActions is just a wrapper around the SynchronizationContext or the UI thread.

public class AutomationActions
{
    SynchronizationContext _UiContext;
    AutoResetEvent _AutoReset = new AutoResetEvent(false);

    public AutomationActions()
    {
        _UiContext = SynchronizationContext.Current;
    }

    public T Ask<T>(Func<T> request)
    {
        T result = default(T);
        Do(end =>
        {
            result = request();
            end();
        });
        return result;
    }
    public void Do(Action action)
    {
        Do(end =>
        {
            try
            {
                action();
            }
            finally
            {
                end();
            }
        });
    }
    public void Do(Action<Action> action)
    {
        if(_UiContext == SynchronizationContext.Current)
        {
            throw new InvalidCastException("Cannot call AutomationActions in the UI thread");
        }
        _UiContext.Post(s =>
        {
            action(() =>
            {
                _AutoReset.Set();
            });
        }, null);
        _AutoReset.WaitOne();
    }
}

Do(Action<Action> action) allow to move on the next action when the UI thread decide it by calling the Action parameter.

I use it for the WhenLoaded action for example.

public Crawler WhenLoaded()
{
    Actions.Do((end) =>
    {
        WebBrowserNavigatedEventHandler onNavigated = null;
        onNavigated = (s, e) =>
        {
            WebBrowser.Navigated -= onNavigated;
            WebBrowser.Document.Window.Load += (s1, e2) =>
            {
                end();
            };
        };
        WebBrowser.Navigated += onNavigated;
    });
    return this;
}

Conclusion and question for you

The obvious limitation of this method of crawling is that it is not suited for web spider or text mining application that need to browser large data.

It is very well suited to code semi automated data entry.

The other limitation is the WebBrowser use IE76 or 7... do you know how I can use another browser instead ? it can make an interesting interoperability article.

I found a way to use all browsers you want : using Selenium. Selenium is an interoperability layer common to several browsers. I don't like the design that is over engineered with lots of interface and features are very hard to find through intellisense, but it does a very good job ! The only limitation compared with WebBrowser is that you can't subscribe to javascript events with C# code ! (but you can execute javacript)... I will update this article if I refactor the code to use Selenium.

Selenium is over fitted to the use case of testing, that explain why it has some friction as a tool for assisted/semi-automated data entry.

License

This article, along with any associated source code and files, is licensed under The Microsoft Public License (Ms-PL)

Share

About the Author

Nicolas Dorier
Software Developer Freelance
France France
I am a trainer and a curious developer.
 
CEO of AO-IS, we created a tool to make IaaS on Azure more easy IaaS Management Studio.
 
If you are interested for working with me, for fun coding stuff, for freelance stuff, or interested in using our cloud training infrastructure freely for a kickass presentation for the dev community ? this way Smile | :)

Comments and Discussions

 
QuestionCrawler Click Pinmemberxbonds12-Dec-13 11:59 
AnswerRe: Crawler Click PinmemberNicolas Dorier13-Dec-13 23:11 
QuestionError while invoking click action in Crawler Click. PinmemberdocLiv4-Oct-13 19:45 
AnswerRe: Error while invoking click action in Crawler Click. PinmemberNicolas Dorier5-Oct-13 0:37 
GeneralMy vote of 5 PinmemberJGranborg7-Aug-13 3:14 
GeneralMy vote of 5 Pinmembersebastienj17-Jul-13 4:24 
GeneralMy vote of 4 Pinmemberyleefree16-Apr-13 6:48 
GeneralMy vote of 5 Pinmemberyleefree16-Apr-13 6:47 
GeneralMy vote of 5 Pinmembermarcelosantaninfor18-Feb-13 3:10 
GeneralMy vote of 5 Pinmemberomarsec8-Feb-13 6:39 
GeneralMy vote of 5 Pinmembersunne14-Jan-13 16:21 
GeneralMy vote of 5 PinmemberPalik-BI30-Oct-12 1:29 
GeneralMy vote of 5 Pinmemberb.brankola23-Oct-12 8:23 
GeneralMy vote of 5 Pinmemberpkruger22-Oct-12 3:33 
GeneralMy vote of 5 PinmemberC Marriott23-Aug-12 17:20 
GeneralMy vote of 5 PinmemberAllen Paul de Rouen26-Jul-12 20:11 
GeneralRe: My vote of 5 PinmemberNicolas Dorier17-Aug-12 14:44 
Thanks, that's the best compliment a coder can have ! Smile | :)
GeneralMy vote of 5 Pinmemberpbh25-Jul-12 12:37 
GeneralMy vote of 5 PinmemberYang Peng15-Jul-12 19:31 
Questiongood Article PinmemberMember 869768711-Jul-12 22:42 
AnswerRe: good Article PinmemberNicolas Dorier11-Jul-12 23:19 
GeneralRe: good Article PinmemberMember 869768713-Jul-12 0:15 
GeneralMy vote of 5 Pinmemberalfcas23-Jun-12 0:58 
GeneralMy vote of 5 Pinmemberthinhtu16-May-12 1:55 
GeneralMy vote of 5 Pinmembersandeep_chauhan19868-May-12 20:04 

General General    News News    Suggestion Suggestion    Question Question    Bug Bug    Answer Answer    Joke Joke    Rant Rant    Admin Admin   

Use Ctrl+Left/Right to switch messages, Ctrl+Up/Down to switch threads, Ctrl+Shift+Left/Right to switch pages.

| Advertise | Privacy | Mobile
Web02 | 2.8.141022.1 | Last Updated 1 Mar 2012
Article Copyright 2012 by Nicolas Dorier
Everything else Copyright © CodeProject, 1999-2014
Terms of Service
Layout: fixed | fluid