Click here to Skip to main content
15,867,488 members
Articles / Programming Languages / C#
Article

Automating web browsing

Rate me:
Please Sign up or sign in to vote.
4.81/5 (37 votes)
11 Dec 20064 min read 239.3K   9.1K   132   54
An article on how to automate web browsing: clicking a button, entering data in a text box etc.

Sample Image

Introduction

In this article I want to present a possibility to automate tasks we are executing with the mouse and keyboard in a web browser. This can mean: to open a page, enter data in a text box, press a button etc. These basics can then be extended into more complex operations such as search, analysis of web data, automatic crawling inside a domain, or all this can be used if you would like to build your own test tool for testing web applications.

Usually, the approach for processing web data is something like this:

C#
WebRequest req = WebRequest.Create("www.codeproject.com");
WebResponse res = req.GetResponse( );

By parsing the response HTML, one can examine the raw HTML content of the page, find out about buttons, hidden fields etc. Then one can make further posts and thus imitate a link click or a button click. But we are introducing here a different approach. We want to use the WebBrowser class from the .NET 2.0 framework. This will make it different from using the request-response model. While using the request–response combination, we are dealing with text files. With the WebBrowser control, we will access a page and its elements on a higher level; e.g., links and buttons will become "objects". So we will say something like:

C#
Button btn = browser.ReturnElementByName("myButton");
btn.Click();

Background

The WebBrowser ActiveX control is an instance of your local Internet Explorer browser, with all its features and problems. All communication with this control is done via the MSHTML library. This library has interfaces for all controls of the web page and these are for all versions of Windows. Basically, you are always getting back an object and converting it to the interface you need. That’s hard, if you don't know what's the right interface. Currently, there is no complete wrapper for this. But you can see the right interface in the DomElement property of the specific element while debugging the WebBrowser's element collection.

You can read a good description of a similar component here. The article refers to the previous versions of the .NET framework.

Using the code

So let's see how we can open a new page in the browser. It is very simple:
C#
public string OpenPage(WebBrowser browser, string urlToLoad)
{
  browser.Navigate(urlToLoad);
}

An important thing to mention here is: loading a page into the browser is an asynchronous operation. We never know when the page will be loaded. Therefore, if we want to load a page and click a button on this page, our load scenario must wait, until the page is loaded. We can achieve this as follows:

C#
public string OpenPage(WebBrowser browser, string urlToLoad){
  bool loadFinished = false;
  
  browser.DocumentCompleted += delegate { loadFinished = true; };
  browser.Navigate(urlToLoad);
  
  while (!loadFinished && counterTimeOut > 0){
    Thread.Sleep(100);
    Application.DoEvents();
    }
}

So what we have done here: we subscribed to the event that happens when the page is loaded fully in our browser. This will set the flag to true, and our while loop will end. In the while loop, we are just waiting, and in the mean time letting the application to serve events.

To click a button we first load a page into the browser. Then we get a button by its name from WebBrowser.HtmlElementCollection. Then we cast the button to the MSHTML interface object HTMLInputElementClass. We can then click a button as follows:

C#
HTMLInputElementClass iElement = (HTMLInputElementClass) button.DomElement;
iElement.click(); 

To click a link we must cast the HTMLElement to a different MSHTML object as follows:

C#
HTMLAnchorElementClass linkElement = 
  (HTMLAnchorElementClass) linkToClick.DomElement;
linkElement.click();

To enter data in an input field we do not even need a type cast. We simply get the input box as a HtmlElement from the WebBrowser, and simply say:

C#
element.InnerText = valueToFill;

To select a radio button, we use code as follows:

C#
HTMLInputElementClass iElement = 
   (HTMLInputElementClass)radioToSelect.DomElement;
iElement.@checked = true;

To select a value from a combo, we use:

C#
HTMLSelectElementClass iElement = 
   (HTMLSelectElementClass) dropdown.DomElement;
iElement.value = value; 

Saving page as a picture (snapshot) can also be useful in many scenarios. A Web browser enables us to save the current page as a snapshot. We can achieve this by this sample function:

C#
Rectangle rec = new Rectangle();
rec.Offset(0, 0);
rec.Size = browser.Document.Window.Size;

Bitmap bmp = new Bitmap(rec.Width, rec.Height);
browser.DrawToBitmap(bmp, rec);
bmp.Save("file.path", ImageFormat.Jpeg);

In the downloads, you have the source code for all these steps, plus a very simple Windows application which presents how the examples can be used. Please note that in order to keep the code sample as simple as possible, examples are without any object oriented principles like inheritance. Running the demo requires the .NET framework 2.0 installed.

Points of interest

From these simple steps, a complex tool for specific purposes can be written very quickly. We have, for example, developed a complete test tool, controlled via an XML file, were you simply write test cases in XML in this manner: go to a page, click a link, enter data, click a button, save page picture for later review etc.

However, the things that we haven't yet addressed are:

  1. frame environment (old ASP sites were usually built using frames),
  2. recording machine (so one can browse a page, and the history of what he did would be saved in a collection of simple tasks).

While using this example, do not forget to use the well known Firefox developer add-on: Web developer toolbar. It helps a lot.

License

This article has no explicit license attached to it but may contain usage terms in the article text or the download files themselves. If in doubt please contact the author via the discussion board below.

A list of licenses authors might use can be found here


Written By
Web Developer
United States United States
This member has not yet provided a Biography. Assume it's interesting and varied, and probably something to do with programming.

Comments and Discussions

 
GeneralRe: Really good article Pin
mitja g29-Jan-07 0:53
mitja g29-Jan-07 0:53 
QuestionWeb page embedded image Pin
salblomo28-Dec-06 12:07
salblomo28-Dec-06 12:07 
AnswerRe: Web page embedded image Pin
mitja g26-Jan-07 2:23
mitja g26-Jan-07 2:23 
QuestionDidn't work with forms and input type=submit buttons Pin
wege26-Dec-06 3:49
wege26-Dec-06 3:49 
AnswerRe: Didn't work with forms and input type=submit buttons Pin
mitja g26-Jan-07 2:37
mitja g26-Jan-07 2:37 
QuestionRe: Didn't work with forms and input type=submit buttons Pin
zzaamine7-Jun-07 4:39
zzaamine7-Jun-07 4:39 
GeneralRe: Didn't work with forms and input type=submit buttons Pin
JackMartinz20-Jun-07 0:44
JackMartinz20-Jun-07 0:44 
AnswerRe: Didn't work with forms and input type=submit buttons Pin
mitja g20-Jun-07 20:59
mitja g20-Jun-07 20:59 
This example worked for me. I only had to repair compile time errors, because you were declaring same variable twice (entryDataTask, clickButtonTask)

So if I have code like this, it works (loades search result 2 times):

//open google page
NavigateTask navTask = new NavigateTask();
navTask.Execute(this.webBrowser, "http://www.google.com");

//enter "codeproject" as a search text
EntryDataTask entryDataTask1 = new EntryDataTask();
HtmlElement textBox1 = this.FindControlByName("q", this.webBrowser.Document.All);
entryDataTask1.Execute(textBox1, "codeproject");

//click button
ClickButtonTask clickButtonTask1 = new ClickButtonTask();
HtmlElement buttonToClick1 = this.FindControlByName("btnG", this.webBrowser.Document.All);
clickButtonTask1.Execute(this.webBrowser, buttonToClick1);

//enter "Thank you" as a search text
EntryDataTask entryDataTask2 = new EntryDataTask();
HtmlElement textBox2 = this.FindControlByName("q", this.webBrowser.Document.All);
entryDataTask2.Execute(textBox2, "Thank you");

// Second Time Using click button
ClickButtonTask clickButtonTask2 = new ClickButtonTask();
HtmlElement buttonToClick2 = this.FindControlByName("btnG", this.webBrowser.Document.All);
clickButtonTask2.Execute(this.webBrowser, buttonToClick2);
GeneralRe: Didn't work with forms and input type=submit buttons Pin
JackMartinz27-Jun-07 23:43
JackMartinz27-Jun-07 23:43 
GeneralWatiN Pin
jvmenen11-Dec-06 21:57
jvmenen11-Dec-06 21:57 
GeneralRe: WatiN Pin
mitja g26-Jan-07 3:54
mitja g26-Jan-07 3:54 
GeneralRe: WatiN Pin
jvmenen26-Jan-07 4:40
jvmenen26-Jan-07 4:40 
GeneralRe: WatiN Pin
mitja g20-Jun-07 21:40
mitja g20-Jun-07 21:40 

General General    News News    Suggestion Suggestion    Question Question    Bug Bug    Answer Answer    Joke Joke    Praise Praise    Rant Rant    Admin Admin   

Use Ctrl+Left/Right to switch messages, Ctrl+Up/Down to switch threads, Ctrl+Shift+Left/Right to switch pages.