Click here to Skip to main content
Click here to Skip to main content

Automating web browsing

, 11 Dec 2006
Rate this:
Please Sign up or sign in to vote.
An article on how to automate web browsing: clicking a button, entering data in a text box etc.

Sample Image

Introduction

In this article I want to present a possibility to automate tasks we are executing with the mouse and keyboard in a web browser. This can mean: to open a page, enter data in a text box, press a button etc. These basics can then be extended into more complex operations such as search, analysis of web data, automatic crawling inside a domain, or all this can be used if you would like to build your own test tool for testing web applications.

Usually, the approach for processing web data is something like this:

WebRequest req = WebRequest.Create("www.codeproject.com");
WebResponse res = req.GetResponse( );

By parsing the response HTML, one can examine the raw HTML content of the page, find out about buttons, hidden fields etc. Then one can make further posts and thus imitate a link click or a button click. But we are introducing here a different approach. We want to use the WebBrowser class from the .NET 2.0 framework. This will make it different from using the request-response model. While using the request–response combination, we are dealing with text files. With the WebBrowser control, we will access a page and its elements on a higher level; e.g., links and buttons will become "objects". So we will say something like:

Button btn = browser.ReturnElementByName(“myButton”);
btn.Click();

Background

The WebBrowser ActiveX control is an instance of your local Internet Explorer browser, with all its features and problems. All communication with this control is done via the MSHTML library. This library has interfaces for all controls of the web page and these are for all versions of Windows. Basically, you are always getting back an object and converting it to the interface you need. That’s hard, if you don't know what's the right interface. Currently, there is no complete wrapper for this. But you can see the right interface in the DomElement property of the specific element while debugging the WebBrowser's element collection.

You can read a good description of a similar component here. The article refers to the previous versions of the .NET framework.

Using the code

So let's see how we can open a new page in the browser. It is very simple:
public string OpenPage(WebBrowser browser, string urlToLoad)
{
  browser.Navigate(urlToLoad);
}

An important thing to mention here is: loading a page into the browser is an asynchronous operation. We never know when the page will be loaded. Therefore, if we want to load a page and click a button on this page, our load scenario must wait, until the page is loaded. We can achieve this as follows:

public string OpenPage(WebBrowser browser, string urlToLoad){
  bool loadFinished = false;
  
  browser.DocumentCompleted += delegate { loadFinished = true; };
  browser.Navigate(urlToLoad);
  
  while (!loadFinished && counterTimeOut > 0){
    Thread.Sleep(100);
    Application.DoEvents();
    }
}

So what we have done here: we subscribed to the event that happens when the page is loaded fully in our browser. This will set the flag to true, and our while loop will end. In the while loop, we are just waiting, and in the mean time letting the application to serve events.

To click a button we first load a page into the browser. Then we get a button by its name from WebBrowser.HtmlElementCollection. Then we cast the button to the MSHTML interface object HTMLInputElementClass. We can then click a button as follows:

HTMLInputElementClass iElement = (HTMLInputElementClass) button.DomElement;
iElement.click(); 

To click a link we must cast the HTMLElement to a different MSHTML object as follows:

HTMLAnchorElementClass linkElement = 
  (HTMLAnchorElementClass) linkToClick.DomElement;
linkElement.click();

To enter data in an input field we do not even need a type cast. We simply get the input box as a HtmlElement from the WebBrowser, and simply say:

element.InnerText = valueToFill;

To select a radio button, we use code as follows:

HTMLInputElementClass iElement = 
   (HTMLInputElementClass)radioToSelect.DomElement;
iElement.@checked = true;

To select a value from a combo, we use:

HTMLSelectElementClass iElement = 
   (HTMLSelectElementClass) dropdown.DomElement;
iElement.value = value; 

Saving page as a picture (snapshot) can also be useful in many scenarios. A Web browser enables us to save the current page as a snapshot. We can achieve this by this sample function:

Rectangle rec = new Rectangle();
rec.Offset(0, 0);
rec.Size = browser.Document.Window.Size;

Bitmap bmp = new Bitmap(rec.Width, rec.Height);
browser.DrawToBitmap(bmp, rec);
bmp.Save(“file.path”, ImageFormat.Jpeg);

In the downloads, you have the source code for all these steps, plus a very simple Windows application which presents how the examples can be used. Please note that in order to keep the code sample as simple as possible, examples are without any object oriented principles like inheritance. Running the demo requires the .NET framework 2.0 installed.

Points of interest

From these simple steps, a complex tool for specific purposes can be written very quickly. We have, for example, developed a complete test tool, controlled via an XML file, were you simply write test cases in XML in this manner: go to a page, click a link, enter data, click a button, save page picture for later review etc.

However, the things that we haven't yet addressed are:

  1. frame environment (old ASP sites were usually built using frames),
  2. recording machine (so one can browse a page, and the history of what he did would be saved in a collection of simple tasks).

While using this example, do not forget to use the well known Firefox developer add-on: Web developer toolbar. It helps a lot.

License

This article has no explicit license attached to it but may contain usage terms in the article text or the download files themselves. If in doubt please contact the author via the discussion board below.

A list of licenses authors might use can be found here

Share

About the Author

mitja g
Web Developer
United States United States
No Biography provided

Comments and Discussions

 
SuggestionMessage Automatically Removed PinmemberMember 1104824331-Aug-14 1:27 
QuestionAwesome Artical PinmemberGauravBaid24-Jun-14 3:41 
QuestionSuperb !! Pinmemberdanishkhawar22-May-12 15:01 
GeneralUnable to cast COM object PinmemberNolubez18-Jun-11 7:15 
GeneralRe: Unable to cast COM object PinmemberRAGolko5-Jul-11 23:39 
GeneralRe: Unable to cast COM object Pinmemberanhlunnhaque29-Sep-11 4:43 
GeneralRe: Unable to cast COM object Pinmembermitja g21-Sep-11 23:00 
GeneralRe: Unable to cast COM object Pinmemberanhlunnhaque4-Oct-11 6:28 
GeneralMy vote of 5 Pinmemberoriono30-Sep-10 23:34 
GeneralRadio Box problem Pinmemberbinhvtt13-Aug-10 2:20 
Questionhow to automatic click to Alert or confirm event Pinmemberskalofrio18-Jun-10 12:46 
QuestionHow to select Combo box option by Index ??? PinmemberToby palladan5-Sep-09 3:24 
QuestionWhat about pages included frames & framesets PinmemberPeyman Ranjbar30-Jul-08 18:39 
GeneralNice Article PinmemberPandu43829-Jul-08 4:28 
GeneralGr8 work!! PinmemberNitin Sawant7-Jun-08 0:59 
Generalinput type="file" handling Pinmembermalco7824-Mar-08 11:13 
GeneralRe: input type="file" handling Pinmemberdjdane7-Apr-09 2:52 
GeneralVs 2003 Pinmembertravis_112112148-Jul-07 23:19 
GeneralReal Browser Pinmembertravis_112112147-Jul-07 18:48 
GeneralRe: Real Browser PinmemberJeff Lehmer2-Sep-07 9:00 
GeneralRe: Real Browser PinmemberAlex Furman9-Oct-07 12:48 
GeneralSystem.NullReferenceException: Object reference not set to an instance of an object PinmemberJackMartinz20-Jun-07 1:46 
GeneralRe: System.NullReferenceException: Object reference not set to an instance of an object Pinmembermitja g20-Jun-07 2:39 
GeneralRe: System.NullReferenceException: Object reference not set to an instance of an object PinmemberJackMartinz20-Jun-07 20:54 
Generalautomation of web browser Pinmembers_khan_87613-Jun-07 21:10 
GeneralRe: automation of web browser PinmemberTom Mat16-Jun-07 20:58 
GeneralRe: automation of web browser Pinmembermitja g20-Jun-07 2:51 
QuestionCombo Box Post Box Issue ???? PinmemberTom Mat10-Jun-07 20:13 
AnswerRe: Combo Box Post Box Issue ???? Pinmembermitja g20-Jun-07 2:46 
GeneralRe: Combo Box Post Box Issue ???? PinmemberTom Mat23-Jun-07 21:24 
Questionno... Postback event !!! PinmemberTom Mat30-Jun-07 20:15 
QuestionPlease help! Pinmemberzzaamine10-Jun-07 13:32 
AnswerRe: Please help! Pinmembermitja g20-Jun-07 2:59 
QuestionRe: Please help! Pinmemberzzaamine20-Jun-07 20:22 
AnswerThank you friend :) Pinmemberzzaamine21-Jun-07 14:43 
GeneralRe: Thank you friend :) Pinmembersubhadeep.sen@gmail.com22-Nov-08 8:28 
GeneralNice work Pinmemberbutschinformatik26-May-07 7:41 
GeneralAutomation of complex DHTML/AJAX applications PinmemberAlex Furman14-Mar-07 17:26 
GeneralAutomating firefox using C# PinmemberMember #382756914-Feb-07 0:23 
GeneralRe: Automating firefox using C# Pinmembermitja g25-Feb-07 22:54 
GeneralRe: Automating firefox using C# Pinmemberfariss24-Apr-07 18:01 
GeneralReally good article PinmemberWackeldackel27-Jan-07 11:50 
GeneralRe: Really good article Pinmembermitja g29-Jan-07 1:53 
QuestionWeb page embedded image Pinmembersalblomo28-Dec-06 13:07 
AnswerRe: Web page embedded image Pinmembermitja g26-Jan-07 3:23 
QuestionDidn't work with forms and input type=submit buttons Pinmemberwege26-Dec-06 4:49 
AnswerRe: Didn't work with forms and input type=submit buttons Pinmembermitja g26-Jan-07 3:37 
QuestionRe: Didn't work with forms and input type=submit buttons Pinmemberzzaamine7-Jun-07 5:39 
GeneralRe: Didn't work with forms and input type=submit buttons PinmemberJackMartinz20-Jun-07 1:44 
AnswerRe: Didn't work with forms and input type=submit buttons Pinmembermitja g20-Jun-07 21:59 

General General    News News    Suggestion Suggestion    Question Question    Bug Bug    Answer Answer    Joke Joke    Rant Rant    Admin Admin   

Use Ctrl+Left/Right to switch messages, Ctrl+Up/Down to switch threads, Ctrl+Shift+Left/Right to switch pages.

| Advertise | Privacy | Terms of Use | Mobile
Web04 | 2.8.1411022.1 | Last Updated 11 Dec 2006
Article Copyright 2006 by mitja g
Everything else Copyright © CodeProject, 1999-2014
Layout: fixed | fluid