Click here to Skip to main content
15,391,989 members
Articles / Programming Languages / C#
Posted 11 Dec 2006


132 bookmarked

Automating web browsing

Rate me:
Please Sign up or sign in to vote.
4.81/5 (37 votes)
11 Dec 20064 min read
An article on how to automate web browsing: clicking a button, entering data in a text box etc.

Sample Image


In this article I want to present a possibility to automate tasks we are executing with the mouse and keyboard in a web browser. This can mean: to open a page, enter data in a text box, press a button etc. These basics can then be extended into more complex operations such as search, analysis of web data, automatic crawling inside a domain, or all this can be used if you would like to build your own test tool for testing web applications.

Usually, the approach for processing web data is something like this:

WebRequest req = WebRequest.Create("");
WebResponse res = req.GetResponse( );

By parsing the response HTML, one can examine the raw HTML content of the page, find out about buttons, hidden fields etc. Then one can make further posts and thus imitate a link click or a button click. But we are introducing here a different approach. We want to use the WebBrowser class from the .NET 2.0 framework. This will make it different from using the request-response model. While using the request–response combination, we are dealing with text files. With the WebBrowser control, we will access a page and its elements on a higher level; e.g., links and buttons will become "objects". So we will say something like:

Button btn = browser.ReturnElementByName("myButton");


The WebBrowser ActiveX control is an instance of your local Internet Explorer browser, with all its features and problems. All communication with this control is done via the MSHTML library. This library has interfaces for all controls of the web page and these are for all versions of Windows. Basically, you are always getting back an object and converting it to the interface you need. That’s hard, if you don't know what's the right interface. Currently, there is no complete wrapper for this. But you can see the right interface in the DomElement property of the specific element while debugging the WebBrowser's element collection.

You can read a good description of a similar component here. The article refers to the previous versions of the .NET framework.

Using the code

So let's see how we can open a new page in the browser. It is very simple:
public string OpenPage(WebBrowser browser, string urlToLoad)

An important thing to mention here is: loading a page into the browser is an asynchronous operation. We never know when the page will be loaded. Therefore, if we want to load a page and click a button on this page, our load scenario must wait, until the page is loaded. We can achieve this as follows:

public string OpenPage(WebBrowser browser, string urlToLoad){
  bool loadFinished = false;
  browser.DocumentCompleted += delegate { loadFinished = true; };
  while (!loadFinished && counterTimeOut > 0){

So what we have done here: we subscribed to the event that happens when the page is loaded fully in our browser. This will set the flag to true, and our while loop will end. In the while loop, we are just waiting, and in the mean time letting the application to serve events.

To click a button we first load a page into the browser. Then we get a button by its name from WebBrowser.HtmlElementCollection. Then we cast the button to the MSHTML interface object HTMLInputElementClass. We can then click a button as follows:

HTMLInputElementClass iElement = (HTMLInputElementClass) button.DomElement;; 

To click a link we must cast the HTMLElement to a different MSHTML object as follows:

HTMLAnchorElementClass linkElement = 
  (HTMLAnchorElementClass) linkToClick.DomElement;;

To enter data in an input field we do not even need a type cast. We simply get the input box as a HtmlElement from the WebBrowser, and simply say:

element.InnerText = valueToFill;

To select a radio button, we use code as follows:

HTMLInputElementClass iElement = 
iElement.@checked = true;

To select a value from a combo, we use:

HTMLSelectElementClass iElement = 
   (HTMLSelectElementClass) dropdown.DomElement;
iElement.value = value; 

Saving page as a picture (snapshot) can also be useful in many scenarios. A Web browser enables us to save the current page as a snapshot. We can achieve this by this sample function:

Rectangle rec = new Rectangle();
rec.Offset(0, 0);
rec.Size = browser.Document.Window.Size;

Bitmap bmp = new Bitmap(rec.Width, rec.Height);
browser.DrawToBitmap(bmp, rec);
bmp.Save("file.path", ImageFormat.Jpeg);

In the downloads, you have the source code for all these steps, plus a very simple Windows application which presents how the examples can be used. Please note that in order to keep the code sample as simple as possible, examples are without any object oriented principles like inheritance. Running the demo requires the .NET framework 2.0 installed.

Points of interest

From these simple steps, a complex tool for specific purposes can be written very quickly. We have, for example, developed a complete test tool, controlled via an XML file, were you simply write test cases in XML in this manner: go to a page, click a link, enter data, click a button, save page picture for later review etc.

However, the things that we haven't yet addressed are:

  1. frame environment (old ASP sites were usually built using frames),
  2. recording machine (so one can browse a page, and the history of what he did would be saved in a collection of simple tasks).

While using this example, do not forget to use the well known Firefox developer add-on: Web developer toolbar. It helps a lot.


This article has no explicit license attached to it but may contain usage terms in the article text or the download files themselves. If in doubt please contact the author via the discussion board below.

A list of licenses authors might use can be found here


About the Author

mitja g
Web Developer
United States United States
No Biography provided

Comments and Discussions

QuestionAwesome Artical Pin
GauravBaid24-Jun-14 2:41
MemberGauravBaid24-Jun-14 2:41 
QuestionSuperb !! Pin
danishkhawar22-May-12 14:01
Memberdanishkhawar22-May-12 14:01 
GeneralUnable to cast COM object Pin
Nolubez18-Jun-11 6:15
MemberNolubez18-Jun-11 6:15 
GeneralRe: Unable to cast COM object Pin
RAGolko5-Jul-11 22:39
MemberRAGolko5-Jul-11 22:39 
GeneralRe: Unable to cast COM object Pin
anhlunnhaque29-Sep-11 3:43
Memberanhlunnhaque29-Sep-11 3:43 
GeneralRe: Unable to cast COM object Pin
mitja g21-Sep-11 22:00
Membermitja g21-Sep-11 22:00 
GeneralRe: Unable to cast COM object Pin
anhlunnhaque4-Oct-11 5:28
Memberanhlunnhaque4-Oct-11 5:28 
GeneralMy vote of 5 Pin
oriono30-Sep-10 22:34
Memberoriono30-Sep-10 22:34 
GeneralRadio Box problem Pin
binhvtt13-Aug-10 1:20
Memberbinhvtt13-Aug-10 1:20 
Questionhow to automatic click to Alert or confirm event Pin
skalofrio18-Jun-10 11:46
Memberskalofrio18-Jun-10 11:46 
QuestionHow to select Combo box option by Index ??? Pin
Toby Palladan5-Sep-09 2:24
MemberToby Palladan5-Sep-09 2:24 
QuestionWhat about pages included frames & framesets Pin
Peyman Ranjbar30-Jul-08 17:39
MemberPeyman Ranjbar30-Jul-08 17:39 
GeneralNice Article Pin
Pandu43829-Jul-08 3:28
MemberPandu43829-Jul-08 3:28 
GeneralGr8 work!! Pin
Nitin S6-Jun-08 23:59
professionalNitin S6-Jun-08 23:59 
Generalinput type="file" handling Pin
malco7824-Mar-08 10:13
Membermalco7824-Mar-08 10:13 
GeneralRe: input type="file" handling Pin
djdane7-Apr-09 1:52
Memberdjdane7-Apr-09 1:52 
GeneralVs 2003 Pin
travis_112112148-Jul-07 22:19
Membertravis_112112148-Jul-07 22:19 
GeneralReal Browser Pin
travis_112112147-Jul-07 17:48
Membertravis_112112147-Jul-07 17:48 
GeneralRe: Real Browser Pin
EntilzhaFini2-Sep-07 8:00
MemberEntilzhaFini2-Sep-07 8:00 
GeneralRe: Real Browser Pin
Alex Furman9-Oct-07 11:48
MemberAlex Furman9-Oct-07 11:48 
GeneralSystem.NullReferenceException: Object reference not set to an instance of an object Pin
JackMartinz20-Jun-07 0:46
MemberJackMartinz20-Jun-07 0:46 
GeneralRe: System.NullReferenceException: Object reference not set to an instance of an object Pin
mitja g20-Jun-07 1:39
Membermitja g20-Jun-07 1:39 
GeneralRe: System.NullReferenceException: Object reference not set to an instance of an object Pin
JackMartinz20-Jun-07 19:54
MemberJackMartinz20-Jun-07 19:54 
Generalautomation of web browser Pin
s_khan_87613-Jun-07 20:10
Members_khan_87613-Jun-07 20:10 
GeneralRe: automation of web browser Pin
Tom Mat16-Jun-07 19:58
MemberTom Mat16-Jun-07 19:58 

General General    News News    Suggestion Suggestion    Question Question    Bug Bug    Answer Answer    Joke Joke    Praise Praise    Rant Rant    Admin Admin   

Use Ctrl+Left/Right to switch messages, Ctrl+Up/Down to switch threads, Ctrl+Shift+Left/Right to switch pages.