Click here to Skip to main content
11,701,855 members (64,667 online)
Click here to Skip to main content

Web Automation

, 23 Mar 2015 CPOL 69K 3.9K 96
Rate this:
Please Sign up or sign in to vote.
You can fill any form, click any button, and extract any data from web by using Web Automation.

Introduction

Web Automation written in C#, using Web Browser control to automate many things. You can do many things by write JavaScript code to control a Web Browser.

Background

You must know basic knowledge about JavaScript and C#, because Web Automation using JavaScript to communication with c# function.

How it work

This code bellow explain communication between JavaScript function and C# function, if you don't use it, when you call JavaScript to C# function it will not work.

// This nested class must be ComVisible for the JavaScript to be able to call it.
[ComVisible(true)]
public partial class frmMain : Form
{
} 

Then you create a Web Browser and a load JavaScript function on the Web Browser like in the code below on Form Load:

private void frmMain_Load(object sender, EventArgs e)
{
    wbMain = new WebBrowser();
    wbMain.ObjectForScripting = this;
    wbMain.DocumentText =
        "<html>" +
        "<head>" +
        "<script type='text/javascript'>" +
            "function go(url) " +
            "{ " +
                "window.external.go(url);" +
            "}" +
        "</script>" +
        "</head>" +
        "<body>" +
        "</body>" +
        "</html>";
    this.Controls.Add(wbMain);
} 

The code below has two important points, first ObjectForScripting and window.external:

The purpose of ObjectForScripting is for communication between JavaScript and the frmMain application, you can replace this by another class if you want, this class will be called from the JavaScript function.

I load the go function in Web Browser when loading the application, the purpose is when you execute script it will call function go in C#:

public void go(string url)
{
    MessageBox.Show(url);
}

How you excute the script:

wbMain.Document.InvokeScript("eval", new object[] { "go('http://www.google.com');" });  

When you excute script bellow, if function is JavaScript it will execute, if not it will find function you load from Web Browser, example function go, it not function JavaScript so that function will call, then window.external.go(url) will communicate with window application by ObjectForScripting and run function go(string url) and it will show dialog with message is 'http://www.google.com';

Combine all code and you will see how it is working:

using System;
using System.Collections.Generic;
using System.ComponentModel;
using System.Data;
using System.Drawing;
using System.Linq;
using System.Text;
using System.Windows.Forms;

using System.Runtime.InteropServices;

namespace WebAutomation
{
    [ComVisible(true)]
    public partial class frmMain : Form
    {
        private WebBrowser wbMain;

        public frmMain()
        {
            InitializeComponent();
        }

        private void frmMain_Load(object sender, EventArgs e)
        {
            wbMain = new WebBrowser();
            wbMain.ObjectForScripting = this;
            wbMain.DocumentText =
                "<html>" +
                "<head>" +
                "<script type='text/javascript'>" +
                    "function go(url) " +
                    "{ " +
                        "window.external.go(url);" +
                    "}" +
                "</script>" +
                "</head>" +
                "<body>" +
                "</body>" +
            "</html>";
            this.Controls.Add(wbMain);
        }

        public void go(string url)
        {
            wbBrowser.Navigate(url);
        }

        private void btnRun_Click(object sender, EventArgs e)
        {
            string code = tbxCode.Text;
            wbMain.Document.InvokeScript("eval", new object[] { code }); 
        }
    }
} 

Techniques

On my Application I using method "Call a C# Method From JavaScript Hosted in a WebBrowser" to control anything JavaScript cannot work like write file, open tab, working with file and folder...

Secure sensitive information like Account, User, Email Server, Connection by using "DotNet Zip Library" and "Secure Password Authentication".

Using "Mouse Key Board Library" to record and playback your mouse and your keyboard by script.

Recognize text from image by "Tesseract OCR", read and write excel file is "NPOI", Speech to Text is "Speech Recognition in C#".

Overview

Web Automation have four sections:

  1. Menu file will help you use some of feature of program like login, logout, register...,
  2. ToolBar menu will help you work with some item you will usually using like record, run,...,
  3. Web Browser will display website when you suffering,
  4. Developer Tools that place will automate every of program by JavaScript language

Using Web Automation to Login Gmail

Open program and enter link http://www.gmail.com and click button go.

Right click every where on Web Browser, click Go then it show question "Do you want go to website?", if you choose yes it will get current URL you enter on the textbox, else it will get XPath of location where you right click then go to the link of the xpath.

Then it will show you Developer Tools with code JavaScript to control this function.

Right click on textbox username, click fill, enter username then click OK, repeat for password.

Right click on Button Login select click.

You must enter two line to ensure website load done before you continue, then you click Run and then program will go to Gmail for you.

go("https://accounts.google.com/ServiceLogin?service=mail&passive=true&
  rm=false&continue=http://mail.google.com/mail/&scc=1&ltmpl=default&ltmplcache=2");
sleep(5,false);
fill("/html/body/div/div[2]/div/div/form/div/input","thang@gmail.com");
fill("/html/body/div/div[2]/div/div/form/div[2]/input","password");
sleep(5,false);
click("/html/body/div/div[2]/div/div/form/input[12]");

What do you do with Web Automation?

Here is function of Web Automation, you can try by write function bellow with your parameter then click run and see what happen, notes some function need authentication to work like function number: 59 to 64, rest of another function you don't need to login.

  1. Open new tab: tabnew();
  2. Close current tab: tabclose();
  3. Go to website by url or xpath: go(url); or go(xpath);
  4. Go back: back();
  5. Go next: next();
  6. Reload browser: reload();
  7. Stop browser: stop();
  8. Wait until Web Browser has loaded or timer to wait Web Browser load: sleep(seconds, isBreakWhenWBCompleted)
  9. Close application: exit();
  10. Click to element: click(xpath);
  11. Write log to Preview Tab: log(value);
  12. Clear log write on Preview: clearlog();
  13. Extract data from xpath: extract(xpath);
  14. Fill data to element on Web Browser: fill(xpath, value);
  15. Select dropdown value: filldropdown(xpath, index);
  16. Convert string to Object javascript: toObject(xpath);
  17. Get all link in the area of xpath, it will stop until program go all of link : browser(xpath);
  18. Reset list website to unread so program can go back and browser continue : resetlistwebsite();
  19. Take snapshot of website: takesnapshot(url);
  20. Create folder: createfolder(path);
  21. Get list folder from path: getfolders(path);
  22. Create file: save(content, location, isOverride);
  23. Get list file from path: getfiles(path);
  24. Download a file: download(url, location);
  25. Read file: read(path);
  26. Remove file: remove(path);
  27. Remove folder: removefolder(path);
  28. Open explorer and select file: explorer(path);
  29. Run code from string: excute(code);
  30. Logoff: logoff();
  31. Lockworkstation: lockworkstation();
  32. Forcelogoff: forcelogoff();
  33. Reboot: reboot();
  34. Shutdown: shutdown();
  35. Hibernate: hibernate();
  36. Standby: standby();
  37. Run application: runcommand(path);
  38. Get current url of current tab: getcurrenturl();
  39. Get browser height: getheight();
  40. Get title of website: gettitle();
  41. Get all link from xpath: getlinks(xpath);
  42. Get content of web browser: getCurrentContent();
  43. Get path of Web Automation: getCurrentPath();
  44. Read cell of excel file: readCellExcel(path, sheetname, row, column);
  45. Write excel file: writeCellExcel(path, sheetname, cellname, value);
  46. Load HTML file: loadHTML(path);
  47. Convert json string to Object: textToJSON(jsonstring);
  48. Login: login(username, password);
  49. Register: register(username, email, password, confirm);
  50. OCR: ocr(path, language);
  51. Mouse Up: MouseUp(mouseButton, lastTime);
  52. Mouse Down: MouseDown(mouseButton, lastTime);
  53. Mouse Click: MouseClick(mouseButton, lastTime);
  54. Mouse Doble Click: MouseDoubleClick(mouseButton, lastTime);
  55. Mouse Move: MouseMove(x, y, isShow, lastTime);
  56. Mouse Wheel: MouseWheel(delta, lastTime);
  57. Key Down: KeyDown(delta, lastTime);
  58. Key Up: KeyUp(delta, lastTime);
  59. Get Account By: getAccountBy(name);
  60. Get Database: getDatabases(name);
  61. Get Tables: getTables(name, dbName);
  62. Get Columns: getColumns(name, dbName, table);
  63. Get Rows: getRows(name, dbName, sql);
  64. Execute SQL Query: excuteQuery(name, dbName, sql);

That all function of Web Automation, every function is JavaScript function so it have return or not return value, some time you get value is string, some time you will get value is object, it base on my C# function.

By combine each function together you can automate a lot of things, automate fill textbox, click button, extract any kind of data, automate create, delete, file, folder, automate extract data from website and save it to excel file easily, update data to excel file, read data from excel file, control your mouse, your keyboard.....

I try to include all knowledge I learn on my program like OCR, Text To Speech, Speech To Text,... My program can be expand by add function on JavaScript then add function on C# to solve some algorithm.

Notes

This is just one function of my software, in my software I give some technique to help automation better like record your mouse and keyboard it help automate some part not work on software, or control program by speech.

License

This article, along with any associated source code and files, is licensed under The Code Project Open License (CPOL)

Share

About the Author

Đinh Công Thắng
Software Developer CSC Viet Nam
Vietnam Vietnam
I worked on Phoenix Software and CSC Viet Nam.
I have knowledge in C#, ASP.NET, Javascript, Jquery, Ajax, SQL Server, MVC 3.

My purpose want apply what I learn to real world, create utility so help another person don't repeat their work day by day, of course it not easy but I try and hope it good.

Website: http://www.thangdc.com
Twitter: http://www.twitter.com/thangdc
Facebook: http://www.facebook.com/tinhyeu24h
Google Plus: https://plus.google.com/117248983435347399719

You may also be interested in...

Comments and Discussions

 
QuestionClear cash whith gecko fx Pin
redouan_beg4-Aug-15 6:01
memberredouan_beg4-Aug-15 6:01 
QuestionNeed to download excel file on button click which prompts to save or open file Pin
Member 23064338-Jul-15 0:05
memberMember 23064338-Jul-15 0:05 
AnswerRe: Need to download excel file on button click which prompts to save or open file Pin
Đinh Công Thắng9-Jul-15 3:44
memberĐinh Công Thắng9-Jul-15 3:44 
GeneralRe: Need to download excel file on button click which prompts to save or open file Pin
Member 23064339-Jul-15 22:05
memberMember 23064339-Jul-15 22:05 
GeneralRe: Need to download excel file on button click which prompts to save or open file Pin
Đinh Công Thắng10-Jul-15 3:53
memberĐinh Công Thắng10-Jul-15 3:53 
GeneralRe: Need to download excel file on button click which prompts to save or open file Pin
Member 230643312-Jul-15 23:02
memberMember 230643312-Jul-15 23:02 
QuestionCustomizations Pin
mrdance1-Jul-15 10:29
membermrdance1-Jul-15 10:29 
Generalapp to record Test case and record mouse movement and keyboard clicks Pin
Member 117448536-Jun-15 20:31
memberMember 117448536-Jun-15 20:31 
QuestionHow execute onclick event for each row in gidview? Pin
ggdppt10-Apr-15 9:50
memberggdppt10-Apr-15 9:50 
AnswerRe: How execute onclick event for each row in gidview? Pin
Đinh Công Thắng12-Apr-15 16:52
memberĐinh Công Thắng12-Apr-15 16:52 
QuestionIs there a way to download PDF files automatically? Pin
Raffaele Lanzetta4-Apr-15 6:46
memberRaffaele Lanzetta4-Apr-15 6:46 
AnswerRe: Is there a way to download PDF files automatically? Pin
Đinh Công Thắng5-Apr-15 22:29
memberĐinh Công Thắng5-Apr-15 22:29 
GeneralRe: Is there a way to download PDF files automatically? Pin
Raffaele Lanzetta7-Apr-15 4:26
memberRaffaele Lanzetta7-Apr-15 4:26 
GeneralRe: Is there a way to download PDF files automatically? Pin
Đinh Công Thắng7-Apr-15 17:09
memberĐinh Công Thắng7-Apr-15 17:09 
QuestionXpcom.Initialize("xulrunner") DLL not found Exception . Pin
Sharad Singh17-Mar-15 1:14
memberSharad Singh17-Mar-15 1:14 
AnswerRe: Xpcom.Initialize("xulrunner") DLL not found Exception . Pin
Đinh Công Thắng22-Mar-15 19:14
memberĐinh Công Thắng22-Mar-15 19:14 
QuestionVisual studio could not copy the file SciLexer.dll because it was not found Pin
Raffaele Lanzetta2-Mar-15 1:00
memberRaffaele Lanzetta2-Mar-15 1:00 
AnswerRe: Visual studio could not copy the file SciLexer.dll because it was not found Pin
g_g_7-Mar-15 17:04
memberg_g_7-Mar-15 17:04 
AnswerRe: Visual studio could not copy the file SciLexer.dll because it was not found Pin
Đinh Công Thắng22-Mar-15 19:29
memberĐinh Công Thắng22-Mar-15 19:29 
QuestionMissing a lot of DLL.Please update the content on the GIT. Pin
hahaclj2-Nov-14 10:52
memberhahaclj2-Nov-14 10:52 
AnswerRe: Missing a lot of DLL.Please update the content on the GIT. Pin
Đinh Công Thắng2-Nov-14 16:11
memberĐinh Công Thắng2-Nov-14 16:11 
GeneralRe: Missing a lot of DLL.Please update the content on the GIT. Pin
hahaclj2-Nov-14 16:25
memberhahaclj2-Nov-14 16:25 
GeneralRe: Missing a lot of DLL.Please update the content on the GIT. Pin
Đinh Công Thắng2-Nov-14 16:38
memberĐinh Công Thắng2-Nov-14 16:38 
GeneralRe: Missing a lot of DLL.Please update the content on the GIT. Pin
hahaclj2-Nov-14 20:23
memberhahaclj2-Nov-14 20:23 
GeneralRe: Missing a lot of DLL.Please update the content on the GIT. Pin
Đinh Công Thắng2-Nov-14 20:45
memberĐinh Công Thắng2-Nov-14 20:45 
GeneralRe: Missing a lot of DLL.Please update the content on the GIT. Pin
hahaclj2-Nov-14 20:50
memberhahaclj2-Nov-14 20:50 
GeneralRe: Missing a lot of DLL.Please update the content on the GIT. Pin
Basio Basha5-Nov-14 2:10
memberBasio Basha5-Nov-14 2:10 
GeneralRe: Missing a lot of DLL.Please update the content on the GIT. Pin
Đinh Công Thắng5-Nov-14 2:35
memberĐinh Công Thắng5-Nov-14 2:35 
GeneralRe: Missing a lot of DLL.Please update the content on the GIT. Pin
Basio Basha5-Nov-14 3:01
memberBasio Basha5-Nov-14 3:01 
GeneralRe: Missing a lot of DLL.Please update the content on the GIT. Pin
Đinh Công Thắng5-Nov-14 3:13
memberĐinh Công Thắng5-Nov-14 3:13 
QuestionMissing references. Pin
Souheil1-Nov-14 1:28
memberSouheil1-Nov-14 1:28 
AnswerRe: Missing references. Pin
Đinh Công Thắng1-Nov-14 1:55
memberĐinh Công Thắng1-Nov-14 1:55 
QuestionActiveX Support? Pin
DBilbao24-Sep-14 3:23
memberDBilbao24-Sep-14 3:23 
AnswerRe: ActiveX Support? Pin
Đinh Công Thắng25-Sep-14 19:23
memberĐinh Công Thắng25-Sep-14 19:23 
QuestionBrowser to run slowly Pin
Đinh Công Thắng19-Sep-14 20:29
memberĐinh Công Thắng19-Sep-14 20:29 
QuestionWhere is ThangDC.dll? Pin
watersonchang1-Sep-14 23:42
memberwatersonchang1-Sep-14 23:42 
AnswerRe: Where is ThangDC.dll? Pin
Đinh Công Thắng19-Sep-14 20:14
memberĐinh Công Thắng19-Sep-14 20:14 
GeneralRe: Where is ThangDC.dll? Pin
Member 1110292925-Sep-14 2:38
memberMember 1110292925-Sep-14 2:38 
GeneralRe: Where is ThangDC.dll? Pin
Đinh Công Thắng25-Sep-14 19:21
memberĐinh Công Thắng25-Sep-14 19:21 
QuestionSequential execution Pin
Anand Elikapati17-Jun-14 22:24
memberAnand Elikapati17-Jun-14 22:24 
AnswerRe: Sequential execution Pin
Đinh Công Thắng19-Jun-14 5:12
memberĐinh Công Thắng19-Jun-14 5:12 
GeneralMy vote of 5 Pin
Prasad Khandekar28-May-14 22:12
professionalPrasad Khandekar28-May-14 22:12 
NewsWeb Automation has opened source code in Github Pin
Đinh Công Thắng27-May-14 4:20
memberĐinh Công Thắng27-May-14 4:20 
Questionvery grate work , really very helpful , but i`m confused Pin
Amr Abd-Rabou15-May-14 22:46
memberAmr Abd-Rabou15-May-14 22:46 
AnswerRe: very grate work , really very helpful , but i`m confused Pin
Đinh Công Thắng16-May-14 3:22
memberĐinh Công Thắng16-May-14 3:22 
GeneralRe: very grate work , really very helpful , but i`m confused Pin
Amr Abd-Rabou20-May-14 1:28
memberAmr Abd-Rabou20-May-14 1:28 
QuestionGood Work but no complete code Pin
Anand Elikapati24-Mar-14 23:41
memberAnand Elikapati24-Mar-14 23:41 
AnswerRe: Good Work but no complete code Pin
Đinh Công Thắng25-Mar-14 2:35
memberĐinh Công Thắng25-Mar-14 2:35 
GeneralMy vote of 5 Pin
Sivaji15655-Jan-14 20:38
memberSivaji15655-Jan-14 20:38 
QuestionJscript is not working. Pin
Sivaji15654-Jan-14 7:24
memberSivaji15654-Jan-14 7:24 

General General    News News    Suggestion Suggestion    Question Question    Bug Bug    Answer Answer    Joke Joke    Rant Rant    Admin Admin   

Use Ctrl+Left/Right to switch messages, Ctrl+Up/Down to switch threads, Ctrl+Shift+Left/Right to switch pages.

| Advertise | Privacy | Terms of Use | Mobile
Web01 | 2.8.150819.1 | Last Updated 23 Mar 2015
Article Copyright 2013 by Đinh Công Thắng
Everything else Copyright © CodeProject, 1999-2015
Layout: fixed | fluid