UI Automation Using Microsoft Active Accessibility (MSAA)

• Download source - 33.7 KB

Introduction

Microsoft Active Accessibility is Microsoft’s User Interface technology. All window controls which are based on this technology can expose information about the UI elements to the outside world. It is a COM based technology and provides the IAccessible interface. Controls implementing this interface expose certain methods which provide information about UI elements like location, type, state, name, value, role, and the default action which can be invoked. We use this technology to write automation libraries for MS Word 2007, while the UI is completely based on MSAA.

I would like to share the same experience on this forum as it has given us very promising results.

Background

We started developing a Windows UI automation library based on Microsoft UIA. We are using this library as a platform to write automated functional test cases for Windows clients. As this library is based on Microsoft UIA, it works well with Win32 and WinForms based applications, but it doesn’t work with Windows applications based on MSAA (e.g., MS Office 2007). That’s where we extended our UI automation library for MSAA support and we successfully achieved our mission.

Approach

We followed the approach as demonstrated in the diagram below:

• Win32 API consumes MSAA services from the Windows OS.
• MSAA Layer is a core automation module. It consumes MSAA services from the Win32 API and provides services to search UI elements on the screen and to get the IAccessible interface as a COM object for that UI element. This layer treats each UI element as an AccessibleUIItem. It can be a window, button, textbox, or any control which implements the IAccessible interface. It provides an efficient searching mechanism to search through an accessible tree.
• UI Manager defines the object model for the application’s UI which we want to automate. It extends the base AssessibleUIItem provided by the MSAA layer. We will see an example of the UI object model designed for MS Office 2007 later in this article.

Now, let's have a look at each layer in detail. A code walkthrough for each layer would be my focus here.

Win32 API

We have developed the automation library in .NET so we used PInvoke to fetch the services of the Win32 API. Following is the set of Win32 APIs we have used:

[DllImport("user32.dll")]
[return: MarshalAs(UnmanagedType.Bool)]
public static extern bool EnumWindows(EnumWindowsProc lpEnumFunc, IntPtr lParam);

[DllImport("user32.dll", SetLastError = true, CharSet = CharSet.Auto)]
public static extern IntPtr FindWindow(string lpClassName, string lpWindowName);

[DllImport("user32.dll", EntryPoint = "FindWindow", 
           SetLastError = true, CharSet = CharSet.Auto)]
public static extern IntPtr FindWindowByClass(string lpClassName, IntPtr zero);

[DllImport("user32.dll", EntryPoint = "FindWindow", 
      SetLastError = true, CharSet = CharSet.Auto)]
public static extern IntPtr FindWindowByCaption(IntPtr zero, string lpWindowName);

[DllImport("user32.dll", CharSet = CharSet.Auto)]
public static extern bool EnumChildWindows(IntPtr hWndParent, 
              EnumWindowsProc lpEnumFunc, int lParam);

[DllImport("user32.dll", SetLastError = true, CharSet = CharSet.Auto)]
public static extern int GetWindowText(IntPtr hwnd, 
                         StringBuilder lpString, int cch);

[DllImport("user32.dll", SetLastError = true, CharSet = CharSet.Auto)]
public static extern int GetWindowTextLength(IntPtr hwnd);

[DllImport("user32.dll", SetLastError = true)]
public static extern bool BringWindowToTop(IntPtr hwnd);

[DllImport("oleacc.dll")]
public static extern uint GetRoleText(uint dwRole, 
       [Out] StringBuilder lpszRole, uint cchRoleMax);

[DllImport("oleacc.dll")]
public static extern uint GetStateText(uint dwStateBit, 
       [Out] StringBuilder lpszStateBit, uint cchStateBitMax);

[DllImport("oleacc.dll")]
public static extern uint WindowFromAccessibleObject(IAccessible pacc, 
                                                     ref IntPtr phwnd);

[DllImport("oleacc.dll", PreserveSig = false)]
[return: MarshalAs(UnmanagedType.Interface)]
public static extern object AccessibleObjectFromWindow(int hwnd, 
                            int dwId, ref Guid riid);

[DllImport("oleacc.dll")]
public static extern int AccessibleObjectFromWindow(
     IntPtr hwnd,
     uint id,
     ref Guid iid,
     [In, Out, MarshalAs(UnmanagedType.IUnknown)] ref object ppvObject);   

[DllImport("oleacc.dll")]
public static extern int AccessibleChildren(IAccessible paccContainer, 
                         int iChildStart, int cChildren,
                         [Out()] [MarshalAs(UnmanagedType.LPArray, 
                         SizeParamIndex = 4)] object[] rgvarChildren, 
                         ref int pcObtained);

public delegate bool EnumWindowsProc(IntPtr hWnd, IntPtr lParam);

MSAA Layer

This layer provides the abstraction over the Win32 API. Its main job is to search the accessible UI item from the whole Accessibility Tree whose root node represents Desktop. After the item is found, it fetches all its properties and provides the MSAAUIItem object. The search mechanism is very optimized. We can ignore the invisible UI items while searching.

"There is a tool available from Microsoft called Accessibility Explorer (AccExplorer32.exe). Using this tool we can easily traverse through all Accessibility Trees."

MSAAUIItme exposes the following UI item properties:

• Name
• Role
• State
• Location
• Value
• Handle
• IsEnabled
• DefaultAction

How to Search the UI Item

It is a two step process.

MSAA.cs

• Step 1: Get the accessible object of the top application window. For example, the "MS Word 2007" window.
C#

public static IAccessible GetTopWindowAccessibleObject(Regex windowName)
{
    foreach (IAccessible accWindowObject in GetTopWindowAccessibleList())
    {
        try
        {
            string accWindowName = accWindowObject.get_accName(0);

            if (!string.IsNullOrEmpty(accWindowName))
            {
                if (windowName.Match(accWindowObject.get_accName(0)).Success)
                {
                    return accWindowObject;
                }
            }
        }
        catch (Exception ex)
        {
        }
    }
    return default(IAccessible);
}

• Step 2: Use the top window accessible object as the parent object and search for the child object under that. E.g., "Ribbon" is a child object of the "MS Word 2007" window.
C#

public static IAccessible GetObjectByName(IAccessible objParent, 
              Regex objName, bool ignoreInvisible)
{
    IAccessible objToReturn = default(IAccessible);

    if (objParent != null)
    {
        IAccessible[] children = GetAccessibleChildren(objParent);
        foreach (IAccessible child in children)
        {
            string childName = null;
            string childState = string.Empty;

            try
            {
                childName = child.get_accName(0);
                childState = 
                  GetStateText(Convert.ToUInt32(child.get_accState(0)));
            }
            catch (Exception)
            {
            }

            if (ignoreInvisible)
            {
                if (childName != null 
                    && objName.Match(childName).Success
                    && !childState.Contains("invisible"))
                {
                    return child;
                }
            }
            else
            {
                if (childName != null 
                    && objName.Match(childName).Success)
                {
                    return child;
                }
            }
                    
            if (ignoreInvisible)
            {
                if (!childState.Contains("invisible"))
                {
                    objToReturn = GetObjectByName(child, objName, ignoreInvisible);
                    if (objToReturn != default(IAccessible))
                    {
                        return objToReturn;
                    }
                }
            }
            else
            {
                objToReturn = GetObjectByName(child, objName, ignoreInvisible);
                if (objToReturn != default(IAccessible))
                {
                    return objToReturn;
                }
            }
            
        }
    }
    return objToReturn;
}

How to Get the State

This is a tricky part. IAccessible.get_accState(object varChild) returns a state ID number which is a combination of one or multiple states. E.g., "MS Word 2007" has three states at a particular instance: focusable, moveable, sizeable. But, the Win32 API GetStateText will return only the first state string against the state ID. So, we have to call this API more than once. How we do it is shown in the following code snippet.

MSAAState.cs

public static string GetStateText(uint stateID)
{
    uint maxLength = 1024;
    var focusableStateText = new StringBuilder((int)maxLength);
    var sizeableStateText = new StringBuilder((int)maxLength);
    var moveableStateText = new StringBuilder((int)maxLength);
    var invisibleStateText = new StringBuilder((int)maxLength);

    //This pattern needs to be used to fetch other available cobination of states. 
    if (stateID == (MSAAStateConstants.STATE_SYSTEM_FOCUSABLE
                   | MSAAStateConstants.STATE_SYSTEM_SIZEABLE
                   | MSAAStateConstants.STATE_SYSTEM_MOVEABLE))
    {
        Win32.GetStateText(MSAAStateConstants.STATE_SYSTEM_FOCUSABLE, 
                           focusableStateText, maxLength);
        Win32.GetStateText(MSAAStateConstants.STATE_SYSTEM_SIZEABLE, 
                           sizeableStateText, maxLength);
        Win32.GetStateText(MSAAStateConstants.STATE_SYSTEM_MOVEABLE, 
                           moveableStateText, maxLength);

        return focusableStateText + "," + sizeableStateText + "," + moveableStateText;

    }

    var stateText = new StringBuilder((int)maxLength);
    Win32.GetStateText(stateID, stateText, maxLength);
    return stateText.ToString();
}

MSAAUIItem answers these questions and abstracts out other complexities as well.

Office 2007 UI Automation Object Model

As the name sounds, this layer is specific to the Windows application which needs to be automated. This layer extends MSAAUIItem to define the Office 2007 UI controls.

OfficeUIItem extends MSAAUIItem and acts as the core Office UI item. All other items like OfficeRibbon, OfficeToolBar, OfficeRibbonTab, OfficePropertyPage, and MSWordWindow extend OfficeUIItem.

Now, let us write some automation code. This code will launch MS Word 2007 and will open the tab named "Insert". Then, it will run the "Cover Page" command on the "Pages" Tool Bar. It is important to select a particular tab because only then MS Word will refresh the UI and load the corresponding toolbars and controls under that.

Process.Start("winword");
MSWordWindow msWordMainWindow = new MSWordWindow("Document1 -");
msWordMainWindow.SelectTab("Insert");
msWordMainWindow.ToolBars["Pages"].Controls["Cover Page"].Invoke();

Attached Sample Code

The sample code attached with this article is tested on Windows XP SP2 and Windows 2008. The attached code will work for Windows XP but may fail for Windows 2008. The reason for this is that the Win32 API which we use to get the Accessible Object is different for both OSs.

Win32.cs

//Windows XP
[DllImport("oleacc.dll")]
internal static extern int AccessibleObjectFromWindow(
     IntPtr hwnd,
     uint id,
     ref Guid iid,
     [In, Out, MarshalAs(UnmanagedType.IUnknown)] ref object ppvObject);    

//Windows 2008
[DllImport("oleacc.dll", PreserveSig = false)]
[return: MarshalAs(UnmanagedType.Interface)]
public static extern object AccessibleObjectFromWindow(int hwnd, 
                            int dwId, ref Guid riid);

Conclusion

Microsoft Active Accessibility is not a comprehensive solution to automate the Windows UI. There are certain limitations. E.g., using accessibility, we can search for a TextBox control on MS Word Ribbon, but we can not get or set a value in it, which can be done using Microsoft UIA. So, a combination of MSAA and UIA can provide a more promising solution for Windows UI automation.