Introduction to Microsoft Active Accessibility

• Download demo project with source - 41.3 KB

Introduction

This is the first is a series of articles that I hope to write concerning the Microsoft Active Accessibility (MSAA) technology. Much of my current work has surrounded this topic and there seems to be very little in the way of useful introductions to the topic. To start, I would like to provide an introduction to MSAA so that an understanding of what it does and why it exists is present. After the groundwork is established we can move into how it would be implemented in an application.

In the final articles in this series I would like to offer a few new tools to aide in the development and debugging of your MSAA enhanced applications and perhaps even go over how MSAA can be used for application automation. The tools are meant to be a replacement for the tools provided by Microsoft which seem to fall very short of usable. These tools will have their source code made available for you to review. The tools are developed with the same concepts that will be shown throughout this series and hence should provide you with a perspective on implementing applications that use MSAA.

The rest of the articles in this series will be written as my time permits. Since there is not sure way to gauge this, each article will stand on its own providing relevant information without the presence of the others. Enjoy!

What is Accessibility?

Accessibility, in this context, is creating your application or framework in such a manner that it will be usable by those users with certain physical handicaps such as blindness, low version, hard-of-hearing or deafness. Depending on where you are developing your application or who you are developing it for you may be legally required to ensure your application is accessible. When working with Microsoft Windows there are two main forms of accessible technologies available to you.

The first and simplest is the system's ability to enter a High Contrast (HC) mode. In this mode the system enters a color scheme, selected by the user, that is supposed to provide great contrast between the foreground and background. This is targeted at low-vision users. If you would like to see what your systems look like in High Contrast mode you can hold down (LEFT-ALT + LEFT-SHIFT + PRNTSCRN). You can also go to Control Panel -> Accessibility Options -> Display. Once activated you will see your system change into a contrasting color scheme. The default high contrast scheme is black backgrounds with white foregrounds. Notice that the title bars also change colors. This can be an important thing to remember when developing your application because the switch is based on system colors. These are the colors that can be obtained through the ::GetSysColor(..) API. In all places that you use custom colors, those colors will stay the same even when in HC mode. This may make your application harder to use for those people with vision impairments. When possible, you should strive to use system colors to keep your application accessible.

The second form of accessibility available to applications is MSAA. If your application uses the standard Win32 controls and that is all then it is very likely you will have little to no work to do to be accessible. Microsoft provides interfaces for all of its controls. If you are developing a toolkit of custom controls or perhaps just a single control for use in your application then it may become necessary for you to add MSAA support for those custom controls. Since Windows does not know the purpose of your control, what it is or how it is structured it is necessary for you to be able to relay that information so that Assistive Technologies (ATs) such as screen readers, magnifiers and other devices such as Braille displays can properly relay information about your control or controls to the user. The sections below will go into more detail concerning how MSAA works and what is required for you to implement it for a given control.

The Pieces to the MSAA Puzzle

There are two main mechanisms to be aware of when in comes to MSAA; the MSAA event model and the IAccessible COM interface. The event model is the mechanism applications/controls use to notify the system that a particular event has taken place within a control. This can be something such as a the control just gained focus or the user modified value of the control. The IAccessible interface is a COM interface that can be implemented in relation to a control so that an AT can query information about the control. This interface would be used to get the current value, determine the state of the control (selected, focused, disabled, etc..) and even perform actions such as moving through the MSAA DOM.

The IAccessible Interface

The first thing to cover is going to be the IAccessible interface and its inner workings. This is necessary because some of the fundamental concepts explained here, as they relate to the MSAA DOM, are required to understand how MSAA events transmitted to the system are then in turn queried by ATs for a greater level of information. Please note that this section goes into a fair bit of detail concerning the various aspects of the MSAA interface including DOM construction, roles, identification elements such as names and descriptions and state data. For a single control it is unlikely you will be concerned with building an entire DOM of elements rather you will be looking to implement a single element in an existing MSAA DOM. Even though most will be looking at the smaller case I wanted to provide as much detail as possible so that those people looking for an understanding of the whole picture can see how everything works. From that point, you should be able to infer all the information you need for a single element. People that are likely to need to build an entire MSAA DOM are those people putting together an entire UI toolkit of custom controls. Since you cannot rely on Microsoft's controls at that point the burden of providing MSAA interfaces falls on you.

The MSAA DOM for an application is a tree structured group of elements that defines the accessible elements for an application. This means that each control that can be used by the end user should show up in the DOM. For a standard application, one that uses Std. Win32 controls, each control element such as edits, lists, list items, trees, tree items, menus, etc... will have a related elements within the MSAA DOM. If you have a custom control that has a visual-only purpose, such as a spacer control or other type of element that helps with the layout of visual/usable controls it it not necessary for it to implement IAccessible. Since the user cannot interact with this control and it provides no information to the user, it can be left out of the DOM. This is can even be considered good practice as the DOM will be more compact, hence more efficient to walk.

Below you can see a view of an MSAA DOM I retrieved by inspecting a folder view through Windows Explorer.

This image shows us a couple of things. First, I started the inspection at the control level rather than the window level. What we get in a standard list control and all my files/folders which show up as elements within the list.

First, let's examine the structure of a MSAA DOM, using the above image as a reference point, and after that we can examine the various information that can be conveyed through the IAccessible interface on each of the elements. Notice in the structure above the first element is that of the list control itself. Each of the items within the list exist as children to the container. You also can see, at the bottom of the tree, where the column headers are shown. Each of these elements exist as children to the list as well, albeit at a different level.

Before we dive headfirst into how the IAccessible interface is used to walk over and read from the MSAA DOM take a look at the code block below. This is what the actual COM interface looks like.

interface IAccessible : IDispatch
{
  HRESULT STDMETHODCALLTYPE get_accParent([retval][out] 
      IDispatch **ppdispParent);        
  HRESULT STDMETHODCALLTYPE get_accChildCount(
      [retval][out] long *pcountChildren);        
  HRESULT STDMETHODCALLTYPE get_accChild([in] VARIANT varChild, 
      [retval][out] IDispatch **ppdispChild);        
  HRESULT STDMETHODCALLTYPE get_accName([in] VARIANT varChild,
      [retval][out] BSTR *pszName);        
  HRESULT STDMETHODCALLTYPE get_accValue([in] VARIANT varChild, 
      [retval][out] BSTR *pszValue);
  HRESULT STDMETHODCALLTYPE get_accDescription([in] VARIANT varChild, 
      [retval][out] BSTR *pszDescription);
  HRESULT STDMETHODCALLTYPE get_accRole([in] VARIANT varChild, 
      [retval][out] VARIANT *pvarRole);
  HRESULT STDMETHODCALLTYPE get_accState([in] VARIANT varChild, 
      [retval][out] VARIANT *pvarState);
  HRESULT STDMETHODCALLTYPE get_accHelp([in] VARIANT varChild, 
      [retval][out] BSTR *pszHelp);
  HRESULT STDMETHODCALLTYPE get_accHelpTopic([out] BSTR *pszHelpFile, 
      [in] VARIANT varChild,[retval][out] long *pidTopic);
  HRESULT STDMETHODCALLTYPE get_accKeyboardShortcut([in] VARIANT varChild,
      [retval][out] BSTR *pszKeyboardShortcut);
  HRESULT STDMETHODCALLTYPE get_accFocus([retval][out] VARIANT *pvarChild);
  HRESULT STDMETHODCALLTYPE get_accSelection([retval][out] 
      VARIANT *pvarChildren);
  HRESULT STDMETHODCALLTYPE get_accDefaultAction([in] VARIANT varChild, 
      [retval][out] BSTR *pszDefaultAction);        
  HRESULT STDMETHODCALLTYPE accSelect([in] long flagsSelect, 
      [in] VARIANT varChild);
  HRESULT STDMETHODCALLTYPE accLocation([out] long *pxLeft, 
      [out] long *pyTop,[out] long *pcxWidth,[out] long *pcyHeight,
      [in] VARIANT varChild);        
  HRESULT STDMETHODCALLTYPE accNavigate([in] long navDir, 
      [in] VARIANT varStart, [retval][out] VARIANT *pvarEndUpAt);
  HRESULT STDMETHODCALLTYPE accHitTest([in] long xLeft, [in] long yTop,
      [retval][out] VARIANT *pvarChild);
  HRESULT STDMETHODCALLTYPE accDoDefaultAction([in] VARIANT varChild);
  HRESULT STDMETHODCALLTYPE put_accName([in] VARIANT varChild,
      [in] BSTR szName);
  HRESULT STDMETHODCALLTYPE put_accValue([in] VARIANT varChild, 
      [in] BSTR szValue);
};

When it comes to constructing the MSAA DOM and in turn, walking through it, there are two main points to understand. Point #1 is that not all elements implement the IAccessible interface. Point #2 is that all elements can be found using a child ID. Children under a given element are located through a 1-based index. There is a special child id called CHILDID_SELF (this constant equals 0) that, when used with a function like get_accChild, returns the element itself rather than a child.

Elements in the MSAA DOM that implement the IAccessible interface are called complex elements while elements that do not implement this interface are called simple elements. If a control has child items then it will be implementing the interface and hence, is a complex element. If it didn't implement IAccessible there would be no way to gain access to the child elements. Elements that do not have any children do not need to implement the interface. Information about the element can be obtained from the complex parent element by passing in the associated child ID. In the DOM shown in the image above we have a list control that is a complex element. Under the list itself exists all of the list items. Since list items have no children and hence do not implement IAccessible they are simple elements.

Accessible Data

The IAccessible interface is provided so that information regarding a non-standard control can be obtained by an AT in order to provide the user feedback as to what is happening on screen. There are a number of different pieces of information available for this purpose. Below is a list of the data bits that can be retrieved through IAccessible that are relevant to the user. I have left out things such as parent, child, child count, keyboard shortcut, etc... as I imagine they are pretty self explanatory.

• Role - The role of a given control is what ATs use to determine what behavior to expect from a control. You should select a role after reviewing all of the available roles provided by MSAA and determining which role best fits your control. For example, if you were creating a control that could be clicked with the mouse and perhaps even activated with the spacebar than it would share very similar traits with a standard button. In that case, you would assign it a role of ROLE_SYSTEM_PUSHBUTTON. See MSDN for a complete list of available MSAA roles.
• Name - The accessible name of the control should be a brief identifying string. In most cases if your control has visible text that will be what the accessible name will be. For example, a button with the text "I'm a Button" on it should give "I'm a Button" back through get name function. A more complex example would be in the case of an edit field. If you have an edit field that the user is supposed to type a username into then your control should return "Username", or some variant thereof, back to the caller. In many cases you would have a static text control showing users what is to go in the field. In those cases it may be easier for you to retrieve the text that a sighted user would see from the static text control and return it instead. Please note, however, it is good practice to keep the name short and to the point. Too much textual information could lead to confusion for blind or low vision users.
• Value - The value of a control depends completely on what the control is. For example, a button has no value. Whether or not it is pressed is determined by its state. A good example of a control with a value would be an edit field. A field with an accessible name of "Username" would have a value of whatever the user has typed into the field or what has been set programmatically. Remember that this must always stay up-to-date with the actual value in the control so the user always has an accurate verbal representation of what is in the control.
• Description - The accessible description is a slightly open ended field. This attribute can be used in many different ways by different AT vendors. In most cases, it is good to populate this field with a slightly more verbose description of what the element is for. Previously we talked about returning "Username" for the username edit field. If we were to come up with a description of this field it could perhaps be set to "Please type in the your username for access to the Delta Corp. secure network".
• State - The state is just that; the state of the control. The state is returned back to the caller as a bunch of bits set in a 32 bit long. The states that are supported will depend on the control you implemented. For a list of all the available state constants please see MSDN.

MSAA Events

As mentioned above, MSAA events are the mechanism by which the application/control notifies the system that an event has happened pertaining to the control. ATs can determine when these events have happened and respond by notifying the user. An example would be if the user had a screen reader, such as Windows Narrator, running while using an application and a focus change event occurs. As focus moves to the MSAA compatible control an event is generated and passed to the OS. The screen reader would be listening for these events and would respond to the event by reading off the name of the newly focused control to the user. The same example can also be applied to states of controls. For example, if focus is sitting on a checkbox and the user uses the spacebar to change the selected state of the control. An event would be fired and eventually the screen reader would respond by notifying the user of the new state of the checkbox.

Microsoft provides a fairly comprehensive list of events that can be sent through the system to notify all listening ATs. If you would like to see a list of all the events available you can view them on this MSDN site. All events are transmitted to the OS by the API call NotifyWinEvent(..). The prototype for this function can be seen below.

void WINAPI NotifyWinEvent(
  DWORD event,
  HWND hwnd,
  LONG idObject,
  LONG idChild
);

Please note that I have been careful to say that all events are transmitted to the system (OS) and not the ATs themselves. All accessible events are pushed through the NotifyWinEvent(..) call and from there the OS does the work. All assistive technologies that wish to respond to the events on the system do so by using a hook. That hook is called after each event and from there the AT can decide on what to do with it. Below you will see the prototype for the function that is used to create such a hook.

HWINEVENTHOOK WINAPI SetWinEventHook(
  UINT eventMin,
  UINT eventMax,
  HMODULE hmodWinEventProc,
  WINEVENTPROC lpfnWinEventProc,
  DWORD idProcess,
  DWORD idThread,
  UINT dwflags
);

// The event procedure need to use this hook must have the 

// following prototype. (WINEVENTPROC)


VOID CALLBACK WinEventProc(
  HWINEVENTHOOK hWinEventHook,
  DWORD event,
  HWND hwnd,
  LONG idObject,
  LONG idChild,
  DWORD dwEventThread,
  DWORD dwmsEventTime
);

Putting it Together

So far we have talked about the IAccessible interface and events that are generated by controls. Most ATs respond to events by notifying the user of that event in some way. To do this, they often need to acquire information from the control itself which means they will need to get an IAccessible interface from us. So for the moment, the question for us is; How do we provide external applications with an IAccessible interface? The answer is simple: WM_GETOBJECT. When an AT gets an event or perhaps arrives at our window by walking down the MSAA DOM of parent windows they can use various Win32 API functions such as AccessibleObjectFromWindow or AccessibleObjectFromEvent to get back an interface. Under the hood Windows sends the server application a WM_GETOBJECT message. We then return the related interface back to the caller as the return value for the message after pushing it through the LresultFromObject API call. To see this in action look for the sample application we are going to build next!

Information Put to the Test

Now that you have a fair bit of background on all the players involved in this little dance its time to start putting some of this to use. To demonstrate how to create a custom control with an MSAA backing to it we are going to build an application that uses a revolutionary new control call "UglyButton". Now this control is no ordinary button mind, it is a button that has two more buttons contained within it. This control will have a single HWND for the control itself. The two buttons contained with the control will no HWND but will be represented to the user as part of the MSAA DOM. We will also have the ability to have a line of text above the two inner buttons since we are just that kind of wild and crazy folk.

Quick Note: I am not going to go over the inner workings of how to create custom controls and/or how to handle standard windows messages or anything else of the like. I expect you already have experience with Windows programming as a whole and if you need to understand how the control itself is being built please look here.

The Application

We are going to build an application that implements our ugly button control and places one in our main window. This application is a standard win32 application with no MFC support. The reason I have elected to do this without MFC is that I would like to expose the true flavor of implementing MSAA on a control that otherwise has no accessibility support. MFC has its own support of the IAccessible interface built into the CWnd base class. Once you understand how everything works together, going back and using MFC's built-in support should be a simple. It is important to understand the more complicated method first though as you may find yourself needing to work with a UI class library that is not MFC based.

Also of note, the control we are going to create does not have much functionality or standard support for things like resizing, moving, etc... This is just here to illustrate the MSAA side of things. Now that the disclaimer is out of the way; onward we march!

Our application has three main pieces; The core application file where the app's message pump exists, the ugly button control file and of course our accessible proxy object which has the implementation for IAccessible. I assume that those reading this article can navigate the UglyButton.cpp/h files and determine most everything that is going on there. There is not much to it and I will not waste time going over it here except in those places it relates to our MSAA implementation for the control. The same goes for the application file. This leaves us with the object implementing IAccessible, CAccProxyObject.

There is something very important to note here. We keep a CAccProxyObject stored in a window property for each ugly button. This object serves as our proxy to our ugly button control. We push a pointer to the button data structure into this proxy for it to hold on to. This is an important concept because if our ugly button goes away before the interface itself is ref counted to zero the button data within the proxy object will be nulled out. This allows all client applications that are holding onto this interface to get a special error code, CO_E_OBJNOTCONNECTED, if they attempt to use an interface after the element it represents is gone. If you didn't implement a proxy you could end up referencing a window and children elements that do not exist anymore through an interface that was obtained when they were still valid.

Reviewing the Proxy Object

At this point you may want to take a moment to review the code for the IAccessible object. Most of the code there is pretty easy to read and I attempted to keep it as well commented as possible to make understanding what was going on and why as easy as possible. While I do not intend to go over each function (I figure that's something you can do on your own) I would like to hit on a couple bullet points about using and interacting with the interface.

Working With Child IDs

It is important to notice that almost every single function in the IAccessible interface requires you pass it a child ID. If you remember from above, child IDs are used to reference elements contained under a complex element. In the case of our ugly button we have the interface being implemented for the button itself which is our complex element. This element can also be referred to by the special child ID, CHILDID_SELF. Under that element there are two simple elements which are the two inner buttons. These buttons are child IDs 1 and 2 respectively. If we had more dynamic content, such as a list box where the list items themselves were child items, we would have to determine the child IDs using a more dynamic process. For our purposes though, just knowing our button has two inner buttons lets us statically reference them by the 1 and 2 child IDs.

You can see the structure of our button control is the image of the MSAA DOM of our sample program below.

Since the concept of the child IDs is the same no matter what function we are dealing with I am just going to go through one of the functions quickly and leave the rest up to you for review. We will take a look at one of the more frequently used functions, get_accName. The implementation for this function is shown below.

STDMETHODIMP CAccProxyObject::get_accName(VARIANT varChild, BSTR *pszName)
{
  HRESULT retCode = DATACHECK(mData);
  if (SUCCEEDED(retCode))
  {
    if (pszName && VALID_CHILDID(varChild))
    {
      GENBTNDATA* pData = NULL;
      switch (varChild.lVal)
      {
        case CHILDID_SELF:
          pData = &mData->btnSelf;
          break;
        case 1:
          pData = &mData->btnOne;
          break;
        case 2:
          pData = &mData->btnTwo;
          break;
      };
            
      if (pData)
      {
        // First preference goes to the set accessible name. 

        // If no accessible name is available 

        if (pData->pszAccName)
          *pszName = ::SysAllocString(pData->pszAccName);
        else if (wcscmp(L"", pData->szText) != 0)
          *pszName = ::SysAllocString(pData->szText);
      }
    }
    else
      retCode = E_INVALIDARG;
  }
  return retCode;
}

As you can see from the code above, there really isn't much to implementing this function. The first parameter for the this function is the child ID of the item that we are to retrieve the name for. There are two macros that are used in this function that are not shown in the code here. DATACHECK is a macro that verifies that mData is non-null. If it isn't null the return code is set to S_OK. If our data element is null then it sets the return code to CO_E_OBJNOTCONNECTED. Remember, this was the whole point of having a proxy object. The other macro is VALID_CHILDID. This macro simply verifies that the type of the variant is VT_I4 and that the lVal member of the variant structure is between CHILDID_SELF (0) and 2. If it isn't within that range then we were given a bad child ID and need to return the invalid argument error code.

Once we are sure that we have a valid data element and that our child ID is correct we can go forward in retrieving the information requested. The switch statement takes the child ID passed in and gets the data block associated with either the outer button or one of the two inner buttons. From there we just create a copy of the string and set it to the storage location provided by the caller. They are responsible for freeing the string's memory. You may also note that we support two different types of accessible names. If someone calls the put_accName function to manually override the accessible name we store it in the pszAccName. If this element is never set then we simply attempt to use whatever visual text is on the button element itself.

Retrieving the Interface

Now that we have covered the interface, it would be helpful to know how it is obtained. The small block of code below is all it takes to get our IAccessible interface back to the client application.

case WM_GETOBJECT:
  {
    // This is the message that we must handle so that we can return the

    // IAccessible pointer that relates to this control.

    CAccProxyObject* pProxy = (CAccProxyObject*)GetProp(hWnd, MAKEINTATOM(
        UBPROPATOM_CACC));
    if (pProxy)
    {
      LRESULT lRes = LresultFromObject(IID_IAccessible, wParam, static_cast(
          pProxy));
      return lRes;
    }
  }
  break;

After a MSAA event is processed or one of the Win32 API function is used on our window a WM_GETOBJECT message will be sent to us requesting we give back an accessible interface for our window. Since we should already have our interface created we simply look it up from the window property and call the LresultFromObject API call to convert it to value we need to return. Since our proxy objects persist this is all we need to do here. If you were creating these objects on an as-needed basis only (this is not recommended) then you would need to call the Release() method after the LresultFromObject call since it does an AddRef() on your behalf. That's it! With this code a client application can now can now retrieve a copy of your IAccessible implementation.

Generating MSAA Events

Last but not least is our implementation of MSAA events. For this sample I did not bother implementing every possible event our button could possible have. Instead I picked the most important ones as they relate to a button control.

As the user interacts with the application, specifically our button, we need to pass MSAA events down to the OS so it can notify all listening ATs that something just happened. You can read the available reference above for a list of all the events but lets take a look at one in specific.

if (PtInRect(&lpData->btnOne.rcBounds, point))
{
  lpData->btnOne.bPushed = TRUE;
  lpData->btnOne.bHasKbFocus = TRUE;
  lpData->btnTwo.bHasKbFocus = FALSE;            
  lpData->btnSelf.bHasKbFocus = FALSE;            
  
  NotifyWinEvent(EVENT_OBJECT_STATECHANGE, hWnd, (LONG)&lpData->btnOne, 1);
}
else if(PtInRect(&lpData->btnTwo.rcBounds, point))
{
  lpData->btnTwo.bPushed = TRUE;            
  lpData->btnOne.bHasKbFocus = FALSE;
  lpData->btnTwo.bHasKbFocus = TRUE;
  lpData->btnSelf.bHasKbFocus = FALSE;            
  
  NotifyWinEvent(EVENT_OBJECT_STATECHANGE, hWnd, (LONG)&lpData->btnTwo, 2);
}
else
{
  lpData->btnSelf.bPushed = TRUE;
  lpData->btnOne.bHasKbFocus = FALSE;
  lpData->btnTwo.bHasKbFocus = FALSE;
  lpData->btnSelf.bHasKbFocus = TRUE;            

  NotifyWinEvent(EVENT_OBJECT_STATECHANGE, hWnd, (LONG)&lpData->btnSelf, 
      CHILDID_SELF);
}

This code comes out of the left button down handler. When someone presses down on the left mouse button the button it is over goes into a pressed state. Since the state of our button has now changed we need to notify the system that the change has happened. To do this we use the NotifyWinEvent Win32 API function and pass in the state change constant. Also note that we have to pass in the child ID that to which the event relates. For our button control we are passing in a custom identifier (a pointer to the button data) as the object ID. While in our sample application, we do not do anything special with this ID it would be possible, in a more complicated application, to use this custom data element for easy access to the element that generated the event. The client does not get to use this data so what it really is doesn't matter to them. It is merely a means for the server application to reference back to the event or item.

This is the same model that you should follow for all event types that your custom control is going to support. Follow the link in the section above to see a list of all the events Microsoft provides.

Conclusion

This concludes this article. I hope it was of some use to those of you actually looking at implementing MSAA for custom controls or libraries. In my next article I will look at using MSAA from the client perspective. I intend to do while writing a replacement for the event32.exe provided by Microsoft to see the accessible events being pushed through the system.

Points of Interest

• MSDN: MSAA Role List
• MSDN: MSAA State Constants
• MSDN: MSAA Events
• Active Accessibility Toolset

History

4/3/2007 -- Initial Posting. Version 1