Introduction
This is the first is a series of articles that I hope to write concerning the
Microsoft Active Accessibility (MSAA) technology. Much of my current work has
surrounded this topic and there seems to be very little in the way of useful
introductions to the topic. To start, I would like to provide an introduction to
MSAA so that an understanding of what it does and why it exists is present.
After the groundwork is established we can move into how it would be implemented in an application.
In the final articles in this series I would like
to offer a few new tools to aide in the development and debugging of your MSAA
enhanced applications and perhaps even go over how MSAA can be used for
application automation. The tools are meant to be a replacement for the tools
provided by Microsoft which seem to fall very short of usable. These tools will
have their source code made available for you to review. The tools are developed
with the same concepts that will be shown throughout this series and hence
should provide you with a perspective on implementing applications that use
MSAA.
The rest of the articles in this series will be written as my time permits.
Since there is not sure way to gauge this, each article will stand
on its own providing relevant information without the presence of the others.
Enjoy!
What is Accessibility?
Accessibility, in this context, is creating your application or framework in
such a manner that it will be usable by those users with certain physical
handicaps such as blindness, low version, hard-of-hearing or deafness. Depending
on where you are developing your application or who you are developing it for
you may be legally required to ensure your application is accessible. When
working with Microsoft Windows there are two main forms of accessible
technologies available to you.
The first and simplest is the system's ability to enter a High Contrast (HC)
mode. In this mode the system enters a color scheme, selected by the user,
that is supposed to provide great contrast between the foreground and
background. This is targeted at low-vision users. If you would like to see what
your systems look like in High Contrast mode you can hold down (LEFT-ALT +
LEFT-SHIFT + PRNTSCRN). You can also go to Control Panel -> Accessibility
Options -> Display. Once activated you will see your system change into a
contrasting color scheme. The default high contrast scheme is black backgrounds
with white foregrounds. Notice that the title bars also change colors. This can
be an important thing to remember when developing your application because the
switch is based on system colors. These are the colors that can be obtained
through the ::GetSysColor(..) API. In all places that you use custom colors,
those colors will stay the same even when in HC mode. This may make your
application harder to use for those people with vision impairments. When
possible, you should strive to use system colors to keep your application
accessible.
The second form of accessibility available to applications is MSAA. If your
application uses the standard Win32 controls and that is all then it is very
likely you will have little to no work to do to be accessible. Microsoft
provides interfaces for all of its controls. If you are developing a toolkit of
custom controls or perhaps just a single control for use in your application
then it may become necessary for you to add MSAA support for those custom
controls. Since Windows does not know the purpose of your control, what it is or
how it is structured it is necessary for you to be able to relay that
information so that Assistive Technologies (ATs) such as screen readers,
magnifiers and other devices such as Braille displays can properly relay
information about your control or controls to the user. The sections below will
go into more detail concerning how MSAA works and what is required for you to
implement it for a given control.
The Pieces to the MSAA Puzzle
There are two main mechanisms to be aware of when in comes to MSAA; the MSAA
event model and the IAccessible COM interface. The event model is the mechanism
applications/controls use to notify the system that a particular event has taken
place within a control. This can be something such as a the control just gained
focus or the user modified value of the control. The IAccessible interface is a
COM interface that can be implemented in relation to a control so that an AT can
query information about the control. This interface would be used to get the
current value, determine the state of the control (selected, focused, disabled,
etc..) and even perform actions such as moving through the MSAA DOM.
The IAccessible Interface
The first thing to cover is going to be the IAccessible interface and its
inner workings. This is necessary because some of the fundamental concepts
explained here, as they relate to the MSAA DOM, are required to understand how
MSAA events transmitted to the system are then in turn queried by ATs for a
greater level of information. Please note that this section goes into a fair bit
of detail concerning the various aspects of the MSAA interface including DOM
construction, roles, identification elements such as names and descriptions and
state data. For a single control it is unlikely you will be concerned with
building an entire DOM of elements rather you will be looking to implement a
single element in an existing MSAA DOM. Even though most will be looking at the
smaller case I wanted to provide as much detail as possible so that those people
looking for an understanding of the whole picture can see how everything works.
From that point, you should be able to infer all the information you need for a
single element. People that are likely to need to build an entire MSAA DOM are
those people putting together an entire UI toolkit of custom controls. Since you
cannot rely on Microsoft's controls at that point the burden of providing MSAA
interfaces falls on you.
The MSAA DOM for an application is a tree structured group of elements that
defines the accessible elements for an application. This means that each control
that can be used by the end user should show up in the DOM. For a standard
application, one that uses Std. Win32 controls, each control element such as
edits, lists, list items, trees, tree items, menus, etc... will have a related
elements within the MSAA DOM. If you have a custom control that has a visual-only
purpose, such as a spacer control or other type of element that helps with the
layout of visual/usable controls it it not necessary for it to implement
IAccessible. Since the user cannot interact with this control and it provides
no information to the user, it can be left out of the DOM. This is can even be
considered good practice as the DOM will be more compact, hence more efficient
to walk.
Below you can see a view of an MSAA DOM I retrieved by inspecting a folder
view through Windows Explorer.
This image shows us a couple of things. First, I started the inspection at the
control level rather than the window level. What we get in a standard list
control and all my files/folders which show up as elements within the list.
First, let's examine the structure of a MSAA DOM, using the above image as a
reference point, and after that we can examine the various information that can
be conveyed through the IAccessible interface on each of the elements. Notice in
the structure above the first element is that of the list control itself. Each
of the items within the list exist as children to the container. You also can
see, at the bottom of the tree, where the column headers are shown. Each of
these elements exist as children to the list as well, albeit at a different
level.
Before we dive headfirst into how the IAccessible interface is used to walk
over and read from the MSAA DOM take a look at the code block below. This is
what the actual COM interface looks like.
interface IAccessible : IDispatch
{
HRESULT STDMETHODCALLTYPE get_accParent([retval][out]
IDispatch **ppdispParent);
HRESULT STDMETHODCALLTYPE get_accChildCount(
[retval][out] long *pcountChildren);
HRESULT STDMETHODCALLTYPE get_accChild([in] VARIANT varChild,
[retval][out] IDispatch **ppdispChild);
HRESULT STDMETHODCALLTYPE get_accName([in] VARIANT varChild,
[retval][out] BSTR *pszName);
HRESULT STDMETHODCALLTYPE get_accValue([in] VARIANT varChild,
[retval][out] BSTR *pszValue);
HRESULT STDMETHODCALLTYPE get_accDescription([in] VARIANT varChild,
[retval][out] BSTR *pszDescription);
HRESULT STDMETHODCALLTYPE get_accRole([in] VARIANT varChild,
[retval][out] VARIANT *pvarRole);
HRESULT STDMETHODCALLTYPE get_accState([in] VARIANT varChild,
[retval][out] VARIANT *pvarState);
HRESULT STDMETHODCALLTYPE get_accHelp([in] VARIANT varChild,
[retval][out] BSTR *pszHelp);
HRESULT STDMETHODCALLTYPE get_accHelpTopic([out] BSTR *pszHelpFile,
[in] VARIANT varChild,[retval][out] long *pidTopic);
HRESULT STDMETHODCALLTYPE get_accKeyboardShortcut([in] VARIANT varChild,
[retval][out] BSTR *pszKeyboardShortcut);
HRESULT STDMETHODCALLTYPE get_accFocus([retval][out] VARIANT *pvarChild);
HRESULT STDMETHODCALLTYPE get_accSelection([retval][out]
VARIANT *pvarChildren);
HRESULT STDMETHODCALLTYPE get_accDefaultAction([in] VARIANT varChild,
[retval][out] BSTR *pszDefaultAction);
HRESULT STDMETHODCALLTYPE accSelect([in] long flagsSelect,
[in] VARIANT varChild);
HRESULT STDMETHODCALLTYPE accLocation([out] long *pxLeft,
[out] long *pyTop,[out] long *pcxWidth,[out] long *pcyHeight,
[in] VARIANT varChild);
HRESULT STDMETHODCALLTYPE accNavigate([in] long navDir,
[in] VARIANT varStart, [retval][out] VARIANT *pvarEndUpAt);
HRESULT STDMETHODCALLTYPE accHitTest([in] long xLeft, [in] long yTop,
[retval][out] VARIANT *pvarChild);
HRESULT STDMETHODCALLTYPE accDoDefaultAction([in] VARIANT varChild);
HRESULT STDMETHODCALLTYPE put_accName([in] VARIANT varChild,
[in] BSTR szName);
HRESULT STDMETHODCALLTYPE put_accValue([in] VARIANT varChild,
[in] BSTR szValue);
};
When it comes to constructing the MSAA DOM and in turn, walking through it,
there are two main points to understand. Point #1 is that not all elements
implement the IAccessible interface. Point #2 is that all elements can be found
using a child ID. Children under a given element are located through a 1-based index. There is a
special child id called CHILDID_SELF (this constant equals 0) that, when
used with a function like get_accChild, returns the element itself rather than a
child.
Elements in the MSAA DOM that implement the IAccessible interface are called
complex elements while elements that do not implement this interface are called
simple elements. If a control has child items then it will be implementing the
interface and hence, is a complex element. If it didn't implement IAccessible there would be no way to gain
access to the child elements. Elements that do not have any children do not need
to implement the interface. Information about the element can be obtained from
the complex parent element by passing in the associated child ID. In the DOM
shown in the image above we have a list control that is a complex element. Under
the list itself exists all of the list items. Since list items have no children
and hence do not implement IAccessible they are simple elements.
Accessible Data
The IAccessible interface is provided so that information regarding a
non-standard control can be obtained by an AT in order to provide the user
feedback as to what is happening on screen. There are a number of different
pieces of information available for this purpose. Below is a list of the data
bits that can be retrieved through IAccessible that are relevant to the user. I
have left out things such as parent, child, child count, keyboard shortcut,
etc... as I imagine they are pretty self explanatory.
- Role - The role of a given control is what ATs use to
determine what behavior to expect from a control. You should select a
role after reviewing all of the available roles provided by MSAA and
determining which role best fits your control. For example, if you were
creating a control that could be clicked with the mouse and perhaps even
activated with the spacebar than it would share very similar traits with
a standard button. In that case, you would assign it a role of
ROLE_SYSTEM_PUSHBUTTON.
See MSDN for a complete list of available MSAA roles.
- Name - The accessible name of the control should be a brief
identifying string. In most cases if your control has visible text that
will be what the accessible name will be. For example, a button with the
text "I'm a Button" on it should give "I'm a Button" back through get
name function. A more complex example would be in the case of an edit
field. If you have an edit field that the user is supposed to type a
username into then your control should return "Username", or some
variant thereof, back to the caller. In many cases you would have a
static text control showing users what is to go in the field. In those
cases it may be easier for you to retrieve the text that a sighted user
would see from the static text control and return it instead. Please
note, however, it is good practice to keep the name short and to the
point. Too much textual information could lead to confusion for blind or
low vision users.
- Value - The value of a control depends completely on what the
control is. For example, a button has no value. Whether or not it is
pressed is determined by its state. A good example of a control with a
value would be an edit field. A field with an accessible name of
"Username" would have a value of whatever the user has typed into the
field or what has been set programmatically. Remember that this must
always stay up-to-date with the actual value in the control so the user
always has an accurate verbal representation of what is in the control.
- Description - The accessible description is a slightly open
ended field. This attribute can be used in many different ways by
different AT vendors. In most cases, it is good to populate this field
with a slightly more verbose description of what the element is for.
Previously we talked about returning "Username" for the username edit
field. If we were to come up with a description of this field it could
perhaps be set to "Please type in the your username for access to the
Delta Corp. secure network".
- State - The state is just that; the state of the control. The
state is returned back to the caller as a bunch of bits set in a 32 bit
long. The states that are supported will depend on the control you
implemented.
For a list of all the available state constants please see MSDN.
MSAA Events
As mentioned above, MSAA events are the mechanism by which the
application/control notifies the system that an event has happened pertaining to
the control. ATs can determine when these events have happened and respond by
notifying the user. An example would be if the user had a screen reader, such as
Windows Narrator, running while using an application and a focus change event
occurs. As focus moves to the MSAA compatible control an event is generated and
passed to the OS. The screen reader would be listening for these events and
would respond to the event by reading off the name of the newly focused control
to the user. The same example can also be applied to states of controls. For
example, if focus is sitting on a checkbox and the user uses the spacebar to
change the selected state of the control. An event would be fired and eventually
the screen reader would respond by notifying the user of the new state of the
checkbox.
Microsoft provides a fairly comprehensive list of events that can be sent
through the system to notify all listening ATs. If you would like to see a list
of all the events available you can view them on
this MSDN site. All events are transmitted to the OS by the API call
NotifyWinEvent(..). The prototype for this function can be seen below.
void WINAPI NotifyWinEvent(
DWORD event,
HWND hwnd,
LONG idObject,
LONG idChild
);
Please note that I have been careful to say that all events are transmitted
to the system (OS) and not the ATs themselves. All accessible events are pushed
through the
NotifyWinEvent(..) call and from there the OS does the work. All
assistive technologies that wish to respond to the events on the system do so by
using a hook. That hook is called after each event and from there the AT
can decide on what to do with it. Below you will see the prototype for the
function that is used to create such a hook.
HWINEVENTHOOK WINAPI SetWinEventHook(
UINT eventMin,
UINT eventMax,
HMODULE hmodWinEventProc,
WINEVENTPROC lpfnWinEventProc,
DWORD idProcess,
DWORD idThread,
UINT dwflags
);
VOID CALLBACK WinEventProc(
HWINEVENTHOOK hWinEventHook,
DWORD event,
HWND hwnd,
LONG idObject,
LONG idChild,
DWORD dwEventThread,
DWORD dwmsEventTime
);
Putting it Together
So far we have talked about the IAccessible interface and events that are
generated by controls.
Most ATs respond to events by notifying the user of that event in some way.
To do this, they often need to acquire information from the control itself
which means they will need to get an IAccessible interface from us. So for the
moment, the question for us is; How do we provide external applications with an
IAccessible interface? The answer is simple: WM_GETOBJECT. When an
AT gets an event or perhaps arrives at our window by walking down the MSAA DOM
of parent windows they can use various Win32 API functions such as
AccessibleObjectFromWindow or AccessibleObjectFromEvent to get back an
interface. Under the hood Windows sends the server application a WM_GETOBJECT
message. We then return the related interface back to the caller as the
return value for the message after pushing it through the LresultFromObject API
call. To see this in action look for the sample application we are going to
build next!
Information Put to the Test
Now that you have a fair bit of background on all the players involved in
this little dance its time to start putting some of this to use. To demonstrate
how to create a custom control with an MSAA backing to it we are going to build
an application that uses a revolutionary new control call "UglyButton". Now
this control is no ordinary button mind, it is a button that has two more
buttons contained within it. This control will have a single HWND for the
control itself. The two buttons contained with the control will no HWND but will
be represented to the user as part of the MSAA DOM. We will also have the
ability to have a line of text above the two inner buttons since we are just
that kind of wild and crazy folk.
Quick Note: I am not going to go over the inner workings of how to create
custom controls and/or how to handle standard windows messages or anything else
of the like. I expect you already have experience with Windows programming as a
whole and if you need to understand how the control itself is being built please
look here.
The Application
We are going to build an application that implements our ugly button control
and places one in our main window. This application
is a standard win32 application with no MFC support. The reason I have elected
to do this without MFC is that I would like to expose the true flavor of
implementing MSAA on a control that otherwise has no accessibility support. MFC
has its own support of the IAccessible interface built into the CWnd base class.
Once you understand how everything works together, going back and using MFC's
built-in support should be a simple. It is important to understand the more
complicated method first though as you may find yourself needing to work with a
UI class library that is not MFC based.
Also of note, the control we are going
to create does not have much functionality or standard support for things like
resizing, moving, etc... This is just here to illustrate the MSAA side of
things. Now that the disclaimer is out of the way; onward we march!
Our
application has three main pieces; The core application file where the app's message
pump exists, the ugly button control file and of course our accessible proxy
object which has the implementation for IAccessible. I assume that those reading
this article can navigate the UglyButton.cpp/h files and determine most
everything that is going on there. There is not much to it and I will not waste
time going over it here except in those places it relates to our MSAA
implementation for the control. The same goes for the application file. This
leaves us with the object implementing IAccessible, CAccProxyObject.
There is something very important to note here. We keep a
CAccProxyObject stored in a window property for each ugly button. This object
serves as our proxy to our ugly button control. We push a pointer to the button
data structure into this proxy for it to hold on to. This is an important
concept because if our ugly button goes away before the interface itself is ref
counted to zero the button data within the proxy object will be nulled out. This
allows all client applications that are holding onto this interface to get a
special error code, CO_E_OBJNOTCONNECTED, if they attempt to use an interface
after the element it represents is gone. If you didn't implement a proxy you
could end up referencing a window and children elements that do not exist
anymore through an interface that was obtained when they were still valid.
Reviewing the Proxy Object
At this point you may want to take a moment to review the code for the
IAccessible object. Most of the code there is pretty easy to read and I
attempted to keep it as well commented as possible to make understanding what
was going on and why as easy as possible. While I do not intend to go over each
function (I figure that's something you can do on your own) I would like to hit
on a couple bullet points about using and interacting with the interface.
Working With Child IDs
It is important to notice that almost every single function in the
IAccessible interface requires you pass it a child ID. If you remember from
above, child IDs are used to reference elements contained under a complex
element. In the case of our ugly button we have the interface being implemented
for the button itself which is our complex element. This element can also be
referred to by the special child ID, CHILDID_SELF. Under that element there are
two simple elements which are the two inner buttons. These buttons are child IDs
1 and 2 respectively. If we had more dynamic content, such as a list box where
the list items themselves were child items, we would have to determine the child
IDs using a more dynamic process. For our purposes though, just knowing our
button has two inner buttons lets us statically reference them by the 1 and 2
child IDs.
You can see the structure of our button control is the image of the
MSAA DOM of our sample program below.
Since the
concept of the child IDs is the same no matter what function we are dealing with
I am just going to go through one of the functions quickly and leave the rest up
to you for review. We will take a look at one of the more frequently used
functions, get_accName. The implementation for this function is shown below.
STDMETHODIMP CAccProxyObject::get_accName(VARIANT varChild, BSTR *pszName)
{
HRESULT retCode = DATACHECK(mData);
if (SUCCEEDED(retCode))
{
if (pszName && VALID_CHILDID(varChild))
{
GENBTNDATA* pData = NULL;
switch (varChild.lVal)
{
case CHILDID_SELF:
pData = &mData->btnSelf;
break;
case 1:
pData = &mData->btnOne;
break;
case 2:
pData = &mData->btnTwo;
break;
};
if (pData)
{
if (pData->pszAccName)
*pszName = ::SysAllocString(pData->pszAccName);
else if (wcscmp(L"", pData->szText) != 0)
*pszName = ::SysAllocString(pData->szText);
}
}
else
retCode = E_INVALIDARG;
}
return retCode;
}
As you can see from the code above, there really isn't much to implementing
this function. The first parameter for the this function is the child ID of the
item that we are to retrieve the name for. There are two macros that are used in
this function that are not shown in the code here. DATACHECK is a macro that
verifies that mData is non-null. If it isn't null the return code is set to S_OK.
If our data element is null then it sets the return code to CO_E_OBJNOTCONNECTED.
Remember, this was the whole point of having a proxy object. The other macro is
VALID_CHILDID. This macro simply verifies that the type of the variant is VT_I4
and that the lVal member of the variant structure is between CHILDID_SELF (0)
and 2. If it isn't within that range then we were given a bad child ID and need
to return the invalid argument error code.
Once we are sure that we have a valid data element and that our child ID is
correct we can go forward in retrieving the information requested. The switch
statement takes the child ID passed in and gets the data block associated with
either the outer button or one of the two inner buttons. From there we just
create a copy of the string and set it to the storage location provided by the
caller. They are responsible for freeing the string's memory. You may also note
that we support two different types of accessible names. If someone calls the
put_accName function to manually override the accessible name we store
it in the pszAccName. If this element is never set then we simply attempt to use
whatever visual text is on the button element itself.
Retrieving the Interface
Now that we have covered the interface, it would be helpful to know how it is
obtained. The small block of code below is all it takes to get our IAccessible
interface back to the client application.
case WM_GETOBJECT:
{
CAccProxyObject* pProxy = (CAccProxyObject*)GetProp(hWnd, MAKEINTATOM(
UBPROPATOM_CACC));
if (pProxy)
{
LRESULT lRes = LresultFromObject(IID_IAccessible, wParam, static_cast(
pProxy));
return lRes;
}
}
break;
After a MSAA event is processed or one of the Win32 API function is used on our
window a WM_GETOBJECT message will be sent to us requesting we give back an
accessible interface for our window. Since we should already have our interface
created we simply look it up from the window property and call the
LresultFromObject API call to convert it to value we need to return. Since our
proxy objects persist this is all we need to do here. If you were creating these
objects on an as-needed basis only (this is not recommended) then you would need
to call the Release() method after the LresultFromObject call since it does an
AddRef() on your behalf. That's it! With this code a client application can now
can now retrieve a copy of your IAccessible implementation.
Generating MSAA Events
Last but not least is our implementation of MSAA events. For this sample I
did not bother implementing every possible event our button could possible have.
Instead I picked the most important ones as they relate to a button control.
As the user interacts with the application, specifically our button, we need
to pass MSAA events down to the OS so it can notify all listening ATs that
something just happened. You can read the available reference above for a list
of all the events but lets take a look at one in specific.
if (PtInRect(&lpData->btnOne.rcBounds, point))
{
lpData->btnOne.bPushed = TRUE;
lpData->btnOne.bHasKbFocus = TRUE;
lpData->btnTwo.bHasKbFocus = FALSE;
lpData->btnSelf.bHasKbFocus = FALSE;
NotifyWinEvent(EVENT_OBJECT_STATECHANGE, hWnd, (LONG)&lpData->btnOne, 1);
}
else if(PtInRect(&lpData->btnTwo.rcBounds, point))
{
lpData->btnTwo.bPushed = TRUE;
lpData->btnOne.bHasKbFocus = FALSE;
lpData->btnTwo.bHasKbFocus = TRUE;
lpData->btnSelf.bHasKbFocus = FALSE;
NotifyWinEvent(EVENT_OBJECT_STATECHANGE, hWnd, (LONG)&lpData->btnTwo, 2);
}
else
{
lpData->btnSelf.bPushed = TRUE;
lpData->btnOne.bHasKbFocus = FALSE;
lpData->btnTwo.bHasKbFocus = FALSE;
lpData->btnSelf.bHasKbFocus = TRUE;
NotifyWinEvent(EVENT_OBJECT_STATECHANGE, hWnd, (LONG)&lpData->btnSelf,
CHILDID_SELF);
}
This code comes out of the left button down handler. When someone presses down on the left mouse button the button it is over
goes into a pressed state. Since the state of our button has now changed we need to notify the system that the change has
happened. To do this we use the NotifyWinEvent Win32 API function and pass in the state change constant. Also note that
we have to pass in the child ID that to which the event relates. For our button control we are passing in a custom identifier (a pointer to the button data)
as the object ID. While in our sample application, we do not do anything special with this ID it would be possible, in a more complicated
application, to use this custom data element for easy access to the element that generated the event. The client does not get to use this data
so what it really is doesn't matter to them. It is merely a means for the server application to reference back to the event or item.
This is the same model that you should follow for all event types that your
custom control is going to support. Follow the link in the section above to see
a list of all the events Microsoft provides.
Conclusion
This concludes this article. I hope it was of some use to those of you
actually looking at implementing MSAA for custom controls or libraries. In my
next article I will look at using MSAA from the client perspective. I intend to
do while writing a replacement for the event32.exe provided by Microsoft to see
the accessible events being pushed through the system.
Points of Interest
History
4/3/2007 -- Initial Posting. Version 1
