Before I begin, I just want to say that I do not consider myself to be an
expert in usability testing, user interface design or just about any other
area. I do, however, have some experience with these areas (both positive
and negative) and I decided to write this article to share my experience with
others. The concepts and ideas I discuss in this article are not
earth-shattering and many of you will already know all of this information.
My current projects have involved a need for wide consumer acceptance and
usability. One project will need to be shipped nation-wide and the other
will be open to web-users from just about anywhere, anytime. The details
of these projects are not important, but what is important is the process I used
to test, refine and re-test these projects and the ongoing assessments
requirements I have uncovered.
The need for software developers to be involved in the usability testing and
UI design is (IMHO) critical to achieving maximum usability. Software
developers have an intimate understanding of what is possible and what is
not. We also understand what is involved in making a particular solution
work. Our involvement in the early stages of UI design and later usability
testing can mean the difference between a mediocre software package which does
the job and an excellent software package that users appreciate, understand and
I have learned from experience that it is important to have a broad knowledge
of technologies, techniques, utilities and tools available to me. With
this knowledge, I can choose the appropriate language (VC++, VB, C#, ASP, PHP,
etc) and technologies (ADO, ODBC, JDBC, etc.) to achieve a desired result.
One area of knowledge that I have omitted in the past has been non-developer
product review. I believe that this is an important area as well. In the
past year I have spent an enormous amount of time reviewing 3rd party consumer
apps, games, business software, etc. By doing this, I have a broader
scope of user-oriented knowledge. I can see how other companies have
solved particular problems and learn from their successes and mistakes.
What is Usability Testing
Usability testing is the process of working with end-users directly and
indirectly to assess how the user perceives a software package and how they
interact with it. This process will uncover areas of difficulty for users
as well as areas of strength. The goal of usability testing should be to
limit and remove difficulties for users and to leverage areas of strength for
This testing should ideally involve direct user feedback, indirect feedback
(observed behavior), and when possible computer supported feedback.
Computer supported feedback is often (if not always) left out of this process.
Computer supported feedback can be as simple as a timer on a dialog to monitor
how long it takes users to use the dialog and counters to determine how often
certain conditions occur (ie. error messages, help messages, etc). Often,
this involves trivial modifications to existing software, but can result in
tremendous return on investment.
Ultimately, usability testing should result in changes to the delivered
product in line with the discoveries made regarding usability. These
changes should be directly related to real-world usability by average
users. As much as possible, documentation should be written supporting
changes so that in the future, similar situations can be handled with ease.
When to Begin
This process can and should begin as early in the development process as
possible. It need not involve a large number of users, in fact 3-10 is
probably ideal depending on the scale of the product. These users should
ideally be intended users for the product (ie. alpha/beta testers) and should
represent a decent cross section of users targeted by the application.
The real question should be "when to end". I believe that
this is an incremental process with many beginnings and endings.
Ultimately, the job is not done until the product has reached the end of its
lifecycle (not just development cycle). Each incremental step should be
relatively short with changes and supporting documentation made often (up until
initial delivery of the product.) Once a product is initially delivered it
can be difficult to make these kind of changes without affecting active
users. Once the product is shipped, changes should be considered more
carefully with special concern for how active users will be affected and how
future users will benefit.
However, it is never too late to start. Even if you are nearing the end
of the development cycle, usability testing can still yield enormous
results. Sometimes even minor changes to the UI, help system, reports, etc
can make the product more appealing to users.
How to Begin
Usability testing can be quite simple. I think there are 4 basic
ingredients that are essential to success. These are
- The usability testing person/team needs to include a
software developer who is open-minded about changes and not offended by
criticism. The goal of usability testing is not to criticize, but
improve and learn. If any member of the team is not ready to receive
criticism with an open-mind, the testing will almost certainly fail.
This person needs to have a good working knowledge of the workflow process the
application is designed to facilitate and needs to have good communication
skills. Good note taking skills are also essential.
- The users selected as test subjects need to be
average users (not all power users, and not all entry-level users.) They
should understand what the application is designed to do and should be able to
communicate their needs reasonably well. Often, these users will not be
able to communicate their needs in technical terms, but will be able to
identify problems they are having.
- The development person/team needs to be prepared to
make changes on an incremental basis in relatively fast paced
environment. The best situation is to make needed changes quickly so
that those changes can be incorporated into the continuing testing process.
- Patience. The usability testing and refinement process can take some
time and will sometimes go in the wrong direction. By keeping changes
small and incremental, it is usually easy to back-track and rework the
problem without significant setbacks.
As I mentioned in the introduction, there are 3 methods of feedback that need
to be incorporated into your testing. These are
- Direct user feedback. This type of feedback
usually occurs by having the test users use the software alone and report back
to the usability team/person. Ideally, reporting back should occur on a
regular basis (daily or weekly at most).
- Observed behavior. This type of feedback can
occur in conjunction with direct user feedback. This occurs when the
testing team/person observes how users use the software.
- Computer supported feedback. This type of feedback can occur on an
ongoing basis throughout the testing process. As mentioned this is
usually quite simple involving timers and hit counters.
Each of these feedback methods should be used to achieve the ultimate goal.
How to best Leverage Feedback Methods
I believe that their are "best practices" for most things, but I am
not sure what those are in all situations. I will, however, tell you some
of the things I found that worked well.
- Provide users with notebooks, pre-printed forms, a
simple program, web page, etc to record their problems as they are having
them. They tend to forget the details if this is not done.
- Regularly review the issues users report.
- Meet with users on a scheduled (somewhat) basis to
discuss their issues and make sure that you fully understand it before
proceeding. Be prepared for this meeting by reviewing their issues
prior to the meeting.
- Keep a good communication dialog open with all
- Prioritize direct-feedback issues highly. Users need to see
results relatively quickly or they get discouraged with the process.
- Use multiple opportunities to observe
behavior. Whether you are training them, discussing problems found
earlier, or just walking past their desks, these are opportunities to
observe their behavior. (I doubt many of us have 1 way glass to watch
users, so I won't go into that here.)
- Take notes (use same forms, software, web pages,
etc as users use if possible. This will insure you do not forget what
- Compare the notes you are taking against the notes
users are taking. If users are not reporting nearly as many problems
as you are finding, it is possible that they are not comfortable with the
- Keep a good communication dialog open with all
- Don't interfere with their normal process.
The goal is not to train them on the "right way", but rather to have the
software work "their way".
- Be prepared to misinterpret user behavior.
Sometimes you might observe a user having an apparent problem, but in-fact
you are misunderstanding what they are doing.
- Prioritize observed behavior after direct-feedback. Users are not
expecting these changes and they can cause confusion. Carefully
review your observations and discuss them with your team and the users.
Computer Supported Feedback
- It is hard to know where to use computer supported
feedback early in the development and testing cycles. Therefore start
simple and grow from there.
- Be careful that the code that supports this does
not interfere with the users workflow and by all means does not crash the
software. (Been there, done that.)
- If possible, log all computer-feedback issues into
a simple database (Access worked well for me.)
- When reviewing the log, be very careful to not
overlook issues and not misinterpret the data. (Your method need not
be statistically valid, just reasonable.)
- When you see an issue in the data, try and support
it through direct feedback and observed behavior methods. This data
can be very helpful in knowing what to look for when working with users.
- Consider leaving this capability in the final
product carefully. Only do so, if the log can be disabled safely and
it has been thoroughly tested against all issues (especially exceptional
growth in log size.)
- I recommend against acting on computer feedback alone.
Real World Examples
To support what I am saying, I will discuss a few of my experiences.
Entering dates and times
In one of my current projects there is a need for users to enter dates on
various forms and for various purposes. Some of these situations require
that the user be able to quickly enter the data on a form (including the dates)
and enter forms in fast succession.
In the early version of the application, dates were entered with a masked
edit-control. Early on I observed that users were having problems
entering dates. This was backed up when I discussed it with the
users. I tried various masks, only to increase confusion and
problems. I tried using a date picker control, but users quickly let me
know that this slowed them down. I decided to just let them enter the
dates in a free-form edit control. When I did this, I integrated a log
to track which users were entering dates and what they were entering.
Users liked this method much better (as they informed me).
I was not pleased with this solution, though, because it allowed too much
freedom and room for error. Upon reviewing the log I discovered that not
only were users entering dates and times in numerous different formats (ie.
MM/DD/YY, MM/DD/YYYY, DD-MM-YYYY, MM-DD), they were entering a lot of
"invalid dates" (ie. "Yesterday", "tomorrow",
etc.) I reviewed other applications which allow date entry and
discovered that MS Outlook allows for date entry in just about any conceivable
I decided to develop an edit control that mimicked the MS Outlook style
date entry edit control. Once this solution was integrated into the
system, I heard nothing more from the users and when reviewing the log, the
number of "invalid dates" had been reduced tremendously. The result
of this control is available here on Code Project at http://www.codeproject.com/useritems/dateparser.asp.
Searching for records in the database
In the same application as the date entry problems occurred, there is a
need for the user to lookup various records from the database. We had
identified this as a need early in the pre-development stage and had provided
a search edit-control on a navigation bar on the left of the screen.
This feature was well received and users did not report any problems using
When observing the users behavior, however, it was apparent that users were
needing to enter multiple search criteria to find the record they were looking
for. They would try entering the name of an organization, only to have
to enter it again with a different spelling before it was found. Since users
were not reporting any problems, I decided to test what was actually
My next step was to begin logging searches attempts, result sets and
whether or not the user opened a record returned by the search. I let
this run for about a week before reviewing the results. When I reviewed
the log, I discovered that people were especially having problems with names
like "Saint Peters", "Mount Saint Helens", etc. I
spoke with the users about my findings and the response was basically that
they had this problem with all software they used and thought it was normal.
I was not happy with this answer, so I looked into ways to improve the
search results. The solution was fairly simple. I created a table
with a set of common name prefixes, suffixes, address labels, etc. I
then wrote an algorithm to take a string and parse it for these common
situations and rebuild the string. I then created an extra column in
every table for each column I needed to search. I placed the
rebuilt-version of the string in this new column each time a database update occurred.
The algorithm basically took a word like "Saint Peters" and made it
into "ST PETERS", "Mount Saint Helens" became "Mt St
Helens". Addresses were also dealt with so that "1600 W Palm
Road" would become "1600 WEST Palm". Other details of the
algorithm attempted to deal with spelling situations by removing duplicate
characters, extra white space, etc.
When this new search algorithm was introduced, I was able to see an
improvement in the search results by reviewing the log. It was obvious
that most search requests were now fulfilled on the first attempt. (This
search mechanism is now known as "quick search", not because it
returns results faster, but because it finds the desired records on the first
attempt.) I hope to post an article with the code for this search
Where to put the buttons on a dialog?
Early on in the development of one project I had a dispute with another
co-worker about the placement of buttons on a dialog. Should they be on
the upper-right aligned vertically or lower right aligned horizontally.
To resolve this dispute I redesigned most of the dialogs (probably about 25)
in the app so that they had buttons in both places. I then logged which
buttons the users clicked.
The results were interesting. On small dialogs there did not appear
to be any preference among users for which buttons they used. For
dialogs of greater height, however, users almost unanimously preferred buttons
horizontally aligned on the lower right.
We redesigned all of the dialogs in the app to consistently have the
buttons on the lower-right aligned horizontally. We decided to change
even the small dialogs to insure consistency.
This was a simple case, but I believe it improved the final product.
What to expect
Users usually appreciate it when they are involved in the development
process. They feel like their opinions matter and once the process begins
usually become better at identifying and describing their problems. In the
early stages their feedback can be sporadic and poorly described. Often
they have difficulty identifying the problems they are having because it is a
new software package and sometimes they seem to think that the problems they are
having occur just because they are not good yet.
As much as possible try to instill in the users the idea that their feedback
is important and that any problems they are having are important to the
process. This can result in a deluge of problems in the early stages
(especially if usability testing occurs late in the development cycle).
This can seem overwhelming but should be considered an opportunity instead of a
problem. By reviewing the problems and prioritizing solutions, you can
manage the process and control the ultimate outcome.
Once the testing has been going on for a while, things will probably slow
down with fewer problems and issues. This is an excellent opportunity to
begin leveraging computer supported feedback. At this point users are
using the software and testing many parts of its implementation. Computer
supported feedback becomes more valuable at this point because usage is more
defined and earlier problems have been dealt with. Computer assisted
feedback can identify more subtle problems that are often overlooked in earlier
testing. (I will discuss some real-world ways to leverage computer
supported feedback in a later section.)
Another thing to expect is set-backs. Sometimes the changes introduced
will result in more problems instead of the desired solution. This is why
it is important to keep changes small and incremental as much as possible.
Set-backs will be minor and easy to overcome. (A good source
control/version control package can help too.) When set-backs do occur,
documentation is critical to avoid the same problem later in this project or
others. Set-backs can be turned into opportunities by discussing why the proposed
solution did not work or wasn't what they needed. Many users will perceive
this as you diligently attempting to meet their needs and will be inclined to
participate even more.
The rewards for the usability testing will be
- Improved user perception of your product.
Because you have involved typical users in the process, the final product is
typically better geared for the target audience. Also, as a result of
the testing you may have established an excellent reference for the quality of
- Reduced technical support requirements.
Hopefully, the usability testing has uncovered and corrected many problems
(not just "bugs") in the application. Also, the overall user experience
is improved, thus eliminating the need for many calls.
- Better documentation of application. Throughout
the testing process notes have been collected and organized. These notes
can be given to the documentation team/person to better develop product
documentation. The "How do I...", "Where is the..." type questions have
already been uncovered and answered. All that remains is to document the
information in a user-friendly way.
- Better preparedness for future projects. As the
usability testing proceeds your ability to predict problems before they occur
will improve. This will expedite later projects greatly.
- Pride in the product. Speaking for myself, I
can safely say that the finished product is of much finer quality after the
usability testing and refinements than it was before. I feel very good
about telling the sales staff to sell it whole heartedly.
- Advantage over competitors.
I hope you have found this article useful or are least interesting. If
you have any comments, please let me know.
I hope to post another article soon to expound further on this subject. I
would also like to develop some simple tools for automating feedback and
expediting the documentation requirements. If I have time, I will develop
these and post them in the future.