Scott Berkun
Microsoft Corporation
November/December 1999
Summary: Discusses the value of usability testing and how important it is to integrate it into the entire project lifecycle. (3 printed pages)
You cannot build a useful product or Web site without usability testing. If you have never watched someone use your designs in a usability lab, you are taking shots in the dark. You can't possibly know whether your hard work is making things better or worse. The features you are focusing on may be things that no one really needs, or could never figure out. Without regular sessions in the usability lab during the development cycle, projects are guaranteed to head in directions that do not benefit the users of the product. As a developer, you should have deep interest as to whether your hard work is making the product better. It's in your interest to make sure your work gets examined in the labs, so that you can make adjustments and ensure that you are making the best possible product for your users.
The value of the usability lab goes beyond the important usability data you obtain. I always think about moments sitting in the lab, watching someone try to use something I designed, and completely failing—sometimes in the most horrific and bizarre ways imaginable. Even when users do well, I'm amazed at the slow pace at which they work. I become aware of the shortcuts and concepts they don't recognize, that I have taken for granted. When I haven't been to the lab in awhile, watching users always resets my expectations for what they really need, and focuses my attention again on solving the real problems they have. As an industry, we continually overestimate the sophistication and technical savvy of the majority of our users. Their problems seem incredibly distant most of the time, and are easy to ignore when we're sitting at our desks writing code. But when you are on the other side of the one-way glass, seeing normal, intelligent people struggling to use something you made, you feel very differently about the work you do. You get a deep sense of how your work really impacts people, and you make a connection with the users of your product. Most computer users flail and stumble, often hopelessly, at tasks that I'd expect anyone on my team to accomplish in seconds. There is just no substitute for regular time in the usability lab. It will humble the most proficient designer or developer into thinking more critically, and provide them with the key information necessary to make better decisions about the product.
I once had a conversation with a developer about a usability test she had just watched. She said, half-seriously, "we need smarter users." I thought for a moment, and responded with "no, just more thoughtful developers." It's a matter of pride: Are you the kind of developer that is driven to make things better for your users, or do you want to make things easier for yourself? From the e-mail that comes in, it seems many of you care deeply about making better products. The most basic step toward that goal is investing in regular usability lab time with a trained usability professional. There is no substitute. It is an investment toward better Web sites and better products. The time it takes to do usability testing is well worth it. It's the only path toward a useful and successful product.
One common misconception is that all you need is one user test to get well-designed products. The belief is that you can do a test, see the problems, make some fixes, and call it a day. This model doesn't get you very far. Usability testing provides significant payback only when it is well integrated into the entire project lifecycle. The value comes from doing regular tests, starting with early prototypes, and repeating trips to the lab at regular intervals as the design and product develops. An individual usability study will identify problems—but without follow-up testing you won't know if your "solutions" have really solved the old problems, without introducing new ones.
There are also situations where understanding the problem is only half the battle. It's true that many of the problems you find in the labs are easy to fix, but many of them aren't. It takes a lot of design effort and exploration to create solutions to hard problems. Some usability engineers may possess interface design skills, and can recommend good alternatives, but the skill set required to create and run a usability test is very different from the skill set required to design products: You probably won't find the same skills in one individual. It usually takes strong interface designers, program managers, and design-savvy developers to work with the usability engineer and make good decisions about how to apply information from a test to the designs. So, learning about the problems through the usability lab is very important, but it's only half-way toward a better product. With usability tests at regular intervals, designers and developers are empowered to learn from one test, explore alternatives, and then return to the lab again with their improved designs, repeating the process as necessary. It's a team effort spanning across the project timeline that makes usability work.
Usability at Microsoft has come a long way in the 10 years it's been around. When the group started, there was one small lab in building 5, in the middle of the oldest part of the Redmond campus. Today there are over 30 individual labs, and even more usability engineers that work with product teams to run the studies. Each usability engineer works as part of a specific product team, participating in meetings, brainstorming sessions, and team reviews. They drive the analytical and analysis aspects of the design process, making sure that key assumptions about users, and interface designs, are supported or challenged with real data. This includes deciding when lab studies, data analysis or site visits are appropriate, and then planning, designing, and executing particular studies. Then they inform the team about the results of the study, with recommendations for how the information should affect the designs in the product. In the last 20 years, many techniques have been developed for gathering data on how people use things, and it's the usability engineer's job to bring the appropriate technique to bear on the product at the right time. You can see some pictures of what the labs at Microsoft are like at http://www.microsoft.com/usability/.
It can be a challenge to find good usability engineers or consultants to work with you or your company on projects. Any reasonably sized Web or software company should have in-house usability resources, but for smaller organizations it often makes sense to outsource this support. There are no official certifications for usability engineers, so word of mouth and recommendations are often the best way to find someone. Two starting points I know of are related to the annual ACM conference on computer human interaction (CHI). The CHI Web site has a list of consultants at http://www.acm.org/sigchi/hci-sites/CONSULTANTS.html. Check references just like you would for any other contract or outsourced relationship. The SIGCHI group, a San Francisco-area chapter of people interested in Human Computer Interaction, has a list of usability consultants on their Web site at http://www.baychi.org/general/consultants.html. Be forewarned: Use these resources at your own risk, because these are not moderated or edited lists. The list to follow includes some related links to how usability labs were used in the development of Microsoft® Windows® 95, as well as information on true low-cost usability testing and evaluation. To learn more about the practice of usability testing, see the UI design resource page listed below.
As a last word: If you're not spending time in the lab, you really have no idea how good or bad your Web site or product is. Stop guessing and get some data.
Have comments, feedback, or a suggestion for future columns? Write to hfactor@microsoft.com.
Check out the previous columns and UI design resources at http://msdn.microsoft.com/ui/.