Here are 5 Things you should Never Say or Do to users (during usability tests): I teach them regularly in my usability training courses…

1) Praise

Example: “How am I doing?” “Good Job! You’re doing great!”

Why is this bad?

Giving your users praise sets up an unhealthy relationship with the researcher and the subject. If the user makes a mistake will you be there to tell them they are doing poorly? Will you provide “therapy” to the user and tell them it’s okay and console them?

I have witnessed colleagues do this. I did it once, and realized I was stuck when the user got angry and insisted I tell them if they were right. To do so would have embarrassed them, I may as well have said “You are a stupid user, don’t worry”.

The problem with Praise during testing is that it violates one of the principles of solid usability testing: there is no right or wrong, the user is not there to make you happy by “doing a good job”.

Best Practice: Take a more neutral interaction with the user. The user does not need to know if they are making a mistake or not or if they succeed or fail- that’s for you to observe, not the user!

2) Feature Like/Dislike

Example: “Do you like this feature?”

Why is this bad?

If you ask users what they like or dislike, you have turned the usability test into a focus group. Focus groups elicit opinions, usability tests elicit behaviors. Margaret Mead once said, “what people say and what they actually do are two very different things”. If you ask people what they like, you’ll miss how they would actually use it when they got home with it.

Best Practice: Give users familiar tasks to perform and watch them! If users really hate a feature they will vocalize it. (Note: usability testing uses a verbalization technique called the “Think Aloud” protocol).

3) Asking about ‘Ease of Use’

Example: “Is this easy to use?”

Why is this bad?

It is really difficult to gauge ease of use from a questionnaire, partly for the reason mentioned in #2 above, and partly because ease of use is relative. Humans are highly flexible and will internalize difficulty with machines, often blaming themselves. What’s easy for some is mind-boggling for others.

How is it that all these years major software manufacturers have given us ease of engineering instead of ease of use?

Best Practice: Again, watch users, don’t ask them. Remember ease of use is not the only usability metric that counts. More on usability metrics in another post.

4) Asking about expectations

Example: “Is this what you were expecting to be on this page?”

Why is this bad?

I once accompanied a usability lab rental customer on site, with a role of observing and acting as technical support. The client was a *major* ad and interactive agency that was conducting usability testing for it’s *major* financial services client. The financial services client was present, but had no idea that a “worst practices” usability test was being delivered by the agency! The facilitator sat with each user and on each screen asked, “Is this what you were expecting here?”… and each user said “I guess so, I don’t know”.

Best Practice: Let users vocalize their expectations by walking through your site or web application with the industry standard “Think Aloud” method. Expectations do not need to be asked, users will tell you what they think should happen 90% of the time (either verbally or through their behavior, non-verbally).

5) Giving Instruction

Example: “Click on that button, scroll down, look at that in the top corner”

Why is this bad?

When a user is lost or confused, common sense tells us to help them. Forget about it! This is one of my cardinal usability testing rules, I stress in my usability testing training with corporate teams. If you instruct or direct the user, like with praise, they will rely on you as their crutch when they need help again.

Another thing, we have realized in over 55 live usability reviews our Portland User Interface Special Interest Group has conducted since 2001 is: let the user go off track if they need to, their confusion will teach you something about their expectations and problem-solving techniques.

Best Practice: Instruction should only be offered if you are consciously moderating and feel it is safe to “reel the user back in” (usually I leave them as long as 5-10 minutes in an off-track path).

Happy Usability Testing!
Frank Spillers, MS (Usability Consultant)

If you enjoyed this article and you are interested in refreshing your usability testing skills, you might check out my exclusive Usability Testing Skills refresher web seminar…

-or-

Join the Portland User Interface SIG, the group meets online and is open to anyone with an interest in learning more about usability!

Formal Usability Testing

Also called “High Fidelity” usability testing.

Where it gets it’s name: Design concepts are typically more finalized. Formal testing can take place in pre-release design, but not always. Websites in their current state (before a re-design) are considered Hi-Fi tests.

How common is it? Very common- even more common than informal usability testing.

Advantages: Click-able prototypes are easier to follow (for stakeholders). Formal usability testing is often the test of choice for including developers, project managers, executives etc.

Drawbacks: A certain level of HTML “smoke and mirrors” design needs to be created (for web sites). More level of coding complexity is involved in testing software applications- in this case Informal testing is better.

Informal Usability Testing

Also called “Low Fidelity” usability testing or “paper prototype” testing.

Where it gets it’s name: Design concepts are tested in draft “wireframe” or unpolished state. No coding or graphic design has occurred at this level. The focus is solely testing the “information architecture” or the “interaction design”.

How common is it? Very common, contrary to what most marketers might think. Usability guru, Jakob Nielsen called this testing “Guerilla HCI” to refer to the fast and frequent use of this technique in corporate environments. However, most low-fi testing is usually done by usability engineers, with users behind closed doors or “in the trenches”.

Advantages: Yes! You can get design feedback early on. Feedback can be rapidly acquired in less than two weeks and inserted into the development lifecycle rapidly. A benefit for the quick turn Agile development cycles.

Drawbacks: You can’t always test dynamic page level interactions. This will become more of a problem as “Web 2.0″ interface design elements become more mainstream (such as fading and hovering elements).

Mixed Fidelity Usability Testing

Mixed-Fi? Mixed fidelity tests are more common for us at Experience Dynamics. We typically test with low-fi concepts that are “taped together” with HTML and some JavaScript. This gives us a rapid “cut and paste” site that can be iterated and refined on the fly. Informal HTML prototypes allow us to prototype and test quickly thereby keeping costs down, but still including stakeholders in our usability labs or remote testing sessions (more on Remote Usability Testing in a future post).

Conclusion: Usability testing can and should be done early on and throughout the product design lifecycle. It is very common for usability practitioners to test concepts that only exist on paper or as static PhotoShop files. Moreover, with basic HTML, a hybrid fidelity can be achieved bringing both the need for speed and user validation to a design.

Happy Usability Testing!
Frank Spillers, MS (Usability Consultant)

If you are wondering where the methodologies you use come from, you ought to know that there is a very long history (and heaps of military, academic and corporate research) behind usability techniques.

(Ivan Sutherland’s Sketchpad 1963 pictured left)

Usability Comes of Age in the “Dot Com” Period

For many, the reference point for usability testing is the dot com boom circa 1998-2001. This is the first time usability testing was used on a wide-scale basis for commercial (e-commerce) purposes. Before this, usability testing was confined to academic or corporate R&D research (Apple, Sun, HP, Bell Labs, AT&T, Microsoft and others). It also marks the first time usability (aka “customer experience”) figured significantly in an executive team’s decision making process (e.g. Amazon, eTrade, Google, Dell etc).

“In our first year we didn’t spend a single dollar on advertising… the best dollars spent are those we use to improve the customer experience.” - Jeff Bezos, Amazon.com

A Historical Time line of Usability Research

For practical purposes we consider World War II the emerging point of usability research. Post WW II is also when the inter-disciplinary field of Cognitive Science was founded. Cognitive Science is the obscure field where usability engineering or HCI (Human Computer Interaction) (aka Human Factors in the US or Ergonomics in Europe) is studied. As early computers and Artificial Intelligence (Nazi code-cracking) emerged, so did the study of how humans process information and perform with computer-based interaction.

Historical Marker 1: 1930-1954

Colonel John C. Flanagan perfects the “Critical Incident Technique“

World War II was the starting point of electronics and electrical systems controlled by human operators through a “user interface”. Industrial psychologists such as John Flanagan discovered that by reducing the amount of buttons, knobs, switches and control panels in new fighter aircraft- they could also dramatically improve operator performance. The P-51 Mustang fighter, for example, “became one of the conflict’s most successful and recognizable aircraft”.*

Developed by Flanagan, the Critical Incident Technique (or CIT) is a set of procedures used for collecting direct observations of human behavior that have critical significance and meet methodically defined criteria. These observations are then kept track of as incidents, which are then used to solve practical problems and develop broad psychological principles. [source: Wikipedia]

In today’s parlance: you need to do usability testing because your customer is interacting with your company, brand, product or service through an informational display (website, software application) creating a self-service situation. Any self-service situation with a computer interface will likely cause errors, confusions and failure (if not designed to meet a user’s expectations). Observing users perform tasks helps find out what those errors are before it’s too late.

For example: compare the Supermarine Spitfire to the P-51 Mustang fighter cockpit (using CIT, an early usability testing technique). [hat tip to Jurek Kirakowski for pointing out this example]

The period 1950-1969 saw an increase in usability research related to computer interfaces (as micro-electronics began its boom). IBM was active in this area early on as were other more academic/ R&D innovators such as Doug Engelbart at SRI; Ivan Sutherland at the University of Utah and Alan Kay (Note: Engelbart, Sutherland and Kay- and others, are known as the inventors of many user interface hardware and software designs that you use today). Early pioneers such as Sutherland developed advanced interfaces in the late 1960’s that are not yet in the public mainstream today, such as virtual reality technology and tablet PC’s.

Historical Marker 2: 1970-1983

Xerox PARC (Palo Alto Research Center)

Xerox is largely responsible for much of the innovation in user interfaces (still in use today!). Many know these as WIMP (Windows, Icons, Menus, Pulldowns). Xerox R&D work and resulting usability and user interface innovations propelled the current age of corporate usability research.

[above: Xerox Star system. Hat tip to Bruce Damer, inventor of the related Xerox Elixer desktop UI]

Historical Marker 3: 1983-1992

Apple unleashes the Macintosh user interface

Apple’s built its design of the personal computer around a strong emotional connection with the user. This was also reflected in their advertising.

above: Apple II- early 1980’s… usability takes root in research circles.

Late 1970’s usability research exploded in the 1980’s (in the R&D sense) with many great achievements in user interfaces adopted by the masses (e.g. Atari jump-started the video game industry with the innovations of usability pioneers Alan Kay and Brenda Laurel for example). Note: If you haven’t read any of Brenda Laurel’s work, you’re missing out…

From the early 1990’s to the mid-1990’s usability research continued but was more of an R&D hang-over from the boom of portable and personal electronics spurred by the mass adoption of Microsoft’s Personal Computer and Windows operating system. Why Microsoft won the PC battle and Mac did not is the subject of David Gelertner’s book Machine Beauty.

Historical Marker 4: 1998-2003

Usability becomes recognized as a strategic win to Web site marketing efforts

The mad rush to build the Internet was triggered by the recognition that information could be “easily” indexed and edited with the new mark-up language (HTML). Furthermore, business could be conducted online and products sold through online catalogs or e-commerce sites.

above: Boo.com. Naive Web design tricks such as those epitomized by Boo.com were a wake up call to usability in the early days of the Web.

Unfortunately the ease of learning HTML meant that anyone could play on the Web. Likewise, anyone could run a usability test. Evangelism favoring “just do it” was promoted by experts like Jakob Nielsen and his “Discount or guerilla HCI” and authors such as Steve Krug with his “going out of business usability testing”. Nielsen’s colleague (Rolf Molich) however, showed that not all usability testing methods and approaches are conducted equally. More in a future post on his findings and the implications of best practice usability testing techniques!

Usability testing solutions exploded with new “bots” like WebCriteria’s Max and online panels like Vividence- replacements to “old school usability testing”– or at least that’s how the Sales VP’s at these companies positioned usability. (Disclosure: I worked for WebCriteria and got a bird’s eye view into this piece of usability history)

Historical Marker 5: 2004-2007

The Web gets a makeover with Web 2.0 and a focus on User Experience

New energy, new thinking and new players are starting to dominate how things are done on the Web (Yahoo, Google, Flickr, etc.). These new approaches signal a maturity never seen before.

New Web 2.0 start-ups aimed at destabilizing the dominant position of traditional software companies and software applications or tools, dominate today’s discussions (just ask a Venture Capitalist what they think about social video or mobile applications– two hot areas of development at the time of writing).

Above- the logos of Web 2.o (interesting analysis of Web 2.0 logos by Stephen Coles)

Usability is now being recognized (in the USA in particular) as a strategic “win” to Web site marketing efforts. No new Web 2.0 start-up would be caught dead without considering user experience it seems. How many are actually doing usability research? (Not all, and I am not sure of the answer to this… but many are serious about improving usability with their tools and applications). Clients, partners, VC’s and end-users are all demanding high standards of usability with your design. I have been tracking this over the past 8 years of my own usability testing at Experience Dynamics. It is amazing to see the tides turning!

Happy Usability Testing!
Frank Spillers, MS (Usability Consultant)

For further interest see: David Meister ’s The History of Human Factors and Ergonomics (Lawrence Erlbaum, Mahwah, NJ, 1999). Chapter 4 covers The Formal History of HFE and Chapter 5 covers The Informal History of HFE.

Disclaimer: the above historical time line is my best effort at explaining what I believe are significant points and players in usability history. If I missed a major detail, please let me know! This history is how I teach it in my usability testing training based on my own understanding as a usability professional.

Answer 1: = 5 users (Jakob Nielsen and Thomas Landauer, 1993).

Answer 2: = 15 users (Laurie Faulkner, 2004), PDF file.

So, which is it, 5 or 15? And why are we arguing about an extra 10 users, doesn’t one need to test with at least 100 or more users for statistical significance, accuracy and validity?

Statistical Validity in Usability Testing

Usability research is largely qual-itative, or driven by insight (why users don’t understand or why they are confused). Qual-itative research follows different research rules to quant-itative research and it is typical that sample size is low (i.e. 15 or 20 participants).

The end result of usability testing is not statistical validity per say (the outcome of quant-itative research) but verification of insights and assumptions based on behavioral observation (the outcome of qual-itative research).

Why don’t we do large numbers in usability testing?

1. We are looking for behavioral based insight (what they do).
2. Statistics tell half the story and often are devoid of context (e.g. Why did they fail?)- Also one of the major problems with gaining insight from web analytics (website traffic statistics).
3. Our objective is to apply findings to fix design problems in a corporate setting (not academic analysis).
4. Research shows that even with low numbers, you can gain valid data.
5. Usability testing is being used industry-wide and has been for past 25 years. Experts, authors and academics put their reputations and credentials behind the methodology.

Behavior vs. Opinion

Usability research is behavior-driven: You observe what people do, not what they say.

In contrast, market research is largely opinion-driven: You ask people what they think and what they think they think. You need big samples for market research because of this (though focus groups bend this because they are somewhat qualitative). This is why phone or web surveys require hundreds or thousands of responses. Behavior-driven research is more predictable. Basically, if 10/15 users are confused you can assume that many more will also be confused as well.

Example:
If you ask someone “what do you think of this homepage?”, you will need several hundred responses to gain statistical validity in order to validate what will be opinion-driven data. Asking someone their opinion does not constitute usability requirements, since usability testing is about isolating “how they will actually use” the design not just “what they think” of the design.

If you give a small set of users a scenario that forces them to interact with home page elements and observe their behavior, and listen to their unsolicited reactions, you will get a better idea of what they think and need. The driver here is expectation (governed by cognitive factors) vs. opinion which can be driven solely by emotional, social or personal factors.

Suggested Sample Sizes for Research

Corporate Usability Research:

• Surveys (phone and web) = ~240-~1,000 +
• Focus Groups = 15-20 (depends on audience segments involved and goals of study)
• Usability Testing = 10-15 participants
• Field Studies = 15-40 participants
• Card Sorting = 15-30 (higher is better since card sorting uses the statistical method of cluster analysis)

Academic Usability Research:

Samples are usually larger depending on size and scope and research objectives (e.g. 15 users per segment or 40-100 users in a usability test).

Jakob Nielsen’s “test with 5 users” assumption

I think it is important to understand that Jakob Nielsen was trying to promote usability testing as a regular usability research activity in corporate environments. I believe he conducted this research (using a call center software application in the early 90’s, rumor has it) in order to demystify the perceived complexity of setting up and running a usability test.

Remember in the early 1990’s, only the hard core research and development labs at Apple, Bell Labs, Microsoft, IBM and Sun were doing usability testing. In Nielsen’s much respected and equally criticized article “Why You Only Need to Test With 5 Users” (written in 2000) he recommends (based on the early 1990’s analysis) that instead of opting for higher accuracy, you go for the “fast and dirty” approach
of conducing three tests instead of one “elaborate” study.

Later on in the article Nielsen says that the rule only applies if your users are comparable. If you have other segments or user types, you will need to test more users.

Translation: 5 users per audience segment or target user group, or for a website with 3 diverse segments you will need 15 users for the one test.

Magic Number 15 for Usability Testing…or Why 5 Users is Not Enough

Laurie Faulkner ( PDF: 2004) has conducted new empirical research showing benefits from increased sample size. In her study, “Beyond the five-user assumption: Benefits of increased sample sizes in usability testing”, she wrote:

It is widely assumed that 5 participants suffice for usability testing. In this study, 60 users were tested and random sets of 5 or more were sampled from the whole, to demonstrate the risks of using only 5 participants and the benefits of using more. Some of the randomly selected sets of 5 participants found 99% of the problems; other sets found only 55%. With 10 users, the lowest percentage of problems revealed by any one set was increased to 80%, and with 20 users, to 95%.

At Experience Dynamics, (usability consultancy) we have found that the cost savings of using fewer users is negligible. In other words, after you spend the time and money to set up, facilitate and report on the test, adding a few more users does not add “that much” time and money to the overall project.

The benefit you get from adding a few more users to the total (or in the case of 5 users, doubling the amount) is far greater than the small test that gives you “quick and dirty” results. In the case of running a series of usability tests or iterating your testing process (recommended for refinements based on evolving design decisions), you may want to choose a smaller number of users: I recommend no less than 8 users.

Happy Usability Testing!
Frank Spillers, MS (Usability Consultant)

Note:

When conducting usability testing, what do you measure and why? How do you capture metrics and what you should be measuring?

In this World Usability Day exclusive web seminar, we will discuss usability testing observation metrics and best practices.

Agenda:

1. Usability Testing metrics: What are the things
you should be measuring? How to measure qualitative vs. quantitative
data (e.g. satisfaction vs. effort).

2. Usability testing observation best practices: Do you
measure time on task every time? What do you need to do a good job
capturing metrics if you are doing “quick and dirty” discount usability
or “guerrilla” testing, without undermining your own efforts?

3. New tool for usability testing logging: LiveLogger.
Just released this week, we will review a new usability test logging
application. We will review the new LiveLogger interface and discuss
what the tool does, how it captures and reports on usability testing
metrics.

Summary: In this 1 hour live web seminar (held twice
on World Usability Day), we will review usability testing observation
best practices.

Length: 60 minutes

Who should attend: People new to usability testing or
want to conduct rapid usability testing; usability managers; user
experience team; anyone responsible for user advocacy or usability
testing.

Note: If you missed this seminar, we will be running it again in future. Stay tuned to Experience Dynamics usability seminars for details.

Here is an independent review of the seminar by Corey Bates at UseTube. Thanks for your comments Corey.

Happy Usability Testing!
Frank Spillers, MS (Usability Consultant)