Usability testing for effective interactivity

Good interaction design is a key requirement for cognitive systems. It takes advantage—of other cognitive system elements, such as natural language understanding, well-structured and integrated data stores, and contextual knowledge—to present an understandable face to the user.

The best interaction design requires meaningful testing. This is quite a challenge because the type of use varies from task to task and from one person to another. From simple question-answering to exploratory research, an appropriate interaction design that fits within the flow of a task is imperative. Users themselves often differ in their domain expertise and linguistic abilities as well. Occasionally, a simple answer may suffice, but not if we need to explore a large information space to spur thinking and creativity. A good interactive design must take all of the variations in underlying data and terminology and couple them with contextual knowledge of the user’s preferences, device, location, current task, and role. But it’s a long leap from stating the need to designing an appropriate interface.

Connecting the seeker to the information she seeks is not a new problem. Interaction design has been a stumbling block since the age of the card catalog. In the early days of online systems, we got around the complexity of creating effective queries by training professional intermediaries. We created taxonomies and event codes to help us structure data and hone queries. This changed with the advent of the internet and the subsequent information accessibility to end users. The trained intermediary was often eliminated.

When we first started our usability tests, my clients needed a quick way to find the most egregious design errors in their interaction designs. They knew that an academic approach with canned questions and answers wouldn’t serve their needs. Instead, they were serving real users with unpredictable questions, variable terminology, and little time. In cooperation with the Cornell HCI Lab and professor Geri Gay, we developed a methodology for testing new interfaces at an early stage in the design process. In those days, that meant paper mockups. We also tested users who had an interest in the content, and who brought their own questions, rather than marching through questions that were not of interest to them. That’s because we were trying to satisfy information needs, rather than answer preset questions.

As we gained more experience, we found some surprises:

♦ Finding design bloopers took very few respondents; it usually took about six before it was clear that a feature was confusing.

♦ Subject experts, novices, software developers, information professionals, and librarians all have different mental models of a collection of information.

♦ Subject experts were aware of hierarchies and relationships among ideas and terms. They were also used to the specialized terminology of their fields. They used these terms in their queries and usually were successful in finding exact matches. However, they missed terms that were very new, very old, or inexact. Their expertise sometimes made them blind to their own biases, so they missed unknown or surprising relationships. When Jason Baron tested e-discovery systems, he found that experts using Boolean queries often miss up to 30% of the relevant documents in a collection. Our own comparison tests of standard dialogue and dialogue target with an early natural language processing search system bore this out.

♦ Novices didn’t know specialized terms and floundered around with general, and often ineffective, queries. They still had a specific information need, but the terms they used didn’t necessarily match the professional literature. They required more help in honing their query before they ever started searching.

♦ Software developers don’t (or didn’t then) think in terms of subject or event hierarchies. They expected all datapoints to exist on the same level. It took customer demand to have them institute drop-down menus and related terms as hints for further directions to search in.

♦ Information professionals thought in hierarchies. They used vocabularies and taxonomies to prepare a query. It was a matter of professional pride to craft a query that accurately reflected what their end user was looking for. They were also more likely to iterate, refining a query if they had access to the user during the seeking process. Intermediaries, however, could only see so far into the mind of the user. They didn’t have that unexpressed information need in the backs of their heads as end users do. The seeker of information will often tell us, “I’ll know it when I see it.”

♦ Librarians were the only group that read directions.

Interaction design has come a long way since those early days. But some of our gleanings are still pertinent:

♦ People are visual animals, but most interfaces are still rooted in the age of text. Color, shape, size, motion, or sound is often easier for the eye to track than plain text, particularly if interfaces change as you interact.

♦ Very often, a good interface design is more effective that the latest technology. I mourn the disappearance of Puffin Search and X1. They may have had relatively simple Boolean search systems, but I could find my stuff because the systems were designed for desktop search. That means that the user knew possible search terms, having written most of the documents, and could sort by date, by keyword, or by folder. Most importantly, X1 gave me a preview of any document in a separate pane so that I didn’t have to go diving into five applications to find the most recent version of an illustration.

♦ The only way to accommodate the variability of users and queries, even within a limited domain, is to build flexibility into the design. It’s very difficult, however, to convey the visual possibilities to the user without turning them off with too many clicks and choices.

♦ As a corollary, we often do better to reduce the variability of users and queries (i.e., focus on the product) rather than to try to come up with a system that tries to be all things to all people. In fact, domain or task specificity is the state of the art today.

♦ Do we need to take some cues from video games and consider 3D interfaces and voice dialogues to improve interaction?

Interaction design challenges

The problems of interaction design are still with us. In fact, they have multiplied as we demand more and more from our information systems, and information becomes ever more central to our personal and business lives. Contact me with your thoughts and your designs.

I’m always looking for new ideas.