Mirel

a Thomson Corporation Company; ASAP

Technical Communication

November, 1998

SECTION: No. 4, Vol. 45; Pg. NA ; ISSN: 0049-3155

IAC-ACC-NO: 53377796

LENGTH: 12030 words

HEADLINE: Visualizations for data exploration and analysis: a critical review of usability research; Visualizing Information

BYLINE: Mirel, Barbara

Data visualization has been called the "killer app" of the 1990s (Card 1996). It is a powerful tool for exploring and analyzing data - for discovering as well as testing hypotheses. This technology has the potential to change the questions that people are able to pose to their data and transform their analytical methods and decision-making processes. It may, in fact, be the next generation of data reporting tools. In this article I argue that the prevailing computer science orientation to data visualizations is severely limited for addressing many of the usability concerns associated with supporting users in three critical problem areas - sophisticated visual literacy, complex data analysis, and new paradigms of visual inquiry. This article critically reviews the usability literature about data visualizations.

I first describe what visualization technology is and what is uncharted about the three usability areas of perceptually rich, interactive displays; complex problem-solving; and visual querying. Then I explain what it means to take a computing - specifically an object-oriented - perspective on the usability of visualizations, emphasizing the limitations of this point of view when it comes to supporting users in complex activities and reasoning.

Against this background, I critically review the current literature on the usability of data visualizations - primarily authored by computing specialists - and show the contributions and blind spots of current usability investigations. I find that this computing orientation causes investigators to overlook key relationships between users' current analytical practices; aspects of these practices that translate readily to data visualizations; aspects that are new, requiring users to learn unfamiliar concepts and procedures relevant to visually structuring, querying, and exploring data; and the interfaces and instructions that facilitate these mappings.

The literature, for example, is mostly silent about the support that users need to monitor where they've been and where they're heading or about the effectiveness of videoclips in online help versus conventional textual online help in guiding users to successfully apply difficult concepts such as creating effective structures in complicated network diagrams. My aim is to prompt technical communicators - with their expertise in rhetoric and situated activities - to conduct additional usability studies that will fill in the current gaps occasioned by a dominant object-oriented perspective.

Three-dimensional visualizations are outside the scope of this article because they warrant their own separate review. In many respects, the advanced visual thinking required for interactive 3-d data displays - motion detection, spatial orientation, and complex image segmentation in multiple planes - is quite different from the visual literacy I discuss in this article.

WHAT IS INFORMATION VISUALIZATION?

Data visualization technology has been around in medicine, science, and geography for some time and is only recently starting to make headway into the business world. Unlike the static presentation graphics found in software such as Excel, SPSS, or Crystal Reports, data visualizations are interactive and linked dynamically. Interactively, they allow people to use mouse clicks and menu functions to move almost seamlessly between overviews, detailed displays, and different perspectives on the same data. In graphic overviews, users can examine the large-scale structure of their data to discern patterns, trends, correlations, and outliers. Then they can click on or sweep across areas of interest and "blow them out" into greater and greater detail, with many graphs open and referenced to each other at once. With dynamic linking, when users change one graphic display, the others automatically update to reflect the changes. This dynamic updating gives users different views of the same information at the same moment. For an example of this interactivity and dynamic linking, see Figure 1.

With visualizations, users can see the data that database, spreadsheet, statistical, and graphing programs report only textually or present through static displays. Instead of having to spend hours and even days searching through 50 or more pages of reports to analyze the relationships that they need for a decision, users may interact with data visualizations to quickly retrieve and interpret data from a 10,000-foot view and from a close-up detailed view almost at the same moment.

The beauty of data visualizations is that they exploit the quickness of the human perceptual processing system. They enable users to see the structure of large amounts of data in one view and to identify outliers, aberrations, trends, and affinities at a glance.

Unlike conventional database queries in which every new question requires users to return to a query screen to write a new statement and wait for the program to provide the new response, querying in data visualizations is immediate and cumulative. Users select the data and graph that they want todisplay; then they proceed through a process of incrementally selecting subsets, adding dimension through perceptual cues such as color coding and sizing of data points, drilling down for detail, zooming out again to relate micro and macro views, transforming data into new variables, reorganizing data structures and relationships, and animating data relationships to see parts in relation to the whole or to see changes over time. Unlike database querying and reporting, interactive, visualized inquiry is nonlinear.

Data visualizations have powerful potential uses in business, some of which can already be realized though products available in the market (Wright 1997). For instance, visualizations extend data warehousing, data mining, and online analytical processing (OLAP) to a visual dimension, literally giving users a hands-on view of large quantities of corporate data for operational and strategic decisions (Boyle, Eick, Hemmje, Keim, Lee, and Sumner 1993). IBM's Business Objects is heading in this visualized direction, as is an initiative in Lucent Technologies' Visual Insights.

Visualizations also effectively display the structure and processes of corporate infrastructures. Platinum, for example, has built software with

Visual Insights' dynamic visual components that pinpoints troublesome programs and code for the Year 2000 (http://www.platinum.com/

products/year2k/y2k.htm). For corporate infrastructures, visualizations also target network traffic, system performance, and efficiencies in distributed and geographically dispersed corporate computing environments. At present, Tivoli and HP OpenView are integrating visualizations into their network monitoring packages. My group in Visual Insights is producing software in this space, as well (Baldonado and Winograd 1997; http://www.visualinsights.com). Finally, all-purpose visualizations such as Spotfire are available and are used, for instance, in the pharmaceutical industry (Alberg and Wistrad 1995; http://www.ivee.com/applications).

Despite these applications, data visualizations have been slow to take hold in the business world due, in part, to usability problems (Roth, Chuah, Kerpedjiev, and Kolojejchick 1997). Three problems are particularly vexing:

* Aiding users in creating and understanding richly encoded displays of data in which visual cues are fluid and open for modification

* Supporting users in complex, open-ended data analysis tasks for nonstructured problems and under-specified goals, tasks for which users have many legitimate and contending options for displaying and analyzing data

* Helping users manage their inquiries, including monitoring progress; evaluating the accuracy and validity of inferences and perceptual assumptions; organizing paths of inquiry, cumulative interpretations, and findings; retracing steps after false moves; and returning to interrupted work

Before data visualizations become common fare, they must overcome these usability problems. The following sections expand on these problems.

1. Understanding and manipulating graphics rich In perceptual cues The strength of interactive visualizations is that they exploit perception and pattern recognition. Yet the more information that visual cues carry, the more usability may be threatened - unless developers and interface and documentation designers help to make visual information and manipulation accessible to users. Visual coding adds dimension to data plots through color, shape, size, or thickness. For example, a bar chart may show the number of products sold and be color-coded by revenue generated, enabling users at a glance to see simultaneously each product's sales and revenues. Users may progressively add more and more encoding to a single graph. Positioning carries information as well - suggesting close relationships and affinities between physically proximate data points. By rearranging or reordering data, users see relationships from different angles.

Drawing meaning from these perceptually sophisticated and modifiable graphs calls for an advanced visual literacy. Visual literacy may be defined as a"basic system of grammar for learning, recognizing, making, and understanding visual messages that are negotiable by all audiences" (Brasseur 1997). People vary in literacy by aptitude, experience, and training. As Brasseur notes, computer technologies have ushered in the need for advanced visual thinking because they offer ways to present and compose more visually complex displays than ever before. To read and interpret these displays, users need to engage in sophisticated Gestalt perception, complicated associations in mental processing, and informed manipulation for interactivity.

Perceptual decoding is particularly challenging in visualizations that present multiple and at times contradictory visual cues at once. For instance, a network diagram may depict projects belonging to various departments in a company. Users may color code projects (the nodes in the diagram) by the risk associated with each project and at the same time color code the links between departments and projects by the project value.

Color-coding the nodes and links on different values can result in a graphic that is difficult to interpret. If blues represent low values and reds represent high values, users will confront contradictory cueing. In looking for prize projects - low risk, high value - they have to coordinate blue nodes and red links, a perceptual inconsistency that is likely to give them pause. Tests conducted in the usability lab by my group show that processing these contradictions slows performance times considerably.

Sophisticated decoding processes may never come easily to most people, much as interpreting complex, conditional sentences with compound negatives requires time and effort even for people familiar with such linguistic constructions. Yet certain dimensions of visual literacy seem to make a difference between the ability to make sense of visually rich displays or not. At the very least, users need to know how to find an entry point into a perceptually rich landscape to render its encoded information intelligible. Even this minimal requirement in advanced visual literacy requires support for users.

2. Supporting optimally complex risks with minimal technological complication

In most analysis and reporting programs, designers try to strike a balance between enabling ad hoc analyses and supporting frequently performed tasks through "canned" views and routines. Striking this balance is difficult in data visualizations because this technology, at heart, targets computing for "wicked problems," that is, for nonstructured problems that lead users toward open-ended exploration more than formulaic methods and solutions. These problems and tasks are notoriously difficult to model (Mirel 1997). They have no clear stopping rules, and people's approaches depend a good deal on contextual factors, contingencies, and constraints, dynamics that exceed the representational capabilities of coding. In addition, complex tasks involve trial and error and serendipity, and they engage users in many problem spaces at once. The hard part is not developing functionality relevant to complex tasks and problems. The hard part is supporting users in their efforts to figure out how to control the visualization software so they can pursue paths of inquiry relevant to their individual purposes.

Designers typically deal with this balance between user and program control by simplifying complex tasks (Berg 1997). They localize tasks to target a specific domain; slice them into generic, decomposed pans; represent these unit-sized parts in program functions and menu options; and offer, on the one hand, a standard version of the software that provides prestructured tasks and routines and on the other hand, a professional version for nonstructured ad hoc inquiries and customizations. One problem with this strategy for data visualizations - as findings from needs-gathering sessions for my group's product underscore - is that experienced data analysts and decision makers want pre-scripted tasks along with ad hoc capabilities in a single, accessible product. As one person who is an inexperienced visualization user but an expert data analyst said in interviews, "Without the ability to create my own views, the program isn't valuable."

Unfortunately, these user desires for control are likely to outstrip users' initial abilities to advantageously manipulate the full power of interactive visualizations. Accompanying the need to strike the right balance, therefore, is the need to support users in the ad hoc tasks that they control. The learning curve is high. Unfortunately, within the prevailing object-oriented perspective in current usability testing, little investigation has been done to identify just how high this learning curve is. Nor have studies closely examined both the information and instruction - form and content - that users need and the most appropriate media to deliver distinct types of information - the interface, online help, or training. These questions are crucial to effective task performance. They also are rhetorical at heart and fall directly within the purview of technical communicators.

Finally, another area of support required for complex tasks is helping users interpret data accurately and manage their inquiries. Users, for example, have to coordinate and manage the iterative processes of calling up and drilling down in relevant data. They have to organize emerging inferences and findings, keep track of various views of the data, and check the validity and accuracy of their perceptions. Many of these activities related to managing inquiries are unique to interactive data visualizations.

3. Facilitating visual querying and exploration

Visualization technology embodies a new paradigm for querying and analyzing data, and users need to learn this paradigm. Over time, this mode of inquiry may become second nature to users, but at first it requires a transition from conventional modes of querying based on database reporting, Boolean logic, and statistical analysis. Conventional querying, unfortunately, does not directly map onto the multiviewed and perceptually rich interpretations and direct manipulations associated with visual queries. In visual querying and analysis, users need to know what graphic or set of graphics should display what data to answer specific - and often fuzzy - questions that they are progressively posing. Figure 2 walks through the steps of visual querying.

Unlike conventional database querying or statistical analyses, visual querying encourages incremental changes and uncertain queries, and it compels users to ask more questions as they move through the data than conventional querying can accommodate (Keim and Kriege 1994; Spoerri 1993). It is an Answer-Question process more than the reverse.

These three usability challenges cover a large span, from broad-based utility issues (for example, supporting users in understanding what they see and how it relates to their questions) to finely detailed overhead of use and learning issues (for example, devising the most intuitive icon to represent a specific action). In general, current usability studies disregard broad-based, utility questions and instead focus on the more finely detailed overhead of use and learning issues. They ask, for example, whether the directional flow of zooming matches users' expectations for spatial movements or whether color-coding operations and mechanisms for executing them are intuitive. As I show in the later sections, they do not ask such utility questions as how the program, instructions, or both facilitate users' efforts to compare outcomes from two different paths of inquiry when the data displays that they create along the way have changed.

Because of the relatively narrow scope in current usability investigations, findings provide only a partial picture of what constitutes usable data visualizations. We have yet to fill in the other parts with studies conducted from more situated and rhetorical perspectives. Given that data visualization is a powerful mode of visual communication with significant usability challenges, it is surprising that technical communicators have virtually ignored it. As a human factors manager for visualization software, my experiences warn me that this neglect may have dire consequences. Just at the time when this breakthrough technology is coming into its own and is still malleable, the influence of technical communicators, with their rhetorical bent and their concern for activity-in-context, is relatively absent.

In the past five years, no issue of Technical communication has included an article on data visualizations although during these same years the literature in human-computer interaction has been awash in such articles-authored primarily by computer science specialists. In this same time span, the Visual Communication stem of the STC Annual Conference has included presentations on topics ranging from user interface design to technical illustrations, hypertext, Web design, international visual communication, teleconferencing, and - increasingly - multimedia design (so much so that the name of the stem changed in 1997 to Multimedia Communication). Again, no presentation has dealt with visualizations for interactive data analysis.

Interactive graphics are prime candidates for study by technical communicators, but this advanced technology and method of visual communication seem to be slipping through the cracks of technical communication. Instead, computing specialists are shaping what will become the conventions for visualization interfaces and instructions, and the standards for visually manipulating and understanding data. Many of these specialists, trained primarily in software graphics and computer science, are in the field of human-computer interaction (HCI). Although HCI is a "sister field" to technical communication, many of these specialists' computing backgrounds give them a perspective and design language distinct from those of technical communicators and contextually oriented task analysts (Agre 1997).

Before I discuss the specific contributions and omissions found in this usability literature, I further explain in the next section the object-oriented perspective that grounds and shapes the scope of much of the research in this literature.

AN OBJECT-ORIENTED PERSPECTIVE

It is not surprising that a focus on finely detailed issues in usability studies dominates the literature. Most visualization designers and developers are trained in computer science and are strongly disposed to thinking in object-oriented terms and categories. By object-oriented, I mean the tendency to break activities down to their smallest parts or basic elements, each with associated properties and events. For example, an object-oriented view of the activity of reading a book would see the book as an object with properties such as pages, binding, printed words, covers. In being read, the book and its properties are subject to some of the events associated with books in general and with this book in particular - for instance, reading silently, reading aloud, opening and closing it, leafing though it, or marking passages for later reference.

This decomposing strategy is the basis for programming languages such as C+ +. It also may frame the ways in which developers and designers treat human activity in the real world, transposing their coding practices onto human activity-in-context. This transposition may happen even when computer professionals understand and appreciate the complexities of human problem-solving. When it does happen, contextual aspects drop out of descriptions of users' work. In designing, therefore, interface and documentation specialists pay little if any attention to these now absent contextual influences on work.

Table 1, derived from Shneiderman (1996), shows that for data visualizations, a number of general objects and events are common.

Object-orientation is neither good nor bad. It - or something akin to it - is a fact of life in programming generally and visualization technology specifically. Shortsightedness in usability arises, however, when investigators rely on object-orientation as their only frame of reference. Then the context and other nonformalizable knowledge and practices that disappear in descriptions of work become lost forever - or at least until users insist on reasserting them because the program and instructions fail to represent their conceptions and the meanings of tasks.

TABLE 1. GENERAL OBJECTS AND EVENTS IN DATA VISUALIZATIONS

What developers construct...

What users see and interact with...

Data objects with types of data displays:

Properties such as: Tables, line graphs, scatterplots, dimension network charts, topographic charts, bar graphs, pie charts.

The elements in these objects such as bars in bar charts...

Dimension: Univariate, bivariate, three-dimensional, temporal...

Events involving data: Zoom, filter, detail-on-demand, objects extract, rotate, pan

Screen objects: Menus, dialogue boxes, tool bar buttons, status bar, controls such as slider bars, function keys

Events involving screen Sort, drag, print, color, size, save, objects, bookmark...

A better strategy is for designers to maintain a focus on the roles played by contextual conditions, pragmatic knowledge, and heuristic strategies, and to find ways for the interfaces and instructions to evoke these factors as needed, including calling them forth without articulating them. Poitou (1998) calls these nonformalizable factors and the designs that call them forth the "intellectual objects" of the design. He claims that within a technology and its instructions, users need to find and activate a knowledge that will support and augment their performances and expertise. This knowledge should be evoked through creatively designed sets of pointers, reminders, landmarks, and signs. They should "describe, [evoke], and transmit some of the conditions under which certain approaches to performance take place . . . and mark paths of inquiry and instrumental navigation" (p. 242). Poitou claims that documentation should play a prime role in providing and managing this supplemental, critical knowledge. But to do so, instructional text cannot simply transmit declarative and procedural knowledge about features (objects) and functions (events). In concert with the "intellectual objects" (or agents) built into interfaces, documentation has to help people integrate, combine, and reorganize their current understandings into new knowledge and practices relevant to the demands of their specific open-ended problems.

As my review of usability investigations below suggests, visions about what constitutes this critical and strategic knowledge for interactive and linked graphics and how it is best represented in instructions and interfaces are often missing in current studies. Instead, interface and information designers fall back on a more narrowly scoped, object-oriented point of view. The rest of this article explores the ways in which this object-oriented perspective has shaped investigators' studies in the three key areas of usability:

* Perceptual sophistication: Supporting users in making meaning from richly encoded graphics

* Complex tasks: Supporting users in conducting and managing their open-ended inquiries

* Visual querying: Supporting users in moving from statistical and database querying to visual querying and exploration

In each of these areas, I assess what we know and don't know from the research.

PERCEPTUAL SOPHISTICATION

What we know

In the relatively new area of the usability of interactive visualization, most studies, including these on perceptual processing, are building blocks for a still-needed, more comprehensive understanding, in this case an understanding of the array of perceptual dynamics and arrangements that lead to high utility, intuitiveness, and ease of learning and use. Usability studies related to perception and visual thinking fall into two categories:

1. Applications of principles from theory to actual designs

2. The effectiveness of applied principles for actual circumstances and purposes

Studies that apply perceptual and visual theory

One group of usability investigations relies - almost in heuristic evaluation fashion - more on a consistent application of perceptual and cognitive design principles than on an empirical assessment of users' situated processes of receiving and interpreting graphics. To facilitate users in constructing and interpreting graphics, designers strive to combine the elements of visual grammar - lines, shape, color, texture, coherence, dominance, and so on - into meaningful compositions. A prime principle, for example, is to use screen real estate primarily for data and nothing else - neither labels nor legends - echoing Tufte's call for a high "data to ink" ratio in print media (Tufte 1983; Ahlberg and Wistrad 1995).

Creating graphics with perceptual integrity involves striving for "at once the simplest and most meaningful representations of data" (Bertin 1983, p. 164). Ideally, a graphic should represent data in a minimal number of images with a minimal number of instants of perception to evoke a maximum amount of meaning. Any given graphic should be the most appropriate display for a particular problem and dataset. It should show the structure of the data so that people can graphically see distributions and variances at a glance. In addition, a single display should answer as many levels of questions as possible: elementary questions that have one correct answer ("What's the longest response time?"); intermediate questions that define a range or subset of items ("What response times are over 60 seconds in this production cycle?"); and advanced questions that describe overviews ("Which of my factories are closest to optimal efficiency in production processes?") (Bertin 1983). These principles for interactive, electronic visualizations are adopted directly from graphics and perceptual processing research.

Though not specifically user tested, graphics based on perceptual and visual theory purportedly satisfy users because they reflect empirically proven strategies for positively tapping into the human perceptual system. It is impossible to list all the heuristic guidelines and principles underlying this approach to usability. Table 2 provides a sample gleaned from the visualization literature to give a flavor of the wide-ranging

levels of generality covered by the principles.

Notably, because they have been heuristically rather than empirically tested, designs based on these principles have not withstood trials of idiosyncratic user needs, contingency, or contextual conditions. Nor do they reveal the inevitable trade-offs between competing goods that

characterize design decisions for actual products developed within the constraints of time, effort, and resources.

Studies that test perception in problem-solving situations

Fortunately, usability investigators in some instances take into account the interplay between perceptual guidelines and users' circumstances and purposes. For actual purposes, a user's graphic analysis, at core, is about making meaning from the observed structure of the data (Cleveland 1983). Therefore, a key goal of interactive data visualization programs s is to help users gain access to and interpret this structure. Programs give users many options for visualizing structure, some more appropriate than others for particular types of problems and questions. Usability specialists need to ask whether the interfaces and instructions give users the support that they need to choose the right graphic view for their problem and purposes.

Graphic theory posits some data structures as better than others for distinct functions. Scatterplots, for example, are effective for showing linear relationships and correlations between two or more variables (Senay and Ignatius 1990). Studies show, however, that for real world problem-solving with interactive graphics, users may know and choose the right structure for their purposes but find that the structure is difficult to manipulate for the next stages of inquiry (Furnas 1997; Furnas and Bederson 1995). For example, data matrices or tables are effective for viewing detail and comparing within and across rows and columns. But for large amounts of data, data tables are too extensive to permit users to select subsets easily for further analysis. Researchers propose that interfaces and instructions should help users negotiate this tension between visualizing the best structure for meaning versus the best structure for interactivity. For example, options such as "folding lists" can fold away irrelevant columns and rows, or instructional cues for interactivity can guide users in taking advantage of dynamic linking. With linking they may bring up one display for structure, another for interactive filtering - with selections from the second graph automatically propagating to the first.

Usability researchers have also user tested perceptual principles for color-coding. Researchers debate how many colors visualization programs should use for color-coding and how users interpret the logic of coloring (Hendee 1997). Many visualization technologies color code using the entire color spectrum. Low values in the data correspond to cool colors (blues) in the spectrum, high values to hot colors (reds), and the values between low to high, respectively, to greens, yellows, and oranges (Eick and Fyock 1996). Color-coding with the full spectrum has obvious pitfalls for people with impaired vision such as color-blindness. In addition, it is a design choice that assumes people intuitively map a spectrum in one-to-one fashion to a linear logic in their data. However, many peoples do not perceive colors in the same way that our culture does. For example, some groups see only three colors in the world - black, white, and red - or four or five - black, white, red, green, and/or yellow (Wilson 1998). As Hendee (1997) stresses, categorizing logical linear orders in terms of wavelength is a modern phenomenon.

Even when people do perceive and think in terms of our full color spectrum, they find color-coding by the full spectrum to be useful only for some problem-solving situations but not for others. Tweedie, Spence, Dawkes, and Su (1996) find, for example, that when users work with significant details to answer pressing questions, fewer colors (green, yellow, and red) convey the meanings that they are looking for better than does the whole spectrum. Similarly, users whom I have interviewed stress that when they troubleshoot, they want only three colors, preferably the traffic light metaphor - red for trouble, yellow for caution, and green for no problem. This metaphor is not surprising because color is most useful when it corresponds closely to the feature or relationship that users are analyzing (Wells 1997). In contrast, for planning purposes when nuances are important, users value a full spectrum scheme for color coding.

On a different note, Hix, Templeman, and Jacob (1995) find that users benefit from having the color cueing used in data displays reiterated in the text that goes with the particular datapoints. When users see a label in the same color as its associated referent - reinforcing the encoded meaning - they are able to perform tasks that were previously impossible for them.

As extensive as the perceptual research is in terms of designing graphics to "standard," a good deal more needs to be conducted to discover when, why, and how users' actual interactions and needs for real world problem-solving may run counter to perceptual guidelines.

What we don't know

By and large, studies on the usability of visual design and perceptual cues focus on isolated features for instance, the number of colors to use for color coding or the data structures that best match both users' purposes and their subsequent actions for analysis (Cleveland 1993; Tukey 1977; Bertin 1983; Tufte 1983). This discrete tendency accords with the "decomposed tasks" framework of object-oriented design and programming.

Numerous aspects of perceptual processing have yet to be studied adequately, as follows:

* The effect of users' prior knowledge and expectations on their interpretations of what they see

Studies show that users' domain knowledge often biases and leads them to mistaken meanings about visual cues; similarly, the more unfamiliar users are with a type of graphic (for example, a network graph), the more likely they are to read things into the view that are not actually there (Wells 1997). Figure 3 shows an unfamiliar graph that may confound users in terms of an entry point and in terms of knowing how to decode all the layers of meaning characteristically embedded in this type of display.

* Trade-off areas such as users' needs for labels and other information competing against the principle of using screen real estate primarily for data

(Kitajima and Polson 1996)

* Limits in users' perceptual processing capabilities

Occlusion and perceptual overloading are strong usability concerns, as are the encoding and decoding of multiple visual cues in a given display at once (size, shape, color, position, thickness). Perceptual processing research shows that because people's "analysis of organized areas of graphs [texture] involves . . . preattentive and attentive vision," the more differences and variations that the texture expresses, "the greater the attentive effort necessary for cognition" (Wells 1997, p. 394). More empirical evidence is needed to know when and why perceptual stress or confusion may occur.

COMPLEX TASKS

What we do know

Visualization specialists agree that the more directly users manipulate displays, the more able they are to solve complex problems. The conventional wisdom in graphic analysis and methodology supports this claim. Any serious graphical, problem-solving effort requires iterative data analysis (Chambers, Cleveland, Kleiner, and Tukey 1983). It involves looking at data in several ways, constructing a number of plots, doing several analyses, letting the results of each step affect the next, and ensuring throughout that one does not discern things in the data that are not there. Within this framework, three aspects of interactive visualizations for solving complex problems are critical:

1. Users' extent of control in an inquiry session 2. Their success in interpreting data relationships 3. Their skills in managing their inquiries

Extent of user's control

In characterizing the most intellectually satisfying computer-supported task situations for users, Frese (1987) argues that users thrive when they have maximum control over their work, optimal complexity in their tasks, and minimal complications from the technology. Achieving this ideal in real contexts is a significant challenge in interactive data visualizations. Recently, as a starting point for this idea, Buckingham Shum, MacLean, Bellotti, and Hammond (1997) have highlighted the rhetorical nature of design, stressing that as an argument, design requires negotiating and collaborating with stakeholders (users) and their workplace perspectives. Invariably in these dialogues, questions will arise about how much control users want over their work processes, how much "canned" support they will accept for generalizable parts of their problem-solving, and how they would like a program to combine the two modes.

Few studies assess this tension. Johnston (1991), who does look at this question, finds that users want customizing capabilities very shortly after they begin using visualizations. They value open-ended tasks, "manipulat[ing] pieces of data in various ways and master[ing] techniques in a 'play-to-learn' environment" (Johnson, p. 85). Other usability issues are likely shaped by decisions that designers make about this control issue. But scant evidence exists to inform this central design decision.

Examining relationships for interpretive purposes

Usability studies that focus on interpretation seek to support users in understanding what they see in relation to their task goals and in interacting further with visualized data for additional insights. Key to interpretation is that users need help in contending with the "unconstrained conditions" of most visualization technologies. Unconstrained conditions are what users encounter when, in a given view, they have more features and functions available to them than they know what to do with. In regard to this situation, issues that investigators have examined include the following.

* Supporting users with labels, intelligent agents, and other content as means for negotiating the unconstrained conditions of visualizations

A fundamental premise for data visualizations is that display-based problem-solving requires learning precisely because both the problem-solving and the technology are complex and powerful (Larkin 1989). It is misguided to think that an improved "look and feel" in user interfaces alone will make visualizations an easy-to-use tool. Even experienced visualization users concentrate anew each time they explore an untried dataset. Users need support from interfaces, online help, and training to figure out the meanings of displays as they relate to the subject matter and domain that they represent (Oostendorp and Walbeehm 1995).

Research findings on the schema that people need for interpreting visualizations reinforce the need for this type of information. Katajima and Polson (1996) find that users need labels to build effective schema. Labels are "intellectual objects" that support users' efforts to tie plans for action to relevant data objects and operations. Importantly, Katajima and Polson identify the actions that users may do to elicit certain outcomes, the actions that are allowable in a given program state, and the strategic interactions that are more efficient than piecemeal operations as crucial for these labels. Halya Yazici (1995) also finds that users need combined text and graphics for complex analyses such as trending and forecasting. Specifically, users need accompanying text and labels to express pragmatic knowledge, including corporate procedures for using a given technology.

Furnas (1997) also advises using labels and echoes Yazici in proposing that labels be specially designed to carry strategic knowledge. Likening labels to highway signage, he believes that every labeled object in a display should be named in a way that evokes points related to it in other parts of that graphic and other linked displays. Such labels are a challenge to create because they require usability specialists to generate an unprecedented richness of categories. These categories can only be discovered by analyzing the ways in which users talk about their data and problem-solving activities in their actual contexts of work.

Furnas's ideas about labels are untested, as are many other solutions to the problem of aiding users in their interpretations and explorations, particularly those involving intelligent agents. Intelligent agents, for example, may indicate the focus of users' current state of analysis and prompt them with the next question to pursue. Or they may flag the key implications of a displayed data structure to help users draw inferences about relationships between attributes. A good number of intelligent agent prototypes and released products incorporating intelligent agents exist, but little empirical evidence of their usability is available (Lieberman 1997; Han and Zukerman 1995). The conventional wisdom about intelligent agents is that novices appreciate them more than experienced users, who often want to be able to "turn them off" or customize them.

* Aiding users instructionally when they lack preexisting schema for solving problems in an interactive graphic medium

Text and visuals again combine in this area of supporting interpretations in interactive visualizations. To support users with no preexisting schema for visually and interactively analyzing and interpreting data, interfaces and documentation need to be framed in the right terminology and language. When people do not see their own work language reflected in the interface or instructions, they have grave difficulty performing this work (Terwillinger and Polson 1997). To technical communicators, this finding is hardly surprising. But Terwillinger and Poison take their study one step farther. They compare the effects of using (1) effective terminology in both the screens and online help; (2) ineffective terminology in screens and in help; (3) effective terminology in screens but not in help; and (4) effective terminology in help but not in screens. Predictably, they find that the first solution produces the best understanding and performance, and the second, the worst. But they also find that if documentation developers try to break from the program-oriented terminology presented on screens - hoping to provide instructions in users' terms - users' understanding and performance is still poor. The same holds true for the reverse. As with labels, informational content and its match to users' ways of thinking as part of the visualization package are paramount for high usability.

* Ensuring that users remain oriented spatially when zooming in for details and out for overviews

It is challenging to keep users oriented as they pan and zoom in particular views. Some investigators have examined orienting strategies by looking for the most usable ways to scale and show the elements in a display when users bring data points closer and farther from view (Hix, Templeman, and Jacob 1995). User testing reveals that the three display elements – graphic elements, icons, and text - all require different strategies. For text, for example, the most usable appearance in a zoomed-in presentation runs counter to actual experience. On screen, users prefer text to disappear gradually as they zoom in and to reappear when they zoom out. By contrast, in real life, text seems magnified when closer and smaller when distanced. This real-life scaling on screen, however, disorients users.

Other efforts to maintain spatial orientation involve "graphic browsers" that show users' their zoomed-in site in relation to the whole landscape. Putting these mini-maps on the screen, however, challenges the perceptual principle of reserving screen real estate primarily for data.

Finally, zooming and panning have evoked studies about the usability of the zooming and panning devices themselves. Researchers' findings are inconclusive as to the most intuitive look and feel for zoom bars, buttons, and keystrokes, and the moves and outcomes of zooming that most closely match users' expectations (Johnson 1995).

Managing inquiry

Users cannot effectively conduct complex, multifaceted inquiries without successfully managing where they have been, where they are headed, and what they know and don't know - but need to know. In interacting with their data, they have to keep track of their place, monitor and evaluate the accuracy and validity of their inferences and conclusions, and iterate between analysis and presentation to share and communicate emerging results. Aside from examining the need for history records, visualization specialists rarely study the usability of visualizations in relation to managing an inquiry. Exceptions include recent investigations into multimodal support (speech in addition to graphics and text) for such communicative interactions as frequently interrupting analysis to save and present a given set of graphics that will persuade a manager about the source of a problem (Roth, Chuah, Kerpedjiev, and Kolojejchick 1997).

Another way into the subject of managing inquiry is to look at window management, a more object-oriented means for naming this activity. Visualization specialists do examine the best windowing system for organizing and interacting with many graphic displays at once. Color-coding window borders or making them fuzzy to signal to users that they are disengaging or re-engaging with a display has proven to be effective (Kandogan and Shneiderman 1997). In terms of window systems, the merits of tiled layouts versus overlapping windows versus elastic windows depend on the type of viewing users do, but in general, elastic windows give users the best support for the most types of viewing (Shneiderman 1996).

What we don't know

In regard to interpreting and managing inquiry, what is lacking most in current usability investigations is a work-in-context perspective. When users interpret and manipulate data displays, they may perform competently in tests that use predefined scenarios and preselected data geared especially to a task scenario. But the same users often founder when they return to their natural work environments and engage in similar tasks but this time with their actual and usually "messy" data and purposes. If they focused studies on users' work-in-context, usability investigators and task analysts would uncover the effects of contingencies and contextually grounded practices and work arrangements.

In terms of managing inquiry, few studies target what it means or what it takes to make visualizations usable. The importance of this dimension of usability cannot be underestimated. Users' success in inquiry depends a great deal on how well they organize and coordinate their analytical activities. In contextual analyses that I have conducted for our visualization software, I find that users commonly go into inquiry management mode at least five times every hour. Often, for accuracy's sake, they should go into this mode more to check the validity of their perceptions. See Figure 4 for an example of cross-referencing to validate accuracy.

In the long run, if people do not get accurate results because they are ill-supported in managing inquiry - monitoring, evaluating, tracking their progress and current state of knowledge - then they will have little use for their visualization program. This rejection is the single most dramatic indicator of poor usability.

In the area of complex tasks, the following investigations are needed to fill current gaps in knowledge.

* Discovering effective language and categories of meaning for labels and other explanatory "intellectual objects" that researchers such as Furnas, Kitajima and Polson, and Poitou deem vital

* Gaining an in-depth understanding of users' actual questions, inferences, misconceptions, and assumptions, and the influence of these factors on real world activities and collaborations

* Generating in-context descriptions of users' patterns of actions and pragmatic, commonsense reasoning underlying them, categorized by specific types of tasks and problems within specified work domains

* For specific problem types in various work domains, identifying points at which users (1) take stock of emerging findings; (2) compare and assess different graphic views for particular subproblems; and (3) validate assumptions and inferences

* Analyzing the designs for saving and recalling data displays that best suit users' actual problem-solving practices, discovering, for example, whether users want a recalled static view to become actively linked to displayed graphs once they bring it back into view

* Exploring the effects that interrupting inquiries to save and print a particular graphic have on users when they resume and reorient themselves to their inquiry

VISUAL QUERYING

What we know

Visual querying often disorients users. It is a new paradigm for information retrieval and analysis, and requires users to move incrementally through linked graphics, progressively building a case that will answer the question-at-hand. In the visual querying example illustrated in Figure 5, the question the user is asking is "Who owns the projects in my division that are slated for the future and what are their values and risks?" To visually query, this user sets up the appropriate graphics - in this case a bar chart on product life cycles and a data list. She then selects only the future projects values, and this selection is updated automatically on the data sheet to show these projects' values and risks.

In attempts to discover the support that users need to engage in this new visual mode of querying, investigators have explored the following questions:

* Is it better for users to learn and engage in querying or have the program prestructure inquiries and resulting displays, hiding the complexity of visual querying?

Studies show that it is better for users to engage in and master visual querying than to have the program "protect" them from it. Results of users' queries are better when users control the result set - when they control what fields are displayed, how results are sequenced, and how they are clustered (Pirolli, Pitkow, and Rao 1996).

* To help users query visually, what processes represent efficient or expert approaches that can be represented by interfaces and instruction?

Bharavani and John (1997) investigate a small slice of the processes associated with visual querying, namely actions for getting data into a desired position, a critical initial move in querying that assures a user that the structure affords an entry point into the data. The researchers identify three generic strategies employed by efficient users: First group items in a clear structure such as a hierarchy, then spread groupings out or reposition them for better discrimination, and finally tweak the individual elements that "don't quite fit." Bharavani and John's study is a first step in developing a taxonomy of subprocesses for the new paradigm of visual querying. Notably, it concentrates on finely detailed interactions typical of object-oriented approaches to users' behaviors, a level at which users' actions map in one-to-one correspondence to program functions and features.

Similarly, Shipman, Marshall, and Moran (1995) address ways to help users make meaning from data structures. They find that users recognize a small set of primitive spatial structures as conventional patterns with specific functions. Users readily relate to lists, stacks, composites, and heaps, knowing, for example, that one sorts lists, puts items in stacks to select a lot of items at once in a small, manageable space, and so on. Although their conclusion is not yet tested, Shipman, Marshall, and Moran contend that users may understand the significance of structure in visualizations better if the visualizations "reveal" themselves incrementally in terms of these composite parts.

* What specific instructions and interface information aid users in learning and engaging in this new visual mode of querying?

Investigators find that users need interfaces to indicate the outcome of each step that they are taking as they progressively refine their views. They also need a readily available history of the steps of their search (Koenemann and Belkin 1996).

* What querying devices give users optimal control for their needs?

Findings are mixed about the devices that support users best in setting up and carrying out visual queries. Options include sliders, drag and drop routines, select-by-sweeping options, and spatially oriented lenses (Fishkin and Stone 1995; Kumar, Plaisant, and Shneiderman 1997). Different devices seem better or worse depending on the specific type of question users are asking and the nature of their data. Sliders, for instance, do not support users well in conducting complex queries (Tweedie, Spence, Dawkes, and Su 1996; Fishkin and Stone 1995).

What we don't know

These investigations into the usability of visual querying focus primarily on discrete components: sliders versus lenses versus mouse selections; interface aids and history records; and matches between querying patterns and corresponding objects and operations. What is missing are studies that take an integrated look at the large scale importance of this new visual querying paradigm. The best way for investigators to understand, describe, and design for this Gestalt is by watching users interact with visualizations as they work with their own data and questions in their natural work settings.

Contextual inquiries that I have conducted on users engaged in the new paradigm of visual querying consistently generate the same comments from users. Users find the visualization tools fairly easy and straightforward to learn and use; they quickly grasp the operations that they may perform and the controls and objects that they should manipulate to perform them. Their difficulty - and it is considerable - comes in knowing how to transform their questions - whether they be in work-related terms or Boolean phrases - into appropriate sets of graphics, filters, and encodings and into optimal incremental steps and paths.

For instance, one quality control manager used our visualizations to display his production, testing, and customer service data to answer the question, "Are customers returning failed circuit boards because of poor testing methods in my factory, faulty production processes in a certain production cycle, or customers' ineffective uses of the boards or misdiagnosis of the problem?" Accustomed to examining root causes with database queries, this manager knew the Boolean queries to write, the reports to examine, and the methods for sifting through multiple reports to answer his question. He was stumped, however, when it came to figuring out where to begin in a visual querying environment. No graphic views or linkings leaped to mind as the equivalent of saying "I want to see correlations between failed boards, disgruntled customers' past returns, factory testing results, and the production cycle history for the boards. "Caught in a vicious circle, users cannot get their data into the right format until they know the optimal graphics to call up for the correlations that they want to explore.

From an object-oriented perspective, interface and instructional designers may evolve better sliders, magic lenses, or selection operations based on pre-defined task scenarios that fit perfectly with preselected data. Yet these studies will not illuminate the following issues that need further investigation:

* Users' understanding of what to do with various query devices once they try to visually tackle their real world problems with their unconditioned or uncleansed data

* Users' actual querying practices, language, and confusions in the context of use

* The combination of improved devices, verbal aids, training, and new interface metaphors that designers should start with - a baseline for expressing in interfaces and instructions the strategic knowledge that users need if they are to transform their current knowledge and conventions for information retrieval into effective approaches for visual querying

* The underlying questions and directions that animate users' current querying approaches - mapping these underlying questions to interactive visualizations

* The picture that users envision when they use terms like "correlation," including the associations that users make between certain classes of graphics and primitives and their analytical questions about data relationships in particular work and problem domains

* The extent to which facility in learning visual querying depends on skills such as knowing the right structures for particular questions and purposes

* Users' expectations from prior experiences for styles of interactivity aimed at getting detail on subsets of interest

CONCLUSIONS

Current investigations into the usability of visualizations reveal a large range of concerns that can easily occupy usability investigators well into the next century. An object-oriented grounding has led researchers to focus on discrete issues and operations: positioning of data, zooming, and selecting a color spectrum. It also has encouraged investigators primarily to examine data analysis in terms of component parts rather than larger, more contextually based activities such as structuring data for meaning in given types of problems and distinct types of data. Admittedly, some studies have wider scopes, for instance studies examining the match between the terminology and language of interfaces, instructions, and users' conceptions of work. Overall, current studies have started the process of laying the usability groundwork for this advanced technology.

Current studies, however, paint an incomplete picture of usability. Implicitly, they model users as operators of visualizations who need to learn discrete functions and features along with the buttons to push to get the views that they want. Unquestionably, users need to be trained on the tools, but that how-to-work it and how-it-works instruction is not sufficient. These skills are important, but they are only a starting point. Users also need to understand displays in relation to their situated work and subject area domains; moreover, they need to control displays, manipulating them to serve their specific problem-solving purposes and tangential questions; and they need to sharpen their visual thinking and develop the deep structures needed to make sense of what they see with assurance.

In current usability investigations, what is missing in the implicit user model is a picture of users as problem-solving experts who know their data and problems intricately and are deeply immersed in their analytical activities. To be usable, visualizations depend on users bringing this

expert intelligence to the displays. A critical dimension of usability for interactive data visualizations, therefore, is users needing cues from interfaces and instructions on how to join and integrate their intelligence with the powerful perceptual capabilities and manipulation techniques of data visualizations. Insights into what these cues and their associated designs for usability should be require a focus on users' situated work and problem-solving.

Technical communicators are particularly well suited to extend current usability studies to users' work-in-context. The rhetorical framework that guides communication professionals in their approach to information and design - verbal and visual - is precisely what is missing in many usability studies of data visualizations today. By applying their insights and perspectives on information design; by looking at the complex relationships between design, users, purpose, context, and medium; and by using their skills in contextual inquiry methodologies, technical communicators could fill in current gaps in the usability research and positively influence the direction of these interactive, data visualization products.

TABLE 2: SAMPLE OF PERCEPTUAL PRINCIPLES FOR USABLE VISUALIZATIONS

* Focus on the data. Avoid unnecessary and distracting features like dialog or overlabeling; instead, hide verbal information and have users brush over to reveal it.

* In network diagrams (for example, tree graphs or hub and spokes), users will perceive relationships between data in close proximity. Yet this perception of relationship due to proximity is overridden by physically distant points that have links drawn between them. These linked points dominate because they cover proportionately more of the screen. The most effective displays of networks exploit this perceptual tension.

* When data is densely displayed, users may fail to see some information due to occlusion or overplotting, causing them to misinterpret what the display is saying.

* People remember the outline of a graphic or image more than what is inside of it.

* Icons facilitate interaction best when their appearance distinctively represents their function.

* Users tend to invert the figure and ground when they experience excessive perceptual cueing or perceptual overload.

(Spence and Lewandowsky 1990; Eick and Wills 1995; Baker and Eick 1995; Draper and Barton 1993; Senay and Ignatius 1990)

ACKNOWLEDGMENTS

My thanks to Russell Borland for helping me think through many usability ideas and issues, for commenting on an earlier draft of this article, and for suggesting revisions. I'm grateful as well to the anonymous reviewers of this article for their suggestions and guidance.

REFERENCES

Agre, Philip. 1997. "Toward a critical technical practice: Lessons learned in trying to reform AI." In Social science, technical systems, and

cooperative work: Beyond the great divide, ed. G. Bowker, S. L. Star., W. Turner, and L. Gasser. Mahwah, NJ: Lawrence Erlbaum Associates, pp. 131-155,

Ahlberg, Chris, and Eric Wistrad. 1995. "IVEE: An information visualization and exploration environment." Proceedings of Information Visualization Symposium 95, ed. Nathan Gershom and Stephen Eick. Los Alamitos, CA: IEEE Computer Society Press, pp. 66-73.

Baker, Marla, and Stephen Eick. 1995. "Space-filling software displays." Journal of visual languages and computing 6:119-133.

Baldonado, M. Q. W., and T. Winograd. 1997. "SenseMaker: An information-exploration interface supporting the contextual evolution of a

user's interest." CHI 97 proceedings. http://www.acm.org/sigchi/chi97/proceedings/paper/mwb.htm

Berg, Marc. 1997. "Formal tools and medical practices: Computer-based decisions to work." In Social science, technical systems, and cooperative work: Beyond the great divide, ed. G. Bowker, S. L. Star., W. Turner, and L. Gasser. Mahwah, NJ: Lawrence Erlbaum Associates, pp. 301-330.

Bertin, Jacques. 1983. Semiology of graphics. Madison, WI: The University of Wisconsin Press.

Bhavnani, Suresh, and Bonnie John. 1998. "Delegation and circumvention: Two faces of efficiency," CHI 98 proceedings. New York, NY: ACM, pp. 273-280.

Boyle, John, Stephen Eick, Matthias Hemmje, Daniel Keim, J. P. Lee, and Eric Sumner. 1993. "Database issues for data visualization: Interaction, user interfaces, and presentation." In IEEE Visualization '93 Workshop: Database issues for visualization, ed. John Lee, George Grinstein. Berlin, Germany: Springer-Verlag, pp. 25-34.

Brasseur, Lee. 1997. "Literacy in the computer age: A complex perceptual landscape." In Computers and technical communication: Pedagogical and programmatic perspectives, ed. Stuart Selber. Norwood, NJ: Ablex, pp. 75-97.

Buckingham Shum, Simon, Allan MacLean, Victoria Bellotti, and Nick Hammond. 1997. "Graphical argument and design cognition. "Human-computer interaction 12:267-300.

Card, Stuart. 1996. "Visualizing retrieved information: A survey." IEEE computer graphics and applications 16 (March): 63-66.

Chambers, John, William Cleveland, Beat Kleiner, and Paul Tukey. 1983. Graphical methods for data analysis. Belmont, CA: Wadsworth International Group.

Cleveland, W. 1993. Visualizing data. Summit, NJ: Hobart Press.

Draper, S., and S. Baron. 1993. Detecting bugs with learning by exploration. Internal report. Glasgow, UK: Glasgow Interactive Systems Center, University of Glasgow.

Eick, Stephen. 1996. "Aspects of network visualization." IEEE computer graphics and applications (March):69-75.

Eick, Stephen, and Daniel Fyock. 1996, "Visualizing corporate data," AT&T technical journal 75:74-86.

Eick, Stephen, and Graham Wills. 1995, "High interaction graphics." European journal of operational research 81:445-459.

Fishkin, Ken, and Maureen Stone, 1995. "Enhanced dynamic queries via movable filters," CHI 95 proceedings, http://www.acm.org/sigchi/chi95/proceedings/papers/kpf_bdy.htm

Frese, Michael, 1987, "A theory of control and complexity: Implications for software design and integration of computer systems into the work place," In Psychological issues of human computer interaction in the work place, ed. M. Frese, E. Ulich, W. Dzida. Amsterdam, Netherlands: Elsevier Science Publishers, pp. 313-337.

Furnas, George, 1997, "Effective view navigation," CHI 97 proceedings, http://www.acm.org/sigchi/chi97/proceedings/paper/gwf.htm

Furnas, George, and Benjamin Bederson, 1995. "Space-scale diagrams: Understanding multiscale interfaces." CHI 95 proceedings, New York, NY: ACM, pp. 234-241.

Han, Yi, and Ingrid Zukerman, 1997. "A mechanism for multimodal presentation planning based on agent cooperation and negotiation." Human-computer interaction 12:187-226.

Hendee, William, 1997. "Cognitive interpretation of visual segments," In Perception of visual information, 2nd ed., ed, William Hendee and Peter Wells. New York, NY: Springer-Verlag, pp. 149-175.

Hix, Deborah, James Templeman, and Robert J. K. Jacob. 1995 "Pre-screen projection: From concept to testing of a new interaction technique," CHI 95 proceedings, http://www.acm. org/sigchi/chi95/proceedings/papers/dh_bdy.htm

Johnson, Jeff. 1995. "A comparison of user interfaces for panning on a touch-controlled display." CHI 95 proceedings, New York, NY: ACM, pp, 218-225.

Johnston, Neal. 1991. "Documenting a scientific visualization tool." Proceedings of the ACM ninth annual International Conference in Systems Documentation. New York, NY: ACM, pp. 83-87.

Kandogan, Eser, and Ben Shneiderman, 1997, "Elastic windows: Evaluation of multi-window operations." CHI 97 proceedings. http:www.acm.org/sigchi/chi97/proceedings/paper/ek.htm

Keim, Daniel, and Hans-Peter Kriegel, 1994, "Using visualization to support data mining of large existing databases," In IEEE Visualization 93 Workshop: Database issues for data visualization, ed, John Lee and George Grinstein, Berlin, Germany: Springer-Verlag, pp, 210-228.

Kitajima, Muneo, and Peter Polson. 1996. "A comprehension-based model of exploration." CHI 96 proceedings, http:/www.acm.org/sigchi/chi96/proceedings/paper/kitajima/mk_txt.htm

Koenemann, Jurgen and Nicholas Belkin. 1996. "A case for interaction: A study of interactive information retrieval behavior and effectiveness," CHI 96 proceedings. http://www.acm.org/sigchi/chi96/proceedings/paper/koenemann/jkl_txt.htm

Kumar, Harsh, Catherine Plaisant, and Ben Shneiderman. 1997. "Browsing hierarchical data with multi-level dynamic queries and pruning." International journal of human-computer studies 46: 103-124.

Larkin, J, 1989, "Display-based problem-solving." In Complex information processing, ed. D. Klahr and K. Kotovsky, Hillsdale, NJ: Lawrence Erhlbaum Associates.

Lieberman, Henry, 1997. "Autonomous interface agents," CHI 97 proceedings, http://www.acm.org/sigchi/chi97/proceedings/paper/hl.htm

Mirel, Barbara, 1997, "Minimalism for complex tasks," In Minimalism beyond the Numberg funnel, ed, John M. Carroll. Cambridge, MA: MIT Press, pp, 179-218.

Oostendorp, Herre, and Benjamin Walbeehm, 1995. "Towards modeling exploratory in the context of direct manipulation interfaces," Interacting with computers 7:3-24.

Pirolli, Peter, James Pitkow, and Ramani Rao, 1996, "Silk from a sow's ear: Extracting usable structures from the Web," CHI 96 proceedings. http://www.acm.org/sigchi/chi96/proceedings/paper/Pirolli_2/ppp2.html

Plaisant, Catherine, Brett Milash, Anne Rose, Seth Widoff, and Ben Shneiderman. 1996. "Lifelines: Visualizing personal histories," CHI 96 proceedings, New York, NY: ACM, pp, 221-227, 518.

Poitou, Jean-Pierre, 1998, "Building a collective knowledge management system: Knowledge-eliciting versus knowledge-eliciting techniques." In Social science, technical systems, and cooperative work, ed, G. Bowker, S. L. Star., W, Turner, and L. Gasser. Mahwah, NJ: Lawrence Erlbaum Associates, pp, 235-256.

Roth, Steven, Mei Chuah, Stephan Kerpedjiev, and John Kolojejchick. 1997. "Toward an information visualization workspace," Human-computer interaction 12: 131-185.

Senay, Hikmet, and Eve Ignatius. 1990, "Rules and principles of scientific data visualization." Technical Report. GWU-IIST-90-13, Institute for Information Science and Technology, The George Washington University.

Shipman, Frank, Catherine Marshall, and Thomas Moran, 1995. "Finding and using implicit structure in human-organized spatial layouts of information," CHI 95 proceedings. New York, NY: ACM, pp, 346-353.

Shneiderman, B. 1996. "The eyes have it: A task by data type taxonomy for information visualizations," Proceedings of Visual Languages 96. Los Alamitos, CA: IEEE Computer Press, pp. 336-343.

Spence, Ian, and Stephan Lewandowsky. 1990. "Graphical perception." In Modern methods of data analysis, ed. John Fox and J. Scott Long. Newbury Park, CA: SAGE, pp. 13-57.

Spoerri, A. 1993. "InfoCrystal: A visual tool for information retrieval." Proceedings of Information, knowledge and management '93. New York, NY: ACM, pp. 150-157.

Terwillinger, Robert, and Peter Polson. 1997. "Relationships between users' and interfaces' task representations," CHI 97 proceedings.

http://www.acm.org/sigchi/chi97/proceedings/paper/pol.htm

Tufte, E. R. 1983, The visual display of quantitative information. Cheshire, CT: Graphics Press.

Tukey, J. W. 1977. Exploring data analysis. Reading, MA: Addison-Wesley.

Tweedie, Lisa, Robert Spence, Huw Dawkes, Hua Su. 1996. "Externalising abstract mathematical models." CHI 96 proceedings. New York, NY: ACM, 406-412.

Wells, Peter, 1997, "Problems and prospects in the perception of visual information." In The perception of visual information, ed. William Hendee and Peter Wells. New York, NY: Springer, pp. 391-400.

Wilson, Edward O. 1997. Consilience. New York, NY: Alfred A. Knopf.

Wright, W. 1997. "Business visualization applications," Computer graphics and applications (July-August):66-70.

Yazici, Hulya. 1995. "A cognitive approach to the influence of graphics on decision-making." In Human factors in information systems, ed. Jane Carey, Norwood, NJ: Ablex, pp. 101-112.

GRAPHIC: Illustration; 3

LANGUAGE: ENGLISH

IAC-CREATE-DATE: December 10, 1998

LOAD-DATE: December 11, 1998

------------------------------------------------------------------------