Testing the Usability of Synchronous Computer-Supported

                        Cooperative Work Products

                                    

                                    

                             Lynellen Perry

                              July 25, 1994

                               Dr. Carter

            CS 9253 Topics in Software Engineering: Usability

                          10 week Summer Term 

                                    

                                    

                                    

                                    

                                    

                                    

                                    

                                    

                                    

                                    

                                    

                                    

                                    

                                    

                                    

                                    

                                    

                                    

                                    

                                    

                                    

                                    

                                    

                                    

         "The needs of a group using a tool collaboratively, are

        different from those of an individual user"  John C. Tang

                                    

        "The needs of the many out-weigh the needs of the few, or

              the one..."  Star Trek II: The Wrath of Khan

                                     

                                       Introduction

     

          Computer-supported Cooperative Work (CSCW) goes by many

     names: groupware, computer-supported collaboration, workflow,

     group decision-support systems (Palmer, 15), electronic

     meeting systems (Valacich, 261), and probably several others. 

     There are nearly as many definitions of CSCW as there are

     authors on the subject, and include the following.  Palmer et

     al. define CSCW as "people working together on a product,

     research area, topic, or scholarly endeavor with help from

     computers" (Palmer, 15), but also as "A system that integrates

     information processing and communications activities to help

     individuals work together as a group" in the same paper

     (Palmer, 16).  Palmer does not make a distinction between the

     term "CSCW" and any of the other terms mentioned above. 

     

          Greenberg, however, states that 'groupware' is merely

     software that "supports and augments group work" while 'CSCW'

     is "the scientific discipline that motivates and validates

     groupware design . . . the study and theory of how people work

     together, and how the computer and related technologies affect

     group behavior" (Greenberg, 133).  In this view, CSCW collects

     research from scientists in the Computer Science, Cognitive

     Science, Psychology, Sociology, Anthropology, Ethnography,

     Management, and Management Information Systems fields.

     

          Many software products can fit into the 'groupware'

     concept: email, bulletin boards, asynchronous conferencing,

     group schedulers, group decision support systems,

     collaborative authoring tools, screen-sharing software,

     computer equivalents to whiteboards, video conferencing

     (Greenberg, 133), multigroup decision-support systems (Palmer,

     16), computer-assisted design/computer-assisted manufacturing

     (CAD/CAM), computer-assisted software engineering (CASE),

     concurrent engineering, workflow management, distance

     learning, telemedicine, real-time network conferences (MUDs

     and MUSHs) (Grudin, 20), and even spreadsheet programs (Nardi,

     161).

     

          Each of the software types above fits its users into one

     of the space/time categories which are shown in the table

     below (Grudin, 25).  There are many research papers describing

     the testing of distributed systems, where the users are not in

     the same place and/or not working at the same time.  However,

     this research review focuses of products where the intended

     users are in the same place (same room), working together at

     the same time on the same project.  This is called synchronous

          work.Grudin's Space/Time Categories

     

     

     

     same time; 

     same place

     different and

     predictable time;

     same place

     different but

     unpredictable

     time; same place

     

     

     same time;

     different and

     predictable place

     

     different and

     predictable time;

     different and

     predictable place

     different but

     unpredictable

     time; different

     and predictable

     place

     

     

     same time; 

     different but

     unpredictable

     place

     different and

     predictable time;

     different but

     unpredictable

     place

     different but

     unpredictable

     place and time

     

     

     

        Research on the Usability of Synchronous CSCW Systems

          Now that some of the terms in the title of this paper

     have been defined, let us turn to the usability testing

     aspect.  Systematic, formal, scientific usability testing is

     still a rather new research area.  The basic methods of

     testing usability include heuristic evaluation, user testing,

     and cognitive walkthroughs.  Nielson describes five usability

     attributes that testing should measure: learnability,

     efficiency, memorability, errors, and satisfaction (Nielson,

     26).  From the literature, it appears that user testing is the

     most widely used form of usability testing and that efficiency

     and satisfaction are the usability attributes most often used

     as measures.  All papers reviewed below implement some variety

     of a user test to study the usability of various groupware

     products.

     

                               Cognoter

          Tatar et al. performed usability tests on Cognoter, a

     software package designed to aid small work groups (two to

     five people) in the creation of a plan or outline.  To test

     the usability of this software, Tator and colleagues ran

     several user tests.  These user tests took the form of a

     series of two hour working sessions of two groups, each of

     which consisted of three users who were experienced with

     computers.  

     

          The test room contained a workstation for each test user,

     and a large screen which was configured to display to the

     group the shared work area available in the software.  The

     Cognoter software is divided into a private editing area and

     a shared work area.  Any individual may type in a brief note

     in the private area and then release it to the shared area. 

     These notes appear randomly in the shared area as icons with

     a keyword.  Any icon may be clicked upon to reveal the full

     annotation that was entered.   

     

          The experiment was run two times.  The first time, a

     pilot test, the experimenters had severe problems observing

     the user groups as they worked.  Given the way the experiment

     was set up, it was impossible for the observers to see the

     details of work because each user had a separate machine. 

     Also, the observers were people who were very familiar with

     the performance characteristics of the Cognoter software, and

     tended to compensate for any problems the users had, thus

     biasing the results.

     

          So for the actual experiment, Tatar et al. videotaped

     each test session and also logged all messages sent between

     machines.  This solved the problem of not being able to

     observe everything that went on during the test, but the other

     problem mentioned above was not corrected.  The three users

     chosen to be tested in each group were expert users of

     Cognoter.  They were long-term collaborators who were familiar

     with the editor, window system, and mouse conventions used in

     the Cognoter software.  Three developers were available to

     help the test users with any problems that arose.

     

          The tasks for each of the two groups tested were not the

     same.  Each group was asked to use the Cognoter software to

     brainstorm about a subject of their own choosing that would be

     useful for their own work.  By not specifying the task for

     both groups, another variable has been introduced into this

     experiment, yet the authors did not comment on this fact or

     analyze what effect this had on the test.  There were no

     usability goals set for this test, and no methods of measuring

     the usability were described.  The goal appears to have been

     simply the discovery usability problems, though problems found

     were not rated by severity after the test.

     

          The results reported were that users expressed "extreme

     frustration and reduced efficiency" (as compared to working

     with traditional paper and/or whiteboard).  Though classed as

     "experts", neither group understood the software well enough

     to use its full potential.  In Group A, everyone first worked

     on their own, using the private editing area but not looking

     at each other's work or talking or sharing ideas in any way. 

     They then left the computer and worked together on paper.  The

     effect of this was that the software was used as a sort of

     word processor, not as a tool for group interaction.  

     

          Group B, on the other hand, figured out how to use the

     video capability of the software so that the whole group could

     see the work of whoever was typing.  They didn't understand

     though, that there could be more than one typist at a time. 

     Despite this bit of success in using the software for group

     interaction, Group B also expressed their frustration visibly,

     with people putting their head in their hands, raising their

     voices, and threatening to walk out.

     

           This rather disastrous user test of the Cognoter

     software indicated that the users were experiencing two types

     of problems.  First, users wanted to see things in the

     workspace that the system would not let them see.  Second,

     users mistook references in one another's speech or actions

     (pointing to individual screen and saying "there", "this",

     "that", etc.) and could not resolve the difficulty

     satisfactorily.

     

          These problems lead to the decision that eight design

     decisions were at fault for the user difficulties (Tatar,

     198).  

          1. Separate screens -- gaze and gestures were missed by

     group members because each was busy looking at their own

     screen.  

          2. Lack of sequentiality -- there was no way to know

     where the next icon would appear, or to know in what order the

     icons had been created.  

          3. Short labels on the icons limited the information the

     group could see.  

          4. Anonymity.  

          5. Private editing allowed someone to change a previous

     contribution, thus losing information.  

          6. There was unpredictable delay between the release of

     a privately edited item and the time it appeared on other

     user's screens.  

          7. Private moving of icons caused an icon to change

     position suddenly on other user's screens and thus lose its

     identifiable position.  

          8. Individually tailorable windows caused confusion when

     attempting to reference a particular item on the screen.  

     

          These design decisions "made Cognoter items more

     difficult both to create and to use than whiteboard objects"

     (Tatar, 203).  In a redesign of the software, only the last

     four of the above design decisions were changed.

     

          In addition to the above software usability problems,

     Tatar's paper also discusses what they learned about groups

     and modes of conversation. These topics, while interesting and

     necessary to understanding groups in order to build worthwhile

     software to support groups, are beyond the scope of this

     research review.

     

                             GroupSystems

          Another electronic meeting support (EMS) system,

     GroupSystems, is described in Valacich et al.  EMS systems can

     be used to support distributed groups.  However, The

     University of Arizona (where this research was conducted) has

     focused on face-to-face (synchronous) meetings.  Tasks that

     can be accomplished with the GroupSystems facilities at The

     University of Arizona include "communication, planning, idea

     generation, negotiation, conflict resolution, systems analysis

     and design, and collaborative group activities such as

     document preparation and sharing" (Valacich, 261).

     

          Valacich et al. measure the usability of the facility in

     terms of the productivity of the meeting, as manifested by the

     reduction or elimination of the "dysfunctions of the group

     interaction (i.e. process losses), so that a group reaches or

     exceeds (i.e. process gains) its task potential" (Valacich,

     262).  There are many process losses, but Appendix A discusses

     the ones relevant to the GroupSystems environment at the

     University of Arizona.  Variables researched that affect the

     productivity of a group include group size, group task,

     anonymity, and proximity.

     

          The hardware setup at the GroupSystems facility is as

     follows: each participant has a work area, all of which are

     arranged to focus on the front of the room.  Each work area

     has a separate color graphics microcomputer networked to the

     others.  There is a facilitator's console to control the EMS,

     at least one large screen video display, and other audio-

     visual support such as white boards and overhead projectors at

     the front of the room.  A control room next door to the

     meeting room has a laser printer and a copier.

     

          Valacich et al. summarize seven laboratory studies

     conducted in the GroupSystems environment where the task for

     the group was idea generation.  They also review two

     laboratory studies where the group task was decision making. 

     Six GroupSystems field and case studies are then presented. 

     All of these studies are conducted via user testing.

     

          Results of these user tests validated the high usability

     of the GroupSystems software and hardware environment. 

     Participants in the studies, which included several groups

     from real-world corporations, stated that meetings supported

     by GroupSystems were much more satisfying, effective, and

     productive than traditional meetings.  IBM has even installed

     more than 36 electronic meeting rooms around the world, using

     the software and hardware environment of GroupSystems.  

     

          In addition to validating the usability of GroupSystems

     for synchronous group work, these studies uncovered and

     confirmed data about how groups work.  Again, details on this

     subject are interesting but are beyond the scope of this

     paper.  Briefly, the studies indicate that CSCW software

     should support large groups (9+ people) over small groups (2-5

     people), and that group members should be anonymous for the

     highest productivity and satisfaction ratings. 

     

     

                  Amsterdam Conversation Environment

          Dykstra and Carasik discuss the "theory and concepts in

     designing a synchronous shared workspace to support human

     interaction" (Dykstra, 419).  They describe an implementation

     of such a system, the Amsterdam Conversation Environment, or

     ACE.  In agreement with Palmer's definition of CSCW (above),

     the authors feel that technology should support groups rather

     than replace or automate activities.  To this end, ACE is not

     task specific.  It is meant to provide users with "a common

     workspace through which they can share and manipulate

     individual products, where the focus is on stimulating

     interaction rather than on producing a product" (Dykstra,

     420).

     

          Dykstra describes how ACE evolved via several iterations

     of user tests.  The current prototype of ACE runs on a network

     of Macintoshs, with a main server maintaining links between

     objects, keeping track of users, and controlling the

     simultaneous update of the user's screens.  However, it was

     designed and user tested first by a physical model, and then

     on overhead transparencies.  If done carefully, these can be

     cheap ways of performing user testing because no software has

     been written at this point.  Mistakes and redirections are

     much cheaper when they occur in the design phase rather than

     after code has been written.

     

          The authors were still developing testing procedures for

     the ACE software prototype at the time they wrote the paper,

     so no results are available on that phase of user testing. 

     Dykstra does mention, though, the usability attributes they

     wish to measure.  These include user satisfaction (especially

     in light of the fact that the software provides so few

     restraints on group process), productivity (as indicated by

     the amount of "process paralysis" that occurs during use), and

     learnability (they hope to avoid needing a specially trained

     facilitator for the software and that the documentation needed

     will be minimal) (Dykstra, 433).

     

                             Spreadsheets

          Nardi and Miller observe that spreadsheets are actually

     developed by the cooperative work of several people most of

     the time.  They use a simple definition of cooperative work,

     "multiple persons working together to produce a product or

     service" (Nardi, 162), and state that there are two forms of

     cooperative work central to CSCW that have not received much

     attention.  Most CSCW research, they argue, is focused on

     computer systems that encourage communication between group

     members.  By having this focus, researchers have overlooked

     the fact that collaboration in programming itself is very

     common.  The sharing of programming expertise and the sharing

     of domain knowledge are obvious in real-world uses of

     spreadsheets.  

     

          Though spreadsheets are usually considered to be a

     single-user application, Nardi and Miller have found that

     "spreadsheet co-development is the rule, not the exception",

     and thus include spreadsheets in the category of synchronous

     groupware.  As further justification of this categorization,

     they note that spreadsheet users: 

          "1) share programming expertise through exchanges of

     code;

          2) transfer domain knowledge via spreadsheet templates

     and the direct editing of spreadsheets;

          3) debug spreadsheets cooperatively;

          4) use spreadsheets for cooperative work in meetings and

     other group settings; and

          5) train each other in new spreadsheet techniques"

     (Nardi, 163).

     

          Nardi and Miller did not perform usability testing of

     synchronously developed spreadsheets in a traditional way. 

     They did not set up their own experiment, find test users, and

     then evaluate the results.  Instead, they tape recorded

     interviews with experienced spreadsheet users in their own

     offices and homes.  In these interviews, they asked a fixed

     set of open-ended questions in whatever order that they came

     up during the conversation.  

     

          Through this process, they found that spreadsheet

     programs are easier to learn cooperatively (spreadsheet

     experts share their programming knowledge with novices),

     spreadsheets are developed more efficiently when developed

     cooperatively (domain knowledge is shared), and there are

     fewer errors in cooperatively developed spreadsheets (the

     collaborator can often spot a programming or logic mistake

     faster than the author).  In addition, novice spreadsheet

     programmers feel more satisfied when they develop spreadsheets

     cooperatively.  These results cover four of the five

     attributes that Nielson says are a part of usability.  Using

     this rather unorthodox method of experimenting, Nardi and

     Miller thus found that a "single-user" application could be

     used in a groupware manner, and they give a few suggestions to

     software developers of "single-user" applications to help make

     those products capable of being used effectively by small

     groups as well as individual users.

     

     

     

                       Summary and Conclusions

          Each of the research papers reviewed contains a slightly

     different idea of the term CSCW, and uses different methods in

     testing the usability of products that support groups.  CSCW

     and Usability testing are still relatively new fields of

     research, so the vocabulary and the methods of testing have

     not yet solidified.  In addition, groupware products can be

     divided into nine space/time areas, making the research arena

     rather large.  

     

          This research review has focused on research that tests

     the usability of synchronous computer-supported cooperative

     work products.  It appears that the most popular method of

     conducting usability tests is the user test.  User tests have

     been used to validate the usability of rather different types

     of groupware products, from spreadsheet applications

     originally intended for use by a single user, to electronic

     meeting rooms that provide both a software and hardware

     environment to support decision making, brainstorming, and

     other typical group activities.

     

          In all of the above research, the authors show that

     usability testing of CSCW or groupware products has many side

     benefits besides the discovery of usability problems of a

     particular implementation.  We do not yet fully understand how

     groups work and how computers can best support group

     processes.  Researchers are trying to understand, among many

     other variables, how group members communicate among

     themselves, how shared work areas (computer-supported and

     traditional) are used, how important are gestures and other

     visual and social cues to getting work done, and how proximity

     and anonymity affect the productivity and satisfaction of the

     group.  Usability studies have, and probably will continue to

     provide many insights into this area at the same time that

     they show developers specific problems with particular

          packages.               Appendix A -- Valacich's Process Losses

     

     Production blocking. "Refers to the fact that only one member

     of a group can speak at a time during verbal communication"

     (Valacich, 262).  This has three effects on the meeting:

          1. others waiting to speak may forget or suppress their

     ideas because they eventually seem less relevant or original

     (attenuation blocking) 

          2. while waiting to speak, group members may not be truly

     paying attention to the speaker, rather they are focusing on

     trying to remember their own idea (concentration blocking) and

          3. while listening to a speaker, other group members are

     not generating their own new ideas (attention blocking).

     

     Unequal air time.  As groups get larger, the amount of time

     that each person could possibly use for verbal communication

     gets smaller.

     

     Evaluation apprehension.  Individuals may shy away from

     sharing ideas and comments for fear of negative evaluation by

     the others present.

     

     Free-riding.  Individuals may try to make the other group

     members accomplish the task without any contributions from

     themselves.  Free-riding may be caused by social loafing, but

     can also be greater when individuals think that their

     contributions are not as necessary for the success of the

     group (i.e. it is a large group, so surely someone else will

     think of the same things that I would have).

     

     Cognitive inertia.  This is the "tendency of discussions to

     move along one line of thought without deviating from the

     current topic" (Valacich, 263).

     

     Socializing.  Chatting, drinking coffee, eating refreshments,

     and other non-task related activities.

     

     Domination.  "Occurs when some group member(s) exercise(s)

     undue influence or monopolize(s) the group's time in an

     inefficient manner" (Valacich, 263).

     

     Failure to remember.  Individuals do not pay attention to

     and/or remember comments that others have said.

     

     Incomplete analysis.  This occurs when the group does not use

     all the information available to it, or fails to challenge

          assumptions.                             


			WORKS CITED

     

     Dykstra, E. A., and R. P. Carasik. 1991.  Structure and

          Support in Cooperative Environments: the Amsterdam

          Conversation Environment. International Journal of Man-

          Machine Studies 34:419-434.

     

     Greenberg, S. 1991. Computer-Supported Cooperative Work and

          Groupware: An Introduction to the Special Issues. 

          International Journal of Man-Machine Studies 34:133-141.

     

     Grudin, J. 1994. Computer-Supported Cooperative Work: History

          and Focus. Computer 27:19-26.

     

     Nardi, B. A., and J. R. Miller. 1991. Twinkling Lights and

          Nested Loops: Distributed Problem Solving and Spreadsheet

          Development. International Journal of Man-Machine Studies

          34:161-183.

     

     Nielson, J. Usability Engineering.  Boston: AP Professional,

          1993.

     

     Palmer, J. D., and N. A. Fields. 1994. Computer-Supported

          Cooperative Work. Computer 27:15-16.

     

     Tang, J. C. 1991. Findings from Observational Studies of

          Collaborative Work. International Journal of Man-Machine

          Studies 34:143-160.

     

     Tator, D. G., G. Foster, and D. Bobrow. 1991. Design for

          Conversation: Lessons from Cognoter. International     

          Journal of Man-Machine Studies 34:185-209. 

     

     Valacich, J. S., A. R. Dennis, and J.F. Nunamaker, Jr. 1991. 

          Electronic Meeting Support: the GroupSystems Concept. 

          International Journal of Man-Machine Studies 34:261-279.