Lessons from our Usability Study: Part I
The most educational part of the original Graphing Calculator 1.0 development process was sitting behind a two-way mirror for five days watching first-time users, students and teachers, trying to use our software. It was a humbling and enlightening experience. Anyone doing any aspect of software development: engineering, design, support, documentation, as well as user interface folks, should be subjected to the experience. Though the our study was done with version 1.0 way back in 1993 many of the lessons we learned were timeless.
Two recent blog posts bring one particular lesson to mind. Coding Horror warns This Is What Happens When You Let Developers Create UI and Arno Gourdol discusses The Design of the Mac OS X Shutdown Feature:
It is the job of the software designer to make choices on behalf of the user. That's what designing is all about.
I worked with Arno on the belated PenMac. He contributed many ideas to the original Graphing Calculator design. (To give you a notion of the extreme attention to detail we paid on version 1.0: in the French translation of the credits page in the online help, the French translators localized his name to Arnaud Gourdol.)
I am in violent agreement with Arno's quote. As I expressed here, giving the user choices often represents our failure as designers. It is often easier to implement a choice than it is to program doing the right thing automatically. Complex dialogs, as parodied in the Coding Horror represent a cop-out.
We found something remarkable while sitting behind the two-way mirror watching users working in teams and talking aloud trying to figure out how to use the software to get their job done. Every group, nearly every time they faced a new question, would explore the software by linearly searching the menus, starting at the upper left with the Apple menu, reading each menu item, then proceeding to File and Edit and the rest. This linear search repeated many times by each group drove home the cost of each menu item. Each extra item was a distraction slowing them down each time they searched the menus.
Though the menus show you a selection of commands that the software can do, they do not tell you when something is impossible. We watched a lot of people read each and every menu item aloud wondering how to create two equations (which we did not have time to implement for 1.0.) Eventually, they sought the online help. At the time, with Mac OS 7.1.2 in 1993, the official Apple User Interface Guidelines directed that application help be brought up using the System Help menu, which was a ? icon on the far right of the menu bar away from the application menus. Most of user study participants did not know this, did not recognize the icon as a menu, and never looked in that menu. The one group which did find the menu, selected the first item in it, which enabled balloon help. Graphing Calculator 1.0 did support balloon help, but most of the messages were innocuous things identifying scroll bars and windows. That group was so distracted and frustrated by the balloons that their first comment was "How do we turn this off?" and they did not notice the "Graphing Calculator Help" menu item at the bottom of that menu.
Eventually, the person running the user study had to show each group how to bring up the online help. This was the only occasion where he had to step in. One thing we noticed while folks were searching the menu items linearly looking for the help is that some stopped at the bottom of the Equation menu saying 'The help should be here." So for the next build, we added a Graphing Calculator Help menu item at the bottom of the Equation menu as well as in the System Help menu as directed by the UI guidelines.
A later version of Mac OS moved the system Help menu to the end of the Application menus and changed its title from the icon to the word "Help", and later versions eliminated Balloon Help.