Evaluation and Usability Committee
From Digital Libraries - Fall 2008
Contents |
[edit] Committee Members
Laura O’Brien, Paul Baker, Pamela Fairlie
Contact/point person: Pamela Fairlie (fairlie@simmons.edu)
[edit] Schedule/Timeline
List of tasks in order
- done: We need to reach Candy/Jeff about the ways that our group may overlap with Web design with regards to Information Architecture. –Pamela will contact (emailed 10-7)
- done: We should find out who Marketing team determined the audience to be so that we can write up test scenarios. -Laura will do.
- got info, now to integrate it: We will contact Environmental Scan committee to see if they have found any usability tools or write-ups in other libraries that may be helpful to us. –Pamela will do. (emailed them about sharing their survey findings with us)
- done: Book Usability Lab for Nov. 20th' -Pamela
- We will talk to Jeff/Candy about a timeline of when we may need to have tool done by. Our proposal is Oct. 20th. Keeping in mind our evaluation should be into the Web Design committee 3 weeks before unveiling. -Pamela (emailed Jeff 10-7)
- revised 11-3 : We will present a draft to the class this Thursday 11-6 to have comments compiled for 11-13
- We have to find subjects for usability testing. ...see min from 11-3 for more details
- revised 11-17: see minutes from 11-17 for more details
- revised 11-18: three undergrads recruited for usability testing, 2 of whom can come to 12-4 in-class testing -Laura
- Usability testing day Nov. 20th. ...see min from 11-3 for more details
[edit] Usability Tool Drafts
- PJF: (first drafts) CHJ relative, Elementary School teacher, Internet browser
- Usability Tool Draft #2
- Please use discussion tab to comment on this usability tool.
- We'd appreciate any comments, but we would especially like feedback on the tasks: 1) Are there tasks that should be altered or rephrased? 2) Is our expected path to find information accurate as to where it will actually be found?
- Final Version of Usability Tool
[edit] Results of Usability Testing
Results Summary of Usability Testing with possible recommendations
[edit] Background Information on Usability
Two basic articles by Jakob Nielsen, standards of the usability field:
- Wrubel, Laura S.(2007). Improving Access to Electronic Resources (ER) Through Usability Testing. Collection Management. 32(1/2). 225-234. link to article
- This article describes what usability testing is and why it is important. It then describes why formal usability testing is necessary for a well designed website. It then describes how there should be special attention paid to the evaluation of the home page of the website since it is a gateway to the rest of the website. It also states that 80% of the problems in the website can be found by 5 participants. The article also describes how to use the information gathered at the Usability testing.
[edit] Meetings and Minutes
Can you meet for our first meeting at 9:30am on Monday, 9-29? -Pamela
Second meeting Monday, November 3 9:30 in Lib
According to the minutes from last week meeting we have a meeting on 11-10 at 9:30, but I guess everyone else forgot. So, when can we meet next?
Usability Testing Notes from 12/4
[edit] White Paper
Usability Testing of Digital Libraries: The digital scrapbook of Catherine Hering Jensen
In doing an environmental survey, we found that there is very little published material available on the evaluation of digital libraries. There is copious material on the evaluation of other library services, and also on the evaluation of websites; we assumed that the best format for evaluation of digital libraries would borrow from both. The structure of the class’s process for building the library, with its compressed schedule, left a limited window for usability testing. The scope of our task of evaluation is the usability of the digital library, which required that it be functional enough to be usable. At the same time, ideally, the evaluation would be done soon enough for the results to be able to have an impact on the final product. Our only viable place in the process was toward the end, evaluating the most preliminary functional versions of the library.
If the structure of the class, and particularly the schedule, limited our options for usability testing, the facilities available at Simmons significantly enhanced our options. In particular, the new Simmons Usability Lab offered capabilities that were frankly overkill for a project the size of the Hering digital library. (Of course, for our class, the creation of the library is actually secondary to our learning the skill involved in its creation.)
Evaluating the usability of the digital library requires knowing who the users might be and to what purpose they might wish to use the library. The timeline involved in this project meant that we would never be able to test actual users; we had to anticipate the needs of potential users and design testing which would emulate that use. It was our assumption that the most likely users would be family or friends of Catherine Hering Jensen or other students who had been at Simmons at the same time, along with others interested in the history of Simmons or Boston in the 1940s.
We decided to create a testing instrument in which test subjects are asked to complete a series of tasks using the library, tasks designed to emulate anticipated actual usage. Subjects were evaluated on their ability to complete the tasks, and were also asked to self-evaluate their level of satisfaction with the library. Questions were designed to test the usability of various aspects of the library. Subjects were encouraged to think out loud, to narrate their thought processes as they perform the tasks. We also observed them as they did so, and recorded our observations. Lastly they were asked to rate the ease of completion of each task on a scale of one to seven, one being extremely difficult and seven being extremely easy. This was intended to strike a balance between objective and subjective data, and also between metric and descriptive data. The instrument we created was also designed to be administrated both using the Morea software available in the Usability Lab and as a paper version in the classroom and elsewhere. In addition, we designed the testing instrument to be usable both with the live online version of the library and with a paper mockup. In actual practice we had to use a combination of live and paper versions.
We settled on a series of seven task-based questions, bracketed by three evaluative questions. Survey takers tended to take between fifteen and twenty minutes to complete the evaluation. While additional questions would have potentially yielded additional information, diminishing returns sets in; doubling the size of the survey would not double its value. Furthermore, the longer the survey, the less likely a test subject would be willing to participate.
All told, the survey was administered to four subjects. Ideally we would have preferred more, but time constraints and other limitations precluded this. (And we are very grateful to our participants.) Laura S. Wrubel, writing in Collection Management (*) , reports that “80% of the problems in the website can be found by 5 participants.” Once again, diminishing returns sets in as more participants are used, but we were below that threshold.
In general, our survey participants were satisfied with the library. They liked the layout and immediately understood the purpose of the library. Several commented favorably on the illustration on the homepage, although it was in fact a provisional illustration with no connection to Catherine Hering Jensen. Most aspects of site navigation were clear, or could be (and have been) corrected with minor tweaking. Perhaps the most problematic issue was the Browse function. Participants were unsure of its purpose or how to use it, problems which were exacerbated by its not being fully functional at the time of testing. Participants also tended to resort to the external Simmons links for information that was available within the Jensen digital library.
A summary of our survey results was forwarded to the relevant committees in the last week of class, time enough for modifications to be made based on our findings, but too late for any major redesign. Fortunately our results did not indicate that any such major changes were necessary. Some of the more minor modifications had already been made in anticipation of our results, based on observations made during the one usability survey conducted in class.
(*)Wrubel, Laura S.(2007). Improving Access to Electronic Resources (ER) Through Usability Testing. Collection Management. 32(1/2). 225-234. Retrieved October 29, 2008, from Library, Information Science & Technology Abstracts with Full Text.
