Presented on June 16, 2008                                                               

Kenny Bellew/ Greg MacKey / John L. Vang

WRIT 599G-01

Topics in Technical Communication: User Research and Usability

with Professor Tori Sadler

 

Group 1’s final report

 

Dr. Sadler,

 

Enclosed is Group 1's findings regarding the usability testing of the automated library checkout system. The report will include the following:

 

1.      Summary of Report

2.      Purpose/Objectives/Problem Statements of test

3.      An overview of our testing methods

4.      Results of testing

5.      Analysis of results

6.      Recommendations

7.      Test Materials

Summary

Using low-fidelity paper prototyping, we tested the task of checking out a single library book. The paper prototype simulated the automated library checkout system at Dayton’s Bluff Library. We found this task to be fairly user-friendly. All participants in our tests were able to scan their library card and follow the simulated on-screen prompts to scan their book and conclude the checkout process using the paper prototype. They were able to do this in a reasonable amount of time without the need for supplemental help. However, for this portion of the usability test, we were tasked with only testing a single-book checkout. It is our belief that had we tested some of the more complicated tasks, such as fine payment and multimedia checkout (DVDs or CDs), more difficulties would have been discovered.

 

For the usability test, we used low-fidelity paper prototyping to simulate the automated library checkout process. We also videotaped the tests and timed certain sub-tasks. We questioned each participant to determine his or her knowledge of existing library checkout systems and knowledge of computer systems in general.  The test results may be biased due to the high-level of knowledge each participant had of library checkout systems. 

 

Purpose of the test

The purpose of the test is to determine if participants will have difficulty checking out a book using the automated checkout system and to identify any other issues or potential improvements to the system.

 

Objectives / problem statements

We hypothesize that there may be a couple of sub-tasks with which participants may have a difficult time. We are especially interested in observing the following:

·         How long it takes participants to scan their library card

·         How long it takes the participant to scan the book's barcode

·         Whether or not the participant would confuse the ISBN barcode with the book's library barcode

·         Whether or not the participant would press the Finish button to conclude the transaction

 

Testing methods

We will use the talk-aloud protocol and low-fidelity paper prototyping. We will measure two subtasks (timed events). We will videotape the test for further evaluation.

 

For the talk-aloud protocol, we used the orientation script to educate the participant in how we wanted the participant to think out loud. We also implemented three questionnaires: A pre-test, a post-task and a post-test questionnaire. We also timed two sub-tasks and videotaped the process for further investigation.

 

Background of Participants

As mentioned, our participants are well versed in usability testing and very knowledgeable of talk-aloud protocols. Due to their own homework assignments, they are very aware of library checkout systems. In our task scenarios, we told the participants that they had never used a library checkout system before the test. However, this did not seem to change the behavior of the participants related to the level of knowledge they exhibited. In our opinion, these participants will do well at pointing out usability problems, but they are poor choices to represent average users.

 

Below are the pre-test questions we asked, with each section divided by participant. Each participant’s response follows the question.

 

We asked the participants the following questions:

 

Participant 1:

  1. Do you have a library card? 

  2. From 1 to 5 (with 1 being never, 3 being monthly and 5 being yearly or more) How often do you pay library fines?  

  3. Have you ever used a library automated checkout system before?

  4. If yes, from 1 to 5, with 1 being the least familiar and 5 being very familiar, how would you rate your experience with automated library checkout systems?

  5. From 1 to 5 (with 1 being never and 5 being daily), how frequently do you use a computer system? 

 

Note that participant #1 took the pre-test questionnaire prior to being given the scenario task card. He had no reason to pretend that he had no experience with library checkout systems; yet, he claims the he is unaware as to whether or not he has ever used a library checkout system. We know that he was very familiar with library checkout systems, as we sat in on his presentation before we did ours. Our orientation script treated him as if he was a novice. Perhaps this lead him to believe he needed to pretend to be one. Nonetheless, this confusion makes his responses less trustworthy for all of his questions.

 

Participant 2:

  1. Do you have a library card? 

  2. From 1 to 5 (with 1 being never, 3 being monthly and 5 being yearly or more) How often do you pay library fines?  

  3. Have you ever used a library automated checkout system before?

  4. If yes, from 1 to 5, with 1 being the least familiar and 5 being very familiar, how would you rate your experience with automated library checkout systems?

  5. From 1 to 5 (with 1 being never and 5 being daily), how frequently do you use a computer system? 

 

Note that both participants were very familiar with using computers. We would expect these participants to find the existing screen messages common-place.

 

Problem statement: Will the user have difficulty with library card and book barcode scanning?

 

Timed-Results

One challenge to timing the scanning of the library card was that the participants started the process by talking aloud. It was difficult to separate the talking from the act of moving the card onto the laser lines of the scanner. However, there was no hesitation by either participant. Both understood that their initial goal was to place the library card's barcode beneath the scanner and the book's library barcode beneath the scanner.

 

Card scanning time:

·         Participant 1: Five seconds

·         Participant 2: Five seconds

 

Book scanning time:

·         Participant 1: Five seconds

·         Participant 2: Five seconds

 

Post-task questionnaire:

 

Problem statement continued: Will the user have difficulty with library card and book barcode scanning?

Post-test questionnaire:

 

Participant 1:

post-test01.jpg

     

Participant 2:

post-test01.jpg

 Video of library card scanning

In the following short video, you can see participant 1 scan his library card. He seems to understand intuitively the relationship between the laser lines and the library card barcode. He places the card's barcode onto the laser lines without hesitation. Observing the video, you can see that participant 1 turns the card upside down to make contact with the scan lines versus hovering the card over the lines. It is our belief that this behavior was the result of the two-dimensional nature of the paper prototype, and that he would have placed the barcode into the scan lines if an actual laser would have been present. We do not believe a flag needs to be raised because he turned the card upside down. He was attempting to place the barcode in contact with the scan lines

 

 

The the video below, participant 2 scans her card without hesitation, showing that she understands the relationship mapping between the scan lines and the library barcode.

 

 

 Video of library book barcode scanning

In the video below, participant 1 scans his book. He ignores the smaller ISBN barcode and properly scans the books library barcode.

 

 

In the video below, participant 2 properly scans the library barcode, also ignoring the smaller ISBN barcode located in the lower corner.

 

Analysis of result

As mentioned earlier, there are challenges to analyzing these two participants. Both were very well versed in library checkout systems due to their having a similar class assignment to this one. However, the behavior of the two participants was very similar to behavior we observed at the library. Most people seemed to understand intuitively to match up the barcode on their library card with the laser lines of the scanner. Similarly, the participants did not have difficulty matching up the book's barcode with the laser lines. However, we were told by the librarian that is it a common problem for users to confuse the ISBN number's barcode with the library's barcode for the book.

 

Severity rating

            Severity rating:

                        1 = Showstopper

                        2 = Major

                        3 = Minor

                        4 = Cosmetic

                        + = Good

            Frequency:

                        A = Always occurs or present

B = Occurs most of the time

C = Occurs occasionally

D = Seldom occurs

 

Trouble scanning library card barcode

·         Severity = +

·         Frequency = D

Trouble scanning library book barcode

·         Severity = 2

·         Frequency = C

While we did not experience the ISBN barcode confusion in our test, this is a severe problem that merits further investigation. If we were to rate the severity of the problem solely based on our two participants, it would receive a rating of “good.” However, to be thorough, we want to mention that a problem does exist. If the system beeps when it encounters the ISBN barcode, the book is likely to be removed from the library without being checked out. This represents a complete failure of the system to check out the book.

Recommendations

We recommend that the beep audio that sounds on error be changed to a different sound that is distinct from the beep that sounds on success. People expect the beeps to be different. Even in our test, participant #1 asked if the beep noise I made represents a "good beep" or a "bad beep."

Problem statement: Will users be able to conclude checkout by pressing Finish button?

 

Post-task questionnaire:

 

Post-test questionnaire:

 

Participant 1:

post-test02.jpg

Participant 2:

post-test02.jpg

Analysis of result

Both participants mentioned that the checkout system has the user press the Finish button twice. After pressing the button the first time, the checkout system summarizes all of the books and media checked out, and then it shows the Finish button a second time. Users seem to have no problems quickly pressing the Finish button a second time, but neither of our participants saw any utility in this additional step. The video shows that participant 2, who answered “yes” to being confused by the Finish button, had no difficulty or hesitation in pressing the Finish button when presented with it.

 

In the video below, participant 2 press the Finish button each time she was presented with the option. Her confusion regarding the Finish button appears to be more related to the utility of seeing it twice than knowing whether or not she should press it.

 

 

Severity rating

            Severity rating:

                        1 = Showstopper

                        2 = Major

                        3 = Minor

                        4 = Cosmetic

                        + = Good

            Frequency:

                        A = Always occurs or present

B = Occurs most of the time

C = Occurs occasionally

D = Seldom occurs

 

Trouble concluding with Finish button

·         Severity = 4

·         Frequency = A

 

Recommendations

We do not recommend any changes for this problem. The utility of this step is realized by users who have more books than fit on the previous screen. The final press of the Finish button shows all items being checked out at a glance. Because it does not cause checkout slow down, we recommend this problem not be proactively addressed. However, in future releases of the software, the software programmers could change the first occurrence of the Finish button from "Finish" to "Continue."

 

Other recommendations

We also recommend that the on-screen messages include prominent text on the final screen that informs the user to take his or her receipt. Our participants both eventually took the receipt, but, in one case, we had to remind the participant to do so.

 

Though it did not affect our testing, we also recommend that Spanish and Hmong be language choices for the on-screen prompts.

 

We also recommend that the receipt be printed out with the materials sorted by due date. This will help the user group items by return date to avoid fines.

 

Appendix

(See physical copies of test materials )