NOTE: This research on DefenseLINK was prepared by the National Center for Supercomputing Applications under a research project for the Federal World Wide Web Consortium.

This information is provided so that others may share the lessons we learned in designing our military sites.

Download this report in MS Word 6.0 format.

DefenseLINK: Usability Study

August 30, 1997

Prepared for the NCSA Federal Consortium by:

Michelle Yeh
Alaina Kanfer, Ph.D.

Technology Research Group
Education & Outreach Division
National Center for Supercomputing Applications
605 E. Springfield
Champaign, Illinois 61820
voice: 217.244.0876
fax: 217.244.1987
alaina@ncsa.uiuc.edu
http://www.ncsa.uiuc.edu/edu/trg

Scope and Purpose

Use of the World Wide Web increases continuously; in the near future, this will provide the greatest means for public access to information, and the most direct way for the Department of Defense to reach the military population and press, as well as the general public. The DefenseLINK site currently is the premiere DOD information-providing site. We conducted a usability test of the DefenseLINK site to evaluate how well the site enables users to accomplish tasks that the site designer has in mind. Subjects completed 10 tasks, which required searching for information representative of what visitors to the DefenseLINK site would be interested in. This report presents the key findings of the study and some recommendations as well as describes the evaluation procedure and general results.

Key Findings

Below are the main findings from the usability test. More details are given in the results section.

1. Subjects had the fewest problems finding information for which the links on the page corresponded directly with the tasks, e.g. finding what time the first tour of the Pentagon required selecting the "Pentagon" link from the home page, and then selecting "Tour Information" from the top of the Pentagon page. On the other hand, completing tasks in which the information location could vary, e.g. the date on which Bernard Rostker, Assistant Secretary to the Navy, was appointed head of the Persian Gulf Illness Action Team, was more difficult (and in some cases, the information was not found within the time allotted).

2. The "Search" function, which allowed subjects to search information from GILS database and press releases, was a source of frustration. Subjects were unable to understand the results of the search or why each entry provided three links. For example, one search produced no results, but still provided the subject with links, which, when selected, displayed what appeared to be the code used to run the search engine. Consequently, rather than use search as an aid, subjects appeared to use search only as a last result. In fact, one subject called the search engine "useless".

3. Subjects did not notice that the search engine presented on the "News" page was different than that presented on the home page. Rather than try both search engines, subjects assumed that if the first search attempted was unsuccessful (usually from the home page), there was no reason to try a second search from a different page.

It is important to note that while a couple of the subjects did navigate to "Frequently Asked Questions" and saw the information about the search engine, they did not spend any time reading it.

4. Subjects often did not notice that information on the "News" page was divided into two sections - that which provided current information, i.e. from the past 7 days, versus that providing information from the last 30 days. Often, when searching for contracts awarded on July 1, 1997, subjects would search for information in "News" from the past 7 days rather than that which occurred in the last 30 days.

5. Subjects did not make the distinction between the "William S. Cohen" site and the "Secretary of Defense" site. Often when trying to find what day Cohen was sworn in as the Secretary of Defense, subjects first navigated to the "Secretary of Defense" site.

6. Each of the tasks was unique and required searching for information on different pages. Thus, we were not able to determine whether subjects were able to "learn" where information was located throughout the site. However, subjects were quick to discover which tools were useful and which were not, and we could observe them learning how to use, or not use, those tools. For example, after failing to find information using the search engine, subjects were reluctant to use the search engine again for other tasks. In fact, one subject commented that she hated dealing with the search engine when encountered with a task requiring information about the search engine (task #9).

7. Universal Access: One subject that we tested was visually impaired. In general, he was able to complete the tasks successfully. Tasks on which he encountered problems were those which required accessing text from image maps or scrolling through a page containing a lot of text. In the case of the latter, he tried to "skim" through the information on the page, jumping from link to link but at times missed the relevant link. He encountered the same problem when accessing pages with text spaced out throughout the page as he could not be sure whether the line was blank or contained text centered in the middle of the line and often assumed that there was no text.

Evaluation Procedure

Subjects

Five subjects participated in the usability test between July 15-31, 1997. All subjects were familiar with the world wide web. The site was evaluated using MacIntosh and PC computers running Netscape Navigator or Internet Explorer with a direct university connection to the Internet.

Two different web designs were used during this usability test as changes were made to the "News" page on July 31, 1997, at 1:00pm CST. Three of the subjects (subject #'s 1, 2, and 5) completed the study prior to this date; two subjects (subject #'s 3 and 4) participated in the study after the changes had been made.

Note that subject #5 was blind; thus, some of the issues he encountered in usability differed from that found by the other 4 subjects.

Tasks

The usability study was designed to address the ease with which users can answer specific questions using DefenseLINK. These tasks were:

1. What date was Secretary of Defense Cohen sworn in?
2. Who is the acting Assistant Secretary of Defense (Command, Control, Communications and Intelligence)?
3. What time does the first walk-in tour of the Pentagon start?
4. What Defense organizations announced contracts on July 1, 1997?
5. What date was it announced that Bernard Rostker, Assistant Secretary of the Navy (Manpower & Reserve Affairs), is to head the Department of Defense's Persian Gulf Illness Action Team?
6. What organization publishes personnel statistics for the DoD?
7. How often is DefenseLINK News Updated?
8. What date was the most recent Quadrennial Defense Review released?
9. What two types of data are currently searchable on DefenseLINK?
10. What is the size of the customer base supported by The Real Estate and Facilities Directorate (RE&F) of the Washington Headquarters Services?

Data Collection

Subjects were asked to complete the tasks in order, starting each task from the DefenseLINK home page. They were allowed to use the search engine to perform the task. Response time data was collected. Additionally, subjects were instructed to "think aloud" as they completed each task in order to give the experimenter better insight into the reasoning behind the subjects' actions. Subjects were instructed to complete each task as quickly as possible. They were given a maximum of 10 minutes to complete each task. Subject #5 was given 15 minutes to complete each task since he was accessing the information with a Braille reader which displayed only one line at a time.

Results

Table 1 gives a summary of the tasks completed and the amount of time required as well as the success rate. Note that the time for subject #5 was not averaged in with subjects 1-4, but his ability to complete the task was included as part of the success rate since the web page should be accessible by all users, regardless of physical disability.

Table 1. Average time and success rate in completing the tasks. Please note that the mean time subjects used in completing the tasks may be misleading, i.e. the amount of time each subject needed to complete each task varied greatly as confirmed by the high standard deviations.

More detailed results are organized by task. Appendix 1 presents tables of issues encountered by each subject for each task.

1. What date was Secretary of Defense Cohen sworn in?

In completing this task, 2 of the 5 subjects first selected the "Secretary of Defense" link from the navigation menu at the top of the page rather than the "William S. Cohen" link embedded in the text. From the "Secretary of Defense" page, one subject selected the link to the "Office of the Secretariat" page; from there, she selected the "Secretary of Defense" page and was caught in a loop until she realized she was seeing the same page over and over.

2. Who is the acting Assistant Secretary of Defense (Command, Control, Communications and Intelligence)?

The information to answer this question was found from a link presented on the "Secretary of Defense" page. Similar to the problems encountered in finding the date on which Cohen was sworn in as Secretary of Defense, subjects were unsure which page would contain this information -- the "Secretary of Defense" page or William Cohen"s page. In some cases, both pages were searched.

One subject was surprised by the appearance of the page for the Assistant Secretary of Defense (Command, Control, Communications, and Intelligence), commenting that there was no real header on the page and criticizing the organization of information as a search engine was presented to the user before information pertaining to the site.

Subject #5 had some difficulty with this task due to the amount of text on the home page. In fact, the subject skimmed the information on the page and missed the link.

3. What time does the first walk-in tour of the Pentagon start?

This task was completed easily; in fact, 4 of the 5 subjects found this information on their first try. Subjects found the link for the Pentagon, then navigated to "Tour Information". The one subject who encountered problems with this task searched for "Pentagon" tour times using the "Other Information" link at the navigation menu at the top of the page rather than scroll through the text in the middle of the page.

4. What Defense organizations announced contracts on July 1, 1997?

The fact that contract announcements were presented as "News" was not obvious to any of the subjects. One subject looked for a "contracts" link on the home page, another looked under "Recent Additions", while another searched under "Doing Business with the Department of Defense". Other pages searched were Defense Agencies, American Forces Press Services, American Forces Information Services, Secretary of Defense, Fact File, and DTIC General Information.

Three of the five subjects searched for all information with the date "July 1" but received no relevant information. One subject even went so far as to search for information with the date formatting "97/07/01".

Four of the five subjects first searched for contract information for July 1 on the page presenting contract announcements for the past 7 days. Note that two of these subjects completed this task using the new design and all of the evaluations took place after one week past July 1. Of the four subjects, two eventually realized why the information on that page did not cover July 1. Of the other two, one attempted a search for "July 1" whereas the other (who was blind) navigated back to the "News" page, and selected a "Contracts" link that presented information from the past 30 days and found the information he needed, without realizing that he had in fact navigated to two different pages. On completing the task successfully, the latter subject commented that he did not understand why contract announcements from July 1 had not been presented originally.

5. What date was it announced that Bernard Rostker, Assistant Secretary of the Navy (Manpower & Reserve Affairs), is to head the Department of Defense's Persian Gulf Illness Action Team?

Of the five subjects participating in the study, 2 did not successfully complete this task.

Due to the phrasing of the question, which pointed out the fact that Rostker was the Assistant Secretary of the Navy, subjects' first inclination was to navigate to the Navy site via a link on the DefenseLINK home page to find more information about Bernard Rostker. One subject did not realize that in doing this, he was leaving the DefenseLINK site and commented later that it was easy for him to leave the site without realizing it. The fact that subjects spent probably half the time searching through the Navy site for this information contributed to the lack of success for the task.

Subjects also tried performing a search for this information, with no success. One subject claimed that she found the results "useless" and stated "I don't understand what it's giving me". Another subject searched the site for "Rostker", which produced no results, but the subject was presented with 3 links, each when selected provided the subject with what appeared to be program code.

Subjects then navigated to the "News" page and were presented with another search engine. Since a search had already been attempted, subjects did not try and perform a second search as there was no indication that this second search engine was different from the one presented on the home page. In fact, one subject commented on this; rather than attempt another search, this subject found the information by attempting to browse all the articles presented in the "News Releases". Luckily, she stumbled across an article in the News Releases, entitled "Troops not exposed to dangerous levels of chemical agent", in which it stated that Rostker had been appointed in October of 1996. She then searched all the news releases for October until she found the one listing his appointment date.

In performing this task, subjects' hesitation to use the search engine made finding the answer difficult. The presentation of search results was not easily interpretable by the subjects. In fact, one commented that he did not want to use the search engine as "it didn't help much before" on an earlier task. Subjects did not spend any time attempting to find out how to interpret the results (presented in "Frequently Asked Questions"); rather, they attempted to complete the tasks without search.

6. What organization publishes personnel statistics for the DoD?

Subjects were able to complete this task relatively easily. Subjects were not always sure where to find the answer -- other pages searched besides "Publications" were "Biographies", "Other Organizations", and the "American Forces Information Services". The name of the organization was presented in an image map and was unreadable by the subject who was blind. Another subject saw the image map and the Department of Defense seal and did not look any longer at the image map.

7. How often is DefenseLINK News Updated?

Subjects were able to complete this task successfully with few problems. One subject first looked for this information under "What's New" but then navigated to the "News" site.

8. What date was the most recent Quadrennial Defense Review released?

Three of the subjects completed this task easily without problems. Subjects who had difficulty with this task were unsure as to what topic area the Quadrennial Defense Review belonged and searched under "News". One subject attempted to search for "Quadrennial" in the DefenseLINK site, but she felt that the results were not helpful and concluded that the search engine was useless.

Subject #5 found the Quadrennial Defense Review successfully but almost missed the date of the release due to the formatting on the cover page, in which the heading (title and date) was centered and spaced throughout the page. The subject commented that information formatted in this manner was easy to miss using a Braille web browser.

9. What two types of data are currently searchable on DefenseLINK?

Subjects were able to complete this task successfully, though one subject's first comment after she saw the question was that she hated dealing with search. Another subject tried searching for this information near the search engine presented on the home page but then navigated to the "search" link.

10. What is the size of the customer base supported by The Real Estate and Facilities Directorate (RE&F) of the Washington Headquarters Services?

Subjects were able to complete this task relatively easily using the search engine. Most searched for "Real Estate and Facilities Directorate". One subject assumed that the results from the search were web links to the actual page, rather than just a summary of that information.

One subject did not complete this task successfully within the allotted time. The subject searched for information under the "Other Organizations", "Frequently Asked Questions", "Public Affairs", "Other Related Sites of Interest", and "Secretary of Defense" pages. The subject successfully requested a search for any documents containing information about "r&f", but when he selected the link, he was presented with the bottom of the page, and he did not scroll up to search for any more information on the page (the answer was at the top of the page).

Appendix 1: Issues Encountered by Task

1. What date was Secretary of Defense Cohen sworn in?

2. Who is the acting Assistant Secretary of Defense (Command, Control, Communications and Intelligence)?

3. What time does the first walk-in tour of the Pentagon start?

4. What Defense organizations announced contracts on July 1, 1997?

5. What date was it announced that Bernard Rostker, Assistant Secretary of the Navy (Manpower & Reserve Affairs), is to head the Department of Defense's Persian Gulf Illness Action Team?

6. What organization publishes personnel statistics for the DoD?

7. How often is DefenseLINK News Updated?

8. What date was the most recent Quadrennial Defense Review released?

9. What two types of data are currently searchable on DefenseLINK?

10. What is the size of the customer base supported by The Real Estate and Facilities Directorate (RE&F) of the Washington Headquarters Services?

ADDENDUM

DefenseLINK: Usability Study Part 2

August 30, 1997

An abbreviated follow-up evaluation of the new DefenseLINK search engine was conducted due to the disappointing findings from the first examination. Previous results showed that subjects found the search feature "frustrating" and "useless" as they were unable to understand the results of their search and dreaded performing any task requiring the search engine. Since that time, the search engine was upgraded, and the additional usability testing determined whether the new search engine was, in fact, more usable.

In this second examination, two subjects who participated in the first evaluation, performed two tasks using the newly re-designed search engine. The subjects were expert web users; one was visually impaired. The subjects were asked to find answers to the following two questions:

1. What date was it announced that Bernard Rostker, Assistant Secretary of the Navy (Manpower & Reserve Affairs), is to head the Department of Defense's Persian Gulf Illness Action Team?
2. What is the size of the customer base supported by The Real Estate and Facilities Directorate (RE&F) of the Washington Headquarters Services?

The same procedure used in the first usability evaluation was used in this study.

The results are summarized in Table 1 presented below.

Table 1. Time and success rate in completing the tasks. For comparison purposes, numbers in parenthesis are those of each subject from the first attempt in completing the task during the previous study.

In addition to finding the required information, subjects actually were able to successfully use the search engine in finding the information they needed. Task #1, which subject #5 had not been able to complete previously in the allotted time limit, was completed successfully with the aid of the new search engine.

However, some confusion still existed. For example, one subject was unsure of whether the search completed was an AND search or an OR search when more than one word was entered. Although the results were more interpretable than they were previously, subjects were now unsure of which category to which they needed to navigate in order to find the answer to the question (e.g. "News", "Publications, "GILS", etc.). Both subjects mentioned that they did not know what GILS meant, and in fact, while the link for the Real Estate and Facilities Directorate home page (which provided the answer to the second task) was located within that category, one subject said "it's not where I would have looked" when he happened to stumble across the link.

Overall, subjects liked the re-design of the search engine; both subjects commented on how much more usable the new search engine was as compared to the previous one.