NOTE: This research on DefenseLINK was prepared by the National Center for Supercomputing Applications under a research project for the Federal World Wide Web Consortium.

Many of the sites and/or problems identified in this report have been changed as a result.  This information is provided so that others may share the lessons we learned in designing our military sites.

Download this report in MS Word 6.0 format.

DefenseLINK: Heuristic Evaluation

June 19, 1997

Prepared for the NCSA Federal Consortium by:

Michelle Yeh
Alaina Kanfer, Ph.D.

Technology Research Group
Education & Outreach Division
National Center for Supercomputing Applications
605 E. Springfield
Champaign, Illinois 61820
voice: 217.244.0876
fax: 217.244.1987
alaina@ncsa.uiuc.edu
http://www.ncsa.uiuc.edu/edu/trg

Scope and Purpose

Use of the World Wide Web increases continuously; this is the largest, most accessible source of information to the public. The purpose of this research was to evaluate the NSF web site in terms of overall look and feel of the pages as well as provide recommendations for possible improvements in their design. This report contains a description of the evaluation procedure, general results and recommendations, as well as detailed comments on each page of the DefenseLINK website resulting from the heuristic evaluation.

Evaluation Procedure

Three evaluators independently analyzed the DefenseLINK web site during June 10-12, 1997. Each page was analyzed according to whether or not it is in accordance with each of ten guidelines for interface usability. The guidelines used in this evaluation were those proposed by Nielsen (1994), which are:

• Visibility of system status: The system should always keep users informed about what is going on, through appropriate feedback within reasonable time.
• Match between system and the real world: The system should speak the usersí language, with words, phrases, and concepts familiar to the user, rather than system-oriented terms. Follow real-world conventions, making information appear in a natural and logical order.
• User control and freedom: Users often choose system functions by mistake and will need a clearly marked ìemergency exitî to leave the unwanted state without having to go through and extended dialogue. Support undo and redo.
• Consistency and standards: Users should not have to wonder whether different words, situations, or actions mean the same thing. Follow platform conventions.
• Error prevention: Even better than good error messages is a careful design which prevents a problem from occurring in the first place.
• Recognition rather than recall: Make objects, actions, and options visible. The user should not have to remember information from one part of the dialogue to another. Instructions for use of the system should be visible or easily retrievable whenever appropriate.
• Flexibility and efficiency of use: Accelerators -- unseen by the novice user -- may often speed up the interaction for the expert user to such an extent that the system can cater to both inexperienced and experienced users. Allow users to tailor frequent actions.
• Aesthetic and minimalist design: Dialogues should not contain information which is irrelevant or rarely needed. Every extra unit of information in a dialogue competes with the relevant units of information and diminishes their relative visibility.
• Help users recognize, diagnose, and recover from errors: Error messages should be expressed in plain language (no codes), precisely indicate the problem, and constructively suggest a solution.
• Help and documentation: Even though it is better if the system can be used without documentation, it may be necessary to provide help and documentation. Any such information should be easy to search, focused on the userís task, list concrete steps to be carried out, and not be too large.

The pages evaluated were those presented on the DefenseLINK home page (and their subsequent links). As many of the sub-pages as possible were examined in the evaluation. The evaluation focused on sites with web addresses that appeared to be maintained by the DefenseLINK group (i.e. http://www.dtic.mil/defenselink/index.html). Links three or more levels (e.g. links from the "Other Information" page) were examined only briefly.

Results

Please note that the goal of a heuristic evaluation is to cite violations of usability guidelines; thus, the result is a list of problems. There were many aspects of good design in this interface, which will not be discussed in great detail, but are mentioned in the last section of this report.

Animation: There was an excessive use of animation throughout the site. Most notably, the National Guard home page (http://www.dtic.mil/defenselink/guardlink/) had an overwhelming amount of movement with flashing colors in "The National Guard" title, a waving flag, page curling for the "Back to the National Guard Home Page" icon, and flashing "hot spot" buttons. Instead of being impressed with the animation, all three evaluators could not wait to leave the site. Jakob Nielsen writes that:

Moving images have an overpowering effect on the human peripheral vision. This is a survival instinct from the time when it was of supreme importance to be aware of any saber-toothed tigers before they could sneak up on you. These days, tiger-avoidance is less of an issue, but anything that moves in your peripheral vision still dominates your awareness: it is very hard to, say, concentrate on reading text in the middle of the a page if there is a spinning logo up in the corner. Never include a permanently moving animation on a web page since it will make it very hard for your users to concentrate on reading the text. (http://www.useit.com/alertbox/9512.html)

See Nielsen's page of multimedia guidelines for web pages (http://www.useit.com/alertbox/9512.html) for more information.

Organization: The page described as the home page must be consistent across all pages in the site. It was assumed that the home page for the DefenseLINK site was located at http://www.dtic.mil/defenselink/, but the "Welcome address" located at http://www.dtic.mil/defenselink/welcome.html greets the user with the following sentence: "Welcome to DefenseLINK, the official World Wide Web Information Service of the Department of Defense. Please use this "Home Page" as your guide and starting point for access to a wide range of information about the Department." The user is then left wondering what the actual home page to the site is.

Information did not flow in any obvious order but seemed scattered. For example, the page began with a short welcome but then rather than stress sites within the DefenseLINK, the user is invited to leave the site to search other pages, e.g. the American Forces Information Service WWW site and BosniaLINK. Afterwards, other sites are described, the purpose of DefenseLINK is finally presented. On a home page, the information describing one's site should not be second to information maintained by other departments. One cause of this problem may be that there was too much information presented on the home page making it difficult for evaluators to orient themselves to the site.As a second example, consider the ordering of information on the page describing how to do business with the Department of Defense (http://www.dtic.mil/defenselink/other_info/business.html). Buying and selling are very closely related ideas, but on this page "selling information to the military" is presented at the top of the page but buying DoD surplus items is not presented until the bottom of the page.

The relationship between the Department of Defense and the four branches of the Armed Forces was not expressed clearly. Often, one branch of the Armed Forces would be emphasized over the others. For example, the page for "Frequently Asked Questions" (http://www.dtic.mil/ defenselink/faq/) presents links to Air Force and Navy frequently asked questions on their respective home pages but no link to the Army or Marines pages. Similarly, on the "Doing Business with the Department of Defense" page, a section is provided specifically for the Air Force but not for the other three branches. It was unclear why certain branches of the Armed Forces received special mention.

Finally, be careful in how text is presented as the use of a different font, centering of text, and making text bold conveys ideas to the user that new information will be presented. For example, one line of text written in a large font centered above a paragraph of text usually suggests to the user that the one-liner is a short title or description for information in the text. However, the "Top Story - Quadrennial Defense Review" looked like a section header, when in fact, it was a link for more information. It did not even have any relevance to the text below it. This is a violation of the match between the system and the real world.

Color: The addition of color may make the page more aesthetically pleasing, but in selecting colors, the designer should consider the conditions under which users may view each page (e.g. lighting conditions and number of colors on the monitor) and select color combinations which are distinguishable and more importantly, easy to read. On the National Guard home page, the use of red text on a black background was difficult to read and hurt the eyes.

Additionally, it was difficult to tell which links had been selected and which had not. The links on some pages updated when they had been selected (e.g. the DefenseLINK home page) but not on others (e.g. the National Guard home page). The inconsistency in updating the links made it difficult to determine whether a page had been visited or not, creating a violation of the match between the system and the real world.

Icons: When using icons, make sure that the graphic has the same action whenever it is presented. For example, the DefenseLINK icon has different navigational capabilities depending on where it is displayed. For example, when presented at the bottom of the page, the icon would take the user to the DefenseLINK home page, but on most headers, there was no navigation, except on the home page, in which selecting the icon in the header took the user to a page on Publications and the Pentagon (http://www.dtic.mil/defenselink/pubs/pentagon/). This lead one evaluator to distrust the DefenseLINK icon to take her back to the DefenseLINK home page.

Navigation: The user needs be told when he will be linked to a separate site because it is most often the case that a page not on the DefenseLINK server will not provide navigation back to the DefenseLINK site.

The presentation of the DefenseLINK icon at the bottom of many of the pages on the site was nice as the user was presented with an "escape" to the home page, but navigation skipped levels in a hierarchy returning the user to the home page for no obvious reason. Consider the hierarchy in Figure 1.

Figure 1. Sample Hierarchy

As Figure 1 shows, A is the home page, which gives access to pages B, C, and D. B1, B2, and B3 are accessible from B. It is important when examining the site to make sure that if one can access B1 from B, then one can also access B from B1. Note that if A does not give a link to B2, then B2 does not (and should not) give a link allowing one to move to A. Navigation can occur within a level of the hierarchy, e.g. between B to C and C to D, but should not jump between levels, e.g. B2 to D. Navigation through the DefenseLINK site required all users to return from A, regardless of the page. It was impossible to move from B3 back to B - rather the user needed to return to A and then navigate down again to B.

Recommendations

The following recommendations are presented as starting points for re-designing the site. More specific recommendations for some of the violations are presented in the next section.

• Color

Use easily discriminable colors.

Select aesthetic color combinations when using colored text against a colored background. Additionally, ensure that the text is readable.

• Animation: Constrain animation so that no graphic objects are continuously moving; this will capture the user's attention and make it difficult for him to concentrate on the information presented. See Nielsen's multimedia guidelines for instances in which animation is useful (http://www.useit.com/alertbox/9512.html).

• Organization:

Organize text on the pages in an order which will make sense to the user. Present information on related topics together.

Emphasize information which relates to the DefenseLINK site before advertising links to other sites.

Changing font will convey information to the user, e.g. a larger font presented above smaller text may be interpreted as a section header.

• Icons

Select icons which convey one idea to the user. Icons which convey more than one idea should not be allowed to stand alone, i.e. without text; a little bit of information presented as part of an icon may be useful.

Each icon should be consistent in the navigation they provide, e.g. one icon should not take the user to two different pages depending on its location within the current page.

• Navigation

Use the same link description to take the user to the same page.

Be consistent in navigation from a page. If one link on page B takes the user to page B1, then there should be a link on B1 to return the user to page B.

Heuristic Evaluation: Violations by page

Home Page: http://www.dtic.mil/defenselink/index.html

• Interesting presentation of navigational buttons.
• There is too much information on the first page - I feel overwhelmed. (Aesthetic and minimalist design)
• Another evaluator said that she did not feel oriented to the site from the home page due to the lack of organization. The page begins with a short welcome and then invites the user to search other pages not presented by the DefenseLINK site, e.g. the American Forces Information Service WWW site and BosniaLINK. After that, the purpose of DefenseLINK is finally presented. (Match between system and the real world)
• Part of the problem may be that there are redundant links to the different organizations (e.g. Army, Navy, and Marines) from the navigation buttons at the top of the page as well as links from departments under the Secretary of Defense. This could be presented on a subsequent page. (Aesthetic and minimalist design)
• It is unclear why the "Top Story" should be presented as a link right after a welcome to the home page. More information about the Department of Defense might be more appropriate. (Match between system and the real world)
• The "Top Story" heading is centered on the page and looks like a heading for the text underneath, but this is not the case. (Match between system and the real world)
• The "Top Story" sounds like it should be found under "News". (Match between system and the real world)
• It is a little confusing that William Cohen, who is the Secretary of Defense, has a separate (and different) link from the home page than the link for the Secretary of Defense. (Match between system and the real world)
• It is surprising that there is a second home page for this site, the "welcome" address located at http://www.dtic.mil/defenselink/welcome.html. The first sentence of the text reads as "Welcome to DefenseLINK, the official World Wide Web Information Service of the Department of Defense. Please use this "Home Page" as your guide and starting point for access to a wide range of information about the Department." (Match between system and the real world)
• The DefenseLINK icon presented at the bottom of many of the sub-sections is selectable at the bottom of the page. However, the same icon is presented at the top of the page as part of the header but has no navigational capabilities. As a result, one evaluator did not trust the DefenseLINK icon to take her back to the DefenseLINK home page. (Consistency and standards)
• It was also not clear to the evaluator that the DefenseLINK icon was the button which would return her to the home page as it was not labeled as such. (Match between system and the real world)

Secretary of Defense: http://www.dtic.mil/defenselink/osd/

• There is good consistency in the page header.
• This site links to other governmental sites, which may not take the user back to the DefenseLINK page. The user should be warned of this. For example, this page takes the user to the "Annual Report to the President and the Congress". At the bottom of the report, the user is presented with a navigation button to the "Executive Secretary" and not back to the DefenseLINK page. (Consistency and standards)
• The sentence: "The Secretary of Defense is the principal defense policy adviser to the President and is responsible for the formulation of general defense policy and policy related to all matters of direct concern to the DoD, and for the execution of approved policy" does not make sense. This may be due to the fact that the word "policy" was mentioned four times. The sentence should be rephrased. (Match between system and the real world)
• Examine how text is viewed across platforms and web browsers. One Mac evaluator saw the bullets on the page as a mixture of squares and circles and was not sure why different bullets were used and what information they conveyed. (Match between system and the real world)
• http://www.dtic.mil/execsec/adr_intro.html: The resolution for the "Executive Secretary" icon at the bottom of the page could be improved. (Aesthetic and minimalist design)

Design Idea: Information describing the Office of the Secretary of Defense (OSD) Organizations could be presented hierarchically (e.g. see http://www.nsf.gov/home/chart/start.htm) so that the relationships can be pictured.

Joint Chiefs of Staffs: http://www.dtic.mil/jcs/

• This page has a very different look and feel than the Department of Defense home page. For example, a purple side bar has been added, and the "DefenseLINK" header found on the home page is replaced with a "JCS Link". (Consistency and standards)
• The "DefenseLINK" icon which takes users back to the Defense home page uses a different icon than that presented on the site of the Secretary of Defense. (Consistency and standards)
• Items in the purple side bar menu are dependent on where the user is in the system - there is a different menu for "Organization" and "People". (Consistency and standards)
• It's nice that there's a sidebar menu, but why present information redundantly at the top of the page and along the sides? When the navigation bar at the top of the page is scrolled out of view, the information on the sidebar menu is out of sight also. (Aesthetic and minimalist design)
• The use of white text against relatively light backgrounds as presented in the navigation buttons makes reading the text difficult. (Aesthetic and minimalist design)
• Additionally, the combination of yellow text on a purple background is not the optimal combination of colors. (Aesthetic and minimalist design)
• The pages within this section are very consistent to each other but not consistent at all with the DefenseLINK site. (Consistency and standards)
• One evaluator especially liked the navigation buttons at the bottom of the page to the Air Force, Army, Navy, Marine Corps, and DefenseLINK pages as the user could most likely find the "originating" page.

Organization: http://www.dtic.mil/jcs/core/jcs_defn.html

• Selecting the link "Joint Chiefs" takes the user to http://www.dtic.mil/jcs/core/s_chiefs.html, in which the side bar menu has changed to the heading of "People". However, selecting the "people" option from the menu bar at the top of the page brings the user to a different page (i.e. http://www.dtic.mil/jcs/core/people.html). It is confusing to have the same text serve as header on a menu and for a navigation button but yet lead the user to two different places in the site. (Consistency and standards)
• The codes, J-1 - J - 8, may mean something to the Department of Defense but may mean nothing to the average user. (Match between system and the real world)

Design Idea: Emphasize the topics which are discussed rather than the topic's number.

• The organizational structure of each position in the Joint Chiefs is described fully in the "People" section of the site (http://www.dtic.mil/jcs/core/s_chiefs.html). Putting information on this page is redundant. (Aesthetic and minimalist design)
• Too much text is presented to the user at once, all in a block, which may overwhelm the user. (Aesthetic and minimalist design)
• There are two different links entitled "Joint Staff": one in the sidebar menu, and the second as part of the long body of text. The user is taken to two different pages. (Consistency and standards)

Design Idea: Provide information specific to the "Joint Chiefs" and "Joint Staff" on their own home page.

People: http://www.dtic.mil/jcs/core/people.html

• There is good internal navigation among the people pages. The page for the Chairman has very good navigation within the three pages.
• The paragraph on this page seems to be very similar to the one describing the Joint Chiefs of Staff on the "Organization" page. This redundancy caused one evaluator to be confused about the organization and hierarchy of the information presented. (Match between system and the real world)
• Selecting "Personnel Information" sounds like the user should receive employment information, not an organization. (Match between system and the real world)
• Some pages have links only at the top of the page (e.g. http://www.dtic.mil/jcs/core/s_chiefs.html, http://www.dtic.mil/jcs/core/j1.html), others only at the bottom (http://www.dtic.mil/jcs/chairman/), and others present links at both the top and the bottom of the page (http://www.af.mil/news/biographies/ralston_jw.html). (Consistency and standards)

Library and Research: http://www.dtic.mil/jcs/core/library.html

• The JCS "Search" is presented on the JCS home page - there is no need to present this information redundantly. (Aesthetic and minimalist design)
• Linking from "Library and Research" information to "Speeches" is nice, but somewhat confusing as the user does not realize he is being taken to the Chairman's home page - and leaving the "Library and Research" section. (Match between system and the real world)
• In fact the transition between the two sections is almost too smooth. One evaluator did not realize that the "Library and Research" menu had been replaced with a "People" menu until she tried to select research information. (Visibility of system status)
• What is "Goldwater-Nichols"? (Recognition rather than recall)
• http://www.dtic.mil/jcs/core/reports.html: text is not lined up with the bullets (Aesthetic and minimalist design)
• The problem with the sidebar menu is that the selections don't stay in place. For example, after selecting information for "Publications and Reports", the menu item resets itself, and to get to the bottom of the information, the user must scroll again. (Aesthetic and minimalist design)
• The "Education and Training" section has a completely new look and feel - the background has patterns, and a new header is used. (Consistency and standards)

Frequently Asked Questions: http://www.dtic.mil/jcs/core/faqs.html

Site Map: http://www.dtic.mil/jcs/core/site_map.html

• It is nice that there's a site map, but it should have the same look and feel as other pages in the JCS site. (Consistency and standards)
• It's hard to tell what text is selectable and what text is not. Black text, which usually suggests no associated links, is actually "hot", but so is the purple text. (Consistency and standards)
• Color is an effective cue for grouping information and calling attention to information, but take caution when using it. Purple text on different pages has indicated links that the user has already selected. However, on this page, the purple text indicates previously unselected links. (Consistency and standards)
• Additionally, the color does not change after a user has visited a site; thus, the user has no idea what pages have been visited and what pages have not. (Recognition rather than recall)

Search: http://www.dtic.mil/jcs/core/search.html

National Guard : http://www.dtic.mil/defenselink/guardlink/

• This page has a different look and feel from the DefenseLINK home page. (Consistency and standards)
• Animation can be nice, but it can also be annoying. Too much animation, in general, is distracting. The changing colors on "The National Guard" title is very patriotic but draws attention away from the information on the page. The flying "Back to the National Guard Home Page" icon makes the text difficult to read. The flashing "hot spot" buttons look like railroad warning signals flashing in unison. There is too much movement on this page. (Aesthetic and minimalist design)
• In fact, updating the animation on some computers can cause continuous updating of the status bar at the bottom of the web browser, making it difficult for users to view addresses of links in that same space when dragging the mouse over a potential link. (Visibility of system status)
• What is the difference between information in the "Hot Spot Nationwide" and "What's Hot" links? It doesn't seem like the two sites would present different information as both use the same icon. (Match between system and the real world)
• Color is nice, but the designer must consider users viewing this page under different conditions (e.g. low lighting, extremely bright lighting, low color displays). The use of red text on a black background is difficult to read and hurts the eyes. (Aesthetic and minimalist design)
• A section header can be presented in ways other than text, e.g. a larger font size. The use of underlining on the web has been associated with linked text, thus, underlining section headers makes users want to select them. (Consistency and standards)
• The links on the page never update to indicate to the user what sites have been visited and what sites have not. (Recognition rather than recall)
• There is no navigation back to the DefenseLINK home page. (Consistency and standards)
• There was no real content on this page. This page does not seem like it would be the home page of the National Guard as it does not give users a good feel about who the National Guard is from the page. The page presents users with a quote about the National Guard and then presents usage information. (Match between system and the real world)
• One evaluator felt that the text for the side bar menu was jumping out at her. (Aesthetic and minimalist design)

National Guard by State: http://www.dtic.mil/defenselink/guardlink/

• Information for many of the states consists only of a contact address displayed on an otherwise empty page. (Aesthetic and minimalist design)

Design Idea: Put all contacts on a single page. Present users with a listing of the 50 states (formatted so that all states fit on one page) - or even using the current map. When a state is selected, take the user to the contact address. Since some states have their own home page for their National Guard, provide links to these pages next to the addresses. Additionally, single spacing the addresses would be a more efficient use of screen real estate.

• The fact that the frames on this site will surround the information from other states' home pages for their National Guards needs to be taken into consideration when selecting colors. For example, the contrast between the black frames and the white background of the Illinois National Guard page is too much. (Aesthetic and minimalist design)
• The presentation of an "E-Mail" icon over the Warning statement on the Illinois National Guard page is annoying. (Aesthetic and minimalist design)

Around the World: http://www.dtic.mil/defenselink/guardlink/

National Guard Bureau: http://www.dtic.mil/defenselink/guardlink/

In the Communities: http://www.dtic.mil/defenselink/guardlink/

• Information on "Emergency Support to our Communities" is presented using what appears to be old slides from a previous talk. This is fine, but the text is too small and unreadable. (Aesthetic and minimalist design)
• There was not enough information on this page to justify its inclusion as a page by itself. (Aesthetic and minimalist design)
• There is extra space on the page to present the three "youth program" links - rather than have the user link to another page of links. (Aesthetic and minimalist design)
• Descriptions on what information is provided by the two topics would be helpful. (Match between system and the real world)

Information and Research Materials: http://www.dtic.mil/defenselink/guardlink/

• The information presented on this page sounds very news oriented, which is not obvious from the title of the page. (Match between system and the real world)

Design Idea: Iincorporate the word "news" somewhere in the name for the section.

Army : http://www.dtic.mil/armylink/

• This page had a nice look, although it was new, i.e. different from the design used in creating the DefenseLINK pages. (Consistency and standards)
• What is the difference between ArmyLINK and the Army's actual home page? (Match between system and the real world)
• The links on the page never update to indicate to the user what sites have been visited and what sites have not. (Recognition rather than recall)
• There is no obvious navigation from this page back to the DefenseLINK home page. The user has to scroll down the page and search the text for the link. (Consistency and standards)
• In designing a page, a certain image to be projected is desired. In this case, whether desirable or not, one evaluator felt the page projected the image of "entertainment" given the links to a Soldiers magazine, Army photos, broadcast services, and the field band. (Match between system and the real world)
• The same links were presented with Army navigation buttons at the top of the page and the text links in the middle of the page. (Aesthetic and minimalist design) However, it is important to note that there was good information provided about the services at each site when presenting the links with text.
• The text presented with "Questions" was "Questions we're frequently asked". A more common phrase is "frequently asked questions". (Match between system and the real world)

News: http://www.dtic.mil/armylink/news/

• A "Search" option should not be presented as part of "Recent News". (Match between system and the real world)
• The News Archive should not present information which is not available. It's good that the information is lowlighted, so that the user can tell that it's not selectable, but why have it on the page at all? (Aesthetic and minimalist design)
• Navigation buttons are different from that on the ArmyLINK home page. (Consistency and standards)
• The navigation buttons at the top of the page served as good orientation cues but icons, similar to that on the first page, would be more consistent with the design of the site. (Consistency and standards)

Soldiers Magazine: http://www.redstone.army.mil/soldiers/home.html [Now at http://www.army.mil/soldiers/]

• This page has a very different look and feel not only from the DefenseLINK site, but from the ArmyLINK page as well. (Consistency and standards)
• If the user is being taken to a site presented by a different department, tell the user that he will be leaving the DefenseLINK site. Otherwise, the user may not know he is no longer within the DefenseLINK site. (Match between system and the real world)

Questions: http://www.dtic.mil/armylink/faq/

Photos: http://www.dtic.mil/armylink/photos/

• The term "Photos" is vague in general, and coming to this site, one may expect general Army pictures - e.g. tanks and soldiers. (Match between system and the real world)
• The "Army Leadership" and "General" photos sections do not contain enough information requiring them to be linked at a lower part on the page. Rather this information can be presented in the "Recent Photos" section. (Aesthetic and minimalist design)
• The pictures need to be explained, i.e. why were they taken, why is this person important (e.g. Major General John E. Longhouser has no description)? (Match between system and the real world)
• The use of identical icons (with only the text changed) made remembering what each page looked like difficult, e.g. the same icon was used for navigation back to the "ArmyLINK" home page as was used to navigate back to the "Photos" page. (Recognition rather than recall)

Army Broadcast Service: http://www.dtic.mil/armylink/abs/

• This page has a different look and feel from the other pages on the ArmyLINK site. (Consistency and standards)
• The background is too bright and the text is very distracting and competes with the actual text on the page. (Aesthetic and minimalist design)
• The use of icons in the text takes up space and prevents more content information from being presented to the user higher on the page. (Aesthetic and minimalist design)

Army Field Band: http://www.mdw.army.mil/fband/usafb.htm

• If the user is being taken to a site presented by a different department, tell the user that he will be leaving the DefenseLINK site. Otherwise, the user may not know he is no longer within the DefenseLINK site. (Match between system and the real world)

Navy : http://www.navy.mil/: [not located on the DTIC server]

Air Force : http://www.af.mil/: [not located on the DTIC server]

Marines : http://www.usmc.mil/: [not located on the DTIC server]

Other Organizations: http://www.dtic.mil/defenselink/pubs/almanac/

• There is no navigation back to the "Other Organizations" page from these subpages - rather, navigation returns the user to the DefenseLINK home page. (Consistency and standards)
• Good consistency within the site - except for the Coast Guard home page. Again, if the user is leaving the site to visit a site maintained by another department, let the user know. (Consistency and standards)

Coast Guard (in time of war): http://www.uscg.mil/

Defense Agencies: http://www.dtic.mil/defenselink/pubs/almanac/def_agencies.html

• There needs to be some explanations as to what the different defense agencies are and what they do. (Match between system and the real world)

DoD Field Activities: http://www.dtic.mil/defenselink/pubs/almanac/def_field_acts.html

• What are field activities? (Match between system and the real world)

Inspector General of the Department of Defense: http://www.dtic.mil/defenselink/ig/

• The font size for the title of the page is smaller than that used on other pages.

Unified Combatant Commands: http://www.dtic.mil/defenselink/unified.html

Other Related Sites of Interest: http://www.dtic.mil/defenselink/pubs/almanac/def_related_orgs.html

Other Information: http://www.dtic.mil/defenselink/other_info/index.html

• In the section describing how to do business with the Department of Defense, the Air Force is listed separately as its own point. It was not clear how the Air Force was different from the other Armed Forces, thereby requiring it to have its own information. (Match between system and the real world)
• http://www.dtic.mil/defenselink/other_info/business.html: This page was not organized in such a way that allowed users to conceive each idea properly. For example, buying and selling are very closely related ideas, but on the page, "selling information to the military" is presented at the top of the page but buying DoD surplus items is not presented until the bottom of the page. (Match between system and the real world)
• http://www.dtic.mil/defenselink/other_info/business.html: It was also unclear how the Office of the Secretary of Defense played a role in doing business with the Department of Defense. (Match between system and the real world)
• The presentation of career opportunities in the military is unnecessary. If someone is interested in being recruited, he would have searched the home pages of the branch of the Armed Forces he were interested in, rather than trying to find information on careers through the DefenseLINK page. (Aesthetic and minimalist design)

News : http://www.dtic.mil/defenselink/news/

• This information was not organized well due to the fact that too much content (i.e. lists and lists of articles) was presented to the user at once. (Aesthetic and minimalist design)
• Some of the information on the page did not seem like news, e.g. the DoD Fact Sheet or selected speeches. (Match between system and the real world)
• There was good navigation from the subpages back to the News page.
• However, the icon used to return the user to the "News" home page was the same icon used on the "News" page to return the user to the "DefenseLINK" home page. The same icon should be used only when taking the user to the same page - navigation to two different pages should be presented via two different icons. (Consistency and standards)
• The "overview" icon is much smaller than the "News" icon and appears out of place. (Consistency and standards; Aesthetic and minimalist design)

Publications : http://www.dtic.mil/defenselink/pubs/

Questions : http://www.dtic.mil/defenselink/faq/

• "About DefenseLINK" should not be presented as part of "Questions". The purpose of the site should be stressed and presented as a link off of the main page. (Match between system and the real world)
• In the "Related Frequently Asked Questions" section, the links presented are to the Air Force and Navy - why not to all the four branches of the Armed Forces? (Match between system and the real world)
• The section on "DoD Frequently Asked Questions" (http://www.dtic.mil/defenselink/faq/pis/index.html) was more of a fact sheet than a list of frequently asked questions. (Match between system and the real world)

Search : http://www.dtic.mil/defenselink/locator/

• The search field should be presented at a more prominent location on the page than at the bottom of the "Search" page. (Match between system and the real world)