Pacific Science & Engineering Group, Inc.

6310 Greenwich Drive, Suite 200, San Diego, CA 92122

Naval Postgraduate School, C3 Academic Group

589 Dyer Road, Monterey, CA 93943

Naval Command, Control and Ocean Surveillance Center, RDT&E Division

Code 44210, 53560 Hull Street, San Diego, CA 92152

Baseline tests in representative littoral scenarios suggested that experienced decision makers were not particularly well served by current systems [Hutchins et al., 1996a]. Teams exhibited periodic losses of situation awareness, often linked with limitations in human memory, problems of shared attention, and decision biases. Problems associated with short term memory limitations included:

Observed problems that were related to decision biases included:

A prototype DSS was developed with the objectives of: (1) minimizing the mismatches between cognitive processes and the information available in the CIC to facilitate the decision making; (2) mitigating the shortcomings of current CIC displays which impose high information processing demands and exceed the limitations of human memory; and (3) transferring the data in the current CIC from numeric to graphical presentations wherever appropriate to facilitate the interpretation of spatial data. It was determined that the resultant system should not filter or extensively process data in the manner of a decision aid because this could reduce the effectiveness of the decision makers.

The prototype DSS and its underlying design principles have been described in detail elsewhere [Hutchins et al., 1996b]. Basically, the DSS consists of several separate display modules arranged as a tiled composite. The modules in the upper-half of the display present a variety of kinematic and status data intended to enable decision makers to determine quickly what a particular track is doing and whether any action is required. Another area summarizes the available data in a form intended to assist decision makers in evaluating a track's status. Presumably, this module would encourage decision makers to consider everything that's known about a track when making an assessment of "threat" vs. "non-threat". The lower portion of the DSS presents an alphanumeric summary of the highest priority tracks. This is intended to support decision makers in allocating their shared attention across concurrent tracks.

An experiment was conducted to examine the impact of the DSS on team performance. Eight expert Navy tactical decision making teams (with emphasis on the CO and TAO) used either an analogue of a current tactical data system alone (the No DSS condition) or in conjunction with the prototype DSS (the DSS condition). Three of the teams were intact CO-TAO teams from ships. The others were ad hoc teams of highly experienced officers currently serving on battle group or type command staffs or as tactics instructors. All had had extensive shipboard experience involving tactical decision making in situations similar to those examined in this study.

A within-subject factorial design was employed across four test scenarios such that each team performed two scenarios with the DSS and two scenarios without it. The order of the scenarios and DSS conditions were counterbalanced using a Latin Square. Criterion-referenced training with the baseline display system and with the DSS was provided, and two practice scenarios were run prior to beginning the test session. In addition to collecting objective data on tactical actions, display usage, and voice communications, subjective assessments (via questionnaires and a structured interview) were solicited from each CO and TAO at the conclusion of the test session.

Results indicated no evidence of a practice effect over the four-scenario test session and no consistent differences between the scenarios themselves. Substantial differences were observed, however, between teams - notably in their subjective workload assessments and in their communication rates and patterns.

The results of primary interest concerned the overall effects of the DSS and its component display modules on the teams' decision making. Two of the measures of decision performance are discussed here: recognition of critical contacts and compliance with the rules of engagement.

During the scenario runs, the experimenter probed the CO-TAO team at prespecified times to identify the tracks that were considered to be of the greatest tactical interest at that time (i.e., highest priority). The teams' responses were contrasted with those of an independent group of subject matter experts. As shown in Figure 1, more of the critical contacts were identified at the early and middle probes when the DSS was available. At the late probe, however, performance was comparable. This effect suggests that the advantage of DSS may disappear late in the scenario since the critical tracks become more obvious by that time, even without the DSS. Another interesting effect is the tendency to report fewer tracks of interest later in the scenario when using the DSS. This could actually reflect a greater situation awareness, whereby decision makers used the DSS to evaluate a wide number of contacts and then chose to concentrate on (and report) only the unresolved, high priority contacts. Nevertheless, earlier recognition of critical tracks is desirable, since it essentially "buys time" for decision makers to consider a broader array of response options and to permit more effective coordination of their responses.

Figure 1. Percent of critical contacts reported as tracks of interest.

Using the rules of engagement as a benchmark for decision performance in the scenarios, a group of subject matter experts assessed whether the CO-TAO teams warned and/or illuminated threat tracks at specified times and took appropriate defensive actions. A modified form of the AAW Team Performance Index (ATPI) [Dwyer, 1992] was used for scoring tactical performance, and these data are summarized in Figure 2. In scenarios when the DSS was available, CO-TAO teams were significantly more likely to take defensive actions in a timely manner against imminent threats. This indicates that the DSS promoted an earlier recognition of the emerging risks of the tactical situation. By contrast, no difference was observed in the number of tracks that were warned or illuminated when the DSS was available. However, several subject matter experts contended that warnings and illuminations may not be valid performance indices since they are provocative tactical actions that commanders may consider to be inappropriate against certain tracks in a littoral situation. Indeed, if the DSS promoted greater understanding of the intent of ambiguous tracks, decision makers might be less willing to issue warnings or to illuminate any but the most threatening tracks. This greater understanding of the tactical situation would of course serve to depress the modified ATPI scores for warnings and illuminations.

Figure 2. Team performance of tactical actions

required by the rules of engagement.

These findings indicate that the DSS contributed to improved decision performance by the CO-TAO teams. In particular, it appears that with the DSS teams were more aware of the tactical situation, which in turn promoted an earlier focus on critical contacts and a greater tendency to take appropriate defensive actions.

It is important to determine whether the DSS altered the process by which decisions are made about which actions to take and when to take them. On one hand, the decision process may be external, triggered by the information flow among the team's communications. On the other hand, it may be internal, whereby decision makers use the information acquired from the DSS and the team to recognize patterns of tactical significance. To explore the locus of the DSS effect, voice communications among the team were examined. Analyses of the rate, pattern, and content of voice communications are reported here.

It was hypothesized that when using the DSS, teams would have less need to exchange data verbally and would, thus, communicate less often. To test this, all voice communications that requested or provided information were tabulated for each of the 32 test runs. Since the length of the scripted test scenarios differed, the total number of voice communications observed was divided by the scenario duration to give a communications rate.

Figure 3 shows the mean rate of communications originating with the CO, TAO, other members of their team, and others external to the ship's combat center (e.g., the battle group commander, the bridge). A general decrease in communications rate with the DSS may be seen. This decrease remains fairly consistent regardless of who originated the communication. In fact, the pattern of communications was unaffected by the presence of the DSS. About 40% of the communications occurred between the CO and TAO, and another 35% occurred between the TAO and the team. Each of the remaining links accounts for about 5% or less of the total communications.

A reduction in communications rate was observed when the DSS was available. It is interesting to note, however, that this reduction was uniform across positions. This suggests that the DSS supported the entire team by providing basic data about tracks, thereby reducing their need to request or provide such data verbally.

While the pattern of communications was not found to be affected by the presence of the DSS, the content of the teams' communications may be altered by the DSS. That is, without the DSS, teams might need to spend more time exchanging basic track data while those with the DSS might spend the bulk of their time assessing track intent or evaluating alternate courses of action. To explore this, voice communications were coded by their message content according to the following scheme:

• Information - exchange of sensor-based data;
• Status - exchange of procedure-based data;
• Clarification - communication to elucidate, interpret, or correct other communications;
• Correlation - association of two or more data;
• Assessment - discussion of expected track behavior, likely intent, or future actions;
• Orders - commands to perform an action.

Figure 4 shows the overall average proportion of communications observed for each of these content categories. The largest proportion involved Information communications, in which sensor-based data were exchanged. This, of course, is not surprising since these data effectively drive the decision processes. The rate of these communications, however, was found to be lower when the DSS was available. Since the DSS provides much of these data, there was less need to exchange it verbally among the team. Similarly, fewer Correlation communications were observed when using the DSS. Although decision makers were less likely to ask about or report correlation data with the DSS (since much of it is displayed automatically), they were somewhat more likely to talk about correlations in the data that they observed on the DSS. No differences between DSS and No DSS runs were observed in the other communications content categories.

Figure 4. Mean proportion of communications by content category.

Overall, about 20% of the communications were for clarification purposes, reflecting uncertainty about track location, kinematics, identification, status, or priority. When the teams had the DSS available, fewer communications were aimed at clarifying the tactical situation, particularly track kinematics, identification, and priority - each of which are directly aided by the DSS. On the other hand, with the DSS, decision makers tended to spend more time clarifying ambiguous communications and checking on the status of actions. While this result may seem counterintuitive, it reveals a greater situational awareness where ambiguous, incomplete, or erroneous communications are more likely to be caught and corrected when the DSS was available.

The tracks to which the teams' communications referred were also examined under the DSS and the No DSS conditions. The hypothesis was that the DSS would enable teams to focus on the critical contacts more quickly, resulting in a greater proportion of their communications about those tracks. The average proportion of communications about the critical contacts was slightly greater (but not significant) when using the DSS. It is not particularly surprising that these teams concentrated the bulk of their communications on the critical contacts regardless of whether or not they were using the DSS. After all, these were highly experienced tactical decision makers who are accustomed to functioning effectively with their current (non-DSS) systems. Thus, greater effects might be obtained with less experienced decision makers.

In reviewing these findings, it appears that when the DSS was available, teams communicated less often but more efficiently, addressing key issues in support of their tactical decisions. Yet, who they communicated with and which tracks they talked about remained pretty much the same.

The absence of strong differences in team communications associated with the DSS suggests that the key processes are largely internal. The DSS presents data in a form that promotes feature matching by experienced decision makers. To the extent that the DSS is a shared display, it also serves as a mediator for collaboration among the CO, TAO, and their team - altering the amount and form of their communications.

It should also be noted that these data were collected during a very brief exposure to the DSS. It seems reasonable to expect that the way in which the DSS is used, and the resultant team communications, would change as greater experience was obtained. A study is currently underway to examine how extended experience with the DSS affects decision performance and team communications.

These findings reveal several interesting characteristics of tactical decision makers and suggest features of a DSS that can best support them. The communications data indicated that these experienced tactical decision makers knew which data they needed about which tracks, who to talk with to get those data, what actions were appropriate to take, and even which tracks were the ones of greatest concern. The major benefits provided by the DSS were the ease with which it enabled decision makers to access the data, to visualize the integral relations among data, and to manage complex response actions. These benefits enabled the CO-TAO team to concentrate more of their efforts on (higher level) tactical decision processes since they did not need to spend as much time on (lower level) information gathering and correlation. The primary result of this is the ability to attend to more critical contacts in greater depth, as indicated by the decision performance data.

The design principles associated with these information processing benefits are illustrated in various DSS features. These include easy access to important track data in a readable format, visualization of track position relative to weapons and ROE thresholds, and interactive reminders of key actions that should be considered and when they should be taken. In general, these features suggest that decision support systems can provide substantial benefits when they reduce the processing involved in gathering and integrating data from diverse sources and in managing the status of concurrent, evolving actions on many tracks.

The authors gratefully acknowledge the many contributions of Steve Francis, Brent Hardy, Pat Kelly, C. C. Johnson, Ron Moore, Connie O'Leary, Mike Quinn, and Will Rogers in data collection, interpretation, DSS development, and technical expertise.