The effectiveness of motion in flight simulators used to train and certify pilots is examined. Two groups of pilots were put through two similar training programs: one was a traditional program in which a full flight simulator (FFS) - including motion - was used for all training and certification, and the other program in which a FFS was used only at the final stage to measure pilot skill. This second group (the experimental group) received training in a simulator featuring all of the FFS features (including visual simulation) except motion. Training effectiveness was measured by using the simulator computer to determine the error in pilot control for six flight maneuvers. A flight instructor rating sheet was also filled out by an independent observer pilot.
Results show no significant differences between the training methods for four of the six maneuvers. In one maneuver, the angle of bank portion of the steep turn maneuver, the control group did perform significantly better than the experimental group. In the Visual approach maneuver, however, the experimental group performed better than the control group.
Flight simulators have reached a high degree of realism in the presentation of visual, audio, tactile and motion cues. The most realistic simulators - full flight simulators (FFSs) - include out-the-window vision systems and motion platforms. Because of the proven training effectiveness of these simulators, aviation authorities have allowed the training and certification of pilots to be performed in these devices rather than the actual aircraft. The advantages to air carriers are lower cost when compared to aircraft training, and the ability to simulate maneuvers which, for safety reasons, would not be possible in the actual aircraft. The cost savings derive not only from the lower hourly cost of the simulator vs. the aircraft, but also from the increased training efficiency (it is possible to go directly to the desired location in space and practice the desired maneuver over and over without concern for the logistical constraints associated with real aircraft).
As the technical capability of simulators has increased over the years, the FAA has allowed and even encouraged their use in pilot training. In 1980, the FAA Advanced Simulator Plan (ASP) allowed the use of simulators for the final stages of training and checking. The ASP allowed the airline industry to further expand the use of simulators in training (Boothe, 1989). The ASP contained standards for three levels of simulators (Phase I through III), which when added to the visual and non-visual simulator levels that previously existed, resulted in five levels of technical sophistication for flight simulators with Phase III being the highest, most realistic level. The intent of this structure was to allow the airlines to use the lower-level devices for lower-level training, preserving time in the more advanced devices for the most advanced training. From the regulatory point of view, the benefits of the program have been an elimination of training accidents and a much improved training environment. The benefit to the airlines has been the lower training cost of simulators when compared to the use of aircraft. However, the actual value of motion to training effectiveness has been the subject of many studies including Koonce (1974); see Waag (1981) for a review.
In 1993, Atlantis Aerospace Corporation performed a detailed analysis of the possible uses of various levels of simulators and trainers in a transition training program. As indicated in the Atlantis study, the objective of the demonstration was to determine whether pilot training costs could be reduced while maintaining the integrity and quality of the training program. Intrinsic to this purpose is the requirement not to compromise training effectiveness or certification standards and not adversely affect aviation safety. There can be more efficient flight simulator use by doing only necessary tasks in the simulator and doing other tasks in less costly devices. The key is to assign each task or event to the device which provides the necessary cues and environment for that task, but not to train in a more sophisticated device than necessary. It must be recognized that it is not sufficient for the pilot to merely accomplish the task. He or she must accomplish the task with the same control strategy and similar control inputs to those that would be used in the respective aircraft. The objective is for the flight training device (FTD) or simulator to provide the same pilot stimulus for the task that the aircraft would provide.
The need for simulator motion can be based on so called "disturbance" inputs which derive from unusual events or disturbances of the flight path as opposed to pilot-induced deviation from the flight path. Events which are known to be independent of motion stimulus can be trained, and for that matter checked, in an FTD. Training for some of the events may benefit from a visual system. In this demonstration, the visual system was used throughout the flight training portion of the program. There was no intent to identify which tasks would benefit by visual cues and which would not.
Current FTD and simulator task assignment is based primarily on realism. The issue is often how realistically does the device represent the total environment, not just how realistically it represents the given task or event. Realism is certainly an acceptable criterion for success, but it may lead to over-specification of the needed training medium. However, since there is no data base except experience that indicates what cues are required for given tasks, there is as yet, no other criteria. The demonstration described in this paper does not attempt to relate pilot response to cues, per se, but is shows that many tasks can be off loaded from the simulator to a less complex device. Hopefully, the results will stimulate further study into cue analysis.
A total of forty-eight pilots, twenty-four volunteer pilots from Embry-Riddle Aeronautical University and twenty-four volunteer furloughed pilots from Delta Air Lines, participated. In each case, half the pilots were in a control group and the other half in a test group. The crew training concept was used and all pilots in the captain position possessed, or were eligible for, an Airline Transport Pilot (ATP) certificate.
The mean flight experience of the Embry-Riddle pilots was 1300 hours with a range of 800 to 10,000 hours. Their mean age was 26 with a range of 22 to 43 years of age. Each pilot held at least a commercial certificate with a multi-engine and instrument rating. The Embry-Riddle pilots were paired so as to avoid having two low-experience pilots together as a crew, neither of which might possess the experience requirements for an air transport rating. After pairing, the subjects were assigned randomly to the control or test groups.
The Delta subjects were furloughed pilots of varying experience all of whom had previously served in a line capacity. They too were paired and then randomly assigned to the control or test groups.
All objective flight performance data were collected using the data collection capabilities of the simulator. All subjects completed the normal ten day MD-88 ground school which was an integral part of the initial training program. The ground training program was unaltered for the demonstration program and utilized the Level 6 FTD, but did not use the visual system.
The pilots in the control groups received flight training in accordance with Delta Air Lines standard all simulator (Level D) initial training program. The pilots in the test groups were trained in a program which used the visual FTD in lieu of the simulator for the first nine training and certification days in the program. Some tasks which require a simulator were learned and practiced in the FTD, but were then repeated in the simulator in the latter part of training program.
All subjects completed the check ride in the MD-88 flight simulator and were evaluated using standard performance criteria required by the FAA-approved Delta Airlines training program. The check rides were administered by an aircrew program designee (APD) who did not know whether the pilot was trained in the all simulator program or in the combined FTD and simulator program. Pass or fail was determined solely by the APD. Any pilot trainee needing more than the allotted time of the training program was given one additional day of training and a second check ride.
A second observer from the Embry-Riddle staff was present during the checking. His sole function was data collection. The observer, a senior check pilot, completed a detailed special performance evaluation form during each check ride. The analysis of the data from the special performance evaluation complemented objective data collected using the simulator computer system. The second observer also managed the collection of the objective data. This involved initializing the computer for data collection before each maneuver.
The data from the Embry-Riddle independent observer consisted of rating sheets with simple dichotomous scores. Pilot performance was evaluated only as to whether or not a procedure, checklist item, or performance item was successfully completed within the parameters of the ATP practical test standards, which are identical to the performance required on a rating ride. Rating sheets were used to assess eight flight maneuvers: a) precision approaches, b) visual approaches, c) approach to stalls, d) non-precision approaches, e) normal takeoffs, f) rejected takeoffs, g) V1 cuts and, h) steep turns.
The frequency of missed items was too small to analyze each of the eight maneuvers independently. Consequently, these data were combined across the maneuvers to develop composite ratings. Non-parametric tests were performed on these data between training conditions for the Embry-Riddle and Delta pilots separately, and combined as larger test and control groups. No differences between control and experimental training conditions were found for Embry-Riddle pilots (X2=.14, ns), Delta pilots (X2=0, ns), or the pilot groups combined (X2=.08, ns).
Simulator-generated data was used in this study as a means to objectively assess and quantify performance while mitigating evaluator biases. The maneuvers and performance parameters were selected based on meeting three criteria: a) the maneuver was a required task in the checkride, b) performance could be assessed using captured relevant parameters, and c) a clear standard of target performance could be developed and used for comparison. All required tasks in the checkride could not be assessed since a clear reference point could not be obtained or because of difficulty in identifying the initialization or termination point. Therefore, only maneuvers and parameters that could be precisely standardized across all checkrides were used. No attempt was made to sample all checkride maneuvers or their components. The six maneuvers sampled and their associated performance parameters are described below.
The important performance criteria for this study, however, were the simulator captured data. Six sampled maneuvers were analyzed to determine if there were significant differences among pilots in critical flight performance measures. In each case, Group 1 represents the control group (full flight simulator throughout training) and Group 2 represents the experimental group (combined FTD and simulator). The reader is cautioned that complete performance data is not available in many instances as a result of simulator problems in capturing and transferring data. All analyses are conducted assuming unequal sample variances using the probabilities for two-tailed tests.
The data reported below have been organized by maneuver. In all cases where there are no significant differences between Embry-Riddle and Delta pilots within training conditions (i.e. control and experimental), the data have been collapsed to increase sample size. Results of these comparisons are not presented here. root mean square (RMS) values were obtained by squaring each deviation value, summing the squares, dividing by the number of samples, and taking the square root of the result for each individual's performance. No distinction was made between first officers and captains.
Steep Turns: The RMS of the deviations from 45 degrees angle of bank (AOB) from initialization (20 degree heading change from initial direction) to completion (within 20 degrees of final heading). Altitude and airspeed RMS deviations were acquired for the entire turn. Target airspeed and altitude is based on nominal target values at the initiation of the maneuver.
Rejected takeoff: Root mean square of heading deviation from runway heading and the total distance to stop in feet. Data collection is initialized at loss of one engine (N2 reverses) and completion is at zero ground speed.
Engine failure at V1: Root mean square of heading deviation from engine failure at V1 to restart.
ILS approach: Root mean square of glide slope and localizer deviations in feet from five miles inbound to touchdown.
Approach to stall: Mean number of feet of altitude lost between stall onset (yoke shaker flag) to recovery (increase in altitude after stall including any secondary stalls). Subsequent secondary stalls were treated as a continuation of the original stall.
Visual approach: Distance from runway centerline at the point of touchdown.
|
Only six maneuvers were evaluated in this study. However, maneuvers were selected which could be objectively measured and evaluated. Other maneuvers and training tasks were not evaluated.
In four of the six maneuvers evaluated no significant differences were found between the control and experimental groups. In one of the maneuvers, Steep Turns, the control group outperformed the experimental group. In the remaining maneuver, Visual Approach, the experimental group was slightly better than the control group. Data gathered by the second flight instructor/observer also showed no significant difference between the groups.
The results of this study lend support to the concept of transferring some of the flight training tasks to devices which are less complex and less costly than full flight simulators. Other tasks, for example steep turns, may benefit from the added realism of motion in simulation.
Boothe, E.M., Cook, E.D. (1989) FAA perspective in increasing benefits of flight simulation. Proceedings 1989 spring convention - flight simulation: Assessing the benefits and economics. London: The Royal Aeronautical Society.
Federal Aviation Administration (June, 1980) Federal Aviation Regulation Part 121, Appendix H - Advance Simulation Plan.
Koonce, J.M. (1974) Effects of ground-based aircraft simulator motion conditions upon prediction of pilot proficiency. Savoy, Ill.: University of Illinois, Aviation Research Laboratory, TR ARL-74-5/AFOSR-74-3 (Ph.D. dissertation, University of Illinois at Urbana-Champaign).
Waag, W.L. (1981) Training effectiveness of visual and motion simulation. Williams AFB, AZ: AFHRL-TR-79-72.