CSRC Homepage
 
 CSRC Site Map

 CSD Publications:
   - Draft Publications
   - Special Publications
   - FIPS Pubs
   - ITL Security Bulletins
   - NIST IRs

 CSD Focus Areas:
   - Cryptographic Standards
       & Application
   - Security Testing
   - Security Research /
       Emerging Technologies
   - Security Management
       & Guidance

 General Information:
   - Site Map
   - List of Acronyms
   - Archived Projects
        & Conferences
   - Virus Information
   - ICAT Alerts

The basic principles of results-based training for information technology (IT) security were discussed in our April 1998 bulletin, Training Requirements for Information Technology Security: An Introduction to Results-Based Learning. This learning approach trains staff members according to their roles and responsibilities within their organizations and recommends that organizations evaluate the results of the training. Both this bulletin and the April bulletin were excerpted from NIST Special Publication 800-16, Information Technology Security Training Requirements: A Role- and Performance-Based Model. The training requirements were developed by the Federal Computer Security Program Managers Forum and by the Federal Information Systems Security Educators' Association (FISSEA). The complete publication is available in paper copy from the Government Printing Office and in electronic format from NIST's Web pages:

http://csrc.nist.gov/publications/nistpubs

The Role- and Performance-Based Model

Results-based training for information technology (IT) security focuses on the job functions that individuals perform, and on their specific roles and responsibilities within their organizations. The results-based approach to learning is based on the premise that individuals have unique backgrounds and different ways of learning. Another consideration is that individuals may have more than one role in the organization, and they may need information technology security training that is tailored for the specific responsibilities of each role.

The role- and performance-based model presented in NIST Special Publication 800-16 provides a systematic and integrated framework for identifying training needs throughout the organization and for ensuring that individuals receive the training that they need. The framework relates job function to the IT security knowledge that is required to carry out a specific job. Managers can then identify needed training for their staff members, understand the consequences of not providing adequate training, and plan and schedule training according to organizational priorities.

In the model, learning is represented as a continuum that starts with awareness, continues with training, and evolves into education. Awareness about IT security is the point-of-entry into the learning process for all employees. The next step is training, which starts with instruction on security basics and literacy, and then carries through with instruction in a wide range of security-related skills needed by employees for their specific roles and responsibilities. The capstone of the learning continuum is education, which creates the expertise needed by the organization's IT security specialists and professionals.

The model is very useful in helping managers meet their responsibilities for training staff members. In addition, the model helps course developers identify the learning outcomes expected for individuals with various roles and responsibilities. As a result, IT security course material can be developed in basic modules and then appropriately customized to meet the needs of individual staff members.

Evaluating the Effectiveness of Training

Training based on organizational and staff members' needs is a good investment for the organization. But to receive the benefits, the organization must ensure that the contents of the training are current and appropriate for the audience, and that the training is delivered in an effective manner.

As part of their regular processes for the wise management of resources, organizations should evaluate the scope of their IT security training needs and the effectiveness of the training provided. Evaluations enable decision-makers to allocate their training resources sensibly and to derive the greatest return on their training investments. When training resources are well-managed, the organization benefits both from improved IT security and from stronger management support for the IT security activities.

The evaluation process starts with identifying what can be measured. Some aspects of training for which evaluation data can be collected include the following:

• The learner's satisfaction with the training - the extent to which the conditions were right for learning.
   
  
• Learning effectiveness - what an individual has learned from a specific course or training event.
   
  
• Teaching effectiveness - the pattern of learner outcomes following a specific course or training event.
   
  
• Program effectiveness - the value of the specific class or training event, compared to other options in the context of an organization's overall IT security training program.

The time and resources spent in conducting training evaluations can be beneficial to employees, managers, and trainers. Employees benefit by being able to assess their own post-training job performance. Managers can use the evaluations to assess the subsequent on-the-job performance of staff members. In addition, managers can use the evaluations to make decisions about how to allocate limited resources among the various IT security activities in the continuum from awareness, security literacy, and training to education. Another benefit is that trainers can use trend data acquired from evaluations to improve both their teaching and student learning.

Developing an Evaluation Plan

It is difficult to get good information for evaluating learner satisfaction and the effectiveness of training programs. Written learning objectives, stated in an observable, measurable way, are needed to evaluate student learning. To evaluate teaching, it is necessary to collect trend and other related data. Mission-related goals must be identified and related to learning objectives in order to evaluate returns on investment.

Before initiating evaluations, organizations should develop plans for gathering the evaluation information that they need. The following are suggested elements to be included when developing the evaluation plan:

• A written description of existing conditions prior to, and in preparation for, the learning activity. Certain conditions must be present to forecast training effectiveness. Does the student need a checklist, a set of items to manipulate, or an outline of the information? Does the instructor need audiovisual equipment, handouts, or a classroom with furniture set up in a specific format? Conditions of the learning activity, including computer-based training (CBT), must be specific and comprehensive.
   
  
• The activity to be performed. The evaluation plan must state the activity to be performed in a manner permitting the evaluator to actually observe the behavior that the student is to learn. Observations can be done in class by the teacher or on the job by the supervisor. It is very difficult and impractical to measure the process of a student's changing attitude or thinking through a task or problem. The evaluator, however, can measure a written exercise, a demonstration of skill, a verbal or written discussion, or any combination of these demonstrable activities. Rather than merely acknowledging that the student was exposed to certain information, the evaluator should observe the skill being performed or the information being applied. In computer-based training, evaluation measurement can be programmed to occur at the instructional unit level, with subsequent units adjusted based on student response. In other instructional methods, adjustments can be made in real time by the instructor based on the nature of student questions during the course. Adjustments can also be made between courses in a student's training sequence.
   
  
• Measures of success derived from the individual's normal work products rather than from classroom testing. This directly ties the individual's performance to the impact on the organization's mission. Written behavioral objectives for a learning activity must include a stated level of success. For example, quantitative skills can be expressed as being performed successfully every time, 10 out of 100 times, or 5 out of 10 times, and the impact or consequences can be noted. Risk management techniques should be used to establish the criticality of quantitative skills. Qualitative skills can be measured by distinguishing satisfactory performance from poor performance, or outstanding performance from satisfactory performance. Measurements of qualitative skills might include the amount of repeat work required on the part of the learner, customer satisfaction, or peer recognition of the employee as a source of IT security information.
   
  
• The nature and purpose of the training activity, and whether it is at a beginning, intermediate, or advanced level, will influence the setting of success measures. This is a subjective goal. If success levels are not documented, an individual student's achievement of the behavioral learning objectives cannot be evaluated, nor can the learning activity itself be evaluated as part of the organization's overall training program.

In addition to the written objectives suggested above, the evaluation plan should show how the data will be collected and used. This step will help to stimulate support for the cost and effort of the data collection.

The Levels of Evaluation

There are four levels of evaluation that progress from relatively simple to rather complex. These levels of evaluation are related to the four purposes of evaluation.

Level 1: End-of-Course Evaluations (Student Satisfaction)

A common term for this type of evaluation is "the 'Smiley Face' evaluation." For example, Likert Scale-type forms ask the student to check a range of options from "poor" to "excellent" indicating the student's feelings about the training activity. The responses indicate the perceptions of the student and provide information about the conditions for learning. At this level of evaluation, questions could be asked about the student's satisfaction with the training facility and instructor, the manner of presentation of the content, and whether or not course objectives met the student's expectations. Although this type of evaluation does not provide in-depth data, it does provide rapid feedback from the learner's perspective.

Measurement of training effectiveness depends on an understanding of the background and skill levels of the people being trained. For example, technical training provided to a group of systems analysts and programmers will have a different level of effect than the same information provided to a group of accountants. Basic demographic data may be collected at the beginning or conclusion of the course. Information about the learners' satisfaction with the course and course material should be collected at the end of the course.

This level of evaluation measures how much information or the level of skill that was transmitted to the student participating in the training activity. The evaluation could be in various formats relative to the level of training. Participants in an IT security basic and literacy course could be given tests both before and after the course. At an intermediate or advanced training level, participants could be given a performance test, such as a case study to analyze. At the education level, essay questions exploring concepts would be appropriate. The evaluation format must relate to the behavioral objectives of the learning activity. This, in turn, drives the content being presented. The Level 2 evaluation provides instant feedback and is more objective than a Level 1 evaluation. Behavior objective testing assesses how much the student remembered or measures skills demonstrated by the student's performance at the end of the program. Level 2 evaluations can be built into each unit of instruction as the course progresses.

A Level 2 evaluation measures success in the transfer of information and skills to the student. It enables the evaluator to determine if a given student needs to repeat the course or attend a different type of learning activity that presents the same material in a different format. The evaluator should be able to see if a pattern of transfer problems exists and to determine whether or not the course itself may need to be revised or dropped from an organization's training program.

Behavior objective testing is possibly the most difficult measurement area to address. It is relatively easy to test the knowledge level of the attendees after completing a course or unit of instruction but it is not easy to determine when that learning took place. An attendee may have had knowledge of the subject area before receiving the instruction and participation in the course may have had little or no impact in expanding knowledge. As a result, information collected solely at the conclusion of a course or instructional unit must be examined with respect to the attendee's background and education.

An approach to determining the learning impact of a specific course or instructional unit is to test at the outset of the training and at the conclusion of the training and to compare the results. At the beginning and intermediate levels, it is appropriate to test a student's knowledge of a particular subject area by including questions or tasks where there is only a single right answer or approach. Questions regarding selection of the "best" answer among possible options should be reserved for those training environments where there is opportunity for analyzing why a particular answer is better than other answers.

Level 3: Job Transfer Skills (Student Performance Effectiveness)