Fri Apr 8 06:47:17 PDT 2016

Management: Security Metrics: What security measurements are taken and when?

Options:

Option 0: Continuously. (C)
Option 1: Shift change. (S)
Option 2: Daily. (D)
Option 3: Weekly. (W)
Option 4: Monthly. (M)
Option 5: Quarterly. (Q)
Option 6: 6 months. (6)
Option 7: Yearly. (Y)
Option A: Randomly. (R)
Option B: Event-driven. (E)
Option C: On hiring. (H)
When required by external mandates.

Basis:

Management typically observes measurements of procedures, documentation, auditing, testing, technology, personnel, incidents, legal events, physical events, the training and awareness program, and organizational changes as part of the management control system for any information-related function. The rate at which this is done depends on the nature of the protection management function.

Continuously: Continuous measurement implies that the system is ever vigilant and whenever something occurs, it is reported and available to management. For example continuous personnel monitoring implies that whenever an observable associated with personnel and their behaviors or situation is identified, it is reported and available for management review, with some pre-defined situations triggering immediate responses.

Shift change: This is a measurement area to be undertaken at the beginning of each shift to check on the previous shift and get a level set for the current situation.

Daily: This implies a daily measurement with reporting to management on a daily basis. For example, in high consequence environments, a daily report might be generated of the security status of the environment and provided to the security manager for review every morning.

Weekly: This implies weekly reporting of roll-up information from the week. For example, the physical security weekly report might include a list of all alarms and incidents with details of how they were resolved (and which ones are not yet resolved). This would be in addition to any real-time response requirements.

Monthly: Many business processes happen monthly, such as billing and payment cycles, accounting reporting cycles, etc. To the extent that rapid reporting is not critical and data is available, monthly reporting to coincide with other monthly business requirements is sensible. For example, monthly review of performance against security procedures may lead to improvements over time or detection of problems associated with changes.

Quarterly: Quarterly reporting is also required for many business functions, but in addition, specific time frames associated with human behaviors and memory produce the basis for quarterly measurements. For example, something like 80% of cases involving insiders turning (changing loyalties) have observables more than 90 days in advance of serious harm. Thus quarterly measurement of personnel issues relevant to detection of turning behaviors might prevent 80% of these incidents from happening - if the response is appropriate and timely.

6 months: People tend to lose performance on many functions over a 6-9 month period. As a result, measurement every 6 months provides the means to prevent these processes from getting to out of sorts before review and serves to remind those participating of their duties and responsibilities.

Yearly: Annual metrics are a minimum for security programs because things change at a pace that mandate review with some period, and annual seems to work well within the normal calendar of most organizations.

Randomly: Random measurements are normally required when those being checked might alter behaviors or records based on advanced knowledge of review.

Event-driven: When things change, they should trigger re-measurement. For example, organizational changes may alter the power and influence structure producing many differences in how what is implemented and measured. Also, when events such as security incidents occur, they produce a need for measurement and reporting on relevant factors as part of the follow-up process that seeks to mitigate harm and reduce the impact of future incidents.

When hired: As part of the hiring process, background checks and other similar security-related measurements of personnel should be done as a matter of course.

When required: Legal, regulatory, management, or other mandates may lead to measurements as well. These always apply.

The basis for the specific positions is provided in the table below:

Element Low consequence Medium consequence High consequence
Management The annual performance review of management personnel should include their performance in security programs to assure that pay and performance of management (and as a side effect their workers) reflects proper attention to protection issues. As consequences increase, management attention must also increase. To assure this, at the Medium consequence level, management is reviewed more often, typically twice a year, to assure that they are performing their work appropriately in managing the protection program and more aware of their protection responsibilities and the potential results of less attentiveness. For High consequence systems, monthly reviews of performance against security management processes helps lead to continuous improvement over time and detection of problems associated with changes. This is also commensurate with the periodic measurement and reporting requirements for those who work for them.
Procedures Annual metrics are a minimum to review procedures since it seems to work well within the normal calendar of most organizations and longer periods get to the point where the issues are no longer remembered by the time they are measured. As consequences grow, the need to review procedures for possible problems becomes more acute. monthly reviews of procedures and, in particular, where they break down, provides a reasonable degree of surety that problems will be remembered and changes wrung out over the measurement period so that improvements or problems associated with procedural changes can be identified and mitigated. Continuous measurement of procedures at the high consequence level provides assurance that any time a procedure fails to meet normal expectations, it can be immediately reviewed and corrections made in a timely fashion.
Procedures When the environment changes or when procedural failures or problems are detected, they need to be addressed. In IT environments, things tend not to get better as they continue to go wrong over longer time periods. Since more harm tends to come over time, events should drive improvement. This then requires measurement, in some cases with specific augmented measurement to meet the needs of the event and changes made and to then identify the normal conditions associated with the changes.
Documentation Quarterly measurement of documentation (e.g., its presence, adequacy, and ability to access) is required in order to continue to keep normal business records and meet normal accounting practices associated with operating almost any kind of system. If and to the extent that documents are missing over an extended period of time, this introduced potential legal liabilities. Monthly measurement of documentation is consistent the other reporting and measurement requirements foe Medium surety situations and is likely to be required for other external mandates. To the extent that documentation is missing or inaccurate, it can often be corrected within a month, but over longer periods, things like backup copies and other business processes tend to become less reliable and disposition processes start to become potential sources of lost records. In addition, because billing and payment cycles tend to be monthly, measurement of documentation is important to assure that these and other related financial processes are accurate and justified based on available records. As consequences increase, it becomes more important to assure that records are bing kept and properly documented. Weekly measurement of documentation associated with processing is timely enough to be useful in finding and correcting otherwise undetected failures in documentation without becoming an excessive and unmanageable burden. In addition, weekly activities tend to reveal periodic problems, such as a particular shift that is regularly off in performance, and reflect changes associated with holiday periods and other similar environmental conditions better than longer time frames.
Auditing Annual audits should be undertaken as part of any substantial business, and thus it is to be expected that these will include reviews of the IT-related activities. As such, and as documents relied upon by management and investors, such measurements as are required for this purpose should be made and completed at least annually. People tend to lose performance on many functions over a 6-9 month period. As a result, measurement every 6 months provides the means to prevent these processes from getting to out of sorts and serves to remind those participating of their duties and responsibilities. For medium surety systems, audit reviews at least once per 6 months, and perhaps once per quarter help to mitigate drift in the operating environment commensurate with the level of management attention to information protection. Monthly audit processes are typically used as part of standard accounting processes associated with billing cycles and related matters. As a matter of normal operation, some level of security audit should be completed on a monthly basis to provide feedback on high consequence systems and to assure that they are operating as they should be based on an independent opinion.
Testing Testing results (i.e., metrics) reported to management on a yearly basis is really the absolute minimum for security. In part, this is because things change at a far faster pace in much of IT. While testing should happen at a far greater rate, measurements resulting from protection testing don't typically need to be reported to management more than monthly in order for trend analysis to be performed, progress measured, and adaptations undertaken.
Random measurements are normally required when those being checked might alter behaviors or records based on advanced knowledge of review.
Technology Even in low risk situations, IT technology measurements should be undertaken monthly to provide feedback to management on performance.
To the extent that events cause protection failures to become known, technology should be measured to determine whether and to what extent changes are necessary and to confirm that changes met the need after completion. 		Continuous measurement of technology is typically required for medium and high consequence situations in order to provide for detection of changes and events that may produce potentially serious negative consequences in time to mitigate those consequences to management specified levels.
At the end of each shift and the beginning of the next shift, technology-related measurements should be provided and taken respectively, so that the IT environment operational and technical status is clearly understood and reflected in the measurements by the next shift, and so that anything missed by the previous shift can be independently measured and potentially detected by the next shift. This limits system drift and reduces the effect of the human tendency to get used to changes that occur slowly.
Personnel No additional requirements beyond the hiring requirements for similar personnel are required for low-consequence environments. Yearly metrics (personnel reviews) are a minimum requirement for personnel suitability in medium consequence IT environments. Because of the relatively low cost of background checks and related HR reporting and review requirements for employees, annual employee performance reviews should include additional requirements for key personnel involved in and producing potential harm to medium consequence IT systems. This typically includes measurement of their security-related behaviors, infractions, and other workplace indicators of less than expected performance. Continuous measurement for personnel implies automated reporting from credit agencies and other similar sources to detect specific indicators that are known to correlate to insider turning behavior, including changes in loyalty, reliability, and suitability. This includes indicators like applying for jobs with competitors, being late to work or insubordinate, and failure to complete necessary training or other similar requirements to maintain currency.
Quarterly measurement is a minimum for high consequence key personnel based on studies performed that suggest that about 80% of cases involving insiders turning (changing loyalties) have observables more than 90 days in advance of serious harm. Thus quarterly measurement of personnel issues relevant to detection of turning behaviors might prevent 80% of these incidents from happening - if the response is appropriate and timely.
Personnel As part of the hiring process, background checks and other similar security-related measurements of personnel should be done as a matter of course.
To the extent that noticeable changes in personnel and their behaviors are identified by a shift supervisor or other team members, these should be reported no later than the end of the shift, and preferably sooner. Supervisors should report these outcomes as metrics at the end of each shift so they become part of the record that then forms longer-term behavioral measurements and so that patterns of behaviors across shifts can be identified, perhaps associated with adverse or hazardous environmental changes not otherwise detected.
Incidents Quarterly measurement and reporting on incidents is important to understanding business implications of security-related risks from IT systems that are required in quarterly reports and projections for most large enterprises, public companies, etc. Thus this sort of reporting should be made quarterly and the supporting measurements taken quarterly. At least monthly incident reports will be required in order for management to make changes resulting from incidents and verify that those changes are taking effect and working as desired. As time frames go beyond this, memory of specifics tends to fade, and as time passes without measurement, risks of further incidents because of failed response or adaptation grow in terms of consequence. Weekly roll-ups of incidents and measurement of progress related to incidents is necessary at high consequence levels to allow enough time for progress to be made against issues while assuring that management and workers remain mindful of the need for resolution and mitigation. By tracking this weekly, the weekend doesn't come until the incidents of the week are understood and properly dealt with, and this is motivating in terms of making progress.
Incidents When events require immediate response or as reportable changes occur, measurements should be taken to provide relevant information to management on an interrupt driven basis. Similarly, events may trigger re-measurement. When security incidents occur, they produce a need for measurement and reporting on relevant factors as part of the follow-up process that seeks to mitigate harm and reduce the impact of future incidents.
Incidents during a shift should be reported to the next shift so that they are aware of the situation as they begin their shift. These reports should also become part of trends measured at the start and end of shifts to help detect systemic changes over time.
Legal Annual metrics are a minimum for legal issues because changes in laws, regulations, or other similar external drivers and duties must be reviewed to assure that event driven changes (e.g., new regulations) were not missed in the normal process of updating duties to protect. This is also part of diligence reporting for public companies where legal and regulatory changes may be material and thus must be reflected in annual reports.
Physical Many business processes happen monthly, such as billing and payment cycles, accounting reporting cycles, etc. Most physical security issues have to be measured at this rate because of these normal business reporting requirements. For example, a break-in resulting in increased facilities costs has to be reported to management so their bookkeeping can accurately reflect the expenditures and/or liabilities. Similarly, alarm companies and other similar providers typically provide monthly invoices along with summary reports that get rolled up into the monthly measurement of the physical security system. This implies weekly reporting of roll-up information from the week. For example, the physical security weekly report might include a list of all alarms and incidents with details of how they were resolved (and which ones are not yet resolved). This would be in addition to any real-time response requirements. Daily measurements of physical security issues should be reviewed in high consequence situations to assure that as the situation changes, adaptations are properly made. These are typically reviewed by the security manager so they can become aware of the situation at the beginning of their daily activities.
Physical Events that cause potential changes in the physical security environment, including naturally occurring (e.g., earth movements) and artificially generated (e.g., a highway accident near the perimeter) physical events, should be measured against known limits of the IT environment (e.g., earth movements measured against the physical building capacity to handle them, highway accident against the perimeter assumptions) and action taken to the extent necessary according to the physical security plan.
Each shift should report physical security events and measurements related to them to the next shift so they are kept aware of the changing environment and the next shift should re-measure relevant physical changes to assure that errors are not propagated or additional changes are identified and compensated for appropriately.
Training Training on security-related matters should be required for all workers at least every 6 months, and metrics on training should indicate and demonstrate the extent to which those workers understand and are able to perform their security-related duties. While history suggests that such training is retained at reasonably levels for only 6 months on average, in low consequence environments, this is commonly done and found acceptable in industry. Quarterly training and measurement of training results is consistent with studies that suggest that such training loses effectiveness in a 6-month time frame. For medium and high consequence environments, allowing training to lapse to the point where workers start to forget or fail to properly respond reliably in some conditions is inappropriate. Measurement is required in order to assure and demonstrate that the training is effective and that workers are able to perform their assigned duties if and when called upon to do so. To the extent that measurement does not show adequate retention and behavioral responses with quarterly refreshers, more frequent training should be applied or the program re-examined for efficacy.
Training As events cause changes to security operations or other similar adaptations, training is required in order to adapt the behavior of workers to the changed environment. This in turn should produce measurement of effectiveness of the changes and related training so that workers do their jobs properly under anticipated circumstances as measured by the training program.
Awareness Annual metrics of the effectiveness of the security awareness program should be undertaken as part of diligence for the evaluation of the overall security program as part of annual review processes. The awareness program for medium consequence environments should be ever-present, but measurement of the program is typically feasible only on a monthly basis because other measurement and reporting that reflects awareness issues only occurs at that rate. Daily awareness should be calculable based on the level of incidental errors and omissions in normal security procedures, such as remembering to and diligently carrying out day-to-day duties and activities. As an embedded part of the daily regimen, things like door lock status checks and perimeter reviews by guards should feed back to management daily as part of incident reporting and be reflective of awareness issues and briefed as part of daily awareness updates on security-related matters. This is normally part of the shift change process and daily management activities.
Organization When there are organizational changes, including personnel changes, hirings, firings, resignations, restructuring, and so forth, these should trigger re-measurement of all affected parts of the protection program. For example, organizational changes may alter the power and influence structure producing many differences in how what is implemented and measured.
Security measurement process times basis
C=Continuously. S=Shift change. D=Daily. W=Weekly. M=Monthly. Q=Quarterly. 6=6 months. Y=Yearly. R=Randomly. E=Event-driven. H=Hiring.

When required by external mandates, measurements should also be taken.

Copyright(c) Fred Cohen, 1988-2015 - All Rights Reserved