Fri Apr 8 06:49:41 PDT 2016
Management: Security Metrics: What security measurements should be taken and when?
Options:
Option 0: Continuously. (C)
Option 1: Shift change. (S)
Option 2: Daily. (D)
Option 3: Weekly. (W)
Option 4: Monthly. (M)
Option 5: Quarterly. (Q)
Option 6: 6 months. (6)
Option 7: Yearly. (Y)
Option A: Randomly. (R)
Option B: Event-driven. (E)
Option C: On hiring. (H)
When required by external mandates.
Decision:
The following table represents advised practice on when to measure what. Review and update as appropriate.
Element | Low consequence | Medium consequence | High consequence |
Management | Y | 6 | M |
Procedures | Y / E | M / E | C / E |
Documentation | Q | M | W |
Auditing | Y | 6 | M |
Testing | Y | M | R / M |
Technology | M / E | C / S | C / S |
Personnel | H / S | H / Y / S | H / C / Q / S |
Incidents | Q / E / S | E / M / S | E / W / S |
Legal | Y | Y | Y |
Physical | M / E | W / S / E | D / S / E |
Training | 6 / E | Q / E | Q / E |
Awareness | Y | M | D |
Organization | E | E | E |
Protection measurement process times
C=Continuously. S=Shift. D=Daily. W=Weekly. M=Monthly. Q=Quarterly. 6=6 months. Y=Yearly. R=Randomly. E=Event-driven. H=Hiring.
When required by external mandates, measurements should also be taken.
Basis:
Management typically observes measurements of
procedures, documentation, auditing, testing, technology, personnel,
incidents, legal events, physical events, the training and awareness
program, and organizational changes as part of the management control
system for any information-related function. The rate at which this
is done depends on the nature of the protection management function.
Continuously: Continuous measurement implies
that the system is ever vigilant and whenever something occurs, it is
reported and available to management. For example continuous personnel
monitoring implies that whenever an observable associated with
personnel and their behaviors or situation is identified, it is
reported and available for management review, with some pre-defined
situations triggering immediate responses.
Shift change: This is a measurement area to be
undertaken at the beginning of each shift to check on the previous
shift and get a level set for the current situation.
Daily: This implies a daily measurement with
reporting to management on a daily basis. For example, in high
consequence environments, a daily report might be generated of the
security status of the environment and provided to the security
manager for review every morning.
Weekly: This implies weekly reporting of
roll-up information from the week. For example, the physical security
weekly report might include a list of all alarms and incidents with
details of how they were resolved (and which ones are not yet
resolved). This would be in addition to any real-time response
requirements.
Monthly: Many business processes happen
monthly, such as billing and payment cycles, accounting reporting
cycles, etc. To the extent that rapid reporting is not critical and
data is available, monthly reporting to coincide with other monthly
business requirements is sensible. For example, monthly review of
performance against security procedures may lead to improvements over
time or detection of problems associated with changes.
Quarterly: Quarterly reporting is also
required for many business functions, but in addition, specific time
frames associated with human behaviors and memory produce the basis
for quarterly measurements. For example, something like 80% of cases
involving insiders turning (changing loyalties) have observables more
than 90 days in advance of serious harm. Thus quarterly measurement of
personnel issues relevant to detection of turning behaviors might
prevent 80% of these incidents from happening - if the response is
appropriate and timely.
6 months: People tend to lose performance on
many functions over a 6-9 month period. As a result, measurement every
6 months provides the means to prevent these processes from getting to
out of sorts before review and serves to remind those participating of
their duties and responsibilities.
Yearly: Annual metrics are a minimum for
security programs because things change at a pace that mandate review
with some period, and annual seems to work well within the normal
calendar of most organizations.
Randomly: Random measurements are normally
required when those being checked might alter behaviors or records
based on advanced knowledge of review.
Event-driven: When things change, they should
trigger re-measurement. For example, organizational changes may alter
the power and influence structure producing many differences in how
what is implemented and measured. Also, when events such as security
incidents occur, they produce a need for measurement and reporting on
relevant factors as part of the follow-up process that seeks to
mitigate harm and reduce the impact of future incidents.
When hired: As part of the hiring process,
background checks and other similar security-related measurements of
personnel should be done as a matter of course.
When required: Legal, regulatory, management,
or other mandates may lead to measurements as well. These always apply.
The basis for the specific positions is provided in the table below:
Element | Low consequence | Medium consequence | High consequence |
Management |
The annual performance review
of management personnel should include their performance in
security programs to assure that pay and performance of
management (and as a side effect their workers) reflects
proper attention to protection issues. |
As consequences increase,
management attention must also increase. To assure this, at
the Medium consequence level, management is reviewed more
often, typically twice a year, to assure that they are
performing their work appropriately in managing the protection
program and more aware of their protection responsibilities
and the potential results of less attentiveness. |
For High consequence systems,
monthly reviews of performance against security management
processes helps lead to continuous improvement over time
and detection of problems associated with changes. This is
also commensurate with the periodic measurement and reporting
requirements for those who work for them. |
Procedures |
Annual metrics are a minimum
to review procedures since it seems to work well within the
normal calendar of most organizations and longer periods get
to the point where the issues are no longer remembered by the
time they are measured. |
As consequences grow, the
need to review procedures for possible problems becomes more
acute. monthly reviews of procedures and, in particular,
where they break down, provides a reasonable degree of surety
that problems will be remembered and changes wrung out over
the measurement period so that improvements or problems
associated with procedural changes can be identified and
mitigated. |
Continuous measurement of
procedures at the high consequence level provides assurance
that any time a procedure fails to meet normal expectations,
it can be immediately reviewed and corrections made in a
timely fashion. |
Procedures |
When the
environment changes or when procedural failures or problems
are detected, they need to be addressed. In IT environments,
things tend not to get better as they continue to go wrong
over longer time periods. Since more harm tends to come over
time, events should drive improvement. This then requires
measurement, in some cases with specific augmented measurement
to meet the needs of the event and changes made and to then
identify the normal conditions associated with the
changes. |
Documentation |
Quarterly measurement of
documentation (e.g., its presence, adequacy, and ability to
access) is required in order to continue to keep normal
business records and meet normal accounting practices
associated with operating almost any kind of system. If and to
the extent that documents are missing over an extended period
of time, this introduced potential legal liabilities. |
Monthly measurement of
documentation is consistent the other reporting and
measurement requirements foe Medium surety situations and is
likely to be required for other external mandates. To the
extent that documentation is missing or inaccurate, it can
often be corrected within a month, but over longer periods,
things like backup copies and other business processes tend to
become less reliable and disposition processes start to become
potential sources of lost records. In addition, because
billing and payment cycles tend to be monthly, measurement of
documentation is important to assure that these and other
related financial processes are accurate and justified based
on available records. |
As consequences increase, it
becomes more important to assure that records are bing kept
and properly documented. Weekly measurement of documentation
associated with processing is timely enough to be useful in
finding and correcting otherwise undetected failures in
documentation without becoming an excessive and unmanageable
burden. In addition, weekly activities tend to reveal periodic
problems, such as a particular shift that is regularly off in
performance, and reflect changes associated with holiday
periods and other similar environmental conditions better than
longer time frames. |
Auditing |
Annual audits should be
undertaken as part of any substantial business, and thus it is
to be expected that these will include reviews of the
IT-related activities. As such, and as documents relied upon
by management and investors, such measurements as are required
for this purpose should be made and completed at least
annually. |
People tend to lose
performance on many functions over a 6-9 month period. As a
result, measurement every 6 months provides the means to
prevent these processes from getting to out of sorts and
serves to remind those participating of their duties and
responsibilities. For medium surety systems, audit reviews at
least once per 6 months, and perhaps once per quarter help to
mitigate drift in the operating environment commensurate with
the level of management attention to information protection. |
Monthly audit processes are
typically used as part of standard accounting processes
associated with billing cycles and related matters. As a
matter of normal operation, some level of security audit should
be completed on a monthly basis to provide feedback on high
consequence systems and to assure that they are operating as
they should be based on an independent opinion. |
Testing |
Testing results (i.e.,
metrics) reported to management on a yearly basis is really
the absolute minimum for security. In part, this is because
things change at a far faster pace in much of IT. |
While testing should happen
at a far greater rate, measurements resulting from protection
testing don't typically need to be reported to management more
than monthly in order for trend analysis to be performed,
progress measured, and adaptations undertaken.
| Random measurements are normally
required when those being checked might alter behaviors or
records based on advanced knowledge of review. |
|
Technology |
Even in low risk situations,
IT technology measurements should be undertaken monthly to
provide feedback to management on performance.
To the extent that events cause protection failures to
become known, technology should be measured to determine
whether and to what extent changes are necessary and to
confirm that changes met the need after completion. |
Continuous
measurement of technology is typically required for medium
and high consequence situations in order to provide for
detection of changes and events that may produce potentially
serious negative consequences in time to mitigate those
consequences to management specified levels.
At the end of each shift and the beginning of the next
shift, technology-related measurements should be provided and
taken respectively, so that the IT environment operational
and technical status is clearly understood and reflected in
the measurements by the next shift, and so that anything
missed by the previous shift can be independently measured and
potentially detected by the next shift. This limits system
drift and reduces the effect of the human tendency to get used
to changes that occur slowly. |
Personnel |
No additional requirements
beyond the hiring requirements for similar personnel are
required for low-consequence environments. |
Yearly metrics (personnel
reviews) are a minimum requirement for personnel suitability
in medium consequence IT environments. Because of the
relatively low cost of background checks and related HR
reporting and review requirements for employees, annual
employee performance reviews should include additional
requirements for key personnel involved in and producing
potential harm to medium consequence IT systems. This
typically includes measurement of their security-related
behaviors, infractions, and other workplace indicators of less
than expected performance. |
Continuous measurement for
personnel implies automated reporting from credit agencies and
other similar sources to detect specific indicators that are
known to correlate to insider turning behavior, including
changes in loyalty, reliability, and suitability. This includes
indicators like applying for jobs with competitors, being
late to work or insubordinate, and failure to complete necessary
training or other similar requirements to maintain currency.
Quarterly measurement is a minimum for high consequence
key personnel based on studies performed that suggest that
about 80% of cases involving insiders turning (changing
loyalties) have observables more than 90 days in advance of
serious harm. Thus quarterly measurement of personnel issues
relevant to detection of turning behaviors might prevent 80%
of these incidents from happening - if the response is
appropriate and timely. |
Personnel |
As part of the
hiring process, background checks and other similar
security-related measurements of personnel should be done as a
matter of course.
To the extent that noticeable changes in personnel and
their behaviors are identified by a shift supervisor or other
team members, these should be reported no later than the end
of the shift, and preferably sooner. Supervisors should
report these outcomes as metrics at the end of each shift so
they become part of the record that then forms longer-term
behavioral measurements and so that patterns of behaviors
across shifts can be identified, perhaps associated with
adverse or hazardous environmental changes not otherwise
detected. |
Incidents |
Quarterly measurement and
reporting on incidents is important to understanding business
implications of security-related risks from IT systems that
are required in quarterly reports and projections for most
large enterprises, public companies, etc. Thus this sort of
reporting should be made quarterly and the supporting
measurements taken quarterly. |
At least monthly incident
reports will be required in order for management to make
changes resulting from incidents and verify that those changes
are taking effect and working as desired. As time frames go
beyond this, memory of specifics tends to fade, and as time
passes without measurement, risks of further incidents because
of failed response or adaptation grow in terms of consequence. |
Weekly roll-ups of incidents
and measurement of progress related to incidents is necessary
at high consequence levels to allow enough time for progress
to be made against issues while assuring that management and
workers remain mindful of the need for resolution and
mitigation. By tracking this weekly, the weekend doesn't come
until the incidents of the week are understood and properly
dealt with, and this is motivating in terms of making
progress. |
Incidents |
When events
require immediate response or as reportable changes occur,
measurements should be taken to provide relevant information
to management on an interrupt driven basis. Similarly, events
may trigger re-measurement. When security incidents occur,
they produce a need for measurement and reporting on relevant
factors as part of the follow-up process that seeks to
mitigate harm and reduce the impact of future incidents.
Incidents during a shift should be reported to the next
shift so that they are aware of the situation as they begin
their shift. These reports should also become part of trends
measured at the start and end of shifts to help detect systemic
changes over time. |
Legal |
Annual metrics
are a minimum for legal issues because changes in laws,
regulations, or other similar external drivers and duties must
be reviewed to assure that event driven changes (e.g., new
regulations) were not missed in the normal process of updating
duties to protect. This is also part of diligence reporting
for public companies where legal and regulatory changes may be
material and thus must be reflected in annual reports. |
Physical |
Many business processes
happen monthly, such as billing and payment cycles, accounting
reporting cycles, etc. Most physical security issues have to
be measured at this rate because of these normal business
reporting requirements. For example, a break-in resulting in
increased facilities costs has to be reported to management so
their bookkeeping can accurately reflect the expenditures
and/or liabilities. Similarly, alarm companies and other
similar providers typically provide monthly invoices along
with summary reports that get rolled up into the monthly
measurement of the physical security system. |
This implies weekly
reporting of roll-up information from the week. For example,
the physical security weekly report might include a list of
all alarms and incidents with details of how they were
resolved (and which ones are not yet resolved). This would be
in addition to any real-time response requirements.
|
Daily measurements of physical
security issues should be reviewed in high consequence
situations to assure that as the situation changes,
adaptations are properly made. These are typically reviewed by
the security manager so they can become aware of the situation
at the beginning of their daily activities. |
Physical |
Events that
cause potential changes in the physical security environment,
including naturally occurring (e.g., earth movements) and
artificially generated (e.g., a highway accident near the
perimeter) physical events, should be measured against known
limits of the IT environment (e.g., earth movements measured
against the physical building capacity to handle them, highway
accident against the perimeter assumptions) and action taken
to the extent necessary according to the physical security
plan.
Each shift should report physical security events and
measurements related to them to the next shift so they are
kept aware of the changing environment and the next shift
should re-measure relevant physical changes to assure that
errors are not propagated or additional changes are identified
and compensated for appropriately. |
Training |
Training on security-related
matters should be required for all workers at least every 6
months, and metrics on training should indicate and
demonstrate the extent to which those workers understand and
are able to perform their security-related duties. While
history suggests that such training is retained at reasonably
levels for only 6 months on average, in low consequence
environments, this is commonly done and found acceptable in
industry. |
Quarterly
training and measurement of training results is consistent
with studies that suggest that such training loses
effectiveness in a 6-month time frame. For medium and high
consequence environments, allowing training to lapse to the
point where workers start to forget or fail to properly
respond reliably in some conditions is inappropriate.
Measurement is required in order to assure and demonstrate
that the training is effective and that workers are able to
perform their assigned duties if and when called upon to do
so. To the extent that measurement does not show adequate
retention and behavioral responses with quarterly refreshers,
more frequent training should be applied or the program
re-examined for efficacy. |
Training |
As events cause
changes to security operations or other similar adaptations,
training is required in order to adapt the behavior of workers
to the changed environment. This in turn should produce
measurement of effectiveness of the changes and related
training so that workers do their jobs properly under
anticipated circumstances as measured by the training
program. |
Awareness |
Annual metrics of the
effectiveness of the security awareness program should be
undertaken as part of diligence for the evaluation of the
overall security program as part of annual review
processes. |
The awareness program for
medium consequence environments should be ever-present, but
measurement of the program is typically feasible only on a
monthly basis because other measurement and reporting that
reflects awareness issues only occurs at that rate.
| Daily awareness should be
calculable based on the level of incidental errors and
omissions in normal security procedures, such as remembering
to and diligently carrying out day-to-day duties and
activities. As an embedded part of the daily regimen, things
like door lock status checks and perimeter reviews by guards
should feed back to management daily as part of incident
reporting and be reflective of awareness issues and briefed as
part of daily awareness updates on security-related
matters. This is normally part of the shift change process and
daily management activities. |
Organization |
When there are
organizational changes, including personnel changes, hirings,
firings, resignations, restructuring, and so forth, these
should trigger re-measurement of all affected parts of the
protection program. For example, organizational changes may
alter the power and influence structure producing many
differences in how what is implemented and measured. |
Security measurement process times basis
C=Continuously. S=Shift change. D=Daily. W=Weekly. M=Monthly. Q=Quarterly. 6=6 months. Y=Yearly. R=Randomly. E=Event-driven. H=Hiring.
When required by external mandates, measurements should also be taken.
Copyright(c) Fred Cohen, 1988-2015 - All Rights Reserved
|