Security Metrics Rehab – Part One

Security Activities that Produce Performance Indicators

In this two-part security governance series, we’ll take a look at the broader picture of security metrics and how to derive them from security activities.  The drug-like fervor around its discovery and cultivation across security and compliance groups has led no where fast; largely due to the same causal factors related to InfoSec group unable to associate operational impact from technical and process related {flaws|vulnerabilities|control gaps|weaknesses}.

This has created a fog of ineptness from which many groups in the Fortune 500 stand today.  This first part aims to level-set on how metrics should be applied in InfoSec and what frameworks to leverage in order to subsequently define a suite of security activities that produce performance indicators that matter.

Security Metrics Defined and Defended

Simply defined, a metric is a unit of measure. In business, metrics are intended to measure progress in various areas such as Sales, Finance, Accounting, and HR. Long before words like ‘cyber’ and ‘cloud’ entered the vernacular, terms such as return on investment, employee turnover, collection efficiency, inventory control, and client churn were commonplace across various industries.

Fluctuations in these aforementioned values are still pivotal data points for decision making in nearly every industry. Metrics’ importance to the information security industry is no different than in more traditional business areas. They substantiate security initiatives by tracking key performance indicators (KPIs) that correspond to security activities such as ‘building security in’, static/ dynamic analysis, risk remediation rates, security training effectiveness, and more.

What Makes a Metric Valuable?

Lets look at what makes a metric worthy of being called a metric in the first place.

1. A metric should foster comparative analysis. If a metric cannot be measured and compared over time, it will not be useful in decision-making.
2. Metrics should be simple to understand. If the scope, source, and calculation of the metric are straightforward and easy to understand, there is a better chance for that metric to be used and useful.
3. Good metrics foster rate or ratio values. By design, rates and ratios provide for comparative analysis. Rates and ratios make for easier such as trend analysis on what is being measured.
4. Metrics change behavior. Good metrics will clearly point to the need for change in the area being analyzed. Since good metrics are based upon evidence from good data sets, sample sizes, and data integrity (e.g. – exclusive of outliers, anomalies, etc.), areas that need to change will be clear.

Unfortunately, many of the metrics we read about today are largely swayed by a vendor’s perspective in Information Security (InfoSec). Even more unfortunate is the fact that many companies abide by a vendor’s perspective on security metrics versus contemplating their own. Vendor-provided metrics wouldn’t necessarily be a problem, however, a lack of transparency into how these tool-derived metrics are produced makes their utility suspect. Having worked at two of the largest product vendors in InfoSec, I can attest to the sometimes less than rigorous methods used to legitimize values such as risk, threat, or impact rating. Somehow vendor-provided metrics have not lost their influence on the masses, but for those looking to find valuable metrics, a more inward approach will be needed.

Drivers for Metrics

Unfortunately for many InfoSec professionals, metrics are usually used as the rationale for more budget or as a way to fulfill regulatory requirements. InfoSec is still in a very reactionary state across every industry including financial services. Larger financial institutions (FIs) do have a greater level of sophistication in their security programs overall. Larger FIs can attribute the maturity of their security programs to years of operating within the cross-hairs of targeted attacks by threat agents and regulatory compliance scrutiny by internal/ external auditors. Most of these larger FIs have learned to evolve their security programs over the past 20 years, but it doesn’t mean those FIs are all running at an optimized level of security maturity.

Today, those same larger FIs face ongoing challenges of integrating security intelligence across various solution suites. Given the more repeatable security processes and available data models around access control, risk, remediation, vulnerabilities, etc., they still do have a better basis for developing meaningful security metrics. Conversely, other organizations rush too quickly to produce metrics and when they do, they fall short of being reproducible and meaningful. One of the reasons for this is that those organizations are mostly trying to rationalize security investments versus harvesting the right data for ongoing analysis, then reporting. Obviously, substantiating security investments is needed but it shouldn’t be the driving force to cultivating them.

Demonstrating trends, improvements and efficacy around prevention, detection, and response is most important and if done correctly, will naturally validate CapEx/ OpEx line items around security. Most smaller to mid-size organizations should definitely consider to isolate what data points would be needed to support security metrics, but their first priority should be to build the necessary people, process, and technology that can support an agreed upon security program, even if the framework is not fully implemented.

The figure (on left) provides both a source of validity to what (unfortunately) drives security metrics today as well as an ironic reflection of bad metrics (ability to change behavior). This small sample size of 40 CISOs conducted nearly 10 years ago hasn’t driven much change. More budget and regulatory pressures continue to be key drivers for security metrics. So what’s still wrong and how do we get better?

Roll Your Own Metric ‘Magic’

Truth in metrics really hinges on a clearly defined security program – one tailored to your industry, culture, and business environment. Many CISOs rush to define a base of metrics without having even defined the various components of their security program. It’s critical to first develop a foundation on security SOPs (standard operating procedures) and actually execute them before attempting to develop a suite of metrics. Otherwise, CISOs run the risk of violating the 4th tenant of good metrics – having metrics that do not influence behavior due to erroneous data values, poor reporting, incomplete data results, or other issues.

InfoSec programs need to first develop their SOPs, perform them to a repeatable level before measuring KPIs, SLAs, SLOs, etc. This is not to say that a blueprint for security metrics shouldn’t be developed, but rather that CISOs should not rush to measure something that is still being developed. There is a reason that many maturity models (e.g. COBIT, CMMI, etc.) don’t address measuring the effectiveness of a program until the program gains some maturity. A foundation of activities needs to be defined and then repeated in a sustainable manner before subsequent metric reporting can provide any level of value to the CISO or their internal customers.

The figure below reflects the key areas where an organization can create their own security ‘magic’. A vital ingredient to this magic is simply being able to align an InfoSec program inwardly to key goals and objectives for the organization. Instead of looking outwardly, an InfoSec program truly needs to be based on an understanding of the company’s business, IT, critical data sources, and infrastructure in order to define metrics that matter. Industry metrics are great for benchmarking, but this is a step that comes later in the maturity of defining metrics. When metrics can be related to the goals and objectives of data, system, and network custodians, as well as product, project, marketing, and sales managers, InfoSec metrics will become relevant data.

This is the kind of data that internal groups can support in order to foster changes in control implementations, investments, remediation efforts, and more.

Senior leaders in any organization are largely accustomed to having a road map provide a vision for their department’s goals and objectives. CISOs are also largely accustomed to this practice in most of the Fortune 500. Road maps provide a blueprint for what areas of a security program are to be developed along with some degree of insight as to how and when.

In developing or refining this roadmap, CISOs should consider tying the scope of their security program to the areas depicted by the OpenSAMM (www.opensamm.org) framework. Designed to organize and measure how InfoSec is governed, constructed, verified, and deployed across an organization, InfoSec programs can use this as a framework to build SOPs supporting a list of activities across those four key areas. Also useful is NIST’s CyberSecurity Framework and it’s ‘Framework Core’ consisting of five key areas: identify, protect, detect, respond, and recover. SOPs for each of the ‘Core’ areas can be defined with supporting activities within an InfoSec program.

It is at this point that the precepts of developing repeatable security activities, from which metrics can later be harvested, can begin to take place. The key is not to let any framework drive your InfoSec road map but simply serve as a framework of reference, as is the true intent behind many of today’s frameworks. Using an unmodified framework without taking into account the complexities of the business, the industry, or the culture doesn’t allow for the integration of security to the business and operational objectives will not be met.

Metrics that for repeated measure of SOP outcomes and are aligned with the business and its goals will translate to metrics that can be used by those outside of InfoSec, particularly since many of the metrics will reflect the implementation of security controls or remediation of security related risks – both largely assigned to personnel outside of InfoSec, who are the true system or information owners across an enterprise.

Part Two Foreshadowing

From the NIST CSF, we’ll identify a list of security activities that can be derived for each category and function as small vignettes of applicability to several different industries.  In additional, we’ll address road-mapping a projection of growing security activities using a security maturity model.

Define, Manage, or Optimize Your Security Program

Wherever you are in the maturity model of your security program, VerSprite can tailor a range of Governance, Risk, and Compliance Services to fit both your near terms goals and capabilities, while still ensuring that a future vision of an optimized model is obtained. Explore GRC Services →