A metric without a calculation method measures nothing

Reports to management often contain the same numbers as last quarter, only higher: more training sessions, more closed tickets, more patched systems. They rarely answer one question: has information security become better as a result?

That is exactly the question ISO/IEC 27001 asks in clause 9.1. There, the organisation evaluates information security performance and the effectiveness of the ISMS. A number on its own is not yet a basis for that.

Six decisions instead of a list

In substance, the standard requires the organisation to determine:

• what needs to be monitored and measured, including information security processes and controls,
• the methods for monitoring, measurement, analysis and evaluation, as applicable, to ensure valid results,
• when monitoring and measuring is performed,
• who monitors and measures,
• when the results are analysed and evaluated,
• and who analyses and evaluates them.

On top of that comes a note that weighs most in practice: the methods selected should produce comparable and reproducible results. The results are retained as documented information. The text of the standard is protected by copyright, so this is in substance only.

Where metrics fail

Activity instead of effect. "Twelve training sessions held" measures the diligence of the training team. Whether staff recognise a phishing email, it does not measure. Effect shows more in the share of people who click in an exercise, or in the time between receiving a suspicious message and reporting it.

The calculation method drifts. In the first quarter one person counts all systems, in the second another counts only servers. The curve rises although nothing has changed. A metric is reproducible only when a second person arrives at the same number from the same data.

No target value. 83 percent is neither good nor bad as long as nobody has defined what is to be achieved and from which point action is taken. Without a target you can measure, but not evaluate.

Zero is not empty. "Zero security incidents" can mean that nothing happened, or that nobody recorded anything. A metric that outputs missing data as zero rewards looking away.

Measured but not evaluated. The standard separates measuring from analysing and evaluating and requires responsibility for both. The results belong in the management review as a trend, not just as a status on a dashboard.

Few metrics, well founded

A robust set is small. Each metric comes with a description of what it is meant to show, the calculation method with its data source, the unit and direction (higher or lower is better), the target value and a warning threshold, the measurement frequency and a responsible person. Whoever defines this properly for five metrics has more than with thirty that nobody can recalculate.

How isidaten represents this

In the KPI Cockpit, every metric has a definition with unit, direction, measurement frequency, target value, warning threshold and calculation type. 44 metrics can be calculated from the platform's own data, for example the mean time to detect incidents from start and detection over the last twelve months. The calculation method is therefore fixed and does not depend on who calculates. The calculation is triggered and stored as a dated measurement noting that it was calculated automatically. If data is missing, no value and no zero is produced, but a message. Metrics can be kept per information network with their own responsible person and target value. The traffic light follows direction, target and warning threshold, and without a measurement or without a target it stays grey instead of green.

The check question

Who calculated your most important security metric last quarter, and would a second person arrive at the same number from the same data?

If you would first have to open a colleague's spreadsheet, the metric is still an opinion with decimal places. How measurement relates to the other clauses of the standard is covered in The standard does not start at Annex A, the internal audit in An audit date is not yet an audit programme.

Sources

ISO/IEC 27001:2022, clause 9.1 (in substance, text of the standard protected by copyright). Guidance on monitoring, measurement, analysis and evaluation: ISO/IEC 27004:2016.