What should security teams measure alongside awareness training completion?
Measure what happens in specific business processes: verification, approval, escalation, and reporting. Connect exercise findings to an owner, a corrective change, and a retest. Keep completion, observed behavior, and technical control results distinct.
A training completion report can tell you who finished an assigned course. It cannot tell you whether your help desk will reject an unsupported MFA reset, or whether accounts payable will hold a suspicious payment change. Those questions need evidence from the process itself.
Security awareness still has work to do: explain expected behavior, make reporting routes usable, and help people recognize when to stop and ask for help. The problem is treating participation in that program as proof that a business process will withstand an attack.
What the research actually supports
An eight-month randomized study at a large healthcare organization evaluated ten simulated phishing campaigns involving more than 19,500 employees. The researchers found no significant relationship between recent annual awareness training and phishing-simulation failure. Embedded training produced small absolute differences between trained and untrained users. Ho et al., Understanding the Efficacy of Phishing Training in Practice
These results concern the programs and setting studied. They challenge the use of completion as a resilience measure; they do not show that every form of security education is ineffective. Nor do email-simulation results establish how someone would handle a recovery call or an unfamiliar payment request.
For a security leader, the useful question is what each measure supports. Course completion measures delivery. An exercise can reveal a decision made under its particular conditions. A technical check can show whether a configured control prevented an action. Report those results separately.
A recovery process needs more than recognition
Mandiant’s June 2024 UNC3944 report describes attackers impersonating employees to corporate help desks and requesting password or MFA resets. The callers used personal information during verification and targeted accounts with significant privilege potential. Mandiant: UNC3944 Targets SaaS Applications
That finding gives a service owner something concrete to inspect: what establishes the caller’s identity before the desk changes account access? Knowing a colleague’s name or job is not sufficient justification for a reset. An agent needs a procedure they can follow, an exception route when the usual check is unavailable, and authority to pause a request.
Training can explain that procedure. It cannot compensate for a procedure that allows the requester to supply the very contact details used to verify them. Our MFA reset review examines the approval, administrative change, and downstream access as separate observations.
Test a decision you can improve
Start with a workflow and a named owner. For account recovery, agree which account category is in scope, how identity should be established, and where the exercise stops. A call that ends before an account change can still test verification and escalation. It cannot establish whether later enrollment or resource access would succeed.
Consider a hypothetical result: the agent accepts the caller’s story, but recovery information goes only to a previously approved destination that the caller cannot access. Report both observations. Verification needs repair; delivery prevented the tested continuation. Neither “account taken over” nor “everything worked” describes that result.
The fix might involve the verification source, approval authority, destination-change rules, or an exception procedure. Give the change to the owner who can implement it. Then repeat the relevant check and record whether it happened before approval. Include a legitimate recovery case so the repaired procedure still helps the right employee regain access.
Give the board evidence of improvement
A useful program report connects each finding to a decision:
- Which workflow was examined, and what could it authorize?
- What happened in the exercise, and what remained untested?
- Which change was made, by whom?
- What did the retest show?
Completion rates can remain in the training report. Findings and retests belong alongside them when leadership asks whether risk has been reduced. Avoid converting a small exercise into a company-wide estimate of breach probability.
We use Human Vulnerability Management to describe this approach: identify relevant exposure, choose a business decision to test, repair the weakness found, and check the change. It is a proposed operating approach, not evidence that every organization needs the same testing frequency or that every employee needs an individualized risk score.
Awareness and operational validation should support each other. A well-designed procedure gives training something specific to teach. Exercise results show where instructions, tools, or authority need to change. The next cycle should reflect those findings rather than repeat the same material by default.
Next in the series: Discover, on reviewing the public information that could make an impersonation plausible.
