How to Measure a Help-Desk Vishing Assessment

Before scheduling a help-desk vishing assessment, write the decision you want to observe. For example: when a caller requests an account change and cannot complete the approved verification route, does the agent leave the change pending and reach the designated escalation owner?

That question produces a different assessment from asking how many calls were answered or how persuasive a simulated caller sounded. It tells observers what to look for and which team owns each step.

Mandiant describes service-desk impersonation in UNC3944 activity and recommends positive verification and monitoring of sensitive account changes. That reporting supports testing consequential support decisions; it does not provide a benchmark failure rate for your organization. Mandiant: vishing analysis, Vishing in the Wild and Recommendations

The recovery tabletop helps a team rehearse decisions. This guide measures a batch of authorized interactions: who reached the decision, what happened, and whether a retest can be compared fairly. The dataset is fictional, with no measured GhostEye customer results. Technical account changes and resource-access tests require their own scope and evidence.

Choose the right denominator

Fictional assessment

Assessment population

• 100 scheduled → 80 connected

  72 reached an assessable decision. Outcome-unknown count: 0.

Decision criterion

• 54 / 72 = 75%

  54 met the criterion; 18 did not.

Required handoffs

• 15 / 24 = 62.5%

  15 acknowledged within 30 minutes; 3 routed but unacknowledged; 6 not routed.

The 24 handoffs overlap the 72 eligible encounters. Using only 18 routed cases gives 83.3% and excludes six missing referrals.

Define the decision and the stopping point

Begin with one account action and the procedure governing it. Ask the process owner to identify the required verification, the unavailable-method branch, and the authorized exception route. Use the actual procedure version as the assessment reference.

Write the primary outcome in observable language. “Agent followed security best practices” is too vague. Prefer: “Before any account change, the agent completed the approved verification or kept the change pending and initiated the required escalation.” State which evidence will distinguish those alternatives.

Define the endpoint before the interaction starts. An exercise may stop once the agent records the decision, before any account is changed. Another may include a designated test-account action under separate authorization. Record the chosen endpoint so a successful human observation does not become an unsupported claim about technical containment.

Also define stop conditions for the assessment team. Include an unexpected production change, a participant reporting a real incident, or a request exceeding the approved scenario. Assign someone who can stop execution and route operational concerns. Review those boundaries with the help-desk owner before the first scheduled attempt.

Choose the unit you will count

For this guide, an assessable eligible encounter is one connected interaction that reaches the predefined decision point and has enough evidence to classify the outcome. The tables shorten this to “eligible.” A scheduled attempt that never connects cannot demonstrate a verification decision.

Keep a separate count for decision reached, outcome unknown. These encounters reached the action being evaluated, but the available evidence cannot establish whether the criterion was met. Report them beside the assessable eligible count and outside that rate’s denominator. They are not pre-decision exclusions or confirmed failures. If many outcomes are unknown, the rate describes only the cases you could assess.

Decide how repeated contacts will be handled. If the same scenario is transferred between agents, count it as one encounter with several handoffs unless your design explicitly measures agent-level decisions. If the team calls the same person twice, retain that relationship in the dataset rather than treating both observations as unrelated people.

Create an eligibility rule before looking at outcomes. The rule might exclude a call that disconnects before the requested action is understood or an encounter that enters a real incident process. Keep an exclusion reason for every excluded record. Do not remove a difficult outcome merely because the scenario did not unfold as expected.

For each metric, write the numerator, denominator, observation window, and missing-data rule in a short dictionary. Make the dictionary available to both observers and the person calculating results. Resolve disagreements about definitions before converting the record into a percentage.

Separate behavior from the quality of the process

Record the action the participant took and the process available to them. An agent may select the required escalation route while its destination is unavailable. That produces a different improvement task from an agent bypassing the route entirely.

Use separate fields for verification decision, escalation initiation, destination receipt, acknowledgment, and next-owner acceptance. Include timestamps where they are available and relevant. For a completed handoff, define whether acknowledgment alone is sufficient or whether the receiving owner must explicitly accept responsibility.

The fictional dataset below uses acknowledgment within 30 minutes of the required escalation decision as its handoff outcome. It does not claim that acknowledgment proves an incident was resolved. Keep any later response task outside that metric unless you collect and define it separately.

Give observers a way to mark uncertainty and record the evidence gap. Use the unknown-outcome count defined above when the decision was reached but its result cannot be established.

Plan coverage instead of chasing one impressive result

List the support conditions you want represented: operating hours, routing teams, supported languages, employee categories, or outsourced arrangements. Choose only the conditions relevant to the assessment objective. Avoid collecting personal attributes that do not support that objective.

Allocate attempts across those conditions before execution. If the after-hours workflow is important, ensure it receives deliberate coverage rather than whatever calls remain at the end of the schedule. Record which conditions were excluded and why.

Treat a convenience sample as a description of the encounters actually observed. Do not present it as the probability that any employee will fail under any real attack. Scenario difficulty, routing, prior exposure, and participant selection should remain visible in the report as design limitations.

For repeated assessment rounds, preserve the scenario family and decision point when comparing outcomes. If you change the available support route or the mix of teams, document the change. A higher overall percentage may reflect a different assessment mix, so inspect the relevant subgroups before attributing improvement to training.

Include a legitimate service case

Pair the suspicious-request objective with an authorized service case. Ask the process owner what a properly verified employee should still be able to accomplish, and how the support team should explain a pending request. This gives the evaluation a way to notice when a proposed improvement makes ordinary recovery impractical.

Keep the legitimate-service observations in their own table. Record the approved route, completion state, escalation needs, and any delay within the chosen window. Do not combine them with the suspicious-request denominator simply to produce one attractive score.

For a rehearsal example, suppose the agent correctly pauses an unverified request but also cannot locate the approved route for a verified test employee. Preserve both observations. The first supports the desired boundary; the second creates a usability question for the process owner. Neither observation should erase the other.

Decide beforehand how the legitimate case will be matched to the tested condition. Use the same routing team and comparable operating window where practical, and record differences. If only one case reaches the relevant decision point, show that limitation rather than claiming a balanced comparison.

Prepare the assessment record before the first call

Use a restricted assessment record with a stable encounter identifier. Keep unnecessary personal information and any real credentials out of the analysis export. The following fields are proposed local fields, not a vendor schema.

Pilot the record with a few authorized rehearsal examples before using it for the full batch. Have two reviewers classify the same examples independently, compare their reasoning, and improve ambiguous definitions. Do not resolve disagreement by simply averaging incompatible labels.

When a definition changes, version it and reconsider affected earlier records. If the change would materially alter the interpretation, show both versions or restart the comparable set. Preserve enough history that the report explains how its final denominator was obtained.

Worked example: 100 attempts, 72 eligible encounters

Suppose the team schedules 100 attempts across daytime and evening support. Eighty connect. Eight connected interactions end before reaching the predefined decision point. The remaining 72 all reach it and have assessable outcomes, so the separate “decision reached, outcome unknown” count is zero. These counts are invented for the worked example.

The connection rate is 80 divided by 100, or 80% of scheduled attempts. Eligibility among connected interactions is 72 divided by 80, or 90%. Neither percentage measures whether the verification procedure held.

The primary decision result is 54 divided by 72, or 75% of eligible encounters. The remaining 18 eligible encounters did not meet the defined decision criterion. Reporting 54 divided by 100 would mix nonconnections and pre-decision exclusions into a behavior metric.

Describe the exclusions alongside the result: six daytime and two evening interactions were not eligible. In a real assessment, list the recorded reasons and assess whether they systematically removed a difficult condition. The fictional example supplies no additional evidence about why those interactions ended.

Now compare the conditions. Daytime is 32 divided by 40, or 80%. Evening is 22 divided by 32, or 68.75%. The difference is 11.25 percentage points. It identifies a question for the process review, not a causal explanation about the people working that shift.

Ask what differs between the workflows: the approved route, available decision authority, or routing destination. Use evidence to investigate those questions. Do not replace the observed gap with an unsupported narrative about motivation, skill, or workload.

Give handoffs their own denominator

In the same fictional set, 24 eligible encounters require an escalation handoff. Eighteen are routed to the designated destination, and 15 receive acknowledgment within the chosen observation window. These encounters are a subset of the 72, not an additional population.

All three figures can be correct while answering different questions. Use 15 out of 24 when describing the overall handoff objective. Use 15 out of 18 when examining what happened after routing. Label the denominator each time so a favorable conditional percentage does not conceal missing referrals.

The dataset leaves six required handoffs unrouted and three routed handoffs unacknowledged within the window. Assign the first gap to the process that initiates routing and the second to the receiving workflow for investigation. Do not assume the same remedy addresses both.

For the real assessment, retain evidence for receipt where possible. If routing is recorded but receipt is unknown, make that missing evidence explicit. A message marked sent and an owner accepting a task answer different operational questions.

Report time without hiding incomplete outcomes

Suppose the 15 acknowledged fictional handoffs have these elapsed times in minutes: 2, 3, 3, 4, 4, 5, 5, 6, 7, 8, 9, 10, 12, 15, 22. The middle value, the eighth observation, is 6 minutes.

Label that result precisely: median time to acknowledgment among the 15 acknowledged handoffs. Pair it with the count of nine required handoffs without acknowledgment within the window. A six-minute figure alone would leave those outcomes invisible.

Define the clock start consistently. For this exercise, use the moment the scenario reaches the required escalation decision, rather than whichever timestamp makes a record look fastest. Record the observation-window end and distinguish an unrouted handoff from one still awaiting a response.

If acknowledgment arrives after the window, preserve the later observation separately. Do not silently rewrite the original result. You can issue a follow-up result with its own cutoff while retaining the original comparison conditions.

Avoid assigning a zero-minute time to an unacknowledged outcome. In this analysis, it has no completed acknowledgment time. Report its status and elapsed observation window rather than forcing it into the completed-time calculation.

Agree on how findings will be used

Before execution, tell the assessment sponsors how individual observations will be handled, who can inspect them, and what information will appear in the wider report. Keep the analysis focused on the agreed decision and process, with access to detailed evidence limited through the organization’s established rules.

Give participants an appropriate feedback route after the exercise. Ask whether the procedure or reporting destination was inaccessible, and retain that context alongside the observation rather than replacing it. Use the debrief to identify a specific process improvement that can be tested again.

Review the observations before drawing conclusions

Have a second reviewer inspect a sample of outcome classifications and all consequential disputed records. Ask them to locate the supporting evidence, apply the metric definition, and identify any unresolved ambiguity. Record disagreements and their resolution.

Review the assessment as a process evaluation. If the primary decision was not met, determine whether the relevant procedure was clear, available, and usable under the tested condition. Assign an appropriate owner to investigate instead of treating the percentage as an explanation.

Also review the successful paths. Identify what evidence shows the agent completed the route and whether the handoff reached its destination. A good review should preserve the conditions under which the process worked so a proposed change does not discard a useful practice.

Choose a few findings and say how you will check that each fix worked. “Improve awareness” is not enough. “Publish the after-hours escalation destination and demonstrate acknowledgment in the next matched scenario” connects the improvement to the gap actually observed.

Design a retest that answers the original question

In this fictional evening group, each of the 32 eligible encounters belongs to a different participant. Suppose the same 32 participants each complete one eligible encounter in a matched retest after the process is revised. In the first round, 22 meet the decision criterion; in the retest, 27 do. The counts remain fictional and do not establish that any particular intervention works.

The first-round rate is 22 divided by 32, or 68.75%. The retest rate is 27 divided by 32, or 84.375%. The descriptive change is 15.625 percentage points, with seven improved classifications and two worsened classifications.

Report that as a result within the paired exercise. Without a design that isolates the intervention, do not call it the intervention’s causal effect. Participants may remember the earlier exercise, and changed conditions may affect the comparison. Document those possibilities rather than claiming the rate predicts a future real-world incident.

If participants differ between rounds, publish the composition and group-specific results instead of implying a paired comparison. If the scenario or scoring rule changes, explain what remains comparable. Keep the retest aligned to the original decision whenever that is the question stakeholders want answered.

Report the result and the decision it supports

Use a short summary that contains the question, population observed, result, limitations, and next action. For the fictional baseline, it could read: “The required verification decision was met in 54 of 72 assessable eligible encounters. No decision-reached outcomes were unknown; eight connected interactions ended before the decision. Twenty-four required handoff; 15 were acknowledged within the observation window. Evening and routing outcomes require process review. The sample describes the assessed conditions and is not a workforce-wide risk estimate.”

Attach the denominator dictionary and an action list with owners. State what leadership needs to decide: approve a procedure change, assign an escalation destination, or authorize a matched retest. These results do not estimate prevented breaches or financial savings.

Use the assessment-design worksheet to define each denominator and stopping point before scheduling the first call. Keep it with the report so another reviewer can reproduce the percentages and identify what remains unknown.