Can Your SOC See the Social Engineering Scenario You Just Ran?

The simulation ends and somebody asks, “Did the SOC detect it?” The answer may depend on which route you mean. An employee can report a suspicious call while an identity event travels through a collector and a rule. One route can work while the other stalls.

A security operations center, or SOC, needs separate evidence for each route: when the record arrived, where it went, who handled it, and whether those steps met the agreed criteria.

Define the two routes before the exercise

For the human-report route, identify where the employee submits the report, which queue receives it, who acknowledges it, and who decides the next action. A report button or support portal is the beginning of that route.

For the sensor route, name the event expected from the permitted technical action, where it should be collected, which analysis should evaluate it, and where any resulting alert should be delivered. Some scenarios produce no meaningful identity or endpoint event because they stop at a conversation. State that before testing instead of expecting a sensor to observe an action that never occurred.

NIST SP 800-61 Revision 3 recommends correlating information from multiple sources, providing adverse-event information to authorized staff and tools, and triaging and validating incident reports. Its profile also distinguishes analysis from incident declaration. Those principles support checking both routes. The acceptance test below is our proposed method, not a set of NIST-mandated timing thresholds. NIST SP 800-61r3: DE.AE-03, DE.AE-06, DE.AE-08, RS.MA-02

Agree on what the test must show

The fictional organization Alder Workshop runs a controlled support-impersonation exercise against a dedicated account. The scope includes one approved test authentication attempt and an employee report. It ends before application access or containment actions. The records and times below are invented; the download uses worksheet columns rather than vendor API fields.

The owners agree on three acceptance criteria for the 14:00–14:30 UTC review window:

1. The designated authentication event is available to the agreed analysis system within two minutes of its source time.
2. The agreed test condition produces an alert delivered to the security queue within three minutes of source time.
3. A human report reaches its agreed queue and receives analyst acknowledgment within five minutes of report submission.

These deadlines are exercise choices. The team should set its own criteria from the use case, staffing, and architecture. Record whether analysts know a test is happening. A coached or announced exercise can validate plumbing, but its response time should not be presented as an unannounced operational benchmark.

Open the fictional evidence packet

Download the fictional evidence CSV and review instructions. The CSV retains source time and observation time separately. An observation time describes when the reviewing system or person obtained the record; it does not replace the event’s own time.

At 14:30, no delivery of A-1 to the required security queue is established. The analyst found it while investigating H-1. That is a different outcome from the alert arriving at its configured destination.

Calculate the delay you actually measured

Source-to-collection delay is 14:03:05 minus 14:00:05, or three minutes. The two-minute collection criterion was missed by one minute. Do not start that calculation from the controller action at 14:00:00; that would measure a different interval.

Collection-to-alert delay is 15 seconds. Alert-to-general-queue delivery is 10 seconds. These short intervals do not rescue the delivery criterion because the queue was wrong. No delivery to security was confirmed in the review window, so there is no delivery time to calculate.

Report-to-acknowledgment delay is 14:04:00 minus 14:01:00, or three minutes, meeting the five-minute human-report criterion. Do not call this time to containment: no containment action was in scope. The distinction lets the team recognize a working reporting path while repairing collection and alert routing.

Keep the original records when linking the case

The case supplies three explicit relationships: C-1 references S-1, A-1 references S-1, and Q-1 references A-1. For the human route, H-2 and H-3 reference H-1. T-1 connects those branches using the designated account, exercise window, and controller’s approved action record.

In production, preserve the identifiers and corroboration your systems actually provide. An exercise identifier can help the controller reconcile results without being visible to analysts during the test. If you add special labels or rules solely for the exercise, disclose them; they change what the outcome demonstrates.

Keep original timestamps and timezones alongside normalized UTC values. Document clock uncertainty before comparing delays against a tight threshold. If the clock difference exceeds the margin you are measuring, resolve it before deciding whether the timing target was met.

Check that the test can distinguish outcomes

Include one approved positive control that meets the selected test rule’s condition and one ordinary event outside that condition. Specify both with the detection owner before execution. If both generate the same alert, investigate whether the test rule is too broad or an exercise marker is doing the work. If neither does, check the source and collection route before declaring the rule ineffective.

A human-report control should also include an ordinary support request so routing can be assessed without treating every message as an incident. Preserve the operational decision to triage rather than requiring analysts to label all exercise traffic malicious.

Introduce a missing-event case

Repeat the exercise on a second designated account. The employee report arrives, but no corresponding identity event is found in the analysis system during the agreed window. The controller confirms the action occurred. What failed?

Check the source before changing the detection rule. If a source record exists but the collector lacks it, inspect forwarding, access permissions, retention, filtering, and ingestion. If the source record itself is unavailable, establish whether the action should generate that record and whether collection was enabled. Licensing or retention limits can also determine which evidence the team can retrieve; verify those conditions for the actual products involved.

The conclusion should identify the search scope: “No matching event was found in the reviewed analysis system for account B during 14:00–14:30; source collection remains unresolved.” That is more useful than classifying every blank result as a detection-rule failure. The sign-in evidence guide develops the identity-specific interpretation.

Fix the collection delay and the wrong alert destination

The collector owner investigates the demonstrated three-minute ingestion delay. The routing owner corrects the destination for A-1’s rule and checks that security can receive the alert. The report-handling owner retains the successful route and tests a legitimate report as well as the exercise case.

Repeat the same permitted action under comparable collection and staffing conditions. Retain both attempts and their configuration differences. Use the assessment-design guide when deciding whether broader results are comparable; one repaired route does not justify a workforce-wide effectiveness percentage.

Our recovery incident case shows why misrouting matters during an incident. This packet gives the SOC a smaller acceptance test it can rerun before the next campaign. The finished result names which route worked, which stage missed its criterion, and which owner can demonstrate the repair.