Phishing Simulations: What a Drill Involves

1. What a phishing simulation is

A social engineering drill, usually called a phishing simulation, is a commissioned security test with a fixed scope. Working from an agreed recipient list and agreed scenarios, the consultant sends mail that looks close to a real attack and records what people do with it. The links point to a logging page built for the drill. No malware is delivered and no data is taken. Anyone who clicks normally lands on a short page telling them what just happened and which details would have given the message away next time.

What it tests is people and process, which makes it the complement to a vulnerability scan rather than a substitute. A scan tells you which holes exist on your machines. A drill tells you under what conditions somebody opens the door themselves. Measure both once and you know which side your next security budget belongs on.

How the data gets handled has to be settled up front too. The simulated login page can be configured to log only the fact that somebody submitted the form, without keeping what they typed. All measurements and raw records belong to the commissioning organisation, and the consultant deletes them at the end of the agreed retention period. These are not points to explain after the fact. They belong in the paperwork before anyone is engaged.

2. Why human judgement decides more than the technical defences

A mail gateway blocks known bad domains, known attachment patterns and known sender blocklists. An attacker registers a domain that week, swaps the attachment for a cloud storage link, and the filter rules no longer match. A plain-text message with no payload at all is harder still to catch by rule, because there is nothing technical in it to compare against. Once the message reaches the inbox, the remaining defence is the few seconds of judgement the recipient gives it.

Trace an incident backwards and the starting point is often a single click, or one set of company credentials typed into a convincing login page. That link in the chain is usually the cheapest part of the whole security budget, and almost nobody has ever measured what state it is currently in.

Most organisations run their annual security awareness session, tick the box, and end up with no before-and-after figures and no idea which department took it in. With no measurement there is nothing to compare and no basis for deciding who needs attention first. That is the value of a drill: it turns staff security awareness from a subjective impression into a number that departments can be compared on.

3. What one drill involves, start to finish

A standard drill runs about two to four weeks from scoping to report, with the sending period usually somewhere between a few days and two weeks. There is nothing mysterious in the process. What decides the outcome is how carefully the preparation is done. If the list is not cleaned, the gateway is not opened, or the internal announcement is worded badly, the numbers at the end are distorted and the report is not worth reading.

The sending itself has one detail worth getting right: split the population into several waves rather than sending to everyone at once. Send it all in one go and the first person to say "that one is fake" by the coffee machine makes the rest of the numbers meaningless.

Then comes the analysis and handover. Four figures get pulled together, meaning opens, clicks, submissions and reports, broken down by department and grade, delivered as a report plus a session to walk through it. Do not skip the session. Numbers only turn into action when somebody explains them.

4. The metrics to measure, and how to break them down

The core output of a drill is a handful of rates. Read any one of them alone and it is easy to draw the wrong conclusion. Put together, they show which stage the problem sits at.

Beyond the headline figures, only a breakdown by department and grade shows where the weakness actually sits. Finance staff tend to be sharp on fake invoices. Sales people, who read external mail all day, are often less alert rather than more. Managers get high volumes and read fast, so their click rate frequently runs above the rank and file. Whether a role deals with customers regularly is worth pulling out as its own group.

5. Scenarios that work, and ones to avoid

Scenarios have to resemble the mail this company genuinely receives, otherwise the difficulty means nothing. Test people with a typo-ridden message from an unknown free mail address and the resulting figure tells you nothing you can use.

Four families of scenario cover most drills. Fake system notices (mailbox full, password about to expire, a shared document waiting for approval) test the reflex click on routine operations. Fake invoices and payment requests (invoice notifications, changed bank details, purchase order confirmations) are aimed mainly at finance and procurement. Fake HR mail (annual review forms, insurance record updates, handbook sign-offs) covers everybody. The fake login page is what produces the submission rate: it looks like an internal system and shows the drill explanation page the moment anything is submitted.

A single drill normally mixes two or three difficulty levels, so the report can separate "anyone should have spotted that" from "even the IT staff had to look twice".

6. After the report: turning the numbers into a training plan

The report exists to order the training. Whichever department has the high submission rate gets the first small-group session. Whichever scenario type concentrated the clicks is the one the training should spend its time on, working through what would have given it away.

Say all of this before the drill, and put it in the written drill policy. When staff know the results are used to schedule training, the report rate goes up. Run it the other way, and one public dressing-down of somebody who clicked is enough to stop anyone reporting again. That costs you the most valuable metric you have. Internal presentations should show aggregate figures only, meaning rates and trends by department and grade.

Individual open and click records can technically be traced back to a named recipient. That data belongs to the commissioning organisation, and in practice access should be limited to a named contact, with the permitted uses written into the authorisation document in advance.

Timing matters for the follow-up as well. The group that submitted credentials goes first, ideally within two weeks of the drill closing, while the memory is fresh. Retest the same group three to six months later and compare the movement. That comparison is far more use than benchmarking yourself against an industry average.

7. Authorisation, personal data, frequency and budget

This kind of exercise sits in a sensitive place, so the boundaries need drawing first and they need drawing on paper. The written authorisation is signed by someone who can commit the organisation, and one line saying "we agree to the test" is not enough.

On cadence, run a baseline drill first, then repeat quarterly or half-yearly, adjusting difficulty gradually so the figures stay comparable. The first set of numbers is usually poor. That is not a bad outcome, that is the baseline.

On budget, a social engineering drill starts at NT$30,000 for up to 300 people, covering scenario design, sending, measurement, the report and one session to present the results. Larger populations, multilingual scenarios, or work that pairs the drill with a physical exercise are quoted separately.