Giving every ESH-SOC analyst the same structure for investigating an alert

In short

In every SOC, junior tier 1 analysts are the first to investigate an alert. They either follow playbooks, which fit only the alerts they were written for, or learn to investigate on the job, so the quality of an investigation depends on who does it.
With TU/e, the ESH-SOC’s senior analysts wrote down which evidence to collect for any alert, in four stages and 13 steps, leaving the decision to the analyst. Every new tier 1 analyst at the SOC now learns this structure, and all analysts follow it.
In an experiment with ten ESH analysts, those who followed the structure had 2.5 times higher odds of classifying an alert correctly. On alerts caused by an attack, their odds were four times higher.

On the left, an alert from the study: a signature named ET JA3 Hash, Abuse.ch, Possible Dridex, with only 9 packets of traffic. A note says that analysts without the structure dismissed it: alerts of this kind are usually false positives, and the traffic was small. On the right, the four stages of the structure, each with what it asks and what it turned up for this alert. Relevance indicators: is the alert specific, recent and within what the SOC monitors for this customer. Additional alerts: the same IP address also appeared in a botnet command-and-control alert. Contextual information: a connection from a suspicious address to the host was established. Attack evidence: the activity is malware-related, so the impact on the customer would be high. At the bottom, the outcome: escalated to tier 2. — **The idea.** An alert from the study, investigated along the four stages. Analysts who followed them found the related alert in the second stage and escalated it; analysts without them dismissed it.

How tier 1 analysts investigate alerts

A security operations centre (SOC) monitors the systems of the organisations it serves and investigates anything suspicious. In a typical SOC, every alert first goes to a tier 1 analyst, who decides whether it is worth escalating to the more experienced tier 2 and 3 analysts. They investigate further and contact the customer. Most alerts are false positives, set off by harmless traffic. The few that point to a real attack need to be escalated.

Tier 1 analysts are usually junior, and across SOCs they tend to stay for a short time before being replaced by new ones. How they carry out an investigation generally follows one of two approaches, and both have drawbacks.

Following playbooks. Many SOCs write step-by-step procedures for specific types of alert, for example one for ransomware. Every type of alert needs its own playbook, kept up to date, which is hard to scale to the variety of alerts a SOC sees. A playbook also leaves the analyst little freedom: when it does not fit the alert at hand, a relevant part of the investigation can be overlooked.
Learning on the job. Without playbooks or any set structure, new analysts learn how to investigate along the way, from colleagues and experience. Investigations then differ from analyst to analyst. Two analysts can look at different evidence for the same alert and reach different conclusions, and nothing ensures a minimum quality for every investigation. When an alert is escalated, the evidence that comes with it differs too.

The ESH-SOC took this question to research: is there a general structure for investigating any alert that ensures every investigation covers the evidence that matters, while leaving the analyst free to judge it?

Four stages of evidence

The structure was developed by Leon Kersten, Tom Mulders, Emmanuele Zambon, Chris Snijders and Luca Allodi at TU/e, together with the ESH-SOC’s own analysts. A tier 2 analyst with more than four years of experience investigated ten alerts from the SOC and noted every piece of information they used. These were grouped into stages and put in order. Two senior experts, with more than 15 and more than 10 years of experience, then each tested the result on new alerts and sent feedback. After four rounds, all three agreed on 13 steps in four stages (Figure 1).

Four columns, one per stage, each listing its steps. Relevance indicators: how specific the signature is, how old the signature is, whether the alert is within the customer’s monitoring scope. Additional alerts: how often this alert triggered before and why, other alerts involving the same hosts around the same time. Contextual information: related logs, the volume and content of the traffic, what the targeted host is, and how the host behaves after the event. Attack evidence: which attack and tools are involved, who the attacker is, whether the attack succeeded, and how it relates to the customer’s use cases. An arrow runs from the first stage to the last, ending in a decision: escalate or not. — **Figure 1.** The structure: 13 steps in four stages, in the order an analyst works through them.

The stages move from the alert itself to the evidence around it. Is the alert relevant at all for this customer? Are there other alerts that point to the same attack? What do the logs and the targeted host show? Is there evidence of an actual attack, and did it succeed? Each step tells the analyst which information to collect. It does not prescribe a decision. At the end, the analyst judges whether the evidence is enough to escalate.

This addresses both drawbacks. The structure applies to every alert, so there is no procedure to write and maintain for each type, and the analyst remains free to follow the evidence where an alert leads. At the same time, every analyst collects the same kinds of evidence in the same order, which sets a minimum for every investigation, whoever carries it out.

Tested with ESH analysts

To test the structure without disturbing operations, the researchers rebuilt the SOC’s analysis environment. They took 2.5 weeks of alerts from the network of one of the SOC’s customers, and added alerts from ten malware attacks by replaying recorded attack traffic as if it took place in that network. From these, they drew 200 alerts, among them 22 from the attacks.

Ten new tier 1 analysts took part, recruited as they joined the SOC. The first five were trained in the structure during their intake and followed it; the next five received only the usual training. Each analyst classified 80 alerts as interesting or not. For 50 of them, a tier 2 analyst had established the correct answer beforehand.

ESH tier 1 analysts, half of them following the structure

200

alerts, 22 of them from injected malware attacks

2.5×

higher odds of a correct classification with the structure

4×

higher odds of correctly escalating an alert caused by an attack

Analysts who followed the structure were right more often: their odds of classifying an alert correctly were about 2.5 times those of analysts without it. The whole difference came from the alerts caused by attacks, where their odds of recognising the alert as worth escalating were four times higher (Figure 2). On harmless alerts, both groups did equally well. In the experiment, each analyst decided alone; in daily operations, tier 2 and 3 analysts back up the work of tier 1.

Bar chart of the odds of a correct classification for analysts who followed the structure, compared with analysts who did not, shown as a dashed line at 1 times. All alerts with a known answer: 2.5 times. Alerts caused by an attack: 4 times. Harmless alerts: 1 time, the same as without the structure. — **Figure 2.** Odds of a correct classification with the structure, compared with those without it (1×), on the 50 alerts with a known answer, each classified by two analysts per group.

The analysts’ written motivations show why. Without the structure, analysts often dismissed an alert after a single check. One dismissed a botnet alert because the traffic was below 50 packets, a threshold that applies to a different type of attack. With the JA3 alert in the illustration above, one analyst who followed the structure had the same doubt at first, but changed their mind on the evidence they collected:

“It’s weird. JA3 is never interesting for us.” (Tier 1 analyst, ESH, who escalated the alert)

What the structure brings to the SOC and its customers

Since the study, the ESH-SOC has fully integrated the structure into its daily operations.

Training. Every new tier 1 analyst learns the structure as part of their training. This gives each new analyst the same way of working from the start, without having to build it up from experience.
Investigations. All analysts follow the four stages when they investigate an alert, so the outcome depends less on who is on shift, and each analyst still decides what the evidence means.
Escalations. Tier 1 analysts write their escalations to tier 2 along the four stages, so tier 2 receives the same kind of evidence every time and can pick up the case faster.
Coaching. Because every investigation goes through the same steps, the SOC can see at which steps analysts struggle, and target coaching and training there.
Beyond network alerts. The structure has been extended to alerts from other sources than the network.

For the organisations the SOC monitors, this means that an alert caused by an attack is more likely to reach a senior analyst, and when it does, it comes with the evidence needed to act on it.