Why “Search and Delete” Won’t Get The Job Done

Introduction

Here at IRONSCALES, we often speak with organizations about their messaging incident response process. And we’ve learned that all too frequently, the primary response tools are 1) an email search and then 2) a deletion operation on the messages identified from that search. We call this a “search-and-delete” incident response strategy. Some time ago, as an email admin, I executed this strategy myself. My company had adopted this strategy for a few different reasons, but primarily because it was the best option given the tools available at the time. And though there are much better strategies and tools available today, countless clients describe the search-and-delete strategy to us daily, while lamenting that it fails to meet the intended goal of eliminating all malicious emails from a single incident from the company’s email environment.

When you don’t know what you don’t know

Typically, clients understand some challenges in their email environments very well, but there is one key area where most have a blind spot: the “unknown unknown” email data. Admins and analysts can’t be everywhere, they won’t see everything, and they can only address what they are able to reasonably perceive or predict. As a result, the “unknown unknown” materializes within a search-and-delete strategy regularly because this strategy can only be as accurate and complete as the person’s ability in creating the search.

The search will focus primarily on two categories of information: 1) information the searcher is explicitly aware of and 2) the information the searcher can be reasonably be expected to extrapolate or predict based on common patterns. If they don’t know of a malicious message and they cannot come up with a search that will find that message, they’ll be in a tough spot. They can either create a search that is so broad it starts to categorize good email messages in with bad email messages or use a huge amount of time to continuously iterate on search criteria when they don’t know what they’re searching for in the environment. Neither option is good, so typically the admin will choose to search what they know via a small number of predictive searches limited in scope. Unless a powerful motivating force exists (e.g., lawsuit, data breach, etc.), there can’t be a companywide search on every email message because there isn’t enough time or resources, but by then the damage is done. Attackers understand this dynamic and exploit it daily, with polymorphic attacks and by launching attacks with incredible volume. They rely on the knowledge that they can generate far more phishing mails than your organization can identify, respond to, and remediate in real time.

While we may clearly see the gaps in this strategy it can be a challenge to articulate to admins for the same reasons mentioned above. How can an administrator or analyst be concerned about something for which they have no clear visibility? It would be analogous to asking how can you know the corner office is on fire if you’re in a different part of the building, you can’t see the flames, smell the smoke, or feel the heat, and you have no smoke alarms? Your only option in this building fire hypothetical is to walk the building continuously and check areas where you suspect fire might be likely. This approach becomes very risky because you don’t have the resources to do this all day, you don’t know for certain you’re checking the right areas, and it’s quite possible a fire will break out where you least expect it. The failure becomes a failure of imagination. 

Real-world scenario: The shortcomings of search-and-delete

Unfortunately, humans don’t scale incredibly well, but we have tools that do and help identify the “unknown unknowns.” Below is an example of a real client interaction showing how search-and-delete can breed a false sense of confidence, and which provides a good example of how attackers take advantage of your organization being spread too thin.

IRONSCALES: Help me understand what normally goes on and how your process flows when you encounter a phishing mail.

Client: Well, first we get alerted to the message in a few different ways but mostly users report the message to us via email. Then we look at that individual message, conduct some research, and finally decide whether we need to take some action. Then we create a search in our environment and tweak the search until we have all the messages. Then we run a script to remove them from inboxes so users can't interact with them.

IRONSCALES: Is that how you handled the phishing email you saw today? How long does that take?

Client: Yep, pretty much. I don't know, it took maybe three or four hours, but we’re good now. We've removed all those messages, so it isn’t a problem now, but we just want to be able to see these kinds of things and deal with them more proactively.

IRONSCALES: Makes sense. Do you have a copy of that mail? I'm curious about it, after we integrate in your environment, why don't we submit one of these mails? See what our system could have told you about it.

Client: Sure, I have the one I was originally sent, I used it to design my search.

Fast-forward five minutes, after we’ve integrated and set up the Report Phishing mailbox. We connect, find the mail, and submit it to the mailbox. Our tool recognizes this as phishing, but the more interesting discovery was in our list of affected mailboxes in the incident. We found a record of 38 messages; of these, 21 were not found, but 17 of them were quarantined. So, what did that mean?

It meant the client was able to search and delete 21 of the 38 malicious messages, but the attacker had changed the metadata around the message to avoid their search. Until our tool acted, 17 malicious messages were sitting in user mailboxes. Our tool used auto-clustering technology to find messages like the baseline attack, quarantining these messages within seconds. This is a common situation, and unfortunately it leaves organizations with the false impression the threat is gone from their environment.

This story is real and, unfortunately, one we hear all too often here at IRONSCALES. The search-and-delete strategy is based on searches that we think are accurate when we're designing them. I ran into this myself as an administrator. As an administrator you attempt to find a match on the subject, a Message ID, an IP address, or a sender name. You do the best you can with the tools that you have, but most of today’s tools have not kept up with an environment regularly encountering volume polymorphic attacks. This is why we believe that search-and-delete is ineffective; it is a strategy that can only be as good as the person writing the search. The strategy and commoditized tools provided to execute that strategy were designed almost a decade ago. At that time, we were focused on virus attachments and needed a device in front of our mail flow to introduce an artificial delay to allow the device time to conduct signature-based scanning. We weren’t dealing with a phishing-as-a-service economy, phishing kits that cost as little as ten dollars (including 24/7 support!), or business email compromise at massive scale. This strategy is inherently backward-looking because it literally seeks to define a pattern based on yesterday’s attacks. 

Embrace change to lessen risk

It’s true this strategy was once an effective and reasonable one. At that time, however, the threat did not morph daily, phishing volumes were much lower, and the industry hadn’t developed into its own self-sustaining economy for attackers. Today, solving the problem of email phishing is inherently a speed game at scale. We know from industry statistics that to have even a chance of keeping users from interacting with a malicious message, we need to detect and respond before eighty-two seconds have elapsed. Anything slower and the level of risk increases significantly. This challenge puts email administrators and SOC analysts in an incredibly tough position, lacking the right tools and with the expectation to know the unknowable.