AI · DEFENSE

Using LLMs to catch phishing.

Classic filters match known-bad strings. Language models can read intent. Here is how that helps on defense, and where it quietly falls down.

July 10, 20268 min readDaniel A. & Óscar S.

Phishing has always beaten simple filters for one reason: the attacker writes new text every time. A rule that blocks the word "verify your account" is trivially dodged by rephrasing. This is exactly the kind of problem large language models are suited to, because they score meaning rather than exact strings, and that makes them a genuinely useful layer in a modern defensive email-security stack.

This article is about detection: using models to protect your own users. It is not about writing more convincing lures. Everything below assumes you are defending an inbox you are responsible for.

Why keyword filters miss modern phishing

Traditional detection leans on signals like known-bad sender domains, blocklisted URLs, and suspicious keywords. These catch high-volume, low-effort spam well. They struggle with targeted attacks: a well-written message from a freshly registered domain, with no misspellings and no blocklisted link, sails straight through. The tell in those messages is not a string: it is the intent: manufactured urgency, an unusual request to move money or credentials, a mismatch between who the sender claims to be and how they write.

What the model actually adds

A language model reads the message the way a cautious human would and flags the semantic red flags:

Intent classification: is this message trying to get the reader to authenticate, pay, or install something, and does that request fit the apparent relationship?
Tone and pressure analysis: phishing leans hard on urgency and consequence ("account will be closed today"). Models pick this up reliably.
Brand-impersonation checks: does the message imitate a known brand's voice while the sender domain and links point somewhere unrelated?
Context mismatch: a "CEO" emailing from a personal address asking a junior employee to buy gift cards is a pattern a model recognises even when every individual word is innocent.

Where an LLM fits in the pipeline

You do not replace your existing filters with a model – you add the model as a later, more expensive stage. A sensible layout:

Stage	What happens
Stage 1 – cheap filters	Domain reputation, SPF/DKIM/DMARC checks, URL blocklists. These reject obvious junk for near-zero cost.
Stage 2 – model scoring	Messages that pass the cheap checks but land in a grey zone get sent to the model for an intent-and-tone assessment. You pay per call, so you only spend it where the cheap layer was inconclusive.
Stage 3 – human review	Anything the model scores as high-risk but not certain goes to a security analyst or a "report phishing" workflow, rather than being silently deleted.

Keeping the model in the middle, not the front, controls both cost and blast radius: a wrong model call quarantines a message for review, it does not delete mail on its own.

The limits you must design around

Anyone selling you an LLM phishing detector as a silver bullet is overselling. The real constraints:

False positives have a cost. Flag too many legitimate emails and users learn to ignore the warnings, or you block a real invoice. A detector that cries wolf is worse than none.
Attackers adapt. Once defenders use models, attackers test their lures against models too. Detection is a moving target, not a solved problem.
Privacy is a first-class concern. Sending corporate email to a third-party model API means sending potentially sensitive content off-site. For many organisations that is a reason to run the model locally or under a strict data-processing agreement.
Prompt injection cuts both ways. A detector that reads email is itself reading attacker-controlled text. A crafted message could try to talk the classifier out of flagging it – see Prompt injection, explained.

The CISA guidance on avoiding phishing still applies underneath all of this: technical detection is one layer, user training and a fast reporting path are the others. The model makes the technical layer smarter; it does not remove the need for the rest.

Measuring whether it actually works

A detector you cannot measure is a detector you cannot trust. Before rolling an LLM stage into production, put numbers on it against a labelled set of real (anonymised) mail – known phishing and known legitimate messages both:

Precision: of everything the model flagged, how much was genuinely phishing. Low precision means noise, and noise trains users to click through your warnings.
Recall: of all the phishing that arrived, how much the model caught. Low recall means a false sense of safety.
The trade-off is a dial, not a fact. Push the threshold to catch more phishing and you flag more legitimate mail; loosen it and the reverse. Choose where to sit deliberately, based on what a miss versus a false alarm costs your organisation.

Re-measure after every change to the prompt, the model, or the surrounding filters: the same discipline a red team applies to a model, turned inward on your own detector. Track the numbers over time; a slow drift in recall is often the first sign attackers have adapted to whatever your model keys on.

A note on scope. This is defensive security education – using models to protect systems and users you are responsible for. It is not a guide to crafting phishing messages or evading someone else's filters.

Used well, an LLM turns your phishing filter from a string matcher into something closer to a wary reader. Used badly – as a single unaccountable gate with no human in the loop, unmeasured and untuned – it just fails in new and more confident ways. The value is in the layering, and in measuring the layer you added.

Frequently asked questions

How do LLMs detect phishing that keyword filters miss?

Keyword and blocklist filters catch high-volume, low-effort spam but miss well-written, targeted messages from freshly registered domains. LLMs score meaning rather than exact strings, so they can flag manufactured urgency, unusual requests to move money or credentials, and a mismatch between a claimed sender identity and how the message actually reads.

Where does an LLM fit in a phishing detection pipeline?

As a later, more expensive stage, not a replacement for existing filters. Cheap checks like domain reputation and SPF/DKIM/DMARC reject obvious junk first; messages that land in a grey zone are then sent to the model for intent-and-tone scoring; anything scored high-risk but uncertain goes to human review rather than being silently deleted.

What are the limits of using LLMs for phishing detection?

False positives train users to ignore warnings or block real mail; attackers adapt their lures once they know defenders use models; sending email content to a third-party model API raises privacy concerns that may require running the model locally; and a detector that reads attacker-controlled email text is itself a target for prompt injection.

How do you measure whether an LLM phishing detector actually works?

Test it against a labelled set of real, anonymised mail and track precision (how much of what it flagged was genuinely phishing) and recall (how much actual phishing it caught). The trade-off between the two is a deliberate choice, and both numbers should be re-measured after every change to the prompt, model, or surrounding filters.