FP-Triage — LLM-powered alert triage

Give me your noisy queue — I'll show you which 4 alerts out of 1,096 were actually worth waking up for.

I build LLM-powered tools that auto-dismiss known-false-positive security alerts, so a cloud team lead who gets paged at 2am can trust their queue enough to book a call with me.

That 4-out-of-1,096 number is the rule, proven offline on a real tenant. Below, watch the agent apply that same rule live to a smaller, harder sample — most of it escalates, because that's what the rule is supposed to do when it doesn't recognize a device.

Book a call see the real run ↓

The Real Run Shipped

Checkpoint 3: the agent reads real Azure AD interactive sign-in events, classifies each as dismiss or escalate with an LLM (Groq / Llama 3.3 70B), gates any dismiss that isn't high-confidence back to escalate, and writes every decision to a real log file — no manual editing mid-run.

Rule validation (offline)

1,096 sign-ins → 4 first-seen devices

A deterministic device-history check, no LLM. Proves the rule is sound.

LLM agent (live, below)

22 sign-ins → 7 dismissed / 15 escalated

The actual agent, real API calls, real log file. Proves the agent implements the rule.

$ python triage_agent.py

Total sign-in events: 22

... 3 distinct (user, device) pairs, classified individually ...

Done.

7 dismissed, 15 escalated, out of 22 events.

Full log written to triage_log.csv

Real events processed

Auto-dismissed by the LLM

Escalated for human review

Where the "1,096" comes from: before building the agent, I ran a rule-based device-history check across 1,096 real sign-ins from a test tenant — the same (user, device) logic this agent's prompt encodes. It found that only 4 of 1,096 were genuinely first-seen devices worth escalating; the rest were known devices being flagged by a naive "unmanaged device" rule alone. That analysis is what told me the rule was worth automating with an LLM. The 22-event run above is the LLM actually doing it, live, end to end — not a re-statement of the same number.

How It Works

Has this exact device signed in for this exact user before?

Every alert gets checked against sign-in history, not treated in isolation.

↓

Yes, known pair

Dismiss — with the specific reason logged.

First-seen, unnamed, or cross-user reuse

Escalate — a human decides, not the model.

Here's the one question the whole thing hinges on: not "is this device unmanaged," but "have I seen this device for this user before."

Try It Shipped

Pick a scenario. This sends it through the real prompt my agent uses, live, to the same LLM — not a scripted response. Sample data, not a live production feed, but the exact same reasoning process.

Why Trust It Shipped

Two separate safety nets, not one.

First, category rules: unnamed devices, first-seen pairs, and a device that's known — but known for a different user — all route to escalate, never auto-dismiss, no LLM judgment call involved. Second, a confidence gate on top: even when a device is recognized, if the model's own confidence isn't "high," the dismiss is overridden to escalate rather than trusted. Both are live in the code, and the confidence gate has been proven to fire correctly on hand-built edge cases, not just designed in theory. During development, a live LLM outage (a provider's free-tier quota hit zero) also triggered the escalate-on-failure fail-safe for real, at length — so that path is proven under an actual failure too.

Honest gap: this test tenant has no confirmed malicious sign-ins, so there's no measured false-negative rate against real attacks yet — only against hand-built adversarial cases. That's a real limitation, not a hidden one.

About

I built this because I was the one reviewing Entra ID sign-in logs and Defender alerts by hand, three years into a cloud security role — the same "unmanaged device" flag, over and over, for devices I already knew. Most of it wasn't a real signal. This is the tool I wanted on my own queue.

What's Next In Progress

No blog yet — the build log lives in the GitHub repo's BUILD_LOG.md in the meantime, written as I go, mistakes included.
Next checkpoint: the confidence gate is live and tested against hand-built edge cases, but not yet against a larger real dataset — next step is running it against a bigger sign-in sample to see how often it actually fires.
Capstone work for the FlyRank ML internship will land here once reviewed.

Also Built

A separate project, different domain — included so you can see how I work when the question isn't security. FlyRank ML Internship capstone: ranking which of 30,000 underperforming SEO pages deserve a content team's limited review time.

74.0%

Precision@50, logistic regression

48.0%

Precision@50, hand-written baseline

51.7%

Base rate, for reference

Same honesty standard as the tool above: evaluated on a client-grouped holdout split, not a naive random split — the naive version inflates the same model's accuracy by 8.0 points and ROC AUC by 9.5 points. The full paper measures that gap directly instead of assuming it away, and states the model's real failure modes (30.5% false positives, 13.2% false negatives on the held-out set) rather than only the headline number.

Read the full paper ↗ View the repo ↗

Get In Touch

Send me your queue. I'll tell you what's actually worth your 2am.

Book a call