Part 1 of the AI Detection Library Hygiene Series
Mike Wade, VP of Customer Success
Here is an uncomfortable exercise. Pick ten detections from your SIEM's detection library at random. Now answer one question for each: if the attack it was written to detect happened tonight, would it fire?
Not "did it fire when we wrote it." Not "does the query run without errors." Would it fire tonight, against tonight's data, and would the alert actually reach a human?
Most teams can't answer that. Not because they're lazy, but because a detection library has no built-in way of telling you it's broken. A web app that goes down throws 500s. A detection that goes down throws nothing. Silence is what a broken detection produces, and silence is also what a healthy environment produces. From the SOC chair, those two states are indistinguishable.
The problem is hard to spot manually but well suited to environment-aware AI: Gravwell’s Logbot AI can evaluate detections against the live environment they’re meant to protect and surface the ones that have quietly stopped working.
Almost no detection dies because someone deleted it. They die upstream, and they die in ways that produce zero errors.
The data source stops. An ingester gets misconfigured during a migration. An API token expires. A firewall rule change cuts off a log shipper. The scheduled search keeps running on time, keeps scanning its window, keeps finding nothing, and keeps reporting success. Your detection for suspicious Okta activity is now a detection for nothing, because the Okta tag hasn't received an entry in six weeks.
The log format drifts. Vendors change their JSON schemas without asking you. A field gets renamed, nested one level deeper, or split into two. Your query still parses, still runs, still matches the syntax it was written against. It just extracts an empty value now, and the comparison downstream never matches again. This one is nastier than a dead data source because the tag still shows healthy ingest volume.
The threshold rots. A detection for anomalous authentication volume was tuned when the company had 400 employees. It has 2,000 now. Or the reverse: a data source that used to be chatty got filtered at the ingest layer, and a threshold that once meant "unusual" now means "impossible." Thresholds encode assumptions about the environment, and the environment does not file a ticket when the assumptions change.
The pipeline breaks after the match. The query fires, but the flow that was supposed to send the alert references a webhook that got rotated, or a downstream integration that got decommissioned. The detection is alive. The alert is dead. Nobody is watching the gap between them.
Notice what all four have in common: no errors, no failed runs, nothing red on a status page. The detection library reports 100% operational while some real fraction of it is decorative.
The honest answer is that checking is miserable work.
To verify a single detection end-to-end, you need to confirm the tag is receiving data, the fields the query depends on still exist in that data, the logic still matches what it's supposed to match, the threshold still makes sense for current volumes, and the alerting path on the far side actually delivers. That's twenty minutes of careful work if you know the detection well. Multiply by a library of 300 detections and you get a project nobody staffs. It becomes an annual purple team engagement at best, and the purple team samples maybe a tenth of the library.
The result is a familiar pattern: a detection library that grows monotonically, because writing new detections is rewarding and auditing old ones is not, sitting on top of a decay process nobody measures.
This is where Logbot earns its keep, because its agents have access to the same powerful toolbox as a Gravwell user. Logbot isn't a chat assistant bolted onto a search bar. It has real access through the same interfaces you do: it can run queries, walk the scheduled search list, inspect flows and alerts, check ingest statistics per tag, and write test entries. Given that access, detection health stops being a project and becomes a job you delegate to your agents.
At Gravwell, we built this out as three agent functions.
The Audit Agent walks every scheduled search in the library and interrogates each one against the live environment. Is the tag it queries still receiving entries, and at roughly the expected rate? Do the fields the query extracts still exist in recent data, or is an extraction now returning empty because the schema moved? It runs each query's core logic over a trailing window and checks whether the intermediate results are plausible or whether some filter has gone from selective to absolute.
The critical distinction the agent can make, because it can query the actual data, is the one humans can't make at a glance: "this detection hasn't fired because the environment is clean" versus "this detection hasn't fired because the field it keys on stopped existing in March." The first is fine. The second is a coverage gap masked by a green checkmark.
The output isn't "review your library." It's a ranked list of detections that cannot currently fire, each with the specific reason and the query that proves it.
Every detection accumulates filters, and nobody ever measures what those filters cost. Logbot can measure that directly by querying the underlying data. For each detection, it runs the core logic over a trailing window twice: once as written, and once with the exclusions peeled back. The difference between those two result sets is what your tuning is eating. A detection whose exclusions filter 30% of candidate matches is tuned. A detection whose exclusions filter 99.8% of them has eroded to uselessness one ticket at a time.
The Audit Agent goes deeper than percentages. Exclusions frequently live in Gravwell resources, such as allowlists and lookup tables the queries reference, and those rot on their own schedule. Logbot cross-checks resource contents against the current reality: hosts on the allowlist that no longer resolve, service accounts that were decommissioned, the scanner subnet from an office you closed. Every stale entry is a hole an attacker can stand in, because an allowlisted identity is an invisible identity. The output is a per-detection tuning ledger: what's excluded, how much data it eats, which entries look dead, and which exclusions are broad enough that they'd hide the attack itself.
None of this says tuning is bad. The point is that tuning decisions were all made locally, one false positive at a time, and nobody has ever seen their cumulative effect. Logbot makes the cumulative effect a number, and once it's a number, it's a review item instead of an archaeology project.
For every threshold-based detection, the Audit Agent periodically re-derives the baseline from current data. What does the actual distribution of failed logins per user per hour look like this quarter? How does that compare to the hardcoded value in the query? Where the two have drifted apart, Logbot proposes a new threshold with the supporting distribution attached, and a human approves or rejects it.
That last part matters. The human keeps judgment and Logbot eliminates the tedious work of checking, pulling the data, and doing the statistics for a few hundred detections.
There’s a lot of attention on using AI to triage alerts. That matters, but triage starts downstream of the detection library. No triage assistant can recover an alert that never fired.
Detection hygiene is a particularly good fit for agents because the work is bounded, repeatable, and verifiable. The agent can inspect a finite library, test each detection against live data, and show the evidence behind every recommendation. Most of the work is read-only, while changes such as threshold adjustments can remain subject to human approval.
Run the sweep once and you're likely to find dead detections. Every team does. The number can be somewhere between embarrassing and alarming. The point isn't the one-time cleanup, though. The point is that after the cleanup, the sweep keeps running, and "would this fire tonight?" becomes a question your SIEM can finally answer.
Don’t wait for a missed alert to discover a detection has gone stale.