Somewhere in building a SIEM and an Active Directory attack lab separately, I kept hitting the same annoying question: how do I actually know a detection rule works? I could write the rule, read the code, feel confident about it — but “I read the code and it looks right” isn’t the same as “I ran a real attack and watched it fire.” So I built Sentinel BAS: a platform that runs real Atomic Red Team attack techniques against the lab and automatically scores whether each one got caught, in time, by which rule.
The number at the end: 70% coverage. 7 Detected, 0 Delayed, 3 Missed, 0 stuck Pending.
Getting there took longer than the number suggests, and the debugging taught me more than a clean first-try run would have.
The scoring logic, briefly
For every technique: log the exact start time, run the attack, log the end time, then check whether a matching alert appeared — matched by MITRE technique (hierarchy-aware, so a test for T1003 correctly credits an alert tagged T1003.001), scoped to the same host, and scored by timing. Within 60 seconds, Detected. Within 5 minutes, Delayed. Nothing by then, Missed.
I want to be specific about one design decision: a single scoring check right after a test ends can’t tell the difference between “this genuinely missed” and “this hasn’t had time to arrive yet.” So Missed isn’t finalized until the full grace window has actually elapsed in real time — otherwise every slightly-slow detection would get mislabeled as a failure.
The bug that took the longest to find, and it wasn’t in the detection logic at all
My Kerberoasting rule checks specifically for RC4 encryption — the exact weak-encryption condition real Kerberoasting attacks target. I sent it real Kerberos ticket events and it kept coming back Missed. I checked the rule logic. Fine. I checked whether Sysmon was even generating the events. It was. I was ready to believe the detection engine itself was broken.
It wasn’t. The actual bug was one field short of the message text the forwarder builds for the SIEM. TicketEncryptionType — the literal field that holds the value 0x17 my rule searches for — was simply never in the list of fields the forwarder pulls out of a raw Windows event. So the alert arrived with the username, the service name, everything except the one field that mattered. The rule was structurally unable to ever fire via the forwarder, no matter what actually happened on the wire. Not a logic bug. A field-extraction gap, invisible until I actually traced a real event field by field instead of trusting that "the message field probably has what I need."
The forwarder was also flooding itself with its own noise
Once that was fixed, I found a second, unrelated problem: 17,000+ logs, and most of them were the forwarder logging itself. Its own PowerShell polling process launching every 10 seconds. Its own HTTP call to the SIEM being seen by Sysmon as a network connection, then dutifully forwarded right back. The tool was watching itself and reporting what it saw about itself, forever. I added a filter that recognizes this specific self-generated pattern and drops it before it ever reaches the SIEM — verified carefully against real attack payloads afterward, since a filter that’s too aggressive and accidentally hides real attacks is a worse bug than the noise it’s meant to remove.
What “Missed” actually means, and why that matters more than the score
Three of ten techniques came back Missed, and I want to be honest about what that means instead of hiding it: none of them are a real detection gap. The Kerberoasting Atomic test’s default LDAP-based targeting failed in my lab, so it fell back to requesting tickets for generic built-in system SPNs using .NET’s Kerberos client, which negotiates modern AES encryption by default — not the RC4 condition against custom service accounts that real Kerberoasting, and my rule, are actually built to catch. Two others failed for the more mundane reason that this is a genuinely internet-isolated lab and some Atomic sub-tests need tools they can’t download.
That distinction — did the simulation fail to reproduce the real technique, versus did my detection actually fail to catch something it should have caught — is the actual skill this project is meant to demonstrate. A tool that reports 100% because nothing hard was ever really tested is less trustworthy than one that reports 70% and can explain exactly why each miss happened.
What’s actually in it now
35 detection rules, a correlation engine that clusters related alerts into incidents automatically, SOAR-style response with hard safety guardrails, and now a Detection Validation Engine that turns “I think my rules work” into a number you can defend. Full code, the bas_runner.ps1 automation script, and the coverage dashboard are on GitHub.