Testing and tuning

The fidelity contract#

Every rule must carry at least one record it must match and one look-alike it must not. A look-alike is the benign thing a careless version of the rule would catch: notrclone.exe, a -ExecutionPolicy switch, "Temp" in an installed program's folder name.

Tests check which records a rule matches; min_count is applied over a run.

Reading a failed check#

FAIL  EXEC-MY-RULE                         C:\signal-rules\mine.json
        test 1: must match, did not - condition "editor"; editor: false (process.name endswith "\notepad.exe"; the record's is "C:\Windows\write.exe")
        test 2 (a look-alike): must not match, did - condition "editor"; editor: true

Each selection is shown as:

• true;

• false, with the first field that did not match and what the record holds there;

• unknown, when the record lacks a field that depends on a policy.

An unknown modifier lists the valid ones; a missing key is named; a JSON mistake gives its line and column.

Tuning on real hosts#

1. Run it on clean hosts first

   Run your rule across hosts you know are clean - a few workstations, a server, a domain controller. Every match there is a false positive.

2. Narrow it on what Windows records

   Exact paths anchored at the drive, a SID instead of a name, a logon session, a signature. Never a broad exclusion an attacker could use.

3. Add each false positive as a look-alike test

   So the rule can never regress.

4. Re-grade if needed

   If benign matches remain, the rule is medium or low - or it is context, a column of a table, not a signal.

See the rules a run would use#

signal rules -m Executions -rules C:\signal-rules

Lists every rule with its origin: built-in, or your file. In Signal Console, the Rules page shows the same, with each rule's outcome on each host.