Feedback Loop Design & Dismissal Pattern Analysis

Production AI systems need feedback loops that systematically improve output quality over time by tracking patterns in failures and dismissals.

The detected_pattern Field

Add a detected_pattern field to structured findings that records which code construct triggered the finding. When developers dismiss findings, you can analyze which patterns produce false positives.

Example

{
  "finding": "Potential null pointer dereference",
  "severity": "high",
  "detected_pattern": "optional_chaining_missing",
  "location": "src/api/handler.ts:45"
}

If developers consistently dismiss optional_chaining_missing findings, you know to refine the prompt criteria for that pattern.

Systematic Improvement

1. Track dismissals: Record which findings developers dismiss
2. Aggregate by pattern: Group dismissals by detected_pattern
3. Identify low-precision patterns: Patterns with >30% dismissal rate
4. Refine prompts: Add explicit criteria or few-shot examples for those patterns
5. Re-enable: Test updated prompts before re-enabling in production

Connection to False Positive Management

This is the data-driven version of the disable-fix-re-enable strategy. Instead of disabling entire categories, you disable specific patterns within categories based on dismissal data.