Classification Consistency & False Positive Reduction

When Claude classifies or categorizes items (like severity ratings), inconsistency is a common problem. High false positive rates in some categories erode trust across ALL categories.

Root Cause of Inconsistency

• Ambiguous category definitions
• No concrete examples for each category
• Relative rather than absolute criteria

Solution: Explicit Criteria with Examples

1. Clear definition for each classification level
2. Concrete code/content examples for each level
3. Absolute criteria (not relative to other items in the batch)

False Positive Trust Erosion

When automated review produces high false positive rates in certain categories (e.g., style at 52%, docs at 48%), developers start dismissing even accurate findings. The fix:

1. Temporarily disable high false-positive categories
2. Keep high-precision categories running (security at 8%, correctness at 8%)
3. Improve prompts for disabled categories
4. Re-enable only when precision meets threshold

Anti-patterns

• "Rate severity relative to other issues" (causes inconsistency across batches)
• Confidence scores (developers who lost trust won't trust self-reported confidence)
• Uniform strictness reduction (hurts high-precision categories unnecessarily)