Retry-with-Error-Feedback Pattern
CoreImplement validation, retry, and feedback loops for extraction quality · Difficulty 3/5
0%
validationretryerror-feedbackextraction
Prerequisites
Explanation
When extraction or generation produces invalid output, a retry-with-error-feedback pattern guides the model toward correction by appending specific validation errors.
The Pattern
- First pass: Model extracts/generates output
- Validate: Check output against schema and semantic rules
- On failure: Send a follow-up request including:
- The original source document
- The failed extraction/output
- Specific validation errors (not just "try again")
- Model corrects: With explicit error context, the model fixes the specific issues
When Retries Work vs Fail
| Scenario | Retry Effective? | Why |
|---|---|---|
| Format mismatch (date format wrong) | Yes | Model can reformat |
| Structural error (wrong nesting) | Yes | Model can restructure |
| Semantic error (values don't sum) | Yes | Model can recalculate |
| Information absent from source | No | Model will fabricate or hallucinate |
| Data in external document not provided | No | Model has no access to the information |
Key Distinction
Schema syntax errors are eliminated by tool use. Semantic validation errors (values don't sum to total, data in wrong fields) persist even with strict schemas and require validation + retry logic.
Self-Correction Validation Flows
- Extract
calculated_totalalongsidestated_totalto flag discrepancies - Add
conflict_detectedbooleans for inconsistent source data - These computed fields enable automated validation without human review.
Key Takeaways
- Append specific validation errors to the retry prompt -- not just 'try again'
- Retries work for format/structural/semantic errors but fail when information is absent from the source
- Tool use eliminates syntax errors; semantic errors require validation + retry logic
- Extract computed validation fields (calculated_total, conflict_detected) for automated checking
Related Concepts