Search Calibration System: Convert Feedback Into Search-Plan Deltas
Make every review round improve the search instead of merely producing more candidates.
Last reviewed: 2026-09-06
Ask SourcingOS about this page
Inputs
- Frozen role artifact
- First 5–10 evidence-backed candidate dossiers
- Reason-tagged recruiter/hiring-manager feedback
- Source-lane and query provenance
1. Review candidates requirement by requirement
Ask reviewers to react to explicit evidence and missing evidence rather than a single overall score. Capture whether the issue is requirement fit, seniority, environment, geography, verification, compensation, or something else.
2. Separate one-person preference from a durable rule
One approved candidate is not automatically the new archetype. Look for repeated feedback before promoting a pattern into the role artifact.
3. Generate a visible before → after delta
Show exactly what changed: a strict requirement added or removed, a flexible criterion reweighted, an adjacent title approved, a donor-company hypothesis expanded, or a false-positive pattern suppressed.
4. Apply the delta to future lanes
Update search hypotheses and ranking behavior while preserving the original brief and the reason for the change.
5. Keep changes reversible
Allow approved learning to be edited or undone so an early calibration mistake does not silently contaminate every subsequent search.
What to measure
- Acceptance rate by slate
- Reason-tag frequency
- Search-plan deltas per review round
- False-positive reduction
- Time from first slate to stable criteria
Failure modes to avoid
- “Find more people like this one” with no explanation
- Converting every comment into a hard requirement
- Hidden preference learning
- No way to undo a learned rule