Quality Assurance
Built-in Quality Control
Ensure annotation accuracy with gold standard comparisons, inter-annotator agreement tracking, automated fraud detection, and a tiered annotator system that rewards consistency and precision.
Key Capabilities
Gold Comparisons
Embed known-answer gold questions into annotation sessions at configurable ratios. Annotators who fall below accuracy thresholds on gold items are automatically flagged, paused, or downtiered.
Agreement Scoring
Compute inter-annotator agreement in real time using Cohen kappa, Krippendorff alpha, and percent agreement. Low-agreement items are automatically re-queued to additional annotators for resolution.
Fraud Detection
Detect speed anomalies, repetitive response patterns, and random clicking through statistical analysis. Fraudulent sessions are quarantined and their annotations excluded from downstream profiles.
Tier Management
Annotators earn tier promotions based on gold accuracy, agreement scores, and throughput. Higher-tier annotators receive priority task routing and higher pay rates, incentivizing quality.
Usage
curl https://api.commandagi.com/v1/projects/proj_abc123/quality \
-H "Authorization: Bearer $COMMANDAGI_API_KEY" \
-G -d "window=7d"
# Response
# {
# "gold_accuracy": 0.96,
# "agreement_kappa": 0.84,
# "fraud_flagged": 3,
# "annotator_tiers": {
# "gold": 12, "silver": 28, "bronze": 45
# },
# "quarantined_sessions": 2,
# "total_annotations": 18420
# }