Track preference evolution
Analytics & Quality
Monitor how preferences shift over time, measure annotator consistency, and drill into demographic breakdowns. Export raw data or connect dashboards to your own BI tools.
Key Capabilities
Trend Analysis
Visualize how preference scores evolve across commits, branches, and time windows. Detect drift early and correlate shifts with specific annotation batches or annotator cohorts.
Quality Dashboards
Real-time dashboards display inter-annotator agreement, gold accuracy, response time distributions, and consistency scores. Filter by project, annotator, or demographic segment.
Demographic Breakdowns
Slice preference data by annotator demographics to uncover systematic differences. Understand how age, region, or expertise level affects aesthetic judgments across your content.
Exports & Integrations
Export annotation data as CSV, JSON, or Parquet. Push metrics to Datadog, Grafana, or any webhook-compatible dashboard. Schedule recurring exports for downstream ML pipelines.
Usage
curl https://api.commandagi.com/v1/projects/proj_abc123/metrics \
-H "Authorization: Bearer $COMMANDAGI_API_KEY" \
-G -d "window=7d" -d "group_by=annotator"
# Response
# {
# "window": "7d",
# "total_annotations": 12480,
# "agreement_kappa": 0.84,
# "gold_accuracy": 0.96,
# "annotators": [
# { "id": "ann_01", "count": 3200, "kappa": 0.91, "gold_acc": 0.98 },
# { "id": "ann_02", "count": 2980, "kappa": 0.82, "gold_acc": 0.95 }
# ]
# }