Multi-modal preference elicitation
Capture Preferences
Collect structured preference data through binary labels, pairwise comparisons, dimension ratings, and free-form feedback. Target specific demographics and enforce quality controls at every step.
Key Capabilities
Binary Labeling
Present frames to annotators for simple accept/reject labeling. Optimized for speed with keyboard shortcuts and batch workflows that scale to thousands of annotations per session.
Dimension Ratings
Define custom rating dimensions such as aesthetics, usability, or brand alignment. Each frame receives granular scores across multiple axes, building rich preference profiles.
Pairwise Comparisons
Show two frames side-by-side and ask which is preferred. Comparison data produces stronger signal than absolute ratings and feeds directly into reward model training.
Quality Control
Embed gold-standard questions, track inter-annotator agreement, and flag inconsistent responses automatically. Ensure every data point meets your accuracy threshold before it enters a profile.
Usage
curl -X POST https://api.commandagi.com/v1/sessions \
-H "Authorization: Bearer $COMMANDAGI_API_KEY" \
-H "Content-Type: application/json" \
-d '{
"project_id": "proj_abc123",
"type": "comparison",
"frames": ["frame_001", "frame_002"],
"dimensions": ["aesthetics", "usability"],
"quality_control": {
"gold_ratio": 0.1,
"min_agreement": 0.8
}
}'