AI Evaluation Rubric Auditor
Mercor · 100% remote · Contract · Posted
- Pay
- $60–85/hr
- Location
- Worldwide
- Languages
- English
- Hours
- Flexible
- Openings
- Not listed
- Level
- Expert
Summary
Audit AI evaluation tasks and grading rubrics to determine if failures are fair, and provide feedback to reviewers on prompt and rubric issues.
How the pay compares
At the middle of its range, this job pays $73 an hour.
What you'll do
- Audit complete tasks and grading rubrics
- Determine if rubric failures are fair
- Check information availability and rubric accuracy
- Evaluate if valid answers would be marked wrong
- Verify rubric structure per guidelines
- Provide feedback to reviewers
Requirements
- Have substantial experience with LLM eval rubrics
- Provide feedback to writers or reviewers
- Maintain consistent availability
- Read long agent transcripts to find errors
- Write precisely and verify data
Skills
- LLM Evaluation
- Rubric Design
- Data Verification
- Feedback
Full description
The work
We are testing AI agents on realistic shopping and delivery tasks. Your role as a Project Auditor is to audit the work done by our Project Reviewers. You will audit complete tasks (prompt and grading rubric) and determine if their rubric failures are fair:
- Was the information available?
- Does the rubric accurately grade what the prompt asked for?
- Would a valid answer have been marked wrong?
- Is the rubric structure correct per the guidelines?
This role includes providing feedback to reviewers regarding prompt and rubric issues.
Must-haves
- Substantial experience writing or reviewing rubrics for LLM evals. Please list the projects in your application.
- Experience providing feedback to writers/reviewers
- Consistent availability
- Can read a long agent transcript and find where it went wrong.
- Precise writing, and you check the data instead of trusting the reference answer.
Similar jobs
Security EngineerCybersecurityLarge Language ModelsAI Research+2$80/hr
🌍 Worldwide
Software Developers: Interview on AI Assistant IntegrationsAI Assistant IntegrationsSoftware EngineeringBackend DevelopmentFull-Stack DevelopmentSystem Architecture+4$85
🇺🇸 US
Strategic Project Lead, AI & Data OperationsTechnical Fluency & Systems ThinkingTechnical Program & Pipeline ManagementData Quality & Analytical JudgmentTechnical Stakeholder Management & Communication+3$80–90/hr
🌍 Worldwide
Strategic Project Lead, U.S. GovernmentTechnical Project Management & ExecutionData Pipeline Design & OperationsRequirements Gathering & Stakeholder ManagementCross-Functional Leadership & Operational Delivery+3$80–90/hr
🌍 Worldwide