Skip to content
AI Trainer Jobs
Companies

AI Evaluation Rubric Auditor

Mercor · 100% remote · Contract · Posted

Pay
$60–85/hr
Location
Worldwide
Languages
English
Hours
Flexible
Openings
Not listed
Level
Expert

Summary

Audit AI evaluation tasks and grading rubrics to determine if failures are fair, and provide feedback to reviewers on prompt and rubric issues.

How the pay compares

At the middle of its range, this job pays $73 an hour.

  • Coding jobs: median $63 an hour across 202 open hourly jobs. This job pays $10 more.
  • Mercor jobs: median $80 an hour across 329 open hourly jobs. This job pays $7 less.

What you'll do

  • Audit complete tasks and grading rubrics
  • Determine if rubric failures are fair
  • Check information availability and rubric accuracy
  • Evaluate if valid answers would be marked wrong
  • Verify rubric structure per guidelines
  • Provide feedback to reviewers

Requirements

  • Have substantial experience with LLM eval rubrics
  • Provide feedback to writers or reviewers
  • Maintain consistent availability
  • Read long agent transcripts to find errors
  • Write precisely and verify data

Skills

  • LLM Evaluation
  • Rubric Design
  • Data Verification
  • Feedback

Full description

The work

We are testing AI agents on realistic shopping and delivery tasks. Your role as a Project Auditor is to audit the work done by our Project Reviewers. You will audit complete tasks (prompt and grading rubric) and determine if their rubric failures are fair:

- Was the information available?

- Does the rubric accurately grade what the prompt asked for?

- Would a valid answer have been marked wrong?

- Is the rubric structure correct per the guidelines?

This role includes providing feedback to reviewers regarding prompt and rubric issues.

Must-haves

- Substantial experience writing or reviewing rubrics for LLM evals. Please list the projects in your application.

- Experience providing feedback to writers/reviewers

- Consistent availability

- Can read a long agent transcript and find where it went wrong.

- Precise writing, and you check the data instead of trusting the reference answer.

Similar jobs

View all jobs