AI Developer Trace Task Auditor
$70 - $90 / hour
Remote · United States · Hourly contract
Loading opportunities…
$60 - $85 / hour
The work
Requirements reviewed 2026-11-21. Official listing
Listed location: Remote · 13 eligible countries; see requirements
Find roles for my country and experience →The work
We are testing AI agents on realistic shopping and delivery tasks. Your role as a Project Auditor is to audit the work done by our Project Reviewers. You will audit complete tasks (prompt and grading rubric) and determine if their rubric failures are fair:
\- Was the information available?
\- Does the rubric accurately grade what the prompt asked for?
\- Would a valid answer have been marked wrong?
\- Is the rubric structure correct per the guidelines?
This role includes providing feedback to reviewers regarding prompt and rubric issues.
Must-haves
\- Substantial experience writing or reviewing rubrics for LLM evals. Please list the projects in your application.
\- Experience providing feedback to writers/reviewers
\- Consistent availability
\- Can read a long agent transcript and find where it went wrong.
\- Precise writing, and you check the data instead of trusting the reference answer.
The listing states $60 - $85 / hour. Confirm the final rate, workload, and payment terms during the official application process.
Use your CV to find relevant roles on HumanitApp. This is separate from a partner application.
Match my CV →Preview the latest role picks before choosing weekly email updates.
Read this week’s picks → · Get weekly role alerts →Before you apply
This listing is not a promise of immediate work. Mercor may send separate offers to prequalified candidates. Read how Instant Work Offers work.
The listing states $60 - $85 / hour. Confirm the final rate, workload, and payment terms during the official application process.
Yes. The button opens the exact verified Mercorlisting using the referral URL published for this role.
No. HumanitApp independently curates the opportunity. The partner platform manages applications and hiring decisions.
No. Choose the primary Apply action to continue directly without giving HumanitApp your name or email address.