SWE-Bench Task Auditor
Evaluate the quality, correctness, and reproducibility of software-engineering benchmark tasks used to train and evaluate a frontier AI lab's models.
- Pay
- Firm hourly pay: $70-$90 per hour
- Location
- Remote — United States
- Eligibility
- Remote; check the U.S. location wording
- Qualification difficulty
- Selective
How current is this information?
The public role and application path were checked. Details can still change; this is not an endorsement or guarantee.
- Platform
- Mercor
- Fit category
- Software engineering
- Listing/source checked
- Sep 1, 2026
- Inventory presence checked
- Sep 28, 2026
- Apply link checked
- Aug 31, 2026
Application
Continue to the current Mercor listing
Apply on MercorOpens the current Mercor page in a new tab. This may be a referral link, and Specialist AI Work may be paid if the platform credits it. That does not change the role's advertised pay or how roles are ordered here.
What this role involves
Evaluate the quality, correctness, and reproducibility of software-engineering benchmark tasks used to train and evaluate a frontier AI lab's models. You'll assess repository-level tasks, reference patches, test harnesses, and grading integrity — and provide clear, rubric-based written feedback.
Basic Qualifications • 3+ years professional software engineering • Real open-source contribution or maintainer experience (merged PRs, committer / maintainer roles) • Strong ability to audit reference patches, test runners, and Docker isolation, and to detect answer leakage / reward hacking • Fluency across common ecosystems (Python and at least one of Java / Go / TypeScript / C++)
Preferred Qualifications
• Familiarity with SWE-Bench (Verified) or similar repository benchmarks • Maintainer history on major Python OSS (Django, Flask, scikit-learn, sympy, pytest, etc.) • Prior code-review or task-grading experience
Before you apply
Review the main fit signals and unresolved details before opening the platform.
Why it may fit
- Professionals whose experience matches the current SWE-Bench Task Auditor requirements.
- Applicants comfortable completing Mercor's role-specific assessment.
Check before applying
Reasons to pause
- You cannot meet the listing's stated remote or location eligibility.
- You need guaranteed acceptance, hours, or project duration.
Still to verify
- Review the official Mercor listing before applying. Requirements, screening, pay, hours, and project availability can change.
- The reviewed listing had limited public detail; check the current platform page for the full requirements.
What to prepare
- Complete Mercor's role-specific application or assessment.
- Review the current Mercor listing before applying.
Role tools
Where these choices are saved
Save, Not for me, Compare, and application tracking remain in this browser.
Application tips
- Complete Mercor's role-specific application or assessment carefully.
- Review the current Mercor listing and its eligibility details before applying.