Software Engineer – LLM Evaluation & Repository Validation
We are looking for software engineers to validate real-world repositories used to evaluate large language models.
- Pay
- Firm hourly pay: USD $50–100 per hour, adjusted by country of residence.
- Location
- Worldwide
- Eligibility
- Remote worldwide
- Qualification difficulty
- Selective
How current is this information?
The public role and application path were checked. Details can still change; this is not an endorsement or guarantee.
- Platform
- REX / RemoExperts
- Fit category
- Software engineering
- Listing/source checked
- Oct 7, 2026
- Apply link checked
- Oct 7, 2026
Application
Continue to the current REX / RemoExperts listing
Apply on REX / RemoExpertsOpens the current REX / RemoExperts page in a new tab. This may be a referral link, and Specialist AI Work may be paid if the platform credits it. That does not change the role's advertised pay or how roles are ordered here.
What this role involves
We are looking for software engineers to validate real-world repositories used to evaluate large language models. You will triage issues, set up reproducible environments, assess test quality, and check how well models fix real bugs.
**
Key Responsibilities
** - Review and triage issues in open-source GitHub projects. - Prepare repositories for evaluation, including containerisation and environment automation. - Assess how thorough and well written the unit tests are. - Edit and run code locally to see how LLMs perform on bug-fixing tasks. - Help pick repositories and issues that are hard for LLMs. - Write up findings clearly for researchers and reviewers. - Follow project guidelines, take part in calibration and review sessions, and incorporate feedback to keep quality consistent.
Before you apply
Review the main fit signals and unresolved details before opening the platform.
Why it may fit
- Required Qualifications**
- 3+ years of software engineering experience.
Check before applying
Reasons to pause
- The opportunity no longer appears in REX's complete public Open Opportunities inventory
- The current role does not show applicant compensation
Still to verify
- Review the current REX opportunity before applying. Country eligibility, exams, identity verification, device requirements, compensation, project availability, and task access can change by role.
What to prepare
- Use examples from your own professional experience and follow REX assessment rules
- Review the current public REX opportunity before applying because project details can change
- Strong experience in at least one of Python, C++, C#, Go, Rust or Ruby.
Role tools
Where these choices are saved
Save, Not for me, Compare, and application tracking remain in this browser.
Requirements
- Proficiency with Git, Docker and basic pipeline setup.
- Ability to navigate complex code bases and run real-world projects locally.
- Fluent written English.
- Reliable, detail-oriented, and able to work independently and meet deadlines in a remote setting.
- Preferred Qualifications**
- Prior LLM research or evaluation projects.
- Experience building or testing developer tools or automation agents.
- Open-source contribution or evaluation experience.
Best for
- Proficiency with Git, Docker and basic pipeline setup.
Skip this if...
- You cannot meet the role-specific country, skill, device, exam, or identity requirements