Software Engineer – LLM Evaluation & Repository Validation

We are looking for software engineers to validate real-world repositories used to evaluate large language models. You will triage issues, set up reproducible environments, assess test quality, and check how well models fix real bugs.

Key Responsibilities

• Review and triage issues in open-source GitHub projects.
• Prepare repositories for evaluation, including containerisation and environment automation.
• Assess how thorough and well written the unit tests are.
• Edit and run code locally to see how LLMs perform on bug-fixing tasks.
• Help pick repositories and issues that are hard for LLMs.
• Write up findings clearly for researchers and reviewers.
• Follow project guidelines, take part in calibration and review sessions, and incorporate feedback to keep quality consistent.

Required Qualifications

• 3+ years of software engineering experience.
• Strong experience in at least one of Python, C++, C#, Go, Rust or Ruby.
• Proficiency with Git, Docker and basic pipeline setup.
• Ability to navigate complex code bases and run real-world projects locally.
• Fluent written English.
• Reliable, detail-oriented, and able to work independently and meet deadlines in a remote setting.

Preferred Qualifications

• Prior LLM research or evaluation projects.
• Experience building or testing developer tools or automation agents.
• Open-source contribution or evaluation experience.

Rights, Confidentiality and Data Handling

• Submit only work that you created yourself or that you have the right to share.
• Do not submit material covered by an employer, client or third-party confidentiality agreement, or material you have no right to share.
• Do not submit export-controlled, military or defence-related technical data.
• Remove credentials, secrets and personal data from anything you submit.
• Rex may ask for evidence of ownership or licence, may verify submissions, and may reject or remove any submission.

Legal and Compliance Notice

• All content you submit must be lawful, and you must have the right to share it.
• Do not submit confidential or proprietary material belonging to an employer, client or third party, or personal data without a lawful basis.
• Rex may verify identity, credentials, ownership and licensing; may reject or remove any submission; and may report unlawful submissions to the relevant authorities.
• Engagement and payment are subject to successful verification and to applicable sanctions, export-control, tax and employment laws.
• The contractor agreement will set out ownership and licence terms for the work you deliver (to be confirmed by Rex Legal).

Where you can work

This role is 100% remote: you can work from anywhere in the world.