Legal Domain Expert - AI Training & Evaluation

Compensation: $60-$100 per hour

A leading AI research organization is seeking an experienced legal professional to work closely with its research and program teams to improve how advanced AI models understand, reason about, and respond to real-world legal work.

As a Legal Domain Expert - AI Training & Evaluation, you will bring substantive legal expertise to the development and evaluation of AI systems. You will assess legal knowledge tasks and model outputs, develop high-quality instructions and reference solutions, and help establish benchmarks that define what accurate, well-reasoned legal work looks like in professional practice.

This role is intended for practicing legal specialists who can apply sound professional judgment, identify subtle issues in legal reasoning, and translate complex legal expertise into clear evaluation standards.

Location: Hybrid - Bay Area, California Commitment: 40 hours per week Initial Engagement: 6 months

Key Responsibilities

• Review & Quality Assurance: Evaluate legal tasks and AI-generated outputs for accuracy, completeness, reasoning quality, and professional relevance. Identify gaps, unsupported conclusions, weak reasoning, and responses that may appear credible but would not meet professional legal standards.
• Develop Instructions & Reference Solutions: Create detailed task specifications, develop high-quality reference answers, and design legal scenarios that accurately represent real-world legal practice.
• Build Evaluation Benchmarks: Develop challenging legal tasks and evaluation datasets that can effectively measure AI capabilities across different legal domains.
• Legal Reasoning & Calibration: Translate professional legal judgment into clear, structured, and teachable evaluation criteria that can be consistently applied across AI outputs.
• Collaborate with Research Teams: Work closely with AI researchers, program managers, and other subject-matter experts to identify knowledge gaps and improve legal reasoning capabilities.
• Provide Structured Feedback: Deliver precise written feedback explaining not only whether an AI response is correct, but why it meets or fails to meet professional standards.

Core Qualifications

• Education: Juris Doctor (JD) from an accredited law school; education from a highly regarded institution is strongly preferred.
• Experience: 5+ years of substantive post-qualification legal experience at an established law firm, corporate legal department, regulatory organization, court, or comparable institution. Internships and clerkships alone do not qualify.
• Legal Specialization: Demonstrated expertise in at least one substantive practice area, such as:
• Corporate & Transactional Law
• Litigation & Dispute Resolution
• Regulatory & Compliance
• Intellectual Property
• Employment & Labor Law
• Tax Law
• Other specialized legal practice areas

-

• Seniority: Demonstrated career progression into a senior position such as Partner, Of Counsel, Counsel, Senior Associate, Senior In-House Counsel, or General Counsel, with meaningful ownership of legal matters.
• Licensure: Active admission to at least one U.S. state bar and in good standing.
• AI Fluency: Practical familiarity with large language models and AI tools, along with the ability to distinguish genuinely sound legal reasoning from plausible but inaccurate AI-generated responses.
• Communication: Excellent written communication skills and the ability to provide clear, concise, and well-structured legal analysis and feedback.
• Availability: Ability to commit reliably to 40 hours per week for an initial six-month engagement.
• Location: Currently based in the Bay Area, California , with the ability to work on-site with the team multiple days per week when required. Candidates outside the Bay Area must be willing to relocate at their own expense before the engagement begins. Relocation assistance is not provided.

What Success Looks Like

The ideal candidate combines deep practical legal expertise with strong analytical and communication skills. You should be comfortable evaluating complex legal work, identifying nuanced errors, explaining your reasoning clearly, and helping transform professional legal judgment into measurable standards for AI training and evaluation.