Expert data by field: what doctors, lawyers and engineers are paid to do

AI labs hire doctors, lawyers, engineers and scientists for the same reason: models now fail in places only an expert can see. The tasks look alike on a vendor’s website (demonstrations, preferences, evaluations) but the work changes a great deal from one field to the next. Here is what each field’s experts are actually paid to do.

What are the four basic tasks?

Almost every expert data project is built from four tasks.

• Demonstrations. The expert writes the answer a model should give, with the reasoning shown. This is supervised fine-tuning data.
• Preferences. The expert reads two or more model answers and ranks them, usually with a reason. This feeds reinforcement learning from human feedback.
• Grading. The expert scores a model answer against a rubric, item by item.
• Rubric design. The expert writes the rubric, or the test, that later graders or programs apply.

Rubric design usually calls for the most senior people, because one senior expert’s judgment is multiplied across thousands of graded items. The vendors’ own pages show the same split: Surge AI lists rubrics and verifiers next to RLHF and demonstrations, and Toloka names SFT demonstrations, preference data and auto-verifiable tasks.

Software engineers: tasks a test can grade

Coding is the field where the expert’s output is most often a program. An engineer writes a problem, a reference solution and the tests that will judge any model’s attempt. Appen, for instance, advertises a set of 500 coding problems that each come with their own automatic checker. Turing goes further and builds whole workplaces in miniature (working copies of Jira, Salesforce and Zendesk screens) in which an agent acts and is scored.

The hard part is not writing code. It is writing tests that cannot be passed by a wrong answer. Senior engineers are paid to find those gaps.

Lawyers: reasoning no test can check

A legal answer can be well written and wrong, or right for the wrong reason. Lawyers are paid to draft model answers to legal questions, redline contracts, rank model outputs and, above all, write rubrics for what a sound answer contains: the controlling rule, the exceptions, the jurisdiction.

Jurisdiction shapes everything. Bar admission is state by state, so a buyer grading California employment questions wants California lawyers, and should say so in the brief.

Finance professionals: license for some tasks, judgment for most

Finance splits in two. Accounting treatment, tax and audit-style review lean on CPAs, whose licenses can be checked on CPAverify for most states. Modeling, valuation, credit analysis and earnings work lean on analysts and former bankers, who hold no public license. For them the evidence is a graded sample: build this model, write this memo.

Scientists: one hard problem, one right answer

Science projects pay PhDs and strong doctoral candidates to write problems that current models get wrong, each with a single checkable answer and a worked solution. Physics derivations, reaction mechanisms, theorems and data interpretation are typical. The second job is step-by-step grading of a model’s reasoning, marking the first wrong step, which is slower and needs more seniority than writing.

Clinicians: expertise without patient data

Physicians, nurses and pharmacists write and grade answers to clinical questions, rank responses for accuracy and safety, and design rubrics that say what a safe answer must include. None of this needs a patient record. The work draws on the clinician’s training, and buyers should keep it that way.

Linguists and translators: reach versus depth

Multilingual projects pay native speakers to write, translate and grade in their language, often combined with a domain: legal Spanish, medical Japanese. Reach is published by some vendors. TELUS Digital claims 500+ languages and a contributor community above one million; Surge AI says it operates in over 70 languages. Depth in a given language and field is what to ask about.

How does it compare across fields?

Only two published pay references turned up on the pages we read, and neither is specific to one field. Labelbox’s Alignerr puts its ceiling at $150 an hour. Prolific publishes a participant minimum of $8 an hour and a recommendation of $12, for research participants generally rather than credentialed experts.

What does this mean for a buyer?

Pick the field first, then the task, then the credential, and only then the vendor. The expert data companies on our short-lists sell across many fields, and few of them say on their websites which professions they can field in depth. The brief you send them should say exactly which credential each task needs and how you will check it.

The field decides who you hire; the task decides what you pay them for.