Skip to content

Research report ยท Live data and sources

Human Data Quality, Fraud and Security

From 13 Jun 2023 to 11 Oct 2026, 41 studies, incidents and vendor statements document the quality risks of human data: workers using AI models to do the tasks (33-46% of crowd workers (range estimate) in the first measurement), fake and resold accounts, quality failures on lab projects and breaches at vendors.

Covers
13 Jun 2023 to 11 Oct 2026
Events by the date they happened
Computed
18 Oct 2026, 17:41 UTC
Live figures daily; sources as reviewed
Source
Job descriptions read every day for the live figure; studies, vendor statements and reporting, each linked.
Method
Version 1. Definitions
CC BY 4.0, how to cite

Every figure on this page as CSV and JSON, sent to your work email.

studies, incidents and vendor statements
41
of crowd workers used an AI model on a task, in the first measurement
33-46%
security records at human-data vendors
9

Key findings

  1. 1

    Veselovsky, Horta Ribeiro and West: LLM use by Amazon Mechanical Turk workers on a summarization task: 33-46% of crowd workers (range estimate) (13 Jun 2023).

    Figure 2
  2. 2

    Vendors report far lower rates on their own platforms: Prolific says internal audits put AI-assisted responding below 1% (below 1% (upper bound, vendor internal audit)).

    Figure 2
  3. 3

    Scale AI late-2023 spreadsheet lists 490 removed contractors (VPN use, duplicate accounts) (490 removed (48 VPN, 70 same name, 11 two accounts, 21 low quality), 1 Dec 2025).

    Figure 3
  4. 4

    The latest security incident recorded: Report: Meta ended its work with Mercor after the breach (10 Jul 2026).

    Figure 5
  5. 5

    micro1 case study, 250 annotators hired from 25,000+ applications (stated, 11 Oct 2026).

    Figure 6
  6. 6

    Identity or location fraud is the most documented risk, with 12 of the 41 records.

    Figure 7

Figure 1

Screening steps stated in open roles today (live)

How often do open roles say applicants will be tested, interviewed or verified?

PlatformRoles readAssessment or testAI-led interviewInterviewBackground checkIdentity check
All described roles3,26014.7%4.5%4.9%0.2%0.7%
RemoExperts365.6%0.0%0.0%0.0%0.0%
SuperAnnotate (SME)37711.1%26.8%0.8%0.0%0.0%
Sigma AI900.0%0.0%0.0%0.0%0.0%
Invisible5323.8%0.0%0.0%0.0%0.0%
micro124823.8%9.3%2.4%0.0%0.8%
Mercor22912.7%2.2%3.9%0.4%0.0%
Terac224.5%9.1%13.6%0.0%0.0%
AfterQuery21640.7%0.0%0.9%2.3%5.1%
Babel Audio430.0%0.0%0.0%0.0%0.0%
Turing19033.7%2.6%14.7%0.0%0.0%
Welo Data23511.1%0.0%13.2%0.0%0.9%
Perle Labs6716.4%0.0%67.2%0.0%0.0%
Claru424.8%0.0%0.0%0.0%0.0%
Vetto2020.0%0.0%10.0%0.0%0.0%
Anyone AI12857.8%0.0%0.0%0.0%0.0%
Handshake AI1705.3%0.0%2.4%0.0%0.0%
OpenTrain AI429.5%0.0%0.0%0.0%0.0%
Pila830.0%0.0%0.0%0.0%0.0%
Braintrust7819.2%15.4%16.7%0.0%0.0%
Pasiflora AI714.3%0.0%0.0%0.0%0.0%
OneForma248.3%0.0%0.0%0.0%0.0%
Sovrano AI2080.0%0.0%0.0%0.0%0.0%
Appen550.0%0.0%0.0%0.0%16.4%
Mindrift2970.0%0.0%0.0%0.0%0.0%
xAI (SpaceXAI)150.0%0.0%100.0%0.0%0.0%
TransPerfect DataForce667.6%0.0%0.0%0.0%0.0%
Remotasks333.3%0.0%0.0%0.0%0.0%
Outlier (Scale AI)5100.0%0.0%0.0%0.0%0.0%

Sample. 3,260 open roles whose job board publishes a full description, read today with the patterns of the requirements report. Recomputed daily.

A mention is not a guarantee: a role can screen without saying so. Only platforms that publish descriptions are read.

Source. AITraining.jobs Research, Human Data Quality, Fraud and Security, live data.

Figure 2

Workers using AI models to do the work

How often do workers use an AI model to do human-data tasks?

RecordDateFigure or finding
Prolific: internal audits put AI-assisted responding below 1%Prolific4 Aug 2026below 1% (upper bound, vendor internal audit)
Prolific: under 0.1% of participants flagged for AI-generated responses (January 2026 audit)Prolific4 Aug 2026<0.1% of participants flagged (January 2026 internal audit)
Prolific: accuracy of its LLM authenticity checksProlific4 Aug 202698.7% precision, 78.9% recall, 88.4% accuracy (vendor testing)
Velutharambath et al.: survey of NLP researchers on LLM use in crowdsourced free-text dataarXiv3 Jun 202644% of respondents observed LLM usage in crowdsourced data
Scale AI project documents discuss banning accounts from some countries for using ChatGPTBusiness Insider (syndicated by AOL)1 Dec 2025According to internal Scale AI documents reviewed by Business Insider, managers on a project for an unnamed client discussed banning certain accounts from Egypt, Kenya, Pakistan and other countries for using ChatGPT, and blocking copy-paste, to stay ahead of spammers.
Zhang, Xu and Alvero: research participants' use of LLMs on Prolific open-ended survey questionsStanford Graduate School of Business21 Nov 2024nearly one-third reported some LLM use; under 10% very frequent
Christoforou, Demartini and Otterbacher (ICWSM 2024): self-reported AI chatbot use in tasks, Amazon Mechanical TurkProceedings of the International AAAI Conference on Web and Social Media (ICWSM)28 May 202480.3% (May 2023), 73.2% (Dec 2023), self-reported
Christoforou, Demartini and Otterbacher (ICWSM 2024): self-reported AI chatbot use in tasks, ProlificProceedings of the International AAAI Conference on Web and Social Media (ICWSM)28 May 202413.1% (May 2023), 13.4% (Dec 2023), self-reported
Christoforou, Demartini and Otterbacher (ICWSM 2024): self-reported AI chatbot use in tasks, ClickworkerProceedings of the International AAAI Conference on Web and Social Media (ICWSM)28 May 202420.7% (May 2023), 15.0% (Dec 2023), self-reported
Veselovsky et al.: prevalence and prevention of LLM use in crowd work (Prolific)arXiv (EPFL, Prolific, Microsoft Research)24 Oct 2023around 30% of responses; roughly halved by mitigations
Veselovsky, Horta Ribeiro and West: LLM use by Amazon Mechanical Turk workers on a summarization taskarXiv (EPFL)13 Jun 202333-46% of crowd workers (range estimate)

Sample. 11 records on LLM use by workers, to 11 Oct 2026.

Rates come from different tasks, platforms and methods (detection, self-report, vendor audits) and are not comparable with one another.

Source. AITraining.jobs Research, Human Data Quality, Fraud and Security, live data.

Figure 3

Identity, account and location fraud

How do fake accounts, account sales and location spoofing reach human-data work?

RecordDateFigure or finding
Outlier location rules: government ID verification and no VPNs from restricted countriesOutlier (Scale AI)11 Oct 2026Outlier's Flexible Working Guidelines say contributors may be required to verify identity and location with a government-issued ID during onboarding, which sets a Primary Location, and that using a VPN to mask presence in a restricted country is not permitted.
19 agencies in nine countries warn of North Korean IT workers on freelancing platformsAl Jazeera31 Jul 202619 agencies, 9 countries
Mercor staff suspected North Korean operatives worked on its platformForbes15 Apr 2026According to multiple former employees who spoke to Forbes, Mercor staff suspected from as early as November 2024 that North Korean operatives had worked for the company using stolen credentials to get past identity checks, in some cases producing data for US AI labs such as Anthropic. Forbes reports Mercor set up a three-person fraud team and now uses Persona for identity checks; a Mercor spokesperson said it has invested heavily in fraud detection, including around-the-clock monitoring and IP-blocking.
Prolific: periodic face-scan re-checks against account sharing and sellingProlific3 Dec 2025Prolific says participants complete identity checks at onboarding and periodically afterwards, and that a participant who fails a check has access to new studies frozen until re-verifying, to prevent account sharing, selling and impersonation.
Business Insider finds 100+ Facebook groups selling AI training accountsBusiness Insider (syndicated by AOL)1 Dec 2025at least 100 groups; about 40 removed by Meta
Scale AI late-2023 spreadsheet lists 490 removed contractors (VPN use, duplicate accounts)Business Insider (syndicated by AOL)1 Dec 2025490 removed (48 VPN, 70 same name, 11 two accounts, 21 low quality)
Prolific: live-video ID verification for every new participant (Entrust)Prolific28 Nov 2025fraud rate <0.1%; false acceptance rate 0.01% (one in 10,000)
Scale AI 'Bulba Experts' spammers returned via VPN; memos targeted contributors from specific countriesFuturism (summarizing Inc.)28 Jun 2025Summarizing Inc.'s reporting on internal documents, Futurism says spammers on Scale AI's Google project could return after being caught by using a VPN, that memos called for denying or removing contributors from Egypt, Pakistan, Kenya and Venezuela, and that supervisors were advised to vet entries with the ZeroGPT detector. Scale AI told Inc. the story contained many inaccuracies and that it had safeguards to remove spam before anything reached customers.
AlgorithmWatch infiltrates WhatsApp market for European Outlier accountsAlgorithmWatch22 May 2025claimed 10 false accounts per day; group of about 1,000 members
Kenyan account manager says Outlier blocked 40 of his team's accountsAlgorithmWatch22 May 202540 accounts blocked (claim by one account manager)
Outlier: every contributor must pass identity verificationAlgorithmWatch22 May 2025Responding to AlgorithmWatch, an Outlier spokesperson said every contributor must pass identity verification before accessing work and that accounts flagged for suspicious activity are investigated and deactivated as necessary.
Remotasks shuts down in Kenya, Nigeria and Pakistan citing security protocolsRest of World7 Mar 20243 countries shut down; new sign-ups stopped in 3 more

Sample. 12 records on identity or location fraud, to 11 Oct 2026.

Source. AITraining.jobs Research, Human Data Quality, Fraud and Security, live data.

Figure 4

Documented quality failures

Where has human data for AI gone wrong, by the documents reported?

RecordDateFigure or finding
Thousands of taskers flagged as 'suspected spammers' or 'cheaters' on a 2024 Scale AI project for GoogleBusiness Insider (syndicated by AOL)1 Dec 2025thousands of taskers flagged
GlobalLogic Gemini rater's time per task cut from 30 to 15 minutesThe Guardian11 Sep 202530 minutes to 15 minutes per task, about 500 words per response
Former Remotasks queue manager: 800 spammers dumped into one teamFuturism (summarizing Inc.)28 Jun 2025800 spammers (single anecdote)
Scale AI's 'Bulba Experts' program for Google flooded with spam (Mar 2023 to Apr 2024)Inc.25 Jun 2025at least 11 months; 'spam' across 83 pages of documents
Gemini raters at GlobalLogic no longer allowed to skip prompts outside their expertiseTechCrunch18 Dec 2024TechCrunch reported, citing internal correspondence, that a new Google guideline passed to GlobalLogic contractors rating Gemini removed the option to skip prompts requiring specialized domain knowledge; raters were told to rate the parts they understand and note their lack of expertise. Google later said it was constantly working to improve factual accuracy and that ratings do not directly impact its algorithms.

Sample. 5 records on quality failure, to 11 Oct 2026.

Source. AITraining.jobs Research, Human Data Quality, Fraud and Security, live data.

Figure 5

Security incidents at human-data vendors

Which breaches and exposures have hit human-data vendors and their clients?

RecordDateFigure or finding
Report: Meta ended its work with Mercor after the breachteiss10 Jul 2026teiss reported that, following the security incident, Meta said it had ended its work with Mercor as part of its security review. This conflicts with Mercor's 25 June statement that all frontier labs had increased their work with it; it was not confirmed by a second outlet in this review.
Mercor concludes breach investigation: 'very limited subset' of nearly five million experts affectedMercor25 Jun 2026nearly five million experts; affected number not disclosed
Meta pauses work with Mercor after the breachForbes15 Apr 2026Meta told Forbes its work with Mercor was paused while it investigated the breach (WIRED first reported an indefinite pause in early April 2026); other frontier labs including OpenAI were evaluating their work with Mercor, according to multiple sources. Mercor said nearly every customer had continued business as usual.
At least six contractor lawsuits against Mercor over the breachForbes15 Apr 2026at least six lawsuits
Hacker group claims 4TB of Mercor dataTechCrunch9 Apr 20264TB (attacker claim)
Class action complaint alleges Mercor breach exposed more than 40,000 peopleThe Next Web1 Apr 2026more than 40,000 people (alleged in complaint)
Mercor confirms data breach via the LiteLLM supply-chain attackTechCrunch31 Mar 2026malicious package live for about 40 minutes
Surge AI spreadsheet of Claude training sources left in an open Google Drive folderTom's Guide (citing Business Insider)23 Jul 2025120+ whitelisted and 50+ blacklisted sites
Scale AI left confidential client and contractor files in public Google DocsTechRepublic (citing Business Insider)25 Jun 202585 documents, thousands of pages

Sample. 9 records on security incident, to 11 Oct 2026.

Source. AITraining.jobs Research, Human Data Quality, Fraud and Security, live data.

Figure 6

How hard vendors screen

How many applicants do vendors accept, and how much work do they reject?

RecordDateFigure or finding
micro1 case study: 250 annotators hired from 25,000+ applicationsmicro111 Oct 2026~85% interview-to-hire; 250 hired of 25,000+ applications
Prolific: 55% of invited applicants pass onboardingProlific4 Aug 202655% of invited applicants pass onboarding
Prolific: overall study rejection rate of 0.5% in 2025Prolific4 Aug 20260.5% of submissions rejected (all studies, 2025)
Prolific: only 13% of waitlist applicants are invited to onboardingProlific28 Nov 202513% of waitlist applicants invited

Sample. 4 records on screening and acceptance, to 11 Oct 2026.

Source. AITraining.jobs Research, Human Data Quality, Fraud and Security, live data.

Figure 7

Records by topic

Which risks are documented most?

TopicRecords
Identity or location fraud
12
LLM use by workers
11
Security incident
9
Quality failure
5
Screening and acceptance
4

Sample. 41 records, to 11 Oct 2026.

Source. AITraining.jobs Research, Human Data Quality, Fraud and Security, live data.

For labs, data vendors and researchers

Buy the data behind this report

The figures above are a summary. Behind them are records we collect ourselves, every day:

  • 9,572 roles tracked, each with its history

    Every role on 29 job boards, read daily since 13 Jul 2026: its pay and place, and the first and last day it was open.

  • 6,200 apply clicks in 28 days

    Which roles people actually apply to, by field, pay and kind of work. It happens on this site, so nobody else can count it.

  • 39 lab and vendor hiring boards

    The job boards of 25 AI labs and 14 human-data vendors, read every day: who is hiring for human data, and who is hiring to buy it.

What you can buy

  • The records

    The rows behind any figure, with their history, not only the totals.

  • A regular feed

    The same figures every week or month, as files.

  • A custom cut or benchmark

    Your countries, languages or fields, or your own platform against the market.

Method and definitions

Job descriptions read every day for the live figure; studies, vendor statements and reporting, each linked. The live figures are computed every day from this site's own data, the same way for every reader; every sourced row links the publication it is taken from, with the figure as that publication states it. Each figure states its sample beneath it.

Screening step (live)
A description that names an assessment, an AI-led or other interview, a background check or identity verification, by the patterns of the requirements report.
Study, incident, vendor statement
A measured rate from research, a documented event in reporting or a court filing, or a vendor's own figure.

What this data cannot tell you

  • Each report sets its sourced records beside a live figure computed from this site's own data every day; the sourced records are added as they are published and reviewed.
  • Compiled from public sources: an event that was never announced or reported is missing, and private companies' figures are what they or the press stated, not audited.
  • Rates come from different tasks, platforms and methods and are not comparable with one another.

How the job feeds are read and roles are classified: site methodology. How fresh, complete and consistent the data is, checked every day: data quality and coverage.

Editions

The live report is the dataset as last reviewed, beside figures recomputed every day from this site's own data; new events are added as they are published. Each complete month is also published as an edition, frozen two days after the month ends, so a figure you cite from an edition never moves.

  • Live dataSources as last reviewed, live figures daily

Cite and reuse

Free to quote, chart and republish under CC BY 4.0 with a link to this page. Live figures change daily: cite a monthly edition for numbers that will read the same later, or keep the access date.

Citation

AITraining.jobs (2026). Human Data Quality, Fraud and Security (live data, 13 Jun 2023 to 11 Oct 2026). https://www.aitraining.team/research/human-data-quality-and-fraud (accessed 18 Oct 2026)

BibTeX

@misc{aitrainingjobs_human_data_quality_and_fraud_live,
  author = {{AITraining.jobs}},
  title = {Human Data Quality, Fraud and Security (live data, 13 Jun 2023 to 11 Oct 2026)},
  year = {2026},
  howpublished = {\url{https://www.aitraining.team/research/human-data-quality-and-fraud}},
  note = {Accessed 18 Oct 2026},
}
Data: every figure as CSV (one line per value) and JSON, sent to your work email.Pay and open-roles data

More research

All 78 research reports