How AI Can Both Detect and Enable Fraudulent Research Irving Washington , Hagere Yilma , and Joel Luther

https://www.kff.org/health-information-trust/how-ai-can-both-detect-and-enable-fraudulent-research/?utm_campaign=22234741-KFF-Information-Trust&utm_medium=email&_hsenc=p2ANqtz-8kUMjU53UsWpFNdk--wGXSz5RtHhmqko4TdmySSsU98p6LqKb6JOZsLB6q5HF_5SWSReZB-1Y5nEtJxH9qVNVZiekD2Q&_hsmi=405757832&utm_content=405757832&utm_source=hs_email Nearly 10% of cancer research papers showed signs of being fabricated by “paper mills” that sell manuscripts at industrial scale, with the share increasing exponentially over time, according to new research. The problem may intensify as generative AI becomes more sophisticated, prompting lawmakers to demand information from federal agencies about safeguards in place. And, persistent claims that physicians are financially incentivized to promote vaccines may be contributing to vaccine hesitancy and declining trust, even as recent analyses show doctors typically break even or lose money when administering vaccines. AI & Emerging Technology Machine Learning Can Help Detect “Paper Mills,” Even as Generative AI May Contribute to Rise in Fraudulent Papers What does new research show about the prevalence of fraudulent papers? As generative AI makes it easier to produce fraudulent papers, researchers are turning to AI-powered detection methods in response. A study published in BMJ developed a machine learning model to identify cancer research papers with similarities to known “paper mill” publications that write and sell manuscripts at industrial scale. When applied to millions of cancer research papers published between 1999 and 2024, the model found that nearly 10% showed signs of coming from these paper mills, sharing textual characteristics with retracted publications. The number of flagged papers increased exponentially over time, rising from about 1% in the early 2000s to over 15% of annual cancer research output by the 2020s. Flagged papers were not limited to low-impact journals, with the share of these papers in high-impact journals also increasing over time to over 10% in recent years. Lawmakers demand safeguards The study comes as trust in medical institutions, including scientific journals, becomes increasingly politicized, with officials questioning the legitimacy of leading medical journals. House Republicans sent oversight letters in early February to five federal agencies, demanding information on safeguards to prevent falsified or fraudulent studies from influencing federal grants and research. The letters specifically raised concern about paper mills linked to the Chinese Communist Party, arguing that pressures imposed on Chinese researchers have increased demand for fabricated research. The letters note that major publishers have retracted thousands of papers linked to paper mill activity, with some forced to shut down journal subsidiaries after discovering widespread fraud. Why this matters The findings suggest that paper mills represent a large and growing threat to research integrity, with generative AI potentially exacerbating the problem through automated text generation. As AI tools become more sophisticated and accessible, fraudulent paper mill activity may increase, requiring ongoing development of detection methods and stronger institutional safeguards to protect research integrity. Fabricated research entering the scientific literature can misdirect research funding and erode public trust in medical research at a time when confidence in scientific institutions is already declining.