Are Scientific Journals Becoming Flooded with AI-Generated Research Papers?
Since the introduction of ChatGPT, many in the scientific research community have been trying to evaluate how to incorporate the benefits of AI into their research workflow without tripping over ethically problematic areas.
The experience of Roy Kishony and his team at the Technion Israel Institute of Technology in Haifa may be typical: As an experiment, Kishony tasked ChatGPT with writing a paper from scratch that evaluated a publicly available CDC data set. Kishony found the resulting paper lacked the kind of unique insights that form a useful paper.
In a survey conducted by Nature, over 1,600 readers weighed in on the pros and cons of using AI in their research work. Many reported positive findings in using AI thanks to it providing helpful, timesaving work – such as generating ideas for hypotheses or optimizing experimental setups. But they also cautioned that AI posed an ethical risk by making it easier to commit fraud or to create irreproducible and/or biased research results.
Sleuthing the Internet to Ferret Out Scientific Papers Written by AI
How many scientific papers are being written by AI?
In late 2022, Ansible Health submitted one of the first papers that intentionally listed ChatGPT as one of the co-authors.
The move generated lots of controversy (perhaps this was their intention?), and suddenly, publications began to quickly amend their submission guidelines to disavow and exclude the very few AI-generated papers they had received to date.
Given the backlash against listing AI as a co-author (more on the policy changes made by journals and government agencies later), we can assume that most recent papers “touched” by AI have been submitted without acknowledging AI’s supporting role.
But sometimes, it’s still possible to identify the hidden hand of AI.
In one hilarious example, a published paper was “outed” as AI-generated because it mistakenly included the original prompt used to generate the paper’s introductory paragraph:
“Certainly, here is a possible introduction for your topic.”
Internet sleuths have stepped into the breach to assist in identifying AI-generated papers.
Two of the prominent internet sleuths who have made it their life’s work to identify problematic research papers are David Bimler (who goes by the online name of “Smut Clyde”) and Elizabeth Bik, a former researcher who now works full-time uncovering faked image data.
Bimler and Bik both got their start investigating problematic, low-quality scientific articles pumped out by so-called “paper mills” – an ongoing problem that pre-dates the introduction of ChatGPT.
Other researchers are using statistical word distribution methods to identify AI-generated papers.
Researcher Andrew Gray of the University College London began to notice a statistically significant rise in the occurrence of certain words, such as “meticulous,” “commendable,” and “intricate” (up 137% since 2022), which may be telltale signs of AI writing. Researchers at Stanford have noticed a similar phenomenon, with the use of the words “realm,” “showcasing,” “intricate,” and “pivotal” up as much as 160%.
Based on these statistics, Gray hypothesizes that at least 1% of all papers published in 2023 were partially written by AI. The Stanford researchers believe the number could be much higher, up to 6.3% of papers in mathematics and natural science and up to 17.5% in computer science articles.
AI-drafted Scientific Articles Can Introduce Inaccuracies and Discrepancies Deep within a Paper’s Reasoning, Supporting Diagrams, and Data Sources
Unfortunately, indiscriminate or unethical use of AI can introduce errors and omissions deep within a scientific research paper – and these can be more difficult to identify without careful case-by-case review.
This certainly makes the job of a peer reviewer more difficult.
In some cases, logic errors and other anomalies found in AI-generated research papers can be traced back to the “hallucination” problem inherent in many large language models (LLMs) that form the backbone of ChatGPT and other AI chat systems.
Here, we can think of hallucinations as a euphemism for “making up facts” to bridge a gap in the knowledge of the AI system.
Another area to be on the lookout for is faked data representations, either stemming from the AI creating what are essentially fraudulent data sets that can’t be reproduced or making changes in the AI-generated data charts and diagrams that don’t accurately reflect the underlying data.
As an aside, the problem with fake data may be with us for some time. That’s because AI systems are increasingly relying on what’s called ‘synthetic data’ for training. In the case of training AI systems, synthetic data is OK, but when it spills over into research articles, it’s bad.
Updated Guidelines for Using AI in Drafting Scientific Papers
As we mentioned at the outset, academic institutions, publishers of scientific journal articles, and government agencies have been scrambling to amend their policies concerning whether they can accept AI-generated works and, if so, how to properly cite them.
It’s a confusing situation that spills out beyond academic publishing into the wider world of copyright law and intellectual property rights (including potentially lucrative patents!).
It will take a long time for policy regulators, lawmakers, and the courts to resolve these issues.
In the meantime, here is what we know.
The journal Nature was among the first to come out and create an outright ban on listing AI tools (such as ChatGPT) as co-authors, calling it a threat to the transparency and integrity of science. They followed up with a few clarifications, such as how AI can be used to help write papers (if it is properly disclosed), but they also issued an outright ban on using AI-generated images or video clips.
The JAMA Network issued its own policies that closely align with those of Nature: AI cannot be cited as an author, all AI-generated content must be disclosed (including details of the AI software version used), and clarified that (human) authors remain responsible for all submissions.
Meanwhile, the European Union has weighed in by passing EU Artificial Intelligence Act (“AI Act”) in March 2024. This landmark legislation, which uses a “risk based” approach to regulating AI, is expected to affect the larger of the Large Language Models (LLMs) by placing them in the “high-impact capabilities” category that carries a higher “systemic risk.”
This means AI software companies operating in the EU will have to disclose their system architectures and data sources. EU users of AI systems, such as scientific researchers, will likely be affected too, e.g. be required to formally disclose the use and scope of AI tools in their published works.
Formaspace is Your Laboratory Research Partner
Evolving Workspaces. It’s in our DNA.
Talk to your Formaspace Sales Representative or Strategic Dealer Partner today to learn more about how we can work together to make your next construction project or remodel a success.
