Between 2020 and 2024, researchers looked at how large language models were used in over a million pre-print and published scientific papers. They found that computer science papers saw the biggest and fastest growth of up to 22% in use of AI systems.
Powered by artificial intelligence (AI), large language models are trained on vast amounts of text and can therefore respond to human requests in the natural language.
From January 2020 to September 2024, researchers from Stanford University and other institutions in the United States examined 1,121,912 pre-print papers published in nature journals in the archives “arXiv” and “bioRxiv.” The team estimated the involvement of a large language model, ChatGPT in this study, in modifying research paper content by focusing on the frequency with which words commonly used by AI systems appeared in the papers.
The findings, which were published in the journal Nature Human Behaviour, “suggest a steady increase in LLM (large language model) usage, with computer science papers experiencing the largest and fastest growth estimated at up to 22%.” The researchers also estimated that pre-print papers in the archive “bioRxiv” written by authors from countries known to have fewer English-speaking people, like China and continental Europe, rely more on AI systems.
According to the analysis, however, there was less evidence of the use of AI in modifying content in mathematics-related and nature-journal-published papers. The study team said that shorter papers and authors posting pre-prints more often showed a higher rate of AI use in writing papers, suggesting that researchers trying to produce a higher quantity of writing are more likely to rely on LLMs.
The team stated that these results “could be an indication of the competitive nature of certain research areas and the pressure to publish quickly.” The researchers also looked at a smaller number of papers to understand how scholars disclose use of AI in their writing.
An inspection of randomly selected 200 computer science papers that were uploaded to the pre-print archive ‘arXiv’ in February 2024 revealed that “only two out of the 200 papers explicitly disclosed the use of LLMs during paper writing”.
Future studies that look at disclosure statements might help researchers understand why they use AI to write papers. For example, policies around disclosing LLM usage in academic writing may still be unclear, or scholars may have other motivations for intentionally avoiding to disclose use of AI, the authors said.
According to a recent study that was published in the scientific journal Science, it was estimated that at least 13% of the research abstracts that were published in 2024 could have benefited from the assistance of a large language model because they contained a greater number of “style” words that these AI systems were thought to favor.
According to an analysis of more than 15 million biomedical papers published between 2010 and 2024 by researchers from the University of Tübingen in Germany, AI models have significantly altered the vocabulary used in academic writing.