PLAGIARISM DETECTION IN TEXTS GENERATED BY LARGE LANGUAGE MODELS
DOI:
https://doi.org/10.32782/IT/2024-4-2Keywords:
Academic misconduct, ChatGPT, fine-tuning, LLM, plagiarism detection, text generation.Abstract
Various algorithms have been developed to detect text generated by large language models, and instruments such as DetectGPT, RADAR, Ghostbuster, GPT-Sentinel, etc. are based on these algorithms. While automated systems can detect some plagiarism, studies show that such software not only fails to find all plagiarism, but also marks original content as plagiarized, thus providing false positive results. State-of-the-art AI-text detectors demonstrate a significant performance degradation when faced with texts created by non-native English speakers. The purpose of this study is to improve the accuracy and reliability of detecting AI-generated text, especially in the educational environment, where plagiarism and academic misconduct are becoming increasingly relevant due to the use of LLMs. The research methodology is based on general scientific methods of analysis and synthesis, experimental testing, and quantitative analysis of the effectiveness of a language model fine-tuned for plagiarism detection. The scientific originality of the study lies in the adaptation of modern methods of plagiarism detection for reliable classification of AI-texts in the context of the Ukrainian language. To do this, a new dataset was created based on paraphrased text fragments generated by ChatGPT, and the mT5 model was fine-tuned on this dataset for a classification task. The model’s performance is evaluated using three different evaluation metrics: F1 score, false positive rate, and false negative rate. The results of the study show that the fine-tuned model effectively detects the differences between the two types of text. The results also provide some insight into the strengths and weaknesses of the model, and demonstrate its potential for application in practical tasks. Further research aims to collect data from different contexts to evaluate the accuracy of the fine-tuned model for different natural language processing tasks.
References
DetectGPT: Zero-Shot Machine-Generated Text Detection using Probability Curvature / E. Mitchell et al. Proceedings of Machine Learning Research : Proceedings of the 40th International Conference on Machine Learning, 3 July 2023. Honolulu, Hawaii, USA, 2023. P. 24950-24962. DOI: 10.48550/arXiv.2301.11305
RADAR: Robust AI-Text Detection via Adversarial Learning / X. Hu et al. Advances in Neural Information Processing Systems, 10-16 December 2023. Vol. 36. New Orleans, USA, 2023. P. 15077–15095. DOI: 10.48550/arXiv.2307.03838
Ghostbuster: Detecting Text Ghostwritten by Large Language Models / V. Verma et al. North American Chapter of the Association for Computational Linguistics. USA, 2023. 16 p. (arXiv preprint). DOI: 10.48550/arXiv.2305.15047
GPT-Sentinel: Distinguishing Human and ChatGPT Generated Content / Y. Chen et al. 2023. 18 p. (arXiv preprint). DOI: 10.48550/arXiv.2305.07969
Cai S., Cui W. Evade ChatGPT Detectors via A Single Space. 2023. 12 p. (arXiv preprint). DOI: 10.48550/arXiv.2307.02599
Cross-domain detection of gpt-2-generated technical text / J. Rodriguez et al. Human Language Technologies: Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics, July 2022. Seattle, USA, 2022. P. 1213–1233. URL: https://aclanthology.org/2022.naacl-main.88 (дата звернення: 07.09.2024)
DeepHider: A Multi-module and Invisibility Watermarking Scheme for Language Model / L. Dai et al. 2022. 16 p. (arXiv preprint). DOI: 10.48550/arXiv.2208.04676
A Watermark for Large Language Models / J. Kirchenbauer et al. Proceedings of Machine Learning Research : Proceedings of the 40th International Conference on Machine Learning, 3 July 2023. Honolulu, Hawaii, USA, 2023. 26 p. DOI: 10.48550/arXiv.2301.10226
Can AI-Generated Text be Reliably Detected? / V.S. Sadasivan et al. 2023. 34 p. (arXiv preprint). DOI: 10.48550/arXiv.2303.11156
Large Language Models can be Guided to Evade AI-Generated Text Detection / N. Lu et al. 2023. 29 p. (arXiv preprint). DOI:10.48550/arXiv.2305.10847
Paraphrasing evades detectors of AI-generated text, but retrieval is an effective defense / K. Krishna et al. Advances in Neural Information Processing Systems, 10-16 December 2023. Vol. 36. New Orleans, USA, 2023. P. 27469-27500. DOI: 10.48550/arXiv.2303.13408
Authorship Obfuscation in Multilingual Machine-Generated Text Detection / D. Macko et al. 2024. 21 p. (arXiv preprint). DOI: 10.48550/arXiv.2401.07867
Foltynek T., Meuschke N., Gipp B. Academic plagiarism detection: a systematic literature review. Association for Computing Machinery Computing Surveys (CSUR). Vol. 52(6). USA, 2019. p. 1–42. DOI: 10.1145/3345317
Survey on Plagiarism Detection in Large Language Models: The Impact of ChatGPT and Gemini on Academic Integrity / S. Pudasaini et al. 2024. 23 p. (arXiv preprint). URL: https://arxiv.org/pdf/2407.13105 (дата звернення: 10.10.2024)
Jawahar G., Abdul-Mageed M., Lakshmanan L. V. Automatic detection of machine generated text: A critical survey. Proceedings of the 28th International Conference on Computational Linguistics, December 2020. Barcelona, Spain, 2020. P. 2296–2309. URL: https://aclanthology.org/2020.coling-main.208 (дата звернення: 10.10.2024)
Chaplynskyi D. Introducing UberText 2.0: A Corpus of Modern Ukrainian at Scale. Proceedings of the Second Ukrainian Natural Language Processing Workshop (UNLP), May 2023. Dubrovnik, Croatia, 2023. P. 1–10. URL: https://aclanthology.org/2023.unlp-1.1 (дата звернення: 21.09.2024)
mT5: A Massively Multilingual Pre-trained Text-to-Text Transformer / L. Xue et al. Human Language Technologies : Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics, June 2021. USA, 2021. P. 483-498. DOI: 10.18653/v1/2021.naacl-main.41
GPT-2 Output Detector Demo. URL: https://openai-openai-detector.hf.space/ (дата звернення: 27.08.2024)
AI Detector – the Original AI Checker for ChatGPT & More. URL: https://gptzero.me/ (дата звернення: 27.08.2024)
