A Mixed-Method Study on the Effectiveness of Using ChatGPT in Academic Writing and Students’ Perceived Experiences

Cover Page

Cite item

Full Text

Abstract

Introduction: The advent of artificial intelligence in education has brought forward tools like ChatGPT, which can potentially enhance students' academic writing abilities. However, there is limited empirical evidence examining its effectiveness and students' perceptions of its utility in academic contexts.

Purpose: This study aimed to measure the effect of using ChatGPT on students' academic writing abilities and to investigate students’ perceived experiences regarding the use of ChatGPT in their writing process.

Method: An explanatory mixed-method design was employed, incorporating a quantitative experiment followed by a qualitative investigation. The quantitative phase involved 102 fifth-semester students from an English education department at a university in Indonesia. These students were randomized into clusters based on their proximate writing test scores, resulting in two homogenous classes of 25 students each. These classes were then assigned to either an experimental group, which received 14 sessions using ChatGPT as a learning tool for academic writing, or a control group, which received 14 sessions using non-generative tools. Pre-tests and post-tests were administered to both groups. The qualitative phase involved interviews with 10 selected students from the experimental group to explore their perceived experiences with ChatGPT.

Results: The pre-test scores indicated homogeneity between the experimental and control groups, with scores of 57.15 and 56.35 respectively. After the intervention, the post-test scores revealed significant improvement in the experimental group, with an average score of 81.11 compared to 60.30 in the control group. Statistical analysis demonstrated a significant disparity between the two groups (p-value = 0.0000 < 0.05 for the experimental group and p-value = 0.067 > 0.05 for the control group), suggesting that the use of ChatGPT significantly enhanced students' academic writing abilities. The qualitative findings supported these results, with students reporting that ChatGPT facilitated idea generation, organization, and construction in their writing process.

Conclusion: The study concludes that ChatGPT significantly improves students' academic writing abilities, as evidenced by both quantitative and qualitative data. The tool's capacity to assist in the formulation and organization of ideas presents substantial potential for its use in academic research and writing. Given these findings, ChatGPT could be a valuable addition to the educational toolkit for enhancing academic writing skills.

About the authors

Eka Apriani

Institut Agama Islam Negeri (IAIN)

Email: eka.apriani@iaincurup.ac.id
ORCID iD: 0000-0002-8427-2817
Indonesia, Curup

Sholihatul Hamidah Daulay

Universitas Islam Negeri Sumatera Utara

Email: eka.apriani@iaincurup.ac.id
ORCID iD: 0000-0002-7660-8763
Indonesia, Medan

Fitria Aprilia

Universitas Bina Darma

Email: eka.apriani@iaincurup.ac.id
ORCID iD: 0000-0002-2439-6607
Indonesia, Palembang

Abdul Gafur Marzuki

UIN Datokarama Palu

Email: eka.apriani@iaincurup.ac.id
ORCID iD: 0000-0002-7110-3440
Indonesia, Palu

Idi Warsah

Institut Agama Islam Negeri (IAIN)

Email: eka.apriani@iaincurup.ac.id
ORCID iD: 0000-0002-9406-1415
Indonesia, Curup

Dadan Supardan

Institut Agama Islam Negeri (IAIN)

Email: eka.apriani@iaincurup.ac.id
ORCID iD: 0000-0003-0886-6143
Indonesia, Curup

. Muthmainnah

Universitas Al Asyariah Mandar

Author for correspondence.
Email: eka.apriani@iaincurup.ac.id
ORCID iD: 0000-0003-3170-2374
Indonesia, Majene

References

  1. Al Yakin, A., Muthmainnah, M., Apriani, E., Obaid, A. J., & Elngar, A. A. (2023). Transforming online learning management: Generative models on ChatGPT to enhance online student engagement scale (OLE). Idarah (Jurnal Pendidikan Dan Kependidikan), 7(2), P. 135–148. https://doi.org/10.47766/idarah.v7i2.1514
  2. AlAfnan, M. A., Dishari, S., Jovic, M., & Lomidze, K. (2023). Chatgpt as an educational tool: Opportunities, challenges, and recommendations for communication, business writing, and composition courses. Journal of Artificial Intelligence and Technology, 3(2), P. 60–68. https://doi.org/10.37965/jait.2023.0184
  3. Alkaissi, H., & McFarlane, S. I. (2023). Artificial hallucinations in ChatGPT: Implications in scientific writing. Cureus, 15(2), P. 1-4. https://doi.org/10.7759/cureus.35179
  4. AlMarwani, M. (2020). Academic Writing: Challenges and Potential Solutions. Arab World English Journal, 6, P. 114–121. https://doi.org/10.24093/awej/call6.8
  5. Altunkaya, H., & Ayranci, B. (2020). The use of Edmodo in academic writing education. Dil ve Dilbilimi Çalışmaları Dergisi, 16(1), P. 89–103. https://doi.org/10.17263/jlls.712659
  6. Amalia, S., Ramdhani, M. I., Apriani, E., & Boulahnane, S. (2024). The Effect of Learning Management System on Reading Comprehension Across 3 Types of Readers. LEARN Journal: Language Education and Acquisition Research Network, 17(1), P. 73-99. https://so04.tci-thaijo.org/index.php/LEARN/index
  7. Apriani, E., Arsyad, S., Syafryadin, S., Supardan, D., Gusmuliana, P., & Santiana, S. (2022). ICT platforms for Indonesian EFL students viewed from gender during the COVID-19 pandemic. Studies in English Language and Education, 9(1), P. 187–202. https://doi.org/10.24815/siele.v9i1.21089
  8. Apriani, E., Santiana, S., & Harmi, H. (2022). Investigating the role of ICT toward students’ higher order thinking in writing skills at Islamic university students. Al-Ishlah: Jurnal Pendidikan, 14(2), P. 2213–2224. https://doi.org/10.35445/alishlah.v14i2.2260
  9. Apriani, E., Syafryadin, S., Inderawati, R., Arianti, A., Wati, S., Hakim, I. N., & Noermanzah, N. (2021). Implementing e-learning training toward English virtual lecturers: The process, perspectives, challenges and solutions. International Journal of Emerging Technologies in Learning (Online), 16(4), P. 240–255. https://doi.org/10.3991/ijet.v16i04.14125
  10. Bagci Ayranci, B., & Mete, F. (2017). Evaluation of teacher candidates writing skills. Educational Research and Reviews, 12(24), P. 1253–1259. https://doi.org/10.5897/ERR2017.3430
  11. Burkhard, M. (2022). Student perceptions of AI-powered writing tools: Towards individualized teaching strategies. 19th International Conference on Cognition and Exploratory Learning in Digital Age, p. 73–81. https://doi.org/10.33965/celda2022_202207l010
  12. Chaves, A. P., & Gerosa, M. A. (2022). The impact of chatbot linguistic register on user perceptions: A replication study. In A. Følstad, T. Araujo, S. Papadopoulos, E. L.-C. Law, E. Luger, M. Goodwin, & P. B. Brandtzaeg (Eds.), Chatbot Research and Design (vol. 13171, pp. 143–159). Springer International Publishing. https://doi.org/10.1007/978-3-030-94890-0_9
  13. Coniam, D. (2014). The linguistic accuracy of chatbots: usability from an ESL perspective. Text & Talk, 34(5), P. 545-567. https://doi.org/10.1515/text-2014-0018
  14. Yu, C. H. (2008). Book Review: Creswell, J., & Plano Clark, V. (2007). Designing and Conducting Mixed Methods Research. Thousand Oaks, CA: Sage. Organizational Research Methods, 12(4), p. 801–804. https://doi.org/10.1177/1094428108318066
  15. Dempere, J., Modugu, K., Hesham, A., & Ramasamy, L. K. (2023). The impact of ChatGPT on higher education. Frontiers in Education, 8, p. 1206936. https://doi.org/10.3389/feduc.2023.1206936
  16. Dergaa, I., Chamari, K., Zmijewski, P., & Saad, H. B. (2023). From human writing to artificial intelligence generated text: Examining the prospects and potential threats of ChatGPT in academic writing. Biology of Sport, 40(2), p. 615–622. https://doi.org/10.5114/biolsport.2023.125623
  17. Firat, M. (2023). What ChatGPT means for universities: Perceptions of scholars and students. Journal of Applied Learning and Teaching, 6(1), p. 57–63. https://doi.org/10.37074/jalt.2023.6.1.22
  18. Frye, B. L. (2022). Should using an AI text generator to produce academic writing be plagiarism? Fordham Intellectual Property, Media and Entertainment Law Journal, Vol. 33, No. 4. P. 946-968.
  19. Fuchs, K. (2022). The importance of competency development in higher education: Letting go of rote learning. Frontiers in Education, 7, p. 1004876. https://doi.org/10.3389/feduc.2022.1004876
  20. Fuchs, K. (2023). Exploring the opportunities and challenges of NLP models in higher education: Is Chat GPT a blessing or a curse? Frontiers in Education, 8. https://doi.org/10.3389/feduc.2023.1166682
  21. Fyfe, P. (2022). How to cheat on your final paper: Assigning AI for student writing. AI & Society, 38(4), P. 1395–1405. https://doi.org/10.1007/s00146-022-01397-z
  22. Gocen, A., & Aydemir, F. (2020). Artificial Intelligence in Education and Schools. Research on Education and Media, 12(1), P. 13–21. https://doi.org/10.2478/rem-2020-0003
  23. Gordijn, B., & Have, H. T. (2023). ChatGPT: Evolution or revolution? Medicine, Health Care and Philosophy, 26(1), P. 1–2. https://doi.org/10.1007/s11019-023-10136-0
  24. Hsiao, J.-C., & Chang, J. S. (2023). Enhancing EFL reading and writing through AI-powered tools: Design, implementation, and evaluation of an online course. Interactive Learning Environments, P. 1–16. https://doi.org/10.1080/10494820.2023.2207187
  25. Hutson, M. (2022). Could AI help you to write your next paper? Nature, 611(7934), P. 192–193. https://doi.org/10.1038/d41586-022-03479-w
  26. Inderawati, R., Apriani, E., Jaya, H. P., Saputri, K., Wijayanti, E., & Muthmainnah, M. (2024). Promoting students’ writing by using essay writing GPT: A mix method. In Advanced Applications of Generative AI and Natural Language Processing Models (pp. 249–264). IGI Global. https://doi.org/10.4018/979-8-3693-0502-7.ch012
  27. Hamp-Lyons, L. (1984). Book reviews : Jacobs, H.L., Zingraf, S.A. Wormuth, D.R., Hartfiel, V.F. and Hughey, J.B. 1981: Testing ESL composition: a practical approach. Rowley, Massachusetts: Newbury House. xii + 139 pp. ISBN 0-88377-225-6 (v.1). Language Testing, 1(2), P. 241–244. https://doi.org/10.1177/026553228400100210
  28. Jiang, R. (2022). How does artificial intelligence empower EFL teaching and learning nowadays? A review on artificial intelligence in the EFL context. Frontiers in Psychology, 13, P. 1–8. https://doi.org/10.3389/fpsyg.2022.1049401
  29. Jobin, A., Ienca, M., & Vayena, E. (2019). The global landscape of AI ethics guidelines. Nature Machine Intelligence, 1(9), P. 389–399. https://doi.org/10.1038/s42256-019-0088-2
  30. Joyce, A. (2023). Embracing AI: Using Cat-GPT to encourage classroom discussion. College Teaching, P. 1–3. https://doi.org/10.1080/87567555.2023.2251074
  31. Kasneci, E., Seßler, K., Küchemann, S., Bannert, M., Dementieva, D., Fischer, F., Gasser, U., Groh, G., Günnemann, S., & Hüllermeier, E. (2023). ChatGPT for good? On opportunities and challenges of large language models for education. Learning and Individual Differences, 103, P. 102274. https://doi.org/10.1016/j.lindif.2023.102274
  32. King, M. R. (2023). The future of AI in medicine: A perspective from a chatbot. Annals of Biomedical Engineering, 51(2), P. 291–295. https://doi.org/10.1007/s10439-022-03121-w
  33. Kirmani, A. R. (2023). Artificial intelligence-enabled science poetry. ACS Energy Letters, 8(1), P. 574–576. https://doi.org/10.1021/acsenergylett.2c02758
  34. Kung, T. H., Cheatham, M., Medenilla, A., Sillos, C., De Leon, L., Elepaño, C., Madriaga, M., Aggabao, R., Diaz-Candido, G., Maningo, J., & Tseng, V. (2023). Performance of ChatGPT on USMLE: Potential for AI-assisted medical education using large language models. PLOS Digital Health, 2(2), e0000198. P. 1-3 https://doi.org/10.1371/journal.pdig.0000198
  35. La’biran, R., Garoga, M., Karuru, P., & Apriani, E. (2023). Development of listening and speaking textbook for general communication based on laboratory language operating software. Journal of English Education and Teaching, 7(2), P. 235–252. https://doi.org/10.33369/jeet.7.2.235-252
  36. Liebrenz, M., Schleifer, R., Buadze, A., Bhugra, D., & Smith, A. (2023). Generating scholarly content with ChatGPT: Ethical challenges for medical publishing. The Lancet Digital Health, 5(3), P. 105–106. https://doi.org/10.1016/S2589-7500(23)00019-5
  37. Lund, B. D., & Wang, T. (2023). Chatting about ChatGPT: How may AI and GPT impact academia and libraries? Library Hi Tech News, 40(3), P. 26–29. https://doi.org/10.1108/LHTN-01-2023-0009
  38. Lund, B. D., Wang, T., Mannuru, N. R., Nie, B., Shimray, S., & Wang, Z. (2023). ChatGPT and a new academic reality: Artificial Intelligence-written research papers and the ethics of the large language models in scholarly publishing. Journal of the Association for Information Science and Technology, 74(5), P. 570–581. https://doi.org/10.1002/asi.24750
  39. Luo, Y. (2023). Understanding Chinese English-major students’ engagement with supervisory feedback on their undergraduate thesis: Insights from a multiple case study. Innovations in Education and Teaching International, 0(0), P. 1–13. https://doi.org/10.1080/14703297.2023.2238693
  40. Macdonald, C., Adeloye, D., Sheikh, A., & Rudan, I. (2023). Can ChatGPT draft a research article? An example of population-level vaccine effectiveness analysis. Journal of Global Health, 13, P. 1–7. https://doi.org/10.7189%2Fjogh.13.01003
  41. Maharrani, D., Saputri, K., Pratiwi, W. R., Apriani, E., Muthmainnah, M., Warsah, I., & Supardan, D. (2023). The era of e-learning and the challenges for rural areas. In Contemporary Challenges in Education: Digitalization, Methodology, and Management (pp. 224–240). IGI Global. https://doi.org/10.4018/979-8-3693-1826-3.ch017
  42. Manohar, N., & Prasad, S. S. (2023). Use of ChatGPT in academic publishing: A rare case of seronegative systemic lupus erythematosus in a patient with HIV infection. Cureus, 15(2), P. 1-8. https://doi.org/10.7759/cureus.34616
  43. McCarthy, J. (2022). Artificial intelligence, logic, and formalising common sense. In S. Carta (Ed.), Machine Learning and the City (1st ed., pp. 69–90). Wiley. https://doi.org/10.1002/9781119815075.ch6
  44. Michel-Villarreal, R., Vilalta-Perdomo, E., Salinas-Navarro, D. E., Thierry-Aguilera, R., & Gerardou, F. S. (2023). Challenges and opportunities of generative AI for higher education as explained by ChatGPT. Education Sciences, 13(9), P. 856. https://doi.org/10.3390/educsci13090856
  45. Molnár, G., & Szüts, Z. (2018). The role of chatbots in formal education. 2018 IEEE 16th International Symposium on Intelligent Systems and Informatics, P. 197–202. https://doi.org/10.1109/SISY.2018.8524609
  46. Muthmainnah, M., Apriani, E., Seraj, P. M. I., Obaid, A. J., & Al Yakin, A. M. (2024). Nudging motivation to learn english through a ChatGPT smartphone-based hybrid model. In Advanced Applications of Generative AI and Natural Language Processing Models (pp. 26–42). IGI Global. https://doi.org/10.4018/979-8-3693-0502-7.ch002
  47. Park, S., Thieme, A., Han, J., Lee, S., Rhee, W., & Suh, B. (2021, June). “I wrote as if I were telling a story to someone I knew.”: Designing Chatbot Interactions for Expressive Writing in Mental Health. In Proceedings of the 2021 ACM Designing Interactive Systems Conference (pp. 926-941). https://doi.org/10.1145/3461778.3462143
  48. Rahman, M. M., Terano, H. J., Rahman, M. N., Salamzadeh, A., & Rahaman, M. S. (2023). ChatGPT and academic research: A review and recommendations based on practical examples. In M. Rahman, H.J.R. Terano, N. Rahman, A. Salamzadeh, & S. Rahaman, (2023). ChatGPT and academic research: A review and recommendations based on practical examples. Journal of Education, Management and Development Studies, 3(1), P. 1–12. https://doi.org/10.52631/jemds.v3i1.175
  49. Rasul, T., Nair, S., Kalendra, D., Robin, M., de Oliveira Santini, F., Ladeira, W. J., Sun, M., Day, I., Rather, R. A., & Heathcote, L. (2023). The role of ChatGPT in higher education: Benefits, challenges, and future research directions. Journal of Applied Learning and Teaching, 6(1), P. 41-56. https://doi.org/10.37074/jalt.2023.6.1.29
  50. Sallam, M. (2023). ChatGPT utility in healthcare education, research, and practice: Systematic review on the promising perspectives and valid concerns. Healthcare, 11(21), P. 1–20. https://doi.org/10.3390/healthcare11060887
  51. Shidiq, M. (2023, May). The use of artificial intelligence-based chat-gpt and its challenges for the world of education; from the viewpoint of the development of creative writing skills. In Proceeding of international conference on education, society and humanity (Vol. 1, No. 1, pp. 353-357).
  52. Song, C., & Song, Y. (2023). Enhancing academic writing skills and motivation: Assessing the efficacy of ChatGPT in AI-assisted language learning for EFL students. Frontiers in Psychology, 14, P. 1260843. https://doi.org/10.3389/fpsyg.2023.1260843
  53. Stokel-Walker, C. (2022). AI bot ChatGPT writes smart essays-should professors worry? Nature. https://doi.org/10.1038/d41586-022-04397-7
  54. Taecharungroj, V. (2023). “What can ChatGPT do?” Analyzing early reactions to the innovative AI chatbot on Twitter. Big Data and Cognitive Computing, 7(1), P. 35. https://doi.org/10.3390/bdcc7010035
  55. Taufik, M., Rijal, A. S., Dahniar, D., & Apriani, E. (2021). The effectiveness of learning English using LMS Google Classroom during covid-19 pandemic. Al-Ishlah: Jurnal Pendidikan, 13(2), P. 960–970. https://doi.org/10.35445/alishlah.v13i2.706
  56. Thorp, H. H. (2023). ChatGPT is fun, but not an author. Science, 379(6630), P. 313–313. https://doi.org/10.1126/science.adg7879
  57. Toba, R., & Noor, W. N. (2019). The current issues of Indonesian EFL students’ writing skills: Ability, problem, and reason in writing comparison and contrast essay. Dinamika Ilmu, 19(1), P. 57–73. https://doi.org/10.21093/di.v19i1.1506
  58. Tuzlukova, V., Al Busaidi, S., Burns, S., & Bugon, G. (2018). EXPLORING TEACHERS’ PERCEPTIONS OF 21ST CENTURY SKILLS IN TEACHING AND LEARNING IN ENGLISH LANGUAGE CLASSROOMS IN OMAN’S HIGHER EDUCATION INSTITUTIONS. Journal of Teaching English for Specific and Academic Purposes, 6(1), P. 191. https://doi.org/10.22190/jtesap1801191t
  59. Utami, D. M. A., Prihantoro, P., Apriani, E., Hidayah, J., & Handrianto, C. (2021). Empowering ICT potentials in English language teaching. Journal Polingua: Scientific Journal of Linguistics, Literature and Language Education, 10(2), 42–48.
  60. Van Dis, E. A., Bollen, J., Zuidema, W., Van Rooij, R., & Bockting, C. L. (2023). ChatGPT: Five priorities for research. Nature, 614(7947), P. 224–226. https://doi.org/10.1038/ d41586-023-00288-7
  61. Van Slyke, C., Johnson, R. D., & Sarabadani, J. (2023). Generative artificial intelligence in information systems education: Challenges, consequences, and responses. Communications of the Association for Information Systems, 53(1), P. 14. https://doi.org/10.17705/1CAIS.05301
  62. Vanichvasin, P. (2021). Chatbot development as a digital learning tool to increase students’ research knowledge. International Education Studies, 14(2), P. 44–53. https://doi.org/10.5539/ies.v14n2p44
  63. Vetter, M. A., Lucia, B., Jiang, J., & Othman, M. (2024). Towards a framework for local Interrogation of AI ethics: A case study on text generators, academic integrity, and composing with ChatGPT. Computers and Composition, 71, P. 1–12. https://doi.org/10.1016/j.compcom.2024.102831
  64. Wilkenfeld, J. N., Yan, B., Huang, J., Luo, G., & Algas, K. (2022). “AI love you”: Linguistic convergence in human-chatbot relationship development. Academy of Management Proceedings, 2022(1), P. 17063. https://doi.org/10.5465/AMBPP.2022.17063abstract
  65. Yan, D. (2023). Impact of ChatGPT on learners in a L2 writing practicum: An exploratory investigation. Education and Information Technologies, 28(11), P. 13943–13967. https://doi.org/10.1007/s10639-023-11742-4
  66. Yu, S., & Liu, C. (2021). Improving student feedback literacy in academic writing: An evidence-based framework. Assessing Writing, P. 48. https://doi.org/10.1016/j.asw.2021.100525

Supplementary files

Supplementary Files
Action
1. JATS XML
Download


Creative Commons License
This work is licensed under a Creative Commons Attribution 4.0 International License.