AUTOMATIC DETECTION OF SPEECH INTENTIONS USING A LARGE LANGUAGE MODEL

Cover Page

Cite item

Full Text

Open Access Open Access
Restricted Access Access granted
Restricted Access Subscription Access

Abstract

The article presents the results of a study of the capabilities of the large language model (GPT-4) to identify the intentional structure of the text. In the course of the research, the following tasks were solved: to develop a methodology for classifying therapist's intentions in psychotherapeutic discourse; to create optimal instructions for working with the model; to analyze the impact of completeness of instructions on the accuracy of identifying intentions by the model; to study the relationship between the accuracy of the classification of intentions by the model with the frequency of occurrence of intentions in the text and the consistency of expert assessments. To analyze the intentional structure of texts, an original Methodology for classifying therapist's intentions in psychotherapeutic discourse has been developed. Using this technique, a team of 3 experts who make decisions about the presence of intent in a remark independently of each other conducted marking up of 14 sessions in Russian (a total of 692 replicas of the therapist). The task of identifying the therapist's intentions was solved by the "GPT-4o-mini" and "GPT-o1" models. The model revealed the intentions of the psychotherapist, realized by him in specific speech utterances (replicas). The conducted research demonstrated the significant capabilities of the large GPT-4 language model in solving the problem of identifying the speaker's speech intentions. The achieved accuracy in classifying the therapist's intentions turned out to be at the level of the best indicators obtained in works on a similar subject. It is shown that improving the instructions significantly increases the quality of the model's operation, and the complexity of the tasks assigned to the model is related to the accuracy of forecasts. Different criteria for the presence of intentions in the therapist's remarks significantly changed the accuracy of the model's predictions.

About the authors

A. V Vanin

FSBI "Institute of Psychology of the Russian Academy of Sciences"

Author for correspondence.
Email: vaninav@ipran.ru
Researcher, Candidate of Psychological Sciences, Master of Information Security Moscow, Russia

A. S Vlasova

FSBI "Institute of Psychology of the Russian Academy of Sciences"; Lomonosov Moscow State University

Email: vlasovaas@ipran.ru
Laboratory assistant. Student Moscow, Russia; Moscow, Russia

E. N Dymova

FSBI "Institute of Psychology of the Russian Academy of Sciences"

Email: dymovaen@ipran.ru
Junior Researcher. Junior Researcher, Laboratory of Developmental Psychology of the subject in normal and post-traumatic conditions Moscow, Russia

V. V Latynov

FSBI "Institute of Psychology of the Russian Academy of Sciences"

Email: latynovv@ipran.ru
Leading researcher. Senior Researcher, Laboratory of Speech Psychology and Psycholinguistics, Candidate of Psychological Sciences Moscow, Russia

A. S Panfilova

FSBI "Institute of Psychology of the Russian Academy of Sciences"

Email: panfilova87@gmail.com
Head of the Laboratory of Artificial Intelligence Technologies in Psychology, Candidate of Technical Sciences Moscow, Russia

P. Y Sereda-Kalinin

FSBI "Institute of Psychology of the Russian Academy of Sciences"

Email: seredapj@ipran.ru
Junior research assistant Moscow, Russia

A. I Tulyankina

FSBI "Institute of Psychology of the Russian Academy of Sciences". Laboratory of Artificial Intelligence Technology in Psychology

Email: anna.tulyankina23@gmail.com
Junior research assistant Moscow, Russia

References

  1. Pavlova N.D., Afinogenova V.A., Grebenshhikova T.A. Diskurs social'nyh media: intencional'nyj podhod. Psikhologicheskii zhurnal. 2023. T. 44. №. 1. S. 81–90.
  2. Pavlova N.D., Grebenshhikova T.A. Postsobytijnyj diskurs v internet-soobshhestvah: intencional'naja struktura i priemy vozdejstvija. Jeksperimental'naja psihologija. 2020. T. 13. № 1. S. 138–148.
  3. Ushakova T.N., Pavlova N.D., Latynov V.V., Cepcov V.A., Alekseev K.I. Slovo v dejstvii: intent-analiz politicheskogo diskursa. Izd-vo Aletejia, 2000.
  4. Breum S.M., Egdal D.V., Mortensen V.G., Moller A.G., Aiello L.M. The persuasive power of large language models. Proceedings of the International AAAI Conference on Web and Social Media. 2024. V. 18. P. 152–163.
  5. Chiu Y.Y., Sharma A., Lin I.W., Althoff T. A computational framework for behavioral assessment of LLM therapists. arXiv preprint arXiv:2401.00820. URL: https://arxiv.org/abs/2401.00820 (date of request: 15.02.2025).
  6. Elyoseph Z., Hadar-Showal D., Asraf K., Lvosky M. ChatGPT outperforms humans in emotional awareness evaluations. Frontiers in Psychology. 2023. V. 14. P. 1199058. DOI: https://doi.org/10.3389/fpsyg.2023.1199058.
  7. Goldstein J.A., Chao J., Grossman S., Stamos A., Tomz M. How persuasive is AI-generated propaganda? PNAS Nexus. 2024. V. 3. № 2. P. pgae034.
  8. Han G., Liu W., Huang X., Borsari B. Chain-of-interaction: Enhancing large language models for psychiatric behavior understanding by dyadic contexts. 2024 IEEE 12th International Conference on Healthcare Informatics (ICHI). IEEE, 2024. P. 392–401.
  9. Hurst A. et al. Gpt-4o system card. arXiv preprint arXiv:2410.21276. URL: https://arxiv.org/abs/2410.21276 (date of request: 20.02.2025).
  10. Karinshak E., Liu S.X., Park J.S., Hancock J.T. Working with AI to persuade: Examining a large language model's ability to generate pro-vaccination messages. Proceedings of the ACM on Human-Computer Interaction. 2023. V. 7. № CSCWI. P. 1–29.
  11. Landis J.R., Koch G.G. The measurement of observer agreement for categorical data. Biometrics. 1977. P. 159–174.
  12. Lee Y., Goldwasser D., Reese L.S. Towards understanding counseling conversations: Domain knowledge and large language models. arXiv preprint arXiv:2402.14200. URL: https://arxiv.org/abs/2402.14200 (date of request: 12.01.2025).
  13. Li A., Lu Y., Song N., Zhang S., Ma L., Lan Z. Understanding the therapeutic relationship between counselors and clients in online text-based counseling using LLMs. arXiv preprint arXiv:2402.11958. URL: https://arxiv.org/abs/2402.11958 (date of request: 05.02.2025).
  14. Matz S.C., Teeny J.D., Vaid S.S., Peters H., Harari G.M., Cerf M. The potential of generative AI for personalized persuasion at scale. Scientific Reports. 2024. V. 14. № 1. P. 4692.
  15. Shin M., Kim J. Large language models can enhance persuasion through linguistic feature alignment. arXiv preprint arXiv:2311.16466. URL: https://arxiv.org/abs/2311.16466 (date of request: 25.12.2024).
  16. Wang T., Zhou N., Chen Z. Enhancing computer programming education with LLMs: A study on effective prompt engineering for Python code generation. arXiv preprint arXiv:2407.05437. URL: https://arxiv.org/abs/2407.05437 (date of request: 10.03.2025).
  17. Wu Y., Hu G. Exploring prompt engineering with GPT language models for document-level machine translation: Insights and findings. Proceedings of the Eighth Conference on Machine Translation. 2023. P. 166–169.
  18. Zhong T. et al. Evaluation of OpenAI O1: Opportunities and challenges of AGI. arXiv preprint arXiv:2409.18486. URL: https://arxiv.org/abs/2409.18486 (date of request: 15.03.2025).

Supplementary files

Supplementary Files
Action
1. JATS XML
Download

Copyright (c) 2025 Russian Academy of Sciences

Согласие на обработку персональных данных

 

Используя сайт https://journals.rcsi.science, я (далее – «Пользователь» или «Субъект персональных данных») даю согласие на обработку персональных данных на этом сайте (текст Согласия) и на обработку персональных данных с помощью сервиса «Яндекс.Метрика» (текст Согласия).