Linguistic Comparison of AI- and Human-Written Responses to Online Mental Health Queries
The ubiquity and widespread use of digital and online technologies have transformed mental health support, with online mental health communities (OMHCs) providing safe spaces for peer support. More recently, generative AI and large language models (LLMs) have introduced new possibilities for scalable, around-the-clock mental health assistance that could potentially augment and supplement the capabilities of OMHCs.
Although generative AI shows promise in delivering immediate and personalized responses, its effectiveness in replicating the nuanced, experience-based support of human peers remains an open question. In this study, we harnessed 24,114 posts and 138,758 online community (OC) responses from 55 OMHCs on Reddit. We prompted several state-of-the-art LLMs (GPT-4-Turbo, Llama-3, and Mistral-7B) with these posts, and compared their responses to human-written (OC) responses based on a variety of linguistic measures across psycholinguistics and lexico-semantics.
Key Findings
Our findings revealed several distinctions between AI-generated and human-written responses:
- Verbosity: AI responses were found to be more verbose compared to their human counterparts.
- Readability: The analytical structure of AI responses made them more readable, facilitating easier understanding.
- Linguistic Diversity: In contrast, AI responses lacked the linguistic diversity and personal narratives that are inherent in human-human interactions.
- Qualitative Insights: Through qualitative examination, we identified that AI responses often maintain a neutral stance and do not seek back-and-forth clarifications that are typical in human interactions.
Ethical and Practical Implications
The integration of generative AI into OMHCs raises a number of ethical and practical considerations. While AI offers scalability and timeliness in mental health support, it lacks the authenticity, social interactiveness, and expertise that form the ethos of online support communities. As such, we advocate for frameworks that ensure a balanced approach, leveraging the strengths of both AI and human interactions.
In conclusion, while generative AI presents exciting opportunities for enhancing mental health support, it is essential to recognize its limitations. The unique qualities of human responses, characterized by emotional depth and personal experience, are irreplaceable in the context of mental health support. Future research should focus on developing hybrid models that combine the efficiency of AI with the empathetic understanding of human interactions.