How Authentic is AI? Comparing AI and Human-Authored EFL Listening Materials

Abstract

This study investigates the linguistic authenticity of AI-generated English as a Foreign Language (EFL) listening materials through a corpus analysis comparing texts produced by ChatGPT-5, Gemini 2.5, and Claude 4.5 against human-authored materials. Building on a pilot study of ChatGPT-4, this research examines how various LLMs replicate spoken discourse features essential for EFL listening and pragmatic skills. Using Sketch Engine, the study analyzed four corpora totaling approximately 101,000 tokens, examining lexical variety, n-gram patterns, and discourse marker usage. Results reveal that while AI-generated texts demonstrate higher type-token ratios (0.128-0.130 vs. 0.107), they significantly underrepresent conversational features crucial for authentic interaction. Human-authored materials contained 50% discourse markers among top keywords compared to 10-25% in AI outputs (χ² = 8.97, p = 0.030). Analysis showed AI corpora, particularly ChatGPT and Claude, exhibited 50-61% formulaic language typical of scripted presentations, contrasting with conversational variability in human texts. Discourse marker frequency was significantly higher in human materials (2.52%) than AI-generated texts (1.15-1.45%). These findings suggest that current LLMs produce language resembling scripted discourse rather than authentic dialogue, limiting their effectiveness for developing listening skills. The study concludes that AI-generated materials require careful supplementation with authentic materials to meet learners’ communicative needs. Implications emphasize balanced integration of AI tools with human-authored content and corpus-based evaluation methods for assessing AI-generated educational materials.