Twenty-First International Conference on Technology, Knowledge, and Society

Abstract

With advancements in technology, generative AI (GenAI) has the potential to adapt text to different reading difficulty levels. However, the appropriateness of AI-generated output has not yet been thoroughly explored, it remains unclear whether GenAI tools align with the linguistic trend of predetermined difficulty level. To address this gap, this study explores the extent to which generative AI tools, i.e., ChatGPT and Brisk Teaching, align with expected linguistic trends when adapting English literature for different grade levels. The primary objective was to examine readability scores, vocabulary level, syntactic complexity, and lexical complexity in AI-generated texts to evaluate their suitability for educational purposes. The research calculated the readability score using the Flesch Reading Ease score, cross-referenced Oxford 3000 and 5000 to understand the distribution of the vocabulary level, and calculated syntactic and lexical complexity metrics using TAALES and TAALED tools. Texts were generated at three reading levels (Grades 6, 9, and 12) and analyzed through ANOVA to identify statistically significant patterns. Results indicate that while Brisk Teaching demonstrated clearer trends in vocabulary control, neither tool consistently aligned with expected patterns of increasing complexity or readability across grade levels. In addition, while variations in lexical difficulty and syntactic complexity are observed across different levels, these changes are not reflected in the readability levels. This suggests that relying solely on a single readability measure cannot comprehensively represent the reading difficulty of a text. Future research is recommended to incorporate multiple evaluation methodologies for a more comprehensive understanding of AI-generated text characteristics.