Applying Natural Language Processing and Hierarchical Machine Learning Approaches to Text Difficulty Classification

Renu Balyan, Kathryn S. McCarthy, Danielle S. McNamara

Research output: Contribution to journalArticlepeer-review

34 Scopus citations


For decades, educators have relied on readability metrics that tend to oversimplify dimensions of text difficulty. This study examines the potential of applying advanced artificial intelligence methods to the educational problem of assessing text difficulty. The combination of hierarchical machine learning and natural language processing (NLP) is leveraged to predict the difficulty of practice texts used in a reading comprehension intelligent tutoring system, iSTART. Human raters estimated the text difficulty level of 262 texts across two text sets (Set A and Set B) in the iSTART library. NLP tools were used to identify linguistic features predictive of text difficulty and these indices were submitted to both flat and hierarchical machine learning algorithms. Results indicated that including NLP indices and machine learning increased accuracy by more than 10% as compared to classic readability metrics (e.g., Flesch-Kincaid Grade Level). Further, hierarchical outperformed non-hierarchical (flat) machine learning classification for Set B (72%) and the combined set A + B (65%), whereas the non-hierarchical approach performed slightly better than the hierarchical approach for Set A (79%). These findings demonstrate the importance of considering deeper features of language related to text difficulty as well as the potential utility of hierarchical machine learning approaches in the development of meaningful text difficulty classification.

Original languageEnglish (US)
Pages (from-to)337-370
Number of pages34
JournalInternational Journal of Artificial Intelligence in Education
Issue number3
StatePublished - Oct 1 2020


  • Hierarchical classification
  • Machine learning
  • Natural language processing
  • Text difficulty

ASJC Scopus subject areas

  • Education
  • Computational Theory and Mathematics


Dive into the research topics of 'Applying Natural Language Processing and Hierarchical Machine Learning Approaches to Text Difficulty Classification'. Together they form a unique fingerprint.

Cite this