Academic

From Flat to Structural: Enhancing Automated Short Answer Grading with GraphRAG

arXiv:2603.19276v1 Announce Type: cross Abstract: Automated short answer grading (ASAG) is critical for scaling educational assessment, yet large language models (LLMs) often struggle with hallucinations and strict rubric adherence due to their reliance on generalized pre-training. While Rretrieval-Augmented Generation (RAG) mitigates these issues, standard "flat" vector retrieval mechanisms treat knowledge as isolated fragments, failing to capture the structural relationships and multi-hop reasoning essential for complex educational content. To address this limitation, we introduce a Graph Retrieval-Augmented Generation (GraphRAG) framework that organizes reference materials into a structured knowledge graph to explicitly model dependencies between concepts. Our methodology employs a dual-phase pipeline: utilizing Microsoft GraphRAG for high-fidelity graph construction and the HippoRAG neurosymbolic algorithm to execute associative graph traversals, thereby retrieving comprehensive,

Y
Yucheng Chu, Haoyu Han, Shen Dong, Hang Li, Kaiqi Yang, Yasemin Copur-Gencturk, Joseph Krajcik, Namsoo Shin, Hui Liu · · 1 min read · 5 views
#cs.CL #cs.AI

arXiv:2603.19276v1 Announce Type: cross Abstract: Automated short answer grading (ASAG) is critical for scaling educational assessment, yet large language models (LLMs) often struggle with hallucinations and strict rubric adherence due to their reliance on generalized pre-training. While Rretrieval-Augmented Generation (RAG) mitigates these issues, standard "flat" vector retrieval mechanisms treat knowledge as isolated fragments, failing to capture the structural relationships and multi-hop reasoning essential for complex educational content. To address this limitation, we introduce a Graph Retrieval-Augmented Generation (GraphRAG) framework that organizes reference materials into a structured knowledge graph to explicitly model dependencies between concepts. Our methodology employs a dual-phase pipeline: utilizing Microsoft GraphRAG for high-fidelity graph construction and the HippoRAG neurosymbolic algorithm to execute associative graph traversals, thereby retrieving comprehensive, connected subgraphs of evidence. Experimental evaluations on a Next Generation Science Standards (NGSS) dataset demonstrate that this structural approach significantly outperforms standard RAG baselines across all metrics. Notably, the HippoRAG implementation achieved substantial improvements in evaluating Science and Engineering Practices (SEP), confirming the superiority of structural retrieval in verifying the logical reasoning chains required for higher-order academic assessment.

Executive Summary

This article introduces GraphRAG, a novel framework for enhancing automated short answer grading (ASAG) by leveraging a structured knowledge graph to address the limitations of standard retrieval-Augmented Generation (RAG) approaches. The GraphRAG framework employs a dual-phase pipeline, utilizing Microsoft GraphRAG for high-fidelity graph construction and the HippoRAG neurosymbolic algorithm for associative graph traversals. Experimental evaluations on a Next Generation Science Standards (NGSS) dataset demonstrate significant improvements over standard RAG baselines, particularly in evaluating Science and Engineering Practices (SEP). The GraphRAG approach has the potential to revolutionize ASAG by enabling the capture of structural relationships and multi-hop reasoning essential for complex educational content.

Key Points

  • ▸ GraphRAG is a novel framework that addresses the limitations of standard RAG approaches in ASAG.
  • ▸ The GraphRAG framework employs a dual-phase pipeline for high-fidelity graph construction and associative graph traversals.
  • ▸ Experimental evaluations demonstrate significant improvements over standard RAG baselines, particularly in evaluating SEP.

Merits

Strength in addressing hallucinations and strict rubric adherence

The GraphRAG framework's structured knowledge graph approach enables the capture of structural relationships and multi-hop reasoning, reducing the likelihood of hallucinations and improving strict rubric adherence.

Improved performance in evaluating complex educational content

The GraphRAG approach enables the capture of complex relationships between concepts, resulting in improved performance in evaluating Science and Engineering Practices (SEP).

Demerits

Complexity and computational requirements

The GraphRAG framework's dual-phase pipeline and reliance on high-fidelity graph construction may introduce complexity and computational requirements, potentially limiting its widespread adoption.

Limited generalizability to other domains

The GraphRAG approach is specifically designed for educational content, and its effectiveness in other domains, such as business or law, may be limited.

Expert Commentary

The GraphRAG approach is a significant innovation in the field of ASAG, and its potential to address the limitations of standard RAG approaches is substantial. However, the complexity and computational requirements of the framework may limit its widespread adoption. Additionally, the limited generalizability of the GraphRAG approach to other domains is a concern. Despite these limitations, the GraphRAG approach has significant implications for education policy and the development of next-generation assessment tools and evaluation methods.

Recommendations

  • ✓ Further research is needed to address the complexity and computational requirements of the GraphRAG framework and to explore its generalizability to other domains.
  • ✓ The GraphRAG approach should be explored in a variety of educational contexts to validate its effectiveness and identify potential limitations.

Sources

Original: arXiv - cs.AI

Related Articles