Multimodal eye-tracking data-driven image feature extraction and classification model for reading behavior

Current reading behavior analysis methods primarily rely on single-modal eye movement features and fail to effectively integrate the visual semantic features of text with the spatiotemporal characteristics of eye movements. This makes it difficult for existing methods to comprehensively capture complex reading cognitive processes. To address this issue, this paper proposes a classification model based on the joint analysis of eye movement trajectories and text images. By integrating multimodal eye-tracking dataincluding fixation sequences, pupil diameter variations, and scanning patterns captured via eye-tracking devices-with semantic features extracted from textual images using deep convolutional networks, a multidimensional feature matrix is constructed. Further, a spatiotemporal attention mechanism is employed to dynamically weight eye-movement data, while a graph convolutional network (GCN) is utilized to model the spatial correlations between eye-movement trajectories and textual regions. Experimental validation on both public reading behavior datasets and a self-constructed multimodal dataset demonstrates that the proposed model achieves 91. 4% accuracy in text-type classification tasks, outperforming mainstream multimodal fusion models by 3. 2%. The proposed model also enables precise discrimination between "intensive reading" and "skimming" patterns. In summary, this work introduces a novel paradigm for the quantitative analysis of reading cognitive processes, with potential applications in educational assessment, human-computer interaction optimization, and other related fields.