A framework for modeling the interaction of syntactic processing and eye movement control

Felix Engelmann, Shravan Vasishth, Ralf Engbert, Reinhold Kliegl

Full Text: PDF   Paper Package: EngelmannVasishthEngbertKliegl2013_1.0 tar.gz PID: 11022/0000-0000-1F2F-B


We explore the interaction between oculomotor control and language comprehension on the sentence level using two well-tested computational accounts of parsing difficulty. Previous work (Boston, Hale, Vasishth, & Kliegl, 2011) has shown that surprisal (Hale, 2001; Levy, 2008) and cue-based memory retrieval (Lewis & Vasishth, 2005) are significant and complementary predictors of reading time in an eyetracking corpus. It remains an open question how the sentence processor interacts with oculomotor control. Using a simple linking hypothesis proposed in Reichle, Warren, and McConnell (2009), we integrated both measures with the eye movement model EMMA (Salvucci, 2001) inside the cognitive architecture ACT-R (Anderson et al., 2004). We built a reading model that could initiate short “Time Out regressions” (Mitchell, Shen, Green, & Hodgson, 2008) that compensate for slow postlexical processing. This simple interaction enabled the model to predict the re-reading of words based on parsing difficulty. The model was evaluated in different configurations on the prediction of frequency effects on the Potsdam Sentence Corpus. The extension of EMMA with postlexical processing improved its predictions and reproduced re-reading rates and durations with a reasonable fit to the data. This demonstration, based on simple and independently motivated assumptions, serves as a foundational step toward a precise investigation of the interaction between high-level language processing and eye movement control.

Topics in Cognitive Science


Anderson, J. R., Bothell, D., Byrne, M. D., Douglass, S., Lebiere, C., & Qin, Y. (2004). An Integrated Theory of the Mind. Psychological Review, 111(4), 1036–1060.

Baayen, R., Piepenbrock, R., & van Rijn, H. (1993). The CELEX lexical data base on CD- ROM. Philadelphia, PA: Linguistic Data Consortium, University of Pennsylvania.

Bartek, B., Lewis, R. L., Vasishth, S., & Smith, M. R. (2011). In search of on-line locality effects in sentence comprehension. Journal of Experimental Psychology: Learning, Memory, and Cognition, 37(5), 1178 –1198.

Bicknell, K., & Levy, R. (2010). A rational model of eye movement control in reading. In Proceedings of the 48th Annual Meeting of the Association for Computational Linguistics (pp. 1168–1178). Uppsala, Sweden: Association for Computational Linguistics.

Binder, K. S., Duffy, S., & Rayner, K. (2001). The effects of thematic fit and discourse context on syntactic ambiguity resolution. Journal of Memory and Language, 44(2),


Booth, R. W., & Weger, U. W. (2013). The function of regressions in reading: Backward eye movements allow rereading. Memory & Cognition, 41(1), 82–97.

Boston, M. F., Hale, J. T., Kliegl, R., Patil, U., & Vasishth, S. (2008). Parsing costs as predictors of reading difficulty: An evaluation using the Potsdam Sentence Corpus. Journal of Eye Movement Research, 2(1), 1–12.

Boston, M. F., Hale, J. T., Vasishth, S., & Kliegl, R. (2011). Parallel processing and sentence comprehension difficulty. Language and Cognitive Processes, 26(3), 301–349.

Budiu, R., & Anderson, J. (2004). Interpretation-based processing: A unified theory of semantic sentence comprehension. Cognitive Science, 28(1), 1–44.

Demberg, V., & Keller, F. (2008). Data from eye-tracking corpora as evidence for theories of syntactic processing complexity. Cognition, 109(2), 193–210.

Elman, J. L., Hare, M., & McRae, K. (2004). Cues, constraints, and competition in sentence processing. In M. Tomasello & D. I. Slobin (Eds.), Beyond nature–nurtur: Essays in honor of elizabeth bates (pp. 111–138). Mahwah, NJ: Lawrence Erlbaum Associates.

Engbert, R., Longtin, A., & Kliegl, R. (2002). A dynamical model of saccade generation in reading based on spatially distributed lexical processing. Vision Research, 42(5), 621–636.

Engbert, R., Nuthmann, A., Richter, E. M., & Kliegl, R. (2005). SWIFT: A dynamical model of saccade generation during reading. Psychological Review, 112(4), 777–813.

Francis, W., & Kucera, H. (1982). Frequency analysis of English usage: Lexicon and grammar. Boston: Houghton Mifflin.

Frank, S. (2009). Surprisal-based comparison between a symbolic and a connectionist model of sentence processing. In N. Taatgen & H. van Rijn (Eds.), Proceedings of the 31st Annual Conference of the Cognitive Science Society (pp. 1139–1144). Amsterdam, Netherlands: Cognitive Science Society.

Frazier, L., & Rayner, K. (1982). Making and correcting errors during sentence comprehension: Eye movements in the analysis of structurally ambiguous sentences. Cognitive Psychology, 14(2), 178–210.

Gibson, E. (2000). The dependency locality theory: A distance-based theory of linguistic complexity. In A. Marantz, Y. Miyashita, & W. O’Neil (Eds.), Image Language Brain: Papers from the first Mind Articulation Project Symposium (pp. 95–126). Cambridge, MA: MIT Press.

Grodner, & Gibson. (2005). Consequences of the Serial Nature of Linguistic Input for Sentenial Complexity. Cognitive Science: A Multidisciplinary Journal, 29(2), 261– 290.

Hale, J. T. (2001). A probabilistic Earley parser as a psycholinguistic model. In Proceedings of the Second Meeting of the North American Chapter of the Association for Computational Linguistics on Language Technologies (pp. 159–166). Pittsburgh, Pennsylvania: Association for Computational Linguistics.

Hale, J. T. (2011). What a rational parser would do. Cognitive Science, 35(3), 399–443.

Inhoff, A., & Weger, U. W. (2005). Memory for word location during reading: Eye movements to previously read words are spatially selective but not precise. Memory & Cognition, 33(3), 447-461.

Just, M. A., & Carpenter, P. A. (1980). A theory of reading: From eye fixations to comprehension. Psychological Review, 87(4), 329–354.

Just, M. A., & Carpenter, P. A. (1992). A capacity theory of comprehension: Individual differences in working memory. Psychological Review, 99(1), 122–149.

Kliegl, R., Grabner, E., Rolfs, M., & Engbert, R. (2004). Length, frequency, and predictability effects of words on eye movements in reading. European Journal of Cogni- tive Psychology, 16(1), 262–284.

Konieczny, L. (2000). Locality and parsing complexity. Journal of Psycholinguistic Re- search, 29(6), 627–645.

Konieczny, L., & Döring, P. (2003). Anticipation of clause-final heads: Evidence from eye-tracking and SRNs. In P. P. Slezak (Ed.), Proceedings of the ICCS/ASCS joint International Conference on Cognitive Science (pp. 330–335). Sydney, Australia: Uni- versity of New South Wales.

Legge, G. E., Hooven, T. A., Klitz, T. S., Mansfield, S. J., & Tjan, B. S. (2002). Mr. Chips 2002: New insights from an ideal-observer model of reading. Vision Research, 42(18), 2219–2234.

Levy, R. (2008). Expectation-based syntactic comprehension. Cognition, 106(3), 1126– 1177.

Lewis, R., & Vasishth, S. (2005). An activation-based model of sentence processing as skilled memory retrieval. Cognitive Science: A Multidisciplinary Journal, 29(3), 375–419.

MacDonald, M. C., & Christiansen, M. H. (2002). Reassessing working memory: Comment on Just and Carpenter (1992) and Waters and Caplan (1996). Psychological Review, 109(1), 35–54.

von der Malsburg, T., & Vasishth, S. (2011). What is the scanpath signature of syntactic reanalysis? Journal of Memory and Language, 65(2), 109–127.

von der Malsburg, T., & Vasishth, S. (2012). Scanpaths reveal syntactic underspecification and reanalysis strategies. Language and Cognitive Processes, in press, available online. doi: 10.1080/01690965.2012.728232

Meseguer, E., Carreiras, M., & Clifton, C. (2002). Overt reanalysis strategies and eye movements during the reading of mild garden path sentences. Memory & Cognition, 30(4), 551–561.

Mitchell, D. C., Shen, X., Green, M. J., & Hodgson, T. L. (2008). Accounting for regressive eye-movements in models of sentence processing: A reappraisal of the Selective Reanalysis hypothesis. Journal of Memory and Language, 59(3), 266–293.

Nilsson, M., & Nivre, J. (2010). Towards a data-driven model of eye movement control in reading. In J. T. Hale (Ed.), Proceedings of the 2010 Workshop on Cognitive Modeling and Computational Linguistics, ACL 2010 (pp. 63–71). Uppsala, Sweden: Association for Computational Linguistics.

Patil, U., Hanne, S., Vasishth, S., Burchert, F., & De Bleser, R. (n.d.). A cue-based retrieval model of offline and online sentence processing in aphasia. (unpublished manuscript)

Patil, U., Vasishth, S., & Kliegl, R. (2009). Compound effect of probabilistic disambiguation and memory retrievals on sentence processing: Evidence from an eye-tracking corpus. In A. Howes, D. Peebles, & R. P. Cooper (Eds.), Proceedings of the 9th International Conference on Cognitive Modeling. Manchester, UK: University of Manchester.

Patil, U., Vasishth, S., & Lewis, R. L. (n.d.). Retrieval interference in syntactic processing: The case of reflexive binding in English. (unpublished manuscript)

R Core Team. (2012). R: A Language and Environment for Statistical Computing. Vienna, Austria: R Foundation for Statistical Computing.

Rayner, K., Kambe, G., & Duffy, S. A. (2000). The effect of clause wrap-up on eye move- ments during reading. The Quarterly Journal of Experimental Psychology, 53A(4), 1061–80.

Reichle, E. D., Pollatsek, A., Fisher, D., & Rayner, K. (1998). Toward a model of eye movement control in reading. Psychological Review, 105(1), 125–157.

Reichle, E. D., Pollatsek, A., & Rayner, K. (2006). E–Z Reader: A cognitive-control, serial-attention model of eye-movement behavior during reading. Cognitive Systems Research, 7(1), 4–22.

Reichle, E. D., Warren, T., & McConnell, K. (2009). Using E-Z Reader to model the effects of higher-level language processing on eye movements during reading. Psychonomic Bulletin & Review, 16(1), 1–21.

Reilly, R. G., & Radach, R. (2006). Some empirical tests of an interactive activation model of eye movement control in reading. Cognitive Systems Research, 7(1), 34–55.

Salvucci, D. (2001). An integrated model of eye movements and visual encoding. Cognitive Systems Research, 1(4), 201–220.

Schilling, H. E. H., Rayner, & Chumbley, J. I. (1998). Comparing naming, lexical decision, and eye fixation times: Word frequency effects and individual differences. Memory & Cognition, 26(6), 1270–1281.

Spivey, M. J., & Tanenhaus, M. K. (1998). Syntactic ambiguity resolution in discourse: Modeling the effects of referential context and lexical frequency. Journal of Experi- mental Psychology: Learning, Memory, and Cognition, 24(6), 1521–1543.

Staub, A. (2010). Eye movements and processing difficulty in object relative clauses. Cognition, 116(1), 71–86.

Van Dyke, J. A., & Lewis, R. L. (2003). Distinguishing effects of structure and decay on attachment and repair: A cue-based parsing account of recovery from misanalyzed ambiguities. Journal of Memory and Language, 49(3), 285–316.

Vasishth, S., Bruessow, Lewis, R., & Drenhaus. (2008). Processing Polarity: How the ungrammatical intrudes on the grammatical. Cognitive Science: A Multidisciplinary Journal, 32(4), 685–712.

Vasishth, S., & Drenhaus, H. (2011). Locality in German. Dialogue & Discourse, 2(1), 59–82.

Vasishth, S., & Lewis, R. L. (2006). Argument-head distance and processing complexity: Explaining both locality and antilocality effects. Language, 82(4), 767–794.

Warren, T., & McConnell, K. (2007). Investigating effects of selectional restriction violations and plausibility violation severity on eye-movements in reading. Psychonomic Bulletin & Review, 14(4), 770–775.

Weger, U. W., & Inhoff, A. (2007). Long-range regressions to previously read words are guided by spatial and verbal memory. Memory & Cognition, 35(6), 1293–1306.