We report a series of eye-tracking studies investigating different facets of how seeing a speaker’s gaze affects listeners’ visual attention and comprehension. We compare the effect of speaker gaze to other cues in the linguistic and non-linguistic context, such as depicted actions and sentence structure. In addition, we discuss top-down influences of the comprehension sub-task, as well as the similarities and differences between using the gaze of a human speaker and that of an artificial agent. Our results suggest that human listeners rapidly make use of speaker gaze as a cue to upcoming conversational content, and that this benefit generalises across a range of situations. At the same time, the extent of gaze-following is affected by sentence structure. These findings are important for processing accounts and models of situated language comprehension, but they also contribute to developing communicative agents that behave in a natural and human-like way.