The prospective safety impact assessment with multi-agent traffic simulation is an important approach for evaluating traffic safety effects of advanced driving assistance systems (ADAS) and automated driving systems, especially at their early development stage. The main challenge in the virtual simulation approach is that it requires a detailed and valid representation of interactive processes between driver, system and environment in traffic. Current driver models lack the consideration of driver individuality and often do not cover the entire chain of cognitive processes from information acquisition to action implementation. This led to the development of the Stochastic Cognitive Model (SCM) in order to consider adequately the individual characteristics as well as the cognitive and physical capabilities and weaknesses of the human driver. Most of the relevant information from the environment is retrieved by the driver’s visual sense channel. Based on these pieces of information the driver forms an understanding of the current situation and makes decisions about his future behavior. Hence, modelling the driver’s gaze behaviour, and thus visual perception of the traffic environment and relevant stimuli for action and decision making, is crucial and one of the most challenging issues for driver modelling. This paper focusses on a variety of factors that can have an impact on the driver’s gaze behaviour. In detail, the impact of driver characteristics, driver states and situational traffic circumstances is investigated. Therefore, two experimental studies in real traffic and in a driving simulator were conducted. A significant impact of driver characteristics and driver states on the driver’s gaze behaviour was observed in a driving simulator study with a with representative German driver sample under controlled conditions concerning traffic flow and surrounding. Differences in gaze behaviour, depending on various situational circumstances, in detail, when being faced with specific traffic scenarios, were investigated in a realistic driving study on a German motorway. Results of both studies are merged and reported and finally, the relevance of considering these factors for modelling cognitive driver behaviour is discussed.