A Reference Model for Driver Attention in Automation: Glance Behavior Changes During Lateral and Longitudinal Assistance

This paper introduces a reference model of glance behavior for driving safety assessment. This model can improve the design of automated and assistive systems. Technological limitations have previously hindered the use of unobtrusive eye trackers to measure glance behavior in naturalistic conditions. This paper presents a comprehensive analysis of eye-tracking data collected in a naturalistic field operation test, using an eye tracker that proved to be robust in real-world driving scenarios. We describe a post-processing technique to enhance the quality of naturalistic eye-tracker data, propose a data-analysis procedure that captures the important features of glance behavior, and develop a model of glance behavior (based on distribution fitting), which was lacking in the literature. The model and its metrics capture key defining characteristics of, and differences between, on- and off-path glance distributions, and during manual driving and driving with adaptive cruise control and lane keeping aid active. The results show that drivers’ visual response is tightly coupled to the driving context (vehicle automation, car-following, and illumination).


I. INTRODUCTION
H UMAN error due to inattention has been identified as the major crash-contributing factor by large-scale naturalistic studies [1], [2] and in-depth crash investigations [3].Demand for advanced driver assistance systems (ADASs) to counteract human errors is increasing, as is evident from the growth of ADAS functionality and market penetration.The effectiveness of these systems at detecting abnormal behaviors depends on the existence of a reference model of normative behavior, which must be able to interpret the driver state (e.g., attentiveness) in relation to the driving context (e.g., level of automation and characteristics of the driving environment) [4].
ADASs that monitor the state of the vehicle and the driving environment are now available on the market.ADASs that monitor the state (e.g., attentiveness) of the driver are still in their infancy [5], but interest in their development is accelerating.There are many practical challenges to be resolved before a driver attention monitor can become a reality, however.These challenges are primarily related to data quality issues and the identification of relevant and robust indicators of attention (taking into account feasibility and equipment cost) [5].Early indicators relied on vehicle control metrics (e.g., headway and lane keeping quality) to measure driver attention [5]- [7].These indirect measures, although easy to collect, are not sufficient to capture the detrimental effects of inattention [8].A significant disadvantage of vehicle control metrics is that, in automated and autonomous driving-hands off wheel, feet off pedals-such metrics would not be available.
A potential approach is to supplement driving metrics with physiological metrics [5].Driver's attention could be inferred by a wide spectrum of physiological signals (for a review see [6]); this paper focuses on measuring eye movements because they are a strong indicator of where attention is directed [9].Visual attention is essential for guiding the vehicle as well as detecting and avoiding hazards on the road [10], [11].Eye movements can be described at different levels of detail.Although it is possible to measure fixations, saccades, and smooth pursuit movements [4], [5], [8], this places high demands on eye-tracker performance.Therefore, an aggregate level of measure is typically used-the glance, which is the transition of the eyes to an area of interest followed by one or more continuous fixations within that area, until the eye moves to another area of interest (ISO 15007-1:2014).Several glance-based metrics can be derived (see e.g., ISO 15007-1:2014), but there is no general 1524-9050 © 2018 IEEE.Personal use is permitted, but republication/redistribution requires IEEE permission.
agreement on the features that best describe attention and inattention (for an overview see [4]).Further, a model of visual behavior for normative driving in the real world is lacking.
Research has shown that drivers look towards the future path (road center region) about 80-90% of the time (see [12]).Hence, a coarse measure of attention based on whether a glance is on-path (as used in this paper) may be sufficient.Frequency and duration of off-path glances have traditionally been central metrics of interest.In fact, driving requires short periods of off-path glances to scan the surroundings and safety-critical in-vehicle information.However, long off-path glances (about 2 s) in single or aggregated form, have been shown to be a sensitive indicator of increasing crash risk, in real-world driving [1], [13] and simulator studies [14].off-path glance duration has also been included as an assessment criterion for in-vehicle electronic devices [15].However, off-path glances that lead to crashes do not necessarily have to be long, because timing in relation to driving context matters more than duration per se [2].To capture context dependencies, off-path glance duration can be complemented with percent road center (PRC) [12].PRC, a simple metric that combines glance duration and frequency to quantify the percent of glances directed at the road center, has been shown to be sensitive to changes in driving demand in simulated [12] and real-world driving [2].Moreover, PRC inherently takes into account the amount of time the drivers spend looking on-path.on-path glance behavior, often discounted but included in our analysis, was found to be an important indicator of safety and crash risk by a recent study [16].
Remote (i.e., unobtrusive) eye trackers can be the perfect companion to ADASs.Distraction detection algorithms that incorporate eye-tracking data have been proposed (for a comparison of the different algorithms see [5] and [17]), and some solutions are also available on the market [5].The main issue is that remote eye trackers, while accurate in constrained laboratory settings, are too expensive and unreliable (due to quality issues) to be used in naturalistic settings [5], [18]- [20].Therefore, large-scale naturalistic driving studies have been relying on the manual annotation of glances from video recordings of the driver's face [1], [2].This method is cumbersome, limiting the size of the dataset available for analysis, and it is not suitable for real-time system development.Recent advances in technology and machine vision algorithms now enable the robust and extensive measurement of glance behavior in real-world driving with simpler setups (for a review see [21] and [22]); the dataset used in this paper is an example of this.
The aims of this paper are fourfold: first, to describe a novel post-processing technique to remove noise and artifacts from eye-tracker signals collected in naturalistic settings; second, to propose a comprehensive data analysis procedure that captures the important features of glance behavior, and offers a more nuanced interpretation of eye tracking data; third, to develop a reference model of visual behavior for real-world driving (including driving with ADASs) that would enable a comprehensive assessment of glance behavior and supplement previously used glance-based metrics; and lastly, to assess how visual behavior adapts to automation and driving context.

A. Data Source
The data used in this study are from the EyesOnRoad naturalistic field operational test (EOR-FOT).The EOR-FOT project was conducted by Volvo Cars and Autoliv to test a new eye tracker system [23], [24].Data were collected from ten Volvo cars (2014 V60 model) in the region of Västra Götaland, Sweden, from December 2014 to September 2015.Most of the data were collected in Göteborg, the secondlargest city in Sweden.The cars were equipped with full-range adaptive cruise control (ACC) and lane keeping aid (LKA).ACC automated the longitudinal control by modulating the speed when the vehicle approached a lead car.LKA supported lateral control at speeds above 60 km/h by providing steering torque when the vehicle approached the lane markings.
Vehicle data were continuously collected at 60 Hz from the controller area network (CAN) bus and three in-car cameras.The cameras recorded the forward and rearward views from the vehicle and the face of the driver.Eye movements were recorded by an eye-tracking system that automatically classified glances as either on-path and off-path, depending on whether they were inside or outside an ellipse 20 • wide and 14 • high with the center reference point being drivers' eyes [23].When the eye tracker was not able to classify a glance sample, the location was coded as unknown.
In total 19 drivers participated in the FOT (10 males and 9 females).The subset of data used for this paper-extracted according to the criteria explained next-included a total of 17 drivers (9 males, 8 females) with an average age of 50 years (SD 12 years).The manual driving group contained 8 drivers (3 males, 5 females) with an average age of 52 years (SD 7 years).The ACC+LKA driving group contained 12 drivers (8 males, 4 females) with an average age of 49 years (SD 12 years).Because only 3 drivers (2 male and 1 female, IDs: 3, 11, 14) belonged to both groups, we considered the manual and ACC+LKA driving groups independent.

B. Driving Segment Extraction and Classification
A segment is an interval of continuous driving data that fulfill specific inclusion criteria.Several signals recorded from the CAN bus were used for segment selection.Overall, the inclusion criteria constrained the analysis to non-safetycritical driving on straight roads (rural roads and highways).The inclusion criteria were as follows: • Curve radius greater than 1500 m, to exclude winding roads and lane changes; • Speed above 60 km/h, to exclude urban driving and match the LKA operating conditions; • Absolute value of longitudinal acceleration less than 1.5 m/s 2 , to remove harsh braking and unusually high acceleration; • No ADAS warnings issued (e.g., forward collision warning), to exclude potential safety-critical events; • Segment longer than 20 s, to ensure steady state driving.Furthermore, each driving segment was classified according to three binary categorical variables: Automation level, Car-following, and Illumination.If the driving segment could not be assigned to any of the categorical variables, it was discarded.The variable Automation level has two levels: Manual, when the assistance systems (i.e., ACC and LKA) were turned off, and ACC+LKA when the assistance systems were turned on and operational.In ACC+LKA driving, the segments were excluded if the driver overrode the ACC by accelerating or braking, or changed the settings of the system (speed and time gap) in the segment.The category Car-following had two levels: CF yes, when the distance to the lead vehicle was less than 50 m, and CF no otherwise.The category Illumination was either Day or Night, which was inferred from the current time, date, and position of the vehicle (derived from the GPS signal).Together, the three categories provide eight combinations, all of which were considered in the analysis.A total of about 163 hours of driving time were available for the analysis.

C. Data Filtering
A multiple-step procedure, based on previous research on glance quality assessment [20], began by verifying the eye-tracking glance classification before analysis.Although the eye-tracker itself provided a real-time data quality assessment, and the EOR-FOT project reported on data reliability [23], we developed a novel and more comprehensive procedure to verify glance data quality.A number of studies have proposed techniques to classify raw eye-tracking data into a binary glance on-and off-path signal [20], [25]; however, previous studies did not produce a post-processing technique to ensure that 1) the filtered signal complies with physiological constraints on human eye motion, and that 2) fragmented glances are discarded as low quality.Our procedure addresses these two issues to improve glance analysis.
First, the quality of the eye-tracking data for individual glances was verified by visual inspection of a sample of 45 driving segments that were 20 s long.The inspection was performed frame by frame with the support of FOTware [26] by comparing the eye-tracker signal with the video of the face of the driver.Second, the eye-tracking signal was post-processed to reject questionable data (e.g., unrealistic glance durations) in the extracted driving segments according to a novel filtering strategy.The filtering strategy consisted of six steps (as depicted in Fig. 1).First, fragmented glances at the beginning or end of the segment were discarded.Second, on-path and off-path glances that were nonconsecutive (because of loss of tracking in between) were discarded altogether if the transition between them was unknown.Third, a brief (less than 0.3 s) loss of tracking between glances of the same category (on-path or off-path) was interpolated.Fourth, glances shorter than 0.15 s were interpolated, as they were shorter than the physiological minimum fixation time [27].Fifth, a brief-less than 0.1 s-loss of tracking between an on/off path glance transition was appended to the subsequent dwell time on the next glance location.Sixth, off-path glances shorter than 0.3 s were interpolated to remove blinks (as suggested in ISO 15007-1:2014).

D. Description of Visual Behavior
1) Empirical Glance Distribution: Glance data were analyzed, at aggregate level, as empirical probability density functions (PDFs), so that the histogram has a total area of one; 500 equally spaced bins, 0.1 s wide, centered at every 0.1 s were used (i.e., the bin centers spanned the interval 0.1-50 s starting from the lowest bin edge at 0.05 s).
As the dataset was unbalanced (because some drivers had more driving time than others), the aggregate distributions were built upon glance distributions weighted per individual driver.This approach was chosen so that drivers with more driving time would not unduly affect the aggregate distribution.The weighting procedure was as follows.First, for each driver, the histogram of glance data was normalized as a relative frequency histogram.That is, the height of each bar of the histogram is the probability of each bin, which is obtained by dividing the count in each bin by the total count (so that the sum of the bin heights sums to one).Then, each corresponding bin probability was summed across drivers.Finally, the aggregate histogram was normalized to a density.
Measures of location (e.g., percentiles and mean) were computed over the aggregate, weighted glance distribution as follows.Each bin's center value was replicated by a factor proportional to the height of each bar, essentially discretizing the range of glance duration at 10 Hz.Because the height of the bar represents a probability, the height can be close to zero.Therefore, the height of each bar was multiplied by a weight (w), chosen to ensure that the observation with minimum non-zero bar height ( p) had unitary frequency: w = 1/mi n( p) with p > 0. Afterwards, each bin center value was replicated w times.
2) Modeling of Visual Behavior: The glance distributions were parametrized by fitting a PDF, in order to obtain an analytical model of glance behavior.The fitting used maximum likelihood estimation, given a class of continuous distributions with unknown parameter values.For the on-path glances, an inverse normal distribution was selected.The PDF is given in (1): For the off-path glances, a lognormal distribution was chosen.The PDF is given in (2): 3) Glance Metric [Percent Road Center (PRC)]: The PRC is the proportion of time that glances fall within a road center area (i.e., on path glances) [12].This metric does not require that the eye-tracking signal to be segmented into glances.Hence, the eye-tracking signals used to calculate PRC included fragmented and nonconsecutive glances; Steps 1 and 2 of the filtering process were not performed.PRC is equivalent to percent time on the area of interest-essentially, the on-path inverse of the percentage of eyes off road time (PEORT) defined in ISO 15007-1:2014.The PRC of each segment was computed via the chunking technique [28] to increase the robustness of measures and facilitate the comparison between driving segments.
Chunking was applied as follows.Each segment was uniformly partitioned into non-overlapping 10 s intervals.If the segment duration was not an integer multiple of the length of the intervals (chunks), the residuals were discarded.To compute the median PRC on a time series, first the median PRC was computed for each chunk in the segment, then the median of the chunks' PRC medians was used as a measure of the driving segment.As the distribution of PRCs is usually skewed toward one, the median was chosen as a more suitable measure of location than mean.

4) Glance Metrics (Percent Glance Duration):
The metric quantifying percent off-path glance durations exceeding 2 s (PGDoff≥2) is comparable to Percentage of extended duration glances (included in ISO 15007-1:2014) which quantifies long glances-previously associated with increased crash risk.Percent on-path glance durations shorter than 1 s (PGDon≤1) is a novel metric, with a newly discovered association with crash risk (see [16]).This metric is intended to quantify on-path information uptake deficiencies when glancing on-path between off-path glances, because short on-path glances may not provide enough information for proper accumulation of information about looming or path trajectories (e.g., see [29] and [30]).Together, PGDoff≥2 and PGDon≤1 identify safe visual scanning, which is characterized by off-path glances that are not too long and on-path glances that are not too short.

5) Statistical Analysis:
A two-sided Wilcoxon rank sum test was used to test the null hypothesis that the samples are from continuous distributions with equal medians.A multiway analysis of variance (ANOVA) was used to test the interaction of the categorical variables with the mean of the dependent variable (if the dependent variable was not normally distributed, an arcsine data transformation was applied first).If the results from the ANOVA were statistically significant, the Tukey's honest significant difference post-hoc was carried out.A chi-square test was used to compare sample proportions and assess the goodness of fit.All tests were run at the 0.05 level of significance.Correlation between the on-path and the following off-path glance duration at aggregate level was assessed with the Spearman's correlation coefficient ρ.

A. Data Quality
The glance distribution data for manual driving were compared to data from two naturalistic driving studies to verify Fig. 2. Comparison of the empirical probability density functions (PDFs) of the glance duration for manual driving from this study with the 100car and SHRP2 studies.Panels (a-b) show, respectively, the comparison to the on-path and off-path glances of the baseline segments from the 100car study [31], [32].Panel (c) shows the comparison to the off-path glances of the baseline segments from the SHRP2 study.the data quality and the filtering procedure detailed in section II-C (Fig. 2).The data were compared to the manual driving baseline from 100car [33] (data available in [32] and further processed in [31]) and SHRP2 [2] (data available as additional material in [34]).The glance data in 100car and SHPR2 were manually labelled.In (a) and (b) the glance distributions were truncated at 6 s, to match the length of the segments in the 100car baseline dataset.The comparison shows good agreement (Fig. 2).Hence, the data used in this study was deemed to be of high quality.
1) Empirical Glance Distributions: Fig. 3 shows that the on-path and off-path glance distributions for manual and ACC+LKA driving have similar shapes.The on-path glance distribution matches an inverse-normal distribution, whereas the off-path glance distribution matches a lognormal distribution (more details will be given in section III-A2).The cumulative distribution functions (CDFs) in panels (b) and (d) in Fig. 3 illustrate a tendency to longer glances in ACC+LKA compared to manual driving, especially on path.There was a weak correlation (ρ = −0.12) between the duration of an on-path glance and the following off-path glance in both manual and ACC+LKA driving.
Fig. 4 and Fig. 5 show the glance distributions (for manual and ACC+LKA driving, respectively) in more detail.The figures also report the results of individual drivers to show the variability between drivers.Panels (a) and (b) in both figures show longer on-path glances in ACC+LKA and a tendency towards longer off-path glances in ACC+LKA.Panel (c) in both figures show that the proportion of off-path glances exceeding 2 s (PGDoff≥2) in manual and ACC+LKA driving was not statistically different (ς 2 (1) = 3.54, p = 0.06, φ = 0.004).However, the proportion of on-path glances shorter than 1 s (PGDon≤1) was significantly higher in manual driving than in ACC+LKA driving (ς 2 (1) = 39.08,p < 0.001, φ = 0.014).Panel (d) in both figures displays the median PRC at aggregate level and per individual driver (the analysis of PRC values is presented in section III-A3).Panel (d) also shows the proportion of data per driver.
3) Visual Response to Driving Context: Fig. 7 presents a series of boxplots showing the distribution of the median PRC with respect to the variables Automation level, Car-following, and Illumination.The upper and lower limits of the notch on the box plots approximate the 95% confidence interval for the median [35], [36].The notch was computed as where M is the median, I Q R is the inter-quantile range (i.e. the range between the 25th quantile [Q1] and the 75th quantile [Q3]), and n is the size of the sample [35].The whiskers extend from Q1 − 1.5 Fig. 7a shows the comparison of samples grouped with respect to the levels of Automation level (Manual and ACC+LKA).The comparison indicates that the median PRC when driving with ACC+LKA is significantly lower than in manual driving (W = 1.11•10 7 , p < 0.001, r = 0.09).Fig. 7b shows the dataset grouped with respect to Automation level and Car-following.In general, the trend is towards a significant increase in median PRC when there is a lead vehicle compared to when there is not, in both manual (W = 7.28 • 10 5 , p < 0.001, r = 0.34) and ACC+LKA driving (W = 1.07•10 7 , p < 0.001, r = 0.10).The increase of median PRC seems to be stronger in manual driving than in ACC+LKA driving, and a 2-way ANOVA (followed by the post-hoc test) confirmed this significant interaction between Automation level and Car-following (F(10653; 1) = 99.5, p < 0.001).Fig. 8a shows that the median THW was longer in manual driving than in ACC+LKA driving (W = 1.65 • 10 6 , p < 0.001, r = 0.19).The boxplots also suggest that THW variation was larger in manual driving than in ACC+LKA driving.In both conditions, the THW was longer than the safety-critical value of 0.5 s [37].
Fig. 7c shows the dataset split with respect to Automation level, Car-following, and Illumination.In general, the trend is towards an increase of the median PRC at night, compared to during the day.The median PRC in manual driving without any vehicle in front at night was significantly higher than in daylight (W = 2.26 • 10 5 , p < 0.001, r = 0.9).However, during car-following, the effect of Illumination was not significant (W = 1.38 • 10 5 , p = 0.11, r = 0.06).In ACC+LKA driving, the effect of Illumination was significant whether there was a vehicle in front (W = 8.26 • 10 5 , p < 0.001, r = 0.36) or not (W = 1 • 10 7 , p < 0.001, r = 0.45).A 3-way ANOVA (followed by the post-hoc test) confirmed the interaction between Car-following and Illumination (F(10649; 1) = 3.92, p < 0.05).Fig. 8b shows that drivers kept a significantly longer THW to the lead vehicle when driving at night compared to driving in daytime, for both manual (W = 7.80 • 10 4 , p < 0.001, r = 0.16) and ACC+LKA driving (W = 8.28 • 10 5 , p < 0.001, r = 0.12).

A. Analysis and Modeling of Glance Behavior
The key strength of the present study is the large amount of eye-tracking data collected in naturalistic settings.The data used in this study were consistent with the naturalistic datasets 100car [33] and SHRP2 [2] (created from manual video reduction, see fig.Fig. 2).The eye-tracking signal was shown to be reliable and robust in real-world driving, and the quality of the data were improved with a sophisticated filtering procedure.This accomplishment has major implications for the real-time assessment of glance behavior in naturalistic settings.The amount of data at our disposal allowed us to set strict inclusion criteria for the analysis and focus on normative visual behavior, excluding safety-critical events.Thus in this paper we were able to overcome a common limitation of naturalistic studies-the lack of control over driving scenarios.
In the literature, the statistical description of glance data usually consists only of mean and standard deviation [14], [38].However, the mean does not accurately describe the population if the distribution is skewed (as is the case with glance data), which can lead to misinterpretation.In this study, median and other quantiles were considered better choices for describing glance distributions.We sought to establish a method to give a deeper understanding of the data, and facilitate comparisons with other datasets.This paper makes use of extensive graphical representations, to provide a better perspective on the empirical data.Furthermore, this paper displays trends, patterns, and uncertainties in the dataset, instead of limiting the analysis to inferential tests comparing groups.Weighting procedures to deal with an unbalanced dataset were also applied.
To our knowledge, this is the first comprehensive study that also provides an analytical model (via parameter fitting of PDFs) of visual behavior in normal real-world driving, including on-and off-path glances.off-path glance data have been shown to follow a lognormal distribution [38], but the parameters have not been published; ours is the first reference model for on-path glance behavior to date.The use of analytic formulas to describe the generative distribution of visual behavior has several benefits.First, each glance distribution can be described by a minimum set of parameters (in this case two).Second, the model can compute any descriptive statistics of choice, which simplifies the comparison between studies.Third, the model could be used in counterfactual  simulations (e.g., see [34]) and in the development of real-time monitoring systems.Because of the weak correlation between on-path glances and subsequent off-path glances, random samples from the marginal distributions (on-path and off-path) may be used to mimic drivers' visual behavior in normal driving.
To our knowledge, this is also the first study that documents the effects of ACC and LKA on glance behavior.Although there were no striking differences in the aggregate glance duration distributions, the effect of vehicle automation was evident in the analysis of PRC (more details in the following section), particularly in response to driving context.
Up to now, few studies have investigated on-path glance behavior in a systematic way.Therefore, there is much more qualitative and quantitative information about off-path glances.However, a recent study highlighted the relevance of on-path glance behavior for evaluating safety and crash risk [16].on-path glance behavior is important because this is when the uptake of information about the driving environment occurs, which is then used to control the vehicle [29], [30].
The study [16] found that drivers who ended up in a safety-critical situation had shown reduced glance duration to on-path locations and an increase in glance frequency between on-and off-path locations in the approach to the precipitating event.Although the present study excluded safety-critical events, the findings suggest that on-path glances may actually be more sensitive to the driving context than off-path glances.For example, the comparison between manual and ACC+LKA driving indicates that the major difference in glance distribution was in the on-path glances rather than in the off-path ones; ACC+LKA driving was characterized by longer glances on path, and fewer on-path glances less than 1 s (PGDon≤1).Notably, no striking difference was found in the off-path glance distributions.As mentioned in the introduction, off-path glances exceeding 2 s (PGDoff≥2) have been associated with increased crash risk.Interestingly, the results from this study show an average percentage of 4% of naturally occurring long glances in normal driving.The safety implication of this finding is not clear; the interaction between glance length and timing with regard to unexpected changes in the driving environment [2] requires further research.

B. Visual Response to Driving Context
1) The Effect of Automation: The significant decrease of PRC (eyes on path) when using ACC+LKA compared to manual driving (Fig. 7a) is consistent with previous studies on the real-world effect of ACC [37], [39], [40].The lower PRC is interpreted as a symptom of a reduction in driving task demand when using vehicle automation, because of vehicle allocation of some of the control tasks-longitudinal and lateral control.Note that Morando et al. [39] showed that this PRC reduction effect occurs in steady-state driving with ACC, and that glances do return on-path in response to vehicle deceleration regulation if the lead vehicle brakes.A difference in PRC in the off-path glance distribution between manual and ACC+LKA driving was also expected.Although the comparison of the glance distributions suggests a tendency towards longer off-path glances when using vehicle automation, this result was not significant.Surprisingly, there were longer on-path glances while driving with ACC+LKA than in manual driving.We found higher PRC in manual driving than in ACC+LKA, yet longer on-path glances while driving with ACC+LKA.That is, more time was spent with eyes on path in manual driving, but while driving with ACC+LKA the on-path glances are longer.This result could be attributed to the fact that PRC embeds glance duration and frequency, whereas the glance distributions do not take into account the aggregate effect.This observation confirms the claim that there is not a single metric able to describe the attentiveness of the driver [4].In fact, frequent short glances and infrequent long glances may yield the same PRC value [12].
The decrease of attention to the forward path when using automation may be considered potentially unsafe.However, it is important to distinguish between eyes off-threat and eyes off-path; for example, drivers look back at the road in response to vehicle deceleration from ACC (see [39]).The detrimental effects of taking the eyes off path are somewhat dependent on whether there is a lead vehicle, which poses a potential threat.The next section substantiates this claim.
2) The Effect of Car-Following: Because a lead vehicle is a potential threat, drivers need to attend to the vehicle in front and assess whether the safety margin-which may be based on the drivers' expectation that the lead vehicle will not suddenly brake-is enough to control the crash risk.Previous studies have shown that, in manual driving, drivers change their visual scanning pattern in car-following.The lead vehicle becomes the focus of attention, and the spatial distribution of eye fixations is narrower [38], [41], [42].
The results from this study are consistent with this previous research.Normal (baseline) driving PRC values seem to range between about 0.70 and 0.89 in the literature, depending on context [12], [42].In a 2004 paper by Tijerina et al. [41], the mean PRC was 0.86 in car-following.Although they gave no values for other scenarios, they mentioned a lower PRC in open road driving.Note also that their on-path study was carried out during the day; the same trend was found in our study (Fig. 7b).The value of PRC, however, was higher (in daylight, the median PRC was 0.96 and the mean was 0.91-computed here for sake of comparison).This difference in PRC between studies could be attributed to different car-following scenarios.In their study the median THW was 1.71 s and the median THW in daylight was lower, 1.36 s.A lower THW probably requires a higher level of attention, which could explain the higher PRC in our study.In [42] the median PRC was 0.89 with lead vehicle vs. 0.87 without lead vehicle (which was found to be a non-significant difference).Most of the trips described in 2014 by Tivesten and Dozza [42] took place in clear weather, without taking illumination into account.As before, the difference in PRC between studies is probably related to different car-following scenarios and road curvatures, among other contextual factors.Unfortunately, the THW was not reported in their paper.However, as the car-following segments were identified when the car ahead was within 150 m-greater than the criterion used in this study (50 m)-it can be assumed that the average THW in their paper was longer than the one in the present study.
The effects of the lead vehicle on glance behavior in manual and ACC+LKA driving are similar, but not identical.In manual driving, the increase in attention in response to a vehicle ahead was greater.A possible explanation is that when driving manually, to keep a safe headway to the lead vehicle, the driver predominantly uses visual cues, such as looming-the optical expansion of the lead vehicle in the eyes of the driver [39], [43], [44].Because looming perception is impaired at the eye periphery [45], [46], an increase in glances to the forward path is expected.
During manual driving, the driver is fully responsible for adjusting the speed to keep a safe headway to the vehicle in front.These adjustments are automated when the ACC is activated.The driver, however, is still required to supervise the system and take control if needed.Although the results show a general decrease of PRC when using ACC+LKA, the results also indicate an increase in visual attention to the forward path when following a lead vehicle.These results suggest that drivers were responsive to the change in driving demand due to the increase of crash risk exposure.Despite the automation of longitudinal control, drivers may proactively increase their attention in order to be able to respond promptly in case of sudden changes in the traffic environment (e.g., harsh braking of the lead vehicle or sensor failures).These findings agree with results from our previous studies of safety-critical events with the use of ACC [39], [40]: drivers proactively increase their attention to the forward path in anticipation of a lead-vehicle conflict.
The median THW when using ACC+LKA was lower than in manual driving, which is not consistent with the findings from on-path studies on the use of ACC (see, for example, [37]).The study by Malta et al. [37] showed that the average THW increased with ACC compared to manual driving; for example, in motorways the average THW in manual driving was 3.51 s, compared to 4.08 s when using ACC.These values are considerably higher than the median THW of 1.38 s in manual driving and 1.12 s in ACC+LKA driving found in this study; the discrepancy could be explained by the different driving-segment inclusion criteria.The inclusion criteria in their study were vehicle speed above 50 km/h and THW > 0 [37].These criteria are looser than the ones used in this study (see section II-B).Their study [37] also showed that ACC reduced the occurrence of critical time gaps (defined as THW < 0.5 s).Although in this study no median THW was shorter than 0.5 s, Fig. 8a shows a tendency towards shorter THW in manual driving than in ACC+LKA (note that 1 s corresponds to the minimum time-gap setting available to the driver).Although this shorter THW could be considered unsafe, it needs to be interpreted taking into account the perception-reaction time of the human compared to an automated system.The average perception-response time for an attentive driver is about 1.5 s [47], but the time varies greatly for different driving situations, with driver expectation being the most important variable [47], [48].The response time of the ACC is arguably faster in certain situations.Moreover, on-path studies have found that in manual driving, drivers tend to maintain a headway that is not sufficient to avoid a collision (less than 1 s) if the lead vehicle brakes suddenly [49].Surprisingly, it was found that drivers were not aware that the headway was unsafe, because they greatly underestimate the THW (by an average of 1.6 s) [49].Commonly, in manual driving, a THW of about 2 s is recommended; in particular, a Swedish regulation sets the minimum safe THW at 1 s [50].ACC has the potential to compensate for these perceptual limitations.
3) The Effect of Illumination: The results from this study indicate a general increase of PRC toward the forward path at night compared to daytime, in both manual and ACC+LKA driving.The interpretation is that, at night, the preview distance of the road ahead is shorter, and the detection and response to obstacles on the road is degraded [38].In addition, there is less to fixate on in darkness.Therefore, eye fixations tend to be more concentrated on the forward path, near the area illuminated by the headlamps [38], [51], [52].Moreover, the lead vehicle-in particular the taillights-may provide additional guidance support for lane keeping.Fig. 8b shows that there is also a general increase in THW at night compared to daytime in manual and ACC+LKA driving.
Unfortunately, a literature search revealed little quantitative information regarding the associations between PRC, carfollowing, and illumination.In manual driving, specifically when following a lead vehicle, the effect of illumination on visual behavior was less noticeable.This result may be explained by the fact that the presence of the lead vehicle already caused the median PRC to be 0.96 in daylight; therefore, a further increase would be hard to detect.This result seems to be in agreement with what was found in the 1989 paper by Olson et al. [51].They found that the mean percent time that the drivers' eyes are directed toward the center road (comparable with PRC) at night appears to be unaffected by the presence of a lead vehicle (around 80% in both conditions).
In ACC+LKA, the effect of illumination was more noticeable; the effect was similar whether or not there was a lead vehicle.These results suggest that the shorter preview of the road ahead due to darkness generally increases the need for higher alertness-whether or not a lead vehicle is present.

C. Limitations
Naturalistic studies have a limitation intrinsic to their design: lack of control over participants and driving scenarios.This limitation was addressed by weighting the glance distribution with respect to the driver and defining clear inclusion criteria for the driving segments.However, the heterogeneity of the drivers sampled made a between-subject analysis unfeasible.Another limitation in our study is that the eye-tracker did not provide detailed information about the off-path areas of interest that the eyes were directed to, or the glance eccentricity (i.e., the radial angle between the forward path and the glance location).For example, some off-path glances (e.g.those to the right or left mirrors) may be critical for safety, but off-path glances further away from the forward pathway may be dangerous.This study did not distinguish between these different off-path glances.

D. Future Directions
In our future research, we intend to: • Identify and quantify naturally occurring distraction events by identifying time-sharing sequences [20], [53].
Visual time-sharing sequences can overcome semantic task types (e.g., as in [1]) by focusing on the visual behavior associated with a distracting task; • Investigate "check" glances, off-path glances that are normally about 0.3 s long.They are usually filtered out because they cannot be easily separated from blinks.Check glances, however, are interesting because they are part of the vision-for-action cognitive mechanism, and presumably are used at the initiation of each new action [11] (e.g., to guide the hand movement before interacting with the infotainment system).

V. CONCLUSIONS
Real-time inattention countermeasures can enhance driving safety by providing feedback to the driver and adapting the functionalities of ADASs to the driver state.Unfortunately, previous attempts to develop real-time driver monitoring algorithms have been hindered by limitations in eye-tracking technology, particularly data quality issues.In this paper, we presented a reference model for driver attention based on data collected as part of a naturalistic FOT with vehicles equipped with a prototype eye tracker that is well suited to real-world applications.
There are four major outcomes from our research.(1) We derived a novel post-processing technique to enhance the quality of eye tracker data collected in real world environment and (2) we proposed a new data analysis procedure that captures the important features of glance behavior.Further, (3) we proposed the first reference model of visual behavior, which was lacking in the literature.This model and its metrics capture key defining characteristics and differences in the distribution of on-and off-path glance distributions while driving manually compared to driving with ACC+LKA, showing that (4) drivers' visual response is tightly coupled to the driving context (use of vehicle automation, car-following, and illumination).
This research has several practical applications.The analysis of visual behavior presented in this paper-in particular, the analytical model-can be valuable for developing driver models that can be used in computer simulations (e.g., counterfactual analyses) for designing and evaluating the safety benefit of ADASs.Further, the results can support the definition of guidelines for in-vehicle electronic devices and improve the rulemaking process.Finally, the findings could inform the design of driver-state monitoring systems based on eye tracking.These systems are becoming increasingly important due to the increased focus on vehicle automation and the need for automated vehicles to cooperate with the human driver.

Fig. 1 .
Fig. 1.Example of data filtering of the eye-tracking signal.The numbered labels identify the steps of the filtering procedure.Details of the filtering procedure are found in the text.

Fig. 3 .
Fig. 3. Comparison of the empirical probability density functions (PDFs, left), and empirical cumulative distribution functions (CDFs, right) of the on-path (top row) and off-path (bottom row) glances for manual driving and driving with ACC+LKA.

Fig. 4 .
Fig. 4. Glance distributions in manual driving.Panel (a) shows the empirical probability density function (PDF) of the on-path glance duration.Panel (b) shows the PDF for the off-path glance duration.The measures of location of the distributions are reported in (a) and (b); the subscript and superscript indicate the min and max value over the individual drivers, respectively.Panel (c) shows the empirical cumulative distribution function (CDF) at aggregate level and for the individual drivers.The proportion of on-path glances shorter than 1 s (PGDon≤1) and the proportion of off-path glances exceeding 2 s (PGDoff≥2) are pinpointed in (c) (the subscript and superscript indicate the min and max value over the individual driver, respectively).Panel (d) displays the median percent road center (PRC) per driver (circles in the chart), with the error bars indicating the 95% confidence interval.The horizontal line in panel (d) indicates the median of the median drivers' PRC with the grey area indicating the 95% confidence interval.The bars in (d) indicates the glance data proportion per driver.

Fig. 5 .
Fig. 5. Glance distributions in ACC+LKA driving.Panel (a) shows the empirical probability density function (PDF) of the on-path glance duration.Panel (b) shows the PDF for the off-path glance duration.The measures of location of the distributions are reported in (a) and (b); the subscript and superscript indicate the min and max value over the individual drivers, respectively.Panel (c) shows the empirical cumulative distribution function (CDF) at aggregate level and for the individual drivers.The proportion of on-path glances shorter than 1 s (PGDon≤1) and the proportion of off-path glances exceeding 2 s (PGDoff≥2) are pinpointed in (c) (the subscript and superscript indicate the min and max value over the individual driver, respectively).Panel (d) displays the median percent road center (PRC) per driver (circles in the chart), with the error bars indicating the 95% confidence interval.The horizontal line in panel (d) indicates the median of the median drivers' PRC with the grey area indicating the 95% confidence interval.The bars in (d) indicates the glance data proportion per driver.

Fig. 6 .
Fig.6.Fitting of the glance distributions in manual (top row) and ACC+LKA driving (bottom row).The fitted probability density function (PDF) is displayed by the thicker line in the chart, whereas the empirical PDF is displayed in gray.

Fig. 7 .
Fig. 7. Boxplots of the percent road center (PRC) distribution, grouped according to the values of the categorical variables Automation level, Car-following and Illumination.In panel (a) the data were grouped solely with respect to the values of Automation level.In panel (b) the categorical variable Car-following was included.In panel (c) the categorical variable Illumination was also incorporated.The thick median line in the boxplot indicates that the pairwise comparison of the medians is significantly different.The value of the grand median is indicated next to the boxplot.Outliers are shown as dots.

Fig. 8 .
Fig. 8. Boxplots of the median time headway (THW) for each driving segment in car-following.The thick median line (value at right) in the boxplot indicates that the pairwise grand medians are significantly different.Outliers are shown as dots.