Determining object of fixation is an important factor in many application of intelligent vehicles including driver’s situational awareness estimation. The objective of this work is to infer the object of fixation given fixation is occurring. We propose a system architecture that identifies object tracks in the scene, derives object characteristics independent of and jointly with gaze behavior, and utilizes a spatio-temporal sensitive machine learning framework to estimate the likelihood of an object being the object of fixation. Performance evaluation is conducted on a dataset of on-road driving, centered around urban intersections, with manual annotations of the object of fixation. Our proposed system can achieve up to 83% average precision accuracy when compared to a baseline of 78%. Furthermore, comparing the effects of different combinations of object characteristics on precision and recall accuracy show promising insights on factors affecting the reliable estimation of object of fixation. Determining object of fixation is an important factor in many application of intelligent vehicles including driver’s situational awareness estimation. The objective of this work is to infer object of fixation given fixation is occurring. We propose a system architecture that identifies object tracks in the scene, derives object characteristics independent of and jointly with gaze behavior, and utilizes a spatio-temporal sensitive machine learning framework to estimate the likelihood of an object being the object of fixation. Performance evaluation is conducted on a dataset of on-road driving, centered around urban intersections, with manual annotations of object of fixation. Our proposed system can achieve up to 83% average precision accuracy when compared to a baseline of 78%. Furthermore, comparing the effects of different combinations of object characteristics on precision and recall accuracy show promising insights on factors affecting the reliable estimation of object of fixation.