As more vehicles on the road are equipped with driver assistance technologies, research on perceived driver discomfort in highly automated vehicles emerged. Driver discomfort estimation is important to automated feature acceptance, user satisfaction and economic factors. Existing literature mainly focused on driver takeover detections, because drivers are more likely to takeover with high level of discomfort. This paper investigated detections of driver discomfort through both takeovers and physiological spikes, to include more subtle and latent discomfort. We present a multimodal dataset from an automated vehicle simulator study, with eye gaze and physiological measurements. The dataset included 32 participants, and each experiment lasted 120 minutes. We recorded their takeover intentions and extracted the physiological spikes that were related to driver discomfort. Machine learning models were then built to detect driver takeovers, physiological spikes and them together through mutli-task learning. Machine learning results showed good performance on takeover and physiological spike detections. The multi-task learning-based detection had improved performance, indicating correlations between takeovers and physiological spikes. Our results demonstrate the potential of driver discomfort prediction through driver’s physiological and behavioral data. The result indicates a potential that AD system can learn user preference from driver without explicit takeovers.