Predicting the behavior of agents such as pedestrians and vehicles presents an intricate challenge due to a myriad of factors including multi-modality, inter-agent interactions, traffic/environmental rules, individual inclinations, and agent dynamics. Consequently, a plethora of neural network-driven prediction models have been introduced in the literature to encompass these intricacies to accurately predict the agent behavior. Nevertheless, many of these approaches falter when confronted with scenarios beyond their training datasets, and lack interpretability, raising concerns about their suitability for real-world applications such as autonomous driving. Moreover, these models frequently demand additional training, substantial computational resources, or specific input features necessitating extensive implementation endeavors. In response, we propose a robust prediction method for autonomous vehicles that can provide a solid baseline for comparison when developing new algorithms as well as a sanity check when deploying prediction on real-world vehicles.