Shared autonomy can improve teleoperating robotic systems in complex manufacturing and assembly tasks by combining human decision-making and robotic capabilities. A key aspect of seamless collaboration and trust in shared autonomy is the robot’s ability to interpret human intentions in a consistent and explainable manner. To achieve this, a graph neural network-based intention estimation framework is introduced, which generates dynamic graphs that capture spatial relationships evolving over time. The framework predicts human intentions at two hierarchical levels: low-level actions and high-level tasks. Furthermore, we empirically and anecdotally verify the correctness and consistency of the predictions using explainability metrics. The algorithm is demonstrated by teleoperating a bi-manual robot to assemble various block structures in a virtual reality simulation environment.