Robot-mediated physical interaction between humans, or physical Human-Robot-Human Interaction (pHRHI for short) can transcend the physical constraints associated with direct human-human interaction such as constant interaction medium and the need for proximity. Control for pHRHI is related to bilateral teleoperation, but pHRHI must necessarily focus on human safety features on both sides of the interaction in a general setting. This paper presents a safety-aware framework for a robot mediated bilateral physical interaction between two humans in general setting while handling safety constraints with different priorities on two sides. The proposed framework also allows a user to haptically feel any number of constraints that are active on the other side, enabling a safe and intuitive interaction. Additionally, this approach is not dependent on the robot model and allows generalization across robots. We have formulated the safety-aware pHRHI problem as a hierarchical optimization framework with higher-priority safety tasks and lower-priority bilateral interaction and constraint communication tasks. We validated this framework by assessing the bilateral interaction, hierarchical constraint handling, and constraint communication performances on a dual human-robot setup. We also compared the performance with current approaches based on constrained optimization without any hierarchy or constraint communication feature. Results show that the proposed framework can handle prioritized physical/safety constraints, enabling bilateral interaction and providing a physical feeling of the other robot’s constraints.