Robots that can physically interact with humans in a safe manner have the potential to revolutionize application domains like home assistance and nursing care. However, to become long-term companions, such robots must learn userspecific preferences and adapt their behaviors in real time. We propose a Constrained Partially Observable Markov Decision Process framework for modeling human safety preferences over representative variables like force, velocity, and proximity. These variables are modeled as adaptive linear constraints, with a belief over their upper bounds that is updated online based on noisy human feedback. By modeling the belief as phase dependent, the model captures varying preferences across different task phases. The robot then solves a hierarchical optimization to select actions that respect both the learned constraints and robot motion limits. Our method does not require offline training data and can be applied directly to diverse physical interaction tasks and operation modes (tele-operated or autonomous). A pilot study shows that our approach effectively learns user preferences and improves perceived safety while reducing user effort compared to baselines.