Dexterousmanipulationwithmulti-fingerhandscan be sensitive tophysical parameters suchasobject shape, pose, and friction coefficients.While simulation enables large-scale datacollectionwithknownparametervalues, simulation-trained policiesmust stillhandleuncertaintyatdeployment,where the trueparametersandthereforethetruedynamicsareunknown. Standarddomainrandomizationstrategiesmaybeinsufficientfor precisetaskslikescrewdriverturning,asmanipulationstrategies may need to change depending on specific parameter values. Toaddress this,weproposeProbabilisticLatentUnifiedworld ModelingandparameterEstimation(PLUME), aworldmodel that jointly learns toevolveabelief overparametervalues as well as the systemdynamics conditionedon thoseparameters. Welearnalatentspacetojointlyrepresentmultiplequalitatively different physical parameters alongwith rewards, themselves functions of partially-observablevariables, to informplanning. Our novel learning framework leads to efficient alignment of theworldmodel to true dynamics through online parameter inferenceasopposedtore-trainingorfine-tuning.Weevaluate ourmethod on simulated screwdriver turning, valve turning, bucket lifting, anddiskflicking tasks, aswell as ahardware screwdriver turning task, where we achieve successful zero shot transfer of our simulation-trainedpolicyandoutperform state-of-the-artofflinereinforcement learningandworld-model augmentedbehaviorcloningbaselines.