Trajectory prediction is an essential task towards understanding object movement and human behavior from observed sequences. Current methods usually assume that the observed sequences are complete while ignoring the potential for missing values in the observations due to object occlusion, scope limitation, sensor failure, etc., hindering the trajectory prediction accuracy. In our paper, a unified framework, Graph-based Conditional Variational Recurrent Neural Network (GC-VRNN), is proposed to jointly conduct trajectory imputation and prediction. Specifically, a novel Multi-Space Graph Neural Network (MS-GNN) is developed to extract spatial features from incomplete observations, leveraging missing patterns. Additionally, we adopt a Conditional VRNN with a specifically designed Temporal Decay (TD) module to model temporal dependencies and temporal missing patterns of incomplete trajectories. Notably, valuable information is shared via temporal flow. We curate and benchmark three practical datasets for the joint problem of trajectory imputation and prediction. Extensive experiments verify the superior performance of our model. To the best of our knowledge, our work is the pioneer which fills the gap in benchmarks and techniques for trajectory imputation and prediction in a unified way.