A novel background subtraction method that can work under complex environments is presented in this paper. The proposed method consists of two stages: coarse foreground detection through the phase based background model we present, and foreground refinement using the distance transform. We first propose a phase feature which is suitable for background modeling. The background model is then built where each pixel is modeled as a group of adaptive phase features. Although the foreground detection result produced by the background model only contains some sparse pixels, the basic structure of the foreground has been captured as a whole. In the next stage, we adopt the distance transform to aggregate the pixels surrounding the foreground so that the final result is more clear and integrated. Our method can handle many complex situations including dynamic background and illumination variations, especially for sudden illumination change. Besides, it has no bootstrapping limitations, which means our method is without background initialization constraints. Experiments on real data sets and comparison with the existing techniques show that the proposed method is effective and robust.