Effective foreground detection in dynamic scenes is a challenging task in computer vision applications. In this paper, we propose a novel background modeling method to tackle this problem. First, we propose a second-order center-symmetric local derivative pattern (CS-LDP) which extracts more detail information compared with the first-order center-symmetric local binary pattern (CS-LBP). Then by concatenating the CS-LBP and CS-LDP histograms, a new hybrid histogram feature is presented. The length of this histogram is much shorter than the local binary pattern (LBP) histogram. Based on this hybrid feature, a novel background modeling method is proposed where the pixel process is modeled with a group of adaptive hybrid histograms. The major advantage of our method is its low complexity. Experiments on three challenging sequences demonstrate that the proposed method is effective and fast, producing comparable results to state-of-art algorithm while reducing the computation time greatly.