Matrix completion (MC) has been successfully applied to many real-world applications, such as recommender systems and image inpainting. Traditional MC methods mainly aim at minimizing the rank function or its surrogate functions. However, the resulting algorithms inevitably calculate the singular value decomposition (SVD) at each iteration, which not only consumes a great deal of time but also greatly limits the potential applications in large-scale data. We propose an MC method that is based on smooth matrix factorization (MF) to deal with the MC problem in the absence of noise. Our method not only inherits the advantage of the MC methods, which are based on the MF and thus avoid calculating the SVD, but also can be easily embedded into many other MF-based MC methods, which makes our method more scalable and flexible. More importantly, different from the existing MF-based MC methods, our method imposes smoothness constraints on each of the factor matrices to help select the factor matrices of high quality, directly leading to the smooth MF, which largely improves the MC performance in both the recovery accuracy and the recovery speed. Two kinds of alternating minimization algorithms are also put forward to numerically solve the proposed model, and extensive experiments further confirm the superior performance of the proposed method over many state-of-the-art methods.