With the wide popularity of surveillance cameras, video synopsis technology has become a research hotspot. The existing methods of surveillance video synopsis usually summarize the input video by shifting the object tube in the video on the time axis, which ignore the serious collision artifacts and chronological disorder between moving objects. To solve these problems, we propose a surveillance video synopsis methodology called “surveillance video synopsis based on spatio-temporal offset (STO)” that can simultaneously shift the moving object in the temporal domain and spatial domain. First, object detection and tracking algorithms are used to extract the object tube from the input video. Two collision relations are proposed by analyzing relationship between tubes to classify collision artifacts. Then, we present two spatial offset cases to find the optimal spatial offset of the object tube. Finally, an adaptive optimization frame density model is proposed to analyze the optimal temporal offset of the object tube. Simultaneously, the object tube and the background are stitched according to the STO to generate the synopsis video. Extensive experimental results demonstrate the effectiveness of the proposed method in improving frame compression rate and alleviating collision artifacts.