Perceptual hashing is an effective compression technology that maps the content of an image into a brief summary, which is essential for efficient processing in the Big Data era. However, most existing methods process images from a single view, leading to the omission of partial information. In addition, many methods utilize labels to construct a pairwise similarity matrix, which can result in significant time and space expenses. We propose a perceptual hashing algorithm based on collective matrix factorization. In particular, we embed the specific representations of each view and label information into a unified binary code learning framework. Specifically, a semantic label offset scheme is adopted to control the margins dynamically, which can avoid computational overhead, enhance semantic information, and improve the discrimination of the hashing. Experiments on several widely used datasets verify that when the threshold is T=45, the true positive rate for similar images stands at 100%, whereas the correct rate for different images is 99.9990%. Furthermore, the area under the receiver operating characteristic curve value of the proposed algorithm is 0.997510, which is higher than that of the comparison algorithms, indicating that our hashing method has better performance than state-of-the-art methods.