Denoising convolutional neural network (DnCNN) has achieved competitive denoising performance for Gaussian noise using residual learning. The same idea can also be applied in the impulse noise (IN) removal by changing the training data. However, as a discriminative learning-based algorithm, DnCNN lacks flexibility in removing spatially varying noise, such as random-valued impulse noise (RVIN). To address this problem, we propose a fine-tuning CNN based on relaxed Bayesian-optimized support vector machine (BO-SVM) for RVIN removal using a relaxed BO-SVM as a detector and a fine-tuning DnCNN as a denoiser. The relaxed BO-SVM is developed to classify all pixels of the test image into two categories: noise and noise-free. A relaxed rule is provided to preserve the information of slightly corrupted pixels, and the system is trained with an optimal feature set, which comprises of newly introduced local ternary pattern with previously used gray-level difference, average background difference, and median value difference. The key hyperparameters of the relaxed SVM are obtained using the Bayesian optimization technique. Then the DnCNN works for image reconstruction through fine-tuning experiments by the use of the corrupted pixels identified by the detector. The experimental results show that the proposed method reports promising results on RVIN removal over other state-of-the-art denoising methods.