Deepfakes and manipulation techniques such as adversarial attacks pose a significant threat to society today. It is essential for the detection methods used against them to be more robust than ever. Generation techniques such as neural texture, along with adversarial attacks, result in visually realistic manipulated media and make it difficult for the current state-of-the-art (SoTA) to detect the manipulation. We propose an ensemble-based approach and a genetic algorithm-based approach for fake imagery detection that has been perturbed using fast gradient sign method and basic iterative method techniques. We collectively put such methods under the umbrella term, “EnsembleDet.” We evaluate the proposed approaches using a perturbed version of the benchmark dataset of FaceForensics++, which consists of data generated from manipulation techniques such as Face2Face, neural textures, among others. We present a comparative analysis of quantitative results with SoTA, XceptionNet, and MesoNet, which depict the superiority of the proposed methods over the current SoTA. We also highlight the aspects where the proposed methods have a scope for improvement.