ANALISIS PERBANDINGAN VISION TRANSFORMER DAN CONVOLUTIONAL NEURAL NETWORK DALAM DETEKSI CITRA DEEPFAKE WAJAH MANUSIA DENGAN EVALUASI ROBUSTNESS DAN INTERPRETABILITAS MODEL

Authors

  • Nomensen Mangaratua Siregar Teknik Informatika, Universitas Krisnadwipayana
  • Avip Kurniawan Teknik Informatika, Universitas Krisnadwipayana
  • Nur Hikmah Teknik Informatika, Universitas Krisnadwipayana

DOI:

https://doi.org/10.35316/jimi.v11i2.10842

Keywords:

Deepfake, Vision Transformer, Convolutional Neural Network (CNN), ResNet-50, Robustness

Abstract

The spread of AI-generated deepfakes poses a growing threat to public information integrity, particularly face manipulations that are hard to distinguish from real images. However, existing detection evaluations often focus only on baseline accuracy, overlooking robustness against image degradation, model interpretability, and real-world applicability. This study compares Vision Transformer (ViT-Base) and Convolutional Neural Network (ResNet-50) using a StyleGAN3-based dataset across four dimensions: baseline accuracy, robustness against seven image degradation scenarios, interpretability via Attention Rollout and Grad-CAM, and real-world validation using a FastAPI interface with a soft-voting ensemble. Unlike previous studies that mostly focus on accuracy or hybrid architectures, this study compares the two pure architectures in a single, unified evaluation design. On clean data, ResNet-50 achieves higher accuracy (97.73%) than ViT-Base (94.67%). However, ViT-Base is far more robust to image degradation, with a maximum relative accuracy drop of 24.21% compared with 48.52% for ResNet-50. Grad-CAM and Attention Rollout correctly predicted 10 and 9 of 10 test samples, respectively, while the soft-voting ensemble corrected all ViT-Base errors on 10 out-of-distribution real-world samples. Overall, this study shows a clear performance trade-off between the two architectures, so model selection should match the intended operating environment.

Downloads

Download data is not yet available.

References

[1] Heather Chen and Kathleen Magramo, “Finance worker pays out $25 million after video call with deepfake ‘chief financial officer,’” CNN. Accessed: Nov. 03, 2026. [Online]. Available: https://edition.cnn.com/2024/02/04/asia/deepfake-cfo-scam-hong-kong-intl-hnk/index.html

[2] CNN Indonesia, “Kominfo Bongkar Cara Produksi Video Deepfake Jokowi Berbahasa Mandarin,” CNN Indonesia. Accessed: Sep. 10, 2026. [Online]. Available: https://www.cnnindonesia.com/teknologi/20231026182710-192-1016431/kominfo-bongkar-cara-produksi-video-deepfake-jokowi-berbahasa-mandarin

[3] Kompas.com, “Kasus Video Deepfake Prabowo, Semua Bisa Jadi Korban AI.” Accessed: Sep. 10, 2026. [Online]. Available: https://nasional.kompas.com/read/2025/01/24/07562761/kasus-video-deepfake-prabowo-semua-bisa-jadi-korban-ai

[4] ANTARA News, “Universitas Udayana investigasi kasus dugaan pelecehan seksual daring.” Accessed: Sep. 10, 2026. [Online]. Available: https://www.antaranews.com/berita/4802537/universitas-udayana-investigasi-kasus-dugaan-pelecehan-seksual-daring

[5] R. Tolosana, R. Vera-Rodriguez, J. Fierrez, A. Morales, and J. Ortega-Garcia, “Deepfakes and beyond: A Survey of face manipulation and fake detection,” Information Fusion, vol. 64, pp. 131–148, 2020, doi: https://doi.org/10.1016/j.inffus.2020.06.014.

[6] Y. Mirsky and W. Lee, “The Creation and Detection of Deepfakes,” Jul. 31, 2021, Association for Computing Machinery. doi: 10.1145/3425780.

[7] Z. Akhtar, “Deepfakes Generation and Detection: A Short Survey,” J. Imaging, vol. 9, no. 1, p. 18, Jan. 2023, doi: 10.3390/jimaging9010018.

[8] T. Karras et al., “Alias-Free Generative Adversarial Networks,” in Advances in Neural Information Processing Systems, M. Ranzato, A. Beygelzimer, Y. Dauphin, P. S. Liang, and J. W. Vaughan, Eds., Curran Associates, Inc., 2021, pp. 852–863. [Online]. Available: https://proceedings.neurips.cc/paper_files/paper/2021/file/076ccd93ad68be51f23707988e934906-Paper.pdf

[9] G. Pei et al., “Deepfake Generation and Detection: A Benchmark and Survey,” Mar. 2026.

[10] P. Liu, Q. Tao, and J. T. Zhou, “Evolving from Single-modal to Multi-modal Facial Deepfake Detection: Progress and Challenges,” Apr. 2025.

[11] L. Alzubaidi et al., “Review of deep learning: concepts, CNN architectures, challenges, applications, future directions,” J. Big Data, vol. 8, no. 1, Dec. 2021, doi: 10.1186/s40537-021-00444-8.

[12] M. M. Naseer, K. Ranasinghe, S. H. Khan, M. Hayat, F. Shahbaz Khan, and M.-H. Yang, “Intriguing Properties of Vision Transformers,” in Advances in Neural Information Processing Systems, M. Ranzato, A. Beygelzimer, Y. Dauphin, P. S. Liang, and J. W. Vaughan, Eds., Curran Associates, Inc., 2021, pp. 23296–23308. [Online]. Available: https://proceedings.neurips.cc/paper_files/paper/2021/file/c404a5adbf90e09631678b13b05d9d7a-Paper.pdf

[13] K. Shiohara and T. Yamasaki, “Detecting Deepfakes with Self-Blended Images,” in 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2022, pp. 18699–18708. doi: 10.1109/CVPR52688.2022.01816.

[14] A. Dosovitskiy et al., “An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale,” Jun. 2021, [Online]. Available: http://arxiv.org/abs/2010.11929

[15] K. Han et al., “A Survey on Vision Transformer,” IEEE Trans. Pattern Anal. Mach. Intell., vol. 45, no. 1, pp. 87–110, 2023, doi: 10.1109/TPAMI.2022.3152247.

[16] Y.-J. Heo, Y.-J. Choi, Y.-W. Lee, and B.-G. Kim, “Deepfake Detection Scheme Based on Vision Transformer and Distillation,” Apr. 2021.

[17] D. A. Coccomini, R. Caldelli, F. Falchi, C. Gennaro, and G. Amato, “Cross-Forgery Analysis of Vision Transformers and CNNs for Deepfake Image Detection,” Jun. 2022, doi: 10.1145/3512732.3533582.

[18] N. and G. C. and F. F. Coccomini Davide Alessandro and Messina, “Combining EfficientNet and Vision Transformers for Video Deepfake Detection,” in Image Analysis and Processing – ICIAP 2022, C. and L. M. and F. G. M. and T. F. Sclaroff Stan and Distante, Ed., Cham: Springer International Publishing, 2022, pp. 219–229.

[19] H. Ha, M. Kim, S. Han, and S. Lee, “Robust DeepFake Detection Method based on Ensemble of ViT and CNN,” in Proceedings of the 38th ACM/SIGAPP Symposium on Applied Computing, New York, NY, USA: ACM, Mar. 2023, pp. 1092–1095. doi: 10.1145/3555776.3577769.

[20] D. Dagar and D. Kumar Vishwakarma, “Tex-ViT: A Generalizable, Robust, Texture-based Dual-Branch Cross-Attention Deepfake Detector.”

[21] S. A. Hasan, M. P. Hera, A. L. Mim, M. M. Jihad Hossain, H. Ali, and M. F. Bulbul, “Beyond Frame‐Level Accuracy: Analyzing Video‐Level Stability in Transformer‐Based Deepfake Forensics,” International Journal of Intelligent Systems, vol. 2026, no. 1, Jan. 2026, doi: 10.1155/int/6801006.

[22] N. A. Chandra et al., “Deepfake-Eval-2024: A Multi-Modal In-the-Wild Benchmark of Deepfakes Circulated in 2024,” May 2026, [Online]. Available: http://arxiv.org/abs/2503.02857

[23] J. Yosinski, J. Clune, Y. Bengio, and H. Lipson, “How transferable are features in deep neural networks?,” Nov. 2014, [Online]. Available: http://arxiv.org/abs/1411.1792

[24] K. He, X. Zhang, S. Ren, and J. Sun, “Deep Residual Learning for Image Recognition,” in 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2016, pp. 770–778. doi: 10.1109/CVPR.2016.90.

[25] H. M and S. M.N, “A Review on Evaluation Metrics for Data Classification Evaluations,” International Journal of Data Mining & Knowledge Management Process, vol. 5, no. 2, pp. 01–11, Mar. 2015, doi: 10.5121/ijdkp.2015.5201.

[26] R. R. Selvaraju, M. Cogswell, A. Das, R. Vedantam, D. Parikh, and D. Batra, “Grad-CAM: Visual Explanations from Deep Networks via Gradient-Based Localization,” in 2017 IEEE International Conference on Computer Vision (ICCV), 2017, pp. 618–626. doi: 10.1109/ICCV.2017.74.

[27] S. Abnar and W. Zuidema, “Quantifying Attention Flow in Transformers.”

Published

15-11-2026

How to Cite

Siregar, N. M., Kurniawan, A., & Hikmah, N. (2026). ANALISIS PERBANDINGAN VISION TRANSFORMER DAN CONVOLUTIONAL NEURAL NETWORK DALAM DETEKSI CITRA DEEPFAKE WAJAH MANUSIA DENGAN EVALUASI ROBUSTNESS DAN INTERPRETABILITAS MODEL. Jurnal Ilmiah Informatika, 11(2), 185–199. https://doi.org/10.35316/jimi.v11i2.10842

Similar Articles

1 2 > >> 

You may also start an advanced similarity search for this article.