A summary of attacks and defenses for VLP models
| Attack/Defense | Method | Year | Category | Subcategory | Target model | Dataset |
|---|---|---|---|---|---|---|
| Adversarial Attack | Co-Attack (Zhang et al., 2022) | 2022 | White-box | Invisible | ALBEF, TCL, CLIP | MS-COCO, Flickr30K, RefCOCO+, SNLI-VE |
| AdvCLIP (Zhou et al., 2023) | 2023 | White-box | Invisible | CLIP | STL10, GTSRB, CIFAR10, ImageNet, Wikipedia, Pascal-Sentence, NUS-WIDE, XmediaNet | |
| Typographical Attacks (Noever and Noever, 2021) | 2021 | White-box | Visible | CLIP | ImageNet | |
| Multi-Image Typographical Attacks (Wang et al., 2025e) | 2025 | White-box | Visible | OpenCLIP, InstructBLIP | ImageNet, LAION | |
| SGA (Lu et al., 2023) | 2023 | Black-box | Sample-wise | ALBEF, TCL, CLIP | Flickr30K, MS-COCO | |
| SA-Attack (He et al., 2023a) | 2023 | Black-box | Sample-wise | ALBEF, TCL, CLIP | Flickr30K, MS-COCO | |
| VLP-Attack (Wang et al., 2023d) | 2023 | Black-box | Sample-wise | ALBEF, TCL, BLIP, BLIP2, MiniGPT-4 | MS-COCO, Flickr30K, SNLI-VE | |
| TMM (Wang et al., 2024c) | 2024 | Black-box | Sample-wise | ALBEF, TCL, X_VLM, CLIP, BLIP, ViLT, METER | MS-COCO, Flickr30K, RefCOCO+, SNLI-VE | |
| VLATTACK (Yin et al., 2023) | 2023 | Black-box | Sample-wise | BLIP, ViLT, CLIP | MS-COCO, VQA v2, NLVR2, SNLI-VE, ImageNet, SVHN | |
| OT-Attack (Han et al., 2023a) | 2023 | Black-box | Sample-wise | CLIP, ALBEF, TCL | Flickr30K, MS-COCO, RefCOCO+ | |
| PRM (Hu et al., 2024a) | 2024 | Black-box | Sample-wise | CLIP, Detic, VL-PLM, FC-CLIP, OpenFlamingo, LLaVA | PASCAL Context, COCO-Stuff, OV-COCO, MS-COCO, OK-VQA | |
| VLPTransferAttack (Gao et al., 2024f) | 2024 | Black-box | Sample-wise | CLIP, ALBEF, TCL | Flickr30K, MS-COCO, RefCOCO+ | |
| C-PGC (Fang et al., 2024a) | 2024 | Black-box | Universal | ALBEF, TCL, X-VLM, CLIP, BLIP | Flickr30K, MS-COCO, SNLI-VE, RefCOCO+ | |
| ETU (Zhang et al., 2024n) | 2024 | Black-box | Universal | ALBEF, TCL, CLIP, BLIP | Flickr30K, MS-COCO | |
| X-Transfer (Huang et al., 2025b) | 2025 | Black-box | Universal | CLIP, OpenFlamingo, LLaVA, BLIP2, MiniGPT4 | Flickr30K, MS-COCO, ImageNet, CIFAR10, CIFAR100, STL10, SUN397, FOOD101, GTSRB StandfordCars, OK-VQA, VizWiz | |
| Adversarial Defense | Defense-Prefix (Azuma and Matsui, 2023) | 2023 | Adversarial Tuning | Prompt Tuning | CLIP | ImageNet |
| AdvPT (Zhang et al., 2024i) | 2023 | Adversarial Tuning | Prompt Tuning | CLIP | ImageNet, Pets, Flowers, Food101, SUN397, DTD, EuroSAT, UCF101, ImageNet-V2, ImageNet-Sketch, ImageNet-A, ImageNet-R | |
| APT (Li et al., 2024k) | 2024 | Adversarial Tuning | Prompt Tuning | CLIP | ImageNet, Caltech101, Pets, StanfordCars, Flowers, Food101, FGVCAircraft, SUN397, DTD, EuroSAT, UCF101, ImageNet-V2, ImageNet-Sketch, ImageNet-R, ObjectNet | |
| MixPrompt (Fan et al., 2024) | 2024 | Adversarial Tuning | Prompt Tuning | CLIP | ImageNet, Pets, Flowers, DTD, EuroSAT, UCF101, SUN397, Food101, ImageNet-V2, ImageNet-Sketch, ImageNet-A, ImageNet-R | |
| PromptSmoot (Hussein et al., 2024) | 2024 | Adversarial Tuning | Prompt Tuning | PLIP, Quilt, MedCLIP | KatherColon, PanNuke, SkinCancer, SICAP v2 | |
| FAP (Zhou et al., 2024g) | 2024 | Adversarial Tuning | Prompt Tuning | CLIP | ImageNet, Caltech101, Pets, StanfordCars, Flowers, Food101, FGVCAircraft, SUN397, DTD, EuroSAT, UCF101 | |
| APD (Luo et al., 2024b) | 2024 | Adversarial Tuning | Prompt Tuning | CLIP | ImageNet, Caltech101, Flowers, Food101, SUN397, DTD, EuroSAT, UCF101 | |
| TAPT (Wang et al., 2025f) | 2025 | Adversarial Tuning | Prompt Tuning | CLIP | ImageNet, Caltech101, Pets, StanfordCars, Flowers, Food101, FGVCAircraft, SUN397, DTD, EuroSAT, UCF101 | |
| TeCoA (Mao et al., 2023) | 2022 | Adversarial Tuning | Contrastive Tuning | CLIP | CIFAR10, CIFAR100, STL10, Caltech101, Caltech256, Pets, StanfordCars, Food101, Flowers, FGVCAircraft, SUN397, DTD, PCAM, HatefulMemes, EuroSAT | |
| PMG-AFT (Wang et al., 2024j) | 2024 | Adversarial Tuning | Contrastive Tuning | CLIP | CIFAR10, CIFAR100, STL10, ImageNet, Caltech101, Caltech256, Pets, Flowers, FGVCAircraft, StanfordCars, SUN397, Food101, EuroSAT, DTD, PCAM | |
| MMCoA (Zhou et al., 2024d) | 2024 | Adversarial Tuning | Contrastive Tuning | CLIP | CIFAR10, CIFAR100, TinyImageNet, STL10, Caltech101, Caltech256, Pets, Flowers, FGVCAircraft, Food101, EuroSAT, DTD, SUN397, Country211 | |
| FARE (Schlarmann et al., 2024) | 2024 | Adversarial Tuning | Contrastive Tuning | OpenFlamingo, LLaVA | COCO, Flickr30k, TextVQA, VQA v2, CalTech101, StanfordCars, CIFAR10, CIFAR100, DTD, EuroSAT, FGVCAircrafts, Flowers, ImageNet-R, ImageNet-Sketch, PCAM, Pets, STL10, ImageNet | |
| VILLA (Gan et al., 2020) | 2020 | Adversarial Training | Two-stage Training | UNITER, LXMERT | MS-COCO, Visual Genome, Conceptual Captions, SBU Captions ImageNet, LAION, DataComp | |
| AdvXL (Wang et al., 2024s) | 2024 | Adversarial Training | Two-stage Training | CLIP | ImageNet, LAION, DataComp | |
| MirrorCheck (Fares et al., 2024) | 2024 | Adversarial Detection | One-shot Detection | UniDiffuser, BLIP, Img2Prompt, BLIP-2, MiniGPT-4 | MS-COCO, CIFAR10, ImageNet | |
| AdvQDet (Wang et al., 2024k) | 2024 | Adversarial Detection | Stateful Detection | CLIP, ViT, ResNet | CIFAR10, GTSRB, ImageNet, Flowers, Pets | |
| Backdoor & Poisoning Attack | PBCL (Carlini and Terzis, 2022) | 2021 | Backdoor&Poisoning | Visual Trigger | CLIP | Conceptual Captions, YFCC |
| BadEncoder (Jia et al., 2022) | 2021 | Backdoor | Visual Trigger | ResNet(SimCLR), CLIP | CIFAR10, STL10, GTSRB, SVHN, Food101 | |
| CorruptEncoder (Zhang et al., 2024l) | 2022 | Backdoor | Visual Trigger | ResNet(SimCLR) | ImageNet, Pets, Flowers | |
| BadCLIP (Liang et al., 2024b) | 2023 | Backdoor | Visual Trigger | CLIP | Conceptual Captions | |
| BadCLIP (Bai et al., 2024a) | 2023 | Backdoor | Multi-modal Trigger | CLIP | ImageNet, Caltech101, Pets, StanfordCars, Flowers, Food101, FGVCAircraft, SUN397, DTD, EuroSAT, UCF101 | |
| MM Poison (Yang et al., 2023) | 2022 | Poisoning | Multi-modal Poisoning | CLIP | Flickr-PASCAL, MS-COCO | |
| MEMLiu et al. 2024n | 2024 | Poisoning | Multi-modal Trigger | CLIP | Flickr8k, Flickr30k, MS-COCO | |
| Backdoor & Poisoning Defense | CleanCLIP (Bansal et al., 2023) | 2023 | Backdoor Removal | Fine-tuning | CLIP | Conceptual Captions, ImageNet |
| SAFECLIP (Yang et al., 2024f) | 2023 | Backdoor Removal | Fine-tuning | CLIP | Conceptual Captions, Visual Genome, MS-COCO, Flowers, Food101, ImageNet, Pets, StanfordCars, Caltech101, CIFAR10, CIFAR100, DTD, FGVCAircraft | |
| RoCLIP (Yang et al., 2024g) | 2023 | Robust Training | Pre-training | CLIP | Conceptual Captions, Flowers, Food101, ImageNet, Pets, StanfordCars, Caltech101, CIFAR10, CIFAR100, DTD, FGVCAircraft | |
| DECREE (Feng et al., 2023) | 2023 | Backdoor Detection | Backdoor Model Detection | CLIP | CIFAR10, GTSRB, SVHN, STL-10, ImageNet | |
| TIJO (Sur et al., 2023) | 2023 | Backdoor Detection | Trigger Inversion | BUTD, MFB, BAN, MCAN, NAS | TrojVQA | |
| Mudjacking (Liu et al., 2024g) | 2024 | Backdoor Detection | Trigger Inversion | CLIP | Conceptual Captions, CIFAR10, STL10, ImageNet, SVHN, Pets, Wiki103-Sub, SST-2, HOSL | |
| SEER (Zhu et al., 2024a) | 2024 | Backdoor Detection | Backdoor Sample Detection | CLIP | MSCOCO, Flickr, STL10, Pet, ImageNet | |
| Outlier Detection (Huang et al., 2025a) | 2025 | Backdoor Detection | Backdoor Sample Detection | CLIP | Conceptual Captions, ImageNet, RedCaps |
| Attack/Defense | Method | Year | Category | Subcategory | Target model | Dataset |
|---|---|---|---|---|---|---|
| Adversarial Attack | Co-Attack ( | 2022 | White-box | Invisible | ALBEF, TCL, | MS-COCO, Flickr30K, RefCOCO+, SNLI-VE |
| AdvCLIP ( | 2023 | White-box | Invisible | STL10, GTSRB, CIFAR10, ImageNet, Wikipedia, Pascal-Sentence, NUS-WIDE, XmediaNet | ||
| Typographical Attacks ( | 2021 | White-box | Visible | ImageNet | ||
| Multi-Image Typographical Attacks ( | 2025 | White-box | Visible | OpenCLIP, InstructBLIP | ImageNet, | |
| 2023 | Black-box | Sample-wise | ALBEF, TCL, | Flickr30K, MS-COCO | ||
| SA-Attack ( | 2023 | Black-box | Sample-wise | ALBEF, TCL, | Flickr30K, MS-COCO | |
| VLP-Attack ( | 2023 | Black-box | Sample-wise | ALBEF, TCL, BLIP, BLIP2, MiniGPT-4 | MS-COCO, Flickr30K, SNLI-VE | |
| 2024 | Black-box | Sample-wise | ALBEF, TCL, X_VLM, CLIP, BLIP, ViLT, | MS-COCO, Flickr30K, RefCOCO+, SNLI-VE | ||
| VLATTACK ( | 2023 | Black-box | Sample-wise | BLIP, ViLT, | MS-COCO, | |
| OT-Attack ( | 2023 | Black-box | Sample-wise | CLIP, ALBEF, | Flickr30K, MS-COCO, RefCOCO+ | |
| 2024 | Black-box | Sample-wise | CLIP, Detic, VL-PLM, FC-CLIP, OpenFlamingo, LLaVA | |||
| VLPTransferAttack ( | 2024 | Black-box | Sample-wise | CLIP, ALBEF, | Flickr30K, MS-COCO, RefCOCO+ | |
| C-PGC ( | 2024 | Black-box | Universal | ALBEF, TCL, X-VLM, CLIP, | Flickr30K, MS-COCO, SNLI-VE, RefCOCO+ | |
| 2024 | Black-box | Universal | ALBEF, TCL, CLIP, | Flickr30K, MS-COCO | ||
| X-Transfer ( | 2025 | Black-box | Universal | CLIP, OpenFlamingo, LLaVA, BLIP2, MiniGPT4 | Flickr30K, MS-COCO, ImageNet, CIFAR10, CIFAR100, STL10, SUN397, FOOD101, | |
| Adversarial Defense | Defense-Prefix ( | 2023 | Adversarial Tuning | Prompt Tuning | ImageNet | |
| AdvPT ( | 2023 | Adversarial Tuning | Prompt Tuning | ImageNet, Pets, Flowers, Food101, SUN397, DTD, EuroSAT, UCF101, ImageNet-V2, ImageNet-Sketch, ImageNet-A, ImageNet-R | ||
| 2024 | Adversarial Tuning | Prompt Tuning | ImageNet, Caltech101, Pets, StanfordCars, Flowers, Food101, FGVCAircraft, SUN397, DTD, EuroSAT, UCF101, ImageNet-V2, ImageNet-Sketch, ImageNet-R, ObjectNet | |||
| MixPrompt ( | 2024 | Adversarial Tuning | Prompt Tuning | ImageNet, Pets, Flowers, DTD, EuroSAT, UCF101, SUN397, Food101, ImageNet-V2, ImageNet-Sketch, ImageNet-A, ImageNet-R | ||
| PromptSmoot ( | 2024 | Adversarial Tuning | Prompt Tuning | PLIP, Quilt, MedCLIP | KatherColon, PanNuke, SkinCancer, | |
| 2024 | Adversarial Tuning | Prompt Tuning | ImageNet, Caltech101, Pets, StanfordCars, Flowers, Food101, FGVCAircraft, SUN397, DTD, EuroSAT, UCF101 | |||
| 2024 | Adversarial Tuning | Prompt Tuning | ImageNet, Caltech101, Flowers, Food101, SUN397, DTD, EuroSAT, UCF101 | |||
| 2025 | Adversarial Tuning | Prompt Tuning | ImageNet, Caltech101, Pets, StanfordCars, Flowers, Food101, FGVCAircraft, SUN397, DTD, EuroSAT, UCF101 | |||
| TeCoA ( | 2022 | Adversarial Tuning | Contrastive Tuning | CIFAR10, CIFAR100, STL10, Caltech101, Caltech256, Pets, StanfordCars, Food101, Flowers, FGVCAircraft, SUN397, DTD, PCAM, HatefulMemes, EuroSAT | ||
| PMG-AFT ( | 2024 | Adversarial Tuning | Contrastive Tuning | CIFAR10, CIFAR100, STL10, ImageNet, Caltech101, Caltech256, Pets, Flowers, FGVCAircraft, StanfordCars, SUN397, Food101, EuroSAT, DTD, | ||
| MMCoA ( | 2024 | Adversarial Tuning | Contrastive Tuning | CIFAR10, CIFAR100, TinyImageNet, STL10, Caltech101, Caltech256, Pets, Flowers, FGVCAircraft, Food101, EuroSAT, DTD, SUN397, Country211 | ||
| 2024 | Adversarial Tuning | Contrastive Tuning | OpenFlamingo, LLaVA | COCO, Flickr30k, TextVQA, | ||
| 2020 | Adversarial Training | Two-stage Training | UNITER, | MS-COCO, Visual Genome, Conceptual Captions, | ||
| AdvXL ( | 2024 | Adversarial Training | Two-stage Training | ImageNet, LAION, DataComp | ||
| MirrorCheck ( | 2024 | Adversarial Detection | One-shot Detection | UniDiffuser, BLIP, Img2Prompt, BLIP-2, MiniGPT-4 | MS-COCO, CIFAR10, ImageNet | |
| AdvQDet ( | 2024 | Adversarial Detection | Stateful Detection | CLIP, ViT, ResNet | CIFAR10, GTSRB, ImageNet, Flowers, Pets | |
| Backdoor & Poisoning Attack | 2021 | Backdoor&Poisoning | Visual Trigger | Conceptual Captions, | ||
| BadEncoder ( | 2021 | Backdoor | Visual Trigger | ResNet(SimCLR), | CIFAR10, STL10, GTSRB, SVHN, Food101 | |
| CorruptEncoder ( | 2022 | Backdoor | Visual Trigger | ResNet(SimCLR) | ImageNet, Pets, Flowers | |
| BadCLIP ( | 2023 | Backdoor | Visual Trigger | Conceptual Captions | ||
| BadCLIP ( | 2023 | Backdoor | Multi-modal Trigger | ImageNet, Caltech101, Pets, StanfordCars, Flowers, Food101, FGVCAircraft, SUN397, DTD, EuroSAT, UCF101 | ||
| 2022 | Poisoning | Multi-modal Poisoning | Flickr-PASCAL, MS-COCO | |||
| 2024 | Poisoning | Multi-modal Trigger | Flickr8k, Flickr30k, MS-COCO | |||
| Backdoor & Poisoning Defense | CleanCLIP ( | 2023 | Backdoor Removal | Fine-tuning | Conceptual Captions, ImageNet | |
| SAFECLIP ( | 2023 | Backdoor Removal | Fine-tuning | Conceptual Captions, Visual Genome, MS-COCO, Flowers, Food101, ImageNet, Pets, StanfordCars, Caltech101, CIFAR10, CIFAR100, DTD, FGVCAircraft | ||
| RoCLIP ( | 2023 | Robust Training | Pre-training | Conceptual Captions, Flowers, Food101, ImageNet, Pets, StanfordCars, Caltech101, CIFAR10, CIFAR100, DTD, FGVCAircraft | ||
| 2023 | Backdoor Detection | Backdoor Model Detection | CIFAR10, GTSRB, SVHN, STL-10, ImageNet | |||
| 2023 | Backdoor Detection | Trigger Inversion | BUTD, MFB, BAN, MCAN, | TrojVQA | ||
| Mudjacking ( | 2024 | Backdoor Detection | Trigger Inversion | Conceptual Captions, CIFAR10, STL10, ImageNet, SVHN, Pets, Wiki103-Sub, SST-2, | ||
| 2024 | Backdoor Detection | Backdoor Sample Detection | MSCOCO, Flickr, STL10, Pet, ImageNet | |||
| Outlier Detection ( | 2025 | Backdoor Detection | Backdoor Sample Detection | Conceptual Captions, ImageNet, RedCaps |
Sharing content requires targeting cookies to be enabled. Please update your cookie preferences to use this feature.