Enhancing instance discrimination for self-supervised adversarial training

Deep neural networks are vulnerable in the presence of adversarial examples. In light of this vulnerability, an emerging research trend is to build a defense model based on adversarial training (AT) (Madry et al., 2018). The adversarial learning objective is typically formulated as a minimax optimization problem (Lio et al., 2022) where the additive perturbations to input data are first estimated by maximizing the loss via gradient ascent, and then the model parameters are trained by minimizing the loss with gradient descent by using AT examples. Such a two-stage training process has been implemented for a variety of downstream tasks in computer vision and natural language processing. The resulting models generally improve the adversarial robustness, but likely degrade the performance on clean data. In Zhu et al. (2020) and Pan et al. (2022), the robustness was even generalized to clean data. For text representation, the resulting model becomes noise-invariant by adding the worst-case perturbations onto word embeddings during training (Chien et al., 2025). The adversarial instances with the perturbed data in embedding space are used to improve model generalization. However, most existing AT methods (Shafahi et al., 2019; Zhang et al., 2019; Wong et al., 2020) were developed under a supervised setting where the labels of training data were required (Li et al., 2023). It is challenging to carry out a practical solution to self-supervised AT. Previously, there have been very few works dedicated to self-supervised AT in image representation, and there has been limited exploration of self-supervised AT for text representation. This paper presents a contrastive view in self-supervised AT by deliberating the perspective of instance discrimination.

Contrastive learning (CL) (Li et al., 2022; Chien and Chen, 2026) is seen as a recent breakthrough in modern machine learning. CL has been advanced to extract vital information which is feasible to upgrade the learning performance. State-of-the-art (SOTA) results have been achieved in numerous applications. The learning process is motivated from the task of instance discrimination (Wu et al., 2018) where each data instance is viewed as an individual class, and a neural encoder is learned to calculate the features which are discriminative among different instances or self-defined classes. It is obvious that such a CL task is essentially implemented for unsupervised learning but implicitly connected to a classification problem in supervised learning (SL). The contrastive perspective is fulfilled for self-supervised learning. CL has its fundamentals originated from metric learning and information theory. In Chen et al. (2020), a simple framework for CL of visual representation (SimCLR) was proposed to upgrade learning an image classifier which was comparable with that based on SL. CL was then extended to enable a significant performance in text representation. In Gao et al. (2021), a simple contrastive sentence embedding (SimCSE) was exploited to train a contrastive sentence encoder, which obtained desirable results on unsupervised or self-supervised learning in various tasks (Sun et al., 2025; Tu et al., 2024).

This study revisits the connection between CL and supervised classification, and leverages this connection to design a self-supervised AT framework. While contrastive objectives such as the information noise-contrastive estimation (InfoNCE) or the normalized temperature-scaled cross entropy (CE) (NT-Xent) in SimCLR can be interpreted as implicit classification over instances, the existing formulations treat each sample differently across views, which may lead to inconsistencies when combined with AT. We explicitly reformulate the instance discrimination as a batch-wise classification problem and integrate it with adversarial optimization. This perspective provides (i) a unified view bridging CL and AT, and (ii) a practical framework that improves robustness while maintaining computational efficiency.

The remaining of this paper is organized as follows. Section 2 introduces the background of CL and supervised/self-supervised AT. Section 3 addresses how the classification perspective is strengthened and merged in a new self-supervised AT. In Section 4, a series of experiments on image and text representations are conducted to evaluate robustness, efficiency and accuracy by using various defense models. Section 5 concludes some findings drawn from this study.

The safety and reliability of deep models have been very crucial when practically deploying an information system. There have been a number of solutions proposed to enhance the robustness to domain shift, random noise, rotation or color changes in images or typos in sentences. The most common robustness highlighted recently is adversarial robustness. The defenses in visual and textual models against adversarial attacks are our main focus. Previously, the adversarial attacks for images (Carlini and Wagner, 2017; Madry et al., 2018) and sentences (Gao et al., 2018; Jin et al., 2020) have been addressed. Among various defense methods, the AT is seen as the most effective method to improve adversarial robustness.

This paper presents a contrastive perspective to develop a self-supervised AT. The instance discrimination is implemented to initiate a unified view toward AT. First, the multi-view CL is studied. CL is used for self-supervised learning (Gan et al., 2024) where class labels are not required. Originally, CL is performed by using SimCLR (Chen et al., 2020) where the representation $fθ(·)$ with parameter $θ$ is trained through a task where individual instances are mutually discriminative. Each sample $xi$ in a batch $x=(x1,…,xB)⊤$ of size B is transformed to a variant $xik=tk(xi)$ to form M batch views $X={x1,…,xM}$ where $xk=(tk(x1),…,tk(xB))⊤$ is the kth view of $X$⁠. Random samples are drawn from strong data augmentation $tk∼TCL$ where $TCL$ denotes the set of variants for CL such as random cropping and resizing, random horizontal flip, strong random color jitter and random gray-scale conversion using image data. In the implementation, each variant $xik$ can be viewed as an anchor, and its latent representation $zik=fθ(xik)$ will attract (shown by green) that of positive sample $zij=fθ(xij)$⁠, i.e. the same sample in different views, and simultaneously repel (shown by red) from those of the remaining samples within and between batches as the negative samples as illustrated in Figure 1 where $B=3$ and $x1a$ is seen as an anchor. For this case of two-view CL (⁠$M=2$⁠), the CL objective $LCL(xa,xb;θ)$ due to individual samples from two batch views $xa={xia}$ (1$st$ term) and $xb={xib}$ (2$nd$ term) as an InfoNCE objective is therefore yielded with encoder $θ$ by:

where $τ$ is the temperature. In equation (1), the encoder outputs are normalized in the calculation of cosine similarity between two samples. A crucial component in such a SimCLR process for image representation is the data augmentation (Luo et al., 2023a) through two or multiple views $x→X$ which are used in the task of instance discrimination (Zhang et al., 2024b). It is worth noting that the contrastive objective in equation (1) (Chien et al., 2024b) can be interpreted as a form of CE loss over a dynamically constructed set of classes defined by the batch samples. However, unlike standard classification, the implicit class assignments depend on the choice of anchor and vary across different views, which will be further analyzed in Section 3.

Relative to image data, the augmentation of text data is harder to visualize due to the discreteness of word tokens. Previously, the augmentation strategies such as back-translation, word deletion, reordering and substitution were used (Yan et al., 2021). For CL of a text model, a successful solution based on SimCSE (Gao et al., 2021) was developed by using an alternative augmentation called dropout. Using dropout, the latent sample $zi=fθp(xi)$ is calculated by using a model $fθp(·)$ with dropout rate p and encoder parameter $θ$⁠. Each instance $xi$ passes through the encoder $fθp(·)$ twice to produce two representations $zia$ and $zib$ due to random dropout. The instance discrimination is then defined within a mini-batch. The representation of an anchor $zia$ gets close to that of positive sample $zib$ and at the same time gets separate from those of negative samples, which correspond to the views $zja$ and $zjb$ of the remaining samples ${xj}j≠i$ in a mini-batch $x$⁠. This study works on AT for image and text defense models from contrastive perspective (Chien et al., 2024a).

AT was developed to estimate a supervised defense model $fθ(·)$ where learning procedure consists of attack generation and model training, as shown in Figure 2, which are performed to achieve adversarial robustness with the number of classes $C=5$⁠. The parameters of perturbation $δ$⁠, encoder $θe$ and classifier $W$ are jointly estimated by solving a minimax optimization problem (Tu et al., 2020; Chien and Sun, 2023), which is decomposed into an upper half and a lower half in the figure. The CE loss between the logit outputs $y^i≜softmax(fθ(·))$⁠, driven by parameters $θ={θe,W}$ and softmax function, and the one-hot target vector $yi$ is first maximized to estimate the worst-case perturbation $δ$ by using attack data $tSL(x)+δ$ as shown in the upper half. In the lower half, the attack data (dashed blocks) are used as the training data to update parameters of encoder $θe$ and classifier $W$ (shaded by grey) by minimizing the loss. The formulation turns out as:

where the augmented data $tSL∼TSL$ from original data $x$ are drawn and used in SL for adversarial robustness. In equation (2), $D$ denotes the set of labeled data ${xi,yi={yic}}$⁠, and $S$ denotes the set of possible perturbations $δ$ which are bounded within $l∞$ norm ball $B(ϵ)$ of a small radius $ϵ$ as $‖δ‖∞≤ϵ$⁠. The classification loss in SL is seen as a negative log likelihood of one-hot targets $y={yi}$ given by inputs $x={xi}$⁠, which is formed as a CE between $y^={y^ic}$⁠, where $y^ic=p(c | xi)$ is the class conditional likelihood, and $y$ based on:

In the first stage, the projected gradient descent (PGD) for $l∞$ bounded adversary in $B(ϵ)$ (Kurakin et al., 2018) is repeatedly performed at each iteration to estimate the perturbation $δ$ via the fast gradient sign method (FGSM) in a form of $δ=ϵsign(∇tSL(x)LCE)$ with a step size $α$⁠. The iterative updating is performed by:

where $fθ(·)=Wfθe(·)$⁠. In the second stage, the model parameters ${θe,W}$ are estimated by allowing perturbation noises $δ$ presented in individual samples $x$⁠.

In general, the adversarial attacks on sentences are very different from those on images. Human eyes are not possible to tell the difference between an original image and its attack image within a small norm ball. For text data, it is hard to evaluate how much the sentence is perturbed when its semantic meaning is changed. The continuous essence of image data naturally results in using the gradient-based attacks. But, the attack in embedding space will produce non-existing tokens in the vocabulary. In Gao et al. (2018) and Jin et al. (2020), the black-box adversarial attacks were developed. Typically, AT for image representation aims to improve the model robustness to adversarial attacks. In this process, the accuracy on clean images would decrease due to the trade-off between robustness and accuracy. In text modeling, the adversarial perturbations were added onto the word embeddings during training. High performance was achieved in various classification tasks. The adversarially trained model was able to prevent overfitting problem. The perturbations in noisy samples were beneficial to protect unknown attacks. In Zhu et al. (2020), a low-cost AT method (Shafahi et al., 2019) was developed to improve model generalization. In Pan et al. (2022), the adversarially perturbed text examples were used to perform CL. All of these methods were designed in a supervised setting which required data labels $y$⁠. This paper presents a self-supervised defense model for adversarial attacks in presence of image and text data.

To handle the AT without labeled data $y$⁠, i.e. $D=x$⁠, the previous methods (Kim et al., 2020; Jiang et al., 2020) generally followed an objective similar to equation (2) while the CE loss was replaced by a CL loss in equation (1). The minimax optimization for self-supervised AT is performed by:

where CL loss $LCL$ is maximized to estimate the perturbation $δ$ and is minimized to train the encoder $θ$⁠. Figure 3 shows the attack procedure of self-supervised AT by using the robust contrastive learning (RoCL) (Kim et al., 2020) with $B=3$⁠. The dashed blocks denote the perturbed instances. In the adversarial procedure, each clean batch view $xb=tb(x)=(tb(x1),…,tb(xB))⊤$ and the perturbation of another batch view $xa=ta(x)$⁠, denoted by:

alternatively take the role of anchor samples, and the negative samples will contain the other samples in both $tb(x)$ and $ta(x)+δ$⁠. The multi-view CL loss $LCLmulti({(xa)adv,xa,xb}; θ)$ due to perturbation and augmentation is then calculated for each individual batch data $x$⁠. The pairs of latent samples ${(zia)adv,zia}$ and ${(zia)adv,zib}$ from the same sample $xi$ are attracting while those ${(zia)adv,(zja)adv}$⁠, ${(zia)adv,zja}$⁠, ${(zib)adv,(zjb)adv}$⁠, ${(zib)adv,zjb}$ from different samples ${xi,xj,j≠i}$ are repelling. An additional regularization term is further introduced to enforce together the clean input $ta(x)$ and the perturbed input $ta(x)+δ$⁠. The CL loss $LCL$ in equation (5) is then replaced by the regularized objective:

where ${xa,xb}∼D$ and $λ$ are regularization parameters.

Different from RoCL, the adversarial contrastive learning (ACL) (Jiang et al., 2020; Lin et al., 2026) presented a dual stream approach to self-supervised AT where the encoder $θ$ was trained according to standard CL via SimCLR by using the clean examples as well as the corresponding adversarial examples. In addition, a potential result of self-supervised AT was obtained by the adversarial CL (AdvCL) (Fan et al., 2021), where an adversarial example view and a high-frequency component view via Fourier transform were introduced to conduct a multi-view CL for classification task. AdvCL used the pseudo labels (PLs) generated by a pretrained encoder for both attack generation and model training. A kind of pseudo-labeled AT was implemented. A recent method called graph ACL (Luo et al., 2023b) was proposed by learning a bank of negative samples for self-supervised graph encoding from graph-structured data. Basically, the above-mentioned methods involved in different methods, which complicated the training procedure. The computation and memory requirements are demanding. The connection to supervised AT was disregarded in previous studies.

This study is motivated by incorporating the contrastive view via instance discrimination into an implementation of a new self-supervised adversarial training (Chien and Chen, 2024; Zhang et al., 2024a). A unified view of CL and AT is presented to allow richer information in a more compact model with lower computation cost. Figure 4 illustrates the flow diagram from a motivation for developing a self-supervised AT to a final solution based on two-view AT and an extended solution based on two-view CL where the self-supervised tricks of conducting classification in instance discrimination and using separate views of anchors and classes are implemented through handling the problems of inconsistencies in class definition and augmentation bias in traditional CL (Chen et al., 2020) and AT (Madry et al., 2018).

To better understand the proposed method, we first formalize the relationship between CL and classification, and clarify the potential inconsistencies discussed in this work.

The contrastive loss in equation (1) can be rewritten as a CE loss or classification loss over a set of batch-wise classes, where each instance is treated as a distinct class, as detailed later in equation (10). Under this interpretation, the probability of assigning an anchor representation $zi$ to its positive counterpart can be expressed as a classification output logit:

This reveals that standard CL implicitly performs classification with dynamically changing class definitions. We analyze two potential inconsistencies in previous CL and supervised adversarial learning.

In conventional CL, each sample alternates between being an anchor and a class representative. As a result, the conditional distributions being optimized considerably differ across views, which may introduce inconsistency when integrating with adversarial objectives that assume a fixed classifier for instance discrimination. In particular, the classes in CL using equation (1) are defined inconsistently at the individual steps for parameter updating. Notice that for those 2B data samples derived from two views ${ta,tb}$ of $X$⁠, each sample follows a different classification problem. Then, even the anchors from the same sample $xi$ but with different views $ta$ and $tb$ are defined and treated as different instances or classes. This is because, as shown in equation (1), each data sample is alternatively acted as the anchor sample and positive/negative sample. When viewing CL as a classification task, there does exist an inconsistency in instance discrimination in equations (1) and (10). Such varying class definitions may introduce ambiguity in optimization and affect downstream performance. Prior contrastive objectives based on InfoNCE implicitly redefine instance classes depending on the anchor view and batch composition. Although effective empirically, this dynamic role switching may introduce ambiguity when adversarial perturbations are iteratively optimized. Our formulation explicitly separates anchor and class representations to stabilize the optimization target during adversarial updates.

In CL and supervised AT, different augmentation pipelines $TCL$ and $TSL$ for the augmented samples $xCL$ and $xAT$ from the same original sample $x$ are used, respectively. This leads to a mismatch in the data distributions during representation learning and adversarial optimization:

The augmented data in previous CL (Chen et al., 2020) and AT (Madry et al., 2018) adopted different views of images in data augmentation as illustrated in Figure 5. Relative to the original image $xi$⁠, the augmented images in previous works on CL and AT were distributed with strong bias $ts(xi)∼TCL$ and weak bias $tw(xi)∼TSL$⁠, respectively. The inconsistent biases in the views of augmented images did exist in the previous CL and AT. The inconsistent distribution view did happen in previous data augmentations for CL and AT. Such discrepancy may affect the stability of AT when combined with contrastive objectives. These observations motivate a unified formulation that aligns instance discrimination with a consistent classification perspective and harmonizes augmentation strategies. This paper presents a self-supervised AT driven by a unified view for instance discrimination where these two inconsistencies are handled through bridging CL and AT via a classification task for individual instances. Without loss of generality, in the application of sentence representation, two views of augmented training data $tw(xi)$ and $ts(xi)$ for self-supervised AT are implemented by using an encoder without and with neural network dropout, respectively. Recent studies in ACL have also observed that discrepancies between augmentation policies and adversarial objectives may affect robustness and representation stability. Dynamic ACL (DynACL) in Luo et al. (2023a) addressed this issue through dynamic augmentation scheduling, while adversarial invariant regularization (AIR) in Xu et al. (2023b) regularized augmentation-induced style variations. These observations are consistent with our motivation that inconsistent augmentation views may hinder stable adversarial optimization. Table 1 shows a convention of notations for ease of comprehension for the proposed method.

Considering an instance discrimination (Wu et al., 2018) task in CL, every instance is viewed as a class, and the model $fθ(·)$ is learned to classify each input sample $xi$ in a B-way classification problem where B is the size of mini-batch. Let a memory bank $V=(v1,…,vB)⊤$ or $Z=(z1,…,zB)⊤$ store the representations of all data samples. The softmax loss in a form of CE loss, or equivalently the negative log conditional likelihood for discrimination of B samples can be minimized via:

where $zi=fθ(xi)$ and $y˜={y˜ij}$ denotes the one-hot target matrix for instance discrimination, which is fixed as a B-dimensional identity matrix $y˜={y˜i}=IB$⁠. Notably, the instance conditional likelihood $y^i=p(y˜i | xi)= softmax(Vzi)$ can be seen as the output logits of an instance classification network for $xi$⁠. A self-supervised learning (Yu and Dai, 2024) is performed without the need of class labels $y={yic}$⁠. When looking at each logarithmic term in equation (1), it is interesting that SimCLR similarly follows a classification task as seen in equation (10). In case of $τ=1$⁠, equation (1) also expresses a softmax function as the CE loss for classification. Such a task plays a key role in CL. Basically, equation (10) shows that the classes in B-way classification problem are consistently defined by individual data samples in the data set throughout the training process. From an optimization perspective, this formulation can be viewed as approximating a robust classification objective over instance identities, where the adversarial perturbation seeks to maximize the classification loss while the encoder minimizes it. Compared to pairwise contrastive loss, the batch-wise classification view provides a more stable gradient signal, especially when integrated with adversarial updates:

Implication. Separating anchor and class samples stabilizes the adversarial optimization objective.

Algorithm 1. Enhancing instance discrimination for a new contrastive learning or self-supervised learning

To cope with these inconsistencies, this paper strengthens the classification perspective in standard CL (Chen et al., 2020) in equation (1) by separating the views of anchor samples and class samples. As the inconsistent class definition is caused by using each sample as both anchor and positive/negative samples, this study explicitly separates both roles and sufficiently uses different views or batches. In addition, data augmentation may cause a severe mismatch between the distributions in training and test data. But, the previous CL (Chen et al., 2020) performed well by using augmented data with strong bias. This study is motivated by incorporating different views by using various augmented data. In particular, the anchor samples$Xanch=(tw(x1),…,tw(xB))⊤$ are drawn from a view batch via weak data augmentation. These anchor samples are analogous to those input samples in previous supervised AT (Madry et al., 2018). Notably, these anchor samples are only used as the anchors in instance discrimination. On the other hand, since strong augmentation is still required for instance discrimination to fulfill CL, the class samples$Xcls=(ts(x1),…,ts(xB))⊤$ are formed as a view batch by using strong data augmentation. These class samples with strong distribution biases are only used as the samples in either positive or negative classes. With the view batches ${Xanch,Xcls}$ defined by the augmented data due to weak and strong biases, respectively, a self-supervised classification task is formulated by feeding the class samples $Xcls$ into a model or an encoder $fθ(·)$ to obtain a batch of class representations for instance discrimination:

Therefore, a two-view classification problem is formulated in a learning procedure. In this procedure, the classification logits $y^i$ of an input example $xi$ are first calculated by feeding an anchor sample $xianch=tw(xi)$ with the view of weak data augmentation, and then passing this sample through an encoder $fθ(·)$⁠, multiplying the encoder output by the class representation matrix $Zcls$ based on the view of strong data augmentation, and measuring the class or instance posterior probabilities by softmax function. Given the class representation matrix $Zcls$⁠, each weakly augmented anchor sample $tw(xi)$ is projected into the same embedding space to obtain the instance classification logits:

Next, the output logits $y^i=((y^i)1,…,(y^i)B)⊤$ of an input $xi$ are used to calculate the sample-based CE loss by:

where $y˜i$ denotes the self-supervision label for an anchor $xianch$⁠. This one-hot label is defined by the index of the corresponding class sample, i.e. $y˜ii=1$ and $y˜ij=0$ if $i≠j$⁠. The number of classes corresponds to that of samples B in a view batch, and the collection of one-hot target vectors is formed as a B-dimensional identity matrix $y˜={y˜i}=IB$⁠. Figure 6 shows an implementation of a new self-supervised classification task for CL. Importantly, the classification perspective in CL is strengthened by consistently handling a unified classification problem from the views of anchor and class samples by using weak $tw$ and strong $ts$ data augmentations, respectively. The issue of inconsistent class definition is addressed because the samples of anchor and class are separately defined over individual examples. The same example $xi$ will not be mixed as different classes. In addition, two views of augmented data are both adopted in a single task so that the inconsistent augmentation bias does not exist. Algorithm 1 illustrates the procedure of enhancing instance discrimination for a new contrastive or self-supervised learning.

This study connects the supervised, unsupervised and self-supervised learning and presents a contrastive view to derive a new self-supervised AT. First, the classifier with parameter $θ$ in SL consists of an encoder and a linear layer with parameters $θe$ and $W$⁠, respectively. The output logits are calculated by:

Alternatively, the softmax or CE loss in equation (13) is minimized to train a self-supervised classifier for instance discrimination in CL. The connection between SL and CL tasks is illustrated by looking at how the classification logits are related in equations (14) and (12). Here, the only alteration is made by replacing C-dimensional $W$ for SL in equation (14) with B-dimensional $Zcls$ for CL in equation (12), i.e. $W←Zcls$⁠. With this connection between classification tasks in SL and CL, it is meaningful to introduce an unsupervised setting for standard AT (Madry et al., 2018) in equation (2). By replacing the supervised classifier weights with the instance representation matrix, the standard supervised AT objective can be reformulated into a self-supervised adversarial optimization:

where the simulated targets $y˜=IB$ are imposed. Notably, two-view augmented data with weak [$tw(x)$] and strong [$ts(x)$ via $Zcls$ in equation (11)] distribution biases relative to $x$ are merged in equation (15). In the implementation, the adversarial attack data are first updated through PGD attack (Kurakin et al., 2018) by maximizing CE loss in equations (13) and (15) with the perturbation $δ$ given by:

or equivalently updated with K attack iterations based on:

The perturbations $δ={δi}$ are then added to anchor samples to obtain attack data $tw(x)+δ$ and used as new anchor samples to minimize CE loss again to train encoder $θ$ where the class samples $Zcls$ from alternative view $ts(x)$ are also used. The proposed self-supervised AT is implemented by incorporating contrastive view in a B-dimensional AT which consistently implements instance discrimination via a classification problem in equation (15).

This study bridges supervised and self-supervised AT and builds an efficient solution to self-supervised AT, which is configured as shown in Figure 7 and implemented by using Algorithm 2. Given an input sample and a set of target classes in standard supervised AT, the perturbation is estimated to trick the model to predict a wrong class. However, in previous self-supervised AT (Kim et al., 2020; Jiang et al., 2020), the classes were defined from positive/negative samples, in which some of them were attack data. Since PGD attack was iterative, in every attack iteration, some of the classes might be changed while the others stayed the same. An ambiguity ascent direction was caused during optimization over attack data. The issue of inconsistent class definition became even severe with adversarial attack.

Compared with RoCL (Kim et al., 2020) and ACL (Jiang et al., 2020), the proposed framework does not rely on additional contrastive streams or auxiliary adversarial objectives. Compared with the DynACL in Luo et al. (2023a), our method does not require dynamic augmentation scheduling. Instead, robustness is achieved through a unified classification-based instance discrimination mechanism with explicit separation between anchor and class representations.

Compared with the solutions to supervised AT in equation (2) and self-supervised AT in equation (5), the proposed self-supervised AT carries out a self-supervised method in equation (15) with three essential refinements including:

Algorithm 2. Enhancing instance discrimination for a new self-supervised adversarial training

1. replacement with the instance class matrix $W←Zcls$⁠;

2. assignment with the self-supervised target matrix $y˜=IB$⁠; and

3. incorporation of two-view augmented data ${tw(x),ts(x)}$ for anchor and class samples ${Xanch,Xcls}$⁠, respectively.

The proposed method addresses the issue of inconsistent instance labels which is caused when the same sample in CL alternatively acts as anchor and positive/negative samples. The classes in this self-supervised AT are defined by the class samples $Xcls$ which are consistent within a training step. The attack perturbation $δ$ is only added to the anchor samples $Xanch$ as seen in equation (17), thus the attack target stays the same throughout the attack iterations. These steps provide a clear ascent direction for the attacker and generate a strong attack during AT, which ultimately results in a robust model. Importantly, this paper presents a meaningful solution to self-supervised AT by transforming a supervised AT into a self-supervised variant through the replacement of $W$ in equation (14) with $Zcls$ in equation (12). Attractively, such a connection provides an avenue to extend any existing or future supervised AT methods to realize their corresponding self-supervised AT methods. This framework takes a further step toward unifying the supervised and self-supervised AT. This paper presents a contrastive perspective for implementing self-supervised AT. The instance discrimination in CL is enhanced to develop a new self-supervised AT. The resulting solution is simple and efficient. The extra views and costs in previous methods (Jiang et al., 2020; Fan et al., 2021) are avoided.

In addition, the proposed two-view AT can be extended by incorporating two-view representation in a previous supervised AT (Zhang et al., 2019) to build a new variant of self-supervised AT as well as in implementation of a robust self-supervised text embedding. For example, the adversarial attack $δ$ in the previous method called TRADES (Zhang et al., 2019) was generated by maximizing the Kullback-Leibler (KL) divergence between $fθe(x)$ and $fθe(x+δ)$⁠. Then, the image encoder parameter in a supervised AT framework was estimated by minimizing:

where $λ$ is a regularization parameter. A robust image encoder was trained in a supervised setting where C-dimensional one-hot target matrix of classifier $y$ was given. This study presents an approach to bridge supervised and self-supervised AT. In particular, a new self-supervised TRADES is proposed and extended by solving:

where two views of augmented data ${tw(x),ts(x)}$ via anchor and class samples ${Zanch,Zcls}$ are adopted to enhance the instance discrimination, respectively. The perturbation is therefore generated by using PGD attack:

A self-supervised TRADES is obtained from supervised TRADES by simply replacing $W←Zcls$ and setting $y˜=IB$⁠.

This study further presents a self-supervised AT method to build a robust text representation. In the implementation, first the latent class samples for individual instances in a mini-batch $x={xi}∼D$ consisting of words or sentences are calculated by:

using an encoder $fθp$ with a dropout rate p. The first view of training data is then formed via strong data augmentation $ts(·)$⁠. This view does contain positive and negative samples for instance discrimination which are used for CL. To enhance the consistencies in instance discrimination, a B-way classification problem is formulated to simulate a SL task by considering the weakly augmented samples $tw(·)$ or simply original samples. The latent anchor inputs $Zanch=(fθ(x1),…,fθ(xB))⊤$ using the encoder without dropout are used to form a second view to predict the corresponding classes for individual instances $xi$ as given in class matrix $Zcls$⁠. Accordingly, a new CL is proposed to simulate an SL task via enriching the instance discrimination by replacing the classification layer $W←Zcls$⁠, specifying the target matrix $y˜=IB$ and separating the views of anchor $Zanch$ and positive/negative class samples $Zcls$⁠. The problem turns out to find the optimal encoder $fθ$ or encoder with dropout $fθp$ based on two views of latent samples from the same original samples $x$⁠. The resulting solution aims to estimate an informative encoder to distinguish individual samples ${xi}$ or latent instances ${zi}$ mutually based on two different views $Zanch$ and $Zcls$ without and with dropout in encoder, respectively. The CE loss $LCE$ between targets $y˜$ and the logits $y^$⁠, which are calculated by softmax function using the dot product of $Zanch$ and $Zcls$ (as shown in Figure 6), is minimized.

More importantly, this new CL is fulfilled to implement AT for a defense text model. From our investigation, this is the first AT framework developed for sentence embedding in self-supervised setting where the encoder $θ$ and the perturbation $δ$ are jointly trained. A minimax optimization problem is formulated according to the same objective for instance discrimination in two-view CL via encoder without (⁠$fθ$⁠) and with (⁠$fθp$⁠) dropout via:

where the worst-case perturbations $δ={δi}$ for a mini-batch of samples $x={xi}$ with maximal CE loss $LCE$ are first calculated and then added to the word embeddings $x$ to estimate the optimal encoder $θ$ with minimum CE loss. The resulting encoder is robust to the worst-case perturbation $δ$⁠. Figure 8 shows the proposed self-supervised AT procedure, which is a direct extension from standard supervised AT in Figure 2 and two-view self-supervised AT in Figure 7. Basically, the adversarial perturbation can be generated by using the proposed self-supervised FGSM attack:

By augmenting the adversarial anchor representation $fθ(x+δ)$ which is obtained via an encoder without dropout, the proposed AT method bridges the supervised and self-supervised AT and compensates the mismatch of adversarial examples between training and test phases.

Further investigation over recent works on improving ACL is addressed from the perspectives of data augmentation and model regularization. In Luo et al. (2023a), a dynamic augmentation scheme was proposed to build a DynACL which compensated the gap due to the inconsistency between strong and weak augmentations in CL and AT, respectively, through an annealing schedule for augmentation strengths. In Xu et al. (2023b), DynACL was further improved by imposing a causal graph to enforce the style-independence and fulfill the AIR in the learned model DynACL-AIR. The robustness in DynACL was enhanced by minimizing an AIR loss. Another approach (Zhang et al., 2022) aimed to decouple the problem in ACL into two-stage sub-problems for a non-robust CL and a pseudo-supervised AT where an efficient solution was implemented. In Xu et al. (2023a), the tremendous running time in ACL was mitigated through a robustness-aware coreset selection which was feasible to run on a large-scale data set. The resulting method was performed by minimizing the representation divergence between natural samples and adversarial variants to assure the robustness transferability (RT). In Wang et al. (2023), the adversarial robustness was enhanced by a contrastive distillation via an adaptive denoising module. Typically, this paper presents a new two-view CL (Figure 6) and a new two-view AT (Figures 7 and 8) where the inconsistencies due to class definition and augmentation bias in previous CL and AT are addressed through an instance discrimination task based on anchor features and class features from two individual views of augmented data. To the best of the authors’ knowledge, this study provides one of the first unified analyses of class-definition inconsistency and augmentation mismatch in self-supervised AT.

The proposed method was examined for image and text representations where the tasks of image classification and sentence embedding were evaluated, respectively. The enhanced instance discrimination was implemented for CL and self-supervised AT.

This paper conducted the evaluation on image classification over CIFAR-10 and CIFAR-100 data sets where each data set had 50 K training images and 10 K test images with a size of 32 $×$ 32 $×$ 3. To ensure fair comparison and reproducibility, all experiments are conducted under a unified training protocol unless otherwise specified:

• Backbone architecture. For image representation, we adopt ResNet-18 (He et al., 2016; Chien and Chang, 2023) as the encoder $fθ$ following standard practice in CL. For text representation, we use a transformer-based encoder initialized from a pre-trained BERT-base model (Devlin et al., 2019).

• Training configuration. All models are trained using the Adam optimizer with an initial learning rate of $3×10−4$⁠, momentum 0.9, weight decay 0.001 and temperature $τ=0.5$⁠. The learning rate is decayed using a cosine annealing schedule. The batch size (B) is set to 256 for image experiments and 128 for text experiments. The models are trained for 800 epochs for image data sets and 200 epochs for text data sets by using the stochastic gradient descent algorithm. No ground-truth labels are used in self-supervised settings. Pseudo-labels are not used unless explicitly stated.

• Adversarial attack settings. We adopt the PGD as the adversarial attack method. For image data, the perturbation is bounded by an $l∞$-norm constraint with $ϵ=8/255$⁠. The number of PGD steps is set to $K=20$ with a step size of $α=2/255$⁠, which is known as strong attacks. For text representation, adversarial perturbations are applied in the embedding space with norm constraint $ϵ=1.0$⁠.

• Data augmentation. For image experiments, the strong random cropping and resizing, random horizontal flip, random color jitter and grayscale conversion were run to find class samples $Xcls$ or $ts(x)$ while the weak random cropping and resizing and random horizontal flip were done to find anchor samples $Xanch$ or $tw(x)$⁠. For text experiments, the weak augmentation was run by using dropout rate $pw$ either 0 or 0.1 while the strong augmentation was done by using $ps$ either 0.2 or 0.3, where each sentence is passed through the encoder twice with independent dropout masks.

In the experiments on image classification, a three-layer projection head was added for CL. An ensemble of parameter-free attack, called AutoAttack (Croce and Hein, 2020), was included in the evaluation. The standard linear evaluation setting was implemented. The trained encoder was fixed and a linear classifier was trained on top of encoder by using the labeled data. The experimental results were compared with current self-supervised AT methods under consistent setting including RoCL (Kim et al., 2020), ACL (Jiang et al., 2020), AdvCL (Fan et al., 2021) and DynACL (Luo et al., 2023a) where the solutions were all developed via CL. The effect of AIR with details in Xu et al. (2023b) was evaluated. The cross-task adversarial and common-corruption (Hendrycks and Dietterich, 2019) RT from pretraining using CIFAR-10 to finetuning using CIFAR-100 was examined. Different methods were further accessed in terms of memory usage and computation time where a personal computer with CPU Intel Xeon E5-2620 v4, RAM 32 G and GPU Tesla V100S was used in the experiments.

The experimental setting in evaluation of sentence embedding followed that in SimCSE (Gao et al., 2021). The semantic textual similarity (STS) between two sentence embeddings was evaluated by using seven STS data sets including STS12 to STS16 (Agirre et al., 2016), STS Benchmark (STSB) (Cer et al., 2017) and SICK-Relatedness (SICKR) (Marelli et al., 2014). The Spearman’s rank correlation was used as the evaluation metric. The CLS token in BERT was extracted as the sentence representation. The model was trained by using $106$ randomly sampled sentences from English Wikipedia the same as those used in SimCSE (Gao et al., 2021). The model was evaluated every 125 training steps on development set to assure the best performance checkpoint. There was no additional regressor on top of encoder. In this comparative study, the results of sentence encoders from the previous supervised methods (Reimers and Gurevych, 2019; Li et al., 2020a; Su et al., 2021) and self-supervised methods (Carlsson et al., 2021; Yan et al., 2021; Kim et al., 2021; Gao et al., 2021) were included.

First of all, the robust accuracy (RA) (%) on attack test images (PGD), standard accuracy (SA) (%) on clean test images, memory usage and training time (CIFAR-10) using SimCLR (Chen et al., 2020), RoCL (Kim et al., 2020), ACL (Jiang et al., 2020), AdvCL (Fan et al., 2021), DynACL (Luo et al., 2023a) and the proposed self-supervised AT are compared in Table 2. This comparison shows the performance of implementing a contrastive view in AT via equations (15) and (19) to carry out a new self-supervised AT and a new self-supervised TRADES, respectively. The proposed TRADES was implemented by merging the new contrastive view in Zhang et al. (2019) for original TRADES. Among different methods, CL using SimCLR is highly affected by attacks and performs the worst. The proposed AT performs better than RoCL and ACL on CIFAR-10 and CIFAR-100 in terms of RA. It is worth noting that the accuracy on attack data (RA) is much more important than that on clean data (SA) for adversarial defense. This is because in a real-world situation one will not know whether the input sample is attacked. RA can be seen as a worst-case guarantee of accuracy assuming all input data are attacked. Notably, the proposed AT (⁠$K=5$⁠) with RA 51.15% even outperforms AdvCL with RA 50.25% on CIFAR-10 where AdvCL additionally used the PLs and extra model architecture. The computation time and memory cost are even reduced by using the proposed AT. It is further found that RAs in two tasks are degraded when the batch size B is reduced to 256. The memory and computation costs are decreased as well. In addition, similar to AdvCL, the proposed AT and TRADES are implemented by using PLs. The resulting RA is therefore increased to 52.52%. In case of CIFAR-100, the highest RA 27.62% is obtained by using the proposed self-supervised TRADES via equation (19). This result without using PLs is comparable with that of AdvCL where PL is required. The SOTA result 28.05% is obtained by using the proposed TRADES with PLs. This is because CIFAR-100 involves more classes, and the resulting performance would be more likely benefited by the KL term in TRADES, which regularizes the entire output probability instead of simply learning from hard labels. When using the proposed TRADES, the time and memory complexities are lower than those of baseline AdvCL.

In addition, Table 3 shows RA and SA of the proposed self-supervised AT, and the new AT by incorporating the proposed contrastive view in TRADES under different hyperparameter $λ$ as well as in the helper-based adversarial training (HAT) (Rade and Moosavi-Dezfooli, 2022). The main goal of the proposed HAT is to increase SA in a self-supervised AT setting. The highest SA among all of AT variants is obtained by using the proposed self-supervised HAT. By increasing $λ$ in the TRADES loss, the model robustness is increased but the SA is decreased. This is consistent with the effect observed in the original TRADES (Zhang et al., 2019). The control over $λ$ enables us to achieve a higher SA while still maintaining a good robustness. This investigation reveals that CL viewpoint does develop the new AT variants while maintaining the benefits and characteristics from their original AT.

A key benefit of using contrastive view in the proposed AT is its intuition, simplicity and consistency. Owing to two perspectives in a hybrid CL and AT objective, the memory requirement is the same as that of the original SimCLR (Chen et al., 2020). The memory consumption is substantially reduced when compared with the other methods, and is even lower than one third of the memory relative to that used in AdvCL. Similar results are obtained in the metric of training time. Training time of the proposed self-supervised AT is generally smaller than or equal to half of the existing methods. The proposed AT with three attack iterations $K=3$ can reduce the training time while its RA is still higher than those using RoCL and ACL. Such a result reflects the robustness in presence of PGD attack as it can generate the challenging attacks for AT even with the reduced attack iterations. Overall, Figure 9 depicts the memory cost in GB versus the training time in seconds per epoch while RAs using different methods are shown. In this comparison, the proposed self-supervised AT or AT+PL obtains higher accuracies but smaller memory cost and computation time.

To analyze the contributions of individual components, we conduct an ablation study with the results as shown in Table 4. Starting from the baseline CL model, introducing AT via RoCL (Kim et al., 2020) significantly improves the robustness but degrades the SA. Reformulating the contrastive objective in CE loss as a classification problem using equation (10) improves the representation quality with slight robustness gain. Combining both CE loss and the proposed AT loss via equation (15) yields a clear improvement in robustness, indicating the compatibility of the classification perspective with adversarial optimization. Aligning augmentation strategies stabilizes the training and improves both standard and robust performance. Finally, an additional loss based on an AIR (Xu et al., 2023b) is merged to achieve the best trade-off, demonstrating the effectiveness of the complete framework.

In general, AT is performed to pursue robust features holding the perturbation invariance as illustrated in Figure 10. In this study, the image data in latent representation using the trained encoder are visualized via t-distributed neighbor embedding. Figure 11 shows two-dimensional (2D) visualizations of the learned representations of clean images $fθ(x)$ (blue dots) and the corresponding adversarial images $fθ(x+δ)$ (orange dots). The image samples of class “cat” on CIFAR-10 are displayed. The trained image encoder using SimCLR (Chen et al., 2020) in Figure 11(a) and the proposed self-supervised AT in Figure 11(b) are compared. It can be seen that CL encoder via SimCLR is susceptible to attack samples so that the clean and attack samples are mapped into different points in latent space. Using the proposed AT, the encoder is learned to hold the invariant property in presence of ambient attack or perturbation. Thus, the resulting classifier is able to classify attack images as good as their original images. The blue and orange dots are mostly overlapping since the proposed model views them as the same, and is accordingly robust to attacks or perturbation noises. For quantitative evaluation, this study calculates the relative invariance $ρ$⁠, which is defined as an inverse ratio of the accumulated distances between clean and attack samples for those latent samples of a method relative to SimCLR. The higher the value $ρ$⁠, the more invariant or robust the adversarial samples obtained by the learned encoder $fθ$⁠. This study compares 2D latent spaces by using the learned image encoders according to different self-supervised AT methods. Relative to CL encoder in Figure 11(a), four AT encoders in Figure 12 have higher overlapping between clean and attack samples which means better robustness. In this comparison, RoCL has the largest number of visible orange dots or sensitive attack images, which results in the lowest RAs in Table 2. The proposed AT obtained the highest relative invariance $ρ=2.01$ among different methods.

The adversarial robustness using different methods is further evaluated across two different data sets transferred from CIFAR-10 as a pretraining task to CIFAR-100 as a finetuning task (CIFAR-10$→$CIFAR-100) under the settings of cross-task adversarial and common-corruption (Hendrycks and Dietterich, 2019) RT. Fifteen types of common corruptions with corruption severity (CS) ranging from ${1,3,5}$ (CS-1, CS-3, CS-5) were adopted with details in Hendrycks and Dietterich (2019) and Xu et al. (2023b). The adversarial full finetuning (Luo et al., 2023a; Li et al., 2023) was performed in a way where the weights in a pre-trained encoder in a source task were used as the initialized weights to train the whole model in a target task by using adversarial data. In addition to the evaluation of RA (%) under PGD attack, this study compares different methods in terms of accuracy under AutoAttack (Croce and Hein, 2020) (denoted as AA [%]). In this set of comparison, an additional AIR loss (Xu et al., 2023b) was minimized to enforce independence from style factors for adversarial invariance in fulfillment of ACL-AIR, DynACL-AIR (Xu et al., 2023b) and ours-AIR where ACL (Jiang et al., 2020), DynACL (Luo et al., 2023a) and the proposed AT were performed, respectively. Table 5 reports different accuracies over different methods under various RT settings. It is found that dynamic data augmentation in DynACL improves the results of ACL under two cases of cross-task RT evaluation. AIR loss for regularization increases the accuracies. The proposed two-view AT with anchor and class features consistently performs better than ACL and DynACL under two RT settings.

The generalizability of adversarial robustness is further investigated toward realistic settings in presence of imbalanced data and small data as illustrated in Table 6. This paper adopted CIFAR-100-LT (Wu et al., 2021) and CIFAR-FS (Goldblum et al., 2020; Dong et al., 2025), constructed from CIFAR-100, as the data sets to evaluate different defense models under long-tailed distribution and few-shot condition, respectively. AIR loss was merged in our TRADES and our AT. The effect of various batch sizes B is evaluated. The settings of CIFAR-100-LT follow (Wu et al., 2021) and ours in CIFAR-100. Again, robust accuracies under AutoAttack (AA) and PGD attack (20 steps) (RA) and SA are reported. Using CIFAR-100-LT, the imbalance rate was set by 10 (Wu et al., 2021). The results of using RoCL (Kim et al., 2020), ACL (Jiang et al., 2020), TRADES (Zhang et al., 2019) and the robust balanced predictions (RoBal), obtained in (Wu et al., 2021), are included for comparison with those of using our TRADES and our AT. It is found that our AT obtains desirable robust accuracies in presence of AutoAttack and PGD attacks although SA of our AT is not as good as that of RoBal. Our two-view TRADES performs better than previous TRADES (Zhang et al., 2019). In addition, the five-way five-shot setting in CIFAR-FS is adopted to assess the effect of adversarial perturbations in few-shot learning (Goldblum et al., 2020). For comparison, a recent method to adversarially robust few-shot learning based on feature purification (FP) and cyclic reconstruction (Dong et al., 2025) is included. Our previous settings in CIFAR-100 are used. In this evaluation, our two-view AT achieves the highest SA with $B=128$ and the highest robust accuracies under AutoAttack and PGD attack with $B=64$ and $B=128$⁠, respectively. These accuracies are higher than those of FP.