Figure 8.
The diagram contains Attack Generation and Model Training stages. Input x is represented as word embeddings and passes through f theta. One path uses no dropout to produce anchor features f theta of x. Another path uses dropout p to produce class features z cls. A dot product combines the anchor and class features. Softmax produces output logits y hat, which are compared with targets y tilde using cross-entropy loss L C E. Delta i is calculated using epsilon sign of the gradient of L C E with respect to x i. Delta is added to x to create x plus delta. The perturbed embeddings pass through f theta with no dropout to produce anchor features f theta of x plus delta. These anchor features are combined with the same class features by dot product. Softmax produces another set of output logits y hat, which are compared with the same targets using L C E. The gradient of L C E with respect to theta is used to update f theta.Self-supervised adversarial training for sentence embedding by using two-view samples without and with dropout