The four architectures take input X with dimensions B, T and C. Panel A shows a token mixer block. LayerNorm is followed by linear projection, self-attention, linear projection and dropout, then a residual addition. A second branch applies LayerNorm, M L P and dropout before another residual addition. Panel b replaces self-attention with average pooling in 2 dimensions. The sequence is LayerNorm, linear projection, average pooling in 2 dimensions, linear projection and dropout, followed by residual addition. LayerNorm, M L P and dropout then form the second residual branch. Panel c applies LayerNorm and linear projection before a channel-split mixing unit. The channels are divided into groups and processed by multiple M L P branches. Their outputs feed linear projection and dropout, followed by residual addition. LayerNorm, M L P and dropout then form the upper residual branch. Panel d applies LayerNorm, unpatching and a convolutional mixing unit. The unit contains patch-wise convolution and activation, depth-wise convolution and activation, and another patch-wise convolution. Its output passes through linear projection and dropout before residual addition. LayerNorm, M L P and dropout then form the final residual branch.Schematic overview of the original token mixer used in VCT and first three token mixers evaluated within the ATEM framework. (a) Vanilla Attention: the standard multi-head self-attention (MHSA) module (Vaswani et al., 2017) used in the VCT baseline. (b) Pooling-Mixer: a parameter-free replacement introduced to establish the performance lower bound, following the methodology of MetaFormer (Yu et al., 2021). (c) MLP-Mixer: adapted from ResMLP (Touvron et al., 2021), this token mixer tests whether the fixed-topology mixing strategies successful in classification tasks can translate to the pixel-precise demands of compression. (d) CNN-Mixer: a convolutional mixer (Sandler et al., 2018) serving as a local-only counterpart to self-attention, designed to evaluate the efficacy of rigid spatial receptive fields in entropy modeling
Source: Authors’ own work