Figure 7.
The left architecture processes frame t minus 2 and frame t minus 1 through separate Encoder sub Separate branches. Their outputs are combined before entering Encoder sub Joint. Joint self-attention then produces features that pass to cross attention. Frame t is processed by Encoder sub Masked and supplies the second input to cross attention. The right architecture processes frame t minus 2 and frame t minus 1 through separate Encoder sub Separate branches. Their outputs undergo channel-wise concatenation, followed by linear projection and joint self-attention in Encoder sub Joint. This joint output enters cross attention. Frame t again passes through Encoder sub Masked and provides the second cross-attention input.Comparison of temporal token selection schemes. Left: The VCT framework applies an sliding window to the latent features of two previous frames. These are concatenated along the sequence dimension ( tokens), resulting in a computationally intensive attention matrix. The spatial context of the current frame is separately extracted via a window. Right: The proposed scheme introduces a more streamlined approach with lower computational cost. Instead of sequence expansion, tokens from previous frames are concatenated along the channel dimension and fused via a single linear projection layer. This maintains the original sequence length of 64, effectively reducing the final attention matrix to
Source: Authors’ own work