The pipeline processes frame x sub t minus 2, frame x sub t minus 1 and the current frame x. The first two frames pass through separate encoders to produce y sub t minus 2 and y sub t minus 1. Each output is tokenised and supplied to the entropy model as temporal context. The current frame x passes through an encoder to produce y sub t. This representation is tokenised, then quantised by Q before arithmetic encoding by A E. The entropy model receives temporal tokens from the two earlier frames and information from the current token stream. It outputs distribution parameters sigma and mu to both A E and arithmetic decoding by A D. The coded bitstream passes from A E to A D. The decoded representation is de-tokenised to obtain y hat sub t. The decoder then converts y hat sub t into reconstructed frame x hat sub t.Illustrations of a generalized design of Transformer-based LVC architectures. The Transformer-based design, led by VCT (Mentzer et al., 2022), eliminates the need for predefined motion priors, resulting in a significantly simpler entropy coding pipeline. Here, temporal context is derived via a cross-attention module that mixes latent features from previous frames with the current representation. In the figure: original frame is individually encoded into latent features by an encoder. The temporal context comes from the concatenation of tokens from previous two encoded frames and . The entropy model queries the current frame tokens on the key and value from the past tokens to predict the PMF needed to encode the current frame
Source: Authors’ own work