Figure 1
The diagram illustrates a multi-stage deep-learning pipeline for intention recognition using sequential video frames and their associated textual descriptions. It is divided into four main sections labeled “Input”, “Feature extraction”, “L S T M and Fusion”, and “Classification and Output”. On the far left, under the heading “Input”, a stack of “sequential images from U T M videos” is shown. These frames depict a person walking in a corridor while carrying a bag. An arrow points from these stacked frames to the next stage. Under the section titled “Feature extraction”, two types of features are generated. At the top, a series of vertically arranged convolution blocks represent a C N N used to extract “General image feature”. Each block is labeled with convolution operations such as “3 times 3 conv, 64” for the first five blocks, “3 times 3 conv, 128, slash 2” for the sixth block, “3 times 3 conv, 128” for blocks 7 to 9, “3 times 3 conv, 256, slash 2” for the tenth block, “3 times 3 conv, 256” for blocks 11 to 13, “3 times 3 conv, 512, slash 2” for block 14, and “3 times 3 conv, 512” for blocks 15 to 17, arranged in repeated groups. Curved arrows above the blocks show the sequential flow through these layers, culminating in a feature vector that is passed to the next stage. Below this, the diagram shows “High-level semantic feature” extraction. Individual images feed into a process labeled “Text for Encoder”. Simultaneously, the lower pathway begins with a selected frame box labeled “Image hash 1” feeding into a funnel-shaped module labeled “Text for Encoder”, which produces a sequence of textual tokens; these tokens are visualized in a tabular grid where the left column lists image identifiers “I subscript 1, I subscript 2, I subscript 3, ellipsis, I subscript n” and the inner cells show paired entries like “I subscript 1 times T subscript 1, I subscript 1 times T subscript 2, I subscript 1 times T subscript 3, ellipsis, I subscript 1 times T subscript n”, “I subscript 2 times T subscript 1, I subscript 2 times T subscript 2, I subscript 2 times T subscript 3, ellipsis, I subscript 2 times T subscript n”, “I subscript 3 times T subscript 1, I subscript 3 times T subscript 2, I subscript 3 times T subscript 3, ellipsis, I subscript 3 times T subscript n”, and “I subscript n times T subscript 1, I subscript n times T subscript 2, I subscript n times T subscript 3, ellipsis, I subscript n times T subscript n”. Above this, another frame box with a dashed outline labeled “Text for image hash 1” leads to a funnel shape labeled “Text for Encoder” and then branches into a horizontal grid pattern labeled “T subscript 1, T subscript 2, T subscript 3, ellipsis, T subscript n”. Both the convolutional feature vector and the semantic-token matrix are routed rightward into the central “L S T M and Fusion” zone: first each feature stream enters its own “L S T M unit” drawn as rounded rectangles containing small internal gate icons and labels for gate operations such as sigma and an inner box labeled “tangent hyperbolic”, with time-step annotations “t minus 1” and “t” to indicate recurrence; arrows labeled “Input slash Previous layer input” feed into the left of each L S T M unit, and arrows labeled “Output slash Next layer input” exit from the top-right. The two LSTM outputs then converge—an explicit downward arrow labeled “cat” marks the concatenation of these feature sequences—and the concatenated vector is fed into a subsequent L S T M diagram at the bottom, which shows the concatenation arrow entering the lower L S T M unit. After recurrent fusion, the final fused L S T M output is sent rightward into the “Classification and Output” zone, where it arrives at a dense feedforward classifier depicted as a tall vertical column of green circular nodes connected by many thin black lines to a parallel column of blue output circles; each blue output node is labeled to the right as “Intention 1”, “Intention 2”, “Intention 3”, “Intention 4”, “Intention 5”, and “Intention 6”, stacked top to bottom.Model architecture of the HIPM. Source: Authors’ own work