Figure 6
The diagram begins with a single rectangular block on the far left, labeled “Input” with “State S subscript t” inside. An arrow points from the “Input” block to a stack of multiple square planes. This stack is labeled “Conv 2 D” above it. Below this stack, details of the convolutional layer are provided: “Kernel equals 3 by 3,” “Output channels equals 16,” “Padding,” and “R e L U.” An arrow, labeled “Max-Pooling 2 by 2,” then connects this first stack of planes to a slightly larger stack of planes, which is labeled “Conv 2 D” above it. Below this second stack, similar details are provided: “Kernel equals 3 by 3,” “Output channels equals 32,” “Padding,” and “R e L U.” An arrow, labeled “Max-Pooling 2 by 2,” then connects this second stack to a thin vertical rectangle to the right, which is labeled “Flatten Layer.” From the “Flatten Layer,” an arrow points to another rectangle labeled “Fully Connected 128.” Another arrow from this layer is labeled “R e L U” and points to a third vertical rectangle labeled “Fully Connected 64.” From this third rectangle, an arrow “SoftMax” is pointing to a final rectangle, which is labeled “Output” at the top and “Q-values Q subscript t” at the bottom.DQN neural network diagram. Source: Authors’ own work