Figure 3.
A flow diagram of event-based video reconstruction from event stream and low frames per second input to high frames per second output.The flow diagram presents an event based video reconstruction pipeline that converts an event stream and low frames per second intensity video into a high frames per second video. The upper path shows event to video conversion where an event stream is transformed into constructed frames. The lower path combines low frames per second video and event stream inputs that pass through flow estimation and synthesis modules. The intermediate outputs are merged in a fusion stage followed by refinement. The final stage produces a high frames per second video with temporally denser reconstructed frames derived from both event and intensity information.

Temporal restoration applications. (Top) Event to video generation models convert a spatio-temporal stream of events with microsecond temporal resolution into a high-quality video, which is typically monochromatic (however, colored outputs also exist through generative coloring). This enables applications such as high-framerate videos and high dynamic range capture. Usually, these models take in a 3D tensor of events along with the feedback from the previous predicted frame to generate the next video frame. (Bottom) Frame interpolation models leverage the high temporal resolution of event streams to estimate non-linear motion information between the frames and insert latent frames between two consecutive frames. As we discuss below, these models can be trained with full supervision, weak supervision or unsupervised methods. The general approach for such models follows processing the RGB and event streams separately and then fusing the information, followed by refinement

or Create an Account

Close subscription notice
Close access options