Video Frame Scheduler Plugin for Improving WebRTC Playback Quality

In WebRTC-based applications, video frames are often delivered at irregular intervals due to the nature of real-time communication. This irregularity can cause issues in waylandsink, which commits frames based on the arrival of frame_redraw_cb signals.

As a result, even when all frames are correctly delivered—for example, in a 60fps video where all 60 frames reach waylandsink—frames may still be dropped in the gst_wayland_sink_show_frame function if the redraw callback is not received in time.

On webOS devices developed by LGE, we tested 60fps video playback using cloud gaming and observed a frame drop rate of over 35% in waylandsink. In contrast, when using non-WebRTC streaming methods with more consistent frame intervals, playback was smooth with almost no frame drops under the same conditions.

To address this issue, we developed a GStreamer plugin called Video Frame Scheduler, which sits between the video decoder and sink. It schedules frame delivery at regular intervals, simulating consistent frame timing. This reduced the frame drop rate to below 10%, significantly improving playback smoothness and reliability.

In this Lightning Talk, we would like to share the challenges we encountered, the design of our solution, and engage with the GStreamer community to explore alternative or complementary approaches. We look forward to exchanging ideas and insights with fellow developers and multimedia experts.