Developing Low latency Video Telephony solutions using GStreamer

GStreamer provides a powerful and flexible way to develop streaming media applications for use-cases such as Video Telephony, Live Audio/Video streaming, Video Conferencing e.t.c, by supporting plugins which utilize both hardware accelerated media components present in SoC and also supporting software based processing entities. To ensure good overall user experience on such streaming media applications, there are various quality factors to ensure both at user-space and kernel-space level such as those related to maintaining Audio/Video Sync, Latency optimization & Performance fine-tuning, Detecting and avoiding Video Frame Skip, Audio distortion & Clipping, maintaining Audio/Video Quality and Error recovery e.t.c to ensure good overall user experience. The talk will go through above design considerations and also cover how to prototype, debug and optimize such low latency audio/video streaming application. Taking a TI K3 based SoC as a reference example for prototyping the video telephony use-case, the talk will go through the basic building blocks of the video telephony use-case covering relevant concepts involved for each component from GStreamer and underlying Linux kernel frameworks perspective. Lastly it will cover tools & techniques for testing, debugging, stabilizing and optimizing such solutions.