Google Summer of Code 2023 Work Product Submission

Student: Soham Gupta

Organisation: VideoLAN

Project: Improve Chromecast Support (subtitles)

Introduction

Over the past 3 months, I had the opportunity to work on the VLC project for Google Summer of Code 2023. It was an invaluable learning experience that helped me grow my skills in software development and open source contribution. In this blog, I'll summarize my work and key learnings.

About the Project

🎥 The project involved enhancing the capabilities of VLC's HTTP Live Streaming (HLS) module by:

• Implementing support for H.265/HEVC video codec
• Improving performance of video casting to Chromecast
• Enabling blending of subtitles with videos

I worked primarily on the chromecast-hls branch under the guidance of my mentor Alaric Sénat and org admins Thomas Guillem and Jean-Baptiste Kempf.

Work Summary

Here are some highlights of what I was able to accomplish over the course of the program:

Transcoding Bug Fix

• 🐛 Fixed a bug in the transcode module to resolve synchronization issues when using transcode:display with paced outputs(UDP, HLS and display)
• ✅ Successfully closed issue #25959 through this bug fix

H.265/HEVC Video Codec Implementation

• 🔍 Researched details of implementing H.265 support
• 🔧 Added code to generate H.265 codec information in codecs.c
• ✅ Transcoded sample videos to H.265 using ffmpeg and tested with hls module
• ❌ Faced challenges manipulating profile/level values to match H.265 codec syntax

Chromecast Performance Improvements

• 🔌 Troubleshot various connectivity issues between VLC and Chromecast device
• 👷 Rebuilt code frequently to address streaming problems
• 📶 Worked around WSL2 network limitations to get video casting working
• 💤 Reduced lag and improved synchronization of audio/video casting

Subtitle Blending

• 💬 Successfully blended SRT subtitles with videos using soverlay parameter
• ⏪ Fixed timing issues causing subtitles to be out of sync
• 📖 Explored advanced subtitle formats like ASS, WebVTT but faced rendering issues at times

Testing Subtitle Blending (Windows Build)

SRT Format

• Used the following command to blend SRT subtitles:

./vlc.exe video.mp4 --input-slave=sub.srt --sout="chromecast{ip=[ip address]}" --demux-filter=cc_demux --no-plugins-cache -vv | cat -

ASS Format

• Used the following command to blend ASS subtitles:

./vlc.exe video.mp4 --input-slave=sub.ass --sout="chromecast{ip=[ip address]}" --demux-filter=cc_demux --no-plugins-cache -vv | cat -  

WebVTT Format

• Used the following command to blend WebVTT subtitles:

./vlc.exe video.mp4 --input-slave=sub.vtt --sout="chromecast{ip=[ip address]}" --demux-filter=cc_demux --no-plugins-cache -vv | cat -

Tracing Flush sout callback

• ❌ Trace module did not log Flush events in the JSON file in the Windows build

• ⏰ Added timestamps for Flush events to enable tracing

• 📈 Traced video frames to analyze timestamp issues

Contributions

Over the course of GSoC 2023, I made the following contributions to the VLC project:

• Created 3 merge requests over the course of the program 🔀
  • Transcoding Bux Fix
  • HEVC Video Codec Implementation
  • Traced Flush events
  • Patches for issues like HTTPD URL deletion, GnuTLS, etc
  • Clear documentation doubts through daily logs and merge request descriptions
  • Responded to code reviews and made changes as per suggestions

  Learning Outcomes

  Through this project, I greatly improved my skills in areas like:

  • 🔧 Tooling - gcc, clang, docker, ffmpeg
  • 💻 Application development - VLC architecture, FFmpeg transcoding, Chromecast protocols
  • Open Source Workflow - version control, build troubleshooting, submitting merge requests
    • Git Version Control --> git commit --amend, git stash, git rebase, git reflog, splitting commits, changing authors & many more in the process
  • ✏️ Technical communication - daily logs, merge request descriptions, collaborating with mentor

  Future Scope

  While I was able to complete core objectives, some areas of improvement remain:

  • Add support of fragmented mp4 in the HLS server
  • Achieve 1:1 matching of H.265 codec profile/level values
  • Reduce audio/video sync issues during Chromecast streaming
  • Support text subtitle formats like WebVTT
  • Improve modularisation and memory management

  I'm excited to continue contributing to VLC and build on the work done during GSoC 2023.

  Challenges Faced 💥

  Throughout the program, I encountered various challenges that tested my problem-solving skills:

  • 🔌 Connectivity issues between VLC and the Chromecast device due to network limitations of WSL2. This required extensive troubleshooting.

  • 🔧 Frequent rebuild errors during cross-compilation that needed debugging of outdated contribs and prebuilt contribs.

  • ❌ Manipulating the profile and level values to match the H.265 codec syntax was difficult without proper documentation.

  • ⏪ Synchronization problems between audio and video streams during Chromecast casting. Identifying the root cause took time.

  • 💤 Limitations on video segment duration and buffers caused intermittent casting failures.

  • 🐛 The initial part of videos without audio failed to render in the transcoded stream.

  • ❓ Absence of logs and abrupt UI closures made it hard to pinpoint errors.

  • ⚒ Frequent merge conflicts and overwrite errors while rebasing branches caused delays.

  By persevering through these roadblocks, I improved my debugging skills and learned how to approach problems methodically. Special thanks to my mentor for their invaluable guidance when I was stuck.

  Final Weeks' Work

  • ⌛️ Added delays using vlc_tick_sleep to extend video casting

  • 🐛 Debugged asserts and crashes during Chromecast streaming

  • ✅ Got WebVTT subtitle blending working with Chromecast

  • 📦 Committed and pushed all code changes to respective branches which I made throughout the project timeline

  Branches Worked On

  Branch	Overview	Commits
  chromecast-hls	Initial work on Chromecast streaming	View
  rebased/chromecast-hls.2	Final work on Chromecast streaming	View
  sout-hls.4	H.265 codec implementation	View
  fix/transcode	Transcoding synchronization fix	View
  trace	Added tracing module	View

  Acknowledgments

  I'm grateful to my mentor Alaric Sénat for his invaluable guidance and support throughout this enriching journey.
  🙏 And a big thanks to the VideoLAN's VLC project maintainers and Google Summer of Code program for giving me this opportunity to learn and grow as a developer👨‍💻!

  README.md