Inclusion of patented video codecs in ParaView/VTK

Development
1

I’m putting this out here for us to learn how an open source project would go about using patented video codecs (h.264/h.265) to

  1. generate video files in a patent-protected non-free container format like mp4
  2. live stream the encoded bytes.

ParaView can generate an mp4 video from a sequence of still images with the help of an h.264/h.265 encoder. I’ll clear some doubts and confusion right away - mp4 is a multimedia container for video, audio, and subtitles. h.26x family are the video codec specification that compresses an image sequence into a media container. Likewise, there are audio codec specs - vopus, vorbis to compress audio bits into a multimedia container.

For ParaView’s sake, we’re concerned about video codecs.

The h.26x spec is implemented by different vendors, it can be a software implementation -

  1. cisco/openh264 ← BSD-2 license
  2. x264 ← GNU GPLv2
  3. x265 ← GNU GPLv2, also available under a commercial license.

… or hardware implementation. All GPU vendors (intel, nvidia, amd) implement the codecs in their driver. As a developer, you are required to agree to the vendor’s terms and obtain an SDK (headers,…) from the vendor. As a user, you only need to set up the drivers properly. If you’re able to use ParaView, chances are your drivers are well set up.

There are some middleware that can be used to lighten the developer burden - ex: ffmpeg and gstreamer - but the licenses are a mix of GPL and LGPL.

  1. Intel on windows: Developer needs media-sdk-windows. User must have installed the intel graphics drivers.
  2. Intel on linux: Developer needs intel-onevpl or libva. A user must have installed intel’s graphics driver properly. Almost every simple distro does this automatically. Advanced distros have detailed documentation on how to DIY.
  3. NVIDIA on windows and Linux: Developer needs NVIDIA VIDEO CODEC SDK - Get Started | NVIDIA Developer. A user must install nvidia drivers properly. On Linux, you need proprietary drivers. nouveau wouldn’t work.
  4. AMD on windows and Linux: Developer needs AMF or libva. A user must have a proper installation of AMD GPU drivers. On linux, the open source radeon driver can use libva.

Question
Can we use the hardware SDKs to build ParaView binaries? The codec implementation sits in the drivers anyway and we’re not redistributing that non-free patent-protected piece of work. Since the hardware is paid for, does it imply that the patent fees are already paid?

If we pick software implementations, Kitware has to either distribute x265/x264 source - which we can’t due to license incompatibility with ParaView or ask users to download libx264/x265 from their package managers or build it from source.

Alternatives
An alternative route is to pick the VP9 or AV1 codec specs which are royalty-free and come with BSD-2,3 software implementations - libvpx, svt-vp9 Although, mp4 cannot contain these bit streams. VP9 requires a WebM file format. It is not as universal as mp4 on the desktop. Ex: not playable in Quicktime.

Thoughts? @berkgeveci @cory.quammen @jourdain

2

Another route is to go through OS-specific frameworks.

  1. Windows - Media Foundation
  2. Mac OS - Video Toolbox
  3. Linux - Intel made an attempt to make libva the single video acceleration framework on linux. But, that was a failed attempt - each vendor came up with their own framework, so the state of video acceleration is quite sad on Linux.
3

Thanks Jaswant, well done.

4

Using libva on linux seems to be reliable enough.

Are the OS-specific frameworks installed by default on MacOS and Windows ?

5

On Mac: Yes.
On Windows:
H.264: Yes
VP9: No. (free download from MS store. VP9 Video Extensions - Microsoft Apps)
HEVC: No. Microsoft charges a dollar for this :smirk: HEVC Video Extensions - Official app in the Microsoft Store

6

Then using commonly available frameworks maybe be a good idea.