GStreamer 1.0 support in Firefox for H.264 HTML5 video

Linux distributions coming up from now on do likely support GStreamer 1.0 only.
The implementation might need to be updated in Gecko.

Given that the Firefox process already maps gstreamer libraries (through libcanberra) it's not really an option to have 0.10 in parallel to 1.0 as there are symbol clashes between both AFAIK.

Porting information:
http://cgit.freedesktop.org/gstreamer/gstreamer/tree/docs/random/porting-to-1.0.txt

Created attachment 714832
First pass at a patch.

I have made a patch to optionally build against 1.0, but it may need work. I'm not sure how best to test it (I get a lot of unexpected mochitest failures and timeouts without my patch; I'm not sure if that's normal or if there is something strange about my set-up).

Great work Mike!

Alessandro Decina should be the one to review your patch, he's got the best understanding of the GStreamer backend.

I've made this bug block bug 833628 because I was tracking fixing a particular shutdown hang that I saw with the GStreamer backend in mochitests there.

Created attachment 729607
Work in progress.

Work in progress to update to apply against the current code base; I'm trying to figure out how the fix for bug 761018 should be handled (ie, one should call gst_buffer_map for access to the data).

Also, do you have advice for testing this? Running the mochitests gives me a timeout if gstreamer is enabled regardless of whether I apply this patch (probably bug 833628).

Created attachment 734530
WIP 1.0 patch

Working 1.0 support. This is still WIP, I need to find a way to reduce the #ifdefs and fix seeking (seek forward works, backwards is broken).

Created attachment 746297
WIP 1.0 support

Seeking works now and I started moving the 0.10 specific code to GStreamerReader-0.10.cpp

I'm getting a few link errors on linux with this new patch:
/usr/bin/ld.gold.real: error: hidden symbol 'gst_video_meta_api_get_type' is not defined locally
/usr/bin/ld.gold.real: error: hidden symbol 'gst_video_meta_map' is not defined locally
/usr/bin/ld.gold.real: error: hidden symbol 'gst_video_meta_unmap' is not defined locally
/usr/bin/ld.gold.real: error: hidden symbol 'gst_buffer_pool_config_get_video_alignment' is not defined locally
/usr/bin/ld.gold.real: error: hidden symbol 'gst_buffer_pool_config_get_video_alignment' is not defined locally
/usr/bin/ld.gold.real: error: hidden symbol 'gst_video_meta_api_get_type' is not defined locally
/usr/bin/ld.gold.real: error: hidden symbol 'gst_video_buffer_pool_get_type' is not defined locally

All of these functions appear to be defined in libgstvideo, and libgstvideo is being linked properly.

There were also some messages about gst_memory_is_type not being defined (doesn't appear to be in 1.0), but got around that by using GST_IS_MOZ_GFX_MEMORY_ALLOCATOR instead.

Compiles fine on my Mac; audio works, but gstreamer fails to build the pipeline for video. I can get video to play from gst-launch with `filesrc ! qtdemux ! vtdec_h264 ! videoconvert ! autovideosink'. When trying to play through Firefox though, video goes through `qtdemux ! multiqueue ! h264parse' and stops there. Will attach pipeline dump.

Created attachment 748341
sadface pipeline dump

(In reply to Edwin Flores [:eflores] [:edwin] from comment #6)
> I'm getting a few link errors on linux with this new patch:

Actually, looks like I've just hosed my gstreamer setup somehow on my linux box. Grr.

(In reply to Edwin Flores [:eflores] [:edwin] from comment #6)

> Compiles fine on my Mac; audio works, but gstreamer fails to build the
> pipeline for video. I can get video to play from gst-launch with `filesrc !
> qtdemux ! vtdec_h264 ! videoconvert ! autovideosink'. When trying to play
> through Firefox though, video goes through `qtdemux ! multiqueue !
> h264parse' and stops there. Will attach pipeline dump.

Hm, have you tried to play from gst-launch with playbin?

Created attachment 748471
1.0 support

New version of the patch. To remove some of the #if GST_VERSION_MAJOR clutter, I refactored the code slightly and split the largest 0.10 specific bits in GStreamerReader-0.10.cpp.

Playback with 1.0 (--enable-gstreamer=1.0) works but there's still some work left to do to pass the test suite. When used with gst 0.10 (--enable-gstreamer=0.10 or simply --enable-gstreamer) the patch introduces no regressions when running the test suite.

Created attachment 748518
1.0 support

Same as the previous patch, with a fix in configure.in so that --enable-gstreamer=X actually works

(In reply to Alessandro Decina from comment #9)
> Hm, have you tried to play from gst-launch with playbin?

Running:

gst-launch-1.0 playbin uri=file:///Users/edwin/bruce.mp4

I get the message:

WARNING: from element /GstPlayBin:playbin0/GstURIDecodeBin:uridecodebin0: No decoder available for type 'video/x-h264, stream-format=(string)avc, alignment=(string)au, level=(string)5, profile=(string)high, codec_data=(buffer)01640032ffe1001b67640032ac34e601e0089f961000000300100000030300f18319a001000668e9784cb22c, width=(int)1920, height=(int)1080, framerate=(fraction)24/1, pixel-aspect-ratio=(fraction)1/1, parsed=(boolean)true'.

The plugin I'm trying to get it to use is the applemedia vtdec_h264 plugin in gst-plugins-bad. The sink caps look like they should accept the above type:

    Capabilities:
      video/x-h264
                  width: [ 1, 2147483647 ]
                 height: [ 1, 2147483647 ]
              framerate: [ 0/1, 2147483647/1 ]
          stream-format: avc

Created attachment 748637
Small fixes

Fixes for a few small issues I had on linux:
 - gst_memory_is_type doesn't exist in gstreamer <= 1.0
 - When upstream doesn't negotiate allocator, CopyIntoImageBuffer would fail because |mBufferPool| isn't configured
 - gst/video/gstvideo{meta,pool}.h needed to be included
 - Had to force visibility in above libs to "default" to avoid link errors

(In reply to Edwin Flores [:eflores] [:edwin] from comment #13)
> Created attachment 748637
> Small fixes

Looks good, I'll integrate these fixes

> Fixes for a few small issues I had on linux:
> - gst_memory_is_type doesn't exist in gstreamer <= 1.0

It was added in 1.2, what 1.x version do you have? Anyway the patch is simple enough so let's put it in for now.

> - When upstream doesn't negotiate allocator, CopyIntoImageBuffer would fail
> because |mBufferPool| isn't configured

Good catch. I'm not 100% sure that passing GST_CAPS_ANY is safe there, i'll double check.

> - gst/video/gstvideo{meta,pool}.h needed to be included
> - Had to force visibility in above libs to "default" to avoid link errors

Hm, I guess these have to do with the fact that you're using an older 1.x than me

I'm using 1.0.5. I think most distros are on 1.0.x, so it's the most desirable target for now.

(In reply to Edwin Flores [:eflores] [:edwin] from comment #15)
> I'm using 1.0.5. I think most distros are on 1.0.x, so it's the most
> desirable target for now.

0.10 is still the default that comes on a lot of systems. Some systems don't even have gstreamer 1.0 (like current Ubuntu LTS). For additional fun, if some system libraries are using gstreamer 0.10, using gstreamer 1.0 is a pile of bugs waiting to happen.

> 0.10 is still the default that comes on a lot of systems. Some systems don't
> even have gstreamer 1.0 (like current Ubuntu LTS). For additional fun, if
> some system libraries are using gstreamer 0.10, using gstreamer 1.0 is a
> pile of bugs waiting to happen.

gstreamer 0.10 is EOF upstream so 1.0 is highly desirable and has much better Mac/Windows support. Even with that they are both parallel installable without too many problems on Linux.

(In reply to pbrobinson from comment #17)
> > 0.10 is still the default that comes on a lot of systems. Some systems don't
> > even have gstreamer 1.0 (like current Ubuntu LTS). For additional fun, if
> > some system libraries are using gstreamer 0.10, using gstreamer 1.0 is a
> > pile of bugs waiting to happen.
>
> gstreamer 0.10 is EOF upstream

Upstream doesn't choose what is on actual user systems, unfortunately.

> so 1.0 is highly desirable and has much
> better Mac/Windows support. Even with that they are both parallel
> installable without too many problems on Linux.

They are parallel installable, but not parallel linkable. So if the user system has, say, libcanberra-gstreamer installed, which is using gstreamer 0.10 on that system, and which we load for event sounds (and that gtk uses too, iirc), and we load gstreamer 1.0, "fun" bugs will show up.

(In reply to Mike Hommey [:glandium] from comment #18)
> (and that gtk uses too, iirc)

That would be libgnome.

(That's why bug 859199 is essential).

(In reply to Edwin Flores [:eflores] [:edwin] from comment #6)
> Compiles fine on my Mac; audio works, but gstreamer fails to build the
> pipeline for video. I can get video to play from gst-launch with `filesrc !
> qtdemux ! vtdec_h264 ! videoconvert ! autovideosink'. When trying to play
> through Firefox though, video goes through `qtdemux ! multiqueue !
> h264parse' and stops there. Will attach pipeline dump.

Found it! vtdec_h264 has GST_RANK_NONE, but playbin filters down to only GST_RANK_MARGINAL.

This should be fine, though. If we're include gstreamer in the build, we can just patch over it to set the rank higher.

What's the progress on this now?

See:

- https://en.opensuse.org/openSUSE:Goals_13.1

- https://en.opensuse.org/openSUSE:Goals_13.1/Port_to_GStreamer_1.0

I'm Interested in firefox playing h264 video under raspberry Pi.

Altought current version of Debian/Raspian linux use Gtreamer 0.10, porting of gstreamer 1.0 is already working (version 1.08 ad apt package)
Gstreamer 1.0 allow use of hardware decoding of h264, but

I kindly ask if your patch could be applied to modify firefox/iceweasel 10.0.12 in Raspberry, so to achieve hardware video decoding.

Could it be possible ?

Hi Giuliano. That's a question for the maintainers of your Raspberry Pi distribution. We don't support Firefox 10 any more; your best bet for this feature would be to wait until a more recent version is packaged for your distro.

thanks RAlph.
Currente updates in debian have realesed firefox/iceseasel 17
http://plugwash.raspbian.org/

should it be compatible for your patch ?
The big question is

which version of firefox is compatible with gstreamer 1.0 ?

(In reply to Giuliano Lotta from comment #26)
> which version of firefox is compatible with gstreamer 1.0 ?

None yet till the patch lands in mozilla source tree.

And the patch itself for which version (= or >= )is realized ?

Created attachment 788730
WIP

Did a bit of hacking on this a couple weeks ago; thought I should put this up.

Fixed a couple seeking issues, and with that a few tests fixed themselves.

Also found that we'd get bogus metadata using anything other than the GST_VIDEO_FRAME_* macros on 1.0.

When upstream fails to negotiate the allocator, we used to instantiate one and then copy the buffer over. Now we just copy to a PlanarYCbCrImage directly.

Outstanding issues:
 * Some tests still fail.
 * gst_buffer_pool_config_get_video_alignment returns total rubbish, even when the option GST_BUFFER_POOL_OPTION_VIDEO_ALIGNMENT is present. For now, gst_video_info_align is commented out.

(In reply to Edwin Flores [:eflores] [:edwin] from comment #29)
> Outstanding issues:
There is also still a shutdown hang sometimes.

This's going to help with va-api also.

1.2 has been released. Should this be the target instead? Seems to be mostly compatible with 1.0.

(In reply to Edwin Flores [:eflores] [:edwin] from comment #29)
> * gst_buffer_pool_config_get_video_alignment returns total rubbish, even
> when the option GST_BUFFER_POOL_OPTION_VIDEO_ALIGNMENT is present. For now,
> gst_video_info_align is commented out.
I'm no expert but shouldn't you check that return value of gst_buffer_pool_config_get_video_alignment is TRUE before using what it produces? Based on quick reading of gst looks like if it doesn't return TRUE, it failed to parse config.

(In reply to seppo.yli-olli from comment #33)
> (In reply to Edwin Flores [:eflores] [:edwin] from comment #29)
> > * gst_buffer_pool_config_get_video_alignment returns total rubbish, even
> > when the option GST_BUFFER_POOL_OPTION_VIDEO_ALIGNMENT is present. For now,
> > gst_video_info_align is commented out.
> I'm no expert but shouldn't you check that return value of
> gst_buffer_pool_config_get_video_alignment is TRUE before using what it
> produces? Based on quick reading of gst looks like if it doesn't return
> TRUE, it failed to parse config.

Yeah. Last weekend I rebased the patch and fixed this and a few more issues (OMTC broke the patch apparently). I will post an update soonish.

Created attachment 819584
0001-Bug-806917-support-GStreamer-1.0.patch

Updated, rebased patch.

Edwin: I couldn't rebase your last patch (https://bugzilla.mozilla.org/attachment.cgi?id=788730), as I think it's applied on some other unpublished branch. I tried to get all the changes in and I *hope* I didn't miss anything.

Comment on attachment 819584
0001-Bug-806917-support-GStreamer-1.0.patch

Setting r? so I remember to review/test this patch soon. If it doesn't regress 0.10 and code review is okay then there's no reason we can't land it.

Hi, I was wandering - considering the fact that the already available GStreamer 1.2 plus the soon-to-be-released version of gstreamer-vaapi allow for hardware acceleration on Linux desktop - if that mean that when using GStreamer (for H.264 decoding) we will get hardware acceleration whereas when using internal Firefox codec (for Theora & VP8 decoding) we will get no hardware acceleration ? Thank you in advance for claryfying that point!

Hardware H.264 support through VA-API is possible but not yet supported. Hardware decoding of VP8 is not supported by VA-API.

Comment on attachment 819584
0001-Bug-806917-support-GStreamer-1.0.patch

+ r? khuey to review the change to configure.in (and moz.build?).

Created attachment 830538
806917-nov-fixes.patch

Needed at least these fixes to get it near working on linux.

Even after this patch, I don't get any video, audio is distorted, and for some reason the time bar only updates once every few seconds at the quickest. This happens on both 0.10 and 1.0.

Comment on attachment 819584
0001-Bug-806917-support-GStreamer-1.0.patch

Putting off review until linux regressions can be dealt with.

Edwin, I filed two issues that were giving similar behaviour on Mac: https://bugzilla.mozilla.org/show_bug.cgi?id=928806 and https://bugzilla.mozilla.org/show_bug.cgi?id=928797

*** Bug 947287 has been marked as a duplicate of this bug. ***