Bug 57831

Summary: androidmedia: add support for video encoding
Product: GStreamer SDK Reporter: Nicola <lists>
Component: GeneralAssignee: bugs
Status: NEW --- QA Contact:
Severity: enhancement    
Priority: high CC: admin, diego.buffa, dovstam, fykcee1
Version: unspecified   
Hardware: All   
OS: All   
Whiteboard:
i915 platform: i915 features:
Attachments: [unworkable]A primary patch adding video encoding support
A primary patch(against sdk 2012.11) adding video encoding support
Patch(against sdk 2012.11, Version 2) adding video encoding support
Patch(against sdk 2012.11, Version 3) adding video encoding support

Description Nicola 2012-12-03 08:28:15 UTC
would be really useful :-)

video from camera can already be captured and sent to gstreamer via appsrc, however actually the only viable encoding is jpeg and even jpeg encoding require a quite high cpu usage, android has hw encoding to h264 and mpeg4 ...
Comment 1 Sebastian Dröge (slomo) 2012-12-03 11:58:54 UTC
Yes, shouldn't be that hard to add either. If you want to work on that I can provide some support and help on IRC :)
Comment 2 cee1 2013-03-14 11:08:03 UTC
Any way to wrap around android native encoder? like currently wrap around  MediaCodec API for decoders...

Also what about gst-omx(http://cgit.freedesktop.org/gstreamer/gst-omx)? It was used to provide hardware accel decoders(though, OMX can also provide hardware accel encoders) under android in the early days, but didn't enter the SDK.(in favor of MediaCodecs ?)
Comment 3 Sebastian Dröge (slomo) 2013-03-14 11:12:38 UTC
(In reply to comment #2)
> Any way to wrap around android native encoder? like currently wrap around 
> MediaCodec API for decoders...

Yes, just do the same ;) Basically need to copy gstamcvideodec.c to gstamcvideoenc.c, change base class to GstVideoEncoder and switch everything to compressed input and uncompressed output. Shouldn't take more than a day of work.

> Also what about gst-omx(http://cgit.freedesktop.org/gstreamer/gst-omx)? It
> was used to provide hardware accel decoders(though, OMX can also provide
> hardware accel encoders) under android in the early days, but didn't enter
> the SDK.(in favor of MediaCodecs ?)

OpenMAX IL is not public API on Android.
Comment 4 cee1 2013-04-03 09:02:55 UTC
Created attachment 77369 [details] [review]
[unworkable]A primary patch adding video encoding support

Note: I haven't tested it as the device at hand has no video encoder registered in gst. I'm also not familiar with the detailed logic, so it's probably broken.

The patch is here for review and further debug purposes.
Comment 5 stic 2013-04-19 13:10:38 UTC
I tried to use it on a pipeline, but was I was only able to produce uncompressed raw video data with it.
I think the main problem with it is that it does not seem to use or to find a codec ....
But I'am not sure of the way how to use it, is there any special parameters to provide to use specific codec ?
Dear Sebastian could you provide us some support on this ?
I would be really happy to help and make it work.

Thank you,
best regards,
Nicolas
Comment 6 cee1 2013-05-04 09:07:01 UTC
I tried another 4.1.1 table, unfortunately I encountered "No supported color formats for video codec" again. Following the code, I find the CodecCapabilities object returned by "getCapabilitiesForType" has zero colorFormats.

It reports a series of "OMX.rk..." codec, seems be wrappen around HW video func of the underlying rockchip.

I tried to work around this by specifying a color format[1], tried each color format with viddec, none of them worked.

So, the MediaCodec APIs are not properly implemented widely? At least for my case.

BTW, what's the difference between amcviddec-omxrkvideodecoderXXX and amcviddec-XXXdecoder?

--------
[1] Attempt to work around(but failed) 0 colorFormats by specifying a color format
--- a/sys/androidmedia/gstamc.c
+++ b/sys/androidmedia/gstamc.c
@@ -1848,9 +1848,23 @@ scan_codecs (GstPlugin * plugin)
 
       if (g_str_has_prefix (gst_codec_type->mime, "video/")) {
         if (!n_elems) {
+#if 1
+          GST_WARNING ("No supported color formats for video codec, assume COLOR_TI_FormatYUV420PackedSemiPlana
+          n_elems = 1;
+          gst_codec_type->n_color_formats = 1;
+          gst_codec_type->color_formats = g_new0 (gint, 1);
+
+          gst_codec_type->color_formats[0] = COLOR_TI_FormatYUV420PackedSemiPlanar;
+          /*
+           * COLOR_FormatYUV420Planar 
+           * COLOR_TI_FormatYUV420PackedSemiPlanar(COLOR_TI_FormatYUV420PackedSemiPlanarInterlaced)
+           * COLOR_QCOM_FormatYUV420SemiPlanar(COLOR_FormatYUV420SemiPlanar)
+           */
+#else
           GST_ERROR ("No supported color formats for video codec");
           valid_codec = FALSE;
           goto next_supported_type;
+#endif
         }
 
         if (!ignore_unknown_color_formats
Comment 7 Sebastian Dröge (slomo) 2013-05-08 07:11:52 UTC
(In reply to comment #6)
> I tried another 4.1.1 table, unfortunately I encountered "No supported color
> formats for video codec" again. Following the code, I find the
> CodecCapabilities object returned by "getCapabilitiesForType" has zero
> colorFormats.
> 
> It reports a series of "OMX.rk..." codec, seems be wrappen around HW video
> func of the underlying rockchip.
> 
> I tried to work around this by specifying a color format[1], tried each
> color format with viddec, none of them worked.
> 
> So, the MediaCodec APIs are not properly implemented widely? At least for my
> case.

Looks like it, to be sure you could create a small Java application for Android and check if it returns 0 there too. This is clearly a bug in the implementation of the MediaCodec API of your device.

> BTW, what's the difference between amcviddec-omxrkvideodecoderXXX and
> amcviddec-XXXdecoder?

They're just wrapper plugins for different codecs of the system. Take a look at /etc/media-codecs.xml for the list of all of them.
Comment 8 stic 2013-05-09 21:21:39 UTC
(In reply to comment #6)
> I tried another 4.1.1 table, unfortunately I encountered "No supported color
> formats for video codec" again. Following the code, I find the
> CodecCapabilities object returned by "getCapabilitiesForType" has zero
> colorFormats.
> 
> It reports a series of "OMX.rk..." codec, seems be wrappen around HW video
> func of the underlying rockchip.
> 
> I tried to work around this by specifying a color format[1], tried each
> color format with viddec, none of them worked.
> 
> So, the MediaCodec APIs are not properly implemented widely? At least for my
> case.
> 

Yes it is the same for me, it seems that every video codec throws a warning saying "Unknown color format 0x00000000".
Comment 9 cee1 2013-05-14 07:30:02 UTC
(In reply to comment #8)
> Yes it is the same for me, it seems that every video codec throws a warning
> saying "Unknown color format 0x00000000".
The returned color format 0x00000000 seems invalid. Here is a list of valid color format codes: http://developer.android.com/reference/android/media/MediaCodecInfo.CodecCapabilities.html#COLOR_Format12bitRGB444

I tried another android 4.1 table, amc found a series of HW-accel codecs with color format code "25"(COLOR_FormatYCbYCr). I tried to map COLOR_FormatYCbYCr to GST_VIDEO_FORMAT_IYU1, it worked for amcvideodec-omxk3videodecoderavc(though the color is not correct), but failed for amcvideodec-omxk3videodecodervp8:
D/GStreamer+amc(15616): 0:00:06.722159000 0x5c6aa600 gstamc.c:118:gst_amc_attach_current_thread Attaching thread 0x5c6aa600
E/OmxVp8DecComponent( 1747): setParameter line 242 ret = 80001005
W/ACodec  (15616): [OMX.k3.video.decoder.vp8] Failed to set standard component role 'video_decoder.vpx'.
E/ACodec  (15616): [OMX.k3.video.decoder.vp8] configureCodec returning error -2147483648
E/MediaCodec(15616): Codec reported an error. (omx error 0x80001001, internalError -2147483648)
E/GStreamer+amc(15616): 0:00:06.739552000 0x5c6aa600 gstamc.c:302:gst_amc_codec_configure Failed to call Java method

BTW, I also tried amcvideodec-omxgooglevpxdecoder and amcviddec-omxgoogleh264decoder, they failed too:
E/OMXNodeInstance(13696): OMX_GetExtensionIndex failed
E/SoftVPX (13696): on2 decoder failed to decode frame.
D/GStreamer+amc(13696): 0:00:02.042483000 0x5c79d6c0 gstamc.c:118:gst_amc_attach_current_thread Attaching thread 0x5c79d6c0
E/ACodec  (13696): [OMX.google.vpx.decoder] ERROR(0x80001001)
E/MediaCodec(13696): Codec reported an error. (omx error 0x80001001, internalError -2147483648)
----
E/OMXNodeInstance(22714): OMX_GetExtensionIndex failed
E/SoftAVC (22714): Decoder failed: -2
E/ACodec  (22714): [OMX.google.h264.decoder] ERROR(0x80001001)
E/MediaCodec(22714): Codec reported an error. (omx error 0x80001001, internalError -2147483648)

Some questions:
* In the failure case, I've got a 0x80001001 ERROR, what does that suggest?
* OMX.google.h264.decoder and OMX.google.vpx.decoder are soft decoding implementations, they should work, unless something wrong with the invocation steps, right?
* For amcvideodec-omxk3videodecoderavc, I saw the following warnings, any ideas?
a) gstamcvideodec.c:1026:gst_amc_video_dec_loop:<amcvideodec-omxk3videodecoderavc0> Frame is too late, dropping (deadline 0:00:00.003577385)
b) I/OmxAvcDecComponent( 1747): in thumbnailmode copy yuv out
Comment 10 Sebastian Dröge (slomo) 2013-05-14 08:00:20 UTC
(In reply to comment #9)
> (In reply to comment #8)
> > Yes it is the same for me, it seems that every video codec throws a warning
> > saying "Unknown color format 0x00000000".
> The returned color format 0x00000000 seems invalid. Here is a list of valid
> color format codes:
> http://developer.android.com/reference/android/media/MediaCodecInfo.
> CodecCapabilities.html#COLOR_Format12bitRGB444
> 
> I tried another android 4.1 table, amc found a series of HW-accel codecs
> with color format code "25"(COLOR_FormatYCbYCr). I tried to map
> COLOR_FormatYCbYCr to GST_VIDEO_FORMAT_IYU1

YCbYCr is GST_VIDEO_FORMAT_YUY2

> it worked for
> amcvideodec-omxk3videodecoderavc(though the color is not correct), but
> failed for amcvideodec-omxk3videodecodervp8:
> D/GStreamer+amc(15616): 0:00:06.722159000 0x5c6aa600
> gstamc.c:118:gst_amc_attach_current_thread Attaching thread 0x5c6aa600
> E/OmxVp8DecComponent( 1747): setParameter line 242 ret = 80001005
> W/ACodec  (15616): [OMX.k3.video.decoder.vp8] Failed to set standard
> component role 'video_decoder.vpx'.
> E/ACodec  (15616): [OMX.k3.video.decoder.vp8] configureCodec returning error
> -2147483648
> E/MediaCodec(15616): Codec reported an error. (omx error 0x80001001,
> internalError -2147483648)

That's OMX_ErrorUndefined, useful :)

> E/GStreamer+amc(15616): 0:00:06.739552000 0x5c6aa600
> gstamc.c:302:gst_amc_codec_configure Failed to call Java method
> 
> BTW, I also tried amcvideodec-omxgooglevpxdecoder and
> amcviddec-omxgoogleh264decoder, they failed too:
> E/OMXNodeInstance(13696): OMX_GetExtensionIndex failed
> E/SoftVPX (13696): on2 decoder failed to decode frame.
> D/GStreamer+amc(13696): 0:00:02.042483000 0x5c79d6c0
> gstamc.c:118:gst_amc_attach_current_thread Attaching thread 0x5c79d6c0
> E/ACodec  (13696): [OMX.google.vpx.decoder] ERROR(0x80001001)
> E/MediaCodec(13696): Codec reported an error. (omx error 0x80001001,
> internalError -2147483648)
> ----
> E/OMXNodeInstance(22714): OMX_GetExtensionIndex failed
> E/SoftAVC (22714): Decoder failed: -2
> E/ACodec  (22714): [OMX.google.h264.decoder] ERROR(0x80001001)
> E/MediaCodec(22714): Codec reported an error. (omx error 0x80001001,
> internalError -2147483648)
> 
> Some questions:
> * In the failure case, I've got a 0x80001001 ERROR, what does that suggest?

That someone was too lazy to add a useful error value there and do proper error reporting.

> * OMX.google.h264.decoder and OMX.google.vpx.decoder are soft decoding
> implementations, they should work, unless something wrong with the
> invocation steps, right?

Yes

> * For amcvideodec-omxk3videodecoderavc, I saw the following warnings, any
> ideas?
> a)
> gstamcvideodec.c:1026:gst_amc_video_dec_loop:<amcvideodec-
> omxk3videodecoderavc0> Frame is too late, dropping (deadline
> 0:00:00.003577385)
> b) I/OmxAvcDecComponent( 1747): in thumbnailmode copy yuv out

a) happens if a frame is decoded too late and would be displayed too late

b) should not be a problem
Comment 11 cee1 2013-06-15 05:35:47 UTC
Created attachment 80833 [details] [review]
A primary patch(against sdk 2012.11) adding video encoding support

This patch works for my Huawei MediaPad 10 FHD.
The testing pipeline:
"videotestsrc ! capsfilter caps=\"video/x-raw-yuv,format=(fourcc)NV12,framerate=(fraction)8/1\" ! amcvidenc-omxk3videoencoderavc bitrate=1024 i-frame-int=5 ! rtph264pay ! queue ! udpsink host=192.168.1.100 port=3057"

The underlying hardware reports three supported color formats:
* COLOR_FormatYUV420SemiPlanar (GST_VIDEO_FORMAT_NV12)
* COLOR_FormatYCbYCr (GST_VIDEO_FORMAT_YUY2)
Tested with "videotestsrc ! video/x-raw-yuv,format=(fourcc)YUY2...", and got a "gst_amc_video_enc_handle_frame: error: Failed to queue input buffer"
* COLOR_Format32bitARGB8888 (GST_VIDEO_FORMAT_ARGB)
Tested with "videotestsrc ! video/x-raw-rgb,bpp=32,depth=32,endianness=4321,alpha_mask=-16777216,red_mask=16711680,green_mask=65280,blue_mask=255,...", and got the same error.

Any idea? It seems there is no stride/crop concept for amc encoder...

Please Review:
1. _find_nearest_frame()
* Is it necessary for encoder? 
* The commented out "gst_video_enoder_drop_frame ...", there is no drop_frame for encoder, any replacement?

2. gst_amc_video_enc_handle_output_frame()
* What do the different branches take care of?

3. gst_amc_video_enc_drain()
* Copy and modify from the "dec" version, need more modification?

= TODO =
No "profile" setting for encoder? At least according to the document of MediaFormat.

----
For references:
* http://developer.android.com/reference/android/media/MediaCodec.html
* http://developer.android.com/reference/android/media/MediaCodecInfo.CodecCapabilities.html
* http://developer.android.com/reference/android/media/MediaFormat.html
* Example1: https://gist.github.com/wobbals/3990442
* Example2: https://android.googlesource.com/platform/cts/+/6289d680cb75fa5a985464b9f362d4a2a007a7bf%5E!/#F0
Comment 12 cee1 2013-06-16 09:40:45 UTC
Created attachment 80894 [details] [review]
Patch(against sdk 2012.11, Version 2) adding video encoding support

Changes since v1:
* gst_amc_video_enc_set_format()
--> Negotiating the src caps (Right now there's no src pad caps negotiation for amcvideo**dec**?)
--> Call gst_amc_video_enc_set_src_caps if !encoder->started (For encoder, it seems will not return INFO_OUTPUT_FORMAT_CHANGED the first time)

Fixups for raw buffer filling, etc.
Comment 13 cee1 2013-06-27 09:44:34 UTC
Created attachment 81539 [details] [review]
Patch(against sdk 2012.11, Version 3) adding video encoding support

Version 3 of the patch against sdk 2012.11, changes since V2:
1. Remove mappings of COLOR_FormatYCbYCr & COLOR_Format32bitARGB8888
2. Set src pad's caps in gst_amc_video_enc_loop:
* On the start of codec, self->first_set_format will be assigned with GstAmcFormat that is passed in gst_amc_codec_configure
* In gst_amc_video_enc_loop, it will set src pad's GstVideoCodecState if self->first_set_format is set or INFO_OUTPUT_FORMAT_CHANGED; self->first_set_format will then be cleared.
3. Convert raw data to respect to "stride > width" case, that's processed in gst_amc_video_enc_fill_buffer.
4. More fileds are set in GstAmcFormat while configring the codec:
"stride": width be rounded up to a multiple of 4 (the same as gst-omx's "safe" logic.)
"slice-height": equal to height
"profile & level": Not set currently. If set "profile" to any value of AVCProfile*, codec configuration failed.
5. Handle incorrectly reported color, that's in color_format_to_video_format_hack & video_format_to_color_format_hack
6. Bug fix & code cleanup

Use of freedesktop.org services, including Bugzilla, is subject to our Code of Conduct. How we collect and use information is described in our Privacy Policy.