Implement an FfmpegVideoRenderer

[rabbitknight]

Merge actual implementation in google/ExoPlayer#7132.

[rabbitknight]

Is there any progress? @microkatz

[FongMi]

@rabbitknight Could you provide me with aar for testing? I don’t have a compilation environment.

[rabbitknight]

@rabbitknight Could you provide me with aar for testing? I don’t have a compilation environment.

Use the provided link.

https://drive.google.com/file/d/1gIZYH6dR780BfPqll1w1VAMaaSkcLqRM/view?usp=drive_link

[FongMi]

@rabbitknight thx

[FongMi]

There is no problem in my test, unlike nextlib which needs to reset the surface.

[rabbitknight]

There is no problem in my test, unlike nextlib which needs to reset the surface.

Thanks for your feedback. There may be slight differences, but overall the code is similar. I will keep watching until successfully merged. ：P

[FongMi]

I finally found the bug
When I randomly seek 3~5 times
The screen is freezes
Use EXTENSION_RENDERER_MODE_PREFER for force ffmpeg decode

video_2024-08-17_00-07-21.mp4

[rabbitknight]

I finally found the bug

When I randomly seek 3~5 times

The screen is freezes

Use EXTENSION_RENDERER_MODE_PREFER for force ffmpeg decode

video_2024-08-17_00-07-21.mp4

Thanks～ let me see what happened. I'll feedback on Monday.

[rabbitknight]

I finally found the bug When I randomly seek 3~5 times The screen is freezes Use EXTENSION_RENDERER_MODE_PREFER for force ffmpeg decode

video_2024-08-17_00-07-21.mp4

@FongMi
I reproduced the issue.
Reason is the BlockingQueue buffers are not cleared when the decoder is flushed. (Causing the decode pipeline to block!!)
So I directly modified the thread used for format conversion and the BlockingQueue.h.
You can get the Lasted aar from : https://drive.google.com/drive/folders/1pwOkFUzMwpfm492IFrUebkn7MUGBA2My?usp=drive_link

[FongMi]

Thx, I will test it.

[FongMi]

@rabbitknight Thers is no problem now, thanks.

[FongMi]

@rabbitknight Can you support mepg and mpeg2 like nextlib

[rabbitknight]

@rabbitknight Can you support mepg and mpeg2 like nextlib

Here is test url http://222.223.250.251:9242/stream/channelid/1587704203?ticket=850d4cf1effb29480cffc7d3022f6a9f7ae59641&profile=pass

@FongMi you can get the latest aar from: https://drive.google.com/drive/folders/1pwOkFUzMwpfm492IFrUebkn7MUGBA2My?usp=drive_link
the commit is :57346bb

[FongMi]

@rabbitknight Can you support mepg and mpeg2 like nextlib

@FongMi you can get the latest aar from: https://drive.google.com/drive/folders/1pwOkFUzMwpfm492IFrUebkn7MUGBA2My?usp=drive_link the commit is :57346bb

Is your audio codecs as good as nextlib?

[FongMi]

@rabbitknight Can you support mepg and mpeg2 like nextlib
@FongMi you can get the latest aar from: https://drive.google.com/drive/folders/1pwOkFUzMwpfm492IFrUebkn7MUGBA2My?usp=drive_link the commit is :57346bb

It worked fine for testing, thank you.

[rabbitknight]

Is your audio codecs as good as nextlib?

@FongMi I will compare the
difference between nexlib tomorrow. Reduce the gap.

[rabbitknight]

Is your audio codecs as good as nextlib?

@FongMi I will compare the difference between nexlib tomorrow. Reduce the gap.

@FongMi, I conducted a comparison:

1. The audio codecs are the same.
2. However, it appears that nextlibs uses an extended VPX decoder.
3. In my test case, FFmpeg can directly decode VP8 and VP9.

[FongMi]

@rabbitknight Thanks, I understand.

[bennettpeter]

I have tried this enhancement and it works well for mpeg2, which many mobile phones do not support. I will be using it in my applications. Please can you merge it into the next version?

[bennettpeter]

I found some issues

• A release build with minifyEnabled true crashes on playback with a message that a variable called I is not defined in a certain class. It seems that minify has renamed variables in the calling code but not in the called classes. If I set minifyEnabled false the release build runs but the apk file is 4 MB larger than with minify.
• When playing ffmpeg decoded video on a google TV device or Amazon fire stick, frames are dropped, resulting in jerky video. Playing on a phone is fine.

[rabbitknight]

I found some issues

• A release build with minifyEnabled true crashes on playback with a message that a variable called I is not defined in a certain class. It seems that minify has renamed variables in the calling code but not in the called classes. If I set minifyEnabled false the release build runs but the apk file is 4 MB larger than with minify.
• When playing ffmpeg decoded video on a google TV device or Amazon fire stick, frames are dropped, resulting in jerky video. Playing on a phone is fine.

Thank you for your feedback.

1. Regarding code minify. I've Fix it in the lastest commit, and also made some MimeTypes compatibility adjustments For AV1 or mp43。
2. Performance on low-end devices: I've also encountered frame dropping issues with FFmpeg software decoding. Since soft decoding is heavily CPU-dependent, I recommend:
   • Monitoring current CPU usage, Using MediaCodec decoder as the Primary option.
   • Modify frame-skipping strategies like:

codecContext->skip_frame = AVDISCARD_BIDIR;  // Skip B-frames

[bennettpeter]

Another mime type that I added, and tested it works:

  case MimeTypes.VIDEO_VC1:
    return "vc1";

[rabbitknight]

Another mime type that I added, and tested it works:

  case MimeTypes.VIDEO_VC1:

    return "vc1";

Awesome!I'm going to test it out!

[bennettpeter]

One more issue I found - anamorphic video is not reproduced properly. This is because the VideoSize returned from player.getVideoSize() always has pixelWidthHeightRatio as 1.0. On anamorphic video this should be values such as 1.212121. A call to player.getVideoFormat() does return the correct value in format.pixelWidthHeightRatio. However the format returns only the initial value, when format changes midstream, the listener gets onVideoSizeChanged with a new value for VideoSize but at that time calling player.getVideoFormat() still returns the old size and pixelWidthHeightRatio.

I worked around this by modifying the onVideoSizeChanged handler to get the pixelWidthHeightRatio from getVideoFormat, but that is not perfect as it will be wrong if the format changes. However format changes are uncommon and until recently, exoplayer did not handle them at all.

Let me know if you want a sample anamorphic video with format change for testing.

[bennettpeter]

Sorry - vc1 was in your list anyway. I had added it previously.

[rabbitknight]

One more issue I found - anamorphic video is not reproduced properly. This is because the VideoSize returned from player.getVideoSize() always has pixelWidthHeightRatio as 1.0. On anamorphic video this should be values such as 1.212121. A call to player.getVideoFormat() does return the correct value in format.pixelWidthHeightRatio. However the format returns only the initial value, when format changes midstream, the listener gets onVideoSizeChanged with a new value for VideoSize but at that time calling player.getVideoFormat() still returns the old size and pixelWidthHeightRatio.

I worked around this by modifying the onVideoSizeChanged handler to get the pixelWidthHeightRatio from getVideoFormat, but that is not perfect as it will be wrong if the format changes. However format changes are uncommon and until recently, exoplayer did not handle them at all.

Let me know if you want a sample anamorphic video with format change for testing.

My test cases not contains this format.
Please provide me with the sample video,thanks!
if privacy's a concern, feel free to email it to mottchenz@gmail.com(mailto:mottchenz@gmail.com)

[bennettpeter]

Here is a sample video. pixelWidthHeightRatio is 1.2121212 at the start with video 528x480 that should be shown at 640x480. After 10 seconds it changes to pixelWidthHeightRatio of 1.0 with resolution 1920x1080.

https://drive.google.com/file/d/1E0gj9B6Kr8-NHUuRVfFDBgKQSotdpD5s/view?usp=sharing

[rabbitknight]

One more issue I found - anamorphic video is not reproduced properly. This is because the VideoSize returned from player.getVideoSize() always has pixelWidthHeightRatio as 1.0. On anamorphic video this should be values such as 1.212121. A call to player.getVideoFormat() does return the correct value in format.pixelWidthHeightRatio. However the format returns only the initial value, when format changes midstream, the listener gets onVideoSizeChanged with a new value for VideoSize but at that time calling player.getVideoFormat() still returns the old size and pixelWidthHeightRatio.

I worked around this by modifying the onVideoSizeChanged handler to get the pixelWidthHeightRatio from getVideoFormat, but that is not perfect as it will be wrong if the format changes. However format changes are uncommon and until recently, exoplayer did not handle them at all.

Let me know if you want a sample anamorphic video with format change for testing.

Maybe The BaseDecoder should compat this case.
FfmpegVideoDecoder only should to Fill the given surface.
This anamorphic video also cannot display will in MediaCodecRenderer or My macOS System player. It's probably an acceptable trade-off – any more work would tank the decoding throughput.

[PratikLahagora]

Hello,
I’m fairly new to video streaming and working with FFmpeg. I’d like to understand the main differences between Nextlib and this FFmpeg library. I’m planning to integrate FFmpeg into my project, but I’m a bit unsure which would be the better choice. From what I’ve seen, Nextlib appears to support more video formats compared to this library. Could you please clarify the differences and help me decide which would be more suitable?

Thank you!

[rabbitknight]

Hello, I’m fairly new to video streaming and working with FFmpeg. I’d like to understand the main differences between Nextlib and this FFmpeg library. I’m planning to integrate FFmpeg into my project, but I’m a bit unsure which would be the better choice. From what I’ve seen, Nextlib appears to support more video formats compared to this library. Could you please clarify the differences and help me decide which would be more suitable?

Thank you!

Nextlib and this PR share the same core objective: to enable FFmpeg extension support for Media3 (ExoPlayer). Their code implementations are nearly identical (likely originating from the same source) with minor subsequent adjustments.

Key differences:

Mediainfo Feature: Nextlib includes an additional mediainfo component, though its purpose remains unclear to me. Notably, its code quality appears suboptimal.
Decoder Support: Nextlib offers no additional decoders compared to this PR. Both are constrained by FFmpeg compilation flags and definitions in FFmpegLibrary.java. To my knowledge, this PR already supports advanced codecs like Apple ProRes Video and AV1.
Rotation Handling: Nextlib currently lacks support for video rotation angle processing.
The solution from this PR has been operating stably in our production environment with no anomalies reported. In summary:

For production use: I recommend this PR implementation.
For educational purposes: Studying this PR's codebase would be more valuable.

[peerless2012]

@rabbitknight Since this pr seems not be merged for a while, is it possible to compile and publish your version (Which always follow this repo) in maven central?