Perf/parallel partial movie encoding - #4899
Conversation
… files concurrently with rendering Add a max_inflight_encoders config option (default 1) bounding how many partial movie files may be encoding while the scene continues rendering. The default keeps the existing serial behavior: each animation's file is fully encoded and closed before the next animation starts. Values > 1 overlap encoding with rendering; 4 is a good value on typical hardware and cuts wall-clock time on an **encode-heavy** 30-animation benchmark from 9.3 s to 4.1 s (-56 %). Output is byte-identical in both modes for mp4, mov, transparent mov, gif, and PNG; webm bytes differ run-to-run either way because the Matroska muxer generates random TrackUIDs. Each partial movie file is now encoded by a self-contained _PartialMovieEncodeJob owning its container, stream, bounded frame queue, and worker thread. - Closing a stream seals the job and defers its join - Jobs are joined oldest-first once the cap is reached, when a cache lookup or a new stream targets the same path, and all are drained at scene end and on the interactive rerun path (which previously replaced the file writer while jobs could still be writing). - Worker failures are captured first-exception-wins across the encode, flush, and close stages and re-raised at join - A failed job deletes its truncated partial file so a later run cannot cache-hit it. - The former SceneFileWriter internals (listen_and_write, encode_and_write_frame, queue, writer_thread, video_container, video_stream) are removed as part of this restructuring (as equivalents now live in the _PartialMovieEncodeJob class). Added tests covering failure propagation and its precedence, removal of failed partial files, the encoder cap and FIFO join order, same-path guards, cache behavior across renders, success logging, and encoder-thread cleanup.
![github-advanced-security[bot]](https://avatars.githubusercontent.com/in/57789?s=60&v=4){kind=small}
|
|
||
| try: | ||
| self._encode_and_write_frame(frame_data, num_frames) | ||
| except BaseException as exception: |
There was a problem hiding this comment.
deferred-exception pattern: captured and re-raised on join(). Narrowing to Exception instead of BaseException risks queue deadlock and silent truncated-file success. The CPython standard library does this in the _WorkItem.run function (/usr/lib/python3.12/concurrent/futures/thread.py)
| try: | ||
| for packet in self.stream.encode(): | ||
| self.container.mux(packet) | ||
| except BaseException as exception: |
There was a problem hiding this comment.
deferred-exception pattern: captured and re-raised on join(). Narrowing to Exception instead of BaseException risks queue deadlock and silent truncated-file success. The CPython standard library does this in the _WorkItem.run function (/usr/lib/python3.12/concurrent/futures/thread.py)
| finally: | ||
| try: | ||
| self.container.close() | ||
| except BaseException as exception: |
There was a problem hiding this comment.
deferred-exception pattern: captured and re-raised on join(). Narrowing to Exception instead of BaseException risks queue deadlock and silent truncated-file success. The CPython standard library does this in the _WorkItem.run function (/usr/lib/python3.12/concurrent/futures/thread.py)
| for job in list(self._inflight_encode_jobs): | ||
| try: | ||
| self._join_job(job) | ||
| except BaseException as exception: |
There was a problem hiding this comment.
deferred-exception pattern: captured and re-raised on join(). Narrowing to Exception instead of BaseException risks queue deadlock and silent truncated-file success. The CPython standard library does this in the _WorkItem.run function (/usr/lib/python3.12/concurrent/futures/thread.py)
|
Note, the python 3.13 builds are failing precisely because of the error identified in this other PR I submitted a couple of days ago): #4896 The basics of the issue are:
I validated this by running a python 3.13, with the commit prior to this PR, I ran the two runs for the test, and the second render, instead of cache hitting all 32 plays, it re-renders 15 of them, which triggers the assert in this new test. The diff is attached. The particular scene I ran for both runs is this Would you prefer I merge that PR into this one? or should this one wait on the other PR to complete? (The .txt file wouldn't upload, so instead the following image is an extract from the diff file [taken from Notepad++]) |
2 participants
Overview: What does this pull request change
Adds a new ability to parallelise the encoding after the render pass to speed up the encoding output of the scene, with a new max_inflight_encoders config option (defaulting to 1) bounding how many partial movie files may be encoding while the scene continues rendering.
The default value of 1 keeps the existing serial behaviour: each animation's file is fully encoded and closed before the next animation starts. Values greater than 1 overlap encoding with rendering. In testing for the performance testing a value of 4 was used.
Output is byte-identical in both modes for mp4, mov, transparent mov, gif, and PNG. Output for webm output has bytes differing run-to-run either way because the Matroska muxer generates random TrackUIDs.
Each partial movie file is now encoded by a self-contained _PartialMovieEncodeJob owning its container, stream, bounded frame queue, and worker thread.
Added tests covering failure propagation and its precedence, removal of failed partial files, the encoder cap and FIFO join order, same-path guards, cache behavior across renders, success logging, and encoder-thread cleanup.
Added the new config flag to the list of all config options in docs/source/guides/configuration.rst (but noted that some other new flags [format, media_embed, save_sections, seed, disable_caching_warning, and zero_pad] are missing from this manual list)
Motivation and Explanation: Why and how do your changes improve the library?
In the library as it stands, the default serial approach where rendering and encoding are performed serially, leaves performance on the table for those who have the hardware to enable parallelism.
During testing the biggest speed improvement is in encoder heavy scenes, where caching is not used. Using an encode-heavy 30-animation benchmark the wall-clock time was reduced from 9.3 s to 4.1 s (-56 %).
In a real world render heavy video generation, consisting of 8 scenes, some with many mobjects, from an as yet unreleased personal project, rendered serially, the wall clock improvement was more modest (~8% - 11% reduction depending on whether or not cache was utilised).
The full video render performance results are:
--disable_caching--disable_caching- cached: ~41.5 mins
- uncached: ~ 24.5 mins
- ~3.5 min saved per full-video pass when caching was on
- ~2.5 mins saved when caching was off
For the encoder heavy performance benchtest this class was used:
Links to added or changed documentation pages
Expected:
Gen Index
Configuration
ManimConfig
SceneFileWriter
manim._config.utils source
manim.scene.scene source
manim.scene.scene_file_writer source
Unexpected (likely due to commits to main after 0.20.1):
Camera
Images
Further Information and Comments
Reviewer Checklist