Skip to content

Commit 6bb00c1

Browse files
bcw222bcw222
committed
Merge branch 'main' of https://github.com/bcw222/torchcodec 
2 parents e073228 + 841cc83 commit 6bb00c1

69 files changed

Lines changed: 2374 additions & 874 deletions

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.

‎.github/workflows/cpp_tests.yaml‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -22,7 +22,7 @@ jobs:
2222
ffmpeg-version-for-tests: ['4.4.2', '5.1.2', '6.1.1', '7.0.1']
2323
steps:
2424
- name: Check out repo
25-
uses: actions/checkout@v3
25+
uses: actions/checkout@v6
2626
- name: Setup conda env
2727
uses: conda-incubator/setup-miniconda@v3
2828
with:

‎.github/workflows/lint.yaml‎

Lines changed: 6 additions & 6 deletions
Original file line numberDiff line numberDiff line change
@@ -22,12 +22,12 @@ jobs:
2222
python-version: ['3.12']
2323
steps:
2424
- name: Check out repo
25-
uses: actions/checkout@v3
25+
uses: actions/checkout@v6
2626
- name: Setup conda env
27-
uses: conda-incubator/setup-miniconda@v2
27+
uses: conda-incubator/setup-miniconda@v3
2828
with:
2929
auto-update-conda: true
30-
miniconda-version: "latest"
30+
miniforge-version: latest
3131
activate-environment: test
3232
python-version: ${{ matrix.python-version }}
3333
- name: Update pip
@@ -50,12 +50,12 @@ jobs:
5050
python-version: ['3.12']
5151
steps:
5252
- name: Check out repo
53-
uses: actions/checkout@v3
53+
uses: actions/checkout@v6
5454
- name: Setup conda env
55-
uses: conda-incubator/setup-miniconda@v2
55+
uses: conda-incubator/setup-miniconda@v3
5656
with:
5757
auto-update-conda: true
58-
miniconda-version: "latest"
58+
miniforge-version: latest
5959
activate-environment: test
6060
python-version: ${{ matrix.python-version }}
6161
- name: Update pip

‎.github/workflows/linux_cuda_aarch64_wheel.yaml‎

Lines changed: 9 additions & 5 deletions
Original file line numberDiff line numberDiff line change
@@ -59,19 +59,23 @@ jobs:
5959

6060
install-and-test:
6161
runs-on: linux.arm64.2xlarge
62-
container:
63-
image: pytorch/manylinuxaarch64-builder:cuda12.6
64-
env:
65-
cuda_version_without_periods: "126"
6662
strategy:
6763
fail-fast: false
6864
matrix:
6965
python-version: ['3.10']
66+
cuda-version: ['12.6']
7067
ffmpeg-version-for-tests: ['4.4.2', '5.1.2', '6.1.1', '7.0.1', '8.0']
68+
container:
69+
image: "pytorch/manylinuxaarch64-builder:cuda${{ matrix.cuda-version }}"
7170
needs: build
7271
steps:
72+
- name: Setup env vars
73+
run: |
74+
cuda_version_without_periods=$(echo "${{ matrix.cuda-version }}" | sed 's/\.//g')
75+
echo cuda_version_without_periods=${cuda_version_without_periods} >> $GITHUB_ENV
76+
7377
- name: Check out repo
74-
uses: actions/checkout@v3
78+
uses: actions/checkout@v6
7579

7680
- name: Remove src/ folder
7781
run: bash packaging/remove_src.sh

‎.github/workflows/linux_cuda_wheel.yaml‎

Lines changed: 80 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -86,7 +86,7 @@ jobs:
8686
echo python_version_without_periods=${python_version_without_periods} >> $GITHUB_ENV
8787
8888
- name: Check out repo
89-
uses: actions/checkout@v3
89+
uses: actions/checkout@v6
9090

9191
- name: Remove src/ folder
9292
run: bash packaging/remove_src.sh
@@ -151,7 +151,7 @@ jobs:
151151
echo python_version_without_periods=${python_version_without_periods} >> $GITHUB_ENV
152152
153153
- name: Check out repo
154-
uses: actions/checkout@v3
154+
uses: actions/checkout@v6
155155

156156
- name: Remove src/ folder
157157
run: bash packaging/remove_src.sh
@@ -199,3 +199,81 @@ jobs:
199199
with:
200200
name: Built-Docs
201201
path: docs/build/html/
202+
203+
doc-preview:
204+
runs-on: [self-hosted, linux.2xlarge]
205+
needs: build-docs
206+
if: github.repository == 'meta-pytorch/torchcodec' && github.event_name == 'pull_request'
207+
steps:
208+
- uses: actions/download-artifact@v4
209+
with:
210+
name: Built-Docs
211+
path: docs
212+
213+
# Update HTML to add the no-index tag so that search engines do not index these ephemeral docs
214+
- name: Add no-index tag
215+
run: |
216+
find docs -name "*.html" -print0 | xargs -0 sed -i '/<head>/a \ \ <meta name="robots" content="noindex">';
217+
218+
- name: Upload docs preview
219+
uses: seemethere/upload-artifact-s3@v5
220+
with:
221+
retention-days: 14
222+
s3-bucket: doc-previews
223+
if-no-files-found: error
224+
path: docs
225+
s3-prefix: meta-pytorch/torchcodec/${{ github.event.pull_request.number }}
226+
227+
upload-docs:
228+
# This job uploads built docs:
229+
# - to the `main` folder of the gh-pages branch (https://meta-pytorch.org/torchcodec/main) on every commit to the `main` branch
230+
# - to the (e.g.) `0.10` folder in the gh-pages branch whenever a corresponding tag is pushed, like `v0.10.0` (https://meta-pytorch.org/torchcodec/0.10).
231+
232+
needs: build-docs
233+
if: github.repository == 'meta-pytorch/torchcodec' && github.event_name == 'push' &&
234+
((github.ref_type == 'branch' && github.ref_name == 'main') || github.ref_type == 'tag')
235+
runs-on: ubuntu-latest
236+
steps:
237+
- name: Check out gh-pages branch
238+
uses: actions/checkout@v6
239+
with:
240+
ref: gh-pages
241+
242+
- uses: actions/download-artifact@v4
243+
with:
244+
name: Built-Docs
245+
path: docs-artifact/
246+
247+
- name: Update docs and push
248+
run: |
249+
set -euo pipefail
250+
251+
REF_TYPE=${{ github.ref_type }}
252+
REF_NAME=${{ github.ref_name }}
253+
254+
if [[ "${REF_TYPE}" == branch ]]; then
255+
TARGET_FOLDER="${REF_NAME}"
256+
elif [[ "${REF_TYPE}" == tag ]]; then
257+
case "${REF_NAME}" in
258+
*-rc*)
259+
echo "Aborting upload since this is an RC tag: ${REF_NAME}"
260+
exit 0
261+
;;
262+
*)
263+
# Strip the leading "v" as well as the trailing patch version. For example:
264+
# 'v0.10.2' -> '0.10'
265+
TARGET_FOLDER=$(echo "${REF_NAME}" | sed 's/v\([0-9]\+\)\.\([0-9]\+\)\.[0-9]\+/\1.\2/')
266+
;;
267+
esac
268+
fi
269+
echo "Target Folder: ${TARGET_FOLDER}"
270+
271+
mkdir -p "${TARGET_FOLDER}"
272+
rm -rf "${TARGET_FOLDER}"/*
273+
cp -r docs-artifact/* "${TARGET_FOLDER}/"
274+
git add "${TARGET_FOLDER}"
275+
276+
git config user.name 'pytorchbot'
277+
git config user.email 'soumith+bot@pytorch.org'
278+
git commit -m "auto-generating sphinx docs for ${TARGET_FOLDER}" || echo "No changes to commit"
279+
git push

‎.github/workflows/linux_wheel.yaml‎

Lines changed: 56 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -67,7 +67,7 @@ jobs:
6767
needs: build
6868
steps:
6969
- name: Check out repo
70-
uses: actions/checkout@v3
70+
uses: actions/checkout@v6
7171

7272
- name: Remove src/ folder
7373
run: bash packaging/remove_src.sh
@@ -107,3 +107,58 @@ jobs:
107107
- name: Run Python tests
108108
run: |
109109
pytest --override-ini="addopts=-v" test
110+
111+
install-and-test-third-party-interface:
112+
runs-on: ubuntu-latest
113+
strategy:
114+
fail-fast: false
115+
matrix:
116+
python-version: ['3.10']
117+
ffmpeg-version-for-tests: ['8.0']
118+
needs: build
119+
steps:
120+
- name: Check out repo
121+
uses: actions/checkout@v3
122+
123+
- name: Remove src/ folder
124+
run: bash packaging/remove_src.sh
125+
126+
- name: Setup conda env
127+
uses: conda-incubator/setup-miniconda@v3
128+
with:
129+
auto-update-conda: true
130+
# Using miniforge instead of miniconda ensures that the default
131+
# conda channel is conda-forge instead of main/default. This ensures
132+
# ABI consistency between dependencies:
133+
# https://conda-forge.org/docs/user/transitioning_from_defaults/
134+
miniforge-version: latest
135+
activate-environment: test
136+
python-version: ${{ matrix.python-version }}
137+
138+
- name: Update pip
139+
run: python -m pip install --upgrade pip
140+
141+
- name: Install PyTorch
142+
run: bash packaging/install_pytorch.sh cpu "torch torchvision"
143+
144+
- uses: actions/download-artifact@v4
145+
with:
146+
name: meta-pytorch_torchcodec__${{ matrix.python-version }}_cpu_x86_64
147+
path: dist/
148+
149+
- name: Install torchcodec from the wheel
150+
run: bash packaging/install_torchcodec_wheel.sh
151+
152+
- name: Install ffmpeg, post build
153+
run: bash packaging/install_ffmpeg.sh ${{ matrix.ffmpeg-version-for-tests }}
154+
155+
- name: Install pkg-config
156+
run: |
157+
conda install pkg-config -c conda-forge
158+
159+
- name: Install test dependencies
160+
run: bash packaging/install_test_dependencies.sh
161+
162+
- name: Run Python tests
163+
run: |
164+
pytest --override-ini="addopts=-v" test/third-party-interface

‎.github/workflows/macos_wheel.yaml‎

Lines changed: 6 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -69,7 +69,7 @@ jobs:
6969
needs: build
7070
steps:
7171
- name: Check out torchcodec repo
72-
uses: actions/checkout@v3
72+
uses: actions/checkout@v6
7373

7474
- name: Remove src/ folder
7575
run: bash packaging/remove_src.sh
@@ -78,7 +78,11 @@ jobs:
7878
uses: conda-incubator/setup-miniconda@v3
7979
with:
8080
auto-update-conda: true
81-
miniconda-version: "latest"
81+
# Using miniforge instead of miniconda ensures that the default
82+
# conda channel is conda-forge instead of main/default. This ensures
83+
# ABI consistency between dependencies:
84+
# https://conda-forge.org/docs/user/transitioning_from_defaults/
85+
miniforge-version: latest
8286
activate-environment: test
8387
python-version: ${{ matrix.python-version }}
8488
- name: Update pip

‎.github/workflows/reference_resources.yaml‎

Lines changed: 23 additions & 24 deletions
Original file line numberDiff line numberDiff line change
@@ -53,42 +53,41 @@ jobs:
5353
fail-fast: false
5454
matrix:
5555
python-version: ['3.10']
56-
ffmpeg-version-for-tests: ['4.4.2', '5.1.2', '6.1.1', '7.0.1']
56+
# Traditionally we generate the resources locally on FFmpeg 4 or 6.
57+
# The exact version shouln't matter as long as the unit tests pass
58+
# across all version for a given generated resource.
59+
ffmpeg-version-for-tests: ['6.1.1']
5760
steps:
58-
- uses: actions/download-artifact@v4
59-
with:
60-
name: meta-pytorch_torchcodec__${{ matrix.python-version }}_cpu_x86_64
61-
path: pytorch/torchcodec/dist/
61+
- name: Check out repo
62+
uses: actions/checkout@v3
63+
6264
- name: Setup conda env
63-
uses: conda-incubator/setup-miniconda@v2
65+
uses: conda-incubator/setup-miniconda@v3
6466
with:
6567
auto-update-conda: true
66-
miniconda-version: "latest"
68+
miniforge-version: latest
6769
activate-environment: test
6870
python-version: ${{ matrix.python-version }}
6971

70-
- name: Install ffmpeg
71-
run: |
72-
conda install "ffmpeg=${{ matrix.ffmpeg-version-for-tests }}" -c conda-forge
73-
ffmpeg -version
74-
7572
- name: Update pip
7673
run: python -m pip install --upgrade pip
7774

78-
- name: Install generation dependencies
79-
run: |
80-
# Note that we're installing stable - this is for running a script where we're a normal PyTorch
81-
# user, not for building TorhCodec.
82-
python -m pip install torch --index-url https://download.pytorch.org/whl/cpu
83-
python -m pip install numpy pillow pytest
75+
- name: Install PyTorch
76+
run: bash packaging/install_pytorch.sh cpu "torch"
77+
78+
- uses: actions/download-artifact@v4
79+
with:
80+
name: meta-pytorch_torchcodec__${{ matrix.python-version }}_cpu_x86_64
81+
path: dist/
8482

8583
- name: Install torchcodec from the wheel
86-
run: |
87-
wheel_path=`find pytorch/torchcodec/dist -type f -name "*.whl"`
88-
echo Installing $wheel_path
89-
python -m pip install $wheel_path -vvv
90-
- name: Check out repo
91-
uses: actions/checkout@v3
84+
run: bash packaging/install_torchcodec_wheel.sh
85+
86+
- name: Install ffmpeg, post build
87+
run: bash packaging/install_ffmpeg.sh ${{ matrix.ffmpeg-version-for-tests }}
88+
89+
- name: Install test dependencies
90+
run: bash packaging/install_test_dependencies.sh
9291

9392
- name: Run generation reference resources
9493
run: |

‎.github/workflows/windows_wheel.yaml‎

Lines changed: 2 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -75,7 +75,7 @@ jobs:
7575
needs: build
7676
steps:
7777
- name: Check out repo
78-
uses: actions/checkout@v3
78+
uses: actions/checkout@v6
7979

8080
- name: Remove src/ folder
8181
run: bash packaging/remove_src.sh
@@ -113,5 +113,4 @@ jobs:
113113
run: bash packaging/install_test_dependencies.sh
114114

115115
- name: Run Python tests
116-
run: |
117-
pytest test -vvv
116+
run: pytest --override-ini="addopts=-v" test

‎README.md‎

Lines changed: 5 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -133,6 +133,7 @@ The following table indicates the compatibility between versions of
133133
| `torchcodec` | `torch` | Python |
134134
| ------------------ | ------------------ | ------------------- |
135135
| `main` / `nightly` | `main` / `nightly` | `>=3.10`, `<=3.14` |
136+
| `0.10` | `2.10` | `>=3.10`, `<=3.14` |
136137
| `0.9` | `2.9` | `>=3.10`, `<=3.14` |
137138
| `0.8` | `2.9` | `>=3.10`, `<=3.13` |
138139
| `0.7` | `2.8` | `>=3.9`, `<=3.13` |
@@ -208,6 +209,10 @@ format you want. Refer to Nvidia's GPU support matrix for more details
208209
conda install -c conda-forge "torchcodec=*=*cuda*"
209210
```
210211

212+
### Specify FFmpeg Path Manually
213+
214+
If torchcodec cannot detect the FFmpeg installation correctly, you can set the `TORCHCODEC_FFMPEG_DIR` environment variable to the directory containing the FFmpeg DLLs.
215+
211216
## Benchmark Results
212217

213218
The following was generated by running [our benchmark script](./benchmarks/decoders/generate_readme_data.py) on a lightly loaded 22-core machine with an Nvidia A100 with

‎benchmarks/encoders/benchmark_encoders.py‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -71,7 +71,7 @@ def report_stats(times, num_frames, nvenc_metrics=None, prefix="", unit="ms"):
7171
f"GPU memory used: med = {mem_used_median:.1f} MB, max = {mem_used_max:.1f} MB"
7272
)
7373
print(
74-
f"NVENC utilization: med = {nvenc_metrics["utilization"].median():.1f}%, max = {util_max:.1f}%"
74+
f"NVENC utilization: med = {nvenc_metrics['utilization'].median():.1f}%, max = {util_max:.1f}%"
7575
)
7676

7777

0 commit comments

Comments
 (0)