model: add Stable-Video-Diffusion by rebel-seinpark · Pull Request #38 · RBLN-SW/optimum-rbln

rebel-seinpark · 2025-03-04T09:01:08Z

Pull Request Description

PR for supporting Stable-video-diffusion(SVD) Pipeline

You need proper compiler hash below

Compiler hash : https://github.com/rebellions-sw/rebel_compiler/pull/6103

Type of Change

Release (dev → main merge for production release)
New Model Support
Bug fix (non-breaking change which fixes an issue)
New feature (non-breaking change which adds functionality)
Breaking change (fix or feature that would cause existing functionality to not work as expected)
Documentation update
Performance improvement
Code refactoring
Other (please describe):

Changes Overview

Add RBLNStableVideoDiffusionPipeline
- Add RBLNUNetSpatioTemporalConditionModel into ./src/optimum/rbln/diffusers/models/unets/unet_spatio_temporal_condition.py
- Add 'RBLNAutoencoderKLTemporalDecoder' into ./src/optimum/rbln/diffusers/models/autoencoders/autoencoder_kl_temporal_decoder.py
- Add _VAETemporalDecoder into ./src/optimum/rbln/diffusers/models/autoencoders/vae.py
We need to discussion about forward function in RBLNCLIPVisionModel and RBLNCLIPVisionModelWithProjection

Motivation and Context

Related Issues

Conventional commit

type(optional scope): description

Type candidate

Model Updates
- model: Adding New models or Bugfix for existing models
  - ex) Add LlavaNext
  - ex) Bugfix Whisper
Enhancements
- performance: Optimizing some models or this library itself
  - ex) Loading RBLNModel faster
  - ex) Optimizing Memory Usage of DecoderOnlyModel
Code Refactor
- refactor: Re-arrange class architecture, or more.
  - ex) Refactor Seq2Seq
Documentation
- doc: Update docstring only
Library Dependencies
- dependency: Update requirements, something like that.
Other
- other: None of above.
  - ex) ci update
  - ex) pdm update

…imum-rbln into stable-video-diffusion

rebel-jongho

Almost looks fine.

rebel-kblee

lgtm! 👍 only check ruff before merge

rebel-dkhong

LGTM!

This reverts commit 79f5928.

This reverts commit dc1c845.

This reverts commit 53d77eb, reversing changes made to 84aa338.

This reverts commit 84aa338.

Co-authored-by: rebel-jongho <112920593+rebel-jongho@users.noreply.github.com> Co-authored-by: rebel-kblee <119555851+rebel-kblee@users.noreply.github.com> Co-authored-by: rebel-thkim <157466331+rebel-thkim@users.noreply.github.com> Co-authored-by: Sungho Shin <87514200+rebel-shshin@users.noreply.github.com> Co-authored-by: rebel-jongho <jongho.choi@rebellions.ai> Co-authored-by: Sunwook Jung <112920070+rebel-sunwook@users.noreply.github.com> Co-authored-by: rebel-junamsong <junam.song@rebellions.ai> Co-authored-by: kblee <kblee@rebellions.ai> Co-authored-by: rebel-dkhong <dk.hong@rebellions.ai>

rebel-seinpark added 13 commits February 3, 2025 17:53

feat svd

05c00b7

rebase

5f8b71a

add multi-device option

d48b605

re-allocate device num

7da25e7

bug fix

a21ec73

rebase

ad486a1

fix: add Union

2891462

feat: model output key

e616073

feat: add pytest with small input size

272c7f9

refactor: remove unnecessary code

6fa4fb7

refactor: modify device number

cd47c12

Merge branch 'stable-video-diffusion' of github.com:rebellions-sw/opt…

56f449a

…imum-rbln into stable-video-diffusion

refactor: ruff

f98fd6d

rebel-seinpark requested review from rebel-jongho, rebel-kblee and rebel-thkim March 4, 2025 09:01

rebel-seinpark commented Mar 4, 2025

View reviewed changes

Comment thread src/optimum/rbln/transformers/models/clip/modeling_clip.py Outdated

rebel-seinpark commented Mar 4, 2025

View reviewed changes

Comment thread src/optimum/rbln/transformers/models/clip/modeling_clip.py Outdated

rebel-jongho reviewed Mar 5, 2025

View reviewed changes

rebel-seinpark added 6 commits March 5, 2025 13:52

refactor: remove duplicated code

5c4b34e

refactor: licences

2ec5300

ruff

95cb174

refactor: import

399e3c8

fix: logger

6247a24

fix: logger

e9428a5

rebel-seinpark commented Mar 5, 2025

View reviewed changes

Comment thread src/optimum/rbln/transformers/models/clip/modeling_clip.py Outdated

rebel-seinpark added 2 commits March 5, 2025 16:27

refactor: remove unnecessary type_checking

3b7c6da

revert clipvisionmodel output logic

082f0ed

rebel-jongho reviewed Mar 6, 2025

View reviewed changes

Comment thread src/optimum/rbln/diffusers/modeling_diffusers.py Outdated

rebel-jongho reviewed Mar 6, 2025

View reviewed changes

Comment thread tests/test_diffusers.py Outdated

rebel-dkhong reviewed Nov 25, 2025

View reviewed changes

Comment thread src/optimum/rbln/diffusers/models/autoencoders/autoencoder_kl_temporal_decoder.py Outdated

rebel-seinpark added 2 commits November 25, 2025 13:33

refactor: funciton internalization

1072912

[skip-ci]applying review

f3f8a5d

rebel-jongho approved these changes Nov 25, 2025

View reviewed changes

Comment thread src/optimum/rbln/diffusers/models/autoencoders/autoencoder_kl_temporal_decoder.py Outdated

Comment thread src/optimum/rbln/diffusers/models/autoencoders/autoencoder_kl_temporal_decoder.py Outdated

rebel-seinpark added 3 commits November 25, 2025 14:02

[skip-transformers] fix: pytest

8fa4f32

[skip-transformers]refactor: simplify _update_rbln_config logic in dec

070d508

[skip-transformers]refactor: ruff

6352389

rebel-kblee approved these changes Nov 25, 2025

View reviewed changes

rebel-dkhong approved these changes Nov 25, 2025

View reviewed changes

rebel-seinpark and others added 16 commits November 25, 2025 14:42

update version

b8fe342

[skip-llm]rebase

86731d2

fix: create_runtimes=False

60eaa3f

[skip-llm][skip-transformers] temp test

84aa338

[skip-llm][skip-transformers] temp test

53d77eb

[skip-llm][skip-transformers] remove debug

dc1c845

fix pytest

79f5928

[skip-llm]

fba8a8b

Revert "fix pytest"

a0b02c2

This reverts commit 79f5928.

Revert "[skip-llm][skip-transformers] remove debug"

8a7e166

This reverts commit dc1c845.

Revert "[skip-llm][skip-transformers] temp test"

f5a5a23

This reverts commit 53d77eb, reversing changes made to 84aa338.

Revert "[skip-llm][skip-transformers] temp test"

7dd9c6e

This reverts commit 84aa338.

fix: pytest

8777280

feat: add timeout for vae

771bfd1

[skip-llm]

2a56d27

[skip-llm]fix: rm create_runtime=True

6c90d03

rebel-seinpark merged commit 2b32ddd into dev Nov 25, 2025
17 checks passed

rebel-seinpark deleted the stable-video-diffusion branch November 25, 2025 12:11

Uh oh!

Conversation

rebel-seinpark commented Mar 4, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Pull Request Description

Type of Change

Changes Overview

Motivation and Context

Related Issues

Conventional commit

Type candidate

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

rebel-jongho left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

rebel-kblee left a comment

Choose a reason for hiding this comment

Uh oh!

rebel-dkhong left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

rebel-seinpark commented Mar 4, 2025 •

edited

Loading