refactor(bindings/python): use declarative pymodule syntax#7812
Conversation
Convert _opendal from the imperative add_pymodule!/add_pyexceptions! macros to PyO3's declarative #[pymodule] mod form with #[pymodule_export]. Export __version__ as a #[pymodule_export] const, and add a register_in_sys helper plus an add_exceptions! macro. Submodule __name__ is now the fully qualified dotted name (e.g. opendal.operator). Refs: https://pyo3.rs/v0.29.0/module.html
There was a problem hiding this comment.
Pull request overview
This PR refactors the Rust-backed Python extension module (_opendal) to use PyO3’s declarative #[pymodule] mod ... syntax with #[pymodule_export], replacing the prior imperative submodule construction macros. It also centralizes submodule registration to ensure sys.modules and __name__ reflect fully-qualified opendal.* module names.
Changes:
- Converted
_opendalfrom afn-based#[pymodule]initializer to a declarative#[pymodule] modwith nested submodules and#[pymodule_export]exports. - Replaced
add_pymodule!/add_pyexceptions!with aregister_in_syshelper and a simplifiedadd_exceptions!macro. - Exported
__version__via#[pymodule_export] pub const __version__instead of setting it during module init.
Reviewed changes
Copilot reviewed 2 out of 3 changed files in this pull request and generated 3 comments.
| File | Description |
|---|---|
| bindings/python/src/utils.rs | Introduces register_in_sys and add_exceptions! to support the new declarative module layout and qualified sys.modules registration. |
| bindings/python/src/lib.rs | Rewrites _opendal module initialization into declarative nested #[pymodule] submodules and exports __version__ as a module constant. |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
|
@erickguan @Xuanwo please have a look when you get a chance. This is in continuation of #7181 |
…larative-pymodule
…dule' into refactor/python-declarative-pymodule
|
@PsiACE you wanna check this one as well? |
erickguan
left a comment
There was a problem hiding this comment.
Generally I like using declarative modules as documentation provided. Perhaps we could stick on one style for majority of the code?
|
|
||
| #[pymodule_init] | ||
| fn init(m: &pyo3::Bound<'_, pyo3::types::PyModule>) -> pyo3::PyResult<()> { | ||
| crate::register_in_sys(m, "operator") |
There was a problem hiding this comment.
Can we use #[pymodule] for submodules too?
I know packaging might be a problem. Here is what I found.
There was a problem hiding this comment.
I looked at the linked comment and the alternative below it and modified the current implementation with a single recursive register_submodules walk called once from the root #[pymodule_init], replacing the per-submodule registration.
For opendal, we don't need the submodule flag or the PyModuleSubmoduleExt trait as all our submodules are nested inside mod _opendal, so PyO3 auto-marks them as submodules.
… walk Replace the per-submodule `register_in_sys` calls (one `#[pymodule_init]` each) with a single `register_submodules` walk invoked once from the root module. It recurses over the declarative submodules and inserts them into `sys.modules` under the public `opendal` name, fixing each `__name__`. Adapted from the approaches in PyO3 issue apache#759. The data submodules are now plain `#[pymodule] mod` blocks; only `exceptions` keeps an init for `add_exceptions!`.
…rsive fn It is only called from the root `#[pymodule_init]`, so fold the thin `sys.modules`-import wrapper into the recursive function itself.
…dule' into refactor/python-declarative-pymodule
erickguan
left a comment
There was a problem hiding this comment.
Looks good. One final question on opendal module
| ] | ||
| )?; | ||
| Ok(()) | ||
| mod _opendal { |
There was a problem hiding this comment.
Does declarative syntax works for modules?
#[pymodule(gil_used = false)]
mod opendal {
}There was a problem hiding this comment.
Yes — mod _opendal here is the declarative #[pymodule] mod form (converted from the old fn _opendal).
It has to stay _opendal, not opendal: this is a mixed layout, so module-name = "opendal._opendal" builds opendal/_opendal.*.so and the hand-written opendal/__init__.py is the real package that re-exports from it. The mod name must match the .so's last path component (PyInit__opendal). Naming it opendal would require the native module to be the top-level package, colliding with the opendal/ package dir.
There was a problem hiding this comment.
Thanks for your detailed explanation! I would recommend documenting:
We are using a mixed Python and native package for python OpenDAL binding.
To avoid conflict of ...
<And a concise explanation of how it works here should be enough>
There was a problem hiding this comment.
Added a comment above mod _opendal explaining the mixed-layout naming (ba8627b).
Which issue does this PR close?
Closes #.
Rationale for this change
The Python binding built its submodules (
operator,file,capability,services,layers,types,exceptions) imperatively via theadd_pymodule!/add_pyexceptions!macros and afn-based#[pymodule]. PyO3 recommends the declarative#[pymodule] mod ...form with#[pymodule_export](PyO3 module guide). This PR moves the binding to that style.What changes are included in this PR?
_opendalis now a#[pymodule] modwith one nested#[pymodule] modper submodule; classes are attached with#[pymodule_export].__version__is exported as a#[pymodule_export] pub constinstead of being set in an init function.add_pymodule!/add_pyexceptions!macros with aregister_in_syshelper (registers each submodule insys.modulesand sets its__name__) and anadd_exceptions!macro.create_exception!types arePyErrsubtypes rather than#[pyclass]es and cannot use#[pymodule_export].uv.lock.Are there any user-facing changes?
No API changes. One minor behavioral correction: submodule
__name__is now the fully qualified dotted name (e.g.opendal.operator) instead of the bare name (operator), matching thesys.moduleskey, the classes'__module__, and standard-library conventions. All public imports (from opendal import Operator,from opendal.exceptions import NotFound, etc.) andopendal.__version__are unchanged.AI Usage Statement