You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@opendal.apache.org by "chitralverma (via GitHub)" <gi...@apache.org> on 2026/06/25 13:06:22 UTC
[PR] refactor(bindings/python): generate type stubs with pyo3 introspection [opendal]
chitralverma opened a new pull request, #7824:
URL: https://github.com/apache/opendal/pull/7824
# Which issue does this PR close?
Closes #.
# Rationale for this change
The Python binding generates its type stubs with `pyo3-stub-gen`. This has a few issues:
- `just stub-gen` has not worked on `main` for months. `pyo3-stub-gen` 0.18 changed the binding's module layout (in a stale, unmerged PR), later bumps moved it to 0.23, and the generator was never re-run. The committed stubs went stale, so the break stayed hidden. Regenerating now fails with a wrong-module error.
- It relies on `#[gen_stub_*]` macros and `override_type` / `override_return_type` attributes spread across the Rust code, kept in sync by hand.
- It ships an extra `stub_gen` binary and a crate dependency.
PyO3 0.29 can describe the extension through its `experimental-inspect` feature, and `maturin --generate-stubs` turns that into `.pyi` files. This is the direction PyO3 is moving in. It removes the extra dependency, the binary, and the hand-maintained macros, and derives the stubs straight from the built binary.
# What changes are included in this PR?
**Switch the generator**
- Drop `pyo3-stub-gen`, the `stub_gen` binary, and `define_stub_info_gatherer!`. Enable `pyo3/experimental-inspect`. The lib is `cdylib`-only now (the `rlib` was only for the deleted binary).
- Remove the `#[gen_stub_*]` attributes and convert the `override_*` attributes into inline `#[pyo3(signature = (arg: "T") -> "R")]` annotations.
- Update the codegen template to emit a lean `Scheme` enum; regenerate `src/services.rs` (−5.3k lines).
**Generate stubs at the public paths**
- `maturin --generate-stubs` writes under `opendal/_opendal/`. `scripts/postprocess_stubs.py` moves them to `opendal/<name>.pyi`, where type checkers resolve `from opendal.operator import …`.
- Wire `just stub-gen` and run ruff's fixers in `just fmt`.
**Guard against drift**
- Add a `stubs` CI job that runs `just stub-gen` and `git diff --exit-code`, so stale stubs fail CI. Release wheels ship the committed `python/opendal` tree as-is, so it must stay fresh.
# Are there any user-facing changes?
Yes. The changes are to the generated stubs only; runtime behavior is unchanged.
**Pros**
- **Stubs derive from the built binary**, so they stay aligned with the Rust source and the runtime types, and cannot silently drift.
- **Stub generation is fast** — it reads the compiled binary instead of running a separate generator pass.
- **Richer, more consistent stubs**: enum members (`EntryMode.*`, `Scheme.*`) and several dunders are now described, and `Metadata.last_modified` is correctly `datetime | None` (was `datetime`, which could not represent `None`).
- **Less code to maintain**: no `#[gen_stub_*]` macros, no `override_*` attributes, no `stub_gen` binary, one fewer dependency.
- **Standard tooling**: generation goes through `maturin` and PyO3 directly, the future-facing path as `experimental-inspect` matures.
- **Freshness is enforced** by the new CI drift check, so the published stubs always match the code.
**Cons**
- The per-service typed constructors are gone. The old stubs had 104 `@overload def __new__(...)` entries (one per service, with a typed `scheme=Literal[...]` and per-service config kwargs); they are now a single generic `__new__(scheme: str | Scheme, **kwargs)`. PyO3 introspection has no overload support. A follow-up will regenerate these from Rust.
- `scripts/postprocess_stubs.py` applies two temporary fixups: it injects the imports PyO3 omits for forward-ref annotations, and it substitutes a hand-written `exceptions.pyi` while the generated one is incomplete (`create_exception!` types are not introspectable). These are PyO3 limitations and should be removed as `experimental-inspect` improves in future releases.
Please add the `breaking-changes` label for the dropped constructor overloads.
# AI Usage Statement
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@opendal.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] refactor(bindings/python): generate type stubs with pyo3 introspection [opendal]
Posted by "erickguan (via GitHub)" <gi...@apache.org>.
erickguan commented on PR #7824:
URL: https://github.com/apache/opendal/pull/7824#issuecomment-4807784750
This is a really nice PR. Deleting code feels always great.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@opendal.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] refactor(bindings/python): generate type stubs with pyo3 introspection [opendal]
Posted by "chitralverma (via GitHub)" <gi...@apache.org>.
chitralverma commented on PR #7824:
URL: https://github.com/apache/opendal/pull/7824#issuecomment-4801139949
@erickguan here a big one, mostly deletions.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@opendal.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] refactor(bindings/python): generate type stubs with pyo3 introspection [opendal]
Posted by "Xuanwo (via GitHub)" <gi...@apache.org>.
Xuanwo merged PR #7824:
URL: https://github.com/apache/opendal/pull/7824
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@opendal.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] refactor(bindings/python): generate type stubs with pyo3 introspection [opendal]
Posted by "Copilot (via GitHub)" <gi...@apache.org>.
Copilot commented on code in PR #7824:
URL: https://github.com/apache/opendal/pull/7824#discussion_r3474649695
##########
bindings/python/src/file.rs:
##########
@@ -194,11 +189,8 @@ impl File {
/// -------
/// int
/// The number of bytes read.
- pub fn readinto(
- &mut self,
- #[gen_stub(override_type(type_repr = "builtins.bytes | builtins.bytearray", imports=("builtins")))]
- buffer: PyBuffer<u8>,
- ) -> PyResult<usize> {
+ #[pyo3(signature = (buffer: "bytes | bytearray"))]
+ pub fn readinto(&mut self, buffer: PyBuffer<u8>) -> PyResult<usize> {
Review Comment:
`readinto` rejects read-only buffers at runtime (`buffer.readonly()` check) but the exposed signature/doc currently advertises `bytes | bytearray`, which implies `bytes` is accepted. This will mislead type-checkers and users; `bytes` should be removed from the accepted types (a writable buffer like `bytearray`/`memoryview` is required).
##########
bindings/python/src/file.rs:
##########
@@ -704,23 +669,26 @@ impl AsyncFile {
})
}
- #[gen_stub(override_return_type(type_repr="typing.Self", imports=("typing")))]
+ // `typing_extensions.Self` (not `typing.Self`) because the binding's
+ // `requires-python` floor is 3.10, where `typing.Self` (3.11+) is absent;
+ // `typing_extensions` back-ports it. Switch to `typing.Self` once the floor
+ // reaches 3.11.
+ #[pyo3(signature = () -> "typing_extensions.Self")]
fn __aenter__<'a>(slf: PyRef<'a, Self>, py: Python<'a>) -> PyResult<Bound<'a, PyAny>> {
let slf = slf.into_py_any(py)?;
future_into_py(py, async move { Ok(slf) })
}
Review Comment:
`AsyncFile.__aenter__` returns an awaitable via `future_into_py`, but the signature annotation currently says it returns `typing_extensions.Self`. For an async context manager, `__aenter__` must return an `Awaitable[...]`; otherwise the generated `.pyi` marks it as a synchronous method returning `Self`, which is incorrect for both runtime calls and type checkers.
##########
bindings/python/src/file.rs:
##########
@@ -790,10 +749,6 @@ impl AsyncFile {
/// -------
/// coroutine
/// An awaitable that returns True if this file is closed.
- #[gen_stub(override_return_type(
- type_repr="collections.abc.Awaitable[builtins.bool]",
- imports=("collections.abc", "builtins")
- ))]
#[getter]
pub fn closed<'p>(&'p self, py: Python<'p>) -> PyResult<Bound<'p, PyAny>> {
let state = self.0.clone();
Review Comment:
`AsyncFile.closed` currently has no `#[pyo3(signature = ...)]` return annotation even though it returns an awaitable. This causes the generated stub to fall back to `Any` for `closed`, losing the intended `Awaitable[bool]` type. Adding an explicit signature keeps stubs accurate without changing runtime behavior.
##########
bindings/python/justfile:
##########
@@ -55,23 +55,24 @@ clean:
stub-gen: setup
@echo "{{ BOLD }}--- Generating Python type stubs ---{{ NORMAL }}"
@cargo run --quiet --manifest-path=../../dev/Cargo.toml -- generate -l python
- @cargo run --quiet --bin stub_gen
- @echo "{{ BOLD }}--- Formatting and fixing generated stubs ---{{ NORMAL }}"
- -@bash -c 'shopt -s globstar; uv run ruff check **/*.pyi --fix --unsafe-fixes --silent || true'
+ @uv run maturin develop --generate-stubs
+ ## This is temporary till the functionalities land in pyo3-introspection. ref: https://github.com/PyO3/pyo3/issues/5137
+ @echo "{{ BOLD }}--- Post processing generated stubs ---{{ NORMAL }}"
+ @uv run python scripts/postprocess_stubs.py
Review Comment:
`just stub-gen` regenerates stubs that are committed and shipped in release wheels, so stub generation should use the same Cargo feature set as wheel builds. The release workflow builds Python wheels with `--features=...,services-all,...`; without passing `--features services-all` here, the stub output can drift depending on maturin defaults / local environment.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@opendal.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] refactor(bindings/python): generate type stubs with pyo3 introspection [opendal]
Posted by "chitralverma (via GitHub)" <gi...@apache.org>.
chitralverma commented on code in PR #7824:
URL: https://github.com/apache/opendal/pull/7824#discussion_r3474851372
##########
bindings/python/src/file.rs:
##########
@@ -790,10 +749,6 @@ impl AsyncFile {
/// -------
/// coroutine
/// An awaitable that returns True if this file is closed.
- #[gen_stub(override_return_type(
- type_repr="collections.abc.Awaitable[builtins.bool]",
- imports=("collections.abc", "builtins")
- ))]
#[getter]
pub fn closed<'p>(&'p self, py: Python<'p>) -> PyResult<Bound<'p, PyAny>> {
let state = self.0.clone();
Review Comment:
not applicable for getters
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@opendal.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org