You are viewing a plain text version of this content. The canonical link for it is here.

Posted to issues@paimon.apache.org by "XiaoHongbo-Hope (via GitHub)" <gi...@apache.org> on 2026/10/05 12:41:45 UTC

[PR] [python] Support native OSS C++ reads with a separate Python backend [paimon]

XiaoHongbo-Hope opened a new pull request, #10393:
URL: https://github.com/apache/paimon/pull/10393

### Purpose

Companion to apache/paimon-rust#1028: allow native OSS plan/read to use the C++ backend while Python I/O, including Blob downloads, keeps using Jindo or legacy implementations.

### Changes

- Accept `fs.oss.impl=cpp` in Python OSS routing; select its Python backend with `fs.oss.python.impl` (`jindo` by default, or `legacy`). Preserve the original options passed to Rust.
- Apply the same selection to PyArrow FileIO and PVFS, including path handling. Keep native-only C++ options out of Jindo configuration.
- Document the separately installed bridge, read-only scope, and worker requirements. Native enablement/fallback rules are unchanged.

### Validation

Five focused suites (`oss_legacy_mode_test`, `pvfs_oss_filesystem_test`, `jindo_file_system_test`, `native_plan_test`, `native_read_test`): **161 passed, 19 skipped, 46 subtests passed**, Python 3.11 / PyArrow 19.0.1.

Tests cover Python backend selection with/without Jindo, invalid options, configuration preservation, and REST native Table reuse. These are isolated routing/reader tests, not a live OSS/C++ performance run. `git diff --check` passes. Whole-file Flake8 reports three existing findings also reproduced on the unmodified base (`pvfs.py` F821; `native_plan_test.py` F821/E305); no new findings.

--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [PR] [python] Support native OSS C++ reads with a separate Python backend [paimon]

Posted by "JingsongLi (via GitHub)" <gi...@apache.org>.

JingsongLi merged PR #10393:
URL: https://github.com/apache/paimon/pull/10393


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [PR] [python] Support native OSS C++ reads with a separate Python backend [paimon]

Posted by "JingsongLi (via GitHub)" <gi...@apache.org>.

JingsongLi commented on PR #10393:
URL: https://github.com/apache/paimon/pull/10393#issuecomment-6118761736

   +1


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [PR] [python] Support native OSS C++ reads with a separate Python backend [paimon]

Posted by "JingsongLi (via GitHub)" <gi...@apache.org>.

JingsongLi commented on code in PR #10393:
URL: https://github.com/apache/paimon/pull/10393#discussion_r4205721577


##########
docs/docs/pypaimon/installation.md:
##########
@@ -109,6 +109,18 @@ The source archive and wheel are written to `dist/` with the same version.
 Development versions append the Git commit date, for example `2.2.dev20260927`.
 Install the wheel with `python -m pip install dist/*.whl`.
 
+## Optional OSS C++ backend
+
+Native OSS reads require a `pypaimon-rust` build with `storage-oss-cpp` and

Review Comment:
   Maybe just bundle `storage-oss-cpp` by default? How large?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [PR] [python] Support native OSS C++ reads with a separate Python backend [paimon]

Posted by "XiaoHongbo-Hope (via GitHub)" <gi...@apache.org>.

XiaoHongbo-Hope commented on code in PR #10393:
URL: https://github.com/apache/paimon/pull/10393#discussion_r4242024521


##########
docs/docs/pypaimon/installation.md:
##########
@@ -109,6 +109,18 @@ The source archive and wheel are written to `dist/` with the same version.
 Development versions append the Git commit date, for example `2.2.dev20260927`.
 Install the wheel with `python -m pip install dist/*.whl`.
 
+## Optional OSS C++ backend
+
+Native OSS reads require a `pypaimon-rust` build with `storage-oss-cpp` and

Review Comment:
   > Maybe just bundle `storage-oss-cpp` by default? How large?
   
   Do you mean bundling the OSS C++ bridge and SDK in the standard pypaimon-rust wheels? On Linux x86_64, bundling them increased the wheel by about 2.4 MB in my initial test.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [PR] [python] Support native OSS C++ reads with a separate Python backend [paimon]

Posted by "JingsongLi (via GitHub)" <gi...@apache.org>.

JingsongLi commented on code in PR #10393:
URL: https://github.com/apache/paimon/pull/10393#discussion_r4245051864


##########
docs/docs/pypaimon/installation.md:
##########
@@ -109,6 +109,18 @@ The source archive and wheel are written to `dist/` with the same version.
 Development versions append the Git commit date, for example `2.2.dev20260927`.
 Install the wheel with `python -m pip install dist/*.whl`.
 
+## Optional OSS C++ backend
+
+Native OSS reads require a `pypaimon-rust` build with `storage-oss-cpp` and

Review Comment:
   Yes, If this is something we’ll be using for the long term, I feel it’s worth it.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org