You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@paimon.apache.org by "JingsongLi (via GitHub)" <gi...@apache.org> on 2026/10/07 14:09:44 UTC
[PR] [python] Keep custom Blob URI readers on native writes [paimon]
JingsongLi opened a new pull request, #10417:
URL: https://github.com/apache/paimon/pull/10417
### Purpose
Keep PyPaimon's existing `with_blob_uri_reader_factory` API on native Arrow writes instead of selecting the Python writer.
Depends on https://github.com/apache/paimon-rust/pull/1065. This PR is draft until the Rust interfaces merge. Native CI continues to build Rust `main`; no branch pin or compatibility fallback is added.
The adapter only forwards the factory and retains it for an explicit row-writer fallback. Descriptor-window copying, successful short reads, stream identity/reuse, failed-seek reopening, unknown-length EOF copying and source closure are implemented in Rust core, following Java `UriReaderFactory`, `UriReader`, `ReusingBlobRefStreamProvider` and managed Blob pack cleanup.
Tests cover batch and stream writes; scalar, array and map Blob values; BlobConsumer composition; nonseekable sources and unknown-length references; original callback exceptions; premature EOF and failed preparation; setter replacement/clearing and row fallback; managed fixed/pending/postpone-fixed writes; and stable or changing reader identity. The multi-column interleaving test reproduced the binding identity bug before its fix.
### Tests
With the locally rebuilt Rust binding from the companion PR:
- Focused URI reader/BlobConsumer/write tests: 48 passed.
- Extended Blob/write/update suite with all Native switches enabled: 518 passed, plus 1 subtest. Actual native paths: 659 plans, 863 reads, 800 writes, 169 commits; row-id/grouped/predicate/upsert/incremental update categories all exercised.
- Companion Rust core: 3,773 passed, 6 ignored. Full Python binding suite: 331 passed.
- `flake8 --config dev/cfg.ini` for changed files, `git diff --check` and Python 3.6 syntax parsing: passed.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [python] Keep custom Blob URI readers on native writes [paimon]
Posted by "leaves12138 (via GitHub)" <gi...@apache.org>.
leaves12138 commented on code in PR #10417:
URL: https://github.com/apache/paimon/pull/10417#discussion_r4208398004
##########
paimon-python/pypaimon/write/native_write.py:
##########
@@ -135,6 +136,8 @@ def _new_python_writer(self):
writer.with_write_type(self._write_cols)
if self._blob_consumer is not None:
writer.with_blob_consumer(self._blob_consumer)
+ if self._blob_uri_reader_factory is not None:
Review Comment:
Blocking: this state is replayed only by `NativeTableWrite._new_python_writer`, but `NativePostponeFixedBucketTableWrite` overrides that method and does not apply `_blob_uri_reader_factory`. If a postpone-fixed writer is configured with `with_blob_uri_reader_factory(factory)` and then a row-oriented API triggers `_switch_to_python()`, the Python writer silently loses the configured factory and cannot resolve custom URI Blob descriptors consistently with the native path. Please propagate this factory in the override and add a regression test for the fallback path.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [python] Keep custom Blob URI readers on native writes [paimon]
Posted by "JingsongLi (via GitHub)" <gi...@apache.org>.
JingsongLi merged PR #10417:
URL: https://github.com/apache/paimon/pull/10417
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org