You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@paimon.apache.org by "XiaoHongbo-Hope (via GitHub)" <gi...@apache.org> on 2026/10/08 11:49:46 UTC
[PR] [python] Fix Arrow S3-compatible client checksums across workers [paimon]
XiaoHongbo-Hope opened a new pull request, #10450:
URL: https://github.com/apache/paimon/pull/10450
### Purpose
While adding newer PyArrow support in #9821 for LeRobot integration, we found that optional AWS SDK request checksums can break writes to OSS and custom S3 endpoints. Pickled FileIO objects also need the same configuration when clients are reconstructed in already-running workers.
Temporarily set `AWS_REQUEST_CHECKSUM_CALCULATION=WHEN_REQUIRED` during compatible client construction, restore the previous environment setting even on failure, and rebuild those clients on deserialization using the worker's PyArrow version. Covers `oss://` with the PyArrow implementation and custom endpoints under `s3://`, `s3a://`, and `s3n://`.
Extracted from #9821 onto master. **Independent of #10425**: no Catalog/deletion changes, Boto3 dependency, or PyArrow dependency-range changes. Batch-delete compatibility and dependency relaxation remain separate work.
The environment setting is process-wide. A lock coordinates Paimon client construction only; unrelated libraries constructing AWS clients concurrently are not protected by this lock.
### Validation
- PyArrow 19.0.1: **51 passed, 1 skipped**; PyArrow 23.0.0: **52 passed** across client, OSS legacy, and FileIO tests.
- Real PyArrow 23 requests to a local HTTP endpoint verify compatible parent/spawn-worker writes omit checksum trailers while an ordinary client retains them.
- Covers environment restoration, S3 scheme aliases, old pickle state, and worker version recomputation.
- Flake8, Python 3.6 syntax, and `git diff --check` pass. No live OSS/DLF rerun for this extracted PR.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [python] Fix optional checksum compatibility for OSS and custom S3 endpoints [paimon]
Posted by "JingsongLi (via GitHub)" <gi...@apache.org>.
JingsongLi merged PR #10450:
URL: https://github.com/apache/paimon/pull/10450
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [python] Fix optional checksum compatibility for OSS and custom S3 endpoints [paimon]
Posted by "JingsongLi (via GitHub)" <gi...@apache.org>.
JingsongLi commented on PR #10450:
URL: https://github.com/apache/paimon/pull/10450#issuecomment-6072784626
+1
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org