You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@paimon.apache.org by "zaynt4606 (via GitHub)" <gi...@apache.org> on 2026/10/20 09:52:42 UTC
[PR] [python][ray] Fix read-task limits and close batch readers [paimon]
zaynt4606 opened a new pull request, #10698:
URL: https://github.com/apache/paimon/pull/10698
### Purpose
Fix limit handling and reader cleanup in `RayDatasource`.
The task can emit fewer rows than its `BlockMetadata.num_rows` reports: for example, a five-row split with a scan limit of two still advertises five rows. Ray's `per_task_row_limit` is also passed only to `ReadTask`, without constraining the underlying Paimon reader. Finally, the worker does not explicitly close its Arrow batch reader.
- Combine the Paimon scan limit and Ray's per-task row limit before constructing the worker reader.
- Cap known task row counts at the effective limit, preserving unknown counts for filtered or primary-key reads.
- Close batch readers in `finally`, including normal completion, empty reads, failures, and early generator closure.
- Add focused regression tests and strengthen the public API test to check exact counts before and after materialization across two partitions, including zero and oversized limits.
This change does not alter public API signatures or catalog authentication behavior.
### Tests
Tested against community Paimon master with official Ray wheels, Python 3.12, and PyArrow 19.0.1:
- Before the fix, the 14 new datasource cases reproduce **11 failures and 3 passes** on Ray 2.58.0.
- After the fix, **69 passed** on Ray 2.58.0 across `ray_datasource_test.py`, `ray_integration_test.py`, `ray_sink_test.py`, and `ray_data_test.py`.
- **30 passed on each of Ray 2.44.0, 2.48.0, 2.50.1, and 2.53.0**: the datasource regressions, the exact-limit integration test, and `RayDataTest`.
- Changed Python files pass repository-configured Flake8; the Python license-header check and `git diff --check` pass.
The new regression tests cover limit combinations, task serialization, unknown filtered counts, reader-level limit pushdown, and batch-reader cleanup. The integration tests use a local Ray cluster and filesystem catalog; remote storage and the optional Rust backend were not exercised.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [python][ray] Fix read-task limits and close batch readers [paimon]
Posted by "zaynt4606 (via GitHub)" <gi...@apache.org>.
zaynt4606 commented on PR #10698:
URL: https://github.com/apache/paimon/pull/10698#issuecomment-6407236986
Rebased onto master and preserved the existing `close_read_source(batch_reader, failure)` pattern, retaining the primary read exception while still surfacing standalone cleanup failures.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [python][ray] Fix read-task limits and close batch readers [paimon]
Posted by "JingsongLi (via GitHub)" <gi...@apache.org>.
JingsongLi commented on PR #10698:
URL: https://github.com/apache/paimon/pull/10698#issuecomment-6408081767
+1
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [python][ray] Fix read-task limits and close batch readers [paimon]
Posted by "zaynt4606 (via GitHub)" <gi...@apache.org>.
zaynt4606 commented on PR #10698:
URL: https://github.com/apache/paimon/pull/10698#issuecomment-6306141779
Fixed the PR-specific Native CI failure in db2e8076bf.
The native planner can return no splits for a zero scan limit, so the new datasource test must not index the first read task unconditionally. The test now checks the metadata of every returned task and asserts the total row count, including an empty plan. No runtime code was changed and no Native tests were skipped.
Validation after this update:
- 69 passed on official Ray 2.58.0: datasource, integration, sink, and RayData tests.
- 14 passed with the split provider patched to reproduce Native's empty-plan behavior for LIMIT 0 (before this test fix: 1 failed, 13 passed).
- Repository-configured Flake8, Python license-header checks, and git diff --check passed.
The real Rust backend was not run locally; the new Native CI run will validate that backend.
The other failures also occur on the unchanged upstream master commit eeb765d405963b619cc0194ecf79e2378df3b77e:
- `BlobConsumerTest::test_blob_consumer_after_write_raises`: Native raises ValueError where the test expects RuntimeError.
- `test_ltz_partition_strings_preserve_submicrosecond_fraction`: timestamp fraction differs from the expected nanosecond value.
Both are present in the [upstream master Native CI log](https://github.com/apache/paimon/actions/runs/39399763121/job/118912025870); the timestamp failure is also present in the [upstream master Python 3.10 CI log](https://github.com/apache/paimon/actions/runs/39399763121/job/118912026269). I have kept these unrelated baseline failures out of this PR.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [python][ray] Fix read-task limits and close batch readers [paimon]
Posted by "zaynt4606 (via GitHub)" <gi...@apache.org>.
zaynt4606 commented on PR #10698:
URL: https://github.com/apache/paimon/pull/10698#issuecomment-6288474509
@Zouxxyy plz help review this pr
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [python][ray] Fix read-task limits and close batch readers [paimon]
Posted by "JingsongLi (via GitHub)" <gi...@apache.org>.
JingsongLi merged PR #10698:
URL: https://github.com/apache/paimon/pull/10698
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org