You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@paimon.apache.org by "TheR1sing3un (via GitHub)" <gi...@apache.org> on 2026/06/17 13:48:13 UTC
[PR] [python] Fix Ray read_paimon dropping nested projection (reads nested leaves as NULL) [paimon]
TheR1sing3un opened a new pull request, #8269:
URL: https://github.com/apache/paimon/pull/8269
## Purpose
`RayDatasource` rebuilds the worker-side `TableRead` from the split provider's
`(table, read_type, predicate, limit)` but **not** its `nested_name_paths`. As a
result `read_paimon(..., projection=['payload.a'])` (a nested-leaf projection)
reads every projected leaf as `NULL` — the worker treats the flattened leaf
name (e.g. `payload_a`) as a missing top-level column. The non-Ray read path is
unaffected because `ReadBuilder.new_read()` already forwards `nested_name_paths`
to `TableRead`.
## Fix
- `SplitProvider` exposes `nested_name_paths()`: resolved via the read builder
for `CatalogSplitProvider`, carried from the source `TableRead` for
`PreResolvedSplitProvider` (`TableRead.to_ray`).
- `RayDatasource` forwards it into the per-task worker `TableRead`.
The change is a no-op for non-nested / top-level-only projections
(`nested_name_paths` is `None` there), so existing reads are unaffected.
## Tests
Adds `RayIntegrationTest.test_read_paimon_with_nested_projection`, asserting a
`['id', 'payload.a']` projection returns the real leaf values instead of `NULL`.
## Does this PR introduce a user-facing change?
No.
## Documentation
No documentation change needed.
---
Generative AI disclosure: drafted with AI assistance and reviewed by the author.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [python] Fix Ray read_paimon dropping nested projection (reads nested leaves as NULL) [paimon]
Posted by "TheR1sing3un (via GitHub)" <gi...@apache.org>.
TheR1sing3un commented on code in PR #8269:
URL: https://github.com/apache/paimon/pull/8269#discussion_r3437094125
##########
paimon-python/pypaimon/tests/ray_integration_test.py:
##########
@@ -119,6 +119,31 @@ def test_read_paimon_with_projection(self):
self.assertEqual(set(df.columns), {'id', 'name'})
self.assertEqual(len(df), 2)
+ def test_read_paimon_with_nested_projection(self):
Review Comment:
> nit: The implementation also fixes the PreResolvedSplitProvider path used by `TableRead.to_ray()`. A small sibling regression test for that entry point would make the coverage a bit more complete.
Nice catch, fixed
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [python] Fix Ray read_paimon dropping nested projection (reads nested leaves as NULL) [paimon]
Posted by "JingsongLi (via GitHub)" <gi...@apache.org>.
JingsongLi merged PR #8269:
URL: https://github.com/apache/paimon/pull/8269
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [python] Fix Ray read_paimon dropping nested projection (reads nested leaves as NULL) [paimon]
Posted by "JingsongLi (via GitHub)" <gi...@apache.org>.
JingsongLi commented on PR #8269:
URL: https://github.com/apache/paimon/pull/8269#issuecomment-4747311322
Reviewed the Ray nested projection fix. The new nested_name_paths plumbing is carried through both CatalogSplitProvider/read_paimon and PreResolvedSplitProvider/TableRead.to_ray, so the worker-side TableRead now has the same nested projection context as the normal read path. The added tests cover both Ray entry points. LGTM.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [python] Fix Ray read_paimon dropping nested projection (reads nested leaves as NULL) [paimon]
Posted by "QuakeWang (via GitHub)" <gi...@apache.org>.
QuakeWang commented on code in PR #8269:
URL: https://github.com/apache/paimon/pull/8269#discussion_r3437008772
##########
paimon-python/pypaimon/tests/ray_integration_test.py:
##########
@@ -119,6 +119,31 @@ def test_read_paimon_with_projection(self):
self.assertEqual(set(df.columns), {'id', 'name'})
self.assertEqual(len(df), 2)
+ def test_read_paimon_with_nested_projection(self):
Review Comment:
nit: The implementation also fixes the PreResolvedSplitProvider path used by `TableRead.to_ray()`. A small sibling regression test for that entry point would make the coverage a bit more complete.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org