You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@paimon.apache.org by "plusplusjiajia (via GitHub)" <gi...@apache.org> on 2026/09/21 07:41:27 UTC
[PR] [spark] Read split statistics through the query authorization wrapper [paimon]
plusplusjiajia opened a new pull request, #10051:
URL: https://github.com/apache/paimon/pull/10051
## Problem
`SplitUtils` narrows a split to `DataSplit` or `FormatDataSplit` and falls through to `0`. A `QueryAuthSplit` is neither, so on a table carrying query authorization every size and file count reads as zero. `PaimonStatistics` reports `sizeInBytes = 0`, which tells the optimizer the table is empty.
`paimon-core` and `paimon-flink` already peel the wrapper through `Splits.underlying`; `paimon-spark` never did.
## Change
`SplitUtils` reads its statistics off the underlying split. Only metadata is read there, so the splits handed to readers keep their wrapper and the row filter and column masks still apply.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [spark] Read split statistics through the query authorization wrapper [paimon]
Posted by "JingsongLi (via GitHub)" <gi...@apache.org>.
JingsongLi commented on PR #10051:
URL: https://github.com/apache/paimon/pull/10051#issuecomment-5770411750
Requirement fit: SUPPORTED. Query authorization wraps splits, and reporting their sizes as zero can mislead Spark's optimizer for a nonempty table. Unwrapping only for metadata keeps the authorization wrapper on the reader path; the focused test checks size and count. Implementation: CLEAN in this diff. I did not rerun the Spark tests locally.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [spark] Read split statistics through the query authorization wrapper [paimon]
Posted by "JingsongLi (via GitHub)" <gi...@apache.org>.
JingsongLi merged PR #10051:
URL: https://github.com/apache/paimon/pull/10051
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org