You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@paimon.apache.org by "Aiden-Dong (via GitHub)" <gi...@apache.org> on 2024/11/25 10:14:07 UTC
[PR] Optimize Parquet Reading by Enhancing Predicate Pushdown to the Page Level [paimon]
Aiden-Dong opened a new pull request, #4587:
URL: https://github.com/apache/paimon/pull/4587
<!-- Please specify the module before the PR name: [core] ... or [flink] ... -->
### Purpose
<!-- Linking this pull request to the issue -->
Linked issue: [#4586](https://github.com/apache/paimon/issues/4586)
<!-- What is the purpose of the change -->
### Tests
<!-- List UT and IT cases to verify this change -->
### API and Format
<!-- Does this change affect API or storage format -->
### Documentation
<!-- Does this change introduce a new feature -->
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [Core] Optimize Parquet Reading by Enhancing Predicate Pushdown to the Page Level [paimon]
Posted by "Aiden-Dong (via GitHub)" <gi...@apache.org>.
Aiden-Dong commented on PR #4587:
URL: https://github.com/apache/paimon/pull/4587#issuecomment-2497541533
> 这个修改没那么简单，它需要我们修改列阅读器。
I found that it works in my local tests, and the column pages obtained based on FilterRowGroup are filtered. Is there any error I might have missed?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [Core] Optimize Parquet Reading by Enhancing Predicate Pushdown to the Page Level [paimon]
Posted by "Aiden-Dong (via GitHub)" <gi...@apache.org>.
Aiden-Dong commented on PR #4587:
URL: https://github.com/apache/paimon/pull/4587#issuecomment-2497565723
> This modification is not that simple, it requires us to modify the column reader.
I found that it works in my local tests, and the column pages obtained based on FilterRowGroup are filtered. Is there any error I might have missed?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [Core] Optimize Parquet Reading by Enhancing Predicate Pushdown to the Page Level [paimon]
Posted by "JingsongLi (via GitHub)" <gi...@apache.org>.
JingsongLi closed pull request #4587: [Core] Optimize Parquet Reading by Enhancing Predicate Pushdown to the Page Level
URL: https://github.com/apache/paimon/pull/4587
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [Core] Optimize Parquet Reading by Enhancing Predicate Pushdown to the Page Level [paimon]
Posted by "Aiden-Dong (via GitHub)" <gi...@apache.org>.
Aiden-Dong commented on PR #4587:
URL: https://github.com/apache/paimon/pull/4587#issuecomment-2499552135
> See implementation in Spark.
ok tks, I'll go check how parquet is implemented when reading in Spark.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [Core] Optimize Parquet Reading by Enhancing Predicate Pushdown to the Page Level [paimon]
Posted by "JingsongLi (via GitHub)" <gi...@apache.org>.
JingsongLi commented on PR #4587:
URL: https://github.com/apache/paimon/pull/4587#issuecomment-2499534087
See implementation in Spark.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [Core] Optimize Parquet Reading by Enhancing Predicate Pushdown to the Page Level [paimon]
Posted by "JingsongLi (via GitHub)" <gi...@apache.org>.
JingsongLi commented on PR #4587:
URL: https://github.com/apache/paimon/pull/4587#issuecomment-2499534533
Close this now, feel free to re-open if you have more questions.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Optimize Parquet Reading by Enhancing Predicate Pushdown to the Page Level [paimon]
Posted by "JingsongLi (via GitHub)" <gi...@apache.org>.
JingsongLi commented on PR #4587:
URL: https://github.com/apache/paimon/pull/4587#issuecomment-2497532407
This modification is not that simple, it requires us to modify the column reader.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [Core] Optimize Parquet Reading by Enhancing Predicate Pushdown to the Page Level [paimon]
Posted by "Aiden-Dong (via GitHub)" <gi...@apache.org>.
Aiden-Dong commented on PR #4587:
URL: https://github.com/apache/paimon/pull/4587#issuecomment-2497566339
In my local tests, I found that with 4 million rows and 3 columns (3 files), after 30 random reads, the time dropped from 14s to 1.9s, and the correct results were returned.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [Core] Optimize Parquet Reading by Enhancing Predicate Pushdown to the Page Level [paimon]
Posted by "JingsongLi (via GitHub)" <gi...@apache.org>.
JingsongLi commented on PR #4587:
URL: https://github.com/apache/paimon/pull/4587#issuecomment-2499533871
https://github.com/apache/paimon/pull/3610
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org