You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@arrow.apache.org by GitBox <gi...@apache.org> on 2022/11/18 11:41:49 UTC
[GitHub] [arrow-datafusion] alamb opened a new pull request, #4279: Push down more predicates into parquet exec
alamb opened a new pull request, #4279:
URL: https://github.com/apache/arrow-datafusion/pull/4279
Draft as it builds on https://github.com/apache/arrow-datafusion/pull/4278/files
# Which issue does this PR close?
Closes https://github.com/apache/arrow-datafusion/issues/4020
connects to #3463
# Rationale for this change
Pruning predicates are for eliminating row groups or pages based on min/max values. Only some predicates can be converted into Pruning Predicates.
However, the only predicates that the parquet exec will attempt to push down are the predicates that can be pruning predicates. It should attempt to push down all of them.
# What changes are included in this PR?
Pass entire pushed down predicate (not just the one attached to pruning predicate)
# Are these changes tested?
Yes
# Are there any user-facing changes?
More predicates can be pushed as row filters
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow-datafusion] alamb merged pull request #4279: Push down more predicates into `ParquetExec`
Posted by GitBox <gi...@apache.org>.
alamb merged PR #4279:
URL: https://github.com/apache/arrow-datafusion/pull/4279
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow-datafusion] Ted-Jiang commented on a diff in pull request #4279: Push down more predicates into `ParquetExec`
Posted by GitBox <gi...@apache.org>.
Ted-Jiang commented on code in PR #4279:
URL: https://github.com/apache/arrow-datafusion/pull/4279#discussion_r1027503408
##########
datafusion/core/src/physical_plan/file_format/parquet.rs:
##########
@@ -1564,6 +1575,9 @@ mod tests {
&display,
"pruning_predicate=c1_min@0 != bar OR bar != c1_max@1"
);
+
+ assert_contains!(&display, r#"predicate=c1 != Utf8("bar")"#);
Review Comment:
👍
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow-datafusion] alamb commented on a diff in pull request #4279: Push down more predicates into `ParquetExec`
Posted by GitBox <gi...@apache.org>.
alamb commented on code in PR #4279:
URL: https://github.com/apache/arrow-datafusion/pull/4279#discussion_r1026629362
##########
datafusion/core/src/physical_plan/file_format/parquet.rs:
##########
@@ -98,6 +100,7 @@ impl ParquetExec {
MetricBuilder::new(&metrics).global_counter("num_predicate_creation_errors");
let pruning_predicate = predicate
+ .clone()
Review Comment:
the key difference here is that the original predicate is kept, even if the creation of the pruning predicate fails
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow-datafusion] alamb commented on a diff in pull request #4279: Push down more predicates into `ParquetExec`
Posted by GitBox <gi...@apache.org>.
alamb commented on code in PR #4279:
URL: https://github.com/apache/arrow-datafusion/pull/4279#discussion_r1026629713
##########
datafusion/core/src/physical_plan/file_format/parquet.rs:
##########
@@ -312,6 +317,12 @@ impl ExecutionPlan for ParquetExec {
) -> std::fmt::Result {
match t {
DisplayFormatType::Default => {
+ let predicate_string = self
Review Comment:
The predicate is now explicitly shown as well in EXPLAIN (as it is difference)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow-datafusion] alamb commented on a diff in pull request #4279: Push down more predicates into `ParquetExec`
Posted by GitBox <gi...@apache.org>.
alamb commented on code in PR #4279:
URL: https://github.com/apache/arrow-datafusion/pull/4279#discussion_r1026452357
##########
datafusion/core/src/physical_plan/file_format/parquet.rs:
##########
@@ -284,6 +288,7 @@ impl ExecutionPlan for ParquetExec {
partition_index,
projection: Arc::from(projection),
batch_size: ctx.session_config().batch_size(),
+ predicate: self.predicate.clone(),
Review Comment:
I don't like all the cloning -- I have a follow on PR (https://github.com/apache/arrow-datafusion/pull/4283) to stop the copying
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow-datafusion] ursabot commented on pull request #4279: Push down more predicates into `ParquetExec`
Posted by GitBox <gi...@apache.org>.
ursabot commented on PR #4279:
URL: https://github.com/apache/arrow-datafusion/pull/4279#issuecomment-1322417377
Benchmark runs are scheduled for baseline = bcd624855778384ee27648161de73951e3fb6ea1 and contender = afe2333a75d2db27518b5bf381b4191d534b9f0e. afe2333a75d2db27518b5bf381b4191d534b9f0e is a master commit associated with this PR. Results will be available as each benchmark for each run completes.
Conbench compare runs links:
[Skipped :warning: Benchmarking of arrow-datafusion-commits is not supported on ec2-t3-xlarge-us-east-2] [ec2-t3-xlarge-us-east-2](https://conbench.ursa.dev/compare/runs/fa5bc2b10c1d494486182cce93331a62...de46f38d97384040bcf8c1e8b538c26a/)
[Skipped :warning: Benchmarking of arrow-datafusion-commits is not supported on test-mac-arm] [test-mac-arm](https://conbench.ursa.dev/compare/runs/7adf1369231049f7aa698fa4e2d01b74...b82c552018694ad9ae165274e6041999/)
[Skipped :warning: Benchmarking of arrow-datafusion-commits is not supported on ursa-i9-9960x] [ursa-i9-9960x](https://conbench.ursa.dev/compare/runs/520496285c604505b5506017b61e5b9e...469318e43a3d44b49846dffd7d04915e/)
[Skipped :warning: Benchmarking of arrow-datafusion-commits is not supported on ursa-thinkcentre-m75q] [ursa-thinkcentre-m75q](https://conbench.ursa.dev/compare/runs/64c0bb2c45f44f28bcdc7c352bc77753...7f873c702e19452ab8be74a39fff9149/)
Buildkite builds:
Supported benchmarks:
ec2-t3-xlarge-us-east-2: Supported benchmark langs: Python, R. Runs only benchmarks with cloud = True
test-mac-arm: Supported benchmark langs: C++, Python, R
ursa-i9-9960x: Supported benchmark langs: Python, R, JavaScript
ursa-thinkcentre-m75q: Supported benchmark langs: C++, Java
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org