You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@arrow.apache.org by GitBox <gi...@apache.org> on 2022/11/09 22:18:08 UTC
[GitHub] [arrow-datafusion] alamb opened a new pull request, #4160: Minor: Extract parquet row group pruning code into its own module
alamb opened a new pull request, #4160:
URL: https://github.com/apache/arrow-datafusion/pull/4160
# Which issue does this PR close?
re #3463
# Rationale for this change
This makes the row group pruning code match the rest of the pushdown rules in DataFusion (filtering and page indexing)
# What changes are included in this PR?
Move a bunch of code out of the parquet.rs module into `row_groups.rs`
# Are these changes tested?
Covered by existing tests
# Are there any user-facing changes?
No it just moves code around. No functional change is intended
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow-datafusion] ursabot commented on pull request #4160: Minor: Extract parquet row group pruning code into its own module
Posted by GitBox <gi...@apache.org>.
ursabot commented on PR #4160:
URL: https://github.com/apache/arrow-datafusion/pull/4160#issuecomment-1310225315
Benchmark runs are scheduled for baseline = 9c24a79118c0ae0aee480bf039385a06d240b499 and contender = 4d621c5c0da387f50c9ab6f3a9ee568062ee9976. 4d621c5c0da387f50c9ab6f3a9ee568062ee9976 is a master commit associated with this PR. Results will be available as each benchmark for each run completes.
Conbench compare runs links:
[Skipped :warning: Benchmarking of arrow-datafusion-commits is not supported on ec2-t3-xlarge-us-east-2] [ec2-t3-xlarge-us-east-2](https://conbench.ursa.dev/compare/runs/861a68dce4ea47a8a4cbbbc7cf6f03b9...f2716ddf4ff0476898e5c965b40d34be/)
[Skipped :warning: Benchmarking of arrow-datafusion-commits is not supported on test-mac-arm] [test-mac-arm](https://conbench.ursa.dev/compare/runs/b7a00cc334e149dab4908c08da02b64f...c73d73eb90fe4db0ae93dcb95e6a736e/)
[Skipped :warning: Benchmarking of arrow-datafusion-commits is not supported on ursa-i9-9960x] [ursa-i9-9960x](https://conbench.ursa.dev/compare/runs/410f7b1a7e8147c7a029fa2b28439889...f860fd6003bd4a58972f2584067bc010/)
[Skipped :warning: Benchmarking of arrow-datafusion-commits is not supported on ursa-thinkcentre-m75q] [ursa-thinkcentre-m75q](https://conbench.ursa.dev/compare/runs/9d9d48c57a924b659662f1cf3a39e28f...dc2664e52f4045bdb38363ee07654df3/)
Buildkite builds:
Supported benchmarks:
ec2-t3-xlarge-us-east-2: Supported benchmark langs: Python, R. Runs only benchmarks with cloud = True
test-mac-arm: Supported benchmark langs: C++, Python, R
ursa-i9-9960x: Supported benchmark langs: Python, R, JavaScript
ursa-thinkcentre-m75q: Supported benchmark langs: C++, Java
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow-datafusion] alamb commented on a diff in pull request #4160: Minor: Extract parquet row group pruning code into its own module
Posted by GitBox <gi...@apache.org>.
alamb commented on code in PR #4160:
URL: https://github.com/apache/arrow-datafusion/pull/4160#discussion_r1019043763
##########
datafusion/core/src/physical_plan/file_format/parquet.rs:
##########
@@ -17,43 +17,36 @@
//! Execution plan for reading Parquet files
+use arrow::datatypes::SchemaRef;
use fmt::Debug;
Review Comment:
I also hope that some day I can split more of this code (file_format) into its own crate -- but one thing at a time !
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow-datafusion] Ted-Jiang commented on a diff in pull request #4160: Minor: Extract parquet row group pruning code into its own module
Posted by GitBox <gi...@apache.org>.
Ted-Jiang commented on code in PR #4160:
URL: https://github.com/apache/arrow-datafusion/pull/4160#discussion_r1018603811
##########
datafusion/core/src/physical_plan/file_format/parquet.rs:
##########
@@ -17,43 +17,36 @@
//! Execution plan for reading Parquet files
+use arrow::datatypes::SchemaRef;
use fmt::Debug;
Review Comment:
Great! before you split. I open this file in Clion it always lagging 😂. Thanks for your work @alamb 😄
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow-datafusion] alamb commented on a diff in pull request #4160: Minor: Extract parquet row group pruning code into its own module
Posted by GitBox <gi...@apache.org>.
alamb commented on code in PR #4160:
URL: https://github.com/apache/arrow-datafusion/pull/4160#discussion_r1018467749
##########
datafusion/core/src/physical_plan/file_format/parquet.rs:
##########
@@ -435,7 +423,7 @@ impl FileOpener for ParquetOpener {
// Row group pruning: attempt to skip entire row_groups
// using metadata on the row groups
let file_metadata = builder.metadata();
- let row_groups = prune_row_groups(
+ let row_groups = row_groups::prune_row_groups(
Review Comment:
The whole point of the PR is to move this function (and its tests) into its own module
##########
datafusion/core/src/physical_plan/file_format/parquet.rs:
##########
@@ -17,43 +17,36 @@
//! Execution plan for reading Parquet files
+use arrow::datatypes::SchemaRef;
use fmt::Debug;
Review Comment:
this file is still pretty massive, but after this PR it is down to under 2000 lines so that is progress I think
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow-datafusion] Ted-Jiang commented on a diff in pull request #4160: Minor: Extract parquet row group pruning code into its own module
Posted by GitBox <gi...@apache.org>.
Ted-Jiang commented on code in PR #4160:
URL: https://github.com/apache/arrow-datafusion/pull/4160#discussion_r1018604494
##########
datafusion/core/src/physical_plan/file_format/parquet/row_groups.rs:
##########
@@ -0,0 +1,744 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+use arrow::{
+ array::ArrayRef,
+ datatypes::{DataType, Schema},
+};
+use datafusion_common::Column;
+use datafusion_common::ScalarValue;
+use log::debug;
+
+use parquet::{
+ file::{metadata::RowGroupMetaData, statistics::Statistics as ParquetStatistics},
+ schema::types::ColumnDescriptor,
+};
+
+use crate::{
+ datasource::listing::FileRange,
+ physical_optimizer::pruning::{PruningPredicate, PruningStatistics},
+};
+use parquet::basic::{ConvertedType, LogicalType};
+
+use super::ParquetFileMetrics;
+
+pub(crate) fn prune_row_groups(
+ groups: &[RowGroupMetaData],
+ range: Option<FileRange>,
+ predicate: Option<PruningPredicate>,
+ metrics: &ParquetFileMetrics,
+) -> Vec<usize> {
+ // TODO: Columnar pruning
Review Comment:
what here `Columnar pruning` means 🤔
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow-datafusion] alamb commented on a diff in pull request #4160: Minor: Extract parquet row group pruning code into its own module
Posted by GitBox <gi...@apache.org>.
alamb commented on code in PR #4160:
URL: https://github.com/apache/arrow-datafusion/pull/4160#discussion_r1019044002
##########
datafusion/core/src/physical_plan/file_format/parquet/row_groups.rs:
##########
@@ -0,0 +1,744 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+use arrow::{
+ array::ArrayRef,
+ datatypes::{DataType, Schema},
+};
+use datafusion_common::Column;
+use datafusion_common::ScalarValue;
+use log::debug;
+
+use parquet::{
+ file::{metadata::RowGroupMetaData, statistics::Statistics as ParquetStatistics},
+ schema::types::ColumnDescriptor,
+};
+
+use crate::{
+ datasource::listing::FileRange,
+ physical_optimizer::pruning::{PruningPredicate, PruningStatistics},
+};
+use parquet::basic::{ConvertedType, LogicalType};
+
+use super::ParquetFileMetrics;
+
+pub(crate) fn prune_row_groups(
+ groups: &[RowGroupMetaData],
+ range: Option<FileRange>,
+ predicate: Option<PruningPredicate>,
+ metrics: &ParquetFileMetrics,
+) -> Vec<usize> {
+ // TODO: Columnar pruning
Review Comment:
I think it means the page index pruning you have implemented -- I 'll remove this comment in a follow on PR
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow-datafusion] alamb merged pull request #4160: Minor: Extract parquet row group pruning code into its own module
Posted by GitBox <gi...@apache.org>.
alamb merged PR #4160:
URL: https://github.com/apache/arrow-datafusion/pull/4160
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org