You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@arrow.apache.org by GitBox <gi...@apache.org> on 2022/11/04 10:09:28 UTC
[GitHub] [arrow-datafusion] Ted-Jiang opened a new pull request, #4105: Add statistics for parquet page level skipping
Ted-Jiang opened a new pull request, #4105:
URL: https://github.com/apache/arrow-datafusion/pull/4105
Signed-off-by: yangjiang <ya...@ebay.com>
# Which issue does this PR close?
Closes #4086 .
# Rationale for this change
we can get metric like
```
ParquetExec: limit=None, partitions=[Users/yangjiang/test-data/1g_tpch_pageIndex/lineitem/part-00000-7d2abab2-a301-4452-9f1d-c641e7f15af4-c000.snappy.parquet],
predicate=l_shipdate_min@0 <= 8037, projection=[l_quantity, l_extendedprice, l_discount, l_tax, l_returnflag, l_linestatus, l_shipdate], metrics=[output_rows=1020000, elapsed_compute=1ns, spill_count=0, spilled_bytes=0, mem_used=0,
pushdown_rows_filtered{filename=Users/yangjiang/test-data/1g_tpch_pageIndex/lineitem/part-00000-7d2abab2-a301-4452-9f1d-c641e7f15af4-c000.snappy.parquet}=0,
page_index_rows_filtered{filename=Users/yangjiang/test-data/1g_tpch_pageIndex/lineitem/part-00000-7d2abab2-a301-4452-9f1d-c641e7f15af4-c000.snappy.parquet}=5000000, predicate_evaluation_errors{filename=Users/yangjiang/test-data/1g_tpch_pageIndex/lineitem/part-00000-7d2abab2-a301-4452-9f1d-c641e7f15af4-c000.snappy.parquet}=0, row_groups_pruned{filename=Users/yangjiang/test-data/1g_tpch_pageIndex/lineitem/part-00000-7d2abab2-a301-4452-9f1d-c641e7f15af4-c000.snappy.parquet}=0, bytes_scanned{filename=Users/yangjiang/test-data/1g_tpch_pageIndex/lineitem/part-00000-7d2abab2-a301-4452-9f1d-c641e7f15af4-c000.snappy.parquet}=13198873, num_predicate_creation_errors=0, time_elapsed_scanning=114.532731ms, time_elapsed_processing=3.561757295s, pushdown_eval_time{filename=Users/yangjiang/test-data/1g_tpch_pageIndex/lineitem/part-00000-7d2abab2-a301-4452-9f1d-c641e7f15af4-c000.snappy.parquet}=2ns,
page_index_eval_time{filename=Users/yangjiang/test-data/1g_tpch_pageIndex/lineitem/part-00000-7d2abab2-a301-4452-9f1d-c641e7f15af4-c000.snappy.parquet}=475.372µs, time_elapsed_opening=58.666766ms] |
```
# What changes are included in this PR?
<!--
There is no need to duplicate the description in the issue here but it is sometimes worth providing a summary of the individual changes in this PR.
-->
# Are there any user-facing changes?
<!--
If there are user-facing changes then we may require documentation to be updated before approving the PR.
-->
We i try to create single column with multi pages, i think there is a bug in page size check. but i found https://github.com/apache/arrow-rs/issues/2941 add page row number check(not release).
<!--
If there are any breaking changes to public APIs, please add the `api change` label.
-->
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow-datafusion] ursabot commented on pull request #4105: Add metrics for parquet page level skipping
Posted by GitBox <gi...@apache.org>.
ursabot commented on PR #4105:
URL: https://github.com/apache/arrow-datafusion/pull/4105#issuecomment-1305701783
Benchmark runs are scheduled for baseline = b7a33317c2abf265f4ab6b3fe636f87c4d01334c and contender = 4d23cae481c8e04fead1f1569dab0dc28b477008. 4d23cae481c8e04fead1f1569dab0dc28b477008 is a master commit associated with this PR. Results will be available as each benchmark for each run completes.
Conbench compare runs links:
[Skipped :warning: Benchmarking of arrow-datafusion-commits is not supported on ec2-t3-xlarge-us-east-2] [ec2-t3-xlarge-us-east-2](https://conbench.ursa.dev/compare/runs/bdc9db9a6d1a460887f02d26e7c6f4d0...9c5334d0f7c1447987c15ce8669d5abc/)
[Skipped :warning: Benchmarking of arrow-datafusion-commits is not supported on test-mac-arm] [test-mac-arm](https://conbench.ursa.dev/compare/runs/e51f18d23c8c406c9873542dade20d6a...a15f60020a434621b179f1a486118bbd/)
[Skipped :warning: Benchmarking of arrow-datafusion-commits is not supported on ursa-i9-9960x] [ursa-i9-9960x](https://conbench.ursa.dev/compare/runs/b09184714450467b97d93bde7b21c285...c4d5c9b4cf42452489f6a27dfdbf708b/)
[Skipped :warning: Benchmarking of arrow-datafusion-commits is not supported on ursa-thinkcentre-m75q] [ursa-thinkcentre-m75q](https://conbench.ursa.dev/compare/runs/77460c7c6e244a19ac99d841ffe2a1a0...b5f08757dc6843b4a9676699e3311160/)
Buildkite builds:
Supported benchmarks:
ec2-t3-xlarge-us-east-2: Supported benchmark langs: Python, R. Runs only benchmarks with cloud = True
test-mac-arm: Supported benchmark langs: C++, Python, R
ursa-i9-9960x: Supported benchmark langs: Python, R, JavaScript
ursa-thinkcentre-m75q: Supported benchmark langs: C++, Java
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow-datafusion] Ted-Jiang commented on pull request #4105: Add metrics for parquet page level skipping
Posted by GitBox <gi...@apache.org>.
Ted-Jiang commented on PR #4105:
URL: https://github.com/apache/arrow-datafusion/pull/4105#issuecomment-1304555628
@alamb fix ut in [use page row limit fix ut](https://github.com/apache/arrow-datafusion/pull/4105/commits/005b10140e0b2da6340d2070c0d6086b475a4f9e)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow-datafusion] Ted-Jiang commented on a diff in pull request #4105: Add statistics for parquet page level skipping
Posted by GitBox <gi...@apache.org>.
Ted-Jiang commented on code in PR #4105:
URL: https://github.com/apache/arrow-datafusion/pull/4105#discussion_r1013849704
##########
datafusion/core/src/physical_plan/file_format/parquet.rs:
##########
@@ -1695,6 +1709,35 @@ mod tests {
Ok(())
}
+ #[tokio::test]
+ async fn parquet_page_index_exec_metrics() {
+ let c1: ArrayRef = Arc::new(Int32Array::from(vec![Some(1), None, Some(2)]));
+ let c2: ArrayRef = Arc::new(Int32Array::from(vec![Some(3), Some(4), Some(5)]));
+ let batch1 = create_batch(vec![("int", c1.clone())]);
+ let batch2 = create_batch(vec![("int", c2.clone())]);
+
+ let filter = col("int").eq(lit(4_i32));
+
+ let rt =
+ round_trip(vec![batch1, batch2], None, None, Some(filter), false, true).await;
+
+ let metrics = rt.parquet_exec.metrics().unwrap();
+
+ // todo fix this https://github.com/apache/arrow-rs/issues/2941 release change to row limit.
Review Comment:
But from the real world file, this metic works fine😂
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow-datafusion] Ted-Jiang commented on a diff in pull request #4105: Add statistics for parquet page level skipping
Posted by GitBox <gi...@apache.org>.
Ted-Jiang commented on code in PR #4105:
URL: https://github.com/apache/arrow-datafusion/pull/4105#discussion_r1014052195
##########
datafusion/core/src/physical_plan/file_format/parquet.rs:
##########
@@ -449,7 +449,7 @@ impl FileOpener for ParquetOpener {
// page index pruning: if all data on individual pages can
// be ruled using page metadata, rows from other columns
// with that range can be skipped as well
- if let Some(row_selection) = enable_page_index
+ if let Some(row_selection) = (enable_page_index && !row_groups.is_empty())
Review Comment:
when all rowGroups are pruned by rg_metadata(min max), we do this skip fast
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow-datafusion] alamb commented on pull request #4105: Add metrics for parquet page level skipping
Posted by GitBox <gi...@apache.org>.
alamb commented on PR #4105:
URL: https://github.com/apache/arrow-datafusion/pull/4105#issuecomment-1305699547
Thanks a lot @Ted-Jiang
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow-datafusion] alamb merged pull request #4105: Add metrics for parquet page level skipping
Posted by GitBox <gi...@apache.org>.
alamb merged PR #4105:
URL: https://github.com/apache/arrow-datafusion/pull/4105
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow-datafusion] Ted-Jiang commented on a diff in pull request #4105: Add statistics for parquet page level skipping
Posted by GitBox <gi...@apache.org>.
Ted-Jiang commented on code in PR #4105:
URL: https://github.com/apache/arrow-datafusion/pull/4105#discussion_r1013849132
##########
datafusion/core/src/physical_plan/file_format/parquet.rs:
##########
@@ -1695,6 +1709,35 @@ mod tests {
Ok(())
}
+ #[tokio::test]
+ async fn parquet_page_index_exec_metrics() {
+ let c1: ArrayRef = Arc::new(Int32Array::from(vec![Some(1), None, Some(2)]));
+ let c2: ArrayRef = Arc::new(Int32Array::from(vec![Some(3), Some(4), Some(5)]));
+ let batch1 = create_batch(vec![("int", c1.clone())]);
+ let batch2 = create_batch(vec![("int", c2.clone())]);
+
+ let filter = col("int").eq(lit(4_i32));
+
+ let rt =
+ round_trip(vec![batch1, batch2], None, None, Some(filter), false, true).await;
+
+ let metrics = rt.parquet_exec.metrics().unwrap();
+
+ // todo fix this https://github.com/apache/arrow-rs/issues/2941 release change to row limit.
Review Comment:
https://github.com/apache/arrow-rs/pull/2942/files#r1013838557
I think theres is a bug in `should_add_data_page ` `self.encoder.num_values() ` always zero.
So no matter how to set `data_pagesize_limit ` and `write_batch_size` always return 1 page in on colunn chunk.
🤔 I think someone metion this before.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow-datafusion] alamb commented on a diff in pull request #4105: Add statistics for parquet page level skipping
Posted by GitBox <gi...@apache.org>.
alamb commented on code in PR #4105:
URL: https://github.com/apache/arrow-datafusion/pull/4105#discussion_r1014283770
##########
datafusion/core/src/physical_plan/file_format/parquet/metrics.rs:
##########
@@ -63,13 +67,22 @@ impl ParquetFileMetrics {
let pushdown_eval_time = MetricBuilder::new(metrics)
.with_new_label("filename", filename.to_string())
.subset_time("pushdown_eval_time", partition);
+ let page_index_rows_filtered = MetricBuilder::new(metrics)
Review Comment:
👍
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org