You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@arrow.apache.org by GitBox <gi...@apache.org> on 2022/03/11 08:55:07 UTC
[GitHub] [arrow-datafusion] yjshen opened a new pull request #1990: WIP: Finer-grained parallelism for Parquet Scan
yjshen opened a new pull request #1990:
URL: https://github.com/apache/arrow-datafusion/pull/1990
# Which issue does this PR close?
Part of #944
It's currently and WIP since it relies on a new Arrow release. To be specific, it needs https://github.com/apache/arrow-rs/commit/2bca71e322fcab6c6d93a47ef71638a617e29f6c
# Rationale for this change
<!--
Why are you proposing this change? If this is already explained clearly in the issue then this section is not needed.
Explaining clearly why changes are proposed helps reviewers understand your changes and offer better suggestions for fixes.
-->
An easier and more feasible way to parallelize parquet file scan.
By introducing a new configuration `max_bytes_per_partition` (TODO), we could generate partitions each of desirable size. And likely to scan only part of the row groups from a big parquet file.
The file range is passed down to the `parquet` crate and filter row groups according to its position in the parquet file.
# What changes are included in this PR?
<!--
There is no need to duplicate the description in the issue here but it is sometimes worth providing a summary of the individual changes in this PR.
-->
# Are there any user-facing changes?
<!--
If there are user-facing changes then we may require documentation to be updated before approving the PR.
-->
Yes, a new configuration to set desirable scan partition size.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow-datafusion] alamb commented on pull request #1990: WIP: Finer-grained parallelism for Parquet Scan
Posted by GitBox <gi...@apache.org>.
alamb commented on pull request #1990:
URL: https://github.com/apache/arrow-datafusion/pull/1990#issuecomment-1065860000
cc @tustvold
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow-datafusion] tustvold commented on pull request #1990: WIP: Finer-grained parallelism for Parquet Scan
Posted by GitBox <gi...@apache.org>.
tustvold commented on pull request #1990:
URL: https://github.com/apache/arrow-datafusion/pull/1990#issuecomment-1065860904
I'm not sure I follow how this will work, parquet files have a block structure internally that is not amenable to seeking. Particularly with RLE data, it is common for a column chunk to consist of a single page. Could you maybe expand a bit on this?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow-datafusion] liukun4515 commented on pull request #1990: WIP: Finer-grained parallelism for Parquet Scan
Posted by GitBox <gi...@apache.org>.
liukun4515 commented on pull request #1990:
URL: https://github.com/apache/arrow-datafusion/pull/1990#issuecomment-1066780774
@yjshen I am very interested in discussing and participating in the parallelism of physical execution.
Oracle has a great feature about parallelism which is called parallel execution.
In the oracle, we can use different dop(degree of parallelism) by hint or other configuration.
I have some questions about this task.
1. how to determine the dop for the query
2. how to determine the `size` of each task
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow-datafusion] alamb commented on pull request #1990: WIP: Finer-grained parallelism for Parquet Scan
Posted by GitBox <gi...@apache.org>.
alamb commented on pull request #1990:
URL: https://github.com/apache/arrow-datafusion/pull/1990#issuecomment-1073220368
@liukun4515 there are already some configuration settings related to parallelism
`target_partitions` (like degree of parallelism)
https://github.com/apache/arrow-datafusion/blob/4994eda81c2280fa78aea1ae0d92ce918947eebd/datafusion/src/execution/context.rs#L822-L823
`batch_size` (that controls how many batches are processed)
https://github.com/apache/arrow-datafusion/blob/4994eda81c2280fa78aea1ae0d92ce918947eebd/datafusion/src/execution/context.rs#L909-L914
I am not sure how well these two parameters are respected in all DataFusion operators, but I think the configurations settings are reasonable
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow-datafusion] yjshen commented on pull request #1990: WIP: Finer-grained parallelism for Parquet Scan
Posted by GitBox <gi...@apache.org>.
yjshen commented on pull request #1990:
URL: https://github.com/apache/arrow-datafusion/pull/1990#issuecomment-1065864478
Hi @tustvold , the filter is based on row-group midpoint position. It was introduced recently in parquet crate with https://github.com/apache/arrow-rs/commit/2bca71e322fcab6c6d93a47ef71638a617e29f6c. The midpoint filtering is modeled after the [ParquetSplit](https://github.com/apache/parquet-mr/blob/master/parquet-hadoop/src/main/java/org/apache/parquet/hadoop/ParquetInputSplit.java#L67-L91) and [MetadataConverter](https://github.com/apache/parquet-mr/blob/master/parquet-hadoop/src/main/java/org/apache/parquet/format/converter/ParquetMetadataConverter.java#L1241-L1292)
The parquet row groups level parallelism is used in MapReduce and Spark. In Spark [`splitFiles`](https://github.com/apache/spark/blob/master/sql/core/src/main/scala/org/apache/spark/sql/execution/PartitionedFileUtil.scala#L26-L45) is used to generate task partitions based on partition size settings. And it may partition bigger parquet file parts to different partitions.
Currently, this PR is still WIP, since only physical plan changes are implemented. And we translate Spark physical plan to DataFusion physical plan to run natively in DataFusion https://github.com/blaze-init/spark-blaze-extension/blob/master/src/main/scala/org/apache/spark/sql/blaze/plan/NativeParquetScanExec.scala#L57-L63
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow-datafusion] tustvold commented on pull request #1990: WIP: Finer-grained parallelism for Parquet Scan
Posted by GitBox <gi...@apache.org>.
tustvold commented on pull request #1990:
URL: https://github.com/apache/arrow-datafusion/pull/1990#issuecomment-1065867378
Oh I think I misunderstood, this is using byte ranges to filter row groups, not to filter within row groups?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow-datafusion] tustvold edited a comment on pull request #1990: WIP: Finer-grained parallelism for Parquet Scan
Posted by GitBox <gi...@apache.org>.
tustvold edited a comment on pull request #1990:
URL: https://github.com/apache/arrow-datafusion/pull/1990#issuecomment-1065867378
Oh I think I misunderstood, this is using byte ranges to filter the row groups to scan, not to filter the rows within the row groups? That makes sense, and sounds like a useful addition 👍
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow-datafusion] yjshen commented on pull request #1990: WIP: Finer-grained parallelism for Parquet Scan
Posted by GitBox <gi...@apache.org>.
yjshen commented on pull request #1990:
URL: https://github.com/apache/arrow-datafusion/pull/1990#issuecomment-1065867712
Yes, to filter row groups, based on RowGroup Metadata as well.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow-datafusion] tustvold edited a comment on pull request #1990: WIP: Finer-grained parallelism for Parquet Scan
Posted by GitBox <gi...@apache.org>.
tustvold edited a comment on pull request #1990:
URL: https://github.com/apache/arrow-datafusion/pull/1990#issuecomment-1065860904
I'm not sure I follow how this will work, parquet files have a block structure internally that is not amenable to seeking. Particularly with RLE data, it is common for a column chunk to consist of a single page. Could you maybe expand a bit on this?
On a more holistic level, is there some prior art on parellelising parquet reads, I've only ever encountered file and rarely row group parallelism...
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org