You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@arrow.apache.org by "alamb (via GitHub)" <gi...@apache.org> on 2023/10/02 14:26:17 UTC
[PR] Rename `bounded_order_preserving_variants` config to `prefer_exising_sort` and update docs [arrow-datafusion]
alamb opened a new pull request, #7723:
URL: https://github.com/apache/arrow-datafusion/pull/7723
## Which issue does this PR close?
closes https://github.com/apache/arrow-datafusion/issues/7722
## Rationale for this change
While debugging an issue upgrading our code to use DataFuson, @ozankabak pointed me at the following config: https://github.com/apache/arrow-datafusion/pull/7671#discussion_r1341636365
This setting (I think) controls if the DataFusion planner should prefer using the existing sort order or trying to maximize paralleilsm using repartition and re-sorting
It turns out to be the right one, but I don't think I would have found it without @ozankabak 's suggestion
I think the core of my challenge is that it more describes how it modifies DataFusion's algorithms rather than what effect it has on the plans
## What changes are included in this PR?
I propose to change the config to `prefer_existing_sort` and update the documentation
## Are these changes tested?
existing tests
<!--
We typically require tests for all PRs in order to:
1. Prevent the code from being accidentally broken by subsequent changes
2. Serve as another way to document the expected behavior of the code
If tests are not included in your PR, please explain why (for example, are they covered by existing tests)?
-->
## Are there any user-facing changes?
yes, a config setting has a different name (and this is a breaking API change)
<!--
If there are user-facing changes then we may require documentation to be updated before approving the PR.
-->
<!--
If there are any breaking changes to public APIs, please add the `api change` label.
-->
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Rename `bounded_order_preserving_variants` config to `prefer_exising_sort` and update docs [arrow-datafusion]
Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on code in PR #7723:
URL: https://github.com/apache/arrow-datafusion/pull/7723#discussion_r1342995326
##########
datafusion/common/src/config.rs:
##########
@@ -453,11 +453,13 @@ config_namespace! {
/// ```
pub repartition_sorts: bool, default = true
- /// When true, DataFusion will opportunistically remove sorts by replacing
- /// `RepartitionExec` with `SortPreservingRepartitionExec`, and
+ /// When true, DataFusion will opportunistically remove sorts when the data is already sorted,
+ /// replacing `RepartitionExec` with `SortPreservingRepartitionExec`, and
Review Comment:
That makes sense, thank you @viirya -- I tried to improve the wording in 35c674852b. Let me know what you think
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Rename `bounded_order_preserving_variants` config to `prefer_exising_sort` and update docs [arrow-datafusion]
Posted by "viirya (via GitHub)" <gi...@apache.org>.
viirya commented on code in PR #7723:
URL: https://github.com/apache/arrow-datafusion/pull/7723#discussion_r1342916873
##########
docs/source/user-guide/configs.md:
##########
@@ -87,7 +87,7 @@ Environment variables are read during `SessionConfig` initialisation so they mus
| datafusion.optimizer.repartition_file_scans | true | When set to `true`, file groups will be repartitioned to achieve maximum parallelism. Currently Parquet and CSV formats are supported. If set to `true`, all files will be repartitioned evenly (i.e., a single large file might be partitioned into smaller chunks) for parallel scanning. If set to `false`, different files will be read in parallel, but repartitioning won't happen within a single file. |
| datafusion.optimizer.repartition_windows | true | Should DataFusion repartition data using the partitions keys to execute window functions in parallel using the provided `target_partitions` level |
| datafusion.optimizer.repartition_sorts | true | Should DataFusion execute sorts in a per-partition fashion and merge afterwards instead of coalescing first and sorting globally. With this flag is enabled, plans in the form below `text "SortExec: [a@0 ASC]", " CoalescePartitionsExec", " RepartitionExec: partitioning=RoundRobinBatch(8), input_partitions=1", ` would turn into the plan below which performs better in multithreaded environments `text "SortPreservingMergeExec: [a@0 ASC]", " SortExec: [a@0 ASC]", " RepartitionExec: partitioning=RoundRobinBatch(8), input_partitions=1", ` |
-| datafusion.optimizer.bounded_order_preserving_variants | false | When true, DataFusion will opportunistically remove sorts by replacing `RepartitionExec` with `SortPreservingRepartitionExec`, and `CoalescePartitionsExec` with `SortPreservingMergeExec`, even when the query is bounded. |
+| datafusion.optimizer.prefer_existing_sort | false | When true, DataFusion will opportunistically remove sorts when the data is already sorted, replacing `RepartitionExec` with `SortPreservingRepartitionExec`, and `CoalescePartitionsExec` with `SortPreservingMergeExec`, When false, DataFusion will prefer to maximize the parallelism using `Repartition/Coalesce` and resort the data subsequently with `SortExec` |
Review Comment:
```suggestion
| datafusion.optimizer.prefer_existing_sort | false | When true, DataFusion will opportunistically remove sorts when the data is already sorted, replacing `RepartitionExec` with `SortPreservingRepartitionExec`, and `CoalescePartitionsExec` with `SortPreservingMergeExec`. When false, DataFusion will prefer to maximize the parallelism using `Repartition/Coalesce` and resort the data subsequently with `SortExec` |
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Rename `bounded_order_preserving_variants` config to `prefer_exising_sort` and update docs [arrow-datafusion]
Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb merged PR #7723:
URL: https://github.com/apache/arrow-datafusion/pull/7723
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Rename `bounded_order_preserving_variants` config to `prefer_exising_sort` and update docs [arrow-datafusion]
Posted by "viirya (via GitHub)" <gi...@apache.org>.
viirya commented on code in PR #7723:
URL: https://github.com/apache/arrow-datafusion/pull/7723#discussion_r1342998378
##########
datafusion/common/src/config.rs:
##########
@@ -453,11 +453,13 @@ config_namespace! {
/// ```
pub repartition_sorts: bool, default = true
- /// When true, DataFusion will opportunistically remove sorts by replacing
- /// `RepartitionExec` with `SortPreservingRepartitionExec`, and
+ /// When true, DataFusion will opportunistically remove sorts when the data is already sorted,
+ /// replacing `RepartitionExec` with `SortPreservingRepartitionExec`, and
Review Comment:
Looks good to me. Thanks @alamb
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Rename `bounded_order_preserving_variants` config to `prefer_exising_sort` and update docs [arrow-datafusion]
Posted by "viirya (via GitHub)" <gi...@apache.org>.
viirya commented on code in PR #7723:
URL: https://github.com/apache/arrow-datafusion/pull/7723#discussion_r1342916873
##########
docs/source/user-guide/configs.md:
##########
@@ -87,7 +87,7 @@ Environment variables are read during `SessionConfig` initialisation so they mus
| datafusion.optimizer.repartition_file_scans | true | When set to `true`, file groups will be repartitioned to achieve maximum parallelism. Currently Parquet and CSV formats are supported. If set to `true`, all files will be repartitioned evenly (i.e., a single large file might be partitioned into smaller chunks) for parallel scanning. If set to `false`, different files will be read in parallel, but repartitioning won't happen within a single file. |
| datafusion.optimizer.repartition_windows | true | Should DataFusion repartition data using the partitions keys to execute window functions in parallel using the provided `target_partitions` level |
| datafusion.optimizer.repartition_sorts | true | Should DataFusion execute sorts in a per-partition fashion and merge afterwards instead of coalescing first and sorting globally. With this flag is enabled, plans in the form below `text "SortExec: [a@0 ASC]", " CoalescePartitionsExec", " RepartitionExec: partitioning=RoundRobinBatch(8), input_partitions=1", ` would turn into the plan below which performs better in multithreaded environments `text "SortPreservingMergeExec: [a@0 ASC]", " SortExec: [a@0 ASC]", " RepartitionExec: partitioning=RoundRobinBatch(8), input_partitions=1", ` |
-| datafusion.optimizer.bounded_order_preserving_variants | false | When true, DataFusion will opportunistically remove sorts by replacing `RepartitionExec` with `SortPreservingRepartitionExec`, and `CoalescePartitionsExec` with `SortPreservingMergeExec`, even when the query is bounded. |
+| datafusion.optimizer.prefer_existing_sort | false | When true, DataFusion will opportunistically remove sorts when the data is already sorted, replacing `RepartitionExec` with `SortPreservingRepartitionExec`, and `CoalescePartitionsExec` with `SortPreservingMergeExec`, When false, DataFusion will prefer to maximize the parallelism using `Repartition/Coalesce` and resort the data subsequently with `SortExec` |
Review Comment:
```suggestion
| datafusion.optimizer.prefer_existing_sort | false | When true, DataFusion will opportunistically remove sorts when the data is already sorted, replacing `RepartitionExec` with `SortPreservingRepartitionExec` (i.e., `RepartitionExec` with `preserve_order` as true), and `CoalescePartitionsExec` with `SortPreservingMergeExec`. When false, DataFusion will prefer to maximize the parallelism using `Repartition/Coalesce` and resort the data subsequently with `SortExec` |
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Rename `bounded_order_preserving_variants` config to `prefer_exising_sort` and update docs [arrow-datafusion]
Posted by "viirya (via GitHub)" <gi...@apache.org>.
viirya commented on code in PR #7723:
URL: https://github.com/apache/arrow-datafusion/pull/7723#discussion_r1342925177
##########
datafusion/common/src/config.rs:
##########
@@ -453,11 +453,13 @@ config_namespace! {
/// ```
pub repartition_sorts: bool, default = true
- /// When true, DataFusion will opportunistically remove sorts by replacing
- /// `RepartitionExec` with `SortPreservingRepartitionExec`, and
+ /// When true, DataFusion will opportunistically remove sorts when the data is already sorted,
+ /// replacing `RepartitionExec` with `SortPreservingRepartitionExec`, and
Review Comment:
Hmm, actually there is no `SortPreservingRepartitionExec` operator but it is a variant of `RepartitionExec` with `preserve_order` as true. It is a little confusion at first if trying to look for `SortPreservingRepartitionExec` type in IDE.
Maybe:
```suggestion
/// replacing `RepartitionExec` with `SortPreservingRepartitionExec` (i.e., `RepartitionExec` with `preserve_order` as true), and
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org