You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@arrow.apache.org by "tustvold (via GitHub)" <gi...@apache.org> on 2023/04/03 16:43:37 UTC
[GitHub] [arrow-datafusion] tustvold opened a new pull request, #5851: Use SortPreservingMerge for in memory sort
tustvold opened a new pull request, #5851:
URL: https://github.com/apache/arrow-datafusion/pull/5851
# Which issue does this PR close?
<!--
We generally require a GitHub issue to be filed for all bug fixes and enhancements and this helps us generate change logs for our releases. You can link an issue to this PR using the GitHub syntax. For example `Closes #123` indicates that this PR will close issue #123.
-->
Closes #.
# Rationale for this change
<!--
Why are you proposing this change? If this is already explained clearly in the issue then this section is not needed.
Explaining clearly why changes are proposed helps reviewers understand your changes and offer better suggestions for fixes.
-->
Investigating https://github.com/apache/arrow-datafusion/pull/5292 and thought I would try to break it into smaller pieces to make it easier to see what is going on. In particular this PR just switches `ExternalSorter` to using `SortPreservingMerge`.
This results in better performance, apart from in the case of a single sort column. This is not hugely surprising, as lexsort_to_indices has optimised kernels for when sorting by a single column.
```
Compiling datafusion v21.0.0 (/home/raphael/repos/external/arrow-datafusion/datafusion/core)
Finished release-nonlto [optimized] target(s) in 22.59s
Running benches/sort.rs (/home/raphael/repos/external/arrow-datafusion/target/release-nonlto/deps/sort-16eed48ca15554e0)
sort i64 time: [7.6330 ms 7.6510 ms 7.6683 ms]
change: [+42.367% +42.983% +43.644%] (p = 0.00 < 0.05)
Performance has regressed.
Found 3 outliers among 100 measurements (3.00%)
1 (1.00%) low severe
2 (2.00%) low mild
Benchmarking sort i64 preserve partitioning: Warming up for 3.0000 s
Warning: Unable to complete 100 samples in 5.0s. You may wish to increase target time to 9.3s, enable flat sampling, or reduce sample count to 50.
sort i64 preserve partitioning
time: [1.8267 ms 1.8308 ms 1.8348 ms]
change: [-5.5979% -5.0486% -4.4890%] (p = 0.00 < 0.05)
Performance has improved.
Found 5 outliers among 100 measurements (5.00%)
1 (1.00%) high mild
4 (4.00%) high severe
sort f64 time: [8.0650 ms 8.0818 ms 8.0983 ms]
change: [+27.214% +27.725% +28.212%] (p = 0.00 < 0.05)
Performance has regressed.
Found 7 outliers among 100 measurements (7.00%)
5 (5.00%) low mild
2 (2.00%) high mild
sort f64 preserve partitioning
time: [2.2920 ms 2.2987 ms 2.3053 ms]
change: [-4.0517% -3.4908% -3.0237%] (p = 0.00 < 0.05)
Performance has improved.
Found 12 outliers among 100 measurements (12.00%)
9 (9.00%) low mild
2 (2.00%) high mild
1 (1.00%) high severe
sort utf8 low cardinality
time: [8.4322 ms 8.4522 ms 8.4707 ms]
change: [-9.3807% -8.9870% -8.6128%] (p = 0.00 < 0.05)
Performance has improved.
Found 16 outliers among 100 measurements (16.00%)
7 (7.00%) low severe
2 (2.00%) low mild
7 (7.00%) high mild
sort utf8 low cardinality preserve partitioning
time: [4.5490 ms 4.5611 ms 4.5723 ms]
change: [-9.4824% -9.0677% -8.6905%] (p = 0.00 < 0.05)
Performance has improved.
Found 24 outliers among 100 measurements (24.00%)
11 (11.00%) low severe
7 (7.00%) low mild
5 (5.00%) high mild
1 (1.00%) high severe
sort utf8 high cardinality
time: [13.918 ms 13.945 ms 13.972 ms]
change: [-25.221% -24.973% -24.715%] (p = 0.00 < 0.05)
Performance has improved.
Found 7 outliers among 100 measurements (7.00%)
3 (3.00%) low severe
1 (1.00%) low mild
1 (1.00%) high mild
2 (2.00%) high severe
sort utf8 high cardinality preserve partitioning
time: [7.4030 ms 7.4190 ms 7.4343 ms]
change: [-9.2608% -8.8695% -8.4917%] (p = 0.00 < 0.05)
Performance has improved.
Found 8 outliers among 100 measurements (8.00%)
1 (1.00%) low severe
5 (5.00%) low mild
2 (2.00%) high mild
sort utf8 tuple time: [14.899 ms 14.924 ms 14.950 ms]
change: [-55.867% -55.718% -55.570%] (p = 0.00 < 0.05)
Performance has improved.
Found 6 outliers among 100 measurements (6.00%)
2 (2.00%) low mild
2 (2.00%) high mild
2 (2.00%) high severe
sort utf8 tuple preserve partitioning
time: [3.4615 ms 3.4720 ms 3.4857 ms]
change: [-18.031% -17.508% -17.055%] (p = 0.00 < 0.05)
Performance has improved.
Found 17 outliers among 100 measurements (17.00%)
4 (4.00%) low severe
9 (9.00%) low mild
3 (3.00%) high mild
1 (1.00%) high severe
sort utf8 dictionary time: [4.2060 ms 4.2164 ms 4.2269 ms]
change: [+106.17% +106.95% +107.78%] (p = 0.00 < 0.05)
Performance has regressed.
Found 3 outliers among 100 measurements (3.00%)
1 (1.00%) low mild
2 (2.00%) high mild
sort utf8 dictionary preserve partitioning
time: [666.64 µs 668.92 µs 671.60 µs]
change: [-16.850% -12.080% -9.1385%] (p = 0.00 < 0.05)
Performance has improved.
Found 4 outliers among 100 measurements (4.00%)
2 (2.00%) high mild
2 (2.00%) high severe
sort utf8 dictionary tuple
time: [10.505 ms 10.523 ms 10.542 ms]
change: [-68.282% -68.190% -68.095%] (p = 0.00 < 0.05)
Performance has improved.
Found 14 outliers among 100 measurements (14.00%)
1 (1.00%) low severe
6 (6.00%) low mild
4 (4.00%) high mild
3 (3.00%) high severe
sort utf8 dictionary tuple preserve partitioning
time: [2.2596 ms 2.2680 ms 2.2768 ms]
change: [+1.9579% +2.5266% +3.0875%] (p = 0.00 < 0.05)
Performance has regressed.
Found 3 outliers among 100 measurements (3.00%)
2 (2.00%) low mild
1 (1.00%) high severe
sort mixed utf8 dictionary tuple
time: [11.538 ms 11.565 ms 11.590 ms]
change: [-66.047% -65.860% -65.706%] (p = 0.00 < 0.05)
Performance has improved.
Found 19 outliers among 100 measurements (19.00%)
8 (8.00%) low severe
7 (7.00%) low mild
2 (2.00%) high mild
2 (2.00%) high severe
sort mixed utf8 dictionary tuple preserve partitioning
time: [2.3764 ms 2.3824 ms 2.3884 ms]
change: [+0.8246% +1.2743% +1.6849%] (p = 0.00 < 0.05)
Change within noise threshold.
Found 2 outliers among 100 measurements (2.00%)
1 (1.00%) low mild
1 (1.00%) high mild
sort mixed tuple time: [12.027 ms 12.081 ms 12.151 ms]
change: [-30.336% -29.962% -29.490%] (p = 0.00 < 0.05)
Performance has improved.
Found 12 outliers among 100 measurements (12.00%)
9 (9.00%) low mild
2 (2.00%) high mild
1 (1.00%) high severe
Benchmarking sort mixed tuple preserve partitioning: Warming up for 3.0000 s
Warning: Unable to complete 100 samples in 5.0s. You may wish to increase target time to 8.0s, enable flat sampling, or reduce sample count to 50.
sort mixed tuple preserve partitioning
time: [1.5646 ms 1.5663 ms 1.5680 ms]
change: [-30.915% -30.601% -30.228%] (p = 0.00 < 0.05)
Performance has improved.
Found 7 outliers among 100 measurements (7.00%)
1 (1.00%) low mild
1 (1.00%) high mild
5 (5.00%) high severe
```
I think the next step is to find a way to make single-column sort expressions perform better in SortPreservingMerge, likely by providing specialized implementations of `SortPreservingMergeStream`'s inner cursor loop
# What changes are included in this PR?
<!--
There is no need to duplicate the description in the issue here but it is sometimes worth providing a summary of the individual changes in this PR.
-->
# Are these changes tested?
<!--
We typically require tests for all PRs in order to:
1. Prevent the code from being accidentally broken by subsequent changes
2. Serve as another way to document the expected behavior of the code
If tests are not included in your PR, please explain why (for example, are they covered by existing tests)?
-->
# Are there any user-facing changes?
<!--
If there are user-facing changes then we may require documentation to be updated before approving the PR.
-->
<!--
If there are any breaking changes to public APIs, please add the `api change` label.
-->
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow-datafusion] tustvold closed pull request #5851: Use SortPreservingMerge for in memory sort
Posted by "tustvold (via GitHub)" <gi...@apache.org>.
tustvold closed pull request #5851: Use SortPreservingMerge for in memory sort
URL: https://github.com/apache/arrow-datafusion/pull/5851
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow-datafusion] tustvold commented on a diff in pull request #5851: Use SortPreservingMerge for in memory sort
Posted by "tustvold (via GitHub)" <gi...@apache.org>.
tustvold commented on code in PR #5851:
URL: https://github.com/apache/arrow-datafusion/pull/5851#discussion_r1156209690
##########
datafusion/core/src/physical_plan/sorts/sort.rs:
##########
@@ -276,279 +276,42 @@ impl Debug for ExternalSorter {
/// consume the non-empty `sorted_batches` and do in_mem_sort
fn in_mem_partial_sort(
- buffered_batches: &mut Vec<BatchWithSortArray>,
+ buffered_batches: &mut Vec<RecordBatch>,
schema: SchemaRef,
expressions: &[PhysicalSortExpr],
batch_size: usize,
tracking_metrics: MemTrackingMetrics,
- fetch: Option<usize>,
+ _fetch: Option<usize>,
) -> Result<SendableRecordBatchStream> {
- assert_ne!(buffered_batches.len(), 0);
- if buffered_batches.len() == 1 {
- let result = buffered_batches.pop();
- Ok(Box::pin(SizedRecordBatchStream::new(
- schema,
- vec![Arc::new(result.unwrap().sorted_batch)],
- tracking_metrics,
- )))
- } else {
- let (sorted_arrays, batches): (Vec<Vec<ArrayRef>>, Vec<RecordBatch>) =
- buffered_batches
- .drain(..)
- .map(|b| {
- let BatchWithSortArray {
- sort_arrays,
- sorted_batch: batch,
- } = b;
- (sort_arrays, batch)
- })
- .unzip();
-
- let sorted_iter = {
- // NB timer records time taken on drop, so there are no
- // calls to `timer.done()` below.
- let _timer = tracking_metrics.elapsed_compute().timer();
- get_sorted_iter(&sorted_arrays, expressions, batch_size, fetch)?
- };
- Ok(Box::pin(SortedSizedRecordBatchStream::new(
- schema,
- batches,
- sorted_iter,
- tracking_metrics,
- )))
+ if buffered_batches.len() < 2 {
+ let batches: Vec<_> = buffered_batches.drain(..).collect();
Review Comment:
It is worth highlighting why this is important to the benchmarks, as there was much discussion of this on https://github.com/apache/arrow-datafusion/issues/5230
The way the "preserve partitioning" benchmarks are setup is they yield a single RecordBatch per partition, they're effectively a special case where the in_mem_partial_sort is a no-op.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow-datafusion] ozankabak commented on pull request #5851: Use SortPreservingMerge for in memory sort
Posted by "ozankabak (via GitHub)" <gi...@apache.org>.
ozankabak commented on PR #5851:
URL: https://github.com/apache/arrow-datafusion/pull/5851#issuecomment-1494701738
Regressions look very significant in some cases, let's see if the proposed next step mitigates that.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow-datafusion] tustvold commented on pull request #5851: Use SortPreservingMerge for in memory sort
Posted by "tustvold (via GitHub)" <gi...@apache.org>.
tustvold commented on PR #5851:
URL: https://github.com/apache/arrow-datafusion/pull/5851#issuecomment-1497728039
Using the benchmarks in https://github.com/apache/arrow-datafusion/pull/5881
```
merge sorted i64 time: [9.3364 ms 9.3444 ms 9.3527 ms]
change: [+1.8876% +2.0141% +2.1272%] (p = 0.00 < 0.05)
Performance has regressed.
Found 5 outliers among 100 measurements (5.00%)
5 (5.00%) high mild
sort merge i64 time: [10.242 ms 10.275 ms 10.308 ms]
change: [+8.8783% +9.2851% +9.6918%] (p = 0.00 < 0.05)
Performance has regressed.
sort i64 time: [14.140 ms 14.159 ms 14.179 ms]
change: [+107.76% +108.22% +108.67%] (p = 0.00 < 0.05)
Performance has regressed.
Found 5 outliers among 100 measurements (5.00%)
4 (4.00%) high mild
1 (1.00%) high severe
sort partitioned i64 time: [1.0789 ms 1.1030 ms 1.1282 ms]
change: [+88.867% +135.91% +186.60%] (p = 0.00 < 0.05)
Performance has regressed.
Found 4 outliers among 100 measurements (4.00%)
1 (1.00%) high mild
3 (3.00%) high severe
merge sorted f64 time: [9.3495 ms 9.3668 ms 9.3846 ms]
change: [+2.1584% +2.4336% +2.6916%] (p = 0.00 < 0.05)
Performance has regressed.
Found 2 outliers among 100 measurements (2.00%)
2 (2.00%) high mild
sort merge f64 time: [10.088 ms 10.118 ms 10.147 ms]
change: [+7.6324% +8.0106% +8.3531%] (p = 0.00 < 0.05)
Performance has regressed.
sort f64 time: [14.030 ms 14.055 ms 14.083 ms]
change: [+89.060% +89.511% +89.963%] (p = 0.00 < 0.05)
Performance has regressed.
Found 2 outliers among 100 measurements (2.00%)
2 (2.00%) high severe
Benchmarking sort partitioned f64: Warming up for 3.0000 s
Warning: Unable to complete 100 samples in 5.0s. You may wish to increase target time to 5.0s, enable flat sampling, or reduce sample count to 70.
sort partitioned f64 time: [1.0817 ms 1.1193 ms 1.1585 ms]
change: [+97.943% +151.36% +215.06%] (p = 0.00 < 0.05)
Performance has regressed.
Found 3 outliers among 100 measurements (3.00%)
1 (1.00%) high mild
2 (2.00%) high severe
merge sorted utf8 low cardinality
time: [6.5726 ms 6.5842 ms 6.5967 ms]
change: [+1.1399% +1.3642% +1.6082%] (p = 0.00 < 0.05)
Performance has regressed.
Found 4 outliers among 100 measurements (4.00%)
2 (2.00%) high mild
2 (2.00%) high severe
sort merge utf8 low cardinality
time: [7.7322 ms 7.7734 ms 7.8145 ms]
change: [+7.3910% +8.1105% +8.7923%] (p = 0.00 < 0.05)
Performance has regressed.
sort utf8 low cardinality
time: [12.349 ms 12.367 ms 12.388 ms]
change: [+37.984% +38.424% +38.870%] (p = 0.00 < 0.05)
Performance has regressed.
Found 3 outliers among 100 measurements (3.00%)
1 (1.00%) high mild
2 (2.00%) high severe
Benchmarking sort partitioned utf8 low cardinality: Warming up for 3.0000 s
Warning: Unable to complete 100 samples in 5.0s. You may wish to increase target time to 6.7s, enable flat sampling, or reduce sample count to 60.
sort partitioned utf8 low cardinality
time: [1.4059 ms 1.4462 ms 1.4889 ms]
change: [+43.382% +70.074% +100.65%] (p = 0.00 < 0.05)
Performance has regressed.
Found 3 outliers among 100 measurements (3.00%)
3 (3.00%) high severe
merge sorted utf8 high cardinality
time: [10.614 ms 10.682 ms 10.759 ms]
change: [+1.6235% +2.2767% +3.0579%] (p = 0.00 < 0.05)
Performance has regressed.
Found 17 outliers among 100 measurements (17.00%)
1 (1.00%) high mild
16 (16.00%) high severe
sort merge utf8 high cardinality
time: [12.340 ms 12.403 ms 12.467 ms]
change: [+4.1153% +4.9344% +5.7495%] (p = 0.00 < 0.05)
Performance has regressed.
Found 1 outliers among 100 measurements (1.00%)
1 (1.00%) high mild
sort utf8 high cardinality
time: [17.160 ms 17.276 ms 17.402 ms]
change: [-10.181% -9.3309% -8.5146%] (p = 0.00 < 0.05)
Performance has improved.
Found 12 outliers among 100 measurements (12.00%)
7 (7.00%) high mild
5 (5.00%) high severe
Benchmarking sort partitioned utf8 high cardinality: Warming up for 3.0000 s
Warning: Unable to complete 100 samples in 5.0s. You may wish to increase target time to 7.4s, enable flat sampling, or reduce sample count to 50.
sort partitioned utf8 high cardinality
time: [1.5979 ms 1.6394 ms 1.6849 ms]
change: [+39.021% +68.819% +101.76%] (p = 0.00 < 0.05)
Performance has regressed.
Found 2 outliers among 100 measurements (2.00%)
2 (2.00%) high severe
merge sorted utf8 tuple time: [17.925 ms 17.957 ms 17.992 ms]
change: [+0.1716% +0.4876% +0.7910%] (p = 0.00 < 0.05)
Change within noise threshold.
Found 5 outliers among 100 measurements (5.00%)
5 (5.00%) high mild
sort merge utf8 tuple time: [22.085 ms 22.189 ms 22.292 ms]
change: [+2.9040% +3.5604% +4.2505%] (p = 0.00 < 0.05)
Performance has regressed.
sort utf8 tuple time: [31.425 ms 31.555 ms 31.685 ms]
change: [-51.893% -51.637% -51.382%] (p = 0.00 < 0.05)
Performance has improved.
sort partitioned utf8 tuple
time: [3.0074 ms 3.1288 ms 3.2508 ms]
change: [+16.143% +22.209% +28.961%] (p = 0.00 < 0.05)
Performance has regressed.
merge sorted utf8 dictionary
time: [5.9205 ms 5.9264 ms 5.9327 ms]
change: [+0.0729% +0.2192% +0.3634%] (p = 0.00 < 0.05)
Change within noise threshold.
Found 2 outliers among 100 measurements (2.00%)
2 (2.00%) high mild
sort merge utf8 dictionary
time: [6.7928 ms 6.8186 ms 6.8448 ms]
change: [+11.083% +11.644% +12.191%] (p = 0.00 < 0.05)
Performance has regressed.
sort utf8 dictionary time: [10.554 ms 10.571 ms 10.588 ms]
change: [+95.615% +96.620% +97.550%] (p = 0.00 < 0.05)
Performance has regressed.
Found 4 outliers among 100 measurements (4.00%)
3 (3.00%) high mild
1 (1.00%) high severe
Benchmarking sort partitioned utf8 dictionary: Warming up for 3.0000 s
Warning: Unable to complete 100 samples in 5.0s. You may wish to increase target time to 5.5s, enable flat sampling, or reduce sample count to 60.
sort partitioned utf8 dictionary
time: [1.1748 ms 1.2139 ms 1.2542 ms]
change: [+71.229% +128.47% +201.74%] (p = 0.00 < 0.05)
Performance has regressed.
Found 5 outliers among 100 measurements (5.00%)
2 (2.00%) high mild
3 (3.00%) high severe
merge sorted utf8 dictionary tuple
time: [8.5027 ms 8.5127 ms 8.5234 ms]
change: [+0.2938% +0.4564% +0.6329%] (p = 0.00 < 0.05)
Change within noise threshold.
Found 3 outliers among 100 measurements (3.00%)
1 (1.00%) high mild
2 (2.00%) high severe
sort merge utf8 dictionary tuple
time: [10.015 ms 10.075 ms 10.136 ms]
change: [+3.0689% +4.0106% +4.9160%] (p = 0.00 < 0.05)
Performance has regressed.
sort utf8 dictionary tuple
time: [17.383 ms 17.407 ms 17.432 ms]
change: [-28.942% -28.653% -28.362%] (p = 0.00 < 0.05)
Performance has improved.
Found 1 outliers among 100 measurements (1.00%)
1 (1.00%) high mild
Benchmarking sort partitioned utf8 dictionary tuple: Warming up for 3.0000 s
Warning: Unable to complete 100 samples in 5.0s. You may wish to increase target time to 8.5s, enable flat sampling, or reduce sample count to 50.
sort partitioned utf8 dictionary tuple
time: [1.7916 ms 1.8438 ms 1.8995 ms]
change: [+11.399% +19.720% +29.069%] (p = 0.00 < 0.05)
Performance has regressed.
Found 2 outliers among 100 measurements (2.00%)
1 (1.00%) high mild
1 (1.00%) high severe
merge sorted mixed dictionary tuple
time: [15.183 ms 15.203 ms 15.227 ms]
change: [+1.1594% +1.3422% +1.5391%] (p = 0.00 < 0.05)
Performance has regressed.
Found 5 outliers among 100 measurements (5.00%)
3 (3.00%) high mild
2 (2.00%) high severe
sort merge mixed dictionary tuple
time: [17.153 ms 17.220 ms 17.285 ms]
change: [+4.3339% +4.9897% +5.6123%] (p = 0.00 < 0.05)
Performance has regressed.
sort mixed dictionary tuple
time: [25.322 ms 25.384 ms 25.454 ms]
change: [-57.042% -56.886% -56.727%] (p = 0.00 < 0.05)
Performance has improved.
Found 3 outliers among 100 measurements (3.00%)
1 (1.00%) high mild
2 (2.00%) high severe
Benchmarking sort partitioned mixed dictionary tuple: Warming up for 3.0000 s
Warning: Unable to complete 100 samples in 5.0s. You may wish to increase target time to 9.3s, enable flat sampling, or reduce sample count to 50.
sort partitioned mixed dictionary tuple
time: [1.9511 ms 1.9985 ms 2.0512 ms]
change: [+5.3542% +12.351% +20.206%] (p = 0.00 < 0.05)
Performance has regressed.
Found 1 outliers among 100 measurements (1.00%)
1 (1.00%) high severe
merge sorted mixed tuple
time: [17.414 ms 17.437 ms 17.461 ms]
change: [+0.8745% +1.0492% +1.2293%] (p = 0.00 < 0.05)
Change within noise threshold.
Found 6 outliers among 100 measurements (6.00%)
6 (6.00%) high mild
sort merge mixed tuple time: [20.549 ms 20.620 ms 20.693 ms]
change: [+4.5839% +5.0752% +5.5380%] (p = 0.00 < 0.05)
Performance has regressed.
sort mixed tuple time: [26.823 ms 26.899 ms 26.977 ms]
change: [-9.2725% -8.8104% -8.3425%] (p = 0.00 < 0.05)
Performance has improved.
sort partitioned mixed tuple
time: [2.3462 ms 2.4398 ms 2.5338 ms]
change: [+40.731% +50.245% +60.230%] (p = 0.00 < 0.05)
Performance has regressed.
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow-datafusion] tustvold commented on pull request #5851: Use SortPreservingMerge for in memory sort
Posted by "tustvold (via GitHub)" <gi...@apache.org>.
tustvold commented on PR #5851:
URL: https://github.com/apache/arrow-datafusion/pull/5851#issuecomment-1494804421
https://github.com/apache/arrow-datafusion/pull/5854 contains an extremely hacky POC of special casing the cursor implementation, and eliminates the regression for "sort i64".
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow-datafusion] tustvold commented on pull request #5851: Use SortPreservingMerge for in memory sort
Posted by "tustvold (via GitHub)" <gi...@apache.org>.
tustvold commented on PR #5851:
URL: https://github.com/apache/arrow-datafusion/pull/5851#issuecomment-1528864119
Closing in favour of https://github.com/apache/arrow-datafusion/pull/6163
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org