You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@datafusion.apache.org by "zhuqi-lucas (via GitHub)" <gi...@apache.org> on 2026/10/12 02:13:14 UTC
[PR] perf(sort-merge): skip null checks in ArrayValues when neither cursor has nulls [datafusion]
zhuqi-lucas opened a new pull request, #26215:
URL: https://github.com/apache/datafusion/pull/26215
## Which issue does this PR close?
- Part of #23840. Second of the three units that umbrella PR was split into; #23990 (the `RowValues` cache) was the first.
## Rationale for this change
`ArrayValues` wraps every sort-key cursor to add null handling, so its `compare` / `eq` / `eq_to_previous` sit in the merge's inner loop. Each one opens with the same shape:
```rust
match (l.is_null(l_idx), r.is_null(r_idx)) {
(true, true) => Ordering::Equal,
(true, false) => ...,
(false, true) => ...,
(false, false) => T::compare(...),
}
```
That is two comparisons against `null_threshold` plus the match, evaluated once per comparison the loser tree makes.
Sort keys are usually null-free, and whether an array has any nulls is known when the cursor is built (`array.null_count()`, already read there to compute `null_threshold`). When neither side has nulls, the match above picks its `(false, false)` arm every single time.
## What changes are included in this PR?
- `ArrayValues` records `has_nulls: bool` at construction, from the `null_count` it already reads.
- `compare`, `eq` and `eq_to_previous` take a direct path when no cursor involved has nulls, skipping the `is_null` checks.
`has_nulls` is purely a shortcut around checks whose outcome is already determined; `is_null` still decides every case where a null can occur, so behaviour is unchanged.
Not touched: `get_value` and `eq_to_single_row_value` also call `is_null`, but they are not in the merge's inner loop, so they keep the plain path rather than growing another branch.
## Are these changes tested?
Yes. `test_array_values_has_nulls_fast_path_agrees_with_slow_path` asserts the fast path is actually taken (`!values.has_nulls` on both cursors) and that it orders the same as the null-aware path, then forces the slow path with a null-containing cursor and checks the two agree.
Red/green verified: making the fast path ignore `options.descending` fails the test.
`cargo test -p datafusion-physical-plan --lib sorts::` β 145 passed. `cargo clippy --all-targets -p datafusion-physical-plan -- -D warnings` clean.
## Benchmark
Measured as part of the full #23840 stack on sort-tpch SF1 (`release-nonlto`, 3 iterations), where this unit is what recovered Q11: StringView cursor caching alone regressed Q11 by ~3%, and adding this fast path moved it to -7.3%. Q11 sorts 6M rows by `l_shipmode` β cardinality 7, so it is almost entirely tied comparisons, which is exactly the inner loop this touches.
Requesting the benchmark bot below for numbers on this change in isolation.
## Are there any user-facing changes?
No. No API change; `ArrayValues` is internal to the sort-merge cursors.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] perf(sort-merge): skip null checks in ArrayValues when neither cursor has nulls [datafusion]
Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on code in PR #26215:
URL: https://github.com/apache/datafusion/pull/26215#discussion_r4251067778
##########
datafusion/physical-plan/src/sorts/cursor.rs:
##########
@@ -643,6 +649,11 @@ impl<T: CursorValues> CursorValues for ArrayValues<T> {
#[inline(always)]
fn eq(l: &Self, l_idx: usize, r: &Self, r_idx: usize) -> bool {
+ // Fast path: neither array has nulls, so the match below would take
+ // the (false, false) arm every time.
+ if !l.has_nulls && !r.has_nulls {
Review Comment:
this still basically dynamically dispatches each row
I wonder if there is some way to parameterize ArrayValues so we could dispatch just once π€
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] perf(sort-merge): skip null checks in ArrayValues when neither cursor has nulls [datafusion]
Posted by "Copilot (via GitHub)" <gi...@apache.org>.
Copilot commented on code in PR #26215:
URL: https://github.com/apache/datafusion/pull/26215#discussion_r4244196857
##########
datafusion/physical-plan/src/sorts/cursor.rs:
##########
@@ -946,4 +968,28 @@ mod tests {
b.advance();
assert_eq!(a.cmp(&b), Ordering::Less);
}
+
+ /// Verifies the `ArrayValues::has_nulls` fast path is actually taken
+ /// (all null-free cursors) and produces the same ordering as the
+ /// null-aware slow path.
+ #[test]
+ fn test_array_values_has_nulls_fast_path_agrees_with_slow_path() {
+ let options = SortOptions {
+ descending: false,
+ nulls_first: false,
+ };
+
+ // Two null-free cursors -- compare takes the has_nulls fast path.
+ let a = new_primitive(options, ScalarBuffer::from(vec![1i32, 2, 3, 4]), 0);
+ let b = new_primitive(options, ScalarBuffer::from(vec![2i32, 3, 4, 5]), 0);
+ assert!(!a.values.has_nulls);
+ assert!(!b.values.has_nulls);
+ assert_eq!(a.cmp(&b), Ordering::Less);
+
+ // A null-containing cursor forces the slow (null-aware) path, and the
+ // two paths must agree.
+ let c = new_primitive(options, ScalarBuffer::from(vec![i32::MAX, 3i32, 4]), 1);
+ assert!(c.values.has_nulls);
+ assert_eq!(a.cmp(&c), Ordering::Less);
Review Comment:
This test does not currently establish the claimed equivalence: the fast path compares `1` with `2`, while the slow path compares `1` with `i32::MAX`, and `descending` is always false. It also never exercises the newly added `eq` or `eq_to_previous` fast paths. In particular, changing the fast-path descending behavior would not fail this test as the PR description claims. Please compare identical logical values through both paths and cover both sort directions and all three methods.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] perf(sort-merge): skip null checks in ArrayValues when neither cursor has nulls [datafusion]
Posted by "zhuqi-lucas (via GitHub)" <gi...@apache.org>.
zhuqi-lucas commented on code in PR #26215:
URL: https://github.com/apache/datafusion/pull/26215#discussion_r4254101474
##########
datafusion/physical-plan/src/sorts/cursor.rs:
##########
@@ -946,4 +968,28 @@ mod tests {
b.advance();
assert_eq!(a.cmp(&b), Ordering::Less);
}
+
+ /// Verifies the `ArrayValues::has_nulls` fast path is actually taken
+ /// (all null-free cursors) and produces the same ordering as the
+ /// null-aware slow path.
+ #[test]
+ fn test_array_values_has_nulls_fast_path_agrees_with_slow_path() {
+ let options = SortOptions {
+ descending: false,
+ nulls_first: false,
+ };
+
+ // Two null-free cursors -- compare takes the has_nulls fast path.
+ let a = new_primitive(options, ScalarBuffer::from(vec![1i32, 2, 3, 4]), 0);
+ let b = new_primitive(options, ScalarBuffer::from(vec![2i32, 3, 4, 5]), 0);
+ assert!(!a.values.has_nulls);
+ assert!(!b.values.has_nulls);
+ assert_eq!(a.cmp(&b), Ordering::Less);
+
+ // A null-containing cursor forces the slow (null-aware) path, and the
+ // two paths must agree.
+ let c = new_primitive(options, ScalarBuffer::from(vec![i32::MAX, 3i32, 4]), 1);
+ assert!(c.values.has_nulls);
+ assert_eq!(a.cmp(&c), Ordering::Less);
Review Comment:
Thanks @alamb , good point, addressed now.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] perf(sort-merge): skip null checks in ArrayValues when neither cursor has nulls [datafusion]
Posted by "zhuqi-lucas (via GitHub)" <gi...@apache.org>.
zhuqi-lucas commented on PR #26215:
URL: https://github.com/apache/datafusion/pull/26215#issuecomment-6137423036
> Thanks @zhuqi-lucas -- this makes sense to me
Thanks @alamb for review and good suggestion!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] perf(sort-merge): skip null checks in ArrayValues when neither cursor has nulls [datafusion]
Posted by "zhuqi-lucas (via GitHub)" <gi...@apache.org>.
zhuqi-lucas commented on code in PR #26215:
URL: https://github.com/apache/datafusion/pull/26215#discussion_r4254332680
##########
datafusion/physical-plan/src/sorts/cursor.rs:
##########
@@ -643,6 +649,11 @@ impl<T: CursorValues> CursorValues for ArrayValues<T> {
#[inline(always)]
fn eq(l: &Self, l_idx: usize, r: &Self, r_idx: usize) -> bool {
+ // Fast path: neither array has nulls, so the match below would take
+ // the (false, false) arm every time.
+ if !l.has_nulls && !r.has_nulls {
Review Comment:
Good point. Prototyped it and it is worth another ~6%, but it needs a new cursor type and a nullability dispatch in `StreamingMergeBuilder`, so I opened #26252 rather than growing this PR.
Thanks @alamb
##########
datafusion/physical-plan/src/sorts/cursor.rs:
##########
@@ -946,4 +968,28 @@ mod tests {
b.advance();
assert_eq!(a.cmp(&b), Ordering::Less);
}
+
+ /// Verifies the `ArrayValues::has_nulls` fast path is actually taken
+ /// (all null-free cursors) and produces the same ordering as the
+ /// null-aware slow path.
+ #[test]
+ fn test_array_values_has_nulls_fast_path_agrees_with_slow_path() {
+ let options = SortOptions {
+ descending: false,
+ nulls_first: false,
+ };
+
+ // Two null-free cursors -- compare takes the has_nulls fast path.
+ let a = new_primitive(options, ScalarBuffer::from(vec![1i32, 2, 3, 4]), 0);
+ let b = new_primitive(options, ScalarBuffer::from(vec![2i32, 3, 4, 5]), 0);
+ assert!(!a.values.has_nulls);
+ assert!(!b.values.has_nulls);
+ assert_eq!(a.cmp(&b), Ordering::Less);
+
+ // A null-containing cursor forces the slow (null-aware) path, and the
+ // two paths must agree.
+ let c = new_primitive(options, ScalarBuffer::from(vec![i32::MAX, 3i32, 4]), 1);
+ assert!(c.values.has_nulls);
+ assert_eq!(a.cmp(&c), Ordering::Less);
Review Comment:
You are both right, the old test proved nothing. Rewrote it to compare the same values through both paths, across all index pairs, both sort directions and both null placements, and it now covers `eq` and `eq_to_previous` too. Verified by mutation: ignoring `descending`, inverting it, and `eq` / `eq_to_previous` returning `true` each fail it.
Thanks @alamb
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] perf(sort-merge): skip null checks in ArrayValues when neither cursor has nulls [datafusion]
Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on code in PR #26215:
URL: https://github.com/apache/datafusion/pull/26215#discussion_r4251060824
##########
datafusion/physical-plan/src/sorts/cursor.rs:
##########
@@ -946,4 +968,28 @@ mod tests {
b.advance();
assert_eq!(a.cmp(&b), Ordering::Less);
}
+
+ /// Verifies the `ArrayValues::has_nulls` fast path is actually taken
+ /// (all null-free cursors) and produces the same ordering as the
+ /// null-aware slow path.
+ #[test]
+ fn test_array_values_has_nulls_fast_path_agrees_with_slow_path() {
+ let options = SortOptions {
+ descending: false,
+ nulls_first: false,
+ };
+
+ // Two null-free cursors -- compare takes the has_nulls fast path.
+ let a = new_primitive(options, ScalarBuffer::from(vec![1i32, 2, 3, 4]), 0);
+ let b = new_primitive(options, ScalarBuffer::from(vec![2i32, 3, 4, 5]), 0);
+ assert!(!a.values.has_nulls);
+ assert!(!b.values.has_nulls);
+ assert_eq!(a.cmp(&b), Ordering::Less);
+
+ // A null-containing cursor forces the slow (null-aware) path, and the
+ // two paths must agree.
+ let c = new_primitive(options, ScalarBuffer::from(vec![i32::MAX, 3i32, 4]), 1);
+ assert!(c.values.has_nulls);
+ assert_eq!(a.cmp(&c), Ordering::Less);
Review Comment:
this looks like a legit suggestion
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] perf(sort-merge): skip null checks in ArrayValues when neither cursor has nulls [datafusion]
Posted by "adriangbot (via GitHub)" <gi...@apache.org>.
adriangbot commented on PR #26215:
URL: https://github.com/apache/datafusion/pull/26215#issuecomment-6116687073
π€ Benchmark completed (GKE) | [trigger](https://github.com/apache/datafusion/pull/26215#issuecomment-6116531912)
**Instance:** `c4a-highmem-16` (12 vCPU / 65 GiB)
Comparing perf/spm-array-values-has-nulls (08944c4893324bbecb606e1e2a3f08e8be19690b) to b295a67 (merge-base) [diff](https://github.com/apache/datafusion/compare/b295a672ce6fcef216c94aaca88987c98449803c..08944c4893324bbecb606e1e2a3f08e8be19690b)
<details><summary>Run configuration</summary>
```yaml
run benchmark sort_tpch
```
</details>
<details><summary>CPU Details (lscpu)</summary>
```
Architecture: aarch64
CPU op-mode(s): 64-bit
Byte Order: Little Endian
CPU(s): 16
On-line CPU(s) list: 0-15
Vendor ID: ARM
Model name: Neoverse-V2
Model: 1
Thread(s) per core: 1
Core(s) per cluster: 16
Socket(s): -
Cluster(s): 1
Stepping: r0p1
BogoMIPS: 2000.00
Flags: fp asimd evtstrm aes pmull sha1 sha2 crc32 atomics fphp asimdhp cpuid asimdrdm jscvt fcma lrcpc dcpop sha3 sm3 sm4 asimddp sha512 sve asimdfhm dit uscat ilrcpc flagm sb paca pacg dcpodp sve2 sveaes svepmull svebitperm svesha3 svesm4 flagm2 frint svei8mm svebf16 i8mm bf16 dgh rng bti
L1d cache: 1 MiB (16 instances)
L1i cache: 1 MiB (16 instances)
L2 cache: 32 MiB (16 instances)
L3 cache: 80 MiB (1 instance)
NUMA node(s): 1
NUMA node0 CPU(s): 0-15
Vulnerability Gather data sampling: Not affected
Vulnerability Indirect target selection: Not affected
Vulnerability Itlb multihit: Not affected
Vulnerability L1tf: Not affected
Vulnerability Mds: Not affected
Vulnerability Meltdown: Not affected
Vulnerability Mmio stale data: Not affected
Vulnerability Reg file data sampling: Not affected
Vulnerability Retbleed: Not affected
Vulnerability Spec rstack overflow: Not affected
Vulnerability Spec store bypass: Mitigation; Speculative Store Bypass disabled via prctl
Vulnerability Spectre v1: Mitigation; __user pointer sanitization
Vulnerability Spectre v2: Mitigation; CSV2, BHB
Vulnerability Srbds: Not affected
Vulnerability Tsa: Not affected
Vulnerability Tsx async abort: Not affected
Vulnerability Vmscape: Not affected
```
</details>
<details><summary>Details</summary>
<p>
```
Comparing HEAD and perf_spm-array-values-has-nulls
--------------------
Benchmark sort_tpch1.json
--------------------
βββββββββ³ββββββββββββ³ββββββββββββββββββββββββββββββββββ³ββββββββββββββββ
β Query β HEAD β perf_spm-array-values-has-nulls β Change β
β‘ββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββ©
β Q1 β 124.35 ms β 110.35 ms β +1.13x faster β
β Q2 β 105.42 ms β 96.86 ms β +1.09x faster β
β Q3 β 631.00 ms β 623.93 ms β no change β
β Q4 β 158.09 ms β 158.13 ms β no change β
β Q5 β 234.33 ms β 232.99 ms β no change β
β Q6 β 245.88 ms β 245.25 ms β no change β
β Q7 β 405.96 ms β 405.61 ms β no change β
β Q8 β 293.11 ms β 291.12 ms β no change β
β Q9 β 316.51 ms β 316.68 ms β no change β
β Q10 β 458.02 ms β 450.93 ms β no change β
β Q11 β 242.47 ms β 242.96 ms β no change β
βββββββββ΄ββββββββββββ΄ββββββββββββββββββββββββββββββββββ΄ββββββββββββββββ
ββββββββββββββββββββββββββββββββββββββββββββββββββ³ββββββββββββ
β Benchmark Summary β β
β‘βββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββ©
β Total Time (HEAD) β 3215.15ms β
β Total Time (perf_spm-array-values-has-nulls) β 3174.82ms β
β Average Time (HEAD) β 292.29ms β
β Average Time (perf_spm-array-values-has-nulls) β 288.62ms β
β Queries Faster β 2 β
β Queries Slower β 0 β
β Queries with No Change β 9 β
β Queries with Failure β 0 β
ββββββββββββββββββββββββββββββββββββββββββββββββββ΄ββββββββββββ
Distribution per query (min / mean Β±stddev / max):
Comparing HEAD and perf_spm-array-values-has-nulls
--------------------
Benchmark sort_tpch1.json
--------------------
βββββββββ³βββββββββββββββββββββββββββββββββββββ³βββββββββββββββββββββββββββββββββββββ³ββββββββββββββββ
β Query β HEAD β perf_spm-array-values-has-nulls β Change β
β‘ββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββ©
β Q1 β 124.35 / 125.16 Β±0.91 / 126.91 ms β 110.35 / 111.10 Β±1.27 / 113.64 ms β +1.13x faster β
β Q2 β 105.42 / 106.91 Β±1.55 / 109.29 ms β 96.86 / 97.74 Β±1.30 / 100.32 ms β +1.09x faster β
β Q3 β 631.00 / 632.70 Β±1.70 / 635.64 ms β 623.93 / 628.34 Β±3.12 / 632.02 ms β no change β
β Q4 β 158.09 / 159.46 Β±0.86 / 160.49 ms β 158.13 / 158.71 Β±0.54 / 159.60 ms β no change β
β Q5 β 234.33 / 234.98 Β±0.45 / 235.38 ms β 232.99 / 234.66 Β±1.52 / 237.52 ms β no change β
β Q6 β 245.88 / 247.79 Β±1.77 / 250.85 ms β 245.25 / 246.96 Β±1.52 / 248.78 ms β no change β
β Q7 β 405.96 / 407.79 Β±1.45 / 410.11 ms β 405.61 / 408.39 Β±2.81 / 413.40 ms β no change β
β Q8 β 293.11 / 297.36 Β±3.34 / 301.92 ms β 291.12 / 296.07 Β±3.13 / 300.34 ms β no change β
β Q9 β 316.51 / 328.33 Β±10.20 / 342.44 ms β 316.68 / 327.43 Β±7.91 / 338.73 ms β no change β
β Q10 β 458.02 / 470.79 Β±14.34 / 498.27 ms β 450.93 / 457.44 Β±3.43 / 460.27 ms β no change β
β Q11 β 242.47 / 251.73 Β±10.89 / 265.93 ms β 242.96 / 255.59 Β±13.77 / 272.96 ms β no change β
βββββββββ΄βββββββββββββββββββββββββββββββββββββ΄βββββββββββββββββββββββββββββββββββββ΄ββββββββββββββββ
ββββββββββββββββββββββββββββββββββββββββββββββββββ³ββββββββββββ
β Benchmark Summary β β
β‘βββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββ©
β Total Time (HEAD) β 3262.99ms β
β Total Time (perf_spm-array-values-has-nulls) β 3222.43ms β
β Average Time (HEAD) β 296.64ms β
β Average Time (perf_spm-array-values-has-nulls) β 292.95ms β
β Queries Faster β 2 β
β Queries Slower β 0 β
β Queries with No Change β 9 β
β Queries with Failure β 0 β
ββββββββββββββββββββββββββββββββββββββββββββββββββ΄ββββββββββββ
```
</p>
</details>
<details><summary>Resource Usage</summary>
**sort_tpch β base (merge-base)**
| Metric | Value |
|--------|-------|
| Wall time | 20.0s |
| Peak memory | 2.4 GiB |
| Avg memory | 1.0 GiB |
| CPU user | 60.7s |
| CPU sys | 3.0s |
| Peak spill | 0 B |
**sort_tpch β branch**
| Metric | Value |
|--------|-------|
| Wall time | 20.0s |
| Peak memory | 2.4 GiB |
| Avg memory | 1.1 GiB |
| CPU user | 60.9s |
| CPU sys | 2.6s |
| Peak spill | 0 B |
</details>
---
[File an issue](https://github.com/adriangb/datafusion-benchmarking/issues) against this benchmark runner
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] perf(sort-merge): skip null checks in ArrayValues when neither cursor has nulls [datafusion]
Posted by "codecov-commenter (via GitHub)" <gi...@apache.org>.
codecov-commenter commented on PR #26215:
URL: https://github.com/apache/datafusion/pull/26215#issuecomment-6116744933
## [Codecov](https://app.codecov.io/gh/apache/datafusion/pull/26215?dropdown=coverage&src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) Report
:x: Patch coverage is `93.33333%` with `2 lines` in your changes missing coverage. Please review.
:white_check_mark: Project coverage is 82.80%. Comparing base ([`b295a67`](https://app.codecov.io/gh/apache/datafusion/commit/b295a672ce6fcef216c94aaca88987c98449803c?dropdown=coverage&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache)) to head ([`08944c4`](https://app.codecov.io/gh/apache/datafusion/commit/08944c4893324bbecb606e1e2a3f08e8be19690b?dropdown=coverage&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache)).
| [Files with missing lines](https://app.codecov.io/gh/apache/datafusion/pull/26215?dropdown=coverage&src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | Patch % | Lines |
|---|---|---|
| [datafusion/physical-plan/src/sorts/cursor.rs](https://app.codecov.io/gh/apache/datafusion/pull/26215?src=pr&el=tree&filepath=datafusion%2Fphysical-plan%2Fsrc%2Fsorts%2Fcursor.rs&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-ZGF0YWZ1c2lvbi9waHlzaWNhbC1wbGFuL3NyYy9zb3J0cy9jdXJzb3IucnM=) | 93.33% | [1 Missing and 1 partial :warning: ](https://app.codecov.io/gh/apache/datafusion/pull/26215?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
<details><summary>Additional details and impacted files</summary>
```diff
@@ Coverage Diff @@
## main #26215 +/- ##
==========================================
- Coverage 82.80% 82.80% -0.01%
==========================================
Files 1149 1149
Lines 452234 452262 +28
Branches 452234 452262 +28
==========================================
+ Hits 374493 374509 +16
- Misses 54946 54954 +8
- Partials 22795 22799 +4
```
</details>
[:umbrella: View full report in Codecov by Harness](https://app.codecov.io/gh/apache/datafusion/pull/26215?dropdown=coverage&src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache).
:loudspeaker: Have feedback on the report? [Share it here](https://about.codecov.io/codecov-pr-comment-feedback/?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache).
<details><summary> :rocket: New features to boost your workflow: </summary>
- :snowflake: [Test Analytics](https://docs.codecov.com/docs/test-analytics): Detect flaky tests, report on failures, and find test suite problems.
- :package: [JS Bundle Analysis](https://docs.codecov.com/docs/javascript-bundle-analysis): Save yourself from yourself by tracking and limiting bundle sizes in JS merges.
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] perf(sort-merge): skip null checks in ArrayValues when neither cursor has nulls [datafusion]
Posted by "zhuqi-lucas (via GitHub)" <gi...@apache.org>.
zhuqi-lucas commented on PR #26215:
URL: https://github.com/apache/datafusion/pull/26215#issuecomment-6116531912
run benchmark sort_tpch
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] perf(sort-merge): skip null checks in ArrayValues when neither cursor has nulls [datafusion]
Posted by "adriangbot (via GitHub)" <gi...@apache.org>.
adriangbot commented on PR #26215:
URL: https://github.com/apache/datafusion/pull/26215#issuecomment-6116604633
π€ Benchmark running (GKE) | [trigger](https://github.com/apache/datafusion/pull/26215#issuecomment-6116531912)
**Instance:** `c4a-highmem-16` (12 vCPU / 65 GiB) | `Linux bench-c6116531912-3310-2v9kh 6.12.94+ #1 SMP Fri Aug 21 08:00:16 UTC 2026 aarch64 GNU/Linux`
<details><summary>CPU Details (lscpu)</summary>
```
Architecture: aarch64
CPU op-mode(s): 64-bit
Byte Order: Little Endian
CPU(s): 16
On-line CPU(s) list: 0-15
Vendor ID: ARM
Model name: Neoverse-V2
Model: 1
Thread(s) per core: 1
Core(s) per cluster: 16
Socket(s): -
Cluster(s): 1
Stepping: r0p1
BogoMIPS: 2000.00
Flags: fp asimd evtstrm aes pmull sha1 sha2 crc32 atomics fphp asimdhp cpuid asimdrdm jscvt fcma lrcpc dcpop sha3 sm3 sm4 asimddp sha512 sve asimdfhm dit uscat ilrcpc flagm sb paca pacg dcpodp sve2 sveaes svepmull svebitperm svesha3 svesm4 flagm2 frint svei8mm svebf16 i8mm bf16 dgh rng bti
L1d cache: 1 MiB (16 instances)
L1i cache: 1 MiB (16 instances)
L2 cache: 32 MiB (16 instances)
L3 cache: 80 MiB (1 instance)
NUMA node(s): 1
NUMA node0 CPU(s): 0-15
Vulnerability Gather data sampling: Not affected
Vulnerability Indirect target selection: Not affected
Vulnerability Itlb multihit: Not affected
Vulnerability L1tf: Not affected
Vulnerability Mds: Not affected
Vulnerability Meltdown: Not affected
Vulnerability Mmio stale data: Not affected
Vulnerability Reg file data sampling: Not affected
Vulnerability Retbleed: Not affected
Vulnerability Spec rstack overflow: Not affected
Vulnerability Spec store bypass: Mitigation; Speculative Store Bypass disabled via prctl
Vulnerability Spectre v1: Mitigation; __user pointer sanitization
Vulnerability Spectre v2: Mitigation; CSV2, BHB
Vulnerability Srbds: Not affected
Vulnerability Tsa: Not affected
Vulnerability Tsx async abort: Not affected
Vulnerability Vmscape: Not affected
```
</details>
Comparing perf/spm-array-values-has-nulls (08944c4893324bbecb606e1e2a3f08e8be19690b) to b295a67 (merge-base) [diff](https://github.com/apache/datafusion/compare/b295a672ce6fcef216c94aaca88987c98449803c..08944c4893324bbecb606e1e2a3f08e8be19690b)
<details><summary>Run configuration</summary>
```yaml
run benchmark sort_tpch
```
</details>
Results will be posted here when complete
---
[File an issue](https://github.com/adriangb/datafusion-benchmarking/issues) against this benchmark runner
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] perf(sort-merge): skip null checks in ArrayValues when neither cursor has nulls [datafusion]
Posted by "zhuqi-lucas (via GitHub)" <gi...@apache.org>.
zhuqi-lucas merged PR #26215:
URL: https://github.com/apache/datafusion/pull/26215
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org