You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@arrow.apache.org by "emilk (via GitHub)" <gi...@apache.org> on 2026/08/19 06:40:26 UTC
[PR] fix: `try_binary` dropped the logical nulls of run and dictionary arrays [arrow-rs]
emilk opened a new pull request, #10748:
URL: https://github.com/apache/arrow-rs/pull/10748
# Which issue does this PR close?
No issue; found while reviewing https://github.com/apache/arrow-rs/pull/10730.
# Rationale for this change
`try_binary` skipped null handling when `a.null_count() == 0 && b.null_count() == 0`. That is a *physical* count. A `RunArray` never has a null buffer of its own, so an array whose values contain nulls took the no-nulls fast path and silently produced a result with no nulls at all:
```
values [Some(10), None, Some(30)] + [1, 1, 1] => [11, 1, 31] // row 1 should be null
```
A `DictionaryArray` with nullable values has the same problem. `binary` is unaffected: it has no such fast path.
# What changes are included in this PR?
Gate the fast path on `is_nullable`, which accounts for logical nulls. That check is allowed to be conservative, so an empty union of the logical nulls now falls back to the no-nulls path instead of `unwrap`ping.
# Are these changes tested?
Yes, a new test for a `RunArray` with nulls in its values. It fails on `main`.
# Are there any user-facing changes?
`try_binary` returns the correct nulls for run and dictionary arrays. No API change.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] fix: `try_binary` dropped the logical nulls of run and dictionary arrays [arrow-rs]
Posted by "Jefffrey (via GitHub)" <gi...@apache.org>.
Jefffrey commented on PR #10748:
URL: https://github.com/apache/arrow-rs/pull/10748#issuecomment-5377894130
thanks @emilk & @Rich-T-kid
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] fix: `try_binary` dropped the logical nulls of run and dictionary arrays [arrow-rs]
Posted by "emilk (via GitHub)" <gi...@apache.org>.
emilk commented on code in PR #10748:
URL: https://github.com/apache/arrow-rs/pull/10748#discussion_r3813177607
##########
arrow-arith/src/arity.rs:
##########
@@ -270,11 +270,17 @@ where
}
let len = a.len();
- if a.null_count() == 0 && b.null_count() == 0 {
+ // Physical nulls are not the whole story: a `RunArray` or `DictionaryArray` can have
+ // logical nulls in its values while its own null buffer is absent. `is_nullable` covers
+ // those, but is allowed to be conservative, so the union of the logical nulls can still
+ // be empty.
Review Comment:
From [the `is_nullable` docs](https://docs.rs/arrow/latest/arrow/array/trait.Array.html#method.is_nullable):
> Implementations will return `true` unless they can cheaply prove no logical nulls are present. For example a [DictionaryArray](https://docs.rs/arrow/latest/arrow/array/struct.DictionaryArray.html) with nullable values will still return `true` […]
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] fix: `try_binary` dropped the logical nulls of run and dictionary arrays [arrow-rs]
Posted by "Jefffrey (via GitHub)" <gi...@apache.org>.
Jefffrey merged PR #10748:
URL: https://github.com/apache/arrow-rs/pull/10748
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] fix: `try_binary` dropped the logical nulls of run and dictionary arrays [arrow-rs]
Posted by "Rich-T-kid (via GitHub)" <gi...@apache.org>.
Rich-T-kid commented on PR #10748:
URL: https://github.com/apache/arrow-rs/pull/10748#issuecomment-5342346088
I think this is related https://github.com/apache/arrow-rs/pull/10485
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] fix: `try_binary` dropped the logical nulls of run and dictionary arrays [arrow-rs]
Posted by "Jefffrey (via GitHub)" <gi...@apache.org>.
Jefffrey commented on code in PR #10748:
URL: https://github.com/apache/arrow-rs/pull/10748#discussion_r3813216601
##########
arrow-arith/src/arity.rs:
##########
@@ -270,11 +270,17 @@ where
}
let len = a.len();
- if a.null_count() == 0 && b.null_count() == 0 {
+ // Physical nulls are not the whole story: a `RunArray` or `DictionaryArray` can have
+ // logical nulls in its values while its own null buffer is absent. `is_nullable` covers
+ // those, but is allowed to be conservative, so the union of the logical nulls can still
+ // be empty.
Review Comment:
thanks i missed that
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] fix: `try_binary` dropped the logical nulls of run and dictionary arrays [arrow-rs]
Posted by "Jefffrey (via GitHub)" <gi...@apache.org>.
Jefffrey commented on PR #10748:
URL: https://github.com/apache/arrow-rs/pull/10748#issuecomment-5342565790
another thing to note is `try_binary_mut` has the same issue
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] fix: `try_binary` dropped the logical nulls of run and dictionary arrays [arrow-rs]
Posted by "Jefffrey (via GitHub)" <gi...@apache.org>.
Jefffrey commented on code in PR #10748:
URL: https://github.com/apache/arrow-rs/pull/10748#discussion_r3813106786
##########
arrow-arith/src/arity.rs:
##########
@@ -270,11 +270,17 @@ where
}
let len = a.len();
- if a.null_count() == 0 && b.null_count() == 0 {
+ // Physical nulls are not the whole story: a `RunArray` or `DictionaryArray` can have
+ // logical nulls in its values while its own null buffer is absent. `is_nullable` covers
+ // those, but is allowed to be conservative, so the union of the logical nulls can still
+ // be empty.
+ if !a.is_nullable() && !b.is_nullable() {
try_binary_no_nulls(len, a, b, op)
} else {
- let nulls =
- NullBuffer::union(a.logical_nulls().as_ref(), b.logical_nulls().as_ref()).unwrap();
+ let Some(nulls) = NullBuffer::union(a.logical_nulls().as_ref(), b.logical_nulls().as_ref())
Review Comment:
~theres an unfortunate side effect in that we calculate the logical_nulls buffer twice; in terms of correctness this fix is good for ensuring we treat nulls properly, but there is a cost to calculating this logical nulls buffer for dictionaries specifically~~ 🤔
edit: this is wrong, disregard
##########
arrow-arith/src/arity.rs:
##########
@@ -270,11 +270,17 @@ where
}
let len = a.len();
- if a.null_count() == 0 && b.null_count() == 0 {
+ // Physical nulls are not the whole story: a `RunArray` or `DictionaryArray` can have
+ // logical nulls in its values while its own null buffer is absent. `is_nullable` covers
+ // those, but is allowed to be conservative, so the union of the logical nulls can still
+ // be empty.
+ if !a.is_nullable() && !b.is_nullable() {
try_binary_no_nulls(len, a, b, op)
} else {
- let nulls =
- NullBuffer::union(a.logical_nulls().as_ref(), b.logical_nulls().as_ref()).unwrap();
+ let Some(nulls) = NullBuffer::union(a.logical_nulls().as_ref(), b.logical_nulls().as_ref())
Review Comment:
~~theres an unfortunate side effect in that we calculate the logical_nulls buffer twice; in terms of correctness this fix is good for ensuring we treat nulls properly, but there is a cost to calculating this logical nulls buffer for dictionaries specifically 🤔~~
edit: this is wrong, disregard
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] fix: `try_binary` dropped the logical nulls of run and dictionary arrays [arrow-rs]
Posted by "Jefffrey (via GitHub)" <gi...@apache.org>.
Jefffrey commented on code in PR #10748:
URL: https://github.com/apache/arrow-rs/pull/10748#discussion_r3813099517
##########
arrow-arith/src/arity.rs:
##########
@@ -270,11 +270,17 @@ where
}
let len = a.len();
- if a.null_count() == 0 && b.null_count() == 0 {
+ // Physical nulls are not the whole story: a `RunArray` or `DictionaryArray` can have
+ // logical nulls in its values while its own null buffer is absent. `is_nullable` covers
+ // those, but is allowed to be conservative, so the union of the logical nulls can still
+ // be empty.
Review Comment:
i dont understand how it can be conservative
##########
arrow-arith/src/arity.rs:
##########
@@ -270,11 +270,17 @@ where
}
let len = a.len();
- if a.null_count() == 0 && b.null_count() == 0 {
+ // Physical nulls are not the whole story: a `RunArray` or `DictionaryArray` can have
+ // logical nulls in its values while its own null buffer is absent. `is_nullable` covers
+ // those, but is allowed to be conservative, so the union of the logical nulls can still
+ // be empty.
+ if !a.is_nullable() && !b.is_nullable() {
try_binary_no_nulls(len, a, b, op)
} else {
- let nulls =
- NullBuffer::union(a.logical_nulls().as_ref(), b.logical_nulls().as_ref()).unwrap();
+ let Some(nulls) = NullBuffer::union(a.logical_nulls().as_ref(), b.logical_nulls().as_ref())
Review Comment:
theres an unfortunate side effect in that we calculate the logical_nulls buffer twice; in terms of correctness this fix is good for ensuring we treat nulls properly, but there is a cost to calculating this logical nulls buffer for dictionaries specifically 🤔
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org