You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@datafusion.apache.org by "comphead (via GitHub)" <gi...@apache.org> on 2025/07/25 20:23:33 UTC
[PR] fix: TrivialValueAccumulators to ignore nulls for `ignore nulls` [datafusion]
comphead opened a new pull request, #16918:
URL: https://github.com/apache/datafusion/pull/16918
## Which issue does this PR close?
<!--
We generally require a GitHub issue to be filed for all bug fixes and enhancements and this helps us generate change logs for our releases. You can link an issue to this PR using the GitHub syntax. For example `Closes #123` indicates that this PR will close issue #123.
-->
- Closes #.
Related to https://github.com/apache/datafusion/issues/16235 and https://github.com/apache/datafusion-comet/pull/2040
## Rationale for this change
After introducing TrivialValueAccumulators in https://github.com/apache/datafusion/pull/16217 we found a `ignore nulls` test failed in Comet. I cannot reproduce the same in DataFusion, although I created the similar environment in DF.
If the key is spread across 2 files and batch_size = 1
file1
|a | b|
|---|---|
|2 | 1|
file2
|a | b|
|---|---|
|2 | 2 |
And run query
```
SELECT a, last_value(case when b==1 then null else b end) IGNORE NULLS FROM t1 GROUP BY a order by a;
```
or
```
SELECT a, first_value(case when b==1 then null else b end) IGNORE NULLS FROM t1 GROUP BY a order by a;
```
The query actually returns
|a | b|
|---|---|
|2 | null |
While debugging a `merge_batch` I found Comet sends sometimes a state `null, true` whereas DF never does that and send s `nulls, false` instead. Probably it is explained that Comet calls physical operations directly, without optimizations.
So this fix is more guess but it works for Comet and Datafusion and unblocks the Comet migration.
Appreciate @ozankabak @alamb @berkaysynnada @mbutrovich for ideas for better fix (if any)
<!--
Why are you proposing this change? If this is already explained clearly in the issue then this section is not needed.
Explaining clearly why changes are proposed helps reviewers understand your changes and offer better suggestions for fixes.
-->
## What changes are included in this PR?
<!--
There is no need to duplicate the description in the issue here but it is sometimes worth providing a summary of the individual changes in this PR.
-->
## Are these changes tested?
<!--
We typically require tests for all PRs in order to:
1. Prevent the code from being accidentally broken by subsequent changes
2. Serve as another way to document the expected behavior of the code
If tests are not included in your PR, please explain why (for example, are they covered by existing tests)?
-->
## Are there any user-facing changes?
<!--
If there are user-facing changes then we may require documentation to be updated before approving the PR.
-->
<!--
If there are any breaking changes to public APIs, please add the `api change` label.
-->
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] fix: `TrivialValueAccumulators` to ignore null states for `ignore nulls` [datafusion]
Posted by "mbutrovich (via GitHub)" <gi...@apache.org>.
mbutrovich commented on PR #16918:
URL: https://github.com/apache/datafusion/pull/16918#issuecomment-3140641647
I've documented my investigation with a new issue on the Comet repo:
https://github.com/apache/datafusion-comet/issues/2051
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] fix: `TrivialValueAccumulators` to ignore null states for `ignore nulls` [datafusion]
Posted by "comphead (via GitHub)" <gi...@apache.org>.
comphead commented on PR #16918:
URL: https://github.com/apache/datafusion/pull/16918#issuecomment-3127884680
> @comphead thanks for the detailed explanation. The `state()`'s are taken from Comet, right?
Thanks @berkaysynnada, the state is taken from TrivialAccumulator. What Comet does is just call physical `LastValue` function https://github.com/apache/datafusion-comet/blob/320ce55eec9f4c846ad7a68bb69c182ae1fcd3ce/native/core/src/execution/planner.rs#L1859 so all internal details, including accumulators, states, rowhash are coming from DF.
> that’s an invalid state, and we shouldn’t be trying to fix it here IMO
I suppose if we know this is an invalid state we still should ignore it like in this PR, if there is a mistake in planner/optimizer it can lead to silent data corruption
I'm trying to reproduce this with unit test if I sent batches with exacly same order and content like Comet does.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] fix: `TrivialValueAccumulators` to ignore null states for `ignore nulls` [datafusion]
Posted by "comphead (via GitHub)" <gi...@apache.org>.
comphead commented on PR #16918:
URL: https://github.com/apache/datafusion/pull/16918#issuecomment-3133671551
I'm still on that, my Comet test likely showing some race conditions. Sounds like depending on incoming data order DF can give incorrect result, but I still trying to create a stable test in DF
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] fix: `TrivialValueAccumulators` to ignore null states for `ignore nulls` [datafusion]
Posted by "comphead (via GitHub)" <gi...@apache.org>.
comphead closed pull request #16918: fix: `TrivialValueAccumulators` to ignore null states for `ignore nulls`
URL: https://github.com/apache/datafusion/pull/16918
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] fix: `TrivialValueAccumulators` to ignore null states for `ignore nulls` [datafusion]
Posted by "berkaysynnada (via GitHub)" <gi...@apache.org>.
berkaysynnada commented on PR #16918:
URL: https://github.com/apache/datafusion/pull/16918#issuecomment-3126923002
@comphead thanks for the detailed explanation. The `state()`'s are taken from Comet, right? and Comet doesn't actually use my proposed change
```rust
fn state(&mut self) -> Result<Vec<ScalarValue>> {
if self.ignore_nulls && self.last.is_null() {
self.is_set = false;
};
Ok(vec![self.last.clone(), ScalarValue::from(self.is_set)])
}
```
?
If that’s the case, the problem seems to originate in Comet, because we shouldn’t be receiving:
``` &states = [
PrimitiveArray<Int32>
[
null,
],
BooleanArray
[
true,
],
]
```
that’s an invalid state, and we shouldn’t be trying to fix it here IMO
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] fix: `TrivialValueAccumulators` to ignore null states for `ignore nulls` [datafusion]
Posted by "comphead (via GitHub)" <gi...@apache.org>.
comphead commented on PR #16918:
URL: https://github.com/apache/datafusion/pull/16918#issuecomment-3141850005
Closing this PR in favor of https://github.com/apache/datafusion-comet/pull/2052
We suspect a FFI problem that flips the boolean `is_set` flag from false to true, and multithreaded execution makes a problem non deterministic, depending on states order.
@mbutrovich provided nice investigation in https://github.com/apache/datafusion-comet/pull/2051 if anyone interested in FFI boundaries
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] fix: `TrivialValueAccumulators` to ignore null states for `ignore nulls` [datafusion]
Posted by "mbutrovich (via GitHub)" <gi...@apache.org>.
mbutrovich commented on PR #16918:
URL: https://github.com/apache/datafusion/pull/16918#issuecomment-3120341095
I'm trying to put together a concise unit test based on the batches I see going through `update_batch` and `merge_batch` in the failing Comet test, but can't make sense of what I'm seeing. A `TrivialFirstValueAccumulator` has `is_set` to false, but when its state ended up at `merge_batch` the second array (which contains `is_set` as boolean) is flipped to true, so I'm trying to sort out how that's happening.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] fix: `TrivialValueAccumulators` to ignore null states for `ignore nulls` [datafusion]
Posted by "berkaysynnada (via GitHub)" <gi...@apache.org>.
berkaysynnada commented on PR #16918:
URL: https://github.com/apache/datafusion/pull/16918#issuecomment-3121734839
> Comet sends sometimes a state null, true whereas DF never does that and send s nulls, false instead. Probably it is explained that Comet calls physical operations directly, without optimizations
I couldn't understand this, can you explain a bit more?
```rust
fn state(&mut self) -> Result<Vec<ScalarValue>> {
let is_set = if self.ignore_nulls && self.first.is_null() {
false
} else {
self.is_set
};
Ok(vec![self.first.clone(), ScalarValue::from(is_set)])
}
```
does it mean converting `fn state()`'s such would fix the problem?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] fix: `TrivialValueAccumulators` to ignore null states for `ignore nulls` [datafusion]
Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on PR #16918:
URL: https://github.com/apache/datafusion/pull/16918#issuecomment-3120300842
> Existing tests, as I cannot reproduce it in DataFusion I'm not able to add a new test
I don't think it is covered by existing tests as they all pass with and without this code change
If we can't make an end to end/sql test I think at least a unit test to make sure we don't break it in the future would be a good idea
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] fix: `TrivialValueAccumulators` to ignore null states for `ignore nulls` [datafusion]
Posted by "ozankabak (via GitHub)" <gi...@apache.org>.
ozankabak commented on PR #16918:
URL: https://github.com/apache/datafusion/pull/16918#issuecomment-3129073690
> I suppose if we know this is an invalid state we still should ignore it like in this PR, if there is a mistake in planner/optimizer it can lead to silent data corruption
I think we should add a `debug_assert!` so that people can catch where/how the invalid state emerges. I don't think it should be swept under the rug
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] fix: `TrivialValueAccumulators` to ignore null states for `ignore nulls` [datafusion]
Posted by "comphead (via GitHub)" <gi...@apache.org>.
comphead commented on PR #16918:
URL: https://github.com/apache/datafusion/pull/16918#issuecomment-3122164585
> does it mean converting `fn state()`'s such would fix the problem?
Thanks @berkaysynnada I tried the code below(which is more concise version of your snippet), it didn't help
```
fn state(&mut self) -> Result<Vec<ScalarValue>> {
if self.ignore_nulls && self.last.is_null() {
self.is_set = false;
};
Ok(vec![self.last.clone(), ScalarValue::from(self.is_set)])
}
```
> I couldn't understand this, can you explain a bit more?
Please find some more details for this issue
First file
|a | b|
|---|---|
|0 | 1|
|0 | 2|
|2 | 1|
Second file
|a | b|
|---|---|
|1 | 1|
|1 | 2|
|2 | 2|
See the row with key 2 split across 2 files.
I'm running the same query
```
SELECT a, last_value(case when b==1 then null else b end) IGNORE NULLS FROM t1 GROUP BY a;
```
and output values from `update_batch` and states from `merge_batch` from `TrivialLastValueAccumulator`
DF
```
> SELECT a, last_value(case when b==1 then null else b end) IGNORE NULLS FROM t1 GROUP BY a;
[datafusion/functions-aggregate/src/first_last.rs:1302:9] &values = [
PrimitiveArray<Int32>
[
2,
],
]
[datafusion/functions-aggregate/src/first_last.rs:1302:9] &values = [
PrimitiveArray<Int32>
[
null,
],
]
[datafusion/functions-aggregate/src/first_last.rs:1302:9] &values = [
PrimitiveArray<Int32>
[
null,
],
]
[datafusion/functions-aggregate/src/first_last.rs:1302:9] &values = [
PrimitiveArray<Int32>
[
2,
],
]
[datafusion/functions-aggregate/src/first_last.rs:1302:9] &values = [
PrimitiveArray<Int32>
[
null,
],
]
[datafusion/functions-aggregate/src/first_last.rs:1302:9] &values = [
PrimitiveArray<Int32>
[
2,
],
]
[datafusion/functions-aggregate/src/first_last.rs:1327:9] &states = [
PrimitiveArray<Int32>
[
2,
],
BooleanArray
[
true,
],
]
[datafusion/functions-aggregate/src/first_last.rs:1327:9] &states = [
PrimitiveArray<Int32>
[
null,
],
BooleanArray
[
false,
],
]
[datafusion/functions-aggregate/src/first_last.rs:1327:9] &states = [
PrimitiveArray<Int32>
[
2,
],
BooleanArray
[
true,
],
]
[datafusion/functions-aggregate/src/first_last.rs:1327:9] &states = [
PrimitiveArray<Int32>
[
2,
],
BooleanArray
[
true,
],
]
[datafusion/functions-aggregate/src/first_last.rs:1327:9] &states = [
PrimitiveArray<Int32>
[
null,
],
BooleanArray
[
false,
],
]
+---+----------------------------------------------------------------------------+
| a | last_value(CASE WHEN t1.b = Int64(1) THEN NULL ELSE t1.b END) IGNORE NULLS |
+---+----------------------------------------------------------------------------+
| 2 | 2 |
| 1 | 2 |
| 0 | 2 |
+---+----------------------------------------------------------------------------+
```
Comet
```
25/07/26 10:06:39 INFO core/src/lib.rs: Comet native library version 0.10.0 initialized
[/Users/ovoievodin/.cargo/git/checkouts/datafusion-11a8b534adb6bd68/afb9099/datafusion/functions-aggregate/src/first_last.rs:1301:9] &values = [
PrimitiveArray<Int32>
[
null,
],
]
[/Users/ovoievodin/.cargo/git/checkouts/datafusion-11a8b534adb6bd68/afb9099/datafusion/functions-aggregate/src/first_last.rs:1301:9] &values = [
PrimitiveArray<Int32>
[
null,
],
]
[/Users/ovoievodin/.cargo/git/checkouts/datafusion-11a8b534adb6bd68/afb9099/datafusion/functions-aggregate/src/first_last.rs:1301:9] &values = [
PrimitiveArray<Int32>
[
2,
],
]
[/Users/ovoievodin/.cargo/git/checkouts/datafusion-11a8b534adb6bd68/afb9099/datafusion/functions-aggregate/src/first_last.rs:1301:9] &values = [
PrimitiveArray<Int32>
[
2,
],
]
[/Users/ovoievodin/.cargo/git/checkouts/datafusion-11a8b534adb6bd68/afb9099/datafusion/functions-aggregate/src/first_last.rs:1301:9] &values = [
PrimitiveArray<Int32>
[
null,
],
]
[/Users/ovoievodin/.cargo/git/checkouts/datafusion-11a8b534adb6bd68/afb9099/datafusion/functions-aggregate/src/first_last.rs:1301:9] &values = [
PrimitiveArray<Int32>
[
2,
],
]
[/Users/ovoievodin/.cargo/git/checkouts/datafusion-11a8b534adb6bd68/afb9099/datafusion/functions-aggregate/src/first_last.rs:1327:9] &states = [
PrimitiveArray<Int32>
[
2,
],
BooleanArray
[
true,
],
]
[/Users/ovoievodin/.cargo/git/checkouts/datafusion-11a8b534adb6bd68/afb9099/datafusion/functions-aggregate/src/first_last.rs:1327:9] &states = [
PrimitiveArray<Int32>
[
2,
],
BooleanArray
[
true,
],
]
[/Users/ovoievodin/.cargo/git/checkouts/datafusion-11a8b534adb6bd68/afb9099/datafusion/functions-aggregate/src/first_last.rs:1327:9] &states = [
PrimitiveArray<Int32>
[
2,
],
BooleanArray
[
true,
],
]
[/Users/ovoievodin/.cargo/git/checkouts/datafusion-11a8b534adb6bd68/afb9099/datafusion/functions-aggregate/src/first_last.rs:1327:9] &states = [
PrimitiveArray<Int32>
[
null,
],
BooleanArray
[
true,
],
]
[/Users/ovoievodin/.cargo/git/checkouts/datafusion-11a8b534adb6bd68/afb9099/datafusion/functions-aggregate/src/first_last.rs:1327:9] &states = [
PrimitiveArray<Int32>
[
2,
],
BooleanArray
[
true,
],
]
[/Users/ovoievodin/.cargo/git/checkouts/datafusion-11a8b534adb6bd68/afb9099/datafusion/functions-aggregate/src/first_last.rs:1327:9] &states = [
PrimitiveArray<Int32>
[
2,
],
BooleanArray
[
true,
],
]
[/Users/ovoievodin/.cargo/git/checkouts/datafusion-11a8b534adb6bd68/afb9099/datafusion/functions-aggregate/src/first_last.rs:1327:9] &states = [
PrimitiveArray<Int32>
[
2,
],
BooleanArray
[
true,
],
]
[/Users/ovoievodin/.cargo/git/checkouts/datafusion-11a8b534adb6bd68/afb9099/datafusion/functions-aggregate/src/first_last.rs:1327:9] &states = [
PrimitiveArray<Int32>
[
null,
],
BooleanArray
[
true,
],
]
```
So values in `update_batch` are the same but ordering is different, not sure if this matters, tbh
for states in `merge_batch`
the Comet sends
```
&states = [
PrimitiveArray<Int32>
[
null,
],
BooleanArray
[
true,
],
]
```
which I believe confuses the accumulator as for DF the state sent as
```
&states = [
PrimitiveArray<Int32>
[
null,
],
BooleanArray
[
false,
],
]
```
and not applied as it is false.
Hope this explains
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org