You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@datafusion.apache.org by "andygrove (via GitHub)" <gi...@apache.org> on 2026/01/15 02:38:54 UTC
[I] [Bug] array_remove returns null when element is null instead of removing null elements [datafusion-comet]
andygrove opened a new issue, #3173:
URL: https://github.com/apache/datafusion-comet/issues/3173
## Summary
When using `array_remove` with a null element to remove, Comet returns null instead of removing all null elements from the array as Spark does.
## Spark Specification
According to Spark's `array_remove` behavior:
- **Null removal element**: Removes all null elements from the array
- **Null array input**: Returns null (null intolerant behavior)
Example:
```sql
SELECT array_remove(array(1, null, 2, null, 3), null);
-- Spark returns: [1, 2, 3]
```
## Current Comet Behavior
The current implementation in `CometArrayRemove` uses a CASE WHEN pattern:
```scala
val caseWhenExpr = ExprOuterClass.CaseWhen
.newBuilder()
.addWhen(isNotNullExpr.get) // if element is not null
.addThen(arrayRemoveScalarExpr.get) // call array_remove_all
.setElseExpr(nullLiteralProto.get) // else return null <-- BUG
.build()
```
When the element to remove is null, instead of removing null elements, Comet returns null for the entire result.
## Expected Behavior
When `array_remove(arr, null)` is called, Comet should remove all null elements from the array (same as Spark).
## Suggested Fix
Modify the implementation to:
1. Detect when the element is null at evaluation time
2. Call a function that removes null values from the array (similar to `array_compact`)
3. Or create a custom Rust implementation that handles null elements correctly
## Impact
This could cause data correctness issues for users who rely on `array_remove` to filter out null values from arrays.
---
> **Note:** This issue was generated with AI assistance.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [I] [Bug] array_remove returns null when element is null instead of removing null elements [datafusion-comet]
Posted by "mbutrovich (via GitHub)" <gi...@apache.org>.
mbutrovich closed issue #3173: [Bug] array_remove returns null when element is null instead of removing null elements
URL: https://github.com/apache/datafusion-comet/issues/3173
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [I] [Bug] array_remove returns null when element is null instead of removing null elements [datafusion-comet]
Posted by "vigneshsiva11 (via GitHub)" <gi...@apache.org>.
vigneshsiva11 commented on issue #3173:
URL: https://github.com/apache/datafusion-comet/issues/3173#issuecomment-3765181829
Hello, @andygrove. I want to work on this problem.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [I] [Bug] array_remove returns null when element is null instead of removing null elements [datafusion-comet]
Posted by "andygrove (via GitHub)" <gi...@apache.org>.
andygrove commented on issue #3173:
URL: https://github.com/apache/datafusion-comet/issues/3173#issuecomment-4143629607
This expression (`ArrayRemove`) is now marked as Incompatible in Comet, which means it falls back to Spark by default and only runs natively when `spark.comet.expr.allowIncompatible=true` is explicitly set. Since default behavior is now correct (via Spark fallback), this is no longer a bug. Relabeling as an enhancement to track implementing compatible native behavior in the future.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org