You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@datafusion.apache.org by "comphead (via GitHub)" <gi...@apache.org> on 2026/03/07 00:43:51 UTC
[I] `arrays_except` type mismatch [datafusion-comet]
comphead opened a new issue, #3646:
URL: https://github.com/apache/datafusion-comet/issues/3646
### Describe the bug
```
- sql-file: expressions/array/array_except.sql [parquet.enable.dictionary=true] *** FAILED *** (733 milliseconds)
java.lang.RuntimeException: Error executing SQL 'SELECT array_except(array(1, 2, 3), array(2, 3, 4)), array_except(array(1, 2), array()), array_except(array(), array(1)), array_except(cast(NULL as array<int>), array(1))'. Reason: 'Job aborted due to stage failure: Task 0 in stage 23.0 failed 1 times, most recent failure: Lost task 0.0 in stage 23.0 (TID 103) (192.168.4.142 executor driver): org.apache.comet.CometNativeException: array_except received incompatible types: List(Int32), List(non-null Int32)
```
This happens because exhaustive datatype check in datafusion in `check_datatypes()` function
### Steps to reproduce
_No response_
### Expected behavior
_No response_
### Additional context
_No response_
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [I] `arrays_except` type mismatch [datafusion-comet]
Posted by "yuboxx (via GitHub)" <gi...@apache.org>.
yuboxx commented on issue #3646:
URL: https://github.com/apache/datafusion-comet/issues/3646#issuecomment-4394243423
@comphead could you share the Spark and Comet version you used to produce this error? I can't seem to reproduce it with a local script running Spark 3.5.8 and the Comet local build based on latest of comet.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [I] `arrays_except` type mismatch [datafusion-comet]
Posted by "comphead (via GitHub)" <gi...@apache.org>.
comphead commented on issue #3646:
URL: https://github.com/apache/datafusion-comet/issues/3646#issuecomment-4398334769
Thanks @yuboxx
I think you can reproduce by running src/test/resources/sql-tests/expressions/array/array_except.sql
```
./mvnw test -Dsuites="org.apache.comet.CometSqlFileTestSuite array_except" -Dtest=none
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [I] `arrays_except` type mismatch [datafusion-comet]
Posted by "stantheman0128 (via GitHub)" <gi...@apache.org>.
stantheman0128 commented on issue #3646:
URL: https://github.com/apache/datafusion-comet/issues/3646#issuecomment-5840765241
I'd like to take this. It still reproduces on a recent main (`1c25b492`) with `allowIncompatible` on: removing the ignore on the literal query in `array_except.sql` fails with `array_except received incompatible types: List(Int32), List(non-null Int32)`.
The plan is to make both sides agree on a nullable element in the serde, reusing `normalizeContainerNullability` as suggested on #4237, and then re-enable that query. #5526 unifies nested element types for the set ops and leaves primitive elements alone, so I'll keep this to the list-level nullability and rebase onto it if it lands first.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [I] `arrays_except` type mismatch [datafusion-comet]
Posted by "n0r0shi (via GitHub)" <gi...@apache.org>.
n0r0shi commented on issue #3646:
URL: https://github.com/apache/datafusion-comet/issues/3646#issuecomment-4016451499
For this, I think there are 2 approaches. One is to fix upstream: make `check_datatypes()` ignore inner field nullability. Another is to fix in Comet: normalize inner `nullability` before calling `array_except`. Avoids upstream change but adds overhead.
I'd lean toward (1), but I'd like to confirm how others think.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [I] `arrays_except` type mismatch [datafusion-comet]
Posted by "comphead (via GitHub)" <gi...@apache.org>.
comphead commented on issue #3646:
URL: https://github.com/apache/datafusion-comet/issues/3646#issuecomment-4016959016
I tried to replace `.equals_datatype` to `.contains` which is more nullability friendly however it would still doesn't work this scenario.
`.contains` ignores nullability if the primary datatype is nullable, but
` let data_type = args[0].data_type();` doesn't guarantee exactly this datatype would be nullable.
I feel we can tweak `check_datatype` and if incoming datatypes are lists then take first NULLABLE datatype instead of just first
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [I] `arrays_except` type mismatch [datafusion-comet]
Posted by "contentforge-press (via GitHub)" <gi...@apache.org>.
contentforge-press commented on issue #3646:
URL: https://github.com/apache/datafusion-comet/issues/3646#issuecomment-5945421205
## Working on a fix for this! 🔧
I've analyzed the issue and identified the root cause:
### Problem
The error occurs because DataFusion's `array_except` function receives arrays with different element nullability:
- One array: `List(Int32)` (nullable elements)
- Other array: `List(non-null Int32)` (non-nullable elements)
### Root Cause
In `CometArrayExcept.convert`, we're passing the raw Spark expressions without normalizing the element type nullability. DataFusion is strict about type compatibility and rejects the mismatch.
### Proposed Fix
Normalize element type nullability before calling the native function:
```scala
// In CometArrayExcept.convert
val leftType = expr.left.dataType.asInstanceOf[ArrayType]
val rightType = expr.right.dataType.asInstanceOf[ArrayType]
// Cast to nullable if needed to ensure compatibility
val normalizedLeft = if (!leftType.containsNull) {
Cast(expr.left, ArrayType(leftType.elementType, containsNull = true))
} else {
expr.left
}
val normalizedRight = if (!rightType.containsNull) {
Cast(expr.right, ArrayType(rightType.elementType, containsNull = true))
} else {
expr.right
}
```
This ensures both arrays have the same element nullability before passing to DataFusion.
### Testing
The fix should enable the ignored test case in `array_except.sql`:
```sql
query ignore(https://github.com/apache/datafusion-comet/issues/3646)
SELECT array_except(array(1, 2, 3), array(2, 3, 4)), array_except(array(1, 2), array()), array_except(array(), array(1)), array_except(cast(NULL as array<int>), array(1))
```
### Next Steps
1. Implement the fix in `spark/src/main/scala/org/apache/comet/serde/arrays.scala`
2. Add unit tests
3. Enable the ignored test case
4. Verify with the full test suite
I'll submit a PR shortly. Working on it now!
cc: @andygrove @parthchandra @kazuyukitanimura
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org