You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@datafusion.apache.org by "andygrove (via GitHub)" <gi...@apache.org> on 2026/04/02 22:36:37 UTC
[PR] Mark array_compact as Compatible and improve test coverage [datafusion-comet]
andygrove opened a new pull request, #3889:
URL: https://github.com/apache/datafusion-comet/pull/3889
## Which issue does this PR close?
Improves test coverage and correctness of `array_compact` support level.
## Rationale for this change
`CometArrayCompact.getSupportLevel` returned `Incompatible(None)`, but this was incorrect:
- On Spark 3.x, the existing tests pass without `allowIncompatible`, confirming the implementation matches Spark behavior.
- On Spark 4.0, `ArrayCompact` is a `RuntimeReplaceable` that Spark rewrites to `ArrayFilter(child, IsNotNull(lambda))` before it reaches Comet's serde, so `CometArrayCompact` is never invoked. The `CometArrayFilter` serde already handles this case as `Compatible()`.
The test coverage was also limited to integer arrays only.
## What changes are included in this PR?
- Change `CometArrayCompact.getSupportLevel` from `Incompatible(None)` to `Compatible()`
- Expand `array_compact.sql` test coverage:
- Add string element type tests (via table with column references)
- Add double element type test
- Add nested array type test (removes null arrays from outer, preserves null elements in inner)
- Add empty string preservation test (not confused with nulls)
- Upgrade existing queries from `spark_answer_only` to `query` mode to assert native execution
- Remove unnecessary `ConfigMatrix` for dictionary encoding
## How are these changes tested?
SQL file tests in `array_compact.sql` run via `CometSqlFileTestSuite`. All tests pass.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] feat: Mark array_compact as Compatible and improve test coverage [datafusion-comet]
Posted by "kazuyukitanimura (via GitHub)" <gi...@apache.org>.
kazuyukitanimura commented on PR #3889:
URL: https://github.com/apache/datafusion-comet/pull/3889#issuecomment-4184797186
I think we will need #3796 as well
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] feat: Mark array_compact as Compatible and improve test coverage [datafusion-comet]
Posted by "kazuyukitanimura (via GitHub)" <gi...@apache.org>.
kazuyukitanimura commented on code in PR #3889:
URL: https://github.com/apache/datafusion-comet/pull/3889#discussion_r3034012476
##########
spark/src/main/scala/org/apache/comet/serde/arrays.scala:
##########
@@ -313,7 +313,7 @@ object CometArrayRepeat extends CometExpressionSerde[ArrayRepeat] {
object CometArrayCompact extends CometExpressionSerde[Expression] {
- override def getSupportLevel(expr: Expression): SupportLevel = Incompatible(None)
+ override def getSupportLevel(expr: Expression): SupportLevel = Compatible()
Review Comment:
I think we can remove `getSupportLevel` entirely
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] feat: Mark array_compact as Compatible and improve test coverage [datafusion-comet]
Posted by "andygrove (via GitHub)" <gi...@apache.org>.
andygrove merged PR #3889:
URL: https://github.com/apache/datafusion-comet/pull/3889
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org