You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@datafusion.apache.org by "andygrove (via GitHub)" <gi...@apache.org> on 2026/04/24 14:13:16 UTC
[PR] chore: promote array_insert to compatible and consolidate expression docs [datafusion-comet]
andygrove opened a new pull request, #4065:
URL: https://github.com/apache/datafusion-comet/pull/4065
## Which issue does this PR close?
Closes #.
## Rationale for this change
The `array_insert` expression was previously marked `Incompatible`, requiring `spark.comet.expr.allowIncompatible=true` to use. A recent audit against Spark 3.4.3, 3.5.8, and 4.0.1 found no behavioral differences beyond a cosmetic error-message mismatch when `pos=0` (Comet throws a generic error while Spark raises `INVALID_INDEX_OF_ZERO`). That falls within the `Compatible` contract ("may have known differences in some specific edge cases that are unlikely to be an issue for most users"), so users no longer need to opt in to incompatible expressions for this function.
The expression audit log has also accumulated a single entry and lives in its own file. Folding that metadata into the main expression support doc keeps audit findings next to the coverage matrix, and moving that doc into the contributor guide makes it discoverable through the Sphinx toctree.
## What changes are included in this PR?
- Change `CometArrayInsert.getSupportLevel` from `Incompatible(None)` to `Compatible()`.
- Move `docs/spark_expressions_support.md` into `docs/source/contributor-guide/` and add it to the contributor-guide toctree.
- Merge the `expression-audit-log.md` contents into the support doc as per-version sub-bullets under `array_insert`, then delete the audit log.
- Update the `audit-comet-expression` skill and `adding_a_new_expression.md` link to point at the new location.
## How are these changes tested?
Covered by existing `array_insert` tests added during the audit. No new behavior is introduced; this PR only relaxes the support-level gate and reorganizes docs.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] chore: promote array_insert to compatible and consolidate expression docs [datafusion-comet]
Posted by "andygrove (via GitHub)" <gi...@apache.org>.
andygrove commented on code in PR #4065:
URL: https://github.com/apache/datafusion-comet/pull/4065#discussion_r3138462061
##########
spark/src/main/scala/org/apache/comet/serde/arrays.scala:
##########
@@ -386,7 +386,7 @@ object CometArrayJoin extends CometExpressionSerde[ArrayJoin] {
object CometArrayInsert extends CometExpressionSerde[ArrayInsert] {
- override def getSupportLevel(expr: ArrayInsert): SupportLevel = Incompatible(None)
+ override def getSupportLevel(expr: ArrayInsert): SupportLevel = Compatible()
Review Comment:
I should have just removed the method, but don't want to wait for another CI run at this point. Will remove in a future PR along with others like this
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] chore: promote array_insert to compatible and consolidate expression docs [datafusion-comet]
Posted by "comphead (via GitHub)" <gi...@apache.org>.
comphead commented on code in PR #4065:
URL: https://github.com/apache/datafusion-comet/pull/4065#discussion_r3138964021
##########
docs/source/user-guide/latest/expressions.md:
##########
@@ -245,7 +245,7 @@ Comet supports using the following aggregate functions within window contexts wi
| ArrayDistinct | Yes | |
| ArrayExcept | No | |
| ArrayFilter | Yes | Only supports case where function is `IsNotNull` |
-| ArrayInsert | No | |
Review Comment:
👍
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] chore: promote array_insert to compatible and consolidate expression docs [datafusion-comet]
Posted by "andygrove (via GitHub)" <gi...@apache.org>.
andygrove merged PR #4065:
URL: https://github.com/apache/datafusion-comet/pull/4065
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] chore: promote array_insert to compatible and consolidate expression docs [datafusion-comet]
Posted by "andygrove (via GitHub)" <gi...@apache.org>.
andygrove commented on PR #4065:
URL: https://github.com/apache/datafusion-comet/pull/4065#issuecomment-4314932164
Thanks @comphead
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org