You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@datafusion.apache.org by "andygrove (via GitHub)" <gi...@apache.org> on 2026/06/24 14:22:12 UTC
[PR] feat: add array_prepend support [datafusion-comet]
andygrove opened a new pull request, #4716:
URL: https://github.com/apache/datafusion-comet/pull/4716
## Which issue does this PR close?
No dedicated issue. `array_prepend` was previously listed as planned (🔜) in the expression reference.
## Rationale for this change
`array_prepend` is a `RuntimeReplaceable` expression in Spark. In every version that defines it (3.5.8, 4.0.1, 4.1.1) the analyzer rewrites it to `ArrayInsert(arr, Literal(1), elem)` before Comet's serde runs, and that replacement is identical across all three versions. Comet already implements `ArrayInsert` as `Compatible`, so `array_prepend` already executes natively end to end. No new Scala serde, protobuf, or Rust code is required. This PR locks that in with a regression test and corrects the documented support status.
## What changes are included in this PR?
- Add a Comet SQL file test at `spark/src/test/resources/sql-tests/expressions/array/array_prepend.sql`. It is guarded with `MinSparkVersion: 3.5` because `array_prepend` was added in Spark 3.5.0 and does not exist in 3.4. Coverage includes column/literal/mixed arguments, NULL array yielding NULL, NULL element prepended, empty array, the int/string/boolean/double (including NaN)/long/multibyte-UTF8 element types, and the three cases Spark's own `DataFrameFunctionsSuite` exercises: type coercion (`array_prepend(array(1, 2), 1.23D)` -> `[1.23, 1.0, 2.0]`), nested-array elements, and binary elements.
- Mark `array_prepend` as supported (✅) in `docs/source/user-guide/latest/expressions.md`.
- Record the cross-version audit in `docs/source/contributor-guide/expression-audits/array_funcs.md`.
The `implement-comet-expression` skill was used to scaffold this work, including the `audit-comet-expression` cross-version comparison against Spark 3.4.3, 3.5.8, 4.0.1, and 4.1.1.
## How are these changes tested?
New SQL file test `array_prepend.sql`, run via `./mvnw test -Dsuites="org.apache.comet.CometSqlFileTestSuite array_prepend" -Dtest=none`. The framework runs each query under both Spark and Comet and asserts the results match and that the expression runs natively.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] feat: add array_prepend support [datafusion-comet]
Posted by "comphead (via GitHub)" <gi...@apache.org>.
comphead merged PR #4716:
URL: https://github.com/apache/datafusion-comet/pull/4716
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] feat: add array_prepend support [datafusion-comet]
Posted by "peterxcli (via GitHub)" <gi...@apache.org>.
peterxcli commented on code in PR #4716:
URL: https://github.com/apache/datafusion-comet/pull/4716#discussion_r3475407851
##########
spark/src/test/resources/sql-tests/expressions/array/array_prepend.sql:
##########
Review Comment:
I notice this sql would fail on comet with error sth like: "INVALID_INDEX_OF_ZERO", and the root cause is because comet's array_insert execution didnt Spark’s short-circuit evaluation.
````sql
statement
CREATE TABLE test_array_prepend(
arr ARRAY<INT>,
idx INT
) USING parquet
statement
INSERT INTO test_array_prepend VALUES
(NULL, 0),
(array(1), 1)
query
SELECT array_prepend(arr, element_at(array(9), idx))
FROM test_array_prepend
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] feat: add array_prepend support [datafusion-comet]
Posted by "peterxcli (via GitHub)" <gi...@apache.org>.
peterxcli commented on code in PR #4716:
URL: https://github.com/apache/datafusion-comet/pull/4716#discussion_r3475457041
##########
spark/src/test/resources/sql-tests/expressions/array/array_prepend.sql:
##########
Review Comment:
I opened a pr for this: https://github.com/apache/datafusion-comet/pull/4726. PTAL, Thanks!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org