You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@datafusion.apache.org by "andygrove (via GitHub)" <gi...@apache.org> on 2026/05/18 14:47:18 UTC
[PR] feat(udf)!: switch ScalarFunction.evaluate to ColumnarValue API (closes #62) [datafusion-java]
andygrove opened a new pull request, #64:
URL: https://github.com/apache/datafusion-java/pull/64
## Which issue does this PR close?
Closes #62.
## Rationale for this change
DataFusion's Rust `ScalarUDFImpl::invoke_with_args` speaks `ColumnarValue` (`Array` or `Scalar`) rather than raw Arrow arrays. The Java binding previously materialised every scalar arg to a length-N array before crossing the JNI boundary, which lost the scalar-vs-array distinction and forced nullary UDFs to learn the batch row count by some out-of-band channel (the workaround proposed in PR #57).
Aligning the Java API with the Rust enum eliminates the workaround: a nullary UDF can return `ColumnarValue.scalar(...)` and the framework broadcasts it, and a UDF that takes literals sees them as Scalars without per-row duplication.
## What changes are included in this PR?
- New `ColumnarValue` sealed interface (`Array`/`Scalar` records, factory enforcing length-1 invariant on scalars).
- New `ScalarFunctionArgs` record bundling `List<ColumnarValue>` and `rowCount`.
- `ScalarFunction.evaluate` is now `evaluate(BufferAllocator, ScalarFunctionArgs) -> ColumnarValue` (source-breaking).
- `JniBridge.invokeScalarUdf` rewritten to ship two struct arrays (length-N Array args + length-1 Scalar args) plus a `byte[] argKinds` positional mask, returning a `byte` indicating the result variant. JNI signature is now `(Lorg/apache/datafusion/ScalarFunction;JJJJ[BJJI)B`.
- Native `invoke_with_args` no longer materialises scalars; it partitions args by `ColumnarValue` variant and reconstructs the result from the returned kind byte via `ScalarValue::try_from_array`.
- `AddOneExample` and `docs/source/user-guide/scalar-udf.md` updated; new "Returning a Scalar" section added to the user guide.
## How are these changes tested?
`make test` — 135 tests pass (12 pre-existing skips). Existing `ScalarUdfTest` cases (`AddOne`, `Concat`, `Square`, error paths, volatility round-trip) adapted to the new signature, plus three new tests:
- `nullaryScalarReturnUdf_overMultiRowQuery_broadcasts` — a nullary `java_pi` returns `ColumnarValue.scalar(...)` and the framework expands it across rows, replacing the rowCount workaround.
- `scalarLiteralArg_arrivesAsScalarColumnarValue` — UDF asserts that a SQL literal arrives as `ColumnarValue.Scalar` (length 1), proving scalar-ness survives the FFI.
- `udfReturningScalar_isBroadcastByFramework` — explicit scalar-return path test.
Also covered by `cargo clippy --all-targets --workspace -- -D warnings` (clean) and `./mvnw spotless:check` (clean).
## Are there any user-facing changes?
Yes — source-breaking signature change to `ScalarFunction.evaluate`. Implementations must:
Before:
```java
public FieldVector evaluate(BufferAllocator allocator, List<FieldVector> args) {
IntVector in = (IntVector) args.get(0);
// ...
return out;
}
```
After:
```java
public ColumnarValue evaluate(BufferAllocator allocator, ScalarFunctionArgs args) {
IntVector in = (IntVector) args.args().get(0).vector();
// ...
return ColumnarValue.array(out);
}
```
Nullary or broadcast-style UDFs can return `ColumnarValue.scalar(...)` over a length-1 vector.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] feat(udf)!: switch ScalarFunction.evaluate to ColumnarValue API (closes #62) [datafusion-java]
Posted by "andygrove (via GitHub)" <gi...@apache.org>.
andygrove commented on PR #64:
URL: https://github.com/apache/datafusion-java/pull/64#issuecomment-4478843312
@pgwhalen fyi
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] feat(udf)!: switch ScalarFunction.evaluate to ColumnarValue API (closes #62) [datafusion-java]
Posted by "andygrove (via GitHub)" <gi...@apache.org>.
andygrove commented on PR #64:
URL: https://github.com/apache/datafusion-java/pull/64#issuecomment-4479723323
> Makes sense to me! My bindings apparently solved/noticed this problem by [passing the row count](https://github.com/pgwhalen/datafusion-java/blob/1c20733aa8b008315af6092a912b58ef3df6a482/datafusion-ffi-java/src/main/java/org/apache/arrow/datafusion/logical_expr/ScalarUDF.java#L134) like the other PR. Unfortunately I missed that that was actually important in your PR and assumed that you just had a cleaner version.
>
> All the more reason to emulate the rust API closely.
Thanks for the review! After chatting with @mbutrovich about making a similar change in Comet, he pointed out that we may still need row count in some cases, like for `uuid()` which has no input arguments.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] feat(udf)!: switch ScalarFunction.evaluate to ColumnarValue API (closes #62) [datafusion-java]
Posted by "andygrove (via GitHub)" <gi...@apache.org>.
andygrove merged PR #64:
URL: https://github.com/apache/datafusion-java/pull/64
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org