You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by "rafafrdz (via GitHub)" <gi...@apache.org> on 2026/03/29 08:50:01 UTC
[PR] [SPARK-52428] Add UDF registration and invocation support [spark-connect-rust]
rafafrdz opened a new pull request, #39:
URL: https://github.com/apache/spark-connect-rust/pull/39
## Summary
- Add `udf.rs` module with `UdfRegistration` struct accessible via `spark.udf()`
- `register_java(name, class_name, return_type)` — register Java UDFs by class name
- `register_java_udaf(name, class_name)` — register Java UDAFs
- Add `call_udf(name, args)` function — calls registered UDFs (`is_user_defined_function=true`)
- Add `call_function(name, args)` function — calls any function by name (uses `CallFunction` proto)
```rust
// Register a Java UDF
spark.udf().register_java("my_udf", "com.example.MyUDF", DataType::String).await?;
// Use it in queries
df.select([call_udf("my_udf", vec![col("input")])]);
```
## Test plan
- [x] `test_call_udf_expression_structure` — verifies proto structure with args
- [x] `test_call_udf_no_args` — verifies empty args case
- [x] `test_call_function_expression_structure` — verifies CallFunction proto
- [x] `test_call_udf_is_user_defined` — verifies `is_user_defined_function` flag differs from `invoke_func`
- [x] `test_register_java_udf_proto_structure` — verifies Java UDF proto fields
- [x] `test_register_java_udf_return_type` — verifies DataType conversion in proto
- [x] `test_register_java_udaf_proto_structure` — verifies UDAF with `aggregate=true`
- [x] `test_register_java_udf_creates_valid_command` — verifies RegisterFunction command wrapping
- [x] `cargo build` passes
- [x] `cargo clippy` passes
- [x] `cargo fmt -- --check` passes
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-52428] Add UDF registration and invocation support [spark-connect-rust]
Posted by "HyukjinKwon (via GitHub)" <gi...@apache.org>.
HyukjinKwon closed pull request #39: [SPARK-52428] Add UDF registration and invocation support
URL: https://github.com/apache/spark-connect-rust/pull/39
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-52428] Add UDF registration and invocation support [spark-connect-rust]
Posted by "HyukjinKwon (via GitHub)" <gi...@apache.org>.
HyukjinKwon commented on PR #39:
URL: https://github.com/apache/spark-connect-rust/pull/39#issuecomment-5403899167
Superseded by #50, which delivers the consolidated Rust core Spark Connect client (now merged to `master`) and incorporates this work. Closing to keep the queue focused on the drop-in Python wrapper (#51). Thanks for the contribution — please reopen if anything here isn't covered by #50.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org