You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@datafusion.apache.org by "mattcuento (via GitHub)" <gi...@apache.org> on 2026/01/27 06:19:39 UTC
[PR] feat: Add spark-compat mode to integrate datafusion-spark features au… [datafusion-ballista]
mattcuento opened a new pull request, #1416:
URL: https://github.com/apache/datafusion-ballista/pull/1416
…tomatically
# Which issue does this PR close?
<!--
We generally require a GitHub issue to be filed for all bug fixes and enhancements and this helps us generate change logs for our releases. You can link an issue to this PR using the GitHub syntax. For example `Closes #123` indicates that this PR will close issue #123.
-->
Closes #1397.
# Rationale for this change
<!--
Why are you proposing this change? If this is already explained clearly in the issue then this section is not needed.
Explaining clearly why changes are proposed helps reviewers understand your changes and offer better suggestions for fixes.
-->
Exposing `datafusion-spark` functions as a 'bundled' feature with Ballista. This simplifies augmenting Ballista with spark features for quicker and more simple experimentation/adoption
# What changes are included in this PR?
<!--
There is no need to duplicate the description in the issue here but it is sometimes worth providing a summary of the individual changes in this PR.
-->
- Expand task definition to include table functions
- Introduce `spark-compat` feature to register all spark scalar/agg/window/table functions by default
# Are there any user-facing changes?
<!--
If there are user-facing changes then we may require documentation to be updated before approving the PR.
-->
<!--
If there are any breaking changes to public APIs, please add the `api change` label.
-->
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] feat: Add spark-compat mode to integrate datafusion-spark features au… [datafusion-ballista]
Posted by "andygrove (via GitHub)" <gi...@apache.org>.
andygrove commented on PR #1416:
URL: https://github.com/apache/datafusion-ballista/pull/1416#issuecomment-3828733473
> > I'm not certain that we have a well-defined place to document compile-time features, just docs for runtime configurations. I'd like to follow this up with a markdown file to describe our features
>
> maybe section in the readme file?
Adding Spark compatibility is a pretty major feature(!), so I think there should be a documentation update as part of this PR. I would be careful to explain it as "using Spark-compatible expressions" where available, rather than full compatibility.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] feat: Add spark-compat mode to integrate datafusion-spark features au… [datafusion-ballista]
Posted by "mattcuento (via GitHub)" <gi...@apache.org>.
mattcuento commented on PR #1416:
URL: https://github.com/apache/datafusion-ballista/pull/1416#issuecomment-3836432022
@milenkovicm
> also if we there are table functions to be supported, we may need to create encoder for them. so I'd suggest to ignore table functions for now and take them as a follow up if you agree
Yup agreed, happy to take the easy wins for scalar/window/aggregate for now and can see what we can do with table functions as a follow up!
@andygrove
> Adding Spark compatibility is a pretty major feature(!), so I think there should be a documentation update as part of this PR. I would be careful to explain it as "using Spark-compatible expressions" where available, rather than full compatibility.
Good point, I can write something up as a part of this PR 👍
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] feat: Add spark-compat mode to integrate datafusion-spark features au… [datafusion-ballista]
Posted by "milenkovicm (via GitHub)" <gi...@apache.org>.
milenkovicm commented on PR #1416:
URL: https://github.com/apache/datafusion-ballista/pull/1416#issuecomment-3805243066
> I'm not certain that we have a well-defined place to document compile-time features, just docs for runtime configurations. I'd like to follow this up with a markdown file to describe our features
maybe section in the readme file?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] feat: Add spark-compat mode to integrate datafusion-spark features au… [datafusion-ballista]
Posted by "mattcuento (via GitHub)" <gi...@apache.org>.
mattcuento commented on PR #1416:
URL: https://github.com/apache/datafusion-ballista/pull/1416#issuecomment-3864860989
No worries! Sounds good to me. If there's any PRs I can help with reviewing as a new contributor to help alleviate a bit, let me know!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] feat: Add spark-compat mode to integrate datafusion-spark features au… [datafusion-ballista]
Posted by "mattcuento (via GitHub)" <gi...@apache.org>.
mattcuento commented on code in PR #1416:
URL: https://github.com/apache/datafusion-ballista/pull/1416#discussion_r2791197316
##########
examples/Cargo.toml:
##########
@@ -71,6 +71,7 @@ uuid = { workspace = true }
[features]
default = ["substrait", "standalone"]
+spark-compat = ["ballista-core/spark-compat"]
Review Comment:
ah good call, added for both, did substrait in a separate commit. Ran both locally. Thanks!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] feat: Add spark-compat mode to integrate datafusion-spark features au… [datafusion-ballista]
Posted by "milenkovicm (via GitHub)" <gi...@apache.org>.
milenkovicm commented on code in PR #1416:
URL: https://github.com/apache/datafusion-ballista/pull/1416#discussion_r2749655377
##########
ballista/executor/src/executor_server.rs:
##########
@@ -661,6 +661,7 @@ impl<T: 'static + AsLogicalPlan, U: 'static + AsExecutionPlan> ExecutorGrpc
self.executor.function_registry.scalar_functions.clone(),
self.executor.function_registry.aggregate_functions.clone(),
self.executor.function_registry.window_functions.clone(),
+ self.executor.function_registry.table_functions.clone(),
Review Comment:
same as previous comment, do se need table functions here?
##########
ballista/core/src/registry.rs:
##########
@@ -34,29 +44,68 @@ pub struct BallistaFunctionRegistry {
pub aggregate_functions: HashMap<String, Arc<AggregateUDF>>,
/// Window user-defined functions.
pub window_functions: HashMap<String, Arc<WindowUDF>>,
+ /// Table functions.
Review Comment:
do we need table functions here? BFR is used at scheduler and executor after logical plan has been ser/de so table function should be resolved to TableProvider (or whatever is exactly) so table function should be encoded between client and scheduler
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] feat: Add spark-compat mode to integrate datafusion-spark features au… [datafusion-ballista]
Posted by "mattcuento (via GitHub)" <gi...@apache.org>.
mattcuento commented on PR #1416:
URL: https://github.com/apache/datafusion-ballista/pull/1416#issuecomment-3803336516
I'm not certain that we have a well-defined place to document compile-time features, just docs for runtime configurations. I'd like to follow this up with a markdown file to describe our features
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] feat: Add spark-compat mode to integrate datafusion-spark features au… [datafusion-ballista]
Posted by "milenkovicm (via GitHub)" <gi...@apache.org>.
milenkovicm commented on PR #1416:
URL: https://github.com/apache/datafusion-ballista/pull/1416#issuecomment-3885951938
Thanks @mattcuento
If we want to improve this further we can open new PRs
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] feat: Add spark-compat mode to integrate datafusion-spark features au… [datafusion-ballista]
Posted by "milenkovicm (via GitHub)" <gi...@apache.org>.
milenkovicm commented on PR #1416:
URL: https://github.com/apache/datafusion-ballista/pull/1416#issuecomment-3855892855
sorry for delay @mattcuento I'm behind reviews (and everything else :) ) had a quick look and it looks ok with me. Will try to do proper review tomorrow or over the weekend.
maybe @andygrove has some more ideas what should be addressed
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] feat: Add spark-compat mode to integrate datafusion-spark features au… [datafusion-ballista]
Posted by "mattcuento (via GitHub)" <gi...@apache.org>.
mattcuento commented on PR #1416:
URL: https://github.com/apache/datafusion-ballista/pull/1416#issuecomment-3812841614
> > I'm not certain that we have a well-defined place to document compile-time features, just docs for runtime configurations. I'd like to follow this up with a markdown file to describe our features
>
> maybe section in the readme file?
Threw up a separate PR: https://github.com/apache/datafusion-ballista/pull/1418/changes
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] feat: Add spark-compat mode to integrate datafusion-spark features au… [datafusion-ballista]
Posted by "mattcuento (via GitHub)" <gi...@apache.org>.
mattcuento commented on code in PR #1416:
URL: https://github.com/apache/datafusion-ballista/pull/1416#discussion_r2737643368
##########
ballista/core/src/extension.rs:
##########
@@ -177,6 +203,9 @@ impl SessionStateExt for SessionState {
.with_query_planner(Arc::new(planner))
.build();
+ #[cfg(feature = "spark-compat")]
+ let session_state = register_spark_functions(session_state);
Review Comment:
Instead of needing to call `register_spark_functions` from wherever we build state, I wonder if it'd be more elegant to push the registration of these functions/the spark-compat feature upstream to the datafusion `SessionStateBuilder`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] feat: Add spark-compat mode to integrate datafusion-spark features au… [datafusion-ballista]
Posted by "milenkovicm (via GitHub)" <gi...@apache.org>.
milenkovicm commented on code in PR #1416:
URL: https://github.com/apache/datafusion-ballista/pull/1416#discussion_r2789974194
##########
examples/Cargo.toml:
##########
@@ -71,6 +71,7 @@ uuid = { workspace = true }
[features]
default = ["substrait", "standalone"]
+spark-compat = ["ballista-core/spark-compat"]
Review Comment:
we need to add
```
[[example]]
name = "remote-spark-functions"
required-features = ["spark-compat"]
```
at the bottom of the file, so it triggers so it triggers build with spark-compact
please add same for `substrait` example and remove it from default
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] feat: Add spark-compat mode to integrate datafusion-spark features au… [datafusion-ballista]
Posted by "milenkovicm (via GitHub)" <gi...@apache.org>.
milenkovicm merged PR #1416:
URL: https://github.com/apache/datafusion-ballista/pull/1416
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org