You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@datafusion.apache.org by "andygrove (via GitHub)" <gi...@apache.org> on 2026/06/13 14:28:37 UTC
[PR] docs: adopt issue #4419 terminology in Scala/Java UDF guide [datafusion-comet]
andygrove opened a new pull request, #4651:
URL: https://github.com/apache/datafusion-comet/pull/4651
## Which issue does this PR close?
Part of #4419.
## Rationale for this change
Issue #4419 proposes a terminology framework to remove the overloaded, bare use of "native" from Comet's docs. The "Scala UDF and Java UDF Support" page describes how UDFs participate in Comet execution and used "native Comet path" / "native path" / "native execution" loosely. This page is a good follow-up to the Understanding Comet Plans cleanup because Scala/Java UDFs are JVM-implemented (codegen'd) yet still run inside the Comet pipeline, so precise wording matters here.
This PR keeps the change scoped to that one page.
## What changes are included in this PR?
In `docs/source/user-guide/latest/scala_java_udfs.md`:
- Use "Comet pipeline" for pipeline membership in place of "native Comet path" / "native path".
- Describe the surrounding accelerated operators as "Rust-implemented" rather than "native".
- Replace "for native execution" / "runs as one native unit" with "to keep execution in the Comet pipeline" / "runs as one unit in the Comet pipeline".
## How are these changes tested?
Documentation-only change. Verified with `prettier --check`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] docs: adopt issue #4419 terminology in Scala/Java UDF guide [datafusion-comet]
Posted by "mbutrovich (via GitHub)" <gi...@apache.org>.
mbutrovich merged PR #4651:
URL: https://github.com/apache/datafusion-comet/pull/4651
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] docs: adopt issue #4419 terminology in Scala/Java UDF guide [datafusion-comet]
Posted by "comphead (via GitHub)" <gi...@apache.org>.
comphead commented on code in PR #4651:
URL: https://github.com/apache/datafusion-comet/pull/4651#discussion_r3408707939
##########
docs/source/user-guide/latest/scala_java_udfs.md:
##########
@@ -19,24 +19,24 @@
# Scala UDF and Java UDF Support
-Comet executes Spark's Scala and Java [scalar user-defined functions (UDFs)](https://spark.apache.org/docs/latest/sql-ref-functions-udf-scalar.html) on the native Comet path. The presence of a UDF does not force the enclosing operator off the native path; surrounding native operators stay native.
+Comet executes Spark's Scala and Java [scalar user-defined functions (UDFs)](https://spark.apache.org/docs/latest/sql-ref-functions-udf-scalar.html) within the Comet pipeline. The presence of a UDF does not force the enclosing operator out of the Comet pipeline; surrounding Rust-implemented operators stay in the pipeline.
This page covers Spark's `ScalaUDF` (Scala `udf(...)`, `spark.udf.register(...)` over Scala or Java functional interfaces, and SQL `CREATE FUNCTION ... AS 'com.example.MyUDF'`). Other UDF kinds (Python / Pandas, Hive, aggregate) are out of scope and continue to fall back to Spark.
This feature is enabled by default. Set `spark.comet.exec.scalaUDF.codegen.enabled` to `false` to route plans containing a `ScalaUDF` back to Spark for the enclosing operator.
## Configuration
-| Key | Default | Description |
-| ------------------------------------------- | ------- | ------------------------------------------------------------------------------------------------------------------ |
-| `spark.comet.exec.scalaUDF.codegen.enabled` | `true` | When `true`, eligible `ScalaUDF`s run on the Comet path. When `false`, the enclosing operator falls back to Spark. |
+| Key | Default | Description |
+| ------------------------------------------- | ------- | ---------------------------------------------------------------------------------------------------------------------- |
+| `spark.comet.exec.scalaUDF.codegen.enabled` | `true` | When `true`, eligible `ScalaUDF`s run in the Comet pipeline. When `false`, the enclosing operator falls back to Spark. |
Review Comment:
perhaps we need to highlight this param contols a fallback for user defined functions codegen, having another config for builtin Spark functions codegen?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org