You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@datafusion.apache.org by "andygrove (via GitHub)" <gi...@apache.org> on 2026/05/05 21:43:49 UTC

[PR] feat: add user-facing CometUDF registration for custom JVM UDFs [datafusion-comet]

andygrove opened a new pull request, #4233:
URL: https://github.com/apache/datafusion-comet/pull/4233

   ## Which issue does this PR close?
   
   Part of https://github.com/apache/datafusion-comet/issues/4193
   
   Builds on https://github.com/apache/datafusion-comet/pull/4232 (JVM UDF framework)
   
   ## Rationale for this change
   
   This PR enables end users to provide their own `CometUDF` implementations that operate on Arrow columnar data, registered alongside standard Spark UDFs. When Comet encounters a matching UDF during planning, it routes to the vectorized Arrow implementation instead of falling back to Spark's row-at-a-time execution.
   
   ## What changes are included in this PR?
   
   - **`CometUdfRegistry`** — a thread-safe registry mapping Spark UDF names to CometUDF implementation class names + metadata. Includes a convenience method that registers both with Spark and Comet in one call.
   - **`CometScalaUdf` serde handler** — intercepts `ScalaUDF` expressions in query planning; if the UDF name is registered in `CometUdfRegistry`, emits a `JvmScalarUdf` proto for native execution.
   - **User guide page** (`custom-jvm-udfs.md`) — documents how to write, register, and deploy custom JVM UDFs.
   
   ### User-facing API:
   
   ```scala
   // Register the Spark UDF (row-at-a-time fallback)
   spark.udf.register("is_positive", (x: Int) => x > 0)
   
   // Register the CometUDF (vectorized Arrow implementation)
   CometUdfRegistry.register(
     "is_positive",
     "com.example.IsPositiveUdf",
     BooleanType,
     nullable = true
   )
   ```
   
   ## How are these changes tested?
   
   - JVM compilation verified (`mvn compile` passes for common + spark modules)
   - End-to-end testing will come in a follow-up PR with a concrete UDF example
   
   ## Test plan
   
   - [ ] Verify `CometUdfRegistry.register` + `ScalaUDF` interception emits `JvmScalarUdf` proto
   - [ ] Verify fallback to Spark when UDF is not registered
   - [ ] Verify user guide renders correctly
   
   🤖 Generated with [Claude Code](https://claude.ai/code)


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org


Re: [PR] feat: add user-facing CometUDF registration for custom JVM UDFs [datafusion-comet]

Posted by "comphead (via GitHub)" <gi...@apache.org>.
comphead commented on code in PR #4233:
URL: https://github.com/apache/datafusion-comet/pull/4233#discussion_r3192109903


##########
spark/src/test/scala/org/apache/comet/CometUserUdfSuite.scala:
##########
@@ -0,0 +1,103 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.comet
+
+import org.apache.spark.sql.CometTestBase
+import org.apache.spark.sql.execution.adaptive.AdaptiveSparkPlanHelper
+import org.apache.spark.sql.types.LongType
+
+import org.apache.comet.udf.CometUdfRegistry
+
+class CometUserUdfSuite extends CometTestBase with AdaptiveSparkPlanHelper {
+
+  override def afterEach(): Unit = {
+    CometUdfRegistry.clear()
+    super.afterEach()
+  }
+
+  test("user CometUDF - basic integer doubling") {
+    CometUdfRegistry.register(

Review Comment:
   non critical, just from dev experience, we prob can have a single facade method that registers the function in all registries



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org


Re: [PR] feat: add user-facing CometUDF registration for custom JVM UDFs [datafusion-comet]

Posted by "comphead (via GitHub)" <gi...@apache.org>.
comphead commented on code in PR #4233:
URL: https://github.com/apache/datafusion-comet/pull/4233#discussion_r3192112297


##########
spark/src/test/scala/org/apache/comet/CometUserUdfSuite.scala:
##########
@@ -0,0 +1,103 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.comet
+
+import org.apache.spark.sql.CometTestBase
+import org.apache.spark.sql.execution.adaptive.AdaptiveSparkPlanHelper
+import org.apache.spark.sql.types.LongType
+
+import org.apache.comet.udf.CometUdfRegistry
+
+class CometUserUdfSuite extends CometTestBase with AdaptiveSparkPlanHelper {
+
+  override def afterEach(): Unit = {
+    CometUdfRegistry.clear()
+    super.afterEach()
+  }
+
+  test("user CometUDF - basic integer doubling") {
+    CometUdfRegistry.register(
+      "double_int",
+      "org.apache.comet.udf.DoubleIntUdf",
+      LongType,
+      nullable = true)
+    spark.udf.register("double_int", (x: Int) => x.toLong * 2L)
+
+    withTable("t") {
+      sql("CREATE TABLE t (x INT) USING parquet")
+      sql("INSERT INTO t VALUES (1), (2), (3), (NULL), (100)")
+      checkSparkAnswerAndOperator(sql("SELECT double_int(x) FROM t"))
+    }
+  }
+
+  test("user CometUDF - unregistered UDF falls back to Spark") {
+    spark.udf.register("triple_int", (x: Int) => x * 3)
+
+    withTable("t") {
+      sql("CREATE TABLE t (x INT) USING parquet")
+      sql("INSERT INTO t VALUES (1), (2), (3)")
+      // Should still produce correct results via Spark fallback
+      checkSparkAnswer(sql("SELECT triple_int(x) FROM t"))

Review Comment:
   should we check fallback msg? 



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org


Re: [PR] feat: add user-facing CometUDF registration for custom JVM UDFs [datafusion-comet]

Posted by "andygrove (via GitHub)" <gi...@apache.org>.
andygrove closed pull request #4233: feat: add user-facing CometUDF registration for custom JVM UDFs
URL: https://github.com/apache/datafusion-comet/pull/4233


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org


Re: [PR] feat: add user-facing CometUDF registration for custom JVM UDFs [datafusion-comet]

Posted by "mbutrovich (via GitHub)" <gi...@apache.org>.
mbutrovich commented on PR #4233:
URL: https://github.com/apache/datafusion-comet/pull/4233#issuecomment-4407698275

   Marking this as draft while we discuss and compare with #4267.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org