You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@datafusion.apache.org by "andygrove (via GitHub)" <gi...@apache.org> on 2026/05/12 17:23:09 UTC
[PR] feat: add JVM UDF engine for Spark JSON expressions [datafusion-comet]
andygrove opened a new pull request, #4305:
URL: https://github.com/apache/datafusion-comet/pull/4305
## Which issue does this PR close?
Closes #.
## Rationale for this change
The native Rust JSON expressions in Comet have known compatibility gaps with Spark and feature restrictions (e.g., `from_json` only supports PERMISSIVE mode with simple schemas; `to_json` does not handle map/array at the top level; `get_json_object` differs from Spark on certain path expressions). Routing through Spark's own expression classes via the JVM UDF framework (introduced in #4232 and exercised for regexp in #4239) guarantees byte-exact compatibility, at the cost of a JNI roundtrip per batch. This is opt-in via a new config and the existing native path remains the default.
## What changes are included in this PR?
- New config `spark.comet.exec.json.engine` (`rust` default, `java` experimental).
- Three new `CometUDF` implementations under `common/src/main/scala/org/apache/comet/udf/`: `GetJsonObjectUDF`, `FromJsonUDF`, `ToJsonUDF`.
- Each UDF builds a fresh Spark expression per `evaluate` call. Spark's JSON evaluators (`GetJsonObjectEvaluator`, `StructsToJsonEvaluator`, `JsonToStructsEvaluator`) hold mutable per-row state, and the JVM UDF framework shares one UDF instance across native worker threads, so a cached cross-thread expression races on its evaluator state.
- `from_json` and `to_json` use a serde-side `CometLambdaRegistry` to pass the configured Spark expression (schema, options, timezone) to the UDF. The serde rebinds the child to `BoundReference(0)` so the UDF can call `eval(row)` against a single-column wrapper row.
- Serde dispatch in three existing serde objects: `CometGetJsonObject` (`strings.scala`), `CometStructsToJson` and `CometJsonToStructs` (`structs.scala`) branch on the engine.
- Compatibility doc `docs/source/user-guide/latest/compatibility/json.md` and updated expressions table.
`json_array_length` and `json_object_keys` are intentionally out of scope. Both are `RuntimeReplaceable` in Spark 4.x and Catalyst's `ReplaceExpressions` rewrites them to `StaticInvoke` before Comet sees the plan, so `classOf[LengthOfJsonArray]` / `classOf[JsonObjectKeys]` registrations never match. Adding support requires recognizing the rewritten `StaticInvoke` form in Comet's serde dispatch and is left to a follow-up.
This PR was scaffolded with the project's brainstorming, writing-plans, and subagent-driven-development skills.
## How are these changes tested?
- Three unit suites under `spark/src/test/scala/org/apache/comet/udf/` covering valid input, null input, empty input vectors, malformed JSON, and (for `from_json`) PERMISSIVE mode all-null struct output.
- `CometJsonJvmSuite` integration suite extending `CometTestBase` that flips `spark.comet.exec.json.engine=java` and runs the three expressions through `checkSparkAnswerAndOperator`, asserting Comet exec and byte-exact match with Spark.
- Existing `CometJsonExpressionSuite` runs unchanged on default `engine=rust` to confirm no regression.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] feat: 100% Spark-compatible JSON support via codegen dispatcher [datafusion-comet]
Posted by "mbutrovich (via GitHub)" <gi...@apache.org>.
mbutrovich commented on code in PR #4305:
URL: https://github.com/apache/datafusion-comet/pull/4305#discussion_r3359173586
##########
spark/src/main/scala/org/apache/comet/serde/CometScalaUDF.scala:
##########
@@ -78,10 +78,20 @@ object CometScalaUDF extends CometExpressionSerde[ScalaUDF] {
return None
}
+ // `RuntimeReplaceable` expressions (e.g. Spark 4's `StructsToJson`) have a `doGenCode` that
+ // always throws "Cannot generate code for expression". Catalyst's `ReplaceExpressions` rule
+ // normally rewrites them to their `replacement` form before codegen runs. Comet's serde
+ // sometimes works with the pre-rewrite form (via shim reconstruction) for matching purposes,
+ // so unwrap to the replacement here before binding so the kernel compiles.
+ val target = expr match {
Review Comment:
Interesting find.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] feat: 100% Spark-compatible JSON support via codegen dispatcher [datafusion-comet]
Posted by "comphead (via GitHub)" <gi...@apache.org>.
comphead commented on code in PR #4305:
URL: https://github.com/apache/datafusion-comet/pull/4305#discussion_r3357645907
##########
spark/src/test/resources/sql-tests/expressions/struct/structs_to_json.sql:
##########
@@ -30,8 +31,10 @@ SELECT to_json(named_struct('a', a, 'b', b, 'f', f, 'd', d)) FROM test_to_json
query
SELECT to_json(named_struct('a', 1, 'b', 'hello'))
-query expect_fallback(StructsToJson with options is not supported)
+-- to_json with options and with array fields are not supported by the native (rust) path, so they
+-- route to the codegen (java) engine, which runs Spark's own implementation inside Comet.
+query
Review Comment:
we need more testing, I believe to_json with structure there are lots of hidden issues, so we need to got through all supported datatypes
Also there is the entire config set for jsons https://spark.apache.org/docs/latest/sql-data-sources-json.html which changes the behavior
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] feat: 100% Spark-compatible JSON support via codegen dispatcher [datafusion-comet]
Posted by "andygrove (via GitHub)" <gi...@apache.org>.
andygrove merged PR #4305:
URL: https://github.com/apache/datafusion-comet/pull/4305
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] feat: 100% Spark-compatible JSON support via codegen dispatcher [datafusion-comet]
Posted by "mbutrovich (via GitHub)" <gi...@apache.org>.
mbutrovich commented on code in PR #4305:
URL: https://github.com/apache/datafusion-comet/pull/4305#discussion_r3359026111
##########
docs/source/user-guide/latest/compatibility/json.md:
##########
@@ -0,0 +1,55 @@
+<!---
+Licensed to the Apache Software Foundation (ASF) under one
+or more contributor license agreements. See the NOTICE file
+distributed with this work for additional information
+regarding copyright ownership. The ASF licenses this file
+to you under the Apache License, Version 2.0 (the
+"License"); you may not use this file except in compliance
+with the License. You may obtain a copy of the License at
+
+http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing,
+software distributed under the License is distributed on an
+"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+KIND, either express or implied. See the License for the
+specific language governing permissions and limitations
+under the License.
+-->
+
+# JSON Compatibility
+
+Comet can evaluate JSON expressions (`get_json_object`, `from_json`, `to_json`)
+two ways:
+
+- **Codegen dispatcher (default):** Spark's own `doGenCode` for the expression
+ runs inside the Comet pipeline (via Comet's Arrow-direct codegen dispatcher),
+ giving byte-exact compatibility with Spark at the cost of a JNI roundtrip per
+ batch. This rides the codegen dispatcher
+ (`spark.comet.exec.scalaUDF.codegen.enabled`, enabled by default); if the
+ dispatcher is disabled, the operator falls back to Spark.
+- **Native (rust) path:** the native DataFusion implementation. Faster, but has
Review Comment:
```suggestion
- **Native (Rust) path:** the native DataFusion implementation. Faster, but has
```
Can be in a followup since we don't want to trigger CI, but Rust should be capitalized.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org