You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@datafusion.apache.org by "kazantsev-maksim (via GitHub)" <gi...@apache.org> on 2026/04/09 18:20:01 UTC
[I] Add support for to_json compatibility with Spark 4.0.0. [datafusion-comet]
kazantsev-maksim opened a new issue, #3920:
URL: https://github.com/apache/datafusion-comet/issues/3920
Since Spark 4.0.0 `to_json` function use runtime expression replacement: https://github.com/apache/spark/blob/master/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/jsonExpressions.scala#L418
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [I] Add support for to_json compatibility with Spark 4.0.0. [datafusion-comet]
Posted by "kazantsev-maksim (via GitHub)" <gi...@apache.org>.
kazantsev-maksim commented on issue #3920:
URL: https://github.com/apache/datafusion-comet/issues/3920#issuecomment-4216482553
take
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [I] Add support for to_json compatibility with Spark 4.0.0. [datafusion-comet]
Posted by "mbutrovich (via GitHub)" <gi...@apache.org>.
mbutrovich closed issue #3920: Add support for to_json compatibility with Spark 4.0.0.
URL: https://github.com/apache/datafusion-comet/issues/3920
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [I] Add support for to_json compatibility with Spark 4.0.0. [datafusion-comet]
Posted by "andygrove (via GitHub)" <gi...@apache.org>.
andygrove commented on issue #3920:
URL: https://github.com/apache/datafusion-comet/issues/3920#issuecomment-4297164752
Four tests are currently skipped with `assume(!isSpark40Plus)` for this issue:
- `CometExpressionSuite`: `to_json`
- `CometExpressionSuite`: `to_json escaping of field names and string values`
- `CometExpressionSuite`: `to_json unicode`
- `CometJsonExpressionSuite`: `to_json - all supported types`
All four fail on Spark 4 with the same fallback: `COMET: invoke is not supported`, and the plan retains a regular `Project` instead of native execution.
Root cause: `StructsToJson` in Spark 4 extends `RuntimeReplaceable` and its `replacement` is `Invoke(Literal.create(evaluator, ObjectType(classOf[StructsToJsonEvaluator])), "evaluate", ...)`. Comet's serde sees the post-replacement `Invoke` (pretty name `"invoke"`), not the original `StructsToJson`. `CometStaticInvoke` handles `StaticInvoke` only; there is no serde for `Invoke`.
Fix direction: add an `Invoke` handler that recognizes `Invoke(Literal(_: StructsToJsonEvaluator, _), "evaluate", _, _)` and routes to `CometStructsToJson`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org