[I] Codegen dispatcher: guard the closure-serialize step so a non-serializable tree falls back cleanly instead of throwing at plan time [datafusion-comet]

["andygrove (via GitHub)" <gi...@apache.org>]

andygrove opened a new issue, #5573:
URL: https://github.com/apache/datafusion-comet/issues/5573

   Part of #5572.
   
   `CometScalaUDF.emitJvmCodegenDispatch` closure-serializes the bound expression with no guard:
   
   ```scala
   val serializer = SparkEnv.get.closureSerializer.newInstance()
   val buffer = serializer.serialize(boundExpr)
   ```
   
   `spark/src/main/scala/org/apache/comet/serde/CometScalaUDF.scala:111-112`
   
   Every other failure mode in that method degrades cleanly — the disabled-config check, `CometBatchKernelCodegen.canHandle`, the literal-payload conversion, the data-arg conversion and the return-type conversion all call `withFallbackReason` and return `None`, which the caller treats as a clean Spark fallback. The serialize call is the one step that can throw instead, and a `NotSerializableException` there escapes during **planning**, which is a much worse failure than falling back.
   
   Today's callers are all curated expression types where this is very unlikely to bite. It matters because of where the sweep wants to go next: routing an unrecognized `StaticInvoke` or `Invoke` through the dispatcher as a catch-all hands it arbitrary trees, including ones holding references the closure serializer will refuse. So this should land before that change.
   
   The fix is small — wrap the serialize in a `try` / `catch`, call `withFallbackReason` with the exception message and return `None`, matching the shape of the checks around it. A unit test can force it with an expression capturing a deliberately non-serializable object.
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org

Re: [I] Codegen dispatcher: guard the closure-serialize step so a non-serializable tree falls back cleanly instead of throwing at plan time [datafusion-comet]

["andygrove (via GitHub)" <gi...@apache.org>]

andygrove closed issue #5573: Codegen dispatcher: guard the closure-serialize step so a non-serializable tree falls back cleanly instead of throwing at plan time
URL: https://github.com/apache/datafusion-comet/issues/5573


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org