You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@datafusion.apache.org by "viirya (via GitHub)" <gi...@apache.org> on 2026/10/23 17:19:44 UTC
[I] Keep native Arrow UDFs on Spark's path when the embedded Python version differs [datafusion-comet]
viirya opened a new issue, #7178:
URL: https://github.com/apache/datafusion-comet/issues/7178
### What is the problem the feature request solves?
The native scalar Arrow UDF path (#6130) checks a UDF's PySpark Python version only when a task creates its callable. A mismatch then fails the task. The embedded interpreter is the `libpython` that libcomet was linked against, so its version is fixed when libcomet is built. Spark only requires the workers' Python to match the driver's. After a Python upgrade without a libcomet rebuild, every query that selects the native path fails, while the same query succeeds on Spark's Python workers.
### Describe the potential solution
Expose the embedded interpreter's `major.minor` version to the JVM through `NativeBase`, read with `Py_GetVersion`, which may be called before the interpreter is initialized. Have `CometArrowEvalPythonExec.getSupportLevel` fall back to Spark when a UDF's `pythonVer` differs, and add a planning test.
### Additional context
Raised in review: https://github.com/apache/datafusion-comet/pull/6130#discussion_r4357444630
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [I] Keep native Arrow UDFs on Spark's path when the embedded Python version differs [datafusion-comet]
Posted by "viirya (via GitHub)" <gi...@apache.org>.
viirya closed issue #7178: Keep native Arrow UDFs on Spark's path when the embedded Python version differs
URL: https://github.com/apache/datafusion-comet/issues/7178
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org