You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@datafusion.apache.org by "viirya (via GitHub)" <gi...@apache.org> on 2026/09/29 23:05:52 UTC
[I] Separate JNI-free core logic and error classification from native JNI entry points [datafusion-comet]
viirya opened a new issue, #6434:
URL: https://github.com/apache/datafusion-comet/issues/6434
### What is the problem the feature request solves?
The native JNI entry points in `native/core/src/execution/jni_api.rs` and `native/core/src/lib.rs`
mix three concerns in one function body: converting JNI arguments (`JLongArray`, `JString`,
`JByteBuffer`, ...), the actual native logic, and raising the JVM exception on failure. As a result:
- The native logic of most entry points cannot be unit-tested or benchmarked without a JVM, even
when it never calls back into the JVM (for example shuffle block decoding or the feature and
object store checks).
- The mapping from `CometError` / `SparkError` to the JVM exception (class, message, Spark error
JSON, rethrown Java throwables, panic backtraces) is interleaved with the JNI calls that throw it
in `throw_exception`, so the classification itself can only be tested through a JVM.
### Describe the potential solution
- Split the exception raising into a JNI-free classification step, which describes the exception
a failed native call surfaces as (with a status code and a serializable form), and a JNI step
that throws it. Every entry point goes through the classification via `try_unwrap_or_throw`,
with exception classes and messages unchanged.
- Turn each `Java_org_apache_comet_*` export into a thin wrapper that converts JNI arguments into
plain Rust values and calls a core function without JNI types in its signature.
- Do this incrementally: the error classification plus a few entry points first, then the
remaining entry points that do not call back into the JVM, then the argument handling of the
plan entry points (`createPlan`, `executePlan`, `releasePlan`, `setShufflePartitionPusher`),
which keep depending on JVM callbacks.
### Additional context
No change to the JVM side or to the exceptions users see.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [I] Separate JNI-free core logic and error classification from native JNI entry points [datafusion-comet]
Posted by "viirya (via GitHub)" <gi...@apache.org>.
viirya commented on issue #6434:
URL: https://github.com/apache/datafusion-comet/issues/6434#issuecomment-6111818544
This is done:
- #6435 split the JVM exception raising into a JNI-free classification (`NativeError`) and a JNI throw step, and moved the logic of `isFeatureEnabled`, `isObjectStoreSchemeSupported` and the shuffle block decoding into core functions.
- #6534 did the same for the remaining entry points that do not call back into the JVM: columnar-to-row conversion, the sorted shuffle file writer, partition sorting, shuffle partition offsets, `NativeBase.init` and `getTzdataVersion`.
- #6695 did it for the plan entry points (`createPlan`, `setShufflePartitionPusher`, `executePlan`, `releasePlan`), passing the steps that need a JNI `Env` in as callbacks, so a plan can be created, run and released in a Rust unit test.
The remaining exports (`columnarToRowInit`/`columnarToRowClose`, `NativeBase.release`, the tracing entry points and `getRustThreadId`) already only call JNI-free functions.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [I] Separate JNI-free core logic and error classification from native JNI entry points [datafusion-comet]
Posted by "viirya (via GitHub)" <gi...@apache.org>.
viirya closed issue #6434: Separate JNI-free core logic and error classification from native JNI entry points
URL: https://github.com/apache/datafusion-comet/issues/6434
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org