You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@datafusion.apache.org by gi...@apache.org on 2026/10/01 01:31:17 UTC
(datafusion-comet) branch gh-readonly-queue/main/pr-6435-ec66afe56b1b9eb5ecc695169feb26495917eedd created (now 62ed9d16d5)
This is an automated email from the ASF dual-hosted git repository.
github-merge-queue[bot] pushed a change to branch gh-readonly-queue/main/pr-6435-ec66afe56b1b9eb5ecc695169feb26495917eedd
in repository https://gitbox.apache.org/repos/asf/datafusion-comet.git
at 62ed9d16d5 refactor: separate JNI-free core logic and error classification at the native boundary (#6435)
This branch includes the following new commits:
new 62ed9d16d5 refactor: separate JNI-free core logic and error classification at the native boundary (#6435)
The 1 revisions listed above as "new" are entirely new to this
repository and will be described in separate emails. The revisions
listed as "add" were already present in the repository and have only
been added to this reference.
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@datafusion.apache.org
For additional commands, e-mail: commits-help@datafusion.apache.org
(datafusion-comet) 01/01: refactor: separate JNI-free core logic and error classification at the native boundary (#6435)
Posted by gi...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
github-merge-queue[bot] pushed a commit to branch gh-readonly-queue/main/pr-6435-ec66afe56b1b9eb5ecc695169feb26495917eedd
in repository https://gitbox.apache.org/repos/asf/datafusion-comet.git
commit 62ed9d16d547fed1ac25260a1910893f076ab638
Author: Liang-Chi Hsieh <vi...@gmail.com>
AuthorDate: Thu Oct 1 01:30:52 2026 +0000
refactor: separate JNI-free core logic and error classification at the native boundary (#6435)
Split the JVM exception raising in errors.rs into a JNI-free classification
step, NativeError::from_comet_error, and a JNI throw step. Every JNI entry
point already goes through try_unwrap_or_throw, so all of them now use the
classification, and the thrown exception classes and messages are unchanged.
The classification can also be serialized as a versioned JSON payload with
a status code, and catch_native provides the panic-catching boundary without
JNI.
Move the logic of isFeatureEnabled, isObjectStoreSchemeSupported,
decodeShuffleBlock, decodeShuffleBlockWithValidation and
createRemoteShuffleDecoder into core functions that take plain Rust values,
leaving the exports as thin wrappers, and split prepare_output into the JNI
array pinning and a core export_batch. The core functions are covered by
Rust unit tests that need no JVM.
---
.../contributor-guide/sql_error_propagation.md | 25 +-
native/Cargo.lock | 1 +
native/core/src/execution/jni_api.rs | 159 ++++-
native/core/src/lib.rs | 57 +-
native/jni-bridge/Cargo.toml | 1 +
native/jni-bridge/src/errors.rs | 657 +++++++++++++++++----
6 files changed, 735 insertions(+), 165 deletions(-)
diff --git a/docs/source/contributor-guide/sql_error_propagation.md b/docs/source/contributor-guide/sql_error_propagation.md
index a27408510d..ac005676ff 100644
--- a/docs/source/contributor-guide/sql_error_propagation.md
+++ b/docs/source/contributor-guide/sql_error_propagation.md
@@ -282,31 +282,38 @@ pub struct SparkErrorWithContext {
## Step 5: Error Is Serialized to JSON
When DataFusion propagates the error all the way up through the execution engine and it
-reaches the JNI boundary, `throw_exception()` in `errors.rs` is called. It detects the
-`SparkErrorWithContext` type and calls `.to_json()` on it:
+reaches the JNI boundary, `try_unwrap_or_throw` in `native/jni-bridge/src/errors.rs` hands it to
+`NativeError::from_comet_error`. That function classifies the error, without using JNI, into the
+JVM exception it must surface as. It detects the `SparkErrorWithContext` type and calls
+`.to_json()` on it:
```rust
-// native/core/src/errors.rs
+// native/jni-bridge/src/errors.rs
-fn throw_exception(env: &mut JNIEnv, error: &CometError, ...) {
+fn jvm_exception<'a>(error: &'a CometError, backtrace: Option<&str>) -> JvmException<'a> {
match error {
CometError::DataFusion {
source: DataFusionError::External(e), ..
} => {
if let Some(spark_err_ctx) = e.downcast_ref::<SparkErrorWithContext>() {
- // Has SQL context → throw with JSON payload
- let json = spark_err_ctx.to_json();
- env.throw_new("org/apache/comet/exceptions/CometQueryExecutionException", json)
+ // Has SQL context → JSON payload
+ JvmException::Spark { json: spark_err_ctx.to_json() }
} else if let Some(spark_err) = e.downcast_ref::<SparkError>() {
- // No SQL context → throw with JSON payload (no context field)
- throw_spark_error_as_json(env, spark_err)
+ // No SQL context → JSON payload (no context field)
+ spark_exception(spark_err)
}
+ // ...
}
// ...
}
}
```
+`throw_native_error` then throws a `JvmException::Spark` as
+`org.apache.comet.exceptions.CometQueryExecutionException` with the JSON as its message. The
+classification is independent of JNI so that a caller that is not a JNI entry point can receive
+the same exception description as a JSON document from `NativeError::to_payload`.
+
The JSON looks like this for a divide-by-zero in `SELECT a/b FROM t`:
```json
diff --git a/native/Cargo.lock b/native/Cargo.lock
index 2e1f24b535..f0748ff71d 100644
--- a/native/Cargo.lock
+++ b/native/Cargo.lock
@@ -2056,6 +2056,7 @@ dependencies = [
"paste",
"prost",
"regex",
+ "serde_json",
"thiserror 2.0.20",
]
diff --git a/native/core/src/execution/jni_api.rs b/native/core/src/execution/jni_api.rs
index 7b370d5b6f..1a9c880e5c 100644
--- a/native/core/src/execution/jni_api.rs
+++ b/native/core/src/execution/jni_api.rs
@@ -16,6 +16,15 @@
// under the License.
//! Define JNI APIs which can be called from Java/Scala.
+//!
+//! An entry point is a thin JNI wrapper: it converts its JNI arguments into plain Rust values,
+//! calls a core function whose signature has no JNI types (e.g. `decode_shuffle_block`), and
+//! converts the result back. Errors and panics are raised as JVM exceptions by
+//! `try_unwrap_or_throw`, through the boundary error protocol in `errors::NativeError`.
+//!
+//! `createPlan`, `setShufflePartitionPusher`, `executePlan` and `releasePlan` also depend on
+//! upcalls into the JVM (input iterators, the task memory manager, metrics, UDFs and scalar
+//! subqueries), so their core logic keeps holding JNI references.
use super::{serde, utils::SparkArrowConvert};
use crate::{
@@ -972,13 +981,25 @@ fn prepare_output(
output_batch: RecordBatch,
validate: bool,
) -> CometResult<jlong> {
- let num_cols = array_addrs.len(env)?;
-
let array_addrs = unsafe { array_addrs.get_elements(env, ReleaseMode::NoCopyBack)? };
- let array_addrs = &*array_addrs;
-
let schema_addrs = unsafe { schema_addrs.get_elements(env, ReleaseMode::NoCopyBack)? };
- let schema_addrs = &*schema_addrs;
+ unsafe { export_batch(&array_addrs, &schema_addrs, output_batch, validate) }
+}
+
+/// Moves the columns of `output_batch` into the Arrow C Data Interface structs at `array_addrs`
+/// and `schema_addrs`, one pair per column, and returns the row count. With no addresses, which
+/// Spark passes when the results of a query are not used, only the row count is returned.
+///
+/// # Safety
+/// Each address must point to a writable `FFI_ArrowArray` / `FFI_ArrowSchema` the caller owns.
+/// Whatever the structs hold is overwritten without being released.
+unsafe fn export_batch(
+ array_addrs: &[i64],
+ schema_addrs: &[i64],
+ output_batch: RecordBatch,
+ validate: bool,
+) -> CometResult<i64> {
+ let num_cols = array_addrs.len();
let output_schema = output_batch.schema();
let results = output_batch.columns();
@@ -1016,7 +1037,7 @@ fn prepare_output(
}
}
- Ok(num_rows as jlong)
+ Ok(num_rows as i64)
}
/// Pull the next input from JVM. Note that we cannot pull input batches in
@@ -1684,7 +1705,7 @@ pub unsafe extern "system" fn Java_org_apache_comet_Native_decodeShuffleBlock(
) -> jlong {
try_unwrap_or_throw(&e, |env| {
with_trace("decodeShuffleBlock", tracing_enabled != JNI_FALSE, || {
- decode_shuffle_block(env, byte_buffer, length, array_addrs, schema_addrs, None)
+ decode_shuffle_block_jni(env, byte_buffer, length, array_addrs, schema_addrs, None)
})
})
}
@@ -1700,12 +1721,7 @@ pub extern "system" fn Java_org_apache_comet_Native_createRemoteShuffleDecoder(
) -> jlong {
try_unwrap_or_throw(&e, |env| {
let bytes = env.convert_byte_array(expected_schema)?;
- let schema = ShuffleScan::decode(bytes.as_slice()).map_err(|error| {
- CometError::Internal(format!("Invalid expected remote shuffle schema: {error}"))
- })?;
- let decoder = RemoteShuffleDecoder {
- expected_types: schema.fields.iter().map(to_arrow_datatype).collect(),
- };
+ let decoder = RemoteShuffleDecoder::try_new(&bytes)?;
Ok(Box::into_raw(Box::new(decoder)) as jlong)
})
}
@@ -1715,6 +1731,19 @@ struct RemoteShuffleDecoder {
expected_types: Vec<ArrowDataType>,
}
+impl RemoteShuffleDecoder {
+ /// Parses the serialized `ShuffleScan` holding the expected schema. Core of
+ /// `Native.createRemoteShuffleDecoder`.
+ fn try_new(expected_schema: &[u8]) -> CometResult<Self> {
+ let schema = ShuffleScan::decode(expected_schema).map_err(|error| {
+ CometError::Internal(format!("Invalid expected remote shuffle schema: {error}"))
+ })?;
+ Ok(RemoteShuffleDecoder {
+ expected_types: schema.fields.iter().map(to_arrow_datatype).collect(),
+ })
+ }
+}
+
#[no_mangle]
/// Release a remote shuffle iterator's decoder.
///
@@ -1755,7 +1784,7 @@ pub unsafe extern "system" fn Java_org_apache_comet_Native_decodeShuffleBlockWit
.ok_or_else(|| {
CometError::Internal("Remote shuffle decoder is not initialized".to_owned())
})?;
- decode_shuffle_block(
+ decode_shuffle_block_jni(
env,
byte_buffer,
length,
@@ -1767,7 +1796,9 @@ pub unsafe extern "system" fn Java_org_apache_comet_Native_decodeShuffleBlockWit
})
}
-fn decode_shuffle_block(
+/// Converts the JNI arguments of the `decodeShuffleBlock` entry points for
+/// [`decode_shuffle_block`].
+fn decode_shuffle_block_jni(
env: &mut Env,
byte_buffer: JByteBuffer,
length: jint,
@@ -1777,15 +1808,33 @@ fn decode_shuffle_block(
) -> CometResult<jlong> {
let raw_pointer = env.get_direct_buffer_address(&byte_buffer)?;
let length = length as usize;
- let slice: &[u8] = unsafe { std::slice::from_raw_parts(raw_pointer, length) };
+ let block: &[u8] = unsafe { std::slice::from_raw_parts(raw_pointer, length) };
+ let array_addrs = unsafe { array_addrs.get_elements(env, ReleaseMode::NoCopyBack)? };
+ let schema_addrs = unsafe { schema_addrs.get_elements(env, ReleaseMode::NoCopyBack)? };
+ unsafe { decode_shuffle_block(block, &array_addrs, &schema_addrs, expected_types) }
+}
+
+/// Decodes one native shuffle block (codec header plus Arrow IPC stream) and exports its columns
+/// as in [`export_batch`], returning the row count. `expected_types` is set for a remote shuffle
+/// block, which is validated against it. Core of `Native.decodeShuffleBlock` and
+/// `Native.decodeShuffleBlockWithValidation`.
+///
+/// # Safety
+/// As for [`export_batch`].
+unsafe fn decode_shuffle_block(
+ block: &[u8],
+ array_addrs: &[i64],
+ schema_addrs: &[i64],
+ expected_types: Option<&[ArrowDataType]>,
+) -> CometResult<i64> {
let batch = if let Some(expected_types) = expected_types {
// Reject incompatible logical types, then decode dictionaries before JVM import. The
// JVM importer supports fewer dictionary key/value layouts than the shuffle writer.
- decode_remote_shuffle_batch(slice, expected_types)?
+ decode_remote_shuffle_batch(block, expected_types)?
} else {
- read_ipc_compressed(slice)?
+ read_ipc_compressed(block)?
};
- prepare_output(env, array_addrs, schema_addrs, batch, false)
+ export_batch(array_addrs, schema_addrs, batch, false)
}
#[no_mangle]
@@ -2835,6 +2884,78 @@ mod tests {
assert_eq!(pool.reserved(), 0, "the stream outlived its producer");
}
+ /// One uncompressed native shuffle block holding `batch`.
+ fn shuffle_block(batch: &RecordBatch) -> Vec<u8> {
+ let mut block = b"NONE".to_vec();
+ let mut writer =
+ arrow::ipc::writer::StreamWriter::try_new(&mut block, batch.schema_ref()).unwrap();
+ writer.write(batch).unwrap();
+ writer.finish().unwrap();
+ drop(writer);
+ block
+ }
+
+ fn int_batch() -> RecordBatch {
+ let schema = Arc::new(Schema::new(vec![Field::new("a", DataType::Int32, true)]));
+ RecordBatch::try_new(
+ schema,
+ vec![Arc::new(Int32Array::from(vec![Some(1), None, Some(3)]))],
+ )
+ .unwrap()
+ }
+
+ /// Decodes `block` into freshly allocated C Data structs and imports the single column back.
+ fn decode_one_column(
+ block: &[u8],
+ expected_types: Option<&[ArrowDataType]>,
+ ) -> CometResult<(i64, ArrayRef)> {
+ let mut ffi_array = Box::new(FFI_ArrowArray::empty());
+ let mut ffi_schema = Box::new(FFI_ArrowSchema::empty());
+ let array_addrs = [ffi_array.as_mut() as *mut FFI_ArrowArray as i64];
+ let schema_addrs = [ffi_schema.as_mut() as *mut FFI_ArrowSchema as i64];
+ let rows =
+ unsafe { decode_shuffle_block(block, &array_addrs, &schema_addrs, expected_types)? };
+ let data = unsafe { from_ffi(*ffi_array, &ffi_schema) }?;
+ Ok((rows, arrow::array::make_array(data)))
+ }
+
+ #[test]
+ fn decode_shuffle_block_exports_columns() {
+ let batch = int_batch();
+ let block = shuffle_block(&batch);
+
+ let (rows, column) = decode_one_column(&block, None).unwrap();
+ assert_eq!(rows, 3);
+ assert_eq!(column.as_ref(), batch.column(0).as_ref());
+
+ let (rows, column) = decode_one_column(&block, Some(&[DataType::Int32])).unwrap();
+ assert_eq!(rows, 3);
+ assert_eq!(column.as_ref(), batch.column(0).as_ref());
+ }
+
+ #[test]
+ fn decode_shuffle_block_without_output_columns_counts_rows() {
+ let block = shuffle_block(&int_batch());
+ let rows = unsafe { decode_shuffle_block(&block, &[], &[], None) }.unwrap();
+ assert_eq!(rows, 3);
+ }
+
+ #[test]
+ fn decode_shuffle_block_rejects_bad_input() {
+ let block = shuffle_block(&int_batch());
+ // A remote block whose column type differs from the expected schema.
+ assert!(decode_one_column(&block, Some(&[DataType::Utf8])).is_err());
+ // A truncated block.
+ assert!(decode_one_column(&block[..block.len() / 2], None).is_err());
+ // An expected schema that is not a serialized `ShuffleScan`.
+ let error = RemoteShuffleDecoder::try_new(&[0xff, 0xff, 0xff])
+ .err()
+ .expect("garbage schema is rejected");
+ assert!(error
+ .to_string()
+ .starts_with("Comet Internal Error: Invalid expected remote shuffle schema"));
+ }
+
/// See issue #6294. A runtime cancels every task it has when it shuts down, which Comet's does
/// when the executor stops. The producer's channel then closes as it does at the end of the
/// stream, and the consumer must fail rather than end with the batches it has read so far.
diff --git a/native/core/src/lib.rs b/native/core/src/lib.rs
index c8868b5254..63eefe3ad4 100644
--- a/native/core/src/lib.rs
+++ b/native/core/src/lib.rs
@@ -183,17 +183,20 @@ pub extern "system" fn Java_org_apache_comet_NativeBase_isFeatureEnabled(
) -> jni::sys::jboolean {
try_unwrap_or_throw(&env, |env| {
let feature: String = feature_name.try_to_string(env)?;
-
- let enabled = match feature.as_str() {
- "jemalloc" => cfg!(feature = "jemalloc"),
- "hdfs-opendal" => cfg!(feature = "hdfs-opendal"),
- _ => false, // Unknown features return false
- };
-
- Ok(enabled)
+ Ok(is_feature_enabled(&feature))
})
}
+/// Whether the native build enables `feature`; unknown features are disabled. Core of
+/// `NativeBase.isFeatureEnabled`.
+pub fn is_feature_enabled(feature: &str) -> bool {
+ match feature {
+ "jemalloc" => cfg!(feature = "jemalloc"),
+ "hdfs-opendal" => cfg!(feature = "hdfs-opendal"),
+ _ => false, // Unknown features return false
+ }
+}
+
/// JNI: can object_store build a store AND an object key for this URL?
///
/// Source of truth for the JVM planner's "can the native reader handle this filesystem?" check.
@@ -218,14 +221,19 @@ pub extern "system" fn Java_org_apache_comet_NativeBase_isObjectStoreSchemeSuppo
) -> jni::sys::jboolean {
try_unwrap_or_throw(&env, |env| {
let url_str: String = url.try_to_string(env)?;
- let supported = url::Url::parse(&url_str)
- .ok()
- .map(|u| object_store::ObjectStoreScheme::parse(&u).is_ok())
- .unwrap_or(false);
- Ok(supported)
+ Ok(is_object_store_scheme_supported(&url_str))
})
}
+/// Whether object_store can build a store and an object key for `url`. Core of
+/// `NativeBase.isObjectStoreSchemeSupported`.
+pub fn is_object_store_scheme_supported(url: &str) -> bool {
+ url::Url::parse(url)
+ .ok()
+ .map(|u| object_store::ObjectStoreScheme::parse(&u).is_ok())
+ .unwrap_or(false)
+}
+
/// JNI: the version of the IANA timezone database native code uses, such as `2025b`. chrono-tz
/// compiles the database into libcomet, so it can differ from the JVM's `tzdb.dat`, and then local
/// times computed natively can differ from Spark's.
@@ -256,3 +264,26 @@ fn default_logger_config(log_level: &str) -> CometResult<Config> {
.build(root)
.map_err(|err| CometError::Config(err.to_string()))
}
+
+#[cfg(test)]
+mod tests {
+ use super::*;
+
+ #[test]
+ fn feature_enabled() {
+ assert_eq!(is_feature_enabled("jemalloc"), cfg!(feature = "jemalloc"));
+ assert_eq!(
+ is_feature_enabled("hdfs-opendal"),
+ cfg!(feature = "hdfs-opendal")
+ );
+ assert!(!is_feature_enabled("no-such-feature"));
+ }
+
+ #[test]
+ fn object_store_scheme_supported() {
+ assert!(is_object_store_scheme_supported("file:///tmp/data.parquet"));
+ assert!(is_object_store_scheme_supported("s3://bucket/key"));
+ assert!(!is_object_store_scheme_supported("unknown://bucket/key"));
+ assert!(!is_object_store_scheme_supported("not a url"));
+ }
+}
diff --git a/native/jni-bridge/Cargo.toml b/native/jni-bridge/Cargo.toml
index 87e87bcc2b..57f35e0aa7 100644
--- a/native/jni-bridge/Cargo.toml
+++ b/native/jni-bridge/Cargo.toml
@@ -36,6 +36,7 @@ jni = "0.22.4"
thiserror = { workspace = true }
regex = { workspace = true }
lazy_static = "1.4.0"
+serde_json = "1.0"
once_cell = "1.18.0"
paste = "1.0.14"
prost = "0.14.3"
diff --git a/native/jni-bridge/src/errors.rs b/native/jni-bridge/src/errors.rs
index 199ad94246..a0a1e4df10 100644
--- a/native/jni-bridge/src/errors.rs
+++ b/native/jni-bridge/src/errors.rs
@@ -482,109 +482,291 @@ pub fn unwrap_or_throw_default<T: JNIDefault>(
match result {
Ok(value) => value,
Err(err) => {
- let backtrace = match err {
- CometError::Panic { msg: _ } => PANIC_BACKTRACE.lock().unwrap().take(),
- _ => None,
- };
- throw_exception(env, &err, backtrace);
+ let failure = NativeFailure::from(err);
+ // Leave an exception already pending on this thread (e.g. raised by an upcall) in
+ // place rather than replacing it.
+ if !env.exception_check() {
+ throw_native_error(env, &failure.native_error());
+ }
T::default()
}
}
}
-fn throw_exception(env: &mut Env, error: &CometError, backtrace: Option<String>) {
- // If there isn't already an exception?
- if !env.exception_check() {
- // DataFusion operators can wrap the original failure in Context, Shared, or External
- // errors. Keep the classifications that own their JVM exception class typed across those
- // wrappers and the JNI boundary.
- if let Some(exception) = typed_jvm_exception(error) {
- let _ = env.throw_new(
- JNIString::new(exception.class),
- JNIString::new(exception.msg),
- );
- return;
+// ----------------------------------------------------------------------
+// Native boundary error protocol
+//
+// A failed native call is reported in two steps. `NativeError::from_comet_error` classifies the
+// error into the JVM exception it must surface as; this step uses no JNI, so a non-JNI caller
+// can consume it through `NativeError::to_payload`. `throw_native_error` then raises that
+// exception through JNI, which is the only transport today.
+
+/// Result of a native boundary call, as a stable code for a non-JNI caller.
+#[repr(i32)]
+#[derive(Debug, Clone, Copy, PartialEq, Eq)]
+pub enum NativeStatus {
+ Ok = 0,
+ /// A Rust error. The payload describes the JVM exception to raise.
+ Error = 1,
+ /// A panic caught at the boundary. The payload describes the JVM exception to raise.
+ Panic = 2,
+ /// A JVM throwable raised during an upcall into the JVM. The caller must rethrow that same
+ /// throwable, which cannot cross a non-JNI boundary: the JVM side has to keep hold of it when
+ /// the upcall returns. The payload's class and message are diagnostic only.
+ JavaThrowable = 3,
+}
+
+impl NativeStatus {
+ fn name(self) -> &'static str {
+ match self {
+ NativeStatus::Ok => "Ok",
+ NativeStatus::Error => "Error",
+ NativeStatus::Panic => "Panic",
+ NativeStatus::JavaThrowable => "JavaThrowable",
}
- // ... then throw new exception
- // Note: in jni 0.22.x, throw/throw_new return Err(JavaException) on success
- // (to signal the pending exception to Rust callers via `?`). We discard the
- // result here because we're in an error-handling path and just need the
- // exception to be pending in the JVM.
- let _ = match error {
- CometError::JavaException {
- class: _,
- msg: _,
+ }
+}
+
+/// Version of the JSON document produced by [`NativeError::to_payload`].
+pub const NATIVE_ERROR_PAYLOAD_VERSION: u32 = 1;
+
+/// JVM exception carrying a JSON-serialized `SparkError`, which `SparkErrorConverter` turns into
+/// the Spark exception with its error class and parameters.
+const COMET_QUERY_EXECUTION_EXCEPTION: &str =
+ "org/apache/comet/exceptions/CometQueryExecutionException";
+
+/// The JVM exception a failed native call surfaces as.
+#[derive(Debug)]
+pub enum JvmException<'a> {
+ /// A new exception of `class` (a JNI internal name such as
+ /// `org/apache/comet/CometNativeException`) with `message`.
+ New { class: String, message: String },
+ /// A `CometQueryExecutionException` whose message is `json`, the output of
+ /// `SparkError::to_json` or `SparkErrorWithContext::to_json`.
+ Spark { json: String },
+ /// A JVM throwable captured during an upcall, rethrown as is.
+ Rethrow {
+ class: &'a str,
+ message: &'a str,
+ throwable: &'a Global<JThrowable<'static>>,
+ },
+}
+
+/// A failed native call, classified into the JVM exception it surfaces as.
+#[derive(Debug)]
+pub struct NativeError<'a> {
+ pub status: NativeStatus,
+ pub exception: JvmException<'a>,
+ /// The captured Rust backtrace of a panic. When the exception carries it, it is already folded
+ /// into the message, so this field is informational.
+ pub backtrace: Option<String>,
+ /// The `Display` of each error in the Rust `source()` chain, outermost first. Diagnostic only:
+ /// no Java cause is attached to the thrown exception.
+ pub causes: Vec<String>,
+}
+
+impl<'a> NativeError<'a> {
+ /// Classifies `error` into the JVM exception it must surface as. `backtrace` is the captured
+ /// backtrace of a panic, if any.
+ pub fn from_comet_error(error: &'a CometError, backtrace: Option<String>) -> Self {
+ let exception = jvm_exception(error, backtrace.as_deref());
+ let status = match (&exception, error) {
+ (JvmException::Rethrow { .. }, _) => NativeStatus::JavaThrowable,
+ (_, CometError::Panic { .. }) => NativeStatus::Panic,
+ _ => NativeStatus::Error,
+ };
+ let mut causes = Vec::new();
+ let mut cause = std::error::Error::source(error);
+ while let Some(e) = cause {
+ causes.push(e.to_string());
+ cause = e.source();
+ }
+ NativeError {
+ status,
+ exception,
+ backtrace,
+ causes,
+ }
+ }
+
+ /// The JNI internal name of the exception class to raise.
+ pub fn exception_class(&self) -> &str {
+ match &self.exception {
+ JvmException::New { class, .. } => class,
+ JvmException::Spark { .. } => COMET_QUERY_EXECUTION_EXCEPTION,
+ JvmException::Rethrow { class, .. } => class,
+ }
+ }
+
+ /// The exact message of the exception to raise.
+ pub fn message(&self) -> &str {
+ match &self.exception {
+ JvmException::New { message, .. } => message,
+ JvmException::Spark { json } => json,
+ JvmException::Rethrow { message, .. } => message,
+ }
+ }
+
+ /// Serializes this error as a versioned JSON document for a caller that cannot raise JVM
+ /// exceptions through JNI. `exceptionClass` and `message` reproduce the exception thrown over
+ /// JNI; `sparkError` is the parsed `message` of a `CometQueryExecutionException`.
+ pub fn to_payload(&self) -> String {
+ let (kind, spark_error) = match &self.exception {
+ JvmException::New { .. } => ("new", serde_json::Value::Null),
+ JvmException::Spark { json } => (
+ "spark",
+ serde_json::from_str(json).unwrap_or(serde_json::Value::Null),
+ ),
+ JvmException::Rethrow { .. } => ("rethrow", serde_json::Value::Null),
+ };
+ serde_json::json!({
+ "version": NATIVE_ERROR_PAYLOAD_VERSION,
+ "status": self.status.name(),
+ "kind": kind,
+ "exceptionClass": self.exception_class(),
+ "message": self.message(),
+ "sparkError": spark_error,
+ "backtrace": self.backtrace,
+ "causes": self.causes,
+ })
+ .to_string()
+ }
+}
+
+/// A failed native call caught at the boundary: the error and, for a panic, its captured
+/// backtrace.
+#[derive(Debug)]
+pub struct NativeFailure {
+ pub error: CometError,
+ pub backtrace: Option<String>,
+}
+
+impl From<CometError> for NativeFailure {
+ fn from(error: CometError) -> Self {
+ let backtrace = match error {
+ CometError::Panic { .. } => PANIC_BACKTRACE.lock().unwrap().take(),
+ _ => None,
+ };
+ NativeFailure { error, backtrace }
+ }
+}
+
+impl NativeFailure {
+ pub fn native_error(&self) -> NativeError<'_> {
+ NativeError::from_comet_error(&self.error, self.backtrace.clone())
+ }
+}
+
+/// Runs a native core function without JNI, converting an error or a panic into a
+/// [`NativeFailure`]. The boundary for a caller that is not a JNI entry point; JNI entry points use
+/// [`try_unwrap_or_throw`].
+pub fn catch_native<T, F>(f: F) -> Result<T, NativeFailure>
+where
+ F: FnOnce() -> CometResult<T> + UnwindSafe,
+{
+ match std::panic::catch_unwind(f) {
+ Ok(Ok(value)) => Ok(value),
+ Ok(Err(error)) => Err(NativeFailure::from(error)),
+ Err(payload) => Err(NativeFailure::from(CometError::from(payload))),
+ }
+}
+
+/// Raises `error` as a pending JVM exception.
+fn throw_native_error(env: &mut Env, error: &NativeError) {
+ // Note: in jni 0.22.x, throw/throw_new return Err(JavaException) on success
+ // (to signal the pending exception to Rust callers via `?`). We discard the
+ // result here because we're in an error-handling path and just need the
+ // exception to be pending in the JVM.
+ let _ = match &error.exception {
+ JvmException::Rethrow { throwable, .. } => env.throw(*throwable),
+ JvmException::New { class, message } => env.throw_new(
+ JNIString::new(class.as_str()),
+ JNIString::new(message.as_str()),
+ ),
+ JvmException::Spark { json } => env.throw_new(
+ JNIString::new(COMET_QUERY_EXECUTION_EXCEPTION),
+ JNIString::new(json.as_str()),
+ ),
+ };
+}
+
+/// The JVM exception `error` surfaces as.
+fn jvm_exception<'a>(error: &'a CometError, backtrace: Option<&str>) -> JvmException<'a> {
+ // DataFusion operators can wrap the original failure in Context, Shared, or External
+ // errors. Keep the classifications that own their JVM exception class typed across those
+ // wrappers and the JNI boundary.
+ if let Some(exception) = typed_jvm_exception(error) {
+ return JvmException::New {
+ class: exception.class,
+ message: exception.msg,
+ };
+ }
+ match error {
+ CometError::JavaException {
+ class,
+ msg,
+ throwable,
+ }
+ | CometError::Execution {
+ source:
+ ExecutionError::JavaException {
+ class,
+ msg,
+ throwable,
+ },
+ } => JvmException::Rethrow {
+ class,
+ message: msg,
+ throwable,
+ },
+ // Handle DataFusion errors containing SparkError or SparkErrorWithContext
+ CometError::DataFusion {
+ msg: _,
+ source: df_error @ DataFusionError::External(e),
+ } => {
+ if let Some(CometError::JavaException {
+ class,
+ msg,
throwable,
- } => env.throw(throwable),
- CometError::Execution {
- source:
- ExecutionError::JavaException {
- class: _,
- msg: _,
- throwable,
- },
- } => env.throw(throwable),
- // Handle DataFusion errors containing SparkError or SparkErrorWithContext
- CometError::DataFusion {
- msg: _,
- source: df_error @ DataFusionError::External(e),
- } => {
- if let Some(CometError::JavaException { throwable, .. }) =
- e.downcast_ref::<CometError>()
- {
- // A Java exception captured inside a JVM UDF kernel (e.g. Spark codegen
- // raising INVALID_REGEXP_REPLACE). Re-throw the original throwable so callers
- // see the exact Spark exception type rather than a wrapped CometNativeException.
- env.throw(throwable)
- } else if let Some(spark_error_with_ctx) = e.downcast_ref::<SparkErrorWithContext>()
- {
- let json_message = spark_error_with_ctx.to_json();
- env.throw_new(
- jni::jni_str!("org/apache/comet/exceptions/CometQueryExecutionException"),
- JNIString::new(json_message),
- )
- } else if let Some(spark_error) = e.downcast_ref::<SparkError>() {
- let json_message = spark_error.to_json();
- env.throw_new(
- jni::jni_str!("org/apache/comet/exceptions/CometQueryExecutionException"),
- JNIString::new(json_message),
- )
- } else if let Some(spark_error) = try_classify_file_read_error(df_error) {
- throw_spark_error_as_json(env, &spark_error)
- } else {
- // Not a SparkError, use generic exception
- let exception = error.to_exception();
- match backtrace {
- Some(backtrace_string) => env.throw_new(
- JNIString::new(exception.class),
- JNIString::new(
- to_stacktrace_string(exception.msg, backtrace_string).unwrap(),
- ),
- ),
- _ => env.throw_new(
- JNIString::new(exception.class),
- JNIString::new(exception.msg),
- ),
- }
+ }) = e.downcast_ref::<CometError>()
+ {
+ // A Java exception captured inside a JVM UDF kernel (e.g. Spark codegen
+ // raising INVALID_REGEXP_REPLACE). Re-throw the original throwable so callers
+ // see the exact Spark exception type rather than a wrapped CometNativeException.
+ JvmException::Rethrow {
+ class,
+ message: msg,
+ throwable,
}
- }
- // Typed file-read errors (corrupt/truncated parquet, object_store) raised by the native
- // scan -- classified by DataFusionError variant, not message text -- surfaced as
- // FAILED_READ_FILE / FileNotFound via the structured SparkError channel. Anything else
- // falls back to generic handling.
- CometError::DataFusion { msg: _, source } => {
- if let Some(spark_error) = parquet_external_spark_error(source) {
- throw_spark_error_as_json(env, spark_error)
- } else if let Some(spark_error) = try_classify_file_read_error(source) {
- throw_spark_error_as_json(env, &spark_error)
- } else {
- throw_generic_exception(env, error, backtrace)
+ } else if let Some(spark_error_with_ctx) = e.downcast_ref::<SparkErrorWithContext>() {
+ JvmException::Spark {
+ json: spark_error_with_ctx.to_json(),
}
+ } else if let Some(spark_error) = e.downcast_ref::<SparkError>() {
+ spark_exception(spark_error)
+ } else if let Some(spark_error) = try_classify_file_read_error(df_error) {
+ spark_exception(&spark_error)
+ } else {
+ // Not a SparkError, use generic exception
+ natural_exception(error, backtrace)
}
- // Handle direct SparkError - serialize to JSON
- CometError::Spark(spark_error) => throw_spark_error_as_json(env, spark_error),
- _ => throw_generic_exception(env, error, backtrace),
- };
+ }
+ // Typed file-read errors (corrupt/truncated parquet, object_store) raised by the native
+ // scan -- classified by DataFusionError variant, not message text -- surfaced as
+ // FAILED_READ_FILE / FileNotFound via the structured SparkError channel. Anything else
+ // falls back to generic handling.
+ CometError::DataFusion { msg: _, source } => {
+ if let Some(spark_error) = parquet_external_spark_error(source) {
+ spark_exception(spark_error)
+ } else if let Some(spark_error) = try_classify_file_read_error(source) {
+ spark_exception(&spark_error)
+ } else {
+ generic_exception(error, backtrace)
+ }
+ }
+ // Handle direct SparkError - serialize to JSON
+ CometError::Spark(spark_error) => spark_exception(spark_error),
+ _ => generic_exception(error, backtrace),
}
}
@@ -605,47 +787,44 @@ fn typed_jvm_exception(error: &(dyn std::error::Error + 'static)) -> Option<Exce
None
}
-/// Generic fallback throw for an error that isn't a structured `SparkError`. Recognises a
-/// file-not-found arriving through non-typed wrapping paths and duplicate-field errors; otherwise
-/// throws the error's natural JVM exception (with the captured backtrace when available).
-fn throw_generic_exception(
- env: &mut Env,
- error: &CometError,
- backtrace: Option<String>,
-) -> jni::errors::Result<()> {
+/// Generic fallback for an error that isn't a structured `SparkError`. Recognises a file-not-found
+/// arriving through non-typed wrapping paths and duplicate-field errors; otherwise the error's
+/// natural JVM exception (with the captured backtrace when available).
+fn generic_exception(error: &CometError, backtrace: Option<&str>) -> JvmException<'static> {
let error_msg = error.to_string();
// A file-not-found that arrived through a non-typed wrapping path (the typed classification
// is handled by `try_classify_file_read_error`).
if error_msg.contains("not found") && error_msg.contains("No such file or directory") {
let spark_error = SparkError::FileNotFound { message: error_msg };
- throw_spark_error_as_json(env, &spark_error)
+ spark_exception(&spark_error)
} else if let Some(spark_error) = try_convert_duplicate_field_error(&error_msg) {
- throw_spark_error_as_json(env, &spark_error)
+ spark_exception(&spark_error)
} else {
- let exception = error.to_exception();
- match backtrace {
- Some(backtrace_string) => env.throw_new(
- JNIString::new(exception.class),
- JNIString::new(to_stacktrace_string(exception.msg, backtrace_string).unwrap()),
- ),
- _ => env.throw_new(
- JNIString::new(exception.class),
- JNIString::new(exception.msg),
- ),
- }
+ natural_exception(error, backtrace)
}
}
-/// Throws a CometQueryExecutionException with JSON-encoded SparkError
-fn throw_spark_error_as_json(env: &mut Env, spark_error: &SparkError) -> jni::errors::Result<()> {
- // Serialize error to JSON
- let json_message = spark_error.to_json();
+/// The error's own JVM exception from `to_exception`, with the backtrace formatted into the
+/// message as a stack trace when available.
+fn natural_exception(error: &CometError, backtrace: Option<&str>) -> JvmException<'static> {
+ let exception = error.to_exception();
+ let message = match backtrace {
+ Some(backtrace_string) => {
+ to_stacktrace_string(exception.msg, backtrace_string.to_string()).unwrap()
+ }
+ None => exception.msg,
+ };
+ JvmException::New {
+ class: exception.class,
+ message,
+ }
+}
- // Throw CometQueryExecutionException with JSON message
- env.throw_new(
- jni::jni_str!("org/apache/comet/exceptions/CometQueryExecutionException"),
- JNIString::new(json_message),
- )
+/// A CometQueryExecutionException with JSON-encoded SparkError
+fn spark_exception(spark_error: &SparkError) -> JvmException<'static> {
+ JvmException::Spark {
+ json: spark_error.to_json(),
+ }
}
/// A `SparkError` the Parquet reader raised on open arrives as
@@ -928,7 +1107,7 @@ mod tests {
};
use jni::{
- objects::{JClass, JIntArray, JString, JThrowable},
+ objects::{JClass, JIntArray, JObject, JString, JThrowable},
sys::{jintArray, jstring},
EnvUnowned, InitArgsBuilder, JNIVersion, JavaVM,
};
@@ -1244,6 +1423,236 @@ mod tests {
Ok(s)
}
+ /// Classifies `error` and returns (status, class, message, payload).
+ fn classify(
+ error: CometError,
+ backtrace: Option<String>,
+ ) -> (NativeStatus, String, String, serde_json::Value) {
+ let native_error = NativeError::from_comet_error(&error, backtrace);
+ let payload: serde_json::Value =
+ serde_json::from_str(&native_error.to_payload()).expect("payload is JSON");
+ (
+ native_error.status,
+ native_error.exception_class().to_string(),
+ native_error.message().to_string(),
+ payload,
+ )
+ }
+
+ const QUERY_EXECUTION_EXCEPTION: &str =
+ "org/apache/comet/exceptions/CometQueryExecutionException";
+
+ #[test]
+ fn native_error_spark_error() {
+ let spark_error = SparkError::DivideByZero;
+ let json = spark_error.to_json();
+ let (status, class, message, payload) = classify(CometError::Spark(spark_error), None);
+ assert_eq!(status, NativeStatus::Error);
+ assert_eq!(class, QUERY_EXECUTION_EXCEPTION);
+ assert_eq!(message, json);
+ assert_eq!(payload["kind"], "spark");
+ assert_eq!(payload["sparkError"]["errorClass"], "DIVIDE_BY_ZERO");
+ assert_eq!(payload["sparkError"]["errorType"], "DivideByZero");
+ }
+
+ #[test]
+ fn native_error_spark_error_through_datafusion() {
+ let with_context = SparkErrorWithContext::new(SparkError::DivideByZero);
+ let json = with_context.to_json();
+ let (_, class, message, _) =
+ classify(CometError::from(DataFusionError::from(with_context)), None);
+ assert_eq!(class, QUERY_EXECUTION_EXCEPTION);
+ assert_eq!(message, json);
+
+ let (_, class, message, _) = classify(
+ CometError::from(DataFusionError::External(Box::new(
+ SparkError::DivideByZero,
+ ))),
+ None,
+ );
+ assert_eq!(class, QUERY_EXECUTION_EXCEPTION);
+ assert_eq!(message, SparkError::DivideByZero.to_json());
+ }
+
+ #[test]
+ fn native_error_file_read_error() {
+ let error = CometError::from(DataFusionError::Context(
+ "reading".to_string(),
+ Box::new(DataFusionError::ParquetError(Box::new(
+ ParquetError::General("Invalid Parquet file. Corrupt footer".to_string()),
+ ))),
+ ));
+ let (status, class, _, payload) = classify(error, None);
+ assert_eq!(status, NativeStatus::Error);
+ assert_eq!(class, QUERY_EXECUTION_EXCEPTION);
+ assert_eq!(payload["sparkError"]["errorType"], "CannotReadFile");
+ }
+
+ #[test]
+ fn native_error_typed_class_through_datafusion_wrappers() {
+ let message = "Remote shuffle exceeds spark.comet.shuffle.rss.maxInFlightBytes";
+ let error = CometError::from(DataFusionError::Shared(Arc::new(DataFusionError::Context(
+ "executing operator".to_string(),
+ Box::new(DataFusionError::from(CometError::ShuffleSizeLimit(
+ message.to_string(),
+ ))),
+ ))));
+ let (status, class, actual_message, payload) = classify(error, None);
+ assert_eq!(status, NativeStatus::Error);
+ assert_eq!(class, "org/apache/comet/CometShuffleSizeLimitException");
+ assert_eq!(actual_message, message);
+ assert_eq!(payload["kind"], "new");
+ assert!(payload["sparkError"].is_null());
+ }
+
+ #[test]
+ fn native_error_natural_exception_classes() {
+ let (_, class, message, _) = classify(CometError::Internal("oops".to_string()), None);
+ assert_eq!(class, "org/apache/comet/CometNativeException");
+ assert_eq!(message, "Comet Internal Error: oops");
+
+ let parse_error = "x".parse::<i32>().unwrap_err();
+ let (_, class, message, _) = classify(CometError::from(parse_error.clone()), None);
+ assert_eq!(class, "java/lang/NumberFormatException");
+ assert_eq!(message, parse_error.to_string());
+
+ let (_, class, _, _) = classify(CometError::from(std::io::Error::other("disk gone")), None);
+ assert_eq!(class, "java/io/IOException");
+
+ let (_, class, message, _) = classify(CometError::IllegalState("bad".to_string()), None);
+ assert_eq!(class, "java/lang/IllegalStateException");
+ assert_eq!(message, "bad");
+ }
+
+ #[test]
+ fn native_error_message_based_fallbacks() {
+ let (_, class, _, payload) = classify(
+ CometError::Internal("file not found: No such file or directory".to_string()),
+ None,
+ );
+ assert_eq!(class, QUERY_EXECUTION_EXCEPTION);
+ assert_eq!(payload["sparkError"]["errorType"], "FileNotFound");
+
+ let (_, class, _, _) = classify(
+ CometError::from(DataFusionError::Execution(
+ r#"Unable to get field named "b". Valid fields: ["A", "B"]"#.to_string(),
+ )),
+ None,
+ );
+ assert_eq!(class, QUERY_EXECUTION_EXCEPTION);
+ }
+
+ #[test]
+ fn native_error_panic_folds_backtrace_into_message() {
+ let backtrace = read_resource("testdata/backtrace.txt").expect("backtrace content");
+ let expected = read_resource("testdata/stacktrace.txt").expect("stacktrace content");
+ let panic = CometError::Panic {
+ msg: "Some Error Message".to_string(),
+ };
+ let (status, class, message, payload) = classify(panic, Some(backtrace.clone()));
+ assert_eq!(status, NativeStatus::Panic);
+ assert_eq!(class, "org/apache/comet/CometNativeException");
+ assert_eq!(message, expected.trim());
+ assert_eq!(payload["status"], "Panic");
+ assert_eq!(payload["backtrace"], backtrace);
+
+ // Without a captured backtrace the message is the panic message alone.
+ let panic = CometError::Panic {
+ msg: "Some Error Message".to_string(),
+ };
+ let (_, _, message, payload) = classify(panic, None);
+ assert_eq!(message, "Some Error Message");
+ assert!(payload["backtrace"].is_null());
+ }
+
+ #[test]
+ fn native_error_payload_shape() {
+ let error = CometError::from(DataFusionError::Context(
+ "outer".to_string(),
+ Box::new(DataFusionError::Execution("inner".to_string())),
+ ));
+ let native_error = NativeError::from_comet_error(&error, None);
+ let payload: serde_json::Value = serde_json::from_str(&native_error.to_payload()).unwrap();
+ assert_eq!(payload["version"], NATIVE_ERROR_PAYLOAD_VERSION);
+ assert_eq!(payload["status"], "Error");
+ assert_eq!(payload["kind"], "new");
+ assert_eq!(payload["exceptionClass"], native_error.exception_class());
+ assert_eq!(payload["message"], native_error.message());
+ assert_eq!(
+ payload["causes"],
+ serde_json::json!(native_error.causes.clone())
+ );
+ assert_eq!(
+ native_error.causes.first(),
+ Some(&error_source_string(&error))
+ );
+ assert_eq!(NativeStatus::Ok as i32, 0);
+ assert_eq!(NativeStatus::Error as i32, 1);
+ assert_eq!(NativeStatus::Panic as i32, 2);
+ assert_eq!(NativeStatus::JavaThrowable as i32, 3);
+ }
+
+ fn error_source_string(error: &CometError) -> String {
+ std::error::Error::source(error).unwrap().to_string()
+ }
+
+ #[test]
+ fn catch_native_reports_errors_and_panics() {
+ assert_eq!(catch_native(|| Ok(7)).unwrap(), 7);
+
+ let failure =
+ catch_native(|| -> CometResult<()> { Err(CometError::Internal("oops".to_string())) })
+ .unwrap_err();
+ assert!(failure.backtrace.is_none());
+ let native_error = failure.native_error();
+ assert_eq!(native_error.status, NativeStatus::Error);
+ assert_eq!(native_error.message(), "Comet Internal Error: oops");
+
+ let failure = catch_native(|| -> CometResult<()> { panic!("boom") }).unwrap_err();
+ let native_error = failure.native_error();
+ assert_eq!(native_error.status, NativeStatus::Panic);
+ assert_eq!(
+ native_error.exception_class(),
+ "org/apache/comet/CometNativeException"
+ );
+ assert!(native_error.message().starts_with("boom"));
+ }
+
+ /// A throwable captured during an upcall is rethrown as the same object.
+ #[test]
+ #[cfg_attr(miri, ignore)] // miri cannot create a JVM.
+ fn java_exception_is_rethrown() {
+ jvm()
+ .attach_current_thread(|env| -> jni::errors::Result<()> {
+ let message: JObject = env.new_string("from jvm")?.into();
+ let throwable = env.new_object(
+ jni::jni_str!("java/lang/IllegalArgumentException"),
+ jni::jni_sig!("(Ljava/lang/String;)V"),
+ &[jni::objects::JValue::Object(&message)],
+ )?;
+ let throwable = unsafe { JThrowable::from_raw(env, throwable.into_raw()) };
+ let error = CometError::JavaException {
+ class: "java.lang.IllegalArgumentException".to_string(),
+ msg: "from jvm".to_string(),
+ throwable: env.new_global_ref(throwable)?,
+ };
+ let native_error = NativeError::from_comet_error(&error, None);
+ assert_eq!(native_error.status, NativeStatus::JavaThrowable);
+ let payload: serde_json::Value =
+ serde_json::from_str(&native_error.to_payload()).unwrap();
+ assert_eq!(payload["kind"], "rethrow");
+
+ unwrap_or_throw_default::<()>(env, Err(error));
+ assert_pending_java_exception_detailed(
+ env,
+ Some("java/lang/IllegalArgumentException"),
+ Some("from jvm"),
+ );
+ Ok(())
+ })
+ .unwrap();
+ }
+
// Example of a simple JNI "Hello World" program. It can be used to demonstrate:
// * returning an object
// * throwing an exception from `.expect()`
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@datafusion.apache.org
For additional commands, e-mail: commits-help@datafusion.apache.org