You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@datafusion.apache.org by "viirya (via GitHub)" <gi...@apache.org> on 2026/10/23 18:26:04 UTC
[PR] feat: interrupt a native scalar Arrow UDF when its task is killed [datafusion-comet]
viirya opened a new pull request, #7181:
URL: https://github.com/apache/datafusion-comet/pull/7181
## Which issue does this PR close?
Closes #6811.
## Rationale for this change
The native scalar Arrow UDF path from #6130 runs the Python callable on the Spark task thread, inside `tokio::task::block_in_place`. A Spark kill does not reach it: the Java interrupt is ignored while the thread is in Python, and Spark's `PythonRunner` `MonitorThread`, which destroys the Python worker of a killed task, does not apply. A cancelled job or the losing copy of a speculative task therefore kept its executor core, and kept competing for the embedded interpreter's GIL, until the UDF returned on its own. A stuck UDF never released them.
## What changes are included in this PR?
Native side (`python_udf.rs`, `arrow_python_udf.rs`):
- A global registry maps a task attempt id to the Python thread ids currently running one of its UDF calls. `ArrowPythonUdfExec::execute` registers the plan under its task attempt (now passed from `createPlan` through `ExecutionContext` and the planner). Each call records its thread right before entering the callable and removes it right after, on success or error. The entry is removed when the task's last plan drops its registration, i.e. when the plan is released.
- `interrupt_task` marks the task as killed and raises `comet.TaskKilled` in its recorded threads with `PyThreadState_SetAsyncExc`. `TaskKilled` subclasses `BaseException`, so a user `except Exception` block does not swallow it. Once a task is killed, later calls fail without entering Python, and a call that returned just as the interrupt arrived clears the pending exception on its thread and fails with the same error.
- Lock ordering: holders of the registry lock never wait for the GIL. `interrupt_task` sets the flag without the GIL and attaches only when a call is running, so it does not touch Python for a task that is not in a UDF call.
- New JNI functions: `NativeBase.interruptPythonUdfs(taskAttemptId)` (a no-op that returns 0 in builds without `python-udf`) and `Native.hasPythonUdf(plan)`.
JVM side:
- `CometExecIterator` asks `hasPythonUdf` only when `NativeBase.supportsPythonUdf()`. For a plan that contains an Arrow UDF, it starts `PythonUdfMonitor`, one daemon thread per task like Spark's `MonitorThread`, which polls `TaskContext.isInterrupted` every 100 ms until the task completes and calls the interrupt while the task is killed. The interrupt is repeated, so it also reaches calls that start after an earlier attempt and calls that caught the exception. Unlike Spark's monitor, it does not wait `spark.python.task.killTimeout`, because the interrupt is cooperative and lets the call unwind instead of killing a process. Plans without a native Arrow UDF, and builds without the feature, start no thread.
- The UDF then fails with a non-fatal error. Spark's `Executor` reports a non-fatal failure of a task whose `reasonIfKilled` is set as `TaskKilled`; this is the same code in Spark 4.1.3 and 4.2.0 (`Executor.scala`, `case _: InterruptedException | NonFatal(_) if task != null && task.reasonIfKilled.isDefined`).
Known limitation, now documented in the PyArrow UDF guide in place of the old "killing a task does not interrupt" note: an asynchronous exception takes effect only when the thread next runs Python bytecode. A call blocked in native code, such as a long PyArrow kernel that released the GIL, keeps the task's core until that code returns.
## How are these changes tested?
- Rust: `interrupts_running_call_of_killed_task` runs a `while True` UDF that wraps its body in `try`/`except Exception`, interrupts it from another thread, and checks that the call fails with `TaskKilled`, that its thread is unregistered, that a later call of the killed task fails without entering Python, and that the registry entry is removed when the registration is dropped. With the `PyThreadState_SetAsyncExc` call stubbed out, the test fails ("the interrupted call did not return").
- Scala: `CometArrowPythonUdfSuite` "killing a task interrupts its running native Arrow UDF" cancels the job group of a query whose native UDF loops forever, and checks that the task ends as `TaskKilled` and the job fails within 30 seconds. With the monitor disabled, the task is never killed and the test fails.
- I ran the Rust tests and the full `CometArrowPythonUdfSuite` with `-Pspark-4.1` on Python 3.10 and 3.13.
This touches the planner and a native operator, so it would be good to run it with the `run-pyarrow-udf-tests` and `run-spark-4.1-tests` labels.
This pull request and its description were written by Isaac.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] feat: interrupt a native scalar Arrow UDF when its task is killed [datafusion-comet]
Posted by "viirya (via GitHub)" <gi...@apache.org>.
viirya commented on code in PR #7181:
URL: https://github.com/apache/datafusion-comet/pull/7181#discussion_r4369216134
##########
spark/src/main/scala/org/apache/comet/CometExecIterator.scala:
##########
@@ -235,6 +235,11 @@ class CometExecIterator(
CometExecIterator.startMemoryUsageLog()
+ // A native Python UDF ignores the task's kill, so watch the task to interrupt its calls.
+ if (PythonUdfMonitor.supported && nativeLib.hasPythonUdf(plan)) {
Review Comment:
Right: `addTaskCompletionListener` runs the listener at once on a completed task, and `close()` releases the plan before the probe read it. Fixed in 3d9fe9ec9. The probe now runs inside the cleanup-protected block that creates the plan, before the listener is registered. That block also binds the task's token (see the token thread) and counts the plan for the monitor, so a failure there releases the plan like any other setup failure.
##########
native/core/src/execution/python_udf.rs:
##########
@@ -29,14 +29,177 @@ use pyo3::ffi::Py_uintptr_t;
use pyo3::prelude::*;
use pyo3::sync::MutexExt;
use pyo3::types::{PyBytes, PyList, PyTuple};
-use std::sync::{Mutex, PoisonError};
+use pyo3::PyTypeInfo;
+use std::collections::HashMap;
+use std::os::raw::{c_long, c_ulong};
+use std::sync::{Arc, LazyLock, Mutex, MutexGuard, PoisonError};
// Tasks create their callables concurrently. Concurrent first imports of a package
// can hand a thread a partially initialized module (observed with PyArrow on Python
// 3.10), so callables are unpickled one at a time. Take it with `lock_py_attached`,
// which waits for the lock without holding the GIL.
static IMPORT_LOCK: Mutex<()> = Mutex::new(());
+pyo3::create_exception!(
+ comet,
+ TaskKilled,
+ pyo3::exceptions::PyBaseException,
+ "Raised in a running Arrow UDF when Spark kills its task."
+);
+
+unsafe extern "C" {
+ // Part of the stable Python C API, but not declared by pyo3-ffi.
+ fn PyThread_get_thread_ident() -> c_ulong;
+}
+
+/// The native plans of one Spark task attempt that hold Arrow UDFs.
+#[derive(Default)]
+struct TaskCalls {
+ plans: usize,
+ killed: bool,
+ /// The calls running now.
+ calls: Vec<Call>,
+}
+
+struct Call {
+ /// Python thread id of the thread running the call.
+ thread: c_ulong,
+ /// Whether [`interrupt_task`] has raised in the call. It raises once, so that a second
+ /// exception does not abort the cleanup the first one started.
+ interrupted: bool,
+}
+
+// Holders of this lock never wait for the GIL, so a thread attached to Python may take it.
+static TASKS: LazyLock<Mutex<HashMap<i64, TaskCalls>>> = LazyLock::new(Default::default);
+
+fn tasks() -> MutexGuard<'static, HashMap<i64, TaskCalls>> {
+ TASKS.lock().unwrap_or_else(PoisonError::into_inner)
+}
+
+/// Registers a native plan's Arrow UDFs under their Spark task attempt, so that
+/// [`interrupt_task`] reaches their calls. The task's entry is removed when its last
+/// registration is dropped, which happens when the plan is released.
+pub struct TaskRegistration {
+ task_attempt_id: i64,
+}
+
+impl TaskRegistration {
+ pub fn new(task_attempt_id: i64) -> Self {
+ tasks().entry(task_attempt_id).or_default().plans += 1;
+ Self { task_attempt_id }
+ }
+
+ fn is_killed(&self) -> bool {
+ tasks()
+ .get(&self.task_attempt_id)
+ .is_some_and(|task| task.killed)
+ }
+
+ fn killed_error(&self) -> ArrowError {
+ ArrowError::ComputeError(format!(
+ "Arrow UDF stopped because task {} was killed",
+ self.task_attempt_id
+ ))
+ }
+}
+
+impl Drop for TaskRegistration {
+ fn drop(&mut self) {
+ let mut tasks = tasks();
+ if let Some(task) = tasks.get_mut(&self.task_attempt_id) {
+ task.plans -= 1;
+ if task.plans == 0 {
+ tasks.remove(&self.task_attempt_id);
+ }
+ }
+ }
+}
+
+/// A call to the user's callable, recorded so that [`interrupt_task`] can raise in its thread.
+struct RunningCall<'a> {
+ task: &'a TaskRegistration,
+ thread: c_ulong,
+}
+
+impl<'a> RunningCall<'a> {
+ /// Fails without recording the call when the task has already been killed.
+ fn enter(_py: Python<'_>, task: &'a TaskRegistration) -> Result<Self> {
+ // SAFETY: the caller is attached to the interpreter.
+ let thread = unsafe { PyThread_get_thread_ident() };
+ let mut tasks = tasks();
+ let calls = tasks.entry(task.task_attempt_id).or_default();
+ if calls.killed {
+ return Err(task.killed_error());
+ }
+ calls.calls.push(Call {
+ thread,
+ interrupted: false,
+ });
+ Ok(Self { task, thread })
+ }
+}
+
+impl Drop for RunningCall<'_> {
+ fn drop(&mut self) {
+ let mut tasks = tasks();
+ let Some(calls) = tasks.get_mut(&self.task.task_attempt_id) else {
+ return;
+ };
+ let Some(index) = calls.calls.iter().position(|c| c.thread == self.thread) else {
+ return;
+ };
+ if calls.calls.swap_remove(index).interrupted {
+ // An interrupt that arrived as the callable returned is still pending on this
+ // thread. Nothing can set another one now that the call is unrecorded, so clear
+ // it rather than let it fire in later Python code on this thread.
+ // SAFETY: the call is made while attached, on the thread it names.
+ unsafe {
+ pyo3::ffi::PyThreadState_SetAsyncExc(self.thread as c_long, std::ptr::null_mut())
+ };
+ }
+ }
+}
+
+/// Marks a Spark task attempt as killed and raises [`TaskKilled`] in its running Arrow UDF
+/// calls that it has not interrupted yet. Later calls of the task fail without entering
+/// Python. Returns the number of calls interrupted. A task with no registered plan is not
+/// recorded, so the caller repeats this until the task completes.
+///
+/// The exception takes effect when the thread next runs Python bytecode: a call blocked in
+/// native code that released the GIL is interrupted only when that code returns.
+pub fn interrupt_task(task_attempt_id: i64) -> usize {
+ let running = match tasks().get_mut(&task_attempt_id) {
+ Some(task) => {
+ task.killed = true;
+ task.calls.iter().any(|call| !call.interrupted)
+ }
+ None => false,
+ };
+ if !running {
+ return 0;
+ }
+ // Calls cannot start after the flag is set, so only calls recorded before it remain.
+ Python::attach(|py| {
+ let exception = TaskKilled::type_object(py);
+ let mut tasks = tasks();
+ let Some(task) = tasks.get_mut(&task_attempt_id) else {
+ return 0;
+ };
+ let mut interrupted = 0;
+ for call in task.calls.iter_mut().filter(|call| !call.interrupted) {
+ // SAFETY: attached to the interpreter, and `exception` is a live type object.
+ let set = unsafe {
+ pyo3::ffi::PyThreadState_SetAsyncExc(call.thread as c_long, exception.as_ptr())
Review Comment:
This one holds and the code cannot rule it out: the exception can fire right after a C `acquire()` returns and before the `try` whose `finally` releases it, and a kill that lands while the call waits for the lock makes that the likely outcome rather than a narrow race. A Python `__enter__` that acquires a lock has the same gap. Moving `acquire()` into the `try` only trades it for releasing a lock that was never taken. Spark avoids the problem by destroying the worker process, which a shared interpreter cannot do.
In 3d9fe9ec9 the monitor reduces and exposes the risk instead:
- A killed task's later calls fail at once without entering Python, which is always safe.
- A running call gets `TaskKilled` only after it has run for `spark.python.task.killTimeout` since the kill, like Spark's monitor, so calls that finish within the timeout are never interrupted.
- `spark.comet.exec.nativeArrowPythonUDF.interruptOnKill.enabled=false` disables raising in running calls; a killed task then waits for its call to return. `kill_without_interrupt_lets_the_running_call_finish` covers that mode.
- The user guide describes the lock hazard, including `logging` before Python 3.13, next to the setting.
Explicit cooperative cancellation would not stop an arbitrary loop, which is the case #6811 is about, so I kept the forced interrupt as the default with the opt-out.
##########
spark/src/main/scala/org/apache/comet/PythonUdfMonitor.scala:
##########
@@ -0,0 +1,70 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.comet
+
+import java.util.concurrent.ConcurrentHashMap
+
+import org.apache.spark.TaskContext
+import org.apache.spark.internal.Logging
+
+/**
+ * Interrupts the native Python UDF calls of killed tasks. Spark's Python runner has a
+ * `MonitorThread` that destroys the worker of a killed task. A native UDF runs in the embedded
+ * interpreter on the task thread instead, where neither a Java interrupt nor a stopped input
+ * reaches it. So one daemon thread per task polls its kill state and, while the task is killed
+ * but still running, raises an exception in the task's running Python calls, once per call so
+ * that its cleanup can run. Calls that start after the first interrupt fail without entering
+ * Python.
+ */
+object PythonUdfMonitor extends Logging {
+
+ // Raising the exception is cheap and lets the call unwind on its own, so unlike Spark's
+ // monitor this one does not wait `spark.python.task.killTimeout` before acting. Repeating it
+ // reaches calls of plans that the task registers after an earlier attempt.
+ private val PollIntervalMs = 100L
+
+ private val watchedTasks = ConcurrentHashMap.newKeySet[java.lang.Long]()
+
+ /** Whether the native library can run Python UDFs, and so may need its calls interrupted. */
+ lazy val supported: Boolean = NativeBase.supportsPythonUdf()
+
+ /** Starts watching the task unless it is watched already. The watch ends with the task. */
+ def watch(context: TaskContext): Unit = {
+ val taskAttemptId = context.taskAttemptId()
+ if (watchedTasks.add(taskAttemptId)) {
+ val thread = new Thread(s"Comet Python UDF monitor for task $taskAttemptId") {
+ override def run(): Unit = {
+ try {
+ while (!context.isCompleted()) {
Review Comment:
Agreed. `BatchProducer.stop` takes the stream's mutex, which the producer holds for the whole poll, including the Python call, so `releasePlan` waits for that call after `isCompleted` is already true. Fixed in 3d9fe9ec9. Each watch counts the native UDF plans bound to it, `CometExecIterator.close` uncounts its plan after `releasePlan` returns, and the monitor runs until the task has completed and no counted plan remains. The new "killing a completed task interrupts the UDF call its plan release waits for" test reads a two-row Parquet file one row per batch, takes the first row while the second loops in the UDF, completes the task, and cancels the job; it fails with the previous lifetime.
##########
native/core/src/execution/python_udf.rs:
##########
@@ -29,14 +29,177 @@ use pyo3::ffi::Py_uintptr_t;
use pyo3::prelude::*;
use pyo3::sync::MutexExt;
use pyo3::types::{PyBytes, PyList, PyTuple};
-use std::sync::{Mutex, PoisonError};
+use pyo3::PyTypeInfo;
+use std::collections::HashMap;
+use std::os::raw::{c_long, c_ulong};
+use std::sync::{Arc, LazyLock, Mutex, MutexGuard, PoisonError};
// Tasks create their callables concurrently. Concurrent first imports of a package
// can hand a thread a partially initialized module (observed with PyArrow on Python
// 3.10), so callables are unpickled one at a time. Take it with `lock_py_attached`,
// which waits for the lock without holding the GIL.
static IMPORT_LOCK: Mutex<()> = Mutex::new(());
+pyo3::create_exception!(
+ comet,
+ TaskKilled,
+ pyo3::exceptions::PyBaseException,
+ "Raised in a running Arrow UDF when Spark kills its task."
+);
+
+unsafe extern "C" {
+ // Part of the stable Python C API, but not declared by pyo3-ffi.
+ fn PyThread_get_thread_ident() -> c_ulong;
+}
+
+/// The native plans of one Spark task attempt that hold Arrow UDFs.
+#[derive(Default)]
+struct TaskCalls {
+ plans: usize,
+ killed: bool,
+ /// The calls running now.
+ calls: Vec<Call>,
+}
+
+struct Call {
+ /// Python thread id of the thread running the call.
+ thread: c_ulong,
+ /// Whether [`interrupt_task`] has raised in the call. It raises once, so that a second
+ /// exception does not abort the cleanup the first one started.
+ interrupted: bool,
+}
+
+// Holders of this lock never wait for the GIL, so a thread attached to Python may take it.
+static TASKS: LazyLock<Mutex<HashMap<i64, TaskCalls>>> = LazyLock::new(Default::default);
+
+fn tasks() -> MutexGuard<'static, HashMap<i64, TaskCalls>> {
+ TASKS.lock().unwrap_or_else(PoisonError::into_inner)
+}
+
+/// Registers a native plan's Arrow UDFs under their Spark task attempt, so that
+/// [`interrupt_task`] reaches their calls. The task's entry is removed when its last
+/// registration is dropped, which happens when the plan is released.
+pub struct TaskRegistration {
+ task_attempt_id: i64,
+}
+
+impl TaskRegistration {
+ pub fn new(task_attempt_id: i64) -> Self {
+ tasks().entry(task_attempt_id).or_default().plans += 1;
Review Comment:
Agreed, `taskAttemptId` is only unique within one SparkContext. Fixed in 3d9fe9ec9. `PythonUdfMonitor` assigns each watched `TaskContext` a token from a process-wide counter. `CometExecIterator` binds it to the plan with a new `Native.setPythonUdfTask` before the first `executePlan`, and the native registry, `ArrowPythonUdfExec`, and `NativeBase.killPythonUdfs` all use the token instead of the attempt id.
##########
native/core/src/execution/python_udf.rs:
##########
@@ -297,14 +468,18 @@ impl ArrowPythonUdf {
kwargs.set_item(name, arg).map_err(python_error)?;
}
}
- let result = self
- .callable
- .bind(py)
- .call(
- PyTuple::new(py, positional).map_err(python_error)?,
- Some(&kwargs),
- )
- .map_err(|error| python_traceback_error(py, error))?;
+ let positional = PyTuple::new(py, positional).map_err(python_error)?;
+ let call = self
+ .task
+ .as_deref()
+ .map(|task| RunningCall::enter(py, task))
+ .transpose()?;
+ let result = self.callable.bind(py).call(positional, Some(&kwargs));
+ drop(call);
Review Comment:
Fixed in 3d9fe9ec9. The call now stays recorded through the Spark 4.2 `RecordBatch.from_arrays` conversion, and the killed check runs after it, so a cancelled conversion fails instead of returning a batch. `interrupts_a_user_defined_array_conversion` interrupts a result whose `__arrow_array__` loops, and fails if the call is unrecorded before the conversion.
##########
native/core/src/execution/python_udf.rs:
##########
@@ -29,14 +29,177 @@ use pyo3::ffi::Py_uintptr_t;
use pyo3::prelude::*;
use pyo3::sync::MutexExt;
use pyo3::types::{PyBytes, PyList, PyTuple};
-use std::sync::{Mutex, PoisonError};
+use pyo3::PyTypeInfo;
+use std::collections::HashMap;
+use std::os::raw::{c_long, c_ulong};
+use std::sync::{Arc, LazyLock, Mutex, MutexGuard, PoisonError};
// Tasks create their callables concurrently. Concurrent first imports of a package
// can hand a thread a partially initialized module (observed with PyArrow on Python
// 3.10), so callables are unpickled one at a time. Take it with `lock_py_attached`,
// which waits for the lock without holding the GIL.
static IMPORT_LOCK: Mutex<()> = Mutex::new(());
+pyo3::create_exception!(
+ comet,
+ TaskKilled,
+ pyo3::exceptions::PyBaseException,
+ "Raised in a running Arrow UDF when Spark kills its task."
+);
+
+unsafe extern "C" {
+ // Part of the stable Python C API, but not declared by pyo3-ffi.
+ fn PyThread_get_thread_ident() -> c_ulong;
+}
+
+/// The native plans of one Spark task attempt that hold Arrow UDFs.
+#[derive(Default)]
+struct TaskCalls {
+ plans: usize,
+ killed: bool,
+ /// The calls running now.
+ calls: Vec<Call>,
+}
+
+struct Call {
+ /// Python thread id of the thread running the call.
+ thread: c_ulong,
+ /// Whether [`interrupt_task`] has raised in the call. It raises once, so that a second
+ /// exception does not abort the cleanup the first one started.
+ interrupted: bool,
+}
+
+// Holders of this lock never wait for the GIL, so a thread attached to Python may take it.
+static TASKS: LazyLock<Mutex<HashMap<i64, TaskCalls>>> = LazyLock::new(Default::default);
+
+fn tasks() -> MutexGuard<'static, HashMap<i64, TaskCalls>> {
+ TASKS.lock().unwrap_or_else(PoisonError::into_inner)
+}
+
+/// Registers a native plan's Arrow UDFs under their Spark task attempt, so that
+/// [`interrupt_task`] reaches their calls. The task's entry is removed when its last
+/// registration is dropped, which happens when the plan is released.
+pub struct TaskRegistration {
+ task_attempt_id: i64,
+}
+
+impl TaskRegistration {
+ pub fn new(task_attempt_id: i64) -> Self {
+ tasks().entry(task_attempt_id).or_default().plans += 1;
+ Self { task_attempt_id }
+ }
+
+ fn is_killed(&self) -> bool {
+ tasks()
+ .get(&self.task_attempt_id)
+ .is_some_and(|task| task.killed)
+ }
+
+ fn killed_error(&self) -> ArrowError {
+ ArrowError::ComputeError(format!(
+ "Arrow UDF stopped because task {} was killed",
+ self.task_attempt_id
+ ))
+ }
+}
+
+impl Drop for TaskRegistration {
+ fn drop(&mut self) {
+ let mut tasks = tasks();
+ if let Some(task) = tasks.get_mut(&self.task_attempt_id) {
+ task.plans -= 1;
+ if task.plans == 0 {
+ tasks.remove(&self.task_attempt_id);
+ }
+ }
+ }
+}
+
+/// A call to the user's callable, recorded so that [`interrupt_task`] can raise in its thread.
+struct RunningCall<'a> {
+ task: &'a TaskRegistration,
+ thread: c_ulong,
+}
+
+impl<'a> RunningCall<'a> {
+ /// Fails without recording the call when the task has already been killed.
+ fn enter(_py: Python<'_>, task: &'a TaskRegistration) -> Result<Self> {
+ // SAFETY: the caller is attached to the interpreter.
+ let thread = unsafe { PyThread_get_thread_ident() };
+ let mut tasks = tasks();
+ let calls = tasks.entry(task.task_attempt_id).or_default();
+ if calls.killed {
+ return Err(task.killed_error());
+ }
+ calls.calls.push(Call {
+ thread,
+ interrupted: false,
+ });
+ Ok(Self { task, thread })
+ }
+}
+
+impl Drop for RunningCall<'_> {
+ fn drop(&mut self) {
+ let mut tasks = tasks();
+ let Some(calls) = tasks.get_mut(&self.task.task_attempt_id) else {
+ return;
+ };
+ let Some(index) = calls.calls.iter().position(|c| c.thread == self.thread) else {
+ return;
+ };
+ if calls.calls.swap_remove(index).interrupted {
+ // An interrupt that arrived as the callable returned is still pending on this
+ // thread. Nothing can set another one now that the call is unrecorded, so clear
+ // it rather than let it fire in later Python code on this thread.
+ // SAFETY: the call is made while attached, on the thread it names.
+ unsafe {
+ pyo3::ffi::PyThreadState_SetAsyncExc(self.thread as c_long, std::ptr::null_mut())
Review Comment:
Agreed. `PyThreadState_SetAsyncExc` signals the interpreter even for a NULL exception, and only a thread that holds an exception clears the flag. Fixed in 3d9fe9ec9. After unregistering an interrupted call and releasing the registry lock, the thread runs a short loop of Python bytecode, so a still-pending `TaskKilled` fires and is discarded there, and nothing is ever cleared with NULL. `consumes_an_interrupt_pending_when_the_call_returns` interrupts a C callable (`functools.partial(time.sleep, 0.3)`) that returns straight to Rust, then runs more Python on the same thread state and checks that the interrupt does not fire there.
##########
native/core/src/execution/python_udf.rs:
##########
@@ -297,14 +468,18 @@ impl ArrowPythonUdf {
kwargs.set_item(name, arg).map_err(python_error)?;
}
}
- let result = self
- .callable
- .bind(py)
- .call(
- PyTuple::new(py, positional).map_err(python_error)?,
- Some(&kwargs),
- )
- .map_err(|error| python_traceback_error(py, error))?;
+ let positional = PyTuple::new(py, positional).map_err(python_error)?;
+ let call = self
+ .task
+ .as_deref()
+ .map(|task| RunningCall::enter(py, task))
Review Comment:
Fixed in 3d9fe9ec9. `evaluate_named` checks the killed flag before `Python::attach`, and `RunningCall::enter` keeps its check for a kill that lands while the batch's arguments are imported.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] feat: interrupt a native scalar Arrow UDF when its task is killed [datafusion-comet]
Posted by "rich7420 (via GitHub)" <gi...@apache.org>.
rich7420 commented on code in PR #7181:
URL: https://github.com/apache/datafusion-comet/pull/7181#discussion_r4391238881
##########
native/core/src/execution/python_udf.rs:
##########
@@ -297,14 +468,18 @@ impl ArrowPythonUdf {
kwargs.set_item(name, arg).map_err(python_error)?;
}
}
- let result = self
- .callable
- .bind(py)
- .call(
- PyTuple::new(py, positional).map_err(python_error)?,
- Some(&kwargs),
- )
- .map_err(|error| python_traceback_error(py, error))?;
+ let positional = PyTuple::new(py, positional).map_err(python_error)?;
+ let call = self
+ .task
+ .as_deref()
+ .map(|task| RunningCall::enter(py, task))
+ .transpose()?;
+ let result = self.callable.bind(py).call(positional, Some(&kwargs));
+ drop(call);
Review Comment:
Keep registration through cast; ExtensionArray.cast still escapes cancellation and returns a successful batch after kill_task.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] feat: interrupt a native scalar Arrow UDF when its task is killed [datafusion-comet]
Posted by "sunchao (via GitHub)" <gi...@apache.org>.
sunchao commented on code in PR #7181:
URL: https://github.com/apache/datafusion-comet/pull/7181#discussion_r4363245616
##########
native/core/src/execution/python_udf.rs:
##########
@@ -29,14 +29,164 @@ use pyo3::ffi::Py_uintptr_t;
use pyo3::prelude::*;
use pyo3::sync::MutexExt;
use pyo3::types::{PyBytes, PyList, PyTuple};
-use std::sync::{Mutex, PoisonError};
+use pyo3::PyTypeInfo;
+use std::collections::HashMap;
+use std::os::raw::{c_long, c_ulong};
+use std::sync::{Arc, LazyLock, Mutex, MutexGuard, PoisonError};
// Tasks create their callables concurrently. Concurrent first imports of a package
// can hand a thread a partially initialized module (observed with PyArrow on Python
// 3.10), so callables are unpickled one at a time. Take it with `lock_py_attached`,
// which waits for the lock without holding the GIL.
static IMPORT_LOCK: Mutex<()> = Mutex::new(());
+pyo3::create_exception!(
+ comet,
+ TaskKilled,
+ pyo3::exceptions::PyBaseException,
+ "Raised in a running Arrow UDF when Spark kills its task."
+);
+
+unsafe extern "C" {
+ // Part of the stable Python C API, but not declared by pyo3-ffi.
+ fn PyThread_get_thread_ident() -> c_ulong;
+}
+
+/// The native plans of one Spark task attempt that hold Arrow UDFs.
+#[derive(Default)]
+struct TaskCalls {
+ plans: usize,
+ killed: bool,
+ /// Python thread ids of the calls running now, one entry per call.
+ threads: Vec<c_ulong>,
+}
+
+// Holders of this lock never wait for the GIL, so a thread attached to Python may take it.
+static TASKS: LazyLock<Mutex<HashMap<i64, TaskCalls>>> = LazyLock::new(Default::default);
+
+fn tasks() -> MutexGuard<'static, HashMap<i64, TaskCalls>> {
+ TASKS.lock().unwrap_or_else(PoisonError::into_inner)
+}
+
+/// Registers a native plan's Arrow UDFs under their Spark task attempt, so that
+/// [`interrupt_task`] reaches their calls. The task's entry is removed when its last
+/// registration is dropped, which happens when the plan is released.
+pub struct TaskRegistration {
+ task_attempt_id: i64,
+}
+
+impl TaskRegistration {
+ pub fn new(task_attempt_id: i64) -> Self {
+ tasks().entry(task_attempt_id).or_default().plans += 1;
+ Self { task_attempt_id }
+ }
+
+ fn is_killed(&self) -> bool {
+ tasks()
+ .get(&self.task_attempt_id)
+ .is_some_and(|task| task.killed)
+ }
+
+ fn killed_error(&self) -> ArrowError {
+ ArrowError::ComputeError(format!(
+ "Arrow UDF stopped because task {} was killed",
+ self.task_attempt_id
+ ))
+ }
+}
+
+impl Drop for TaskRegistration {
+ fn drop(&mut self) {
+ let mut tasks = tasks();
+ if let Some(task) = tasks.get_mut(&self.task_attempt_id) {
+ task.plans -= 1;
+ if task.plans == 0 {
+ tasks.remove(&self.task_attempt_id);
+ }
+ }
+ }
+}
+
+/// A call to the user's callable, recorded so that [`interrupt_task`] can raise in its thread.
+struct RunningCall<'a> {
+ task: &'a TaskRegistration,
+ thread: c_ulong,
+}
+
+impl<'a> RunningCall<'a> {
+ /// Fails without recording the call when the task has already been killed.
+ fn enter(_py: Python<'_>, task: &'a TaskRegistration) -> Result<Self> {
+ // SAFETY: the caller is attached to the interpreter.
+ let thread = unsafe { PyThread_get_thread_ident() };
+ let mut tasks = tasks();
+ let calls = tasks.entry(task.task_attempt_id).or_default();
+ if calls.killed {
+ return Err(task.killed_error());
+ }
+ calls.threads.push(thread);
+ Ok(Self { task, thread })
+ }
+}
+
+impl Drop for RunningCall<'_> {
+ fn drop(&mut self) {
+ let mut tasks = tasks();
+ let Some(calls) = tasks.get_mut(&self.task.task_attempt_id) else {
+ return;
+ };
+ if let Some(index) = calls.threads.iter().position(|t| *t == self.thread) {
+ calls.threads.swap_remove(index);
+ }
+ if calls.killed {
+ // An interrupt that arrived as the callable returned is still pending on this
+ // thread. Nothing can set another one now that the call is unrecorded, so clear
+ // it rather than let it fire in later Python code on this thread.
+ // SAFETY: the call is made while attached, on the thread it names.
+ unsafe {
+ pyo3::ffi::PyThreadState_SetAsyncExc(self.thread as c_long, std::ptr::null_mut())
+ };
+ }
+ }
+}
+
+/// Marks a Spark task attempt as killed and raises [`TaskKilled`] in its running Arrow UDF
+/// calls. Later calls of the task fail without entering Python. Returns the number of calls
+/// interrupted. A task with no registered plan is not recorded, so the caller repeats this
+/// until the task completes.
+///
+/// The exception takes effect when the thread next runs Python bytecode: a call blocked in
+/// native code that released the GIL is interrupted only when that code returns.
+pub fn interrupt_task(task_attempt_id: i64) -> usize {
+ let running = match tasks().get_mut(&task_attempt_id) {
+ Some(task) => {
+ task.killed = true;
+ !task.threads.is_empty()
+ }
+ None => false,
+ };
+ if !running {
+ return 0;
+ }
+ // Calls cannot start after the flag is set, so only calls recorded before it remain.
+ Python::attach(|py| {
+ let exception = TaskKilled::type_object(py);
+ let tasks = tasks();
+ let Some(task) = tasks.get(&task_attempt_id) else {
+ return 0;
+ };
+ task.threads
+ .iter()
+ .map(|thread| {
+ // SAFETY: attached to the interpreter, and `exception` is a live type object.
+ let set = unsafe {
+ pyo3::ffi::PyThreadState_SetAsyncExc(*thread as c_long, exception.as_ptr())
Review Comment:
[P2] [P2] Avoid reinjecting cancellation while a call is unwinding
A call remains in `task.threads` until its entire Python invocation returns, including its `finally` blocks. Because the monitor calls this method every 100 ms, subsequent `TaskKilled` exceptions also interrupt cleanup that is already handling the first one. For example, a UDF holding a module-level lock with `finally: flush(); lock.release()` leaves that lock permanently held when the cleanup takes longer than a polling interval. Other tasks share that module and can then block indefinitely. A single interrupt lets the same cleanup finish. Could cancellation delivery be recorded per active call so subsequent polls do not reinject into it, while retaining the killed flag that rejects later calls?
Evidence: A bounded Rust harness copied this head's TaskRegistration, RunningCall, and interrupt_task implementation unchanged. With PyO3 0.28.3 and CPython 3.12.13, the callable acquired a shared threading.Lock, entered a loop, and used `finally: cleaning.set(); time.sleep(0.4); lock.release()`. One interrupt produced TaskKilled and left the lock available. A second interrupt 100 ms later, after confirming cleanup had begun, also produced TaskKilled but left the lock unavailable after the call ended. Both calls were correctly unregistered, so the retained lock outlived cancellation. Command: `cargo test --offline -- --nocapture --test-threads=1` in `/tmp/comet7181-908ae-root-probe-xk1ssxaq`. Output: `repeat=false, shared lock available after cancellation=true` and `repeat=true, shared lock available after cancellation=false`. The production monitor repeats delivery at this same interval.
##########
spark/src/main/scala/org/apache/comet/PythonUdfMonitor.scala:
##########
@@ -0,0 +1,69 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.comet
+
+import java.util.concurrent.ConcurrentHashMap
+
+import org.apache.spark.TaskContext
+import org.apache.spark.internal.Logging
+
+/**
+ * Interrupts the native Python UDF calls of killed tasks. Spark's Python runner has a
+ * `MonitorThread` that destroys the worker of a killed task. A native UDF runs in the embedded
+ * interpreter on the task thread instead, where neither a Java interrupt nor a stopped input
+ * reaches it. So one daemon thread per task polls its kill state and, while the task is killed
+ * but still running, raises an exception in the task's running Python calls. Calls that start
+ * after the first interrupt fail without entering Python.
+ */
+object PythonUdfMonitor extends Logging {
+
+ // Raising the exception is cheap and lets the call unwind on its own, so unlike Spark's
+ // monitor this one does not wait `spark.python.task.killTimeout` before acting. Repeating it
+ // reaches calls that start after an earlier attempt and calls that caught the exception.
+ private val PollIntervalMs = 100L
+
+ private val watchedTasks = ConcurrentHashMap.newKeySet[java.lang.Long]()
+
+ /** Whether the native library can run Python UDFs, and so may need its calls interrupted. */
+ lazy val supported: Boolean = NativeBase.supportsPythonUdf()
+
+ /** Starts watching the task unless it is watched already. The watch ends with the task. */
+ def watch(context: TaskContext): Unit = {
+ val taskAttemptId = context.taskAttemptId()
+ if (watchedTasks.add(taskAttemptId)) {
+ val thread = new Thread(s"Comet Python UDF monitor for task $taskAttemptId") {
+ override def run(): Unit = {
+ try {
+ while (!context.isCompleted()) {
+ if (context.isInterrupted() && NativeBase.interruptPythonUdfs(taskAttemptId) > 0) {
+ logInfo(s"Interrupted Python UDF calls of killed task $taskAttemptId")
+ }
+ Thread.sleep(PollIntervalMs)
+ }
+ } finally {
+ watchedTasks.remove(taskAttemptId)
Review Comment:
[P2] [P2] Explicitly discard the removal result for strict Scala compilation
`watchedTasks.remove(taskAttemptId)` returns a Boolean, which is discarded by this `finally` block. The supported Spark 3.5 strict-warnings profile treats that discard as a compilation error, so the new shared source fails the PR's strict Scala job even though native Arrow UDF execution is disabled on that Spark version. Explicitly discard the result, for example with `val _ = watchedTasks.remove(taskAttemptId)`.
Evidence: Exact-head run 40003235556, job 120591854928, executes `./mvnw -B test-compile -Pspark-3.5,delta -Pstrict-warnings -DskipTests`. Its compiler output reports `PythonUdfMonitor.scala:61: discarded non-Unit value`, then exits with compilation failure. This diagnostic points to the newly added file. Log: https://github.com/apache/datafusion-comet/actions/runs/40003235556/job/120591854928.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] feat: interrupt a native scalar Arrow UDF when its task is killed [datafusion-comet]
Posted by "viirya (via GitHub)" <gi...@apache.org>.
viirya commented on code in PR #7181:
URL: https://github.com/apache/datafusion-comet/pull/7181#discussion_r4418695204
##########
spark/src/main/scala/org/apache/comet/PythonUdfMonitor.scala:
##########
@@ -0,0 +1,111 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.comet
+
+import java.util.concurrent.{ConcurrentHashMap, TimeUnit}
+import java.util.concurrent.atomic.{AtomicInteger, AtomicLong}
+
+import org.apache.spark.{SparkEnv, TaskContext}
+import org.apache.spark.internal.Logging
+
+/**
+ * Stops the native Python UDF calls of killed tasks. Spark's Python runner has a `MonitorThread`
+ * that destroys the worker of a killed task. A native UDF runs in the embedded interpreter on the
+ * task thread instead, where neither a Java interrupt nor a stopped input reaches it. So one
+ * daemon thread per task polls its kill state. Once the task is killed, its later calls fail
+ * without entering Python. A call still running `spark.python.task.killTimeout` after the kill
+ * gets an exception raised in it, once so that its cleanup can run, unless that is disabled.
+ */
+object PythonUdfMonitor extends Logging {
Review Comment:
Thanks. I'd rather keep the scope here, because a general name would sit on top of behavior that is specific to Python UDFs, and plan cancellation would need each of these to be different:
- The watch runs only for plans that contain an Arrow UDF (`hasPythonUdf`). #6295 needs every plan watched.
- It waits `spark.python.task.killTimeout` before interrupting, to give a call the grace period Spark's `MonitorThread` gives a worker. Cancelling a plan would act at once.
- `interruptOnKill.enabled` exists because an asynchronous exception can leave a lock held in the shared interpreter, which does not apply to stopping a native stream.
What does carry over is the plumbing: a token unique in the process, bound to the plan before the first `executePlan`, and a watch that keeps running after the task completes until its plans are released. I've left a pointer on #6295.
##########
spark/src/test/spark-4.1+/org/apache/spark/sql/comet/CometArrowPythonUdfSuite.scala:
##########
@@ -583,6 +590,101 @@ class CometArrowPythonUdfSuite extends CometTestBase {
}
}
+ /**
+ * Runs `job` in a job group, waits until `started` exists, cancels the group, and checks that
+ * the task ends as killed.
+ */
+ private def assertCancelKillsTask(started: File)(job: => Any): Unit = {
+ val killed = new CountDownLatch(1)
+ val listener = new SparkListener {
+ override def onTaskEnd(taskEnd: SparkListenerTaskEnd): Unit = taskEnd.reason match {
+ case _: TaskKilled => killed.countDown()
+ case _ =>
+ }
+ }
+ spark.sparkContext.addSparkListener(listener)
+ try {
+ val group = "comet-arrow-udf-kill"
+ val running = Future {
+ spark.sparkContext.setJobGroup(group, "native Arrow UDF loop", interruptOnCancel = true)
+ job
+ }(ExecutionContext.global)
+ val deadline = 60.seconds.fromNow
+ while (!started.exists()) {
+ assert(deadline.hasTimeLeft(), "the UDF did not start")
Review Comment:
Good catch, both fixed in 53b6a5303. The `finally` now cancels the group as well, which does nothing when the job has already ended. The `Future` clears the job group from its pooled thread once the job returns.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] feat: interrupt a native scalar Arrow UDF when its task is killed [datafusion-comet]
Posted by "viirya (via GitHub)" <gi...@apache.org>.
viirya commented on code in PR #7181:
URL: https://github.com/apache/datafusion-comet/pull/7181#discussion_r4369216569
##########
native/core/src/execution/python_udf.rs:
##########
@@ -600,6 +775,122 @@ mod tests {
});
}
+ fn running_calls(task_attempt_id: i64) -> Option<usize> {
+ tasks().get(&task_attempt_id).map(|task| task.calls.len())
+ }
+
+ #[test]
+ fn interrupts_running_call_of_killed_task() {
+ const TASK: i64 = -6811;
+ initialize_python().unwrap();
+ // `except Exception` must not swallow the interrupt.
+ let command = Python::attach(|py| {
+ cloudpickled_lambda(
+ py,
+ c"(lambda ns: (exec('def f(a):\\n n = 0\\n while True:\\n n += 1\\n try:\\n n %= 7\\n except Exception:\\n pass\\n', ns), ns['f'])[1])({})",
+ )
+ });
+ let task = Arc::new(TaskRegistration::new(TASK));
+ let udf = Arc::new(
+ ArrowPythonUdf::from_command(&command, DataType::Int64, true, true, false, "")
+ .unwrap()
+ .with_task(Arc::clone(&task)),
+ );
+ let input: ArrayRef = Arc::new(Int64Array::from(vec![1]));
+ let (sender, receiver) = std::sync::mpsc::channel();
+ {
+ let (udf, input) = (Arc::clone(&udf), Arc::clone(&input));
+ // Not joined: a call that ignores the interrupt would never return.
+ std::thread::spawn(move || sender.send(udf.evaluate(&[input], 1)));
Review Comment:
Fixed in 3d9fe9ec9. The worker drops its `Arc<ArrowPythonUdf>` before it sends the result, in a shared `start_call` helper that the interrupt tests now use.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] feat: interrupt a native scalar Arrow UDF when its task is killed [datafusion-comet]
Posted by "viirya (via GitHub)" <gi...@apache.org>.
viirya commented on PR #7181:
URL: https://github.com/apache/datafusion-comet/pull/7181#issuecomment-6510296911
Thanks, I've rewritten the description to match the current code: the JVM token and `setPythonUdfTask`, `killPythonUdfs` with its once-per-call interrupt after `spark.python.task.killTimeout`, the watch lifetime, `spark.comet.exec.nativeArrowPythonUDF.interruptOnKill.enabled`, and the full list of tests.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] feat: interrupt a native scalar Arrow UDF when its task is killed [datafusion-comet]
Posted by "sunchao (via GitHub)" <gi...@apache.org>.
sunchao commented on code in PR #7181:
URL: https://github.com/apache/datafusion-comet/pull/7181#discussion_r4366518997
##########
spark/src/main/scala/org/apache/comet/CometExecIterator.scala:
##########
@@ -235,6 +235,11 @@ class CometExecIterator(
CometExecIterator.startMemoryUsageLog()
+ // A native Python UDF ignores the task's kill, so watch the task to interrupt its calls.
+ if (PythonUdfMonitor.supported && nativeLib.hasPythonUdf(plan)) {
Review Comment:
[P1] Compute `hasPythonUdf` before registering the listener that calls `close()`. Spark can invoke a completion listener immediately when an input feeder constructs its parent iterator after the consuming task completes. `close()` then frees `plan` before this line reads it. That can terminate the executor for any plan in a Python-enabled build, even with native UDF execution disabled in the session. Cache the Boolean in the cleanup-protected `createdPlan` block.
Evidence: A bounded Spark 4.1.3 `PipedRDD` fixture recorded `completed=true closeRanBeforeProbe=true` using an inert handle. At this head, line 232 publishes the listener, line 375 calls `releasePlan`, and `hasPythonUdf` dereferences the released `ExecutionContext` to read `spark_plan`. The base constructor made no native plan access after publishing that listener.
##########
spark/src/main/scala/org/apache/comet/PythonUdfMonitor.scala:
##########
@@ -0,0 +1,70 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.comet
+
+import java.util.concurrent.ConcurrentHashMap
+
+import org.apache.spark.TaskContext
+import org.apache.spark.internal.Logging
+
+/**
+ * Interrupts the native Python UDF calls of killed tasks. Spark's Python runner has a
+ * `MonitorThread` that destroys the worker of a killed task. A native UDF runs in the embedded
+ * interpreter on the task thread instead, where neither a Java interrupt nor a stopped input
+ * reaches it. So one daemon thread per task polls its kill state and, while the task is killed
+ * but still running, raises an exception in the task's running Python calls, once per call so
+ * that its cleanup can run. Calls that start after the first interrupt fail without entering
+ * Python.
+ */
+object PythonUdfMonitor extends Logging {
+
+ // Raising the exception is cheap and lets the call unwind on its own, so unlike Spark's
+ // monitor this one does not wait `spark.python.task.killTimeout` before acting. Repeating it
+ // reaches calls of plans that the task registers after an earlier attempt.
+ private val PollIntervalMs = 100L
+
+ private val watchedTasks = ConcurrentHashMap.newKeySet[java.lang.Long]()
+
+ /** Whether the native library can run Python UDFs, and so may need its calls interrupted. */
+ lazy val supported: Boolean = NativeBase.supportsPythonUdf()
+
+ /** Starts watching the task unless it is watched already. The watch ends with the task. */
+ def watch(context: TaskContext): Unit = {
+ val taskAttemptId = context.taskAttemptId()
+ if (watchedTasks.add(taskAttemptId)) {
+ val thread = new Thread(s"Comet Python UDF monitor for task $taskAttemptId") {
+ override def run(): Unit = {
+ try {
+ while (!context.isCompleted()) {
Review Comment:
[P2] Keep the watch alive until the last native UDF plan finishes closing. Spark sets `isCompleted` before invoking completion listeners. When a consumer takes the first batch and the producer is evaluating the next one, `releasePlan` can wait in `BatchProducer.stop()` after the monitor exits. A kill during that wait cannot interrupt even a pure Python loop, which can retain the executor slot indefinitely.
Evidence: A Spark 4.1.3 `take(1)` fixture used the head’s monitor, UDF module, and byte-identical `BatchProducer`. After `cancelJobGroup`, it observed `completed=true`, `interrupted=true`, zero native interrupts, and both plan close and task end pending. A direct interrupt returned `1`, released close, and Spark reported `TaskKilled`. [Spark completion order](https://github.com/apache/spark/blob/v4.2.0/core/src/main/scala/org/apache/spark/TaskContextImpl.scala#L159-L164)
##########
native/core/src/execution/python_udf.rs:
##########
@@ -29,14 +29,177 @@ use pyo3::ffi::Py_uintptr_t;
use pyo3::prelude::*;
use pyo3::sync::MutexExt;
use pyo3::types::{PyBytes, PyList, PyTuple};
-use std::sync::{Mutex, PoisonError};
+use pyo3::PyTypeInfo;
+use std::collections::HashMap;
+use std::os::raw::{c_long, c_ulong};
+use std::sync::{Arc, LazyLock, Mutex, MutexGuard, PoisonError};
// Tasks create their callables concurrently. Concurrent first imports of a package
// can hand a thread a partially initialized module (observed with PyArrow on Python
// 3.10), so callables are unpickled one at a time. Take it with `lock_py_attached`,
// which waits for the lock without holding the GIL.
static IMPORT_LOCK: Mutex<()> = Mutex::new(());
+pyo3::create_exception!(
+ comet,
+ TaskKilled,
+ pyo3::exceptions::PyBaseException,
+ "Raised in a running Arrow UDF when Spark kills its task."
+);
+
+unsafe extern "C" {
+ // Part of the stable Python C API, but not declared by pyo3-ffi.
+ fn PyThread_get_thread_ident() -> c_ulong;
+}
+
+/// The native plans of one Spark task attempt that hold Arrow UDFs.
+#[derive(Default)]
+struct TaskCalls {
+ plans: usize,
+ killed: bool,
+ /// The calls running now.
+ calls: Vec<Call>,
+}
+
+struct Call {
+ /// Python thread id of the thread running the call.
+ thread: c_ulong,
+ /// Whether [`interrupt_task`] has raised in the call. It raises once, so that a second
+ /// exception does not abort the cleanup the first one started.
+ interrupted: bool,
+}
+
+// Holders of this lock never wait for the GIL, so a thread attached to Python may take it.
+static TASKS: LazyLock<Mutex<HashMap<i64, TaskCalls>>> = LazyLock::new(Default::default);
+
+fn tasks() -> MutexGuard<'static, HashMap<i64, TaskCalls>> {
+ TASKS.lock().unwrap_or_else(PoisonError::into_inner)
+}
+
+/// Registers a native plan's Arrow UDFs under their Spark task attempt, so that
+/// [`interrupt_task`] reaches their calls. The task's entry is removed when its last
+/// registration is dropped, which happens when the plan is released.
+pub struct TaskRegistration {
+ task_attempt_id: i64,
+}
+
+impl TaskRegistration {
+ pub fn new(task_attempt_id: i64) -> Self {
+ tasks().entry(task_attempt_id).or_default().plans += 1;
+ Self { task_attempt_id }
+ }
+
+ fn is_killed(&self) -> bool {
+ tasks()
+ .get(&self.task_attempt_id)
+ .is_some_and(|task| task.killed)
+ }
+
+ fn killed_error(&self) -> ArrowError {
+ ArrowError::ComputeError(format!(
+ "Arrow UDF stopped because task {} was killed",
+ self.task_attempt_id
+ ))
+ }
+}
+
+impl Drop for TaskRegistration {
+ fn drop(&mut self) {
+ let mut tasks = tasks();
+ if let Some(task) = tasks.get_mut(&self.task_attempt_id) {
+ task.plans -= 1;
+ if task.plans == 0 {
+ tasks.remove(&self.task_attempt_id);
+ }
+ }
+ }
+}
+
+/// A call to the user's callable, recorded so that [`interrupt_task`] can raise in its thread.
+struct RunningCall<'a> {
+ task: &'a TaskRegistration,
+ thread: c_ulong,
+}
+
+impl<'a> RunningCall<'a> {
+ /// Fails without recording the call when the task has already been killed.
+ fn enter(_py: Python<'_>, task: &'a TaskRegistration) -> Result<Self> {
+ // SAFETY: the caller is attached to the interpreter.
+ let thread = unsafe { PyThread_get_thread_ident() };
+ let mut tasks = tasks();
+ let calls = tasks.entry(task.task_attempt_id).or_default();
+ if calls.killed {
+ return Err(task.killed_error());
+ }
+ calls.calls.push(Call {
+ thread,
+ interrupted: false,
+ });
+ Ok(Self { task, thread })
+ }
+}
+
+impl Drop for RunningCall<'_> {
+ fn drop(&mut self) {
+ let mut tasks = tasks();
+ let Some(calls) = tasks.get_mut(&self.task.task_attempt_id) else {
+ return;
+ };
+ let Some(index) = calls.calls.iter().position(|c| c.thread == self.thread) else {
+ return;
+ };
+ if calls.calls.swap_remove(index).interrupted {
+ // An interrupt that arrived as the callable returned is still pending on this
+ // thread. Nothing can set another one now that the call is unrecorded, so clear
+ // it rather than let it fire in later Python code on this thread.
+ // SAFETY: the call is made while attached, on the thread it names.
+ unsafe {
+ pyo3::ffi::PyThreadState_SetAsyncExc(self.thread as c_long, std::ptr::null_mut())
Review Comment:
[P2] Avoid unconditionally clearing an interrupt that Python already delivered. On CPython 3.11 and 3.12, `PyThreadState_SetAsyncExc(id, NULL)` re-arms the interpreter-wide pending flag even when no thread has an exception to consume. Other UDFs then repeatedly enter the pending-event path. After unregistering the call and releasing the mutex, consume any genuinely late interrupt on its originating thread before reusing it.
Evidence: For a surviving UDF summing three million integers, median thread CPU over 13 samples rose from 100.22 to 111.71 ms on 3.11.15 and from 134.66 to 154.77 ms on 3.12.13. Both reported a pending flag of `1` with zero target threads. A disposable Python checkpoint kept the flag at `0` and also passed the case where a C callable returns directly to Rust with an interrupt pending. [CPython setter](https://github.com/python/cpython/blob/v3.12.13/Python/pystate.c#L1832-L1838) · [Pending-event handler](https://github.com/python/cpython/blob/v3.12.13/Python/ceval_gil.c#L1065-L1073)
##########
native/core/src/execution/python_udf.rs:
##########
@@ -29,14 +29,177 @@ use pyo3::ffi::Py_uintptr_t;
use pyo3::prelude::*;
use pyo3::sync::MutexExt;
use pyo3::types::{PyBytes, PyList, PyTuple};
-use std::sync::{Mutex, PoisonError};
+use pyo3::PyTypeInfo;
+use std::collections::HashMap;
+use std::os::raw::{c_long, c_ulong};
+use std::sync::{Arc, LazyLock, Mutex, MutexGuard, PoisonError};
// Tasks create their callables concurrently. Concurrent first imports of a package
// can hand a thread a partially initialized module (observed with PyArrow on Python
// 3.10), so callables are unpickled one at a time. Take it with `lock_py_attached`,
// which waits for the lock without holding the GIL.
static IMPORT_LOCK: Mutex<()> = Mutex::new(());
+pyo3::create_exception!(
+ comet,
+ TaskKilled,
+ pyo3::exceptions::PyBaseException,
+ "Raised in a running Arrow UDF when Spark kills its task."
+);
+
+unsafe extern "C" {
+ // Part of the stable Python C API, but not declared by pyo3-ffi.
+ fn PyThread_get_thread_ident() -> c_ulong;
+}
+
+/// The native plans of one Spark task attempt that hold Arrow UDFs.
+#[derive(Default)]
+struct TaskCalls {
+ plans: usize,
+ killed: bool,
+ /// The calls running now.
+ calls: Vec<Call>,
+}
+
+struct Call {
+ /// Python thread id of the thread running the call.
+ thread: c_ulong,
+ /// Whether [`interrupt_task`] has raised in the call. It raises once, so that a second
+ /// exception does not abort the cleanup the first one started.
+ interrupted: bool,
+}
+
+// Holders of this lock never wait for the GIL, so a thread attached to Python may take it.
+static TASKS: LazyLock<Mutex<HashMap<i64, TaskCalls>>> = LazyLock::new(Default::default);
+
+fn tasks() -> MutexGuard<'static, HashMap<i64, TaskCalls>> {
+ TASKS.lock().unwrap_or_else(PoisonError::into_inner)
+}
+
+/// Registers a native plan's Arrow UDFs under their Spark task attempt, so that
+/// [`interrupt_task`] reaches their calls. The task's entry is removed when its last
+/// registration is dropped, which happens when the plan is released.
+pub struct TaskRegistration {
+ task_attempt_id: i64,
+}
+
+impl TaskRegistration {
+ pub fn new(task_attempt_id: i64) -> Self {
+ tasks().entry(task_attempt_id).or_default().plans += 1;
Review Comment:
[P2] Key cancellation by a process-unique token shared by the plans of one `TaskContext`. A cancelled local task can remain in native code after its `SparkContext` stops. If the replacement context reuses its attempt ID, this registration inherits the old `killed` flag and rejects a healthy UDF. Use the same token for `watchedTasks` and the native registry.
Evidence: Two sequential Spark 4.1.3 `local[1]` contexts both assigned attempt ID `0`. While the first context’s cancelled call was still waiting, the replacement’s identity UDF received input `73` and returned `Arrow UDF stopped because task 0 was killed` with `isInterrupted=false`. Attempt `1` returned `73` for the same UDF and input. [Spark attempt-ID contract](https://github.com/apache/spark/blob/v4.1.3/core/src/main/scala/org/apache/spark/TaskContext.scala#L231-L235)
##########
native/core/src/execution/python_udf.rs:
##########
@@ -297,14 +468,18 @@ impl ArrowPythonUdf {
kwargs.set_item(name, arg).map_err(python_error)?;
}
}
- let result = self
- .callable
- .bind(py)
- .call(
- PyTuple::new(py, positional).map_err(python_error)?,
- Some(&kwargs),
- )
- .map_err(|error| python_traceback_error(py, error))?;
+ let positional = PyTuple::new(py, positional).map_err(python_error)?;
+ let call = self
+ .task
+ .as_deref()
+ .map(|task| RunningCall::enter(py, task))
+ .transpose()?;
+ let result = self.callable.bind(py).call(positional, Some(&kwargs));
+ drop(call);
Review Comment:
[P2] Keep the call registered through user-defined result conversion and check cancellation before returning the converted batch. Spark 4.2 accepts array-like results whose `__arrow_array__` executes Python inside `RecordBatch.from_arrays`. Dropping `RunningCall` here makes that code invisible to the monitor. A cancelled conversion can keep running and ultimately return a successful batch.
Evidence: On CPython 3.11.15 and 3.12.13, two interrupts during a bounded `__arrow_array__` loop both returned `0`. After the loop was released, native evaluation returned `[73]`. Stock Spark 4.2 accepted the same result shape and stopped the worker during conversion on cancellation. [Spark result materialization](https://github.com/apache/spark/blob/v4.2.0/python/pyspark/worker.py#L2563-L2581)
##########
native/core/src/execution/python_udf.rs:
##########
@@ -297,14 +468,18 @@ impl ArrowPythonUdf {
kwargs.set_item(name, arg).map_err(python_error)?;
}
}
- let result = self
- .callable
- .bind(py)
- .call(
- PyTuple::new(py, positional).map_err(python_error)?,
- Some(&kwargs),
- )
- .map_err(|error| python_traceback_error(py, error))?;
+ let positional = PyTuple::new(py, positional).map_err(python_error)?;
+ let call = self
+ .task
+ .as_deref()
+ .map(|task| RunningCall::enter(py, task))
Review Comment:
[P2] Check the killed flag before `Python::attach`, and retain this check in `RunningCall::enter` for races. A batch submitted after the registration is already marked killed currently waits for the GIL and imports its arguments before it can fail. An unrelated UDF holding the GIL can therefore retain this cancelled task’s executor slot even though its callable will never run.
Evidence: A CPython 3.12.13 probe marked task `101` killed with no active calls, then evaluated its next batch while another UDF held the GIL in `math.factorial(420000)`. Returning the kill error took 1,481 ms. With a pre-attach check, the same evaluation returned in under 0.1 ms.
##########
native/core/src/execution/python_udf.rs:
##########
@@ -600,6 +775,122 @@ mod tests {
});
}
+ fn running_calls(task_attempt_id: i64) -> Option<usize> {
+ tasks().get(&task_attempt_id).map(|task| task.calls.len())
+ }
+
+ #[test]
+ fn interrupts_running_call_of_killed_task() {
+ const TASK: i64 = -6811;
+ initialize_python().unwrap();
+ // `except Exception` must not swallow the interrupt.
+ let command = Python::attach(|py| {
+ cloudpickled_lambda(
+ py,
+ c"(lambda ns: (exec('def f(a):\\n n = 0\\n while True:\\n n += 1\\n try:\\n n %= 7\\n except Exception:\\n pass\\n', ns), ns['f'])[1])({})",
+ )
+ });
+ let task = Arc::new(TaskRegistration::new(TASK));
+ let udf = Arc::new(
+ ArrowPythonUdf::from_command(&command, DataType::Int64, true, true, false, "")
+ .unwrap()
+ .with_task(Arc::clone(&task)),
+ );
+ let input: ArrayRef = Arc::new(Int64Array::from(vec![1]));
+ let (sender, receiver) = std::sync::mpsc::channel();
+ {
+ let (udf, input) = (Arc::clone(&udf), Arc::clone(&input));
+ // Not joined: a call that ignores the interrupt would never return.
+ std::thread::spawn(move || sender.send(udf.evaluate(&[input], 1)));
Review Comment:
[P2] Release the worker’s cloned `Arc<ArrowPythonUdf>` before sending the result, or join the worker after the timed receive succeeds. The receiver can wake while the detached worker still owns the registration. The final assertion then intermittently fails even though the Python call was interrupted and unregistered correctly.
Evidence: The unmodified `interrupts_running_call_of_killed_task` failed 5 of 24 isolated CPython 3.12.13 runs at line 825 with actual `Some(0)` and expected `None`. A control changing only the worker to drop its `Arc` before `send` passed all 24 runs.
##########
native/core/src/execution/python_udf.rs:
##########
@@ -29,14 +29,177 @@ use pyo3::ffi::Py_uintptr_t;
use pyo3::prelude::*;
use pyo3::sync::MutexExt;
use pyo3::types::{PyBytes, PyList, PyTuple};
-use std::sync::{Mutex, PoisonError};
+use pyo3::PyTypeInfo;
+use std::collections::HashMap;
+use std::os::raw::{c_long, c_ulong};
+use std::sync::{Arc, LazyLock, Mutex, MutexGuard, PoisonError};
// Tasks create their callables concurrently. Concurrent first imports of a package
// can hand a thread a partially initialized module (observed with PyArrow on Python
// 3.10), so callables are unpickled one at a time. Take it with `lock_py_attached`,
// which waits for the lock without holding the GIL.
static IMPORT_LOCK: Mutex<()> = Mutex::new(());
+pyo3::create_exception!(
+ comet,
+ TaskKilled,
+ pyo3::exceptions::PyBaseException,
+ "Raised in a running Arrow UDF when Spark kills its task."
+);
+
+unsafe extern "C" {
+ // Part of the stable Python C API, but not declared by pyo3-ffi.
+ fn PyThread_get_thread_ident() -> c_ulong;
+}
+
+/// The native plans of one Spark task attempt that hold Arrow UDFs.
+#[derive(Default)]
+struct TaskCalls {
+ plans: usize,
+ killed: bool,
+ /// The calls running now.
+ calls: Vec<Call>,
+}
+
+struct Call {
+ /// Python thread id of the thread running the call.
+ thread: c_ulong,
+ /// Whether [`interrupt_task`] has raised in the call. It raises once, so that a second
+ /// exception does not abort the cleanup the first one started.
+ interrupted: bool,
+}
+
+// Holders of this lock never wait for the GIL, so a thread attached to Python may take it.
+static TASKS: LazyLock<Mutex<HashMap<i64, TaskCalls>>> = LazyLock::new(Default::default);
+
+fn tasks() -> MutexGuard<'static, HashMap<i64, TaskCalls>> {
+ TASKS.lock().unwrap_or_else(PoisonError::into_inner)
+}
+
+/// Registers a native plan's Arrow UDFs under their Spark task attempt, so that
+/// [`interrupt_task`] reaches their calls. The task's entry is removed when its last
+/// registration is dropped, which happens when the plan is released.
+pub struct TaskRegistration {
+ task_attempt_id: i64,
+}
+
+impl TaskRegistration {
+ pub fn new(task_attempt_id: i64) -> Self {
+ tasks().entry(task_attempt_id).or_default().plans += 1;
+ Self { task_attempt_id }
+ }
+
+ fn is_killed(&self) -> bool {
+ tasks()
+ .get(&self.task_attempt_id)
+ .is_some_and(|task| task.killed)
+ }
+
+ fn killed_error(&self) -> ArrowError {
+ ArrowError::ComputeError(format!(
+ "Arrow UDF stopped because task {} was killed",
+ self.task_attempt_id
+ ))
+ }
+}
+
+impl Drop for TaskRegistration {
+ fn drop(&mut self) {
+ let mut tasks = tasks();
+ if let Some(task) = tasks.get_mut(&self.task_attempt_id) {
+ task.plans -= 1;
+ if task.plans == 0 {
+ tasks.remove(&self.task_attempt_id);
+ }
+ }
+ }
+}
+
+/// A call to the user's callable, recorded so that [`interrupt_task`] can raise in its thread.
+struct RunningCall<'a> {
+ task: &'a TaskRegistration,
+ thread: c_ulong,
+}
+
+impl<'a> RunningCall<'a> {
+ /// Fails without recording the call when the task has already been killed.
+ fn enter(_py: Python<'_>, task: &'a TaskRegistration) -> Result<Self> {
+ // SAFETY: the caller is attached to the interpreter.
+ let thread = unsafe { PyThread_get_thread_ident() };
+ let mut tasks = tasks();
+ let calls = tasks.entry(task.task_attempt_id).or_default();
+ if calls.killed {
+ return Err(task.killed_error());
+ }
+ calls.calls.push(Call {
+ thread,
+ interrupted: false,
+ });
+ Ok(Self { task, thread })
+ }
+}
+
+impl Drop for RunningCall<'_> {
+ fn drop(&mut self) {
+ let mut tasks = tasks();
+ let Some(calls) = tasks.get_mut(&self.task.task_attempt_id) else {
+ return;
+ };
+ let Some(index) = calls.calls.iter().position(|c| c.thread == self.thread) else {
+ return;
+ };
+ if calls.calls.swap_remove(index).interrupted {
+ // An interrupt that arrived as the callable returned is still pending on this
+ // thread. Nothing can set another one now that the call is unrecorded, so clear
+ // it rather than let it fire in later Python code on this thread.
+ // SAFETY: the call is made while attached, on the thread it names.
+ unsafe {
+ pyo3::ffi::PyThreadState_SetAsyncExc(self.thread as c_long, std::ptr::null_mut())
+ };
+ }
+ }
+}
+
+/// Marks a Spark task attempt as killed and raises [`TaskKilled`] in its running Arrow UDF
+/// calls that it has not interrupted yet. Later calls of the task fail without entering
+/// Python. Returns the number of calls interrupted. A task with no registered plan is not
+/// recorded, so the caller repeats this until the task completes.
+///
+/// The exception takes effect when the thread next runs Python bytecode: a call blocked in
+/// native code that released the GIL is interrupted only when that code returns.
+pub fn interrupt_task(task_attempt_id: i64) -> usize {
+ let running = match tasks().get_mut(&task_attempt_id) {
+ Some(task) => {
+ task.killed = true;
+ task.calls.iter().any(|call| !call.interrupted)
+ }
+ None => false,
+ };
+ if !running {
+ return 0;
+ }
+ // Calls cannot start after the flag is set, so only calls recorded before it remain.
+ Python::attach(|py| {
+ let exception = TaskKilled::type_object(py);
+ let mut tasks = tasks();
+ let Some(task) = tasks.get_mut(&task_attempt_id) else {
+ return 0;
+ };
+ let mut interrupted = 0;
+ for call in task.calls.iter_mut().filter(|call| !call.interrupted) {
+ // SAFETY: attached to the interpreter, and `exception` is a live type object.
+ let set = unsafe {
+ pyo3::ffi::PyThreadState_SetAsyncExc(call.thread as c_long, exception.as_ptr())
Review Comment:
[P2] A first `TaskKilled` can strand a shared Python lock. If a UDF is waiting in `logging.Handler.acquire()`, delivery can occur after it acquires the `RLock` and before `Handler.handle()` enters its `try/finally`. The call exits owning the lock, and healthy tasks using the same handler block. Use explicit cooperative cancellation points in the shared interpreter, or isolate calls that require forced interruption.
Evidence: On CPython 3.11.15 and 3.12.13, one interrupt made the cancelled UDF return `TaskKilled` while still owning the handler lock. A peer’s `logger.warning()` timed out after 300 ms and completed when the victim thread released the lock. With zero interrupts, all three UDFs completed. This happens during acquisition, before the cleanup covered by the earlier review. [CPython handler](https://github.com/python/cpython/blob/v3.12.13/Lib/logging/__init__.py#L1025-L1030)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] feat: interrupt a native scalar Arrow UDF when its task is killed [datafusion-comet]
Posted by "comphead (via GitHub)" <gi...@apache.org>.
comphead commented on code in PR #7181:
URL: https://github.com/apache/datafusion-comet/pull/7181#discussion_r4407318798
##########
spark/src/main/scala/org/apache/comet/PythonUdfMonitor.scala:
##########
@@ -0,0 +1,111 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.comet
+
+import java.util.concurrent.{ConcurrentHashMap, TimeUnit}
+import java.util.concurrent.atomic.{AtomicInteger, AtomicLong}
+
+import org.apache.spark.{SparkEnv, TaskContext}
+import org.apache.spark.internal.Logging
+
+/**
+ * Stops the native Python UDF calls of killed tasks. Spark's Python runner has a `MonitorThread`
+ * that destroys the worker of a killed task. A native UDF runs in the embedded interpreter on the
+ * task thread instead, where neither a Java interrupt nor a stopped input reaches it. So one
+ * daemon thread per task polls its kill state. Once the task is killed, its later calls fail
+ * without entering Python. A call still running `spark.python.task.killTimeout` after the kill
+ * gets an exception raised in it, once so that its cleanup can run, unless that is disabled.
+ */
+object PythonUdfMonitor extends Logging {
Review Comment:
#6295 is the same problem for every native plan: a killed task stays in `executePlan` until the plan yields its next batch, and rich7420 reported a prototype there with a native cancellation token and a JVM watcher. This PR adds most of what that needs, a per-task kill poller, a process-unique task token and a JNI call that binds the token to a plan before the first `executePlan`. All of it is named and gated on Python UDFs, though (`hasPythonUdf`, `setPythonUdfTask`, `killPythonUdfs`, `python_udf_task`). `CelebornNativeShuffleDestination.watchForCancellation` already polls the same signal for another consumer. Could the token and the watcher be named after the task, so #6295 can add plan cancellation as a second consumer instead of a second watcher and a second per-plan binding? If you would rather keep the scope, a pointer on #6295 to this mechanism would help whoever picks it up.
##########
spark/src/test/spark-4.1+/org/apache/spark/sql/comet/CometArrowPythonUdfSuite.scala:
##########
@@ -583,6 +590,101 @@ class CometArrowPythonUdfSuite extends CometTestBase {
}
}
+ /**
+ * Runs `job` in a job group, waits until `started` exists, cancels the group, and checks that
+ * the task ends as killed.
+ */
+ private def assertCancelKillsTask(started: File)(job: => Any): Unit = {
+ val killed = new CountDownLatch(1)
+ val listener = new SparkListener {
+ override def onTaskEnd(taskEnd: SparkListenerTaskEnd): Unit = taskEnd.reason match {
+ case _: TaskKilled => killed.countDown()
+ case _ =>
+ }
+ }
+ spark.sparkContext.addSparkListener(listener)
+ try {
+ val group = "comet-arrow-udf-kill"
+ val running = Future {
+ spark.sparkContext.setJobGroup(group, "native Arrow UDF loop", interruptOnCancel = true)
+ job
+ }(ExecutionContext.global)
+ val deadline = 60.seconds.fromNow
+ while (!started.exists()) {
+ assert(deadline.hasTimeLeft(), "the UDF did not start")
Review Comment:
If the UDF never creates `started`, this assertion throws before `cancelJobGroup` runs, and the looping UDF keeps its task alive for the rest of the suite, holding one of the `local[5]` cores. Could the `finally` cancel the group as well? The `Future` also leaves the job group and `interruptOnCancel` properties on an `ExecutionContext.global` thread for whatever runs there next, which `CometTaskMetricsSuite` avoids with `clearJobGroup()`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] feat: interrupt a native scalar Arrow UDF when its task is killed [datafusion-comet]
Posted by "viirya (via GitHub)" <gi...@apache.org>.
viirya commented on code in PR #7181:
URL: https://github.com/apache/datafusion-comet/pull/7181#discussion_r4365458903
##########
native/core/src/execution/python_udf.rs:
##########
@@ -29,14 +29,164 @@ use pyo3::ffi::Py_uintptr_t;
use pyo3::prelude::*;
use pyo3::sync::MutexExt;
use pyo3::types::{PyBytes, PyList, PyTuple};
-use std::sync::{Mutex, PoisonError};
+use pyo3::PyTypeInfo;
+use std::collections::HashMap;
+use std::os::raw::{c_long, c_ulong};
+use std::sync::{Arc, LazyLock, Mutex, MutexGuard, PoisonError};
// Tasks create their callables concurrently. Concurrent first imports of a package
// can hand a thread a partially initialized module (observed with PyArrow on Python
// 3.10), so callables are unpickled one at a time. Take it with `lock_py_attached`,
// which waits for the lock without holding the GIL.
static IMPORT_LOCK: Mutex<()> = Mutex::new(());
+pyo3::create_exception!(
+ comet,
+ TaskKilled,
+ pyo3::exceptions::PyBaseException,
+ "Raised in a running Arrow UDF when Spark kills its task."
+);
+
+unsafe extern "C" {
+ // Part of the stable Python C API, but not declared by pyo3-ffi.
+ fn PyThread_get_thread_ident() -> c_ulong;
+}
+
+/// The native plans of one Spark task attempt that hold Arrow UDFs.
+#[derive(Default)]
+struct TaskCalls {
+ plans: usize,
+ killed: bool,
+ /// Python thread ids of the calls running now, one entry per call.
+ threads: Vec<c_ulong>,
+}
+
+// Holders of this lock never wait for the GIL, so a thread attached to Python may take it.
+static TASKS: LazyLock<Mutex<HashMap<i64, TaskCalls>>> = LazyLock::new(Default::default);
+
+fn tasks() -> MutexGuard<'static, HashMap<i64, TaskCalls>> {
+ TASKS.lock().unwrap_or_else(PoisonError::into_inner)
+}
+
+/// Registers a native plan's Arrow UDFs under their Spark task attempt, so that
+/// [`interrupt_task`] reaches their calls. The task's entry is removed when its last
+/// registration is dropped, which happens when the plan is released.
+pub struct TaskRegistration {
+ task_attempt_id: i64,
+}
+
+impl TaskRegistration {
+ pub fn new(task_attempt_id: i64) -> Self {
+ tasks().entry(task_attempt_id).or_default().plans += 1;
+ Self { task_attempt_id }
+ }
+
+ fn is_killed(&self) -> bool {
+ tasks()
+ .get(&self.task_attempt_id)
+ .is_some_and(|task| task.killed)
+ }
+
+ fn killed_error(&self) -> ArrowError {
+ ArrowError::ComputeError(format!(
+ "Arrow UDF stopped because task {} was killed",
+ self.task_attempt_id
+ ))
+ }
+}
+
+impl Drop for TaskRegistration {
+ fn drop(&mut self) {
+ let mut tasks = tasks();
+ if let Some(task) = tasks.get_mut(&self.task_attempt_id) {
+ task.plans -= 1;
+ if task.plans == 0 {
+ tasks.remove(&self.task_attempt_id);
+ }
+ }
+ }
+}
+
+/// A call to the user's callable, recorded so that [`interrupt_task`] can raise in its thread.
+struct RunningCall<'a> {
+ task: &'a TaskRegistration,
+ thread: c_ulong,
+}
+
+impl<'a> RunningCall<'a> {
+ /// Fails without recording the call when the task has already been killed.
+ fn enter(_py: Python<'_>, task: &'a TaskRegistration) -> Result<Self> {
+ // SAFETY: the caller is attached to the interpreter.
+ let thread = unsafe { PyThread_get_thread_ident() };
+ let mut tasks = tasks();
+ let calls = tasks.entry(task.task_attempt_id).or_default();
+ if calls.killed {
+ return Err(task.killed_error());
+ }
+ calls.threads.push(thread);
+ Ok(Self { task, thread })
+ }
+}
+
+impl Drop for RunningCall<'_> {
+ fn drop(&mut self) {
+ let mut tasks = tasks();
+ let Some(calls) = tasks.get_mut(&self.task.task_attempt_id) else {
+ return;
+ };
+ if let Some(index) = calls.threads.iter().position(|t| *t == self.thread) {
+ calls.threads.swap_remove(index);
+ }
+ if calls.killed {
+ // An interrupt that arrived as the callable returned is still pending on this
+ // thread. Nothing can set another one now that the call is unrecorded, so clear
+ // it rather than let it fire in later Python code on this thread.
+ // SAFETY: the call is made while attached, on the thread it names.
+ unsafe {
+ pyo3::ffi::PyThreadState_SetAsyncExc(self.thread as c_long, std::ptr::null_mut())
+ };
+ }
+ }
+}
+
+/// Marks a Spark task attempt as killed and raises [`TaskKilled`] in its running Arrow UDF
+/// calls. Later calls of the task fail without entering Python. Returns the number of calls
+/// interrupted. A task with no registered plan is not recorded, so the caller repeats this
+/// until the task completes.
+///
+/// The exception takes effect when the thread next runs Python bytecode: a call blocked in
+/// native code that released the GIL is interrupted only when that code returns.
+pub fn interrupt_task(task_attempt_id: i64) -> usize {
+ let running = match tasks().get_mut(&task_attempt_id) {
+ Some(task) => {
+ task.killed = true;
+ !task.threads.is_empty()
+ }
+ None => false,
+ };
+ if !running {
+ return 0;
+ }
+ // Calls cannot start after the flag is set, so only calls recorded before it remain.
+ Python::attach(|py| {
+ let exception = TaskKilled::type_object(py);
+ let tasks = tasks();
+ let Some(task) = tasks.get(&task_attempt_id) else {
+ return 0;
+ };
+ task.threads
+ .iter()
+ .map(|thread| {
+ // SAFETY: attached to the interpreter, and `exception` is a live type object.
+ let set = unsafe {
+ pyo3::ffi::PyThreadState_SetAsyncExc(*thread as c_long, exception.as_ptr())
Review Comment:
Good catch, thanks. I reproduced it on Python 3.10 and 3.13 with a lock released in a `finally` after a 0.4 s sleep: a second injection during the cleanup leaves the lock held. Fixed in 89da483c1. Each recorded call now carries an `interrupted` flag, and `interrupt_task` raises only in calls it has not interrupted yet, so a running call gets `TaskKilled` exactly once. The killed flag is unchanged, so later calls of the task still fail without entering Python. The monitor keeps polling, because a task killed before its plan registers is not recorded, and a plan the task registers later still needs the flag set.
The trade-off is that a call that catches `TaskKilled` (e.g. `except BaseException`) and keeps running is no longer interrupted again. The user guide and the `PythonUdfMonitor` comment now say so. The new `interrupts_a_call_once_so_its_cleanup_runs` test interrupts a call, waits until its `finally` cleanup has started, checks that a second `interrupt_task` raises in nothing, and checks that the cleanup released its lock. It fails with the previous repeated delivery.
##########
spark/src/main/scala/org/apache/comet/PythonUdfMonitor.scala:
##########
@@ -0,0 +1,69 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.comet
+
+import java.util.concurrent.ConcurrentHashMap
+
+import org.apache.spark.TaskContext
+import org.apache.spark.internal.Logging
+
+/**
+ * Interrupts the native Python UDF calls of killed tasks. Spark's Python runner has a
+ * `MonitorThread` that destroys the worker of a killed task. A native UDF runs in the embedded
+ * interpreter on the task thread instead, where neither a Java interrupt nor a stopped input
+ * reaches it. So one daemon thread per task polls its kill state and, while the task is killed
+ * but still running, raises an exception in the task's running Python calls. Calls that start
+ * after the first interrupt fail without entering Python.
+ */
+object PythonUdfMonitor extends Logging {
+
+ // Raising the exception is cheap and lets the call unwind on its own, so unlike Spark's
+ // monitor this one does not wait `spark.python.task.killTimeout` before acting. Repeating it
+ // reaches calls that start after an earlier attempt and calls that caught the exception.
+ private val PollIntervalMs = 100L
+
+ private val watchedTasks = ConcurrentHashMap.newKeySet[java.lang.Long]()
+
+ /** Whether the native library can run Python UDFs, and so may need its calls interrupted. */
+ lazy val supported: Boolean = NativeBase.supportsPythonUdf()
+
+ /** Starts watching the task unless it is watched already. The watch ends with the task. */
+ def watch(context: TaskContext): Unit = {
+ val taskAttemptId = context.taskAttemptId()
+ if (watchedTasks.add(taskAttemptId)) {
+ val thread = new Thread(s"Comet Python UDF monitor for task $taskAttemptId") {
+ override def run(): Unit = {
+ try {
+ while (!context.isCompleted()) {
+ if (context.isInterrupted() && NativeBase.interruptPythonUdfs(taskAttemptId) > 0) {
+ logInfo(s"Interrupted Python UDF calls of killed task $taskAttemptId")
+ }
+ Thread.sleep(PollIntervalMs)
+ }
+ } finally {
+ watchedTasks.remove(taskAttemptId)
Review Comment:
Fixed in 89da483c1 with `val _ = watchedTasks.remove(taskAttemptId)`, matching `CometExecIterator`. `test-compile -Pspark-3.5,delta -Pstrict-warnings` now passes. The four `@nowarn` errors in the shuffle managers came from the same failed compile and are gone with it.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] feat: interrupt a native scalar Arrow UDF when its task is killed [datafusion-comet]
Posted by "viirya (via GitHub)" <gi...@apache.org>.
viirya commented on code in PR #7181:
URL: https://github.com/apache/datafusion-comet/pull/7181#discussion_r4391838906
##########
native/core/src/execution/python_udf.rs:
##########
@@ -297,14 +468,18 @@ impl ArrowPythonUdf {
kwargs.set_item(name, arg).map_err(python_error)?;
}
}
- let result = self
- .callable
- .bind(py)
- .call(
- PyTuple::new(py, positional).map_err(python_error)?,
- Some(&kwargs),
- )
- .map_err(|error| python_traceback_error(py, error))?;
+ let positional = PyTuple::new(py, positional).map_err(python_error)?;
+ let call = self
+ .task
+ .as_deref()
+ .map(|task| RunningCall::enter(py, task))
+ .transpose()?;
+ let result = self.callable.bind(py).call(positional, Some(&kwargs));
+ drop(call);
Review Comment:
Good catch, thanks. An extension type's `__arrow_ext_class__` can return an `ExtensionArray` subclass that overrides `cast`, and the same goes for `__len__`, the type comparison, and the type's `__str__` in the mismatch error. All of them ran after the call was unrecorded and after the kill check. Fixed in 69697372f. Everything from the callable to `_export_to_c` now runs in `call_and_export` while the call is still recorded, so a kill interrupts any user code in that span. The call is unrecorded only after the export, on success and on error alike, and the kill check follows it. `interrupts_a_user_defined_result_cast` returns an extension array whose `cast` loops, interrupts it, and checks that the call fails with `TaskKilled`. It fails when the call is unrecorded right after the callable.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org