You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@flink.apache.org by "eemario (via GitHub)" <gi...@apache.org> on 2026/09/20 07:10:32 UTC
[PR] [runtime] Support parallel action execution on JDK < 21 [flink-agents]
eemario opened a new pull request, #1147:
URL: https://github.com/apache/flink-agents/pull/1147
Execute plans whose actions are implemented in Java on coordinator workers when continuation-based execution is unavailable. Shared operator state remains protected by the parallel execution lock, while blocking asynchronous calls release the lock and restore task context after worker re-acquisition.
Runtime flow:
The mailbox thread admits each input record under the in-flight budget and appends its action tasks to the per-key queue. The coordinator grants workers the parallel execution lock in ascending (recordIndex, taskIndex) order, with absolute priority for the mailbox thread. A worker pulls its task from keyed state, prepares and runs the action; a blocking asynchronous call releases the lock before awaiting and restores the task context after re-acquisition. Commits happen on the mailbox thread, so keyed state is only ever mutated under the lock.
Key decisions:
- Worker-pool engine on JDK < 21, where continuation-based execution is unavailable; engine eligibility is validated at setup and ineligible plans fall back to the previous engine.
- Drain in-flight action tasks at prepareSnapshotPreBarrier instead of snapshotting them: pulled-but-uncommitted results live only in memory and would be lost on restore, so the barrier waits until the engine is quiesced and the snapshot captures no half-pulled intermediate state. Dispatch resumes only after the synchronous snapshot capture, because a worker woken any earlier could pull a queued task out of keyed state before it is captured. Restore semantics are therefore identical to the serial engine.
- Mailbox-priority lock, so checkpoint and commit work on the mailbox thread is never starved by workers.
- Bound concurrent input records with in-flight backpressure instead of unbounded admission.
- Align lifecycle notifications with the serial path: prepared/started/reused fire in execute(), finishing/finished/ failed fire in commit(), with noop-silent guards.
- Register the durable-execution context in the lookup map on setup so workers resolve it during replay.
Interaction decisions (engine selection):
Plan parallel-execution.enabled Engine
pure-Java, JDK < 21 true (default) parallel
pure-Java, JDK < 21 false serial
coroutine engine available any (ignored) coroutine
contains Python actions any (ignored) serial
Behavioral contracts:
- Same-key input records commit in input order.
- Fan-out siblings, broadcast listeners and multi-key inputs run concurrently; each action observes its own action-scoped context.
- At most max-in-flight-input-records (default 100) records are in flight; admission then blocks until an in-flight record retires.
- A checkpoint contains no in-flight execution state; after restore the engine behaves exactly as the serial path and exactly-once output is preserved.
- A failing action task fails the job on the mailbox thread without hanging.
- Task lifecycle notifications are indistinguishable from the serial path.
Failure behavior:
- Pull/prepare/run failures on a worker are captured and rethrown on the mailbox thread; the lock is always returned and no pool thread dies silently.
- A failed prepareSnapshotPreBarrier rolls the drain back via stopDraining and resumes dispatch, because such a failure never reaches snapshotState's stopDraining.
- Aborting an old checkpoint does not resume a drain belonging to the current checkpoint.
- Unsupported configurations (Python actions, coroutine engine) are not errors: the parallel options are ignored and execution falls back to the previous engine.
Compatibility impact:
- Adds AgentExecutionOptions.PARALLEL_EXECUTION_ENABLED (default true) and MAX_IN_FLIGHT_INPUT_RECORDS (default 100) to the Java and Python APIs; no existing option changes meaning.
- Pure-Java agents on JDK < 21 switch from serial to parallel execution by default; per-key ordering and exactly-once semantics are unchanged, and parallel-execution.enabled=false restores the previous behavior.
- JDK 21 coroutine execution and plans containing Python actions are unaffected.
Contracts to tests:
- Same-key order: ParallelExecutionE2ETest.sameKeyInputsCommitInOrder; ParallelExecutionTaskQueueTest
- Concurrent fan-out/broadcast/multi-key and action-scoped context: ParallelExecutionE2ETest.fanoutRunsSiblingActionsInParallel, broadcastRunsAllListenersInParallel, multiKeyInputsRunInParallel, parallelSiblingsKeepActionScopedContext; ActionTaskContextManagerTest
- In-flight bound: ParallelExecutionE2ETest. maxInFlightInputRecordsThrottlesAdmission; ParallelExecutionCoordinatorTest
- Checkpoint consistency and recovery: ActionExecutionOperatorDrainDeadlockIntegrationTest; ActionExecutionOperatorCapacityIntegrationTest. actionsBeyondWorkerCountRecoverFromCheckpointWithExactlyOnceOutput; ParallelExecutionLockTest
- Failure propagation: ParallelExecutionE2ETest. asyncFailureFailsTheJob, failingSiblingFailsTheJobWithoutHanging
- Lifecycle notifications: TaskLifecycleListenerNotificationTest; ActionExecutionOperatorTest
- Lock handoff on blocking async calls: ContinuationActionExecutorTest; JavaRunnerContextImplDurableExecuteAsyncTest
- Beyond-worker capacity: ParallelExecutionE2ETest. tasksBeyondWorkerCountCompleteExactlyOnce; ActionExecutionOperatorCapacityIntegrationTest
Not verified: plans containing Python actions (the engine never applies by design); JobManager failover while a drain is in progress.
Generated-by: Qoder
<!--
* Thank you very much for contributing to Flink Agents.
* Please add the relevant components in the PR title. E.g., [api], [runtime], [java], [python], [hotfix], etc.
-->
<!-- Please link the PR to the relevant issue(s). Hotfix doesn't need this. -->
Linked issue: #1119
### Purpose of change
<!-- What is the purpose of this change? -->
### Behavioral Semantics
<!-- For a non-trivial code change whose implementation is largely AI-assisted: interaction decisions, behavioral contracts, and failure behavior. See `contribution-guides/ai-assisted-pr.md`. Remove this heading and this comment otherwise. -->
### Tests
<!-- How is this change verified? -->
### API
<!-- Does this change touches any public APIs? -->
### Documentation
<!-- Do not remove this section. Check the proper box only. -->
- [ ] `doc-needed` <!-- Your PR changes impact docs -->
- [ ] `doc-not-needed` <!-- Your PR changes do not impact docs -->
- [ ] `doc-included` <!-- Your PR already contains the necessary documentation updates -->
### Was this patch authored or co-authored using generative AI tooling?
<!-- Do not remove this section. Check the proper box only. -->
- [ ] Yes
- [ ] No
If yes, include a `Generated-by: <tool name and version> (<model name and version>)` line, for example `Generated-by: Claude Code 2.1.226 (Claude Opus 4.6)`, in the commit message so it reaches Git history. Repeat the same line here for reviewer visibility. See the [ASF generative tooling guidance](https://www.apache.org/legal/generative-tooling.html).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [api][runtime] Support parallel action execution on JDK < 21 [flink-agents]
Posted by "wenjin272 (via GitHub)" <gi...@apache.org>.
wenjin272 commented on code in PR #1147:
URL: https://github.com/apache/flink-agents/pull/1147#discussion_r4096402406
##########
runtime/src/main/java/org/apache/flink/agents/runtime/operator/ActionExecutionOperator.java:
##########
@@ -236,10 +347,23 @@ public void open() throws Exception {
getContainingTask().getEnvironment().getTaskManagerInfo().getTmpDirectories(),
getRuntimeContext().getJobInfo().getJobId(),
metricGroup,
- this::checkMailboxThread,
+ parallelExecutionWithoutCoroutineEnabled
+ ? parallelExecutionLock::checkReentrant
+ : this::checkMailboxThread,
jobIdentifier,
getRuntimeContext().getUserCodeClassLoader());
+ if (parallelExecutionWithoutCoroutineEnabled) {
+ executionCoordinator =
+ new ParallelExecutionCoordinator(
+ parallelExecutionLock,
+ mailboxExecutor::execute,
+ Work::new,
+ pythonBridge::releaseCurrentThreadInterpreter,
Review Comment:
Thanks for addressing the earlier findings! The managed-worker cleanup introduces a resource-lifetime issue.
On the parallel path, `ctx.getResource()` checks lock ownership rather than mailbox-thread identity, so a Python resource can be lazily created on an action worker. Its cached `PyObject` retains that worker interpreter's native state.
During operator shutdown, the coordinator closes first, and worker cleanup closes the interpreter. `ResourceCache.close()` then closes the cached resource, whose `PyObject.close()` accesses the already-freed native state.
I reproduced this with the actual coordinator, resource cache, adapters, and native Pemja: the current shutdown order crashes in `PyObject.decRef → PyEval_AcquireThread`. Closing the cache while the worker interpreter is still alive succeeds, even from the mailbox thread, so this is a lifetime issue rather than cross-thread access alone.
Could we ensure cached resource handles cannot outlive their interpreter state? Reordering shutdown alone would not cover surplus workers retiring after the idle timeout. The current mocked tests do not exercise cached resource cleanup after interpreter teardown.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [api][runtime] Support parallel action execution on JDK < 21 [flink-agents]
Posted by "eemario (via GitHub)" <gi...@apache.org>.
eemario commented on code in PR #1147:
URL: https://github.com/apache/flink-agents/pull/1147#discussion_r4121626562
##########
runtime/src/main/java/org/apache/flink/agents/runtime/operator/ActionExecutionOperator.java:
##########
@@ -236,10 +347,23 @@ public void open() throws Exception {
getContainingTask().getEnvironment().getTaskManagerInfo().getTmpDirectories(),
getRuntimeContext().getJobInfo().getJobId(),
metricGroup,
- this::checkMailboxThread,
+ parallelExecutionWithoutCoroutineEnabled
+ ? parallelExecutionLock::checkReentrant
+ : this::checkMailboxThread,
jobIdentifier,
getRuntimeContext().getUserCodeClassLoader());
+ if (parallelExecutionWithoutCoroutineEnabled) {
+ executionCoordinator =
+ new ParallelExecutionCoordinator(
+ parallelExecutionLock,
+ mailboxExecutor::execute,
+ Work::new,
+ pythonBridge::releaseCurrentThreadInterpreter,
Review Comment:
Thanks for the detailed reproduction!
I fixed this by creating cached resource `PyObject` handles through the owner interpreter, even when initialization is triggered by a managed worker. Worker access is serialized by the parallel execution lock, while resource opening remains on the worker to preserve reentrant cache lookup.
This decouples cached resource lifetime from worker lifetime: idle worker retirement can safely close its interpreter, and cached handles remain valid until `ResourceCache` closes before the owner interpreter.
I also added a native Pemja E2E test covering resource creation, worker retirement, reuse by another worker, and final cleanup. Reverting to worker-interpreter creation reproduces the original native crash.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [api][runtime] Support parallel action execution on JDK < 21 [flink-agents]
Posted by "wenjin272 (via GitHub)" <gi...@apache.org>.
wenjin272 commented on code in PR #1147:
URL: https://github.com/apache/flink-agents/pull/1147#discussion_r4068158267
##########
runtime/src/main/java/org/apache/flink/agents/runtime/operator/parallel/ParallelExecutionCoordinator.java:
##########
@@ -0,0 +1,299 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.flink.agents.runtime.operator.parallel;
+
+import org.apache.flink.annotation.VisibleForTesting;
+import org.apache.flink.util.function.ThrowingRunnable;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.util.concurrent.LinkedBlockingQueue;
+import java.util.concurrent.ThreadPoolExecutor;
+import java.util.concurrent.TimeUnit;
+import java.util.concurrent.atomic.AtomicInteger;
+import java.util.function.Supplier;
+
+import static org.apache.flink.util.Preconditions.checkNotNull;
+
+/**
+ * Coordinates worker execution of {@link ParallelExecutionTask}: callers {@link #addTask(Object)
+ * add} one node per queued task, and the coordinator submits one anonymous <em>dispatch permit</em>
+ * per node to an elastic {@link ThreadPoolExecutor}. A pool thread redeems a permit under the lock
+ * (node poll and FIFO task pull share the lock hold, keeping their binding correct — permits never
+ * name a task); commits always run on the mailbox thread in per-key taskIndex order.
+ */
+public final class ParallelExecutionCoordinator implements AutoCloseable {
+
+ private static final Logger LOG = LoggerFactory.getLogger(ParallelExecutionCoordinator.class);
+
+ private static final long WORKER_TERMINATION_TIMEOUT_SECONDS = 180L;
+
+ @FunctionalInterface
+ public interface MailboxDispatcher {
+ void execute(ThrowingRunnable<? extends Exception> command, String description);
+ }
+
+ private final ParallelExecutionTaskQueue schedule;
+ private final ParallelExecutionLock parallelExecutionLock;
+ private final ThreadPoolExecutor workerExecutor;
+ private final MailboxDispatcher mailboxDispatcher;
+
+ /** Supplies a fresh work; the work pulls and prepares its own task inside {@code execute()}. */
+ private final Supplier<ParallelExecutionTask> workFactory;
+
+ /** Tasks pulled but not yet committed; read by {@link #isQuiesced()} for checkpoint quiesce. */
+ private final AtomicInteger inFlightExecuting = new AtomicInteger();
+
+ /**
+ * When {@code true}, workers finish in-flight tasks but pull no new nodes. Set before a
+ * checkpoint, cleared once the snapshot is taken.
+ */
+ private volatile boolean draining;
+
+ private final Object drainMonitor = new Object();
+
+ /**
+ * Permits submitted and not yet fully redeemed; compared against the pool size by {@link
+ * DispatchQueue#offer} to decide when the pool must grow.
+ */
+ private final AtomicInteger submittedPermits = new AtomicInteger();
+
+ /**
+ * Creates a coordinator with an elastic worker pool: one resident thread, on-demand growth up
+ * to {@code maxWorkers}, and idle retirement after {@code workerIdleTimeoutMillis}. Admission
+ * is not gated here; the operator bounds outstanding work via input-record backpressure.
+ */
+ public ParallelExecutionCoordinator(
+ ParallelExecutionLock parallelExecutionLock,
+ MailboxDispatcher mailboxDispatcher,
+ Supplier<ParallelExecutionTask> workFactory,
+ int maxWorkers,
+ long workerIdleTimeoutMillis) {
+ this.parallelExecutionLock = checkNotNull(parallelExecutionLock);
+ this.schedule = new ParallelExecutionTaskQueue(parallelExecutionLock);
+ this.mailboxDispatcher = checkNotNull(mailboxDispatcher);
+ this.workFactory = checkNotNull(workFactory);
+ DispatchQueue dispatchQueue = new DispatchQueue();
+ this.workerExecutor =
+ new ThreadPoolExecutor(
Review Comment:
These workers need to participate in the existing managed-worker interpreter lifecycle from #1092. With the default thread factory, `PythonInterpreterManager` treats them as unmanaged callers and forwards Python invocations to a separate callback worker. This introduces two regressions for Java actions using Python resources:
- **Cross-thread Python object access:** `PythonVectorStore` receives `PyObject`s created on the callback worker, but reads their attributes and closes them on the original action worker.
- **Resource-initialization deadlock:** the action worker holds the `ResourceCache.getResource()` monitor while initializing a Python chat model setup. It waits for the callback worker to run Python `open()`. That `open()` resolves the model connection through the same resource cache, so the callback worker waits for the monitor held by the action worker. Neither can proceed. Previously, this lookup ran on the same thread and could re-enter the monitor.
I reproduced both with the production Java components and mocked native interpreter calls. Could we integrate the new pool with managed-worker interpreter ownership and cleanup, and add regression coverage for these paths?
##########
runtime/src/main/java/org/apache/flink/agents/runtime/operator/ActionExecutionOperator.java:
##########
@@ -1049,17 +1398,256 @@ private void tryResumeProcessActionTasks() throws Exception {
});
long[] pendingInputEvents = {0L};
+ // Recovered pending input records will each be dequeued, processed, and decremented when
+ // they finish, so they must occupy in-flight budget too — count one unit per pending event.
stateManager.forEachPendingInputEventKey(
getKeyedStateBackend(),
- (key, state) -> {
- for (Event ignored : state.get()) {
- eventRouter.getKeySegmentQueue().addKeyToLastSegment(key);
- pendingInputEvents[0]++;
- }
- });
+ (key, state) ->
+ state.get()
+ .forEach(
+ event -> {
+ inFlightInputRecords++;
+ pendingInputEvents[0]++;
+ eventRouter
+ .getKeySegmentQueue()
+ .addKeyToLastSegment(key);
+ }));
builtInMetrics.restorePendingInputEvents(pendingInputEvents[0]);
}
+ private void runWithParallelExecutionLock(ThrowingRunnable<? extends Exception> action)
+ throws Exception {
+ boolean acquired = false;
+ try {
+ parallelExecutionLock.acquireByMain();
+ acquired = true;
+ action.run();
+ } finally {
+ if (acquired) {
+ parallelExecutionLock.release();
+ }
+ }
+ }
+
+ /**
+ * A single action task's full worker lifecycle on the parallel engine: a pool thread {@code
+ * setup}s and {@code execute}s it under the lock; the mailbox thread later restores and commits
+ * it. All task-specific processing lives here; the operator keeps only mailbox flow.
+ */
+ private final class Work implements ParallelExecutionTask {
+ private Object key;
+ private long recordIndex;
+ private long taskIndex;
+ private String contextKey;
+
+ private ActionTask actionTask;
+ private long sequenceNumber;
+ private boolean replayCompletedAction;
+
+ @Nullable private ActionTask.ActionTaskResult result;
+ @Nullable private Throwable failure;
+
+ @Override
+ public void setup(Object key, long recordIndex, long taskIndex) {
+ this.key = key;
+ this.recordIndex = recordIndex;
+ this.taskIndex = taskIndex;
+ this.contextKey = resolveContextKey(key);
+ }
+
+ @Override
+ public void execute() {
+ try {
+ // Pull + prepare under the lock hold: pairing the FIFO poll with the under-lock
+ // taskIndex keeps the (taskIndex, task) binding correct across concurrent workers.
+ // createAndSetRunnerContext already binds the context on this thread.
+ setCurrentKey(key);
+ ActionTask task = stateManager.pollNextActionTask();
Review Comment:
This direct call to `stateManager.pollNextActionTask()` bypasses the operator's existing dequeue helper, which calls `builtInMetrics.markActionTaskDequeued()`.
Enqueueing an action still stores its execution ID and timestamp in `initialTaskEnqueueNanos`. The skipped dequeue notification is the only place that removes this entry; action completion does not clean it up. Consequently, the parallel path retains an entry for every completed action indefinitely, causing unbounded memory growth as well as an incorrect pending-action gauge.
Using the actual operator harness, 10 fully completed input records producing 20 actions left **20 retained entries and a pending count of 20** in parallel mode; serial mode left **zero**.
Could we preserve the existing dequeue bookkeeping here, accounting for internal noop tasks? The direct `pollNextPendingInputEvent()` call in `maybeFinishCurrentInput()` also skips the corresponding pending-input metric decrement.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [api][runtime] Support parallel action execution on JDK < 21 [flink-agents]
Posted by "wenjin272 (via GitHub)" <gi...@apache.org>.
wenjin272 merged PR #1147:
URL: https://github.com/apache/flink-agents/pull/1147
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [api][runtime] Support parallel action execution on JDK < 21 [flink-agents]
Posted by "eemario (via GitHub)" <gi...@apache.org>.
eemario commented on PR #1147:
URL: https://github.com/apache/flink-agents/pull/1147#issuecomment-5748752573
Thanks @pltbkd and @Ryan-Nightwish - this builds on your earlier work. PTAL when you get a chance!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org