You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@flink.apache.org by "Bowen Li (Jira)" <ji...@apache.org> on 2026/09/10 17:00:00 UTC
[jira] [Created] (FLINK-40628) PyFlink TTL initialization can hang task threads and leave jobs stuck in INITIALIZING
Bowen Li created FLINK-40628:
--------------------------------
Summary: PyFlink TTL initialization can hang task threads and leave jobs stuck in INITIALIZING
Key: FLINK-40628
URL: https://issues.apache.org/jira/browse/FLINK-40628
Project: Flink
Issue Type: Bug
Components: API / Python
Affects Versions: 1.19.3
Reporter: Bowen Li
Assignee: Bowen Li
Fix For: 2.4.0
*User impact*
Python operators can remain stuck in INITIALIZING, preventing the job from processing data and completing checkpoints. The TaskManager process can remain alive while its task threads are blocked, so process-liveness checks may not detect the failure. Restarting the application can recover it, but the problem can recur.
*Cause*
With default TTL background cleanup, embedded state initialization follows:
RuntimeContext.get_state()
→ default RocksDB cleanup strategy
→ Duration.of_days(30)
→ get_gateway()
→ subprocess.Popen(preexec_fn=...)
This unexpectedly launches a separate gateway process from the multithreaded TaskManager. The child process can deadlock before exec, leaving the calling task thread waiting and the job unable to finish initialization. Python documents this deadlock risk.
*Verification*
The unexpected gateway request was reproduced through a real JVM/Pemja/RocksDB job using a Flink 1.19.1-derived distribution, Python 3.12.9, and Pemja 0.5.5. The same code path remains in upstream 1.19.3, 1.20.5, 2.3.0, and inspected master.
The production initialization hang was observed; its exact blocked lock was not captured.
*Proposed fix*
Avoid constructing a Py4J-backed Duration during embedded TTL initialization. Keep the default duration as a Python value and convert it through Pemja. Add a regression test ensuring RuntimeContext.get_state() with default TTL never requests a gateway.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)