You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@flink.apache.org by "Bowen Li (Jira)" <ji...@apache.org> on 2026/09/10 17:00:00 UTC

[jira] [Created] (FLINK-40628) PyFlink TTL initialization can hang task threads and leave jobs stuck in INITIALIZING

Bowen Li created FLINK-40628:
--------------------------------

             Summary: PyFlink TTL initialization can hang task threads and leave jobs stuck in INITIALIZING
                 Key: FLINK-40628
                 URL: https://issues.apache.org/jira/browse/FLINK-40628
             Project: Flink
          Issue Type: Bug
          Components: API / Python
    Affects Versions: 1.19.3
            Reporter: Bowen Li
            Assignee: Bowen Li
             Fix For: 2.4.0


*User impact*

Python operators can remain stuck in INITIALIZING, preventing the job from processing data and completing checkpoints. The TaskManager process can remain alive while its task threads are blocked, so process-liveness checks may not detect the failure. Restarting the application can recover it, but the problem can recur.

*Cause*

With default TTL background cleanup, embedded state initialization follows:

RuntimeContext.get_state()
  → default RocksDB cleanup strategy
  → Duration.of_days(30)
  → get_gateway()
  → subprocess.Popen(preexec_fn=...)

This unexpectedly launches a separate gateway process from the multithreaded TaskManager. The child process can deadlock before exec, leaving the calling task thread waiting and the job unable to finish initialization. Python documents this deadlock risk.

*Verification*

The unexpected gateway request was reproduced through a real JVM/Pemja/RocksDB job using a Flink 1.19.1-derived distribution, Python 3.12.9, and Pemja 0.5.5. The same code path remains in upstream 1.19.3, 1.20.5, 2.3.0, and inspected master.
The production initialization hang was observed; its exact blocked lock was not captured.

*Proposed fix*

Avoid constructing a Py4J-backed Duration during embedded TTL initialization. Keep the default duration as a Python value and convert it through Pemja. Add a regression test ensuring RuntimeContext.get_state() with default TTL never requests a gateway.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)