You are viewing a plain text version of this content. The canonical link for it is here.

Posted to issues@flink.apache.org by "HuangZhenQiu (via GitHub)" <gi...@apache.org> on 2026/04/22 23:08:14 UTC

[PR] [FLINK-39491][runtime] move lineage Listener creation to Dispatcher [flink]

HuangZhenQiu opened a new pull request, #28002:
URL: https://github.com/apache/flink/pull/28002

## What is the purpose of the change

Currently, the job creation event happens in client side, and any follow up status change events propagation happen in DefaultExecutionGraph. In this case, there are two instances of the same job listener created. We probably may consider moving the job creation event to dispatcher. With the change, a single job listener instance for each type in JVM needs to be created. Then, states within the instance will be shared within the whole job lifecycle.

## Brief change log
- Move JobStatusChangedListener creation for Job Creation Event to Dispatcher
- Change JobStatusChangedListener to singleton for each type in JobStatusChangedListenerUtils
- Add test cases for JobStatusChangedListenerUtils.

## Verifying this change

This change added tests and can be verified as follows:

- JobStatusChangedListenerUtilsTest is added to test JobStatusChangedListener creation

## Does this pull request potentially affect one of the following parts:

- Dependencies (does it add or upgrade a dependency): (no)
- The public API, i.e., is any changed class annotated with `@Public(Evolving)`: (no)
- The serializers: (no)
- The runtime per-record code paths (performance sensitive): (no)
- Anything that affects deployment or recovery: JobManager (and its components), Checkpointing, Kubernetes/Yarn, ZooKeeper: (no)
- The S3 file system connector: (no)

## Documentation

- Does this pull request introduce a new feature? (no)
- If yes, how is the feature documented? (not applicable)

---

##### Was generative AI tooling used to co-author this PR?

<!--
If generative AI tooling has been used in the process of authoring this PR, please
change the checkbox below to `[X]` followed by the name of the tool, and uncomment the
"Generated-by" line. See the ASF Generative Tooling Guidance for details:
https://www.apache.org/legal/generative-tooling.html

You are responsible for the quality and correctness of every change in this PR
regardless of the tooling used. Low-effort AI-generated PRs will be closed. See
AGENTS.md for the full guidance.
-->

- [ ] Yes (please specify the tool below)

--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [PR] [FLINK-39491][runtime] move lineage Listener creation to Dispatcher [flink]

Posted by "swapna267 (via GitHub)" <gi...@apache.org>.

swapna267 commented on PR #28002:
URL: https://github.com/apache/flink/pull/28002#issuecomment-4337737116

Specific to lineage listener , this seems to break for code using the flink cli to submit the job to session cluster.

As per my understanding, StreamGraph is generated and serialized over network in those use cases, for example,
https://github.com/apache/flink/blob/master/flink-clients/src/main/java/org/apache/flink/client/deployment/executors/AbstractSessionClusterExecutor.java#L104

LineageGraph is transient and is not serialized over.
https://github.com/apache/flink/blob/master/flink-runtime/src/main/java/org/apache/flink/streaming/api/graph/StreamGraph.java#L166 .

So LineageGraph details are not available on Dispatcher running as part of JobManage JVM.
Please correct me if my understanding is not right.

I see in https://cwiki.apache.org/confluence/display/FLINK/FLIP-314%3A+Support+Customized+Job+Lineage+Listener,

`In this FLIP the JobStatusChangedListener will be in Client and JobMaster, which will report lineage information and job status independently. Another proposal is to **put the lineage information into JobGraph and then report it in JobMaster** . Considering that the lineage information may be relatively large and affect JobGraph , we will report separately in the first phase and then consider merging in the future.`

That seems to be the right way to go about if we want to ensure,
1. Single Lineage listener context
2. JobCreationEvent is sent before any JobTransitionEvent.

--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org