You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@flink.apache.org by "arvindKandpal-ksolves (via GitHub)" <gi...@apache.org> on 2026/06/18 09:36:23 UTC

[PR] [FLINK-37351][runtime] Ensure writer and committer are colocated for non-pre-commit topologies [flink]

arvindKandpal-ksolves opened a new pull request, #28479:
URL: https://github.com/apache/flink/pull/28479

   ## What is the purpose of the change
   
   This pull request addresses the issue detailed in FLINK-37351, which aims to ensure writer and committer colocation specifically for non-pre-commit topologies. This optimization eliminates unnecessary network hops between the writer and committer, thereby improving overall performance and potentially enabling backchannels (Statefun style). 
   
   *Note: This PR takes over the stale work originally started in PR #27486. It fixes the CI propagation issue from the previous PR by moving the colocation group key assignment to the very beginning of the translation lifecycle (`translateInternal`).*
   
   ## Brief change log
   
     - Assigned a default co-location group key in `SinkTransformationTranslator.translateInternal()` for `SinkTransformation`s where the sink implements `SupportsCommitter` but does NOT implement `SupportsPreCommitTopology`.
     - Setting this early ensures the co-location property propagates correctly to all expanded child transformations (Writer and Committer).
     - Ensured that any user-specified co-location group keys are strictly respected and not overwritten.
   
   ## Verifying this change
   
   This change added tests and can be verified as follows:
   
     - Added `testWriterAndCommitterColocatedWithoutPreCommitTopology` to verify writer/committer colocation is strictly applied for non-pre-commit topologies.
     - Added `testWriterAndCommitterNotColocatedWithPreCommitTopology` to verify sinks with pre-commit topologies bypass this co-location logic.
     - Added `testUserSpecifiedCoLocationGroupIsRespected` to verify that manually defined co-location groups by the user are preserved across the translation pipeline.
     - Verified that all downstream properties (parallelism, slot sharing groups, etc.) propagate seamlessly.
   
   ## Does this pull request potentially affect one of the following parts:
   
     - Dependencies (does it add or upgrade a dependency): **no**
     - The public API, i.e., is any changed class annotated with `@Public(Evolving)`: **no**
     - The serializers: **no**
     - The runtime per-record code paths (performance sensitive): **no**
     - Anything that affects deployment or recovery: JobManager (and its components), Checkpointing, Kubernetes/Yarn, ZooKeeper: **yes** *(Modifies task scheduling/colocation for Sinks)*
     - The S3 file system connector: **no**
   
   ## Documentation
   
     - Does this pull request introduce a new feature? **no** *(Optimization/Bugfix)*
     - If yes, how is the feature documented? **not applicable**
   
   ---
   
   ##### Was generative AI tooling used to co-author this PR?
   
   <!--
   If generative AI tooling has been used in the process of authoring this PR, please
   change the checkbox below to `[X]` followed by the name of the tool, and uncomment the
   "Generated-by" line. See the ASF Generative Tooling Guidance for details:
   https://www.apache.org/legal/generative-tooling.html
   
   You are responsible for the quality and correctness of every change in this PR
   regardless of the tooling used. Low-effort AI-generated PRs will be closed. See
   AGENTS.md for the full guidance.
   -->
   
   - [ ] Yes (please specify the tool below)
   
   <!--
   Generated-by: [Tool Name and Version]
   -->
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [FLINK-37351][runtime] Ensure writer and committer are colocated for non-pre-commit topologies [flink]

Posted by "arvindKandpal-ksolves (via GitHub)" <gi...@apache.org>.
arvindKandpal-ksolves commented on PR #28479:
URL: https://github.com/apache/flink/pull/28479#issuecomment-4745370876

   > @arvindKandpal-ksolves Can you please check if you have followed https://flink.apache.org/how-to-contribute/contribute-code/#ai-assisted-contributions correctly?
   
   I am bit confused on this like in temple we have option to disclose AI tool usage if it is co-author, but in my case I use gemini in some cases but chat version, and all the test case was run before raising the PR. 
   
   What I think, if I use tool like Claude Code , Codex in that case I should fill the check in PR template as they are AI agent?
   
   I am not sure , Please let me know if you about more.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [FLINK-37351][runtime] Ensure writer and committer are colocated for non-pre-commit topologies [flink]

Posted by "arvindKandpal-ksolves (via GitHub)" <gi...@apache.org>.
arvindKandpal-ksolves commented on PR #28479:
URL: https://github.com/apache/flink/pull/28479#issuecomment-4745285552

   And I ask in jira about this
   
   >  it's also didn't check if the previous PR was really stale / asked if there was anyone working on it. 
   
   And I don't get the response, and that PR already marked stale 3 week ago by GitHub action.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [FLINK-37351][runtime] Ensure writer and committer are colocated for non-pre-commit topologies [flink]

Posted by "MartijnVisser (via GitHub)" <gi...@apache.org>.
MartijnVisser commented on PR #28479:
URL: https://github.com/apache/flink/pull/28479#issuecomment-4743981517

   @arvindKandpal-ksolves Can you please check if you have followed https://flink.apache.org/how-to-contribute/contribute-code/#ai-assisted-contributions correctly?


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [FLINK-37351][runtime] Ensure writer and committer are colocated for non-pre-commit topologies [flink]

Posted by "arvindKandpal-ksolves (via GitHub)" <gi...@apache.org>.
arvindKandpal-ksolves commented on PR #28479:
URL: https://github.com/apache/flink/pull/28479#issuecomment-4745253816

   Hi @MartijnVisser , Other PR already marked stale , but still open. But if this not meet the code quality surely I will close my PR for now.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [FLINK-37351][runtime] Ensure writer and committer are colocated for non-pre-commit topologies [flink]

Posted by "arvindKandpal-ksolves (via GitHub)" <gi...@apache.org>.
arvindKandpal-ksolves commented on PR #28479:
URL: https://github.com/apache/flink/pull/28479#issuecomment-4740472113

   Hi @davidradl , Please take a look and review it whenever have time.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [FLINK-37351][runtime] Ensure writer and committer are colocated for non-pre-commit topologies [flink]

Posted by "MartijnVisser (via GitHub)" <gi...@apache.org>.
MartijnVisser commented on PR #28479:
URL: https://github.com/apache/flink/pull/28479#issuecomment-4745541935

   > What I think, if I use tool like Claude Code , Codex in that case I should fill the check in PR template as they are AI agent?
   
   Yes. For every AI coding tool, you must tick the box and you need to make sure that the commit message adheres to include the used tool, from the guidelines
   > Disclose AI usage by checking the AI disclosure checkbox in the pull request template and adding a Generated-by: <Tool Name and Version> trailer to each commit, per the [ASF Generative Tooling Guidance](https://www.apache.org/legal/generative-tooling.html).


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [FLINK-37351][runtime] Ensure writer and committer are colocated for non-pre-commit topologies [flink]

Posted by "flinkbot (via GitHub)" <gi...@apache.org>.
flinkbot commented on PR #28479:
URL: https://github.com/apache/flink/pull/28479#issuecomment-4740659254

   <!--
   Meta data
   {
     "version" : 1,
     "metaDataEntries" : [ {
       "hash" : "73642393dc8d5fdf365b0a16faa1537bf382a4ca",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "73642393dc8d5fdf365b0a16faa1537bf382a4ca",
       "triggerType" : "PUSH"
     } ]
   }-->
   ## CI report:
   
   * 73642393dc8d5fdf365b0a16faa1537bf382a4ca UNKNOWN
   
   <details>
   <summary>Bot commands</summary>
     The @flinkbot bot supports the following commands:
   
    - `@flinkbot run azure` re-run the last Azure build
   </details>


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [FLINK-37351][runtime] Ensure writer and committer are colocated for non-pre-commit topologies [flink]

Posted by "MartijnVisser (via GitHub)" <gi...@apache.org>.
MartijnVisser commented on PR #28479:
URL: https://github.com/apache/flink/pull/28479#issuecomment-4744048542

   Besides this highly likely to being an AI PR, it's also didn't check if the previous PR was really stale / asked if there was anyone working on it. The code quality is also lower compared to the previous PR, but my main worry is that this isn't how I think we should work together in the community


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [FLINK-37351][runtime] Ensure writer and committer are colocated for non-pre-commit topologies [flink]

Posted by "arvindKandpal-ksolves (via GitHub)" <gi...@apache.org>.
arvindKandpal-ksolves closed pull request #28479: [FLINK-37351][runtime] Ensure writer and committer are colocated for non-pre-commit topologies
URL: https://github.com/apache/flink/pull/28479


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org