You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@flink.apache.org by "RocMarshal (via GitHub)" <gi...@apache.org> on 2026/01/13 11:15:48 UTC
[PR] [FLINK-33123][runtime] Fix the wrong dynamic replacement of partitioner from FORWARD to REBLANCE for autoscaler and adaptive scheduler [flink]
RocMarshal opened a new pull request, #27404:
URL: https://github.com/apache/flink/pull/27404
<!--
*Thank you very much for contributing to Apache Flink - we are happy that you want to help us improve Flink. To help the community review your contribution in the best possible way, please go through the checklist below, which will get the contribution into a shape in which it can be best reviewed.*
*Please understand that we do not do this to make contributions to Flink a hassle. In order to uphold a high standard of quality for code contributions, while at the same time managing a large number of contributions, we need contributors to prepare the contributions well, and give reviewers enough contextual information for the review. Please also understand that contributions that do not follow this guide will take longer to review and thus typically be picked up with lower priority by the community.*
## Contribution Checklist
- Make sure that the pull request corresponds to a [JIRA issue](https://issues.apache.org/jira/projects/FLINK/issues). Exceptions are made for typos in JavaDoc or documentation files, which need no JIRA issue.
- Name the pull request in the form "[FLINK-XXXX] [component] Title of the pull request", where *FLINK-XXXX* should be replaced by the actual issue number. Skip *component* if you are unsure about which is the best component.
Typo fixes that have no associated JIRA issue should be named following this pattern: `[hotfix] [docs] Fix typo in event time introduction` or `[hotfix] [javadocs] Expand JavaDoc for PuncuatedWatermarkGenerator`.
- Fill out the template below to describe the changes contributed by the pull request. That will give reviewers the context they need to do the review.
- Make sure that the change passes the automated tests, i.e., `mvn clean verify` passes. You can set up Azure Pipelines CI to do that following [this guide](https://cwiki.apache.org/confluence/display/FLINK/Azure+Pipelines#AzurePipelines-Tutorial:SettingupAzurePipelinesforaforkoftheFlinkrepository).
- Each pull request should address only one issue, not mix up code from multiple issues.
- Each commit in the pull request has a meaningful commit message (including the JIRA id)
- Once all items of the checklist are addressed, remove the above text and this checklist, leaving only the filled out template below.
**(The sections below can be removed for hotfixes of typos)**
-->
## What is the purpose of the change
[FLINK-33123][runtime] Fix the wrong dynamic replacement of partitioner from FORWARD to REBLANCE for autoscaler and adaptive scheduler
## Brief change log
[FLINK-33123][runtime] Fix the wrong dynamic replacement of partitioner from FORWARD to REBLANCE for autoscaler and adaptive scheduler
## Verifying this change
- ParallelismBasedForwardEdgesAdapterTest
## Does this pull request potentially affect one of the following parts:
- Dependencies (does it add or upgrade a dependency): (yes / **no**)
- The public API, i.e., is any changed class annotated with `@Public(Evolving)`: (yes / **no**)
- The serializers: (yes / **no** / don't know)
- The runtime per-record code paths (performance sensitive): (yes / **no** / don't know)
- Anything that affects deployment or recovery: JobManager (and its components), Checkpointing, Kubernetes/Yarn, ZooKeeper: (yes / **no** / don't know)
- The S3 file system connector: (yes / **no** / don't know)
## Documentation
- Does this pull request introduce a new feature? (yes / **no**)
- If yes, how is the feature documented? (**not applicable** / docs / JavaDocs / not documented)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [FLINK-33123][runtime] Fix the wrong dynamic replacement of partitioner from FORWARD to REBALANCE for autoscaler and adaptive scheduler [flink]
Posted by "mxm (via GitHub)" <gi...@apache.org>.
mxm commented on code in PR #27404:
URL: https://github.com/apache/flink/pull/27404#discussion_r2716535489
##########
flink-runtime/src/main/java/org/apache/flink/runtime/jobgraph/JobEdge.java:
##########
@@ -111,6 +111,20 @@ public JobEdge(
this.intraInputKeyCorrelated = intraInputKeyCorrelated;
}
+ public void forceUpdate(
+ boolean isForward,
+ DistributionPattern distributionPattern,
+ SubtaskStateMapper upstreamSubtaskStateMapper,
+ SubtaskStateMapper downstreamSubtaskStateMapper,
+ String shipStrategyName) {
+ this.isForward = isForward;
+ this.distributionPattern = distributionPattern;
+ this.upstreamSubtaskStateMapper = checkNotNull(upstreamSubtaskStateMapper);
+ this.downstreamSubtaskStateMapper = checkNotNull(downstreamSubtaskStateMapper);
+ this.shipStrategyName = shipStrategyName;
+ this.source.forceUpdate(isForward, distributionPattern);
+ }
Review Comment:
I know this might be tricky, but could we prevent adding methods which mutate JobGraph? Any methods on JobGraph which mutate it, should yield a new JobGraph and not touch the existing JobGraph instance.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [FLINK-33123][runtime] Fix the wrong dynamic replacement of partitioner from FORWARD to REBLANCE for autoscaler and adaptive scheduler [flink]
Posted by "flinkbot (via GitHub)" <gi...@apache.org>.
flinkbot commented on PR #27404:
URL: https://github.com/apache/flink/pull/27404#issuecomment-3743799610
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "11d83b8bf80d1f8591da8714a8f01fb3a3661ed1",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "11d83b8bf80d1f8591da8714a8f01fb3a3661ed1",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 11d83b8bf80d1f8591da8714a8f01fb3a3661ed1 UNKNOWN
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [FLINK-33123][runtime] Fix the wrong dynamic replacement of partitioner from FORWARD to REBALANCE for autoscaler and adaptive scheduler [flink]
Posted by "mxm (via GitHub)" <gi...@apache.org>.
mxm commented on code in PR #27404:
URL: https://github.com/apache/flink/pull/27404#discussion_r2720479561
##########
flink-runtime/src/main/java/org/apache/flink/runtime/jobgraph/IntermediateDataSet.java:
##########
@@ -146,6 +146,11 @@ public void updateOutputPattern(
this.isForward = isForward;
}
+ public void forceUpdate(boolean isForward, DistributionPattern distributionPattern) {
Review Comment:
Same here. Change the name to something like updateDistributionPattern.
##########
flink-runtime/src/main/java/org/apache/flink/runtime/jobgraph/JobEdge.java:
##########
@@ -111,6 +111,20 @@ public JobEdge(
this.intraInputKeyCorrelated = intraInputKeyCorrelated;
}
+ public void forceUpdate(
Review Comment:
Can we change the name to make this easier to understand?
```suggestion
public void updateDistributionPattern(
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [FLINK-33123][runtime] Fix the wrong dynamic replacement of partitioner from FORWARD to REBLANCE for autoscaler and adaptive scheduler [flink]
Posted by "davidradl (via GitHub)" <gi...@apache.org>.
davidradl commented on PR #27404:
URL: https://github.com/apache/flink/pull/27404#issuecomment-3783652538
@RocMarshal nit - the title has a typo should be REBALANCE I think. It is worth changing so searches pick this up properly I think.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [FLINK-33123][runtime] Fix the wrong dynamic replacement of partitioner from FORWARD to REBLANCE for autoscaler and adaptive scheduler [flink]
Posted by "RocMarshal (via GitHub)" <gi...@apache.org>.
RocMarshal commented on code in PR #27404:
URL: https://github.com/apache/flink/pull/27404#discussion_r2713678251
##########
flink-runtime/src/main/java/org/apache/flink/runtime/jobgraph/JobEdge.java:
##########
@@ -35,7 +35,7 @@ public class JobEdge implements java.io.Serializable {
private final JobVertex target;
/** The distribution pattern that should be used for this job edge. */
- private final DistributionPattern distributionPattern;
+ private DistributionPattern distributionPattern;
Review Comment:
nice proposal & updated.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [FLINK-33123][runtime] Fix the wrong dynamic replacement of partitioner from FORWARD to REBALANCE for autoscaler and adaptive scheduler [flink]
Posted by "mxm (via GitHub)" <gi...@apache.org>.
mxm commented on code in PR #27404:
URL: https://github.com/apache/flink/pull/27404#discussion_r2717179093
##########
flink-runtime/src/main/java/org/apache/flink/runtime/jobgraph/JobEdge.java:
##########
@@ -111,6 +111,20 @@ public JobEdge(
this.intraInputKeyCorrelated = intraInputKeyCorrelated;
}
+ public void forceUpdate(
+ boolean isForward,
+ DistributionPattern distributionPattern,
+ SubtaskStateMapper upstreamSubtaskStateMapper,
+ SubtaskStateMapper downstreamSubtaskStateMapper,
+ String shipStrategyName) {
+ this.isForward = isForward;
+ this.distributionPattern = distributionPattern;
+ this.upstreamSubtaskStateMapper = checkNotNull(upstreamSubtaskStateMapper);
+ this.downstreamSubtaskStateMapper = checkNotNull(downstreamSubtaskStateMapper);
+ this.shipStrategyName = shipStrategyName;
+ this.source.forceUpdate(isForward, distributionPattern);
+ }
Review Comment:
I was imagining that we create a new instance of JobEdge via its constructor which takes the DistributionPattern and `isForward`, but this approach may not be feasible. I realized that JobEdge and JobVertex are already mutable. So the current approach may be fine.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [FLINK-33123][runtime] Fix the wrong dynamic replacement of partitioner from FORWARD to REBALANCE for autoscaler and adaptive scheduler [flink]
Posted by "RocMarshal (via GitHub)" <gi...@apache.org>.
RocMarshal commented on PR #27404:
URL: https://github.com/apache/flink/pull/27404#issuecomment-3783713866
> @RocMarshal nit - the title has a typo should be REBALANCE I think. It is worth changing so searches pick this up properly I think.
Thanks @davidradl and Nice catch!
Updated.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [FLINK-33123][runtime] Fix the wrong dynamic replacement of partitioner from FORWARD to REBALANCE for autoscaler and adaptive scheduler [flink]
Posted by "mxm (via GitHub)" <gi...@apache.org>.
mxm merged PR #27404:
URL: https://github.com/apache/flink/pull/27404
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [FLINK-33123][runtime] Fix the wrong dynamic replacement of partitioner from FORWARD to REBALANCE for autoscaler and adaptive scheduler [flink]
Posted by "mxm (via GitHub)" <gi...@apache.org>.
mxm commented on PR #27404:
URL: https://github.com/apache/flink/pull/27404#issuecomment-3824682153
Thank you @RocMarshal!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [FLINK-33123][runtime] Fix the wrong dynamic replacement of partitioner from FORWARD to REBALANCE for autoscaler and adaptive scheduler [flink]
Posted by "RocMarshal (via GitHub)" <gi...@apache.org>.
RocMarshal commented on code in PR #27404:
URL: https://github.com/apache/flink/pull/27404#discussion_r2717275103
##########
flink-runtime/src/main/java/org/apache/flink/runtime/jobgraph/JobEdge.java:
##########
@@ -111,6 +111,20 @@ public JobEdge(
this.intraInputKeyCorrelated = intraInputKeyCorrelated;
}
+ public void forceUpdate(
+ boolean isForward,
+ DistributionPattern distributionPattern,
+ SubtaskStateMapper upstreamSubtaskStateMapper,
+ SubtaskStateMapper downstreamSubtaskStateMapper,
+ String shipStrategyName) {
+ this.isForward = isForward;
+ this.distributionPattern = distributionPattern;
+ this.upstreamSubtaskStateMapper = checkNotNull(upstreamSubtaskStateMapper);
+ this.downstreamSubtaskStateMapper = checkNotNull(downstreamSubtaskStateMapper);
+ this.shipStrategyName = shipStrategyName;
+ this.source.forceUpdate(isForward, distributionPattern);
+ }
Review Comment:
> I was imagining that we create a new instance of JobEdge via its constructor which takes the DistributionPattern and isForward
Yes, the introduced new method is required for the `final` attributes.
We could also wait a few days to see if any new, concise, and safe methods become available.
Thanks @mxm .
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [FLINK-33123][runtime] Fix the wrong dynamic replacement of partitioner from FORWARD to REBLANCE for autoscaler and adaptive scheduler [flink]
Posted by "RocMarshal (via GitHub)" <gi...@apache.org>.
RocMarshal commented on code in PR #27404:
URL: https://github.com/apache/flink/pull/27404#discussion_r2713678251
##########
flink-runtime/src/main/java/org/apache/flink/runtime/jobgraph/JobEdge.java:
##########
@@ -35,7 +35,7 @@ public class JobEdge implements java.io.Serializable {
private final JobVertex target;
/** The distribution pattern that should be used for this job edge. */
- private final DistributionPattern distributionPattern;
+ private DistributionPattern distributionPattern;
Review Comment:
nice proposal
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [FLINK-33123][runtime] Fix the wrong dynamic replacement of partitioner from FORWARD to REBLANCE for autoscaler and adaptive scheduler [flink]
Posted by "RocMarshal (via GitHub)" <gi...@apache.org>.
RocMarshal commented on PR #27404:
URL: https://github.com/apache/flink/pull/27404#issuecomment-3777522736
Hi, @mxm @X-czh Could you help take a look on it if you had the free time ?
I'd appreciate any input~
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [FLINK-33123][runtime] Fix the wrong dynamic replacement of partitioner from FORWARD to REBALANCE for autoscaler and adaptive scheduler [flink]
Posted by "RocMarshal (via GitHub)" <gi...@apache.org>.
RocMarshal commented on code in PR #27404:
URL: https://github.com/apache/flink/pull/27404#discussion_r2716911325
##########
flink-runtime/src/main/java/org/apache/flink/runtime/jobgraph/JobEdge.java:
##########
@@ -111,6 +111,20 @@ public JobEdge(
this.intraInputKeyCorrelated = intraInputKeyCorrelated;
}
+ public void forceUpdate(
+ boolean isForward,
+ DistributionPattern distributionPattern,
+ SubtaskStateMapper upstreamSubtaskStateMapper,
+ SubtaskStateMapper downstreamSubtaskStateMapper,
+ String shipStrategyName) {
+ this.isForward = isForward;
+ this.distributionPattern = distributionPattern;
+ this.upstreamSubtaskStateMapper = checkNotNull(upstreamSubtaskStateMapper);
+ this.downstreamSubtaskStateMapper = checkNotNull(downstreamSubtaskStateMapper);
+ this.shipStrategyName = shipStrategyName;
+ this.source.forceUpdate(isForward, distributionPattern);
+ }
Review Comment:
Hi, @mxm Please let me have a clarify:
The mutated JogGraph is a copy from the original jog graph of JobGraphInformation, which is almost equivalent to what you expressed. And the action didn't change the original existing JobGraph instance.
IIUC, Did you mean that instead of introducing this method `forceUpdate ` , we would regenerate the JobGraph based on the existing JobGraph and reuse the methods already defined in it?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [FLINK-33123][runtime] Fix the wrong dynamic replacement of partitioner from FORWARD to REBALANCE for autoscaler and adaptive scheduler [flink]
Posted by "mxm (via GitHub)" <gi...@apache.org>.
mxm commented on code in PR #27404:
URL: https://github.com/apache/flink/pull/27404#discussion_r2717040404
##########
flink-runtime/src/main/java/org/apache/flink/runtime/jobgraph/JobEdge.java:
##########
@@ -111,6 +111,20 @@ public JobEdge(
this.intraInputKeyCorrelated = intraInputKeyCorrelated;
}
+ public void forceUpdate(
+ boolean isForward,
+ DistributionPattern distributionPattern,
+ SubtaskStateMapper upstreamSubtaskStateMapper,
+ SubtaskStateMapper downstreamSubtaskStateMapper,
+ String shipStrategyName) {
+ this.isForward = isForward;
+ this.distributionPattern = distributionPattern;
+ this.upstreamSubtaskStateMapper = checkNotNull(upstreamSubtaskStateMapper);
+ this.downstreamSubtaskStateMapper = checkNotNull(downstreamSubtaskStateMapper);
+ this.shipStrategyName = shipStrategyName;
+ this.source.forceUpdate(isForward, distributionPattern);
+ }
Review Comment:
>IIUC, Did you mean that instead of introducing this method forceUpdate , we would regenerate the JobGraph based on the existing JobGraph and reuse the methods already defined in it?
Yes, exactly. I want to avoid adding any methods for mutation in JobGraph / JobVertex / JobEdge. Let me know if that is feasible.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [FLINK-33123][runtime] Fix the wrong dynamic replacement of partitioner from FORWARD to REBALANCE for autoscaler and adaptive scheduler [flink]
Posted by "RocMarshal (via GitHub)" <gi...@apache.org>.
RocMarshal commented on code in PR #27404:
URL: https://github.com/apache/flink/pull/27404#discussion_r2717151403
##########
flink-runtime/src/main/java/org/apache/flink/runtime/jobgraph/JobEdge.java:
##########
@@ -111,6 +111,20 @@ public JobEdge(
this.intraInputKeyCorrelated = intraInputKeyCorrelated;
}
+ public void forceUpdate(
+ boolean isForward,
+ DistributionPattern distributionPattern,
+ SubtaskStateMapper upstreamSubtaskStateMapper,
+ SubtaskStateMapper downstreamSubtaskStateMapper,
+ String shipStrategyName) {
+ this.isForward = isForward;
+ this.distributionPattern = distributionPattern;
+ this.upstreamSubtaskStateMapper = checkNotNull(upstreamSubtaskStateMapper);
+ this.downstreamSubtaskStateMapper = checkNotNull(downstreamSubtaskStateMapper);
+ this.shipStrategyName = shipStrategyName;
+ this.source.forceUpdate(isForward, distributionPattern);
+ }
Review Comment:
Hi, @mxm
sorry, in my limited reading:
Currently, the JobGraph does not carry any StreamGraph-related information, so it is impossible for the JobGraph to be created based on the StreamGraph in the same way as it is during the initial generation.
We can only generate a new JobGraph from the original JobGraph. Frankly speaking, when constructing the JobEdges of the new graph, if we do not introduce any additional methods or resort to reflection, we have no way to modify the JobEdge#distributionPattern and JobEdge#isForward attributes.
Pls correct me if anything wrong.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [FLINK-33123][runtime] Fix the wrong dynamic replacement of partitioner from FORWARD to REBLANCE for autoscaler and adaptive scheduler [flink]
Posted by "RocMarshal (via GitHub)" <gi...@apache.org>.
RocMarshal commented on code in PR #27404:
URL: https://github.com/apache/flink/pull/27404#discussion_r2713682517
##########
flink-runtime/src/main/java/org/apache/flink/runtime/scheduler/adaptive/AdaptiveScheduler.java:
##########
@@ -1473,8 +1474,11 @@ private ExecutionGraph createExecutionGraphAndRestoreState(
new ExecutionStateUpdateListener[0]));
}
+ final JobGraph adjustedJobGraph = jobInformation.copyJobGraph();
+ adaptiveChangeForwardJobEdges(adjustedJobGraph, adjustedParallelismStore);
Review Comment:
Thanks @mxm .
That makes sense to me.
Updated.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [FLINK-33123][runtime] Fix the wrong dynamic replacement of partitioner from FORWARD to REBLANCE for autoscaler and adaptive scheduler [flink]
Posted by "mxm (via GitHub)" <gi...@apache.org>.
mxm commented on code in PR #27404:
URL: https://github.com/apache/flink/pull/27404#discussion_r2713197883
##########
flink-runtime/src/main/java/org/apache/flink/runtime/jobgraph/JobEdge.java:
##########
@@ -54,7 +54,8 @@ public class JobEdge implements java.io.Serializable {
private final boolean isBroadcast;
- private final boolean isForward;
+ private boolean isForward;
+ private final boolean initialForward;
Review Comment:
Same here, let's not add extra fields for modification.
##########
flink-runtime/src/main/java/org/apache/flink/runtime/jobgraph/JobEdge.java:
##########
@@ -35,7 +35,7 @@ public class JobEdge implements java.io.Serializable {
private final JobVertex target;
/** The distribution pattern that should be used for this job edge. */
- private final DistributionPattern distributionPattern;
+ private DistributionPattern distributionPattern;
Review Comment:
I think it is better not to mutate the existing JobGraph.
##########
flink-runtime/src/main/java/org/apache/flink/runtime/scheduler/adaptive/AdaptiveScheduler.java:
##########
@@ -1473,8 +1474,11 @@ private ExecutionGraph createExecutionGraphAndRestoreState(
new ExecutionStateUpdateListener[0]));
}
+ final JobGraph adjustedJobGraph = jobInformation.copyJobGraph();
+ adaptiveChangeForwardJobEdges(adjustedJobGraph, adjustedParallelismStore);
Review Comment:
Can we return a JobGraph copy with the adjustments from this method? I want to avoid mutating an existing JobGraph.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [FLINK-33123][runtime] Fix the wrong dynamic replacement of partitioner from FORWARD to REBALANCE for autoscaler and adaptive scheduler [flink]
Posted by "RocMarshal (via GitHub)" <gi...@apache.org>.
RocMarshal commented on code in PR #27404:
URL: https://github.com/apache/flink/pull/27404#discussion_r2716911325
##########
flink-runtime/src/main/java/org/apache/flink/runtime/jobgraph/JobEdge.java:
##########
@@ -111,6 +111,20 @@ public JobEdge(
this.intraInputKeyCorrelated = intraInputKeyCorrelated;
}
+ public void forceUpdate(
+ boolean isForward,
+ DistributionPattern distributionPattern,
+ SubtaskStateMapper upstreamSubtaskStateMapper,
+ SubtaskStateMapper downstreamSubtaskStateMapper,
+ String shipStrategyName) {
+ this.isForward = isForward;
+ this.distributionPattern = distributionPattern;
+ this.upstreamSubtaskStateMapper = checkNotNull(upstreamSubtaskStateMapper);
+ this.downstreamSubtaskStateMapper = checkNotNull(downstreamSubtaskStateMapper);
+ this.shipStrategyName = shipStrategyName;
+ this.source.forceUpdate(isForward, distributionPattern);
+ }
Review Comment:
Hi, @mxm
Thanks for the comments.
Please let me have a clarify:
The mutated JogGraph is a copy from the original jog graph of JobGraphInformation, which is almost equivalent to what you expressed. And the action didn't change the original existing JobGraph instance.
IIUC, Did you mean that instead of introducing this method `forceUpdate ` , we would regenerate the JobGraph based on the existing JobGraph and reuse the methods already defined in it?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [FLINK-33123][runtime] Fix the wrong dynamic replacement of partitioner from FORWARD to REBALANCE for autoscaler and adaptive scheduler [flink]
Posted by "RocMarshal (via GitHub)" <gi...@apache.org>.
RocMarshal commented on code in PR #27404:
URL: https://github.com/apache/flink/pull/27404#discussion_r2717275103
##########
flink-runtime/src/main/java/org/apache/flink/runtime/jobgraph/JobEdge.java:
##########
@@ -111,6 +111,20 @@ public JobEdge(
this.intraInputKeyCorrelated = intraInputKeyCorrelated;
}
+ public void forceUpdate(
+ boolean isForward,
+ DistributionPattern distributionPattern,
+ SubtaskStateMapper upstreamSubtaskStateMapper,
+ SubtaskStateMapper downstreamSubtaskStateMapper,
+ String shipStrategyName) {
+ this.isForward = isForward;
+ this.distributionPattern = distributionPattern;
+ this.upstreamSubtaskStateMapper = checkNotNull(upstreamSubtaskStateMapper);
+ this.downstreamSubtaskStateMapper = checkNotNull(downstreamSubtaskStateMapper);
+ this.shipStrategyName = shipStrategyName;
+ this.source.forceUpdate(isForward, distributionPattern);
+ }
Review Comment:
> I was imagining that we create a new instance of JobEdge via its constructor which takes the DistributionPattern and isForward
Yes, the introduced new method is required for changing the `final` attributes.
We could also wait a few days to see if any new, concise, and safe methods become available.
Thanks @mxm .
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [FLINK-33123][runtime] Fix the wrong dynamic replacement of partitioner from FORWARD to REBALANCE for autoscaler and adaptive scheduler [flink]
Posted by "RocMarshal (via GitHub)" <gi...@apache.org>.
RocMarshal commented on code in PR #27404:
URL: https://github.com/apache/flink/pull/27404#discussion_r2720517999
##########
flink-runtime/src/main/java/org/apache/flink/runtime/jobgraph/IntermediateDataSet.java:
##########
@@ -146,6 +146,11 @@ public void updateOutputPattern(
this.isForward = isForward;
}
+ public void forceUpdate(boolean isForward, DistributionPattern distributionPattern) {
Review Comment:
Thanks @mxm
That makes sense to me.
Updated.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [FLINK-33123][runtime] Fix the wrong dynamic replacement of partitioner from FORWARD to REBALANCE for autoscaler and adaptive scheduler [flink]
Posted by "RocMarshal (via GitHub)" <gi...@apache.org>.
RocMarshal commented on PR #27404:
URL: https://github.com/apache/flink/pull/27404#issuecomment-3827262948
Thanks all of you involved in the related work.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [FLINK-33123][runtime] Fix the wrong dynamic replacement of partitioner from FORWARD to REBALANCE for autoscaler and adaptive scheduler [flink]
Posted by "1996fanrui (via GitHub)" <gi...@apache.org>.
1996fanrui commented on PR #27404:
URL: https://github.com/apache/flink/pull/27404#issuecomment-3826053427
Thanks @RocMarshal for picking this long-standing issue up, and thanks @mxm and @X-czh for the detailed discussion and review!
It's great to see the issue being resolved in the community.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org