You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@flink.apache.org by Yuepeng Pan <pa...@apache.org> on 2024/12/17 15:18:45 UTC

[DISCUSS] FLIP-495: Support AdaptiveScheduler record and query the rescale history

Hi community,




We discussed several aspects of FLIP-487[1] 'Show history of rescales in Web UI for AdaptiveScheduler' 
and received a lot of valuable feedback. Based on the suggestions from the email thread[2],
we plan to split the original proposal for FLIP-487[1]. 




The current email thread and the FLIP-495[3] wiki will be used to discuss 'Support AdaptiveScheduler in recording and querying the rescale history', 
while FLIP-487[1] will primarily focus on displaying-related design content




Looking forward to any feedback and opinions on FLIP-495[3].




[1] https://cwiki.apache.org/confluence/display/FLINK/%5BWIP%5D+FLIP-487%3A+Show+history+of+rescales+in+Web+UI+for+AdaptiveScheduler

[2] https://lists.apache.org/thread/f4md4btkf006mxcxf66bng1kfz0rsn8c

[3] https://cwiki.apache.org/confluence/display/FLINK/%5BWIP%5D+FLIP-495%3A+Support+AdaptiveScheduler+record+and+query+the+rescale+history




Thank you very much.




Best,

Regards.

Yuepeng Pan

Re: Re: [DISCUSS] FLIP-495: Support AdaptiveScheduler record and query the rescale history

Posted by Yuepeng Pan <pa...@apache.org>.
Hi, Matthias, So sorry for the long delay in completing and following up. 
I'm still making the necessary research to address the issues mentioned previously in this FLIP.

FYI, the new round of adjustments has been completed, and I've attached an overview of a few issues.

> Also keep in mind that it's not only the JobResourceRequirements update that can trigger a rescale. 
> Updating the available resources through newResourcesAvailable [2] can also initiate a rescale.
> I don't see this being clearly laid out in FLIP-495, yet.

Thanks for the reminding. I added it in the section of ‘Current core adaptive scheduler state transition about rescale’[1] 

> I'm wondering whether that could be even considered a bug that manifests itself 
> if the stopWithSavepoint mechanism fails and the job is executed.

Sorry, Matthias, I am not very familiar with the original design intentions of this part. 
However, from the current perspective, since the current behavior is 
allowed and the community has not received any risk-related feedback, 
it seems acceptable to not address it in the current FLIP.
Of course, if we now define it as an unintended behavior, 
I think that would also be reasonable, because, if I remember correctly, 
this aligns with the semantics of the DefaultScheduler for streaming jobs.

> It might be useful to make this clearer in the FLIP as well (i.e. adding more context aside from the bullet points).

Thanks Matthias for the reminder. I have added some perspectives in the Rescale state definition section and stated some reasons. [2]

> I agree that we might want to have this information being stored in DFS. 
> That way the information would survive a JobManager failover. 
> It would be still nice to have all three options being reflected in the FLIP, 
> though, with the pro's and con's having the feature properly documented.

Thanks Matthias for the reminder and sharing. I have updated the corresponding document in the About rescale events storage section.[3]

> I also want to point out that we have four different notions of resource configurations:
> Desired resources: The ideal resource configuration that we want to achieve for a job if 
> enough Task slots are available (essentially the upper bound of the job's parallelism)
> Sufficient resources: A minimum resource configuration that the job can run on (the lower bound of the job's parallelism)
> Current resources: The resource configuration the job runs on before rescaling
> Follow-up resources
> The first two are the resource configurations the rescale decision is based on. 
> The last two are the actual applied resource configurations. 
> Keep in mind that the latter two are not necessarily matching the resource configurations 
> that were considered when deciding on the rescaling.
> Especially the case where the desired resources were met when rescaling was triggered but where task slots are lost while rescaling can have a surprising outcome. 
> We might want to have this reflected in the rescale event.

Thank you Matthias for your sharing! This is very reasonable and
will enhance the readability of the information as well as its usability for users.
I have made some adjustments in the corresponding part[4].

Any suggestions or comments would be greatly appreciated!

Best regards,
Yuepeng.

[1] https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=334760525#id-[WIP]FLIP495:SupportAdaptiveSchedulerrecordandquerytherescalehistory-Currentcoreadaptiveschedulerstatetransitionaboutrescale
[2] https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=334760525#id-[WIP]FLIP495:SupportAdaptiveSchedulerrecordandquerytherescalehistory-Rescalestatus
[3] https://cwiki.apache.org/confluence/display/FLINK/%5BWIP%5D+FLIP-495%3A+Support+AdaptiveScheduler+record+and+query+the+rescale+history#id-[WIP]FLIP495:SupportAdaptiveSchedulerrecordandquerytherescalehistory-Aboutrescaleeventsstorage
[4] https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=334760525#id-[WIP]FLIP495:SupportAdaptiveSchedulerrecordandquerytherescalehistory-Slots/Resources


On 2025/01/13 07:53:25 Matthias Pohl wrote:
> Hi Yupeng,
> I managed to find some time to respond. See my answers inlined below.
> 
> Matthias
> 
> On Fri, Jan 3, 2025 at 11:48 AM Yuepeng Pan <pa...@apache.org> wrote:
> 
> > [...]
> > Sorry for not expressing this part clearly earlier.
> > IIUC, based on the Adaptive Scheduler state diagram [1],
> > when a stop-with-savepoint operation fails but the job is restartable,
> > it transitions to the WaitingForResources state, which implies a
> > potential rescaling process may trigger.
> >
> 
> Yes, you're right. That's a scenario that should be covered by the FLIP but
> doesn't include having the Executing state as an initial AdaptiveScheduler
> state. That's also a corner case which is not covered by the current
> implementation. ...at least we're not considering any job resource
> requirement updates during StopWithSavepoint.
> 
> I'm wondering whether that could be even considered a bug that manifests
> itself if the stopWithSavepoint mechanism fails and the job is executed.
> 
> 
> > From the current logic, a rescale event may be triggered under the
> > following circumstances, called 'rescale triggers' here:
> > - updateJobResourceRequirement
> > - Restart due to recoverable failure
> > - newResourceAvailable
> >
> > From Fig [1], the entire chain of rescale-related scheduler
> > states typically involves several loops:
> > - WaitingForResources -> CreatingExecutionGraph -> Executing ->
> > StopWithSavepoint (error & restartable) -> Restarting -> WaitingForResources
> > - CreatingExecutionGraph [-> WaitingForResources ->
> > CreatingExecutionGraph](optinal loop ) -> Executing(rescale triggers) ->
> > Restarting -> CreatingExecutionGraph
> >
> > Following your shared ideas: > Based on what I pointed out in my previous
> > post, I would think that a rescale operation has its >  starting point in
> > the AdaptiveScheduler's Executing state (i.e. when the job is running).  I
> > attempted to interpret the boundary definition of rescale events.
> >
> > The historical states of the Adaptive Scheduler during amsuccessful
> > rescale event would likely match one of the following patterns ?
> > - (Starting) Executing(rescale triggers) -> Restarting ->
> > CreatingExecutionGraph [ -> WaitingForResources -> CreatingExecutionGraph]
> > -> (Ending) Executing
> > - (Starting) Executing -> StopWithSavepoint (error & restartable) ->
> > Restarting -> WaitingForResources -> CreatingExecutionGraph [ ->
> > WaitingForResources -> CreatingExecutionGraph] -> (Ending) Executing
> >
> > If my understanding is incorrect, please feel free to correct me.
> >
> 
> No, I see your point now. I was too focused on the Executing state. But
> you're right: WaitingForRescale can also be part of error handling that
> happened while Executing the job or trying to stop the job (i.e.
> StopWithSavepoint). These cases need to be considered when collecting the
> rescale events. Sorry for not grasping your intention earlier and thanks
> for your clarification.
> 
> It might be useful to make this clearer in the FLIP as well (i.e. adding
> more context aside from the bullet points).
> 
> 2 - About the status of a rescale event.
> > > - Rescale status: I would say that this section needs to be reworked.
> > Can I understand that Rescale Event does indeed need some status
> > information,
> > such as FAILED, IGNORED, SUCCESS, and PENDING, etc,
> > to indicate the final status of a completed event or the current status of
> > an ongoing event?
> > In other words, the current status fields and their associations are
> > unreasonable,
> > so its need to be redesigned rather than discarding this descriptive
> > mechanism.
> >
> 
> Sounds reasonable.
> 
> 
> > [1]
> > https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=334760525#id-[WIP]FLIP495:SupportAdaptiveSchedulerrecordandquerytherescalehistory-CurrentadaptiveschedulerstatetransitionFig
> > .
> >
> > Thank you.
> >
> > Best,
> > Yuepeng
> >
> 

Re: Re: [DISCUSS] FLIP-495: Support AdaptiveScheduler record and query the rescale history

Posted by Matthias Pohl <ma...@apache.org>.
Hi Yupeng,
I managed to find some time to respond. See my answers inlined below.

Matthias

On Fri, Jan 3, 2025 at 11:48 AM Yuepeng Pan <pa...@apache.org> wrote:

> [...]
> Sorry for not expressing this part clearly earlier.
> IIUC, based on the Adaptive Scheduler state diagram [1],
> when a stop-with-savepoint operation fails but the job is restartable,
> it transitions to the WaitingForResources state, which implies a
> potential rescaling process may trigger.
>

Yes, you're right. That's a scenario that should be covered by the FLIP but
doesn't include having the Executing state as an initial AdaptiveScheduler
state. That's also a corner case which is not covered by the current
implementation. ...at least we're not considering any job resource
requirement updates during StopWithSavepoint.

I'm wondering whether that could be even considered a bug that manifests
itself if the stopWithSavepoint mechanism fails and the job is executed.


> From the current logic, a rescale event may be triggered under the
> following circumstances, called 'rescale triggers' here:
> - updateJobResourceRequirement
> - Restart due to recoverable failure
> - newResourceAvailable
>
> From Fig [1], the entire chain of rescale-related scheduler
> states typically involves several loops:
> - WaitingForResources -> CreatingExecutionGraph -> Executing ->
> StopWithSavepoint (error & restartable) -> Restarting -> WaitingForResources
> - CreatingExecutionGraph [-> WaitingForResources ->
> CreatingExecutionGraph](optinal loop ) -> Executing(rescale triggers) ->
> Restarting -> CreatingExecutionGraph
>
> Following your shared ideas: > Based on what I pointed out in my previous
> post, I would think that a rescale operation has its >  starting point in
> the AdaptiveScheduler's Executing state (i.e. when the job is running).  I
> attempted to interpret the boundary definition of rescale events.
>
> The historical states of the Adaptive Scheduler during amsuccessful
> rescale event would likely match one of the following patterns ?
> - (Starting) Executing(rescale triggers) -> Restarting ->
> CreatingExecutionGraph [ -> WaitingForResources -> CreatingExecutionGraph]
> -> (Ending) Executing
> - (Starting) Executing -> StopWithSavepoint (error & restartable) ->
> Restarting -> WaitingForResources -> CreatingExecutionGraph [ ->
> WaitingForResources -> CreatingExecutionGraph] -> (Ending) Executing
>
> If my understanding is incorrect, please feel free to correct me.
>

No, I see your point now. I was too focused on the Executing state. But
you're right: WaitingForRescale can also be part of error handling that
happened while Executing the job or trying to stop the job (i.e.
StopWithSavepoint). These cases need to be considered when collecting the
rescale events. Sorry for not grasping your intention earlier and thanks
for your clarification.

It might be useful to make this clearer in the FLIP as well (i.e. adding
more context aside from the bullet points).

2 - About the status of a rescale event.
> > - Rescale status: I would say that this section needs to be reworked.
> Can I understand that Rescale Event does indeed need some status
> information,
> such as FAILED, IGNORED, SUCCESS, and PENDING, etc,
> to indicate the final status of a completed event or the current status of
> an ongoing event?
> In other words, the current status fields and their associations are
> unreasonable,
> so its need to be redesigned rather than discarding this descriptive
> mechanism.
>

Sounds reasonable.


> [1]
> https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=334760525#id-[WIP]FLIP495:SupportAdaptiveSchedulerrecordandquerytherescalehistory-CurrentadaptiveschedulerstatetransitionFig
> .
>
> Thank you.
>
> Best,
> Yuepeng
>

Re:Re: [DISCUSS] FLIP-495: Support AdaptiveScheduler record and query the rescale history

Posted by Yuepeng Pan <pa...@apache.org>.
Thanks Matthias very much for the sharing and suggestions.

The ideas make sense to me on the whole and I'd be willing to update it.

I just want to confirm a few questions before starting the next round of wiki adjustments:







1 - About the Boundary Definition of Rescale Events




> A few other items I'd like to point out are the following ones:

> - Is the "Rescale Event" section still out-dated or do we have a different

> understanding of what a rescale operation is? Based on what I pointed out

> in my previous post, I would think that a rescale operation has its

> starting point in the AdaptiveScheduler's Executing state (i.e. when the

> job is running). That's how it is implemented right now. The "Rescale

> Event" section always starts from WaitingForResources, though. Do we

> disagree here?




Sorry for not expressing this part clearly earlier.

IIUC, based on the Adaptive Scheduler state diagram [1], 

when a stop-with-savepoint operation fails but the job is restartable, 

it transitions to the WaitingForResources state, which implies a 

potential rescaling process may trigger.

From the current logic, a rescale event may be triggered under the following circumstances, called 'rescale triggers' here:




- updateJobResourceRequirement

- Restart due to recoverable failure

- newResourceAvailable




From Fig [1], the entire chain of rescale-related scheduler 

states typically involves several loops:




- WaitingForResources -> CreatingExecutionGraph -> Executing -> StopWithSavepoint (error & restartable) -> Restarting -> WaitingForResources

- CreatingExecutionGraph [-> WaitingForResources -> CreatingExecutionGraph](optinal loop ) -> Executing(rescale triggers) -> Restarting -> CreatingExecutionGraph




Following your shared ideas: > Based on what I pointed out in my previous post, I would think that a rescale operation has its >  starting point in the AdaptiveScheduler's Executing state (i.e. when the job is running).  I attempted to interpret the boundary definition of rescale events. 

The historical states of the Adaptive Scheduler during a 

successful rescale event would likely match one of the following patterns ?




- (Starting) Executing(rescale triggers) -> Restarting -> CreatingExecutionGraph [ -> WaitingForResources -> CreatingExecutionGraph] -> (Ending) Executing

- (Starting) Executing -> StopWithSavepoint (error & restartable) -> Restarting -> WaitingForResources -> CreatingExecutionGraph [ -> WaitingForResources -> CreatingExecutionGraph] -> (Ending) Executing




If my understanding is incorrect, please feel free to correct me.







2 - About the status of a rescale event.




> - Rescale status: I would say that this section needs to be reworked. 




Can I understand that Rescale Event does indeed need some status information,
such as FAILED, IGNORED, SUCCESS, and PENDING, etc,
to indicate the final status of a completed event or the current status of an ongoing event?
In other words, the current status fields and their associations are unreasonable,
so its need to be redesigned rather than discarding this descriptive mechanism.







[1]https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=334760525#id-[WIP]FLIP495:SupportAdaptiveSchedulerrecordandquerytherescalehistory-CurrentadaptiveschedulerstatetransitionFig.

Thank you.


Best,

Yuepeng








At 2025-01-03 01:15:54, "Matthias Pohl" <ma...@apache.org> wrote:
>Thanks Yuepeng for your response. I added my comments to the individual
>paragraphs below:
>
>
>> Thank you very much for the reminding The proposal makes sense to me.
>> Additionally, I'd like to confirm whether each rescale cycle/event
>> requires a status field, such as FAILED, IGNORED, SUCCESS, PENDING, etc. If
>> such state fields are not needed, how do we record that a particular
>> rescale request was ignored? Or do we not care about this situation and
>> only plan to record successful rescale events?
>>
>
>Ignored events (due to the AdaptiveScheduler not being in Executing state)
>can be still collected by the AdaptiveScheduler in
>updateJobResourceRequirements [1]. That's where the rescale history could
>be populated.
>
>Also keep in mind that it's not only the JobResourceRequirements update
>that can trigger a rescale. Updating the available resources through
>newResourcesAvailable [2] can also initiate a rescale. I don't see this
>being clearly laid out in FLIP-495, yet.
>
>[1]
>https://github.com/apache/flink/blob/a5258a015d553196d34e082e75ca4ae916addbaf/flink-runtime/src/main/java/org/apache/flink/runtime/scheduler/adaptive/AdaptiveScheduler.java#L1115
>
>[2]
>https://github.com/apache/flink/blob/fc0ccf325527c3589d5cd5ae7397b22c22321cec/flink-runtime/src/main/java/org/apache/flink/runtime/scheduler/adaptive/ResourceListener.java#L28
>
>Yes, this could happen during the reserving slots phase, and it's important
>> to record this. In my limited read, this is feasible. We can collect the
>> exceptions from this phase while gathering the scheduler state history, and
>> record the specific information using the previously mentioned exception
>> field and comments field, or use failed status  as the final status of the
>> rescale event/record, WDYTA?
>>
>
>I struggle to follow why we put this in a comment and/or exception field.
>Shouldn't we have the information provided in structured fields for each
>event? Something like:
>- pre-rescale resources
>- on-rescale-trigger sufficient resources
>- on-rescale-trigger desired resources
>- actual resources used when rescaling finished (i.e. reaching Executing
>state)
> That would give a clear summary of what was available when deciding to
>rescale and what was used in the end. WDYT?
>
>
>> > I feel like on disk approach (analogously to the exception
>> >history) makes the most sense here. WDYT?
>>
>> Sorry,Matthias, IIUC,. If the storage mechanism here is similar to that of
>> the exception history, then we should choose the DFS approach, such as
>> HDFS. Please correct me if I’m wrong.
>>
>
>I agree that we might want to have this information being stored in DFS.
>That way the information would survive a JobManager failover. It would be
>still nice to have all three options being reflected in the FLIP, though,
>with the pro's and con's having the feature properly documented.
>
>As a side note: There is also FLIP-360 [3] which proposes merging the
>ExecutionGraphInfoStore and the JobResultStore into a single component.
>That would give us a single store for any completed job that could include
>the job's result, its exception history and the rescale history. No need to
>rely on the HistoryServer anymore. But that's out-of-scope for FLIP-495.
>But implementing the JM-local-disk approach and working on FLIP-360 would
>be another option.
>
>[3]
>https://cwiki.apache.org/confluence/display/FLINK/FLIP-360%3A+Merging+the+ExecutionGraphInfoStore+and+the+JobResultStore+into+a+single+component+CompletedJobStore
>
>
>> BTW, the subsequent FLIP content will be maintained in the wiki page, and
>> the version in Google Docs will be deprecated.
>>
>
>A few other items I'd like to point out are the following ones:
>- Is the "Rescale Event" section still out-dated or do we have a different
>understanding of what a rescale operation is? Based on what I pointed out
>in my previous post, I would think that a rescale operation has its
>starting point in the AdaptiveScheduler's Executing state (i.e. when the
>job is running). That's how it is implemented right now. The "Rescale
>Event" section always starts from WaitingForResources, though. Do we
>disagree here?
>
>- Rescale status: I would say that this section needs to be reworked. For
>instance:
>  - Why is a rescale having the STARTING status when the scheduler is in
>StopWithSavepoint state? We shouldn't rescale if the user wants to stop the
>job? Right now, rescale events are ignored in StopWithSavepoint.
>  - The section seems to be out-dated. I already pointed out in my previous
>message that the rescale operation doesn't touch WaitingForResources but
>goes from Restarting straight to CreatingExecutionGraph.
>  - Can the content of this section be visualized in a proper control flow
>diagram, instead? That might help understanding the goal of this section a
>bit more.
>
>- The Rescale ID section:
>  - You're talking about resourceRequirementsEpochID, rescale ID and
>rescale attempt here: The resourceRequirementsEpochID is used as a general
>UUID, the attempt ID is a monotonically increasing number in the scope of a
>single resource requirements update. And the rescale ID is a globally
>monotonically increasing ID? Can you elaborate on the purpose of each of
>the IDs? Why do we need two globally-scoped IDs here?
>  - Can you document what an attempt is? Is a failed attempt defined by the
>outcome of the rescale operation (i.e. the desired parallelism isn't
>reached)?
>
>Generally, it might help to add more diagrams (especially for documenting
>state machines). That might be easier to understand than plain text.
>
>I'm looking forward to your response.
>
>Best,
>Matthias

Re: [DISCUSS] FLIP-495: Support AdaptiveScheduler record and query the rescale history

Posted by Matthias Pohl <ma...@apache.org>.
Thanks Yuepeng for your response. I added my comments to the individual
paragraphs below:


> Thank you very much for the reminding The proposal makes sense to me.
> Additionally, I'd like to confirm whether each rescale cycle/event
> requires a status field, such as FAILED, IGNORED, SUCCESS, PENDING, etc. If
> such state fields are not needed, how do we record that a particular
> rescale request was ignored? Or do we not care about this situation and
> only plan to record successful rescale events?
>

Ignored events (due to the AdaptiveScheduler not being in Executing state)
can be still collected by the AdaptiveScheduler in
updateJobResourceRequirements [1]. That's where the rescale history could
be populated.

Also keep in mind that it's not only the JobResourceRequirements update
that can trigger a rescale. Updating the available resources through
newResourcesAvailable [2] can also initiate a rescale. I don't see this
being clearly laid out in FLIP-495, yet.

[1]
https://github.com/apache/flink/blob/a5258a015d553196d34e082e75ca4ae916addbaf/flink-runtime/src/main/java/org/apache/flink/runtime/scheduler/adaptive/AdaptiveScheduler.java#L1115

[2]
https://github.com/apache/flink/blob/fc0ccf325527c3589d5cd5ae7397b22c22321cec/flink-runtime/src/main/java/org/apache/flink/runtime/scheduler/adaptive/ResourceListener.java#L28

Yes, this could happen during the reserving slots phase, and it's important
> to record this. In my limited read, this is feasible. We can collect the
> exceptions from this phase while gathering the scheduler state history, and
> record the specific information using the previously mentioned exception
> field and comments field, or use failed status  as the final status of the
> rescale event/record, WDYTA?
>

I struggle to follow why we put this in a comment and/or exception field.
Shouldn't we have the information provided in structured fields for each
event? Something like:
- pre-rescale resources
- on-rescale-trigger sufficient resources
- on-rescale-trigger desired resources
- actual resources used when rescaling finished (i.e. reaching Executing
state)
 That would give a clear summary of what was available when deciding to
rescale and what was used in the end. WDYT?


> > I feel like on disk approach (analogously to the exception
> >history) makes the most sense here. WDYT?
>
> Sorry,Matthias, IIUC,. If the storage mechanism here is similar to that of
> the exception history, then we should choose the DFS approach, such as
> HDFS. Please correct me if I’m wrong.
>

I agree that we might want to have this information being stored in DFS.
That way the information would survive a JobManager failover. It would be
still nice to have all three options being reflected in the FLIP, though,
with the pro's and con's having the feature properly documented.

As a side note: There is also FLIP-360 [3] which proposes merging the
ExecutionGraphInfoStore and the JobResultStore into a single component.
That would give us a single store for any completed job that could include
the job's result, its exception history and the rescale history. No need to
rely on the HistoryServer anymore. But that's out-of-scope for FLIP-495.
But implementing the JM-local-disk approach and working on FLIP-360 would
be another option.

[3]
https://cwiki.apache.org/confluence/display/FLINK/FLIP-360%3A+Merging+the+ExecutionGraphInfoStore+and+the+JobResultStore+into+a+single+component+CompletedJobStore


> BTW, the subsequent FLIP content will be maintained in the wiki page, and
> the version in Google Docs will be deprecated.
>

A few other items I'd like to point out are the following ones:
- Is the "Rescale Event" section still out-dated or do we have a different
understanding of what a rescale operation is? Based on what I pointed out
in my previous post, I would think that a rescale operation has its
starting point in the AdaptiveScheduler's Executing state (i.e. when the
job is running). That's how it is implemented right now. The "Rescale
Event" section always starts from WaitingForResources, though. Do we
disagree here?

- Rescale status: I would say that this section needs to be reworked. For
instance:
  - Why is a rescale having the STARTING status when the scheduler is in
StopWithSavepoint state? We shouldn't rescale if the user wants to stop the
job? Right now, rescale events are ignored in StopWithSavepoint.
  - The section seems to be out-dated. I already pointed out in my previous
message that the rescale operation doesn't touch WaitingForResources but
goes from Restarting straight to CreatingExecutionGraph.
  - Can the content of this section be visualized in a proper control flow
diagram, instead? That might help understanding the goal of this section a
bit more.

- The Rescale ID section:
  - You're talking about resourceRequirementsEpochID, rescale ID and
rescale attempt here: The resourceRequirementsEpochID is used as a general
UUID, the attempt ID is a monotonically increasing number in the scope of a
single resource requirements update. And the rescale ID is a globally
monotonically increasing ID? Can you elaborate on the purpose of each of
the IDs? Why do we need two globally-scoped IDs here?
  - Can you document what an attempt is? Is a failed attempt defined by the
outcome of the rescale operation (i.e. the desired parallelism isn't
reached)?

Generally, it might help to add more diagrams (especially for documenting
state machines). That might be easier to understand than plain text.

I'm looking forward to your response.

Best,
Matthias

Re: [DISCUSS] FLIP-495: Support AdaptiveScheduler record and query the rescale history

Posted by Yuepeng Pan <pa...@apache.org>.
Thanks Matthias for the comments.

>- How is the resourceRequirementsEpochID generated?

Please let me have a try on clarifying it.
Each rescale requirement corresponds to a resourceRequirementsEpochID. 
A new value(UUID) will be generated for resourceRequirementsEpochID when the job is first started and each time an update resource requirements request is received from the REST API.

>- Why is the Resource ID section "hidden" (at least, I don't have access)
>in GoogleDocs and not added to the FLIP?

Sorry for the typo. I corrected it in the wiki page.

>- Can you add more details of where this ID is coming from?

Added it~.

>Slots section
>- I was surprised that we have a name for the SlotSharingGroup. Then I
>realized that we have 3 different implementations of the SlotSharingGroup
>in Flink. The one that's used in the scheduler doesn't have the name
>preserved but works with SlotSharingGroupIds. That, we might need to
>consider. If we pass down the name (which might make sense for the UI), we
>still want to expose the SlotSharingGroupId as well, I guess.

Yes, In simple terms, the group name is user-oriented, while the group ID is engine-oriented.
And adding the group name would indeed be friendly to users and easy to use.

>- What about exposing the ResourceProfile of the SlotSharingGroup here as
>Well?

That makes sense to me! And I updated it in wiki FLIP.

>Rescale Event/Rescale status sections
>- I'm not sure about the AdaptiveScheduler state to Rescale event state
>mapping that's included in the FLIP right now: Triggering rescaling only
>happens in the Executing state of the AdaptiveScheduler right now. Waiting
>for resources also happens while the job is running (i.e. Executing state).
>The AdaptiveScheduler will immediately transition from Executing to
>CreatingExecutionGraph state in case of rescaling (WaitingForResources is
>omitted). This was introduced FLIP-472 [3]
>
>- I'm wondering whether we can rely on the StateTransitionManager here
>(which was also introduced with FLIP-472 [3]). That instance is coupled
>with the Executing state (aside from WaitingForResources where it serves a
>different purpose) and holds the information about the rescale trigger
>event (and subsequent ignored rescale trigger events) and when the
>rescaling was actually initiated. There might not be a need to work

Thank you very much for the reminding The proposal makes sense to me. 
Additionally, I'd like to confirm whether each rescale cycle/event requires a status field, such as FAILED, IGNORED, SUCCESS, PENDING, etc. If such state fields are not needed, how do we record that a particular rescale request was ignored? Or do we not care about this situation and only plan to record successful rescale events?

>The first two are the resource configurations the rescale decision is based
>on. The last two are the actual applied resource configurations. Keep in
>mind that the latter two are not necessarily matching the resource
>configurations that were considered when deciding on the rescaling.
>Especially the case where the desired resources were met when rescaling was
>triggered but where task slots are lost while rescaling can have a
>surprising outcome. We might want to have this reflected in the rescale
>Event.

Yes, this could happen during the reserving slots phase, and it's important to record this. In my limited read, this is feasible. We can collect the exceptions from this phase while gathering the scheduler state history, and record the specific information using the previously mentioned exception field and comments field, or use failed status  as the final status of the rescale event/record, WDYTA?

>How/Where to store rescale events section
>- It makes sense to have the rescale event history be stored in the
>AdaptiveScheduler (analogously to what is done for the exception history).
>But can you elaborate a bit more on the different approaches (in-memory, on
>disk, DFS). Each of them have different outcome (in-memory: the history is
>gone as soon as the job reaches a globally-terminal state; on disk: rescale
>history survives the job termination; DFS: rescale history survives a JM
>failover).

Thanks for the comment.
I updated the wiki in the corresponding sections based on the exception history mechanism. Please take a look~

> I feel like on disk approach (analogously to the exception
>history) makes the most sense here. WDYT?

Sorry,Matthias, IIUC,. If the storage mechanism here is similar to that of the exception history, then we should choose the DFS approach, such as HDFS. Please correct me if I’m wrong.

BTW, the subsequent FLIP content will be maintained in the wiki page, and the version in Google Docs will be deprecated.

Thank you.

Best, 
Yuepeng Pan


On 2024/12/18 08:49:32 Matthias Pohl wrote:
> Hi Yuepeng,
> Sorry for not finding the time to respond earlier. I went over FLIP-495 [1]
> and the previous FLIP-487 discussion [2]. Thanks for putting it all
> together in a FLIP. That makes it easier to discuss the next iteration.
> Here are a few comments I have:
> 
> Rescale ID section
> - How is the resourceRequirementsEpochID generated?
> - Why is the Resource ID section "hidden" (at least, I don't have access)
> in GoogleDocs and not added to the FLIP?
> - Can you add more details of where this ID is coming from?
> 
> Slots section
> - I was surprised that we have a name for the SlotSharingGroup. Then I
> realized that we have 3 different implementations of the SlotSharingGroup
> in Flink. The one that's used in the scheduler doesn't have the name
> preserved but works with SlotSharingGroupIds. That, we might need to
> consider. If we pass down the name (which might make sense for the UI), we
> still want to expose the SlotSharingGroupId as well, I guess.
> - What about exposing the ResourceProfile of the SlotSharingGroup here as
> well?
> 
> Rescale Event/Rescale status sections
> - I'm not sure about the AdaptiveScheduler state to Rescale event state
> mapping that's included in the FLIP right now: Triggering rescaling only
> happens in the Executing state of the AdaptiveScheduler right now. Waiting
> for resources also happens while the job is running (i.e. Executing state).
> The AdaptiveScheduler will immediately transition from Executing to
> CreatingExecutionGraph state in case of rescaling (WaitingForResources is
> omitted). This was introduced FLIP-472 [3]
> 
> - I'm wondering whether we can rely on the StateTransitionManager here
> (which was also introduced with FLIP-472 [3]). That instance is coupled
> with the Executing state (aside from WaitingForResources where it serves a
> different purpose) and holds the information about the rescale trigger
> event (and subsequent ignored rescale trigger events) and when the
> rescaling was actually initiated. There might not be a need to work
> 
> - I also want to point out that we have four different notions of resource
> configurations:
>   - Desired resources: The ideal resource configuration that we want to
> achieve for a job if enough Task slots are available (essentially the upper
> bound of the job's parallelism)
>   - Sufficient resources: A minimum resource configuration that the job can
> run on (the lower bound of the job's parallelism)
>   - Current resources: The resource configuration the job runs on before
> rescaling
>   - Follow-up resources
> The first two are the resource configurations the rescale decision is based
> on. The last two are the actual applied resource configurations. Keep in
> mind that the latter two are not necessarily matching the resource
> configurations that were considered when deciding on the rescaling.
> Especially the case where the desired resources were met when rescaling was
> triggered but where task slots are lost while rescaling can have a
> surprising outcome. We might want to have this reflected in the rescale
> event.
> 
> How/Where to store rescale events section
> - It makes sense to have the rescale event history be stored in the
> AdaptiveScheduler (analogously to what is done for the exception history).
> But can you elaborate a bit more on the different approaches (in-memory, on
> disk, DFS). Each of them have different outcome (in-memory: the history is
> gone as soon as the job reaches a globally-terminal state; on disk: rescale
> history survives the job termination; DFS: rescale history survives a JM
> failover). I feel like on disk approach (analogously to the exception
> history) makes the most sense here. WDYT?
> 
> Best,
> Matthias
> 
> [1]
> https://cwiki.apache.org/confluence/display/FLINK/%5BWIP%5D+FLIP-495%3A+Support+AdaptiveScheduler+record+and+query+the+rescale+history
> [2] https://lists.apache.org/thread/f4md4btkf006mxcxf66bng1kfz0rsn8c
> [3]
> https://cwiki.apache.org/confluence/display/FLINK/FLIP-472%3A+Aligning+timeout+logic+in+the+AdaptiveScheduler%27s+WaitingForResources+and+Executing+states
> 
> On Tue, 17 Dec 2024, 16:21 Yuepeng Pan, <pa...@apache.org> wrote:
> 
> > Hi community,
> >
> >
> >
> >
> > We discussed several aspects of FLIP-487[1] 'Show history of rescales in
> > Web UI for AdaptiveScheduler'
> > and received a lot of valuable feedback. Based on the suggestions from the
> > email thread[2],
> > we plan to split the original proposal for FLIP-487[1].
> >
> >
> >
> >
> > The current email thread and the FLIP-495[3] wiki will be used to discuss
> > 'Support AdaptiveScheduler in recording and querying the rescale history',
> > while FLIP-487[1] will primarily focus on displaying-related design content
> >
> >
> >
> >
> > Looking forward to any feedback and opinions on FLIP-495[3].
> >
> >
> >
> >
> > [1]
> > https://cwiki.apache.org/confluence/display/FLINK/%5BWIP%5D+FLIP-487%3A+Show+history+of+rescales+in+Web+UI+for+AdaptiveScheduler
> >
> > [2] https://lists.apache.org/thread/f4md4btkf006mxcxf66bng1kfz0rsn8c
> >
> > [3]
> > https://cwiki.apache.org/confluence/display/FLINK/%5BWIP%5D+FLIP-495%3A+Support+AdaptiveScheduler+record+and+query+the+rescale+history
> >
> >
> >
> >
> > Thank you very much.
> >
> >
> >
> >
> > Best,
> >
> > Regards.
> >
> > Yuepeng Pan
> 

Re: [DISCUSS] FLIP-495: Support AdaptiveScheduler record and query the rescale history

Posted by Matthias Pohl <ma...@apache.org>.
Hi Yuepeng,
Sorry for not finding the time to respond earlier. I went over FLIP-495 [1]
and the previous FLIP-487 discussion [2]. Thanks for putting it all
together in a FLIP. That makes it easier to discuss the next iteration.
Here are a few comments I have:

Rescale ID section
- How is the resourceRequirementsEpochID generated?
- Why is the Resource ID section "hidden" (at least, I don't have access)
in GoogleDocs and not added to the FLIP?
- Can you add more details of where this ID is coming from?

Slots section
- I was surprised that we have a name for the SlotSharingGroup. Then I
realized that we have 3 different implementations of the SlotSharingGroup
in Flink. The one that's used in the scheduler doesn't have the name
preserved but works with SlotSharingGroupIds. That, we might need to
consider. If we pass down the name (which might make sense for the UI), we
still want to expose the SlotSharingGroupId as well, I guess.
- What about exposing the ResourceProfile of the SlotSharingGroup here as
well?

Rescale Event/Rescale status sections
- I'm not sure about the AdaptiveScheduler state to Rescale event state
mapping that's included in the FLIP right now: Triggering rescaling only
happens in the Executing state of the AdaptiveScheduler right now. Waiting
for resources also happens while the job is running (i.e. Executing state).
The AdaptiveScheduler will immediately transition from Executing to
CreatingExecutionGraph state in case of rescaling (WaitingForResources is
omitted). This was introduced FLIP-472 [3]

- I'm wondering whether we can rely on the StateTransitionManager here
(which was also introduced with FLIP-472 [3]). That instance is coupled
with the Executing state (aside from WaitingForResources where it serves a
different purpose) and holds the information about the rescale trigger
event (and subsequent ignored rescale trigger events) and when the
rescaling was actually initiated. There might not be a need to work

- I also want to point out that we have four different notions of resource
configurations:
  - Desired resources: The ideal resource configuration that we want to
achieve for a job if enough Task slots are available (essentially the upper
bound of the job's parallelism)
  - Sufficient resources: A minimum resource configuration that the job can
run on (the lower bound of the job's parallelism)
  - Current resources: The resource configuration the job runs on before
rescaling
  - Follow-up resources
The first two are the resource configurations the rescale decision is based
on. The last two are the actual applied resource configurations. Keep in
mind that the latter two are not necessarily matching the resource
configurations that were considered when deciding on the rescaling.
Especially the case where the desired resources were met when rescaling was
triggered but where task slots are lost while rescaling can have a
surprising outcome. We might want to have this reflected in the rescale
event.

How/Where to store rescale events section
- It makes sense to have the rescale event history be stored in the
AdaptiveScheduler (analogously to what is done for the exception history).
But can you elaborate a bit more on the different approaches (in-memory, on
disk, DFS). Each of them have different outcome (in-memory: the history is
gone as soon as the job reaches a globally-terminal state; on disk: rescale
history survives the job termination; DFS: rescale history survives a JM
failover). I feel like on disk approach (analogously to the exception
history) makes the most sense here. WDYT?

Best,
Matthias

[1]
https://cwiki.apache.org/confluence/display/FLINK/%5BWIP%5D+FLIP-495%3A+Support+AdaptiveScheduler+record+and+query+the+rescale+history
[2] https://lists.apache.org/thread/f4md4btkf006mxcxf66bng1kfz0rsn8c
[3]
https://cwiki.apache.org/confluence/display/FLINK/FLIP-472%3A+Aligning+timeout+logic+in+the+AdaptiveScheduler%27s+WaitingForResources+and+Executing+states

On Tue, 17 Dec 2024, 16:21 Yuepeng Pan, <pa...@apache.org> wrote:

> Hi community,
>
>
>
>
> We discussed several aspects of FLIP-487[1] 'Show history of rescales in
> Web UI for AdaptiveScheduler'
> and received a lot of valuable feedback. Based on the suggestions from the
> email thread[2],
> we plan to split the original proposal for FLIP-487[1].
>
>
>
>
> The current email thread and the FLIP-495[3] wiki will be used to discuss
> 'Support AdaptiveScheduler in recording and querying the rescale history',
> while FLIP-487[1] will primarily focus on displaying-related design content
>
>
>
>
> Looking forward to any feedback and opinions on FLIP-495[3].
>
>
>
>
> [1]
> https://cwiki.apache.org/confluence/display/FLINK/%5BWIP%5D+FLIP-487%3A+Show+history+of+rescales+in+Web+UI+for+AdaptiveScheduler
>
> [2] https://lists.apache.org/thread/f4md4btkf006mxcxf66bng1kfz0rsn8c
>
> [3]
> https://cwiki.apache.org/confluence/display/FLINK/%5BWIP%5D+FLIP-495%3A+Support+AdaptiveScheduler+record+and+query+the+rescale+history
>
>
>
>
> Thank you very much.
>
>
>
>
> Best,
>
> Regards.
>
> Yuepeng Pan

Re: [DISCUSS] FLIP-495: Support AdaptiveScheduler record and query the rescale history

Posted by Yuepeng Pan <hi...@gmail.com>.
Hi, David.

Thank you for your comments. And your suggestion makes sense.

What I originally intended here is something like:
“Provide rescale history and setting information that can be used to
analyze and optimize Adaptive Scheduler parameters.”

Regarding the addition of examples,
my original plan was to address this in the documentation ticket [1].
The documentation will include such scenarios. For example,
if during one or multiple successful scale-up attempts a job still
does not reach the maximum expected resources,
users can look at the WaitingForResources duration from the rescale
history.
Based on this information, they may decide whether to increase the
resource waiting time or related timeout parameters,
and adjust the configuration accordingly.

Hope it helps.

[1] https://issues.apache.org/jira/browse/FLINK-38902

Best,
Yuepeng Pan

David Radley <da...@uk.ibm.com> 于2026年3月6日周五 19:41写道:

> Hi Yuepeng,
> I was looking the motivation section of the flip. It says:
>
>   *   Facilitate users to trace the history of rescale and make rescale
> information more transparent
>   *   Provide users with information on optimizing Adaptive Scheduler
> parameters
>
> I wonder if the second point could read :
>  "Provide users with information that can be used to optimize Adaptive
> Scheduler parameters." Or is it recommending optimizations?
>
> I would find it easier to read to have some motivating examples, of what
> the history might show that could lead the user to decide to optimise. The
> value of the Flip would then be more explicit.
>
>  Kind regards, David.
>
>
>
>
> From: Yuepeng Pan <pa...@apache.org>
> Date: Wednesday, 25 February 2026 at 02:58
> To: dev@flink.apache.org <de...@flink.apache.org>
> Subject: [EXTERNAL] Re: [DISCUSS] FLIP-495: Support AdaptiveScheduler
> record and query the rescale history
>
> Hi, community.
>
> FYI,
> To ensure that the rescale history stored and recorded in FLIP-495 can be
> accessed by external systems/users, we'd plan to release the FLIP-495
> functionality together with at least two sub-tasks[1][2] of FLIP-487[3].
>
> These two sub-tasks will respectively support:
> - retrieving all current rescale history records
> - retrieving the detailed record of a specific rescale by its rescale UUID
>
> [1] https://issues.apache.org/jira/browse/FLINK-38894
> [2] https://issues.apache.org/jira/browse/FLINK-38895
> [3] https://issues.apache.org/jira/browse/FLINK-22258
>
> Best,
> Yuepeng Pan
>
> On 2025/09/18 04:03:22 Yuepeng Pan wrote:
> > Hi, community.
> >
> > FYI:
> > Since the design work of the query interface of rescale history was
> separated into FLIP-487[1] during the discussion, we have therefore changed
> the title of the FLIP to:
> >
> > FLIP-495: Support AdaptiveScheduler record and store the rescale history.
> >
> > [1] https://cwiki.apache.org/confluence/x/vZCMEw
> >
> > Best regards,
> > Yuepeng Pan
> >
> > On 2025/08/19 09:13:22 Yuepeng Pan wrote:
> > > Bumping this thread kindly. Thanks!
> > >
> > > Best,
> > > Yuepeng Pan
> > >
> > >
> > >
> > >
> > > At 2025-08-13 14:52:26, "Yuepeng Pan" <pa...@apache.org> wrote:
> > >
> > > Hi, Matthias,
> > > Thank you  very much for your comments!
> > > I have carefully read your reply and made some changes in the hope of
> making improvements.
> > > Please help take a look.
> > >
> > > For your comments:
> > >
> > > > 1. You mention a few options for when it comes to storing the data
> which is
> > > > good. The FLIP doesn't point out, though, what option you're going
> to go
> > > > for as part of this FLIP (as far as I can see). It would be good to
> only
> > > > outline the option to go for in the FLIP and list the other options
> as
> > > > rejected alternatives (with the pro's and con's). I think it make
> sense to
> > > > go for option 3 (i.e. following what's done for the
> ExecutionGraphInfoStore
> > > > for now). The other options can be considered as a follow-up.
> > >
> > > This is very meaningful. Based on this comment, I have kept option 3
> in its original place and moved the other candidate options to [1].
> > >
> > > > 2. About the terminal states of a rescaling (i.e. IGNORED, FAILED,
> > > > COMPLETED): Can we we clarify in the FLIP under what conditions the
> > > > rescaling transitions into each of the three terminal states?
> > >
> > > Yes, this is a reasonable request for understanding and explaining the
> logic of transitions to terminated states.
> > > A new subsection [2] has been added to address this.
> > >
> > > > 3. The section "The information to record in a rescale event" could
> be
> > > > restructured in four sections (to remove redundancy):
> > > > a) The IDs (Rescale
> > > > ID, resourceRequirementsEpochID,
> subRescaleIdOfResourceRequirementsEpochID):
> > > > What about making these names easier to read: GlobalRescaleID,
> RescaleUUID,
> > > > RescaleAttemptId)
> > > > b) Per-vertex data which includes: JobVertexID, JobVertexName,
> > > > SlotSharingGroupId, the different parallelisms (pre-rescale,
> sufficient,
> > > > desired, post-rescale)
> > > > c) The SlotSharingGroup information: SlotSharingGroupId, name,
> > > > ResourceProfile
> > > > d) Other information: Timestamps of state transitions, etc. as laid
> out in
> > > > the FLIP already
> > >
> > > That makes sense to me. Please check [3] for the latest updates in
> this part.
> > >
> > > > 4. The FLIP doesn't explain how the data is passed through the
> > > > AdaptiveScheduler states. We should be handling some kind of
> > > > RescaleSnapshot that is passed through the different states and
> updated and
> > > > its final state is stored somewhere within AdaptiveScheduler in the
> end, I
> > > > guess. Can we clarify that in the FLIP?
> > >
> > > Indeed — this was missing in the original FLIP. To address this, I
> have added [4], which focuses on describing how a Rescale is represented,
> > > and how we can quickly pass and maintain the Rescale history.
> > >
> > > > 5. You mention the config parameters for the cache in the public
> interface
> > > > section. But there's no mentioning of any caching and how that is
> used
> > > > within the FLIP.
> > >
> > > Sorry for the rough description in the previous version.
> > > Since this part belongs to the REST API acceleration mechanism for
> rescaling, and Option 6 seems reasonable to me,
> > > I plan to add it to FLIP-487 once the design of FLIP-495 has reached
> consensus.
> > > Of course, if needed, I'd be happy to clarify the usage and purpose of
> this parameter in the current email thread.
> > >
> > > > 6. The REST endpoint is probably better suited in FLIP-487. FLIP-495
> should
> > > > be about the actual implementation details and how the data is stored
> > > > internally whereas FLIP-487 is about exposing the information to the
> > > > outside through the REST API and the Flink UI. That would be a way to
> > > > decrease the scope of FLIP-495. WDYT?
> > >
> > > That sounds nice to me. Therefore, I have moved all REST API–related
> changes to FLIP-487.
> > > BTW, to avoid repetitive changes in FLIP-487, I'll start organizing
> FLIP-487 after FLIP-495 has been finalized.
> > >
> > > Looking forward to your next review!
> > >
> > > [1]
> https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=334760525#FLIP495:SupportAdaptiveSchedulerrecordandquerytherescalehistory-Aboutrescaleeventsstorage.1
> > > [2]
> https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=334760525#FLIP495:SupportAdaptiveSchedulerrecordandquerytherescalehistory-ThemainscenarioswhereRescalestatusswitchestoterminated
> > > [3]
> https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=334760525#FLIP495:SupportAdaptiveSchedulerrecordandquerytherescalehistory-Theinformationtorecordinarescaleevent
> > > [4]
> https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=334760525#FLIP495:SupportAdaptiveSchedulerrecordandquerytherescalehistory-InternalInterfaces
> > >
> > >
> > >
> > >
> > >
> > > Best regards,
> > > Yuepeng Pan
> > >
> > >
> > >
> > >
> > >
> > >
> > >
> > >
> > >
> > >
> > >
> > > At 2025-08-10 23:54:37, "Matthias Pohl" <ma...@apache.org> wrote:
> > > >Hi Yuepeng,
> > > >thanks for reminding me of this FLIP. I went over it and have a few
> items
> > > >which we might need to address before we can actually finalize the
> vote:
> > > >
> > > >1. You mention a few options for when it comes to storing the data
> which is
> > > >good. The FLIP doesn't point out, though, what option you're going to
> go
> > > >for as part of this FLIP (as far as I can see). It would be good to
> only
> > > >outline the option to go for in the FLIP and list the other options as
> > > >rejected alternatives (with the pro's and con's). I think it make
> sense to
> > > >go for option 3 (i.e. following what's done for the
> ExecutionGraphInfoStore
> > > >for now). The other options can be considered as a follow-up.
> > > >2. About the terminal states of a rescaling (i.e. IGNORED, FAILED,
> > > >COMPLETED): Can we we clarify in the FLIP under what conditions the
> > > >rescaling transitions into each of the three terminal states?
> > > >3. The section "The information to record in a rescale event" could be
> > > >restructured in four sections (to remove redundancy):
> > > > a) The IDs (Rescale
> > > >ID, resourceRequirementsEpochID,
> subRescaleIdOfResourceRequirementsEpochID):
> > > >What about making these names easier to read: GlobalRescaleID,
> RescaleUUID,
> > > >RescaleAttemptId)
> > > > b) Per-vertex data which includes: JobVertexID, JobVertexName,
> > > >SlotSharingGroupId, the different parallelisms (pre-rescale,
> sufficient,
> > > >desired, post-rescale)
> > > > c) The SlotSharingGroup information: SlotSharingGroupId, name,
> > > >ResourceProfile
> > > > d) Other information: Timestamps of state transitions, etc. as laid
> out in
> > > >the FLIP already
> > > >4. The FLIP doesn't explain how the data is passed through the
> > > >AdaptiveScheduler states. We should be handling some kind of
> > > >RescaleSnapshot that is passed through the different states and
> updated and
> > > >its final state is stored somewhere within AdaptiveScheduler in the
> end, I
> > > >guess. Can we clarify that in the FLIP?
> > > >5. You mention the config parameters for the cache in the public
> interface
> > > >section. But there's no mentioning of any caching and how that is used
> > > >within the FLIP.
> > > >6. The REST endpoint is probably better suited in FLIP-487. FLIP-495
> should
> > > >be about the actual implementation details and how the data is stored
> > > >internally whereas FLIP-487 is about exposing the information to the
> > > >outside through the REST API and the Flink UI. That would be a way to
> > > >decrease the scope of FLIP-495. WDYT?
> > > >
> > > >Best,
> > > >Matthias
> > > >
> > > >
> > > >On Mon, Mar 24, 2025 at 11:37 AM Yuepeng Pan <pa...@apache.org>
> wrote:
> > > >
> > > >> Hi, Community,
> > > >>
> > > >> There haven’t been any further responses to this email over the
> past few
> > > >> days.
> > > >> I'd like to initiate a vote on the current proposal[1] in the next
> few
> > > >> days.
> > > >> Please rest assured that I’m proceeding cautiously and not rushing
> the
> > > >> process.
> > > >> If there are any concerns about this FLIP-495[1],
> > > >> I will gladly pause and make the adjustments.
> > > >>
> > > >> Best regards,
> > > >> Yuepeng Pan
> > > >>
> > > >> [1]
> > > >>
> https://cwiki.apache.org/confluence/display/FLINK/FLIP-495%3A+Support+AdaptiveScheduler+record+and+query+the+rescale+history
> > > >>
> > > >>
> > > >> On 2024/12/17 15:18:45 Yuepeng Pan wrote:
> > > >> > Hi community,
> > > >> >
> > > >> >
> > > >> >
> > > >> >
> > > >> > We discussed several aspects of FLIP-487[1] 'Show history of
> rescales in
> > > >> Web UI for AdaptiveScheduler'
> > > >> > and received a lot of valuable feedback. Based on the suggestions
> from
> > > >> the email thread[2],
> > > >> > we plan to split the original proposal for FLIP-487[1].
> > > >> >
> > > >> >
> > > >> >
> > > >> >
> > > >> > The current email thread and the FLIP-495[3] wiki will be used to
> > > >> discuss 'Support AdaptiveScheduler in recording and querying the
> rescale
> > > >> history',
> > > >> > while FLIP-487[1] will primarily focus on displaying-related
> design
> > > >> content
> > > >> >
> > > >> >
> > > >> >
> > > >> >
> > > >> > Looking forward to any feedback and opinions on FLIP-495[3].
> > > >> >
> > > >> >
> > > >> >
> > > >> >
> > > >> > [1]
> > > >>
> https://cwiki.apache.org/confluence/display/FLINK/%5BWIP%5D+FLIP-487%3A+Show+history+of+rescales+in+Web+UI+for+AdaptiveScheduler
> > > >> >
> > > >> > [2]
> https://lists.apache.org/thread/f4md4btkf006mxcxf66bng1kfz0rsn8c
> > > >> >
> > > >> > [3]
> > > >>
> https://cwiki.apache.org/confluence/display/FLINK/%5BWIP%5D+FLIP-495%3A+Support+AdaptiveScheduler+record+and+query+the+rescale+history
> > > >> >
> > > >> >
> > > >> >
> > > >> >
> > > >> > Thank you very much.
> > > >> >
> > > >> >
> > > >> >
> > > >> >
> > > >> > Best,
> > > >> >
> > > >> > Regards.
> > > >> >
> > > >> > Yuepeng Pan
> > > >>
> > >
> >
>
> Unless otherwise stated above:
>
> IBM United Kingdom Limited
> Registered in England and Wales with number 741598
> Registered office: Building C, IBM Hursley Office, Hursley Park Road,
> Winchester, Hampshire SO21 2JN
>

RE: [DISCUSS] FLIP-495: Support AdaptiveScheduler record and query the rescale history

Posted by David Radley <da...@uk.ibm.com>.
Hi Yuepeng,
I was looking the motivation section of the flip. It says:

  *   Facilitate users to trace the history of rescale and make rescale information more transparent
  *   Provide users with information on optimizing Adaptive Scheduler parameters

I wonder if the second point could read :
 "Provide users with information that can be used to optimize Adaptive Scheduler parameters." Or is it recommending optimizations?

I would find it easier to read to have some motivating examples, of what the history might show that could lead the user to decide to optimise. The value of the Flip would then be more explicit.

 Kind regards, David.




From: Yuepeng Pan <pa...@apache.org>
Date: Wednesday, 25 February 2026 at 02:58
To: dev@flink.apache.org <de...@flink.apache.org>
Subject: [EXTERNAL] Re: [DISCUSS] FLIP-495: Support AdaptiveScheduler record and query the rescale history

Hi, community.

FYI,
To ensure that the rescale history stored and recorded in FLIP-495 can be accessed by external systems/users, we'd plan to release the FLIP-495 functionality together with at least two sub-tasks[1][2] of FLIP-487[3].

These two sub-tasks will respectively support:
- retrieving all current rescale history records
- retrieving the detailed record of a specific rescale by its rescale UUID

[1] https://issues.apache.org/jira/browse/FLINK-38894
[2] https://issues.apache.org/jira/browse/FLINK-38895
[3] https://issues.apache.org/jira/browse/FLINK-22258

Best,
Yuepeng Pan

On 2025/09/18 04:03:22 Yuepeng Pan wrote:
> Hi, community.
>
> FYI:
> Since the design work of the query interface of rescale history was separated into FLIP-487[1] during the discussion, we have therefore changed the title of the FLIP to:
>
> FLIP-495: Support AdaptiveScheduler record and store the rescale history.
>
> [1] https://cwiki.apache.org/confluence/x/vZCMEw
>
> Best regards,
> Yuepeng Pan
>
> On 2025/08/19 09:13:22 Yuepeng Pan wrote:
> > Bumping this thread kindly. Thanks!
> >
> > Best,
> > Yuepeng Pan
> >
> >
> >
> >
> > At 2025-08-13 14:52:26, "Yuepeng Pan" <pa...@apache.org> wrote:
> >
> > Hi, Matthias,
> > Thank you  very much for your comments!
> > I have carefully read your reply and made some changes in the hope of making improvements.
> > Please help take a look.
> >
> > For your comments:
> >
> > > 1. You mention a few options for when it comes to storing the data which is
> > > good. The FLIP doesn't point out, though, what option you're going to go
> > > for as part of this FLIP (as far as I can see). It would be good to only
> > > outline the option to go for in the FLIP and list the other options as
> > > rejected alternatives (with the pro's and con's). I think it make sense to
> > > go for option 3 (i.e. following what's done for the ExecutionGraphInfoStore
> > > for now). The other options can be considered as a follow-up.
> >
> > This is very meaningful. Based on this comment, I have kept option 3 in its original place and moved the other candidate options to [1].
> >
> > > 2. About the terminal states of a rescaling (i.e. IGNORED, FAILED,
> > > COMPLETED): Can we we clarify in the FLIP under what conditions the
> > > rescaling transitions into each of the three terminal states?
> >
> > Yes, this is a reasonable request for understanding and explaining the logic of transitions to terminated states.
> > A new subsection [2] has been added to address this.
> >
> > > 3. The section "The information to record in a rescale event" could be
> > > restructured in four sections (to remove redundancy):
> > > a) The IDs (Rescale
> > > ID, resourceRequirementsEpochID, subRescaleIdOfResourceRequirementsEpochID):
> > > What about making these names easier to read: GlobalRescaleID, RescaleUUID,
> > > RescaleAttemptId)
> > > b) Per-vertex data which includes: JobVertexID, JobVertexName,
> > > SlotSharingGroupId, the different parallelisms (pre-rescale, sufficient,
> > > desired, post-rescale)
> > > c) The SlotSharingGroup information: SlotSharingGroupId, name,
> > > ResourceProfile
> > > d) Other information: Timestamps of state transitions, etc. as laid out in
> > > the FLIP already
> >
> > That makes sense to me. Please check [3] for the latest updates in this part.
> >
> > > 4. The FLIP doesn't explain how the data is passed through the
> > > AdaptiveScheduler states. We should be handling some kind of
> > > RescaleSnapshot that is passed through the different states and updated and
> > > its final state is stored somewhere within AdaptiveScheduler in the end, I
> > > guess. Can we clarify that in the FLIP?
> >
> > Indeed — this was missing in the original FLIP. To address this, I have added [4], which focuses on describing how a Rescale is represented,
> > and how we can quickly pass and maintain the Rescale history.
> >
> > > 5. You mention the config parameters for the cache in the public interface
> > > section. But there's no mentioning of any caching and how that is used
> > > within the FLIP.
> >
> > Sorry for the rough description in the previous version.
> > Since this part belongs to the REST API acceleration mechanism for rescaling, and Option 6 seems reasonable to me,
> > I plan to add it to FLIP-487 once the design of FLIP-495 has reached consensus.
> > Of course, if needed, I'd be happy to clarify the usage and purpose of this parameter in the current email thread.
> >
> > > 6. The REST endpoint is probably better suited in FLIP-487. FLIP-495 should
> > > be about the actual implementation details and how the data is stored
> > > internally whereas FLIP-487 is about exposing the information to the
> > > outside through the REST API and the Flink UI. That would be a way to
> > > decrease the scope of FLIP-495. WDYT?
> >
> > That sounds nice to me. Therefore, I have moved all REST API–related changes to FLIP-487.
> > BTW, to avoid repetitive changes in FLIP-487, I'll start organizing FLIP-487 after FLIP-495 has been finalized.
> >
> > Looking forward to your next review!
> >
> > [1]https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=334760525#FLIP495:SupportAdaptiveSchedulerrecordandquerytherescalehistory-Aboutrescaleeventsstorage.1
> > [2]https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=334760525#FLIP495:SupportAdaptiveSchedulerrecordandquerytherescalehistory-ThemainscenarioswhereRescalestatusswitchestoterminated
> > [3]https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=334760525#FLIP495:SupportAdaptiveSchedulerrecordandquerytherescalehistory-Theinformationtorecordinarescaleevent
> > [4]https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=334760525#FLIP495:SupportAdaptiveSchedulerrecordandquerytherescalehistory-InternalInterfaces
> >
> >
> >
> >
> >
> > Best regards,
> > Yuepeng Pan
> >
> >
> >
> >
> >
> >
> >
> >
> >
> >
> >
> > At 2025-08-10 23:54:37, "Matthias Pohl" <ma...@apache.org> wrote:
> > >Hi Yuepeng,
> > >thanks for reminding me of this FLIP. I went over it and have a few items
> > >which we might need to address before we can actually finalize the vote:
> > >
> > >1. You mention a few options for when it comes to storing the data which is
> > >good. The FLIP doesn't point out, though, what option you're going to go
> > >for as part of this FLIP (as far as I can see). It would be good to only
> > >outline the option to go for in the FLIP and list the other options as
> > >rejected alternatives (with the pro's and con's). I think it make sense to
> > >go for option 3 (i.e. following what's done for the ExecutionGraphInfoStore
> > >for now). The other options can be considered as a follow-up.
> > >2. About the terminal states of a rescaling (i.e. IGNORED, FAILED,
> > >COMPLETED): Can we we clarify in the FLIP under what conditions the
> > >rescaling transitions into each of the three terminal states?
> > >3. The section "The information to record in a rescale event" could be
> > >restructured in four sections (to remove redundancy):
> > > a) The IDs (Rescale
> > >ID, resourceRequirementsEpochID, subRescaleIdOfResourceRequirementsEpochID):
> > >What about making these names easier to read: GlobalRescaleID, RescaleUUID,
> > >RescaleAttemptId)
> > > b) Per-vertex data which includes: JobVertexID, JobVertexName,
> > >SlotSharingGroupId, the different parallelisms (pre-rescale, sufficient,
> > >desired, post-rescale)
> > > c) The SlotSharingGroup information: SlotSharingGroupId, name,
> > >ResourceProfile
> > > d) Other information: Timestamps of state transitions, etc. as laid out in
> > >the FLIP already
> > >4. The FLIP doesn't explain how the data is passed through the
> > >AdaptiveScheduler states. We should be handling some kind of
> > >RescaleSnapshot that is passed through the different states and updated and
> > >its final state is stored somewhere within AdaptiveScheduler in the end, I
> > >guess. Can we clarify that in the FLIP?
> > >5. You mention the config parameters for the cache in the public interface
> > >section. But there's no mentioning of any caching and how that is used
> > >within the FLIP.
> > >6. The REST endpoint is probably better suited in FLIP-487. FLIP-495 should
> > >be about the actual implementation details and how the data is stored
> > >internally whereas FLIP-487 is about exposing the information to the
> > >outside through the REST API and the Flink UI. That would be a way to
> > >decrease the scope of FLIP-495. WDYT?
> > >
> > >Best,
> > >Matthias
> > >
> > >
> > >On Mon, Mar 24, 2025 at 11:37 AM Yuepeng Pan <pa...@apache.org> wrote:
> > >
> > >> Hi, Community,
> > >>
> > >> There haven’t been any further responses to this email over the past few
> > >> days.
> > >> I'd like to initiate a vote on the current proposal[1] in the next few
> > >> days.
> > >> Please rest assured that I’m proceeding cautiously and not rushing the
> > >> process.
> > >> If there are any concerns about this FLIP-495[1],
> > >> I will gladly pause and make the adjustments.
> > >>
> > >> Best regards,
> > >> Yuepeng Pan
> > >>
> > >> [1]
> > >> https://cwiki.apache.org/confluence/display/FLINK/FLIP-495%3A+Support+AdaptiveScheduler+record+and+query+the+rescale+history
> > >>
> > >>
> > >> On 2024/12/17 15:18:45 Yuepeng Pan wrote:
> > >> > Hi community,
> > >> >
> > >> >
> > >> >
> > >> >
> > >> > We discussed several aspects of FLIP-487[1] 'Show history of rescales in
> > >> Web UI for AdaptiveScheduler'
> > >> > and received a lot of valuable feedback. Based on the suggestions from
> > >> the email thread[2],
> > >> > we plan to split the original proposal for FLIP-487[1].
> > >> >
> > >> >
> > >> >
> > >> >
> > >> > The current email thread and the FLIP-495[3] wiki will be used to
> > >> discuss 'Support AdaptiveScheduler in recording and querying the rescale
> > >> history',
> > >> > while FLIP-487[1] will primarily focus on displaying-related design
> > >> content
> > >> >
> > >> >
> > >> >
> > >> >
> > >> > Looking forward to any feedback and opinions on FLIP-495[3].
> > >> >
> > >> >
> > >> >
> > >> >
> > >> > [1]
> > >> https://cwiki.apache.org/confluence/display/FLINK/%5BWIP%5D+FLIP-487%3A+Show+history+of+rescales+in+Web+UI+for+AdaptiveScheduler
> > >> >
> > >> > [2] https://lists.apache.org/thread/f4md4btkf006mxcxf66bng1kfz0rsn8c
> > >> >
> > >> > [3]
> > >> https://cwiki.apache.org/confluence/display/FLINK/%5BWIP%5D+FLIP-495%3A+Support+AdaptiveScheduler+record+and+query+the+rescale+history
> > >> >
> > >> >
> > >> >
> > >> >
> > >> > Thank you very much.
> > >> >
> > >> >
> > >> >
> > >> >
> > >> > Best,
> > >> >
> > >> > Regards.
> > >> >
> > >> > Yuepeng Pan
> > >>
> >
>

Unless otherwise stated above:

IBM United Kingdom Limited
Registered in England and Wales with number 741598
Registered office: Building C, IBM Hursley Office, Hursley Park Road, Winchester, Hampshire SO21 2JN

Re: [DISCUSS] FLIP-495: Support AdaptiveScheduler record and query the rescale history

Posted by Yuepeng Pan <pa...@apache.org>.
Hi, community.

FYI,
To ensure that the rescale history stored and recorded in FLIP-495 can be accessed by external systems/users, we'd plan to release the FLIP-495 functionality together with at least two sub-tasks[1][2] of FLIP-487[3].

These two sub-tasks will respectively support:
- retrieving all current rescale history records
- retrieving the detailed record of a specific rescale by its rescale UUID

[1] https://issues.apache.org/jira/browse/FLINK-38894
[2] https://issues.apache.org/jira/browse/FLINK-38895
[3] https://issues.apache.org/jira/browse/FLINK-22258

Best, 
Yuepeng Pan

On 2025/09/18 04:03:22 Yuepeng Pan wrote:
> Hi, community.
> 
> FYI:
> Since the design work of the query interface of rescale history was separated into FLIP-487[1] during the discussion, we have therefore changed the title of the FLIP to:
> 
> FLIP-495: Support AdaptiveScheduler record and store the rescale history.
> 
> [1] https://cwiki.apache.org/confluence/x/vZCMEw
> 
> Best regards,
> Yuepeng Pan
> 
> On 2025/08/19 09:13:22 Yuepeng Pan wrote:
> > Bumping this thread kindly. Thanks!
> > 
> > Best,
> > Yuepeng Pan 
> > 
> > 
> > 
> > 
> > At 2025-08-13 14:52:26, "Yuepeng Pan" <pa...@apache.org> wrote:
> > 
> > Hi, Matthias,
> > Thank you  very much for your comments!
> > I have carefully read your reply and made some changes in the hope of making improvements.
> > Please help take a look.
> > 
> > For your comments:
> > 
> > > 1. You mention a few options for when it comes to storing the data which is
> > > good. The FLIP doesn't point out, though, what option you're going to go
> > > for as part of this FLIP (as far as I can see). It would be good to only
> > > outline the option to go for in the FLIP and list the other options as
> > > rejected alternatives (with the pro's and con's). I think it make sense to
> > > go for option 3 (i.e. following what's done for the ExecutionGraphInfoStore
> > > for now). The other options can be considered as a follow-up.
> > 
> > This is very meaningful. Based on this comment, I have kept option 3 in its original place and moved the other candidate options to [1].
> > 
> > > 2. About the terminal states of a rescaling (i.e. IGNORED, FAILED,
> > > COMPLETED): Can we we clarify in the FLIP under what conditions the
> > > rescaling transitions into each of the three terminal states?
> > 
> > Yes, this is a reasonable request for understanding and explaining the logic of transitions to terminated states.
> > A new subsection [2] has been added to address this.
> > 
> > > 3. The section "The information to record in a rescale event" could be
> > > restructured in four sections (to remove redundancy):
> > > a) The IDs (Rescale
> > > ID, resourceRequirementsEpochID, subRescaleIdOfResourceRequirementsEpochID):
> > > What about making these names easier to read: GlobalRescaleID, RescaleUUID,
> > > RescaleAttemptId)
> > > b) Per-vertex data which includes: JobVertexID, JobVertexName,
> > > SlotSharingGroupId, the different parallelisms (pre-rescale, sufficient,
> > > desired, post-rescale)
> > > c) The SlotSharingGroup information: SlotSharingGroupId, name,
> > > ResourceProfile
> > > d) Other information: Timestamps of state transitions, etc. as laid out in
> > > the FLIP already
> > 
> > That makes sense to me. Please check [3] for the latest updates in this part.
> > 
> > > 4. The FLIP doesn't explain how the data is passed through the
> > > AdaptiveScheduler states. We should be handling some kind of
> > > RescaleSnapshot that is passed through the different states and updated and
> > > its final state is stored somewhere within AdaptiveScheduler in the end, I
> > > guess. Can we clarify that in the FLIP?
> > 
> > Indeed — this was missing in the original FLIP. To address this, I have added [4], which focuses on describing how a Rescale is represented,
> > and how we can quickly pass and maintain the Rescale history.
> > 
> > > 5. You mention the config parameters for the cache in the public interface
> > > section. But there's no mentioning of any caching and how that is used
> > > within the FLIP.
> > 
> > Sorry for the rough description in the previous version.
> > Since this part belongs to the REST API acceleration mechanism for rescaling, and Option 6 seems reasonable to me,
> > I plan to add it to FLIP-487 once the design of FLIP-495 has reached consensus.
> > Of course, if needed, I'd be happy to clarify the usage and purpose of this parameter in the current email thread.
> > 
> > > 6. The REST endpoint is probably better suited in FLIP-487. FLIP-495 should
> > > be about the actual implementation details and how the data is stored
> > > internally whereas FLIP-487 is about exposing the information to the
> > > outside through the REST API and the Flink UI. That would be a way to
> > > decrease the scope of FLIP-495. WDYT?
> > 
> > That sounds nice to me. Therefore, I have moved all REST API–related changes to FLIP-487. 
> > BTW, to avoid repetitive changes in FLIP-487, I'll start organizing FLIP-487 after FLIP-495 has been finalized.
> > 
> > Looking forward to your next review!
> > 
> > [1]https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=334760525#FLIP495:SupportAdaptiveSchedulerrecordandquerytherescalehistory-Aboutrescaleeventsstorage.1
> > [2]https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=334760525#FLIP495:SupportAdaptiveSchedulerrecordandquerytherescalehistory-ThemainscenarioswhereRescalestatusswitchestoterminated
> > [3]https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=334760525#FLIP495:SupportAdaptiveSchedulerrecordandquerytherescalehistory-Theinformationtorecordinarescaleevent
> > [4]https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=334760525#FLIP495:SupportAdaptiveSchedulerrecordandquerytherescalehistory-InternalInterfaces
> > 
> > 
> > 
> > 
> > 
> > Best regards,
> > Yuepeng Pan
> > 
> > 
> > 
> > 
> > 
> > 
> > 
> > 
> > 
> > 
> > 
> > At 2025-08-10 23:54:37, "Matthias Pohl" <ma...@apache.org> wrote:
> > >Hi Yuepeng,
> > >thanks for reminding me of this FLIP. I went over it and have a few items
> > >which we might need to address before we can actually finalize the vote:
> > >
> > >1. You mention a few options for when it comes to storing the data which is
> > >good. The FLIP doesn't point out, though, what option you're going to go
> > >for as part of this FLIP (as far as I can see). It would be good to only
> > >outline the option to go for in the FLIP and list the other options as
> > >rejected alternatives (with the pro's and con's). I think it make sense to
> > >go for option 3 (i.e. following what's done for the ExecutionGraphInfoStore
> > >for now). The other options can be considered as a follow-up.
> > >2. About the terminal states of a rescaling (i.e. IGNORED, FAILED,
> > >COMPLETED): Can we we clarify in the FLIP under what conditions the
> > >rescaling transitions into each of the three terminal states?
> > >3. The section "The information to record in a rescale event" could be
> > >restructured in four sections (to remove redundancy):
> > > a) The IDs (Rescale
> > >ID, resourceRequirementsEpochID, subRescaleIdOfResourceRequirementsEpochID):
> > >What about making these names easier to read: GlobalRescaleID, RescaleUUID,
> > >RescaleAttemptId)
> > > b) Per-vertex data which includes: JobVertexID, JobVertexName,
> > >SlotSharingGroupId, the different parallelisms (pre-rescale, sufficient,
> > >desired, post-rescale)
> > > c) The SlotSharingGroup information: SlotSharingGroupId, name,
> > >ResourceProfile
> > > d) Other information: Timestamps of state transitions, etc. as laid out in
> > >the FLIP already
> > >4. The FLIP doesn't explain how the data is passed through the
> > >AdaptiveScheduler states. We should be handling some kind of
> > >RescaleSnapshot that is passed through the different states and updated and
> > >its final state is stored somewhere within AdaptiveScheduler in the end, I
> > >guess. Can we clarify that in the FLIP?
> > >5. You mention the config parameters for the cache in the public interface
> > >section. But there's no mentioning of any caching and how that is used
> > >within the FLIP.
> > >6. The REST endpoint is probably better suited in FLIP-487. FLIP-495 should
> > >be about the actual implementation details and how the data is stored
> > >internally whereas FLIP-487 is about exposing the information to the
> > >outside through the REST API and the Flink UI. That would be a way to
> > >decrease the scope of FLIP-495. WDYT?
> > >
> > >Best,
> > >Matthias
> > >
> > >
> > >On Mon, Mar 24, 2025 at 11:37 AM Yuepeng Pan <pa...@apache.org> wrote:
> > >
> > >> Hi, Community,
> > >>
> > >> There haven’t been any further responses to this email over the past few
> > >> days.
> > >> I'd like to initiate a vote on the current proposal[1] in the next few
> > >> days.
> > >> Please rest assured that I’m proceeding cautiously and not rushing the
> > >> process.
> > >> If there are any concerns about this FLIP-495[1],
> > >> I will gladly pause and make the adjustments.
> > >>
> > >> Best regards,
> > >> Yuepeng Pan
> > >>
> > >> [1]
> > >> https://cwiki.apache.org/confluence/display/FLINK/FLIP-495%3A+Support+AdaptiveScheduler+record+and+query+the+rescale+history
> > >>
> > >>
> > >> On 2024/12/17 15:18:45 Yuepeng Pan wrote:
> > >> > Hi community,
> > >> >
> > >> >
> > >> >
> > >> >
> > >> > We discussed several aspects of FLIP-487[1] 'Show history of rescales in
> > >> Web UI for AdaptiveScheduler'
> > >> > and received a lot of valuable feedback. Based on the suggestions from
> > >> the email thread[2],
> > >> > we plan to split the original proposal for FLIP-487[1].
> > >> >
> > >> >
> > >> >
> > >> >
> > >> > The current email thread and the FLIP-495[3] wiki will be used to
> > >> discuss 'Support AdaptiveScheduler in recording and querying the rescale
> > >> history',
> > >> > while FLIP-487[1] will primarily focus on displaying-related design
> > >> content
> > >> >
> > >> >
> > >> >
> > >> >
> > >> > Looking forward to any feedback and opinions on FLIP-495[3].
> > >> >
> > >> >
> > >> >
> > >> >
> > >> > [1]
> > >> https://cwiki.apache.org/confluence/display/FLINK/%5BWIP%5D+FLIP-487%3A+Show+history+of+rescales+in+Web+UI+for+AdaptiveScheduler
> > >> >
> > >> > [2] https://lists.apache.org/thread/f4md4btkf006mxcxf66bng1kfz0rsn8c
> > >> >
> > >> > [3]
> > >> https://cwiki.apache.org/confluence/display/FLINK/%5BWIP%5D+FLIP-495%3A+Support+AdaptiveScheduler+record+and+query+the+rescale+history
> > >> >
> > >> >
> > >> >
> > >> >
> > >> > Thank you very much.
> > >> >
> > >> >
> > >> >
> > >> >
> > >> > Best,
> > >> >
> > >> > Regards.
> > >> >
> > >> > Yuepeng Pan
> > >>
> > 
> 

Re: [DISCUSS] FLIP-495: Support AdaptiveScheduler record and query the rescale history

Posted by Yuepeng Pan <pa...@apache.org>.
Hi, community.

FYI:
Since the design work of the query interface of rescale history was separated into FLIP-487[1] during the discussion, we have therefore changed the title of the FLIP to:

FLIP-495: Support AdaptiveScheduler record and store the rescale history.

[1] https://cwiki.apache.org/confluence/x/vZCMEw

Best regards,
Yuepeng Pan

On 2025/08/19 09:13:22 Yuepeng Pan wrote:
> Bumping this thread kindly. Thanks!
> 
> Best,
> Yuepeng Pan 
> 
> 
> 
> 
> At 2025-08-13 14:52:26, "Yuepeng Pan" <pa...@apache.org> wrote:
> 
> Hi, Matthias,
> Thank you  very much for your comments!
> I have carefully read your reply and made some changes in the hope of making improvements.
> Please help take a look.
> 
> For your comments:
> 
> > 1. You mention a few options for when it comes to storing the data which is
> > good. The FLIP doesn't point out, though, what option you're going to go
> > for as part of this FLIP (as far as I can see). It would be good to only
> > outline the option to go for in the FLIP and list the other options as
> > rejected alternatives (with the pro's and con's). I think it make sense to
> > go for option 3 (i.e. following what's done for the ExecutionGraphInfoStore
> > for now). The other options can be considered as a follow-up.
> 
> This is very meaningful. Based on this comment, I have kept option 3 in its original place and moved the other candidate options to [1].
> 
> > 2. About the terminal states of a rescaling (i.e. IGNORED, FAILED,
> > COMPLETED): Can we we clarify in the FLIP under what conditions the
> > rescaling transitions into each of the three terminal states?
> 
> Yes, this is a reasonable request for understanding and explaining the logic of transitions to terminated states.
> A new subsection [2] has been added to address this.
> 
> > 3. The section "The information to record in a rescale event" could be
> > restructured in four sections (to remove redundancy):
> > a) The IDs (Rescale
> > ID, resourceRequirementsEpochID, subRescaleIdOfResourceRequirementsEpochID):
> > What about making these names easier to read: GlobalRescaleID, RescaleUUID,
> > RescaleAttemptId)
> > b) Per-vertex data which includes: JobVertexID, JobVertexName,
> > SlotSharingGroupId, the different parallelisms (pre-rescale, sufficient,
> > desired, post-rescale)
> > c) The SlotSharingGroup information: SlotSharingGroupId, name,
> > ResourceProfile
> > d) Other information: Timestamps of state transitions, etc. as laid out in
> > the FLIP already
> 
> That makes sense to me. Please check [3] for the latest updates in this part.
> 
> > 4. The FLIP doesn't explain how the data is passed through the
> > AdaptiveScheduler states. We should be handling some kind of
> > RescaleSnapshot that is passed through the different states and updated and
> > its final state is stored somewhere within AdaptiveScheduler in the end, I
> > guess. Can we clarify that in the FLIP?
> 
> Indeed — this was missing in the original FLIP. To address this, I have added [4], which focuses on describing how a Rescale is represented,
> and how we can quickly pass and maintain the Rescale history.
> 
> > 5. You mention the config parameters for the cache in the public interface
> > section. But there's no mentioning of any caching and how that is used
> > within the FLIP.
> 
> Sorry for the rough description in the previous version.
> Since this part belongs to the REST API acceleration mechanism for rescaling, and Option 6 seems reasonable to me,
> I plan to add it to FLIP-487 once the design of FLIP-495 has reached consensus.
> Of course, if needed, I'd be happy to clarify the usage and purpose of this parameter in the current email thread.
> 
> > 6. The REST endpoint is probably better suited in FLIP-487. FLIP-495 should
> > be about the actual implementation details and how the data is stored
> > internally whereas FLIP-487 is about exposing the information to the
> > outside through the REST API and the Flink UI. That would be a way to
> > decrease the scope of FLIP-495. WDYT?
> 
> That sounds nice to me. Therefore, I have moved all REST API–related changes to FLIP-487. 
> BTW, to avoid repetitive changes in FLIP-487, I'll start organizing FLIP-487 after FLIP-495 has been finalized.
> 
> Looking forward to your next review!
> 
> [1]https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=334760525#FLIP495:SupportAdaptiveSchedulerrecordandquerytherescalehistory-Aboutrescaleeventsstorage.1
> [2]https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=334760525#FLIP495:SupportAdaptiveSchedulerrecordandquerytherescalehistory-ThemainscenarioswhereRescalestatusswitchestoterminated
> [3]https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=334760525#FLIP495:SupportAdaptiveSchedulerrecordandquerytherescalehistory-Theinformationtorecordinarescaleevent
> [4]https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=334760525#FLIP495:SupportAdaptiveSchedulerrecordandquerytherescalehistory-InternalInterfaces
> 
> 
> 
> 
> 
> Best regards,
> Yuepeng Pan
> 
> 
> 
> 
> 
> 
> 
> 
> 
> 
> 
> At 2025-08-10 23:54:37, "Matthias Pohl" <ma...@apache.org> wrote:
> >Hi Yuepeng,
> >thanks for reminding me of this FLIP. I went over it and have a few items
> >which we might need to address before we can actually finalize the vote:
> >
> >1. You mention a few options for when it comes to storing the data which is
> >good. The FLIP doesn't point out, though, what option you're going to go
> >for as part of this FLIP (as far as I can see). It would be good to only
> >outline the option to go for in the FLIP and list the other options as
> >rejected alternatives (with the pro's and con's). I think it make sense to
> >go for option 3 (i.e. following what's done for the ExecutionGraphInfoStore
> >for now). The other options can be considered as a follow-up.
> >2. About the terminal states of a rescaling (i.e. IGNORED, FAILED,
> >COMPLETED): Can we we clarify in the FLIP under what conditions the
> >rescaling transitions into each of the three terminal states?
> >3. The section "The information to record in a rescale event" could be
> >restructured in four sections (to remove redundancy):
> > a) The IDs (Rescale
> >ID, resourceRequirementsEpochID, subRescaleIdOfResourceRequirementsEpochID):
> >What about making these names easier to read: GlobalRescaleID, RescaleUUID,
> >RescaleAttemptId)
> > b) Per-vertex data which includes: JobVertexID, JobVertexName,
> >SlotSharingGroupId, the different parallelisms (pre-rescale, sufficient,
> >desired, post-rescale)
> > c) The SlotSharingGroup information: SlotSharingGroupId, name,
> >ResourceProfile
> > d) Other information: Timestamps of state transitions, etc. as laid out in
> >the FLIP already
> >4. The FLIP doesn't explain how the data is passed through the
> >AdaptiveScheduler states. We should be handling some kind of
> >RescaleSnapshot that is passed through the different states and updated and
> >its final state is stored somewhere within AdaptiveScheduler in the end, I
> >guess. Can we clarify that in the FLIP?
> >5. You mention the config parameters for the cache in the public interface
> >section. But there's no mentioning of any caching and how that is used
> >within the FLIP.
> >6. The REST endpoint is probably better suited in FLIP-487. FLIP-495 should
> >be about the actual implementation details and how the data is stored
> >internally whereas FLIP-487 is about exposing the information to the
> >outside through the REST API and the Flink UI. That would be a way to
> >decrease the scope of FLIP-495. WDYT?
> >
> >Best,
> >Matthias
> >
> >
> >On Mon, Mar 24, 2025 at 11:37 AM Yuepeng Pan <pa...@apache.org> wrote:
> >
> >> Hi, Community,
> >>
> >> There haven’t been any further responses to this email over the past few
> >> days.
> >> I'd like to initiate a vote on the current proposal[1] in the next few
> >> days.
> >> Please rest assured that I’m proceeding cautiously and not rushing the
> >> process.
> >> If there are any concerns about this FLIP-495[1],
> >> I will gladly pause and make the adjustments.
> >>
> >> Best regards,
> >> Yuepeng Pan
> >>
> >> [1]
> >> https://cwiki.apache.org/confluence/display/FLINK/FLIP-495%3A+Support+AdaptiveScheduler+record+and+query+the+rescale+history
> >>
> >>
> >> On 2024/12/17 15:18:45 Yuepeng Pan wrote:
> >> > Hi community,
> >> >
> >> >
> >> >
> >> >
> >> > We discussed several aspects of FLIP-487[1] 'Show history of rescales in
> >> Web UI for AdaptiveScheduler'
> >> > and received a lot of valuable feedback. Based on the suggestions from
> >> the email thread[2],
> >> > we plan to split the original proposal for FLIP-487[1].
> >> >
> >> >
> >> >
> >> >
> >> > The current email thread and the FLIP-495[3] wiki will be used to
> >> discuss 'Support AdaptiveScheduler in recording and querying the rescale
> >> history',
> >> > while FLIP-487[1] will primarily focus on displaying-related design
> >> content
> >> >
> >> >
> >> >
> >> >
> >> > Looking forward to any feedback and opinions on FLIP-495[3].
> >> >
> >> >
> >> >
> >> >
> >> > [1]
> >> https://cwiki.apache.org/confluence/display/FLINK/%5BWIP%5D+FLIP-487%3A+Show+history+of+rescales+in+Web+UI+for+AdaptiveScheduler
> >> >
> >> > [2] https://lists.apache.org/thread/f4md4btkf006mxcxf66bng1kfz0rsn8c
> >> >
> >> > [3]
> >> https://cwiki.apache.org/confluence/display/FLINK/%5BWIP%5D+FLIP-495%3A+Support+AdaptiveScheduler+record+and+query+the+rescale+history
> >> >
> >> >
> >> >
> >> >
> >> > Thank you very much.
> >> >
> >> >
> >> >
> >> >
> >> > Best,
> >> >
> >> > Regards.
> >> >
> >> > Yuepeng Pan
> >>
> 

Re: [DISCUSS] FLIP-495: Support AdaptiveScheduler record and query the rescale history

Posted by Yuepeng Pan <fl...@126.com>.
Hi, Matthias.


Thanks for your reply !


> - The TerminalState has the state PENDING. Not sure whether that's the right naming considering PENDING not being a terminal state. We could remove PENDING from the enum and make the terminalState field @Nullable to cover the pending state.

That sounds good to me! And I made the corresponding change.

> - nit: In the FLIP, you describe desired and sufficient resources well now. But you use the word "sufficient" to describe having the right amount of task slots available for the scaling which is a bit confusing. Maybe changing that one to "desired" can cause confusion with the subsequent description of the "sufficient resources.

A temporary use of 'minimalRequiredSlots' might be a good naming choice, or we can also revisit this issue during the code review stage of development.



> Otherwise, I have nothing to add. I will let the discussion sit for a bit
> in case someone else wants to contribute before approving the FLIP
> considering that there are already 2 binding votes and the FLIP changed in
> the meantime.



Thank you very much for your help and review, which has led to exciting progress on this FLIP!
BTW, I will prepare the corresponding backend interface design and frontend sketch updates for FLIP-487 based on the current consensus.


Best regards,
Yuepeng Pan




---- Replied Message ----
| From | Matthias Pohl<ma...@apache.org> |
| Date | 09/1/2025 14:21 |
| To | <de...@flink.apache.org> |
| Cc | Yuepeng Pan<pa...@apache.org> |
| Subject | Re: [DISCUSS] FLIP-495: Support AdaptiveScheduler record and query the rescale history |
Thanks for responding to my comments. 


In the current design, AdaptiveScheduler has a method #getRescaleTimeline,
so we are already able to access the RescaleTimeline object within the AdaptiveScheduler instance and use it to handle rescale-related logic.
Sorry, I may not understand your point.
[...] 
I would appreciate any corrections or suggestions.


Sorry for the misunderstanding: My remark was actually only meant as a clarification. Not as an action item. 


I went over the FLIP once more and do not have any major things to add. Here are a few minor items:


- The TerminalState has the state PENDING. Not sure whether that's the right naming considering PENDING not being a terminal state. We could remove PENDING from the enum and make the terminalState field @Nullable to cover the pending state.
- nit: In the FLIP, you describe desired and sufficient resources well now. But you use the word "sufficient" to describe having the right amount of task slots available for the scaling which is a bit confusing. Maybe changing that one to "desired" can cause confusion with the subsequent description of the "sufficient resources.


Otherwise, I have nothing to add. I will let the discussion sit for a bit in case someone else wants to contribute before approving the FLIP considering that there are already 2 binding votes and the FLIP changed in the meantime. 


Thanks for pushing this topic, Yuepeng.


Best,
Matthias




On Mon, Aug 25, 2025 at 12:38 PM Roc Marshal <fl...@126.com> wrote:

Hi, Matthias.
Thank you very much for the comments.



> I was thinking about the case where a rescale was initiated by
> calling updateJobResourcesRequirements but it's not executed, yet, before
> the error occurs. In that case, a restart will be triggered that implicitly
> executes the rescale based on the available resources. If the available
> resources match the desired resource requirements of the
> updateJobResourcesRequirements, we execute the rescale through the job
> restart which was triggered by the error handling. That would result in a
> rescale that was actually "successful/completed".

IIUC, did you mean making some adjustments in such cases to reduce the number of ignored rescale records triggered by updateJobResourcesRequirements in the mentioned about case?
If so, I think that is reasonable, and I will handle it accordingly during the implementation.

> The State implementations handle AdaptiveScheduler State-specific
> functionality but the corresponding Context interfaces provide the methods
> to actually modify the AdaptiveScheduler instance itself. The
> RescaleTimeline is part of the AdaptiveScheduler instance and would be
> exposed through that instance (analogously to the exception history in
> AdaptiveScheduler#requestJob).

In the current design, AdaptiveScheduler has a method #getRescaleTimeline,
so we are already able to access the RescaleTimeline object within the AdaptiveScheduler instance and use it to handle rescale-related logic.
Sorry, I may not understand your point.
Did you mean that we still need some changes like the following?
- Remove AdaptiveScheduler#getRescaleTimeline
- Remove StateWithExecutionGraph#getRescaleTimeline
- Remove StateWithoutExecutionGraph#getRescaleTimeline
- Add interfaces in State.Context to operate on rescale in RescaleTimeline directly, so as to avoid an extra #getRescaleTimeline method calling.
I would appreciate any corrections or suggestions.

For Other notes:

> - nit (implementation detail): I'm not sure whether we need to keep the
> latestRescales as part of the RescalesStatsSnapshot. That could be quickly
> derived from RescalesStatsSnapshot#rescaleHistory (to reduce redundancy in
> the code) assuming that this list doesn't contain millions of entries. But
> that's an implementation detail. Not sure whether that even needs to be
> mentioned in the FLIP.
> - nit (implementation detail): VertexParallelismRescale#pre/postParallelism
> should be named pre/postRescaleParallelism. Same applies
> to SlotSharingGroupRescale#pre/postSlot
> - nit (implementation detail): The name of the SlotSharingGroup does not
> need to be stored in the VertexParallelismRescale as far as I can see. The
> SlotSharingGroup which holds the name is referenced through the ID.

Reasonable! Updated it.

> - Do we still need RescaleStatus in the Rescale structure? That seems to be
> a transient field that should be derived via the AdaptiveScheduler's state.

Thank you very much for the reminder.
This is exactly the point I have been reconsidering.
After introducing TerminalState and TerminatedReason, this field has lost its meaning.
If users need to pay attention to the AdaptiveScheduler state information, they can obtain it from schedulerStates.
Therefore, removing this field is reasonable to me, too.



Looking forward to your comments!


Best regards,
Yuepeng Pan
---- Replied Message ----
| From | Matthias Pohl<ma...@apache.org> |
| Date | 08/25/2025 15:14 |
| To | <de...@flink.apache.org> |
| Cc | Roc Marshal<fl...@126.com> |
| Subject | Re: [DISCUSS] FLIP-495: Support AdaptiveScheduler record and query the rescale history |
Hi Yuepeng,
thanks for the update. Please see my responses below:

If, during failure handling and job recovery (i.e., when there is an
ongoing rescale), a new resource requirement change or new available
resource change occurs (provided that such a change can trigger a new
rescale), the ongoing rescale will be forcibly terminated and saved.


I was thinking about the case where a rescale was initiated by
calling updateJobResourcesRequirements but it's not executed, yet, before
the error occurs. In that case, a restart will be triggered that implicitly
executes the rescale based on the available resources. If the available
resources match the desired resource requirements of the
updateJobResourcesRequirements, we execute the rescale through the job
restart which was triggered by the error handling. That would result in a
rescale that was actually "successful/completed".

- Why does StateTransitions need to be extended with
getRescaleTimeline()?



- This is because we want to quickly obtain a reference to the
RescaleTimeline, and maintain rescale information through it.
- It was introduced into the StateTransitions class so that all states can
call #getRescaleTimeline via this interface.
- After re-examining the scope affected by rescale maintenance operations,
it might be more precise to introduce this method into
StateWithExecutionGraph.Context / StateWithoutExecutionGraph.Context [1]
instead.


The State implementations handle AdaptiveScheduler State-specific
functionality but the corresponding Context interfaces provide the methods
to actually modify the AdaptiveScheduler instance itself. The
RescaleTimeline is part of the AdaptiveScheduler instance and would be
exposed through that instance (analogously to the exception history in
AdaptiveScheduler#requestJob).

Other notes:
- nit (implementation detail): I'm not sure whether we need to keep the
latestRescales as part of the RescalesStatsSnapshot. That could be quickly
derived from RescalesStatsSnapshot#rescaleHistory (to reduce redundancy in
the code) assuming that this list doesn't contain millions of entries. But
that's an implementation detail. Not sure whether that even needs to be
mentioned in the FLIP.

- nit (implementation detail): VertexParallelismRescale#pre/postParallelism
should be named pre/postRescaleParallelism. Same applies
to SlotSharingGroupRescale#pre/postSlot

- nit (implementation detail): The name of the SlotSharingGroup does not
need to be stored in the VertexParallelismRescale as far as I can see. The
SlotSharingGroup which holds the name is referenced through the ID.

- Do we still need RescaleStatus in the Rescale structure? That seems to be
a transient field that should be derived via the AdaptiveScheduler's state.

Thanks,
Matthias

On Fri, Aug 22, 2025 at 1:19 PM Roc Marshal <fl...@126.com> wrote:

Hi, Matthias.
Thank you very much for the comments.

- Why do we mark the rescale operation as IGNORED if an (repeatable)
exception causes a restart?

Here we mainly discuss two scenarios:
- If a recoverable exception occurs while there is an ongoing rescale, the
current rescale will be forcibly terminated and saved. Then, a new rescale
will be created to record the subsequent rescale information.
- If there is no ongoing rescale, a new rescale will be created directly
to record the information.

That's because the failure will create a new rescale instance that will
be saved if the resources changed as part of the failure handling/job
restart process?

If, during failure handling and job recovery (i.e., when there is an
ongoing rescale), a new resource requirement change or new available
resource change occurs (provided that such a change can trigger a new
rescale), the ongoing rescale will be forcibly terminated and saved.
Then, the system will create a new rescale to continue recording the
subsequent process.

- Thinking about the GlobalRescaleId: Should we still keep it
considering that we're not preserving the value on JM failovers (because of
the missing HA functionality) in this FLIP? We could remove it and add it
as part of a HA-support follow-up. Providing it despite the missing HA
support might be misleading in case a JM failover actually happens. WDYT?

Nice proposal! That makes sense to me~

- For the job vertices, you introduced a 5th resource type (previous,
acquired, required, min bound, max bound). What's the difference? I guess,
min bound corresponds to "sufficient resources" and max bound corresponds
to "desired resources". What are required resources here in that case? It
might be good to stick to the naming that was introduced in FLIP-472 [1].

I’ve already made the latest changes.
BTW, the original intention here was to record information as similar as
possible to the one from #updateJobResourcesRequirements requests. Now I
believe it’s quite necessary to remove the previous design and use a
unified description instead, since this makes things more consistent and
easier to understand. Many thanks for your suggestions.

- "Job Vertices Rescaled Information" and "Slots / Resources Rescaled
Information" seem to have redundant information now: "parallelism of job
vertex" and "slot number of the slot sharing group" are more or less the
same. The parallelism is defined per job vertex (along the
SlotSharingGroupId). The SlotSharingGroup holds the ResourceProfile (but no
parallelism).

Please let me to explain the reasoning:
- The parallelism of job vertices is kept so that users can clearly see
the parallelism of each job vertex.
- Adding information about the number of slots in a SlotSharingGroup is
intended to give users a more intuitive understanding of how many resources
this type of group consumes. Of course, even if we remove the slot number
in the SlotSharingGroup, users could still calculate it by themselves based
on job vertices paralllelsims, but that would require them to be familiar
with the AdaptiveScheduler’s logic for computing slot numbers.
- In addition, if we only keep the information about the number of slots
in the SlotSharingGroup and drop the parallelism information at the
job-vertex level, then users would lose the fine-grained parallelism
details of job vertices during a rescale. They would only be able to see
SlotSharingGroup-level information, which is somewhat like parallelism but
less precise.
- If necessary, I’m willing to remove the slot-number-related field from
the SlotSharingGroup description.
Please let me know your thoughts.

- "Other information" section contains a comment message for the reason
why a rescale event terminates. Reason might be a better term than comment
(especially because we're relying on an enum here rather than plain text).

Good idea — I’ve already made the corresponding changes.

- Based on the current proposal of the FLIP, the intermediate states of
the rescale event actually match the AdaptiveScheduler's states. Only the
final states are exclusive to the scale event?  Would it make sense to rely
on the AdaptiveScheduler state entirely for the intermediate state and a
dedicated field "terminalState"?

Yes, your understanding is correct. This approach works well for me, and
I’ve already made the corresponding changes in the wiki document.

- Why does StateTransitions need to be extended with
getRescaleTimeline()?

- This is because we want to quickly obtain a reference to the
RescaleTimeline, and maintain rescale information through it.
- It was introduced into the StateTransitions class so that all states can
call #getRescaleTimeline via this interface.
- After re-examining the scope affected by rescale maintenance operations,
it might be more precise to introduce this method into
StateWithExecutionGraph.Context / StateWithoutExecutionGraph.Context [1]
instead.

Shouldn't the RescaleTimeline be located in the AdaptiveScheduler?

Yes. Whether #getRescaleTimeline is introduced in StateTransitions or in
StateWithExecutionGraph.Context / StateWithoutExecutionGraph.Context, the
RescaleTimeline should still be part of the AdaptiveScheduler.

Instead, the State.Context (essentially, the AdaptiveScheduler) should
provide methods for updating the current rescale process object that lives
in the AdaptiveScheduler. WDYT?

Sounds good to me. Once #getRescaleTimeline is introduced in
State.Context, it can directly obtain a reference to the RescaleTimeline,
allowing quick operations on the current rescale.
What do you think?


- Section "How to process rescales storage ?": ExecutionGraphInfo could
be modified to include the Rescale history (along the exception history).
The data can be then saved in the ExecutionGraphInfoStore. All you have to
do is to store the rescale history in AdaptiveScheduler and an object that
holds the infos of the current rescale process. If this current rescale
process terminates, an immutable rescale snapshot of this event is created
that is saved in the rescale history.

Sorry for my previous wording was ambiguous and inconsistent with the
documentation. Thanks a lot for pointing it out. This is actually what I
intended to do, and I’ve already corrected the corresponding description in
the document.

[1]
https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=334760525#FLIP495:SupportAdaptiveSchedulerrecordandquerytherescalehistory-Howtohold/maintaintherescaleshistory
?

Best,
Yuepeng Pan



Roc Marshal
flinker@126.com

---- Replied Message ----
From Matthias Pohl<ma...@apache.org> <ma...@apache.org>
Date 08/21/2025 18:54
To dev@flink.apache.org<de...@flink.apache.org> <de...@flink.apache.org>
Cc Yuepeng Pan<pa...@apache.org> <pa...@apache.org>
Subject Re: [DISCUSS] FLIP-495: Support AdaptiveScheduler record and
query the rescale history
Thanks for the update, Yuepeng. Here are my remarks:

- Why do we mark the rescale operation as IGNORED if an (repeatable)
exception causes a restart? That's because the failure will create a new
rescale instance that will be saved if the resources changed as part of the
failure handling/job restart process?

- Thinking about the GlobalRescaleId: Should we still keep it considering
that we're not preserving the value on JM failovers (because of the missing
HA functionality) in this FLIP? We could remove it and add it as part of a
HA-support follow-up. Providing it despite the missing HA support might be
misleading in case a JM failover actually happens. WDYT?

- For the job vertices, you introduced a 5th resource type (previous,
acquired, required, min bound, max bound). What's the difference? I guess,
min bound corresponds to "sufficient resources" and max bound corresponds
to "desired resources". What are required resources here in that case? It
might be good to stick to the naming that was introduced in FLIP-472 [1].

- "Job Vertices Rescaled Information" and "Slots / Resources Rescaled
Information" seem to have redundant information now: "parallelism of job
vertex" and "slot number of the slot sharing group" are more or less the
same. The parallelism is defined per job vertex (along the
SlotSharingGroupId). The SlotSharingGroup holds the ResourceProfile (but no
parallelism).

- "Other information" section contains a comment message for the reason why
a rescale event terminates. Reason might be a better term than comment
(especially because we're relying on an enum here rather than plain text).

- Based on the current proposal of the FLIP, the intermediate states of the
rescale event actually match the AdaptiveScheduler's states. Only the final
states are exclusive to the scale event? Would it make sense to rely on the
AdaptiveScheduler state entirely for the intermediate state and a dedicated
field "terminalState"?

- Why does StateTransitions need to be extended with getRescaleTimeline()?
Shouldn't the RescaleTimeline be located in the AdaptiveScheduler? Instead,
the State.Context (essentially, the AdaptiveScheduler) should provide
methods for updating the current rescale process object that lives in the
AdaptiveScheduler. WDYT?

- Section "How to process rescales storage ?": ExecutionGraphInfo could be
modified to include the Rescale history (along the exception history). The
data can be then saved in the ExecutionGraphInfoStore. All you have to do
is to store the rescale history in AdaptiveScheduler and an object that
holds the infos of the current rescale process. If this current rescale
process terminates, an immutable rescale snapshot of this event is created
that is saved in the rescale history.

Matthias

[1]

https://cwiki.apache.org/confluence/display/FLINK/FLIP-472%3A+Aligning+timeout+logic+in+the+AdaptiveScheduler%27s+WaitingForResources+and+Executing+states

On Tue, Aug 19, 2025 at 11:14 AM Yuepeng Pan <pa...@apache.org>
wrote:

Bumping this thread kindly. Thanks!

Best,
Yuepeng Pan


At 2025-08-13 14:52:26, "Yuepeng Pan" <pa...@apache.org> wrote:

Hi, Matthias,
Thank you  very much for your comments!
I have carefully read your reply and made some changes in the hope of
making improvements.
Please help take a look.

For your comments:

1. You mention a few options for when it comes to storing the data which is
good. The FLIP doesn't point out, though, what option you're going to go
for as part of this FLIP (as far as I can see). It would be good to only
outline the option to go for in the FLIP and list the other options as
rejected alternatives (with the pro's and con's). I think it make sense to
go for option 3 (i.e. following what's done for the ExecutionGraphInfoStore
for now). The other options can be considered as a follow-up.


This is very meaningful. Based on this comment, I have kept option 3 in
its original place and moved the other candidate options to [1].

2. About the terminal states of a rescaling (i.e. IGNORED, FAILED,
COMPLETED): Can we we clarify in the FLIP under what conditions the
rescaling transitions into each of the three terminal states?


Yes, this is a reasonable request for understanding and explaining the
logic of transitions to terminated states.
A new subsection [2] has been added to address this.

3. The section "The information to record in a rescale event" could be
restructured in four sections (to remove redundancy):
a) The IDs (Rescale
ID, resourceRequirementsEpochID,
subRescaleIdOfResourceRequirementsEpochID):
What about making these names easier to read: GlobalRescaleID, RescaleUUID,
RescaleAttemptId)
b) Per-vertex data which includes: JobVertexID, JobVertexName,
SlotSharingGroupId, the different parallelisms (pre-rescale, sufficient,
desired, post-rescale)
c) The SlotSharingGroup information: SlotSharingGroupId, name,
ResourceProfile
d) Other information: Timestamps of state transitions, etc. as laid out in
the FLIP already


That makes sense to me. Please check [3] for the latest updates in this
part.

4. The FLIP doesn't explain how the data is passed through the
AdaptiveScheduler states. We should be handling some kind of
RescaleSnapshot that is passed through the different states and updated and
its final state is stored somewhere within AdaptiveScheduler in the end, I
guess. Can we clarify that in the FLIP?


Indeed — this was missing in the original FLIP. To address this, I have
added [4], which focuses on describing how a Rescale is represented,
and how we can quickly pass and maintain the Rescale history.

5. You mention the config parameters for the cache in the public interface
section. But there's no mentioning of any caching and how that is used
within the FLIP.


Sorry for the rough description in the previous version.
Since this part belongs to the REST API acceleration mechanism for
rescaling, and Option 6 seems reasonable to me,
I plan to add it to FLIP-487 once the design of FLIP-495 has reached
consensus.
Of course, if needed, I'd be happy to clarify the usage and purpose of
this parameter in the current email thread.

6. The REST endpoint is probably better suited in FLIP-487. FLIP-495 should
be about the actual implementation details and how the data is stored
internally whereas FLIP-487 is about exposing the information to the
outside through the REST API and the Flink UI. That would be a way to
decrease the scope of FLIP-495. WDYT?


That sounds nice to me. Therefore, I have moved all REST API–related
changes to FLIP-487.
BTW, to avoid repetitive changes in FLIP-487, I'll start organizing
FLIP-487 after FLIP-495 has been finalized.

Looking forward to your next review!

[1]
https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=334760525#FLIP495:SupportAdaptiveSchedulerrecordandquerytherescalehistory-Aboutrescaleeventsstorage.1
[2]
https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=334760525#FLIP495:SupportAdaptiveSchedulerrecordandquerytherescalehistory-ThemainscenarioswhereRescalestatusswitchestoterminated
[3]
https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=334760525#FLIP495:SupportAdaptiveSchedulerrecordandquerytherescalehistory-Theinformationtorecordinarescaleevent
[4]
https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=334760525#FLIP495:SupportAdaptiveSchedulerrecordandquerytherescalehistory-InternalInterfaces


Best regards,
Yuepeng Pan





At 2025-08-10 23:54:37, "Matthias Pohl" <ma...@apache.org> wrote:

Hi Yuepeng,
thanks for reminding me of this FLIP. I went over it and have a few items
which we might need to address before we can actually finalize the vote:

1. You mention a few options for when it comes to storing the data which is
good. The FLIP doesn't point out, though, what option you're going to go
for as part of this FLIP (as far as I can see). It would be good to only
outline the option to go for in the FLIP and list the other options as
rejected alternatives (with the pro's and con's). I think it make sense to
go for option 3 (i.e. following what's done for the ExecutionGraphInfoStore
for now). The other options can be considered as a follow-up.
2. About the terminal states of a rescaling (i.e. IGNORED, FAILED,
COMPLETED): Can we we clarify in the FLIP under what conditions the
rescaling transitions into each of the three terminal states?
3. The section "The information to record in a rescale event" could be
restructured in four sections (to remove redundancy):
a) The IDs (Rescale
ID, resourceRequirementsEpochID,
subRescaleIdOfResourceRequirementsEpochID):
What about making these names easier to read: GlobalRescaleID, RescaleUUID,
RescaleAttemptId)
b) Per-vertex data which includes: JobVertexID, JobVertexName,
SlotSharingGroupId, the different parallelisms (pre-rescale, sufficient,
desired, post-rescale)
c) The SlotSharingGroup information: SlotSharingGroupId, name,
ResourceProfile
d) Other information: Timestamps of state transitions, etc. as laid out in
the FLIP already
4. The FLIP doesn't explain how the data is passed through the
AdaptiveScheduler states. We should be handling some kind of
RescaleSnapshot that is passed through the different states and updated and
its final state is stored somewhere within AdaptiveScheduler in the end, I
guess. Can we clarify that in the FLIP?
5. You mention the config parameters for the cache in the public interface
section. But there's no mentioning of any caching and how that is used
within the FLIP.
6. The REST endpoint is probably better suited in FLIP-487. FLIP-495 should
be about the actual implementation details and how the data is stored
internally whereas FLIP-487 is about exposing the information to the
outside through the REST API and the Flink UI. That would be a way to
decrease the scope of FLIP-495. WDYT?

Best,
Matthias


On Mon, Mar 24, 2025 at 11:37 AM Yuepeng Pan <pa...@apache.org>
wrote:

Hi, Community,

There haven’t been any further responses to this email over the past few
days.
I'd like to initiate a vote on the current proposal[1] in the next few
days.
Please rest assured that I’m proceeding cautiously and not rushing the
process.
If there are any concerns about this FLIP-495[1],
I will gladly pause and make the adjustments.

Best regards,
Yuepeng Pan

[1]

https://cwiki.apache.org/confluence/display/FLINK/FLIP-495%3A+Support+AdaptiveScheduler+record+and+query+the+rescale+history


On 2024/12/17 15:18:45 Yuepeng Pan wrote:

Hi community,




We discussed several aspects of FLIP-487[1] 'Show history of rescales in

Web UI for AdaptiveScheduler'

and received a lot of valuable feedback. Based on the suggestions from

the email thread[2],

we plan to split the original proposal for FLIP-487[1].




The current email thread and the FLIP-495[3] wiki will be used to

discuss 'Support AdaptiveScheduler in recording and querying the rescale
history',

while FLIP-487[1] will primarily focus on displaying-related design

content





Looking forward to any feedback and opinions on FLIP-495[3].




[1]


https://cwiki.apache.org/confluence/display/FLINK/%5BWIP%5D+FLIP-487%3A+Show+history+of+rescales+in+Web+UI+for+AdaptiveScheduler


[2] https://lists.apache.org/thread/f4md4btkf006mxcxf66bng1kfz0rsn8c

[3]


https://cwiki.apache.org/confluence/display/FLINK/%5BWIP%5D+FLIP-495%3A+Support+AdaptiveScheduler+record+and+query+the+rescale+history





Thank you very much.




Best,

Regards.

Yuepeng Pan






Re: [DISCUSS] FLIP-495: Support AdaptiveScheduler record and query the rescale history

Posted by Matthias Pohl <ma...@apache.org>.
Thanks for responding to my comments.

In the current design, AdaptiveScheduler has a method #getRescaleTimeline,
> so we are already able to access the RescaleTimeline object within the
> AdaptiveScheduler instance and use it to handle rescale-related logic.
> Sorry, I may not understand your point.
> [...]
> I would appreciate any corrections or suggestions.


Sorry for the misunderstanding: My remark was actually only meant as a
clarification. Not as an action item.

I went over the FLIP once more and do not have any major things to add.
Here are a few minor items:

- The TerminalState has the state PENDING. Not sure whether that's the
right naming considering PENDING not being a terminal state. We could
remove PENDING from the enum and make the terminalState field @Nullable to
cover the pending state.
- nit: In the FLIP, you describe desired and sufficient resources well now.
But you use the word "sufficient" to describe having the right amount of
task slots available for the scaling which is a bit confusing. Maybe
changing that one to "desired" can cause confusion with the subsequent
description of the "sufficient resources.

Otherwise, I have nothing to add. I will let the discussion sit for a bit
in case someone else wants to contribute before approving the FLIP
considering that there are already 2 binding votes and the FLIP changed in
the meantime.

Thanks for pushing this topic, Yuepeng.

Best,
Matthias


On Mon, Aug 25, 2025 at 12:38 PM Roc Marshal <fl...@126.com> wrote:

> Hi, Matthias.
> Thank you very much for the comments.
>
>
>
> > I was thinking about the case where a rescale was initiated by
> > calling updateJobResourcesRequirements but it's not executed, yet, before
> > the error occurs. In that case, a restart will be triggered that
> implicitly
> > executes the rescale based on the available resources. If the available
> > resources match the desired resource requirements of the
> > updateJobResourcesRequirements, we execute the rescale through the job
> > restart which was triggered by the error handling. That would result in a
> > rescale that was actually "successful/completed".
>
> IIUC, did you mean making some adjustments in such cases to reduce the
> number of ignored rescale records triggered by
> updateJobResourcesRequirements in the mentioned about case?
> If so, I think that is reasonable, and I will handle it accordingly during
> the implementation.
>
> > The State implementations handle AdaptiveScheduler State-specific
> > functionality but the corresponding Context interfaces provide the
> methods
> > to actually modify the AdaptiveScheduler instance itself. The
> > RescaleTimeline is part of the AdaptiveScheduler instance and would be
> > exposed through that instance (analogously to the exception history in
> > AdaptiveScheduler#requestJob).
>
> In the current design, AdaptiveScheduler has a method #getRescaleTimeline,
> so we are already able to access the RescaleTimeline object within the
> AdaptiveScheduler instance and use it to handle rescale-related logic.
> Sorry, I may not understand your point.
> Did you mean that we still need some changes like the following?
> - Remove AdaptiveScheduler#getRescaleTimeline
> - Remove StateWithExecutionGraph#getRescaleTimeline
> - Remove StateWithoutExecutionGraph#getRescaleTimeline
> - Add interfaces in State.Context to operate on rescale in RescaleTimeline
> directly, so as to avoid an extra #getRescaleTimeline method calling.
> I would appreciate any corrections or suggestions.
>
> For Other notes:
>
> > - nit (implementation detail): I'm not sure whether we need to keep the
> > latestRescales as part of the RescalesStatsSnapshot. That could be
> quickly
> > derived from RescalesStatsSnapshot#rescaleHistory (to reduce redundancy
> in
> > the code) assuming that this list doesn't contain millions of entries.
> But
> > that's an implementation detail. Not sure whether that even needs to be
> > mentioned in the FLIP.
> > - nit (implementation detail):
> VertexParallelismRescale#pre/postParallelism
> > should be named pre/postRescaleParallelism. Same applies
> > to SlotSharingGroupRescale#pre/postSlot
> > - nit (implementation detail): The name of the SlotSharingGroup does not
> > need to be stored in the VertexParallelismRescale as far as I can see.
> The
> > SlotSharingGroup which holds the name is referenced through the ID.
>
> Reasonable! Updated it.
>
> > - Do we still need RescaleStatus in the Rescale structure? That seems to
> be
> > a transient field that should be derived via the AdaptiveScheduler's
> state.
>
> Thank you very much for the reminder.
> This is exactly the point I have been reconsidering.
> After introducing TerminalState and TerminatedReason, this field has lost
> its meaning.
> If users need to pay attention to the AdaptiveScheduler state information,
> they can obtain it from schedulerStates.
> Therefore, removing this field is reasonable to me, too.
>
>
>
> Looking forward to your comments!
>
>
> Best regards,
> Yuepeng Pan
> ---- Replied Message ----
> | From | Matthias Pohl<ma...@apache.org> |
> | Date | 08/25/2025 15:14 |
> | To | <de...@flink.apache.org> |
> | Cc | Roc Marshal<fl...@126.com> |
> | Subject | Re: [DISCUSS] FLIP-495: Support AdaptiveScheduler record and
> query the rescale history |
> Hi Yuepeng,
> thanks for the update. Please see my responses below:
>
> If, during failure handling and job recovery (i.e., when there is an
> ongoing rescale), a new resource requirement change or new available
> resource change occurs (provided that such a change can trigger a new
> rescale), the ongoing rescale will be forcibly terminated and saved.
>
>
> I was thinking about the case where a rescale was initiated by
> calling updateJobResourcesRequirements but it's not executed, yet, before
> the error occurs. In that case, a restart will be triggered that implicitly
> executes the rescale based on the available resources. If the available
> resources match the desired resource requirements of the
> updateJobResourcesRequirements, we execute the rescale through the job
> restart which was triggered by the error handling. That would result in a
> rescale that was actually "successful/completed".
>
> - Why does StateTransitions need to be extended with
> getRescaleTimeline()?
>
>
>
> - This is because we want to quickly obtain a reference to the
> RescaleTimeline, and maintain rescale information through it.
> - It was introduced into the StateTransitions class so that all states can
> call #getRescaleTimeline via this interface.
> - After re-examining the scope affected by rescale maintenance operations,
> it might be more precise to introduce this method into
> StateWithExecutionGraph.Context / StateWithoutExecutionGraph.Context [1]
> instead.
>
>
> The State implementations handle AdaptiveScheduler State-specific
> functionality but the corresponding Context interfaces provide the methods
> to actually modify the AdaptiveScheduler instance itself. The
> RescaleTimeline is part of the AdaptiveScheduler instance and would be
> exposed through that instance (analogously to the exception history in
> AdaptiveScheduler#requestJob).
>
> Other notes:
> - nit (implementation detail): I'm not sure whether we need to keep the
> latestRescales as part of the RescalesStatsSnapshot. That could be quickly
> derived from RescalesStatsSnapshot#rescaleHistory (to reduce redundancy in
> the code) assuming that this list doesn't contain millions of entries. But
> that's an implementation detail. Not sure whether that even needs to be
> mentioned in the FLIP.
>
> - nit (implementation detail): VertexParallelismRescale#pre/postParallelism
> should be named pre/postRescaleParallelism. Same applies
> to SlotSharingGroupRescale#pre/postSlot
>
> - nit (implementation detail): The name of the SlotSharingGroup does not
> need to be stored in the VertexParallelismRescale as far as I can see. The
> SlotSharingGroup which holds the name is referenced through the ID.
>
> - Do we still need RescaleStatus in the Rescale structure? That seems to be
> a transient field that should be derived via the AdaptiveScheduler's state.
>
> Thanks,
> Matthias
>
> On Fri, Aug 22, 2025 at 1:19 PM Roc Marshal <fl...@126.com> wrote:
>
> Hi, Matthias.
> Thank you very much for the comments.
>
> - Why do we mark the rescale operation as IGNORED if an (repeatable)
> exception causes a restart?
>
> Here we mainly discuss two scenarios:
> - If a recoverable exception occurs while there is an ongoing rescale, the
> current rescale will be forcibly terminated and saved. Then, a new rescale
> will be created to record the subsequent rescale information.
> - If there is no ongoing rescale, a new rescale will be created directly
> to record the information.
>
> That's because the failure will create a new rescale instance that will
> be saved if the resources changed as part of the failure handling/job
> restart process?
>
> If, during failure handling and job recovery (i.e., when there is an
> ongoing rescale), a new resource requirement change or new available
> resource change occurs (provided that such a change can trigger a new
> rescale), the ongoing rescale will be forcibly terminated and saved.
> Then, the system will create a new rescale to continue recording the
> subsequent process.
>
> - Thinking about the GlobalRescaleId: Should we still keep it
> considering that we're not preserving the value on JM failovers (because of
> the missing HA functionality) in this FLIP? We could remove it and add it
> as part of a HA-support follow-up. Providing it despite the missing HA
> support might be misleading in case a JM failover actually happens. WDYT?
>
> Nice proposal! That makes sense to me~
>
> - For the job vertices, you introduced a 5th resource type (previous,
> acquired, required, min bound, max bound). What's the difference? I guess,
> min bound corresponds to "sufficient resources" and max bound corresponds
> to "desired resources". What are required resources here in that case? It
> might be good to stick to the naming that was introduced in FLIP-472 [1].
>
> I’ve already made the latest changes.
> BTW, the original intention here was to record information as similar as
> possible to the one from #updateJobResourcesRequirements requests. Now I
> believe it’s quite necessary to remove the previous design and use a
> unified description instead, since this makes things more consistent and
> easier to understand. Many thanks for your suggestions.
>
> - "Job Vertices Rescaled Information" and "Slots / Resources Rescaled
> Information" seem to have redundant information now: "parallelism of job
> vertex" and "slot number of the slot sharing group" are more or less the
> same. The parallelism is defined per job vertex (along the
> SlotSharingGroupId). The SlotSharingGroup holds the ResourceProfile (but no
> parallelism).
>
> Please let me to explain the reasoning:
> - The parallelism of job vertices is kept so that users can clearly see
> the parallelism of each job vertex.
> - Adding information about the number of slots in a SlotSharingGroup is
> intended to give users a more intuitive understanding of how many resources
> this type of group consumes. Of course, even if we remove the slot number
> in the SlotSharingGroup, users could still calculate it by themselves based
> on job vertices paralllelsims, but that would require them to be familiar
> with the AdaptiveScheduler’s logic for computing slot numbers.
> - In addition, if we only keep the information about the number of slots
> in the SlotSharingGroup and drop the parallelism information at the
> job-vertex level, then users would lose the fine-grained parallelism
> details of job vertices during a rescale. They would only be able to see
> SlotSharingGroup-level information, which is somewhat like parallelism but
> less precise.
> - If necessary, I’m willing to remove the slot-number-related field from
> the SlotSharingGroup description.
> Please let me know your thoughts.
>
> - "Other information" section contains a comment message for the reason
> why a rescale event terminates. Reason might be a better term than comment
> (especially because we're relying on an enum here rather than plain text).
>
> Good idea — I’ve already made the corresponding changes.
>
> - Based on the current proposal of the FLIP, the intermediate states of
> the rescale event actually match the AdaptiveScheduler's states. Only the
> final states are exclusive to the scale event?  Would it make sense to rely
> on the AdaptiveScheduler state entirely for the intermediate state and a
> dedicated field "terminalState"?
>
> Yes, your understanding is correct. This approach works well for me, and
> I’ve already made the corresponding changes in the wiki document.
>
> - Why does StateTransitions need to be extended with
> getRescaleTimeline()?
>
> - This is because we want to quickly obtain a reference to the
> RescaleTimeline, and maintain rescale information through it.
> - It was introduced into the StateTransitions class so that all states can
> call #getRescaleTimeline via this interface.
> - After re-examining the scope affected by rescale maintenance operations,
> it might be more precise to introduce this method into
> StateWithExecutionGraph.Context / StateWithoutExecutionGraph.Context [1]
> instead.
>
> Shouldn't the RescaleTimeline be located in the AdaptiveScheduler?
>
> Yes. Whether #getRescaleTimeline is introduced in StateTransitions or in
> StateWithExecutionGraph.Context / StateWithoutExecutionGraph.Context, the
> RescaleTimeline should still be part of the AdaptiveScheduler.
>
> Instead, the State.Context (essentially, the AdaptiveScheduler) should
> provide methods for updating the current rescale process object that lives
> in the AdaptiveScheduler. WDYT?
>
> Sounds good to me. Once #getRescaleTimeline is introduced in
> State.Context, it can directly obtain a reference to the RescaleTimeline,
> allowing quick operations on the current rescale.
> What do you think?
>
>
> - Section "How to process rescales storage ?": ExecutionGraphInfo could
> be modified to include the Rescale history (along the exception history).
> The data can be then saved in the ExecutionGraphInfoStore. All you have to
> do is to store the rescale history in AdaptiveScheduler and an object that
> holds the infos of the current rescale process. If this current rescale
> process terminates, an immutable rescale snapshot of this event is created
> that is saved in the rescale history.
>
> Sorry for my previous wording was ambiguous and inconsistent with the
> documentation. Thanks a lot for pointing it out. This is actually what I
> intended to do, and I’ve already corrected the corresponding description in
> the document.
>
> [1]
>
> https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=334760525#FLIP495:SupportAdaptiveSchedulerrecordandquerytherescalehistory-Howtohold/maintaintherescaleshistory
> ?
>
> Best,
> Yuepeng Pan
>
>
>
> Roc Marshal
> flinker@126.com
>
> ---- Replied Message ----
> From Matthias Pohl<ma...@apache.org> <ma...@apache.org>
> Date 08/21/2025 18:54
> To dev@flink.apache.org<de...@flink.apache.org> <de...@flink.apache.org>
> Cc Yuepeng Pan<pa...@apache.org> <pa...@apache.org>
> Subject Re: [DISCUSS] FLIP-495: Support AdaptiveScheduler record and
> query the rescale history
> Thanks for the update, Yuepeng. Here are my remarks:
>
> - Why do we mark the rescale operation as IGNORED if an (repeatable)
> exception causes a restart? That's because the failure will create a new
> rescale instance that will be saved if the resources changed as part of the
> failure handling/job restart process?
>
> - Thinking about the GlobalRescaleId: Should we still keep it considering
> that we're not preserving the value on JM failovers (because of the missing
> HA functionality) in this FLIP? We could remove it and add it as part of a
> HA-support follow-up. Providing it despite the missing HA support might be
> misleading in case a JM failover actually happens. WDYT?
>
> - For the job vertices, you introduced a 5th resource type (previous,
> acquired, required, min bound, max bound). What's the difference? I guess,
> min bound corresponds to "sufficient resources" and max bound corresponds
> to "desired resources". What are required resources here in that case? It
> might be good to stick to the naming that was introduced in FLIP-472 [1].
>
> - "Job Vertices Rescaled Information" and "Slots / Resources Rescaled
> Information" seem to have redundant information now: "parallelism of job
> vertex" and "slot number of the slot sharing group" are more or less the
> same. The parallelism is defined per job vertex (along the
> SlotSharingGroupId). The SlotSharingGroup holds the ResourceProfile (but no
> parallelism).
>
> - "Other information" section contains a comment message for the reason why
> a rescale event terminates. Reason might be a better term than comment
> (especially because we're relying on an enum here rather than plain text).
>
> - Based on the current proposal of the FLIP, the intermediate states of the
> rescale event actually match the AdaptiveScheduler's states. Only the final
> states are exclusive to the scale event? Would it make sense to rely on the
> AdaptiveScheduler state entirely for the intermediate state and a dedicated
> field "terminalState"?
>
> - Why does StateTransitions need to be extended with getRescaleTimeline()?
> Shouldn't the RescaleTimeline be located in the AdaptiveScheduler? Instead,
> the State.Context (essentially, the AdaptiveScheduler) should provide
> methods for updating the current rescale process object that lives in the
> AdaptiveScheduler. WDYT?
>
> - Section "How to process rescales storage ?": ExecutionGraphInfo could be
> modified to include the Rescale history (along the exception history). The
> data can be then saved in the ExecutionGraphInfoStore. All you have to do
> is to store the rescale history in AdaptiveScheduler and an object that
> holds the infos of the current rescale process. If this current rescale
> process terminates, an immutable rescale snapshot of this event is created
> that is saved in the rescale history.
>
> Matthias
>
> [1]
>
>
> https://cwiki.apache.org/confluence/display/FLINK/FLIP-472%3A+Aligning+timeout+logic+in+the+AdaptiveScheduler%27s+WaitingForResources+and+Executing+states
>
> On Tue, Aug 19, 2025 at 11:14 AM Yuepeng Pan <pa...@apache.org>
> wrote:
>
> Bumping this thread kindly. Thanks!
>
> Best,
> Yuepeng Pan
>
>
> At 2025-08-13 14:52:26, "Yuepeng Pan" <pa...@apache.org> wrote:
>
> Hi, Matthias,
> Thank you  very much for your comments!
> I have carefully read your reply and made some changes in the hope of
> making improvements.
> Please help take a look.
>
> For your comments:
>
> 1. You mention a few options for when it comes to storing the data which is
> good. The FLIP doesn't point out, though, what option you're going to go
> for as part of this FLIP (as far as I can see). It would be good to only
> outline the option to go for in the FLIP and list the other options as
> rejected alternatives (with the pro's and con's). I think it make sense to
> go for option 3 (i.e. following what's done for the ExecutionGraphInfoStore
> for now). The other options can be considered as a follow-up.
>
>
> This is very meaningful. Based on this comment, I have kept option 3 in
> its original place and moved the other candidate options to [1].
>
> 2. About the terminal states of a rescaling (i.e. IGNORED, FAILED,
> COMPLETED): Can we we clarify in the FLIP under what conditions the
> rescaling transitions into each of the three terminal states?
>
>
> Yes, this is a reasonable request for understanding and explaining the
> logic of transitions to terminated states.
> A new subsection [2] has been added to address this.
>
> 3. The section "The information to record in a rescale event" could be
> restructured in four sections (to remove redundancy):
> a) The IDs (Rescale
> ID, resourceRequirementsEpochID,
> subRescaleIdOfResourceRequirementsEpochID):
> What about making these names easier to read: GlobalRescaleID, RescaleUUID,
> RescaleAttemptId)
> b) Per-vertex data which includes: JobVertexID, JobVertexName,
> SlotSharingGroupId, the different parallelisms (pre-rescale, sufficient,
> desired, post-rescale)
> c) The SlotSharingGroup information: SlotSharingGroupId, name,
> ResourceProfile
> d) Other information: Timestamps of state transitions, etc. as laid out in
> the FLIP already
>
>
> That makes sense to me. Please check [3] for the latest updates in this
> part.
>
> 4. The FLIP doesn't explain how the data is passed through the
> AdaptiveScheduler states. We should be handling some kind of
> RescaleSnapshot that is passed through the different states and updated and
> its final state is stored somewhere within AdaptiveScheduler in the end, I
> guess. Can we clarify that in the FLIP?
>
>
> Indeed — this was missing in the original FLIP. To address this, I have
> added [4], which focuses on describing how a Rescale is represented,
> and how we can quickly pass and maintain the Rescale history.
>
> 5. You mention the config parameters for the cache in the public interface
> section. But there's no mentioning of any caching and how that is used
> within the FLIP.
>
>
> Sorry for the rough description in the previous version.
> Since this part belongs to the REST API acceleration mechanism for
> rescaling, and Option 6 seems reasonable to me,
> I plan to add it to FLIP-487 once the design of FLIP-495 has reached
> consensus.
> Of course, if needed, I'd be happy to clarify the usage and purpose of
> this parameter in the current email thread.
>
> 6. The REST endpoint is probably better suited in FLIP-487. FLIP-495 should
> be about the actual implementation details and how the data is stored
> internally whereas FLIP-487 is about exposing the information to the
> outside through the REST API and the Flink UI. That would be a way to
> decrease the scope of FLIP-495. WDYT?
>
>
> That sounds nice to me. Therefore, I have moved all REST API–related
> changes to FLIP-487.
> BTW, to avoid repetitive changes in FLIP-487, I'll start organizing
> FLIP-487 after FLIP-495 has been finalized.
>
> Looking forward to your next review!
>
> [1]
>
> https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=334760525#FLIP495:SupportAdaptiveSchedulerrecordandquerytherescalehistory-Aboutrescaleeventsstorage.1
> [2]
>
> https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=334760525#FLIP495:SupportAdaptiveSchedulerrecordandquerytherescalehistory-ThemainscenarioswhereRescalestatusswitchestoterminated
> [3]
>
> https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=334760525#FLIP495:SupportAdaptiveSchedulerrecordandquerytherescalehistory-Theinformationtorecordinarescaleevent
> [4]
>
> https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=334760525#FLIP495:SupportAdaptiveSchedulerrecordandquerytherescalehistory-InternalInterfaces
>
>
> Best regards,
> Yuepeng Pan
>
>
>
>
>
> At 2025-08-10 23:54:37, "Matthias Pohl" <ma...@apache.org> wrote:
>
> Hi Yuepeng,
> thanks for reminding me of this FLIP. I went over it and have a few items
> which we might need to address before we can actually finalize the vote:
>
> 1. You mention a few options for when it comes to storing the data which is
> good. The FLIP doesn't point out, though, what option you're going to go
> for as part of this FLIP (as far as I can see). It would be good to only
> outline the option to go for in the FLIP and list the other options as
> rejected alternatives (with the pro's and con's). I think it make sense to
> go for option 3 (i.e. following what's done for the ExecutionGraphInfoStore
> for now). The other options can be considered as a follow-up.
> 2. About the terminal states of a rescaling (i.e. IGNORED, FAILED,
> COMPLETED): Can we we clarify in the FLIP under what conditions the
> rescaling transitions into each of the three terminal states?
> 3. The section "The information to record in a rescale event" could be
> restructured in four sections (to remove redundancy):
> a) The IDs (Rescale
> ID, resourceRequirementsEpochID,
> subRescaleIdOfResourceRequirementsEpochID):
> What about making these names easier to read: GlobalRescaleID, RescaleUUID,
> RescaleAttemptId)
> b) Per-vertex data which includes: JobVertexID, JobVertexName,
> SlotSharingGroupId, the different parallelisms (pre-rescale, sufficient,
> desired, post-rescale)
> c) The SlotSharingGroup information: SlotSharingGroupId, name,
> ResourceProfile
> d) Other information: Timestamps of state transitions, etc. as laid out in
> the FLIP already
> 4. The FLIP doesn't explain how the data is passed through the
> AdaptiveScheduler states. We should be handling some kind of
> RescaleSnapshot that is passed through the different states and updated and
> its final state is stored somewhere within AdaptiveScheduler in the end, I
> guess. Can we clarify that in the FLIP?
> 5. You mention the config parameters for the cache in the public interface
> section. But there's no mentioning of any caching and how that is used
> within the FLIP.
> 6. The REST endpoint is probably better suited in FLIP-487. FLIP-495 should
> be about the actual implementation details and how the data is stored
> internally whereas FLIP-487 is about exposing the information to the
> outside through the REST API and the Flink UI. That would be a way to
> decrease the scope of FLIP-495. WDYT?
>
> Best,
> Matthias
>
>
> On Mon, Mar 24, 2025 at 11:37 AM Yuepeng Pan <pa...@apache.org>
> wrote:
>
> Hi, Community,
>
> There haven’t been any further responses to this email over the past few
> days.
> I'd like to initiate a vote on the current proposal[1] in the next few
> days.
> Please rest assured that I’m proceeding cautiously and not rushing the
> process.
> If there are any concerns about this FLIP-495[1],
> I will gladly pause and make the adjustments.
>
> Best regards,
> Yuepeng Pan
>
> [1]
>
>
> https://cwiki.apache.org/confluence/display/FLINK/FLIP-495%3A+Support+AdaptiveScheduler+record+and+query+the+rescale+history
>
>
> On 2024/12/17 15:18:45 Yuepeng Pan wrote:
>
> Hi community,
>
>
>
>
> We discussed several aspects of FLIP-487[1] 'Show history of rescales in
>
> Web UI for AdaptiveScheduler'
>
> and received a lot of valuable feedback. Based on the suggestions from
>
> the email thread[2],
>
> we plan to split the original proposal for FLIP-487[1].
>
>
>
>
> The current email thread and the FLIP-495[3] wiki will be used to
>
> discuss 'Support AdaptiveScheduler in recording and querying the rescale
> history',
>
> while FLIP-487[1] will primarily focus on displaying-related design
>
> content
>
>
>
>
>
> Looking forward to any feedback and opinions on FLIP-495[3].
>
>
>
>
> [1]
>
>
>
> https://cwiki.apache.org/confluence/display/FLINK/%5BWIP%5D+FLIP-487%3A+Show+history+of+rescales+in+Web+UI+for+AdaptiveScheduler
>
>
> [2] https://lists.apache.org/thread/f4md4btkf006mxcxf66bng1kfz0rsn8c
>
> [3]
>
>
>
> https://cwiki.apache.org/confluence/display/FLINK/%5BWIP%5D+FLIP-495%3A+Support+AdaptiveScheduler+record+and+query+the+rescale+history
>
>
>
>
>
> Thank you very much.
>
>
>
>
> Best,
>
> Regards.
>
> Yuepeng Pan
>
>
>
>
>
>

Re: [DISCUSS] FLIP-495: Support AdaptiveScheduler record and query the rescale history

Posted by Roc Marshal <fl...@126.com>.
Hi, Matthias.
Thank you very much for the comments.



> I was thinking about the case where a rescale was initiated by
> calling updateJobResourcesRequirements but it's not executed, yet, before
> the error occurs. In that case, a restart will be triggered that implicitly
> executes the rescale based on the available resources. If the available
> resources match the desired resource requirements of the
> updateJobResourcesRequirements, we execute the rescale through the job
> restart which was triggered by the error handling. That would result in a
> rescale that was actually "successful/completed".

IIUC, did you mean making some adjustments in such cases to reduce the number of ignored rescale records triggered by updateJobResourcesRequirements in the mentioned about case?
If so, I think that is reasonable, and I will handle it accordingly during the implementation.

> The State implementations handle AdaptiveScheduler State-specific
> functionality but the corresponding Context interfaces provide the methods
> to actually modify the AdaptiveScheduler instance itself. The
> RescaleTimeline is part of the AdaptiveScheduler instance and would be
> exposed through that instance (analogously to the exception history in
> AdaptiveScheduler#requestJob).

In the current design, AdaptiveScheduler has a method #getRescaleTimeline,
so we are already able to access the RescaleTimeline object within the AdaptiveScheduler instance and use it to handle rescale-related logic.
Sorry, I may not understand your point.
Did you mean that we still need some changes like the following?
- Remove AdaptiveScheduler#getRescaleTimeline
- Remove StateWithExecutionGraph#getRescaleTimeline
- Remove StateWithoutExecutionGraph#getRescaleTimeline
- Add interfaces in State.Context to operate on rescale in RescaleTimeline directly, so as to avoid an extra #getRescaleTimeline method calling.
I would appreciate any corrections or suggestions.

For Other notes:

> - nit (implementation detail): I'm not sure whether we need to keep the
> latestRescales as part of the RescalesStatsSnapshot. That could be quickly
> derived from RescalesStatsSnapshot#rescaleHistory (to reduce redundancy in
> the code) assuming that this list doesn't contain millions of entries. But
> that's an implementation detail. Not sure whether that even needs to be
> mentioned in the FLIP.
> - nit (implementation detail): VertexParallelismRescale#pre/postParallelism
> should be named pre/postRescaleParallelism. Same applies
> to SlotSharingGroupRescale#pre/postSlot
> - nit (implementation detail): The name of the SlotSharingGroup does not
> need to be stored in the VertexParallelismRescale as far as I can see. The
> SlotSharingGroup which holds the name is referenced through the ID.

Reasonable! Updated it.

> - Do we still need RescaleStatus in the Rescale structure? That seems to be
> a transient field that should be derived via the AdaptiveScheduler's state.

Thank you very much for the reminder.
This is exactly the point I have been reconsidering.
After introducing TerminalState and TerminatedReason, this field has lost its meaning.
If users need to pay attention to the AdaptiveScheduler state information, they can obtain it from schedulerStates.
Therefore, removing this field is reasonable to me, too.



Looking forward to your comments!


Best regards,
Yuepeng Pan
---- Replied Message ----
| From | Matthias Pohl<ma...@apache.org> |
| Date | 08/25/2025 15:14 |
| To | <de...@flink.apache.org> |
| Cc | Roc Marshal<fl...@126.com> |
| Subject | Re: [DISCUSS] FLIP-495: Support AdaptiveScheduler record and query the rescale history |
Hi Yuepeng,
thanks for the update. Please see my responses below:

If, during failure handling and job recovery (i.e., when there is an
ongoing rescale), a new resource requirement change or new available
resource change occurs (provided that such a change can trigger a new
rescale), the ongoing rescale will be forcibly terminated and saved.


I was thinking about the case where a rescale was initiated by
calling updateJobResourcesRequirements but it's not executed, yet, before
the error occurs. In that case, a restart will be triggered that implicitly
executes the rescale based on the available resources. If the available
resources match the desired resource requirements of the
updateJobResourcesRequirements, we execute the rescale through the job
restart which was triggered by the error handling. That would result in a
rescale that was actually "successful/completed".

- Why does StateTransitions need to be extended with
getRescaleTimeline()?



- This is because we want to quickly obtain a reference to the
RescaleTimeline, and maintain rescale information through it.
- It was introduced into the StateTransitions class so that all states can
call #getRescaleTimeline via this interface.
- After re-examining the scope affected by rescale maintenance operations,
it might be more precise to introduce this method into
StateWithExecutionGraph.Context / StateWithoutExecutionGraph.Context [1]
instead.


The State implementations handle AdaptiveScheduler State-specific
functionality but the corresponding Context interfaces provide the methods
to actually modify the AdaptiveScheduler instance itself. The
RescaleTimeline is part of the AdaptiveScheduler instance and would be
exposed through that instance (analogously to the exception history in
AdaptiveScheduler#requestJob).

Other notes:
- nit (implementation detail): I'm not sure whether we need to keep the
latestRescales as part of the RescalesStatsSnapshot. That could be quickly
derived from RescalesStatsSnapshot#rescaleHistory (to reduce redundancy in
the code) assuming that this list doesn't contain millions of entries. But
that's an implementation detail. Not sure whether that even needs to be
mentioned in the FLIP.

- nit (implementation detail): VertexParallelismRescale#pre/postParallelism
should be named pre/postRescaleParallelism. Same applies
to SlotSharingGroupRescale#pre/postSlot

- nit (implementation detail): The name of the SlotSharingGroup does not
need to be stored in the VertexParallelismRescale as far as I can see. The
SlotSharingGroup which holds the name is referenced through the ID.

- Do we still need RescaleStatus in the Rescale structure? That seems to be
a transient field that should be derived via the AdaptiveScheduler's state.

Thanks,
Matthias

On Fri, Aug 22, 2025 at 1:19 PM Roc Marshal <fl...@126.com> wrote:

Hi, Matthias.
Thank you very much for the comments.

- Why do we mark the rescale operation as IGNORED if an (repeatable)
exception causes a restart?

Here we mainly discuss two scenarios:
- If a recoverable exception occurs while there is an ongoing rescale, the
current rescale will be forcibly terminated and saved. Then, a new rescale
will be created to record the subsequent rescale information.
- If there is no ongoing rescale, a new rescale will be created directly
to record the information.

That's because the failure will create a new rescale instance that will
be saved if the resources changed as part of the failure handling/job
restart process?

If, during failure handling and job recovery (i.e., when there is an
ongoing rescale), a new resource requirement change or new available
resource change occurs (provided that such a change can trigger a new
rescale), the ongoing rescale will be forcibly terminated and saved.
Then, the system will create a new rescale to continue recording the
subsequent process.

- Thinking about the GlobalRescaleId: Should we still keep it
considering that we're not preserving the value on JM failovers (because of
the missing HA functionality) in this FLIP? We could remove it and add it
as part of a HA-support follow-up. Providing it despite the missing HA
support might be misleading in case a JM failover actually happens. WDYT?

Nice proposal! That makes sense to me~

- For the job vertices, you introduced a 5th resource type (previous,
acquired, required, min bound, max bound). What's the difference? I guess,
min bound corresponds to "sufficient resources" and max bound corresponds
to "desired resources". What are required resources here in that case? It
might be good to stick to the naming that was introduced in FLIP-472 [1].

I’ve already made the latest changes.
BTW, the original intention here was to record information as similar as
possible to the one from #updateJobResourcesRequirements requests. Now I
believe it’s quite necessary to remove the previous design and use a
unified description instead, since this makes things more consistent and
easier to understand. Many thanks for your suggestions.

- "Job Vertices Rescaled Information" and "Slots / Resources Rescaled
Information" seem to have redundant information now: "parallelism of job
vertex" and "slot number of the slot sharing group" are more or less the
same. The parallelism is defined per job vertex (along the
SlotSharingGroupId). The SlotSharingGroup holds the ResourceProfile (but no
parallelism).

Please let me to explain the reasoning:
- The parallelism of job vertices is kept so that users can clearly see
the parallelism of each job vertex.
- Adding information about the number of slots in a SlotSharingGroup is
intended to give users a more intuitive understanding of how many resources
this type of group consumes. Of course, even if we remove the slot number
in the SlotSharingGroup, users could still calculate it by themselves based
on job vertices paralllelsims, but that would require them to be familiar
with the AdaptiveScheduler’s logic for computing slot numbers.
- In addition, if we only keep the information about the number of slots
in the SlotSharingGroup and drop the parallelism information at the
job-vertex level, then users would lose the fine-grained parallelism
details of job vertices during a rescale. They would only be able to see
SlotSharingGroup-level information, which is somewhat like parallelism but
less precise.
- If necessary, I’m willing to remove the slot-number-related field from
the SlotSharingGroup description.
Please let me know your thoughts.

- "Other information" section contains a comment message for the reason
why a rescale event terminates. Reason might be a better term than comment
(especially because we're relying on an enum here rather than plain text).

Good idea — I’ve already made the corresponding changes.

- Based on the current proposal of the FLIP, the intermediate states of
the rescale event actually match the AdaptiveScheduler's states. Only the
final states are exclusive to the scale event?  Would it make sense to rely
on the AdaptiveScheduler state entirely for the intermediate state and a
dedicated field "terminalState"?

Yes, your understanding is correct. This approach works well for me, and
I’ve already made the corresponding changes in the wiki document.

- Why does StateTransitions need to be extended with
getRescaleTimeline()?

- This is because we want to quickly obtain a reference to the
RescaleTimeline, and maintain rescale information through it.
- It was introduced into the StateTransitions class so that all states can
call #getRescaleTimeline via this interface.
- After re-examining the scope affected by rescale maintenance operations,
it might be more precise to introduce this method into
StateWithExecutionGraph.Context / StateWithoutExecutionGraph.Context [1]
instead.

Shouldn't the RescaleTimeline be located in the AdaptiveScheduler?

Yes. Whether #getRescaleTimeline is introduced in StateTransitions or in
StateWithExecutionGraph.Context / StateWithoutExecutionGraph.Context, the
RescaleTimeline should still be part of the AdaptiveScheduler.

Instead, the State.Context (essentially, the AdaptiveScheduler) should
provide methods for updating the current rescale process object that lives
in the AdaptiveScheduler. WDYT?

Sounds good to me. Once #getRescaleTimeline is introduced in
State.Context, it can directly obtain a reference to the RescaleTimeline,
allowing quick operations on the current rescale.
What do you think?


- Section "How to process rescales storage ?": ExecutionGraphInfo could
be modified to include the Rescale history (along the exception history).
The data can be then saved in the ExecutionGraphInfoStore. All you have to
do is to store the rescale history in AdaptiveScheduler and an object that
holds the infos of the current rescale process. If this current rescale
process terminates, an immutable rescale snapshot of this event is created
that is saved in the rescale history.

Sorry for my previous wording was ambiguous and inconsistent with the
documentation. Thanks a lot for pointing it out. This is actually what I
intended to do, and I’ve already corrected the corresponding description in
the document.

[1]
https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=334760525#FLIP495:SupportAdaptiveSchedulerrecordandquerytherescalehistory-Howtohold/maintaintherescaleshistory
?

Best,
Yuepeng Pan



Roc Marshal
flinker@126.com

---- Replied Message ----
From Matthias Pohl<ma...@apache.org> <ma...@apache.org>
Date 08/21/2025 18:54
To dev@flink.apache.org<de...@flink.apache.org> <de...@flink.apache.org>
Cc Yuepeng Pan<pa...@apache.org> <pa...@apache.org>
Subject Re: [DISCUSS] FLIP-495: Support AdaptiveScheduler record and
query the rescale history
Thanks for the update, Yuepeng. Here are my remarks:

- Why do we mark the rescale operation as IGNORED if an (repeatable)
exception causes a restart? That's because the failure will create a new
rescale instance that will be saved if the resources changed as part of the
failure handling/job restart process?

- Thinking about the GlobalRescaleId: Should we still keep it considering
that we're not preserving the value on JM failovers (because of the missing
HA functionality) in this FLIP? We could remove it and add it as part of a
HA-support follow-up. Providing it despite the missing HA support might be
misleading in case a JM failover actually happens. WDYT?

- For the job vertices, you introduced a 5th resource type (previous,
acquired, required, min bound, max bound). What's the difference? I guess,
min bound corresponds to "sufficient resources" and max bound corresponds
to "desired resources". What are required resources here in that case? It
might be good to stick to the naming that was introduced in FLIP-472 [1].

- "Job Vertices Rescaled Information" and "Slots / Resources Rescaled
Information" seem to have redundant information now: "parallelism of job
vertex" and "slot number of the slot sharing group" are more or less the
same. The parallelism is defined per job vertex (along the
SlotSharingGroupId). The SlotSharingGroup holds the ResourceProfile (but no
parallelism).

- "Other information" section contains a comment message for the reason why
a rescale event terminates. Reason might be a better term than comment
(especially because we're relying on an enum here rather than plain text).

- Based on the current proposal of the FLIP, the intermediate states of the
rescale event actually match the AdaptiveScheduler's states. Only the final
states are exclusive to the scale event? Would it make sense to rely on the
AdaptiveScheduler state entirely for the intermediate state and a dedicated
field "terminalState"?

- Why does StateTransitions need to be extended with getRescaleTimeline()?
Shouldn't the RescaleTimeline be located in the AdaptiveScheduler? Instead,
the State.Context (essentially, the AdaptiveScheduler) should provide
methods for updating the current rescale process object that lives in the
AdaptiveScheduler. WDYT?

- Section "How to process rescales storage ?": ExecutionGraphInfo could be
modified to include the Rescale history (along the exception history). The
data can be then saved in the ExecutionGraphInfoStore. All you have to do
is to store the rescale history in AdaptiveScheduler and an object that
holds the infos of the current rescale process. If this current rescale
process terminates, an immutable rescale snapshot of this event is created
that is saved in the rescale history.

Matthias

[1]

https://cwiki.apache.org/confluence/display/FLINK/FLIP-472%3A+Aligning+timeout+logic+in+the+AdaptiveScheduler%27s+WaitingForResources+and+Executing+states

On Tue, Aug 19, 2025 at 11:14 AM Yuepeng Pan <pa...@apache.org>
wrote:

Bumping this thread kindly. Thanks!

Best,
Yuepeng Pan


At 2025-08-13 14:52:26, "Yuepeng Pan" <pa...@apache.org> wrote:

Hi, Matthias,
Thank you  very much for your comments!
I have carefully read your reply and made some changes in the hope of
making improvements.
Please help take a look.

For your comments:

1. You mention a few options for when it comes to storing the data which is
good. The FLIP doesn't point out, though, what option you're going to go
for as part of this FLIP (as far as I can see). It would be good to only
outline the option to go for in the FLIP and list the other options as
rejected alternatives (with the pro's and con's). I think it make sense to
go for option 3 (i.e. following what's done for the ExecutionGraphInfoStore
for now). The other options can be considered as a follow-up.


This is very meaningful. Based on this comment, I have kept option 3 in
its original place and moved the other candidate options to [1].

2. About the terminal states of a rescaling (i.e. IGNORED, FAILED,
COMPLETED): Can we we clarify in the FLIP under what conditions the
rescaling transitions into each of the three terminal states?


Yes, this is a reasonable request for understanding and explaining the
logic of transitions to terminated states.
A new subsection [2] has been added to address this.

3. The section "The information to record in a rescale event" could be
restructured in four sections (to remove redundancy):
a) The IDs (Rescale
ID, resourceRequirementsEpochID,
subRescaleIdOfResourceRequirementsEpochID):
What about making these names easier to read: GlobalRescaleID, RescaleUUID,
RescaleAttemptId)
b) Per-vertex data which includes: JobVertexID, JobVertexName,
SlotSharingGroupId, the different parallelisms (pre-rescale, sufficient,
desired, post-rescale)
c) The SlotSharingGroup information: SlotSharingGroupId, name,
ResourceProfile
d) Other information: Timestamps of state transitions, etc. as laid out in
the FLIP already


That makes sense to me. Please check [3] for the latest updates in this
part.

4. The FLIP doesn't explain how the data is passed through the
AdaptiveScheduler states. We should be handling some kind of
RescaleSnapshot that is passed through the different states and updated and
its final state is stored somewhere within AdaptiveScheduler in the end, I
guess. Can we clarify that in the FLIP?


Indeed — this was missing in the original FLIP. To address this, I have
added [4], which focuses on describing how a Rescale is represented,
and how we can quickly pass and maintain the Rescale history.

5. You mention the config parameters for the cache in the public interface
section. But there's no mentioning of any caching and how that is used
within the FLIP.


Sorry for the rough description in the previous version.
Since this part belongs to the REST API acceleration mechanism for
rescaling, and Option 6 seems reasonable to me,
I plan to add it to FLIP-487 once the design of FLIP-495 has reached
consensus.
Of course, if needed, I'd be happy to clarify the usage and purpose of
this parameter in the current email thread.

6. The REST endpoint is probably better suited in FLIP-487. FLIP-495 should
be about the actual implementation details and how the data is stored
internally whereas FLIP-487 is about exposing the information to the
outside through the REST API and the Flink UI. That would be a way to
decrease the scope of FLIP-495. WDYT?


That sounds nice to me. Therefore, I have moved all REST API–related
changes to FLIP-487.
BTW, to avoid repetitive changes in FLIP-487, I'll start organizing
FLIP-487 after FLIP-495 has been finalized.

Looking forward to your next review!

[1]
https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=334760525#FLIP495:SupportAdaptiveSchedulerrecordandquerytherescalehistory-Aboutrescaleeventsstorage.1
[2]
https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=334760525#FLIP495:SupportAdaptiveSchedulerrecordandquerytherescalehistory-ThemainscenarioswhereRescalestatusswitchestoterminated
[3]
https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=334760525#FLIP495:SupportAdaptiveSchedulerrecordandquerytherescalehistory-Theinformationtorecordinarescaleevent
[4]
https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=334760525#FLIP495:SupportAdaptiveSchedulerrecordandquerytherescalehistory-InternalInterfaces


Best regards,
Yuepeng Pan





At 2025-08-10 23:54:37, "Matthias Pohl" <ma...@apache.org> wrote:

Hi Yuepeng,
thanks for reminding me of this FLIP. I went over it and have a few items
which we might need to address before we can actually finalize the vote:

1. You mention a few options for when it comes to storing the data which is
good. The FLIP doesn't point out, though, what option you're going to go
for as part of this FLIP (as far as I can see). It would be good to only
outline the option to go for in the FLIP and list the other options as
rejected alternatives (with the pro's and con's). I think it make sense to
go for option 3 (i.e. following what's done for the ExecutionGraphInfoStore
for now). The other options can be considered as a follow-up.
2. About the terminal states of a rescaling (i.e. IGNORED, FAILED,
COMPLETED): Can we we clarify in the FLIP under what conditions the
rescaling transitions into each of the three terminal states?
3. The section "The information to record in a rescale event" could be
restructured in four sections (to remove redundancy):
a) The IDs (Rescale
ID, resourceRequirementsEpochID,
subRescaleIdOfResourceRequirementsEpochID):
What about making these names easier to read: GlobalRescaleID, RescaleUUID,
RescaleAttemptId)
b) Per-vertex data which includes: JobVertexID, JobVertexName,
SlotSharingGroupId, the different parallelisms (pre-rescale, sufficient,
desired, post-rescale)
c) The SlotSharingGroup information: SlotSharingGroupId, name,
ResourceProfile
d) Other information: Timestamps of state transitions, etc. as laid out in
the FLIP already
4. The FLIP doesn't explain how the data is passed through the
AdaptiveScheduler states. We should be handling some kind of
RescaleSnapshot that is passed through the different states and updated and
its final state is stored somewhere within AdaptiveScheduler in the end, I
guess. Can we clarify that in the FLIP?
5. You mention the config parameters for the cache in the public interface
section. But there's no mentioning of any caching and how that is used
within the FLIP.
6. The REST endpoint is probably better suited in FLIP-487. FLIP-495 should
be about the actual implementation details and how the data is stored
internally whereas FLIP-487 is about exposing the information to the
outside through the REST API and the Flink UI. That would be a way to
decrease the scope of FLIP-495. WDYT?

Best,
Matthias


On Mon, Mar 24, 2025 at 11:37 AM Yuepeng Pan <pa...@apache.org>
wrote:

Hi, Community,

There haven’t been any further responses to this email over the past few
days.
I'd like to initiate a vote on the current proposal[1] in the next few
days.
Please rest assured that I’m proceeding cautiously and not rushing the
process.
If there are any concerns about this FLIP-495[1],
I will gladly pause and make the adjustments.

Best regards,
Yuepeng Pan

[1]

https://cwiki.apache.org/confluence/display/FLINK/FLIP-495%3A+Support+AdaptiveScheduler+record+and+query+the+rescale+history


On 2024/12/17 15:18:45 Yuepeng Pan wrote:

Hi community,




We discussed several aspects of FLIP-487[1] 'Show history of rescales in

Web UI for AdaptiveScheduler'

and received a lot of valuable feedback. Based on the suggestions from

the email thread[2],

we plan to split the original proposal for FLIP-487[1].




The current email thread and the FLIP-495[3] wiki will be used to

discuss 'Support AdaptiveScheduler in recording and querying the rescale
history',

while FLIP-487[1] will primarily focus on displaying-related design

content





Looking forward to any feedback and opinions on FLIP-495[3].




[1]


https://cwiki.apache.org/confluence/display/FLINK/%5BWIP%5D+FLIP-487%3A+Show+history+of+rescales+in+Web+UI+for+AdaptiveScheduler


[2] https://lists.apache.org/thread/f4md4btkf006mxcxf66bng1kfz0rsn8c

[3]


https://cwiki.apache.org/confluence/display/FLINK/%5BWIP%5D+FLIP-495%3A+Support+AdaptiveScheduler+record+and+query+the+rescale+history





Thank you very much.




Best,

Regards.

Yuepeng Pan






Re: [DISCUSS] FLIP-495: Support AdaptiveScheduler record and query the rescale history

Posted by Matthias Pohl <ma...@apache.org>.
Hi Yuepeng,
thanks for the update. Please see my responses below:

If, during failure handling and job recovery (i.e., when there is an
> ongoing rescale), a new resource requirement change or new available
> resource change occurs (provided that such a change can trigger a new
> rescale), the ongoing rescale will be forcibly terminated and saved.


I was thinking about the case where a rescale was initiated by
calling updateJobResourcesRequirements but it's not executed, yet, before
the error occurs. In that case, a restart will be triggered that implicitly
executes the rescale based on the available resources. If the available
resources match the desired resource requirements of the
updateJobResourcesRequirements, we execute the rescale through the job
restart which was triggered by the error handling. That would result in a
rescale that was actually "successful/completed".

> - Why does StateTransitions need to be extended with
> getRescaleTimeline()?



- This is because we want to quickly obtain a reference to the
> RescaleTimeline, and maintain rescale information through it.
> - It was introduced into the StateTransitions class so that all states can
> call #getRescaleTimeline via this interface.
> - After re-examining the scope affected by rescale maintenance operations,
> it might be more precise to introduce this method into
> StateWithExecutionGraph.Context / StateWithoutExecutionGraph.Context [1]
> instead.


The State implementations handle AdaptiveScheduler State-specific
functionality but the corresponding Context interfaces provide the methods
to actually modify the AdaptiveScheduler instance itself. The
RescaleTimeline is part of the AdaptiveScheduler instance and would be
exposed through that instance (analogously to the exception history in
AdaptiveScheduler#requestJob).

Other notes:
- nit (implementation detail): I'm not sure whether we need to keep the
latestRescales as part of the RescalesStatsSnapshot. That could be quickly
derived from RescalesStatsSnapshot#rescaleHistory (to reduce redundancy in
the code) assuming that this list doesn't contain millions of entries. But
that's an implementation detail. Not sure whether that even needs to be
mentioned in the FLIP.

- nit (implementation detail): VertexParallelismRescale#pre/postParallelism
should be named pre/postRescaleParallelism. Same applies
to SlotSharingGroupRescale#pre/postSlot

- nit (implementation detail): The name of the SlotSharingGroup does not
need to be stored in the VertexParallelismRescale as far as I can see. The
SlotSharingGroup which holds the name is referenced through the ID.

- Do we still need RescaleStatus in the Rescale structure? That seems to be
a transient field that should be derived via the AdaptiveScheduler's state.

Thanks,
Matthias

On Fri, Aug 22, 2025 at 1:19 PM Roc Marshal <fl...@126.com> wrote:

> Hi, Matthias.
> Thank you very much for the comments.
>
> > - Why do we mark the rescale operation as IGNORED if an (repeatable)
> exception causes a restart?
>
> Here we mainly discuss two scenarios:
> - If a recoverable exception occurs while there is an ongoing rescale, the
> current rescale will be forcibly terminated and saved. Then, a new rescale
> will be created to record the subsequent rescale information.
> - If there is no ongoing rescale, a new rescale will be created directly
> to record the information.
>
> > That's because the failure will create a new rescale instance that will
> be saved if the resources changed as part of the failure handling/job
> restart process?
>
> If, during failure handling and job recovery (i.e., when there is an
> ongoing rescale), a new resource requirement change or new available
> resource change occurs (provided that such a change can trigger a new
> rescale), the ongoing rescale will be forcibly terminated and saved.
> Then, the system will create a new rescale to continue recording the
> subsequent process.
>
> > - Thinking about the GlobalRescaleId: Should we still keep it
> considering that we're not preserving the value on JM failovers (because of
> the missing HA functionality) in this FLIP? We could remove it and add it
> as part of a HA-support follow-up. Providing it despite the missing HA
> support might be misleading in case a JM failover actually happens. WDYT?
>
> Nice proposal! That makes sense to me~
>
> > - For the job vertices, you introduced a 5th resource type (previous,
> acquired, required, min bound, max bound). What's the difference? I guess,
> min bound corresponds to "sufficient resources" and max bound corresponds
> to "desired resources". What are required resources here in that case? It
> might be good to stick to the naming that was introduced in FLIP-472 [1].
>
> I’ve already made the latest changes.
> BTW, the original intention here was to record information as similar as
> possible to the one from #updateJobResourcesRequirements requests. Now I
> believe it’s quite necessary to remove the previous design and use a
> unified description instead, since this makes things more consistent and
> easier to understand. Many thanks for your suggestions.
>
> > - "Job Vertices Rescaled Information" and "Slots / Resources Rescaled
> Information" seem to have redundant information now: "parallelism of job
> vertex" and "slot number of the slot sharing group" are more or less the
> same. The parallelism is defined per job vertex (along the
> SlotSharingGroupId). The SlotSharingGroup holds the ResourceProfile (but no
> parallelism).
>
> Please let me to explain the reasoning:
> - The parallelism of job vertices is kept so that users can clearly see
> the parallelism of each job vertex.
> - Adding information about the number of slots in a SlotSharingGroup is
> intended to give users a more intuitive understanding of how many resources
> this type of group consumes. Of course, even if we remove the slot number
> in the SlotSharingGroup, users could still calculate it by themselves based
> on job vertices paralllelsims, but that would require them to be familiar
> with the AdaptiveScheduler’s logic for computing slot numbers.
> - In addition, if we only keep the information about the number of slots
> in the SlotSharingGroup and drop the parallelism information at the
> job-vertex level, then users would lose the fine-grained parallelism
> details of job vertices during a rescale. They would only be able to see
> SlotSharingGroup-level information, which is somewhat like parallelism but
> less precise.
> - If necessary, I’m willing to remove the slot-number-related field from
> the SlotSharingGroup description.
> Please let me know your thoughts.
>
> > - "Other information" section contains a comment message for the reason
> why a rescale event terminates. Reason might be a better term than comment
> (especially because we're relying on an enum here rather than plain text).
>
> Good idea — I’ve already made the corresponding changes.
>
> > - Based on the current proposal of the FLIP, the intermediate states of
> the rescale event actually match the AdaptiveScheduler's states. Only the
> final states are exclusive to the scale event?  Would it make sense to rely
> on the AdaptiveScheduler state entirely for the intermediate state and a
> dedicated field "terminalState"?
>
> Yes, your understanding is correct. This approach works well for me, and
> I’ve already made the corresponding changes in the wiki document.
>
> > - Why does StateTransitions need to be extended with
> getRescaleTimeline()?
>
> - This is because we want to quickly obtain a reference to the
> RescaleTimeline, and maintain rescale information through it.
> - It was introduced into the StateTransitions class so that all states can
> call #getRescaleTimeline via this interface.
> - After re-examining the scope affected by rescale maintenance operations,
> it might be more precise to introduce this method into
> StateWithExecutionGraph.Context / StateWithoutExecutionGraph.Context [1]
> instead.
>
> > Shouldn't the RescaleTimeline be located in the AdaptiveScheduler?
>
> Yes. Whether #getRescaleTimeline is introduced in StateTransitions or in
> StateWithExecutionGraph.Context / StateWithoutExecutionGraph.Context, the
> RescaleTimeline should still be part of the AdaptiveScheduler.
>
> > Instead, the State.Context (essentially, the AdaptiveScheduler) should
> provide methods for updating the current rescale process object that lives
> in the AdaptiveScheduler. WDYT?
>
> Sounds good to me. Once #getRescaleTimeline is introduced in
> State.Context, it can directly obtain a reference to the RescaleTimeline,
> allowing quick operations on the current rescale.
> What do you think?
>
>
> > - Section "How to process rescales storage ?": ExecutionGraphInfo could
> be modified to include the Rescale history (along the exception history).
> The data can be then saved in the ExecutionGraphInfoStore. All you have to
> do is to store the rescale history in AdaptiveScheduler and an object that
> holds the infos of the current rescale process. If this current rescale
> process terminates, an immutable rescale snapshot of this event is created
> that is saved in the rescale history.
>
> Sorry for my previous wording was ambiguous and inconsistent with the
> documentation. Thanks a lot for pointing it out. This is actually what I
> intended to do, and I’ve already corrected the corresponding description in
> the document.
>
> [1]
> https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=334760525#FLIP495:SupportAdaptiveSchedulerrecordandquerytherescalehistory-Howtohold/maintaintherescaleshistory
> ?
>
> Best,
> Yuepeng Pan
>
>
>
> Roc Marshal
> flinker@126.com
>
> ---- Replied Message ----
> From Matthias Pohl<ma...@apache.org> <ma...@apache.org>
> Date 08/21/2025 18:54
> To dev@flink.apache.org<de...@flink.apache.org> <de...@flink.apache.org>
> Cc Yuepeng Pan<pa...@apache.org> <pa...@apache.org>
> Subject Re: [DISCUSS] FLIP-495: Support AdaptiveScheduler record and
> query the rescale history
> Thanks for the update, Yuepeng. Here are my remarks:
>
> - Why do we mark the rescale operation as IGNORED if an (repeatable)
> exception causes a restart? That's because the failure will create a new
> rescale instance that will be saved if the resources changed as part of the
> failure handling/job restart process?
>
> - Thinking about the GlobalRescaleId: Should we still keep it considering
> that we're not preserving the value on JM failovers (because of the missing
> HA functionality) in this FLIP? We could remove it and add it as part of a
> HA-support follow-up. Providing it despite the missing HA support might be
> misleading in case a JM failover actually happens. WDYT?
>
> - For the job vertices, you introduced a 5th resource type (previous,
> acquired, required, min bound, max bound). What's the difference? I guess,
> min bound corresponds to "sufficient resources" and max bound corresponds
> to "desired resources". What are required resources here in that case? It
> might be good to stick to the naming that was introduced in FLIP-472 [1].
>
> - "Job Vertices Rescaled Information" and "Slots / Resources Rescaled
> Information" seem to have redundant information now: "parallelism of job
> vertex" and "slot number of the slot sharing group" are more or less the
> same. The parallelism is defined per job vertex (along the
> SlotSharingGroupId). The SlotSharingGroup holds the ResourceProfile (but no
> parallelism).
>
> - "Other information" section contains a comment message for the reason why
> a rescale event terminates. Reason might be a better term than comment
> (especially because we're relying on an enum here rather than plain text).
>
> - Based on the current proposal of the FLIP, the intermediate states of the
> rescale event actually match the AdaptiveScheduler's states. Only the final
> states are exclusive to the scale event? Would it make sense to rely on the
> AdaptiveScheduler state entirely for the intermediate state and a dedicated
> field "terminalState"?
>
> - Why does StateTransitions need to be extended with getRescaleTimeline()?
> Shouldn't the RescaleTimeline be located in the AdaptiveScheduler? Instead,
> the State.Context (essentially, the AdaptiveScheduler) should provide
> methods for updating the current rescale process object that lives in the
> AdaptiveScheduler. WDYT?
>
> - Section "How to process rescales storage ?": ExecutionGraphInfo could be
> modified to include the Rescale history (along the exception history). The
> data can be then saved in the ExecutionGraphInfoStore. All you have to do
> is to store the rescale history in AdaptiveScheduler and an object that
> holds the infos of the current rescale process. If this current rescale
> process terminates, an immutable rescale snapshot of this event is created
> that is saved in the rescale history.
>
> Matthias
>
> [1]
>
> https://cwiki.apache.org/confluence/display/FLINK/FLIP-472%3A+Aligning+timeout+logic+in+the+AdaptiveScheduler%27s+WaitingForResources+and+Executing+states
>
> On Tue, Aug 19, 2025 at 11:14 AM Yuepeng Pan <pa...@apache.org>
> wrote:
>
> Bumping this thread kindly. Thanks!
>
> Best,
> Yuepeng Pan
>
>
> At 2025-08-13 14:52:26, "Yuepeng Pan" <pa...@apache.org> wrote:
>
> Hi, Matthias,
> Thank you  very much for your comments!
> I have carefully read your reply and made some changes in the hope of
> making improvements.
> Please help take a look.
>
> For your comments:
>
> 1. You mention a few options for when it comes to storing the data which is
> good. The FLIP doesn't point out, though, what option you're going to go
> for as part of this FLIP (as far as I can see). It would be good to only
> outline the option to go for in the FLIP and list the other options as
> rejected alternatives (with the pro's and con's). I think it make sense to
> go for option 3 (i.e. following what's done for the ExecutionGraphInfoStore
> for now). The other options can be considered as a follow-up.
>
>
> This is very meaningful. Based on this comment, I have kept option 3 in
> its original place and moved the other candidate options to [1].
>
> 2. About the terminal states of a rescaling (i.e. IGNORED, FAILED,
> COMPLETED): Can we we clarify in the FLIP under what conditions the
> rescaling transitions into each of the three terminal states?
>
>
> Yes, this is a reasonable request for understanding and explaining the
> logic of transitions to terminated states.
> A new subsection [2] has been added to address this.
>
> 3. The section "The information to record in a rescale event" could be
> restructured in four sections (to remove redundancy):
> a) The IDs (Rescale
> ID, resourceRequirementsEpochID,
> subRescaleIdOfResourceRequirementsEpochID):
> What about making these names easier to read: GlobalRescaleID, RescaleUUID,
> RescaleAttemptId)
> b) Per-vertex data which includes: JobVertexID, JobVertexName,
> SlotSharingGroupId, the different parallelisms (pre-rescale, sufficient,
> desired, post-rescale)
> c) The SlotSharingGroup information: SlotSharingGroupId, name,
> ResourceProfile
> d) Other information: Timestamps of state transitions, etc. as laid out in
> the FLIP already
>
>
> That makes sense to me. Please check [3] for the latest updates in this
> part.
>
> 4. The FLIP doesn't explain how the data is passed through the
> AdaptiveScheduler states. We should be handling some kind of
> RescaleSnapshot that is passed through the different states and updated and
> its final state is stored somewhere within AdaptiveScheduler in the end, I
> guess. Can we clarify that in the FLIP?
>
>
> Indeed — this was missing in the original FLIP. To address this, I have
> added [4], which focuses on describing how a Rescale is represented,
> and how we can quickly pass and maintain the Rescale history.
>
> 5. You mention the config parameters for the cache in the public interface
> section. But there's no mentioning of any caching and how that is used
> within the FLIP.
>
>
> Sorry for the rough description in the previous version.
> Since this part belongs to the REST API acceleration mechanism for
> rescaling, and Option 6 seems reasonable to me,
> I plan to add it to FLIP-487 once the design of FLIP-495 has reached
> consensus.
> Of course, if needed, I'd be happy to clarify the usage and purpose of
> this parameter in the current email thread.
>
> 6. The REST endpoint is probably better suited in FLIP-487. FLIP-495 should
> be about the actual implementation details and how the data is stored
> internally whereas FLIP-487 is about exposing the information to the
> outside through the REST API and the Flink UI. That would be a way to
> decrease the scope of FLIP-495. WDYT?
>
>
> That sounds nice to me. Therefore, I have moved all REST API–related
> changes to FLIP-487.
> BTW, to avoid repetitive changes in FLIP-487, I'll start organizing
> FLIP-487 after FLIP-495 has been finalized.
>
> Looking forward to your next review!
>
> [1]
> https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=334760525#FLIP495:SupportAdaptiveSchedulerrecordandquerytherescalehistory-Aboutrescaleeventsstorage.1
> [2]
> https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=334760525#FLIP495:SupportAdaptiveSchedulerrecordandquerytherescalehistory-ThemainscenarioswhereRescalestatusswitchestoterminated
> [3]
> https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=334760525#FLIP495:SupportAdaptiveSchedulerrecordandquerytherescalehistory-Theinformationtorecordinarescaleevent
> [4]
> https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=334760525#FLIP495:SupportAdaptiveSchedulerrecordandquerytherescalehistory-InternalInterfaces
>
>
> Best regards,
> Yuepeng Pan
>
>
>
>
>
> At 2025-08-10 23:54:37, "Matthias Pohl" <ma...@apache.org> wrote:
>
> Hi Yuepeng,
> thanks for reminding me of this FLIP. I went over it and have a few items
> which we might need to address before we can actually finalize the vote:
>
> 1. You mention a few options for when it comes to storing the data which is
> good. The FLIP doesn't point out, though, what option you're going to go
> for as part of this FLIP (as far as I can see). It would be good to only
> outline the option to go for in the FLIP and list the other options as
> rejected alternatives (with the pro's and con's). I think it make sense to
> go for option 3 (i.e. following what's done for the ExecutionGraphInfoStore
> for now). The other options can be considered as a follow-up.
> 2. About the terminal states of a rescaling (i.e. IGNORED, FAILED,
> COMPLETED): Can we we clarify in the FLIP under what conditions the
> rescaling transitions into each of the three terminal states?
> 3. The section "The information to record in a rescale event" could be
> restructured in four sections (to remove redundancy):
> a) The IDs (Rescale
> ID, resourceRequirementsEpochID,
> subRescaleIdOfResourceRequirementsEpochID):
> What about making these names easier to read: GlobalRescaleID, RescaleUUID,
> RescaleAttemptId)
> b) Per-vertex data which includes: JobVertexID, JobVertexName,
> SlotSharingGroupId, the different parallelisms (pre-rescale, sufficient,
> desired, post-rescale)
> c) The SlotSharingGroup information: SlotSharingGroupId, name,
> ResourceProfile
> d) Other information: Timestamps of state transitions, etc. as laid out in
> the FLIP already
> 4. The FLIP doesn't explain how the data is passed through the
> AdaptiveScheduler states. We should be handling some kind of
> RescaleSnapshot that is passed through the different states and updated and
> its final state is stored somewhere within AdaptiveScheduler in the end, I
> guess. Can we clarify that in the FLIP?
> 5. You mention the config parameters for the cache in the public interface
> section. But there's no mentioning of any caching and how that is used
> within the FLIP.
> 6. The REST endpoint is probably better suited in FLIP-487. FLIP-495 should
> be about the actual implementation details and how the data is stored
> internally whereas FLIP-487 is about exposing the information to the
> outside through the REST API and the Flink UI. That would be a way to
> decrease the scope of FLIP-495. WDYT?
>
> Best,
> Matthias
>
>
> On Mon, Mar 24, 2025 at 11:37 AM Yuepeng Pan <pa...@apache.org>
> wrote:
>
> Hi, Community,
>
> There haven’t been any further responses to this email over the past few
> days.
> I'd like to initiate a vote on the current proposal[1] in the next few
> days.
> Please rest assured that I’m proceeding cautiously and not rushing the
> process.
> If there are any concerns about this FLIP-495[1],
> I will gladly pause and make the adjustments.
>
> Best regards,
> Yuepeng Pan
>
> [1]
>
> https://cwiki.apache.org/confluence/display/FLINK/FLIP-495%3A+Support+AdaptiveScheduler+record+and+query+the+rescale+history
>
>
> On 2024/12/17 15:18:45 Yuepeng Pan wrote:
>
> Hi community,
>
>
>
>
> We discussed several aspects of FLIP-487[1] 'Show history of rescales in
>
> Web UI for AdaptiveScheduler'
>
> and received a lot of valuable feedback. Based on the suggestions from
>
> the email thread[2],
>
> we plan to split the original proposal for FLIP-487[1].
>
>
>
>
> The current email thread and the FLIP-495[3] wiki will be used to
>
> discuss 'Support AdaptiveScheduler in recording and querying the rescale
> history',
>
> while FLIP-487[1] will primarily focus on displaying-related design
>
> content
>
>
>
>
>
> Looking forward to any feedback and opinions on FLIP-495[3].
>
>
>
>
> [1]
>
>
> https://cwiki.apache.org/confluence/display/FLINK/%5BWIP%5D+FLIP-487%3A+Show+history+of+rescales+in+Web+UI+for+AdaptiveScheduler
>
>
> [2] https://lists.apache.org/thread/f4md4btkf006mxcxf66bng1kfz0rsn8c
>
> [3]
>
>
> https://cwiki.apache.org/confluence/display/FLINK/%5BWIP%5D+FLIP-495%3A+Support+AdaptiveScheduler+record+and+query+the+rescale+history
>
>
>
>
>
> Thank you very much.
>
>
>
>
> Best,
>
> Regards.
>
> Yuepeng Pan
>
>
>
>
>

Re: [DISCUSS] FLIP-495: Support AdaptiveScheduler record and query the rescale history

Posted by Roc Marshal <fl...@126.com>.
Hi, Matthias.
Thank you very much for the comments.

> - Why do we mark the rescale operation as IGNORED if an (repeatable) exception causes a restart?

Here we mainly discuss two scenarios:
- If a recoverable exception occurs while there is an ongoing rescale, the current rescale will be forcibly terminated and saved. Then, a new rescale will be created to record the subsequent rescale information.
- If there is no ongoing rescale, a new rescale will be created directly to record the information.

> That's because the failure will create a new rescale instance that will be saved if the resources changed as part of the failure handling/job restart process?

If, during failure handling and job recovery (i.e., when there is an ongoing rescale), a new resource requirement change or new available resource change occurs (provided that such a change can trigger a new rescale), the ongoing rescale will be forcibly terminated and saved.
Then, the system will create a new rescale to continue recording the subsequent process.

> - Thinking about the GlobalRescaleId: Should we still keep it considering that we're not preserving the value on JM failovers (because of the missing HA functionality) in this FLIP? We could remove it and add it as part of a HA-support follow-up. Providing it despite the missing HA support might be misleading in case a JM failover actually happens. WDYT?

Nice proposal! That makes sense to me~

> - For the job vertices, you introduced a 5th resource type (previous, acquired, required, min bound, max bound). What's the difference? I guess, min bound corresponds to "sufficient resources" and max bound corresponds to "desired resources". What are required resources here in that case? It might be good to stick to the naming that was introduced in FLIP-472 [1].

I’ve already made the latest changes.
BTW, the original intention here was to record information as similar as possible to the one from #updateJobResourcesRequirements requests. Now I believe it’s quite necessary to remove the previous design and use a unified description instead, since this makes things more consistent and easier to understand. Many thanks for your suggestions.

> - "Job Vertices Rescaled Information" and "Slots / Resources Rescaled Information" seem to have redundant information now: "parallelism of job vertex" and "slot number of the slot sharing group" are more or less the same. The parallelism is defined per job vertex (along the SlotSharingGroupId). The SlotSharingGroup holds the ResourceProfile (but no parallelism).

Please let me to explain the reasoning:
- The parallelism of job vertices is kept so that users can clearly see the parallelism of each job vertex.
- Adding information about the number of slots in a SlotSharingGroup is intended to give users a more intuitive understanding of how many resources this type of group consumes. Of course, even if we remove the slot number in the SlotSharingGroup, users could still calculate it by themselves based on job vertices paralllelsims, but that would require them to be familiar with the AdaptiveScheduler’s logic for computing slot numbers.
- In addition, if we only keep the information about the number of slots in the SlotSharingGroup and drop the parallelism information at the job-vertex level, then users would lose the fine-grained parallelism details of job vertices during a rescale. They would only be able to see SlotSharingGroup-level information, which is somewhat like parallelism but less precise.
- If necessary, I’m willing to remove the slot-number-related field from the SlotSharingGroup description.
Please let me know your thoughts.

> - "Other information" section contains a comment message for the reason why a rescale event terminates. Reason might be a better term than comment (especially because we're relying on an enum here rather than plain text).

Good idea — I’ve already made the corresponding changes.

> - Based on the current proposal of the FLIP, the intermediate states of the rescale event actually match the AdaptiveScheduler's states. Only the final states are exclusive to the scale event?  Would it make sense to rely on the AdaptiveScheduler state entirely for the intermediate state and a dedicated field "terminalState"?

Yes, your understanding is correct. This approach works well for me, and I’ve already made the corresponding changes in the wiki document.

> - Why does StateTransitions need to be extended with getRescaleTimeline()?

- This is because we want to quickly obtain a reference to the RescaleTimeline, and maintain rescale information through it.
- It was introduced into the StateTransitions class so that all states can call #getRescaleTimeline via this interface.
- After re-examining the scope affected by rescale maintenance operations, it might be more precise to introduce this method into StateWithExecutionGraph.Context / StateWithoutExecutionGraph.Context [1] instead.

> Shouldn't the RescaleTimeline be located in the AdaptiveScheduler?

Yes. Whether #getRescaleTimeline is introduced in StateTransitions or in StateWithExecutionGraph.Context / StateWithoutExecutionGraph.Context, the RescaleTimeline should still be part of the AdaptiveScheduler.

> Instead, the State.Context (essentially, the AdaptiveScheduler) should provide methods for updating the current rescale process object that lives in the AdaptiveScheduler. WDYT?

Sounds good to me. Once #getRescaleTimeline is introduced in State.Context, it can directly obtain a reference to the RescaleTimeline, allowing quick operations on the current rescale.
What do you think?


> - Section "How to process rescales storage ?": ExecutionGraphInfo could be modified to include the Rescale history (along the exception history). The data can be then saved in the ExecutionGraphInfoStore. All you have to do is to store the rescale history in AdaptiveScheduler and an object that holds the infos of the current rescale process. If this current rescale process terminates, an immutable rescale snapshot of this event is created that is saved in the rescale history.

Sorry for my previous wording was ambiguous and inconsistent with the documentation. Thanks a lot for pointing it out. This is actually what I intended to do, and I’ve already corrected the corresponding description in the document.

[1] https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=334760525#FLIP495:SupportAdaptiveSchedulerrecordandquerytherescalehistory-Howtohold/maintaintherescaleshistory?

Best,
Yuepeng Pan






| |
Roc Marshal
|
|
flinker@126.com
|


---- Replied Message ----
| From | Matthias Pohl<ma...@apache.org> |
| Date | 08/21/2025 18:54 |
| To | dev@flink.apache.org<de...@flink.apache.org> |
| Cc | Yuepeng Pan<pa...@apache.org> |
| Subject | Re: [DISCUSS] FLIP-495: Support AdaptiveScheduler record and query the rescale history |
Thanks for the update, Yuepeng. Here are my remarks:

- Why do we mark the rescale operation as IGNORED if an (repeatable)
exception causes a restart? That's because the failure will create a new
rescale instance that will be saved if the resources changed as part of the
failure handling/job restart process?

- Thinking about the GlobalRescaleId: Should we still keep it considering
that we're not preserving the value on JM failovers (because of the missing
HA functionality) in this FLIP? We could remove it and add it as part of a
HA-support follow-up. Providing it despite the missing HA support might be
misleading in case a JM failover actually happens. WDYT?

- For the job vertices, you introduced a 5th resource type (previous,
acquired, required, min bound, max bound). What's the difference? I guess,
min bound corresponds to "sufficient resources" and max bound corresponds
to "desired resources". What are required resources here in that case? It
might be good to stick to the naming that was introduced in FLIP-472 [1].

- "Job Vertices Rescaled Information" and "Slots / Resources Rescaled
Information" seem to have redundant information now: "parallelism of job
vertex" and "slot number of the slot sharing group" are more or less the
same. The parallelism is defined per job vertex (along the
SlotSharingGroupId). The SlotSharingGroup holds the ResourceProfile (but no
parallelism).

- "Other information" section contains a comment message for the reason why
a rescale event terminates. Reason might be a better term than comment
(especially because we're relying on an enum here rather than plain text).

- Based on the current proposal of the FLIP, the intermediate states of the
rescale event actually match the AdaptiveScheduler's states. Only the final
states are exclusive to the scale event? Would it make sense to rely on the
AdaptiveScheduler state entirely for the intermediate state and a dedicated
field "terminalState"?

- Why does StateTransitions need to be extended with getRescaleTimeline()?
Shouldn't the RescaleTimeline be located in the AdaptiveScheduler? Instead,
the State.Context (essentially, the AdaptiveScheduler) should provide
methods for updating the current rescale process object that lives in the
AdaptiveScheduler. WDYT?

- Section "How to process rescales storage ?": ExecutionGraphInfo could be
modified to include the Rescale history (along the exception history). The
data can be then saved in the ExecutionGraphInfoStore. All you have to do
is to store the rescale history in AdaptiveScheduler and an object that
holds the infos of the current rescale process. If this current rescale
process terminates, an immutable rescale snapshot of this event is created
that is saved in the rescale history.

Matthias

[1]
https://cwiki.apache.org/confluence/display/FLINK/FLIP-472%3A+Aligning+timeout+logic+in+the+AdaptiveScheduler%27s+WaitingForResources+and+Executing+states

On Tue, Aug 19, 2025 at 11:14 AM Yuepeng Pan <pa...@apache.org> wrote:

Bumping this thread kindly. Thanks!

Best,
Yuepeng Pan


At 2025-08-13 14:52:26, "Yuepeng Pan" <pa...@apache.org> wrote:

Hi, Matthias,
Thank you  very much for your comments!
I have carefully read your reply and made some changes in the hope of making improvements.
Please help take a look.

For your comments:

1. You mention a few options for when it comes to storing the data which is
good. The FLIP doesn't point out, though, what option you're going to go
for as part of this FLIP (as far as I can see). It would be good to only
outline the option to go for in the FLIP and list the other options as
rejected alternatives (with the pro's and con's). I think it make sense to
go for option 3 (i.e. following what's done for the ExecutionGraphInfoStore
for now). The other options can be considered as a follow-up.

This is very meaningful. Based on this comment, I have kept option 3 in its original place and moved the other candidate options to [1].

2. About the terminal states of a rescaling (i.e. IGNORED, FAILED,
COMPLETED): Can we we clarify in the FLIP under what conditions the
rescaling transitions into each of the three terminal states?

Yes, this is a reasonable request for understanding and explaining the logic of transitions to terminated states.
A new subsection [2] has been added to address this.

3. The section "The information to record in a rescale event" could be
restructured in four sections (to remove redundancy):
a) The IDs (Rescale
ID, resourceRequirementsEpochID, subRescaleIdOfResourceRequirementsEpochID):
What about making these names easier to read: GlobalRescaleID, RescaleUUID,
RescaleAttemptId)
b) Per-vertex data which includes: JobVertexID, JobVertexName,
SlotSharingGroupId, the different parallelisms (pre-rescale, sufficient,
desired, post-rescale)
c) The SlotSharingGroup information: SlotSharingGroupId, name,
ResourceProfile
d) Other information: Timestamps of state transitions, etc. as laid out in
the FLIP already

That makes sense to me. Please check [3] for the latest updates in this part.

4. The FLIP doesn't explain how the data is passed through the
AdaptiveScheduler states. We should be handling some kind of
RescaleSnapshot that is passed through the different states and updated and
its final state is stored somewhere within AdaptiveScheduler in the end, I
guess. Can we clarify that in the FLIP?

Indeed — this was missing in the original FLIP. To address this, I have added [4], which focuses on describing how a Rescale is represented,
and how we can quickly pass and maintain the Rescale history.

5. You mention the config parameters for the cache in the public interface
section. But there's no mentioning of any caching and how that is used
within the FLIP.

Sorry for the rough description in the previous version.
Since this part belongs to the REST API acceleration mechanism for rescaling, and Option 6 seems reasonable to me,
I plan to add it to FLIP-487 once the design of FLIP-495 has reached consensus.
Of course, if needed, I'd be happy to clarify the usage and purpose of this parameter in the current email thread.

6. The REST endpoint is probably better suited in FLIP-487. FLIP-495 should
be about the actual implementation details and how the data is stored
internally whereas FLIP-487 is about exposing the information to the
outside through the REST API and the Flink UI. That would be a way to
decrease the scope of FLIP-495. WDYT?

That sounds nice to me. Therefore, I have moved all REST API–related changes to FLIP-487.
BTW, to avoid repetitive changes in FLIP-487, I'll start organizing FLIP-487 after FLIP-495 has been finalized.

Looking forward to your next review!

[1]https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=334760525#FLIP495:SupportAdaptiveSchedulerrecordandquerytherescalehistory-Aboutrescaleeventsstorage.1
[2]https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=334760525#FLIP495:SupportAdaptiveSchedulerrecordandquerytherescalehistory-ThemainscenarioswhereRescalestatusswitchestoterminated
[3]https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=334760525#FLIP495:SupportAdaptiveSchedulerrecordandquerytherescalehistory-Theinformationtorecordinarescaleevent
[4]https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=334760525#FLIP495:SupportAdaptiveSchedulerrecordandquerytherescalehistory-InternalInterfaces


Best regards,
Yuepeng Pan





At 2025-08-10 23:54:37, "Matthias Pohl" <ma...@apache.org> wrote:
Hi Yuepeng,
thanks for reminding me of this FLIP. I went over it and have a few items
which we might need to address before we can actually finalize the vote:

1. You mention a few options for when it comes to storing the data which is
good. The FLIP doesn't point out, though, what option you're going to go
for as part of this FLIP (as far as I can see). It would be good to only
outline the option to go for in the FLIP and list the other options as
rejected alternatives (with the pro's and con's). I think it make sense to
go for option 3 (i.e. following what's done for the ExecutionGraphInfoStore
for now). The other options can be considered as a follow-up.
2. About the terminal states of a rescaling (i.e. IGNORED, FAILED,
COMPLETED): Can we we clarify in the FLIP under what conditions the
rescaling transitions into each of the three terminal states?
3. The section "The information to record in a rescale event" could be
restructured in four sections (to remove redundancy):
a) The IDs (Rescale
ID, resourceRequirementsEpochID, subRescaleIdOfResourceRequirementsEpochID):
What about making these names easier to read: GlobalRescaleID, RescaleUUID,
RescaleAttemptId)
b) Per-vertex data which includes: JobVertexID, JobVertexName,
SlotSharingGroupId, the different parallelisms (pre-rescale, sufficient,
desired, post-rescale)
c) The SlotSharingGroup information: SlotSharingGroupId, name,
ResourceProfile
d) Other information: Timestamps of state transitions, etc. as laid out in
the FLIP already
4. The FLIP doesn't explain how the data is passed through the
AdaptiveScheduler states. We should be handling some kind of
RescaleSnapshot that is passed through the different states and updated and
its final state is stored somewhere within AdaptiveScheduler in the end, I
guess. Can we clarify that in the FLIP?
5. You mention the config parameters for the cache in the public interface
section. But there's no mentioning of any caching and how that is used
within the FLIP.
6. The REST endpoint is probably better suited in FLIP-487. FLIP-495 should
be about the actual implementation details and how the data is stored
internally whereas FLIP-487 is about exposing the information to the
outside through the REST API and the Flink UI. That would be a way to
decrease the scope of FLIP-495. WDYT?

Best,
Matthias


On Mon, Mar 24, 2025 at 11:37 AM Yuepeng Pan <pa...@apache.org> wrote:

Hi, Community,

There haven’t been any further responses to this email over the past few
days.
I'd like to initiate a vote on the current proposal[1] in the next few
days.
Please rest assured that I’m proceeding cautiously and not rushing the
process.
If there are any concerns about this FLIP-495[1],
I will gladly pause and make the adjustments.

Best regards,
Yuepeng Pan

[1]
https://cwiki.apache.org/confluence/display/FLINK/FLIP-495%3A+Support+AdaptiveScheduler+record+and+query+the+rescale+history


On 2024/12/17 15:18:45 Yuepeng Pan wrote:
Hi community,




We discussed several aspects of FLIP-487[1] 'Show history of rescales in
Web UI for AdaptiveScheduler'
and received a lot of valuable feedback. Based on the suggestions from
the email thread[2],
we plan to split the original proposal for FLIP-487[1].




The current email thread and the FLIP-495[3] wiki will be used to
discuss 'Support AdaptiveScheduler in recording and querying the rescale
history',
while FLIP-487[1] will primarily focus on displaying-related design
content




Looking forward to any feedback and opinions on FLIP-495[3].




[1]
https://cwiki.apache.org/confluence/display/FLINK/%5BWIP%5D+FLIP-487%3A+Show+history+of+rescales+in+Web+UI+for+AdaptiveScheduler

[2] https://lists.apache.org/thread/f4md4btkf006mxcxf66bng1kfz0rsn8c

[3]
https://cwiki.apache.org/confluence/display/FLINK/%5BWIP%5D+FLIP-495%3A+Support+AdaptiveScheduler+record+and+query+the+rescale+history




Thank you very much.




Best,

Regards.

Yuepeng Pan




Re: [DISCUSS] FLIP-495: Support AdaptiveScheduler record and query the rescale history

Posted by Matthias Pohl <ma...@apache.org>.
Thanks for the update, Yuepeng. Here are my remarks:

- Why do we mark the rescale operation as IGNORED if an (repeatable)
exception causes a restart? That's because the failure will create a new
rescale instance that will be saved if the resources changed as part of the
failure handling/job restart process?

- Thinking about the GlobalRescaleId: Should we still keep it considering
that we're not preserving the value on JM failovers (because of the missing
HA functionality) in this FLIP? We could remove it and add it as part of a
HA-support follow-up. Providing it despite the missing HA support might be
misleading in case a JM failover actually happens. WDYT?

- For the job vertices, you introduced a 5th resource type (previous,
acquired, required, min bound, max bound). What's the difference? I guess,
min bound corresponds to "sufficient resources" and max bound corresponds
to "desired resources". What are required resources here in that case? It
might be good to stick to the naming that was introduced in FLIP-472 [1].

- "Job Vertices Rescaled Information" and "Slots / Resources Rescaled
Information" seem to have redundant information now: "parallelism of job
vertex" and "slot number of the slot sharing group" are more or less the
same. The parallelism is defined per job vertex (along the
SlotSharingGroupId). The SlotSharingGroup holds the ResourceProfile (but no
parallelism).

- "Other information" section contains a comment message for the reason why
a rescale event terminates. Reason might be a better term than comment
(especially because we're relying on an enum here rather than plain text).

- Based on the current proposal of the FLIP, the intermediate states of the
rescale event actually match the AdaptiveScheduler's states. Only the final
states are exclusive to the scale event? Would it make sense to rely on the
AdaptiveScheduler state entirely for the intermediate state and a dedicated
field "terminalState"?

- Why does StateTransitions need to be extended with getRescaleTimeline()?
Shouldn't the RescaleTimeline be located in the AdaptiveScheduler? Instead,
the State.Context (essentially, the AdaptiveScheduler) should provide
methods for updating the current rescale process object that lives in the
AdaptiveScheduler. WDYT?

- Section "How to process rescales storage ?": ExecutionGraphInfo could be
modified to include the Rescale history (along the exception history). The
data can be then saved in the ExecutionGraphInfoStore. All you have to do
is to store the rescale history in AdaptiveScheduler and an object that
holds the infos of the current rescale process. If this current rescale
process terminates, an immutable rescale snapshot of this event is created
that is saved in the rescale history.

Matthias

[1]
https://cwiki.apache.org/confluence/display/FLINK/FLIP-472%3A+Aligning+timeout+logic+in+the+AdaptiveScheduler%27s+WaitingForResources+and+Executing+states

On Tue, Aug 19, 2025 at 11:14 AM Yuepeng Pan <pa...@apache.org> wrote:

> Bumping this thread kindly. Thanks!
>
> Best,
> Yuepeng Pan
>
>
> At 2025-08-13 14:52:26, "Yuepeng Pan" <pa...@apache.org> wrote:
>
> Hi, Matthias,
> Thank you  very much for your comments!
> I have carefully read your reply and made some changes in the hope of making improvements.
> Please help take a look.
>
> For your comments:
>
> > 1. You mention a few options for when it comes to storing the data which is
> > good. The FLIP doesn't point out, though, what option you're going to go
> > for as part of this FLIP (as far as I can see). It would be good to only
> > outline the option to go for in the FLIP and list the other options as
> > rejected alternatives (with the pro's and con's). I think it make sense to
> > go for option 3 (i.e. following what's done for the ExecutionGraphInfoStore
> > for now). The other options can be considered as a follow-up.
>
> This is very meaningful. Based on this comment, I have kept option 3 in its original place and moved the other candidate options to [1].
>
> > 2. About the terminal states of a rescaling (i.e. IGNORED, FAILED,
> > COMPLETED): Can we we clarify in the FLIP under what conditions the
> > rescaling transitions into each of the three terminal states?
>
> Yes, this is a reasonable request for understanding and explaining the logic of transitions to terminated states.
> A new subsection [2] has been added to address this.
>
> > 3. The section "The information to record in a rescale event" could be
> > restructured in four sections (to remove redundancy):
> > a) The IDs (Rescale
> > ID, resourceRequirementsEpochID, subRescaleIdOfResourceRequirementsEpochID):
> > What about making these names easier to read: GlobalRescaleID, RescaleUUID,
> > RescaleAttemptId)
> > b) Per-vertex data which includes: JobVertexID, JobVertexName,
> > SlotSharingGroupId, the different parallelisms (pre-rescale, sufficient,
> > desired, post-rescale)
> > c) The SlotSharingGroup information: SlotSharingGroupId, name,
> > ResourceProfile
> > d) Other information: Timestamps of state transitions, etc. as laid out in
> > the FLIP already
>
> That makes sense to me. Please check [3] for the latest updates in this part.
>
> > 4. The FLIP doesn't explain how the data is passed through the
> > AdaptiveScheduler states. We should be handling some kind of
> > RescaleSnapshot that is passed through the different states and updated and
> > its final state is stored somewhere within AdaptiveScheduler in the end, I
> > guess. Can we clarify that in the FLIP?
>
> Indeed — this was missing in the original FLIP. To address this, I have added [4], which focuses on describing how a Rescale is represented,
> and how we can quickly pass and maintain the Rescale history.
>
> > 5. You mention the config parameters for the cache in the public interface
> > section. But there's no mentioning of any caching and how that is used
> > within the FLIP.
>
> Sorry for the rough description in the previous version.
> Since this part belongs to the REST API acceleration mechanism for rescaling, and Option 6 seems reasonable to me,
> I plan to add it to FLIP-487 once the design of FLIP-495 has reached consensus.
> Of course, if needed, I'd be happy to clarify the usage and purpose of this parameter in the current email thread.
>
> > 6. The REST endpoint is probably better suited in FLIP-487. FLIP-495 should
> > be about the actual implementation details and how the data is stored
> > internally whereas FLIP-487 is about exposing the information to the
> > outside through the REST API and the Flink UI. That would be a way to
> > decrease the scope of FLIP-495. WDYT?
>
> That sounds nice to me. Therefore, I have moved all REST API–related changes to FLIP-487.
> BTW, to avoid repetitive changes in FLIP-487, I'll start organizing FLIP-487 after FLIP-495 has been finalized.
>
> Looking forward to your next review!
>
> [1]https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=334760525#FLIP495:SupportAdaptiveSchedulerrecordandquerytherescalehistory-Aboutrescaleeventsstorage.1
> [2]https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=334760525#FLIP495:SupportAdaptiveSchedulerrecordandquerytherescalehistory-ThemainscenarioswhereRescalestatusswitchestoterminated
> [3]https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=334760525#FLIP495:SupportAdaptiveSchedulerrecordandquerytherescalehistory-Theinformationtorecordinarescaleevent
> [4]https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=334760525#FLIP495:SupportAdaptiveSchedulerrecordandquerytherescalehistory-InternalInterfaces
>
>
> Best regards,
> Yuepeng Pan
>
>
>
>
>
> At 2025-08-10 23:54:37, "Matthias Pohl" <ma...@apache.org> wrote:
> >Hi Yuepeng,
> >thanks for reminding me of this FLIP. I went over it and have a few items
> >which we might need to address before we can actually finalize the vote:
> >
> >1. You mention a few options for when it comes to storing the data which is
> >good. The FLIP doesn't point out, though, what option you're going to go
> >for as part of this FLIP (as far as I can see). It would be good to only
> >outline the option to go for in the FLIP and list the other options as
> >rejected alternatives (with the pro's and con's). I think it make sense to
> >go for option 3 (i.e. following what's done for the ExecutionGraphInfoStore
> >for now). The other options can be considered as a follow-up.
> >2. About the terminal states of a rescaling (i.e. IGNORED, FAILED,
> >COMPLETED): Can we we clarify in the FLIP under what conditions the
> >rescaling transitions into each of the three terminal states?
> >3. The section "The information to record in a rescale event" could be
> >restructured in four sections (to remove redundancy):
> > a) The IDs (Rescale
> >ID, resourceRequirementsEpochID, subRescaleIdOfResourceRequirementsEpochID):
> >What about making these names easier to read: GlobalRescaleID, RescaleUUID,
> >RescaleAttemptId)
> > b) Per-vertex data which includes: JobVertexID, JobVertexName,
> >SlotSharingGroupId, the different parallelisms (pre-rescale, sufficient,
> >desired, post-rescale)
> > c) The SlotSharingGroup information: SlotSharingGroupId, name,
> >ResourceProfile
> > d) Other information: Timestamps of state transitions, etc. as laid out in
> >the FLIP already
> >4. The FLIP doesn't explain how the data is passed through the
> >AdaptiveScheduler states. We should be handling some kind of
> >RescaleSnapshot that is passed through the different states and updated and
> >its final state is stored somewhere within AdaptiveScheduler in the end, I
> >guess. Can we clarify that in the FLIP?
> >5. You mention the config parameters for the cache in the public interface
> >section. But there's no mentioning of any caching and how that is used
> >within the FLIP.
> >6. The REST endpoint is probably better suited in FLIP-487. FLIP-495 should
> >be about the actual implementation details and how the data is stored
> >internally whereas FLIP-487 is about exposing the information to the
> >outside through the REST API and the Flink UI. That would be a way to
> >decrease the scope of FLIP-495. WDYT?
> >
> >Best,
> >Matthias
> >
> >
> >On Mon, Mar 24, 2025 at 11:37 AM Yuepeng Pan <pa...@apache.org> wrote:
> >
> >> Hi, Community,
> >>
> >> There haven’t been any further responses to this email over the past few
> >> days.
> >> I'd like to initiate a vote on the current proposal[1] in the next few
> >> days.
> >> Please rest assured that I’m proceeding cautiously and not rushing the
> >> process.
> >> If there are any concerns about this FLIP-495[1],
> >> I will gladly pause and make the adjustments.
> >>
> >> Best regards,
> >> Yuepeng Pan
> >>
> >> [1]
> >> https://cwiki.apache.org/confluence/display/FLINK/FLIP-495%3A+Support+AdaptiveScheduler+record+and+query+the+rescale+history
> >>
> >>
> >> On 2024/12/17 15:18:45 Yuepeng Pan wrote:
> >> > Hi community,
> >> >
> >> >
> >> >
> >> >
> >> > We discussed several aspects of FLIP-487[1] 'Show history of rescales in
> >> Web UI for AdaptiveScheduler'
> >> > and received a lot of valuable feedback. Based on the suggestions from
> >> the email thread[2],
> >> > we plan to split the original proposal for FLIP-487[1].
> >> >
> >> >
> >> >
> >> >
> >> > The current email thread and the FLIP-495[3] wiki will be used to
> >> discuss 'Support AdaptiveScheduler in recording and querying the rescale
> >> history',
> >> > while FLIP-487[1] will primarily focus on displaying-related design
> >> content
> >> >
> >> >
> >> >
> >> >
> >> > Looking forward to any feedback and opinions on FLIP-495[3].
> >> >
> >> >
> >> >
> >> >
> >> > [1]
> >> https://cwiki.apache.org/confluence/display/FLINK/%5BWIP%5D+FLIP-487%3A+Show+history+of+rescales+in+Web+UI+for+AdaptiveScheduler
> >> >
> >> > [2] https://lists.apache.org/thread/f4md4btkf006mxcxf66bng1kfz0rsn8c
> >> >
> >> > [3]
> >> https://cwiki.apache.org/confluence/display/FLINK/%5BWIP%5D+FLIP-495%3A+Support+AdaptiveScheduler+record+and+query+the+rescale+history
> >> >
> >> >
> >> >
> >> >
> >> > Thank you very much.
> >> >
> >> >
> >> >
> >> >
> >> > Best,
> >> >
> >> > Regards.
> >> >
> >> > Yuepeng Pan
> >>
>
>

Re: [DISCUSS] FLIP-495: Support AdaptiveScheduler record and query the rescale history

Posted by Yuepeng Pan <pa...@apache.org>.
Bumping this thread kindly. Thanks!

Best,
Yuepeng Pan 




At 2025-08-13 14:52:26, "Yuepeng Pan" <pa...@apache.org> wrote:

Hi, Matthias,
Thank you  very much for your comments!
I have carefully read your reply and made some changes in the hope of making improvements.
Please help take a look.

For your comments:

> 1. You mention a few options for when it comes to storing the data which is
> good. The FLIP doesn't point out, though, what option you're going to go
> for as part of this FLIP (as far as I can see). It would be good to only
> outline the option to go for in the FLIP and list the other options as
> rejected alternatives (with the pro's and con's). I think it make sense to
> go for option 3 (i.e. following what's done for the ExecutionGraphInfoStore
> for now). The other options can be considered as a follow-up.

This is very meaningful. Based on this comment, I have kept option 3 in its original place and moved the other candidate options to [1].

> 2. About the terminal states of a rescaling (i.e. IGNORED, FAILED,
> COMPLETED): Can we we clarify in the FLIP under what conditions the
> rescaling transitions into each of the three terminal states?

Yes, this is a reasonable request for understanding and explaining the logic of transitions to terminated states.
A new subsection [2] has been added to address this.

> 3. The section "The information to record in a rescale event" could be
> restructured in four sections (to remove redundancy):
> a) The IDs (Rescale
> ID, resourceRequirementsEpochID, subRescaleIdOfResourceRequirementsEpochID):
> What about making these names easier to read: GlobalRescaleID, RescaleUUID,
> RescaleAttemptId)
> b) Per-vertex data which includes: JobVertexID, JobVertexName,
> SlotSharingGroupId, the different parallelisms (pre-rescale, sufficient,
> desired, post-rescale)
> c) The SlotSharingGroup information: SlotSharingGroupId, name,
> ResourceProfile
> d) Other information: Timestamps of state transitions, etc. as laid out in
> the FLIP already

That makes sense to me. Please check [3] for the latest updates in this part.

> 4. The FLIP doesn't explain how the data is passed through the
> AdaptiveScheduler states. We should be handling some kind of
> RescaleSnapshot that is passed through the different states and updated and
> its final state is stored somewhere within AdaptiveScheduler in the end, I
> guess. Can we clarify that in the FLIP?

Indeed — this was missing in the original FLIP. To address this, I have added [4], which focuses on describing how a Rescale is represented,
and how we can quickly pass and maintain the Rescale history.

> 5. You mention the config parameters for the cache in the public interface
> section. But there's no mentioning of any caching and how that is used
> within the FLIP.

Sorry for the rough description in the previous version.
Since this part belongs to the REST API acceleration mechanism for rescaling, and Option 6 seems reasonable to me,
I plan to add it to FLIP-487 once the design of FLIP-495 has reached consensus.
Of course, if needed, I'd be happy to clarify the usage and purpose of this parameter in the current email thread.

> 6. The REST endpoint is probably better suited in FLIP-487. FLIP-495 should
> be about the actual implementation details and how the data is stored
> internally whereas FLIP-487 is about exposing the information to the
> outside through the REST API and the Flink UI. That would be a way to
> decrease the scope of FLIP-495. WDYT?

That sounds nice to me. Therefore, I have moved all REST API–related changes to FLIP-487. 
BTW, to avoid repetitive changes in FLIP-487, I'll start organizing FLIP-487 after FLIP-495 has been finalized.

Looking forward to your next review!

[1]https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=334760525#FLIP495:SupportAdaptiveSchedulerrecordandquerytherescalehistory-Aboutrescaleeventsstorage.1
[2]https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=334760525#FLIP495:SupportAdaptiveSchedulerrecordandquerytherescalehistory-ThemainscenarioswhereRescalestatusswitchestoterminated
[3]https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=334760525#FLIP495:SupportAdaptiveSchedulerrecordandquerytherescalehistory-Theinformationtorecordinarescaleevent
[4]https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=334760525#FLIP495:SupportAdaptiveSchedulerrecordandquerytherescalehistory-InternalInterfaces





Best regards,
Yuepeng Pan











At 2025-08-10 23:54:37, "Matthias Pohl" <ma...@apache.org> wrote:
>Hi Yuepeng,
>thanks for reminding me of this FLIP. I went over it and have a few items
>which we might need to address before we can actually finalize the vote:
>
>1. You mention a few options for when it comes to storing the data which is
>good. The FLIP doesn't point out, though, what option you're going to go
>for as part of this FLIP (as far as I can see). It would be good to only
>outline the option to go for in the FLIP and list the other options as
>rejected alternatives (with the pro's and con's). I think it make sense to
>go for option 3 (i.e. following what's done for the ExecutionGraphInfoStore
>for now). The other options can be considered as a follow-up.
>2. About the terminal states of a rescaling (i.e. IGNORED, FAILED,
>COMPLETED): Can we we clarify in the FLIP under what conditions the
>rescaling transitions into each of the three terminal states?
>3. The section "The information to record in a rescale event" could be
>restructured in four sections (to remove redundancy):
> a) The IDs (Rescale
>ID, resourceRequirementsEpochID, subRescaleIdOfResourceRequirementsEpochID):
>What about making these names easier to read: GlobalRescaleID, RescaleUUID,
>RescaleAttemptId)
> b) Per-vertex data which includes: JobVertexID, JobVertexName,
>SlotSharingGroupId, the different parallelisms (pre-rescale, sufficient,
>desired, post-rescale)
> c) The SlotSharingGroup information: SlotSharingGroupId, name,
>ResourceProfile
> d) Other information: Timestamps of state transitions, etc. as laid out in
>the FLIP already
>4. The FLIP doesn't explain how the data is passed through the
>AdaptiveScheduler states. We should be handling some kind of
>RescaleSnapshot that is passed through the different states and updated and
>its final state is stored somewhere within AdaptiveScheduler in the end, I
>guess. Can we clarify that in the FLIP?
>5. You mention the config parameters for the cache in the public interface
>section. But there's no mentioning of any caching and how that is used
>within the FLIP.
>6. The REST endpoint is probably better suited in FLIP-487. FLIP-495 should
>be about the actual implementation details and how the data is stored
>internally whereas FLIP-487 is about exposing the information to the
>outside through the REST API and the Flink UI. That would be a way to
>decrease the scope of FLIP-495. WDYT?
>
>Best,
>Matthias
>
>
>On Mon, Mar 24, 2025 at 11:37 AM Yuepeng Pan <pa...@apache.org> wrote:
>
>> Hi, Community,
>>
>> There haven’t been any further responses to this email over the past few
>> days.
>> I'd like to initiate a vote on the current proposal[1] in the next few
>> days.
>> Please rest assured that I’m proceeding cautiously and not rushing the
>> process.
>> If there are any concerns about this FLIP-495[1],
>> I will gladly pause and make the adjustments.
>>
>> Best regards,
>> Yuepeng Pan
>>
>> [1]
>> https://cwiki.apache.org/confluence/display/FLINK/FLIP-495%3A+Support+AdaptiveScheduler+record+and+query+the+rescale+history
>>
>>
>> On 2024/12/17 15:18:45 Yuepeng Pan wrote:
>> > Hi community,
>> >
>> >
>> >
>> >
>> > We discussed several aspects of FLIP-487[1] 'Show history of rescales in
>> Web UI for AdaptiveScheduler'
>> > and received a lot of valuable feedback. Based on the suggestions from
>> the email thread[2],
>> > we plan to split the original proposal for FLIP-487[1].
>> >
>> >
>> >
>> >
>> > The current email thread and the FLIP-495[3] wiki will be used to
>> discuss 'Support AdaptiveScheduler in recording and querying the rescale
>> history',
>> > while FLIP-487[1] will primarily focus on displaying-related design
>> content
>> >
>> >
>> >
>> >
>> > Looking forward to any feedback and opinions on FLIP-495[3].
>> >
>> >
>> >
>> >
>> > [1]
>> https://cwiki.apache.org/confluence/display/FLINK/%5BWIP%5D+FLIP-487%3A+Show+history+of+rescales+in+Web+UI+for+AdaptiveScheduler
>> >
>> > [2] https://lists.apache.org/thread/f4md4btkf006mxcxf66bng1kfz0rsn8c
>> >
>> > [3]
>> https://cwiki.apache.org/confluence/display/FLINK/%5BWIP%5D+FLIP-495%3A+Support+AdaptiveScheduler+record+and+query+the+rescale+history
>> >
>> >
>> >
>> >
>> > Thank you very much.
>> >
>> >
>> >
>> >
>> > Best,
>> >
>> > Regards.
>> >
>> > Yuepeng Pan
>>

Re:Re: [DISCUSS] FLIP-495: Support AdaptiveScheduler record and query the rescale history

Posted by Yuepeng Pan <pa...@apache.org>.
Hi, Matthias,
Thank you  very much for your comments!
I have carefully read your reply and made some changes in the hope of making improvements.
Please help take a look.

For your comments:

> 1. You mention a few options for when it comes to storing the data which is
> good. The FLIP doesn't point out, though, what option you're going to go
> for as part of this FLIP (as far as I can see). It would be good to only
> outline the option to go for in the FLIP and list the other options as
> rejected alternatives (with the pro's and con's). I think it make sense to
> go for option 3 (i.e. following what's done for the ExecutionGraphInfoStore
> for now). The other options can be considered as a follow-up.

This is very meaningful. Based on this comment, I have kept option 3 in its original place and moved the other candidate options to [1].

> 2. About the terminal states of a rescaling (i.e. IGNORED, FAILED,
> COMPLETED): Can we we clarify in the FLIP under what conditions the
> rescaling transitions into each of the three terminal states?

Yes, this is a reasonable request for understanding and explaining the logic of transitions to terminated states.
A new subsection [2] has been added to address this.

> 3. The section "The information to record in a rescale event" could be
> restructured in four sections (to remove redundancy):
> a) The IDs (Rescale
> ID, resourceRequirementsEpochID, subRescaleIdOfResourceRequirementsEpochID):
> What about making these names easier to read: GlobalRescaleID, RescaleUUID,
> RescaleAttemptId)
> b) Per-vertex data which includes: JobVertexID, JobVertexName,
> SlotSharingGroupId, the different parallelisms (pre-rescale, sufficient,
> desired, post-rescale)
> c) The SlotSharingGroup information: SlotSharingGroupId, name,
> ResourceProfile
> d) Other information: Timestamps of state transitions, etc. as laid out in
> the FLIP already

That makes sense to me. Please check [3] for the latest updates in this part.

> 4. The FLIP doesn't explain how the data is passed through the
> AdaptiveScheduler states. We should be handling some kind of
> RescaleSnapshot that is passed through the different states and updated and
> its final state is stored somewhere within AdaptiveScheduler in the end, I
> guess. Can we clarify that in the FLIP?

Indeed — this was missing in the original FLIP. To address this, I have added [4], which focuses on describing how a Rescale is represented,
and how we can quickly pass and maintain the Rescale history.

> 5. You mention the config parameters for the cache in the public interface
> section. But there's no mentioning of any caching and how that is used
> within the FLIP.

Sorry for the rough description in the previous version.
Since this part belongs to the REST API acceleration mechanism for rescaling, and Option 6 seems reasonable to me,
I plan to add it to FLIP-487 once the design of FLIP-495 has reached consensus.
Of course, if needed, I'd be happy to clarify the usage and purpose of this parameter in the current email thread.

> 6. The REST endpoint is probably better suited in FLIP-487. FLIP-495 should
> be about the actual implementation details and how the data is stored
> internally whereas FLIP-487 is about exposing the information to the
> outside through the REST API and the Flink UI. That would be a way to
> decrease the scope of FLIP-495. WDYT?

That sounds nice to me. Therefore, I have moved all REST API–related changes to FLIP-487. 
BTW, to avoid repetitive changes in FLIP-487, I'll start organizing FLIP-487 after FLIP-495 has been finalized.

Looking forward to your next review!

[1]https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=334760525#FLIP495:SupportAdaptiveSchedulerrecordandquerytherescalehistory-Aboutrescaleeventsstorage.1
[2]https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=334760525#FLIP495:SupportAdaptiveSchedulerrecordandquerytherescalehistory-ThemainscenarioswhereRescalestatusswitchestoterminated
[3]https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=334760525#FLIP495:SupportAdaptiveSchedulerrecordandquerytherescalehistory-Theinformationtorecordinarescaleevent
[4]https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=334760525#FLIP495:SupportAdaptiveSchedulerrecordandquerytherescalehistory-InternalInterfaces





Best regards,
Yuepeng Pan











At 2025-08-10 23:54:37, "Matthias Pohl" <ma...@apache.org> wrote:
>Hi Yuepeng,
>thanks for reminding me of this FLIP. I went over it and have a few items
>which we might need to address before we can actually finalize the vote:
>
>1. You mention a few options for when it comes to storing the data which is
>good. The FLIP doesn't point out, though, what option you're going to go
>for as part of this FLIP (as far as I can see). It would be good to only
>outline the option to go for in the FLIP and list the other options as
>rejected alternatives (with the pro's and con's). I think it make sense to
>go for option 3 (i.e. following what's done for the ExecutionGraphInfoStore
>for now). The other options can be considered as a follow-up.
>2. About the terminal states of a rescaling (i.e. IGNORED, FAILED,
>COMPLETED): Can we we clarify in the FLIP under what conditions the
>rescaling transitions into each of the three terminal states?
>3. The section "The information to record in a rescale event" could be
>restructured in four sections (to remove redundancy):
> a) The IDs (Rescale
>ID, resourceRequirementsEpochID, subRescaleIdOfResourceRequirementsEpochID):
>What about making these names easier to read: GlobalRescaleID, RescaleUUID,
>RescaleAttemptId)
> b) Per-vertex data which includes: JobVertexID, JobVertexName,
>SlotSharingGroupId, the different parallelisms (pre-rescale, sufficient,
>desired, post-rescale)
> c) The SlotSharingGroup information: SlotSharingGroupId, name,
>ResourceProfile
> d) Other information: Timestamps of state transitions, etc. as laid out in
>the FLIP already
>4. The FLIP doesn't explain how the data is passed through the
>AdaptiveScheduler states. We should be handling some kind of
>RescaleSnapshot that is passed through the different states and updated and
>its final state is stored somewhere within AdaptiveScheduler in the end, I
>guess. Can we clarify that in the FLIP?
>5. You mention the config parameters for the cache in the public interface
>section. But there's no mentioning of any caching and how that is used
>within the FLIP.
>6. The REST endpoint is probably better suited in FLIP-487. FLIP-495 should
>be about the actual implementation details and how the data is stored
>internally whereas FLIP-487 is about exposing the information to the
>outside through the REST API and the Flink UI. That would be a way to
>decrease the scope of FLIP-495. WDYT?
>
>Best,
>Matthias
>
>
>On Mon, Mar 24, 2025 at 11:37 AM Yuepeng Pan <pa...@apache.org> wrote:
>
>> Hi, Community,
>>
>> There haven’t been any further responses to this email over the past few
>> days.
>> I'd like to initiate a vote on the current proposal[1] in the next few
>> days.
>> Please rest assured that I’m proceeding cautiously and not rushing the
>> process.
>> If there are any concerns about this FLIP-495[1],
>> I will gladly pause and make the adjustments.
>>
>> Best regards,
>> Yuepeng Pan
>>
>> [1]
>> https://cwiki.apache.org/confluence/display/FLINK/FLIP-495%3A+Support+AdaptiveScheduler+record+and+query+the+rescale+history
>>
>>
>> On 2024/12/17 15:18:45 Yuepeng Pan wrote:
>> > Hi community,
>> >
>> >
>> >
>> >
>> > We discussed several aspects of FLIP-487[1] 'Show history of rescales in
>> Web UI for AdaptiveScheduler'
>> > and received a lot of valuable feedback. Based on the suggestions from
>> the email thread[2],
>> > we plan to split the original proposal for FLIP-487[1].
>> >
>> >
>> >
>> >
>> > The current email thread and the FLIP-495[3] wiki will be used to
>> discuss 'Support AdaptiveScheduler in recording and querying the rescale
>> history',
>> > while FLIP-487[1] will primarily focus on displaying-related design
>> content
>> >
>> >
>> >
>> >
>> > Looking forward to any feedback and opinions on FLIP-495[3].
>> >
>> >
>> >
>> >
>> > [1]
>> https://cwiki.apache.org/confluence/display/FLINK/%5BWIP%5D+FLIP-487%3A+Show+history+of+rescales+in+Web+UI+for+AdaptiveScheduler
>> >
>> > [2] https://lists.apache.org/thread/f4md4btkf006mxcxf66bng1kfz0rsn8c
>> >
>> > [3]
>> https://cwiki.apache.org/confluence/display/FLINK/%5BWIP%5D+FLIP-495%3A+Support+AdaptiveScheduler+record+and+query+the+rescale+history
>> >
>> >
>> >
>> >
>> > Thank you very much.
>> >
>> >
>> >
>> >
>> > Best,
>> >
>> > Regards.
>> >
>> > Yuepeng Pan
>>

Re: [DISCUSS] FLIP-495: Support AdaptiveScheduler record and query the rescale history

Posted by Matthias Pohl <ma...@apache.org>.
Hi Yuepeng,
thanks for reminding me of this FLIP. I went over it and have a few items
which we might need to address before we can actually finalize the vote:

1. You mention a few options for when it comes to storing the data which is
good. The FLIP doesn't point out, though, what option you're going to go
for as part of this FLIP (as far as I can see). It would be good to only
outline the option to go for in the FLIP and list the other options as
rejected alternatives (with the pro's and con's). I think it make sense to
go for option 3 (i.e. following what's done for the ExecutionGraphInfoStore
for now). The other options can be considered as a follow-up.
2. About the terminal states of a rescaling (i.e. IGNORED, FAILED,
COMPLETED): Can we we clarify in the FLIP under what conditions the
rescaling transitions into each of the three terminal states?
3. The section "The information to record in a rescale event" could be
restructured in four sections (to remove redundancy):
 a) The IDs (Rescale
ID, resourceRequirementsEpochID, subRescaleIdOfResourceRequirementsEpochID):
What about making these names easier to read: GlobalRescaleID, RescaleUUID,
RescaleAttemptId)
 b) Per-vertex data which includes: JobVertexID, JobVertexName,
SlotSharingGroupId, the different parallelisms (pre-rescale, sufficient,
desired, post-rescale)
 c) The SlotSharingGroup information: SlotSharingGroupId, name,
ResourceProfile
 d) Other information: Timestamps of state transitions, etc. as laid out in
the FLIP already
4. The FLIP doesn't explain how the data is passed through the
AdaptiveScheduler states. We should be handling some kind of
RescaleSnapshot that is passed through the different states and updated and
its final state is stored somewhere within AdaptiveScheduler in the end, I
guess. Can we clarify that in the FLIP?
5. You mention the config parameters for the cache in the public interface
section. But there's no mentioning of any caching and how that is used
within the FLIP.
6. The REST endpoint is probably better suited in FLIP-487. FLIP-495 should
be about the actual implementation details and how the data is stored
internally whereas FLIP-487 is about exposing the information to the
outside through the REST API and the Flink UI. That would be a way to
decrease the scope of FLIP-495. WDYT?

Best,
Matthias


On Mon, Mar 24, 2025 at 11:37 AM Yuepeng Pan <pa...@apache.org> wrote:

> Hi, Community,
>
> There haven’t been any further responses to this email over the past few
> days.
> I'd like to initiate a vote on the current proposal[1] in the next few
> days.
> Please rest assured that I’m proceeding cautiously and not rushing the
> process.
> If there are any concerns about this FLIP-495[1],
> I will gladly pause and make the adjustments.
>
> Best regards,
> Yuepeng Pan
>
> [1]
> https://cwiki.apache.org/confluence/display/FLINK/FLIP-495%3A+Support+AdaptiveScheduler+record+and+query+the+rescale+history
>
>
> On 2024/12/17 15:18:45 Yuepeng Pan wrote:
> > Hi community,
> >
> >
> >
> >
> > We discussed several aspects of FLIP-487[1] 'Show history of rescales in
> Web UI for AdaptiveScheduler'
> > and received a lot of valuable feedback. Based on the suggestions from
> the email thread[2],
> > we plan to split the original proposal for FLIP-487[1].
> >
> >
> >
> >
> > The current email thread and the FLIP-495[3] wiki will be used to
> discuss 'Support AdaptiveScheduler in recording and querying the rescale
> history',
> > while FLIP-487[1] will primarily focus on displaying-related design
> content
> >
> >
> >
> >
> > Looking forward to any feedback and opinions on FLIP-495[3].
> >
> >
> >
> >
> > [1]
> https://cwiki.apache.org/confluence/display/FLINK/%5BWIP%5D+FLIP-487%3A+Show+history+of+rescales+in+Web+UI+for+AdaptiveScheduler
> >
> > [2] https://lists.apache.org/thread/f4md4btkf006mxcxf66bng1kfz0rsn8c
> >
> > [3]
> https://cwiki.apache.org/confluence/display/FLINK/%5BWIP%5D+FLIP-495%3A+Support+AdaptiveScheduler+record+and+query+the+rescale+history
> >
> >
> >
> >
> > Thank you very much.
> >
> >
> >
> >
> > Best,
> >
> > Regards.
> >
> > Yuepeng Pan
>

Re: [DISCUSS] FLIP-495: Support AdaptiveScheduler record and query the rescale history

Posted by Yuepeng Pan <pa...@apache.org>.
Hi, Community,

There haven’t been any further responses to this email over the past few days.
I'd like to initiate a vote on the current proposal[1] in the next few days.
Please rest assured that I’m proceeding cautiously and not rushing the process.
If there are any concerns about this FLIP-495[1],
I will gladly pause and make the adjustments.

Best regards,
Yuepeng Pan

[1] https://cwiki.apache.org/confluence/display/FLINK/FLIP-495%3A+Support+AdaptiveScheduler+record+and+query+the+rescale+history


On 2024/12/17 15:18:45 Yuepeng Pan wrote:
> Hi community,
> 
> 
> 
> 
> We discussed several aspects of FLIP-487[1] 'Show history of rescales in Web UI for AdaptiveScheduler' 
> and received a lot of valuable feedback. Based on the suggestions from the email thread[2],
> we plan to split the original proposal for FLIP-487[1]. 
> 
> 
> 
> 
> The current email thread and the FLIP-495[3] wiki will be used to discuss 'Support AdaptiveScheduler in recording and querying the rescale history', 
> while FLIP-487[1] will primarily focus on displaying-related design content
> 
> 
> 
> 
> Looking forward to any feedback and opinions on FLIP-495[3].
> 
> 
> 
> 
> [1] https://cwiki.apache.org/confluence/display/FLINK/%5BWIP%5D+FLIP-487%3A+Show+history+of+rescales+in+Web+UI+for+AdaptiveScheduler
> 
> [2] https://lists.apache.org/thread/f4md4btkf006mxcxf66bng1kfz0rsn8c
> 
> [3] https://cwiki.apache.org/confluence/display/FLINK/%5BWIP%5D+FLIP-495%3A+Support+AdaptiveScheduler+record+and+query+the+rescale+history
> 
> 
> 
> 
> Thank you very much.
> 
> 
> 
> 
> Best,
> 
> Regards.
> 
> Yuepeng Pan