You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@flink.apache.org by "Yangze Guo (Jira)" <ji...@apache.org> on 2024/07/15 11:09:00 UTC
[jira] [Commented] (FLINK-20048) Make Approximate Local Recovery Compatible With PipelinedRegionSchedulingStrategy
[ https://issues.apache.org/jira/browse/FLINK-20048?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17866027#comment-17866027 ]
Yangze Guo commented on FLINK-20048:
------------------------------------
This issue is fixed by FLINK-27902, am I understand it correctly? [~Weijie Guo] [~ym]
> Make Approximate Local Recovery Compatible With PipelinedRegionSchedulingStrategy
> ---------------------------------------------------------------------------------
>
> Key: FLINK-20048
> URL: https://issues.apache.org/jira/browse/FLINK-20048
> Project: Flink
> Issue Type: Sub-task
> Components: Runtime / Coordination
> Affects Versions: 1.12.0
> Reporter: Yuan Mei
> Priority: Major
>
> in the case where approximate_failover is enabled, *only source task* is scheduled to be deployed, and consumer tasks never deployed. So I guess it may be related to how the region is used in the region PipelinedRegionSchedulingStrategy.
> * For *pipeline_(bounded)*, all vertices connected are considered as one region; there are not any dependent relations between regions, so PipelinedRegionSchedulingStrategy works roughly the same as EagerSchedulingStrategy
> * For *blocking*, regions are scheduled after dependent regions’ produced partitions are consumable.
> * From this point, it sounds like PipelinedRegionSchedulingStrategy should also work for pipeline_(approximate), but depends on how “produced partitions are consumable” is notified.
> The current version of how “produced partitions are consumable” is notified is very “blocking” specific.
> consumerRegions are maybe scheduled upon
> {{PipelinedRegionSchedulingStrategy#onExecutionStateChange}}
> First of all, it needs the “executionState == ExecutionState.FINISHED”;
> Second, only FINISHED and FAILED are notifiable in {{SchedulerBase#updateTaskExecutionState}}
> and e.t.c.
> I think if we make the “produced partitions are consumable” notification propagated properly for pipeline_approximated, it should work with pipeline_approximated as well.
> But the question is whether it is worthing the change? Because later, we probably won’t make each task in approximate mode a region after it has its own restart strategy?
> In short, the reason is approximate failover is restarted regionally but expected to be deployed as one region (if connected).
--
This message was sent by Atlassian Jira
(v8.20.10#820010)