You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@flink.apache.org by Ken Krugler <kk...@transpac.com> on 2022/12/16 15:48:37 UTC

Re: Slow restart from savepoint with large broadcast state when increasing parallelism

Hi Jun,

Thanks for following up.

The state storage is internal at a client, and isn’t throttled.

Also restoring from the savepoint when we didn’t change the parallelism was fine.

I didn’t see any errors in the TM logs, but I didn’t carefully inspect them - I’ll do that when we give this another test.

Broadcast state is weird in that it’s duplicated, apparently avoid “hot spots” when restoring from state. So I’m wondering how Flink handles the case of restoring broadcast state when the parallelism increases.

Regards,

— Ken
 

> On Dec 15, 2022, at 4:33 PM, Jun Qin <qi...@gmail.com> wrote:
> 
> Hi Ken,
> 
> Without knowning the details, the first thing I would suggest to check is whether you have reached a threshold which is configured in your state storage (e.g., s3) therefore your further download were throttled. Checking your storage metrics or logs should help to confirm whether this is the case.
> 
> In addition, in those TMs where the restarting was slow, do you see anything suspicious in the logs, e.g., reconnecting?
> 
> Thanks
> Jun
> 
> 
> 
> 
> 发自我的手机
> 
> 
> -------- 原始邮件 --------
> 发件人: Ken Krugler <kk...@transpac.com>
> 日期: 2022年12月14日周三 19:32
> 收件人: User <us...@flink.apache.org>
> 主 题: Slow restart from savepoint with large broadcast state when
> increasing parallelism
> Hi all,
> 
> I have a job with a large amount of broadcast state (62MB).
> 
> I took a savepoint when my workflow was running with parallelism 300.
> 
> I then restarted the workflow with parallelism 400.
> 
> The first 297 sub-tasks restored their broadcast state fairly quickly, but after that it slowed to a crawl (maybe 2 sub-tasks finished per minute)
> 
> After 10 minutes we killed the job, so I don’t know if it would have ultimately succeeded.
> 
> Is this expected? Seems like it could lead to a bad situation, where it would take an hour to restart the workflow.
> 
> Thanks,
> 
> — Ken
> 
> --------------------------
> Ken Krugler
> http://www.scaleunlimited.com <http://www.scaleunlimited.com/>
> Custom big data solutions
> Flink, Pinot, Solr, Elasticsearch
> 

--------------------------
Ken Krugler
http://www.scaleunlimited.com
Custom big data solutions
Flink, Pinot, Solr, Elasticsearch




Re: Slow restart from savepoint with large broadcast state when increasing parallelism

Posted by Jun Qin <qi...@gmail.com>.
Hi Ken,

> Broadcast state is weird in that it’s duplicated, apparently avoid “hot spots” when restoring from state. So I’m wondering how Flink handles the case of restoring broadcast state when the parallelism increases.

The Flink doc is here: https://nightlies.apache.org/flink/flink-docs-release-1.16/docs/dev/datastream/fault-tolerance/broadcast_state/. Particularly,
> Upon scaling up, each task reads its own state, and the remaining tasks (p_new-p_old) read checkpoints of previous tasks in a round-robin manner.

You could also consider to enable DEBUG logs (for necessary classes) when you give another try to see what happens in TMs. I also suggest to check all of your state storage metrics for any possible indication.

Thanks
Jun

> On Dec 16, 2022, at 4:48 PM, Ken Krugler <kk...@transpac.com> wrote:
> 
> Hi Jun,
> 
> Thanks for following up.
> 
> The state storage is internal at a client, and isn’t throttled.
> 
> Also restoring from the savepoint when we didn’t change the parallelism was fine.
> 
> I didn’t see any errors in the TM logs, but I didn’t carefully inspect them - I’ll do that when we give this another test.
> 
> Broadcast state is weird in that it’s duplicated, apparently avoid “hot spots” when restoring from state. So I’m wondering how Flink handles the case of restoring broadcast state when the parallelism increases.
> 
> Regards,
> 
> — Ken
>  
> 
>> On Dec 15, 2022, at 4:33 PM, Jun Qin <qinjunjerry@gmail.com <ma...@gmail.com>> wrote:
>> 
>> Hi Ken,
>> 
>> Without knowning the details, the first thing I would suggest to check is whether you have reached a threshold which is configured in your state storage (e.g., s3) therefore your further download were throttled. Checking your storage metrics or logs should help to confirm whether this is the case.
>> 
>> In addition, in those TMs where the restarting was slow, do you see anything suspicious in the logs, e.g., reconnecting?
>> 
>> Thanks
>> Jun
>> 
>> 
>> 
>> 
>> 发自我的手机
>> 
>> 
>> -------- 原始邮件 --------
>> 发件人: Ken Krugler <kkrugler_lists@transpac.com <ma...@transpac.com>>
>> 日期: 2022年12月14日周三 19:32
>> 收件人: User <user@flink.apache.org <ma...@flink.apache.org>>
>> 主 题: Slow restart from savepoint with large broadcast state when
>> increasing parallelism
>> Hi all,
>> 
>> I have a job with a large amount of broadcast state (62MB).
>> 
>> I took a savepoint when my workflow was running with parallelism 300.
>> 
>> I then restarted the workflow with parallelism 400.
>> 
>> The first 297 sub-tasks restored their broadcast state fairly quickly, but after that it slowed to a crawl (maybe 2 sub-tasks finished per minute)
>> 
>> After 10 minutes we killed the job, so I don’t know if it would have ultimately succeeded.
>> 
>> Is this expected? Seems like it could lead to a bad situation, where it would take an hour to restart the workflow.
>> 
>> Thanks,
>> 
>> — Ken
>> 
>> --------------------------
>> Ken Krugler
>> http://www.scaleunlimited.com <http://www.scaleunlimited.com/>
>> Custom big data solutions
>> Flink, Pinot, Solr, Elasticsearch
>> 
> 
> --------------------------
> Ken Krugler
> http://www.scaleunlimited.com <http://www.scaleunlimited.com/>
> Custom big data solutions
> Flink, Pinot, Solr, Elasticsearch
> 
> 
>