You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@flink.apache.org by Paul Lam <pa...@gmail.com> on 2020/10/12 04:43:31 UTC
TM heartbeat timeout due to ResourceManager being busy
Hi,
After FLINK-13184 is implemented (even with Flink 1.11), occasionally there would still be jobs
with high parallelism getting TM-RM heartbeat timeouts when RM is busy creating TM contexts
on cluster initialization and HDFS is slow at that moment.
Apart from increasing the TM heartbeat timeout, is there any recommended out of the box
approach that can reduce the chance of getting the timeouts?
In the long run, is it possible to limit the number of taskmanager contexts that RM creates at
a time, so that the heartbeat triggers can chime in?
Thanks!
Best,
Paul Lam
Re: TM heartbeat timeout due to ResourceManager being busy
Posted by Xintong Song <to...@gmail.com>.
FYI, I just created FLINK-19568 for tracking this issue.
Thank you~
Xintong Song
[1] https://issues.apache.org/jira/browse/FLINK-19568
On Mon, Oct 12, 2020 at 2:18 PM Xintong Song <to...@gmail.com> wrote:
> Hi Paul,
>
> Thanks for reporting this.
>
> Indeed, Flink's RM currently performs several HDFS operations in the rpc
> main thread when preparing the TM context, which may block the main thread
> when HDFS is slow.
>
> Unfortunately, I don't see any out-of-box approach that fixes the problem
> at the moment, except for increasing the heartbeat timeout.
>
> As for the long run solution, I think there's an easier approach. We can
> move creating of the TM contexts away from the rpc main thread. Ideally, we
> should try to avoid performing any heavy operations which do not modify the
> RM's internal states in the rpc main thread. With FLINK-19241, this can be
> achieved easily by delegating the work to the io executor.
>
> Thank you~
>
> Xintong Song
>
>
>
> On Mon, Oct 12, 2020 at 12:44 PM Paul Lam <pa...@gmail.com> wrote:
>
>> Hi,
>>
>> After FLINK-13184 is implemented (even with Flink 1.11), occasionally
>> there would still be jobs
>> with high parallelism getting TM-RM heartbeat timeouts when RM is busy
>> creating TM contexts
>> on cluster initialization and HDFS is slow at that moment.
>>
>> Apart from increasing the TM heartbeat timeout, is there any recommended
>> out of the box
>> approach that can reduce the chance of getting the timeouts?
>>
>> In the long run, is it possible to limit the number of taskmanager
>> contexts that RM creates at
>> a time, so that the heartbeat triggers can chime in?
>>
>> Thanks!
>>
>> Best,
>> Paul Lam
>>
>
Re: TM heartbeat timeout due to ResourceManager being busy
Posted by Xintong Song <to...@gmail.com>.
No worries :)
Thank you~
Xintong Song
On Mon, Oct 12, 2020 at 2:48 PM Paul Lam <pa...@gmail.com> wrote:
> Sorry for the misspelled name, Xintong
>
> Best,
> Paul Lam
>
> 2020年10月12日 14:46，Paul Lam <pa...@gmail.com> 写道：
>
> Hi Xingtong,
>
> Thanks a lot for the pointer!
>
> It’s good to see there would be a new IO executor to take care of the TM
> contexts. Looking forward to the 1.12 release!
>
> Best,
> Paul Lam
>
> 2020年10月12日 14:18，Xintong Song <to...@gmail.com> 写道：
>
> Hi Paul,
>
> Thanks for reporting this.
>
> Indeed, Flink's RM currently performs several HDFS operations in the rpc
> main thread when preparing the TM context, which may block the main thread
> when HDFS is slow.
>
> Unfortunately, I don't see any out-of-box approach that fixes the problem
> at the moment, except for increasing the heartbeat timeout.
>
> As for the long run solution, I think there's an easier approach. We can
> move creating of the TM contexts away from the rpc main thread. Ideally, we
> should try to avoid performing any heavy operations which do not modify the
> RM's internal states in the rpc main thread. With FLINK-19241, this can be
> achieved easily by delegating the work to the io executor.
>
> Thank you~
> Xintong Song
>
>
>
> On Mon, Oct 12, 2020 at 12:44 PM Paul Lam <pa...@gmail.com> wrote:
>
>> Hi,
>>
>> After FLINK-13184 is implemented (even with Flink 1.11), occasionally
>> there would still be jobs
>> with high parallelism getting TM-RM heartbeat timeouts when RM is busy
>> creating TM contexts
>> on cluster initialization and HDFS is slow at that moment.
>>
>> Apart from increasing the TM heartbeat timeout, is there any recommended
>> out of the box
>> approach that can reduce the chance of getting the timeouts?
>>
>> In the long run, is it possible to limit the number of taskmanager
>> contexts that RM creates at
>> a time, so that the heartbeat triggers can chime in?
>>
>> Thanks!
>>
>> Best,
>> Paul Lam
>>
>
>
>
Re: TM heartbeat timeout due to ResourceManager being busy
Posted by Paul Lam <pa...@gmail.com>.
Sorry for the misspelled name, Xintong
Best,
Paul Lam
> 2020年10月12日 14:46，Paul Lam <pa...@gmail.com> 写道：
>
> Hi Xingtong,
>
> Thanks a lot for the pointer!
>
> It’s good to see there would be a new IO executor to take care of the TM contexts. Looking forward to the 1.12 release!
>
> Best,
> Paul Lam
>
>> 2020年10月12日 14:18，Xintong Song <tonysong820@gmail.com <ma...@gmail.com>> 写道：
>>
>> Hi Paul,
>>
>> Thanks for reporting this.
>>
>> Indeed, Flink's RM currently performs several HDFS operations in the rpc main thread when preparing the TM context, which may block the main thread when HDFS is slow.
>>
>> Unfortunately, I don't see any out-of-box approach that fixes the problem at the moment, except for increasing the heartbeat timeout.
>>
>> As for the long run solution, I think there's an easier approach. We can move creating of the TM contexts away from the rpc main thread. Ideally, we should try to avoid performing any heavy operations which do not modify the RM's internal states in the rpc main thread. With FLINK-19241, this can be achieved easily by delegating the work to the io executor.
>>
>> Thank you~
>> Xintong Song
>>
>>
>> On Mon, Oct 12, 2020 at 12:44 PM Paul Lam <paullin3280@gmail.com <ma...@gmail.com>> wrote:
>> Hi,
>>
>> After FLINK-13184 is implemented (even with Flink 1.11), occasionally there would still be jobs
>> with high parallelism getting TM-RM heartbeat timeouts when RM is busy creating TM contexts
>> on cluster initialization and HDFS is slow at that moment.
>>
>> Apart from increasing the TM heartbeat timeout, is there any recommended out of the box
>> approach that can reduce the chance of getting the timeouts?
>>
>> In the long run, is it possible to limit the number of taskmanager contexts that RM creates at
>> a time, so that the heartbeat triggers can chime in?
>>
>> Thanks!
>>
>> Best,
>> Paul Lam
>
Re: TM heartbeat timeout due to ResourceManager being busy
Posted by Paul Lam <pa...@gmail.com>.
Hi Xingtong,
Thanks a lot for the pointer!
It’s good to see there would be a new IO executor to take care of the TM contexts. Looking forward to the 1.12 release!
Best,
Paul Lam
> 2020年10月12日 14:18，Xintong Song <to...@gmail.com> 写道：
>
> Hi Paul,
>
> Thanks for reporting this.
>
> Indeed, Flink's RM currently performs several HDFS operations in the rpc main thread when preparing the TM context, which may block the main thread when HDFS is slow.
>
> Unfortunately, I don't see any out-of-box approach that fixes the problem at the moment, except for increasing the heartbeat timeout.
>
> As for the long run solution, I think there's an easier approach. We can move creating of the TM contexts away from the rpc main thread. Ideally, we should try to avoid performing any heavy operations which do not modify the RM's internal states in the rpc main thread. With FLINK-19241, this can be achieved easily by delegating the work to the io executor.
>
> Thank you~
> Xintong Song
>
>
> On Mon, Oct 12, 2020 at 12:44 PM Paul Lam <paullin3280@gmail.com <ma...@gmail.com>> wrote:
> Hi,
>
> After FLINK-13184 is implemented (even with Flink 1.11), occasionally there would still be jobs
> with high parallelism getting TM-RM heartbeat timeouts when RM is busy creating TM contexts
> on cluster initialization and HDFS is slow at that moment.
>
> Apart from increasing the TM heartbeat timeout, is there any recommended out of the box
> approach that can reduce the chance of getting the timeouts?
>
> In the long run, is it possible to limit the number of taskmanager contexts that RM creates at
> a time, so that the heartbeat triggers can chime in?
>
> Thanks!
>
> Best,
> Paul Lam
Re: TM heartbeat timeout due to ResourceManager being busy
Posted by Xintong Song <to...@gmail.com>.
Hi Paul,
Thanks for reporting this.
Indeed, Flink's RM currently performs several HDFS operations in the rpc
main thread when preparing the TM context, which may block the main thread
when HDFS is slow.
Unfortunately, I don't see any out-of-box approach that fixes the problem
at the moment, except for increasing the heartbeat timeout.
As for the long run solution, I think there's an easier approach. We can
move creating of the TM contexts away from the rpc main thread. Ideally, we
should try to avoid performing any heavy operations which do not modify the
RM's internal states in the rpc main thread. With FLINK-19241, this can be
achieved easily by delegating the work to the io executor.
Thank you~
Xintong Song
On Mon, Oct 12, 2020 at 12:44 PM Paul Lam <pa...@gmail.com> wrote:
> Hi,
>
> After FLINK-13184 is implemented (even with Flink 1.11), occasionally
> there would still be jobs
> with high parallelism getting TM-RM heartbeat timeouts when RM is busy
> creating TM contexts
> on cluster initialization and HDFS is slow at that moment.
>
> Apart from increasing the TM heartbeat timeout, is there any recommended
> out of the box
> approach that can reduce the chance of getting the timeouts?
>
> In the long run, is it possible to limit the number of taskmanager
> contexts that RM creates at
> a time, so that the heartbeat triggers can chime in?
>
> Thanks!
>
> Best,
> Paul Lam
>