You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@flink.apache.org by Jai Patel <ja...@cloudkitchens.com> on 2022/04/15 21:51:05 UTC
Re: java.lang.Exception: Job leader for job id 0efd8681eda64b072b72baef58722bc0 lost leadership.
Hi Nico,
Wanted to close the loop here. We did end up find a number of problems in
our code:
1. Our operator was slow. It was iterating over several large Protobufs in
a MapState then filtering it down to 1. We were able to identify that one
up-front and significantly improve the runtime of the operator.
2. We did increase the total memory and raised the managed memory fraction
from 40% to 70%.
Both solutions were needed in order to bring us the stability we were
looking for. Since then, everything has been running great.
Thanks.
Jai
On Thu, Feb 24, 2022 at 2:42 AM Nicolaus Weidner <
nicolaus.weidner@ververica.com> wrote:
> Hi Jai,
>
> Do writes to ValueStates/MapStates have a direct on churn of the Flink
>> State or is the data buffered in between?
>>
>
> Writes to keyed state go directly to RocksDB. So there shouldn't be any
> memory issues with buffers overflowing or similar. In general, more memory
> should increase performance (larger cache sizes before having to write to
> disk), but less memory shouldn't cause crashes.
>
> Since the errors you encountered are not that specific, can you provide
> full logs surrounding such incidents? There is not much to go on without
> further info.
>
> Best,
> Nico
>
>>
Re: java.lang.Exception: Job leader for job id 0efd8681eda64b072b72baef58722bc0 lost leadership.
Posted by Zhanghao Chen <zh...@outlook.com>.
Hi, to unsubscribe you need to send a mail to user-unsubscribe@flink.apache.org<ma...@flink.apache.org>.
Best,
Zhanghao Chen
________________________________
From: Samir Vasani <sa...@gmail.com>
Sent: Saturday, April 16, 2022 1:08:20 PM
To: Jai Patel <ja...@cloudkitchens.com>
Cc: Nicolaus Weidner <ni...@ververica.com>; user <us...@flink.apache.org>; Weixiang Sun <we...@cloudkitchens.com>
Subject: Re: java.lang.Exception: Job leader for job id 0efd8681eda64b072b72baef58722bc0 lost leadership.
how to unsubscribe?
On Sat, Apr 16, 2022 at 3:21 AM Jai Patel <ja...@cloudkitchens.com>> wrote:
Hi Nico,
Wanted to close the loop here. We did end up find a number of problems in our code:
1. Our operator was slow. It was iterating over several large Protobufs in a MapState then filtering it down to 1. We were able to identify that one up-front and significantly improve the runtime of the operator.
2. We did increase the total memory and raised the managed memory fraction from 40% to 70%.
Both solutions were needed in order to bring us the stability we were looking for. Since then, everything has been running great.
Thanks.
Jai
On Thu, Feb 24, 2022 at 2:42 AM Nicolaus Weidner <ni...@ververica.com>> wrote:
Hi Jai,
Do writes to ValueStates/MapStates have a direct on churn of the Flink State or is the data buffered in between?
Writes to keyed state go directly to RocksDB. So there shouldn't be any memory issues with buffers overflowing or similar. In general, more memory should increase performance (larger cache sizes before having to write to disk), but less memory shouldn't cause crashes.
Since the errors you encountered are not that specific, can you provide full logs surrounding such incidents? There is not much to go on without further info.
Best,
Nico
Re: java.lang.Exception: Job leader for job id 0efd8681eda64b072b72baef58722bc0 lost leadership.
Posted by Samir Vasani <sa...@gmail.com>.
how to unsubscribe?
On Sat, Apr 16, 2022 at 3:21 AM Jai Patel <ja...@cloudkitchens.com>
wrote:
> Hi Nico,
>
> Wanted to close the loop here. We did end up find a number of problems in
> our code:
> 1. Our operator was slow. It was iterating over several large Protobufs
> in a MapState then filtering it down to 1. We were able to identify that
> one up-front and significantly improve the runtime of the operator.
> 2. We did increase the total memory and raised the managed memory fraction
> from 40% to 70%.
>
> Both solutions were needed in order to bring us the stability we were
> looking for. Since then, everything has been running great.
>
> Thanks.
> Jai
>
> On Thu, Feb 24, 2022 at 2:42 AM Nicolaus Weidner <
> nicolaus.weidner@ververica.com> wrote:
>
>> Hi Jai,
>>
>> Do writes to ValueStates/MapStates have a direct on churn of the Flink
>>> State or is the data buffered in between?
>>>
>>
>> Writes to keyed state go directly to RocksDB. So there shouldn't be any
>> memory issues with buffers overflowing or similar. In general, more memory
>> should increase performance (larger cache sizes before having to write to
>> disk), but less memory shouldn't cause crashes.
>>
>> Since the errors you encountered are not that specific, can you provide
>> full logs surrounding such incidents? There is not much to go on without
>> further info.
>>
>> Best,
>> Nico
>>
>>>