You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@storm.apache.org by Ethan Li <et...@gmail.com> on 2020/03/06 03:08:13 UTC
Re: Storm 2.0 worker heartbeat
Hi Surajeet,
In 2.0, with the change in STORM-2693 <https://issues.apache.org/jira/browse/STORM-2693>, supervisors will send worker heartbeat to nimbus directly. Pacemaker is not required but still can be used.
Links:
[1] https://issues.apache.org/jira/browse/STORM-2693 <https://issues.apache.org/jira/browse/STORM-2693>
Best,
Ethan
> On Feb 18, 2020, at 1:03 PM, Surajeet Dev <su...@gmail.com> wrote:
>
> I am aware that Pacemaker was introduced to avoid Zookeeper becoming a bottleneck when there is high volume of worker heartbeat.
>
> Will Pacemaker be still required if we upgrade to Storm 2.0? Or is it with 2.0 , the whole thing has been re-architected?
>
> Regards
> Surajeet
Re: Storm 2.0 worker heartbeat
Posted by Andrew Neilson <ar...@gmail.com>.
Thank you!
On Wed, Apr 8, 2020 at 20:47 Ethan Li <et...@gmail.com> wrote:
> In summary, on 2.x, you shouldn't need to worry about PaceMaker. And I
> don't suggest to support running older version of topologies running on 2.x
> storm cluster. It is painful in practice.
>
> On Wed, Apr 8, 2020 at 10:44 PM Ethan Li <et...@gmail.com>
> wrote:
>
>> Hi Andrew,
>>
>> There are two places Pacemaker can be used.
>>
>> 1. Storm 2.x actually supports running older version of topologies (i.e.
>> topology compiled with storm 0.x or 1.x). We have been doing it inside my
>> company for a long time. It's kind of painful to support it but it works.
>> When older version of topologies running on 2.x Storm cluster, it runs
>> older worker code and it can use PaceMaker to mitigate the performance
>> issue on zookeeper. Hence the code here still check the heartbeat from
>> Zookeeper if any.
>> https://github.com/apache/storm/blob/master/storm-server/src/main/java/org/apache/storm/daemon/nimbus/Nimbus.java#L2219-L2222
>>
>>
>> 2. In 2.x, workers write heartbeats to disk and supervisor pick them up
>> and send to nimbus directly. It has another timer (run every 60s by
>> default) to send executor status through PaceMaker/Zookeeper.
>> https://github.com/apache/storm/blob/master/storm-client/src/jvm/org/apache/storm/daemon/worker/Worker.java#L217-L220
>> And these are executor metrics (emitted, executed, etc.) used on UI.
>> This timer only happens every 60s. So it shouldn't overload zookeeper. But
>> Pacemaker can still be used here. The graph at
>> https://github.com/apache/storm/pull/2389 might help to understand this.
>>
>> The code is merged at https://github.com/apache/storm/pull/2433
>>
>>
>> On Wed, Mar 18, 2020 at 6:38 PM Andrew Neilson <ar...@gmail.com>
>> wrote:
>>
>>> Hi Ethan,
>>>
>>> Pacemaker is not required but still can be used.
>>>>
>>>
>>> Under what circumstances would Pacemaker be used on v2.0+? It's not
>>> totally clear to me from how that ticket is written but it looks like that
>>> replaced all of the heartbeat logic that was managed by Pacemaker and ZK in
>>> older versions.
>>>
>>> Thanks,
>>> Andrew
>>>
>>> On Thu, Mar 5, 2020 at 7:08 PM Ethan Li <et...@gmail.com>
>>> wrote:
>>>
>>>> Hi Surajeet,
>>>>
>>>> In 2.0, with the change in STORM-2693
>>>> <https://issues.apache.org/jira/browse/STORM-2693>, supervisors will
>>>> send worker heartbeat to nimbus directly. Pacemaker is not required but
>>>> still can be used.
>>>>
>>>>
>>>> Links:
>>>> [1] https://issues.apache.org/jira/browse/STORM-2693
>>>>
>>>> Best,
>>>> Ethan
>>>>
>>>> On Feb 18, 2020, at 1:03 PM, Surajeet Dev <su...@gmail.com>
>>>> wrote:
>>>>
>>>> I am aware that Pacemaker was introduced to avoid Zookeeper becoming a
>>>> bottleneck when there is high volume of worker heartbeat.
>>>>
>>>> Will Pacemaker be still required if we upgrade to Storm 2.0? Or is it
>>>> with 2.0 , the whole thing has been re-architected?
>>>>
>>>> Regards
>>>> Surajeet
>>>>
>>>>
>>>>
Re: Storm 2.0 worker heartbeat
Posted by Ethan Li <et...@gmail.com>.
In summary, on 2.x, you shouldn't need to worry about PaceMaker. And I
don't suggest to support running older version of topologies running on 2.x
storm cluster. It is painful in practice.
On Wed, Apr 8, 2020 at 10:44 PM Ethan Li <et...@gmail.com> wrote:
> Hi Andrew,
>
> There are two places Pacemaker can be used.
>
> 1. Storm 2.x actually supports running older version of topologies (i.e.
> topology compiled with storm 0.x or 1.x). We have been doing it inside my
> company for a long time. It's kind of painful to support it but it works.
> When older version of topologies running on 2.x Storm cluster, it runs
> older worker code and it can use PaceMaker to mitigate the performance
> issue on zookeeper. Hence the code here still check the heartbeat from
> Zookeeper if any.
> https://github.com/apache/storm/blob/master/storm-server/src/main/java/org/apache/storm/daemon/nimbus/Nimbus.java#L2219-L2222
>
>
> 2. In 2.x, workers write heartbeats to disk and supervisor pick them up
> and send to nimbus directly. It has another timer (run every 60s by
> default) to send executor status through PaceMaker/Zookeeper.
> https://github.com/apache/storm/blob/master/storm-client/src/jvm/org/apache/storm/daemon/worker/Worker.java#L217-L220
> And these are executor metrics (emitted, executed, etc.) used on UI.
> This timer only happens every 60s. So it shouldn't overload zookeeper. But
> Pacemaker can still be used here. The graph at
> https://github.com/apache/storm/pull/2389 might help to understand this.
>
> The code is merged at https://github.com/apache/storm/pull/2433
>
>
> On Wed, Mar 18, 2020 at 6:38 PM Andrew Neilson <ar...@gmail.com>
> wrote:
>
>> Hi Ethan,
>>
>> Pacemaker is not required but still can be used.
>>>
>>
>> Under what circumstances would Pacemaker be used on v2.0+? It's not
>> totally clear to me from how that ticket is written but it looks like that
>> replaced all of the heartbeat logic that was managed by Pacemaker and ZK in
>> older versions.
>>
>> Thanks,
>> Andrew
>>
>> On Thu, Mar 5, 2020 at 7:08 PM Ethan Li <et...@gmail.com>
>> wrote:
>>
>>> Hi Surajeet,
>>>
>>> In 2.0, with the change in STORM-2693
>>> <https://issues.apache.org/jira/browse/STORM-2693>, supervisors will
>>> send worker heartbeat to nimbus directly. Pacemaker is not required but
>>> still can be used.
>>>
>>>
>>> Links:
>>> [1] https://issues.apache.org/jira/browse/STORM-2693
>>>
>>> Best,
>>> Ethan
>>>
>>> On Feb 18, 2020, at 1:03 PM, Surajeet Dev <su...@gmail.com>
>>> wrote:
>>>
>>> I am aware that Pacemaker was introduced to avoid Zookeeper becoming a
>>> bottleneck when there is high volume of worker heartbeat.
>>>
>>> Will Pacemaker be still required if we upgrade to Storm 2.0? Or is it
>>> with 2.0 , the whole thing has been re-architected?
>>>
>>> Regards
>>> Surajeet
>>>
>>>
>>>
Re: Storm 2.0 worker heartbeat
Posted by Ethan Li <et...@gmail.com>.
Hi Andrew,
There are two places Pacemaker can be used.
1. Storm 2.x actually supports running older version of topologies (i.e.
topology compiled with storm 0.x or 1.x). We have been doing it inside my
company for a long time. It's kind of painful to support it but it works.
When older version of topologies running on 2.x Storm cluster, it runs
older worker code and it can use PaceMaker to mitigate the performance
issue on zookeeper. Hence the code here still check the heartbeat from
Zookeeper if any.
https://github.com/apache/storm/blob/master/storm-server/src/main/java/org/apache/storm/daemon/nimbus/Nimbus.java#L2219-L2222
2. In 2.x, workers write heartbeats to disk and supervisor pick them up and
send to nimbus directly. It has another timer (run every 60s by default) to
send executor status through PaceMaker/Zookeeper.
https://github.com/apache/storm/blob/master/storm-client/src/jvm/org/apache/storm/daemon/worker/Worker.java#L217-L220
And these are executor metrics (emitted, executed, etc.) used on UI.
This timer only happens every 60s. So it shouldn't overload zookeeper. But
Pacemaker can still be used here. The graph at
https://github.com/apache/storm/pull/2389 might help to understand this.
The code is merged at https://github.com/apache/storm/pull/2433
On Wed, Mar 18, 2020 at 6:38 PM Andrew Neilson <ar...@gmail.com> wrote:
> Hi Ethan,
>
> Pacemaker is not required but still can be used.
>>
>
> Under what circumstances would Pacemaker be used on v2.0+? It's not
> totally clear to me from how that ticket is written but it looks like that
> replaced all of the heartbeat logic that was managed by Pacemaker and ZK in
> older versions.
>
> Thanks,
> Andrew
>
> On Thu, Mar 5, 2020 at 7:08 PM Ethan Li <et...@gmail.com> wrote:
>
>> Hi Surajeet,
>>
>> In 2.0, with the change in STORM-2693
>> <https://issues.apache.org/jira/browse/STORM-2693>, supervisors will
>> send worker heartbeat to nimbus directly. Pacemaker is not required but
>> still can be used.
>>
>>
>> Links:
>> [1] https://issues.apache.org/jira/browse/STORM-2693
>>
>> Best,
>> Ethan
>>
>> On Feb 18, 2020, at 1:03 PM, Surajeet Dev <su...@gmail.com>
>> wrote:
>>
>> I am aware that Pacemaker was introduced to avoid Zookeeper becoming a
>> bottleneck when there is high volume of worker heartbeat.
>>
>> Will Pacemaker be still required if we upgrade to Storm 2.0? Or is it
>> with 2.0 , the whole thing has been re-architected?
>>
>> Regards
>> Surajeet
>>
>>
>>
Re: Storm 2.0 worker heartbeat
Posted by Andrew Neilson <ar...@gmail.com>.
Hi Ethan,
Pacemaker is not required but still can be used.
>
Under what circumstances would Pacemaker be used on v2.0+? It's not totally
clear to me from how that ticket is written but it looks like that replaced
all of the heartbeat logic that was managed by Pacemaker and ZK in older
versions.
Thanks,
Andrew
On Thu, Mar 5, 2020 at 7:08 PM Ethan Li <et...@gmail.com> wrote:
> Hi Surajeet,
>
> In 2.0, with the change in STORM-2693
> <https://issues.apache.org/jira/browse/STORM-2693>, supervisors will send
> worker heartbeat to nimbus directly. Pacemaker is not required but still
> can be used.
>
>
> Links:
> [1] https://issues.apache.org/jira/browse/STORM-2693
>
> Best,
> Ethan
>
> On Feb 18, 2020, at 1:03 PM, Surajeet Dev <su...@gmail.com>
> wrote:
>
> I am aware that Pacemaker was introduced to avoid Zookeeper becoming a
> bottleneck when there is high volume of worker heartbeat.
>
> Will Pacemaker be still required if we upgrade to Storm 2.0? Or is it with
> 2.0 , the whole thing has been re-architected?
>
> Regards
> Surajeet
>
>
>