You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@storm.apache.org by Ethan Li <et...@gmail.com> on 2019/08/22 16:06:20 UTC
Re: pacemaker question
Hi Ignor,
If you are running 2.0.0 or higher version of topology on 2.x Storm cluster, thanks to the changes of https://issues.apache.org/jira/browse/STORM-2693 <https://issues.apache.org/jira/browse/STORM-2693>, https://github.com/apache/storm/pull/2433 <https://github.com/apache/storm/pull/2433>, Pacemaker is no longer needed. It won’t do anything.
If you are running older version of topology on 2.x Storm cluster, Pacemaker is an additional daemon that takes some load from Zookeeper so it will not be overloaded, which means Zookeeper is still required. https://github.com/apache/storm/blob/master/docs/Pacemaker.md <https://github.com/apache/storm/blob/master/docs/Pacemaker.md>. So you will see supervisors connecting to zookeeper.
storm.zookeeper.port is used to connect to zookeeper so you want to set it as your zookeeper port, default to 2181. https://github.com/apache/storm/blob/master/storm-client/src/jvm/org/apache/storm/Config.java#L1067 <https://github.com/apache/storm/blob/master/storm-client/src/jvm/org/apache/storm/Config.java#L1067>
Thanks
Ethan
> On Aug 22, 2019, at 10:24 AM, Igor A. <ig...@gmail.com> wrote:
>
> Hello all,
>
> I'm trying to run Storm version 2.0.0 with Pacemaker.
> I've set up everything like it is described in documentation (there is not so much parameters to miss something). Pacemaker is listening on default :6699.
> The issue is despite the fact storm.cluster.state.store is set to "[...].PaceMakerStateStorageFactory", Storm Supervisor tries to connect to Zookeeper on :2181.
>
> I can see in logs that Storm launched Zookeeper client and it runs in 3.4 compatibility mode.
> I tried to set storm.zookeeper.port to 6699, but this caused Pacemaker to die with an NPE. Note: after reading sources i've found that PacemakerClient doesn't use zookeeper.port
> Does anyone know how to make Storm communicate with Pacemaker? May be somebody have a working storm.yaml to enable Pacemake Client?
>
>
>
Re: pacemaker question
Posted by "Igor A." <ig...@gmail.com>.
Thank you, Stig, Ethan!
On Fri, Aug 23, 2019, 18:52 Ethan Li <et...@gmail.com> wrote:
> Yes. Zookeeper is required.
> https://storm.apache.org/releases/2.0.0/Setting-up-a-Storm-cluster.html
>
> Even if you run Apache Storm in localMode,
> https://storm.apache.org/releases/2.0.0/Local-mode.html, an in process
> Zookeeper will be launched. (
> https://github.com/apache/storm/blob/master/storm-server/src/main/java/org/apache/storm/LocalCluster.java#L144
> )
>
>
>
> On Aug 23, 2019, at 8:32 AM, Stig Rohde Døssing <st...@gmail.com>
> wrote:
>
> As far as I know you can't use Storm without Zookeeper. Pacemaker is just
> for receiving heartbeats, Storm still needs to store other information in
> Zookeeper, e.g. which topologies are deployed.
>
> Den fre. 23. aug. 2019 kl. 10.57 skrev Igor A. <ig...@gmail.com>:
>
>> Ethan,
>>
>> >Pacemaker is an additional daemon that takes some load from Zookeeper
>> so it will not be overloaded, which means Zookeeper is still required.
>>
>> could you please confirm that Zookeeper is required in any Storm
>> configuration?
>>
>> On Thu, Aug 22, 2019, 19:06 Ethan Li <et...@gmail.com> wrote:
>>
>>> Hi Ignor,
>>>
>>> If you are running 2.0.0 or higher version of topology on 2.x Storm
>>> cluster, thanks to the changes of
>>> https://issues.apache.org/jira/browse/STORM-2693,
>>> https://github.com/apache/storm/pull/2433, Pacemaker is no longer
>>> needed. It won’t do anything.
>>>
>>>
>>> If you are running older version of topology on 2.x Storm cluster,
>>> Pacemaker is an additional daemon that takes some load from Zookeeper so
>>> it will not be overloaded, which means Zookeeper is still required.
>>> https://github.com/apache/storm/blob/master/docs/Pacemaker.md. So you
>>> will see supervisors connecting to zookeeper.
>>>
>>> storm.zookeeper.port is used to connect to zookeeper so you want to set
>>> it as your zookeeper port, default to 2181.
>>> https://github.com/apache/storm/blob/master/storm-client/src/jvm/org/apache/storm/Config.java#L1067
>>>
>>>
>>> Thanks
>>> Ethan
>>>
>>>
>>> On Aug 22, 2019, at 10:24 AM, Igor A. <ig...@gmail.com> wrote:
>>>
>>> Hello all,
>>>
>>> I'm trying to run Storm version 2.0.0 with Pacemaker.
>>> I've set up everything like it is described in documentation (there is
>>> not so much parameters to miss something). Pacemaker is listening on
>>> default :6699.
>>> The issue is despite the fact storm.cluster.state.store is set to
>>> "[...].PaceMakerStateStorageFactory", Storm Supervisor tries to connect to
>>> Zookeeper on :2181.
>>>
>>> I can see in logs that Storm launched Zookeeper client and it runs in
>>> 3.4 compatibility mode.
>>> I tried to set storm.zookeeper.port to 6699, but this caused Pacemaker
>>> to die with an NPE. Note: after reading sources i've found that
>>> PacemakerClient doesn't use zookeeper.port
>>> Does anyone know how to make Storm communicate with Pacemaker? May be
>>> somebody have a working storm.yaml to enable Pacemake Client?
>>>
>>>
>>>
>>>
>>>
>
Re: pacemaker question
Posted by Ethan Li <et...@gmail.com>.
Yes. Zookeeper is required. https://storm.apache.org/releases/2.0.0/Setting-up-a-Storm-cluster.html <https://storm.apache.org/releases/2.0.0/Setting-up-a-Storm-cluster.html>
Even if you run Apache Storm in localMode, https://storm.apache.org/releases/2.0.0/Local-mode.html <https://storm.apache.org/releases/2.0.0/Local-mode.html>, an in process Zookeeper will be launched. (https://github.com/apache/storm/blob/master/storm-server/src/main/java/org/apache/storm/LocalCluster.java#L144 <https://github.com/apache/storm/blob/master/storm-server/src/main/java/org/apache/storm/LocalCluster.java#L144>)
> On Aug 23, 2019, at 8:32 AM, Stig Rohde Døssing <st...@gmail.com> wrote:
>
> As far as I know you can't use Storm without Zookeeper. Pacemaker is just for receiving heartbeats, Storm still needs to store other information in Zookeeper, e.g. which topologies are deployed.
>
> Den fre. 23. aug. 2019 kl. 10.57 skrev Igor A. <igor.fraa@gmail.com <ma...@gmail.com>>:
> Ethan,
>
> >Pacemaker is an additional daemon that takes some load from Zookeeper so it will not be overloaded, which means Zookeeper is still required.
>
> could you please confirm that Zookeeper is required in any Storm configuration?
>
> On Thu, Aug 22, 2019, 19:06 Ethan Li <ethanopensource@gmail.com <ma...@gmail.com>> wrote:
> Hi Ignor,
>
> If you are running 2.0.0 or higher version of topology on 2.x Storm cluster, thanks to the changes of https://issues.apache.org/jira/browse/STORM-2693 <https://issues.apache.org/jira/browse/STORM-2693>, https://github.com/apache/storm/pull/2433 <https://github.com/apache/storm/pull/2433>, Pacemaker is no longer needed. It won’t do anything.
>
>
> If you are running older version of topology on 2.x Storm cluster, Pacemaker is an additional daemon that takes some load from Zookeeper so it will not be overloaded, which means Zookeeper is still required. https://github.com/apache/storm/blob/master/docs/Pacemaker.md <https://github.com/apache/storm/blob/master/docs/Pacemaker.md>. So you will see supervisors connecting to zookeeper.
>
> storm.zookeeper.port is used to connect to zookeeper so you want to set it as your zookeeper port, default to 2181. https://github.com/apache/storm/blob/master/storm-client/src/jvm/org/apache/storm/Config.java#L1067 <https://github.com/apache/storm/blob/master/storm-client/src/jvm/org/apache/storm/Config.java#L1067>
>
>
> Thanks
> Ethan
>
>
>> On Aug 22, 2019, at 10:24 AM, Igor A. <igor.fraa@gmail.com <ma...@gmail.com>> wrote:
>>
>> Hello all,
>>
>> I'm trying to run Storm version 2.0.0 with Pacemaker.
>> I've set up everything like it is described in documentation (there is not so much parameters to miss something). Pacemaker is listening on default :6699.
>> The issue is despite the fact storm.cluster.state.store is set to "[...].PaceMakerStateStorageFactory", Storm Supervisor tries to connect to Zookeeper on :2181.
>>
>> I can see in logs that Storm launched Zookeeper client and it runs in 3.4 compatibility mode.
>> I tried to set storm.zookeeper.port to 6699, but this caused Pacemaker to die with an NPE. Note: after reading sources i've found that PacemakerClient doesn't use zookeeper.port
>> Does anyone know how to make Storm communicate with Pacemaker? May be somebody have a working storm.yaml to enable Pacemake Client?
>>
>>
>>
>
Re: pacemaker question
Posted by Stig Rohde Døssing <st...@gmail.com>.
As far as I know you can't use Storm without Zookeeper. Pacemaker is just
for receiving heartbeats, Storm still needs to store other information in
Zookeeper, e.g. which topologies are deployed.
Den fre. 23. aug. 2019 kl. 10.57 skrev Igor A. <ig...@gmail.com>:
> Ethan,
>
> >Pacemaker is an additional daemon that takes some load from Zookeeper so
> it will not be overloaded, which means Zookeeper is still required.
>
> could you please confirm that Zookeeper is required in any Storm
> configuration?
>
> On Thu, Aug 22, 2019, 19:06 Ethan Li <et...@gmail.com> wrote:
>
>> Hi Ignor,
>>
>> If you are running 2.0.0 or higher version of topology on 2.x Storm
>> cluster, thanks to the changes of
>> https://issues.apache.org/jira/browse/STORM-2693,
>> https://github.com/apache/storm/pull/2433, Pacemaker is no longer
>> needed. It won’t do anything.
>>
>>
>> If you are running older version of topology on 2.x Storm cluster,
>> Pacemaker is an additional daemon that takes some load from Zookeeper so
>> it will not be overloaded, which means Zookeeper is still required.
>> https://github.com/apache/storm/blob/master/docs/Pacemaker.md. So you
>> will see supervisors connecting to zookeeper.
>>
>> storm.zookeeper.port is used to connect to zookeeper so you want to set
>> it as your zookeeper port, default to 2181.
>> https://github.com/apache/storm/blob/master/storm-client/src/jvm/org/apache/storm/Config.java#L1067
>>
>>
>> Thanks
>> Ethan
>>
>>
>> On Aug 22, 2019, at 10:24 AM, Igor A. <ig...@gmail.com> wrote:
>>
>> Hello all,
>>
>> I'm trying to run Storm version 2.0.0 with Pacemaker.
>> I've set up everything like it is described in documentation (there is
>> not so much parameters to miss something). Pacemaker is listening on
>> default :6699.
>> The issue is despite the fact storm.cluster.state.store is set to
>> "[...].PaceMakerStateStorageFactory", Storm Supervisor tries to connect to
>> Zookeeper on :2181.
>>
>> I can see in logs that Storm launched Zookeeper client and it runs in 3.4
>> compatibility mode.
>> I tried to set storm.zookeeper.port to 6699, but this caused Pacemaker to
>> die with an NPE. Note: after reading sources i've found that
>> PacemakerClient doesn't use zookeeper.port
>> Does anyone know how to make Storm communicate with Pacemaker? May be
>> somebody have a working storm.yaml to enable Pacemake Client?
>>
>>
>>
>>
>>
Re: pacemaker question
Posted by "Igor A." <ig...@gmail.com>.
Ethan,
>Pacemaker is an additional daemon that takes some load from Zookeeper so
it will not be overloaded, which means Zookeeper is still required.
could you please confirm that Zookeeper is required in any Storm
configuration?
On Thu, Aug 22, 2019, 19:06 Ethan Li <et...@gmail.com> wrote:
> Hi Ignor,
>
> If you are running 2.0.0 or higher version of topology on 2.x Storm
> cluster, thanks to the changes of
> https://issues.apache.org/jira/browse/STORM-2693,
> https://github.com/apache/storm/pull/2433, Pacemaker is no longer needed.
> It won’t do anything.
>
>
> If you are running older version of topology on 2.x Storm cluster,
> Pacemaker is an additional daemon that takes some load from Zookeeper so
> it will not be overloaded, which means Zookeeper is still required.
> https://github.com/apache/storm/blob/master/docs/Pacemaker.md. So you
> will see supervisors connecting to zookeeper.
>
> storm.zookeeper.port is used to connect to zookeeper so you want to set it
> as your zookeeper port, default to 2181.
> https://github.com/apache/storm/blob/master/storm-client/src/jvm/org/apache/storm/Config.java#L1067
>
>
> Thanks
> Ethan
>
>
> On Aug 22, 2019, at 10:24 AM, Igor A. <ig...@gmail.com> wrote:
>
> Hello all,
>
> I'm trying to run Storm version 2.0.0 with Pacemaker.
> I've set up everything like it is described in documentation (there is not
> so much parameters to miss something). Pacemaker is listening on default
> :6699.
> The issue is despite the fact storm.cluster.state.store is set to
> "[...].PaceMakerStateStorageFactory", Storm Supervisor tries to connect to
> Zookeeper on :2181.
>
> I can see in logs that Storm launched Zookeeper client and it runs in 3.4
> compatibility mode.
> I tried to set storm.zookeeper.port to 6699, but this caused Pacemaker to
> die with an NPE. Note: after reading sources i've found that
> PacemakerClient doesn't use zookeeper.port
> Does anyone know how to make Storm communicate with Pacemaker? May be
> somebody have a working storm.yaml to enable Pacemake Client?
>
>
>
>
>
Re: pacemaker question
Posted by Ethan Li <et...@gmail.com>.
Sorry Igor for not typing your name correctly.
> On Aug 22, 2019, at 11:06 AM, Ethan Li <et...@gmail.com> wrote:
>
> Hi Ignor,
>
> If you are running 2.0.0 or higher version of topology on 2.x Storm cluster, thanks to the changes of https://issues.apache.org/jira/browse/STORM-2693 <https://issues.apache.org/jira/browse/STORM-2693>, https://github.com/apache/storm/pull/2433 <https://github.com/apache/storm/pull/2433>, Pacemaker is no longer needed. It won’t do anything.
>
>
> If you are running older version of topology on 2.x Storm cluster, Pacemaker is an additional daemon that takes some load from Zookeeper so it will not be overloaded, which means Zookeeper is still required. https://github.com/apache/storm/blob/master/docs/Pacemaker.md <https://github.com/apache/storm/blob/master/docs/Pacemaker.md>. So you will see supervisors connecting to zookeeper.
>
> storm.zookeeper.port is used to connect to zookeeper so you want to set it as your zookeeper port, default to 2181. https://github.com/apache/storm/blob/master/storm-client/src/jvm/org/apache/storm/Config.java#L1067 <https://github.com/apache/storm/blob/master/storm-client/src/jvm/org/apache/storm/Config.java#L1067>
>
>
> Thanks
> Ethan
>
>
>> On Aug 22, 2019, at 10:24 AM, Igor A. <igor.fraa@gmail.com <ma...@gmail.com>> wrote:
>>
>> Hello all,
>>
>> I'm trying to run Storm version 2.0.0 with Pacemaker.
>> I've set up everything like it is described in documentation (there is not so much parameters to miss something). Pacemaker is listening on default :6699.
>> The issue is despite the fact storm.cluster.state.store is set to "[...].PaceMakerStateStorageFactory", Storm Supervisor tries to connect to Zookeeper on :2181.
>>
>> I can see in logs that Storm launched Zookeeper client and it runs in 3.4 compatibility mode.
>> I tried to set storm.zookeeper.port to 6699, but this caused Pacemaker to die with an NPE. Note: after reading sources i've found that PacemakerClient doesn't use zookeeper.port
>> Does anyone know how to make Storm communicate with Pacemaker? May be somebody have a working storm.yaml to enable Pacemake Client?
>>
>>
>>
>
Re: pacemaker question
Posted by "Igor A." <ig...@gmail.com>.
Hello Ethan,
Thank you for your help.
Actually we are running Storm 1.1.0 . We are planning upgrade to 2.x but
for later.
According to https://issues.apache.org/jira/browse/STORM-2693 request for
1.1.0 was not merged so 1.1.x branch lacks support for collecting heartbits
in the supervisor itself.
Thank you again for clarification of Zookeeper role.
On Thu, Aug 22, 2019, 19:06 Ethan Li <et...@gmail.com> wrote:
> Hi Ignor,
>
> If you are running 2.0.0 or higher version of topology on 2.x Storm
> cluster, thanks to the changes of
> https://issues.apache.org/jira/browse/STORM-2693,
> https://github.com/apache/storm/pull/2433, Pacemaker is no longer needed.
> It won’t do anything.
>
>
> If you are running older version of topology on 2.x Storm cluster,
> Pacemaker is an additional daemon that takes some load from Zookeeper so
> it will not be overloaded, which means Zookeeper is still required.
> https://github.com/apache/storm/blob/master/docs/Pacemaker.md. So you
> will see supervisors connecting to zookeeper.
>
> storm.zookeeper.port is used to connect to zookeeper so you want to set it
> as your zookeeper port, default to 2181.
> https://github.com/apache/storm/blob/master/storm-client/src/jvm/org/apache/storm/Config.java#L1067
>
>
> Thanks
> Ethan
>
>
> On Aug 22, 2019, at 10:24 AM, Igor A. <ig...@gmail.com> wrote:
>
> Hello all,
>
> I'm trying to run Storm version 2.0.0 with Pacemaker.
> I've set up everything like it is described in documentation (there is not
> so much parameters to miss something). Pacemaker is listening on default
> :6699.
> The issue is despite the fact storm.cluster.state.store is set to
> "[...].PaceMakerStateStorageFactory", Storm Supervisor tries to connect to
> Zookeeper on :2181.
>
> I can see in logs that Storm launched Zookeeper client and it runs in 3.4
> compatibility mode.
> I tried to set storm.zookeeper.port to 6699, but this caused Pacemaker to
> die with an NPE. Note: after reading sources i've found that
> PacemakerClient doesn't use zookeeper.port
> Does anyone know how to make Storm communicate with Pacemaker? May be
> somebody have a working storm.yaml to enable Pacemake Client?
>
>
>
>
>