You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@storm.apache.org by "Kazansky, Michael" <mi...@jpmchase.com> on 2016/06/09 17:11:23 UTC
not all the servers in storm cluster processing messages
Hi All,
We have 3 servers in storm cluster. To prevent the order of the messages which come to Kafka the Kafka partition was set to 1. KafkaSpout parallelism hint is set to 1 as well. When messages flow through the Storm we noticed that only one server in the cluster is busy processing messages other 2 are idle although the number of workers is set to 3 in Config. Is it excepted behavior or something is missing?
If it is expected behavior is there any way to have all 3 servers in the cluster processing messages?
Storm version is 0.9.5
Thanks,
Michael Kazansky
This communication is for informational purposes only. It is not intended as an offer or solicitation for the purchase or sale of any financial instrument or as an official confirmation of any transaction. All market prices, data and other information are not warranted as to completeness or accuracy and are subject to change without notice. Any comments or statements made herein do not necessarily reflect those of JPMorgan Chase & Co., its subsidiaries and affiliates (collectively, "JPMC").
This transmission may contain information that is proprietary, privileged, confidential and/or exempt from disclosure under applicable law. If you are not the intended recipient, you are hereby notified that any disclosure, copying, distribution, or use of the information contained herein (including any reliance thereon) is STRICTLY PROHIBITED. If you received this transmission in error, please immediately contact the sender and destroy the material in its entirety, whether in electronic or hard copy format. Although this transmission and any attachments are believed to be free of any virus or other defect that might affect any computer system into which it is received and opened, it is the responsibility of the recipient to ensure that it is virus free and no responsibility is accepted by JPMC for any loss or damage arising in any way from its use. Please note that any electronic communication that is conducted within or through JPMC's systems is subject to interception, monitoring, review, retention and external production in accordance with JPMC's policy and local laws, rules and regulations; may be stored or otherwise processed in countries other than the country in which you are located; and will be treated in accordance with JPMC policies and applicable laws and regulations.
Please refer to http://www.jpmorgan.com/pages/disclosures for disclosures relating to European legal entities.
Re: not all the servers in storm cluster processing messages
Posted by Satish Duggana <sa...@gmail.com>.
Below link maybe helpful in understanding parallelism in storm
http://storm.apache.org/releases/1.0.1/Understanding-the-parallelism-of-a-Storm-topology.html
On Thu, Jun 9, 2016 at 11:24 PM, Satish Duggana <sa...@gmail.com>
wrote:
> Hi Michael,
> Kafka spout is given parallelism-hint as 1 and the topic has only one
> partition. So, only one kafka spout task is run in one of the workers of
> the topology. Are you asking that bolts connected directly/indirectly from
> that kafka spout are all executed in the same worker as kafka spout? If so,
> What are the groupings and parallellism-hint that you have configured on
> those bolts?
>
> Thanks,
> Satish.
>
>
> On Thu, Jun 9, 2016 at 10:41 PM, Kazansky, Michael <
> michael.kazansky@jpmchase.com> wrote:
>
>> Hi All,
>>
>> We have 3 servers in storm cluster. To prevent the order of the messages
>> which come to Kafka the Kafka partition was set to 1. KafkaSpout
>> parallelism hint is set to 1 as well. When messages flow through the Storm
>> we noticed that only one server in the cluster is busy processing messages
>> other 2 are idle although the number of workers is set to 3 in Config. Is
>> it excepted behavior or something is missing?
>>
>> If it is expected behavior is there any way to have all 3 servers in the
>> cluster processing messages?
>>
>> Storm version is 0.9.5
>>
>> Thanks,
>>
>> Michael Kazansky
>>
>>
>>
>> This communication is for informational purposes only. It is not intended
>> as an offer or solicitation for the purchase or sale of any financial
>> instrument or as an official confirmation of any transaction. All market
>> prices, data and other information are not warranted as to completeness or
>> accuracy and are subject to change without notice. Any comments or
>> statements made herein do not necessarily reflect those of JPMorgan Chase &
>> Co., its subsidiaries and affiliates (collectively, "JPMC"). This
>> transmission may contain information that is proprietary, privileged,
>> confidential and/or exempt from disclosure under applicable law. If you are
>> not the intended recipient, you are hereby notified that any disclosure,
>> copying, distribution, or use of the information contained herein
>> (including any reliance thereon) is STRICTLY PROHIBITED. If you received
>> this transmission in error, please immediately contact the sender and
>> destroy the material in its entirety, whether in electronic or hard copy
>> format. Although this transmission and any attachments are believed to be
>> free of any virus or other defect that might affect any computer system
>> into which it is received and opened, it is the responsibility of the
>> recipient to ensure that it is virus free and no responsibility is accepted
>> by JPMC for any loss or damage arising in any way from its use. Please note
>> that any electronic communication that is conducted within or through
>> JPMC's systems is subject to interception, monitoring, review, retention
>> and external production in accordance with JPMC's policy and local laws,
>> rules and regulations; may be stored or otherwise processed in countries
>> other than the country in which you are located; and will be treated in
>> accordance with JPMC policies and applicable laws and regulations. Please
>> refer to http://www.jpmorgan.com/pages/disclosures for disclosures
>> relating to European legal entities.
>>
>
>
RE: not all the servers in storm cluster processing messages
Posted by "Kazansky, Michael" <mi...@jpmchase.com>.
Thanks I will try it out.
Thanks,
Michael Kazansky
From: Nathan Leung [mailto:ncleung@gmail.com]
Sent: Thursday, June 09, 2016 3:07 PM
To: user
Subject: Re: not all the servers in storm cluster processing messages
If you use localOrShuffleGrouping, the tuple will stay in process if possible. The solution, assuming your bolts are distributed, is to use shuffleGrouping to send tuples from spout->bolt so that the tuples are distributed to all of your workers, then use localOrShuffleGrouping from bolt->bolt to reduce inter-worker communication overhead.
On Thu, Jun 9, 2016 at 3:00 PM, Kazansky, Michael <mi...@jpmchase.com>> wrote:
We are using localOrShuffleGrouping
Thanks,
Michael Kazansky
From: Satish Duggana [mailto:satish.duggana@gmail.com<ma...@gmail.com>]
Sent: Thursday, June 09, 2016 1:54 PM
To: user@storm.apache.org<ma...@storm.apache.org>
Subject: Re: not all the servers in storm cluster processing messages
Hi Michael,
Kafka spout is given parallelism-hint as 1 and the topic has only one partition. So, only one kafka spout task is run in one of the workers of the topology. Are you asking that bolts connected directly/indirectly from that kafka spout are all executed in the same worker as kafka spout? If so, What are the groupings and parallellism-hint that you have configured on those bolts?
Thanks,
Satish.
On Thu, Jun 9, 2016 at 10:41 PM, Kazansky, Michael <mi...@jpmchase.com>> wrote:
Hi All,
We have 3 servers in storm cluster. To prevent the order of the messages which come to Kafka the Kafka partition was set to 1. KafkaSpout parallelism hint is set to 1 as well. When messages flow through the Storm we noticed that only one server in the cluster is busy processing messages other 2 are idle although the number of workers is set to 3 in Config. Is it excepted behavior or something is missing?
If it is expected behavior is there any way to have all 3 servers in the cluster processing messages?
Storm version is 0.9.5
Thanks,
Michael Kazansky
This communication is for informational purposes only. It is not intended as an offer or solicitation for the purchase or sale of any financial instrument or as an official confirmation of any transaction. All market prices, data and other information are not warranted as to completeness or accuracy and are subject to change without notice. Any comments or statements made herein do not necessarily reflect those of JPMorgan Chase & Co., its subsidiaries and affiliates (collectively, "JPMC"). This transmission may contain information that is proprietary, privileged, confidential and/or exempt from disclosure under applicable law. If you are not the intended recipient, you are hereby notified that any disclosure, copying, distribution, or use of the information contained herein (including any reliance thereon) is STRICTLY PROHIBITED. If you received this transmission in error, please immediately contact the sender and destroy the material in its entirety, whether in electronic or hard copy format. Although this transmission and any attachments are believed to be free of any virus or other defect that might affect any computer system into which it is received and opened, it is the responsibility of the recipient to ensure that it is virus free and no responsibility is accepted by JPMC for any loss or damage arising in any way from its use. Please note that any electronic communication that is conducted within or through JPMC's systems is subject to interception, monitoring, review, retention and external production in accordance with JPMC's policy and local laws, rules and regulations; may be stored or otherwise processed in countries other than the country in which you are located; and will be treated in accordance with JPMC policies and applicable laws and regulations. Please refer to http://www.jpmorgan.com/pages/disclosures for disclosures relating to European legal entities.
This communication is for informational purposes only. It is not intended as an offer or solicitation for the purchase or sale of any financial instrument or as an official confirmation of any transaction. All market prices, data and other information are not warranted as to completeness or accuracy and are subject to change without notice. Any comments or statements made herein do not necessarily reflect those of JPMorgan Chase & Co., its subsidiaries and affiliates (collectively, "JPMC"). This transmission may contain information that is proprietary, privileged, confidential and/or exempt from disclosure under applicable law. If you are not the intended recipient, you are hereby notified that any disclosure, copying, distribution, or use of the information contained herein (including any reliance thereon) is STRICTLY PROHIBITED. If you received this transmission in error, please immediately contact the sender and destroy the material in its entirety, whether in electronic or hard copy format. Although this transmission and any attachments are believed to be free of any virus or other defect that might affect any computer system into which it is received and opened, it is the responsibility of the recipient to ensure that it is virus free and no responsibility is accepted by JPMC for any loss or damage arising in any way from its use. Please note that any electronic communication that is conducted within or through JPMC's systems is subject to interception, monitoring, review, retention and external production in accordance with JPMC's policy and local laws, rules and regulations; may be stored or otherwise processed in countries other than the country in which you are located; and will be treated in accordance with JPMC policies and applicable laws and regulations. Please refer to http://www.jpmorgan.com/pages/disclosures for disclosures relating to European legal entities.
This communication is for informational purposes only. It is not intended as an offer or solicitation for the purchase or sale of any financial instrument or as an official confirmation of any transaction. All market prices, data and other information are not warranted as to completeness or accuracy and are subject to change without notice. Any comments or statements made herein do not necessarily reflect those of JPMorgan Chase & Co., its subsidiaries and affiliates (collectively, "JPMC").
This transmission may contain information that is proprietary, privileged, confidential and/or exempt from disclosure under applicable law. If you are not the intended recipient, you are hereby notified that any disclosure, copying, distribution, or use of the information contained herein (including any reliance thereon) is STRICTLY PROHIBITED. If you received this transmission in error, please immediately contact the sender and destroy the material in its entirety, whether in electronic or hard copy format. Although this transmission and any attachments are believed to be free of any virus or other defect that might affect any computer system into which it is received and opened, it is the responsibility of the recipient to ensure that it is virus free and no responsibility is accepted by JPMC for any loss or damage arising in any way from its use. Please note that any electronic communication that is conducted within or through JPMC's systems is subject to interception, monitoring, review, retention and external production in accordance with JPMC's policy and local laws, rules and regulations; may be stored or otherwise processed in countries other than the country in which you are located; and will be treated in accordance with JPMC policies and applicable laws and regulations.
Please refer to http://www.jpmorgan.com/pages/disclosures for disclosures relating to European legal entities.
Re: not all the servers in storm cluster processing messages
Posted by Nathan Leung <nc...@gmail.com>.
If you use localOrShuffleGrouping, the tuple will stay in process if
possible. The solution, assuming your bolts are distributed, is to use
shuffleGrouping to send tuples from spout->bolt so that the tuples are
distributed to all of your workers, then use localOrShuffleGrouping from
bolt->bolt to reduce inter-worker communication overhead.
On Thu, Jun 9, 2016 at 3:00 PM, Kazansky, Michael <
michael.kazansky@jpmchase.com> wrote:
> We are using localOrShuffleGrouping
>
>
>
> Thanks,
>
> Michael Kazansky
>
>
>
> *From:* Satish Duggana [mailto:satish.duggana@gmail.com]
> *Sent:* Thursday, June 09, 2016 1:54 PM
> *To:* user@storm.apache.org
> *Subject:* Re: not all the servers in storm cluster processing messages
>
>
>
> Hi Michael,
>
> Kafka spout is given parallelism-hint as 1 and the topic has only one
> partition. So, only one kafka spout task is run in one of the workers of
> the topology. Are you asking that bolts connected directly/indirectly from
> that kafka spout are all executed in the same worker as kafka spout? If so,
> What are the groupings and parallellism-hint that you have configured on
> those bolts?
>
>
>
> Thanks,
>
> Satish.
>
>
>
>
>
> On Thu, Jun 9, 2016 at 10:41 PM, Kazansky, Michael <
> michael.kazansky@jpmchase.com> wrote:
>
> Hi All,
>
> We have 3 servers in storm cluster. To prevent the order of the messages
> which come to Kafka the Kafka partition was set to 1. KafkaSpout
> parallelism hint is set to 1 as well. When messages flow through the Storm
> we noticed that only one server in the cluster is busy processing messages
> other 2 are idle although the number of workers is set to 3 in Config. Is
> it excepted behavior or something is missing?
>
> If it is expected behavior is there any way to have all 3 servers in the
> cluster processing messages?
>
> Storm version is 0.9.5
>
> Thanks,
>
> Michael Kazansky
>
>
>
> This communication is for informational purposes only. It is not intended
> as an offer or solicitation for the purchase or sale of any financial
> instrument or as an official confirmation of any transaction. All market
> prices, data and other information are not warranted as to completeness or
> accuracy and are subject to change without notice. Any comments or
> statements made herein do not necessarily reflect those of JPMorgan Chase &
> Co., its subsidiaries and affiliates (collectively, "JPMC"). This
> transmission may contain information that is proprietary, privileged,
> confidential and/or exempt from disclosure under applicable law. If you are
> not the intended recipient, you are hereby notified that any disclosure,
> copying, distribution, or use of the information contained herein
> (including any reliance thereon) is STRICTLY PROHIBITED. If you received
> this transmission in error, please immediately contact the sender and
> destroy the material in its entirety, whether in electronic or hard copy
> format. Although this transmission and any attachments are believed to be
> free of any virus or other defect that might affect any computer system
> into which it is received and opened, it is the responsibility of the
> recipient to ensure that it is virus free and no responsibility is accepted
> by JPMC for any loss or damage arising in any way from its use. Please note
> that any electronic communication that is conducted within or through
> JPMC's systems is subject to interception, monitoring, review, retention
> and external production in accordance with JPMC's policy and local laws,
> rules and regulations; may be stored or otherwise processed in countries
> other than the country in which you are located; and will be treated in
> accordance with JPMC policies and applicable laws and regulations. Please
> refer to http://www.jpmorgan.com/pages/disclosures for disclosures
> relating to European legal entities.
>
>
>
> This communication is for informational purposes only. It is not intended
> as an offer or solicitation for the purchase or sale of any financial
> instrument or as an official confirmation of any transaction. All market
> prices, data and other information are not warranted as to completeness or
> accuracy and are subject to change without notice. Any comments or
> statements made herein do not necessarily reflect those of JPMorgan Chase &
> Co., its subsidiaries and affiliates (collectively, "JPMC"). This
> transmission may contain information that is proprietary, privileged,
> confidential and/or exempt from disclosure under applicable law. If you are
> not the intended recipient, you are hereby notified that any disclosure,
> copying, distribution, or use of the information contained herein
> (including any reliance thereon) is STRICTLY PROHIBITED. If you received
> this transmission in error, please immediately contact the sender and
> destroy the material in its entirety, whether in electronic or hard copy
> format. Although this transmission and any attachments are believed to be
> free of any virus or other defect that might affect any computer system
> into which it is received and opened, it is the responsibility of the
> recipient to ensure that it is virus free and no responsibility is accepted
> by JPMC for any loss or damage arising in any way from its use. Please note
> that any electronic communication that is conducted within or through
> JPMC's systems is subject to interception, monitoring, review, retention
> and external production in accordance with JPMC's policy and local laws,
> rules and regulations; may be stored or otherwise processed in countries
> other than the country in which you are located; and will be treated in
> accordance with JPMC policies and applicable laws and regulations. Please
> refer to http://www.jpmorgan.com/pages/disclosures for disclosures
> relating to European legal entities.
>
RE: not all the servers in storm cluster processing messages
Posted by "Kazansky, Michael" <mi...@jpmchase.com>.
We are using localOrShuffleGrouping
Thanks,
Michael Kazansky
From: Satish Duggana [mailto:satish.duggana@gmail.com]
Sent: Thursday, June 09, 2016 1:54 PM
To: user@storm.apache.org
Subject: Re: not all the servers in storm cluster processing messages
Hi Michael,
Kafka spout is given parallelism-hint as 1 and the topic has only one partition. So, only one kafka spout task is run in one of the workers of the topology. Are you asking that bolts connected directly/indirectly from that kafka spout are all executed in the same worker as kafka spout? If so, What are the groupings and parallellism-hint that you have configured on those bolts?
Thanks,
Satish.
On Thu, Jun 9, 2016 at 10:41 PM, Kazansky, Michael <mi...@jpmchase.com>> wrote:
Hi All,
We have 3 servers in storm cluster. To prevent the order of the messages which come to Kafka the Kafka partition was set to 1. KafkaSpout parallelism hint is set to 1 as well. When messages flow through the Storm we noticed that only one server in the cluster is busy processing messages other 2 are idle although the number of workers is set to 3 in Config. Is it excepted behavior or something is missing?
If it is expected behavior is there any way to have all 3 servers in the cluster processing messages?
Storm version is 0.9.5
Thanks,
Michael Kazansky
This communication is for informational purposes only. It is not intended as an offer or solicitation for the purchase or sale of any financial instrument or as an official confirmation of any transaction. All market prices, data and other information are not warranted as to completeness or accuracy and are subject to change without notice. Any comments or statements made herein do not necessarily reflect those of JPMorgan Chase & Co., its subsidiaries and affiliates (collectively, "JPMC"). This transmission may contain information that is proprietary, privileged, confidential and/or exempt from disclosure under applicable law. If you are not the intended recipient, you are hereby notified that any disclosure, copying, distribution, or use of the information contained herein (including any reliance thereon) is STRICTLY PROHIBITED. If you received this transmission in error, please immediately contact the sender and destroy the material in its entirety, whether in electronic or hard copy format. Although this transmission and any attachments are believed to be free of any virus or other defect that might affect any computer system into which it is received and opened, it is the responsibility of the recipient to ensure that it is virus free and no responsibility is accepted by JPMC for any loss or damage arising in any way from its use. Please note that any electronic communication that is conducted within or through JPMC's systems is subject to interception, monitoring, review, retention and external production in accordance with JPMC's policy and local laws, rules and regulations; may be stored or otherwise processed in countries other than the country in which you are located; and will be treated in accordance with JPMC policies and applicable laws and regulations. Please refer to http://www.jpmorgan.com/pages/disclosures for disclosures relating to European legal entities.
This communication is for informational purposes only. It is not intended as an offer or solicitation for the purchase or sale of any financial instrument or as an official confirmation of any transaction. All market prices, data and other information are not warranted as to completeness or accuracy and are subject to change without notice. Any comments or statements made herein do not necessarily reflect those of JPMorgan Chase & Co., its subsidiaries and affiliates (collectively, "JPMC").
This transmission may contain information that is proprietary, privileged, confidential and/or exempt from disclosure under applicable law. If you are not the intended recipient, you are hereby notified that any disclosure, copying, distribution, or use of the information contained herein (including any reliance thereon) is STRICTLY PROHIBITED. If you received this transmission in error, please immediately contact the sender and destroy the material in its entirety, whether in electronic or hard copy format. Although this transmission and any attachments are believed to be free of any virus or other defect that might affect any computer system into which it is received and opened, it is the responsibility of the recipient to ensure that it is virus free and no responsibility is accepted by JPMC for any loss or damage arising in any way from its use. Please note that any electronic communication that is conducted within or through JPMC's systems is subject to interception, monitoring, review, retention and external production in accordance with JPMC's policy and local laws, rules and regulations; may be stored or otherwise processed in countries other than the country in which you are located; and will be treated in accordance with JPMC policies and applicable laws and regulations.
Please refer to http://www.jpmorgan.com/pages/disclosures for disclosures relating to European legal entities.
Re: not all the servers in storm cluster processing messages
Posted by Satish Duggana <sa...@gmail.com>.
Hi Michael,
Kafka spout is given parallelism-hint as 1 and the topic has only one
partition. So, only one kafka spout task is run in one of the workers of
the topology. Are you asking that bolts connected directly/indirectly from
that kafka spout are all executed in the same worker as kafka spout? If so,
What are the groupings and parallellism-hint that you have configured on
those bolts?
Thanks,
Satish.
On Thu, Jun 9, 2016 at 10:41 PM, Kazansky, Michael <
michael.kazansky@jpmchase.com> wrote:
> Hi All,
>
> We have 3 servers in storm cluster. To prevent the order of the messages
> which come to Kafka the Kafka partition was set to 1. KafkaSpout
> parallelism hint is set to 1 as well. When messages flow through the Storm
> we noticed that only one server in the cluster is busy processing messages
> other 2 are idle although the number of workers is set to 3 in Config. Is
> it excepted behavior or something is missing?
>
> If it is expected behavior is there any way to have all 3 servers in the
> cluster processing messages?
>
> Storm version is 0.9.5
>
> Thanks,
>
> Michael Kazansky
>
>
>
> This communication is for informational purposes only. It is not intended
> as an offer or solicitation for the purchase or sale of any financial
> instrument or as an official confirmation of any transaction. All market
> prices, data and other information are not warranted as to completeness or
> accuracy and are subject to change without notice. Any comments or
> statements made herein do not necessarily reflect those of JPMorgan Chase &
> Co., its subsidiaries and affiliates (collectively, "JPMC"). This
> transmission may contain information that is proprietary, privileged,
> confidential and/or exempt from disclosure under applicable law. If you are
> not the intended recipient, you are hereby notified that any disclosure,
> copying, distribution, or use of the information contained herein
> (including any reliance thereon) is STRICTLY PROHIBITED. If you received
> this transmission in error, please immediately contact the sender and
> destroy the material in its entirety, whether in electronic or hard copy
> format. Although this transmission and any attachments are believed to be
> free of any virus or other defect that might affect any computer system
> into which it is received and opened, it is the responsibility of the
> recipient to ensure that it is virus free and no responsibility is accepted
> by JPMC for any loss or damage arising in any way from its use. Please note
> that any electronic communication that is conducted within or through
> JPMC's systems is subject to interception, monitoring, review, retention
> and external production in accordance with JPMC's policy and local laws,
> rules and regulations; may be stored or otherwise processed in countries
> other than the country in which you are located; and will be treated in
> accordance with JPMC policies and applicable laws and regulations. Please
> refer to http://www.jpmorgan.com/pages/disclosures for disclosures
> relating to European legal entities.
>