You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@kafka.apache.org by Nilesh Chhapru <ni...@ugamsolutions.com> on 2014/12/18 08:53:35 UTC
Issues With Parallelism In Kafka Spout
Hi All,
I have implemented a high level Kafka consumer in Storm but looks like the parallelism isn't getting achieved as I have 3 partitions and 2 task for the spout, but only one of it is emitting the data.
PFB the screen grab for number of task of spout and data emitted by only one of them.
Please assist on how to achieve parallelism using high level Kafka spout.
[cid:image001.png@01D01ABD.FA56FB90]
Regards,
Nilesh Chhapru.
________________________________
---------------------------------------------------------------------------------------Disclaimer----------------------------------------------------------------------------------------------
****Opinions expressed in this e-mail are those of the author and do not necessarily represent those of Ugam. Ugam does not accept any responsibility or liability for it. This e-mail message may contain proprietary, confidential or legally privileged information for the sole use of the person or entity to whom this message was originally addressed. Any review, re-transmission, dissemination or other use of or taking of any action in reliance upon this information by persons or entities other than the intended recipient is prohibited. If you have received this e-mail in error, please delete it and all attachments from any servers, hard drives or any other media.
Warning: Sufficient measures have been taken to scan any presence of viruses however the recipient should check this email and any attachments for the presence of viruses. Ugam accepts no liability for any damage caused by any virus transmitted by this email. ****
Re: Issues With Parallelism In Kafka Spout
Posted by Jun Rao <ju...@confluent.io>.
Perhaps you can try getting some help from the Storm mailing list.
Thanks,
Jun
On Thu, Dec 18, 2014 at 7:22 AM, Nilesh Chhapru <
nilesh.chhapru@ugamsolutions.com> wrote:
> Hi All,
>
> Please give some inputs as this is pending since long and need to meet the
> deadlines
>
> Regards,
> Nilesh Chhapru.
>
> From: Nilesh Chhapru [mailto:nilesh.chhapru@ugamsolutions.com]
> Sent: 18 December 2014 01:24 PM
> To: user@storm.apache.org; users@kafka.apache.org
> Subject: Issues With Parallelism In Kafka Spout
>
> Hi All,
>
> I have implemented a high level Kafka consumer in Storm but looks like the
> parallelism isn't getting achieved as I have 3 partitions and 2 task for
> the spout, but only one of it is emitting the data.
> PFB the screen grab for number of task of spout and data emitted by only
> one of them.
>
> Please assist on how to achieve parallelism using high level Kafka spout.
>
> [cid:image001.png@01D01ABD.FA56FB90]
>
> Regards,
> Nilesh Chhapru.
>
>
> ________________________________
>
> ---------------------------------------------------------------------------------------Disclaimer----------------------------------------------------------------------------------------------
>
> ****Opinions expressed in this e-mail are those of the author and do not
> necessarily represent those of Ugam. Ugam does not accept any
> responsibility or liability for it. This e-mail message may contain
> proprietary, confidential or legally privileged information for the sole
> use of the person or entity to whom this message was originally addressed.
> Any review, re-transmission, dissemination or other use of or taking of any
> action in reliance upon this information by persons or entities other than
> the intended recipient is prohibited. If you have received this e-mail in
> error, please delete it and all attachments from any servers, hard drives
> or any other media.
>
> Warning: Sufficient measures have been taken to scan any presence of
> viruses however the recipient should check this email and any attachments
> for the presence of viruses. Ugam accepts no liability for any damage
> caused by any virus transmitted by this email. ****
>
> ________________________________
>
> ---------------------------------------------------------------------------------------Disclaimer----------------------------------------------------------------------------------------------
>
> ****Opinions expressed in this e-mail are those of the author and do not
> necessarily represent those of Ugam. Ugam does not accept any
> responsibility or liability for it. This e-mail message may contain
> proprietary, confidential or legally privileged information for the sole
> use of the person or entity to whom this message was originally addressed.
> Any review, re-transmission, dissemination or other use of or taking of any
> action in reliance upon this information by persons or entities other than
> the intended recipient is prohibited. If you have received this e-mail in
> error, please delete it and all attachments from any servers, hard drives
> or any other media.
>
> Warning: Sufficient measures have been taken to scan any presence of
> viruses however the recipient should check this email and any attachments
> for the presence of viruses. Ugam accepts no liability for any damage
> caused by any virus transmitted by this email. ****
>
RE: Issues With Parallelism In Kafka Spout
Posted by Nilesh Chhapru <ni...@ugamsolutions.com>.
Hi All / Harsha,
PFB the link for git.
https://github.com/nchhapru/storm-kafka-consumer/tree/master/src/main/java/com/ugam/crawler/core/consumer
Also there is the caller program in with method : setHighLevelKafkaSpout
https://github.com/nchhapru/storm-kafka-consumer/blob/master/src/main/java/com/ugam/crawler/core/topology/PlatformTopology.java
Regards,
Nilesh Chhapru.
From: Harsha [mailto:storm@harsha.io]
Sent: 18 December 2014 10:34 PM
To: user@storm.apache.org<ma...@storm.apache.org>
Subject: Re: Issues With Parallelism In Kafka Spout
Its going to be difficult without looking your code. If its on github do give us a link to it.
-Harsha
On Thu, Dec 18, 2014, at 07:47 AM, Nilesh Chhapru wrote:
Hi Harsha,
The partitions and topics are having the data and the zookeeper also shows the group id of all the consumer in its logs.
Regards,
Nilesh Chhapru.
From: Harsha [mailto:storm@harsha.io]
Sent: 18 December 2014 09:13 PM
To: user@storm.apache.org<ma...@storm.apache.org>
Subject: Re: Issues With Parallelism In Kafka Spout
Hi Nilesh,
Did you check if your kafka topic partitions all being writtten to.
-Harsha
On Thu, Dec 18, 2014, at 07:22 AM, Nilesh Chhapru wrote:
Hi All,
Please give some inputs as this is pending since long and need to meet the deadlines
Regards,
Nilesh Chhapru.
From: Nilesh Chhapru [mailto:nilesh.chhapru@ugamsolutions.com]
Sent: 18 December 2014 01:24 PM
To: user@storm.apache.org<ma...@storm.apache.org>; users@kafka.apache.org<ma...@kafka.apache.org>
Subject: Issues With Parallelism In Kafka Spout
Hi All,
I have implemented a high level Kafka consumer in Storm but looks like the parallelism isn’t getting achieved as I have 3 partitions and 2 task for the spout, but only one of it is emitting the data.
PFB the screen grab for number of task of spout and data emitted by only one of them.
Please assist on how to achieve parallelism using high level Kafka spout.
Regards,
Nilesh Chhapru.
________________________________
---------------------------------------------------------------------------------------Disclaimer----------------------------------------------------------------------------------------------
****Opinions expressed in this e-mail are those of the author and do not necessarily represent those of Ugam. Ugam does not accept any responsibility or liability for it. This e-mail message may contain proprietary, confidential or legally privileged information for the sole use of the person or entity to whom this message was originally addressed. Any review, re-transmission, dissemination or other use of or taking of any action in reliance upon this information by persons or entities other than the intended recipient is prohibited. If you have received this e-mail in error, please delete it and all attachments from any servers, hard drives or any other media.
Warning: Sufficient measures have been taken to scan any presence of viruses however the recipient should check this email and any attachments for the presence of viruses. Ugam accepts no liability for any damage caused by any virus transmitted by this email. ****
________________________________
---------------------------------------------------------------------------------------Disclaimer----------------------------------------------------------------------------------------------
****Opinions expressed in this e-mail are those of the author and do not necessarily represent those of Ugam. Ugam does not accept any responsibility or liability for it. This e-mail message may contain proprietary, confidential or legally privileged information for the sole use of the person or entity to whom this message was originally addressed. Any review, re-transmission, dissemination or other use of or taking of any action in reliance upon this information by persons or entities other than the intended recipient is prohibited. If you have received this e-mail in error, please delete it and all attachments from any servers, hard drives or any other media.
Warning: Sufficient measures have been taken to scan any presence of viruses however the recipient should check this email and any attachments for the presence of viruses. Ugam accepts no liability for any damage caused by any virus transmitted by this email. ****
________________________________
---------------------------------------------------------------------------------------Disclaimer----------------------------------------------------------------------------------------------
****Opinions expressed in this e-mail are those of the author and do not necessarily represent those of Ugam. Ugam does not accept any responsibility or liability for it. This e-mail message may contain proprietary, confidential or legally privileged information for the sole use of the person or entity to whom this message was originally addressed. Any review, re-transmission, dissemination or other use of or taking of any action in reliance upon this information by persons or entities other than the intended recipient is prohibited. If you have received this e-mail in error, please delete it and all attachments from any servers, hard drives or any other media.
Warning: Sufficient measures have been taken to scan any presence of viruses however the recipient should check this email and any attachments for the presence of viruses. Ugam accepts no liability for any damage caused by any virus transmitted by this email. ****
________________________________
---------------------------------------------------------------------------------------Disclaimer----------------------------------------------------------------------------------------------
****Opinions expressed in this e-mail are those of the author and do not necessarily represent those of Ugam. Ugam does not accept any responsibility or liability for it. This e-mail message may contain proprietary, confidential or legally privileged information for the sole use of the person or entity to whom this message was originally addressed. Any review, re-transmission, dissemination or other use of or taking of any action in reliance upon this information by persons or entities other than the intended recipient is prohibited. If you have received this e-mail in error, please delete it and all attachments from any servers, hard drives or any other media.
Warning: Sufficient measures have been taken to scan any presence of viruses however the recipient should check this email and any attachments for the presence of viruses. Ugam accepts no liability for any damage caused by any virus transmitted by this email. ****
________________________________
---------------------------------------------------------------------------------------Disclaimer----------------------------------------------------------------------------------------------
****Opinions expressed in this e-mail are those of the author and do not necessarily represent those of Ugam. Ugam does not accept any responsibility or liability for it. This e-mail message may contain proprietary, confidential or legally privileged information for the sole use of the person or entity to whom this message was originally addressed. Any review, re-transmission, dissemination or other use of or taking of any action in reliance upon this information by persons or entities other than the intended recipient is prohibited. If you have received this e-mail in error, please delete it and all attachments from any servers, hard drives or any other media.
Warning: Sufficient measures have been taken to scan any presence of viruses however the recipient should check this email and any attachments for the presence of viruses. Ugam accepts no liability for any damage caused by any virus transmitted by this email. ****
RE: Issues With Parallelism In Kafka Spout
Posted by Nilesh Chhapru <ni...@ugamsolutions.com>.
Hi Harsha,
PFB the link for git.
https://github.com/nchhapru/storm-kafka-consumer/tree/master/src/main/java/com/ugam/crawler/core/consumer
Also there is the caller program in with method : setHighLevelKafkaSpout
https://github.com/nchhapru/storm-kafka-consumer/blob/master/src/main/java/com/ugam/crawler/core/topology/PlatformTopology.java
Regards,
Nilesh Chhapru.
From: Harsha [mailto:storm@harsha.io]
Sent: 18 December 2014 10:34 PM
To: user@storm.apache.org
Subject: Re: Issues With Parallelism In Kafka Spout
Its going to be difficult without looking your code. If its on github do give us a link to it.
-Harsha
On Thu, Dec 18, 2014, at 07:47 AM, Nilesh Chhapru wrote:
Hi Harsha,
The partitions and topics are having the data and the zookeeper also shows the group id of all the consumer in its logs.
Regards,
Nilesh Chhapru.
From: Harsha [mailto:storm@harsha.io]
Sent: 18 December 2014 09:13 PM
To: user@storm.apache.org<ma...@storm.apache.org>
Subject: Re: Issues With Parallelism In Kafka Spout
Hi Nilesh,
Did you check if your kafka topic partitions all being writtten to.
-Harsha
On Thu, Dec 18, 2014, at 07:22 AM, Nilesh Chhapru wrote:
Hi All,
Please give some inputs as this is pending since long and need to meet the deadlines
Regards,
Nilesh Chhapru.
From: Nilesh Chhapru [mailto:nilesh.chhapru@ugamsolutions.com]
Sent: 18 December 2014 01:24 PM
To: user@storm.apache.org<ma...@storm.apache.org>; users@kafka.apache.org<ma...@kafka.apache.org>
Subject: Issues With Parallelism In Kafka Spout
Hi All,
I have implemented a high level Kafka consumer in Storm but looks like the parallelism isn’t getting achieved as I have 3 partitions and 2 task for the spout, but only one of it is emitting the data.
PFB the screen grab for number of task of spout and data emitted by only one of them.
Please assist on how to achieve parallelism using high level Kafka spout.
Regards,
Nilesh Chhapru.
________________________________
---------------------------------------------------------------------------------------Disclaimer----------------------------------------------------------------------------------------------
****Opinions expressed in this e-mail are those of the author and do not necessarily represent those of Ugam. Ugam does not accept any responsibility or liability for it. This e-mail message may contain proprietary, confidential or legally privileged information for the sole use of the person or entity to whom this message was originally addressed. Any review, re-transmission, dissemination or other use of or taking of any action in reliance upon this information by persons or entities other than the intended recipient is prohibited. If you have received this e-mail in error, please delete it and all attachments from any servers, hard drives or any other media.
Warning: Sufficient measures have been taken to scan any presence of viruses however the recipient should check this email and any attachments for the presence of viruses. Ugam accepts no liability for any damage caused by any virus transmitted by this email. ****
________________________________
---------------------------------------------------------------------------------------Disclaimer----------------------------------------------------------------------------------------------
****Opinions expressed in this e-mail are those of the author and do not necessarily represent those of Ugam. Ugam does not accept any responsibility or liability for it. This e-mail message may contain proprietary, confidential or legally privileged information for the sole use of the person or entity to whom this message was originally addressed. Any review, re-transmission, dissemination or other use of or taking of any action in reliance upon this information by persons or entities other than the intended recipient is prohibited. If you have received this e-mail in error, please delete it and all attachments from any servers, hard drives or any other media.
Warning: Sufficient measures have been taken to scan any presence of viruses however the recipient should check this email and any attachments for the presence of viruses. Ugam accepts no liability for any damage caused by any virus transmitted by this email. ****
________________________________
---------------------------------------------------------------------------------------Disclaimer----------------------------------------------------------------------------------------------
****Opinions expressed in this e-mail are those of the author and do not necessarily represent those of Ugam. Ugam does not accept any responsibility or liability for it. This e-mail message may contain proprietary, confidential or legally privileged information for the sole use of the person or entity to whom this message was originally addressed. Any review, re-transmission, dissemination or other use of or taking of any action in reliance upon this information by persons or entities other than the intended recipient is prohibited. If you have received this e-mail in error, please delete it and all attachments from any servers, hard drives or any other media.
Warning: Sufficient measures have been taken to scan any presence of viruses however the recipient should check this email and any attachments for the presence of viruses. Ugam accepts no liability for any damage caused by any virus transmitted by this email. ****
________________________________
---------------------------------------------------------------------------------------Disclaimer----------------------------------------------------------------------------------------------
****Opinions expressed in this e-mail are those of the author and do not necessarily represent those of Ugam. Ugam does not accept any responsibility or liability for it. This e-mail message may contain proprietary, confidential or legally privileged information for the sole use of the person or entity to whom this message was originally addressed. Any review, re-transmission, dissemination or other use of or taking of any action in reliance upon this information by persons or entities other than the intended recipient is prohibited. If you have received this e-mail in error, please delete it and all attachments from any servers, hard drives or any other media.
Warning: Sufficient measures have been taken to scan any presence of viruses however the recipient should check this email and any attachments for the presence of viruses. Ugam accepts no liability for any damage caused by any virus transmitted by this email. ****
Re: Issues With Parallelism In Kafka Spout
Posted by Andrew Neilson <ar...@gmail.com>.
Harsha asked a similar question, but I would definitely make certain that
messages being written to kafka are being partitioned the way you are
expecting. If you are trying to consume live events coming from a lone
kafka producer using default configuration, then messages are only going to
appear in one partition at a time.
See the Kafka FAQ for details
https://cwiki.apache.org/confluence/display/KAFKA/FAQ#FAQ-Whyisdatanotevenlydistributedamongpartitionswhenapartitioningkeyisnotspecified
?
On Tue, Dec 23, 2014 at 10:22 AM, Nilesh Chhapru <
nilesh.chhapru@ugamsolutions.com> wrote:
> Hi Harsha,
>
>
>
> PFB the link for git.
>
>
>
>
> https://github.com/nchhapru/storm-kafka-consumer/tree/master/src/main/java/com/ugam/crawler/core/consumer
>
>
>
> Also there is the caller program in with method : setHighLevelKafkaSpout
>
>
> https://github.com/nchhapru/storm-kafka-consumer/blob/master/src/main/java/com/ugam/crawler/core/topology/PlatformTopology.java
>
>
>
>
>
> *Regards*,
>
> *Nilesh Chhapru.*
>
>
>
> *From:* Harsha [mailto:storm@harsha.io <st...@harsha.io>]
> *Sent:* 18 December 2014 10:34 PM
>
> *To:* user@storm.apache.org
> *Subject:* Re: Issues With Parallelism In Kafka Spout
>
>
>
> Its going to be difficult without looking your code. If its on github do
> give us a link to it.
>
> -Harsha
>
>
>
>
>
> On Thu, Dec 18, 2014, at 07:47 AM, Nilesh Chhapru wrote:
>
> Hi Harsha,
>
>
>
> The partitions and topics are having the data and the zookeeper also shows
> the group id of all the consumer in its logs.
>
>
>
> *Regards*,
>
> *Nilesh Chhapru.*
>
>
>
> *From:* Harsha [mailto:storm@harsha.io <st...@harsha.io>]
> *Sent:* 18 December 2014 09:13 PM
> *To:* user@storm.apache.org
> *Subject:* Re: Issues With Parallelism In Kafka Spout
>
>
>
> Hi Nilesh,
>
> Did you check if your kafka topic partitions all being
> writtten to.
>
> -Harsha
>
>
>
>
>
> On Thu, Dec 18, 2014, at 07:22 AM, Nilesh Chhapru wrote:
>
> Hi All,
>
>
>
> Please give some inputs as this is pending since long and need to meet the
> deadlines
>
>
>
> *Regards*,
>
> *Nilesh Chhapru.*
>
>
>
> *From:* Nilesh Chhapru [mailto:nilesh.chhapru@ugamsolutions.com
> <ni...@ugamsolutions.com>]
> *Sent:* 18 December 2014 01:24 PM
> *To:* user@storm.apache.org; users@kafka.apache.org
> *Subject:* Issues With Parallelism In Kafka Spout
>
>
>
> Hi All,
>
>
>
> I have implemented a high level Kafka consumer in Storm but looks like the
> parallelism isn’t getting achieved as I have 3 partitions and 2 task for
> the spout, but only one of it is emitting the data.
>
> PFB the screen grab for number of task of spout and data emitted by only
> one of them.
>
>
>
> Please assist on how to achieve parallelism using high level Kafka spout.
>
>
>
>
>
>
>
> *Regards*,
>
> *Nilesh Chhapru.*
>
>
>
>
> ------------------------------
>
>
>
>
>
>
> ---------------------------------------------------------------------------------------Disclaimer----------------------------------------------------------------------------------------------
>
> ****Opinions expressed in this e-mail are those of the author and do not
> necessarily represent those of Ugam. Ugam does not accept any
> responsibility or liability for it. This e-mail message may contain
> proprietary, confidential or legally privileged information for the sole
> use of the person or entity to whom this message was originally addressed.
> Any review, re-transmission, dissemination or other use of or taking of any
> action in reliance upon this information by persons or entities other than
> the intended recipient is prohibited. If you have received this e-mail in
> error, please delete it and all attachments from any servers, hard drives
> or any other media.
>
> Warning: Sufficient measures have been taken to scan any presence of
> viruses however the recipient should check this email and any attachments
> for the presence of viruses. Ugam accepts no liability for any damage
> caused by any virus transmitted by this email. ****
>
>
> ------------------------------
>
>
>
>
> ---------------------------------------------------------------------------------------Disclaimer----------------------------------------------------------------------------------------------
>
>
>
> ****Opinions expressed in this e-mail are those of the author and do not
> necessarily represent those of Ugam. Ugam does not accept any
> responsibility or liability for it. This e-mail message may contain
> proprietary, confidential or legally privileged information for the sole
> use of the person or entity to whom this message was originally addressed.
> Any review, re-transmission, dissemination or other use of or taking of any
> action in reliance upon this information by persons or entities other than
> the intended recipient is prohibited. If you have received this e-mail in
> error, please delete it and all attachments from any servers, hard drives
> or any other media.
>
>
>
> Warning: Sufficient measures have been taken to scan any presence of
> viruses however the recipient should check this email and any attachments
> for the presence of viruses. Ugam accepts no liability for any damage
> caused by any virus transmitted by this email. ****
>
>
>
>
> ------------------------------
>
>
> ---------------------------------------------------------------------------------------Disclaimer----------------------------------------------------------------------------------------------
>
>
>
> ****Opinions expressed in this e-mail are those of the author and do not
> necessarily represent those of Ugam. Ugam does not accept any
> responsibility or liability for it. This e-mail message may contain
> proprietary, confidential or legally privileged information for the sole
> use of the person or entity to whom this message was originally addressed.
> Any review, re-transmission, dissemination or other use of or taking of any
> action in reliance upon this information by persons or entities other than
> the intended recipient is prohibited. If you have received this e-mail in
> error, please delete it and all attachments from any servers, hard drives
> or any other media.
>
>
>
> Warning: Sufficient measures have been taken to scan any presence of
> viruses however the recipient should check this email and any attachments
> for the presence of viruses. Ugam accepts no liability for any damage
> caused by any virus transmitted by this email. ****
>
>
>
> ------------------------------
>
> ---------------------------------------------------------------------------------------Disclaimer----------------------------------------------------------------------------------------------
>
> ****Opinions expressed in this e-mail are those of the author and do not
> necessarily represent those of Ugam. Ugam does not accept any
> responsibility or liability for it. This e-mail message may contain
> proprietary, confidential or legally privileged information for the sole
> use of the person or entity to whom this message was originally addressed.
> Any review, re-transmission, dissemination or other use of or taking of any
> action in reliance upon this information by persons or entities other than
> the intended recipient is prohibited. If you have received this e-mail in
> error, please delete it and all attachments from any servers, hard drives
> or any other media.
>
> Warning: Sufficient measures have been taken to scan any presence of
> viruses however the recipient should check this email and any attachments
> for the presence of viruses. Ugam accepts no liability for any damage
> caused by any virus transmitted by this email. ****
>
RE: Issues With Parallelism In Kafka Spout
Posted by Nilesh Chhapru <ni...@ugamsolutions.com>.
Hi Harsha,
In the code shared, I’m ending the loop by giving a size criteria in the nextTuple(), which is very similar to HomeNL code.
Regards,
Nilesh Chhapru.
From: Harsha [mailto:storm@harsha.io]
Sent: 24 December 2014 12:37 AM
To: user@storm.apache.org
Subject: Re: Issues With Parallelism In Kafka Spout
Nilesh,
I recommend you to take a look at this code https://github.com/HolmesNL/kafka-spout/blob/develop/src/main/java/nl/minvenj/nfi/storm/kafka/KafkaSpout.java . It uses high level consumer api.
In your code , in nextTuple() you just emit all the tuples from your topic. This never ends until your topic is exhausted. My suspicion is when your spout initializes and one of the instances will just go through your topic since nextTuple never ends ( until your topic is exhausted).
https://github.com/nchhapru/storm-kafka-consumer/blob/master/src/main/java/com/ugam/crawler/core/consumer/StormKafkaConsumer.java#L71
This is handled properly in HolmesNL kafka-spout.
-Harsha.
On Tue, Dec 23, 2014, at 10:22 AM, Nilesh Chhapru wrote:
Hi Harsha,
PFB the link for git.
https://github.com/nchhapru/storm-kafka-consumer/tree/master/src/main/java/com/ugam/crawler/core/consumer
Also there is the caller program in with method : setHighLevelKafkaSpout
https://github.com/nchhapru/storm-kafka-consumer/blob/master/src/main/java/com/ugam/crawler/core/topology/PlatformTopology.java
Regards,
Nilesh Chhapru.
From: Harsha [mailto:storm@harsha.io]
Sent: 18 December 2014 10:34 PM
To: user@storm.apache.org<ma...@storm.apache.org>
Subject: Re: Issues With Parallelism In Kafka Spout
Its going to be difficult without looking your code. If its on github do give us a link to it.
-Harsha
On Thu, Dec 18, 2014, at 07:47 AM, Nilesh Chhapru wrote:
Hi Harsha,
The partitions and topics are having the data and the zookeeper also shows the group id of all the consumer in its logs.
Regards,
Nilesh Chhapru.
From: Harsha [mailto:storm@harsha.io]
Sent: 18 December 2014 09:13 PM
To: user@storm.apache.org<ma...@storm.apache.org>
Subject: Re: Issues With Parallelism In Kafka Spout
Hi Nilesh,
Did you check if your kafka topic partitions all being writtten to.
-Harsha
On Thu, Dec 18, 2014, at 07:22 AM, Nilesh Chhapru wrote:
Hi All,
Please give some inputs as this is pending since long and need to meet the deadlines
Regards,
Nilesh Chhapru.
From: Nilesh Chhapru [mailto:nilesh.chhapru@ugamsolutions.com]
Sent: 18 December 2014 01:24 PM
To: user@storm.apache.org<ma...@storm.apache.org>; users@kafka.apache.org<ma...@kafka.apache.org>
Subject: Issues With Parallelism In Kafka Spout
Hi All,
I have implemented a high level Kafka consumer in Storm but looks like the parallelism isn’t getting achieved as I have 3 partitions and 2 task for the spout, but only one of it is emitting the data.
PFB the screen grab for number of task of spout and data emitted by only one of them.
Please assist on how to achieve parallelism using high level Kafka spout.
Regards,
Nilesh Chhapru.
________________________________
---------------------------------------------------------------------------------------Disclaimer----------------------------------------------------------------------------------------------
****Opinions expressed in this e-mail are those of the author and do not necessarily represent those of Ugam. Ugam does not accept any responsibility or liability for it. This e-mail message may contain proprietary, confidential or legally privileged information for the sole use of the person or entity to whom this message was originally addressed. Any review, re-transmission, dissemination or other use of or taking of any action in reliance upon this information by persons or entities other than the intended recipient is prohibited. If you have received this e-mail in error, please delete it and all attachments from any servers, hard drives or any other media.
Warning: Sufficient measures have been taken to scan any presence of viruses however the recipient should check this email and any attachments for the presence of viruses. Ugam accepts no liability for any damage caused by any virus transmitted by this email. ****
________________________________
---------------------------------------------------------------------------------------Disclaimer----------------------------------------------------------------------------------------------
****Opinions expressed in this e-mail are those of the author and do not necessarily represent those of Ugam. Ugam does not accept any responsibility or liability for it. This e-mail message may contain proprietary, confidential or legally privileged information for the sole use of the person or entity to whom this message was originally addressed. Any review, re-transmission, dissemination or other use of or taking of any action in reliance upon this information by persons or entities other than the intended recipient is prohibited. If you have received this e-mail in error, please delete it and all attachments from any servers, hard drives or any other media.
Warning: Sufficient measures have been taken to scan any presence of viruses however the recipient should check this email and any attachments for the presence of viruses. Ugam accepts no liability for any damage caused by any virus transmitted by this email. ****
________________________________
---------------------------------------------------------------------------------------Disclaimer----------------------------------------------------------------------------------------------
****Opinions expressed in this e-mail are those of the author and do not necessarily represent those of Ugam. Ugam does not accept any responsibility or liability for it. This e-mail message may contain proprietary, confidential or legally privileged information for the sole use of the person or entity to whom this message was originally addressed. Any review, re-transmission, dissemination or other use of or taking of any action in reliance upon this information by persons or entities other than the intended recipient is prohibited. If you have received this e-mail in error, please delete it and all attachments from any servers, hard drives or any other media.
Warning: Sufficient measures have been taken to scan any presence of viruses however the recipient should check this email and any attachments for the presence of viruses. Ugam accepts no liability for any damage caused by any virus transmitted by this email. ****
________________________________
---------------------------------------------------------------------------------------Disclaimer----------------------------------------------------------------------------------------------
****Opinions expressed in this e-mail are those of the author and do not necessarily represent those of Ugam. Ugam does not accept any responsibility or liability for it. This e-mail message may contain proprietary, confidential or legally privileged information for the sole use of the person or entity to whom this message was originally addressed. Any review, re-transmission, dissemination or other use of or taking of any action in reliance upon this information by persons or entities other than the intended recipient is prohibited. If you have received this e-mail in error, please delete it and all attachments from any servers, hard drives or any other media.
Warning: Sufficient measures have been taken to scan any presence of viruses however the recipient should check this email and any attachments for the presence of viruses. Ugam accepts no liability for any damage caused by any virus transmitted by this email. ****
________________________________
---------------------------------------------------------------------------------------Disclaimer----------------------------------------------------------------------------------------------
****Opinions expressed in this e-mail are those of the author and do not necessarily represent those of Ugam. Ugam does not accept any responsibility or liability for it. This e-mail message may contain proprietary, confidential or legally privileged information for the sole use of the person or entity to whom this message was originally addressed. Any review, re-transmission, dissemination or other use of or taking of any action in reliance upon this information by persons or entities other than the intended recipient is prohibited. If you have received this e-mail in error, please delete it and all attachments from any servers, hard drives or any other media.
Warning: Sufficient measures have been taken to scan any presence of viruses however the recipient should check this email and any attachments for the presence of viruses. Ugam accepts no liability for any damage caused by any virus transmitted by this email. ****
Re: Issues With Parallelism In Kafka Spout
Posted by Harsha <st...@harsha.io>.
Nilesh, I recommend you to take a look at this code
https://github.com/HolmesNL/kafka-spout/blob/develop/src/main/java/nl/minvenj/nfi/storm/kafka/KafkaSpout.java
. It uses high level consumer api. In your code , in nextTuple() you
just emit all the tuples from your topic. This never ends until your
topic is exhausted. My suspicion is when your spout initializes and one
of the instances will just go through your topic since nextTuple never
ends ( until your topic is exhausted).
https://github.com/nchhapru/storm-kafka-consumer/blob/master/src/main/java/com/ugam/crawler/core/consumer/StormKafkaConsumer.java#L71
This is handled properly in HolmesNL kafka-spout.
-Harsha.
On Tue, Dec 23, 2014, at 10:22 AM, Nilesh Chhapru wrote:
> Hi Harsha,
>
> PFB the link for git.
>
> https://github.com/nchhapru/storm-kafka-consumer/tree/master/src/main/java/com/ugam/crawler/core/consumer
>
> Also there is the caller program in with method :
> setHighLevelKafkaSpout
> https://github.com/nchhapru/storm-kafka-consumer/blob/master/src/main/java/com/ugam/crawler/core/topology/PlatformTopology.java
>
>
> *Regards*,
> *Nilesh Chhapru.*
>
> *From:* Harsha [mailto:storm@harsha.io]
>
> *Sent:* 18 December 2014 10:34 PM *To:* user@storm.apache.org
> *Subject:* Re: Issues With Parallelism In Kafka Spout
>
> Its going to be difficult without looking your code. If its on github
> do give us a link to it.
> -Harsha
>
>
> On Thu, Dec 18, 2014, at 07:47 AM, Nilesh Chhapru wrote:
>> Hi Harsha,
>>
>> The partitions and topics are having the data and the zookeeper also
>> shows the group id of all the consumer in its
logs.
>>
>> *Regards*,
>> *Nilesh Chhapru.*
>>
>> *From:* Harsha [mailto:storm@harsha.io]
>>
>> *Sent:* 18 December 2014 09:13 PM *To:* user@storm.apache.org
>> *Subject:* Re: Issues With Parallelism In Kafka Spout
>>
>> Hi Nilesh,
>> Did you check if your kafka topic partitions all being writtten to.
>> -Harsha
>>
>>
>> On Thu, Dec 18, 2014, at 07:22 AM, Nilesh Chhapru wrote:
>>> Hi All,
>>>
>>> Please give some inputs as this is pending since long and need to
>>> meet the deadlines
>>>
>>> *Regards*,
>>> *Nilesh Chhapru.*
>>>
>>> *From:* Nilesh Chhapru [mailto:nilesh.chhapru@ugamsolutions.com]
>>>
>>> *Sent:* 18 December 2014 01:24 PM *To:* user@storm.apache.org;
>>> users@kafka.apache.org *Subject:* Issues With Parallelism In
>>> Kafka Spout
>>>
>>> Hi All,
>>>
>>> I have implemented a high level Kafka consumer in Storm but looks
>>> like the parallelism isn’t getting achieved as I have 3 partitions
and 2 task for the spout, but only one of it is emitting the data.
>>> PFB the screen grab for number of task of spout and data emitted by
>>> only one of them.
>>>
>>> Please assist on how to achieve parallelism using high level Kafka
>>> spout.
>>>
>>>
>>>
>>> *Regards*,
>>> *Nilesh Chhapru.*
>>>
>>>
>>>
>>>
>>>
>>> ---------------------------------------------------------------------------------------Disclaimer----------------------------------------------------------------------------------------------
>>>
>>> ****Opinions expressed in this e-mail are those of the author and do
>>> not necessarily represent those of Ugam. Ugam does not accept any
>>> responsibility or liability for it. This e-mail message may contain
>>> proprietary, confidential or legally
privileged information for the sole use of the person or entity to whom
this message was originally addressed. Any review, re-transmission,
dissemination or other use of or taking of any action in reliance upon
this information by persons or entities other than the intended
recipient is prohibited. If you have received this e-mail in error,
please delete it and all attachments from any servers, hard drives or
any other media.
>>>
>>> Warning: Sufficient measures have been taken to scan any presence of
>>> viruses however the recipient should check this email and any
>>> attachments for the presence of viruses. Ugam accepts no liability
>>> for any damage caused by any virus transmitted
by this email. ****
>>>
>>>
>>>
>>> ---------------------------------------------------------------------------------------Disclaimer----------------------------------------------------------------------------------------------
>>>
>>> ****Opinions expressed in this e-mail are those of the author and do
>>> not necessarily represent those of Ugam. Ugam does
not accept any responsibility or liability for it. This e-mail message
may contain proprietary, confidential or legally privileged information
for the sole use of the person or entity to whom this message was
originally addressed. Any review, re-transmission, dissemination or
other use of or taking of any action in reliance upon this information
by persons or entities other than the intended recipient is prohibited.
If you have received this e-mail in error, please delete it and all
attachments from any servers, hard drives or any other media.
>>>
>>> Warning: Sufficient measures have been taken to scan any presence of
>>> viruses however the recipient should check this
email and any attachments for the presence of viruses. Ugam accepts no
liability for any damage caused by any virus transmitted by this
email. ****
>>
>>
>>
>> ---------------------------------------------------------------------------------------Disclaimer----------------------------------------------------------------------------------------------
>>
>> ****Opinions expressed in this e-mail are those of the author and do
>> not necessarily represent those of Ugam. Ugam does not accept any
>> responsibility
or liability for it. This e-mail message may contain proprietary,
confidential or legally privileged information for the sole use of the
person or entity to whom this message was originally addressed. Any
review, re-transmission, dissemination or other use of or taking of any
action in reliance upon this information by persons or entities other
than the intended recipient is prohibited. If you have received this
e-mail in error, please delete it and all attachments from any servers,
hard drives or any other media.
>>
>> Warning: Sufficient measures have been taken to scan any presence of
>> viruses however the recipient should check this email and any
>> attachments
for the presence of viruses. Ugam accepts no liability for any damage
caused by any virus transmitted by this email. ****
>
>
> ---------------------------------------------------------------------------------------Disclaimer----------------------------------------------------------------------------------------------
>
>
****Opinions expressed in this e-mail are those of the author and do not
necessarily represent those of Ugam. Ugam does not accept any
responsibility or liability for it. This e-mail message may contain
proprietary, confidential or legally privileged information for the sole
use of the person or entity to whom this message was originally
addressed. Any review, re-transmission, dissemination or other use of or
taking of any action in reliance upon this information by persons or
entities other than the intended recipient is prohibited. If you have
received this e-mail in error, please delete it and all attachments from
any servers, hard drives or any other media.
>
>
Warning: Sufficient measures have been taken to scan any presence of
viruses however the recipient should check this email and any
attachments for the presence of viruses. Ugam accepts no liability for
any damage caused by any virus transmitted by this email.
****
RE: Issues With Parallelism In Kafka Spout
Posted by Nilesh Chhapru <ni...@ugamsolutions.com>.
Hi Harsha,
PFB the link for git.
https://github.com/nchhapru/storm-kafka-consumer/tree/master/src/main/java/com/ugam/crawler/core/consumer
Also there is the caller program in with method : setHighLevelKafkaSpout
https://github.com/nchhapru/storm-kafka-consumer/blob/master/src/main/java/com/ugam/crawler/core/topology/PlatformTopology.java
Regards,
Nilesh Chhapru.
From: Harsha [mailto:storm@harsha.io]
Sent: 18 December 2014 10:34 PM
To: user@storm.apache.org<ma...@storm.apache.org>
Subject: Re: Issues With Parallelism In Kafka Spout
Its going to be difficult without looking your code. If its on github do give us a link to it.
-Harsha
On Thu, Dec 18, 2014, at 07:47 AM, Nilesh Chhapru wrote:
Hi Harsha,
The partitions and topics are having the data and the zookeeper also shows the group id of all the consumer in its logs.
Regards,
Nilesh Chhapru.
From: Harsha [mailto:storm@harsha.io]
Sent: 18 December 2014 09:13 PM
To: user@storm.apache.org<ma...@storm.apache.org>
Subject: Re: Issues With Parallelism In Kafka Spout
Hi Nilesh,
Did you check if your kafka topic partitions all being writtten to.
-Harsha
On Thu, Dec 18, 2014, at 07:22 AM, Nilesh Chhapru wrote:
Hi All,
Please give some inputs as this is pending since long and need to meet the deadlines
Regards,
Nilesh Chhapru.
From: Nilesh Chhapru [mailto:nilesh.chhapru@ugamsolutions.com]
Sent: 18 December 2014 01:24 PM
To: user@storm.apache.org<ma...@storm.apache.org>; users@kafka.apache.org<ma...@kafka.apache.org>
Subject: Issues With Parallelism In Kafka Spout
Hi All,
I have implemented a high level Kafka consumer in Storm but looks like the parallelism isn’t getting achieved as I have 3 partitions and 2 task for the spout, but only one of it is emitting the data.
PFB the screen grab for number of task of spout and data emitted by only one of them.
Please assist on how to achieve parallelism using high level Kafka spout.
Regards,
Nilesh Chhapru.
________________________________
---------------------------------------------------------------------------------------Disclaimer----------------------------------------------------------------------------------------------
****Opinions expressed in this e-mail are those of the author and do not necessarily represent those of Ugam. Ugam does not accept any responsibility or liability for it. This e-mail message may contain proprietary, confidential or legally privileged information for the sole use of the person or entity to whom this message was originally addressed. Any review, re-transmission, dissemination or other use of or taking of any action in reliance upon this information by persons or entities other than the intended recipient is prohibited. If you have received this e-mail in error, please delete it and all attachments from any servers, hard drives or any other media.
Warning: Sufficient measures have been taken to scan any presence of viruses however the recipient should check this email and any attachments for the presence of viruses. Ugam accepts no liability for any damage caused by any virus transmitted by this email. ****
________________________________
---------------------------------------------------------------------------------------Disclaimer----------------------------------------------------------------------------------------------
****Opinions expressed in this e-mail are those of the author and do not necessarily represent those of Ugam. Ugam does not accept any responsibility or liability for it. This e-mail message may contain proprietary, confidential or legally privileged information for the sole use of the person or entity to whom this message was originally addressed. Any review, re-transmission, dissemination or other use of or taking of any action in reliance upon this information by persons or entities other than the intended recipient is prohibited. If you have received this e-mail in error, please delete it and all attachments from any servers, hard drives or any other media.
Warning: Sufficient measures have been taken to scan any presence of viruses however the recipient should check this email and any attachments for the presence of viruses. Ugam accepts no liability for any damage caused by any virus transmitted by this email. ****
________________________________
---------------------------------------------------------------------------------------Disclaimer----------------------------------------------------------------------------------------------
****Opinions expressed in this e-mail are those of the author and do not necessarily represent those of Ugam. Ugam does not accept any responsibility or liability for it. This e-mail message may contain proprietary, confidential or legally privileged information for the sole use of the person or entity to whom this message was originally addressed. Any review, re-transmission, dissemination or other use of or taking of any action in reliance upon this information by persons or entities other than the intended recipient is prohibited. If you have received this e-mail in error, please delete it and all attachments from any servers, hard drives or any other media.
Warning: Sufficient measures have been taken to scan any presence of viruses however the recipient should check this email and any attachments for the presence of viruses. Ugam accepts no liability for any damage caused by any virus transmitted by this email. ****
________________________________
---------------------------------------------------------------------------------------Disclaimer----------------------------------------------------------------------------------------------
****Opinions expressed in this e-mail are those of the author and do not necessarily represent those of Ugam. Ugam does not accept any responsibility or liability for it. This e-mail message may contain proprietary, confidential or legally privileged information for the sole use of the person or entity to whom this message was originally addressed. Any review, re-transmission, dissemination or other use of or taking of any action in reliance upon this information by persons or entities other than the intended recipient is prohibited. If you have received this e-mail in error, please delete it and all attachments from any servers, hard drives or any other media.
Warning: Sufficient measures have been taken to scan any presence of viruses however the recipient should check this email and any attachments for the presence of viruses. Ugam accepts no liability for any damage caused by any virus transmitted by this email. ****
Re: Issues With Parallelism In Kafka Spout
Posted by Harsha <st...@harsha.io>.
Its going to be difficult without looking your code. If its on github do
give us a link to it. -Harsha
On Thu, Dec 18, 2014, at 07:47 AM, Nilesh Chhapru wrote:
> Hi Harsha,
>
> The partitions and topics are having the data and the zookeeper also
> shows the group id of all the consumer in its logs.
>
> *Regards*,
> *Nilesh Chhapru.*
>
> *From:* Harsha [mailto:storm@harsha.io]
>
> *Sent:* 18 December 2014 09:13 PM *To:* user@storm.apache.org
> *Subject:* Re: Issues With Parallelism In Kafka Spout
>
> Hi Nilesh,
> Did you check if your kafka topic partitions all being writtten to.
> -Harsha
>
>
> On Thu, Dec 18, 2014, at 07:22 AM, Nilesh Chhapru wrote:
>> Hi All,
>>
>> Please give some inputs as this is pending since long and need to
>> meet the deadlines
>>
>> *Regards*,
>> *Nilesh Chhapru.*
>>
>> *From:* Nilesh Chhapru [mailto:nilesh.chhapru@ugamsolutions.com]
>>
>> *Sent:* 18 December 2014 01:24 PM *To:* user@storm.apache.org;
>> users@kafka.apache.org *Subject:* Issues With Parallelism In
>> Kafka Spout
>>
>> Hi All,
>>
>> I have implemented a high level Kafka consumer in Storm but looks
>> like the parallelism isn’t getting achieved as I have 3 partitions
>> and 2 task for
the spout, but only one of it is emitting the data.
>> PFB the screen grab for number of task of spout and data emitted by
>> only one of them.
>>
>> Please assist on how to achieve parallelism using high level
>> Kafka spout.
>>
>>
>>
>> *Regards*,
>> *Nilesh Chhapru.*
>>
>>
>>
>>
>> ---------------------------------------------------------------------------------------Disclaimer----------------------------------------------------------------------------------------------
>>
>> ****Opinions expressed in this e-mail are those of the author and do
>> not necessarily represent those of Ugam. Ugam does not accept any
>> responsibility or liability for it. This e-mail message may contain
>> proprietary, confidential or legally
privileged information for the sole use of the person or entity to whom
this message was originally addressed. Any review, re-transmission,
dissemination or other use of or taking of any action in reliance upon
this information by persons or entities other than the intended
recipient is prohibited. If you have received this e-mail in error,
please delete it and all attachments from any servers, hard drives or
any other media.
>>
>> Warning: Sufficient measures have been taken to scan any presence of
>> viruses however the recipient should check this email and any
>> attachments for the presence of viruses. Ugam accepts no liability
>> for any damage caused by any virus transmitted
by this email. ****
>>
>>
>> ---------------------------------------------------------------------------------------Disclaimer----------------------------------------------------------------------------------------------
>>
>> ****Opinions expressed in this e-mail are those of the author and do
>> not necessarily represent those of Ugam. Ugam does not accept any
>> responsibility
or liability for it. This e-mail message may contain proprietary,
confidential or legally privileged information for the sole use of the
person or entity to whom this message was originally addressed. Any
review, re-transmission, dissemination or other use of or taking of any
action in reliance upon this information by persons or entities other
than the intended recipient is prohibited. If you have received this
e-mail in error, please delete it and all attachments from any servers,
hard drives or any other media.
>>
>> Warning: Sufficient measures have been taken to scan any presence of
>> viruses however the recipient should check this email and any
>> attachments
for the presence of viruses. Ugam accepts no liability for any damage
caused by any virus transmitted by this email. ****
>
>
> ---------------------------------------------------------------------------------------Disclaimer----------------------------------------------------------------------------------------------
>
>
****Opinions expressed in this e-mail are those of the author and do not
necessarily represent those of Ugam. Ugam does not accept any
responsibility or liability for it. This e-mail message may contain
proprietary, confidential or legally privileged information for the sole
use of the person or entity to whom this message was originally
addressed. Any review, re-transmission, dissemination or other use of or
taking of any action in reliance upon this information by persons or
entities other than the intended recipient is prohibited. If you have
received this e-mail in error, please delete it and all attachments from
any servers, hard drives or any other media.
>
>
Warning: Sufficient measures have been taken to scan any presence of
viruses however the recipient should check this email and any
attachments for the presence of viruses. Ugam accepts no liability for
any damage caused by any virus transmitted by this email.
****
RE: Issues With Parallelism In Kafka Spout
Posted by Nilesh Chhapru <ni...@ugamsolutions.com>.
Hi Harsha,
The partitions and topics are having the data and the zookeeper also shows the group id of all the consumer in its logs.
Regards,
Nilesh Chhapru.
From: Harsha [mailto:storm@harsha.io]
Sent: 18 December 2014 09:13 PM
To: user@storm.apache.org
Subject: Re: Issues With Parallelism In Kafka Spout
Hi Nilesh,
Did you check if your kafka topic partitions all being writtten to.
-Harsha
On Thu, Dec 18, 2014, at 07:22 AM, Nilesh Chhapru wrote:
Hi All,
Please give some inputs as this is pending since long and need to meet the deadlines
Regards,
Nilesh Chhapru.
From: Nilesh Chhapru [mailto:nilesh.chhapru@ugamsolutions.com]
Sent: 18 December 2014 01:24 PM
To: user@storm.apache.org<ma...@storm.apache.org>; users@kafka.apache.org<ma...@kafka.apache.org>
Subject: Issues With Parallelism In Kafka Spout
Hi All,
I have implemented a high level Kafka consumer in Storm but looks like the parallelism isn’t getting achieved as I have 3 partitions and 2 task for the spout, but only one of it is emitting the data.
PFB the screen grab for number of task of spout and data emitted by only one of them.
Please assist on how to achieve parallelism using high level Kafka spout.
[/static/docs/badimage.png]
Regards,
Nilesh Chhapru.
________________________________
---------------------------------------------------------------------------------------Disclaimer----------------------------------------------------------------------------------------------
****Opinions expressed in this e-mail are those of the author and do not necessarily represent those of Ugam. Ugam does not accept any responsibility or liability for it. This e-mail message may contain proprietary, confidential or legally privileged information for the sole use of the person or entity to whom this message was originally addressed. Any review, re-transmission, dissemination or other use of or taking of any action in reliance upon this information by persons or entities other than the intended recipient is prohibited. If you have received this e-mail in error, please delete it and all attachments from any servers, hard drives or any other media.
Warning: Sufficient measures have been taken to scan any presence of viruses however the recipient should check this email and any attachments for the presence of viruses. Ugam accepts no liability for any damage caused by any virus transmitted by this email. ****
________________________________
---------------------------------------------------------------------------------------Disclaimer----------------------------------------------------------------------------------------------
****Opinions expressed in this e-mail are those of the author and do not necessarily represent those of Ugam. Ugam does not accept any responsibility or liability for it. This e-mail message may contain proprietary, confidential or legally privileged information for the sole use of the person or entity to whom this message was originally addressed. Any review, re-transmission, dissemination or other use of or taking of any action in reliance upon this information by persons or entities other than the intended recipient is prohibited. If you have received this e-mail in error, please delete it and all attachments from any servers, hard drives or any other media.
Warning: Sufficient measures have been taken to scan any presence of viruses however the recipient should check this email and any attachments for the presence of viruses. Ugam accepts no liability for any damage caused by any virus transmitted by this email. ****
________________________________
---------------------------------------------------------------------------------------Disclaimer----------------------------------------------------------------------------------------------
****Opinions expressed in this e-mail are those of the author and do not necessarily represent those of Ugam. Ugam does not accept any responsibility or liability for it. This e-mail message may contain proprietary, confidential or legally privileged information for the sole use of the person or entity to whom this message was originally addressed. Any review, re-transmission, dissemination or other use of or taking of any action in reliance upon this information by persons or entities other than the intended recipient is prohibited. If you have received this e-mail in error, please delete it and all attachments from any servers, hard drives or any other media.
Warning: Sufficient measures have been taken to scan any presence of viruses however the recipient should check this email and any attachments for the presence of viruses. Ugam accepts no liability for any damage caused by any virus transmitted by this email. ****
Re: Issues With Parallelism In Kafka Spout
Posted by Harsha <st...@harsha.io>.
Hi Nilesh, Did you check if your kafka topic partitions all being
writtten to. -Harsha
On Thu, Dec 18, 2014, at 07:22 AM, Nilesh Chhapru wrote:
> Hi All,
>
> Please give some inputs as this is pending since long and need to meet
> the deadlines
>
> *Regards*,
> *Nilesh Chhapru.*
>
> *From:* Nilesh Chhapru [mailto:nilesh.chhapru@ugamsolutions.com]
>
> *Sent:* 18 December 2014 01:24 PM *To:* user@storm.apache.org;
> users@kafka.apache.org *Subject:* Issues With Parallelism In
> Kafka Spout
>
> Hi All,
>
> I have implemented a high level Kafka consumer in Storm but looks like
> the parallelism isn’t getting achieved as I have 3 partitions and 2
> task for the spout, but only one of it is emitting the data.
> PFB the screen grab for number of task of spout and data emitted by
> only one of them.
>
> Please assist on how to achieve parallelism using high level
> Kafka spout.
>
>
>
> *Regards*,
> *Nilesh Chhapru.*
>
>
>
> ---------------------------------------------------------------------------------------Disclaimer----------------------------------------------------------------------------------------------
>
>
****Opinions expressed in this e-mail are those of the author and do not
necessarily represent those of Ugam. Ugam does not accept any
responsibility or liability for it. This e-mail message may contain
proprietary, confidential or legally privileged information for the sole
use of the person or entity to whom this message was originally
addressed. Any review, re-transmission, dissemination or other use of or
taking of any action in reliance upon this information by persons or
entities other than the intended recipient is prohibited. If you have
received this e-mail in error, please delete it and all attachments from
any servers, hard drives or any other media.
>
>
Warning: Sufficient measures have been taken to scan any presence of
viruses however the recipient should check this email and any
attachments for the presence of viruses. Ugam accepts no liability for
any damage caused by any virus transmitted by this email.
****
>
> ---------------------------------------------------------------------------------------Disclaimer----------------------------------------------------------------------------------------------
>
>
****Opinions expressed in this e-mail are those of the author and do not
necessarily represent those of Ugam. Ugam does not accept any
responsibility or liability for it. This e-mail message may contain
proprietary, confidential or legally privileged information for the sole
use of the person or entity to whom this message was originally
addressed. Any review, re-transmission, dissemination or other use of or
taking of any action in reliance upon this information by persons or
entities other than the intended recipient is prohibited. If you have
received this e-mail in error, please delete it and all attachments from
any servers, hard drives or any other media.
>
>
Warning: Sufficient measures have been taken to scan any presence of
viruses however the recipient should check this email and any
attachments for the presence of viruses. Ugam accepts no liability for
any damage caused by any virus transmitted by this email.
****
RE: Issues With Parallelism In Kafka Spout
Posted by Nilesh Chhapru <ni...@ugamsolutions.com>.
Hi All,
Please give some inputs as this is pending since long and need to meet the deadlines
Regards,
Nilesh Chhapru.
From: Nilesh Chhapru [mailto:nilesh.chhapru@ugamsolutions.com]
Sent: 18 December 2014 01:24 PM
To: user@storm.apache.org; users@kafka.apache.org
Subject: Issues With Parallelism In Kafka Spout
Hi All,
I have implemented a high level Kafka consumer in Storm but looks like the parallelism isn't getting achieved as I have 3 partitions and 2 task for the spout, but only one of it is emitting the data.
PFB the screen grab for number of task of spout and data emitted by only one of them.
Please assist on how to achieve parallelism using high level Kafka spout.
[cid:image001.png@01D01ABD.FA56FB90]
Regards,
Nilesh Chhapru.
________________________________
---------------------------------------------------------------------------------------Disclaimer----------------------------------------------------------------------------------------------
****Opinions expressed in this e-mail are those of the author and do not necessarily represent those of Ugam. Ugam does not accept any responsibility or liability for it. This e-mail message may contain proprietary, confidential or legally privileged information for the sole use of the person or entity to whom this message was originally addressed. Any review, re-transmission, dissemination or other use of or taking of any action in reliance upon this information by persons or entities other than the intended recipient is prohibited. If you have received this e-mail in error, please delete it and all attachments from any servers, hard drives or any other media.
Warning: Sufficient measures have been taken to scan any presence of viruses however the recipient should check this email and any attachments for the presence of viruses. Ugam accepts no liability for any damage caused by any virus transmitted by this email. ****
________________________________
---------------------------------------------------------------------------------------Disclaimer----------------------------------------------------------------------------------------------
****Opinions expressed in this e-mail are those of the author and do not necessarily represent those of Ugam. Ugam does not accept any responsibility or liability for it. This e-mail message may contain proprietary, confidential or legally privileged information for the sole use of the person or entity to whom this message was originally addressed. Any review, re-transmission, dissemination or other use of or taking of any action in reliance upon this information by persons or entities other than the intended recipient is prohibited. If you have received this e-mail in error, please delete it and all attachments from any servers, hard drives or any other media.
Warning: Sufficient measures have been taken to scan any presence of viruses however the recipient should check this email and any attachments for the presence of viruses. Ugam accepts no liability for any damage caused by any virus transmitted by this email. ****
RE: Issues With Parallelism In Kafka Spout
Posted by Nilesh Chhapru <ni...@ugamsolutions.com>.
Hi All,
Please give some inputs as this is pending since long and need to meet the deadlines
Regards,
Nilesh Chhapru.
From: Nilesh Chhapru [mailto:nilesh.chhapru@ugamsolutions.com]
Sent: 18 December 2014 01:24 PM
To: user@storm.apache.org; users@kafka.apache.org
Subject: Issues With Parallelism In Kafka Spout
Hi All,
I have implemented a high level Kafka consumer in Storm but looks like the parallelism isn't getting achieved as I have 3 partitions and 2 task for the spout, but only one of it is emitting the data.
PFB the screen grab for number of task of spout and data emitted by only one of them.
Please assist on how to achieve parallelism using high level Kafka spout.
[cid:image001.png@01D01ABD.FA56FB90]
Regards,
Nilesh Chhapru.
________________________________
---------------------------------------------------------------------------------------Disclaimer----------------------------------------------------------------------------------------------
****Opinions expressed in this e-mail are those of the author and do not necessarily represent those of Ugam. Ugam does not accept any responsibility or liability for it. This e-mail message may contain proprietary, confidential or legally privileged information for the sole use of the person or entity to whom this message was originally addressed. Any review, re-transmission, dissemination or other use of or taking of any action in reliance upon this information by persons or entities other than the intended recipient is prohibited. If you have received this e-mail in error, please delete it and all attachments from any servers, hard drives or any other media.
Warning: Sufficient measures have been taken to scan any presence of viruses however the recipient should check this email and any attachments for the presence of viruses. Ugam accepts no liability for any damage caused by any virus transmitted by this email. ****
________________________________
---------------------------------------------------------------------------------------Disclaimer----------------------------------------------------------------------------------------------
****Opinions expressed in this e-mail are those of the author and do not necessarily represent those of Ugam. Ugam does not accept any responsibility or liability for it. This e-mail message may contain proprietary, confidential or legally privileged information for the sole use of the person or entity to whom this message was originally addressed. Any review, re-transmission, dissemination or other use of or taking of any action in reliance upon this information by persons or entities other than the intended recipient is prohibited. If you have received this e-mail in error, please delete it and all attachments from any servers, hard drives or any other media.
Warning: Sufficient measures have been taken to scan any presence of viruses however the recipient should check this email and any attachments for the presence of viruses. Ugam accepts no liability for any damage caused by any virus transmitted by this email. ****