You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@spark.apache.org by hotdog <li...@163.com> on 2015/05/13 17:33:46 UTC
force the kafka consumer process to different machines
I 'm using streaming integrated with streaming-kafka.
My kafka topic has 80 partitions, while my machines have 40 cores. I found
that when the job is running, the kafka consumer processes are only deploy
to 2 machines, the bandwidth of the 2 machines will be very very high.
I wonder is there any way to control the kafka consumer's dispatch？
--
View this message in context: http://apache-spark-user-list.1001560.n3.nabble.com/force-the-kafka-consumer-process-to-different-machines-tp22872.html
Sent from the Apache Spark User List mailing list archive at Nabble.com.
---------------------------------------------------------------------
To unsubscribe, e-mail: user-unsubscribe@spark.apache.org
For additional commands, e-mail: user-help@spark.apache.org
Re: force the kafka consumer process to different machines
Posted by Cody Koeninger <co...@koeninger.org>.
I assume you're using the receiver based approach? Have you tried the
createDirectStream api?
https://spark.apache.org/docs/1.3.0/streaming-kafka-integration.html
If you're sticking with the receiver based approach I think your only
option would be to create more consumer streams and union them. That
doesn't give you control over where they're run, but should increase the
consumer parallelism.
On Wed, May 13, 2015 at 10:33 AM, hotdog <li...@163.com> wrote:
> I 'm using streaming integrated with streaming-kafka.
>
> My kafka topic has 80 partitions, while my machines have 40 cores. I found
> that when the job is running, the kafka consumer processes are only deploy
> to 2 machines, the bandwidth of the 2 machines will be very very high.
>
> I wonder is there any way to control the kafka consumer's dispatch？
>
>
>
> --
> View this message in context:
> http://apache-spark-user-list.1001560.n3.nabble.com/force-the-kafka-consumer-process-to-different-machines-tp22872.html
> Sent from the Apache Spark User List mailing list archive at Nabble.com.
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: user-unsubscribe@spark.apache.org
> For additional commands, e-mail: user-help@spark.apache.org
>
>
Re: force the kafka consumer process to different machines
Posted by Akhil Das <ak...@sigmoidanalytics.com>.
With this lowlevel Kafka API
<https://github.com/dibbhatt/kafka-spark-consumer/>, you can actually
specify how many receivers that you want to spawn and most of the time it
spawns evenly, usually you can put a sleep just after creating the context
for the executors to connect to the driver and then spark will evenly
distribute the receivers.
Thanks
Best Regards
On Wed, May 13, 2015 at 9:03 PM, hotdog <li...@163.com> wrote:
> I 'm using streaming integrated with streaming-kafka.
>
> My kafka topic has 80 partitions, while my machines have 40 cores. I found
> that when the job is running, the kafka consumer processes are only deploy
> to 2 machines, the bandwidth of the 2 machines will be very very high.
>
> I wonder is there any way to control the kafka consumer's dispatch？
>
>
>
> --
> View this message in context:
> http://apache-spark-user-list.1001560.n3.nabble.com/force-the-kafka-consumer-process-to-different-machines-tp22872.html
> Sent from the Apache Spark User List mailing list archive at Nabble.com.
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: user-unsubscribe@spark.apache.org
> For additional commands, e-mail: user-help@spark.apache.org
>
>