Number of consumers in Kafka with Spark Streaming

[Guillermo Ortiz <ko...@gmail.com>]

I use Spark Streaming with Kafka and I'd like to know how many consumers
are generated. I guess that as many as partitions in Kafka but I'm not
sure.
Is there a way to know the name of the groupId generated in Spark to Kafka?

Re: Number of consumers in Kafka with Spark Streaming

[Cody Koeninger <co...@koeninger.org>]

If you're using the direct stream, and don't have speculative
execution turned on, there is one executor consumer created per
partition, plus a driver consumer for getting the latest offsets.  If
you have fewer executors than partitions, not all of those consumers
will be running at the same time.

The direct stream doesn't use consumer groups in the same way the
kafka high level consumer does, but you should be able to pass group
id in the kafka parameters.

On Tue, Jun 21, 2016 at 9:56 AM, Guillermo Ortiz <ko...@gmail.com> wrote:
> I use Spark Streaming with Kafka and I'd like to know how many consumers are
> generated. I guess that as many as partitions in Kafka but I'm not sure.
> Is there a way to know the name of the groupId generated in Spark to Kafka?

---------------------------------------------------------------------
To unsubscribe, e-mail: user-unsubscribe@spark.apache.org
For additional commands, e-mail: user-help@spark.apache.org