You are viewing a plain text version of this content. The canonical link for it is here.

Posted to users@kafka.apache.org by Nishanth S <ni...@gmail.com> on 2017/09/11 21:48:46 UTC

Identifying Number of Kafka Consumers

All,
I am very new to kafka . We have a  case where we need to ingest multiple
avro record types . These avro record types vary vastly in volume  and size
and I am thinking of sending  each of these message types to a different
topic and creating partitions based on volume and through put needed.  What
the  kafka consumer has to do is read the record of from  partitions and
 write to  different hdfs locations based on record type  . I am guessing
we should at least start with one consumer per  topic  . Is this
understanding correct or is there a better way to look at it?

Re: Identifying Number of Kafka Consumers

Posted by Nishanth S <ni...@gmail.com>.

Thanks David!. We are not  using  confluent at the moment .Since the work
 that needs to be  done by each consumer is the same (read > write to hdfs)
  I am guessing my consumer code will just look the same and will need just
one consumer group.

Thanks,
Nishanth

On Tue, Sep 12, 2017 at 8:53 AM, David Garcia <da...@spiceworks.com> wrote:

> Consumers can be split up based on partitions.  So, you can tell a
> consumer group to listen to several topics and it will divvy up the work.
> Your use case sounds very canonical.  I would take a look at Kafka connect
> (if you’re using the confluent stack).
>
> -Daivd
>
> http://docs.confluent.io/current/connect/connect-hdfs/docs/index.html
>
> On 9/11/17, 4:48 PM, "Nishanth S" <ni...@gmail.com> wrote:
>
>     All,
>     I am very new to kafka . We have a  case where we need to ingest
> multiple
>     avro record types . These avro record types vary vastly in volume  and
> size
>     and I am thinking of sending  each of these message types to a
> different
>     topic and creating partitions based on volume and through put needed.
> What
>     the  kafka consumer has to do is read the record of from  partitions
> and
>      write to  different hdfs locations based on record type  . I am
> guessing
>     we should at least start with one consumer per  topic  . Is this
>     understanding correct or is there a better way to look at it?
>
>
>

Re: Identifying Number of Kafka Consumers

Posted by David Garcia <da...@spiceworks.com>.

Consumers can be split up based on partitions.  So, you can tell a consumer group to listen to several topics and it will divvy up the work.  Your use case sounds very canonical.  I would take a look at Kafka connect (if you’re using the confluent stack).

-Daivd

http://docs.confluent.io/current/connect/connect-hdfs/docs/index.html

On 9/11/17, 4:48 PM, "Nishanth S" <ni...@gmail.com> wrote:

    All,
    I am very new to kafka . We have a  case where we need to ingest multiple
    avro record types . These avro record types vary vastly in volume  and size
    and I am thinking of sending  each of these message types to a different
    topic and creating partitions based on volume and through put needed.  What
    the  kafka consumer has to do is read the record of from  partitions and
     write to  different hdfs locations based on record type  . I am guessing
    we should at least start with one consumer per  topic  . Is this
    understanding correct or is there a better way to look at it?