You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@kafka.apache.org by BYEONG-GI KIM <bg...@bluedigm.com> on 2016/03/23 01:21:51 UTC
Multi-threaded consumer?
Hello.
I'd like to know how to implement a multi-threaded consumer, which retrieve
message(s) from a topic per thread.
I read the Kafka Consumer 0.9.0.1 API document from
https://kafka.apache.org/090/javadoc/index.html?org/apache/kafka/clients/consumer/KafkaConsumer.html,
and I copied and pasted the example source code from here. And I tried to
manage threads via ThreadPoolTaskExecutor, but exception occurrd while
executing multi-thread.
The API document mentioned that multi-threaded access should be properly
synchronized, but I think the example code seems missing something for
that. My understanding of Kafka is probably bad so that I guess I may be
using the API and functions of kafka wrongly...
Could you give me a good sample code how to develop the multi-threaded
consumer or any advice? Do I need to use commitSync() method somewhere to
solve the problem?
Thanks in advance!
Best regards
bgkim
Re: Multi-threaded consumer?
Posted by BYEONG-GI KIM <bg...@bluedigm.com>.
Thank you very much for sharing the information!
I'll look into it.
Best regards
bgkim
2016-03-23 9:34 GMT+09:00 James Cheng <jc...@tivo.com>:
> Here's a good introductory blog post on the 0.9.0 consumer:
>
>
> http://www.confluent.io/blog/tutorial-getting-started-with-the-new-apache-kafka-0.9-consumer-client
>
> It shows the basics of using the consumer, as well as a section where they
> launch 3 threads, each with one consumer, to consume a single topic.
>
> -James
>
> > On Mar 22, 2016, at 5:21 PM, BYEONG-GI KIM <bg...@bluedigm.com> wrote:
> >
> > Hello.
> >
> > I'd like to know how to implement a multi-threaded consumer, which
> retrieve
> > message(s) from a topic per thread.
> >
> > I read the Kafka Consumer 0.9.0.1 API document from
> >
> https://kafka.apache.org/090/javadoc/index.html?org/apache/kafka/clients/consumer/KafkaConsumer.html
> ,
> > and I copied and pasted the example source code from here. And I tried to
> > manage threads via ThreadPoolTaskExecutor, but exception occurrd while
> > executing multi-thread.
> >
> > The API document mentioned that multi-threaded access should be properly
> > synchronized, but I think the example code seems missing something for
> > that. My understanding of Kafka is probably bad so that I guess I may be
> > using the API and functions of kafka wrongly...
> >
> >
> > Could you give me a good sample code how to develop the multi-threaded
> > consumer or any advice? Do I need to use commitSync() method somewhere to
> > solve the problem?
> >
> > Thanks in advance!
> >
> > Best regards
> >
> > bgkim
>
>
> ________________________________
>
> This email and any attachments may contain confidential and privileged
> material for the sole use of the intended recipient. Any review, copying,
> or distribution of this email (or any attachments) by others is prohibited.
> If you are not the intended recipient, please contact the sender
> immediately and permanently delete this email and any attachments. No
> employee or agent of TiVo Inc. is authorized to conclude any binding
> agreement on behalf of TiVo Inc. by email. Binding agreements with TiVo
> Inc. may only be made by a signed written agreement.
>
Re: Multi-threaded consumer?
Posted by James Cheng <jc...@tivo.com>.
Here's a good introductory blog post on the 0.9.0 consumer:
http://www.confluent.io/blog/tutorial-getting-started-with-the-new-apache-kafka-0.9-consumer-client
It shows the basics of using the consumer, as well as a section where they launch 3 threads, each with one consumer, to consume a single topic.
-James
> On Mar 22, 2016, at 5:21 PM, BYEONG-GI KIM <bg...@bluedigm.com> wrote:
>
> Hello.
>
> I'd like to know how to implement a multi-threaded consumer, which retrieve
> message(s) from a topic per thread.
>
> I read the Kafka Consumer 0.9.0.1 API document from
> https://kafka.apache.org/090/javadoc/index.html?org/apache/kafka/clients/consumer/KafkaConsumer.html,
> and I copied and pasted the example source code from here. And I tried to
> manage threads via ThreadPoolTaskExecutor, but exception occurrd while
> executing multi-thread.
>
> The API document mentioned that multi-threaded access should be properly
> synchronized, but I think the example code seems missing something for
> that. My understanding of Kafka is probably bad so that I guess I may be
> using the API and functions of kafka wrongly...
>
>
> Could you give me a good sample code how to develop the multi-threaded
> consumer or any advice? Do I need to use commitSync() method somewhere to
> solve the problem?
>
> Thanks in advance!
>
> Best regards
>
> bgkim
________________________________
This email and any attachments may contain confidential and privileged material for the sole use of the intended recipient. Any review, copying, or distribution of this email (or any attachments) by others is prohibited. If you are not the intended recipient, please contact the sender immediately and permanently delete this email and any attachments. No employee or agent of TiVo Inc. is authorized to conclude any binding agreement on behalf of TiVo Inc. by email. Binding agreements with TiVo Inc. may only be made by a signed written agreement.