You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@kafka.apache.org by Fajar Maulana Firdaus <fa...@gmail.com> on 2015/10/23 17:35:27 UTC
Consumer of multiple topic
I am using kafka to implement event sourcing. Based on a threat in
http://search-hadoop.com/kafka, I created sepparate topic for my
events, e.g. UserCreatedEvent and UserUpdateEmailEvent. Both events
are using userId as the message key. However assuming each topic has
several partition, how to setup the consumer?
My consumer is listening to both events / topics. But I am not sure if
my consumer didn't miss the event for user A from both topics because
in topic UserCreated, key userA can end up in partition 1 but on
UserUpdateEmail topic, key userA can end up in other partition.
Or does it mean I need to make consumer to listen to all partition?
But this approach is not scalable.
Thank you,
Fajar
Re: Consumer of multiple topic
Posted by Fajar Maulana Firdaus <fa...@gmail.com>.
oh that's great. thank you
On Fri, Oct 23, 2015 at 10:39 PM, pushkar priyadarshi
<pr...@gmail.com> wrote:
> Currently there are no partitions based subscription inside topic.So when
> you subscribe to both topics your consumer will get data from each
> partitions in these two topics, i dont think you would be missing anything.
>
>
>
> On Fri, Oct 23, 2015 at 11:35 AM, Fajar Maulana Firdaus <fa...@gmail.com>
> wrote:
>
>> I am using kafka to implement event sourcing. Based on a threat in
>> http://search-hadoop.com/kafka, I created sepparate topic for my
>> events, e.g. UserCreatedEvent and UserUpdateEmailEvent. Both events
>> are using userId as the message key. However assuming each topic has
>> several partition, how to setup the consumer?
>>
>> My consumer is listening to both events / topics. But I am not sure if
>> my consumer didn't miss the event for user A from both topics because
>> in topic UserCreated, key userA can end up in partition 1 but on
>> UserUpdateEmail topic, key userA can end up in other partition.
>>
>> Or does it mean I need to make consumer to listen to all partition?
>> But this approach is not scalable.
>>
>> Thank you,
>> Fajar
>>
Re: Consumer of multiple topic
Posted by pushkar priyadarshi <pr...@gmail.com>.
Currently there are no partitions based subscription inside topic.So when
you subscribe to both topics your consumer will get data from each
partitions in these two topics, i dont think you would be missing anything.
On Fri, Oct 23, 2015 at 11:35 AM, Fajar Maulana Firdaus <fa...@gmail.com>
wrote:
> I am using kafka to implement event sourcing. Based on a threat in
> http://search-hadoop.com/kafka, I created sepparate topic for my
> events, e.g. UserCreatedEvent and UserUpdateEmailEvent. Both events
> are using userId as the message key. However assuming each topic has
> several partition, how to setup the consumer?
>
> My consumer is listening to both events / topics. But I am not sure if
> my consumer didn't miss the event for user A from both topics because
> in topic UserCreated, key userA can end up in partition 1 but on
> UserUpdateEmail topic, key userA can end up in other partition.
>
> Or does it mean I need to make consumer to listen to all partition?
> But this approach is not scalable.
>
> Thank you,
> Fajar
>