You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@kafka.apache.org by xiaoyu wang <xi...@gmail.com> on 2012/05/08 21:41:29 UTC
will my topic be stored in one broker if I let the partitioner always
return 0?
Hello,
I want to keep the order of record in my topic so I created the producer
with a partitioner that always returns 0. I have the following questions
- will my topic be stored in 1 broker only?
- If so, which broker does my topic go to?
- what happens if I change the partitioner later to e.g. random
partitoner?
Thanks,
-Xiaoyu
Re: will my topic be stored in one broker if I let the partitioner always return 0?
Posted by "Xiaoyu.Wang" <xi...@gmail.com>.
On May 10, 2012, at 10:21 PM, Jun Rao <ju...@gmail.com> wrote:
> If broker A comes back, messages are routed back to A again. Ordering is
> guaranteed within a partition, not cross partitions.
Thanks. To clarify, partition here is the logical partition on each broker. So, The topic has two partitions (on broker A and B) and therefore ordering is not guaranteed.
>
> Jun
>
> On Thu, May 10, 2012 at 6:28 PM, xiaoyu wang <xi...@gmail.com> wrote:
>
>> OK. I was wrong. Hmm, assume messages all goes to broker A and was routed
>> to broker B after A goes down.what happens when A comes back online? If I
>> have a consumer that consumes the topic, is ordering guaranteed? If so, how
>> does kafka guarantee ordering in this case?
>>
>>
>>
>> On Wed, May 9, 2012 at 6:17 PM, Jun Rao <ju...@gmail.com> wrote:
>>
>>> If you have a partitioner that always return 0, produce requests will
>>> always be routed to 1 broker. However, if that broker goes down, the
>>> requests will be routed to another broker. This is because the producer
>>> only route requests to live partitions.
>>>
>>> Jun
>>>
>>> On Wed, May 9, 2012 at 5:57 PM, xiaoyu wang <xi...@gmail.com>
>> wrote:
>>>
>>>> OK. I have the answer now. when I connect to brokers through zookeeper,
>>> the
>>>> record may go to any broker and the partitioner is for partition on the
>>>> broker.
>>>>
>>>> On Wed, May 9, 2012 at 5:48 PM, xiaoyu wang <xi...@gmail.com>
>>> wrote:
>>>>
>>>>> Thanks Jun.
>>>>>
>>>>> I think my question is if I have a program which produces a topic
>> with
>>> a
>>>>> partitioner that always returns 0, does all the messages go the same
>>>>> broker? If so, what happens if I restart the program, does the
>> messages
>>>>> still go the same broker? Note that all the connections are through
>>>>> zookeeper.
>>>>>
>>>>>
>>>>> Thanks,
>>>>>
>>>>>
>>>>>
>>>>> On Tue, May 8, 2012 at 5:37 PM, Jun Rao <ju...@gmail.com> wrote:
>>>>>
>>>>>> Xiaoyu,
>>>>>>
>>>>>> A simpler way to do that is to send all produce requests to 1 broker
>>> and
>>>>>> configure that broker to have 1 partition per topic.
>>>>>>
>>>>>> Jun
>>>>>>
>>>>>> On Tue, May 8, 2012 at 12:41 PM, xiaoyu wang <xiaoyu.wang@gmail.com
>>>
>>>>>> wrote:
>>>>>>
>>>>>>> Hello,
>>>>>>>
>>>>>>> I want to keep the order of record in my topic so I created the
>>>> producer
>>>>>>> with a partitioner that always returns 0. I have the following
>>>> questions
>>>>>>>
>>>>>>>
>>>>>>> - will my topic be stored in 1 broker only?
>>>>>>> - If so, which broker does my topic go to?
>>>>>>> - what happens if I change the partitioner later to e.g. random
>>>>>>> partitoner?
>>>>>>>
>>>>>>>
>>>>>>> Thanks,
>>>>>>>
>>>>>>> -Xiaoyu
>>>>>>>
>>>>>>
>>>>>
>>>>>
>>>>
>>>
>>
Re: will my topic be stored in one broker if I let the partitioner
always return 0?
Posted by Jun Rao <ju...@gmail.com>.
If broker A comes back, messages are routed back to A again. Ordering is
guaranteed within a partition, not cross partitions.
Jun
On Thu, May 10, 2012 at 6:28 PM, xiaoyu wang <xi...@gmail.com> wrote:
> OK. I was wrong. Hmm, assume messages all goes to broker A and was routed
> to broker B after A goes down.what happens when A comes back online? If I
> have a consumer that consumes the topic, is ordering guaranteed? If so, how
> does kafka guarantee ordering in this case?
>
>
>
> On Wed, May 9, 2012 at 6:17 PM, Jun Rao <ju...@gmail.com> wrote:
>
> > If you have a partitioner that always return 0, produce requests will
> > always be routed to 1 broker. However, if that broker goes down, the
> > requests will be routed to another broker. This is because the producer
> > only route requests to live partitions.
> >
> > Jun
> >
> > On Wed, May 9, 2012 at 5:57 PM, xiaoyu wang <xi...@gmail.com>
> wrote:
> >
> > > OK. I have the answer now. when I connect to brokers through zookeeper,
> > the
> > > record may go to any broker and the partitioner is for partition on the
> > > broker.
> > >
> > > On Wed, May 9, 2012 at 5:48 PM, xiaoyu wang <xi...@gmail.com>
> > wrote:
> > >
> > > > Thanks Jun.
> > > >
> > > > I think my question is if I have a program which produces a topic
> with
> > a
> > > > partitioner that always returns 0, does all the messages go the same
> > > > broker? If so, what happens if I restart the program, does the
> messages
> > > > still go the same broker? Note that all the connections are through
> > > > zookeeper.
> > > >
> > > >
> > > > Thanks,
> > > >
> > > >
> > > >
> > > > On Tue, May 8, 2012 at 5:37 PM, Jun Rao <ju...@gmail.com> wrote:
> > > >
> > > >> Xiaoyu,
> > > >>
> > > >> A simpler way to do that is to send all produce requests to 1 broker
> > and
> > > >> configure that broker to have 1 partition per topic.
> > > >>
> > > >> Jun
> > > >>
> > > >> On Tue, May 8, 2012 at 12:41 PM, xiaoyu wang <xiaoyu.wang@gmail.com
> >
> > > >> wrote:
> > > >>
> > > >> > Hello,
> > > >> >
> > > >> > I want to keep the order of record in my topic so I created the
> > > producer
> > > >> > with a partitioner that always returns 0. I have the following
> > > questions
> > > >> >
> > > >> >
> > > >> > - will my topic be stored in 1 broker only?
> > > >> > - If so, which broker does my topic go to?
> > > >> > - what happens if I change the partitioner later to e.g. random
> > > >> > partitoner?
> > > >> >
> > > >> >
> > > >> > Thanks,
> > > >> >
> > > >> > -Xiaoyu
> > > >> >
> > > >>
> > > >
> > > >
> > >
> >
>
Re: will my topic be stored in one broker if I let the partitioner
always return 0?
Posted by xiaoyu wang <xi...@gmail.com>.
OK. I was wrong. Hmm, assume messages all goes to broker A and was routed
to broker B after A goes down.what happens when A comes back online? If I
have a consumer that consumes the topic, is ordering guaranteed? If so, how
does kafka guarantee ordering in this case?
On Wed, May 9, 2012 at 6:17 PM, Jun Rao <ju...@gmail.com> wrote:
> If you have a partitioner that always return 0, produce requests will
> always be routed to 1 broker. However, if that broker goes down, the
> requests will be routed to another broker. This is because the producer
> only route requests to live partitions.
>
> Jun
>
> On Wed, May 9, 2012 at 5:57 PM, xiaoyu wang <xi...@gmail.com> wrote:
>
> > OK. I have the answer now. when I connect to brokers through zookeeper,
> the
> > record may go to any broker and the partitioner is for partition on the
> > broker.
> >
> > On Wed, May 9, 2012 at 5:48 PM, xiaoyu wang <xi...@gmail.com>
> wrote:
> >
> > > Thanks Jun.
> > >
> > > I think my question is if I have a program which produces a topic with
> a
> > > partitioner that always returns 0, does all the messages go the same
> > > broker? If so, what happens if I restart the program, does the messages
> > > still go the same broker? Note that all the connections are through
> > > zookeeper.
> > >
> > >
> > > Thanks,
> > >
> > >
> > >
> > > On Tue, May 8, 2012 at 5:37 PM, Jun Rao <ju...@gmail.com> wrote:
> > >
> > >> Xiaoyu,
> > >>
> > >> A simpler way to do that is to send all produce requests to 1 broker
> and
> > >> configure that broker to have 1 partition per topic.
> > >>
> > >> Jun
> > >>
> > >> On Tue, May 8, 2012 at 12:41 PM, xiaoyu wang <xi...@gmail.com>
> > >> wrote:
> > >>
> > >> > Hello,
> > >> >
> > >> > I want to keep the order of record in my topic so I created the
> > producer
> > >> > with a partitioner that always returns 0. I have the following
> > questions
> > >> >
> > >> >
> > >> > - will my topic be stored in 1 broker only?
> > >> > - If so, which broker does my topic go to?
> > >> > - what happens if I change the partitioner later to e.g. random
> > >> > partitoner?
> > >> >
> > >> >
> > >> > Thanks,
> > >> >
> > >> > -Xiaoyu
> > >> >
> > >>
> > >
> > >
> >
>
Re: will my topic be stored in one broker if I let the partitioner
always return 0?
Posted by Jun Rao <ju...@gmail.com>.
If you have a partitioner that always return 0, produce requests will
always be routed to 1 broker. However, if that broker goes down, the
requests will be routed to another broker. This is because the producer
only route requests to live partitions.
Jun
On Wed, May 9, 2012 at 5:57 PM, xiaoyu wang <xi...@gmail.com> wrote:
> OK. I have the answer now. when I connect to brokers through zookeeper, the
> record may go to any broker and the partitioner is for partition on the
> broker.
>
> On Wed, May 9, 2012 at 5:48 PM, xiaoyu wang <xi...@gmail.com> wrote:
>
> > Thanks Jun.
> >
> > I think my question is if I have a program which produces a topic with a
> > partitioner that always returns 0, does all the messages go the same
> > broker? If so, what happens if I restart the program, does the messages
> > still go the same broker? Note that all the connections are through
> > zookeeper.
> >
> >
> > Thanks,
> >
> >
> >
> > On Tue, May 8, 2012 at 5:37 PM, Jun Rao <ju...@gmail.com> wrote:
> >
> >> Xiaoyu,
> >>
> >> A simpler way to do that is to send all produce requests to 1 broker and
> >> configure that broker to have 1 partition per topic.
> >>
> >> Jun
> >>
> >> On Tue, May 8, 2012 at 12:41 PM, xiaoyu wang <xi...@gmail.com>
> >> wrote:
> >>
> >> > Hello,
> >> >
> >> > I want to keep the order of record in my topic so I created the
> producer
> >> > with a partitioner that always returns 0. I have the following
> questions
> >> >
> >> >
> >> > - will my topic be stored in 1 broker only?
> >> > - If so, which broker does my topic go to?
> >> > - what happens if I change the partitioner later to e.g. random
> >> > partitoner?
> >> >
> >> >
> >> > Thanks,
> >> >
> >> > -Xiaoyu
> >> >
> >>
> >
> >
>
Re: will my topic be stored in one broker if I let the partitioner
always return 0?
Posted by xiaoyu wang <xi...@gmail.com>.
OK. I have the answer now. when I connect to brokers through zookeeper, the
record may go to any broker and the partitioner is for partition on the
broker.
On Wed, May 9, 2012 at 5:48 PM, xiaoyu wang <xi...@gmail.com> wrote:
> Thanks Jun.
>
> I think my question is if I have a program which produces a topic with a
> partitioner that always returns 0, does all the messages go the same
> broker? If so, what happens if I restart the program, does the messages
> still go the same broker? Note that all the connections are through
> zookeeper.
>
>
> Thanks,
>
>
>
> On Tue, May 8, 2012 at 5:37 PM, Jun Rao <ju...@gmail.com> wrote:
>
>> Xiaoyu,
>>
>> A simpler way to do that is to send all produce requests to 1 broker and
>> configure that broker to have 1 partition per topic.
>>
>> Jun
>>
>> On Tue, May 8, 2012 at 12:41 PM, xiaoyu wang <xi...@gmail.com>
>> wrote:
>>
>> > Hello,
>> >
>> > I want to keep the order of record in my topic so I created the producer
>> > with a partitioner that always returns 0. I have the following questions
>> >
>> >
>> > - will my topic be stored in 1 broker only?
>> > - If so, which broker does my topic go to?
>> > - what happens if I change the partitioner later to e.g. random
>> > partitoner?
>> >
>> >
>> > Thanks,
>> >
>> > -Xiaoyu
>> >
>>
>
>
Re: will my topic be stored in one broker if I let the partitioner
always return 0?
Posted by xiaoyu wang <xi...@gmail.com>.
Thanks Jun.
I think my question is if I have a program which produces a topic with a
partitioner that always returns 0, does all the messages go the same
broker? If so, what happens if I restart the program, does the messages
still go the same broker? Note that all the connections are through
zookeeper.
Thanks,
On Tue, May 8, 2012 at 5:37 PM, Jun Rao <ju...@gmail.com> wrote:
> Xiaoyu,
>
> A simpler way to do that is to send all produce requests to 1 broker and
> configure that broker to have 1 partition per topic.
>
> Jun
>
> On Tue, May 8, 2012 at 12:41 PM, xiaoyu wang <xi...@gmail.com>
> wrote:
>
> > Hello,
> >
> > I want to keep the order of record in my topic so I created the producer
> > with a partitioner that always returns 0. I have the following questions
> >
> >
> > - will my topic be stored in 1 broker only?
> > - If so, which broker does my topic go to?
> > - what happens if I change the partitioner later to e.g. random
> > partitoner?
> >
> >
> > Thanks,
> >
> > -Xiaoyu
> >
>
Re: will my topic be stored in one broker if I let the partitioner
always return 0?
Posted by Jun Rao <ju...@gmail.com>.
Xiaoyu,
A simpler way to do that is to send all produce requests to 1 broker and
configure that broker to have 1 partition per topic.
Jun
On Tue, May 8, 2012 at 12:41 PM, xiaoyu wang <xi...@gmail.com> wrote:
> Hello,
>
> I want to keep the order of record in my topic so I created the producer
> with a partitioner that always returns 0. I have the following questions
>
>
> - will my topic be stored in 1 broker only?
> - If so, which broker does my topic go to?
> - what happens if I change the partitioner later to e.g. random
> partitoner?
>
>
> Thanks,
>
> -Xiaoyu
>