You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@kafka.apache.org by Chitra Raveendran <ch...@fluturasolutions.com> on 2013/05/14 14:20:22 UTC
can producers(from same system) send messages to separate broker systems?
Hi
Can there be 2 producers in the same server, both sending their own
separate topics to separate broker systems?
--
Chitra Raveendran
Data Scientist
*Flutura Business Solutions Pvt. Ltd*
email : chitra.raveendran@fluturasoutions.com
Re: can producers(from same system) send messages to separate broker systems?
Posted by Neha Narkhede <ne...@gmail.com>.
The feature I mentioned is only available on 0.8. In 0.7.2, you can tweak
producer batch size and the flush interval on the broker for the high
priority topics. Note that setting those too low will have performance
implications.
Thanks,
Neha
On May 17, 2013 2:25 PM, "Neha Narkhede" <ne...@gmail.com> wrote:
> Do you have any tests that measure that your high priority data is being
> delayed ? Assuming you are using 0.8, the end to end latency can be reduced
> by tuning some configs on the consumer (fetch.min.bytes, fetch.wait.max.ms
> ).
> The defaults for these configs are already tuned for low latency though.
>
> Thanks,
> Neha
>
>
> On Tue, May 14, 2013 at 11:46 AM, Chitra Raveendran <
> chitra.raveendran@fluturasolutions.com> wrote:
>
>> Thanks for the reply,
>>
>> I have a 3 node kafka cluster, and i have 2 topics( one of very high
>> priority and other normal data), i need to transmit the normal data to two
>> brokers in the cluster , and the high priority data directly to the 3rd
>> broker.
>>
>> This is so that my high priority data has a clear path and can be
>> transmitted without any delay at all.
>>
>> What should I do to achieve this ? Just specifying appropriate broker.list
>> for each producer , is enough?
>>
>>
>> On Tue, May 14, 2013 at 9:11 PM, Neha Narkhede <neha.narkhede@gmail.com
>> >wrote:
>>
>> > Yes there can be. You just need to make sure they are configured with
>> > separate broker.list.
>> >
>> > Thanks,
>> > Neha
>> > On May 14, 2013 5:28 AM, "Andrea Gazzarini" <andrea.gazzarini@gmail.com
>> >
>> > wrote:
>> >
>> > > Of course...what is the problem? Or maybe you're missing some other
>> > > constraint?
>> > >
>> > > On 05/14/2013 02:20 PM, Chitra Raveendran wrote:
>> > >
>> > >> Hi
>> > >>
>> > >> Can there be 2 producers in the same server, both sending their own
>> > >> separate topics to separate broker systems?
>> > >>
>> > >>
>> > >
>> >
>>
>>
>>
>> --
>> Chitra Raveendran
>> Data Scientist
>> *Flutura Business Solutions Pvt. Ltd*
>> Tel : +918197563660
>> email : chitra.raveendran@fluturasoutions.com
>>
>
>
Re: can producers(from same system) send messages to separate broker systems?
Posted by Chitra Raveendran <ch...@fluturasolutions.com>.
Hi Neha
No i haven't experienced any noticeable latency as of now, the high
priority data is too critical for any sort of latency, that's why i wanted
to optimize everything before deployment.
I'm using 0.7.2 since my consumer is a storm spout, and i read that storm
is most compatible with 0.7.2
Does 0.7.2 have that feature as well ??
On Sat, May 18, 2013 at 2:55 AM, Neha Narkhede <ne...@gmail.com>wrote:
> Do you have any tests that measure that your high priority data is being
> delayed ? Assuming you are using 0.8, the end to end latency can be reduced
> by tuning some configs on the consumer (fetch.min.bytes, fetch.wait.max.ms
> ).
> The defaults for these configs are already tuned for low latency though.
>
> Thanks,
> Neha
>
>
> On Tue, May 14, 2013 at 11:46 AM, Chitra Raveendran <
> chitra.raveendran@fluturasolutions.com> wrote:
>
> > Thanks for the reply,
> >
> > I have a 3 node kafka cluster, and i have 2 topics( one of very high
> > priority and other normal data), i need to transmit the normal data to
> two
> > brokers in the cluster , and the high priority data directly to the 3rd
> > broker.
> >
> > This is so that my high priority data has a clear path and can be
> > transmitted without any delay at all.
> >
> > What should I do to achieve this ? Just specifying appropriate
> broker.list
> > for each producer , is enough?
> >
> >
> > On Tue, May 14, 2013 at 9:11 PM, Neha Narkhede <neha.narkhede@gmail.com
> > >wrote:
> >
> > > Yes there can be. You just need to make sure they are configured with
> > > separate broker.list.
> > >
> > > Thanks,
> > > Neha
> > > On May 14, 2013 5:28 AM, "Andrea Gazzarini" <
> andrea.gazzarini@gmail.com>
> > > wrote:
> > >
> > > > Of course...what is the problem? Or maybe you're missing some other
> > > > constraint?
> > > >
> > > > On 05/14/2013 02:20 PM, Chitra Raveendran wrote:
> > > >
> > > >> Hi
> > > >>
> > > >> Can there be 2 producers in the same server, both sending their own
> > > >> separate topics to separate broker systems?
> > > >>
> > > >>
> > > >
> > >
> >
> >
> >
> > --
> > Chitra Raveendran
> > Data Scientist
> > *Flutura Business Solutions Pvt. Ltd*
> > Tel : +918197563660
> > email : chitra.raveendran@fluturasoutions.com
> >
>
--
Chitra Raveendran
Data Scientist
*Flutura Business Solutions Pvt. Ltd*
email : chitra.raveendran@fluturasoutions.com
Re: can producers(from same system) send messages to separate broker systems?
Posted by Neha Narkhede <ne...@gmail.com>.
Do you have any tests that measure that your high priority data is being
delayed ? Assuming you are using 0.8, the end to end latency can be reduced
by tuning some configs on the consumer (fetch.min.bytes, fetch.wait.max.ms
).
The defaults for these configs are already tuned for low latency though.
Thanks,
Neha
On Tue, May 14, 2013 at 11:46 AM, Chitra Raveendran <
chitra.raveendran@fluturasolutions.com> wrote:
> Thanks for the reply,
>
> I have a 3 node kafka cluster, and i have 2 topics( one of very high
> priority and other normal data), i need to transmit the normal data to two
> brokers in the cluster , and the high priority data directly to the 3rd
> broker.
>
> This is so that my high priority data has a clear path and can be
> transmitted without any delay at all.
>
> What should I do to achieve this ? Just specifying appropriate broker.list
> for each producer , is enough?
>
>
> On Tue, May 14, 2013 at 9:11 PM, Neha Narkhede <neha.narkhede@gmail.com
> >wrote:
>
> > Yes there can be. You just need to make sure they are configured with
> > separate broker.list.
> >
> > Thanks,
> > Neha
> > On May 14, 2013 5:28 AM, "Andrea Gazzarini" <an...@gmail.com>
> > wrote:
> >
> > > Of course...what is the problem? Or maybe you're missing some other
> > > constraint?
> > >
> > > On 05/14/2013 02:20 PM, Chitra Raveendran wrote:
> > >
> > >> Hi
> > >>
> > >> Can there be 2 producers in the same server, both sending their own
> > >> separate topics to separate broker systems?
> > >>
> > >>
> > >
> >
>
>
>
> --
> Chitra Raveendran
> Data Scientist
> *Flutura Business Solutions Pvt. Ltd*
> Tel : +918197563660
> email : chitra.raveendran@fluturasoutions.com
>
Re: can producers(from same system) send messages to separate broker systems?
Posted by Chitra Raveendran <ch...@fluturasolutions.com>.
Thanks for the reply,
I have a 3 node kafka cluster, and i have 2 topics( one of very high
priority and other normal data), i need to transmit the normal data to two
brokers in the cluster , and the high priority data directly to the 3rd
broker.
This is so that my high priority data has a clear path and can be
transmitted without any delay at all.
What should I do to achieve this ? Just specifying appropriate broker.list
for each producer , is enough?
On Tue, May 14, 2013 at 9:11 PM, Neha Narkhede <ne...@gmail.com>wrote:
> Yes there can be. You just need to make sure they are configured with
> separate broker.list.
>
> Thanks,
> Neha
> On May 14, 2013 5:28 AM, "Andrea Gazzarini" <an...@gmail.com>
> wrote:
>
> > Of course...what is the problem? Or maybe you're missing some other
> > constraint?
> >
> > On 05/14/2013 02:20 PM, Chitra Raveendran wrote:
> >
> >> Hi
> >>
> >> Can there be 2 producers in the same server, both sending their own
> >> separate topics to separate broker systems?
> >>
> >>
> >
>
--
Chitra Raveendran
Data Scientist
*Flutura Business Solutions Pvt. Ltd*
Tel : +918197563660
email : chitra.raveendran@fluturasoutions.com
Re: can producers(from same system) send messages to separate broker systems?
Posted by Neha Narkhede <ne...@gmail.com>.
Yes there can be. You just need to make sure they are configured with
separate broker.list.
Thanks,
Neha
On May 14, 2013 5:28 AM, "Andrea Gazzarini" <an...@gmail.com>
wrote:
> Of course...what is the problem? Or maybe you're missing some other
> constraint?
>
> On 05/14/2013 02:20 PM, Chitra Raveendran wrote:
>
>> Hi
>>
>> Can there be 2 producers in the same server, both sending their own
>> separate topics to separate broker systems?
>>
>>
>
Re: can producers(from same system) send messages to separate broker
systems?
Posted by Andrea Gazzarini <an...@gmail.com>.
Of course...what is the problem? Or maybe you're missing some other
constraint?
On 05/14/2013 02:20 PM, Chitra Raveendran wrote:
> Hi
>
> Can there be 2 producers in the same server, both sending their own
> separate topics to separate broker systems?
>