You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@kafka.apache.org by Apoorva Gaurav <ap...@gmail.com> on 2013/02/02 03:47:37 UTC
Transferring events across data centers
Hello All,
We are working on Kafka based event collection system. This needs to gather
events from across data centers. Lets say all the events will be produced
in DC1 while kafka brokers and consumers are lying in DC2. Round trip
between DC1 and DC2 can be around ~80 ms. Number of events should be around
~50 million a day, peak being ~5K events a day, data volume ~100GB a day,
peak being ~10MB a day. What is the best way to do it.
--- Is keeping the producer is DC1 sending events to DC2 a good idea.
--- Should my ZK quorum lie only in DC2 or should it spawn across both DC1
and DC2.
--- Will this problem be solved easily in version .8.0 through broker
replication by keeping brokers in both DC1 and DC2.
--
Thanks & Regards,
Apoorvave
Re: Transferring events across data centers
Posted by S Ahmed <sa...@gmail.com>.
what do you mean by this:
>>Number of events should be around
>>~50 million a day, peak being ~5K events a day, data volume ~100GB a day,
>>peak being ~10MB a day.
How can it be 50 million a day, with peak of 5K a day? do you mean per
second?
On Fri, Feb 1, 2013 at 9:47 PM, Apoorva Gaurav <ap...@gmail.com>wrote:
> Hello All,
>
> We are working on Kafka based event collection system. This needs to gather
> events from across data centers. Lets say all the events will be produced
> in DC1 while kafka brokers and consumers are lying in DC2. Round trip
> between DC1 and DC2 can be around ~80 ms. Number of events should be around
> ~50 million a day, peak being ~5K events a day, data volume ~100GB a day,
> peak being ~10MB a day. What is the best way to do it.
>
> --- Is keeping the producer is DC1 sending events to DC2 a good idea.
> --- Should my ZK quorum lie only in DC2 or should it spawn across both DC1
> and DC2.
> --- Will this problem be solved easily in version .8.0 through broker
> replication by keeping brokers in both DC1 and DC2.
>
> --
> Thanks & Regards,
> Apoorvave
>
Re: Transferring events across data centers
Posted by Jun Rao <ju...@gmail.com>.
There is buffer.size in producer config for setting producer socket send
buffer size.
Thanks,
Jun
On Mon, Feb 4, 2013 at 9:29 AM, Apoorva Gaurav <ap...@gmail.com>wrote:
> Thanks Jun. Is there any parameter exposed through .properties files? I can
> see socket.send.buffer, socket.receive.buffer and max.socket.request.bytes
> in broker properties files but nothing in producers'.
>
> On Mon, Feb 4, 2013 at 10:05 PM, Jun Rao <ju...@gmail.com> wrote:
>
> > It's probably fine to have a remote producer too. You will need to do the
> > same socket buffer tuning on the producer side to amortize the long
> network
> > delay.
> >
> > Thanks,
> >
> > Jun
> >
> > On Sun, Feb 3, 2013 at 10:01 PM, Apoorva Gaurav <apoorvagaurav@gmail.com
> > >wrote:
> >
> > > Thanks Jun.
> > >
> > > So we'll have to maintain Zookeepers and Brokers in both the DCs while
> > > Producers can be in DC1 and Consumers can be in target DC2.
> > >
> > > Are there any issues if we keep only Producer in DC1 talking
> > > to Zookeepers and Brokers in DC2. I've been able to achieve this by
> > making
> > > a "hostname" entry in Broker properties which will have internal IP in
> > DC2
> > > and public IP in DC1.
> > >
> > > On Mon, Feb 4, 2013 at 10:55 AM, Jun Rao <ju...@gmail.com> wrote:
> > >
> > > > Apoorva,
> > > >
> > > > Kafka replication in 0.8 is designed for a Kafka cluster within the
> > same
> > > > DC. The following wiki describes cross DC mirroring using the tool
> > > > MirrorMaker and how to optimize the throughput for long network
> > latency.
> > > >
> > > > https://cwiki.apache.org/KAFKA/kafka-mirroring-mirrormaker.html
> > > >
> > > > Thanks,
> > > >
> > > > Jun
> > > >
> > > > On Fri, Feb 1, 2013 at 6:47 PM, Apoorva Gaurav <
> > apoorvagaurav@gmail.com
> > > > >wrote:
> > > >
> > > > > Hello All,
> > > > >
> > > > > We are working on Kafka based event collection system. This needs
> to
> > > > gather
> > > > > events from across data centers. Lets say all the events will be
> > > produced
> > > > > in DC1 while kafka brokers and consumers are lying in DC2. Round
> trip
> > > > > between DC1 and DC2 can be around ~80 ms. Number of events should
> be
> > > > around
> > > > > ~50 million a day, peak being ~5K events a day, data volume ~100GB
> a
> > > day,
> > > > > peak being ~10MB a day. What is the best way to do it.
> > > > >
> > > > > --- Is keeping the producer is DC1 sending events to DC2 a good
> idea.
> > > > > --- Should my ZK quorum lie only in DC2 or should it spawn across
> > both
> > > > DC1
> > > > > and DC2.
> > > > > --- Will this problem be solved easily in version .8.0 through
> broker
> > > > > replication by keeping brokers in both DC1 and DC2.
> > > > >
> > > > > --
> > > > > Thanks & Regards,
> > > > > Apoorvave
> > > > >
> > > >
> > >
> > >
> > >
> > > --
> > > Thanks & Regards,
> > > Apoorva
> > >
> >
>
>
>
> --
> Thanks & Regards,
> Apoorva
>
Re: Transferring events across data centers
Posted by Apoorva Gaurav <ap...@gmail.com>.
Thanks Jun. Is there any parameter exposed through .properties files? I can
see socket.send.buffer, socket.receive.buffer and max.socket.request.bytes
in broker properties files but nothing in producers'.
On Mon, Feb 4, 2013 at 10:05 PM, Jun Rao <ju...@gmail.com> wrote:
> It's probably fine to have a remote producer too. You will need to do the
> same socket buffer tuning on the producer side to amortize the long network
> delay.
>
> Thanks,
>
> Jun
>
> On Sun, Feb 3, 2013 at 10:01 PM, Apoorva Gaurav <apoorvagaurav@gmail.com
> >wrote:
>
> > Thanks Jun.
> >
> > So we'll have to maintain Zookeepers and Brokers in both the DCs while
> > Producers can be in DC1 and Consumers can be in target DC2.
> >
> > Are there any issues if we keep only Producer in DC1 talking
> > to Zookeepers and Brokers in DC2. I've been able to achieve this by
> making
> > a "hostname" entry in Broker properties which will have internal IP in
> DC2
> > and public IP in DC1.
> >
> > On Mon, Feb 4, 2013 at 10:55 AM, Jun Rao <ju...@gmail.com> wrote:
> >
> > > Apoorva,
> > >
> > > Kafka replication in 0.8 is designed for a Kafka cluster within the
> same
> > > DC. The following wiki describes cross DC mirroring using the tool
> > > MirrorMaker and how to optimize the throughput for long network
> latency.
> > >
> > > https://cwiki.apache.org/KAFKA/kafka-mirroring-mirrormaker.html
> > >
> > > Thanks,
> > >
> > > Jun
> > >
> > > On Fri, Feb 1, 2013 at 6:47 PM, Apoorva Gaurav <
> apoorvagaurav@gmail.com
> > > >wrote:
> > >
> > > > Hello All,
> > > >
> > > > We are working on Kafka based event collection system. This needs to
> > > gather
> > > > events from across data centers. Lets say all the events will be
> > produced
> > > > in DC1 while kafka brokers and consumers are lying in DC2. Round trip
> > > > between DC1 and DC2 can be around ~80 ms. Number of events should be
> > > around
> > > > ~50 million a day, peak being ~5K events a day, data volume ~100GB a
> > day,
> > > > peak being ~10MB a day. What is the best way to do it.
> > > >
> > > > --- Is keeping the producer is DC1 sending events to DC2 a good idea.
> > > > --- Should my ZK quorum lie only in DC2 or should it spawn across
> both
> > > DC1
> > > > and DC2.
> > > > --- Will this problem be solved easily in version .8.0 through broker
> > > > replication by keeping brokers in both DC1 and DC2.
> > > >
> > > > --
> > > > Thanks & Regards,
> > > > Apoorvave
> > > >
> > >
> >
> >
> >
> > --
> > Thanks & Regards,
> > Apoorva
> >
>
--
Thanks & Regards,
Apoorva
Re: Transferring events across data centers
Posted by Jun Rao <ju...@gmail.com>.
It's probably fine to have a remote producer too. You will need to do the
same socket buffer tuning on the producer side to amortize the long network
delay.
Thanks,
Jun
On Sun, Feb 3, 2013 at 10:01 PM, Apoorva Gaurav <ap...@gmail.com>wrote:
> Thanks Jun.
>
> So we'll have to maintain Zookeepers and Brokers in both the DCs while
> Producers can be in DC1 and Consumers can be in target DC2.
>
> Are there any issues if we keep only Producer in DC1 talking
> to Zookeepers and Brokers in DC2. I've been able to achieve this by making
> a "hostname" entry in Broker properties which will have internal IP in DC2
> and public IP in DC1.
>
> On Mon, Feb 4, 2013 at 10:55 AM, Jun Rao <ju...@gmail.com> wrote:
>
> > Apoorva,
> >
> > Kafka replication in 0.8 is designed for a Kafka cluster within the same
> > DC. The following wiki describes cross DC mirroring using the tool
> > MirrorMaker and how to optimize the throughput for long network latency.
> >
> > https://cwiki.apache.org/KAFKA/kafka-mirroring-mirrormaker.html
> >
> > Thanks,
> >
> > Jun
> >
> > On Fri, Feb 1, 2013 at 6:47 PM, Apoorva Gaurav <apoorvagaurav@gmail.com
> > >wrote:
> >
> > > Hello All,
> > >
> > > We are working on Kafka based event collection system. This needs to
> > gather
> > > events from across data centers. Lets say all the events will be
> produced
> > > in DC1 while kafka brokers and consumers are lying in DC2. Round trip
> > > between DC1 and DC2 can be around ~80 ms. Number of events should be
> > around
> > > ~50 million a day, peak being ~5K events a day, data volume ~100GB a
> day,
> > > peak being ~10MB a day. What is the best way to do it.
> > >
> > > --- Is keeping the producer is DC1 sending events to DC2 a good idea.
> > > --- Should my ZK quorum lie only in DC2 or should it spawn across both
> > DC1
> > > and DC2.
> > > --- Will this problem be solved easily in version .8.0 through broker
> > > replication by keeping brokers in both DC1 and DC2.
> > >
> > > --
> > > Thanks & Regards,
> > > Apoorvave
> > >
> >
>
>
>
> --
> Thanks & Regards,
> Apoorva
>
Re: Transferring events across data centers
Posted by Apoorva Gaurav <ap...@gmail.com>.
Reason I'm asking this is that I don't want to maintain Brokers in DC1 as
my consumers are exclusively in DC2.
On Mon, Feb 4, 2013 at 11:31 AM, Apoorva Gaurav <ap...@gmail.com>wrote:
> Thanks Jun.
>
> So we'll have to maintain Zookeepers and Brokers in both the DCs while
> Producers can be in DC1 and Consumers can be in target DC2.
>
> Are there any issues if we keep only Producer in DC1 talking
> to Zookeepers and Brokers in DC2. I've been able to achieve this by making
> a "hostname" entry in Broker properties which will have internal IP in DC2
> and public IP in DC1.
>
>
> On Mon, Feb 4, 2013 at 10:55 AM, Jun Rao <ju...@gmail.com> wrote:
>
>> Apoorva,
>>
>> Kafka replication in 0.8 is designed for a Kafka cluster within the same
>> DC. The following wiki describes cross DC mirroring using the tool
>> MirrorMaker and how to optimize the throughput for long network latency.
>>
>> https://cwiki.apache.org/KAFKA/kafka-mirroring-mirrormaker.html
>>
>> Thanks,
>>
>> Jun
>>
>> On Fri, Feb 1, 2013 at 6:47 PM, Apoorva Gaurav <apoorvagaurav@gmail.com
>> >wrote:
>>
>> > Hello All,
>> >
>> > We are working on Kafka based event collection system. This needs to
>> gather
>> > events from across data centers. Lets say all the events will be
>> produced
>> > in DC1 while kafka brokers and consumers are lying in DC2. Round trip
>> > between DC1 and DC2 can be around ~80 ms. Number of events should be
>> around
>> > ~50 million a day, peak being ~5K events a day, data volume ~100GB a
>> day,
>> > peak being ~10MB a day. What is the best way to do it.
>> >
>> > --- Is keeping the producer is DC1 sending events to DC2 a good idea.
>> > --- Should my ZK quorum lie only in DC2 or should it spawn across both
>> DC1
>> > and DC2.
>> > --- Will this problem be solved easily in version .8.0 through broker
>> > replication by keeping brokers in both DC1 and DC2.
>> >
>> > --
>> > Thanks & Regards,
>> > Apoorvave
>> >
>>
>
>
>
> --
> Thanks & Regards,
> Apoorva
>
--
Thanks & Regards,
Apoorva
Re: Transferring events across data centers
Posted by Apoorva Gaurav <ap...@gmail.com>.
Thanks Jun.
So we'll have to maintain Zookeepers and Brokers in both the DCs while
Producers can be in DC1 and Consumers can be in target DC2.
Are there any issues if we keep only Producer in DC1 talking
to Zookeepers and Brokers in DC2. I've been able to achieve this by making
a "hostname" entry in Broker properties which will have internal IP in DC2
and public IP in DC1.
On Mon, Feb 4, 2013 at 10:55 AM, Jun Rao <ju...@gmail.com> wrote:
> Apoorva,
>
> Kafka replication in 0.8 is designed for a Kafka cluster within the same
> DC. The following wiki describes cross DC mirroring using the tool
> MirrorMaker and how to optimize the throughput for long network latency.
>
> https://cwiki.apache.org/KAFKA/kafka-mirroring-mirrormaker.html
>
> Thanks,
>
> Jun
>
> On Fri, Feb 1, 2013 at 6:47 PM, Apoorva Gaurav <apoorvagaurav@gmail.com
> >wrote:
>
> > Hello All,
> >
> > We are working on Kafka based event collection system. This needs to
> gather
> > events from across data centers. Lets say all the events will be produced
> > in DC1 while kafka brokers and consumers are lying in DC2. Round trip
> > between DC1 and DC2 can be around ~80 ms. Number of events should be
> around
> > ~50 million a day, peak being ~5K events a day, data volume ~100GB a day,
> > peak being ~10MB a day. What is the best way to do it.
> >
> > --- Is keeping the producer is DC1 sending events to DC2 a good idea.
> > --- Should my ZK quorum lie only in DC2 or should it spawn across both
> DC1
> > and DC2.
> > --- Will this problem be solved easily in version .8.0 through broker
> > replication by keeping brokers in both DC1 and DC2.
> >
> > --
> > Thanks & Regards,
> > Apoorvave
> >
>
--
Thanks & Regards,
Apoorva
Re: Transferring events across data centers
Posted by Jun Rao <ju...@gmail.com>.
Apoorva,
Kafka replication in 0.8 is designed for a Kafka cluster within the same
DC. The following wiki describes cross DC mirroring using the tool
MirrorMaker and how to optimize the throughput for long network latency.
https://cwiki.apache.org/KAFKA/kafka-mirroring-mirrormaker.html
Thanks,
Jun
On Fri, Feb 1, 2013 at 6:47 PM, Apoorva Gaurav <ap...@gmail.com>wrote:
> Hello All,
>
> We are working on Kafka based event collection system. This needs to gather
> events from across data centers. Lets say all the events will be produced
> in DC1 while kafka brokers and consumers are lying in DC2. Round trip
> between DC1 and DC2 can be around ~80 ms. Number of events should be around
> ~50 million a day, peak being ~5K events a day, data volume ~100GB a day,
> peak being ~10MB a day. What is the best way to do it.
>
> --- Is keeping the producer is DC1 sending events to DC2 a good idea.
> --- Should my ZK quorum lie only in DC2 or should it spawn across both DC1
> and DC2.
> --- Will this problem be solved easily in version .8.0 through broker
> replication by keeping brokers in both DC1 and DC2.
>
> --
> Thanks & Regards,
> Apoorvave
>