cross-colo writing/reading?

[Yang <te...@gmail.com>]

we tested our new application that reads and writes to kafka.

at first we found the access latency is very high. then we realized that
it's because the client and server are in different colos. moving them
together reduces down the access time to < 4 ms.


I was wondering if there are any techniques/properties to set , or maybe
enhancements to be made to the kafka code, that could cater to such
situations ?

Thanks
Yang

Re: cross-colo writing/reading?

[Todd Palino <tp...@gmail.com>]

I don't know if I'd go as far as "elegant". Functional, definitely. :)

Yang, I'm not entirely sure what you're looking for here. You can already
specify the acks setting in the producer if you do not care about
acknowledgement from your remote produce requests (setting it to 0). If you
do care, you just have to tolerate the latency. The speed of light isn't
just a good idea, it's the law.

-Todd


On Tue, Mar 3, 2015 at 3:34 PM, Jeff Schroeder <je...@computer.org>
wrote:

> Mirror maker is about separating latency and failure domains. I think it is
> a very elegant solution to a difficult problem. My suspicion is that the
> LinkedIn / Confluent team agrees.
>
> On Tue, Mar 3, 2015 at 3:50 PM, Yang <te...@gmail.com> wrote:
>
> > thanks guys.
> >
> > it's just quite a lot of ops cost to setup and monitor a separate
> cluster,
> > connected through mirror maker. sometimes if I have just a single
> > producer/consumer in a new cluster, it would be more desirable to just
> > connect it directly to an existing kafka setup.
> >
> > I remember at least in an earlier version of Cassandra (about 2 years
> ago,
> > when I still actively worked on that), you could define network
> topologies
> > , and define the different acknowledgement requirements for different
> parts
> > of the topology.  if this kind of flexibility is built into the internal
> > protocol of Kafka, it would be much nicer. personally I think the mirror
> > maker is more of a patch work than a systematic design.
> >
> > On Mon, Mar 2, 2015 at 7:53 AM, Todd Palino <tp...@gmail.com> wrote:
> >
> > > Latencies like this are one of the big reasons that we run our Kafka
> > > clusters local to the producers and consumers. Another is network
> > > partition. As Jeff noted, mirror maker is the way to connect them
> > together.
> > > Our architecture uses a local cluster in each datacenter, and then an
> > > aggregate cluster (either in those datacenters or in another) where we
> > > mirror all the traffic from the local clusters together into one place.
> > > This way clients can pick a local or a global view of the data, and we
> > can
> > > have everyone still stay in their local Kafka clusters.
> > >
> > > -Todd
> > >
> > >
> > > On Fri, Feb 27, 2015 at 4:21 PM, Yang <te...@gmail.com> wrote:
> > >
> > > > we tested our new application that reads and writes to kafka.
> > > >
> > > > at first we found the access latency is very high. then we realized
> > that
> > > > it's because the client and server are in different colos. moving
> them
> > > > together reduces down the access time to < 4 ms.
> > > >
> > > >
> > > > I was wondering if there are any techniques/properties to set , or
> > maybe
> > > > enhancements to be made to the kafka code, that could cater to such
> > > > situations ?
> > > >
> > > > Thanks
> > > > Yang
> > > >
> > >
> >
>
>
>
> --
> Jeff Schroeder
>
> Don't drink and derive, alcohol and analysis don't mix.
> http://www.digitalprognosis.com
>

Re: cross-colo writing/reading?

[Jeff Schroeder <je...@computer.org>]

Mirror maker is about separating latency and failure domains. I think it is
a very elegant solution to a difficult problem. My suspicion is that the
LinkedIn / Confluent team agrees.

On Tue, Mar 3, 2015 at 3:50 PM, Yang <te...@gmail.com> wrote:

> thanks guys.
>
> it's just quite a lot of ops cost to setup and monitor a separate cluster,
> connected through mirror maker. sometimes if I have just a single
> producer/consumer in a new cluster, it would be more desirable to just
> connect it directly to an existing kafka setup.
>
> I remember at least in an earlier version of Cassandra (about 2 years ago,
> when I still actively worked on that), you could define network topologies
> , and define the different acknowledgement requirements for different parts
> of the topology.  if this kind of flexibility is built into the internal
> protocol of Kafka, it would be much nicer. personally I think the mirror
> maker is more of a patch work than a systematic design.
>
> On Mon, Mar 2, 2015 at 7:53 AM, Todd Palino <tp...@gmail.com> wrote:
>
> > Latencies like this are one of the big reasons that we run our Kafka
> > clusters local to the producers and consumers. Another is network
> > partition. As Jeff noted, mirror maker is the way to connect them
> together.
> > Our architecture uses a local cluster in each datacenter, and then an
> > aggregate cluster (either in those datacenters or in another) where we
> > mirror all the traffic from the local clusters together into one place.
> > This way clients can pick a local or a global view of the data, and we
> can
> > have everyone still stay in their local Kafka clusters.
> >
> > -Todd
> >
> >
> > On Fri, Feb 27, 2015 at 4:21 PM, Yang <te...@gmail.com> wrote:
> >
> > > we tested our new application that reads and writes to kafka.
> > >
> > > at first we found the access latency is very high. then we realized
> that
> > > it's because the client and server are in different colos. moving them
> > > together reduces down the access time to < 4 ms.
> > >
> > >
> > > I was wondering if there are any techniques/properties to set , or
> maybe
> > > enhancements to be made to the kafka code, that could cater to such
> > > situations ?
> > >
> > > Thanks
> > > Yang
> > >
> >
>



-- 
Jeff Schroeder

Don't drink and derive, alcohol and analysis don't mix.
http://www.digitalprognosis.com

Re: cross-colo writing/reading?

[Yang <te...@gmail.com>]

thanks guys.

it's just quite a lot of ops cost to setup and monitor a separate cluster,
connected through mirror maker. sometimes if I have just a single
producer/consumer in a new cluster, it would be more desirable to just
connect it directly to an existing kafka setup.

I remember at least in an earlier version of Cassandra (about 2 years ago,
when I still actively worked on that), you could define network topologies
, and define the different acknowledgement requirements for different parts
of the topology.  if this kind of flexibility is built into the internal
protocol of Kafka, it would be much nicer. personally I think the mirror
maker is more of a patch work than a systematic design.

On Mon, Mar 2, 2015 at 7:53 AM, Todd Palino <tp...@gmail.com> wrote:

> Latencies like this are one of the big reasons that we run our Kafka
> clusters local to the producers and consumers. Another is network
> partition. As Jeff noted, mirror maker is the way to connect them together.
> Our architecture uses a local cluster in each datacenter, and then an
> aggregate cluster (either in those datacenters or in another) where we
> mirror all the traffic from the local clusters together into one place.
> This way clients can pick a local or a global view of the data, and we can
> have everyone still stay in their local Kafka clusters.
>
> -Todd
>
>
> On Fri, Feb 27, 2015 at 4:21 PM, Yang <te...@gmail.com> wrote:
>
> > we tested our new application that reads and writes to kafka.
> >
> > at first we found the access latency is very high. then we realized that
> > it's because the client and server are in different colos. moving them
> > together reduces down the access time to < 4 ms.
> >
> >
> > I was wondering if there are any techniques/properties to set , or maybe
> > enhancements to be made to the kafka code, that could cater to such
> > situations ?
> >
> > Thanks
> > Yang
> >
>

Re: cross-colo writing/reading?

[Todd Palino <tp...@gmail.com>]

Latencies like this are one of the big reasons that we run our Kafka
clusters local to the producers and consumers. Another is network
partition. As Jeff noted, mirror maker is the way to connect them together.
Our architecture uses a local cluster in each datacenter, and then an
aggregate cluster (either in those datacenters or in another) where we
mirror all the traffic from the local clusters together into one place.
This way clients can pick a local or a global view of the data, and we can
have everyone still stay in their local Kafka clusters.

-Todd


On Fri, Feb 27, 2015 at 4:21 PM, Yang <te...@gmail.com> wrote:

> we tested our new application that reads and writes to kafka.
>
> at first we found the access latency is very high. then we realized that
> it's because the client and server are in different colos. moving them
> together reduces down the access time to < 4 ms.
>
>
> I was wondering if there are any techniques/properties to set , or maybe
> enhancements to be made to the kafka code, that could cater to such
> situations ?
>
> Thanks
> Yang
>

Re: cross-colo writing/reading?

[Jeff Schroeder <je...@computer.org>]

Perhaps mirror maker is what you want?

https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=27846330

On Friday, February 27, 2015, Yang <te...@gmail.com> wrote:

> we tested our new application that reads and writes to kafka.
>
> at first we found the access latency is very high. then we realized that
> it's because the client and server are in different colos. moving them
> together reduces down the access time to < 4 ms.
>
>
> I was wondering if there are any techniques/properties to set , or maybe
> enhancements to be made to the kafka code, that could cater to such
> situations ?
>
> Thanks
> Yang
>


-- 
Text by Jeff, typos by iPhone