You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@kafka.apache.org by Cory Kolbeck <ck...@gmail.com> on 2015/10/09 22:11:39 UTC
Urban Airship Connect
Hi folks,
I thought this list might find a case study interesting. Urban Airship just
released a new product, Connect, that uses Kafka heavily. We released some
discussion of the why and how we built it at
https://www.urbanairship.com/blog/why-we-built-urban-airship-connect and
https://www.urbanairship.com/blog/how-we-built-urban-airship-connect
respectively. The posts aren't hugely in depth engineering wise, but I hope
folks find them interesting.
Cory Kolbeck
Senior Data Engineer
Urban Airship
Re: Urban Airship Connect
Posted by Prabhjot Bharaj <pr...@gmail.com>.
Hi,
Great info.
What is your approximate/average message size ?
And, how many topics and partitions do you use in your 4-node cluster.
Also, please share your hardware specs
Thanks,
Prabhjot
On Sat, Oct 10, 2015 at 2:00 AM, Cory Kolbeck <ck...@gmail.com> wrote:
> The first level Kafkas are actually two separate clusters, and do about avg
> 50k messages/s, max 120k messages/s on 4 node clusters. The second level
> cluster doesn't handle all that load, as we filter down to the subset of
> our customers using the service. We currently use a retention window of 7
> days/100GB and double replication. Most of the time end-to-end is less than
> 500ms, but that's subject to the latency of the decoration services.
>
> On Fri, Oct 9, 2015 at 1:23 PM, Prabhjot Bharaj <pr...@gmail.com>
> wrote:
>
> > Hi,
> >
> > Nice to know another usage of Kafka
> >
> > Could you tell how much load does the first and second level Kafka
> clusters
> > intake?
> >
> > What is your write throughput, and end-to-end latency, for how long do
> you
> > hold data in Kafka and how many minimum in-sync replicas you've chosen?
> >
> > Thanks,
> > Prabhjot
> > On Oct 10, 2015 1:41 AM, "Cory Kolbeck" <ck...@gmail.com> wrote:
> >
> > > Hi folks,
> > >
> > > I thought this list might find a case study interesting. Urban Airship
> > just
> > > released a new product, Connect, that uses Kafka heavily. We released
> > some
> > > discussion of the why and how we built it at
> > > https://www.urbanairship.com/blog/why-we-built-urban-airship-connect
> and
> > > https://www.urbanairship.com/blog/how-we-built-urban-airship-connect
> > > respectively. The posts aren't hugely in depth engineering wise, but I
> > hope
> > > folks find them interesting.
> > >
> > > Cory Kolbeck
> > > Senior Data Engineer
> > > Urban Airship
> > >
> >
>
--
---------------------------------------------------------
"There are only 10 types of people in the world: Those who understand
binary, and those who don't"
Re: Urban Airship Connect
Posted by Cory Kolbeck <ck...@gmail.com>.
The first level Kafkas are actually two separate clusters, and do about avg
50k messages/s, max 120k messages/s on 4 node clusters. The second level
cluster doesn't handle all that load, as we filter down to the subset of
our customers using the service. We currently use a retention window of 7
days/100GB and double replication. Most of the time end-to-end is less than
500ms, but that's subject to the latency of the decoration services.
On Fri, Oct 9, 2015 at 1:23 PM, Prabhjot Bharaj <pr...@gmail.com>
wrote:
> Hi,
>
> Nice to know another usage of Kafka
>
> Could you tell how much load does the first and second level Kafka clusters
> intake?
>
> What is your write throughput, and end-to-end latency, for how long do you
> hold data in Kafka and how many minimum in-sync replicas you've chosen?
>
> Thanks,
> Prabhjot
> On Oct 10, 2015 1:41 AM, "Cory Kolbeck" <ck...@gmail.com> wrote:
>
> > Hi folks,
> >
> > I thought this list might find a case study interesting. Urban Airship
> just
> > released a new product, Connect, that uses Kafka heavily. We released
> some
> > discussion of the why and how we built it at
> > https://www.urbanairship.com/blog/why-we-built-urban-airship-connect and
> > https://www.urbanairship.com/blog/how-we-built-urban-airship-connect
> > respectively. The posts aren't hugely in depth engineering wise, but I
> hope
> > folks find them interesting.
> >
> > Cory Kolbeck
> > Senior Data Engineer
> > Urban Airship
> >
>
Re: Urban Airship Connect
Posted by Prabhjot Bharaj <pr...@gmail.com>.
Hi,
Nice to know another usage of Kafka
Could you tell how much load does the first and second level Kafka clusters
intake?
What is your write throughput, and end-to-end latency, for how long do you
hold data in Kafka and how many minimum in-sync replicas you've chosen?
Thanks,
Prabhjot
On Oct 10, 2015 1:41 AM, "Cory Kolbeck" <ck...@gmail.com> wrote:
> Hi folks,
>
> I thought this list might find a case study interesting. Urban Airship just
> released a new product, Connect, that uses Kafka heavily. We released some
> discussion of the why and how we built it at
> https://www.urbanairship.com/blog/why-we-built-urban-airship-connect and
> https://www.urbanairship.com/blog/how-we-built-urban-airship-connect
> respectively. The posts aren't hugely in depth engineering wise, but I hope
> folks find them interesting.
>
> Cory Kolbeck
> Senior Data Engineer
> Urban Airship
>