You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@kafka.apache.org by Philip O'Toole <ph...@loggly.com> on 2013/12/02 03:17:22 UTC

Loggly's use of Kafka on AWS

A couple of us here at Loggly recently spoke at AWS reinvent, on how we use Kafka 0.72 in our ingestion pipeline. The slides are at the link below, and may be of interest to people on this list. 

http://www.slideshare.net/AmazonWebServices/infrastructure-at-scale-apache-kafka-twitter-storm-elastic-search-arc303-aws-reinvent-2013

Any questions, let me know, though I can't promise I can answer everything. Can't give the complete game away. :-)

As always, Kafka rocks!

Philip




Re: Loggly's use of Kafka on AWS

Posted by Jonathan Hodges <ho...@gmail.com>.
A cheaper approach is to use ephemeral disk and have the replicas across
AZs.  You can lose up to 2 AZs and still avoid data loss.  You mitigate
complete region failure using mirror maker to replicate to another region.


On Tue, Dec 3, 2013 at 8:41 AM, Philip O'Toole <ph...@loggly.com> wrote:

> Yeah, we use provisioned IOPs EBS so we have good performance.
>
> We run 3 ZK nodes, one in each AZ. Quorum 2.
>
> Philip
>
> On Dec 3, 2013, at 7:31 AM, Timothy Chen <tn...@gmail.com> wrote:
>
> > Hi Philip,
> >
> > So I wonder if you guys hit disk perf problems with EBS? It seems quite
> common in the past but I haven't tried recently.
> >
> > Also can you share how you guys deployed zookeeper in AWS so that a
> qurom is always available?
> >
> > Tim
> >
> > Sent from my iPhone
> >
> >> On Dec 2, 2013, at 5:15 PM, Steve Morin <st...@gmail.com> wrote:
> >>
> >> Philip this is definitely useful
> >>
> >>> On Dec 2, 2013, at 14:55, Surendranauth Hiraman <
> suren.hiraman@sociocast.com> wrote:
> >>>
> >>> S Ahmed,
> >>>
> >>> This combination of Kafka and Storm to process streaming data is
> becoming
> >>> pretty common. Definitely worth looking at.
> >>>
> >>> The throughput will vary depending on your workload (cpu usage, etc.)
> and
> >>> if you're talking to a backend, of course. But it scales very well.
> >>>
> >>> -Suren
> >>>
> >>>
> >>>
> >>>> On Mon, Dec 2, 2013 at 5:49 PM, S Ahmed <sa...@gmail.com> wrote:
> >>>>
> >>>> Interesting.  So twitter storm is used to basically process the
> messages on
> >>>> kafka?   I'll have to read-up on storm b/c I always thought the use
> case
> >>>> was a bit different.
> >>>>
> >>>>
> >>>>
> >>>>> On Sun, Dec 1, 2013 at 9:59 PM, Joe Stein <jo...@stealth.ly>
> wrote:
> >>>>>
> >>>>> Awesome Philip, thanks for sharing!
> >>>>>
> >>>>>> On Sun, Dec 1, 2013 at 9:17 PM, Philip O'Toole <ph...@loggly.com>
> >>>>> wrote:
> >>>>>
> >>>>>> A couple of us here at Loggly recently spoke at AWS reinvent, on
> how we
> >>>>>> use Kafka 0.72 in our ingestion pipeline. The slides are at the link
> >>>>> below,
> >>>>>> and may be of interest to people on this list.
> >>>>
> http://www.slideshare.net/AmazonWebServices/infrastructure-at-scale-apache-kafka-twitter-storm-elastic-search-arc303-aws-reinvent-2013
> >>>>>>
> >>>>>> Any questions, let me know, though I can't promise I can answer
> >>>>>> everything. Can't give the complete game away. :-)
> >>>>>>
> >>>>>> As always, Kafka rocks!
> >>>>>>
> >>>>>> Philip
> >>>
> >>>
> >>>
> >>> --
> >>> ___________________________________________
> >>> Available at these partners:
> >>> [image: CloudFlare | shopify | Bigcommerce]
> >>>
> >>> SUREN HIRAMAN, VP TECHNOLOGY
> >>> SOCIOCAST
> >>> Simple. Powerful. Predictions.
> >>>
> >>> 96 SPRING STREET, 7TH FLOOR
> >>> NEW YORK, NY 10012
> >>> O: (917) 525-2466 ext. 105
> >>> F: 646.349.4063
> >>> E: suren.hiraman@sociocast.com
> >>> W: www.sociocast.com
> >>>
> >>> Increase Conversion Rates up to 500%. Go to www.sociocast.com and
> enter
> >>> your URL for a free trial!
>

Re: Loggly's use of Kafka on AWS

Posted by Philip O'Toole <ph...@loggly.com>.
Yeah, we use provisioned IOPs EBS so we have good performance. 

We run 3 ZK nodes, one in each AZ. Quorum 2. 

Philip

On Dec 3, 2013, at 7:31 AM, Timothy Chen <tn...@gmail.com> wrote:

> Hi Philip,
> 
> So I wonder if you guys hit disk perf problems with EBS? It seems quite common in the past but I haven't tried recently.
> 
> Also can you share how you guys deployed zookeeper in AWS so that a qurom is always available?
> 
> Tim
> 
> Sent from my iPhone
> 
>> On Dec 2, 2013, at 5:15 PM, Steve Morin <st...@gmail.com> wrote:
>> 
>> Philip this is definitely useful
>> 
>>> On Dec 2, 2013, at 14:55, Surendranauth Hiraman <su...@sociocast.com> wrote:
>>> 
>>> S Ahmed,
>>> 
>>> This combination of Kafka and Storm to process streaming data is becoming
>>> pretty common. Definitely worth looking at.
>>> 
>>> The throughput will vary depending on your workload (cpu usage, etc.) and
>>> if you're talking to a backend, of course. But it scales very well.
>>> 
>>> -Suren
>>> 
>>> 
>>> 
>>>> On Mon, Dec 2, 2013 at 5:49 PM, S Ahmed <sa...@gmail.com> wrote:
>>>> 
>>>> Interesting.  So twitter storm is used to basically process the messages on
>>>> kafka?   I'll have to read-up on storm b/c I always thought the use case
>>>> was a bit different.
>>>> 
>>>> 
>>>> 
>>>>> On Sun, Dec 1, 2013 at 9:59 PM, Joe Stein <jo...@stealth.ly> wrote:
>>>>> 
>>>>> Awesome Philip, thanks for sharing!
>>>>> 
>>>>>> On Sun, Dec 1, 2013 at 9:17 PM, Philip O'Toole <ph...@loggly.com>
>>>>> wrote:
>>>>> 
>>>>>> A couple of us here at Loggly recently spoke at AWS reinvent, on how we
>>>>>> use Kafka 0.72 in our ingestion pipeline. The slides are at the link
>>>>> below,
>>>>>> and may be of interest to people on this list.
>>>> http://www.slideshare.net/AmazonWebServices/infrastructure-at-scale-apache-kafka-twitter-storm-elastic-search-arc303-aws-reinvent-2013
>>>>>> 
>>>>>> Any questions, let me know, though I can't promise I can answer
>>>>>> everything. Can't give the complete game away. :-)
>>>>>> 
>>>>>> As always, Kafka rocks!
>>>>>> 
>>>>>> Philip
>>> 
>>> 
>>> 
>>> -- 
>>> ___________________________________________
>>> Available at these partners:
>>> [image: CloudFlare | shopify | Bigcommerce]
>>> 
>>> SUREN HIRAMAN, VP TECHNOLOGY
>>> SOCIOCAST
>>> Simple. Powerful. Predictions.
>>> 
>>> 96 SPRING STREET, 7TH FLOOR
>>> NEW YORK, NY 10012
>>> O: (917) 525-2466 ext. 105
>>> F: 646.349.4063
>>> E: suren.hiraman@sociocast.com
>>> W: www.sociocast.com
>>> 
>>> Increase Conversion Rates up to 500%. Go to www.sociocast.com and enter
>>> your URL for a free trial!

Re: Loggly's use of Kafka on AWS

Posted by Timothy Chen <tn...@gmail.com>.
Hi Philip,

So I wonder if you guys hit disk perf problems with EBS? It seems quite common in the past but I haven't tried recently.

Also can you share how you guys deployed zookeeper in AWS so that a qurom is always available?

Tim

Sent from my iPhone

> On Dec 2, 2013, at 5:15 PM, Steve Morin <st...@gmail.com> wrote:
> 
> Philip this is definitely useful
> 
>> On Dec 2, 2013, at 14:55, Surendranauth Hiraman <su...@sociocast.com> wrote:
>> 
>> S Ahmed,
>> 
>> This combination of Kafka and Storm to process streaming data is becoming
>> pretty common. Definitely worth looking at.
>> 
>> The throughput will vary depending on your workload (cpu usage, etc.) and
>> if you're talking to a backend, of course. But it scales very well.
>> 
>> -Suren
>> 
>> 
>> 
>>> On Mon, Dec 2, 2013 at 5:49 PM, S Ahmed <sa...@gmail.com> wrote:
>>> 
>>> Interesting.  So twitter storm is used to basically process the messages on
>>> kafka?   I'll have to read-up on storm b/c I always thought the use case
>>> was a bit different.
>>> 
>>> 
>>> 
>>>> On Sun, Dec 1, 2013 at 9:59 PM, Joe Stein <jo...@stealth.ly> wrote:
>>>> 
>>>> Awesome Philip, thanks for sharing!
>>>> 
>>>>> On Sun, Dec 1, 2013 at 9:17 PM, Philip O'Toole <ph...@loggly.com>
>>>> wrote:
>>>> 
>>>>> A couple of us here at Loggly recently spoke at AWS reinvent, on how we
>>>>> use Kafka 0.72 in our ingestion pipeline. The slides are at the link
>>>> below,
>>>>> and may be of interest to people on this list.
>>> http://www.slideshare.net/AmazonWebServices/infrastructure-at-scale-apache-kafka-twitter-storm-elastic-search-arc303-aws-reinvent-2013
>>>>> 
>>>>> Any questions, let me know, though I can't promise I can answer
>>>>> everything. Can't give the complete game away. :-)
>>>>> 
>>>>> As always, Kafka rocks!
>>>>> 
>>>>> Philip
>> 
>> 
>> 
>> -- 
>> ___________________________________________
>> Available at these partners:
>> [image: CloudFlare | shopify | Bigcommerce]
>> 
>> SUREN HIRAMAN, VP TECHNOLOGY
>> SOCIOCAST
>> Simple. Powerful. Predictions.
>> 
>> 96 SPRING STREET, 7TH FLOOR
>> NEW YORK, NY 10012
>> O: (917) 525-2466 ext. 105
>> F: 646.349.4063
>> E: suren.hiraman@sociocast.com
>> W: www.sociocast.com
>> 
>> Increase Conversion Rates up to 500%. Go to www.sociocast.com and enter
>> your URL for a free trial!

Re: Loggly's use of Kafka on AWS

Posted by Steve Morin <st...@gmail.com>.
Philip this is definitely useful

> On Dec 2, 2013, at 14:55, Surendranauth Hiraman <su...@sociocast.com> wrote:
> 
> S Ahmed,
> 
> This combination of Kafka and Storm to process streaming data is becoming
> pretty common. Definitely worth looking at.
> 
> The throughput will vary depending on your workload (cpu usage, etc.) and
> if you're talking to a backend, of course. But it scales very well.
> 
> -Suren
> 
> 
> 
>> On Mon, Dec 2, 2013 at 5:49 PM, S Ahmed <sa...@gmail.com> wrote:
>> 
>> Interesting.  So twitter storm is used to basically process the messages on
>> kafka?   I'll have to read-up on storm b/c I always thought the use case
>> was a bit different.
>> 
>> 
>> 
>>> On Sun, Dec 1, 2013 at 9:59 PM, Joe Stein <jo...@stealth.ly> wrote:
>>> 
>>> Awesome Philip, thanks for sharing!
>>> 
>>> On Sun, Dec 1, 2013 at 9:17 PM, Philip O'Toole <ph...@loggly.com>
>> wrote:
>>> 
>>>> A couple of us here at Loggly recently spoke at AWS reinvent, on how we
>>>> use Kafka 0.72 in our ingestion pipeline. The slides are at the link
>>> below,
>>>> and may be of interest to people on this list.
>>>> 
>>>> 
>>>> 
>>> 
>> http://www.slideshare.net/AmazonWebServices/infrastructure-at-scale-apache-kafka-twitter-storm-elastic-search-arc303-aws-reinvent-2013
>>>> 
>>>> Any questions, let me know, though I can't promise I can answer
>>>> everything. Can't give the complete game away. :-)
>>>> 
>>>> As always, Kafka rocks!
>>>> 
>>>> Philip
>>>> 
>>>> 
>>>> 
>>>> 
>>> 
>> 
> 
> 
> 
> -- 
> ___________________________________________
> Available at these partners:
> [image: CloudFlare | shopify | Bigcommerce]
> 
> SUREN HIRAMAN, VP TECHNOLOGY
> SOCIOCAST
> Simple. Powerful. Predictions.
> 
> 96 SPRING STREET, 7TH FLOOR
> NEW YORK, NY 10012
> O: (917) 525-2466 ext. 105
> F: 646.349.4063
> E: suren.hiraman@sociocast.com
> W: www.sociocast.com
> 
> Increase Conversion Rates up to 500%. Go to www.sociocast.com and enter
> your URL for a free trial!

Re: Loggly's use of Kafka on AWS

Posted by Surendranauth Hiraman <su...@sociocast.com>.
S Ahmed,

This combination of Kafka and Storm to process streaming data is becoming
pretty common. Definitely worth looking at.

The throughput will vary depending on your workload (cpu usage, etc.) and
if you're talking to a backend, of course. But it scales very well.

-Suren



On Mon, Dec 2, 2013 at 5:49 PM, S Ahmed <sa...@gmail.com> wrote:

> Interesting.  So twitter storm is used to basically process the messages on
> kafka?   I'll have to read-up on storm b/c I always thought the use case
> was a bit different.
>
>
>
> On Sun, Dec 1, 2013 at 9:59 PM, Joe Stein <jo...@stealth.ly> wrote:
>
> > Awesome Philip, thanks for sharing!
> >
> > On Sun, Dec 1, 2013 at 9:17 PM, Philip O'Toole <ph...@loggly.com>
> wrote:
> >
> > > A couple of us here at Loggly recently spoke at AWS reinvent, on how we
> > > use Kafka 0.72 in our ingestion pipeline. The slides are at the link
> > below,
> > > and may be of interest to people on this list.
> > >
> > >
> > >
> >
> http://www.slideshare.net/AmazonWebServices/infrastructure-at-scale-apache-kafka-twitter-storm-elastic-search-arc303-aws-reinvent-2013
> > >
> > > Any questions, let me know, though I can't promise I can answer
> > > everything. Can't give the complete game away. :-)
> > >
> > > As always, Kafka rocks!
> > >
> > > Philip
> > >
> > >
> > >
> > >
> >
>



-- 
___________________________________________
Available at these partners:
[image: CloudFlare | shopify | Bigcommerce]

SUREN HIRAMAN, VP TECHNOLOGY
SOCIOCAST
Simple. Powerful. Predictions.

96 SPRING STREET, 7TH FLOOR
NEW YORK, NY 10012
O: (917) 525-2466 ext. 105
F: 646.349.4063
E: suren.hiraman@sociocast.com
W: www.sociocast.com

Increase Conversion Rates up to 500%. Go to www.sociocast.com and enter
your URL for a free trial!

Re: Loggly's use of Kafka on AWS

Posted by S Ahmed <sa...@gmail.com>.
Interesting.  So twitter storm is used to basically process the messages on
kafka?   I'll have to read-up on storm b/c I always thought the use case
was a bit different.



On Sun, Dec 1, 2013 at 9:59 PM, Joe Stein <jo...@stealth.ly> wrote:

> Awesome Philip, thanks for sharing!
>
> On Sun, Dec 1, 2013 at 9:17 PM, Philip O'Toole <ph...@loggly.com> wrote:
>
> > A couple of us here at Loggly recently spoke at AWS reinvent, on how we
> > use Kafka 0.72 in our ingestion pipeline. The slides are at the link
> below,
> > and may be of interest to people on this list.
> >
> >
> >
> http://www.slideshare.net/AmazonWebServices/infrastructure-at-scale-apache-kafka-twitter-storm-elastic-search-arc303-aws-reinvent-2013
> >
> > Any questions, let me know, though I can't promise I can answer
> > everything. Can't give the complete game away. :-)
> >
> > As always, Kafka rocks!
> >
> > Philip
> >
> >
> >
> >
>

Re: Loggly's use of Kafka on AWS

Posted by Joe Stein <jo...@stealth.ly>.
Awesome Philip, thanks for sharing!

On Sun, Dec 1, 2013 at 9:17 PM, Philip O'Toole <ph...@loggly.com> wrote:

> A couple of us here at Loggly recently spoke at AWS reinvent, on how we
> use Kafka 0.72 in our ingestion pipeline. The slides are at the link below,
> and may be of interest to people on this list.
>
>
> http://www.slideshare.net/AmazonWebServices/infrastructure-at-scale-apache-kafka-twitter-storm-elastic-search-arc303-aws-reinvent-2013
>
> Any questions, let me know, though I can't promise I can answer
> everything. Can't give the complete game away. :-)
>
> As always, Kafka rocks!
>
> Philip
>
>
>
>