Is there anyone who have experience of using storm with druid?

[이승진 <sw...@navercorp.com>]

Hello all,
 
We are about to implement aggregation feature in our system and for that we think it's a good option to use Druid with Storm. 
 
If any of you have experience, I want to here from you about these things.
 
- data feeding : Do you feed data from kafka to druid directly or use Storm as a intermeidate refiner?
- Did you develop custom druid driver for that?
- any inconvenience during implementation?
 
Thanks in advance

Re: Is there anyone who have experience of using storm with druid?

[Brian O'Neill <bo...@alumni.brown.edu>]

Ironically, we dedicated two chapters to a Storm/Druid integration when we
wrote the book:
http://www.amazon.com/Storm-Blueprints-Distributed-Real-time-Computation/dp/
178216829X/

If you want to contact me directly, I’d be interested in hearing more about
your use case, and what specifically you are looking to do.  (I might also
have code you can use)

-brian


---
Brian O'Neill 
Chief Technology Officer
Health Market Science, a LexisNexis Company
215.588.6024 Mobile ? @boneill42 <http://www.twitter.com/boneill42>


This information transmitted in this email message is for the intended
recipient only and may contain confidential and/or privileged material. If
you received this email in error and are not the intended recipient, or the
person responsible to deliver it to the intended recipient, please contact
the sender at the email above and delete this email and any attachments and
destroy any copies thereof. Any review, retransmission, dissemination,
copying or other use of, or taking any action in reliance upon, this
information by persons or entities other than the intended recipient is
strictly prohibited.
 


From:  Andrew Neilson <ar...@gmail.com>
Reply-To:  <us...@storm.apache.org>
Date:  Wednesday, February 4, 2015 at 3:49 PM
To:  <us...@storm.apache.org>, 이승진 <sw...@navercorp.com>
Subject:  Re: Is there anyone who have experience of using storm with druid?

I can't answer your question directly as I haven't used Druid, but I will
point out that Druid was developed by Metamarkets and they are using Druid
with Storm and Kafka in their architecture
(https://metamarkets.com/2014/building-a-data-pipeline-that-handles-billions
-of-events-in-real-time/). They've open-sourced Tranquility, which includes
components for streaming data from Storm to Druid:

https://github.com/metamx/tranquility

Hope this helps.

On Tue, Feb 3, 2015 at 7:17 PM, 이승진 <sw...@navercorp.com> wrote:
> Hello all,
> 
>  
> 
> We are about to implement aggregation feature in our system and for that we
> think it's a good option to use Druid with Storm.
> 
>  
> 
> If any of you have experience, I want to here from you about these things.
> 
>  
> 
> - data feeding : Do you feed data from kafka to druid directly or use Storm as
> a intermeidate refiner?
> 
> - Did you develop custom druid driver for that?
> 
> - any inconvenience during implementation?
> 
>  
> 
> Thanks in advance
> 

Re: Is there anyone who have experience of using storm with druid?

[Andrew Neilson <ar...@gmail.com>]

I can't answer your question directly as I haven't used Druid, but I will
point out that Druid was developed by Metamarkets and they are using Druid
with Storm and Kafka in their architecture (
https://metamarkets.com/2014/building-a-data-pipeline-that-handles-billions-of-events-in-real-time/).
They've open-sourced Tranquility, which includes components for streaming
data from Storm to Druid:

https://github.com/metamx/tranquility

Hope this helps.

On Tue, Feb 3, 2015 at 7:17 PM, 이승진 <sw...@navercorp.com> wrote:

> Hello all,
>
>
>
> We are about to implement aggregation feature in our system and for that
> we think it's a good option to use Druid with Storm.
>
>
>
> If any of you have experience, I want to here from you about these things.
>
>
>
> - data feeding : Do you feed data from kafka to druid directly or use
> Storm as a intermeidate refiner?
>
> - Did you develop custom druid driver for that?
>
> - any inconvenience during implementation?
>
>
>
> Thanks in advance
>