You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@storm.apache.org by Satish Mittal <sa...@inmobi.com> on 2015/10/19 13:24:20 UTC
Storm vs Spark Streaming Tech Evaluation
Hi All,
The data platform team at Inmobi recently performed an extensive evaluation
exercise in the process of finalizing the real-time Stream processing stack
as the choice of our platform.
We have captured all the details of our evaluation as the following series
of 4 blogs which have been published at Inmobi technology site:
1) Introduction; Identifying stream processing use-cases at Inmobi;
Identifying potential Technology Candidates. In the interest of time, we
limited the
http://technology.inmobi.com/blog/real-time-stream-processing-at-inmobi-part-1
2) Detailed overview of Storm and Spark Streaming platforms
http://technology.inmobi.com/blog/real-time-stream-processing-at-inmobi-part-2
3) Identify and define various important evaluation criteria
http://technology.inmobi.com/blog/real-time-stream-processing-at-inmobi-part-3
4) Detailed findings on various evaluation criteria, evaluation summary
along with the final recommendation.
http://technology.inmobi.com/blog/real-time-stream-processing-at-inmobi-part-4
We hope that this analysis would be useful in general to anyone who is
starting to explore the world of real-time stream processing and decide
upon a particular tech stack.
Please go through the blogs and let us know your thoughts!
Regards,
Satish
--
_____________________________________________________________
The information contained in this communication is intended solely for the
use of the individual or entity to whom it is addressed and others
authorized to receive it. It may contain confidential or legally privileged
information. If you are not the intended recipient you are hereby notified
that any disclosure, copying, distribution or taking any action in reliance
on the contents of this information is strictly prohibited and may be
unlawful. If you have received this communication in error, please notify
us immediately by responding to this email and then delete it from your
system. The firm is neither liable for the proper and complete transmission
of the information contained in this communication nor for any delay in its
receipt.
Re: Storm vs Spark Streaming Tech Evaluation
Posted by padma priya chitturi <pa...@gmail.com>.
Yes, its a functional language but industry adoption is very very less.
On Wed, Oct 21, 2015 at 4:21 PM, Javier Gonzalez <ja...@gmail.com> wrote:
> Clojure is, AFAIK, a functional language. ;)
> On Oct 20, 2015 11:42 PM, "padma priya chitturi" <pa...@gmail.com>
> wrote:
>
>> Very nice post :) Storm is very good in terms of the capabilities it has.
>> The only thing is they could have provided API in Scala as well as Python.
>> Also, when debugging, understanding clojure code is bit hard. They should
>> have written code using functional programming language like Scala.
>>
>> On Wed, Oct 21, 2015 at 1:18 AM, Ankur Garg <an...@gmail.com> wrote:
>>
>>> Very Nice post Satish . Thanks for sharing :) .
>>>
>>> On Wed, Oct 21, 2015 at 12:33 AM, P. Taylor Goetz <pt...@gmail.com>
>>> wrote:
>>>
>>>> Hi Satish,
>>>>
>>>> Great series of blog posts. Thanks for sharing!
>>>>
>>>> -Taylor
>>>>
>>>> On Oct 19, 2015, at 4:24 AM, Satish Mittal <sa...@inmobi.com>
>>>> wrote:
>>>>
>>>> Hi All,
>>>>
>>>> The data platform team at Inmobi recently performed an extensive
>>>> evaluation exercise in the process of finalizing the real-time Stream
>>>> processing stack as the choice of our platform.
>>>>
>>>> We have captured all the details of our evaluation as the following
>>>> series of 4 blogs which have been published at Inmobi technology site:
>>>>
>>>> 1) Introduction; Identifying stream processing use-cases at Inmobi;
>>>> Identifying potential Technology Candidates. In the interest of time, we
>>>> limited the
>>>>
>>>> http://technology.inmobi.com/blog/real-time-stream-processing-at-inmobi-part-1
>>>>
>>>> 2) Detailed overview of Storm and Spark Streaming platforms
>>>>
>>>> http://technology.inmobi.com/blog/real-time-stream-processing-at-inmobi-part-2
>>>>
>>>> 3) Identify and define various important evaluation criteria
>>>>
>>>> http://technology.inmobi.com/blog/real-time-stream-processing-at-inmobi-part-3
>>>>
>>>> 4) Detailed findings on various evaluation criteria, evaluation summary
>>>> along with the final recommendation.
>>>>
>>>> http://technology.inmobi.com/blog/real-time-stream-processing-at-inmobi-part-4
>>>>
>>>> We hope that this analysis would be useful in general to anyone who is
>>>> starting to explore the world of real-time stream processing and decide
>>>> upon a particular tech stack.
>>>>
>>>> Please go through the blogs and let us know your thoughts!
>>>>
>>>> Regards,
>>>> Satish
>>>>
>>>>
>>>>
>>>>
>>>>
>>>> _____________________________________________________________
>>>> The information contained in this communication is intended solely for
>>>> the use of the individual or entity to whom it is addressed and others
>>>> authorized to receive it. It may contain confidential or legally privileged
>>>> information. If you are not the intended recipient you are hereby notified
>>>> that any disclosure, copying, distribution or taking any action in reliance
>>>> on the contents of this information is strictly prohibited and may be
>>>> unlawful. If you have received this communication in error, please notify
>>>> us immediately by responding to this email and then delete it from your
>>>> system. The firm is neither liable for the proper and complete transmission
>>>> of the information contained in this communication nor for any delay in its
>>>> receipt.
>>>>
>>>>
>>>>
>>>
>>
Re: Storm vs Spark Streaming Tech Evaluation
Posted by Javier Gonzalez <ja...@gmail.com>.
Clojure is, AFAIK, a functional language. ;)
On Oct 20, 2015 11:42 PM, "padma priya chitturi" <pa...@gmail.com>
wrote:
> Very nice post :) Storm is very good in terms of the capabilities it has.
> The only thing is they could have provided API in Scala as well as Python.
> Also, when debugging, understanding clojure code is bit hard. They should
> have written code using functional programming language like Scala.
>
> On Wed, Oct 21, 2015 at 1:18 AM, Ankur Garg <an...@gmail.com> wrote:
>
>> Very Nice post Satish . Thanks for sharing :) .
>>
>> On Wed, Oct 21, 2015 at 12:33 AM, P. Taylor Goetz <pt...@gmail.com>
>> wrote:
>>
>>> Hi Satish,
>>>
>>> Great series of blog posts. Thanks for sharing!
>>>
>>> -Taylor
>>>
>>> On Oct 19, 2015, at 4:24 AM, Satish Mittal <sa...@inmobi.com>
>>> wrote:
>>>
>>> Hi All,
>>>
>>> The data platform team at Inmobi recently performed an extensive
>>> evaluation exercise in the process of finalizing the real-time Stream
>>> processing stack as the choice of our platform.
>>>
>>> We have captured all the details of our evaluation as the following
>>> series of 4 blogs which have been published at Inmobi technology site:
>>>
>>> 1) Introduction; Identifying stream processing use-cases at Inmobi;
>>> Identifying potential Technology Candidates. In the interest of time, we
>>> limited the
>>>
>>> http://technology.inmobi.com/blog/real-time-stream-processing-at-inmobi-part-1
>>>
>>> 2) Detailed overview of Storm and Spark Streaming platforms
>>>
>>> http://technology.inmobi.com/blog/real-time-stream-processing-at-inmobi-part-2
>>>
>>> 3) Identify and define various important evaluation criteria
>>>
>>> http://technology.inmobi.com/blog/real-time-stream-processing-at-inmobi-part-3
>>>
>>> 4) Detailed findings on various evaluation criteria, evaluation summary
>>> along with the final recommendation.
>>>
>>> http://technology.inmobi.com/blog/real-time-stream-processing-at-inmobi-part-4
>>>
>>> We hope that this analysis would be useful in general to anyone who is
>>> starting to explore the world of real-time stream processing and decide
>>> upon a particular tech stack.
>>>
>>> Please go through the blogs and let us know your thoughts!
>>>
>>> Regards,
>>> Satish
>>>
>>>
>>>
>>>
>>>
>>> _____________________________________________________________
>>> The information contained in this communication is intended solely for
>>> the use of the individual or entity to whom it is addressed and others
>>> authorized to receive it. It may contain confidential or legally privileged
>>> information. If you are not the intended recipient you are hereby notified
>>> that any disclosure, copying, distribution or taking any action in reliance
>>> on the contents of this information is strictly prohibited and may be
>>> unlawful. If you have received this communication in error, please notify
>>> us immediately by responding to this email and then delete it from your
>>> system. The firm is neither liable for the proper and complete transmission
>>> of the information contained in this communication nor for any delay in its
>>> receipt.
>>>
>>>
>>>
>>
>
Re: Storm vs Spark Streaming Tech Evaluation
Posted by padma priya chitturi <pa...@gmail.com>.
Very nice post :) Storm is very good in terms of the capabilities it has.
The only thing is they could have provided API in Scala as well as Python.
Also, when debugging, understanding clojure code is bit hard. They should
have written code using functional programming language like Scala.
On Wed, Oct 21, 2015 at 1:18 AM, Ankur Garg <an...@gmail.com> wrote:
> Very Nice post Satish . Thanks for sharing :) .
>
> On Wed, Oct 21, 2015 at 12:33 AM, P. Taylor Goetz <pt...@gmail.com>
> wrote:
>
>> Hi Satish,
>>
>> Great series of blog posts. Thanks for sharing!
>>
>> -Taylor
>>
>> On Oct 19, 2015, at 4:24 AM, Satish Mittal <sa...@inmobi.com>
>> wrote:
>>
>> Hi All,
>>
>> The data platform team at Inmobi recently performed an extensive
>> evaluation exercise in the process of finalizing the real-time Stream
>> processing stack as the choice of our platform.
>>
>> We have captured all the details of our evaluation as the following
>> series of 4 blogs which have been published at Inmobi technology site:
>>
>> 1) Introduction; Identifying stream processing use-cases at Inmobi;
>> Identifying potential Technology Candidates. In the interest of time, we
>> limited the
>>
>> http://technology.inmobi.com/blog/real-time-stream-processing-at-inmobi-part-1
>>
>> 2) Detailed overview of Storm and Spark Streaming platforms
>>
>> http://technology.inmobi.com/blog/real-time-stream-processing-at-inmobi-part-2
>>
>> 3) Identify and define various important evaluation criteria
>>
>> http://technology.inmobi.com/blog/real-time-stream-processing-at-inmobi-part-3
>>
>> 4) Detailed findings on various evaluation criteria, evaluation summary
>> along with the final recommendation.
>>
>> http://technology.inmobi.com/blog/real-time-stream-processing-at-inmobi-part-4
>>
>> We hope that this analysis would be useful in general to anyone who is
>> starting to explore the world of real-time stream processing and decide
>> upon a particular tech stack.
>>
>> Please go through the blogs and let us know your thoughts!
>>
>> Regards,
>> Satish
>>
>>
>>
>>
>>
>> _____________________________________________________________
>> The information contained in this communication is intended solely for
>> the use of the individual or entity to whom it is addressed and others
>> authorized to receive it. It may contain confidential or legally privileged
>> information. If you are not the intended recipient you are hereby notified
>> that any disclosure, copying, distribution or taking any action in reliance
>> on the contents of this information is strictly prohibited and may be
>> unlawful. If you have received this communication in error, please notify
>> us immediately by responding to this email and then delete it from your
>> system. The firm is neither liable for the proper and complete transmission
>> of the information contained in this communication nor for any delay in its
>> receipt.
>>
>>
>>
>
Re: Storm vs Spark Streaming Tech Evaluation
Posted by Ankur Garg <an...@gmail.com>.
Very Nice post Satish . Thanks for sharing :) .
On Wed, Oct 21, 2015 at 12:33 AM, P. Taylor Goetz <pt...@gmail.com> wrote:
> Hi Satish,
>
> Great series of blog posts. Thanks for sharing!
>
> -Taylor
>
> On Oct 19, 2015, at 4:24 AM, Satish Mittal <sa...@inmobi.com>
> wrote:
>
> Hi All,
>
> The data platform team at Inmobi recently performed an extensive
> evaluation exercise in the process of finalizing the real-time Stream
> processing stack as the choice of our platform.
>
> We have captured all the details of our evaluation as the following series
> of 4 blogs which have been published at Inmobi technology site:
>
> 1) Introduction; Identifying stream processing use-cases at Inmobi;
> Identifying potential Technology Candidates. In the interest of time, we
> limited the
>
> http://technology.inmobi.com/blog/real-time-stream-processing-at-inmobi-part-1
>
> 2) Detailed overview of Storm and Spark Streaming platforms
>
> http://technology.inmobi.com/blog/real-time-stream-processing-at-inmobi-part-2
>
> 3) Identify and define various important evaluation criteria
>
> http://technology.inmobi.com/blog/real-time-stream-processing-at-inmobi-part-3
>
> 4) Detailed findings on various evaluation criteria, evaluation summary
> along with the final recommendation.
>
> http://technology.inmobi.com/blog/real-time-stream-processing-at-inmobi-part-4
>
> We hope that this analysis would be useful in general to anyone who is
> starting to explore the world of real-time stream processing and decide
> upon a particular tech stack.
>
> Please go through the blogs and let us know your thoughts!
>
> Regards,
> Satish
>
>
>
>
>
> _____________________________________________________________
> The information contained in this communication is intended solely for the
> use of the individual or entity to whom it is addressed and others
> authorized to receive it. It may contain confidential or legally privileged
> information. If you are not the intended recipient you are hereby notified
> that any disclosure, copying, distribution or taking any action in reliance
> on the contents of this information is strictly prohibited and may be
> unlawful. If you have received this communication in error, please notify
> us immediately by responding to this email and then delete it from your
> system. The firm is neither liable for the proper and complete transmission
> of the information contained in this communication nor for any delay in its
> receipt.
>
>
>
Re: Storm vs Spark Streaming Tech Evaluation
Posted by padma priya chitturi <pa...@gmail.com>.
That's very useful information. What is the DB store you guys were using ??
Is it Cassandra. Although Storm has very good capabilities, I see the
development of db connectors or version releases is quite slow when
compared to Spark Streaming. If I look at storm-cassandra connector, it has
bee 2 years since its updated (https://github.com/hmsonline/storm-cassandra).
Why is it that slow in making progress ??
On Sat, Oct 24, 2015 at 12:56 AM, Satish Mittal <sa...@inmobi.com>
wrote:
> Thanks!
>
> On Wed, Oct 21, 2015 at 12:33 AM, P. Taylor Goetz <pt...@gmail.com>
> wrote:
>
>> Hi Satish,
>>
>> Great series of blog posts. Thanks for sharing!
>>
>> -Taylor
>>
>> On Oct 19, 2015, at 4:24 AM, Satish Mittal <sa...@inmobi.com>
>> wrote:
>>
>> Hi All,
>>
>> The data platform team at Inmobi recently performed an extensive
>> evaluation exercise in the process of finalizing the real-time Stream
>> processing stack as the choice of our platform.
>>
>> We have captured all the details of our evaluation as the following
>> series of 4 blogs which have been published at Inmobi technology site:
>>
>> 1) Introduction; Identifying stream processing use-cases at Inmobi;
>> Identifying potential Technology Candidates. In the interest of time, we
>> limited the
>>
>> http://technology.inmobi.com/blog/real-time-stream-processing-at-inmobi-part-1
>>
>> 2) Detailed overview of Storm and Spark Streaming platforms
>>
>> http://technology.inmobi.com/blog/real-time-stream-processing-at-inmobi-part-2
>>
>> 3) Identify and define various important evaluation criteria
>>
>> http://technology.inmobi.com/blog/real-time-stream-processing-at-inmobi-part-3
>>
>> 4) Detailed findings on various evaluation criteria, evaluation summary
>> along with the final recommendation.
>>
>> http://technology.inmobi.com/blog/real-time-stream-processing-at-inmobi-part-4
>>
>> We hope that this analysis would be useful in general to anyone who is
>> starting to explore the world of real-time stream processing and decide
>> upon a particular tech stack.
>>
>> Please go through the blogs and let us know your thoughts!
>>
>> Regards,
>> Satish
>>
>>
>>
>>
>>
>> _____________________________________________________________
>> The information contained in this communication is intended solely for
>> the use of the individual or entity to whom it is addressed and others
>> authorized to receive it. It may contain confidential or legally privileged
>> information. If you are not the intended recipient you are hereby notified
>> that any disclosure, copying, distribution or taking any action in reliance
>> on the contents of this information is strictly prohibited and may be
>> unlawful. If you have received this communication in error, please notify
>> us immediately by responding to this email and then delete it from your
>> system. The firm is neither liable for the proper and complete transmission
>> of the information contained in this communication nor for any delay in its
>> receipt.
>>
>>
>>
>
> _____________________________________________________________
> The information contained in this communication is intended solely for the
> use of the individual or entity to whom it is addressed and others
> authorized to receive it. It may contain confidential or legally privileged
> information. If you are not the intended recipient you are hereby notified
> that any disclosure, copying, distribution or taking any action in reliance
> on the contents of this information is strictly prohibited and may be
> unlawful. If you have received this communication in error, please notify
> us immediately by responding to this email and then delete it from your
> system. The firm is neither liable for the proper and complete transmission
> of the information contained in this communication nor for any delay in its
> receipt.
>
Re: Storm vs Spark Streaming Tech Evaluation
Posted by padma priya chitturi <pa...@gmail.com>.
That's very useful information. What is the DB store you guys were using ??
Is it Cassandra. Although Storm has very good capabilities, I see the
development of db connectors or version releases is quite slow when
compared to Spark Streaming. If I look at storm-cassandra connector, it has
bee 2 years since its updated (https://github.com/hmsonline/storm-cassandra).
Why is it that slow in making progress ??
On Sat, Oct 24, 2015 at 12:56 AM, Satish Mittal <sa...@inmobi.com>
wrote:
> Thanks!
>
> On Wed, Oct 21, 2015 at 12:33 AM, P. Taylor Goetz <pt...@gmail.com>
> wrote:
>
>> Hi Satish,
>>
>> Great series of blog posts. Thanks for sharing!
>>
>> -Taylor
>>
>> On Oct 19, 2015, at 4:24 AM, Satish Mittal <sa...@inmobi.com>
>> wrote:
>>
>> Hi All,
>>
>> The data platform team at Inmobi recently performed an extensive
>> evaluation exercise in the process of finalizing the real-time Stream
>> processing stack as the choice of our platform.
>>
>> We have captured all the details of our evaluation as the following
>> series of 4 blogs which have been published at Inmobi technology site:
>>
>> 1) Introduction; Identifying stream processing use-cases at Inmobi;
>> Identifying potential Technology Candidates. In the interest of time, we
>> limited the
>>
>> http://technology.inmobi.com/blog/real-time-stream-processing-at-inmobi-part-1
>>
>> 2) Detailed overview of Storm and Spark Streaming platforms
>>
>> http://technology.inmobi.com/blog/real-time-stream-processing-at-inmobi-part-2
>>
>> 3) Identify and define various important evaluation criteria
>>
>> http://technology.inmobi.com/blog/real-time-stream-processing-at-inmobi-part-3
>>
>> 4) Detailed findings on various evaluation criteria, evaluation summary
>> along with the final recommendation.
>>
>> http://technology.inmobi.com/blog/real-time-stream-processing-at-inmobi-part-4
>>
>> We hope that this analysis would be useful in general to anyone who is
>> starting to explore the world of real-time stream processing and decide
>> upon a particular tech stack.
>>
>> Please go through the blogs and let us know your thoughts!
>>
>> Regards,
>> Satish
>>
>>
>>
>>
>>
>> _____________________________________________________________
>> The information contained in this communication is intended solely for
>> the use of the individual or entity to whom it is addressed and others
>> authorized to receive it. It may contain confidential or legally privileged
>> information. If you are not the intended recipient you are hereby notified
>> that any disclosure, copying, distribution or taking any action in reliance
>> on the contents of this information is strictly prohibited and may be
>> unlawful. If you have received this communication in error, please notify
>> us immediately by responding to this email and then delete it from your
>> system. The firm is neither liable for the proper and complete transmission
>> of the information contained in this communication nor for any delay in its
>> receipt.
>>
>>
>>
>
> _____________________________________________________________
> The information contained in this communication is intended solely for the
> use of the individual or entity to whom it is addressed and others
> authorized to receive it. It may contain confidential or legally privileged
> information. If you are not the intended recipient you are hereby notified
> that any disclosure, copying, distribution or taking any action in reliance
> on the contents of this information is strictly prohibited and may be
> unlawful. If you have received this communication in error, please notify
> us immediately by responding to this email and then delete it from your
> system. The firm is neither liable for the proper and complete transmission
> of the information contained in this communication nor for any delay in its
> receipt.
>
Re: Storm vs Spark Streaming Tech Evaluation
Posted by Satish Mittal <sa...@inmobi.com>.
Thanks!
On Wed, Oct 21, 2015 at 12:33 AM, P. Taylor Goetz <pt...@gmail.com> wrote:
> Hi Satish,
>
> Great series of blog posts. Thanks for sharing!
>
> -Taylor
>
> On Oct 19, 2015, at 4:24 AM, Satish Mittal <sa...@inmobi.com>
> wrote:
>
> Hi All,
>
> The data platform team at Inmobi recently performed an extensive
> evaluation exercise in the process of finalizing the real-time Stream
> processing stack as the choice of our platform.
>
> We have captured all the details of our evaluation as the following series
> of 4 blogs which have been published at Inmobi technology site:
>
> 1) Introduction; Identifying stream processing use-cases at Inmobi;
> Identifying potential Technology Candidates. In the interest of time, we
> limited the
>
> http://technology.inmobi.com/blog/real-time-stream-processing-at-inmobi-part-1
>
> 2) Detailed overview of Storm and Spark Streaming platforms
>
> http://technology.inmobi.com/blog/real-time-stream-processing-at-inmobi-part-2
>
> 3) Identify and define various important evaluation criteria
>
> http://technology.inmobi.com/blog/real-time-stream-processing-at-inmobi-part-3
>
> 4) Detailed findings on various evaluation criteria, evaluation summary
> along with the final recommendation.
>
> http://technology.inmobi.com/blog/real-time-stream-processing-at-inmobi-part-4
>
> We hope that this analysis would be useful in general to anyone who is
> starting to explore the world of real-time stream processing and decide
> upon a particular tech stack.
>
> Please go through the blogs and let us know your thoughts!
>
> Regards,
> Satish
>
>
>
>
>
> _____________________________________________________________
> The information contained in this communication is intended solely for the
> use of the individual or entity to whom it is addressed and others
> authorized to receive it. It may contain confidential or legally privileged
> information. If you are not the intended recipient you are hereby notified
> that any disclosure, copying, distribution or taking any action in reliance
> on the contents of this information is strictly prohibited and may be
> unlawful. If you have received this communication in error, please notify
> us immediately by responding to this email and then delete it from your
> system. The firm is neither liable for the proper and complete transmission
> of the information contained in this communication nor for any delay in its
> receipt.
>
>
>
--
_____________________________________________________________
The information contained in this communication is intended solely for the
use of the individual or entity to whom it is addressed and others
authorized to receive it. It may contain confidential or legally privileged
information. If you are not the intended recipient you are hereby notified
that any disclosure, copying, distribution or taking any action in reliance
on the contents of this information is strictly prohibited and may be
unlawful. If you have received this communication in error, please notify
us immediately by responding to this email and then delete it from your
system. The firm is neither liable for the proper and complete transmission
of the information contained in this communication nor for any delay in its
receipt.
Re: Storm vs Spark Streaming Tech Evaluation
Posted by Ankur Garg <an...@gmail.com>.
Very Nice post Satish . Thanks for sharing :) .
On Wed, Oct 21, 2015 at 12:33 AM, P. Taylor Goetz <pt...@gmail.com> wrote:
> Hi Satish,
>
> Great series of blog posts. Thanks for sharing!
>
> -Taylor
>
> On Oct 19, 2015, at 4:24 AM, Satish Mittal <sa...@inmobi.com>
> wrote:
>
> Hi All,
>
> The data platform team at Inmobi recently performed an extensive
> evaluation exercise in the process of finalizing the real-time Stream
> processing stack as the choice of our platform.
>
> We have captured all the details of our evaluation as the following series
> of 4 blogs which have been published at Inmobi technology site:
>
> 1) Introduction; Identifying stream processing use-cases at Inmobi;
> Identifying potential Technology Candidates. In the interest of time, we
> limited the
>
> http://technology.inmobi.com/blog/real-time-stream-processing-at-inmobi-part-1
>
> 2) Detailed overview of Storm and Spark Streaming platforms
>
> http://technology.inmobi.com/blog/real-time-stream-processing-at-inmobi-part-2
>
> 3) Identify and define various important evaluation criteria
>
> http://technology.inmobi.com/blog/real-time-stream-processing-at-inmobi-part-3
>
> 4) Detailed findings on various evaluation criteria, evaluation summary
> along with the final recommendation.
>
> http://technology.inmobi.com/blog/real-time-stream-processing-at-inmobi-part-4
>
> We hope that this analysis would be useful in general to anyone who is
> starting to explore the world of real-time stream processing and decide
> upon a particular tech stack.
>
> Please go through the blogs and let us know your thoughts!
>
> Regards,
> Satish
>
>
>
>
>
> _____________________________________________________________
> The information contained in this communication is intended solely for the
> use of the individual or entity to whom it is addressed and others
> authorized to receive it. It may contain confidential or legally privileged
> information. If you are not the intended recipient you are hereby notified
> that any disclosure, copying, distribution or taking any action in reliance
> on the contents of this information is strictly prohibited and may be
> unlawful. If you have received this communication in error, please notify
> us immediately by responding to this email and then delete it from your
> system. The firm is neither liable for the proper and complete transmission
> of the information contained in this communication nor for any delay in its
> receipt.
>
>
>
Re: Storm vs Spark Streaming Tech Evaluation
Posted by Satish Mittal <sa...@inmobi.com>.
Thanks!
On Wed, Oct 21, 2015 at 12:33 AM, P. Taylor Goetz <pt...@gmail.com> wrote:
> Hi Satish,
>
> Great series of blog posts. Thanks for sharing!
>
> -Taylor
>
> On Oct 19, 2015, at 4:24 AM, Satish Mittal <sa...@inmobi.com>
> wrote:
>
> Hi All,
>
> The data platform team at Inmobi recently performed an extensive
> evaluation exercise in the process of finalizing the real-time Stream
> processing stack as the choice of our platform.
>
> We have captured all the details of our evaluation as the following series
> of 4 blogs which have been published at Inmobi technology site:
>
> 1) Introduction; Identifying stream processing use-cases at Inmobi;
> Identifying potential Technology Candidates. In the interest of time, we
> limited the
>
> http://technology.inmobi.com/blog/real-time-stream-processing-at-inmobi-part-1
>
> 2) Detailed overview of Storm and Spark Streaming platforms
>
> http://technology.inmobi.com/blog/real-time-stream-processing-at-inmobi-part-2
>
> 3) Identify and define various important evaluation criteria
>
> http://technology.inmobi.com/blog/real-time-stream-processing-at-inmobi-part-3
>
> 4) Detailed findings on various evaluation criteria, evaluation summary
> along with the final recommendation.
>
> http://technology.inmobi.com/blog/real-time-stream-processing-at-inmobi-part-4
>
> We hope that this analysis would be useful in general to anyone who is
> starting to explore the world of real-time stream processing and decide
> upon a particular tech stack.
>
> Please go through the blogs and let us know your thoughts!
>
> Regards,
> Satish
>
>
>
>
>
> _____________________________________________________________
> The information contained in this communication is intended solely for the
> use of the individual or entity to whom it is addressed and others
> authorized to receive it. It may contain confidential or legally privileged
> information. If you are not the intended recipient you are hereby notified
> that any disclosure, copying, distribution or taking any action in reliance
> on the contents of this information is strictly prohibited and may be
> unlawful. If you have received this communication in error, please notify
> us immediately by responding to this email and then delete it from your
> system. The firm is neither liable for the proper and complete transmission
> of the information contained in this communication nor for any delay in its
> receipt.
>
>
>
--
_____________________________________________________________
The information contained in this communication is intended solely for the
use of the individual or entity to whom it is addressed and others
authorized to receive it. It may contain confidential or legally privileged
information. If you are not the intended recipient you are hereby notified
that any disclosure, copying, distribution or taking any action in reliance
on the contents of this information is strictly prohibited and may be
unlawful. If you have received this communication in error, please notify
us immediately by responding to this email and then delete it from your
system. The firm is neither liable for the proper and complete transmission
of the information contained in this communication nor for any delay in its
receipt.
Re: Storm vs Spark Streaming Tech Evaluation
Posted by "P. Taylor Goetz" <pt...@gmail.com>.
Hi Satish,
Great series of blog posts. Thanks for sharing!
-Taylor
> On Oct 19, 2015, at 4:24 AM, Satish Mittal <sa...@inmobi.com> wrote:
>
> Hi All,
>
> The data platform team at Inmobi recently performed an extensive evaluation exercise in the process of finalizing the real-time Stream processing stack as the choice of our platform.
>
> We have captured all the details of our evaluation as the following series of 4 blogs which have been published at Inmobi technology site:
>
> 1) Introduction; Identifying stream processing use-cases at Inmobi; Identifying potential Technology Candidates. In the interest of time, we limited the
> http://technology.inmobi.com/blog/real-time-stream-processing-at-inmobi-part-1 <http://technology.inmobi.com/blog/real-time-stream-processing-at-inmobi-part-1>
>
> 2) Detailed overview of Storm and Spark Streaming platforms
> http://technology.inmobi.com/blog/real-time-stream-processing-at-inmobi-part-2 <http://technology.inmobi.com/blog/real-time-stream-processing-at-inmobi-part-2>
>
> 3) Identify and define various important evaluation criteria
> http://technology.inmobi.com/blog/real-time-stream-processing-at-inmobi-part-3 <http://technology.inmobi.com/blog/real-time-stream-processing-at-inmobi-part-3>
>
> 4) Detailed findings on various evaluation criteria, evaluation summary along with the final recommendation.
> http://technology.inmobi.com/blog/real-time-stream-processing-at-inmobi-part-4 <http://technology.inmobi.com/blog/real-time-stream-processing-at-inmobi-part-4>
>
> We hope that this analysis would be useful in general to anyone who is starting to explore the world of real-time stream processing and decide upon a particular tech stack.
>
> Please go through the blogs and let us know your thoughts!
>
> Regards,
> Satish
>
>
>
>
>
> _____________________________________________________________
> The information contained in this communication is intended solely for the use of the individual or entity to whom it is addressed and others authorized to receive it. It may contain confidential or legally privileged information. If you are not the intended recipient you are hereby notified that any disclosure, copying, distribution or taking any action in reliance on the contents of this information is strictly prohibited and may be unlawful. If you have received this communication in error, please notify us immediately by responding to this email and then delete it from your system. The firm is neither liable for the proper and complete transmission of the information contained in this communication nor for any delay in its receipt.
Re: Storm vs Spark Streaming Tech Evaluation
Posted by "P. Taylor Goetz" <pt...@gmail.com>.
Hi Satish,
Great series of blog posts. Thanks for sharing!
-Taylor
> On Oct 19, 2015, at 4:24 AM, Satish Mittal <sa...@inmobi.com> wrote:
>
> Hi All,
>
> The data platform team at Inmobi recently performed an extensive evaluation exercise in the process of finalizing the real-time Stream processing stack as the choice of our platform.
>
> We have captured all the details of our evaluation as the following series of 4 blogs which have been published at Inmobi technology site:
>
> 1) Introduction; Identifying stream processing use-cases at Inmobi; Identifying potential Technology Candidates. In the interest of time, we limited the
> http://technology.inmobi.com/blog/real-time-stream-processing-at-inmobi-part-1 <http://technology.inmobi.com/blog/real-time-stream-processing-at-inmobi-part-1>
>
> 2) Detailed overview of Storm and Spark Streaming platforms
> http://technology.inmobi.com/blog/real-time-stream-processing-at-inmobi-part-2 <http://technology.inmobi.com/blog/real-time-stream-processing-at-inmobi-part-2>
>
> 3) Identify and define various important evaluation criteria
> http://technology.inmobi.com/blog/real-time-stream-processing-at-inmobi-part-3 <http://technology.inmobi.com/blog/real-time-stream-processing-at-inmobi-part-3>
>
> 4) Detailed findings on various evaluation criteria, evaluation summary along with the final recommendation.
> http://technology.inmobi.com/blog/real-time-stream-processing-at-inmobi-part-4 <http://technology.inmobi.com/blog/real-time-stream-processing-at-inmobi-part-4>
>
> We hope that this analysis would be useful in general to anyone who is starting to explore the world of real-time stream processing and decide upon a particular tech stack.
>
> Please go through the blogs and let us know your thoughts!
>
> Regards,
> Satish
>
>
>
>
>
> _____________________________________________________________
> The information contained in this communication is intended solely for the use of the individual or entity to whom it is addressed and others authorized to receive it. It may contain confidential or legally privileged information. If you are not the intended recipient you are hereby notified that any disclosure, copying, distribution or taking any action in reliance on the contents of this information is strictly prohibited and may be unlawful. If you have received this communication in error, please notify us immediately by responding to this email and then delete it from your system. The firm is neither liable for the proper and complete transmission of the information contained in this communication nor for any delay in its receipt.
Re: Storm vs Spark Streaming Tech Evaluation
Posted by Satish Mittal <sa...@inmobi.com>.
Hi,
Initially we did start with a bigger list of Storm, Spark Streaming, Samza,
and Flink. Flink provides very nice abstractions and sounds quite
promising, but being fairly new it was ruled out in early stage. Soon we
realized that in order to do true justice with evaluation exercise, we need
to narrow down on 2 of them.
The decision to prune the list was purely in the interest of time and
resources, rather than any technical limitations. Storm was a majority
choice of candidate. Amongst Samza and Spark Streaming, probably what
prompted us to consider Spark Streaming was the fact that there were
already some teams within Inmobi who were using Spark for batch processing,
and hence there was a natural inclination/expectation from Spark Streaming.
Thanks,
Satish
On Tue, Oct 20, 2015 at 11:51 PM, Banias H <ba...@gmail.com> wrote:
> In the first blog post, you mentioned Inmobi considered Storm, Spark
> Streaming, Samza, and Flink. Flink is promising but is relatively new. What
> about Samza? Would you share the reason why it is not considered for
> evaluation?
>
> Thanks,
> BH
>
> On Mon, Oct 19, 2015 at 6:24 AM, Satish Mittal <sa...@inmobi.com>
> wrote:
>
>> Hi All,
>>
>> The data platform team at Inmobi recently performed an extensive
>> evaluation exercise in the process of finalizing the real-time Stream
>> processing stack as the choice of our platform.
>>
>> We have captured all the details of our evaluation as the following
>> series of 4 blogs which have been published at Inmobi technology site:
>>
>> 1) Introduction; Identifying stream processing use-cases at Inmobi;
>> Identifying potential Technology Candidates. In the interest of time, we
>> limited the
>>
>> http://technology.inmobi.com/blog/real-time-stream-processing-at-inmobi-part-1
>>
>> 2) Detailed overview of Storm and Spark Streaming platforms
>>
>> http://technology.inmobi.com/blog/real-time-stream-processing-at-inmobi-part-2
>>
>> 3) Identify and define various important evaluation criteria
>>
>> http://technology.inmobi.com/blog/real-time-stream-processing-at-inmobi-part-3
>>
>> 4) Detailed findings on various evaluation criteria, evaluation summary
>> along with the final recommendation.
>>
>> http://technology.inmobi.com/blog/real-time-stream-processing-at-inmobi-part-4
>>
>> We hope that this analysis would be useful in general to anyone who is
>> starting to explore the world of real-time stream processing and decide
>> upon a particular tech stack.
>>
>> Please go through the blogs and let us know your thoughts!
>>
>> Regards,
>> Satish
>>
>>
>>
>>
>>
>> _____________________________________________________________
>> The information contained in this communication is intended solely for
>> the use of the individual or entity to whom it is addressed and others
>> authorized to receive it. It may contain confidential or legally privileged
>> information. If you are not the intended recipient you are hereby notified
>> that any disclosure, copying, distribution or taking any action in reliance
>> on the contents of this information is strictly prohibited and may be
>> unlawful. If you have received this communication in error, please notify
>> us immediately by responding to this email and then delete it from your
>> system. The firm is neither liable for the proper and complete transmission
>> of the information contained in this communication nor for any delay in its
>> receipt.
>
>
>
--
_____________________________________________________________
The information contained in this communication is intended solely for the
use of the individual or entity to whom it is addressed and others
authorized to receive it. It may contain confidential or legally privileged
information. If you are not the intended recipient you are hereby notified
that any disclosure, copying, distribution or taking any action in reliance
on the contents of this information is strictly prohibited and may be
unlawful. If you have received this communication in error, please notify
us immediately by responding to this email and then delete it from your
system. The firm is neither liable for the proper and complete transmission
of the information contained in this communication nor for any delay in its
receipt.
Re: Storm vs Spark Streaming Tech Evaluation
Posted by Banias H <ba...@gmail.com>.
In the first blog post, you mentioned Inmobi considered Storm, Spark
Streaming, Samza, and Flink. Flink is promising but is relatively new. What
about Samza? Would you share the reason why it is not considered for
evaluation?
Thanks,
BH
On Mon, Oct 19, 2015 at 6:24 AM, Satish Mittal <sa...@inmobi.com>
wrote:
> Hi All,
>
> The data platform team at Inmobi recently performed an extensive
> evaluation exercise in the process of finalizing the real-time Stream
> processing stack as the choice of our platform.
>
> We have captured all the details of our evaluation as the following series
> of 4 blogs which have been published at Inmobi technology site:
>
> 1) Introduction; Identifying stream processing use-cases at Inmobi;
> Identifying potential Technology Candidates. In the interest of time, we
> limited the
>
> http://technology.inmobi.com/blog/real-time-stream-processing-at-inmobi-part-1
>
> 2) Detailed overview of Storm and Spark Streaming platforms
>
> http://technology.inmobi.com/blog/real-time-stream-processing-at-inmobi-part-2
>
> 3) Identify and define various important evaluation criteria
>
> http://technology.inmobi.com/blog/real-time-stream-processing-at-inmobi-part-3
>
> 4) Detailed findings on various evaluation criteria, evaluation summary
> along with the final recommendation.
>
> http://technology.inmobi.com/blog/real-time-stream-processing-at-inmobi-part-4
>
> We hope that this analysis would be useful in general to anyone who is
> starting to explore the world of real-time stream processing and decide
> upon a particular tech stack.
>
> Please go through the blogs and let us know your thoughts!
>
> Regards,
> Satish
>
>
>
>
>
> _____________________________________________________________
> The information contained in this communication is intended solely for the
> use of the individual or entity to whom it is addressed and others
> authorized to receive it. It may contain confidential or legally privileged
> information. If you are not the intended recipient you are hereby notified
> that any disclosure, copying, distribution or taking any action in reliance
> on the contents of this information is strictly prohibited and may be
> unlawful. If you have received this communication in error, please notify
> us immediately by responding to this email and then delete it from your
> system. The firm is neither liable for the proper and complete transmission
> of the information contained in this communication nor for any delay in its
> receipt.
RE: Storm vs Spark Streaming Tech Evaluation
Posted by pr...@wipro.com.
Hi Satish,
Great insights and analysis. Thanks for sharing: The right way to work with open source.
Couple of points:
1. You could add a “weight” column in blog 4(table under Evaluation summary), to give weights to a specific criteria. Someone else may use different weights.
2. Was cluster deployment tooling covered in any of the criteria items(“East of operability” perhaps ?). I found the lack of good native cluster deployment tools for storm, a challenge.
Regards,
Prajod
From: Satish Mittal [mailto:satish.mittal@inmobi.com]
Sent: 19 October 2015 16:54
To: user@storm.apache.org; dev@storm.apache.org
Cc: Srikanth Sundarrajan <Sr...@inmobi.com>
Subject: Storm vs Spark Streaming Tech Evaluation
Hi All,
The data platform team at Inmobi recently performed an extensive evaluation exercise in the process of finalizing the real-time Stream processing stack as the choice of our platform.
We have captured all the details of our evaluation as the following series of 4 blogs which have been published at Inmobi technology site:
1) Introduction; Identifying stream processing use-cases at Inmobi; Identifying potential Technology Candidates. In the interest of time, we limited the
http://technology.inmobi.com/blog/real-time-stream-processing-at-inmobi-part-1
2) Detailed overview of Storm and Spark Streaming platforms
http://technology.inmobi.com/blog/real-time-stream-processing-at-inmobi-part-2
3) Identify and define various important evaluation criteria
http://technology.inmobi.com/blog/real-time-stream-processing-at-inmobi-part-3
4) Detailed findings on various evaluation criteria, evaluation summary along with the final recommendation.
http://technology.inmobi.com/blog/real-time-stream-processing-at-inmobi-part-4
We hope that this analysis would be useful in general to anyone who is starting to explore the world of real-time stream processing and decide upon a particular tech stack.
Please go through the blogs and let us know your thoughts!
Regards,
Satish
_____________________________________________________________
The information contained in this communication is intended solely for the use of the individual or entity to whom it is addressed and others authorized to receive it. It may contain confidential or legally privileged information. If you are not the intended recipient you are hereby notified that any disclosure, copying, distribution or taking any action in reliance on the contents of this information is strictly prohibited and may be unlawful. If you have received this communication in error, please notify us immediately by responding to this email and then delete it from your system. The firm is neither liable for the proper and complete transmission of the information contained in this communication nor for any delay in its receipt.
The information contained in this electronic message and any attachments to this message are intended for the exclusive use of the addressee(s) and may contain proprietary, confidential or privileged information. If you are not the intended recipient, you should not disseminate, distribute or copy this e-mail. Please notify the sender immediately and destroy all copies of this message and any attachments. WARNING: Computer viruses can be transmitted via email. The recipient should check this email and any attachments for the presence of viruses. The company accepts no liability for any damage caused by any virus transmitted by this email. www.wipro.com