You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@storm.apache.org by John Fang <xi...@alibaba-inc.com> on 2015/11/18 01:20:43 UTC
答复: Storm typical application
Yes,storm can do it. I suggest you read some storm�� example.:
https://github.com/apache/storm/tree/master/examples/storm-starter
������: Aliza Nagauker [mailto:Aliza.Nagauker@ecitele.com]
����ʱ��: 2015��11��17�� 23:23
�ռ���: user@storm.apache.org
����: Storm typical application
Hello all,
I am new at Storm. I read Storm Doc and tutorial as published in storm site
and have few basic questions.
I am trying to learn and understand whether Storm is suitable for my
application.
Is Storm mainly intended for distributed real time applications that has to
handle "massive input data" and apply "data analytics over this data"?
Is it indented to application where the data-size is large and need analytic
over the data itself (word count, search words, convert formats, write it to
DB etc.)?
Assuming my application is a kind of a Controller that:
�� receive messages from multiple sources: Management Systems,
Network Elements, Internal timers, Internal modules
�� Act upon these messages: update protocol �Cstate-machines, it may
send messages to other servers/applications.
�� Messages are typically short ones �C control protocols messages
(Not HTTP pages, Not Documents, Not Database info).
�� We may need to run this application in multiple machines.
In this case, is Storm is the right choice for this application?
I understand that Storm is indeed very recommended for Distributed Real Time
application, yet, I am not sure it is intended for network applications that
are mainly control application and not Data Processing Applications (Not Big
Data applications)
I'll appreciate your consult on this.
Thanks, Aliza
Re: Storm typical application
Posted by Craig Charleton <cr...@gmail.com>.
I think your company is in South Florida, where I also reside.
Craig Charleton
craig.charleton@gmail.com
> On Nov 18, 2015, at 3:25 AM, Aliza Nagauker <Al...@ecitele.com> wrote:
>
> Hi,
>
> Thanks for your response. I will try the examples.
>
> I understand that Storm can do the functionality required in my application, yet my question is whether it is the right platform.
> So far we worked with Karaf-framework for our applications, and I am trying to understand what should be the motivations to move to Storm framework?
> Is it for cases of:
> · large amount of real time data processing (Big Data: files, DB, WEB pages) over distributed machines?
> · Large amount of real time events processing – usually control protocols (network protocols – like routing protocol, VOIP protocols, SNMP, REST) over distributed machines?
>
> Thanks, Aliza
>
> From: John Fang [mailto:xiaojian.fxj@alibaba-inc.com]
> Sent: Wednesday, November 18, 2015 2:21 AM
> To: user@storm.apache.org
> Subject: 答复: Storm typical application
>
> Yes,storm can do it. I suggest you read some storm’ example.: https://github.com/apache/storm/tree/master/examples/storm-starter
>
>
> 发件人: Aliza Nagauker [mailto:Aliza.Nagauker@ecitele.com]
> 发送时间: 2015年11月17日 23:23
> 收件人: user@storm.apache.org
> 主题: Storm typical application
>
> Hello all,
>
> I am new at Storm. I read Storm Doc and tutorial as published in storm site and have few basic questions.
> I am trying to learn and understand whether Storm is suitable for my application.
>
> Is Storm mainly intended for distributed real time applications that has to handle "massive input data" and apply "data analytics over this data"?
> Is it indented to application where the data-size is large and need analytic over the data itself (word count, search words, convert formats, write it to DB etc.)?
>
> Assuming my application is a kind of a Controller that:
> �. receive messages from multiple sources: Management Systems, Network Elements, Internal timers, Internal modules
> �. Act upon these messages: update protocol –state-machines, it may send messages to other servers/applications.
> �. Messages are typically short ones – control protocols messages (Not HTTP pages, Not Documents, Not Database info).
> �. We may need to run this application in multiple machines.
>
> In this case, is Storm is the right choice for this application?
> I understand that Storm is indeed very recommended for Distributed Real Time application, yet, I am not sure it is intended for network applications that are mainly control application and not Data Processing Applications (Not Big Data applications)
>
> I'll appreciate your consult on this.
>
> Thanks, Aliza
>
>
>
>
Re: Storm typical application
Posted by Craig Charleton <cr...@gmail.com>.
Aliza,
Below is a link to a book that goes over a number of use cases that may help you.
https://www.packtpub.com/big-data-and-business-intelligence/storm-blueprints-patterns-distributed-real-time-computation
Craig Charleton
craig.charleton@gmail.com
> On Nov 18, 2015, at 9:21 AM, Aliza Nagauker <Al...@ecitele.com> wrote:
>
> Hi,
>
> Thank you all, you defiantly cleared the issue (and it wasn't boring at all J )
>
> So Craig, is it the "Big data" processing that makes the "Storm" the right choice?
> If I have ONLY large number of events, and they do not add-up to "Big data", is it still relevant to consider Storm? (I am sure Storm can handle it…yet is it a criteria to select Storm as a platform)
>
> Aliza
>
>
> From: Craig Charleton [mailto:craig.charleton@gmail.com]
> Sent: Wednesday, November 18, 2015 3:44 PM
> To: user@storm.apache.org
> Cc: xiaojian.fxj@alibaba-inc.com
> Subject: Re: Storm typical application
>
> Sorry I didn't make it clear in my response but market data messages are typically very small, received super-fast and (in my situation) coming from a number of sources that trigger other processes in addition to being aggregated up for large-set, real-time analysis the results of which might be run through a separate Storm Topology.
>
> Craig Charleton
> craig.charleton@gmail.com
>
> On Nov 18, 2015, at 8:24 AM, <pr...@wipro.com> <pr...@wipro.com> wrote:
>
> Hi,
>
> There was a point in the question that, I think, translates to : Will Storm be useful if my data packets are small(“Messages are typically short ones”), but they add up to the size of Big Data ?
>
> The answer is Yes.
>
> The other parts of the questions have been answered by others, I hope.
>
> Regards,
> Prajod
>
> From: Craig Charleton [mailto:craig.charleton@gmail.com]
> Sent: 18 November 2015 18:39
> To: user@storm.apache.org
> Cc: John Fang <xi...@alibaba-inc.com>
> Subject: Re: Storm typical application
>
> Aliza,
>
> If I may, I would like to share A few random thoughts about your question.
>
> I worked for a large enterprise software company and our customers were always struggling with how to use the massive amounts of data that were being input/created by their systems to understand their business and make decisions. Traditionally the data had to come to its final resting place before it could be analyzed for decision support. There was no way to reformat, clean, analyze, aggregate the data as it was flowing through the systems, let alone for different user populations to apply their own perspective to the "streams" without affecting the operations of others.
>
> That is where I see the value to large organizations. In fact, it was the limitations of traditional enterprise systems that became obvious once companies like Twitter, Linked-In, Google, Yahoo, Facebook needed to do things to large volumes of data in real time. They not only needed to perform these operations quickly, the load was continually growing, so solutions needed to be able to scale beyond one server on an ongoing basis.
>
> This is what Storm is for in my opinion. I am currently implementing it to perform a lot of operations on stock trade and quote information as it is received from the markets. The number of stocks that need to be handled by the system is unknown. Therefore I am able to use Storm to write the operations once and then scale the load across an unlimited number of servers.
>
> Hope this wasn't too boring.
>
>
> Craig Charleton
> craig.charleton@gmail.com
>
> On Nov 18, 2015, at 3:25 AM, Aliza Nagauker <Al...@ecitele.com> wrote:
>
> Hi,
>
> Thanks for your response. I will try the examples.
>
> I understand that Storm can do the functionality required in my application, yet my question is whether it is the right platform.
> So far we worked with Karaf-framework for our applications, and I am trying to understand what should be the motivations to move to Storm framework?
> Is it for cases of:
> large amount of real time data processing (Big Data: files, DB, WEB pages) over distributed machines?
> Large amount of real time events processing – usually control protocols (network protocols – like routing protocol, VOIP protocols, SNMP, REST) over distributed machines?
>
> Thanks, Aliza
>
> From: John Fang [mailto:xiaojian.fxj@alibaba-inc.com]
> Sent: Wednesday, November 18, 2015 2:21 AM
> To: user@storm.apache.org
> Subject: 答复: Storm typical application
>
> Yes,storm can do it. I suggest you read some storm’ example.: https://github.com/apache/storm/tree/master/examples/storm-starter
>
>
> 发件人: Aliza Nagauker [mailto:Aliza.Nagauker@ecitele.com]
> 发送时间: 2015年11月17日 23:23
> 收件人: user@storm.apache.org
> 主题: Storm typical application
>
> Hello all,
>
> I am new at Storm. I read Storm Doc and tutorial as published in storm site and have few basic questions.
> I am trying to learn and understand whether Storm is suitable for my application.
>
> Is Storm mainly intended for distributed real time applications that has to handle "massive input data" and apply "data analytics over this data"?
> Is it indented to application where the data-size is large and need analytic over the data itself (word count, search words, convert formats, write it to DB etc.)?
>
> Assuming my application is a kind of a Controller that:
> receive messages from multiple sources: Management Systems, Network Elements, Internal timers, Internal modules
> Act upon these messages: update protocol –state-machines, it may send messages to other servers/applications.
> Messages are typically short ones – control protocols messages (Not HTTP pages, Not Documents, Not Database info).
> We may need to run this application in multiple machines.
>
> In this case, is Storm is the right choice for this application?
> I understand that Storm is indeed very recommended for Distributed Real Time application, yet, I am not sure it is intended for network applications that are mainly control application and not Data Processing Applications (Not Big Data applications)
>
> I'll appreciate your consult on this.
>
> Thanks, Aliza
>
>
>
>
> The information contained in this electronic message and any attachments to this message are intended for the exclusive use of the addressee(s) and may contain proprietary, confidential or privileged information. If you are not the intended recipient, you should not disseminate, distribute or copy this e-mail. Please notify the sender immediately and destroy all copies of this message and any attachments. WARNING: Computer viruses can be transmitted via email. The recipient should check this email and any attachments for the presence of viruses. The company accepts no liability for any damage caused by any virus transmitted by this email. www.wipro.com
Re: Storm typical application
Posted by Craig Charleton <cr...@gmail.com>.
Aliza,
The short answer is you can use it for stream processing without BigData. I will add more detail later but I am not able to at the moment. However I didn't want to leave you hanging.
Craig Charleton
craig.charleton@gmail.com
> On Nov 18, 2015, at 9:21 AM, Aliza Nagauker <Al...@ecitele.com> wrote:
>
> Hi,
>
> Thank you all, you defiantly cleared the issue (and it wasn't boring at all J )
>
> So Craig, is it the "Big data" processing that makes the "Storm" the right choice?
> If I have ONLY large number of events, and they do not add-up to "Big data", is it still relevant to consider Storm? (I am sure Storm can handle it…yet is it a criteria to select Storm as a platform)
>
> Aliza
>
>
> From: Craig Charleton [mailto:craig.charleton@gmail.com]
> Sent: Wednesday, November 18, 2015 3:44 PM
> To: user@storm.apache.org
> Cc: xiaojian.fxj@alibaba-inc.com
> Subject: Re: Storm typical application
>
> Sorry I didn't make it clear in my response but market data messages are typically very small, received super-fast and (in my situation) coming from a number of sources that trigger other processes in addition to being aggregated up for large-set, real-time analysis the results of which might be run through a separate Storm Topology.
>
> Craig Charleton
> craig.charleton@gmail.com
>
> On Nov 18, 2015, at 8:24 AM, <pr...@wipro.com> <pr...@wipro.com> wrote:
>
> Hi,
>
> There was a point in the question that, I think, translates to : Will Storm be useful if my data packets are small(“Messages are typically short ones”), but they add up to the size of Big Data ?
>
> The answer is Yes.
>
> The other parts of the questions have been answered by others, I hope.
>
> Regards,
> Prajod
>
> From: Craig Charleton [mailto:craig.charleton@gmail.com]
> Sent: 18 November 2015 18:39
> To: user@storm.apache.org
> Cc: John Fang <xi...@alibaba-inc.com>
> Subject: Re: Storm typical application
>
> Aliza,
>
> If I may, I would like to share A few random thoughts about your question.
>
> I worked for a large enterprise software company and our customers were always struggling with how to use the massive amounts of data that were being input/created by their systems to understand their business and make decisions. Traditionally the data had to come to its final resting place before it could be analyzed for decision support. There was no way to reformat, clean, analyze, aggregate the data as it was flowing through the systems, let alone for different user populations to apply their own perspective to the "streams" without affecting the operations of others.
>
> That is where I see the value to large organizations. In fact, it was the limitations of traditional enterprise systems that became obvious once companies like Twitter, Linked-In, Google, Yahoo, Facebook needed to do things to large volumes of data in real time. They not only needed to perform these operations quickly, the load was continually growing, so solutions needed to be able to scale beyond one server on an ongoing basis.
>
> This is what Storm is for in my opinion. I am currently implementing it to perform a lot of operations on stock trade and quote information as it is received from the markets. The number of stocks that need to be handled by the system is unknown. Therefore I am able to use Storm to write the operations once and then scale the load across an unlimited number of servers.
>
> Hope this wasn't too boring.
>
>
> Craig Charleton
> craig.charleton@gmail.com
>
> On Nov 18, 2015, at 3:25 AM, Aliza Nagauker <Al...@ecitele.com> wrote:
>
> Hi,
>
> Thanks for your response. I will try the examples.
>
> I understand that Storm can do the functionality required in my application, yet my question is whether it is the right platform.
> So far we worked with Karaf-framework for our applications, and I am trying to understand what should be the motivations to move to Storm framework?
> Is it for cases of:
> large amount of real time data processing (Big Data: files, DB, WEB pages) over distributed machines?
> Large amount of real time events processing – usually control protocols (network protocols – like routing protocol, VOIP protocols, SNMP, REST) over distributed machines?
>
> Thanks, Aliza
>
> From: John Fang [mailto:xiaojian.fxj@alibaba-inc.com]
> Sent: Wednesday, November 18, 2015 2:21 AM
> To: user@storm.apache.org
> Subject: 答复: Storm typical application
>
> Yes,storm can do it. I suggest you read some storm’ example.: https://github.com/apache/storm/tree/master/examples/storm-starter
>
>
> 发件人: Aliza Nagauker [mailto:Aliza.Nagauker@ecitele.com]
> 发送时间: 2015年11月17日 23:23
> 收件人: user@storm.apache.org
> 主题: Storm typical application
>
> Hello all,
>
> I am new at Storm. I read Storm Doc and tutorial as published in storm site and have few basic questions.
> I am trying to learn and understand whether Storm is suitable for my application.
>
> Is Storm mainly intended for distributed real time applications that has to handle "massive input data" and apply "data analytics over this data"?
> Is it indented to application where the data-size is large and need analytic over the data itself (word count, search words, convert formats, write it to DB etc.)?
>
> Assuming my application is a kind of a Controller that:
> receive messages from multiple sources: Management Systems, Network Elements, Internal timers, Internal modules
> Act upon these messages: update protocol –state-machines, it may send messages to other servers/applications.
> Messages are typically short ones – control protocols messages (Not HTTP pages, Not Documents, Not Database info).
> We may need to run this application in multiple machines.
>
> In this case, is Storm is the right choice for this application?
> I understand that Storm is indeed very recommended for Distributed Real Time application, yet, I am not sure it is intended for network applications that are mainly control application and not Data Processing Applications (Not Big Data applications)
>
> I'll appreciate your consult on this.
>
> Thanks, Aliza
>
>
>
>
> The information contained in this electronic message and any attachments to this message are intended for the exclusive use of the addressee(s) and may contain proprietary, confidential or privileged information. If you are not the intended recipient, you should not disseminate, distribute or copy this e-mail. Please notify the sender immediately and destroy all copies of this message and any attachments. WARNING: Computer viruses can be transmitted via email. The recipient should check this email and any attachments for the presence of viruses. The company accepts no liability for any damage caused by any virus transmitted by this email. www.wipro.com
RE: Storm typical application
Posted by Aliza Nagauker <Al...@ecitele.com>.
Hi,
Thank you all, you defiantly cleared the issue (and it wasn't boring at all ☺ )
So Craig, is it the "Big data" processing that makes the "Storm" the right choice?
If I have ONLY large number of events, and they do not add-up to "Big data", is it still relevant to consider Storm? (I am sure Storm can handle it…yet is it a criteria to select Storm as a platform)
Aliza
From: Craig Charleton [mailto:craig.charleton@gmail.com]
Sent: Wednesday, November 18, 2015 3:44 PM
To: user@storm.apache.org
Cc: xiaojian.fxj@alibaba-inc.com
Subject: Re: Storm typical application
Sorry I didn't make it clear in my response but market data messages are typically very small, received super-fast and (in my situation) coming from a number of sources that trigger other processes in addition to being aggregated up for large-set, real-time analysis the results of which might be run through a separate Storm Topology.
Craig Charleton
craig.charleton@gmail.com<ma...@gmail.com>
On Nov 18, 2015, at 8:24 AM, <pr...@wipro.com>> <pr...@wipro.com>> wrote:
Hi,
There was a point in the question that, I think, translates to : Will Storm be useful if my data packets are small(“Messages are typically short ones”), but they add up to the size of Big Data ?
The answer is Yes.
The other parts of the questions have been answered by others, I hope.
Regards,
Prajod
From: Craig Charleton [mailto:craig.charleton@gmail.com]
Sent: 18 November 2015 18:39
To: user@storm.apache.org<ma...@storm.apache.org>
Cc: John Fang <xi...@alibaba-inc.com>>
Subject: Re: Storm typical application
Aliza,
If I may, I would like to share A few random thoughts about your question.
I worked for a large enterprise software company and our customers were always struggling with how to use the massive amounts of data that were being input/created by their systems to understand their business and make decisions. Traditionally the data had to come to its final resting place before it could be analyzed for decision support. There was no way to reformat, clean, analyze, aggregate the data as it was flowing through the systems, let alone for different user populations to apply their own perspective to the "streams" without affecting the operations of others.
That is where I see the value to large organizations. In fact, it was the limitations of traditional enterprise systems that became obvious once companies like Twitter, Linked-In, Google, Yahoo, Facebook needed to do things to large volumes of data in real time. They not only needed to perform these operations quickly, the load was continually growing, so solutions needed to be able to scale beyond one server on an ongoing basis.
This is what Storm is for in my opinion. I am currently implementing it to perform a lot of operations on stock trade and quote information as it is received from the markets. The number of stocks that need to be handled by the system is unknown. Therefore I am able to use Storm to write the operations once and then scale the load across an unlimited number of servers.
Hope this wasn't too boring.
Craig Charleton
craig.charleton@gmail.com<ma...@gmail.com>
On Nov 18, 2015, at 3:25 AM, Aliza Nagauker <Al...@ecitele.com>> wrote:
Hi,
Thanks for your response. I will try the examples.
I understand that Storm can do the functionality required in my application, yet my question is whether it is the right platform.
So far we worked with Karaf-framework for our applications, and I am trying to understand what should be the motivations to move to Storm framework?
Is it for cases of:
large amount of real time data processing (Big Data: files, DB, WEB pages) over distributed machines?
Large amount of real time events processing – usually control protocols (network protocols – like routing protocol, VOIP protocols, SNMP, REST) over distributed machines?
Thanks, Aliza
From: John Fang [mailto:xiaojian.fxj@alibaba-inc.com]
Sent: Wednesday, November 18, 2015 2:21 AM
To: user@storm.apache.org<ma...@storm.apache.org>
Subject: 答复: Storm typical application
Yes,storm can do it. I suggest you read some storm’ example.: https://github.com/apache/storm/tree/master/examples/storm-starter
发件人: Aliza Nagauker [mailto:Aliza.Nagauker@ecitele.com]
发送时间: 2015年11月17日 23:23
收件人: user@storm.apache.org<ma...@storm.apache.org>
主题: Storm typical application
Hello all,
I am new at Storm. I read Storm Doc and tutorial as published in storm site and have few basic questions.
I am trying to learn and understand whether Storm is suitable for my application.
Is Storm mainly intended for distributed real time applications that has to handle "massive input data" and apply "data analytics over this data"?
Is it indented to application where the data-size is large and need analytic over the data itself (word count, search words, convert formats, write it to DB etc.)?
Assuming my application is a kind of a Controller that:
receive messages from multiple sources: Management Systems, Network Elements, Internal timers, Internal modules
Act upon these messages: update protocol –state-machines, it may send messages to other servers/applications.
Messages are typically short ones – control protocols messages (Not HTTP pages, Not Documents, Not Database info).
We may need to run this application in multiple machines.
In this case, is Storm is the right choice for this application?
I understand that Storm is indeed very recommended for Distributed Real Time application, yet, I am not sure it is intended for network applications that are mainly control application and not Data Processing Applications (Not Big Data applications)
I'll appreciate your consult on this.
Thanks, Aliza
The information contained in this electronic message and any attachments to this message are intended for the exclusive use of the addressee(s) and may contain proprietary, confidential or privileged information. If you are not the intended recipient, you should not disseminate, distribute or copy this e-mail. Please notify the sender immediately and destroy all copies of this message and any attachments. WARNING: Computer viruses can be transmitted via email. The recipient should check this email and any attachments for the presence of viruses. The company accepts no liability for any damage caused by any virus transmitted by this email. www.wipro.com<http://www.wipro.com>
Re: Storm typical application
Posted by Craig Charleton <cr...@gmail.com>.
Sorry I didn't make it clear in my response but market data messages are typically very small, received super-fast and (in my situation) coming from a number of sources that trigger other processes in addition to being aggregated up for large-set, real-time analysis the results of which might be run through a separate Storm Topology.
Craig Charleton
craig.charleton@gmail.com
> On Nov 18, 2015, at 8:24 AM, <pr...@wipro.com> <pr...@wipro.com> wrote:
>
> Hi,
>
> There was a point in the question that, I think, translates to : Will Storm be useful if my data packets are small(“Messages are typically short ones”), but they add up to the size of Big Data ?
>
> The answer is Yes.
>
> The other parts of the questions have been answered by others, I hope.
>
> Regards,
> Prajod
>
> From: Craig Charleton [mailto:craig.charleton@gmail.com]
> Sent: 18 November 2015 18:39
> To: user@storm.apache.org
> Cc: John Fang <xi...@alibaba-inc.com>
> Subject: Re: Storm typical application
>
> Aliza,
>
> If I may, I would like to share A few random thoughts about your question.
>
> I worked for a large enterprise software company and our customers were always struggling with how to use the massive amounts of data that were being input/created by their systems to understand their business and make decisions. Traditionally the data had to come to its final resting place before it could be analyzed for decision support. There was no way to reformat, clean, analyze, aggregate the data as it was flowing through the systems, let alone for different user populations to apply their own perspective to the "streams" without affecting the operations of others.
>
> That is where I see the value to large organizations. In fact, it was the limitations of traditional enterprise systems that became obvious once companies like Twitter, Linked-In, Google, Yahoo, Facebook needed to do things to large volumes of data in real time. They not only needed to perform these operations quickly, the load was continually growing, so solutions needed to be able to scale beyond one server on an ongoing basis.
>
> This is what Storm is for in my opinion. I am currently implementing it to perform a lot of operations on stock trade and quote information as it is received from the markets. The number of stocks that need to be handled by the system is unknown. Therefore I am able to use Storm to write the operations once and then scale the load across an unlimited number of servers.
>
> Hope this wasn't too boring.
>
>
> Craig Charleton
> craig.charleton@gmail.com
>
> On Nov 18, 2015, at 3:25 AM, Aliza Nagauker <Al...@ecitele.com> wrote:
>
> Hi,
>
> Thanks for your response. I will try the examples.
>
> I understand that Storm can do the functionality required in my application, yet my question is whether it is the right platform.
> So far we worked with Karaf-framework for our applications, and I am trying to understand what should be the motivations to move to Storm framework?
> Is it for cases of:
> · large amount of real time data processing (Big Data: files, DB, WEB pages) over distributed machines?
> · Large amount of real time events processing – usually control protocols (network protocols – like routing protocol, VOIP protocols, SNMP, REST) over distributed machines?
>
> Thanks, Aliza
>
> From: John Fang [mailto:xiaojian.fxj@alibaba-inc.com]
> Sent: Wednesday, November 18, 2015 2:21 AM
> To: user@storm.apache.org
> Subject: 答复: Storm typical application
>
> Yes,storm can do it. I suggest you read some storm’ example.: https://github.com/apache/storm/tree/master/examples/storm-starter
>
>
> 发件人: Aliza Nagauker [mailto:Aliza.Nagauker@ecitele.com]
> 发送时间: 2015年11月17日 23:23
> 收件人: user@storm.apache.org
> 主题: Storm typical application
>
> Hello all,
>
> I am new at Storm. I read Storm Doc and tutorial as published in storm site and have few basic questions.
> I am trying to learn and understand whether Storm is suitable for my application.
>
> Is Storm mainly intended for distributed real time applications that has to handle "massive input data" and apply "data analytics over this data"?
> Is it indented to application where the data-size is large and need analytic over the data itself (word count, search words, convert formats, write it to DB etc.)?
>
> Assuming my application is a kind of a Controller that:
> �. receive messages from multiple sources: Management Systems, Network Elements, Internal timers, Internal modules
> �. Act upon these messages: update protocol –state-machines, it may send messages to other servers/applications.
> �. Messages are typically short ones – control protocols messages (Not HTTP pages, Not Documents, Not Database info).
> �. We may need to run this application in multiple machines.
>
> In this case, is Storm is the right choice for this application?
> I understand that Storm is indeed very recommended for Distributed Real Time application, yet, I am not sure it is intended for network applications that are mainly control application and not Data Processing Applications (Not Big Data applications)
>
> I'll appreciate your consult on this.
>
> Thanks, Aliza
>
>
>
>
> The information contained in this electronic message and any attachments to this message are intended for the exclusive use of the addressee(s) and may contain proprietary, confidential or privileged information. If you are not the intended recipient, you should not disseminate, distribute or copy this e-mail. Please notify the sender immediately and destroy all copies of this message and any attachments. WARNING: Computer viruses can be transmitted via email. The recipient should check this email and any attachments for the presence of viruses. The company accepts no liability for any damage caused by any virus transmitted by this email. www.wipro.com
RE: Storm typical application
Posted by pr...@wipro.com.
Hi,
There was a point in the question that, I think, translates to : Will Storm be useful if my data packets are small(“Messages are typically short ones”), but they add up to the size of Big Data ?
The answer is Yes.
The other parts of the questions have been answered by others, I hope.
Regards,
Prajod
From: Craig Charleton [mailto:craig.charleton@gmail.com]
Sent: 18 November 2015 18:39
To: user@storm.apache.org
Cc: John Fang <xi...@alibaba-inc.com>
Subject: Re: Storm typical application
Aliza,
If I may, I would like to share A few random thoughts about your question.
I worked for a large enterprise software company and our customers were always struggling with how to use the massive amounts of data that were being input/created by their systems to understand their business and make decisions. Traditionally the data had to come to its final resting place before it could be analyzed for decision support. There was no way to reformat, clean, analyze, aggregate the data as it was flowing through the systems, let alone for different user populations to apply their own perspective to the "streams" without affecting the operations of others.
That is where I see the value to large organizations. In fact, it was the limitations of traditional enterprise systems that became obvious once companies like Twitter, Linked-In, Google, Yahoo, Facebook needed to do things to large volumes of data in real time. They not only needed to perform these operations quickly, the load was continually growing, so solutions needed to be able to scale beyond one server on an ongoing basis.
This is what Storm is for in my opinion. I am currently implementing it to perform a lot of operations on stock trade and quote information as it is received from the markets. The number of stocks that need to be handled by the system is unknown. Therefore I am able to use Storm to write the operations once and then scale the load across an unlimited number of servers.
Hope this wasn't too boring.
Craig Charleton
craig.charleton@gmail.com<ma...@gmail.com>
On Nov 18, 2015, at 3:25 AM, Aliza Nagauker <Al...@ecitele.com>> wrote:
Hi,
Thanks for your response. I will try the examples.
I understand that Storm can do the functionality required in my application, yet my question is whether it is the right platform.
So far we worked with Karaf-framework for our applications, and I am trying to understand what should be the motivations to move to Storm framework?
Is it for cases of:
· large amount of real time data processing (Big Data: files, DB, WEB pages) over distributed machines?
· Large amount of real time events processing – usually control protocols (network protocols – like routing protocol, VOIP protocols, SNMP, REST) over distributed machines?
Thanks, Aliza
From: John Fang [mailto:xiaojian.fxj@alibaba-inc.com]
Sent: Wednesday, November 18, 2015 2:21 AM
To: user@storm.apache.org<ma...@storm.apache.org>
Subject: 答复: Storm typical application
Yes,storm can do it. I suggest you read some storm’ example.: https://github.com/apache/storm/tree/master/examples/storm-starter
发件人: Aliza Nagauker [mailto:Aliza.Nagauker@ecitele.com]
发送时间: 2015年11月17日 23:23
收件人: user@storm.apache.org<ma...@storm.apache.org>
主题: Storm typical application
Hello all,
I am new at Storm. I read Storm Doc and tutorial as published in storm site and have few basic questions.
I am trying to learn and understand whether Storm is suitable for my application.
Is Storm mainly intended for distributed real time applications that has to handle "massive input data" and apply "data analytics over this data"?
Is it indented to application where the data-size is large and need analytic over the data itself (word count, search words, convert formats, write it to DB etc.)?
Assuming my application is a kind of a Controller that:
�. receive messages from multiple sources: Management Systems, Network Elements, Internal timers, Internal modules
�. Act upon these messages: update protocol –state-machines, it may send messages to other servers/applications.
�. Messages are typically short ones – control protocols messages (Not HTTP pages, Not Documents, Not Database info).
�. We may need to run this application in multiple machines.
In this case, is Storm is the right choice for this application?
I understand that Storm is indeed very recommended for Distributed Real Time application, yet, I am not sure it is intended for network applications that are mainly control application and not Data Processing Applications (Not Big Data applications)
I'll appreciate your consult on this.
Thanks, Aliza
The information contained in this electronic message and any attachments to this message are intended for the exclusive use of the addressee(s) and may contain proprietary, confidential or privileged information. If you are not the intended recipient, you should not disseminate, distribute or copy this e-mail. Please notify the sender immediately and destroy all copies of this message and any attachments. WARNING: Computer viruses can be transmitted via email. The recipient should check this email and any attachments for the presence of viruses. The company accepts no liability for any damage caused by any virus transmitted by this email. www.wipro.com
Re: Storm typical application
Posted by Craig Charleton <cr...@gmail.com>.
Aliza,
If I may, I would like to share A few random thoughts about your question.
I worked for a large enterprise software company and our customers were always struggling with how to use the massive amounts of data that were being input/created by their systems to understand their business and make decisions. Traditionally the data had to come to its final resting place before it could be analyzed for decision support. There was no way to reformat, clean, analyze, aggregate the data as it was flowing through the systems, let alone for different user populations to apply their own perspective to the "streams" without affecting the operations of others.
That is where I see the value to large organizations. In fact, it was the limitations of traditional enterprise systems that became obvious once companies like Twitter, Linked-In, Google, Yahoo, Facebook needed to do things to large volumes of data in real time. They not only needed to perform these operations quickly, the load was continually growing, so solutions needed to be able to scale beyond one server on an ongoing basis.
This is what Storm is for in my opinion. I am currently implementing it to perform a lot of operations on stock trade and quote information as it is received from the markets. The number of stocks that need to be handled by the system is unknown. Therefore I am able to use Storm to write the operations once and then scale the load across an unlimited number of servers.
Hope this wasn't too boring.
Craig Charleton
craig.charleton@gmail.com
> On Nov 18, 2015, at 3:25 AM, Aliza Nagauker <Al...@ecitele.com> wrote:
>
> Hi,
>
> Thanks for your response. I will try the examples.
>
> I understand that Storm can do the functionality required in my application, yet my question is whether it is the right platform.
> So far we worked with Karaf-framework for our applications, and I am trying to understand what should be the motivations to move to Storm framework?
> Is it for cases of:
> · large amount of real time data processing (Big Data: files, DB, WEB pages) over distributed machines?
> · Large amount of real time events processing – usually control protocols (network protocols – like routing protocol, VOIP protocols, SNMP, REST) over distributed machines?
>
> Thanks, Aliza
>
> From: John Fang [mailto:xiaojian.fxj@alibaba-inc.com]
> Sent: Wednesday, November 18, 2015 2:21 AM
> To: user@storm.apache.org
> Subject: 答复: Storm typical application
>
> Yes,storm can do it. I suggest you read some storm’ example.: https://github.com/apache/storm/tree/master/examples/storm-starter
>
>
> 发件人: Aliza Nagauker [mailto:Aliza.Nagauker@ecitele.com]
> 发送时间: 2015年11月17日 23:23
> 收件人: user@storm.apache.org
> 主题: Storm typical application
>
> Hello all,
>
> I am new at Storm. I read Storm Doc and tutorial as published in storm site and have few basic questions.
> I am trying to learn and understand whether Storm is suitable for my application.
>
> Is Storm mainly intended for distributed real time applications that has to handle "massive input data" and apply "data analytics over this data"?
> Is it indented to application where the data-size is large and need analytic over the data itself (word count, search words, convert formats, write it to DB etc.)?
>
> Assuming my application is a kind of a Controller that:
> �. receive messages from multiple sources: Management Systems, Network Elements, Internal timers, Internal modules
> �. Act upon these messages: update protocol –state-machines, it may send messages to other servers/applications.
> �. Messages are typically short ones – control protocols messages (Not HTTP pages, Not Documents, Not Database info).
> �. We may need to run this application in multiple machines.
>
> In this case, is Storm is the right choice for this application?
> I understand that Storm is indeed very recommended for Distributed Real Time application, yet, I am not sure it is intended for network applications that are mainly control application and not Data Processing Applications (Not Big Data applications)
>
> I'll appreciate your consult on this.
>
> Thanks, Aliza
>
>
>
>
RE: Storm typical application
Posted by Aliza Nagauker <Al...@ecitele.com>.
Hi,
Thanks for your response. I will try the examples.
I understand that Storm can do the functionality required in my application, yet my question is whether it is the right platform.
So far we worked with Karaf-framework for our applications, and I am trying to understand what should be the motivations to move to Storm framework?
Is it for cases of:
�� large amount of real time data processing (Big Data: files, DB, WEB pages) over distributed machines?
�� Large amount of real time events processing �C usually control protocols (network protocols �C like routing protocol, VOIP protocols, SNMP, REST) over distributed machines?
Thanks, Aliza
From: John Fang [mailto:xiaojian.fxj@alibaba-inc.com]
Sent: Wednesday, November 18, 2015 2:21 AM
To: user@storm.apache.org
Subject: ��: Storm typical application
Yes,storm can do it. I suggest you read some storm�� example.: https://github.com/apache/storm/tree/master/examples/storm-starter
������: Aliza Nagauker [mailto:Aliza.Nagauker@ecitele.com]
����ʱ��: 2015��11��17�� 23:23
�ռ���: user@storm.apache.org<ma...@storm.apache.org>
����: Storm typical application
Hello all,
I am new at Storm. I read Storm Doc and tutorial as published in storm site and have few basic questions.
I am trying to learn and understand whether Storm is suitable for my application.
Is Storm mainly intended for distributed real time applications that has to handle "massive input data" and apply "data analytics over this data"?
Is it indented to application where the data-size is large and need analytic over the data itself (word count, search words, convert formats, write it to DB etc.)?
Assuming my application is a kind of a Controller that:
�. receive messages from multiple sources: Management Systems, Network Elements, Internal timers, Internal modules
�. Act upon these messages: update protocol �Cstate-machines, it may send messages to other servers/applications.
�. Messages are typically short ones �C control protocols messages (Not HTTP pages, Not Documents, Not Database info).
�. We may need to run this application in multiple machines.
In this case, is Storm is the right choice for this application?
I understand that Storm is indeed very recommended for Distributed Real Time application, yet, I am not sure it is intended for network applications that are mainly control application and not Data Processing Applications (Not Big Data applications)
I'll appreciate your consult on this.
Thanks, Aliza