You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@storm.apache.org by Kang Xiao <kx...@gmail.com> on 2014/02/22 08:51:37 UTC
回复： Understand the processing flow of storm
Hi R.RAMPRASAD
I try to answer your questions and hope that will help you.
> 1. I have a topology(1 spout and 3 bolts) and 3 machines(1 nimbus and 2
> worker nodes).Whether I want to run my topology on all 3 nodes or run on
> nimbus is enough? Is it nimbus will take care of distributing the code to
> other nodes?
>
Just submit your topology to nimbus and nimbus will take care of distributing the code to worker nodes.
>
> 2. My spout will run on nimbus or any of the worker nodes?
Spouts and Bolts will all run on worker nodes.
>
> 3. The 3 bolts are run on 3 separate nodes or run on same node? is it will
> take care by nimbus?
Spouts and Bolts will run in workers. Workers will distribute evenly on worker nodes. If you specify enough workers eg. 4 workers in your case, they will run on separate nodes.
>
> 4. How do we track processing of bolt in nodes?
Storm UI keep track of tuple emit/ack/fail count of spout/bolt. For more detail information you can turn on topology debug mode and worker will log every tuple.
>
> 5. Is it any documentation available to understand the complete flow of
> processing a message in Storm?
Your can refer to the storm documentation site: http://storm.incubator.apache.org/documentation/Implementation-docs.html
--
Best Regards!
肖康(Kang Xiao,<kxiao.tiger@gmail.com (mailto:kxiao.tiger@gmail.com)>)
Distributed Software Engineer
在 2014年2月22日 星期六，14:51，Ramprasad Ranganathan 写道：
> Hi,
>
> I am beginner in Storm.I am trying to understand the processing flow of
> storm by following questions.
>
> 1. I have a topology(1 spout and 3 bolts) and 3 machines(1 nimbus and 2
> worker nodes).Whether I want to run my topology on all 3 nodes or run on
> nimbus is enough? Is it nimbus will take care of distributing the code to
> other nodes?
>
> 2. My spout will run on nimbus or any of the worker nodes?
>
> 3. The 3 bolts are run on 3 separate nodes or run on same node? is it will
> take care by nimbus?
>
> 4. How do we track processing of bolt in nodes?
>
> 5. Is it any documentation available to understand the complete flow of
> processing a message in Storm?
>
> --
> by
>
> *R.RAMPRASAD*
Re: 回复： Understand the processing flow of storm
Posted by Milinda Pathirage <mp...@umail.iu.edu>.
Hi Ramprasad,
Following articles [1] from Michael Noll contains lots of information
about how storm works. You should be able to get answers to most of
your new questions (mainly questions about how streams are delivered
and processed) from [2].
Thanks
Milinda
[1] http://www.michael-noll.com/blog/categories/storm/
[2] http://www.michael-noll.com/blog/2013/06/21/understanding-storm-internal-message-buffers/
On Sat, Feb 22, 2014 at 12:59 PM, 肖康(Kang Xiao) <kx...@gmail.com> wrote:
> Hi R.RAMPRASAD
>
> I'm sorry I can not see the image you post. PS: It seems that the questions
> are more suitable to be discussed in storm-user mail list. Can we transfer
> to user@storm.incubator.apache.org?
>
>
> On Sat, Feb 22, 2014 at 5:03 PM, Ramprasad Ranganathan <ra...@gmail.com>wrote:
>
>> Hi,Kang Xiao
>>
>> Thank you. From your answer I understood, After submitting topology
>> to Nimbus node it does only distributing code and other management
>> works.Then the spouts and bolts are distributed evenly to worker nodes. I
>> have some other questions,
>>
>> - If I have only 3 machines,how 1 spout and 3 bolts will distributed
>> to nodes?
>> - Storm is for real time processing, If all data in the queue is
>> processed, spouts and bolts will go to "Waiting" state or what will happen?
>> - In following Image, If 3 bolts are distributed to each 3 nodes and
>> processing parallel,how Bolt 2 will process without the result of Bolt 1?
>> is it go to waiting state to receive bolt 1 result tuple?
>> - Bolts process tuples synchronously or asynchronously? If 1 million
>> tuples emitted by spout, each tuple process one by one?
>> - If 2 tuples emitted by spout,1st bolt on node 1 process and send to
>> 2nd bolt on Bolt. Same way tuple 2 process on 1 bolt on 1 node and send to
>> 2 bolt on 2 node. Is my understanding is correct or not?
>>
>> [image: Inline image 1]
>>
>>
>>
>> On Sat, Feb 22, 2014 at 1:21 PM, Kang Xiao <kx...@gmail.com> wrote:
>>
>>> Hi R.RAMPRASAD
>>>
>>> I try to answer your questions and hope that will help you.
>>>
>>> > 1. I have a topology(1 spout and 3 bolts) and 3 machines(1 nimbus and 2
>>> > worker nodes).Whether I want to run my topology on all 3 nodes or run on
>>> > nimbus is enough? Is it nimbus will take care of distributing the code
>>> to
>>> > other nodes?
>>> >
>>>
>>> Just submit your topology to nimbus and nimbus will take care of
>>> distributing the code to worker nodes.
>>>
>>> >
>>> > 2. My spout will run on nimbus or any of the worker nodes?
>>>
>>> Spouts and Bolts will all run on worker nodes.
>>>
>>> >
>>> > 3. The 3 bolts are run on 3 separate nodes or run on same node? is it
>>> will
>>> > take care by nimbus?
>>>
>>> Spouts and Bolts will run in workers. Workers will distribute evenly on
>>> worker nodes. If you specify enough workers eg. 4 workers in your case,
>>> they will run on separate nodes.
>>>
>>> >
>>> > 4. How do we track processing of bolt in nodes?
>>> Storm UI keep track of tuple emit/ack/fail count of spout/bolt. For more
>>> detail information you can turn on topology debug mode and worker will log
>>> every tuple.
>>>
>>> >
>>> > 5. Is it any documentation available to understand the complete flow of
>>> > processing a message in Storm?
>>>
>>>
>>> Your can refer to the storm documentation site:
>>> http://storm.incubator.apache.org/documentation/Implementation-docs.html
>>>
>>>
>>> --
>>> Best Regards!
>>>
>>> 肖康(Kang Xiao,<kxiao.tiger@gmail.com (mailto:kxiao.tiger@gmail.com)>)
>>>
>>> Distributed Software Engineer
>>>
>>> 在 2014年2月22日 星期六，14:51，Ramprasad Ranganathan 写道：
>>> > Hi,
>>> >
>>> > I am beginner in Storm.I am trying to understand the processing flow of
>>> > storm by following questions.
>>> >
>>> > 1. I have a topology(1 spout and 3 bolts) and 3 machines(1 nimbus and 2
>>> > worker nodes).Whether I want to run my topology on all 3 nodes or run on
>>> > nimbus is enough? Is it nimbus will take care of distributing the code
>>> to
>>> > other nodes?
>>> >
>>> > 2. My spout will run on nimbus or any of the worker nodes?
>>> >
>>> > 3. The 3 bolts are run on 3 separate nodes or run on same node? is it
>>> will
>>> > take care by nimbus?
>>> >
>>> > 4. How do we track processing of bolt in nodes?
>>> >
>>> > 5. Is it any documentation available to understand the complete flow of
>>> > processing a message in Storm?
>>> >
>>> > --
>>> > by
>>> >
>>> > *R.RAMPRASAD*
>>>
>>>
>>
>>
>> --
>> by
>>
>> *R.RAMPRASAD*
>>
>
>
>
> --
> Best Regards!
>
> 肖康(Kang Xiao,<kx...@gmail.com>)
> Distributed Software Engineer
--
Milinda Pathirage
twitter: milindalakmal
skype: milinda.pathirage
blog: http://milinda.pathirage.org
Re: 回复： Understand the processing flow of storm
Posted by "肖康 (Kang Xiao)" <kx...@gmail.com>.
Hi R.RAMPRASAD
I'm sorry I can not see the image you post. PS: It seems that the questions
are more suitable to be discussed in storm-user mail list. Can we transfer
to user@storm.incubator.apache.org?
On Sat, Feb 22, 2014 at 5:03 PM, Ramprasad Ranganathan <ra...@gmail.com>wrote:
> Hi,Kang Xiao
>
> Thank you. From your answer I understood, After submitting topology
> to Nimbus node it does only distributing code and other management
> works.Then the spouts and bolts are distributed evenly to worker nodes. I
> have some other questions,
>
> - If I have only 3 machines,how 1 spout and 3 bolts will distributed
> to nodes?
> - Storm is for real time processing, If all data in the queue is
> processed, spouts and bolts will go to "Waiting" state or what will happen?
> - In following Image, If 3 bolts are distributed to each 3 nodes and
> processing parallel,how Bolt 2 will process without the result of Bolt 1?
> is it go to waiting state to receive bolt 1 result tuple?
> - Bolts process tuples synchronously or asynchronously? If 1 million
> tuples emitted by spout, each tuple process one by one?
> - If 2 tuples emitted by spout,1st bolt on node 1 process and send to
> 2nd bolt on Bolt. Same way tuple 2 process on 1 bolt on 1 node and send to
> 2 bolt on 2 node. Is my understanding is correct or not?
>
> [image: Inline image 1]
>
>
>
> On Sat, Feb 22, 2014 at 1:21 PM, Kang Xiao <kx...@gmail.com> wrote:
>
>> Hi R.RAMPRASAD
>>
>> I try to answer your questions and hope that will help you.
>>
>> > 1. I have a topology(1 spout and 3 bolts) and 3 machines(1 nimbus and 2
>> > worker nodes).Whether I want to run my topology on all 3 nodes or run on
>> > nimbus is enough? Is it nimbus will take care of distributing the code
>> to
>> > other nodes?
>> >
>>
>> Just submit your topology to nimbus and nimbus will take care of
>> distributing the code to worker nodes.
>>
>> >
>> > 2. My spout will run on nimbus or any of the worker nodes?
>>
>> Spouts and Bolts will all run on worker nodes.
>>
>> >
>> > 3. The 3 bolts are run on 3 separate nodes or run on same node? is it
>> will
>> > take care by nimbus?
>>
>> Spouts and Bolts will run in workers. Workers will distribute evenly on
>> worker nodes. If you specify enough workers eg. 4 workers in your case,
>> they will run on separate nodes.
>>
>> >
>> > 4. How do we track processing of bolt in nodes?
>> Storm UI keep track of tuple emit/ack/fail count of spout/bolt. For more
>> detail information you can turn on topology debug mode and worker will log
>> every tuple.
>>
>> >
>> > 5. Is it any documentation available to understand the complete flow of
>> > processing a message in Storm?
>>
>>
>> Your can refer to the storm documentation site:
>> http://storm.incubator.apache.org/documentation/Implementation-docs.html
>>
>>
>> --
>> Best Regards!
>>
>> 肖康(Kang Xiao,<kxiao.tiger@gmail.com (mailto:kxiao.tiger@gmail.com)>)
>>
>> Distributed Software Engineer
>>
>> 在 2014年2月22日 星期六，14:51，Ramprasad Ranganathan 写道：
>> > Hi,
>> >
>> > I am beginner in Storm.I am trying to understand the processing flow of
>> > storm by following questions.
>> >
>> > 1. I have a topology(1 spout and 3 bolts) and 3 machines(1 nimbus and 2
>> > worker nodes).Whether I want to run my topology on all 3 nodes or run on
>> > nimbus is enough? Is it nimbus will take care of distributing the code
>> to
>> > other nodes?
>> >
>> > 2. My spout will run on nimbus or any of the worker nodes?
>> >
>> > 3. The 3 bolts are run on 3 separate nodes or run on same node? is it
>> will
>> > take care by nimbus?
>> >
>> > 4. How do we track processing of bolt in nodes?
>> >
>> > 5. Is it any documentation available to understand the complete flow of
>> > processing a message in Storm?
>> >
>> > --
>> > by
>> >
>> > *R.RAMPRASAD*
>>
>>
>
>
> --
> by
>
> *R.RAMPRASAD*
>
--
Best Regards!
肖康(Kang Xiao,<kx...@gmail.com>)
Distributed Software Engineer
Re: 回复： Understand the processing flow of storm
Posted by Ramprasad Ranganathan <ra...@gmail.com>.
Hi,Kang Xiao
Thank you. From your answer I understood, After submitting topology
to Nimbus node it does only distributing code and other management
works.Then the spouts and bolts are distributed evenly to worker nodes. I
have some other questions,
- If I have only 3 machines,how 1 spout and 3 bolts will distributed to
nodes?
- Storm is for real time processing, If all data in the queue is
processed, spouts and bolts will go to "Waiting" state or what will happen?
- In following Image, If 3 bolts are distributed to each 3 nodes and
processing parallel,how Bolt 2 will process without the result of Bolt 1?
is it go to waiting state to receive bolt 1 result tuple?
- Bolts process tuples synchronously or asynchronously? If 1 million
tuples emitted by spout, each tuple process one by one?
- If 2 tuples emitted by spout,1st bolt on node 1 process and send to
2nd bolt on Bolt. Same way tuple 2 process on 1 bolt on 1 node and send to
2 bolt on 2 node. Is my understanding is correct or not?
[image: Inline image 1]
On Sat, Feb 22, 2014 at 1:21 PM, Kang Xiao <kx...@gmail.com> wrote:
> Hi R.RAMPRASAD
>
> I try to answer your questions and hope that will help you.
>
> > 1. I have a topology(1 spout and 3 bolts) and 3 machines(1 nimbus and 2
> > worker nodes).Whether I want to run my topology on all 3 nodes or run on
> > nimbus is enough? Is it nimbus will take care of distributing the code to
> > other nodes?
> >
>
> Just submit your topology to nimbus and nimbus will take care of
> distributing the code to worker nodes.
>
> >
> > 2. My spout will run on nimbus or any of the worker nodes?
>
> Spouts and Bolts will all run on worker nodes.
>
> >
> > 3. The 3 bolts are run on 3 separate nodes or run on same node? is it
> will
> > take care by nimbus?
>
> Spouts and Bolts will run in workers. Workers will distribute evenly on
> worker nodes. If you specify enough workers eg. 4 workers in your case,
> they will run on separate nodes.
>
> >
> > 4. How do we track processing of bolt in nodes?
> Storm UI keep track of tuple emit/ack/fail count of spout/bolt. For more
> detail information you can turn on topology debug mode and worker will log
> every tuple.
>
> >
> > 5. Is it any documentation available to understand the complete flow of
> > processing a message in Storm?
>
>
> Your can refer to the storm documentation site:
> http://storm.incubator.apache.org/documentation/Implementation-docs.html
>
>
> --
> Best Regards!
>
> 肖康(Kang Xiao,<kxiao.tiger@gmail.com (mailto:kxiao.tiger@gmail.com)>)
> Distributed Software Engineer
>
> 在 2014年2月22日 星期六，14:51，Ramprasad Ranganathan 写道：
> > Hi,
> >
> > I am beginner in Storm.I am trying to understand the processing flow of
> > storm by following questions.
> >
> > 1. I have a topology(1 spout and 3 bolts) and 3 machines(1 nimbus and 2
> > worker nodes).Whether I want to run my topology on all 3 nodes or run on
> > nimbus is enough? Is it nimbus will take care of distributing the code to
> > other nodes?
> >
> > 2. My spout will run on nimbus or any of the worker nodes?
> >
> > 3. The 3 bolts are run on 3 separate nodes or run on same node? is it
> will
> > take care by nimbus?
> >
> > 4. How do we track processing of bolt in nodes?
> >
> > 5. Is it any documentation available to understand the complete flow of
> > processing a message in Storm?
> >
> > --
> > by
> >
> > *R.RAMPRASAD*
>
>
--
by
*R.RAMPRASAD*