You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@storm.apache.org by Junguk Cho <jm...@gmail.com> on 2016/06/27 15:41:51 UTC
What is roles of __system-executor?
Hi,
In one worker, by default Storm creates one-executor and
one-executor-disruptor threads pair.
In detail, it created __system, __acker, and __bolt or __spout.
I understood __acker and __bolt or __spout, but what is the roles of
__system and __system-disruptor threads?
Another question is a benchmark.
I found one from Yahoo which they used it for netty vs zeromq test (
https://github.com/yahoo/storm-perf-test).
Are there well built benchmark like this to measure storm performances?
Thanks,
Junguk
Re: What is roles of __system-executor?
Posted by Jungtaek Lim <ka...@gmail.com>.
Here is JIRA link for Apache Stotm :
https://issues.apache.org/jira/browse/STORM
There's no format for filing an issue, but you're encouraged to provide
enough information to reproduce, or detailed symptom.
Please set affected versions accordingly. It should reduce contributors to
spent time with other versions.
Thanks,
Jungtaek Lim (HeartSaVioR)
2016년 7월 6일 (수) 오후 12:32, Junguk Cho <jm...@gmail.com>님이 작성:
> Hi, Jungtaek.
>
> Thank you for reply.
>
> Just a curious, do you have a reason to disable SystemBolt?
> -> Not really. I just want to use the most basic bolt. So, it is easy to
> understand storm framework.
>
>
> Regarding __acker, if you set acker to 0 by modifying config and it doesn't
> work, please file a JIRA issue.
> -> Thanks. I will try.
>
> Filing a JIRA issue means reporting this issue to somewhere?
> Is there a format for this? If so, please let me know.
> If I can, I will do it.
>
> Thanks,
> Junguk
>
>
> 2016-07-05 22:44 GMT-04:00 Jungtaek Lim <ka...@gmail.com>:
>
> > Hi Junguk,
> >
> > Topology metric consumer receives task level metrics for all tasks, and
> > SystemBolt is a task which exists only one for all workers to gather
> worker
> > specific metrics.
> > Please refer http://storm.apache.org/releases/1.0.1/Metrics.html for
> more
> > details on metrics feature.
> > Just a curious, do you have a reason to disable SystemBolt?
> >
> > Regarding __acker, if you set acker to 0 by modifying config and it
> doesn't
> > work, please file a JIRA issue.
> >
> > Thanks,
> > Jungtaek Lim (HeartSaVioR)
> >
> >
> > 2016년 7월 6일 (수) 오전 11:01, Junguk Cho <jm...@gmail.com>님이 작성:
> >
> > > One more following question about __acker bolt.
> > >
> > > Even though I created un-anchored topology (
> > >
> > >
> >
> http://storm.apache.org/releases/current/Guaranteeing-message-processing.html
> > > ),
> > > it still created __acker bolt in each worker.
> > >
> > > Are there reasons?
> > >
> > > Thanks,
> > > Junguk
> > >
> > > 2016-07-05 21:45 GMT-04:00 Junguk Cho <jm...@gmail.com>:
> > >
> > > > Hi,
> > > >
> > > > Thank you for reply.
> > > > I checked the code.
> > > >
> > > > I have some questions.
> > > > There is no code for sending data to "central node" by using
> "emit-like
> > > > function".
> > > > Who will get the worker level metrics from this SystemBolt?
> > > >
> > > > It seemed we cannot disable this SystemBolt.
> > > > Can we also disable this in storm.yaml?
> > > >
> > > >
> > > > Thanks,
> > > > Junguk
> > > >
> > > > 2016-06-28 9:57 GMT-04:00 Bobby Evans <ev...@yahoo-inc.com.invalid>:
> > > >
> > > >> Every worker gets a SystemBolt. The bolt does not really process
> any
> > > >> data, it just provides a set of worker level metrics.
> > > >>
> > > >>
> > >
> >
> https://github.com/apache/storm/blob/master/storm-core/src/jvm/org/apache/storm/metric/SystemBolt.java
> > > >> - Bobby
> > > >>
> > > >> On Monday, June 27, 2016 10:42 AM, Junguk Cho <
> jmanbal@gmail.com>
> > > >> wrote:
> > > >>
> > > >>
> > > >> Hi,
> > > >>
> > > >> In one worker, by default Storm creates one-executor and
> > > >> one-executor-disruptor threads pair.
> > > >>
> > > >> In detail, it created __system, __acker, and __bolt or __spout.
> > > >> I understood __acker and __bolt or __spout, but what is the roles of
> > > >> __system and __system-disruptor threads?
> > > >>
> > > >>
> > > >> Another question is a benchmark.
> > > >> I found one from Yahoo which they used it for netty vs zeromq test (
> > > >> https://github.com/yahoo/storm-perf-test).
> > > >>
> > > >> Are there well built benchmark like this to measure storm
> > performances?
> > > >>
> > > >> Thanks,
> > > >> Junguk
> > > >>
> > > >>
> > > >>
> > > >>
> > > >
> > > >
> > >
> >
>
Re: What is roles of __system-executor?
Posted by Junguk Cho <jm...@gmail.com>.
Hi, Jungtaek.
Thank you for reply.
Just a curious, do you have a reason to disable SystemBolt?
-> Not really. I just want to use the most basic bolt. So, it is easy to
understand storm framework.
Regarding __acker, if you set acker to 0 by modifying config and it doesn't
work, please file a JIRA issue.
-> Thanks. I will try.
Filing a JIRA issue means reporting this issue to somewhere?
Is there a format for this? If so, please let me know.
If I can, I will do it.
Thanks,
Junguk
2016-07-05 22:44 GMT-04:00 Jungtaek Lim <ka...@gmail.com>:
> Hi Junguk,
>
> Topology metric consumer receives task level metrics for all tasks, and
> SystemBolt is a task which exists only one for all workers to gather worker
> specific metrics.
> Please refer http://storm.apache.org/releases/1.0.1/Metrics.html for more
> details on metrics feature.
> Just a curious, do you have a reason to disable SystemBolt?
>
> Regarding __acker, if you set acker to 0 by modifying config and it doesn't
> work, please file a JIRA issue.
>
> Thanks,
> Jungtaek Lim (HeartSaVioR)
>
>
> 2016년 7월 6일 (수) 오전 11:01, Junguk Cho <jm...@gmail.com>님이 작성:
>
> > One more following question about __acker bolt.
> >
> > Even though I created un-anchored topology (
> >
> >
> http://storm.apache.org/releases/current/Guaranteeing-message-processing.html
> > ),
> > it still created __acker bolt in each worker.
> >
> > Are there reasons?
> >
> > Thanks,
> > Junguk
> >
> > 2016-07-05 21:45 GMT-04:00 Junguk Cho <jm...@gmail.com>:
> >
> > > Hi,
> > >
> > > Thank you for reply.
> > > I checked the code.
> > >
> > > I have some questions.
> > > There is no code for sending data to "central node" by using "emit-like
> > > function".
> > > Who will get the worker level metrics from this SystemBolt?
> > >
> > > It seemed we cannot disable this SystemBolt.
> > > Can we also disable this in storm.yaml?
> > >
> > >
> > > Thanks,
> > > Junguk
> > >
> > > 2016-06-28 9:57 GMT-04:00 Bobby Evans <ev...@yahoo-inc.com.invalid>:
> > >
> > >> Every worker gets a SystemBolt. The bolt does not really process any
> > >> data, it just provides a set of worker level metrics.
> > >>
> > >>
> >
> https://github.com/apache/storm/blob/master/storm-core/src/jvm/org/apache/storm/metric/SystemBolt.java
> > >> - Bobby
> > >>
> > >> On Monday, June 27, 2016 10:42 AM, Junguk Cho <jm...@gmail.com>
> > >> wrote:
> > >>
> > >>
> > >> Hi,
> > >>
> > >> In one worker, by default Storm creates one-executor and
> > >> one-executor-disruptor threads pair.
> > >>
> > >> In detail, it created __system, __acker, and __bolt or __spout.
> > >> I understood __acker and __bolt or __spout, but what is the roles of
> > >> __system and __system-disruptor threads?
> > >>
> > >>
> > >> Another question is a benchmark.
> > >> I found one from Yahoo which they used it for netty vs zeromq test (
> > >> https://github.com/yahoo/storm-perf-test).
> > >>
> > >> Are there well built benchmark like this to measure storm
> performances?
> > >>
> > >> Thanks,
> > >> Junguk
> > >>
> > >>
> > >>
> > >>
> > >
> > >
> >
>
Re: What is roles of __system-executor?
Posted by Jungtaek Lim <ka...@gmail.com>.
Hi Junguk,
Topology metric consumer receives task level metrics for all tasks, and
SystemBolt is a task which exists only one for all workers to gather worker
specific metrics.
Please refer http://storm.apache.org/releases/1.0.1/Metrics.html for more
details on metrics feature.
Just a curious, do you have a reason to disable SystemBolt?
Regarding __acker, if you set acker to 0 by modifying config and it doesn't
work, please file a JIRA issue.
Thanks,
Jungtaek Lim (HeartSaVioR)
2016년 7월 6일 (수) 오전 11:01, Junguk Cho <jm...@gmail.com>님이 작성:
> One more following question about __acker bolt.
>
> Even though I created un-anchored topology (
>
> http://storm.apache.org/releases/current/Guaranteeing-message-processing.html
> ),
> it still created __acker bolt in each worker.
>
> Are there reasons?
>
> Thanks,
> Junguk
>
> 2016-07-05 21:45 GMT-04:00 Junguk Cho <jm...@gmail.com>:
>
> > Hi,
> >
> > Thank you for reply.
> > I checked the code.
> >
> > I have some questions.
> > There is no code for sending data to "central node" by using "emit-like
> > function".
> > Who will get the worker level metrics from this SystemBolt?
> >
> > It seemed we cannot disable this SystemBolt.
> > Can we also disable this in storm.yaml?
> >
> >
> > Thanks,
> > Junguk
> >
> > 2016-06-28 9:57 GMT-04:00 Bobby Evans <ev...@yahoo-inc.com.invalid>:
> >
> >> Every worker gets a SystemBolt. The bolt does not really process any
> >> data, it just provides a set of worker level metrics.
> >>
> >>
> https://github.com/apache/storm/blob/master/storm-core/src/jvm/org/apache/storm/metric/SystemBolt.java
> >> - Bobby
> >>
> >> On Monday, June 27, 2016 10:42 AM, Junguk Cho <jm...@gmail.com>
> >> wrote:
> >>
> >>
> >> Hi,
> >>
> >> In one worker, by default Storm creates one-executor and
> >> one-executor-disruptor threads pair.
> >>
> >> In detail, it created __system, __acker, and __bolt or __spout.
> >> I understood __acker and __bolt or __spout, but what is the roles of
> >> __system and __system-disruptor threads?
> >>
> >>
> >> Another question is a benchmark.
> >> I found one from Yahoo which they used it for netty vs zeromq test (
> >> https://github.com/yahoo/storm-perf-test).
> >>
> >> Are there well built benchmark like this to measure storm performances?
> >>
> >> Thanks,
> >> Junguk
> >>
> >>
> >>
> >>
> >
> >
>
Re: What is roles of __system-executor?
Posted by Junguk Cho <jm...@gmail.com>.
One more following question about __acker bolt.
Even though I created un-anchored topology (
http://storm.apache.org/releases/current/Guaranteeing-message-processing.html),
it still created __acker bolt in each worker.
Are there reasons?
Thanks,
Junguk
2016-07-05 21:45 GMT-04:00 Junguk Cho <jm...@gmail.com>:
> Hi,
>
> Thank you for reply.
> I checked the code.
>
> I have some questions.
> There is no code for sending data to "central node" by using "emit-like
> function".
> Who will get the worker level metrics from this SystemBolt?
>
> It seemed we cannot disable this SystemBolt.
> Can we also disable this in storm.yaml?
>
>
> Thanks,
> Junguk
>
> 2016-06-28 9:57 GMT-04:00 Bobby Evans <ev...@yahoo-inc.com.invalid>:
>
>> Every worker gets a SystemBolt. The bolt does not really process any
>> data, it just provides a set of worker level metrics.
>>
>> https://github.com/apache/storm/blob/master/storm-core/src/jvm/org/apache/storm/metric/SystemBolt.java
>> - Bobby
>>
>> On Monday, June 27, 2016 10:42 AM, Junguk Cho <jm...@gmail.com>
>> wrote:
>>
>>
>> Hi,
>>
>> In one worker, by default Storm creates one-executor and
>> one-executor-disruptor threads pair.
>>
>> In detail, it created __system, __acker, and __bolt or __spout.
>> I understood __acker and __bolt or __spout, but what is the roles of
>> __system and __system-disruptor threads?
>>
>>
>> Another question is a benchmark.
>> I found one from Yahoo which they used it for netty vs zeromq test (
>> https://github.com/yahoo/storm-perf-test).
>>
>> Are there well built benchmark like this to measure storm performances?
>>
>> Thanks,
>> Junguk
>>
>>
>>
>>
>
>
Re: What is roles of __system-executor?
Posted by Junguk Cho <jm...@gmail.com>.
Hi,
Thank you for reply.
I checked the code.
I have some questions.
There is no code for sending data to "central node" by using "emit-like
function".
Who will get the worker level metrics from this SystemBolt?
It seemed we cannot disable this SystemBolt.
Can we also disable this in storm.yaml?
Thanks,
Junguk
2016-06-28 9:57 GMT-04:00 Bobby Evans <ev...@yahoo-inc.com.invalid>:
> Every worker gets a SystemBolt. The bolt does not really process any
> data, it just provides a set of worker level metrics.
>
> https://github.com/apache/storm/blob/master/storm-core/src/jvm/org/apache/storm/metric/SystemBolt.java
> - Bobby
>
> On Monday, June 27, 2016 10:42 AM, Junguk Cho <jm...@gmail.com>
> wrote:
>
>
> Hi,
>
> In one worker, by default Storm creates one-executor and
> one-executor-disruptor threads pair.
>
> In detail, it created __system, __acker, and __bolt or __spout.
> I understood __acker and __bolt or __spout, but what is the roles of
> __system and __system-disruptor threads?
>
>
> Another question is a benchmark.
> I found one from Yahoo which they used it for netty vs zeromq test (
> https://github.com/yahoo/storm-perf-test).
>
> Are there well built benchmark like this to measure storm performances?
>
> Thanks,
> Junguk
>
>
>
>
Re: What is roles of __system-executor?
Posted by Bobby Evans <ev...@yahoo-inc.com.INVALID>.
Every worker gets a SystemBolt. The bolt does not really process any data, it just provides a set of worker level metrics.
https://github.com/apache/storm/blob/master/storm-core/src/jvm/org/apache/storm/metric/SystemBolt.java
- Bobby
On Monday, June 27, 2016 10:42 AM, Junguk Cho <jm...@gmail.com> wrote:
Hi,
In one worker, by default Storm creates one-executor and
one-executor-disruptor threads pair.
In detail, it created __system, __acker, and __bolt or __spout.
I understood __acker and __bolt or __spout, but what is the roles of
__system and __system-disruptor threads?
Another question is a benchmark.
I found one from Yahoo which they used it for netty vs zeromq test (
https://github.com/yahoo/storm-perf-test).
Are there well built benchmark like this to measure storm performances?
Thanks,
Junguk