You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@storm.apache.org by S G <sg...@gmail.com> on 2017/03/16 19:15:56 UTC
Too many threads per bolt instance (as seen with jconsole images)
Hi,
I am trying to make sense of number of threads seen in JConsole.
It seems like a very high number of threads are launched per bolt thread.
*Experiment 1*
topology.workers.val=16
spout.parallelism.val=1
bolt1.parallelism.val=900
bolt2.parallelism.val=160
observation:
Threads seen per node in jconsole = 1200
Bolt threads per node = (900 + 160)/16 = 66
Threads per bolt = 1200/66 = 18
*Experiment 2*
topology.workers.val=16
spout.parallelism.val=1
bolt1.parallelism.val=16
bolt2.parallelism.val=16
observation:
Threads seen per node in jconsole = 61
Bolt threads per node = (16+16)/16 = 2
Threads per bolt = 61/2 = 30
There are no other topologies running in my cluster.
There are no other spouts/bolts running in my cluster except the ones
mentioned above.
I am running only one worker process per machine.
So the question is how many threads per bolt are launched by storm?
I am not interested in the exact number, but concerned about the high
number of extra threads (18+ thread) for running a single bolt.
Can we limit or optimize it somehow?
Or if all of them are required, it would be good to document them somewhere.
Thanks
SG
Re: Too many threads per bolt instance (as seen with jconsole
images)
Posted by Roshan Naik <ro...@hortonworks.com>.
Can you provide a jstack dump of all the threads in one worker ?
-roshan
On 3/16/17, 8:40 PM, "S G" <sg...@gmail.com> wrote:
I did not say 3 thds per spout/bolt.
I said "When 60 bolts are run per node, there are 1200 total threads/node"
That gets me 20 threads/bolt.
And I am not creating any new threads inside my bolts.
On Thu, Mar 16, 2017 at 6:12 PM, Roshan Naik <ro...@hortonworks.com> wrote:
> I assume you mean ..3 thds per spout/bolt is on the high side ?
> Currently there is an executor thd, a xsfer thd and flusher thd. The plan
> (in the redesign – STORM-2284 subtask 2) is to have 1 thread per spout/bolt.
> Also have some (untested) thoughts on reducing the number of remaining
> threads in the worker.
> -roshan
>
>
> On 3/16/17, 5:07 PM, "S G" <sg...@gmail.com> wrote:
>
> Thanks for sharing the doc Roshan.
> It is very informative.
>
> I think number of threads per bolt is on the high side (When 60 bolts
> are
> run per node, there are 1200 total threads/node).
>
> Some of these would be essential for the worker's book-keeping but
> still it
> seems we can get a lot of performance boost if we can somehow reduce
> these
> threads and the associated context switching between them.
>
>
> On Thu, Mar 16, 2017 at 12:53 PM, Roshan Naik <ro...@hortonworks.com>
> wrote:
>
> > Typically there are 3 threads per spout or bolt instance (unless the
> > spout or bolt is spawning its own threads). There are the acker
> bolts,
> > event loggers and system bolt running there too.
> >
> > Then there are several more per worker. See a summary of it in
> section 3.1
> > of this document
> >
> > https://docs.google.com/document/d/1EzeHL3d7EE-
> > RyyBEpN7CwRmWz3oqjbbKiVVAlzFp2Nc/edit?usp=sharing
> >
> >
> >
> >
> > Get Outlook for iOS<https://aka.ms/o0ukef>
> >
> >
> >
> >
> > On Thu, Mar 16, 2017 at 12:16 PM -0700, "S G" <
> sg.online.email@gmail.com<
> > mailto:sg.online.email@gmail.com>> wrote:
> >
> > Hi,
> >
> > I am trying to make sense of number of threads seen in JConsole.
> > It seems like a very high number of threads are launched per bolt
> thread.
> >
> >
> > [cid:ii_j0crtbtm0_15ad8874d0924ae2]
> >
> > Experiment 1
> > topology.workers.val=16
> > spout.parallelism.val=1
> > bolt1.parallelism.val=900
> > bolt2.parallelism.val=160
> > observation:
> > Threads seen per node in jconsole = 1200
> > Bolt threads per node = (900 + 160)/16 = 66
> > Threads per bolt = 1200/66 = 18
> >
> >
> > Experiment 2
> > topology.workers.val=16
> > spout.parallelism.val=1
> > bolt1.parallelism.val=16
> > bolt2.parallelism.val=16
> > observation:
> > Threads seen per node in jconsole = 61
> > Bolt threads per node = (16+16)/16 = 2
> > Threads per bolt = 61/2 = 30
> >
> > There are no other topologies running in my cluster.
> > There are no other spouts/bolts running in my cluster except the ones
> > mentioned above.
> > I am running only one worker process per machine.
> >
> >
> > So the question is how many threads per bolt are launched by storm?
> > I am not interested in the exact number, but concerned about the high
> > number of extra threads (18+ thread) for running a single bolt.
> >
> > Can we limit or optimize it somehow?
> > Or if all of them are required, it would be good to document them
> > somewhere.
> >
> > Thanks
> > SG
> >
> >
> >
> >
>
>
>
Re: Too many threads per bolt instance (as seen with jconsole
images)
Posted by Roshan Naik <ro...@hortonworks.com>.
Can you provide a jstack dump of all the threads in one worker ?
-roshan
On 3/16/17, 8:40 PM, "S G" <sg...@gmail.com> wrote:
I did not say 3 thds per spout/bolt.
I said "When 60 bolts are run per node, there are 1200 total threads/node"
That gets me 20 threads/bolt.
And I am not creating any new threads inside my bolts.
On Thu, Mar 16, 2017 at 6:12 PM, Roshan Naik <ro...@hortonworks.com> wrote:
> I assume you mean ..3 thds per spout/bolt is on the high side ?
> Currently there is an executor thd, a xsfer thd and flusher thd. The plan
> (in the redesign – STORM-2284 subtask 2) is to have 1 thread per spout/bolt.
> Also have some (untested) thoughts on reducing the number of remaining
> threads in the worker.
> -roshan
>
>
> On 3/16/17, 5:07 PM, "S G" <sg...@gmail.com> wrote:
>
> Thanks for sharing the doc Roshan.
> It is very informative.
>
> I think number of threads per bolt is on the high side (When 60 bolts
> are
> run per node, there are 1200 total threads/node).
>
> Some of these would be essential for the worker's book-keeping but
> still it
> seems we can get a lot of performance boost if we can somehow reduce
> these
> threads and the associated context switching between them.
>
>
> On Thu, Mar 16, 2017 at 12:53 PM, Roshan Naik <ro...@hortonworks.com>
> wrote:
>
> > Typically there are 3 threads per spout or bolt instance (unless the
> > spout or bolt is spawning its own threads). There are the acker
> bolts,
> > event loggers and system bolt running there too.
> >
> > Then there are several more per worker. See a summary of it in
> section 3.1
> > of this document
> >
> > https://docs.google.com/document/d/1EzeHL3d7EE-
> > RyyBEpN7CwRmWz3oqjbbKiVVAlzFp2Nc/edit?usp=sharing
> >
> >
> >
> >
> > Get Outlook for iOS<https://aka.ms/o0ukef>
> >
> >
> >
> >
> > On Thu, Mar 16, 2017 at 12:16 PM -0700, "S G" <
> sg.online.email@gmail.com<
> > mailto:sg.online.email@gmail.com>> wrote:
> >
> > Hi,
> >
> > I am trying to make sense of number of threads seen in JConsole.
> > It seems like a very high number of threads are launched per bolt
> thread.
> >
> >
> > [cid:ii_j0crtbtm0_15ad8874d0924ae2]
> >
> > Experiment 1
> > topology.workers.val=16
> > spout.parallelism.val=1
> > bolt1.parallelism.val=900
> > bolt2.parallelism.val=160
> > observation:
> > Threads seen per node in jconsole = 1200
> > Bolt threads per node = (900 + 160)/16 = 66
> > Threads per bolt = 1200/66 = 18
> >
> >
> > Experiment 2
> > topology.workers.val=16
> > spout.parallelism.val=1
> > bolt1.parallelism.val=16
> > bolt2.parallelism.val=16
> > observation:
> > Threads seen per node in jconsole = 61
> > Bolt threads per node = (16+16)/16 = 2
> > Threads per bolt = 61/2 = 30
> >
> > There are no other topologies running in my cluster.
> > There are no other spouts/bolts running in my cluster except the ones
> > mentioned above.
> > I am running only one worker process per machine.
> >
> >
> > So the question is how many threads per bolt are launched by storm?
> > I am not interested in the exact number, but concerned about the high
> > number of extra threads (18+ thread) for running a single bolt.
> >
> > Can we limit or optimize it somehow?
> > Or if all of them are required, it would be good to document them
> > somewhere.
> >
> > Thanks
> > SG
> >
> >
> >
> >
>
>
>
Re: Too many threads per bolt instance (as seen with jconsole images)
Posted by S G <sg...@gmail.com>.
I did not say 3 thds per spout/bolt.
I said "When 60 bolts are run per node, there are 1200 total threads/node"
That gets me 20 threads/bolt.
And I am not creating any new threads inside my bolts.
On Thu, Mar 16, 2017 at 6:12 PM, Roshan Naik <ro...@hortonworks.com> wrote:
> I assume you mean ..3 thds per spout/bolt is on the high side ?
> Currently there is an executor thd, a xsfer thd and flusher thd. The plan
> (in the redesign – STORM-2284 subtask 2) is to have 1 thread per spout/bolt.
> Also have some (untested) thoughts on reducing the number of remaining
> threads in the worker.
> -roshan
>
>
> On 3/16/17, 5:07 PM, "S G" <sg...@gmail.com> wrote:
>
> Thanks for sharing the doc Roshan.
> It is very informative.
>
> I think number of threads per bolt is on the high side (When 60 bolts
> are
> run per node, there are 1200 total threads/node).
>
> Some of these would be essential for the worker's book-keeping but
> still it
> seems we can get a lot of performance boost if we can somehow reduce
> these
> threads and the associated context switching between them.
>
>
> On Thu, Mar 16, 2017 at 12:53 PM, Roshan Naik <ro...@hortonworks.com>
> wrote:
>
> > Typically there are 3 threads per spout or bolt instance (unless the
> > spout or bolt is spawning its own threads). There are the acker
> bolts,
> > event loggers and system bolt running there too.
> >
> > Then there are several more per worker. See a summary of it in
> section 3.1
> > of this document
> >
> > https://docs.google.com/document/d/1EzeHL3d7EE-
> > RyyBEpN7CwRmWz3oqjbbKiVVAlzFp2Nc/edit?usp=sharing
> >
> >
> >
> >
> > Get Outlook for iOS<https://aka.ms/o0ukef>
> >
> >
> >
> >
> > On Thu, Mar 16, 2017 at 12:16 PM -0700, "S G" <
> sg.online.email@gmail.com<
> > mailto:sg.online.email@gmail.com>> wrote:
> >
> > Hi,
> >
> > I am trying to make sense of number of threads seen in JConsole.
> > It seems like a very high number of threads are launched per bolt
> thread.
> >
> >
> > [cid:ii_j0crtbtm0_15ad8874d0924ae2]
> >
> > Experiment 1
> > topology.workers.val=16
> > spout.parallelism.val=1
> > bolt1.parallelism.val=900
> > bolt2.parallelism.val=160
> > observation:
> > Threads seen per node in jconsole = 1200
> > Bolt threads per node = (900 + 160)/16 = 66
> > Threads per bolt = 1200/66 = 18
> >
> >
> > Experiment 2
> > topology.workers.val=16
> > spout.parallelism.val=1
> > bolt1.parallelism.val=16
> > bolt2.parallelism.val=16
> > observation:
> > Threads seen per node in jconsole = 61
> > Bolt threads per node = (16+16)/16 = 2
> > Threads per bolt = 61/2 = 30
> >
> > There are no other topologies running in my cluster.
> > There are no other spouts/bolts running in my cluster except the ones
> > mentioned above.
> > I am running only one worker process per machine.
> >
> >
> > So the question is how many threads per bolt are launched by storm?
> > I am not interested in the exact number, but concerned about the high
> > number of extra threads (18+ thread) for running a single bolt.
> >
> > Can we limit or optimize it somehow?
> > Or if all of them are required, it would be good to document them
> > somewhere.
> >
> > Thanks
> > SG
> >
> >
> >
> >
>
>
>
Re: Too many threads per bolt instance (as seen with jconsole images)
Posted by S G <sg...@gmail.com>.
I did not say 3 thds per spout/bolt.
I said "When 60 bolts are run per node, there are 1200 total threads/node"
That gets me 20 threads/bolt.
And I am not creating any new threads inside my bolts.
On Thu, Mar 16, 2017 at 6:12 PM, Roshan Naik <ro...@hortonworks.com> wrote:
> I assume you mean ..3 thds per spout/bolt is on the high side ?
> Currently there is an executor thd, a xsfer thd and flusher thd. The plan
> (in the redesign – STORM-2284 subtask 2) is to have 1 thread per spout/bolt.
> Also have some (untested) thoughts on reducing the number of remaining
> threads in the worker.
> -roshan
>
>
> On 3/16/17, 5:07 PM, "S G" <sg...@gmail.com> wrote:
>
> Thanks for sharing the doc Roshan.
> It is very informative.
>
> I think number of threads per bolt is on the high side (When 60 bolts
> are
> run per node, there are 1200 total threads/node).
>
> Some of these would be essential for the worker's book-keeping but
> still it
> seems we can get a lot of performance boost if we can somehow reduce
> these
> threads and the associated context switching between them.
>
>
> On Thu, Mar 16, 2017 at 12:53 PM, Roshan Naik <ro...@hortonworks.com>
> wrote:
>
> > Typically there are 3 threads per spout or bolt instance (unless the
> > spout or bolt is spawning its own threads). There are the acker
> bolts,
> > event loggers and system bolt running there too.
> >
> > Then there are several more per worker. See a summary of it in
> section 3.1
> > of this document
> >
> > https://docs.google.com/document/d/1EzeHL3d7EE-
> > RyyBEpN7CwRmWz3oqjbbKiVVAlzFp2Nc/edit?usp=sharing
> >
> >
> >
> >
> > Get Outlook for iOS<https://aka.ms/o0ukef>
> >
> >
> >
> >
> > On Thu, Mar 16, 2017 at 12:16 PM -0700, "S G" <
> sg.online.email@gmail.com<
> > mailto:sg.online.email@gmail.com>> wrote:
> >
> > Hi,
> >
> > I am trying to make sense of number of threads seen in JConsole.
> > It seems like a very high number of threads are launched per bolt
> thread.
> >
> >
> > [cid:ii_j0crtbtm0_15ad8874d0924ae2]
> >
> > Experiment 1
> > topology.workers.val=16
> > spout.parallelism.val=1
> > bolt1.parallelism.val=900
> > bolt2.parallelism.val=160
> > observation:
> > Threads seen per node in jconsole = 1200
> > Bolt threads per node = (900 + 160)/16 = 66
> > Threads per bolt = 1200/66 = 18
> >
> >
> > Experiment 2
> > topology.workers.val=16
> > spout.parallelism.val=1
> > bolt1.parallelism.val=16
> > bolt2.parallelism.val=16
> > observation:
> > Threads seen per node in jconsole = 61
> > Bolt threads per node = (16+16)/16 = 2
> > Threads per bolt = 61/2 = 30
> >
> > There are no other topologies running in my cluster.
> > There are no other spouts/bolts running in my cluster except the ones
> > mentioned above.
> > I am running only one worker process per machine.
> >
> >
> > So the question is how many threads per bolt are launched by storm?
> > I am not interested in the exact number, but concerned about the high
> > number of extra threads (18+ thread) for running a single bolt.
> >
> > Can we limit or optimize it somehow?
> > Or if all of them are required, it would be good to document them
> > somewhere.
> >
> > Thanks
> > SG
> >
> >
> >
> >
>
>
>
Re: Too many threads per bolt instance (as seen with jconsole
images)
Posted by Roshan Naik <ro...@hortonworks.com>.
I assume you mean ..3 thds per spout/bolt is on the high side ?
Currently there is an executor thd, a xsfer thd and flusher thd. The plan (in the redesign – STORM-2284 subtask 2) is to have 1 thread per spout/bolt.
Also have some (untested) thoughts on reducing the number of remaining threads in the worker.
-roshan
On 3/16/17, 5:07 PM, "S G" <sg...@gmail.com> wrote:
Thanks for sharing the doc Roshan.
It is very informative.
I think number of threads per bolt is on the high side (When 60 bolts are
run per node, there are 1200 total threads/node).
Some of these would be essential for the worker's book-keeping but still it
seems we can get a lot of performance boost if we can somehow reduce these
threads and the associated context switching between them.
On Thu, Mar 16, 2017 at 12:53 PM, Roshan Naik <ro...@hortonworks.com>
wrote:
> Typically there are 3 threads per spout or bolt instance (unless the
> spout or bolt is spawning its own threads). There are the acker bolts,
> event loggers and system bolt running there too.
>
> Then there are several more per worker. See a summary of it in section 3.1
> of this document
>
> https://docs.google.com/document/d/1EzeHL3d7EE-
> RyyBEpN7CwRmWz3oqjbbKiVVAlzFp2Nc/edit?usp=sharing
>
>
>
>
> Get Outlook for iOS<https://aka.ms/o0ukef>
>
>
>
>
> On Thu, Mar 16, 2017 at 12:16 PM -0700, "S G" <sg.online.email@gmail.com<
> mailto:sg.online.email@gmail.com>> wrote:
>
> Hi,
>
> I am trying to make sense of number of threads seen in JConsole.
> It seems like a very high number of threads are launched per bolt thread.
>
>
> [cid:ii_j0crtbtm0_15ad8874d0924ae2]
>
> Experiment 1
> topology.workers.val=16
> spout.parallelism.val=1
> bolt1.parallelism.val=900
> bolt2.parallelism.val=160
> observation:
> Threads seen per node in jconsole = 1200
> Bolt threads per node = (900 + 160)/16 = 66
> Threads per bolt = 1200/66 = 18
>
>
> Experiment 2
> topology.workers.val=16
> spout.parallelism.val=1
> bolt1.parallelism.val=16
> bolt2.parallelism.val=16
> observation:
> Threads seen per node in jconsole = 61
> Bolt threads per node = (16+16)/16 = 2
> Threads per bolt = 61/2 = 30
>
> There are no other topologies running in my cluster.
> There are no other spouts/bolts running in my cluster except the ones
> mentioned above.
> I am running only one worker process per machine.
>
>
> So the question is how many threads per bolt are launched by storm?
> I am not interested in the exact number, but concerned about the high
> number of extra threads (18+ thread) for running a single bolt.
>
> Can we limit or optimize it somehow?
> Or if all of them are required, it would be good to document them
> somewhere.
>
> Thanks
> SG
>
>
>
>
Re: Too many threads per bolt instance (as seen with jconsole
images)
Posted by Roshan Naik <ro...@hortonworks.com>.
I assume you mean ..3 thds per spout/bolt is on the high side ?
Currently there is an executor thd, a xsfer thd and flusher thd. The plan (in the redesign – STORM-2284 subtask 2) is to have 1 thread per spout/bolt.
Also have some (untested) thoughts on reducing the number of remaining threads in the worker.
-roshan
On 3/16/17, 5:07 PM, "S G" <sg...@gmail.com> wrote:
Thanks for sharing the doc Roshan.
It is very informative.
I think number of threads per bolt is on the high side (When 60 bolts are
run per node, there are 1200 total threads/node).
Some of these would be essential for the worker's book-keeping but still it
seems we can get a lot of performance boost if we can somehow reduce these
threads and the associated context switching between them.
On Thu, Mar 16, 2017 at 12:53 PM, Roshan Naik <ro...@hortonworks.com>
wrote:
> Typically there are 3 threads per spout or bolt instance (unless the
> spout or bolt is spawning its own threads). There are the acker bolts,
> event loggers and system bolt running there too.
>
> Then there are several more per worker. See a summary of it in section 3.1
> of this document
>
> https://docs.google.com/document/d/1EzeHL3d7EE-
> RyyBEpN7CwRmWz3oqjbbKiVVAlzFp2Nc/edit?usp=sharing
>
>
>
>
> Get Outlook for iOS<https://aka.ms/o0ukef>
>
>
>
>
> On Thu, Mar 16, 2017 at 12:16 PM -0700, "S G" <sg.online.email@gmail.com<
> mailto:sg.online.email@gmail.com>> wrote:
>
> Hi,
>
> I am trying to make sense of number of threads seen in JConsole.
> It seems like a very high number of threads are launched per bolt thread.
>
>
> [cid:ii_j0crtbtm0_15ad8874d0924ae2]
>
> Experiment 1
> topology.workers.val=16
> spout.parallelism.val=1
> bolt1.parallelism.val=900
> bolt2.parallelism.val=160
> observation:
> Threads seen per node in jconsole = 1200
> Bolt threads per node = (900 + 160)/16 = 66
> Threads per bolt = 1200/66 = 18
>
>
> Experiment 2
> topology.workers.val=16
> spout.parallelism.val=1
> bolt1.parallelism.val=16
> bolt2.parallelism.val=16
> observation:
> Threads seen per node in jconsole = 61
> Bolt threads per node = (16+16)/16 = 2
> Threads per bolt = 61/2 = 30
>
> There are no other topologies running in my cluster.
> There are no other spouts/bolts running in my cluster except the ones
> mentioned above.
> I am running only one worker process per machine.
>
>
> So the question is how many threads per bolt are launched by storm?
> I am not interested in the exact number, but concerned about the high
> number of extra threads (18+ thread) for running a single bolt.
>
> Can we limit or optimize it somehow?
> Or if all of them are required, it would be good to document them
> somewhere.
>
> Thanks
> SG
>
>
>
>
Re: Too many threads per bolt instance (as seen with jconsole images)
Posted by S G <sg...@gmail.com>.
Thanks for sharing the doc Roshan.
It is very informative.
I think number of threads per bolt is on the high side (When 60 bolts are
run per node, there are 1200 total threads/node).
Some of these would be essential for the worker's book-keeping but still it
seems we can get a lot of performance boost if we can somehow reduce these
threads and the associated context switching between them.
On Thu, Mar 16, 2017 at 12:53 PM, Roshan Naik <ro...@hortonworks.com>
wrote:
> Typically there are 3 threads per spout or bolt instance (unless the
> spout or bolt is spawning its own threads). There are the acker bolts,
> event loggers and system bolt running there too.
>
> Then there are several more per worker. See a summary of it in section 3.1
> of this document
>
> https://docs.google.com/document/d/1EzeHL3d7EE-
> RyyBEpN7CwRmWz3oqjbbKiVVAlzFp2Nc/edit?usp=sharing
>
>
>
>
> Get Outlook for iOS<https://aka.ms/o0ukef>
>
>
>
>
> On Thu, Mar 16, 2017 at 12:16 PM -0700, "S G" <sg.online.email@gmail.com<
> mailto:sg.online.email@gmail.com>> wrote:
>
> Hi,
>
> I am trying to make sense of number of threads seen in JConsole.
> It seems like a very high number of threads are launched per bolt thread.
>
>
> [cid:ii_j0crtbtm0_15ad8874d0924ae2]
>
> Experiment 1
> topology.workers.val=16
> spout.parallelism.val=1
> bolt1.parallelism.val=900
> bolt2.parallelism.val=160
> observation:
> Threads seen per node in jconsole = 1200
> Bolt threads per node = (900 + 160)/16 = 66
> Threads per bolt = 1200/66 = 18
>
>
> Experiment 2
> topology.workers.val=16
> spout.parallelism.val=1
> bolt1.parallelism.val=16
> bolt2.parallelism.val=16
> observation:
> Threads seen per node in jconsole = 61
> Bolt threads per node = (16+16)/16 = 2
> Threads per bolt = 61/2 = 30
>
> There are no other topologies running in my cluster.
> There are no other spouts/bolts running in my cluster except the ones
> mentioned above.
> I am running only one worker process per machine.
>
>
> So the question is how many threads per bolt are launched by storm?
> I am not interested in the exact number, but concerned about the high
> number of extra threads (18+ thread) for running a single bolt.
>
> Can we limit or optimize it somehow?
> Or if all of them are required, it would be good to document them
> somewhere.
>
> Thanks
> SG
>
>
>
>
Re: Too many threads per bolt instance (as seen with jconsole images)
Posted by S G <sg...@gmail.com>.
Thanks for sharing the doc Roshan.
It is very informative.
I think number of threads per bolt is on the high side (When 60 bolts are
run per node, there are 1200 total threads/node).
Some of these would be essential for the worker's book-keeping but still it
seems we can get a lot of performance boost if we can somehow reduce these
threads and the associated context switching between them.
On Thu, Mar 16, 2017 at 12:53 PM, Roshan Naik <ro...@hortonworks.com>
wrote:
> Typically there are 3 threads per spout or bolt instance (unless the
> spout or bolt is spawning its own threads). There are the acker bolts,
> event loggers and system bolt running there too.
>
> Then there are several more per worker. See a summary of it in section 3.1
> of this document
>
> https://docs.google.com/document/d/1EzeHL3d7EE-
> RyyBEpN7CwRmWz3oqjbbKiVVAlzFp2Nc/edit?usp=sharing
>
>
>
>
> Get Outlook for iOS<https://aka.ms/o0ukef>
>
>
>
>
> On Thu, Mar 16, 2017 at 12:16 PM -0700, "S G" <sg.online.email@gmail.com<
> mailto:sg.online.email@gmail.com>> wrote:
>
> Hi,
>
> I am trying to make sense of number of threads seen in JConsole.
> It seems like a very high number of threads are launched per bolt thread.
>
>
> [cid:ii_j0crtbtm0_15ad8874d0924ae2]
>
> Experiment 1
> topology.workers.val=16
> spout.parallelism.val=1
> bolt1.parallelism.val=900
> bolt2.parallelism.val=160
> observation:
> Threads seen per node in jconsole = 1200
> Bolt threads per node = (900 + 160)/16 = 66
> Threads per bolt = 1200/66 = 18
>
>
> Experiment 2
> topology.workers.val=16
> spout.parallelism.val=1
> bolt1.parallelism.val=16
> bolt2.parallelism.val=16
> observation:
> Threads seen per node in jconsole = 61
> Bolt threads per node = (16+16)/16 = 2
> Threads per bolt = 61/2 = 30
>
> There are no other topologies running in my cluster.
> There are no other spouts/bolts running in my cluster except the ones
> mentioned above.
> I am running only one worker process per machine.
>
>
> So the question is how many threads per bolt are launched by storm?
> I am not interested in the exact number, but concerned about the high
> number of extra threads (18+ thread) for running a single bolt.
>
> Can we limit or optimize it somehow?
> Or if all of them are required, it would be good to document them
> somewhere.
>
> Thanks
> SG
>
>
>
>
Re: Too many threads per bolt instance (as seen with jconsole
images)
Posted by Roshan Naik <ro...@hortonworks.com>.
Typically there are 3 threads per spout or bolt instance (unless the spout or bolt is spawning its own threads). There are the acker bolts, event loggers and system bolt running there too.
Then there are several more per worker. See a summary of it in section 3.1 of this document
https://docs.google.com/document/d/1EzeHL3d7EE-RyyBEpN7CwRmWz3oqjbbKiVVAlzFp2Nc/edit?usp=sharing
Get Outlook for iOS<https://aka.ms/o0ukef>
On Thu, Mar 16, 2017 at 12:16 PM -0700, "S G" <sg...@gmail.com>> wrote:
Hi,
I am trying to make sense of number of threads seen in JConsole.
It seems like a very high number of threads are launched per bolt thread.
[cid:ii_j0crtbtm0_15ad8874d0924ae2]
Experiment 1
topology.workers.val=16
spout.parallelism.val=1
bolt1.parallelism.val=900
bolt2.parallelism.val=160
observation:
Threads seen per node in jconsole = 1200
Bolt threads per node = (900 + 160)/16 = 66
Threads per bolt = 1200/66 = 18
Experiment 2
topology.workers.val=16
spout.parallelism.val=1
bolt1.parallelism.val=16
bolt2.parallelism.val=16
observation:
Threads seen per node in jconsole = 61
Bolt threads per node = (16+16)/16 = 2
Threads per bolt = 61/2 = 30
There are no other topologies running in my cluster.
There are no other spouts/bolts running in my cluster except the ones mentioned above.
I am running only one worker process per machine.
So the question is how many threads per bolt are launched by storm?
I am not interested in the exact number, but concerned about the high number of extra threads (18+ thread) for running a single bolt.
Can we limit or optimize it somehow?
Or if all of them are required, it would be good to document them somewhere.
Thanks
SG
Re: Too many threads per bolt instance (as seen with jconsole
images)
Posted by Roshan Naik <ro...@hortonworks.com>.
Typically there are 3 threads per spout or bolt instance (unless the spout or bolt is spawning its own threads). There are the acker bolts, event loggers and system bolt running there too.
Then there are several more per worker. See a summary of it in section 3.1 of this document
https://docs.google.com/document/d/1EzeHL3d7EE-RyyBEpN7CwRmWz3oqjbbKiVVAlzFp2Nc/edit?usp=sharing
Get Outlook for iOS<https://aka.ms/o0ukef>
On Thu, Mar 16, 2017 at 12:16 PM -0700, "S G" <sg...@gmail.com>> wrote:
Hi,
I am trying to make sense of number of threads seen in JConsole.
It seems like a very high number of threads are launched per bolt thread.
[cid:ii_j0crtbtm0_15ad8874d0924ae2]
Experiment 1
topology.workers.val=16
spout.parallelism.val=1
bolt1.parallelism.val=900
bolt2.parallelism.val=160
observation:
Threads seen per node in jconsole = 1200
Bolt threads per node = (900 + 160)/16 = 66
Threads per bolt = 1200/66 = 18
Experiment 2
topology.workers.val=16
spout.parallelism.val=1
bolt1.parallelism.val=16
bolt2.parallelism.val=16
observation:
Threads seen per node in jconsole = 61
Bolt threads per node = (16+16)/16 = 2
Threads per bolt = 61/2 = 30
There are no other topologies running in my cluster.
There are no other spouts/bolts running in my cluster except the ones mentioned above.
I am running only one worker process per machine.
So the question is how many threads per bolt are launched by storm?
I am not interested in the exact number, but concerned about the high number of extra threads (18+ thread) for running a single bolt.
Can we limit or optimize it somehow?
Or if all of them are required, it would be good to document them somewhere.
Thanks
SG