How dynamic are spouts/bolts in a topology?

[Otis Gospodnetic <ot...@gmail.com>]

Hi,

I'm wondering how dynamic topologies are for most Storm users?
That is, do people tend to have topologies with a more or less fixed type
(and number?) of spouts & bolts?  Or do people have topologies where
different types of spouts & bolts are regularly being added/removed?

I'm asking because some of our SPM users who use it to monitor Storm
performance metrics are asking if they can see metrics for their spouts and
bolts.  We didn't collect these metrics in the SPM based on the assumption
that topologies are very dynamic, meaning that spouts and bolts change a
lot and one would end up with a messy pile of different kinds of bolts &
spouts in the SPM UI, but.....

.... now I'm wondering, is this actually a correct assumption?  I now get
the feeling most people have 1 or more topologies with a more or less fixed
types (and even number?) of spouts & bolts.

Could anyone please enlighten me?

Thanks,
Otis
--
Performance Monitoring * Log Analytics * Search Analytics
Solr & Elasticsearch Support * http://sematext.com/

Re: How dynamic are spouts/bolts in a topology?

[Jason Jackson <ja...@gmail.com>]

storm only allows static topologies. however you could always create a
fully connected graph, and then implement CustomStreamGrouping with your
own dynamic logic.


On Sat, Apr 12, 2014 at 2:45 AM, M.Tarkeshwar Rao <ta...@gmail.com>wrote:

> Hi All,
>
> Any one have idea about how reportError is working.
>
> Can you please share some links for storm internals and how ackers are
> working.
>
> Regards
>
>
> On Sat, Apr 12, 2014 at 6:44 AM, Srinath C <sr...@gmail.com> wrote:
>
>> Hi Otis,
>>    The type of spouts and bolts and the inter-connectivity between them
>> is fixed. The number of (tasks) spouts and bolts is configurable but is
>> also fixed for the lifetime of the topology. The topology should be
>> re-deployed if it changes. However, the number of threads that work with
>> these tasks are updateable by deactivating and re-activating the topology.
>>
>>    But, I did feel the need to dynamically add/remove bolts. In our
>> environment, users configure policies which do specific aggregations over
>> the data streams. Since these policies would be created/deleted by user, I
>> thought it would be great to be able to dynamically alter the topology.
>> Since that is not possible using Storm, we had to circumvent it by
>> re-designing our streams.
>>
>> Regards,
>> Srinath.
>>
>>
>>
>>
>> On Fri, Apr 11, 2014 at 6:41 PM, Otis Gospodnetic <
>> otis.gospodnetic@gmail.com> wrote:
>>
>>> Hi,
>>>
>>> I'm wondering how dynamic topologies are for most Storm users?
>>> That is, do people tend to have topologies with a more or less fixed
>>> type (and number?) of spouts & bolts?  Or do people have topologies where
>>> different types of spouts & bolts are regularly being added/removed?
>>>
>>> I'm asking because some of our SPM users who use it to monitor Storm
>>> performance metrics are asking if they can see metrics for their spouts and
>>> bolts.  We didn't collect these metrics in the SPM based on the assumption
>>> that topologies are very dynamic, meaning that spouts and bolts change a
>>> lot and one would end up with a messy pile of different kinds of bolts &
>>> spouts in the SPM UI, but.....
>>>
>>> .... now I'm wondering, is this actually a correct assumption?  I now
>>> get the feeling most people have 1 or more topologies with a more or less
>>> fixed types (and even number?) of spouts & bolts.
>>>
>>> Could anyone please enlighten me?
>>>
>>> Thanks,
>>> Otis
>>> --
>>> Performance Monitoring * Log Analytics * Search Analytics
>>> Solr & Elasticsearch Support * http://sematext.com/
>>>
>>>
>>
>

Re: How dynamic are spouts/bolts in a topology?

["M.Tarkeshwar Rao" <ta...@gmail.com>]

Hi All,

Any one have idea about how reportError is working.

Can you please share some links for storm internals and how ackers are
working.

Regards


On Sat, Apr 12, 2014 at 6:44 AM, Srinath C <sr...@gmail.com> wrote:

> Hi Otis,
>    The type of spouts and bolts and the inter-connectivity between them is
> fixed. The number of (tasks) spouts and bolts is configurable but is also
> fixed for the lifetime of the topology. The topology should be re-deployed
> if it changes. However, the number of threads that work with these tasks
> are updateable by deactivating and re-activating the topology.
>
>    But, I did feel the need to dynamically add/remove bolts. In our
> environment, users configure policies which do specific aggregations over
> the data streams. Since these policies would be created/deleted by user, I
> thought it would be great to be able to dynamically alter the topology.
> Since that is not possible using Storm, we had to circumvent it by
> re-designing our streams.
>
> Regards,
> Srinath.
>
>
>
>
> On Fri, Apr 11, 2014 at 6:41 PM, Otis Gospodnetic <
> otis.gospodnetic@gmail.com> wrote:
>
>> Hi,
>>
>> I'm wondering how dynamic topologies are for most Storm users?
>> That is, do people tend to have topologies with a more or less fixed type
>> (and number?) of spouts & bolts?  Or do people have topologies where
>> different types of spouts & bolts are regularly being added/removed?
>>
>> I'm asking because some of our SPM users who use it to monitor Storm
>> performance metrics are asking if they can see metrics for their spouts and
>> bolts.  We didn't collect these metrics in the SPM based on the assumption
>> that topologies are very dynamic, meaning that spouts and bolts change a
>> lot and one would end up with a messy pile of different kinds of bolts &
>> spouts in the SPM UI, but.....
>>
>> .... now I'm wondering, is this actually a correct assumption?  I now get
>> the feeling most people have 1 or more topologies with a more or less fixed
>> types (and even number?) of spouts & bolts.
>>
>> Could anyone please enlighten me?
>>
>> Thanks,
>> Otis
>> --
>> Performance Monitoring * Log Analytics * Search Analytics
>> Solr & Elasticsearch Support * http://sematext.com/
>>
>>
>

Re: How dynamic are spouts/bolts in a topology?

[Srinath C <sr...@gmail.com>]

Hi Otis,
   The type of spouts and bolts and the inter-connectivity between them is
fixed. The number of (tasks) spouts and bolts is configurable but is also
fixed for the lifetime of the topology. The topology should be re-deployed
if it changes. However, the number of threads that work with these tasks
are updateable by deactivating and re-activating the topology.

   But, I did feel the need to dynamically add/remove bolts. In our
environment, users configure policies which do specific aggregations over
the data streams. Since these policies would be created/deleted by user, I
thought it would be great to be able to dynamically alter the topology.
Since that is not possible using Storm, we had to circumvent it by
re-designing our streams.

Regards,
Srinath.




On Fri, Apr 11, 2014 at 6:41 PM, Otis Gospodnetic <
otis.gospodnetic@gmail.com> wrote:

> Hi,
>
> I'm wondering how dynamic topologies are for most Storm users?
> That is, do people tend to have topologies with a more or less fixed type
> (and number?) of spouts & bolts?  Or do people have topologies where
> different types of spouts & bolts are regularly being added/removed?
>
> I'm asking because some of our SPM users who use it to monitor Storm
> performance metrics are asking if they can see metrics for their spouts and
> bolts.  We didn't collect these metrics in the SPM based on the assumption
> that topologies are very dynamic, meaning that spouts and bolts change a
> lot and one would end up with a messy pile of different kinds of bolts &
> spouts in the SPM UI, but.....
>
> .... now I'm wondering, is this actually a correct assumption?  I now get
> the feeling most people have 1 or more topologies with a more or less fixed
> types (and even number?) of spouts & bolts.
>
> Could anyone please enlighten me?
>
> Thanks,
> Otis
> --
> Performance Monitoring * Log Analytics * Search Analytics
> Solr & Elasticsearch Support * http://sematext.com/
>
>