ScheduledTopology

[Muhammad Ali A'råby <an...@yahoo.com.INVALID>]

Dear all,
I am implementing a set of classes for scheduled topology works. For example when you want to disable/enable topologies periodically. This codes are having mechanisms for deciding whether a topology is finished working (are all tuples fully processed), mechanism for bolt to have an alternate prepare for every reboot of topology, etc.
I am wondering where should I commit my code? Each class separately in storm-core package (e.g. ScheduledBolt in storm/storm-core/src/jvm/backtype/storm/bolt, ScheduledTopology in storm/storm-core/src/jvm/backtype/storm/topology, etc.) or all of them in a package somewhere?
Sincerely,Muhammad-Ali A'rabi

Re: ScheduledTopology

[Itai Frenkel <It...@forter.com>]

Muhammad,

We are thinking of using storm for scheduled non-realtime processing (things like you would do with map-reduce once a week or daily). The benefit of using Storm is that the same code could be used in realtime and non-realtime processing (it has some disadvantages ofcourse). 

For that use case we thought there would be a cron job that would deploy the new topology (and perhaps also start virtual machine instances). The spout would probably be in charge of sending an event to remove the topology (and perhaps the instances) when all tuples have been processed.

What are the benefits of using the classes you described for this use case?

Regards,
Itai

________________________________________
From: Harsha <st...@harsha.io>
Sent: Tuesday, January 13, 2015 6:25 PM
To: dev@storm.apache.org
Subject: Re: ScheduledTopology

Muhammad,
         Can you give us bit more details on how this works. Topologies
         supposed to be long running. I do see use for this, just
         curious that users can automate a topology activate or
         deactivate using UI api.
Thanks,
Harsha

On Tue, Jan 13, 2015, at 08:13 AM, Muhammad Ali A'råby wrote:
> Dear all,
> I am implementing a set of classes for scheduled topology works. For
> example when you want to disable/enable topologies periodically. This
> codes are having mechanisms for deciding whether a topology is finished
> working (are all tuples fully processed), mechanism for bolt to have an
> alternate prepare for every reboot of topology, etc.
> I am wondering where should I commit my code? Each class separately in
> storm-core package (e.g. ScheduledBolt in
> storm/storm-core/src/jvm/backtype/storm/bolt, ScheduledTopology in
> storm/storm-core/src/jvm/backtype/storm/topology, etc.) or all of them in
> a package somewhere?
> Sincerely,Muhammad-Ali A'rabi

Re: ScheduledTopology

[Harsha <st...@harsha.io>]

Muhammad,
         Can you give us bit more details on how this works. Topologies
         supposed to be long running. I do see use for this, just
         curious that users can automate a topology activate or
         deactivate using UI api.
Thanks,
Harsha

On Tue, Jan 13, 2015, at 08:13 AM, Muhammad Ali A'råby wrote:
> Dear all,
> I am implementing a set of classes for scheduled topology works. For
> example when you want to disable/enable topologies periodically. This
> codes are having mechanisms for deciding whether a topology is finished
> working (are all tuples fully processed), mechanism for bolt to have an
> alternate prepare for every reboot of topology, etc.
> I am wondering where should I commit my code? Each class separately in
> storm-core package (e.g. ScheduledBolt in
> storm/storm-core/src/jvm/backtype/storm/bolt, ScheduledTopology in
> storm/storm-core/src/jvm/backtype/storm/topology, etc.) or all of them in
> a package somewhere?
> Sincerely,Muhammad-Ali A'rabi