Is dynamic update of topologies possible?

[Kevin Stembridge <ke...@gmail.com>]

Hi all,

I'm just looking at Storm to see if it would be suitable for my use case.
What I need to be able to do is allow users to edit topologies on the fly.
Is it possible to do this?

From what I can see, a topology is basically baked into a jar file and
deployed.

Thanks in advance.

Kevin

Re: Is dynamic update of topologies possible?

[Kevin Stembridge <ke...@gmail.com>]

Thanks very much for all the options. I think I can probably achieve what
I'm looking for.

I appreciate the help very much.

Cheers,
Kevin

On 8 July 2016 at 06:33, Abhishek Agarwal <ab...@gmail.com> wrote:

> I agree with Nathan. It would be great if you can narrow down your
> problem. You should think on how will you edit your application on the fly,
> if it were a standalone java process. If the answer is yes, the same
> approach will likely work in storm as well.
>
> On Fri, Jul 8, 2016 at 6:03 AM, Nathan Leung <nc...@gmail.com> wrote:
>
>> It depends how your computation changes too. If it's something like
>> changing models you can load them dynamically in a bolt and that will
>> change its computation without a topology reload.
>> On Jul 7, 2016 7:24 PM, "Jungtaek Lim" <ka...@gmail.com> wrote:
>>
>>> Yeah sure that's possible. I thought "on the fly" means updating
>>> topology without downtime which is not possible yet.
>>>
>>> As you may know, flux helps us to compose topology dynamically but
>>> limited to components of compiled jar. So flux can't help if you want to
>>> update logic / algorithm of some components. In that case you need to
>>> update your code and rebuild and relaunch.
>>>
>>> And you may want to consider about the moment for both topologies
>>> running in same time. If you're tolerant to have downtime it would be easy
>>> to make but if not there're something to think about while switching.
>>>
>>> 2016년 7월 8일 (금) 오전 7:27, Kevin Stembridge <
>>> kevin.stembridge.dev@gmail.com>님이 작성:
>>>
>>>> Hi Jungtaek,
>>>> Thanks for the response.
>>>>
>>>> So would it be possible to achieve what I want by using Flux in some
>>>> way? I'm assuming the Flux config file just has to be available on the file
>>>> system when running the storm jar command. So I could generate a Flux
>>>> config file based on user input from a UI and then invoke storm cli to
>>>> start the new topology and stop the old one. Does that sound feasible?
>>>>
>>>>
>>>>
>>>>
>>>> On 7 July 2016 at 22:56, Jungtaek Lim <ka...@gmail.com> wrote:
>>>>
>>>>> Unfortunately no. You need to redeploy topology when you need to
>>>>> update content of topology.
>>>>> But, if your dynamic update is restricted to just replace resource
>>>>> files, using distributed cache on 1.0.0 makes it possible.
>>>>>
>>>>> Hope it helps.
>>>>>
>>>>> Thanks,
>>>>> Jungtaek Lim (HeartSaVioR)
>>>>>
>>>>> 2016년 7월 8일 (금) 오전 6:30, Kevin Stembridge <
>>>>> kevin.stembridge.dev@gmail.com>님이 작성:
>>>>>
>>>>>> Hi all,
>>>>>>
>>>>>> I'm just looking at Storm to see if it would be suitable for my use
>>>>>> case. What I need to be able to do is allow users to edit topologies on the
>>>>>> fly. Is it possible to do this?
>>>>>>
>>>>>> From what I can see, a topology is basically baked into a jar file
>>>>>> and deployed.
>>>>>>
>>>>>> Thanks in advance.
>>>>>>
>>>>>> Kevin
>>>>>>
>>>>>>
>>>>>>
>>>>
>
>
> --
> Regards,
> Abhishek Agarwal
>
>

Re: Is dynamic update of topologies possible?

[Abhishek Agarwal <ab...@gmail.com>]

I agree with Nathan. It would be great if you can narrow down your problem.
You should think on how will you edit your application on the fly, if it
were a standalone java process. If the answer is yes, the same approach
will likely work in storm as well.

On Fri, Jul 8, 2016 at 6:03 AM, Nathan Leung <nc...@gmail.com> wrote:

> It depends how your computation changes too. If it's something like
> changing models you can load them dynamically in a bolt and that will
> change its computation without a topology reload.
> On Jul 7, 2016 7:24 PM, "Jungtaek Lim" <ka...@gmail.com> wrote:
>
>> Yeah sure that's possible. I thought "on the fly" means updating topology
>> without downtime which is not possible yet.
>>
>> As you may know, flux helps us to compose topology dynamically but
>> limited to components of compiled jar. So flux can't help if you want to
>> update logic / algorithm of some components. In that case you need to
>> update your code and rebuild and relaunch.
>>
>> And you may want to consider about the moment for both topologies running
>> in same time. If you're tolerant to have downtime it would be easy to make
>> but if not there're something to think about while switching.
>>
>> 2016년 7월 8일 (금) 오전 7:27, Kevin Stembridge <ke...@gmail.com>님이
>> 작성:
>>
>>> Hi Jungtaek,
>>> Thanks for the response.
>>>
>>> So would it be possible to achieve what I want by using Flux in some
>>> way? I'm assuming the Flux config file just has to be available on the file
>>> system when running the storm jar command. So I could generate a Flux
>>> config file based on user input from a UI and then invoke storm cli to
>>> start the new topology and stop the old one. Does that sound feasible?
>>>
>>>
>>>
>>>
>>> On 7 July 2016 at 22:56, Jungtaek Lim <ka...@gmail.com> wrote:
>>>
>>>> Unfortunately no. You need to redeploy topology when you need to update
>>>> content of topology.
>>>> But, if your dynamic update is restricted to just replace resource
>>>> files, using distributed cache on 1.0.0 makes it possible.
>>>>
>>>> Hope it helps.
>>>>
>>>> Thanks,
>>>> Jungtaek Lim (HeartSaVioR)
>>>>
>>>> 2016년 7월 8일 (금) 오전 6:30, Kevin Stembridge <
>>>> kevin.stembridge.dev@gmail.com>님이 작성:
>>>>
>>>>> Hi all,
>>>>>
>>>>> I'm just looking at Storm to see if it would be suitable for my use
>>>>> case. What I need to be able to do is allow users to edit topologies on the
>>>>> fly. Is it possible to do this?
>>>>>
>>>>> From what I can see, a topology is basically baked into a jar file and
>>>>> deployed.
>>>>>
>>>>> Thanks in advance.
>>>>>
>>>>> Kevin
>>>>>
>>>>>
>>>>>
>>>


-- 
Regards,
Abhishek Agarwal

Re: Is dynamic update of topologies possible?

[Nathan Leung <nc...@gmail.com>]

It depends how your computation changes too. If it's something like
changing models you can load them dynamically in a bolt and that will
change its computation without a topology reload.
On Jul 7, 2016 7:24 PM, "Jungtaek Lim" <ka...@gmail.com> wrote:

> Yeah sure that's possible. I thought "on the fly" means updating topology
> without downtime which is not possible yet.
>
> As you may know, flux helps us to compose topology dynamically but limited
> to components of compiled jar. So flux can't help if you want to update
> logic / algorithm of some components. In that case you need to update your
> code and rebuild and relaunch.
>
> And you may want to consider about the moment for both topologies running
> in same time. If you're tolerant to have downtime it would be easy to make
> but if not there're something to think about while switching.
>
> 2016년 7월 8일 (금) 오전 7:27, Kevin Stembridge <ke...@gmail.com>님이
> 작성:
>
>> Hi Jungtaek,
>> Thanks for the response.
>>
>> So would it be possible to achieve what I want by using Flux in some way?
>> I'm assuming the Flux config file just has to be available on the file
>> system when running the storm jar command. So I could generate a Flux
>> config file based on user input from a UI and then invoke storm cli to
>> start the new topology and stop the old one. Does that sound feasible?
>>
>>
>>
>>
>> On 7 July 2016 at 22:56, Jungtaek Lim <ka...@gmail.com> wrote:
>>
>>> Unfortunately no. You need to redeploy topology when you need to update
>>> content of topology.
>>> But, if your dynamic update is restricted to just replace resource
>>> files, using distributed cache on 1.0.0 makes it possible.
>>>
>>> Hope it helps.
>>>
>>> Thanks,
>>> Jungtaek Lim (HeartSaVioR)
>>>
>>> 2016년 7월 8일 (금) 오전 6:30, Kevin Stembridge <
>>> kevin.stembridge.dev@gmail.com>님이 작성:
>>>
>>>> Hi all,
>>>>
>>>> I'm just looking at Storm to see if it would be suitable for my use
>>>> case. What I need to be able to do is allow users to edit topologies on the
>>>> fly. Is it possible to do this?
>>>>
>>>> From what I can see, a topology is basically baked into a jar file and
>>>> deployed.
>>>>
>>>> Thanks in advance.
>>>>
>>>> Kevin
>>>>
>>>>
>>>>
>>

Re: Is dynamic update of topologies possible?

[Jungtaek Lim <ka...@gmail.com>]

Yeah sure that's possible. I thought "on the fly" means updating topology
without downtime which is not possible yet.

As you may know, flux helps us to compose topology dynamically but limited
to components of compiled jar. So flux can't help if you want to update
logic / algorithm of some components. In that case you need to update your
code and rebuild and relaunch.

And you may want to consider about the moment for both topologies running
in same time. If you're tolerant to have downtime it would be easy to make
but if not there're something to think about while switching.

2016년 7월 8일 (금) 오전 7:27, Kevin Stembridge <ke...@gmail.com>님이
작성:

> Hi Jungtaek,
> Thanks for the response.
>
> So would it be possible to achieve what I want by using Flux in some way?
> I'm assuming the Flux config file just has to be available on the file
> system when running the storm jar command. So I could generate a Flux
> config file based on user input from a UI and then invoke storm cli to
> start the new topology and stop the old one. Does that sound feasible?
>
>
>
>
> On 7 July 2016 at 22:56, Jungtaek Lim <ka...@gmail.com> wrote:
>
>> Unfortunately no. You need to redeploy topology when you need to update
>> content of topology.
>> But, if your dynamic update is restricted to just replace resource files,
>> using distributed cache on 1.0.0 makes it possible.
>>
>> Hope it helps.
>>
>> Thanks,
>> Jungtaek Lim (HeartSaVioR)
>>
>> 2016년 7월 8일 (금) 오전 6:30, Kevin Stembridge <ke...@gmail.com>님이
>> 작성:
>>
>>> Hi all,
>>>
>>> I'm just looking at Storm to see if it would be suitable for my use
>>> case. What I need to be able to do is allow users to edit topologies on the
>>> fly. Is it possible to do this?
>>>
>>> From what I can see, a topology is basically baked into a jar file and
>>> deployed.
>>>
>>> Thanks in advance.
>>>
>>> Kevin
>>>
>>>
>>>
>

Re: Is dynamic update of topologies possible?

[Kevin Stembridge <ke...@gmail.com>]

Hi Jungtaek,
Thanks for the response.

So would it be possible to achieve what I want by using Flux in some way?
I'm assuming the Flux config file just has to be available on the file
system when running the storm jar command. So I could generate a Flux
config file based on user input from a UI and then invoke storm cli to
start the new topology and stop the old one. Does that sound feasible?




On 7 July 2016 at 22:56, Jungtaek Lim <ka...@gmail.com> wrote:

> Unfortunately no. You need to redeploy topology when you need to update
> content of topology.
> But, if your dynamic update is restricted to just replace resource files,
> using distributed cache on 1.0.0 makes it possible.
>
> Hope it helps.
>
> Thanks,
> Jungtaek Lim (HeartSaVioR)
>
> 2016년 7월 8일 (금) 오전 6:30, Kevin Stembridge <ke...@gmail.com>님이
> 작성:
>
>> Hi all,
>>
>> I'm just looking at Storm to see if it would be suitable for my use case.
>> What I need to be able to do is allow users to edit topologies on the fly.
>> Is it possible to do this?
>>
>> From what I can see, a topology is basically baked into a jar file and
>> deployed.
>>
>> Thanks in advance.
>>
>> Kevin
>>
>>
>>

Re: Is dynamic update of topologies possible?

[Jungtaek Lim <ka...@gmail.com>]

Unfortunately no. You need to redeploy topology when you need to update
content of topology.
But, if your dynamic update is restricted to just replace resource files,
using distributed cache on 1.0.0 makes it possible.

Hope it helps.

Thanks,
Jungtaek Lim (HeartSaVioR)

2016년 7월 8일 (금) 오전 6:30, Kevin Stembridge <ke...@gmail.com>님이
작성:

> Hi all,
>
> I'm just looking at Storm to see if it would be suitable for my use case.
> What I need to be able to do is allow users to edit topologies on the fly.
> Is it possible to do this?
>
> From what I can see, a topology is basically baked into a jar file and
> deployed.
>
> Thanks in advance.
>
> Kevin
>
>
>