You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@nifi.apache.org by Jeff - Data Bean Australia <da...@gmail.com> on 2016/02/16 03:10:06 UTC
Does NiFi support Hot Deploy?
Here is my use case: I have a quite complex Data Flow system implemented by
NiFi and keep streaming data to another system. There is part of the system
I might want to update on-the-fly without turning down the whole NiFi
platform. And occasionally I might want to add a new processor, or upgrade
an existing one, then bring up this processor right away.
Can I do it in NiFi now?
Thanks,
Jeff
--
Data Bean - A Big Data Solution Provider in Australia.
Re: Does NiFi support Hot Deploy?
Posted by Jeff - Data Bean Australia <da...@gmail.com>.
Thanks Joe. I thought NiFi Clusters, or Clustered NiFis are used for HA,
Load Balancing, and Scalability, but it seems like it can use for
modularization.
On Tue, Feb 16, 2016 at 3:45 PM, Joe Witt <jo...@gmail.com> wrote:
> Jeff
>
> Clustered NiFis (multiple nodes) is quite common. You can have
> hundreds of processors representing a quite large number of distinct
> flows on there. Templates are the mechanism by which flows can be
> shared among clusters and between dev/prod environments. We have some
> important issues to tackle to make templates as powerful as they
> should be such as having environment variable mapping and
> consistent/repeatable serialization which lends to better version
> control of templates.
>
> Thanks
> Joe
>
> On Mon, Feb 15, 2016 at 9:39 PM, Jeff - Data Bean Australia
> <da...@gmail.com> wrote:
> > Thanks Joe for the clarification. And it does make sense to me with
> runtime
> > reliability and change requirement in enterprise environment.
> >
> > I know that there are solutions that put multiple NiFi instances working
> > together. How common a solution like this in real world? Should we,
> > generally speaking, prefer simple flow.xml configuration and combine NiFi
> > instances together for more complex scenarios?
> >
> > On Tue, Feb 16, 2016 at 3:16 PM, Joe Witt <jo...@gmail.com> wrote:
> >>
> >> Jeff,
> >>
> >> Regarding hot deploy of new dataflows:
> >> Absolutely. The ability to have interactive command and control is a
> >> key feature/concept of NiFi so you can certainly create, alter, remove
> >> dataflows while the systems is running by design. This isn't just for
> >> a person controlling the flow through UI but also for external
> >> systems to have automated interactions with NiFi through its REST API
> >> which can do things like change the flow, alter priorities, etc..
> >>
> >> Regarding hot deploy of new code/extensions:
> >> We've avoided live deploy of new extensions to this point. Largely
> >> due to the understanding that while adding new extensions at runtime
> >> is pretty doable it is less reliable/clear to change/update. That
> >> said, we're trending toward this registry and it would back versions
> >> of extensions at which point maybe this becomes more reasonable.
> >>
> >> Anyway, in the mean time one option may be the fact that the
> >> ExecuteScript,InvokeScriptedProcessor processors do support live
> >> alteration of the code behind them. While clearly not a complete
> >> solution this may help with some of your cases.
> >>
> >> Thanks
> >> Joe
> >>
> >>
> >>
> >> On Mon, Feb 15, 2016 at 9:10 PM, Jeff - Data Bean Australia
> >> <da...@gmail.com> wrote:
> >> > Here is my use case: I have a quite complex Data Flow system
> implemented
> >> > by
> >> > NiFi and keep streaming data to another system. There is part of the
> >> > system
> >> > I might want to update on-the-fly without turning down the whole NiFi
> >> > platform. And occasionally I might want to add a new processor, or
> >> > upgrade
> >> > an existing one, then bring up this processor right away.
> >> >
> >> > Can I do it in NiFi now?
> >> >
> >> > Thanks,
> >> > Jeff
> >> >
> >> > --
> >> > Data Bean - A Big Data Solution Provider in Australia.
> >
> >
> >
> >
> > --
> > Data Bean - A Big Data Solution Provider in Australia.
>
--
Data Bean - A Big Data Solution Provider in Australia.
Re: Does NiFi support Hot Deploy?
Posted by Joe Witt <jo...@gmail.com>.
Jeff
Clustered NiFis (multiple nodes) is quite common. You can have
hundreds of processors representing a quite large number of distinct
flows on there. Templates are the mechanism by which flows can be
shared among clusters and between dev/prod environments. We have some
important issues to tackle to make templates as powerful as they
should be such as having environment variable mapping and
consistent/repeatable serialization which lends to better version
control of templates.
Thanks
Joe
On Mon, Feb 15, 2016 at 9:39 PM, Jeff - Data Bean Australia
<da...@gmail.com> wrote:
> Thanks Joe for the clarification. And it does make sense to me with runtime
> reliability and change requirement in enterprise environment.
>
> I know that there are solutions that put multiple NiFi instances working
> together. How common a solution like this in real world? Should we,
> generally speaking, prefer simple flow.xml configuration and combine NiFi
> instances together for more complex scenarios?
>
> On Tue, Feb 16, 2016 at 3:16 PM, Joe Witt <jo...@gmail.com> wrote:
>>
>> Jeff,
>>
>> Regarding hot deploy of new dataflows:
>> Absolutely. The ability to have interactive command and control is a
>> key feature/concept of NiFi so you can certainly create, alter, remove
>> dataflows while the systems is running by design. This isn't just for
>> a person controlling the flow through UI but also for external
>> systems to have automated interactions with NiFi through its REST API
>> which can do things like change the flow, alter priorities, etc..
>>
>> Regarding hot deploy of new code/extensions:
>> We've avoided live deploy of new extensions to this point. Largely
>> due to the understanding that while adding new extensions at runtime
>> is pretty doable it is less reliable/clear to change/update. That
>> said, we're trending toward this registry and it would back versions
>> of extensions at which point maybe this becomes more reasonable.
>>
>> Anyway, in the mean time one option may be the fact that the
>> ExecuteScript,InvokeScriptedProcessor processors do support live
>> alteration of the code behind them. While clearly not a complete
>> solution this may help with some of your cases.
>>
>> Thanks
>> Joe
>>
>>
>>
>> On Mon, Feb 15, 2016 at 9:10 PM, Jeff - Data Bean Australia
>> <da...@gmail.com> wrote:
>> > Here is my use case: I have a quite complex Data Flow system implemented
>> > by
>> > NiFi and keep streaming data to another system. There is part of the
>> > system
>> > I might want to update on-the-fly without turning down the whole NiFi
>> > platform. And occasionally I might want to add a new processor, or
>> > upgrade
>> > an existing one, then bring up this processor right away.
>> >
>> > Can I do it in NiFi now?
>> >
>> > Thanks,
>> > Jeff
>> >
>> > --
>> > Data Bean - A Big Data Solution Provider in Australia.
>
>
>
>
> --
> Data Bean - A Big Data Solution Provider in Australia.
Re: Does NiFi support Hot Deploy?
Posted by Jeff - Data Bean Australia <da...@gmail.com>.
Thanks Joe for the clarification. And it does make sense to me with runtime
reliability and change requirement in enterprise environment.
I know that there are solutions that put multiple NiFi instances working
together. How common a solution like this in real world? Should we,
generally speaking, prefer simple flow.xml configuration and combine NiFi
instances together for more complex scenarios?
On Tue, Feb 16, 2016 at 3:16 PM, Joe Witt <jo...@gmail.com> wrote:
> Jeff,
>
> Regarding hot deploy of new dataflows:
> Absolutely. The ability to have interactive command and control is a
> key feature/concept of NiFi so you can certainly create, alter, remove
> dataflows while the systems is running by design. This isn't just for
> a person controlling the flow through UI but also for external
> systems to have automated interactions with NiFi through its REST API
> which can do things like change the flow, alter priorities, etc..
>
> Regarding hot deploy of new code/extensions:
> We've avoided live deploy of new extensions to this point. Largely
> due to the understanding that while adding new extensions at runtime
> is pretty doable it is less reliable/clear to change/update. That
> said, we're trending toward this registry and it would back versions
> of extensions at which point maybe this becomes more reasonable.
>
> Anyway, in the mean time one option may be the fact that the
> ExecuteScript,InvokeScriptedProcessor processors do support live
> alteration of the code behind them. While clearly not a complete
> solution this may help with some of your cases.
>
> Thanks
> Joe
>
>
>
> On Mon, Feb 15, 2016 at 9:10 PM, Jeff - Data Bean Australia
> <da...@gmail.com> wrote:
> > Here is my use case: I have a quite complex Data Flow system implemented
> by
> > NiFi and keep streaming data to another system. There is part of the
> system
> > I might want to update on-the-fly without turning down the whole NiFi
> > platform. And occasionally I might want to add a new processor, or
> upgrade
> > an existing one, then bring up this processor right away.
> >
> > Can I do it in NiFi now?
> >
> > Thanks,
> > Jeff
> >
> > --
> > Data Bean - A Big Data Solution Provider in Australia.
>
--
Data Bean - A Big Data Solution Provider in Australia.
Re: Does NiFi support Hot Deploy?
Posted by Joe Witt <jo...@gmail.com>.
Jeff,
Regarding hot deploy of new dataflows:
Absolutely. The ability to have interactive command and control is a
key feature/concept of NiFi so you can certainly create, alter, remove
dataflows while the systems is running by design. This isn't just for
a person controlling the flow through UI but also for external
systems to have automated interactions with NiFi through its REST API
which can do things like change the flow, alter priorities, etc..
Regarding hot deploy of new code/extensions:
We've avoided live deploy of new extensions to this point. Largely
due to the understanding that while adding new extensions at runtime
is pretty doable it is less reliable/clear to change/update. That
said, we're trending toward this registry and it would back versions
of extensions at which point maybe this becomes more reasonable.
Anyway, in the mean time one option may be the fact that the
ExecuteScript,InvokeScriptedProcessor processors do support live
alteration of the code behind them. While clearly not a complete
solution this may help with some of your cases.
Thanks
Joe
On Mon, Feb 15, 2016 at 9:10 PM, Jeff - Data Bean Australia
<da...@gmail.com> wrote:
> Here is my use case: I have a quite complex Data Flow system implemented by
> NiFi and keep streaming data to another system. There is part of the system
> I might want to update on-the-fly without turning down the whole NiFi
> platform. And occasionally I might want to add a new processor, or upgrade
> an existing one, then bring up this processor right away.
>
> Can I do it in NiFi now?
>
> Thanks,
> Jeff
>
> --
> Data Bean - A Big Data Solution Provider in Australia.