You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@nifi.apache.org by Phillip Lord <ph...@onyxpoint.com> on 2021/07/01 21:16:09 UTC
Apache NiFi Registry
Hello,
My organization is considering utilizing the Registry. From my testing it
appears that versioning doesn't keep track of the state of components
(stopped/started/etc). Is this accurate? Are there plans to have
versioning keep track of this in future releases?
I'm using NiFi 1.11.4 and Registry version 0.8.0
Thanks,
Phil Lord
Re: Apache NiFi Registry
Posted by Mark Bean <ma...@gmail.com>.
And, for clarification, when a PG is imported from NiFi Registry, the
initial state of non-disabled components is 'STOPPED'. It takes user
interaction to put the components into a running state.
Agree with Andrew to disable (in the versioned flow in the Registry) any
component which you do not want to start when starting the entire versioned
flow, i.e. the Process Group level.
-Mark
On Thu, Jul 1, 2021 at 5:35 PM Andrew Grande <ap...@gmail.com> wrote:
> Isn't the proper state for this use case enabled/disabled? NiFi will start
> a PG and every schedulable component in it. If one needs to prevent this,
> disable a processor.
>
> Andrew
>
> On Thu, Jul 1, 2021, 2:17 PM Phillip Lord <ph...@onyxpoint.com>
> wrote:
>
> > Hello,
> >
> > My organization is considering utilizing the Registry. From my testing
> it
> > appears that versioning doesn't keep track of the state of components
> > (stopped/started/etc). Is this accurate? Are there plans to have
> > versioning keep track of this in future releases?
> >
> > I'm using NiFi 1.11.4 and Registry version 0.8.0
> >
> > Thanks,
> > Phil Lord
> >
>
Re: Apache NiFi Registry
Posted by Phillip Lord <ph...@onyxpoint.com>.
Thanks Bryan,
We can definitely automate most things by following up with an API call... although making the option available would be beneficial.
One issue I've noticed though is that it seems to track/reflect a change when a component is disabled... however if an input/output port is disabled that is not tracked/reflected as a change to the existing version. We never leave anything in a "stopped" state... so that's not an issue. But there are certainly scenarios where things may be in a disabled state. Automating those kinds of scenarios I don't think is really realistic...
On 2021/07/06 18:53:50, Bryan Bende <bb...@gmail.com> wrote:
> It was done as a precaution that a user may not realize that after
> import or change-version, that the entire flow is going to
> automatically start running. For cases where there is an existing flow
> and the next version maybe adds one processor in the middle, it
> probably wouldn't be a big deal, but that can't easily be
> differentiated from a whole new section of the flow that starts
> pulling data from an external system in production. We could probably
> add an option to make it easier, but I think most people are
> automating this in some way and it ends up being two API calls to
> enable valid services and start valid processors.
>
> On Tue, Jul 6, 2021 at 2:45 PM Phillip Lord <ph...@onyxpoint.com> wrote:
> >
> > Thanks Andrew,
> >
> > Yes we generally do not leave any components in a "stopped" state. They're always "running/enabled/disabled".
> > What we're hoping to do is utilize a "staging" environment for all changes as opposed to having to have any direct manipulation to the production clusters. Where the staging environment would then check-in changes to the registry... The production environment then recognized the pg isn't utilizing the newest available version and we can then tell the production environment to check in the latest version. As it stands today if the change that I checked in is aa new component in a flow, the new component that the production environment checks-in is in a stopped state. This then requires additional manual intervention to "enable/start" the flow.
> >
> > I can see it potentially being a precautionary type thing... but would definitely be useful to have the option to have components retain status/state when checked into the registry.
> >
> > I imagined it would be somewhat similar to how MiNiFi works where components are assumed to be running at all times... at least I think that's how it's working in the background?
> >
> >
> >
> > On 2021/07/01 21:34:52, Andrew Grande <ap...@gmail.com> wrote:
> > > Isn't the proper state for this use case enabled/disabled? NiFi will start
> > > a PG and every schedulable component in it. If one needs to prevent this,
> > > disable a processor.
> > >
> > > Andrew
> > >
> > > On Thu, Jul 1, 2021, 2:17 PM Phillip Lord <ph...@onyxpoint.com>
> > > wrote:
> > >
> > > > Hello,
> > > >
> > > > My organization is considering utilizing the Registry. From my testing it
> > > > appears that versioning doesn't keep track of the state of components
> > > > (stopped/started/etc). Is this accurate? Are there plans to have
> > > > versioning keep track of this in future releases?
> > > >
> > > > I'm using NiFi 1.11.4 and Registry version 0.8.0
> > > >
> > > > Thanks,
> > > > Phil Lord
> > > >
> > >
>
Re: Apache NiFi Registry
Posted by Bryan Bende <bb...@gmail.com>.
It was done as a precaution that a user may not realize that after
import or change-version, that the entire flow is going to
automatically start running. For cases where there is an existing flow
and the next version maybe adds one processor in the middle, it
probably wouldn't be a big deal, but that can't easily be
differentiated from a whole new section of the flow that starts
pulling data from an external system in production. We could probably
add an option to make it easier, but I think most people are
automating this in some way and it ends up being two API calls to
enable valid services and start valid processors.
On Tue, Jul 6, 2021 at 2:45 PM Phillip Lord <ph...@onyxpoint.com> wrote:
>
> Thanks Andrew,
>
> Yes we generally do not leave any components in a "stopped" state. They're always "running/enabled/disabled".
> What we're hoping to do is utilize a "staging" environment for all changes as opposed to having to have any direct manipulation to the production clusters. Where the staging environment would then check-in changes to the registry... The production environment then recognized the pg isn't utilizing the newest available version and we can then tell the production environment to check in the latest version. As it stands today if the change that I checked in is aa new component in a flow, the new component that the production environment checks-in is in a stopped state. This then requires additional manual intervention to "enable/start" the flow.
>
> I can see it potentially being a precautionary type thing... but would definitely be useful to have the option to have components retain status/state when checked into the registry.
>
> I imagined it would be somewhat similar to how MiNiFi works where components are assumed to be running at all times... at least I think that's how it's working in the background?
>
>
>
> On 2021/07/01 21:34:52, Andrew Grande <ap...@gmail.com> wrote:
> > Isn't the proper state for this use case enabled/disabled? NiFi will start
> > a PG and every schedulable component in it. If one needs to prevent this,
> > disable a processor.
> >
> > Andrew
> >
> > On Thu, Jul 1, 2021, 2:17 PM Phillip Lord <ph...@onyxpoint.com>
> > wrote:
> >
> > > Hello,
> > >
> > > My organization is considering utilizing the Registry. From my testing it
> > > appears that versioning doesn't keep track of the state of components
> > > (stopped/started/etc). Is this accurate? Are there plans to have
> > > versioning keep track of this in future releases?
> > >
> > > I'm using NiFi 1.11.4 and Registry version 0.8.0
> > >
> > > Thanks,
> > > Phil Lord
> > >
> >
Re: Apache NiFi Registry
Posted by Phillip Lord <ph...@onyxpoint.com>.
Thanks Andrew,
Yes we generally do not leave any components in a "stopped" state. They're always "running/enabled/disabled".
What we're hoping to do is utilize a "staging" environment for all changes as opposed to having to have any direct manipulation to the production clusters. Where the staging environment would then check-in changes to the registry... The production environment then recognized the pg isn't utilizing the newest available version and we can then tell the production environment to check in the latest version. As it stands today if the change that I checked in is aa new component in a flow, the new component that the production environment checks-in is in a stopped state. This then requires additional manual intervention to "enable/start" the flow.
I can see it potentially being a precautionary type thing... but would definitely be useful to have the option to have components retain status/state when checked into the registry.
I imagined it would be somewhat similar to how MiNiFi works where components are assumed to be running at all times... at least I think that's how it's working in the background?
On 2021/07/01 21:34:52, Andrew Grande <ap...@gmail.com> wrote:
> Isn't the proper state for this use case enabled/disabled? NiFi will start
> a PG and every schedulable component in it. If one needs to prevent this,
> disable a processor.
>
> Andrew
>
> On Thu, Jul 1, 2021, 2:17 PM Phillip Lord <ph...@onyxpoint.com>
> wrote:
>
> > Hello,
> >
> > My organization is considering utilizing the Registry. From my testing it
> > appears that versioning doesn't keep track of the state of components
> > (stopped/started/etc). Is this accurate? Are there plans to have
> > versioning keep track of this in future releases?
> >
> > I'm using NiFi 1.11.4 and Registry version 0.8.0
> >
> > Thanks,
> > Phil Lord
> >
>
Re: Apache NiFi Registry
Posted by Andrew Grande <ap...@gmail.com>.
Isn't the proper state for this use case enabled/disabled? NiFi will start
a PG and every schedulable component in it. If one needs to prevent this,
disable a processor.
Andrew
On Thu, Jul 1, 2021, 2:17 PM Phillip Lord <ph...@onyxpoint.com>
wrote:
> Hello,
>
> My organization is considering utilizing the Registry. From my testing it
> appears that versioning doesn't keep track of the state of components
> (stopped/started/etc). Is this accurate? Are there plans to have
> versioning keep track of this in future releases?
>
> I'm using NiFi 1.11.4 and Registry version 0.8.0
>
> Thanks,
> Phil Lord
>