What happens to in-progess flow files during a process group version update?

[Tim Dean <ti...@gmail.com>]

I’m trying to understand what can and cannot happen to in-process flow files during an upgrade for a versioned process group. For example, what if version 1 of a PG has several flow files queued up waiting to be processed when I upgrade to version 2? Will NiFi prevent me from upgrading until all flow files have been processed? Or will it attempt to upgrade while flow files are still in process?

I know that the update will cause processors within the PG to be stopped and then I presume they get restarted. If the only change in the new version are things like processor property changes, it seems like q	queued flow files would just continue being processed. But what if connections were added or removed? Seems like queued work could get lost or improperly routed if NiFi attempted to handle it automatically.

I’ve searched through various NiFi registry documents but not found any explanation about how this kind of update problem is handled.

Thanks

-Tim

Re: What happens to in-progess flow files during a process group version update?

[Tim Dean <ti...@gmail.com>]

Thanks Bryan -

Are there any other considerations that are used to determine if an upgrade should be prevented? Or is it strictly a matter of making sure that any queued flow files are in queues that will continue to exist after upgrade. I assume, for example, that there is no way to prevent upgrades if the schema of a flow file changes incompatibly from one version to the next? 

What I’m really trying to understand is the rules/guidelines I need to follow in my flow designs to make sure that upgrading from one version to another will not result in loss of data.

Thanks

-Tim

> On Mar 4, 2019, at 1:18 PM, Bryan Bende <bb...@gmail.com> wrote:
> 
> Tim,
> 
> The upgrade will be prevented if there are any queues that currently
> have flow files in them that would be removed during the upgrade.
> 
> For example, processor A is connected to B and there flow files in the
> queue, but in v2 A is no longer connected to B, you won't be able to
> upgrade until you clear the queue which can be done by stopping A
> first and letting it bleed out.
> 
> All other changes besides this can be applied in place and everything
> will continue as normal.
> 
> -Bryan
> 
> On Mon, Mar 4, 2019 at 1:48 PM Tim Dean <ti...@gmail.com> wrote:
>> 
>> I’m trying to understand what can and cannot happen to in-process flow files during an upgrade for a versioned process group. For example, what if version 1 of a PG has several flow files queued up waiting to be processed when I upgrade to version 2? Will NiFi prevent me from upgrading until all flow files have been processed? Or will it attempt to upgrade while flow files are still in process?
>> 
>> I know that the update will cause processors within the PG to be stopped and then I presume they get restarted. If the only change in the new version are things like processor property changes, it seems like q       queued flow files would just continue being processed. But what if connections were added or removed? Seems like queued work could get lost or improperly routed if NiFi attempted to handle it automatically.
>> 
>> I’ve searched through various NiFi registry documents but not found any explanation about how this kind of update problem is handled.
>> 
>> Thanks
>> 
>> -Tim

Re: What happens to in-progess flow files during a process group version update?

[Bryan Bende <bb...@gmail.com>]

Tim,

The upgrade will be prevented if there are any queues that currently
have flow files in them that would be removed during the upgrade.

For example, processor A is connected to B and there flow files in the
queue, but in v2 A is no longer connected to B, you won't be able to
upgrade until you clear the queue which can be done by stopping A
first and letting it bleed out.

All other changes besides this can be applied in place and everything
will continue as normal.

-Bryan

On Mon, Mar 4, 2019 at 1:48 PM Tim Dean <ti...@gmail.com> wrote:
>
> I’m trying to understand what can and cannot happen to in-process flow files during an upgrade for a versioned process group. For example, what if version 1 of a PG has several flow files queued up waiting to be processed when I upgrade to version 2? Will NiFi prevent me from upgrading until all flow files have been processed? Or will it attempt to upgrade while flow files are still in process?
>
> I know that the update will cause processors within the PG to be stopped and then I presume they get restarted. If the only change in the new version are things like processor property changes, it seems like q       queued flow files would just continue being processed. But what if connections were added or removed? Seems like queued work could get lost or improperly routed if NiFi attempted to handle it automatically.
>
> I’ve searched through various NiFi registry documents but not found any explanation about how this kind of update problem is handled.
>
> Thanks
>
> -Tim