[jira] [Commented] (NIFIREG-218) Integrity issue when git flow provider fails

["Bryan Bende (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/NIFIREG-218?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16723030#comment-16723030 ] 

Bryan Bende commented on NIFIREG-218:
-------------------------------------

[~jgondron] thanks for the thorough explanation of the issue!

I think the reason this scenario can happen is because we chose to make pushing to remote an async process since it was fairly slow and thus we didn't want the user thread in NiFi to be waiting every time they commit changes for a PG.

If it was done synchronously then presumably if an exception was thrown during the push, then it will rollback the DB transaction and none of the entries the DB would get created.

The downside would be that your registry would basically be unusable when the remote was unreachable, where as currently you can continue working with everything getting saved to the local repo, and then pushed as soon as the remote comes back online.

> Integrity issue when git flow provider fails
> --------------------------------------------
>
>                 Key: NIFIREG-218
>                 URL: https://issues.apache.org/jira/browse/NIFIREG-218
>             Project: NiFi Registry
>          Issue Type: Bug
>            Reporter: Justin Gondron
>            Priority: Major
>
> While working through https://github.com/apache/nifi-registry/pull/152, I came across a related issue when using a git flow provider. It's possible to successfully push a version from nifi to the registry, without it pushing to github. When this happens, it appears to the nifi instance that the version was pushed successfully, and on the registry side it has metadata in the database about that version, but the corresponding flow file for that version will not be in the repo. This means a few things:
>  * If you ever restart the registry host and try to rebuild it, there will be lost changes.
>  * The registry will show a bucket or flow version that cannot actually be used by a nifi instance
>  * You will be unable to delete the bucket and/or flow version via the registry UI
> One way to recreate:
>  * First, create the inconsistent state between an external database and flow files
>  ** Setup a read only user in your upstream repository
>  ** Clone the repo into your registry using this user
>  ** Configure the flow provider using this read only user
>  ** Configure the registry to use an external db
>  ** Start the registry
>  ** Create a new bucket
>  ** Push a version from Nifi to the registry using the new bucket
>  ** It will appear to Nifi that the version was successfully committed
>  ** The registry UI will show the new bucket and new flow
>  ** The bucket/flow will not be in the upstream repository
>  * Recreate the registry instance
>  ** Start a new registry service
>  ** Configure the registry to use the same external db
>  ** Configure the flow provider/clone the repo as before
>  ** Start the registry
>  ** The bucket/flow will appear in the registry UI, but you will be unable to delete them and nifi instances will be unable to track to them since there are no corresponding flow files
> This is primarily an issue in a Docker environment where you're trying to persist metadata to an external database and the flow files to github. But, I can see this also occurring if you had problems pushing to github for an indeterminate period of time and then had to perform an emergency restoration of the registry.
>  
> Also, this may be related to https://issues.apache.org/jira/browse/NIFIREG-204, not sure.
>  



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)