Nifi Registry git presistence

[Tomislav Novosel <to...@clearpeaks.com>]

Hi to all,

i sit possible to have one central Git repo for multiple NiFi registries
as persistance layer?
Ii it possible to configure, for example, three NiFi registries (through providers.xml file)
to communicate with one git repo?

Thanks,
Tom

RE: Nifi Registry git presistence

[Tomislav Novosel <to...@clearpeaks.com>]

Hi Bryan,

Thanks for the explanation.

Unfortunately, we don't have possibility at all to integrate all the envs with one central NiFi registry nor to integrate dedicated NiFi registries between
the envs, security reasons etc, no connectivity between them.

The way we are having is to sync underlaying git repos when transferring the flows to higher logical env.

We have also parameter context which needs to be env specific. Is there a way how to automate this, to have like some kind of tmeplate file which can be populated
with specific env values?
Just to avoid manually modifying parameter context file and then to import it to the higher env.

Best regards,
Tom

-----Original Message-----
From: Bryan Bende <bb...@gmail.com> 
Sent: 23 June 2021 16:34
To: users@nifi.apache.org
Subject: Re: Nifi Registry git presistence

There is no automatic pulling from git. The way registry works is that there is a database (H2/MySql/Postgres) that stores all the metadata and knowledge of what items exist, and then a persistence provider for each time of versioned item (flows, extension bundles). Git is just an implementation of a flow persistence provider, so it was really meant to be treated like just another place to store a blob. There are also file system and database providers for flow persistence.

When registry starts up, if the database is empty and git is configured, then it reads the git repo to rebuild the database. So the trick some people are using, is that in higher environments like prod where the registry is read-only, they periodically stop registry, delete the H2 database, git pull, and start again, and now everything is up to date.

On Wed, Jun 23, 2021 at 10:21 AM Tomislav Novosel <to...@clearpeaks.com> wrote:
>
> Hi Chris, Bryan,
>
> Thanks for the answers.
>
> How to configure "follower" Registries to pull from the "leader" automatically? What if one of the Registries goes down, how to recover, is it tricky?
>
> We have three separated envs, with separated nifi registries. Is it even required to have git as a persistence layer?
> What is the difference if we setup just file persistence layer in comparison to git?
>
> I know Toolkit CLI can be used in our case for transfering the flows 
> between registries, but I'm curious and want to know pros and cons of having one git repo for all the registries.
>
> Thanks,
> Tom
> -----Original Message-----
> From: Bryan Bende <bb...@gmail.com>
> Sent: 23 June 2021 14:58
> To: users@nifi.apache.org
> Subject: Re: Nifi Registry git presistence
>
> It wasn't really meant for that, but as others pointed out, if only one of the registries is doing writes and the others are read-only for pulling into nifi in another environment, then it can work.
>
> The best option is to have one central registry that all nifi's can access (if you are willing to access the same git repo from all environments, then why not the same nifi registry?).
>
> The second best option is the database persistence provider, with all registries pointing at the same database.
>
> On Wed, Jun 23, 2021 at 7:56 AM Chris McKeever <cg...@gmail.com> wrote:
> >
> > Yes -- however:
> > - only one should be the authority
> > - see the older thread of how you need to have the read-only-clients re-poll the remote git in order ot pick up the latest changes.
> >
> > On Wed, Jun 23, 2021 at 5:25 AM Tomislav Novosel <to...@clearpeaks.com> wrote:
> >>
> >> Hi to all,
> >>
> >>
> >>
> >> i sit possible to have one central Git repo for multiple NiFi 
> >> registries
> >>
> >> as persistance layer?
> >>
> >> Ii it possible to configure, for example, three NiFi registries 
> >> (through providers.xml file)
> >>
> >> to communicate with one git repo?
> >>
> >>
> >>
> >> Thanks,
> >>
> >> Tom

Re: Nifi Registry git presistence

[Bryan Bende <bb...@gmail.com>]

There is no automatic pulling from git. The way registry works is that
there is a database (H2/MySql/Postgres) that stores all the metadata
and knowledge of what items exist, and then a persistence provider for
each time of versioned item (flows, extension bundles). Git is just an
implementation of a flow persistence provider, so it was really meant
to be treated like just another place to store a blob. There are also
file system and database providers for flow persistence.

When registry starts up, if the database is empty and git is
configured, then it reads the git repo to rebuild the database. So the
trick some people are using, is that in higher environments like prod
where the registry is read-only, they periodically stop registry,
delete the H2 database, git pull, and start again, and now everything
is up to date.

On Wed, Jun 23, 2021 at 10:21 AM Tomislav Novosel
<to...@clearpeaks.com> wrote:
>
> Hi Chris, Bryan,
>
> Thanks for the answers.
>
> How to configure "follower" Registries to pull from the "leader" automatically? What if one of the Registries goes down, how to recover, is it tricky?
>
> We have three separated envs, with separated nifi registries. Is it even required to have git as a persistence layer?
> What is the difference if we setup just file persistence layer in comparison to git?
>
> I know Toolkit CLI can be used in our case for transfering the flows between registries, but I'm curious and want to know pros and cons of having
> one git repo for all the registries.
>
> Thanks,
> Tom
> -----Original Message-----
> From: Bryan Bende <bb...@gmail.com>
> Sent: 23 June 2021 14:58
> To: users@nifi.apache.org
> Subject: Re: Nifi Registry git presistence
>
> It wasn't really meant for that, but as others pointed out, if only one of the registries is doing writes and the others are read-only for pulling into nifi in another environment, then it can work.
>
> The best option is to have one central registry that all nifi's can access (if you are willing to access the same git repo from all environments, then why not the same nifi registry?).
>
> The second best option is the database persistence provider, with all registries pointing at the same database.
>
> On Wed, Jun 23, 2021 at 7:56 AM Chris McKeever <cg...@gmail.com> wrote:
> >
> > Yes -- however:
> > - only one should be the authority
> > - see the older thread of how you need to have the read-only-clients re-poll the remote git in order ot pick up the latest changes.
> >
> > On Wed, Jun 23, 2021 at 5:25 AM Tomislav Novosel <to...@clearpeaks.com> wrote:
> >>
> >> Hi to all,
> >>
> >>
> >>
> >> i sit possible to have one central Git repo for multiple NiFi
> >> registries
> >>
> >> as persistance layer?
> >>
> >> Ii it possible to configure, for example, three NiFi registries
> >> (through providers.xml file)
> >>
> >> to communicate with one git repo?
> >>
> >>
> >>
> >> Thanks,
> >>
> >> Tom

RE: Nifi Registry git presistence

[Tomislav Novosel <to...@clearpeaks.com>]

Hi Chris, Bryan,

Thanks for the answers.

How to configure "follower" Registries to pull from the "leader" automatically? What if one of the Registries goes down, how to recover, is it tricky?

We have three separated envs, with separated nifi registries. Is it even required to have git as a persistence layer?
What is the difference if we setup just file persistence layer in comparison to git?

I know Toolkit CLI can be used in our case for transfering the flows between registries, but I'm curious and want to know pros and cons of having
one git repo for all the registries.

Thanks,
Tom
-----Original Message-----
From: Bryan Bende <bb...@gmail.com> 
Sent: 23 June 2021 14:58
To: users@nifi.apache.org
Subject: Re: Nifi Registry git presistence

It wasn't really meant for that, but as others pointed out, if only one of the registries is doing writes and the others are read-only for pulling into nifi in another environment, then it can work.

The best option is to have one central registry that all nifi's can access (if you are willing to access the same git repo from all environments, then why not the same nifi registry?).

The second best option is the database persistence provider, with all registries pointing at the same database.

On Wed, Jun 23, 2021 at 7:56 AM Chris McKeever <cg...@gmail.com> wrote:
>
> Yes -- however:
> - only one should be the authority
> - see the older thread of how you need to have the read-only-clients re-poll the remote git in order ot pick up the latest changes.
>
> On Wed, Jun 23, 2021 at 5:25 AM Tomislav Novosel <to...@clearpeaks.com> wrote:
>>
>> Hi to all,
>>
>>
>>
>> i sit possible to have one central Git repo for multiple NiFi 
>> registries
>>
>> as persistance layer?
>>
>> Ii it possible to configure, for example, three NiFi registries 
>> (through providers.xml file)
>>
>> to communicate with one git repo?
>>
>>
>>
>> Thanks,
>>
>> Tom

Re: Nifi Registry git presistence

[Bryan Bende <bb...@gmail.com>]

It wasn't really meant for that, but as others pointed out, if only
one of the registries is doing writes and the others are read-only for
pulling into nifi in another environment, then it can work.

The best option is to have one central registry that all nifi's can
access (if you are willing to access the same git repo from all
environments, then why not the same nifi registry?).

The second best option is the database persistence provider, with all
registries pointing at the same database.

On Wed, Jun 23, 2021 at 7:56 AM Chris McKeever <cg...@gmail.com> wrote:
>
> Yes -- however:
> - only one should be the authority
> - see the older thread of how you need to have the read-only-clients re-poll the remote git in order ot pick up the latest changes.
>
> On Wed, Jun 23, 2021 at 5:25 AM Tomislav Novosel <to...@clearpeaks.com> wrote:
>>
>> Hi to all,
>>
>>
>>
>> i sit possible to have one central Git repo for multiple NiFi registries
>>
>> as persistance layer?
>>
>> Ii it possible to configure, for example, three NiFi registries (through providers.xml file)
>>
>> to communicate with one git repo?
>>
>>
>>
>> Thanks,
>>
>> Tom

Re: Nifi Registry git presistence

[Chris McKeever <cg...@gmail.com>]

Yes -- however:
- only one should be the authority
- see the older thread of how you need to have the read-only-clients
re-poll the remote git in order ot pick up the latest changes.

On Wed, Jun 23, 2021 at 5:25 AM Tomislav Novosel <
tomislav.novosel@clearpeaks.com> wrote:

> Hi to all,
>
>
>
> i sit possible to have one central Git repo for multiple NiFi registries
>
> as persistance layer?
>
> Ii it possible to configure, for example, three NiFi registries (through
> providers.xml file)
>
> to communicate with one git repo?
>
>
>
> Thanks,
>
> Tom
>