Registry and db connection services

[Max <na...@gmail.com>]

Hello!

I’d like to use registry for deploying my flows in various environments.

Let’s say on dev I create a flow that has an ExecuteSQL processor and this
processor references a db controller service, which is defined on the root
level.

I then version my flow and commit. Next, I go to my production env and
import the flow. The result is that the ExecuteSQL processor on prod
will be not enabled because the property “Database Connection Pooling
Service” will say “Incompatible Controller Service configured” where on dev
I had the db referenced.

Maybe I don’t get versioning in NiFi yet but how is that supposed to work
to reference controller services (db connections in particular) that are
defined on the root level when using registry?

Re: Registry and db connection services

[Bryan Bende <bb...@gmail.com>]

There really isn't a correct answer, it depends on how you are using
NiFi and how you want to organize your flows.

You can make a self-contained versioned process group that contains
all of the services with in it, and may or may not contain child
process groups.

You could also have services at the root level that are external to
all of the versioned process groups, and when importing to the next
environment you have the same corresponding service at the root level
and need to re-link the references.

On Wed, Apr 10, 2019 at 9:01 PM Max <na...@gmail.com> wrote:
>
> OK that makes sense. But I could create a parent process group, add my controller services there, and reference them from the child process groups. Then I put the parent process group under version control and I have my desired result, I think.
>
> Is that common/useful pattern to have multiple child process groups and only version the parent?
>
> Sorry for the basic questions here, I'm still trying to figure out a good pattern how to work with NiFi and deploy the result. :-)
>
> On Wed, 10 Apr 2019 at 21:47, Bryan Bende <bb...@gmail.com> wrote:
>>
>> Hello,
>>
>> When you version control a process group, it captures all components
>> in that group and child groups. If a component references a controller
>> service from a parent group, then when you import this process group
>> to the next environment you will have to re-link that component to the
>> appropriate service in the parent group of the new environment. You
>> only have to make this change when time during initial import. After
>> that on future upgrades it will retain the link to the service.
>>
>> The reason it works this way is because NiFi can't know for sure what
>> service in the new environment to select since its a service that was
>> not part of version control. There is a JIRA for a future improvement
>> to attempt to auto-select the service from a parent group when there
>> is only one service of the required type with the same name (name is
>> not unique so if there is more than one with the same name we can't
>> know which one to select).
>>
>> -Bryan
>>
>> On Wed, Apr 10, 2019 at 5:17 AM Max <na...@gmail.com> wrote:
>> >
>> > Hello!
>> >
>> > I’d like to use registry for deploying my flows in various environments.
>> >
>> > Let’s say on dev I create a flow that has an ExecuteSQL processor and this processor references a db controller service, which is defined on the root level.
>> >
>> > I then version my flow and commit. Next, I go to my production env and import the flow. The result is that the ExecuteSQL processor on prod
>> > will be not enabled because the property “Database Connection Pooling Service” will say “Incompatible Controller Service configured” where on dev I had the db referenced.
>> >
>> > Maybe I don’t get versioning in NiFi yet but how is that supposed to work to reference controller services (db connections in particular) that are defined on the root level when using registry?

Re: Registry and db connection services

[Max <na...@gmail.com>]

OK that makes sense. But I could create a parent process group, add my
controller services there, and reference them from the child process
groups. Then I put the parent process group under version control and I
have my desired result, I think.

Is that common/useful pattern to have multiple child process groups and
only version the parent?

Sorry for the basic questions here, I'm still trying to figure out a good
pattern how to work with NiFi and deploy the result. :-)

On Wed, 10 Apr 2019 at 21:47, Bryan Bende <bb...@gmail.com> wrote:

> Hello,
>
> When you version control a process group, it captures all components
> in that group and child groups. If a component references a controller
> service from a parent group, then when you import this process group
> to the next environment you will have to re-link that component to the
> appropriate service in the parent group of the new environment. You
> only have to make this change when time during initial import. After
> that on future upgrades it will retain the link to the service.
>
> The reason it works this way is because NiFi can't know for sure what
> service in the new environment to select since its a service that was
> not part of version control. There is a JIRA for a future improvement
> to attempt to auto-select the service from a parent group when there
> is only one service of the required type with the same name (name is
> not unique so if there is more than one with the same name we can't
> know which one to select).
>
> -Bryan
>
> On Wed, Apr 10, 2019 at 5:17 AM Max <na...@gmail.com> wrote:
> >
> > Hello!
> >
> > I’d like to use registry for deploying my flows in various environments.
> >
> > Let’s say on dev I create a flow that has an ExecuteSQL processor and
> this processor references a db controller service, which is defined on the
> root level.
> >
> > I then version my flow and commit. Next, I go to my production env and
> import the flow. The result is that the ExecuteSQL processor on prod
> > will be not enabled because the property “Database Connection Pooling
> Service” will say “Incompatible Controller Service configured” where on dev
> I had the db referenced.
> >
> > Maybe I don’t get versioning in NiFi yet but how is that supposed to
> work to reference controller services (db connections in particular) that
> are defined on the root level when using registry?
>

Re: Registry and db connection services

[Bryan Bende <bb...@gmail.com>]

Hello,

When you version control a process group, it captures all components
in that group and child groups. If a component references a controller
service from a parent group, then when you import this process group
to the next environment you will have to re-link that component to the
appropriate service in the parent group of the new environment. You
only have to make this change when time during initial import. After
that on future upgrades it will retain the link to the service.

The reason it works this way is because NiFi can't know for sure what
service in the new environment to select since its a service that was
not part of version control. There is a JIRA for a future improvement
to attempt to auto-select the service from a parent group when there
is only one service of the required type with the same name (name is
not unique so if there is more than one with the same name we can't
know which one to select).

-Bryan

On Wed, Apr 10, 2019 at 5:17 AM Max <na...@gmail.com> wrote:
>
> Hello!
>
> I’d like to use registry for deploying my flows in various environments.
>
> Let’s say on dev I create a flow that has an ExecuteSQL processor and this processor references a db controller service, which is defined on the root level.
>
> I then version my flow and commit. Next, I go to my production env and import the flow. The result is that the ExecuteSQL processor on prod
> will be not enabled because the property “Database Connection Pooling Service” will say “Incompatible Controller Service configured” where on dev I had the db referenced.
>
> Maybe I don’t get versioning in NiFi yet but how is that supposed to work to reference controller services (db connections in particular) that are defined on the root level when using registry?