You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@nifi.apache.org by Jorge Machado <jo...@me.com> on 2022/10/07 13:07:16 UTC
Workflows deployments across environments
Hello Nifi users,
Question in the round. Let’s say I have 3 Nifi instances with dev, test and prod. What is the recommend way of deploying specific workflows from one environment to another ?
We are using Nifi registry + GitHub as storage for the flows.
What we want to achieve:
* in production there should be only read rights
* Deployment in prod should happen when a commit on master or a new git release is created.
I found this: https://chamila.dev/blog/2021-07-17_ci-cd-for-apache-nifi-01-intro-and-setup/ and https://medium.com/tech-at-nordstrom/building-ci-cd-pipelines-with-apache-nifi-and-terraform-acedd653b356 both ways seems to be like custom made and there is not a standard way of doing this.
Would be great if we had a wiki page on nifi for this.
Thx
Jorge
Re: Workflows deployments across environments
Posted by Pierre Villard <pi...@gmail.com>.
Hi,
You may find this video useful: https://www.youtube.com/watch?v=XYHMExiWM6k
The title is misleading since the video does cover automated deployments
across environments with NiFi Registry.
Note that the recently released NiFi 1.18 version introduces the concept of
parameter providers which can greatly simplify automation of flow
deployments. I'll try to spend some time recording a demo/video leveraging
this new feature in the context of CI/CD & NiFi Registry.
HTH,
Pierre
Le ven. 7 oct. 2022 à 09:45, Bryan Bende <bb...@gmail.com> a écrit :
> In NiFi Registry, each of your NIFi servers must be defined as a user
> with Proxy permissions.
>
> To make production be read-only, give the production NiFi user only
> Proxy - READ in NiFi Registry. This will cause it to fail
> authorization if it ever attempted to save anything to registry.
>
> Then give your dev NiFi Proxy - READ/WRITE/DELETE.
>
> On Fri, Oct 7, 2022 at 9:22 AM Marel, J. van der (Jasper) via users
> <us...@nifi.apache.org> wrote:
> >
> > Hi,
> >
> > Regarding read rights depends on nifi internal rights, cannot be set on
> workflows itself. But that can be accomplished with a nipyapi calls.
> >
> > Regarding deployments, this is more a ci/cd setup with triggers
> >
> > With kind regards,
> >
> > Jasper
> >
> > Outlook voor Android downloaden
> >
> > ________________________________
> > Van: Jorge Machado <jo...@me.com>
> > Verstuurd: vrijdag 7 oktober 2022 15:07
> > Aan: users@nifi.apache.org <us...@nifi.apache.org>
> > Onderwerp: Workflows deployments across environments
> >
> > Hello Nifi users,
> >
> >
> > Question in the round. Let’s say I have 3 Nifi instances with dev, test
> and prod. What is the recommend way of deploying specific workflows from
> one environment to another ?
> > We are using Nifi registry + GitHub as storage for the flows.
> >
> > What we want to achieve:
> > * in production there should be only read rights
> > * Deployment in prod should happen when a commit on master or a new git
> release is created.
> >
> >
> > I found this:
> https://eur02.safelinks.protection.outlook.com/?url=https%3A%2F%2Fchamila.dev%2Fblog%2F2021-07-17_ci-cd-for-apache-nifi-01-intro-and-setup%2F&data=05%7C01%7CJasper.van.der.Marel%40ing.com%7Ccb499dffa57649fd0d0e08daa864e989%7C587b6ea13db94fe1a9d785d4c64ce5cc%7C0%7C0%7C638007448610342559%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=04bmOAo7vZAajkGJRn31jwMpkRVqg9%2FIPmx4OiLwG9k%3D&reserved=0
> and
> https://eur02.safelinks.protection.outlook.com/?url=https%3A%2F%2Fmedium.com%2Ftech-at-nordstrom%2Fbuilding-ci-cd-pipelines-with-apache-nifi-and-terraform-acedd653b356&data=05%7C01%7CJasper.van.der.Marel%40ing.com%7Ccb499dffa57649fd0d0e08daa864e989%7C587b6ea13db94fe1a9d785d4c64ce5cc%7C0%7C0%7C638007448610342559%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=2WLVMUD2zA2gPqfkU62mRZDa4ftzYuvgegJ70WUOtNU%3D&reserved=0
> both ways seems to be like custom made and there is not a standard way of
> doing this.
> > Would be great if we had a wiki page on nifi for this.
> >
> >
> > Thx
> > Jorge
> >
> >
> > -----------------------------------------------------------------
> > ATTENTION:
> > The information in this e-mail is confidential and only meant for the
> intended recipient. If you are not the intended recipient, don't use or
> disclose it in any way. Please let the sender know and delete the message
> immediately.
> > -----------------------------------------------------------------
>
how to debug PutDatabaseRecord "Unable to setObject() with value desc at index 1 of type 12"
Posted by Richard Staehli <Ri...@expeditors.com>.
The details of my current problem shown below are probably less relevant that my bigger question of "how to debug" configuration of a processor like PutDatabaseRecord? It would help to have a working example, updating a DB2 database table with data from json records. When I get the error "Unable to setObject() with value desc at index 1 of type 12" I can tell from the data "desc" that it is referring to the demo data value "desc" that I'm trying to use in the update to the DESCRIPTION column in this table. But this is just a "string" in the json schema and the column type "12" is just a varchar(160) so it's hard to guess at what the actual problem is.
The processor is configured like this:
[cid:image001.png@01D8DCC8.CB3BBF50]
And the JsonTreeReader like this:
[cid:image002.png@01D8DCC9.38DE2750]
And Database Connection Pooling Service:
[cid:image003.png@01D8DCCA.560A4DD0]
Re: Workflows deployments across environments
Posted by Bryan Bende <bb...@gmail.com>.
In NiFi Registry, each of your NIFi servers must be defined as a user
with Proxy permissions.
To make production be read-only, give the production NiFi user only
Proxy - READ in NiFi Registry. This will cause it to fail
authorization if it ever attempted to save anything to registry.
Then give your dev NiFi Proxy - READ/WRITE/DELETE.
On Fri, Oct 7, 2022 at 9:22 AM Marel, J. van der (Jasper) via users
<us...@nifi.apache.org> wrote:
>
> Hi,
>
> Regarding read rights depends on nifi internal rights, cannot be set on workflows itself. But that can be accomplished with a nipyapi calls.
>
> Regarding deployments, this is more a ci/cd setup with triggers
>
> With kind regards,
>
> Jasper
>
> Outlook voor Android downloaden
>
> ________________________________
> Van: Jorge Machado <jo...@me.com>
> Verstuurd: vrijdag 7 oktober 2022 15:07
> Aan: users@nifi.apache.org <us...@nifi.apache.org>
> Onderwerp: Workflows deployments across environments
>
> Hello Nifi users,
>
>
> Question in the round. Let’s say I have 3 Nifi instances with dev, test and prod. What is the recommend way of deploying specific workflows from one environment to another ?
> We are using Nifi registry + GitHub as storage for the flows.
>
> What we want to achieve:
> * in production there should be only read rights
> * Deployment in prod should happen when a commit on master or a new git release is created.
>
>
> I found this: https://eur02.safelinks.protection.outlook.com/?url=https%3A%2F%2Fchamila.dev%2Fblog%2F2021-07-17_ci-cd-for-apache-nifi-01-intro-and-setup%2F&data=05%7C01%7CJasper.van.der.Marel%40ing.com%7Ccb499dffa57649fd0d0e08daa864e989%7C587b6ea13db94fe1a9d785d4c64ce5cc%7C0%7C0%7C638007448610342559%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=04bmOAo7vZAajkGJRn31jwMpkRVqg9%2FIPmx4OiLwG9k%3D&reserved=0 and https://eur02.safelinks.protection.outlook.com/?url=https%3A%2F%2Fmedium.com%2Ftech-at-nordstrom%2Fbuilding-ci-cd-pipelines-with-apache-nifi-and-terraform-acedd653b356&data=05%7C01%7CJasper.van.der.Marel%40ing.com%7Ccb499dffa57649fd0d0e08daa864e989%7C587b6ea13db94fe1a9d785d4c64ce5cc%7C0%7C0%7C638007448610342559%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=2WLVMUD2zA2gPqfkU62mRZDa4ftzYuvgegJ70WUOtNU%3D&reserved=0 both ways seems to be like custom made and there is not a standard way of doing this.
> Would be great if we had a wiki page on nifi for this.
>
>
> Thx
> Jorge
>
>
> -----------------------------------------------------------------
> ATTENTION:
> The information in this e-mail is confidential and only meant for the intended recipient. If you are not the intended recipient, don't use or disclose it in any way. Please let the sender know and delete the message immediately.
> -----------------------------------------------------------------
Re: Workflows deployments across environments
Posted by "Marel, J. van der (Jasper) via users" <us...@nifi.apache.org>.
Hi,
Regarding read rights depends on nifi internal rights, cannot be set on workflows itself. But that can be accomplished with a nipyapi calls.
Regarding deployments, this is more a ci/cd setup with triggers
With kind regards,
Jasper
Outlook voor Android<https://aka.ms/AAb9ysg> downloaden
________________________________
Van: Jorge Machado <jo...@me.com>
Verstuurd: vrijdag 7 oktober 2022 15:07
Aan: users@nifi.apache.org <us...@nifi.apache.org>
Onderwerp: Workflows deployments across environments
Hello Nifi users,
Question in the round. Let’s say I have 3 Nifi instances with dev, test and prod. What is the recommend way of deploying specific workflows from one environment to another ?
We are using Nifi registry + GitHub as storage for the flows.
What we want to achieve:
* in production there should be only read rights
* Deployment in prod should happen when a commit on master or a new git release is created.
I found this: https://eur02.safelinks.protection.outlook.com/?url=https%3A%2F%2Fchamila.dev%2Fblog%2F2021-07-17_ci-cd-for-apache-nifi-01-intro-and-setup%2F&data=05%7C01%7CJasper.van.der.Marel%40ing.com%7Ccb499dffa57649fd0d0e08daa864e989%7C587b6ea13db94fe1a9d785d4c64ce5cc%7C0%7C0%7C638007448610342559%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=04bmOAo7vZAajkGJRn31jwMpkRVqg9%2FIPmx4OiLwG9k%3D&reserved=0 and https://eur02.safelinks.protection.outlook.com/?url=https%3A%2F%2Fmedium.com%2Ftech-at-nordstrom%2Fbuilding-ci-cd-pipelines-with-apache-nifi-and-terraform-acedd653b356&data=05%7C01%7CJasper.van.der.Marel%40ing.com%7Ccb499dffa57649fd0d0e08daa864e989%7C587b6ea13db94fe1a9d785d4c64ce5cc%7C0%7C0%7C638007448610342559%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=2WLVMUD2zA2gPqfkU62mRZDa4ftzYuvgegJ70WUOtNU%3D&reserved=0 both ways seems to be like custom made and there is not a standard way of doing this.
Would be great if we had a wiki page on nifi for this.
Thx
Jorge
-----------------------------------------------------------------
ATTENTION:
The information in this e-mail is confidential and only meant for the intended recipient. If you are not the intended recipient, don't use or disclose it in any way. Please let the sender know and delete the message immediately.
-----------------------------------------------------------------