CI/CD like setup for configuring NIFI for different environment

[Midhun Mohan <mi...@esginc.us>]

Hi all,

I need to know if there is a possible way to have CI/CD like pipeline setup
on NIFI.

So that I can run deployments of nifi processors to different setups of
NIFI, without even bothering about setting it up manually or trying to use
NIFI-registry.


-- 


Regards,
Midhun Mohan

Re: CI/CD like setup for configuring NIFI for different environment

[Javi Roman <jr...@gmail.com>]

Maybe this article from Abdelkrim Hadjidj [1] can give you some ideas.

[1] http://tiny.cc/zx5lsz
--
Javi Roman

Twitter: @javiromanrh
GitHub: github.com/javiroman
Linkedin: es.linkedin.com/in/javiroman
Big Data Blog: dataintensive.info
Apache Id: javiroman


On Wed, Jul 29, 2020 at 7:11 PM Midhun Mohan <mi...@esginc.us> wrote:

> Hi all,
>
> I need to know if there is a possible way to have CI/CD like pipeline setup
> on NIFI.
>
> So that I can run deployments of nifi processors to different setups of
> NIFI, without even bothering about setting it up manually or trying to use
> NIFI-registry.
>
>
> --
>
>
> Regards,
> Midhun Mohan
>

Re: CI/CD like setup for configuring NIFI for different environment

[Mike Thomsen <mi...@gmail.com>]

You can also just dump flow.xml.gz from your dev environment and use a
Dockerfile like this:

FROM apache/nifi:1.11.4
COPY flow.xml.gz /opt/nifi/nifi-current/conf/flow.xml.gz

At the end of the day, you're going to have to write some scripts to
automate reconfiguring it in an elevated environment. Things like
changing out service endpoints, credentials, etc.

What we do is have Jenkins push custom code to standardized locations
on the file system per environment and then manually update with the
Registry.

On Wed, Jul 29, 2020 at 2:41 PM Chris Sampson
<ch...@naimuri.com.invalid> wrote:
>
> What do you mean by supplying nifi processors to different environments?
>
> Do you mean you want to deploy new versions of custom processors that
> someone could configure in the nifi UI?
>
> And/or do you mean you want to deploy updates flow definitions that have
> been defined in one environment and you want to promote to another?
>
>
> We've achieved this kind of thing in kubernetes using the apache/nifi
> docker image (securely clustered) as a statefulset.
>
> We have custom processors that we build and publish to a (private) maven
> repository (Artifactory in this case).
>
> Our nifi instances are configured using a set of scripts that use a mix of
> nifi toolkit and nipyapi library to do this such as global configuration,
> controller services, reporting tasks, parameter contexts and flow
> definitions (from registry).
>
> Custom processors and pulled from the maven repository into the nifi
> instances extensions/ directories (so they're auto-loaded for use in the
> UI).
>
> We do use nifi registry (I note you say you don't want to) - again it's
> deployed via a ci/cd pipeline and configured with a set of scripts in the
> same way as nifi itself. We use the git flow persistence so our flows are
> persisted outside of kubernetes and shared between environments.
>
> This way, we use our custom processors in dev to author our flows, which
> are committed to registry. We then deploy the flows to a new environment
> through our registry and nifi pipelines.
>
> If you really don't want to use registry as well, I'd imagine you could use
> the older templates & variable registry approaches.
>
>
> As a general note, nifi is all driven via a restful api, so everything
> you'd like to do is possible using a script.
>
>
> Cheers,
>
> Chris Sampson
>
> On Wed, 29 Jul 2020, 18:11 Midhun Mohan, <mi...@esginc.us> wrote:
>
> > Hi all,
> >
> > I need to know if there is a possible way to have CI/CD like pipeline setup
> > on NIFI.
> >
> > So that I can run deployments of nifi processors to different setups of
> > NIFI, without even bothering about setting it up manually or trying to use
> > NIFI-registry.
> >
> >
> > --
> >
> >
> > Regards,
> > Midhun Mohan
> >

Re: CI/CD like setup for configuring NIFI for different environment

[Chris Sampson <ch...@naimuri.com.INVALID>]

What do you mean by supplying nifi processors to different environments?

Do you mean you want to deploy new versions of custom processors that
someone could configure in the nifi UI?

And/or do you mean you want to deploy updates flow definitions that have
been defined in one environment and you want to promote to another?


We've achieved this kind of thing in kubernetes using the apache/nifi
docker image (securely clustered) as a statefulset.

We have custom processors that we build and publish to a (private) maven
repository (Artifactory in this case).

Our nifi instances are configured using a set of scripts that use a mix of
nifi toolkit and nipyapi library to do this such as global configuration,
controller services, reporting tasks, parameter contexts and flow
definitions (from registry).

Custom processors and pulled from the maven repository into the nifi
instances extensions/ directories (so they're auto-loaded for use in the
UI).

We do use nifi registry (I note you say you don't want to) - again it's
deployed via a ci/cd pipeline and configured with a set of scripts in the
same way as nifi itself. We use the git flow persistence so our flows are
persisted outside of kubernetes and shared between environments.

This way, we use our custom processors in dev to author our flows, which
are committed to registry. We then deploy the flows to a new environment
through our registry and nifi pipelines.

If you really don't want to use registry as well, I'd imagine you could use
the older templates & variable registry approaches.


As a general note, nifi is all driven via a restful api, so everything
you'd like to do is possible using a script.


Cheers,

Chris Sampson

On Wed, 29 Jul 2020, 18:11 Midhun Mohan, <mi...@esginc.us> wrote:

> Hi all,
>
> I need to know if there is a possible way to have CI/CD like pipeline setup
> on NIFI.
>
> So that I can run deployments of nifi processors to different setups of
> NIFI, without even bothering about setting it up manually or trying to use
> NIFI-registry.
>
>
> --
>
>
> Regards,
> Midhun Mohan
>