CI/CD for custom processor in aws nifi cluster

[sanjeet rath <ra...@gmail.com>]

Hi Team,

I have a requirement of building CI/CD pipeline using jenkin & concourse to
build and deploy custom nars in AWS nifi cluster.
The CI part is completed successfully , but in the CD while doing the
deployment using concourse pipeline, we have a problem ,  we can not ssh to
port 22 of the nifi cluster nodes due to restrictions.
So we planed to copy the nar files from artifactory(jfrog) to s3 bucket  &
from S3 bucket we will copy it to custom_lib folder of the nodes of aws
nifi cluster.
we have another restriction to avoid use of cron tab to (run on each niffi
cluster nodes )to pull it from s3 .

Could you please suggest , how to copy files from s3 to cutom_lib folder of
nifi cluster and restart nifi service using jenkin and concourse

Thanks
-- 
Sanjeet Kumar Rath,
mob- +91 8777577470

Re: CI/CD for custom processor in aws nifi cluster

[Bryan Bende <bb...@gmail.com>]

Actually, I forgot auto-loading is in 1.9.0 [1], but even without that
you could do the same thing and instead write it to your custom_lib,
and then you'd have to call ExecuteScript to call "bin/nifi.sh
restart". With auto-loading then no restart is required.

[1] https://issues.apache.org/jira/browse/NIFI-5673

On Mon, Nov 18, 2019 at 12:09 PM sanjeet rath <ra...@gmail.com> wrote:
>
> Thanks Bryan for your quick response.
> we are using nifi 1.8 version.Could you please suggest something regards to 1.8 version.
>
> Regards,
> Sanjeet
>
> On Mon, 18 Nov, 2019, 10:21 PM Bryan Bende, <bb...@gmail.com> wrote:
>>
>> If you are using NiFi 1.9.x or newer, then there is a new feature to
>> auto-load new NARs in a directory specified by
>> nifi.nar.library.autoload.directory in nifi.properties.
>>
>> You could make part of your NiFi flow run ListS3 -> FetchS3 -> PutFile
>> into the auto-load directory, then NiFi will update itself and you
>> won't need to restart.
>>
>> You'll want to make List3 run on all nodes since each node needs to
>> get its own copy of the NAR.
>>
>> On Mon, Nov 18, 2019 at 11:42 AM sanjeet rath <ra...@gmail.com> wrote:
>> >
>> >
>> > Hi Team,
>> >
>> > I have a requirement of building CI/CD pipeline using jenkin & concourse to build and deploy custom nars in AWS nifi cluster.
>> > The CI part is completed successfully , but in the CD while doing the deployment using concourse pipeline, we have a problem ,  we can not ssh to port 22 of the nifi cluster nodes due to restrictions.
>> > So we planed to copy the nar files from artifactory(jfrog) to s3 bucket  & from S3 bucket we will copy it to custom_lib folder of the nodes of aws nifi cluster.
>> > we have another restriction to avoid use of cron tab to (run on each niffi cluster nodes )to pull it from s3 .
>> >
>> > Could you please suggest , how to copy files from s3 to cutom_lib folder of nifi cluster and restart nifi service using jenkin and concourse
>> >
>> > Thanks
>> > --
>> > Sanjeet Kumar Rath,
>> > mob- +91 8777577470
>> >

Re: CI/CD for custom processor in aws nifi cluster

[sanjeet rath <ra...@gmail.com>]

Thanks Bryan for your quick response.
we are using nifi 1.8 version.Could you please suggest something regards to
1.8 version.

Regards,
Sanjeet

On Mon, 18 Nov, 2019, 10:21 PM Bryan Bende, <bb...@gmail.com> wrote:

> If you are using NiFi 1.9.x or newer, then there is a new feature to
> auto-load new NARs in a directory specified by
> nifi.nar.library.autoload.directory in nifi.properties.
>
> You could make part of your NiFi flow run ListS3 -> FetchS3 -> PutFile
> into the auto-load directory, then NiFi will update itself and you
> won't need to restart.
>
> You'll want to make List3 run on all nodes since each node needs to
> get its own copy of the NAR.
>
> On Mon, Nov 18, 2019 at 11:42 AM sanjeet rath <ra...@gmail.com>
> wrote:
> >
> >
> > Hi Team,
> >
> > I have a requirement of building CI/CD pipeline using jenkin & concourse
> to build and deploy custom nars in AWS nifi cluster.
> > The CI part is completed successfully , but in the CD while doing the
> deployment using concourse pipeline, we have a problem ,  we can not ssh to
> port 22 of the nifi cluster nodes due to restrictions.
> > So we planed to copy the nar files from artifactory(jfrog) to s3 bucket
> & from S3 bucket we will copy it to custom_lib folder of the nodes of aws
> nifi cluster.
> > we have another restriction to avoid use of cron tab to (run on each
> niffi cluster nodes )to pull it from s3 .
> >
> > Could you please suggest , how to copy files from s3 to cutom_lib folder
> of nifi cluster and restart nifi service using jenkin and concourse
> >
> > Thanks
> > --
> > Sanjeet Kumar Rath,
> > mob- +91 8777577470
> >
>

Re: CI/CD for custom processor in aws nifi cluster

[Bryan Bende <bb...@gmail.com>]

If you are using NiFi 1.9.x or newer, then there is a new feature to
auto-load new NARs in a directory specified by
nifi.nar.library.autoload.directory in nifi.properties.

You could make part of your NiFi flow run ListS3 -> FetchS3 -> PutFile
into the auto-load directory, then NiFi will update itself and you
won't need to restart.

You'll want to make List3 run on all nodes since each node needs to
get its own copy of the NAR.

On Mon, Nov 18, 2019 at 11:42 AM sanjeet rath <ra...@gmail.com> wrote:
>
>
> Hi Team,
>
> I have a requirement of building CI/CD pipeline using jenkin & concourse to build and deploy custom nars in AWS nifi cluster.
> The CI part is completed successfully , but in the CD while doing the deployment using concourse pipeline, we have a problem ,  we can not ssh to port 22 of the nifi cluster nodes due to restrictions.
> So we planed to copy the nar files from artifactory(jfrog) to s3 bucket  & from S3 bucket we will copy it to custom_lib folder of the nodes of aws nifi cluster.
> we have another restriction to avoid use of cron tab to (run on each niffi cluster nodes )to pull it from s3 .
>
> Could you please suggest , how to copy files from s3 to cutom_lib folder of nifi cluster and restart nifi service using jenkin and concourse
>
> Thanks
> --
> Sanjeet Kumar Rath,
> mob- +91 8777577470
>