Invoking a Processor through REST API

[Arun Manivannan <ar...@arunma.com>]

Hi,

Good evening.

I've been using NiFi for a month now and have all good things to say about
it. Truly amazing work !! Thanks a lot!

*Gist* :  What's the correct way to invoke a NiFi flow from an external
scheduler such as Control-M (https://en.wikipedia.org/wiki/BMC_Control-M)

We have around 40 jobs (one per country) generating about 100 small/medium
sized files each (about 1-20 MB).  Each of these jobs end at different
times and they need to be moved from one Hadoop cluster to another for
further processing.  We intend to use NiFi for this purpose and we have a
very simple initial prototype for this (attached screenshot).  The reason
for not going for distcp is because we would like to add more provisions
into the flow such as reconciliation and Hive external table creation.

The last Control-M job must kick off this Flow in NiFi.  I did explore the
nifi-api (https://nifi.apache.org/docs/nifi-docs/rest-api/) but it seems
like I need to know the processor id and I am required to start and stop
the processor via the API to achieve this.

*Questions : *

1. Is there a way to have the processor running all the time and invoke it
by "name" when needed.
2. Is there a way I could use the HandleHttpRequest to activate the
ListHDFS Processor.
3. I understand that the ListHDFS has it's INPUT_FORBIDDEN and is it okay
to remove it on my local copy?  Is there a reason why it is designed that
way?
4. For all the 40 countries, is it possible to re-use the same flow and
have them parameterized on the "source cluster", "destination cluster",
"source path" and "destination path"?  Do you think there is a possible
concurrency bug waiting to happen (especially on the ListHDFS component
where I shouldn't be updating the directory while the files are being
listed. A lock, may be?).

Apologies if the questions are on a tangent.

Best Regards,
Arun

[image: Screen Shot 2017-04-04 at 7.21.43 PM.png]

Fwd: Invoking a Processor through REST API

[Arun Manivannan <ar...@arunma.com>]

Hi,

Good morning.


(Reposting this on dev from users because this is a better place.)

I've been using NiFi for a month now and have all good things to say about
it. Truly amazing work !! Thanks a lot!

*Gist* :  What's the correct way to invoke a NiFi flow from an external
scheduler such as Control-M (https://en.wikipedia.org/wiki/BMC_Control-M)

We have around 40 jobs (one per country) generating about 100 small/medium
sized files each (about 1-20 MB).  Each of these jobs end at different
times and they need to be moved from one Hadoop cluster to another for
further processing.  We intend to use NiFi for this purpose and we have a
very simple initial prototype for this (attached screenshot).  The reason
for not going for distcp is because we would like to add more provisions
into the flow such as reconciliation and Hive external table creation.

The last Control-M job must kick off this Flow in NiFi.  I did explore the
nifi-api (https://nifi.apache.org/docs/nifi-docs/rest-api/) but it seems
like I need to know the processor id and I am required to start and stop
the processor via the API to achieve this.

*Questions : *

1. Is there a way to have the processor running all the time and invoke it
by "name" when needed.
2. Is there a way I could use the HandleHttpRequest to activate the
ListHDFS Processor.
3. I understand that the ListHDFS has it's INPUT_FORBIDDEN and is it okay
to remove it on my local copy?  Is there a reason why it is designed that
way?
4. For all the 40 countries, is it possible to re-use the same flow and
have them parameterized on the "source cluster", "destination cluster",
"source path" and "destination path"?  Do you think there is a possible
concurrency bug waiting to happen (especially on the ListHDFS component
where I shouldn't be updating the directory while the files are being
listed. A lock, may be?).

Apologies if the questions are on a tangent.

Best Regards,
Arun

[image: Screen Shot 2017-04-04 at 7.21.43 PM.png]