You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@nifi.apache.org by Russell Bateman <ru...@perfectsearchcorp.com> on 2016/07/08 21:35:50 UTC
A few NiFi ReST questions...
I'm trying to figure out how to use the ReST (nifi-api) interface to
accomplish a number of things. First, I've played successfully with it
doing simple things like getting configuration, a list of existing
processors, and the like. I've even discovered that in
controller/history/processors/{processorId} I can use the processor name
for {processorId} instead of an intangible or impossible number.
What puzzles me is that I need a {process-group-id} to do so many things.
I'm not using a process group, my flow is very simple for now. Maybe, think
I, I have one process group and I just need its id, but how do I get that?
I see no list API for it.
The long list of things I'm hoping to do down the road from a UI is to be
able to see things like:
1. How many times a processor processed anything?
2. How many flowfiles were processed?
3. How many flowfiles were produced?
4. Can I get this information per processor?
5. How can I tell different instances of the same processor apart?
6. Why can't I see the name I gave a processor in configuration? (For
example, I named an instance of GetFile to "Get PDF files from test fodder".
7. How can I get a list of items on the NiFi canvas like processors and
relationships.
8. How many times processors or a processor failed?
9. What is the profile of resource usage, like memory in use?
10. What is the profile of processor latency?
- flowfiles backed up awaiting processing
- flowfiles in other states (?)
Any nudge on any of this would be very welcome. NiFi is new enough that
there's precious few samples of people using the ReST interface.
Thanks
Re: A few NiFi ReST questions...
Posted by Russell Bateman <ru...@perfectsearchcorp.com>.
Yup, this works very well:
http://localhost:8080/nifi-api/controller/process-groups/*root*/status
Thanks,
Russ
On Fri, Jul 8, 2016 at 5:11 PM, Andy LoPresto <al...@apache.org> wrote:
> Russell,
>
> To echo Matt’s point, the NiFi UI uses the REST API on every command, and
> developer tools is a great way to observe this.
>
> Another example using the REST API is Andrew Grande’s “nifi-deploy-api”
> project on GitHub [1]. It is a groovy script that can automate the
> deployment of NiFi instances and does this using the API. If you run it
> with the ‘--debug’ flag, you can see the HTTP endpoints it invokes to
> accomplish this.
>
> Finally, the REST API is extremely well documented here [2] (also locally
> in the installation). It is extensive, but should be able to answer a
> number of the questions you asked. For example, to get processor
> statistics, you would invoke:
>
> GET /controller/process-groups/{process-group-id}/status
>
> Gets the status for a process group
> The status for a process group includes status for all descendent
> components. When invoked on the root group with recursive set to true, it
> will return the current status of every component in the flow.
>
> When I invoke that endpoint on a simple cluster I have running with a
> single Base64EncodeContent processor named “Token Example” on the canvas,
> I get this response:
>
>
> https://ncm.nifi.apache.org:4567/nifi-api/controller/process-groups/root/status
>
> <processGroupStatusEntity>
> <revision>
> <clientId>392b161b-d9cc-442a-9394-8db9cee33ba7</clientId>
> </revision>
> <processGroupStatus>
> <activeThreadCount>0</activeThreadCount>
> <id>d17d7a95-6301-46ff-b003-11c58698bd31</id>
> <input>0 / 0 bytes</input>
> <name>NiFi Flow</name>
> <output>0 / 0 bytes</output>
> <processorStatus>
> <activeThreadCount>0</activeThreadCount>
> <groupId>d17d7a95-6301-46ff-b003-11c58698bd31</groupId>
> <id>b642a34d-c18c-4bb1-b836-6bbc9d32f31e</id>
> <input>0 / 0 bytes</input>
> <name>*Token Example*</name>
> <output>0 / 0 bytes</output>
> <read>0 bytes</read>
> <runStatus>Invalid</runStatus>
> <tasks>0</tasks>
> <tasksDuration>00:00:00.000</tasksDuration>
> <type>*Base64EncodeContent*</type>
> <written>0 bytes</written>
> </processorStatus>
> <queued>0 / 0 bytes</queued>
> <queuedCount>0</queuedCount>
> <queuedSize>0 bytes</queuedSize>
> <read>0 bytes</read>
> <received>0 / 0 bytes</received>
> <sent>0 / 0 bytes</sent>
> <statsLastRefreshed>16:09:46 PDT</statsLastRefreshed>
> <transferred>0 / 0 bytes</transferred>
> <written>0 bytes</written>
> </processGroupStatus>
> </processGroupStatusEntity>
>
> [1] https://github.com/aperepel/nifi-api-deploy
> [2] https://nifi.apache.org/docs/nifi-docs/rest-api/index.html
>
> Andy LoPresto
> alopresto@apache.org
> *alopresto.apache@gmail.com <al...@gmail.com>*
> PGP Fingerprint: 70EC B3E5 98A6 5A3F D3C4 BACE 3C6E F65B 2F7D EF69
>
> On Jul 8, 2016, at 3:18 PM, Matt Gilman <ma...@gmail.com> wrote:
>
> Russell,
>
> I can provide a more detailed response when I'm back in front of my
> computer but thought I'd offer this as a quick suggestion.
>
> All flows use process groups. The blank canvas when you load your nifi
> instance is the root level process group. If you don't know the actual ID,
> you can use the alias 'root' as the path parameter for the process group ID.
>
> Please check out your browser's Dev Tools to see these API's in action.
> The UI uses these API's exclusively.
>
> Also the API's have been completely refactored in the upcoming 1.0.0
> release to align with the authorization policies of each resource. So the
> answers to your questions will depend on which version your running.
>
> Matt
>
> Sent from my iPhone
>
> On Jul 8, 2016, at 5:35 PM, Russell Bateman <
> russell.bateman@perfectsearchcorp.com> wrote:
>
> I'm trying to figure out how to use the ReST (nifi-api) interface to
> accomplish a number of things. First, I've played successfully with it
> doing simple things like getting configuration, a list of existing
> processors, and the like. I've even discovered that in
> controller/history/processors/{processorId} I can use the processor name
> for {processorId} instead of an intangible or impossible number.
>
> What puzzles me is that I need a {process-group-id} to do so many things.
> I'm not using a process group, my flow is very simple for now. Maybe, think
> I, I have one process group and I just need its id, but how do I get that?
> I see no list API for it.
>
> The long list of things I'm hoping to do down the road from a UI is to be
> able to see things like:
>
> 1. How many times a processor processed anything?
> 2. How many flowfiles were processed?
> 3. How many flowfiles were produced?
> 4. Can I get this information per processor?
> 5. How can I tell different instances of the same processor apart?
> 6. Why can't I see the name I gave a processor in configuration? (For
> example, I named an instance of GetFile to "Get PDF files from test
> fodder".
> 7. How can I get a list of items on the NiFi canvas like processors and
> relationships.
> 8. How many times processors or a processor failed?
> 9. What is the profile of resource usage, like memory in use?
> 10. What is the profile of processor latency?
> - flowfiles backed up awaiting processing
> - flowfiles in other states (?)
>
> Any nudge on any of this would be very welcome. NiFi is new enough that
> there's precious few samples of people using the ReST interface.
>
> Thanks
>
>
Re: A few NiFi ReST questions...
Posted by Russell Bateman <ru...@perfectsearchcorp.com>.
Andy,
Thanks. Yes, I know how Ajax works having used it before. It's looking at
it happening in the browser developer tools that I've never learned to do.
I've been using </> RESTED so that I can set Accept application/json which
works perfectly.
Russ
On Fri, Jul 8, 2016 at 10:55 PM, Andy LoPresto <al...@apache.org> wrote:
> Russell,
>
> No worries. The Javascript application that runs the UI is simply invoking
> Ajax (Asynchronous Javascript and XML) requests to normal HTTP endpoints
> provided by the embedded Jetty server in NiFi. You don’t need to understand
> anything about front-end development to monitor these requests using the
> Dev Tools. In fact, you could simply type the entire API endpoint into the
> browser address bar and receive the XML (or JSON-formatted, if you provide
> a header requesting it) response from the server directly. If you are more
> comfortable on the command-line or in an IDE, you can use Java, Groovy,
> etc. or even curl to invoke the API purely through code. There are also API
> tools (no endorsement, but some are listed here [1]) for hitting these APIs
> from a GUI application.
>
> Here is a good starting point for using the Network tab of the developer
> tools window [2].
>
> Glad to hear you have been able to write some custom processors. Let us
> know if you have any additional questions.
>
> [1]
> https://stackoverflow.com/questions/4486658/what-tools-do-you-use-to-test-your-public-rest-api
> [2] http://commandlinefanatic.com/cgi-bin/showarticle.cgi?article=art034
>
> Andy LoPresto
> alopresto@apache.org
> *alopresto.apache@gmail.com <al...@gmail.com>*
> PGP Fingerprint: 70EC B3E5 98A6 5A3F D3C4 BACE 3C6E F65B 2F7D EF69
>
> On Jul 8, 2016, at 7:42 PM, Russell Bateman <
> russell.bateman@perfectsearchcorp.com> wrote:
>
> (Make that "not much of a web page developer". I'm a back-end Java guy
> mostly, don't much value my UI skills and so have stayed away from the
> browser except for straight HTML and CSS.)
>
> On Fri, Jul 8, 2016 at 8:39 PM, Russell Bateman <
> russell.bateman@perfectsearchcorp.com> wrote:
>
> Thanks, Andy and Matt, I had actually tried out the Developer Tools
> earlier today as suggested by the doc, but, I'm not much of a web
> developer, don't have a lot of familiarity with those tools in Chrome or in
> Firefox, and I couldn't see anything going on (in Firefox) that was
> helpful. I may have to grow strong that way.
>
> For now, we're in 0.6.1. I'm hoping that soon after 1.0 makes its
> appearance, we can move up, but you point out something we're already
> beginning to pay attention to: incompatiblities. Still we're committed to
> (and love!) the technology. I've written a dozen custom processors (for
> proprietary ETL stuff) already and my colleague has done the same.
>
> Thanks, again.
>
>
> On Fri, Jul 8, 2016 at 5:11 PM, Andy LoPresto <al...@apache.org>
> wrote:
>
> Russell,
>
> To echo Matt’s point, the NiFi UI uses the REST API on every command, and
> developer tools is a great way to observe this.
>
> Another example using the REST API is Andrew Grande’s “nifi-deploy-api”
> project on GitHub [1]. It is a groovy script that can automate the
> deployment of NiFi instances and does this using the API. If you run it
> with the ‘--debug’ flag, you can see the HTTP endpoints it invokes to
> accomplish this.
>
> Finally, the REST API is extremely well documented here [2] (also locally
> in the installation). It is extensive, but should be able to answer a
> number of the questions you asked. For example, to get processor
> statistics, you would invoke:
>
> GET /controller/process-groups/{process-group-id}/status
>
> Gets the status for a process group
> The status for a process group includes status for all descendent
> components. When invoked on the root group with recursive set to true, it
> will return the current status of every component in the flow.
>
> When I invoke that endpoint on a simple cluster I have running with a
> single Base64EncodeContent processor named “Token Example” on the canvas,
> I get this response:
>
>
>
> https://ncm.nifi.apache.org:4567/nifi-api/controller/process-groups/root/status
>
> <processGroupStatusEntity>
> <revision>
> <clientId>392b161b-d9cc-442a-9394-8db9cee33ba7</clientId>
> </revision>
> <processGroupStatus>
> <activeThreadCount>0</activeThreadCount>
> <id>d17d7a95-6301-46ff-b003-11c58698bd31</id>
> <input>0 / 0 bytes</input>
> <name>NiFi Flow</name>
> <output>0 / 0 bytes</output>
> <processorStatus>
> <activeThreadCount>0</activeThreadCount>
> <groupId>d17d7a95-6301-46ff-b003-11c58698bd31</groupId>
> <id>b642a34d-c18c-4bb1-b836-6bbc9d32f31e</id>
> <input>0 / 0 bytes</input>
> <name>*Token Example*</name>
> <output>0 / 0 bytes</output>
> <read>0 bytes</read>
> <runStatus>Invalid</runStatus>
> <tasks>0</tasks>
> <tasksDuration>00:00:00.000</tasksDuration>
> <type>*Base64EncodeContent*</type>
> <written>0 bytes</written>
> </processorStatus>
> <queued>0 / 0 bytes</queued>
> <queuedCount>0</queuedCount>
> <queuedSize>0 bytes</queuedSize>
> <read>0 bytes</read>
> <received>0 / 0 bytes</received>
> <sent>0 / 0 bytes</sent>
> <statsLastRefreshed>16:09:46 PDT</statsLastRefreshed>
> <transferred>0 / 0 bytes</transferred>
> <written>0 bytes</written>
> </processGroupStatus>
> </processGroupStatusEntity>
>
> [1] https://github.com/aperepel/nifi-api-deploy
> [2] https://nifi.apache.org/docs/nifi-docs/rest-api/index.html
>
> Andy LoPresto
> alopresto@apache.org
> *alopresto.apache@gmail.com <al...@gmail.com>*
> PGP Fingerprint: 70EC B3E5 98A6 5A3F D3C4 BACE 3C6E F65B 2F7D EF69
>
> On Jul 8, 2016, at 3:18 PM, Matt Gilman <ma...@gmail.com> wrote:
>
> Russell,
>
> I can provide a more detailed response when I'm back in front of my
> computer but thought I'd offer this as a quick suggestion.
>
> All flows use process groups. The blank canvas when you load your nifi
> instance is the root level process group. If you don't know the actual ID,
> you can use the alias 'root' as the path parameter for the process group
> ID.
>
> Please check out your browser's Dev Tools to see these API's in action.
> The UI uses these API's exclusively.
>
> Also the API's have been completely refactored in the upcoming 1.0.0
> release to align with the authorization policies of each resource. So the
> answers to your questions will depend on which version your running.
>
> Matt
>
> Sent from my iPhone
>
> On Jul 8, 2016, at 5:35 PM, Russell Bateman <
> russell.bateman@perfectsearchcorp.com> wrote:
>
> I'm trying to figure out how to use the ReST (nifi-api) interface to
> accomplish a number of things. First, I've played successfully with it
> doing simple things like getting configuration, a list of existing
> processors, and the like. I've even discovered that in
> controller/history/processors/{processorId} I can use the processor name
> for {processorId} instead of an intangible or impossible number.
>
> What puzzles me is that I need a {process-group-id} to do so many things.
> I'm not using a process group, my flow is very simple for now. Maybe,
> think
> I, I have one process group and I just need its id, but how do I get that?
> I see no list API for it.
>
> The long list of things I'm hoping to do down the road from a UI is to be
> able to see things like:
>
> 1. How many times a processor processed anything?
> 2. How many flowfiles were processed?
> 3. How many flowfiles were produced?
> 4. Can I get this information per processor?
> 5. How can I tell different instances of the same processor apart?
> 6. Why can't I see the name I gave a processor in configuration? (For
> example, I named an instance of GetFile to "Get PDF files from test
> fodder".
> 7. How can I get a list of items on the NiFi canvas like processors and
> relationships.
> 8. How many times processors or a processor failed?
> 9. What is the profile of resource usage, like memory in use?
> 10. What is the profile of processor latency?
> - flowfiles backed up awaiting processing
> - flowfiles in other states (?)
>
> Any nudge on any of this would be very welcome. NiFi is new enough that
> there's precious few samples of people using the ReST interface.
>
> Thanks
>
>
>
>
>
>
Re: A few NiFi ReST questions...
Posted by Andy LoPresto <al...@apache.org>.
Russell,
No worries. The Javascript application that runs the UI is simply invoking Ajax (Asynchronous Javascript and XML) requests to normal HTTP endpoints provided by the embedded Jetty server in NiFi. You don’t need to understand anything about front-end development to monitor these requests using the Dev Tools. In fact, you could simply type the entire API endpoint into the browser address bar and receive the XML (or JSON-formatted, if you provide a header requesting it) response from the server directly. If you are more comfortable on the command-line or in an IDE, you can use Java, Groovy, etc. or even curl to invoke the API purely through code. There are also API tools (no endorsement, but some are listed here [1]) for hitting these APIs from a GUI application.
Here is a good starting point for using the Network tab of the developer tools window [2].
Glad to hear you have been able to write some custom processors. Let us know if you have any additional questions.
[1] https://stackoverflow.com/questions/4486658/what-tools-do-you-use-to-test-your-public-rest-api <https://stackoverflow.com/questions/4486658/what-tools-do-you-use-to-test-your-public-rest-api>
[2] http://commandlinefanatic.com/cgi-bin/showarticle.cgi?article=art034 <http://commandlinefanatic.com/cgi-bin/showarticle.cgi?article=art034>
Andy LoPresto
alopresto@apache.org
alopresto.apache@gmail.com
PGP Fingerprint: 70EC B3E5 98A6 5A3F D3C4 BACE 3C6E F65B 2F7D EF69
> On Jul 8, 2016, at 7:42 PM, Russell Bateman <ru...@perfectsearchcorp.com> wrote:
>
> (Make that "not much of a web page developer". I'm a back-end Java guy
> mostly, don't much value my UI skills and so have stayed away from the
> browser except for straight HTML and CSS.)
>
> On Fri, Jul 8, 2016 at 8:39 PM, Russell Bateman <
> russell.bateman@perfectsearchcorp.com> wrote:
>
>> Thanks, Andy and Matt, I had actually tried out the Developer Tools
>> earlier today as suggested by the doc, but, I'm not much of a web
>> developer, don't have a lot of familiarity with those tools in Chrome or in
>> Firefox, and I couldn't see anything going on (in Firefox) that was
>> helpful. I may have to grow strong that way.
>>
>> For now, we're in 0.6.1. I'm hoping that soon after 1.0 makes its
>> appearance, we can move up, but you point out something we're already
>> beginning to pay attention to: incompatiblities. Still we're committed to
>> (and love!) the technology. I've written a dozen custom processors (for
>> proprietary ETL stuff) already and my colleague has done the same.
>>
>> Thanks, again.
>>
>>
>> On Fri, Jul 8, 2016 at 5:11 PM, Andy LoPresto <al...@apache.org>
>> wrote:
>>
>>> Russell,
>>>
>>> To echo Matt’s point, the NiFi UI uses the REST API on every command, and
>>> developer tools is a great way to observe this.
>>>
>>> Another example using the REST API is Andrew Grande’s “nifi-deploy-api”
>>> project on GitHub [1]. It is a groovy script that can automate the
>>> deployment of NiFi instances and does this using the API. If you run it
>>> with the ‘--debug’ flag, you can see the HTTP endpoints it invokes to
>>> accomplish this.
>>>
>>> Finally, the REST API is extremely well documented here [2] (also locally
>>> in the installation). It is extensive, but should be able to answer a
>>> number of the questions you asked. For example, to get processor
>>> statistics, you would invoke:
>>>
>>> GET /controller/process-groups/{process-group-id}/status
>>>
>>> Gets the status for a process group
>>> The status for a process group includes status for all descendent
>>> components. When invoked on the root group with recursive set to true, it
>>> will return the current status of every component in the flow.
>>>
>>> When I invoke that endpoint on a simple cluster I have running with a
>>> single Base64EncodeContent processor named “Token Example” on the canvas,
>>> I get this response:
>>>
>>>
>>> https://ncm.nifi.apache.org:4567/nifi-api/controller/process-groups/root/status
>>>
>>> <processGroupStatusEntity>
>>> <revision>
>>> <clientId>392b161b-d9cc-442a-9394-8db9cee33ba7</clientId>
>>> </revision>
>>> <processGroupStatus>
>>> <activeThreadCount>0</activeThreadCount>
>>> <id>d17d7a95-6301-46ff-b003-11c58698bd31</id>
>>> <input>0 / 0 bytes</input>
>>> <name>NiFi Flow</name>
>>> <output>0 / 0 bytes</output>
>>> <processorStatus>
>>> <activeThreadCount>0</activeThreadCount>
>>> <groupId>d17d7a95-6301-46ff-b003-11c58698bd31</groupId>
>>> <id>b642a34d-c18c-4bb1-b836-6bbc9d32f31e</id>
>>> <input>0 / 0 bytes</input>
>>> <name>*Token Example*</name>
>>> <output>0 / 0 bytes</output>
>>> <read>0 bytes</read>
>>> <runStatus>Invalid</runStatus>
>>> <tasks>0</tasks>
>>> <tasksDuration>00:00:00.000</tasksDuration>
>>> <type>*Base64EncodeContent*</type>
>>> <written>0 bytes</written>
>>> </processorStatus>
>>> <queued>0 / 0 bytes</queued>
>>> <queuedCount>0</queuedCount>
>>> <queuedSize>0 bytes</queuedSize>
>>> <read>0 bytes</read>
>>> <received>0 / 0 bytes</received>
>>> <sent>0 / 0 bytes</sent>
>>> <statsLastRefreshed>16:09:46 PDT</statsLastRefreshed>
>>> <transferred>0 / 0 bytes</transferred>
>>> <written>0 bytes</written>
>>> </processGroupStatus>
>>> </processGroupStatusEntity>
>>>
>>> [1] https://github.com/aperepel/nifi-api-deploy
>>> [2] https://nifi.apache.org/docs/nifi-docs/rest-api/index.html
>>>
>>> Andy LoPresto
>>> alopresto@apache.org
>>> *alopresto.apache@gmail.com <al...@gmail.com>*
>>> PGP Fingerprint: 70EC B3E5 98A6 5A3F D3C4 BACE 3C6E F65B 2F7D EF69
>>>
>>> On Jul 8, 2016, at 3:18 PM, Matt Gilman <ma...@gmail.com> wrote:
>>>
>>> Russell,
>>>
>>> I can provide a more detailed response when I'm back in front of my
>>> computer but thought I'd offer this as a quick suggestion.
>>>
>>> All flows use process groups. The blank canvas when you load your nifi
>>> instance is the root level process group. If you don't know the actual ID,
>>> you can use the alias 'root' as the path parameter for the process group ID.
>>>
>>> Please check out your browser's Dev Tools to see these API's in action.
>>> The UI uses these API's exclusively.
>>>
>>> Also the API's have been completely refactored in the upcoming 1.0.0
>>> release to align with the authorization policies of each resource. So the
>>> answers to your questions will depend on which version your running.
>>>
>>> Matt
>>>
>>> Sent from my iPhone
>>>
>>> On Jul 8, 2016, at 5:35 PM, Russell Bateman <
>>> russell.bateman@perfectsearchcorp.com> wrote:
>>>
>>> I'm trying to figure out how to use the ReST (nifi-api) interface to
>>> accomplish a number of things. First, I've played successfully with it
>>> doing simple things like getting configuration, a list of existing
>>> processors, and the like. I've even discovered that in
>>> controller/history/processors/{processorId} I can use the processor name
>>> for {processorId} instead of an intangible or impossible number.
>>>
>>> What puzzles me is that I need a {process-group-id} to do so many things.
>>> I'm not using a process group, my flow is very simple for now. Maybe,
>>> think
>>> I, I have one process group and I just need its id, but how do I get that?
>>> I see no list API for it.
>>>
>>> The long list of things I'm hoping to do down the road from a UI is to be
>>> able to see things like:
>>>
>>> 1. How many times a processor processed anything?
>>> 2. How many flowfiles were processed?
>>> 3. How many flowfiles were produced?
>>> 4. Can I get this information per processor?
>>> 5. How can I tell different instances of the same processor apart?
>>> 6. Why can't I see the name I gave a processor in configuration? (For
>>> example, I named an instance of GetFile to "Get PDF files from test
>>> fodder".
>>> 7. How can I get a list of items on the NiFi canvas like processors and
>>> relationships.
>>> 8. How many times processors or a processor failed?
>>> 9. What is the profile of resource usage, like memory in use?
>>> 10. What is the profile of processor latency?
>>> - flowfiles backed up awaiting processing
>>> - flowfiles in other states (?)
>>>
>>> Any nudge on any of this would be very welcome. NiFi is new enough that
>>> there's precious few samples of people using the ReST interface.
>>>
>>> Thanks
>>>
>>>
>>>
>>
Re: A few NiFi ReST questions...
Posted by Russell Bateman <ru...@perfectsearchcorp.com>.
(Make that "not much of a web page developer". I'm a back-end Java guy
mostly, don't much value my UI skills and so have stayed away from the
browser except for straight HTML and CSS.)
On Fri, Jul 8, 2016 at 8:39 PM, Russell Bateman <
russell.bateman@perfectsearchcorp.com> wrote:
> Thanks, Andy and Matt, I had actually tried out the Developer Tools
> earlier today as suggested by the doc, but, I'm not much of a web
> developer, don't have a lot of familiarity with those tools in Chrome or in
> Firefox, and I couldn't see anything going on (in Firefox) that was
> helpful. I may have to grow strong that way.
>
> For now, we're in 0.6.1. I'm hoping that soon after 1.0 makes its
> appearance, we can move up, but you point out something we're already
> beginning to pay attention to: incompatiblities. Still we're committed to
> (and love!) the technology. I've written a dozen custom processors (for
> proprietary ETL stuff) already and my colleague has done the same.
>
> Thanks, again.
>
>
> On Fri, Jul 8, 2016 at 5:11 PM, Andy LoPresto <al...@apache.org>
> wrote:
>
>> Russell,
>>
>> To echo Matt’s point, the NiFi UI uses the REST API on every command, and
>> developer tools is a great way to observe this.
>>
>> Another example using the REST API is Andrew Grande’s “nifi-deploy-api”
>> project on GitHub [1]. It is a groovy script that can automate the
>> deployment of NiFi instances and does this using the API. If you run it
>> with the ‘--debug’ flag, you can see the HTTP endpoints it invokes to
>> accomplish this.
>>
>> Finally, the REST API is extremely well documented here [2] (also locally
>> in the installation). It is extensive, but should be able to answer a
>> number of the questions you asked. For example, to get processor
>> statistics, you would invoke:
>>
>> GET /controller/process-groups/{process-group-id}/status
>>
>> Gets the status for a process group
>> The status for a process group includes status for all descendent
>> components. When invoked on the root group with recursive set to true, it
>> will return the current status of every component in the flow.
>>
>> When I invoke that endpoint on a simple cluster I have running with a
>> single Base64EncodeContent processor named “Token Example” on the canvas,
>> I get this response:
>>
>>
>> https://ncm.nifi.apache.org:4567/nifi-api/controller/process-groups/root/status
>>
>> <processGroupStatusEntity>
>> <revision>
>> <clientId>392b161b-d9cc-442a-9394-8db9cee33ba7</clientId>
>> </revision>
>> <processGroupStatus>
>> <activeThreadCount>0</activeThreadCount>
>> <id>d17d7a95-6301-46ff-b003-11c58698bd31</id>
>> <input>0 / 0 bytes</input>
>> <name>NiFi Flow</name>
>> <output>0 / 0 bytes</output>
>> <processorStatus>
>> <activeThreadCount>0</activeThreadCount>
>> <groupId>d17d7a95-6301-46ff-b003-11c58698bd31</groupId>
>> <id>b642a34d-c18c-4bb1-b836-6bbc9d32f31e</id>
>> <input>0 / 0 bytes</input>
>> <name>*Token Example*</name>
>> <output>0 / 0 bytes</output>
>> <read>0 bytes</read>
>> <runStatus>Invalid</runStatus>
>> <tasks>0</tasks>
>> <tasksDuration>00:00:00.000</tasksDuration>
>> <type>*Base64EncodeContent*</type>
>> <written>0 bytes</written>
>> </processorStatus>
>> <queued>0 / 0 bytes</queued>
>> <queuedCount>0</queuedCount>
>> <queuedSize>0 bytes</queuedSize>
>> <read>0 bytes</read>
>> <received>0 / 0 bytes</received>
>> <sent>0 / 0 bytes</sent>
>> <statsLastRefreshed>16:09:46 PDT</statsLastRefreshed>
>> <transferred>0 / 0 bytes</transferred>
>> <written>0 bytes</written>
>> </processGroupStatus>
>> </processGroupStatusEntity>
>>
>> [1] https://github.com/aperepel/nifi-api-deploy
>> [2] https://nifi.apache.org/docs/nifi-docs/rest-api/index.html
>>
>> Andy LoPresto
>> alopresto@apache.org
>> *alopresto.apache@gmail.com <al...@gmail.com>*
>> PGP Fingerprint: 70EC B3E5 98A6 5A3F D3C4 BACE 3C6E F65B 2F7D EF69
>>
>> On Jul 8, 2016, at 3:18 PM, Matt Gilman <ma...@gmail.com> wrote:
>>
>> Russell,
>>
>> I can provide a more detailed response when I'm back in front of my
>> computer but thought I'd offer this as a quick suggestion.
>>
>> All flows use process groups. The blank canvas when you load your nifi
>> instance is the root level process group. If you don't know the actual ID,
>> you can use the alias 'root' as the path parameter for the process group ID.
>>
>> Please check out your browser's Dev Tools to see these API's in action.
>> The UI uses these API's exclusively.
>>
>> Also the API's have been completely refactored in the upcoming 1.0.0
>> release to align with the authorization policies of each resource. So the
>> answers to your questions will depend on which version your running.
>>
>> Matt
>>
>> Sent from my iPhone
>>
>> On Jul 8, 2016, at 5:35 PM, Russell Bateman <
>> russell.bateman@perfectsearchcorp.com> wrote:
>>
>> I'm trying to figure out how to use the ReST (nifi-api) interface to
>> accomplish a number of things. First, I've played successfully with it
>> doing simple things like getting configuration, a list of existing
>> processors, and the like. I've even discovered that in
>> controller/history/processors/{processorId} I can use the processor name
>> for {processorId} instead of an intangible or impossible number.
>>
>> What puzzles me is that I need a {process-group-id} to do so many things.
>> I'm not using a process group, my flow is very simple for now. Maybe,
>> think
>> I, I have one process group and I just need its id, but how do I get that?
>> I see no list API for it.
>>
>> The long list of things I'm hoping to do down the road from a UI is to be
>> able to see things like:
>>
>> 1. How many times a processor processed anything?
>> 2. How many flowfiles were processed?
>> 3. How many flowfiles were produced?
>> 4. Can I get this information per processor?
>> 5. How can I tell different instances of the same processor apart?
>> 6. Why can't I see the name I gave a processor in configuration? (For
>> example, I named an instance of GetFile to "Get PDF files from test
>> fodder".
>> 7. How can I get a list of items on the NiFi canvas like processors and
>> relationships.
>> 8. How many times processors or a processor failed?
>> 9. What is the profile of resource usage, like memory in use?
>> 10. What is the profile of processor latency?
>> - flowfiles backed up awaiting processing
>> - flowfiles in other states (?)
>>
>> Any nudge on any of this would be very welcome. NiFi is new enough that
>> there's precious few samples of people using the ReST interface.
>>
>> Thanks
>>
>>
>>
>
Re: A few NiFi ReST questions...
Posted by Russell Bateman <ru...@perfectsearchcorp.com>.
Thanks, Andy and Matt, I had actually tried out the Developer Tools earlier
today as suggested by the doc,but, I'm not much of a web developer, don't
have a lot of familiarity with those tools in Chrome or in Firefox, and I
couldn't see anything going on (in Firefox) that was helpful. I may have to
grow strong that way.
For now, we're in 0.6.1. I'm hoping that soon after 1.0 makes its
appearance, we can move up, but you point out something we're already
beginning to pay attention to: incompatiblities. Still we're committed to
(and love!) the technology. I've written a dozen custom processors (for
proprietary ETL stuff) already and my colleague has done the same.
Thanks, again.
On Fri, Jul 8, 2016 at 5:11 PM, Andy LoPresto <al...@apache.org> wrote:
> Russell,
>
> To echo Matt’s point, the NiFi UI uses the REST API on every command, and
> developer tools is a great way to observe this.
>
> Another example using the REST API is Andrew Grande’s “nifi-deploy-api”
> project on GitHub [1]. It is a groovy script that can automate the
> deployment of NiFi instances and does this using the API. If you run it
> with the ‘--debug’ flag, you can see the HTTP endpoints it invokes to
> accomplish this.
>
> Finally, the REST API is extremely well documented here [2] (also locally
> in the installation). It is extensive, but should be able to answer a
> number of the questions you asked. For example, to get processor
> statistics, you would invoke:
>
> GET /controller/process-groups/{process-group-id}/status
>
> Gets the status for a process group
> The status for a process group includes status for all descendent
> components. When invoked on the root group with recursive set to true, it
> will return the current status of every component in the flow.
>
> When I invoke that endpoint on a simple cluster I have running with a
> single Base64EncodeContent processor named “Token Example” on the canvas,
> I get this response:
>
>
> https://ncm.nifi.apache.org:4567/nifi-api/controller/process-groups/root/status
>
> <processGroupStatusEntity>
> <revision>
> <clientId>392b161b-d9cc-442a-9394-8db9cee33ba7</clientId>
> </revision>
> <processGroupStatus>
> <activeThreadCount>0</activeThreadCount>
> <id>d17d7a95-6301-46ff-b003-11c58698bd31</id>
> <input>0 / 0 bytes</input>
> <name>NiFi Flow</name>
> <output>0 / 0 bytes</output>
> <processorStatus>
> <activeThreadCount>0</activeThreadCount>
> <groupId>d17d7a95-6301-46ff-b003-11c58698bd31</groupId>
> <id>b642a34d-c18c-4bb1-b836-6bbc9d32f31e</id>
> <input>0 / 0 bytes</input>
> <name>*Token Example*</name>
> <output>0 / 0 bytes</output>
> <read>0 bytes</read>
> <runStatus>Invalid</runStatus>
> <tasks>0</tasks>
> <tasksDuration>00:00:00.000</tasksDuration>
> <type>*Base64EncodeContent*</type>
> <written>0 bytes</written>
> </processorStatus>
> <queued>0 / 0 bytes</queued>
> <queuedCount>0</queuedCount>
> <queuedSize>0 bytes</queuedSize>
> <read>0 bytes</read>
> <received>0 / 0 bytes</received>
> <sent>0 / 0 bytes</sent>
> <statsLastRefreshed>16:09:46 PDT</statsLastRefreshed>
> <transferred>0 / 0 bytes</transferred>
> <written>0 bytes</written>
> </processGroupStatus>
> </processGroupStatusEntity>
>
> [1] https://github.com/aperepel/nifi-api-deploy
> [2] https://nifi.apache.org/docs/nifi-docs/rest-api/index.html
>
> Andy LoPresto
> alopresto@apache.org
> *alopresto.apache@gmail.com <al...@gmail.com>*
> PGP Fingerprint: 70EC B3E5 98A6 5A3F D3C4 BACE 3C6E F65B 2F7D EF69
>
> On Jul 8, 2016, at 3:18 PM, Matt Gilman <ma...@gmail.com> wrote:
>
> Russell,
>
> I can provide a more detailed response when I'm back in front of my
> computer but thought I'd offer this as a quick suggestion.
>
> All flows use process groups. The blank canvas when you load your nifi
> instance is the root level process group. If you don't know the actual ID,
> you can use the alias 'root' as the path parameter for the process group ID.
>
> Please check out your browser's Dev Tools to see these API's in action.
> The UI uses these API's exclusively.
>
> Also the API's have been completely refactored in the upcoming 1.0.0
> release to align with the authorization policies of each resource. So the
> answers to your questions will depend on which version your running.
>
> Matt
>
> Sent from my iPhone
>
> On Jul 8, 2016, at 5:35 PM, Russell Bateman <
> russell.bateman@perfectsearchcorp.com> wrote:
>
> I'm trying to figure out how to use the ReST (nifi-api) interface to
> accomplish a number of things. First, I've played successfully with it
> doing simple things like getting configuration, a list of existing
> processors, and the like. I've even discovered that in
> controller/history/processors/{processorId} I can use the processor name
> for {processorId} instead of an intangible or impossible number.
>
> What puzzles me is that I need a {process-group-id} to do so many things.
> I'm not using a process group, my flow is very simple for now. Maybe, think
> I, I have one process group and I just need its id, but how do I get that?
> I see no list API for it.
>
> The long list of things I'm hoping to do down the road from a UI is to be
> able to see things like:
>
> 1. How many times a processor processed anything?
> 2. How many flowfiles were processed?
> 3. How many flowfiles were produced?
> 4. Can I get this information per processor?
> 5. How can I tell different instances of the same processor apart?
> 6. Why can't I see the name I gave a processor in configuration? (For
> example, I named an instance of GetFile to "Get PDF files from test
> fodder".
> 7. How can I get a list of items on the NiFi canvas like processors and
> relationships.
> 8. How many times processors or a processor failed?
> 9. What is the profile of resource usage, like memory in use?
> 10. What is the profile of processor latency?
> - flowfiles backed up awaiting processing
> - flowfiles in other states (?)
>
> Any nudge on any of this would be very welcome. NiFi is new enough that
> there's precious few samples of people using the ReST interface.
>
> Thanks
>
>
>
Re: A few NiFi ReST questions...
Posted by Andy LoPresto <al...@apache.org>.
Russell,
To echo Matt’s point, the NiFi UI uses the REST API on every command, and developer tools is a great way to observe this.
Another example using the REST API is Andrew Grande’s “nifi-deploy-api” project on GitHub [1]. It is a groovy script that can automate the deployment of NiFi instances and does this using the API. If you run it with the ‘--debug’ flag, you can see the HTTP endpoints it invokes to accomplish this.
Finally, the REST API is extremely well documented here [2] (also locally in the installation). It is extensive, but should be able to answer a number of the questions you asked. For example, to get processor statistics, you would invoke:
GET /controller/process-groups/{process-group-id}/status
Gets the status for a process group
The status for a process group includes status for all descendent components. When invoked on the root group with recursive set to true, it will return the current status of every component in the flow.
When I invoke that endpoint on a simple cluster I have running with a single Base64EncodeContent processor named “Token Example” on the canvas, I get this response:
https://ncm.nifi.apache.org:4567/nifi-api/controller/process-groups/root/status
<processGroupStatusEntity>
<revision>
<clientId>392b161b-d9cc-442a-9394-8db9cee33ba7</clientId>
</revision>
<processGroupStatus>
<activeThreadCount>0</activeThreadCount>
<id>d17d7a95-6301-46ff-b003-11c58698bd31</id>
<input>0 / 0 bytes</input>
<name>NiFi Flow</name>
<output>0 / 0 bytes</output>
<processorStatus>
<activeThreadCount>0</activeThreadCount>
<groupId>d17d7a95-6301-46ff-b003-11c58698bd31</groupId>
<id>b642a34d-c18c-4bb1-b836-6bbc9d32f31e</id>
<input>0 / 0 bytes</input>
<name>Token Example</name>
<output>0 / 0 bytes</output>
<read>0 bytes</read>
<runStatus>Invalid</runStatus>
<tasks>0</tasks>
<tasksDuration>00:00:00.000</tasksDuration>
<type>Base64EncodeContent</type>
<written>0 bytes</written>
</processorStatus>
<queued>0 / 0 bytes</queued>
<queuedCount>0</queuedCount>
<queuedSize>0 bytes</queuedSize>
<read>0 bytes</read>
<received>0 / 0 bytes</received>
<sent>0 / 0 bytes</sent>
<statsLastRefreshed>16:09:46 PDT</statsLastRefreshed>
<transferred>0 / 0 bytes</transferred>
<written>0 bytes</written>
</processGroupStatus>
</processGroupStatusEntity>
[1] https://github.com/aperepel/nifi-api-deploy <https://github.com/aperepel/nifi-api-deploy>
[2] https://nifi.apache.org/docs/nifi-docs/rest-api/index.html
Andy LoPresto
alopresto@apache.org
alopresto.apache@gmail.com
PGP Fingerprint: 70EC B3E5 98A6 5A3F D3C4 BACE 3C6E F65B 2F7D EF69
> On Jul 8, 2016, at 3:18 PM, Matt Gilman <ma...@gmail.com> wrote:
>
> Russell,
>
> I can provide a more detailed response when I'm back in front of my computer but thought I'd offer this as a quick suggestion.
>
> All flows use process groups. The blank canvas when you load your nifi instance is the root level process group. If you don't know the actual ID, you can use the alias 'root' as the path parameter for the process group ID.
>
> Please check out your browser's Dev Tools to see these API's in action. The UI uses these API's exclusively.
>
> Also the API's have been completely refactored in the upcoming 1.0.0 release to align with the authorization policies of each resource. So the answers to your questions will depend on which version your running.
>
> Matt
>
> Sent from my iPhone
>
>> On Jul 8, 2016, at 5:35 PM, Russell Bateman <ru...@perfectsearchcorp.com> wrote:
>>
>> I'm trying to figure out how to use the ReST (nifi-api) interface to
>> accomplish a number of things. First, I've played successfully with it
>> doing simple things like getting configuration, a list of existing
>> processors, and the like. I've even discovered that in
>> controller/history/processors/{processorId} I can use the processor name
>> for {processorId} instead of an intangible or impossible number.
>>
>> What puzzles me is that I need a {process-group-id} to do so many things.
>> I'm not using a process group, my flow is very simple for now. Maybe, think
>> I, I have one process group and I just need its id, but how do I get that?
>> I see no list API for it.
>>
>> The long list of things I'm hoping to do down the road from a UI is to be
>> able to see things like:
>>
>> 1. How many times a processor processed anything?
>> 2. How many flowfiles were processed?
>> 3. How many flowfiles were produced?
>> 4. Can I get this information per processor?
>> 5. How can I tell different instances of the same processor apart?
>> 6. Why can't I see the name I gave a processor in configuration? (For
>> example, I named an instance of GetFile to "Get PDF files from test fodder".
>> 7. How can I get a list of items on the NiFi canvas like processors and
>> relationships.
>> 8. How many times processors or a processor failed?
>> 9. What is the profile of resource usage, like memory in use?
>> 10. What is the profile of processor latency?
>> - flowfiles backed up awaiting processing
>> - flowfiles in other states (?)
>>
>> Any nudge on any of this would be very welcome. NiFi is new enough that
>> there's precious few samples of people using the ReST interface.
>>
>> Thanks
Re: A few NiFi ReST questions...
Posted by Matt Gilman <ma...@gmail.com>.
Russell,
I can provide a more detailed response when I'm back in front of my computer but thought I'd offer this as a quick suggestion.
All flows use process groups. The blank canvas when you load your nifi instance is the root level process group. If you don't know the actual ID, you can use the alias 'root' as the path parameter for the process group ID.
Please check out your browser's Dev Tools to see these API's in action. The UI uses these API's exclusively.
Also the API's have been completely refactored in the upcoming 1.0.0 release to align with the authorization policies of each resource. So the answers to your questions will depend on which version your running.
Matt
Sent from my iPhone
> On Jul 8, 2016, at 5:35 PM, Russell Bateman <ru...@perfectsearchcorp.com> wrote:
>
> I'm trying to figure out how to use the ReST (nifi-api) interface to
> accomplish a number of things. First, I've played successfully with it
> doing simple things like getting configuration, a list of existing
> processors, and the like. I've even discovered that in
> controller/history/processors/{processorId} I can use the processor name
> for {processorId} instead of an intangible or impossible number.
>
> What puzzles me is that I need a {process-group-id} to do so many things.
> I'm not using a process group, my flow is very simple for now. Maybe, think
> I, I have one process group and I just need its id, but how do I get that?
> I see no list API for it.
>
> The long list of things I'm hoping to do down the road from a UI is to be
> able to see things like:
>
> 1. How many times a processor processed anything?
> 2. How many flowfiles were processed?
> 3. How many flowfiles were produced?
> 4. Can I get this information per processor?
> 5. How can I tell different instances of the same processor apart?
> 6. Why can't I see the name I gave a processor in configuration? (For
> example, I named an instance of GetFile to "Get PDF files from test fodder".
> 7. How can I get a list of items on the NiFi canvas like processors and
> relationships.
> 8. How many times processors or a processor failed?
> 9. What is the profile of resource usage, like memory in use?
> 10. What is the profile of processor latency?
> - flowfiles backed up awaiting processing
> - flowfiles in other states (?)
>
> Any nudge on any of this would be very welcome. NiFi is new enough that
> there's precious few samples of people using the ReST interface.
>
> Thanks
Re: A few NiFi ReST questions...
Posted by Russell Bateman <ru...@perfectsearchcorp.com>.
Ah, for (9), I used system-diagnostics. (I had not spelled it correctly
when trying it before.)
On Fri, Jul 8, 2016 at 3:35 PM, Russell Bateman <
russell.bateman@perfectsearchcorp.com> wrote:
> I'm trying to figure out how to use the ReST (nifi-api) interface to
> accomplish a number of things. First, I've played successfully with it
> doing simple things like getting configuration, a list of existing
> processors, and the like. I've even discovered that in
> controller/history/processors/{processorId} I can use the processor name
> for {processorId} instead of an intangible or impossible number.
>
> What puzzles me is that I need a {process-group-id} to do so many things.
> I'm not using a process group, my flow is very simple for now. Maybe, think
> I, I have one process group and I just need its id, but how do I get that?
> I see no list API for it.
>
> The long list of things I'm hoping to do down the road from a UI is to be
> able to see things like:
>
> 1. How many times a processor processed anything?
> 2. How many flowfiles were processed?
> 3. How many flowfiles were produced?
> 4. Can I get this information per processor?
> 5. How can I tell different instances of the same processor apart?
> 6. Why can't I see the name I gave a processor in configuration? (For
> example, I named an instance of GetFile to "Get PDF files from test fodder".
> 7. How can I get a list of items on the NiFi canvas like processors
> and relationships.
> 8. How many times processors or a processor failed?
> 9. What is the profile of resource usage, like memory in use?
> 10. What is the profile of processor latency?
> - flowfiles backed up awaiting processing
> - flowfiles in other states (?)
>
> Any nudge on any of this would be very welcome. NiFi is new enough that
> there's precious few samples of people using the ReST interface.
>
> Thanks
>