Questions arising from use of ExecuteProcess, batching, stopping tasks and user-defined menus

[Rick Braddy <rb...@softnas.com>]

Hi,

I have been doing some testing with ExecuteProcess processor to learn more about how Nifi processors work and perform I/O.  This has led to a couple of questions.


1)      Nifi API docs - are there API docs planned for what appear to be "internal" API's that are commonly used by processors?  (e.g., session, context, process(), run(), ... )?  The Developer Guide provides a lot of useful "conceptual" level information, but I haven't yet found an API Reference Guide


2)      Batching - in ExecuteProcess I see where the external process is launched, but it's unclear how "batching" occurs, whereby a series of FlowFiles is created.  On the surface, it appears that after the process is launched, there's a small wait that occurs based upon the specified batch time (e.g., "2 millis"), and that after that a single FlowFile is created.  My testing confirms that multiple FlowFiles do, indeed, get created when batching is enabled.  Perhaps if the API's were documented it would be easier to understand (or I could spend more time in the debugger looking at stack traces, I suppose)


3)      External process and Stop command - with a long-running "batched" process, the external process runs and does not end for a very long time (by design). When I "stop" the processor, I was expecting the external process to be killed (also stop).  Is this a bug or is ExecuteProcess not intended to also stop the spawned processes?  (I found that I had to shut the entire Nifi service down to kill the spawned process - not good if an external process were to get hung up)


4)      How does one add a new menu command to a processor?  For example, if I wanted to add a "Cancel task" menu item to ExecuteProcess processor, how would that be accomplished?  I'm hoping there's any easy way to associate a "menu property" and callback to a processor (maybe that's an enhancement request)

Thanks!
Rick

RE: Questions arising from use of ExecuteProcess, batching, stopping tasks and user-defined menus

[Rick Braddy <rb...@softnas.com>]

Matt,

Thanks for the detailed response. I will look into the JavaDocs (agree it would be great to have a link on the website for convenience and Googling).

I suppose the user-defined menus would also likely "grow hair" very quickly, as the menus must activate some kind of form or other means of user interaction, beyond something as simple as "cancel task", which makes sense as a built-in feature anyway, so hung or long-running processes can be interrupted whenever needed, without shutting the entire framework down (not possible without a downtime window for many production systems.

Thanks again for listening and incorporating as tickets.

Best,
Rick

-----Original Message-----
From: Matt Gilman [mailto:matt.c.gilman@gmail.com] 
Sent: Wednesday, September 09, 2015 3:52 PM
To: dev@nifi.apache.org
Subject: Re: Questions arising from use of ExecuteProcess, batching, stopping tasks and user-defined menus

Rick,

1) Any classes that interface with an extension point (like a processor) would be considered a public API and does have JavaDocs. The JavaDocs are usually deployed when a release is performed. However, it appears those JavaDocs were not successfully deployed when 0.2.1 was released. You should be able to generate locally or attach sources so your IDE can render the documentation in the content assist. Also, we should add those JavaDocs to the website [1].

2) It appears that this processor handles it's own batching internally via a BATCH_DURATION property. It allows you to basically break the output of the process into chunks and treat each as it's own FlowFile.

3) When a processor is stopped, it is simply not scheduled anymore. This allows the processor to complete its current execution. However, when a processor has long running tasks it may require the user to wait prior to being able to make changes to that processor. We've created a ticket to support interrupting a processor [2].

4) There is currently no way to create new menu items. However, it sounds like the interrupt option which was going to be available to all processors is what your looking for. Additionally, we've considered supporting a 'reporting UI' that would allow a processor to rendering specific details about it's processing (like showing how full the 'bins' are in MergeContent).

Matt

[1] https://issues.apache.org/jira/browse/NIFI-943
[2] https://issues.apache.org/jira/browse/NIFI-78

On Sun, Sep 6, 2015 at 6:59 PM, Rick Braddy <rb...@softnas.com> wrote:

> Hi,
>
> I have been doing some testing with ExecuteProcess processor to learn 
> more about how Nifi processors work and perform I/O.  This has led to 
> a couple of questions.
>
>
> 1)      Nifi API docs - are there API docs planned for what appear to be
> "internal" API's that are commonly used by processors?  (e.g., 
> session, context, process(), run(), ... )?  The Developer Guide 
> provides a lot of useful "conceptual" level information, but I haven't 
> yet found an API Reference Guide
>
>
> 2)      Batching - in ExecuteProcess I see where the external process is
> launched, but it's unclear how "batching" occurs, whereby a series of 
> FlowFiles is created.  On the surface, it appears that after the 
> process is launched, there's a small wait that occurs based upon the 
> specified batch time (e.g., "2 millis"), and that after that a single FlowFile is created.
> My testing confirms that multiple FlowFiles do, indeed, get created 
> when batching is enabled.  Perhaps if the API's were documented it 
> would be easier to understand (or I could spend more time in the 
> debugger looking at stack traces, I suppose)
>
>
> 3)      External process and Stop command - with a long-running "batched"
> process, the external process runs and does not end for a very long 
> time (by design). When I "stop" the processor, I was expecting the 
> external process to be killed (also stop).  Is this a bug or is 
> ExecuteProcess not intended to also stop the spawned processes?  (I 
> found that I had to shut the entire Nifi service down to kill the 
> spawned process - not good if an external process were to get hung up)
>
>
> 4)      How does one add a new menu command to a processor?  For example,
> if I wanted to add a "Cancel task" menu item to ExecuteProcess 
> processor, how would that be accomplished?  I'm hoping there's any 
> easy way to associate a "menu property" and callback to a processor 
> (maybe that's an enhancement request)
>
> Thanks!
> Rick
>

Re: Questions arising from use of ExecuteProcess, batching, stopping tasks and user-defined menus

[Matt Gilman <ma...@gmail.com>]

Rick,

1) Any classes that interface with an extension point (like a processor)
would be considered a public API and does have JavaDocs. The JavaDocs are
usually deployed when a release is performed. However, it appears those
JavaDocs were not successfully deployed when 0.2.1 was released. You should
be able to generate locally or attach sources so your IDE can render the
documentation in the content assist. Also, we should add those JavaDocs to
the website [1].

2) It appears that this processor handles it's own batching internally via
a BATCH_DURATION property. It allows you to basically break the output of
the process into chunks and treat each as it's own FlowFile.

3) When a processor is stopped, it is simply not scheduled anymore. This
allows the processor to complete its current execution. However, when a
processor has long running tasks it may require the user to wait prior to
being able to make changes to that processor. We've created a ticket to
support interrupting a processor [2].

4) There is currently no way to create new menu items. However, it sounds
like the interrupt option which was going to be available to all processors
is what your looking for. Additionally, we've considered supporting a
'reporting UI' that would allow a processor to rendering specific details
about it's processing (like showing how full the 'bins' are in
MergeContent).

Matt

[1] https://issues.apache.org/jira/browse/NIFI-943
[2] https://issues.apache.org/jira/browse/NIFI-78

On Sun, Sep 6, 2015 at 6:59 PM, Rick Braddy <rb...@softnas.com> wrote:

> Hi,
>
> I have been doing some testing with ExecuteProcess processor to learn more
> about how Nifi processors work and perform I/O.  This has led to a couple
> of questions.
>
>
> 1)      Nifi API docs - are there API docs planned for what appear to be
> "internal" API's that are commonly used by processors?  (e.g., session,
> context, process(), run(), ... )?  The Developer Guide provides a lot of
> useful "conceptual" level information, but I haven't yet found an API
> Reference Guide
>
>
> 2)      Batching - in ExecuteProcess I see where the external process is
> launched, but it's unclear how "batching" occurs, whereby a series of
> FlowFiles is created.  On the surface, it appears that after the process is
> launched, there's a small wait that occurs based upon the specified batch
> time (e.g., "2 millis"), and that after that a single FlowFile is created.
> My testing confirms that multiple FlowFiles do, indeed, get created when
> batching is enabled.  Perhaps if the API's were documented it would be
> easier to understand (or I could spend more time in the debugger looking at
> stack traces, I suppose)
>
>
> 3)      External process and Stop command - with a long-running "batched"
> process, the external process runs and does not end for a very long time
> (by design). When I "stop" the processor, I was expecting the external
> process to be killed (also stop).  Is this a bug or is ExecuteProcess not
> intended to also stop the spawned processes?  (I found that I had to shut
> the entire Nifi service down to kill the spawned process - not good if an
> external process were to get hung up)
>
>
> 4)      How does one add a new menu command to a processor?  For example,
> if I wanted to add a "Cancel task" menu item to ExecuteProcess processor,
> how would that be accomplished?  I'm hoping there's any easy way to
> associate a "menu property" and callback to a processor (maybe that's an
> enhancement request)
>
> Thanks!
> Rick
>