Re: Service to handle workflow start/stop

[Joe Witt <jo...@gmail.com>]

Alessio

I might not quite follow all of the question but I'll try to respond
to what I believe you're asking.

First, it is important to note that whether a human is looking at the
UI or a system/script/process is interacting with the REST API of NiFi
in the end the effect is the same.  It is all happening through the
REST API.

Presently we offer the following scheduling styles available within
the framework
- timer based
- cron based
- event based

Whether a processor/component is given a thread to use at any given
time depends on whether one of those above triggers fire AND whether
or not back pressure prevents it from getting a thread or whether it
depends on there being work (data) in an inbound queue first or
whether that processor/component is presently in a yield/hold pattern
for some period of time.  That is all predicated on whether some thing
(user through UI or some service/process invoking the REST API) has
told the nifi framework that a given component may be scheduled to
execute.

So how best to inject your desire to have an external system dictate
whether a component or set of components (your notion of a workflow)
depends on how you feel the above best reflects your case.  You can
control whether processors/components are scheduled to run (not
whether they actually get triggered) by using our REST API
exclusively.  Or, you might feel that you want to add a new triggering
style such as 'external service'.  That would require framework level
code changes in an area that is not really designed for extension most
likely.  So that might be a lot harder.  So, in that sense I'd
probably recommend more evaluation and research going into leveraging
that first approach I mention to see if that meets your needs.

Thanks
Joe

On Mon, Jan 23, 2017 at 10:59 AM, Alessio Palma
<al...@docomodigital.com> wrote:
> Hello all,
>
> before starting diving into the code I have a little question:
>
> It's possible to write a service connected to processor which can be used to start workflows ?
> Scenario is this:
> Currently we can start workflows using a schedule strategy, I like to move this logic into a service which broadcast to registered processors the names of the workflows which are supposed to start in the current time frame.
> So the basic idea is to have these custom processor which keep polling the service asking the permission to start.
>
> Is there a better option which does not requires polling?
>
> If I read it carefully the  REST API can do the work, but it cuts all the advantages of the graphical environment. So this is not an option.
>
>
> AP

Re: Service to handle workflow start/stop

[Alessio Palma <al...@docomodigital.com>]

Hello Witt,


you give me good and bad news. U wrote:


"Presently we offer the following scheduling styles available within
the framework
- timer based
- cron based
- event based ..."


event based sounds good and googling it I found a doc, locate to https://cwiki.apache.org/confluence/display/NIFI/Event+Processor  which says:


- - - - -

Goals

  *   Provide a mechanism for Processor developers to create Processors that are event-driven in such a way that the framework is able to more efficiently trigger the Processor to run


... There is a mechanism for scheduling processors to run via an Event-Driven schedule. However, this scheduling strategy has been 'experimental' since it was developed"

- - - - - -

All this looks like there is already a solution in NIFI, which perhaps is not complete but is still able to get async events and forward them to listeners. Is it correct ?
Is there an available and undocumented REST API which allow us to send events to NIFI framework and  in turn forwards them it to all processors which are listening?
Is there a way to allow processors to subscribe for specific event types?
Where can I find more info and code example about this?

Since something about event handling should be available, I also wonder if there is a simple way to have processors started by a service, which is a better solution to have processors polling the service for the authorization to start, this was the quick and dirty solution that I was thinking to develop. In my scenario, a custom processor using a timer based scheduling strategy  ask a  custom service the authorization to start; of course, if the polling is removed and the service can send a fake/empty flowfile to start the processor, we gain efficiency.

We need to have a centralized point from which start/stop/track/resume all the workflows because our data lake needs tons of workflows.


Alternative solutions to a centralized management are welcomed.

________________________________
From: Joe Witt <jo...@gmail.com>
Sent: Monday, January 23, 2017 6:37:28 PM
To: dev@nifi.apache.org
Subject: Re: Service to handle workflow start/stop

Alessio

I might not quite follow all of the question but I'll try to respond
to what I believe you're asking.

First, it is important to note that whether a human is looking at the
UI or a system/script/process is interacting with the REST API of NiFi
in the end the effect is the same.  It is all happening through the
REST API.

Presently we offer the following scheduling styles available within
the framework
- timer based
- cron based
- event based

Whether a processor/component is given a thread to use at any given
time depends on whether one of those above triggers fire AND whether
or not back pressure prevents it from getting a thread or whether it
depends on there being work (data) in an inbound queue first or
whether that processor/component is presently in a yield/hold pattern
for some period of time.  That is all predicated on whether some thing
(user through UI or some service/process invoking the REST API) has
told the nifi framework that a given component may be scheduled to
execute.

So how best to inject your desire to have an external system dictate
whether a component or set of components (your notion of a workflow)
depends on how you feel the above best reflects your case.  You can
control whether processors/components are scheduled to run (not
whether they actually get triggered) by using our REST API
exclusively.  Or, you might feel that you want to add a new triggering
style such as 'external service'.  That would require framework level
code changes in an area that is not really designed for extension most
likely.  So that might be a lot harder.  So, in that sense I'd
probably recommend more evaluation and research going into leveraging
that first approach I mention to see if that meets your needs.

Thanks
Joe

On Mon, Jan 23, 2017 at 10:59 AM, Alessio Palma
<al...@docomodigital.com> wrote:
> Hello all,
>
> before starting diving into the code I have a little question:
>
> It's possible to write a service connected to processor which can be used to start workflows ?
> Scenario is this:
> Currently we can start workflows using a schedule strategy, I like to move this logic into a service which broadcast to registered processors the names of the workflows which are supposed to start in the current time frame.
> So the basic idea is to have these custom processor which keep polling the service asking the permission to start.
>
> Is there a better option which does not requires polling?
>
> If I read it carefully the  REST API can do the work, but it cuts all the advantages of the graphical environment. So this is not an option.
>
>
> AP