You are viewing a plain text version of this content. The canonical link for it is here.

Posted to users@nifi.apache.org by Boris Tyukin <bo...@boristyukin.com> on 2018/09/17 18:06:44 UTC

Run another PG once first one is finished

Hi,

Let's say I have two totally independent projects / PGs - PG ProjectA and
PG ProjectB.

What would be the best way to kick off PG ProjectB when PG ProjectA is
finished?

I know I can use output/input ports but I do not want to wire them like
that since they are independent projects.

The only I idea I came up is to use Kafka and push completion event to a
common topic and then have Project B subscribe to it.

Thanks,
Boris

Re: Run another PG once first one is finished

Posted by Boris Tyukin <bo...@boristyukin.com>.

hi Joe,

this is a very valid question. And something I am still thinking about if
this is worth it. Technically, both flows are totally independent and the
second one can run even if the second one is not finished or failed - it
will still produce correct but stale data.

On Mon, Sep 17, 2018 at 9:56 PM Joe Witt <jo...@gmail.com> wrote:

> Boris,
>
> It is of course a little bit interesting to state that two groups are
> completely independent while then also stating you want to trigger
> group B on the completion of group A.  Clearly they have a dependency.
> I dont quite follow the desire then to decouple them from the flow
> configuration and where you can easily see and manage this
> relationship but then establish another mechanism to establish this
> linkage such as Kafka, files, etc..  I would recommend sharing more
> about why you would want to hide this relationship.
>
> That said, another option in the list of ways to accomplish this is
> use of Wait/Notify processors.
>
> Thanks
> Joe
> On Mon, Sep 17, 2018 at 9:23 PM Boris Tyukin <bo...@boristyukin.com>
> wrote:
> >
> > thanks Ed! Totally forgot about S2S - we already use it to monitor error
> bulletins. Thanks for the ideas!
> >
> > Boris
> >
> > On Mon, Sep 17, 2018 at 6:43 PM Ed B <bd...@gmail.com> wrote:
> >>
> >> Boris,
> >> My first idea while reading your question was - Kafka, looks like very
> valid approach (event driven implementation?)
> >>
> >> But you can also check on:
> >> 1. S2S
> >> 2. ListenHTTP in project B (project A can use postHTTP or invokeHTTP)
> (not really cluster-node)
> >> 3. If it is important to start Project B on the same node where Project
> A finished its part - you can use PutFile (Project A)/GetFile (Project B)
> (signal files) instead of Kafka.
> >>
> >> hope that helps.
> >> Ed.
> >>
> >>
> >> On Mon, Sep 17, 2018 at 2:08 PM Boris Tyukin <bo...@boristyukin.com>
> wrote:
> >>>
> >>> Hi,
> >>>
> >>> Let's say I have two totally independent projects / PGs - PG ProjectA
> and PG ProjectB.
> >>>
> >>> What would be the best way to kick off PG ProjectB when PG ProjectA is
> finished?
> >>>
> >>> I know I can use output/input ports but I do not want to wire them
> like that since they are independent projects.
> >>>
> >>> The only I idea I came up is to use Kafka and push completion event to
> a common topic and then have Project B subscribe to it.
> >>>
> >>> Thanks,
> >>> Boris
>

Re: Run another PG once first one is finished

Posted by Joe Witt <jo...@gmail.com>.

Boris,

It is of course a little bit interesting to state that two groups are
completely independent while then also stating you want to trigger
group B on the completion of group A.  Clearly they have a dependency.
I dont quite follow the desire then to decouple them from the flow
configuration and where you can easily see and manage this
relationship but then establish another mechanism to establish this
linkage such as Kafka, files, etc..  I would recommend sharing more
about why you would want to hide this relationship.

That said, another option in the list of ways to accomplish this is
use of Wait/Notify processors.

Thanks
Joe
On Mon, Sep 17, 2018 at 9:23 PM Boris Tyukin <bo...@boristyukin.com> wrote:
>
> thanks Ed! Totally forgot about S2S - we already use it to monitor error bulletins. Thanks for the ideas!
>
> Boris
>
> On Mon, Sep 17, 2018 at 6:43 PM Ed B <bd...@gmail.com> wrote:
>>
>> Boris,
>> My first idea while reading your question was - Kafka, looks like very valid approach (event driven implementation?)
>>
>> But you can also check on:
>> 1. S2S
>> 2. ListenHTTP in project B (project A can use postHTTP or invokeHTTP) (not really cluster-node)
>> 3. If it is important to start Project B on the same node where Project A finished its part - you can use PutFile (Project A)/GetFile (Project B) (signal files) instead of Kafka.
>>
>> hope that helps.
>> Ed.
>>
>>
>> On Mon, Sep 17, 2018 at 2:08 PM Boris Tyukin <bo...@boristyukin.com> wrote:
>>>
>>> Hi,
>>>
>>> Let's say I have two totally independent projects / PGs - PG ProjectA and PG ProjectB.
>>>
>>> What would be the best way to kick off PG ProjectB when PG ProjectA is finished?
>>>
>>> I know I can use output/input ports but I do not want to wire them like that since they are independent projects.
>>>
>>> The only I idea I came up is to use Kafka and push completion event to a common topic and then have Project B subscribe to it.
>>>
>>> Thanks,
>>> Boris

Re: Run another PG once first one is finished

Posted by Boris Tyukin <bo...@boristyukin.com>.

thanks Ed! Totally forgot about S2S - we already use it to monitor error
bulletins. Thanks for the ideas!

Boris

On Mon, Sep 17, 2018 at 6:43 PM Ed B <bd...@gmail.com> wrote:

> Boris,
> My first idea while reading your question was - Kafka, looks like very
> valid approach (event driven implementation?)
>
> But you can also check on:
> 1. S2S
> 2. ListenHTTP in project B (project A can use postHTTP or invokeHTTP) (not
> really cluster-node)
> 3. If it is important to start Project B on the same node where Project A
> finished its part - you can use PutFile (Project A)/GetFile (Project B)
> (signal files) instead of Kafka.
>
> hope that helps.
> Ed.
>
>
> On Mon, Sep 17, 2018 at 2:08 PM Boris Tyukin <bo...@boristyukin.com>
> wrote:
>
>> Hi,
>>
>> Let's say I have two totally independent projects / PGs - PG ProjectA and
>> PG ProjectB.
>>
>> What would be the best way to kick off PG ProjectB when PG ProjectA is
>> finished?
>>
>> I know I can use output/input ports but I do not want to wire them like
>> that since they are independent projects.
>>
>> The only I idea I came up is to use Kafka and push completion event to a
>> common topic and then have Project B subscribe to it.
>>
>> Thanks,
>> Boris
>>
>

Re: Run another PG once first one is finished

Posted by Ed B <bd...@gmail.com>.

Boris,
My first idea while reading your question was - Kafka, looks like very
valid approach (event driven implementation?)

But you can also check on:
1. S2S
2. ListenHTTP in project B (project A can use postHTTP or invokeHTTP) (not
really cluster-node)
3. If it is important to start Project B on the same node where Project A
finished its part - you can use PutFile (Project A)/GetFile (Project B)
(signal files) instead of Kafka.

hope that helps.
Ed.

On Mon, Sep 17, 2018 at 2:08 PM Boris Tyukin <bo...@boristyukin.com> wrote:

> Hi,
>
> Let's say I have two totally independent projects / PGs - PG ProjectA and
> PG ProjectB.
>
> What would be the best way to kick off PG ProjectB when PG ProjectA is
> finished?
>
> I know I can use output/input ports but I do not want to wire them like
> that since they are independent projects.
>
> The only I idea I came up is to use Kafka and push completion event to a
> common topic and then have Project B subscribe to it.
>
> Thanks,
> Boris
>