wait/notify - serial processing (gate flipflop)

[Emanuel Oliveira <em...@gmail.com>]

Hi,

I been testing wait/notify pattern specially on a process one-by-one
pattern (an http rest API can only process 2nd request only after previous
completed).

1. i managed to have it work using technique i found here
https://community.cloudera.com/t5/Community-Articles/Trigger-based-Serial-Data-processing-in-NiFi-using-Wait-and/ta-p/248308
Basically it tags each ff via statefull UpdateAttribute and then creates a
distributedcache key "Release Signal Identifier" with this token (1, 2, ..)
and works fine. Wait removes key after consuming/releasing it.
2. but i was unable to understand - i guess - how to implement it uding
tooltip suggestion of special values zero(0) or one(1) on both
wait.Releasable FlowFile Count and notify.xxxxx which seems suggestion its
possible to implement exactly what i want (a flip-flop gate,. where some
processing happens serial way one at a time.. but i wasn't able to have
this work. It seems both wait/notify Release Signal Identifier would/can be
a static value (doesn't matter) as the magic values of zero(close gate) or
one(open gate) would be enough... but wasn't able to have this work. Any
suggestion please how to have a flip-flop or Singleton or serial processing
pattern using the magic values and just 1 Release Signal Identifier?

wait.Releasable FlowFile Count
"(..)Zero (0) has a special meaning, any number of FlowFiles can be
released as long as signal count matches target.(..)"

notify.Signal Counter Delta
"(..)Zero (0) has a special meaning, it clears target count back to 0,
which is especially useful when used with Wait Releasable FlowFile Count =
Zero (0) mode, to provide 'open-close-gate' type of flow control. One (1)
can open a corresponding Wait processor, and Zero (0) can negate it as if
closing a gate.(..)".

Ideally would be great if someone could share a minimalistic xml template
just demo this capacity suggested by both wait/notify processors
documentation.


Thanks,
Emanuel Oliveira

Re: wait/notify - serial processing (gate flipflop)

[Etienne Jouvin <la...@gmail.com>]

Hello.

Not sure to correctly understand the expected behavior.
But if this is for a "singleton" execution, I try to do it in previous
project, but I had errors. Not only singleton, always 2 flow files were
proceeded.

But this morning (thanks 2020 ?) I made it works.
Here is a screenshot :
[image: image.png]

I put an XML Template in attached file.

It uses the wait and notify processors with exactly the same release signal.

I use the generate flowfiles to generate 4 flow files with dummy content.

The wait configuration
[image: image.png]

The notify configuration
[image: image.png]


I put a back pressure object threshold to 1 between the "Dummy process" and
the funnel, between funnel and Wait

The dummy process seems really important. I tried to put the link between
Generate and the funnel. But this is not a good idea, because all flowfiles
will be process in parallel.

To verify this is working, just stop the Notify processor. You will see
that there is only one in queue, one in queue before the wait and 2 in
queue before the "dummy processor".


Hope it helps.

Etienne Jouvin


Le mar. 24 déc. 2019 à 15:17, Emanuel Oliveira <em...@gmail.com> a
écrit :

> Hi,
>
> I been testing wait/notify pattern specially on a process one-by-one
> pattern (an http rest API can only process 2nd request only after previous
> completed).
>
> 1. i managed to have it work using technique i found here
> https://community.cloudera.com/t5/Community-Articles/Trigger-based-Serial-Data-processing-in-NiFi-using-Wait-and/ta-p/248308
> Basically it tags each ff via statefull UpdateAttribute and then creates a
> distributedcache key "Release Signal Identifier" with this token (1, 2, ..)
> and works fine. Wait removes key after consuming/releasing it.
> 2. but i was unable to understand - i guess - how to implement it uding
> tooltip suggestion of special values zero(0) or one(1) on both
> wait.Releasable FlowFile Count and notify.xxxxx which seems suggestion its
> possible to implement exactly what i want (a flip-flop gate,. where some
> processing happens serial way one at a time.. but i wasn't able to have
> this work. It seems both wait/notify Release Signal Identifier would/can be
> a static value (doesn't matter) as the magic values of zero(close gate) or
> one(open gate) would be enough... but wasn't able to have this work. Any
> suggestion please how to have a flip-flop or Singleton or serial processing
> pattern using the magic values and just 1 Release Signal Identifier?
>
> wait.Releasable FlowFile Count
> "(..)Zero (0) has a special meaning, any number of FlowFiles can be
> released as long as signal count matches target.(..)"
>
> notify.Signal Counter Delta
> "(..)Zero (0) has a special meaning, it clears target count back to 0,
> which is especially useful when used with Wait Releasable FlowFile Count =
> Zero (0) mode, to provide 'open-close-gate' type of flow control. One (1)
> can open a corresponding Wait processor, and Zero (0) can negate it as if
> closing a gate.(..)".
>
> Ideally would be great if someone could share a minimalistic xml template
> just demo this capacity suggested by both wait/notify processors
> documentation.
>
>
> Thanks,
> Emanuel Oliveira
>
>