Re: How do I atomically increment a variable in NiFi ?

[Márcio Faria <fa...@ymail.com>]

Hi Bryan,
I didn't know about AtomicDistributedMapCacheClient before. I should surely spend more time perusing the NiFi's master branch in the future.
Your suggestion reminds of the compare-and-swap technique (which, by the way, java.util.concurrent.atomic.Atomic* classes use). I'm now wondering if having a Controller Service created around an AtomicInteger could become a bottleneck in a distributed setting... Your thoughts?
Anyway, would the following still be a good start if I decide to create my own SequenceControllerService for NiFi 1.2.0 ?

http://www.nifi.rocks/developing-a-custom-apache-nifi-controller-service/

Thank you,
Marcio
 

    On Tuesday, February 28, 2017 9:37 AM, Bryan Bende <bb...@gmail.com> wrote:
 

 Marcio,

There is a new controller service that is in the master branch, but
hasn't been released yet, AtomicDistributedMapCacheClient [1]. You
could probably use this to implement a new processor that provides the
ability to generate a global sequence with concurrent access. It is
basically an extension of the DistributedMapCacheClient that lets you
do an atomic replace.

I'm thinking of something like "GenerateSequence" with a property like
"Sequence Key" where the user can specify a key in the cache where the
sequence will be held, and as each flow file passes through the
processor it would fetch the value of that key from the cache and call
replace with the value incremented by 1, and then some type of retry
logic if it fails to replace because it was already updated by another
process.

Hope this helps.

-Bryan

[1] https://github.com/apache/nifi/blob/master/nifi-nar-bundles/nifi-standard-services/nifi-distributed-cache-client-service-api/src/main/java/org/apache/nifi/distributed/cache/client/AtomicDistributedMapCacheClient.java

On Sun, Feb 26, 2017 at 5:19 PM, Afonso Murakami <mu...@icloud.com> wrote:
> The question is How do I create my own specialized controller service ? I
> have no idea how to do that .
>
> On Feb 26, 2017, at 1:48 PM, Márcio Faria <fa...@ymail.com> wrote:
>
> Hi,
>
> I need to generate unique numbers globally across NiFi nodes in an atomic,
> persistent way. Ideally, something like a sequence but with no actual
> database involved.
>
> Maybe I'm missing something here, but I don't think a Distributed Map Cache,
> like described in Use of Sequence in Nifi without using a custom processor,
> is a solution. How would that prevent concurrent access issues?
>
> Counters don't help either because they only keep values in memory, right?
>
> Do I need to create my own specialized Controller Service?
>
> Thank you,
>
> Marcio
>
>
>
>
>
>
>