Re: Processor with State

[Joe Percivall <jo...@yahoo.com>]

Hello Claudio,

Your use-case actually could leverage a couple of recently added features to create a really cool open-source processor. The two key features that were added are State Management and the ability to reference processor specific variables in expression language. You can take a look at RouteText to see both in action. 

By utilizing both you can create a processor that is configured with multiple Expression language expressions. There would be dynamic properties which would accept expression language and then store the evaluated value via state management. Then there would be a routing property (that supports expression language) that could simply add an attribute to the flowfile with the evaluated value which would allow it to be used by flowing processors for routing.

This would allow you to do your use-case where you store the value for the incoming stream and route differently once you go over a threshold. It could even allow more complex use-cases. One instance, I believe, would be possible is to have a running average and standard deviation and route data to different locations based on it's standard deviation.


You can think of this like an UpdateAttribute with the ability to store and calculate variables using expression language.
Joe

- - - - - - 
Joseph Percivall
linkedin.com/in/Percivall
e: joepercivall@yahoo.com



On Thursday, February 25, 2016 1:12 PM, Claudio Caldato <cl...@microsoft.com> wrote:



I expect that in the future I’ll need something a little more sophisticated but for now my problem is very simple:
I want to be able to trigger an alert (only once) when an attribute in an incoming stream, for instance, goes over a predefined threshold. The Processor should then trigger (only once again) another trigger when the signal goes back to normal (below threshold). Basically a RouteByAttribute but with memory.

Thanks 
Claudio





On 2/24/16, 8:56 PM, "Joe Witt" <jo...@gmail.com> wrote:

>Claudio
>
>Hello there and welcome to the nifi community.  There are some
>processors available now that allow you to store values in distributed
>(across the cluster) maps and to retrieve them.  And now within
>processors there is the ability interact with state management
>features built into the framework.  So the basic pieces are there.  I
>would like to better understand the idea though because it may be even
>more straight forward.
>
>Where does the state or signal come from that would prompt you to
>store a value away?  And is this source/signal separate from the feed
>of data you'd like to tag with this value?
>
>For example, we have the UpdateAttribute processor which can be used
>to tag attributes onto flow files going by.  You can of course simply
>call the rest api to change the tag being applied as needed and that
>can be done by whatever the signal/source is potentially.
>
>Thanks
>Joe
>
>On Wed, Feb 24, 2016 at 11:49 PM, Claudio Caldato
><cl...@microsoft.com> wrote:
>>
>> I need to be able to store a simple value (it can be true/false) in the
>> processor across messages, basically I need a processor with a local state
>> (set of properties) that I can use to set the value of properties on output
>> messages
>>
>> Can it be done or do I need to build a custom processor?
>>
>> Thanks
>> Claudio
>>

Re: Processor with State

[Joe Percivall <jo...@yahoo.com>]

I created a jira ticket to track this idea for a processor that enables updating an attribute using state, which should enable the very basics of data science: https://issues.apache.org/jira/browse/NIFI-1582
 
Joe- - - - - - 
Joseph Percivall
linkedin.com/in/Percivall
e: joepercivall@yahoo.com




On Wednesday, March 2, 2016 11:19 AM, Joe Percivall <jo...@yahoo.com> wrote:
Hello Claudio,

Your use-case actually could leverage a couple of recently added features to create a really cool open-source processor. The two key features that were added are State Management and the ability to reference processor specific variables in expression language. You can take a look at RouteText to see both in action. 

By utilizing both you can create a processor that is configured with multiple Expression language expressions. There would be dynamic properties which would accept expression language and then store the evaluated value via state management. Then there would be a routing property (that supports expression language) that could simply add an attribute to the flowfile with the evaluated value which would allow it to be used by flowing processors for routing.

This would allow you to do your use-case where you store the value for the incoming stream and route differently once you go over a threshold. It could even allow more complex use-cases. One instance, I believe, would be possible is to have a running average and standard deviation and route data to different locations based on it's standard deviation.


You can think of this like an UpdateAttribute with the ability to store and calculate variables using expression language.
Joe

- - - - - - 
Joseph Percivall
linkedin.com/in/Percivall
e: joepercivall@yahoo.com




On Thursday, February 25, 2016 1:12 PM, Claudio Caldato <cl...@microsoft.com> wrote:



I expect that in the future I’ll need something a little more sophisticated but for now my problem is very simple:
I want to be able to trigger an alert (only once) when an attribute in an incoming stream, for instance, goes over a predefined threshold. The Processor should then trigger (only once again) another trigger when the signal goes back to normal (below threshold). Basically a RouteByAttribute but with memory.

Thanks 
Claudio





On 2/24/16, 8:56 PM, "Joe Witt" <jo...@gmail.com> wrote:

>Claudio
>
>Hello there and welcome to the nifi community.  There are some
>processors available now that allow you to store values in distributed
>(across the cluster) maps and to retrieve them.  And now within
>processors there is the ability interact with state management
>features built into the framework.  So the basic pieces are there.  I
>would like to better understand the idea though because it may be even
>more straight forward.
>
>Where does the state or signal come from that would prompt you to
>store a value away?  And is this source/signal separate from the feed
>of data you'd like to tag with this value?
>
>For example, we have the UpdateAttribute processor which can be used
>to tag attributes onto flow files going by.  You can of course simply
>call the rest api to change the tag being applied as needed and that
>can be done by whatever the signal/source is potentially.
>
>Thanks
>Joe
>
>On Wed, Feb 24, 2016 at 11:49 PM, Claudio Caldato
><cl...@microsoft.com> wrote:
>>
>> I need to be able to store a simple value (it can be true/false) in the
>> processor across messages, basically I need a processor with a local state
>> (set of properties) that I can use to set the value of properties on output
>> messages
>>
>> Can it be done or do I need to build a custom processor?
>>
>> Thanks
>> Claudio
>>