You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@nifi.apache.org by Andrew Grande <ap...@gmail.com> on 2016/07/11 02:23:55 UTC
Re: State sharing
Sumo,
IIRC there's a node one selects when setting state. If you invoke with a
cluster mode, the state will be set to a ZK by default. Otherwise just
local to this processor node.
Andrew
On Sun, Jul 10, 2016, 10:17 PM Sumanth Chinthagunta <xm...@gmail.com>
wrote:
> If I set state from one ExecuteScript processor via stateManager , can I
> access same state from other processor ?
> Thanks
> Sumo
>
> Sent from my iPhone
Re: State sharing
Posted by Sumanth Chinthagunta <xm...@gmail.com>.
Andrew,
Thanks for the link. My current use case need to store data in centralized key/value store, that can be updated by one Processor and read by couple of different processors in the cluster. Data speed is low (tracking database schema changes) , so concurrently is not big concern.
My use case is explained little bit here: https://github.com/zendesk/maxwell/pull/344 <https://github.com/zendesk/maxwell/pull/344>
+---------+------------+
| Table | Version |
+---------+------------+
| Book | v1 |
+---------+------------+
| Author | v2 |
+---------+------------+
| Sales | v1 |
+---------+------------+
| Cart | v1 |
+---------+------------+
One processor update schema version number (increment) whenever new schema change event arrived to it.
Other processors start writing new DML events to a sub folder, based on schema version number updated by first processes .
Thanks
Sumanth
> On Jul 11, 2016, at 10:33 AM, Andrew Grande <ap...@gmail.com> wrote:
>
> Sumo,
>
> Something lightweight I devised here, backed by a simple in mem concurrent
> map, for cases when a distributed map is too much
> https://github.com/aperepel/nifi-csv-bundle/tree/master/nifi-csv-processors/src/main/java/org/apache/nifi/processors/lookup
>
> In a cluster, though, it's the true distributed replicated cache that one
> must explicitly design for and use. While state framework is ok, the
> default implementation backed by a ZK is not meant for high speed
> concurrent use. Distributed caches are, however.
>
> Andrew
>
> On Sun, Jul 10, 2016, 10:58 PM Sumanth Chinthagunta <xm...@gmail.com>
> wrote:
>
>> Thanks Bryan.
>> Would be nice if we get support for state sharing across diffrent
>> processors in the future.
>> -Sumo
>>
>> Sent from my iPhone
>>
>>> On Jul 10, 2016, at 7:39 PM, Bryan Bende <bb...@gmail.com> wrote:
>>>
>>> Sumo,
>>>
>>> Two different processor instances (different UUIDs) can not share state
>>> that is stored through the state manager. For something like this you
>> would
>>> likely use the distributed map cache.
>>>
>>> As Andrew mentioned, the state is accessible across the cluster, so a
>>> given processor can access the state from any node because the processor
>>> will have the same UUID on each node.
>>>
>>> -Bryan
>>>
>>>> On Sunday, July 10, 2016, Andrew Grande <ap...@gmail.com> wrote:
>>>>
>>>> Sumo,
>>>>
>>>> IIRC there's a node one selects when setting state. If you invoke with a
>>>> cluster mode, the state will be set to a ZK by default. Otherwise just
>>>> local to this processor node.
>>>>
>>>> Andrew
>>>>
>>>> On Sun, Jul 10, 2016, 10:17 PM Sumanth Chinthagunta <xmlking@gmail.com
>>>> <javascript:;>>
>>>> wrote:
>>>>
>>>>> If I set state from one ExecuteScript processor via stateManager , can
>> I
>>>>> access same state from other processor ?
>>>>> Thanks
>>>>> Sumo
>>>>>
>>>>> Sent from my iPhone
>>>
>>>
>>> --
>>> Sent from Gmail Mobile
>>
Re: State sharing
Posted by Andrew Grande <ap...@gmail.com>.
Sumo,
Something lightweight I devised here, backed by a simple in mem concurrent
map, for cases when a distributed map is too much
https://github.com/aperepel/nifi-csv-bundle/tree/master/nifi-csv-processors/src/main/java/org/apache/nifi/processors/lookup
In a cluster, though, it's the true distributed replicated cache that one
must explicitly design for and use. While state framework is ok, the
default implementation backed by a ZK is not meant for high speed
concurrent use. Distributed caches are, however.
Andrew
On Sun, Jul 10, 2016, 10:58 PM Sumanth Chinthagunta <xm...@gmail.com>
wrote:
> Thanks Bryan.
> Would be nice if we get support for state sharing across diffrent
> processors in the future.
> -Sumo
>
> Sent from my iPhone
>
> > On Jul 10, 2016, at 7:39 PM, Bryan Bende <bb...@gmail.com> wrote:
> >
> > Sumo,
> >
> > Two different processor instances (different UUIDs) can not share state
> > that is stored through the state manager. For something like this you
> would
> > likely use the distributed map cache.
> >
> > As Andrew mentioned, the state is accessible across the cluster, so a
> > given processor can access the state from any node because the processor
> > will have the same UUID on each node.
> >
> > -Bryan
> >
> >> On Sunday, July 10, 2016, Andrew Grande <ap...@gmail.com> wrote:
> >>
> >> Sumo,
> >>
> >> IIRC there's a node one selects when setting state. If you invoke with a
> >> cluster mode, the state will be set to a ZK by default. Otherwise just
> >> local to this processor node.
> >>
> >> Andrew
> >>
> >> On Sun, Jul 10, 2016, 10:17 PM Sumanth Chinthagunta <xmlking@gmail.com
> >> <javascript:;>>
> >> wrote:
> >>
> >>> If I set state from one ExecuteScript processor via stateManager , can
> I
> >>> access same state from other processor ?
> >>> Thanks
> >>> Sumo
> >>>
> >>> Sent from my iPhone
> >
> >
> > --
> > Sent from Gmail Mobile
>
Re: State sharing
Posted by Sumanth Chinthagunta <xm...@gmail.com>.
Thanks Bryan.
Would be nice if we get support for state sharing across diffrent processors in the future.
-Sumo
Sent from my iPhone
> On Jul 10, 2016, at 7:39 PM, Bryan Bende <bb...@gmail.com> wrote:
>
> Sumo,
>
> Two different processor instances (different UUIDs) can not share state
> that is stored through the state manager. For something like this you would
> likely use the distributed map cache.
>
> As Andrew mentioned, the state is accessible across the cluster, so a
> given processor can access the state from any node because the processor
> will have the same UUID on each node.
>
> -Bryan
>
>> On Sunday, July 10, 2016, Andrew Grande <ap...@gmail.com> wrote:
>>
>> Sumo,
>>
>> IIRC there's a node one selects when setting state. If you invoke with a
>> cluster mode, the state will be set to a ZK by default. Otherwise just
>> local to this processor node.
>>
>> Andrew
>>
>> On Sun, Jul 10, 2016, 10:17 PM Sumanth Chinthagunta <xmlking@gmail.com
>> <javascript:;>>
>> wrote:
>>
>>> If I set state from one ExecuteScript processor via stateManager , can I
>>> access same state from other processor ?
>>> Thanks
>>> Sumo
>>>
>>> Sent from my iPhone
>
>
> --
> Sent from Gmail Mobile
Re: State sharing
Posted by Bryan Bende <bb...@gmail.com>.
Sumo,
Two different processor instances (different UUIDs) can not share state
that is stored through the state manager. For something like this you would
likely use the distributed map cache.
As Andrew mentioned, the state is accessible across the cluster, so a
given processor can access the state from any node because the processor
will have the same UUID on each node.
-Bryan
On Sunday, July 10, 2016, Andrew Grande <ap...@gmail.com> wrote:
> Sumo,
>
> IIRC there's a node one selects when setting state. If you invoke with a
> cluster mode, the state will be set to a ZK by default. Otherwise just
> local to this processor node.
>
> Andrew
>
> On Sun, Jul 10, 2016, 10:17 PM Sumanth Chinthagunta <xmlking@gmail.com
> <javascript:;>>
> wrote:
>
> > If I set state from one ExecuteScript processor via stateManager , can I
> > access same state from other processor ?
> > Thanks
> > Sumo
> >
> > Sent from my iPhone
>
--
Sent from Gmail Mobile
Re: State sharing
Posted by Sumanth Chinthagunta <xm...@gmail.com>.
Andrew,
If I use cluster mode, if one processor writes state, a diffrent processor can read it ?
I understand that same processor on multiple nodes in the cluster can share state.
Thanks
Sumo
Sent from my iPhone
> On Jul 10, 2016, at 7:23 PM, Andrew Grande <ap...@gmail.com> wrote:
>
> Sumo,
>
> IIRC there's a node one selects when setting state. If you invoke with a
> cluster mode, the state will be set to a ZK by default. Otherwise just
> local to this processor node.
>
> Andrew
>
> On Sun, Jul 10, 2016, 10:17 PM Sumanth Chinthagunta <xm...@gmail.com>
> wrote:
>
>> If I set state from one ExecuteScript processor via stateManager , can I
>> access same state from other processor ?
>> Thanks
>> Sumo
>>
>> Sent from my iPhone