Singleton data inside processors

[Phil H <gi...@gmail.com>]

Hi there,

I have a custom processor that I want to contain some shared data across
threads (this data is based on analysis of flow files, so constantly
changing). I can’t use a static variable though, because I use the same
processor more than once within my flow in different contexts. If there
isn’t a “proper” way to do this in the NiFi API, is there a way to
programmatically access the componentID (as shown in the UI) so I could use
that as a key to a Map containing the data?

Thanks,
Phil

Re: Singleton data inside processors

[Otto Fowler <ot...@gmail.com>]

Take look at other processors that use AtomicXXXXX for such things ( like
the relationships), as here
https://github.com/apache/nifi/blob/master/nifi-nar-bundles/nifi-standard-bundle/nifi-standard-processors/src/main/java/org/apache/nifi/processors/standard/DistributeLoad.java




On December 8, 2019 at 20:41:28, Joe Witt (joe.witt@gmail.com) wrote:

Phil

You can have a class member variable and just protect access to it. This
is quite common in many processors.

Thanks

On Sun, Dec 8, 2019 at 7:30 PM Phil H <gi...@gmail.com> wrote:

> Hi there,
>
> I have a custom processor that I want to contain some shared data across
> threads (this data is based on analysis of flow files, so constantly
> changing). I can’t use a static variable though, because I use the same
> processor more than once within my flow in different contexts. If there
> isn’t a “proper” way to do this in the NiFi API, is there a way to
> programmatically access the componentID (as shown in the UI) so I could
use
> that as a key to a Map containing the data?
>
> Thanks,
> Phil
>

Re: Singleton data inside processors

[Phil H <gi...@gmail.com>]

Ah okay, that makes sense. Thanks for clarifying that for me

On Tue, 10 Dec 2019 at 10:29, Joe Witt <jo...@gmail.com> wrote:

> Phil
>
> A single instance of a processor on a flow is like instantiating one
> instance of that processor class.  Your member variable is for that
> instance of the processor.  Each processor can have one or more threads so
> as long as you protect that member variable properly you're good to go.
>
> If what you meant was "I have multiple instances of the processor on the
> flow and I want all of them to have some sort of shared state mechanism"
> then this is what controller services are designed to accommodate but it is
> really important you're careful here as shared state like that is where
> bottlenecks can occur.  Most controller services are meant for a write
> once/read many type pattern.
>
> Thanks
>
> On Mon, Dec 9, 2019 at 6:18 PM Phil H <gi...@gmail.com> wrote:
>
> > Hi Joe,
> >
> > But if I have multiple threads running for that processor, won’t each
> > instance have its own copy of that member variable?
> >
> > Thanks,
> > Phil
> >
> > On Mon, 9 Dec 2019 at 12:41, Joe Witt <jo...@gmail.com> wrote:
> >
> > > Phil
> > >
> > > You can have a class member variable and just protect access to it.
> This
> > > is quite common in many processors.
> > >
> > > Thanks
> > >
> > > On Sun, Dec 8, 2019 at 7:30 PM Phil H <gi...@gmail.com> wrote:
> > >
> > > > Hi there,
> > > >
> > > > I have a custom processor that I want to contain some shared data
> > across
> > > > threads (this data is based on analysis of flow files, so constantly
> > > > changing). I can’t use a static variable though, because I use the
> same
> > > > processor more than once within my flow in different contexts. If
> there
> > > > isn’t a “proper” way to do this in the NiFi API, is there a way to
> > > > programmatically access the componentID (as shown in the UI) so I
> could
> > > use
> > > > that as a key to a Map containing the data?
> > > >
> > > > Thanks,
> > > > Phil
> > > >
> > >
> >
>

Re: Singleton data inside processors

[Joe Witt <jo...@gmail.com>]

Phil

A single instance of a processor on a flow is like instantiating one
instance of that processor class.  Your member variable is for that
instance of the processor.  Each processor can have one or more threads so
as long as you protect that member variable properly you're good to go.

If what you meant was "I have multiple instances of the processor on the
flow and I want all of them to have some sort of shared state mechanism"
then this is what controller services are designed to accommodate but it is
really important you're careful here as shared state like that is where
bottlenecks can occur.  Most controller services are meant for a write
once/read many type pattern.

Thanks

On Mon, Dec 9, 2019 at 6:18 PM Phil H <gi...@gmail.com> wrote:

> Hi Joe,
>
> But if I have multiple threads running for that processor, won’t each
> instance have its own copy of that member variable?
>
> Thanks,
> Phil
>
> On Mon, 9 Dec 2019 at 12:41, Joe Witt <jo...@gmail.com> wrote:
>
> > Phil
> >
> > You can have a class member variable and just protect access to it.  This
> > is quite common in many processors.
> >
> > Thanks
> >
> > On Sun, Dec 8, 2019 at 7:30 PM Phil H <gi...@gmail.com> wrote:
> >
> > > Hi there,
> > >
> > > I have a custom processor that I want to contain some shared data
> across
> > > threads (this data is based on analysis of flow files, so constantly
> > > changing). I can’t use a static variable though, because I use the same
> > > processor more than once within my flow in different contexts. If there
> > > isn’t a “proper” way to do this in the NiFi API, is there a way to
> > > programmatically access the componentID (as shown in the UI) so I could
> > use
> > > that as a key to a Map containing the data?
> > >
> > > Thanks,
> > > Phil
> > >
> >
>

Re: Singleton data inside processors

[Phil H <gi...@gmail.com>]

Hi Joe,

But if I have multiple threads running for that processor, won’t each
instance have its own copy of that member variable?

Thanks,
Phil

On Mon, 9 Dec 2019 at 12:41, Joe Witt <jo...@gmail.com> wrote:

> Phil
>
> You can have a class member variable and just protect access to it.  This
> is quite common in many processors.
>
> Thanks
>
> On Sun, Dec 8, 2019 at 7:30 PM Phil H <gi...@gmail.com> wrote:
>
> > Hi there,
> >
> > I have a custom processor that I want to contain some shared data across
> > threads (this data is based on analysis of flow files, so constantly
> > changing). I can’t use a static variable though, because I use the same
> > processor more than once within my flow in different contexts. If there
> > isn’t a “proper” way to do this in the NiFi API, is there a way to
> > programmatically access the componentID (as shown in the UI) so I could
> use
> > that as a key to a Map containing the data?
> >
> > Thanks,
> > Phil
> >
>

Re: Singleton data inside processors

[Joe Witt <jo...@gmail.com>]

Phil

You can have a class member variable and just protect access to it.  This
is quite common in many processors.

Thanks

On Sun, Dec 8, 2019 at 7:30 PM Phil H <gi...@gmail.com> wrote:

> Hi there,
>
> I have a custom processor that I want to contain some shared data across
> threads (this data is based on analysis of flow files, so constantly
> changing). I can’t use a static variable though, because I use the same
> processor more than once within my flow in different contexts. If there
> isn’t a “proper” way to do this in the NiFi API, is there a way to
> programmatically access the componentID (as shown in the UI) so I could use
> that as a key to a Map containing the data?
>
> Thanks,
> Phil
>