You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@nifi.apache.org by Russell Bateman <ru...@windofkeltia.com> on 2024/05/07 17:38:11 UTC

...not the most recent version of this FlowFile within this session...

In /pom.xml/ I specify using NiFi framework libraries from 1.13.2.

There are peculiar reasons (that we are trying to fix) that inhibit us 
from moving forward as all of our customer machines are running 1.1.2. 
(Don't shoot me, I'm not DevOps, but just the guy who writes custom 
processors.)

I have custom processor code that, built as noted using 1.13.2, runs 
_perfectly_ on any version of NiFi from 1.13.2 to 1.25.0. When loaded on 
1.1.2, one processor fails to process a very simple flowfile with the 
error message below.

I'm wondering if there's a dance I can do to tweak one or more of what I 
suspect are the calls where this is happening (in onTrigger()):

    FlowFile result = session.*write*( flowfile, new StreamCallback()
                       session.*read*( flowfile, new InputStreamCallback()
                       session.*removeAllAttributes*( result, keys );
                       session.*putAllAttributes*( result, attributes );
                       session.*transfer*( result, SUCCESS );


Error message:

10:32:47 MDTERRORc78b2d48-79a6-39ae-1454-4ee57bbf8928
ExcerptWarehouseResponse[id=c78b2d48-79a6-39ae-1454-4ee57bbf8928] \
failed to process session due to
org.apache.nifi.processor.exception.FlowFileHandlingException: \
StandardFlowFileRecord[uuid=cfb2a124-4b07-46bd-a70e-3848502b3df1,claim=StandardContentClaim \
[resourceClaim=StandardResourceClaim[id=1715094869203-1, container=default, section=1], offset=0, length=1077766],offset=0,name=30575811037248611,size=1077766] \
is not the most recent version of this FlowFile within this session (StandardProcessSession[id=3923]): \
org.apache.nifi.processor.exception.FlowFileHandlingException: StandardFlowFileRecord[uuid=cfb2a124-4b07-46bd-a70e-3848502b3df1,claim=StandardContentClaim \
[resourceClaim=StandardResourceClaim[id=1715094869203-1, container=default, section=1], offset=0, length=1077766],offset=0,name=30575811037248611,size=1077766] \
is not the most recent version of this FlowFile within this session (StandardProcessSession[id=3923])

10:32:47 MDTWARNINGc78b2d48-79a6-39ae-1454-4ee57bbf8928
ExcerptWarehouseResponse[id=c78b2d48-79a6-39ae-1454-4ee57bbf8928] Processor Administratively Yielded for 1 sec due to processing failure


I know this is old stuff and I'm prepared be told that I'm out of luck 
or that the solution is unfeasible or would take too much effort to dig 
through. I'm just hoping that someone remembers something quick off the 
top of the head and can give advice.

Thanks,

Russ






Re: ...not the most recent version of this FlowFile within this session...

Posted by Dan S <ds...@gmail.com>.
NIFI-13288 <https://issues.apache.org/jira/browse/NIFI-13288>

On Thu, May 23, 2024 at 12:46 PM Dan S <ds...@gmail.com> wrote:

> I will make a ticket for this. Thanks!
>
> On Thu, May 23, 2024 at 12:45 PM Mark Payne <ma...@hotmail.com> wrote:
>
>> Hey Dan,
>>
>> Yessir, it absolutely would! Probably would be good to clean those up.
>>
>> Thanks
>> -Mark
>>
>>
>> > On May 23, 2024, at 12:29 PM, Dan S <ds...@gmail.com> wrote:
>> >
>> > Mark,
>> > In regards to your last comments
>> >
>> >> Not really related, but on the note of things you may not realize, with
>> >> that code snippet :)
>> >> If you have access to update the processor mentioned here, you should
>> >> avoid using session.putAttribute many times.
>> >> Under the hood, in order to maintain object immutability it has to
>> create
>> >> a new FlowFile object (and a new HashMap of all attributes!)
>> >> for every call to putAttribute. So if there are 100 attributes that
>> match
>> >> that’s potentially a huge amount of garbage getting created.
>> >
>> >
>> > I noticed that some of the split processors SplitJson, SplitXml, and
>> > SplitAvro all have loops to create a new flow file for each split and it
>> > calls putAttribute more than once (in order to populate the split
>> > attributes FRAGMENT_ID, FRAGMENT_INDEX etc)  for each flow file created.
>> > Would this all suffer from "a huge amount of garbage getting created"
>> > since putAttribute is called multiple times for each iteration of the
>> loop?
>> >
>> > On Wed, May 8, 2024 at 5:27 PM Michael Moser <mo...@gmail.com>
>> wrote:
>> >
>> >> Oh yeah, I do love this behavior of ProcessSession.  And thanks for the
>> >> tip, it's easy to forget that there are efficiencies to be gained by
>> using
>> >> different parts of an API.
>> >>
>> >> -- Mke
>> >>
>> >>
>> >> On Wed, May 8, 2024 at 11:04 AM Mark Payne <ma...@hotmail.com>
>> wrote:
>> >>
>> >>> Yeah, that was something that kinda flew under the radar. Definitely
>> >>> improved the API.
>> >>>
>> >>> Not really related, but on the note of things you may not realize,
>> with
>> >>> that code snippet :)
>> >>> If you have access to update the processor mentioned here, you should
>> >>> avoid using session.putAttribute many times.
>> >>> Under the hood, in order to maintain object immutability it has to
>> create
>> >>> a new FlowFile object (and a new HashMap of all attributes!)
>> >>> for every call to putAttribute. So if there are 100 attributes that
>> match
>> >>> that’s potentially a huge amount of garbage getting created. Instead,
>> >>> you could just use:
>> >>>
>> >>> ```
>> >>> final Map<String, String> attributes = new HashMap<>();
>> >>> flowFile.getAttributes().forEach( (key, value) -> {
>> >>>  if (key.startsWith(“foo”)) {
>> >>>    attributes.put(“original-“ + key, value);
>> >>>  }
>> >>> }
>> >>>
>> >>> flowFIle = session.putAllAttributes(flowFile, attributes);
>> >>> ```
>> >>>
>> >>> Thanks
>> >>> -Mark
>> >>>
>> >>>
>> >>>
>> >>>> On May 8, 2024, at 10:30 AM, Michael Moser <mo...@gmail.com>
>> wrote:
>> >>>>
>> >>>> Wow, thanks for this information!  Just last week I saw code that
>> >>> modified
>> >>>> attributes in a stream:
>> >>>>
>> >>>> flowFile.getAttributes().entrySet().stream().filter(e ->
>> >>>> e.getKey().startsWith("foo"))
>> >>>>   .forEach(e -> session.putAttribute(flowFile, "original-" +
>> e.getKey,
>> >>>> e.getValue()));
>> >>>>
>> >>>> and I wondered how that could possibly work since the return value of
>> >>>> session.putAttribute is ignored!  Now I know.
>> >>>>
>> >>>> -- Mike
>> >>>>
>> >>>> On Tue, May 7, 2024 at 3:02 PM Russell Bateman <
>> russ@windofkeltia.com>
>> >>>> wrote:
>> >>>>
>> >>>>> Yes, what you described is what was happening, Mark. I didn't
>> display
>> >>>>> all of the code to the session methods, and I did re-read the
>> >> in-coming
>> >>>>> flowfile for different purposes than I had already read and written
>> >> it.
>> >>>>> So, I wasn't helpful enough. In the end, however, I had forgotten,
>> >>>>> immediately after the call to session.putAllAttributes(), to update
>> >> the
>> >>>>> resulting flowfile for passing to session.transfer(). That solved it
>> >> for
>> >>>>> 1.1.2 which wasn't necessary for 1.13.2 or later versions. Being
>> >>>>> helpful, the newer versions made me a spoiled, entitled child and I
>> >> will
>> >>>>> repent immediately.
>> >>>>>
>> >>>>> Thanks, guys! DevOps are happy they don't have to upgrade the
>> >> customers
>> >>>>> to NiFi 1.13.2. (In a way, I'm unhappy about that, but...).
>> >>>>>
>> >>>>> Best regards,
>> >>>>>
>> >>>>> Russ
>> >>>>>
>> >>>>> On 5/7/24 11:53, Mark Payne wrote:
>> >>>>>> The call to session.putAttribute would throw an Exception because
>> you
>> >>>>>> provided an outdated version of the flowFile (did not capturing the
>> >>>>>> result of calling session.write)
>> >>>>>>
>> >>>>>> Now, as NiFi matured, we found that:
>> >>>>>> (a) for more complex processors that aren’t just a series of
>> >> sequential
>> >>>>>> steps it becomes difficult to manage all of that bookkeeping.
>> >>>>>> (b) it was not intuitive to require this
>> >>>>>> (c) the ProcessSession already had more or less what it needed in
>> >> order
>> >>>>>> to determine what the most up-to-date version of the FlowFile was.
>> >>>>>>
>> >>>>>> So we updated the ProcessSession to automatically grab the latest
>> >>>>>> version of the FlowFile for these methods. But since you’re trying
>> to
>> >>>>>> run an old version, you’ll need to make sure that you capture all
>> of
>> >>>>>> those outputs and always keep track of the most recent version of a
>> >>>>>> FlowFile.
>> >>>>>
>> >>>
>> >>>
>> >>
>>
>>

Re: ...not the most recent version of this FlowFile within this session...

Posted by Dan S <ds...@gmail.com>.
I will make a ticket for this. Thanks!

On Thu, May 23, 2024 at 12:45 PM Mark Payne <ma...@hotmail.com> wrote:

> Hey Dan,
>
> Yessir, it absolutely would! Probably would be good to clean those up.
>
> Thanks
> -Mark
>
>
> > On May 23, 2024, at 12:29 PM, Dan S <ds...@gmail.com> wrote:
> >
> > Mark,
> > In regards to your last comments
> >
> >> Not really related, but on the note of things you may not realize, with
> >> that code snippet :)
> >> If you have access to update the processor mentioned here, you should
> >> avoid using session.putAttribute many times.
> >> Under the hood, in order to maintain object immutability it has to
> create
> >> a new FlowFile object (and a new HashMap of all attributes!)
> >> for every call to putAttribute. So if there are 100 attributes that
> match
> >> that’s potentially a huge amount of garbage getting created.
> >
> >
> > I noticed that some of the split processors SplitJson, SplitXml, and
> > SplitAvro all have loops to create a new flow file for each split and it
> > calls putAttribute more than once (in order to populate the split
> > attributes FRAGMENT_ID, FRAGMENT_INDEX etc)  for each flow file created.
> > Would this all suffer from "a huge amount of garbage getting created"
> > since putAttribute is called multiple times for each iteration of the
> loop?
> >
> > On Wed, May 8, 2024 at 5:27 PM Michael Moser <mo...@gmail.com> wrote:
> >
> >> Oh yeah, I do love this behavior of ProcessSession.  And thanks for the
> >> tip, it's easy to forget that there are efficiencies to be gained by
> using
> >> different parts of an API.
> >>
> >> -- Mke
> >>
> >>
> >> On Wed, May 8, 2024 at 11:04 AM Mark Payne <ma...@hotmail.com>
> wrote:
> >>
> >>> Yeah, that was something that kinda flew under the radar. Definitely
> >>> improved the API.
> >>>
> >>> Not really related, but on the note of things you may not realize, with
> >>> that code snippet :)
> >>> If you have access to update the processor mentioned here, you should
> >>> avoid using session.putAttribute many times.
> >>> Under the hood, in order to maintain object immutability it has to
> create
> >>> a new FlowFile object (and a new HashMap of all attributes!)
> >>> for every call to putAttribute. So if there are 100 attributes that
> match
> >>> that’s potentially a huge amount of garbage getting created. Instead,
> >>> you could just use:
> >>>
> >>> ```
> >>> final Map<String, String> attributes = new HashMap<>();
> >>> flowFile.getAttributes().forEach( (key, value) -> {
> >>>  if (key.startsWith(“foo”)) {
> >>>    attributes.put(“original-“ + key, value);
> >>>  }
> >>> }
> >>>
> >>> flowFIle = session.putAllAttributes(flowFile, attributes);
> >>> ```
> >>>
> >>> Thanks
> >>> -Mark
> >>>
> >>>
> >>>
> >>>> On May 8, 2024, at 10:30 AM, Michael Moser <mo...@gmail.com>
> wrote:
> >>>>
> >>>> Wow, thanks for this information!  Just last week I saw code that
> >>> modified
> >>>> attributes in a stream:
> >>>>
> >>>> flowFile.getAttributes().entrySet().stream().filter(e ->
> >>>> e.getKey().startsWith("foo"))
> >>>>   .forEach(e -> session.putAttribute(flowFile, "original-" + e.getKey,
> >>>> e.getValue()));
> >>>>
> >>>> and I wondered how that could possibly work since the return value of
> >>>> session.putAttribute is ignored!  Now I know.
> >>>>
> >>>> -- Mike
> >>>>
> >>>> On Tue, May 7, 2024 at 3:02 PM Russell Bateman <russ@windofkeltia.com
> >
> >>>> wrote:
> >>>>
> >>>>> Yes, what you described is what was happening, Mark. I didn't display
> >>>>> all of the code to the session methods, and I did re-read the
> >> in-coming
> >>>>> flowfile for different purposes than I had already read and written
> >> it.
> >>>>> So, I wasn't helpful enough. In the end, however, I had forgotten,
> >>>>> immediately after the call to session.putAllAttributes(), to update
> >> the
> >>>>> resulting flowfile for passing to session.transfer(). That solved it
> >> for
> >>>>> 1.1.2 which wasn't necessary for 1.13.2 or later versions. Being
> >>>>> helpful, the newer versions made me a spoiled, entitled child and I
> >> will
> >>>>> repent immediately.
> >>>>>
> >>>>> Thanks, guys! DevOps are happy they don't have to upgrade the
> >> customers
> >>>>> to NiFi 1.13.2. (In a way, I'm unhappy about that, but...).
> >>>>>
> >>>>> Best regards,
> >>>>>
> >>>>> Russ
> >>>>>
> >>>>> On 5/7/24 11:53, Mark Payne wrote:
> >>>>>> The call to session.putAttribute would throw an Exception because
> you
> >>>>>> provided an outdated version of the flowFile (did not capturing the
> >>>>>> result of calling session.write)
> >>>>>>
> >>>>>> Now, as NiFi matured, we found that:
> >>>>>> (a) for more complex processors that aren’t just a series of
> >> sequential
> >>>>>> steps it becomes difficult to manage all of that bookkeeping.
> >>>>>> (b) it was not intuitive to require this
> >>>>>> (c) the ProcessSession already had more or less what it needed in
> >> order
> >>>>>> to determine what the most up-to-date version of the FlowFile was.
> >>>>>>
> >>>>>> So we updated the ProcessSession to automatically grab the latest
> >>>>>> version of the FlowFile for these methods. But since you’re trying
> to
> >>>>>> run an old version, you’ll need to make sure that you capture all of
> >>>>>> those outputs and always keep track of the most recent version of a
> >>>>>> FlowFile.
> >>>>>
> >>>
> >>>
> >>
>
>

Re: ...not the most recent version of this FlowFile within this session...

Posted by Mark Payne <ma...@hotmail.com>.
Hey Dan,

Yessir, it absolutely would! Probably would be good to clean those up.

Thanks
-Mark


> On May 23, 2024, at 12:29 PM, Dan S <ds...@gmail.com> wrote:
> 
> Mark,
> In regards to your last comments
> 
>> Not really related, but on the note of things you may not realize, with
>> that code snippet :)
>> If you have access to update the processor mentioned here, you should
>> avoid using session.putAttribute many times.
>> Under the hood, in order to maintain object immutability it has to create
>> a new FlowFile object (and a new HashMap of all attributes!)
>> for every call to putAttribute. So if there are 100 attributes that match
>> that’s potentially a huge amount of garbage getting created.
> 
> 
> I noticed that some of the split processors SplitJson, SplitXml, and
> SplitAvro all have loops to create a new flow file for each split and it
> calls putAttribute more than once (in order to populate the split
> attributes FRAGMENT_ID, FRAGMENT_INDEX etc)  for each flow file created.
> Would this all suffer from "a huge amount of garbage getting created"
> since putAttribute is called multiple times for each iteration of the loop?
> 
> On Wed, May 8, 2024 at 5:27 PM Michael Moser <mo...@gmail.com> wrote:
> 
>> Oh yeah, I do love this behavior of ProcessSession.  And thanks for the
>> tip, it's easy to forget that there are efficiencies to be gained by using
>> different parts of an API.
>> 
>> -- Mke
>> 
>> 
>> On Wed, May 8, 2024 at 11:04 AM Mark Payne <ma...@hotmail.com> wrote:
>> 
>>> Yeah, that was something that kinda flew under the radar. Definitely
>>> improved the API.
>>> 
>>> Not really related, but on the note of things you may not realize, with
>>> that code snippet :)
>>> If you have access to update the processor mentioned here, you should
>>> avoid using session.putAttribute many times.
>>> Under the hood, in order to maintain object immutability it has to create
>>> a new FlowFile object (and a new HashMap of all attributes!)
>>> for every call to putAttribute. So if there are 100 attributes that match
>>> that’s potentially a huge amount of garbage getting created. Instead,
>>> you could just use:
>>> 
>>> ```
>>> final Map<String, String> attributes = new HashMap<>();
>>> flowFile.getAttributes().forEach( (key, value) -> {
>>>  if (key.startsWith(“foo”)) {
>>>    attributes.put(“original-“ + key, value);
>>>  }
>>> }
>>> 
>>> flowFIle = session.putAllAttributes(flowFile, attributes);
>>> ```
>>> 
>>> Thanks
>>> -Mark
>>> 
>>> 
>>> 
>>>> On May 8, 2024, at 10:30 AM, Michael Moser <mo...@gmail.com> wrote:
>>>> 
>>>> Wow, thanks for this information!  Just last week I saw code that
>>> modified
>>>> attributes in a stream:
>>>> 
>>>> flowFile.getAttributes().entrySet().stream().filter(e ->
>>>> e.getKey().startsWith("foo"))
>>>>   .forEach(e -> session.putAttribute(flowFile, "original-" + e.getKey,
>>>> e.getValue()));
>>>> 
>>>> and I wondered how that could possibly work since the return value of
>>>> session.putAttribute is ignored!  Now I know.
>>>> 
>>>> -- Mike
>>>> 
>>>> On Tue, May 7, 2024 at 3:02 PM Russell Bateman <ru...@windofkeltia.com>
>>>> wrote:
>>>> 
>>>>> Yes, what you described is what was happening, Mark. I didn't display
>>>>> all of the code to the session methods, and I did re-read the
>> in-coming
>>>>> flowfile for different purposes than I had already read and written
>> it.
>>>>> So, I wasn't helpful enough. In the end, however, I had forgotten,
>>>>> immediately after the call to session.putAllAttributes(), to update
>> the
>>>>> resulting flowfile for passing to session.transfer(). That solved it
>> for
>>>>> 1.1.2 which wasn't necessary for 1.13.2 or later versions. Being
>>>>> helpful, the newer versions made me a spoiled, entitled child and I
>> will
>>>>> repent immediately.
>>>>> 
>>>>> Thanks, guys! DevOps are happy they don't have to upgrade the
>> customers
>>>>> to NiFi 1.13.2. (In a way, I'm unhappy about that, but...).
>>>>> 
>>>>> Best regards,
>>>>> 
>>>>> Russ
>>>>> 
>>>>> On 5/7/24 11:53, Mark Payne wrote:
>>>>>> The call to session.putAttribute would throw an Exception because you
>>>>>> provided an outdated version of the flowFile (did not capturing the
>>>>>> result of calling session.write)
>>>>>> 
>>>>>> Now, as NiFi matured, we found that:
>>>>>> (a) for more complex processors that aren’t just a series of
>> sequential
>>>>>> steps it becomes difficult to manage all of that bookkeeping.
>>>>>> (b) it was not intuitive to require this
>>>>>> (c) the ProcessSession already had more or less what it needed in
>> order
>>>>>> to determine what the most up-to-date version of the FlowFile was.
>>>>>> 
>>>>>> So we updated the ProcessSession to automatically grab the latest
>>>>>> version of the FlowFile for these methods. But since you’re trying to
>>>>>> run an old version, you’ll need to make sure that you capture all of
>>>>>> those outputs and always keep track of the most recent version of a
>>>>>> FlowFile.
>>>>> 
>>> 
>>> 
>> 


Re: ...not the most recent version of this FlowFile within this session...

Posted by Dan S <ds...@gmail.com>.
Mark,
 In regards to your last comments

> Not really related, but on the note of things you may not realize, with
> that code snippet :)
> If you have access to update the processor mentioned here, you should
> avoid using session.putAttribute many times.
> Under the hood, in order to maintain object immutability it has to create
> a new FlowFile object (and a new HashMap of all attributes!)
> for every call to putAttribute. So if there are 100 attributes that match
> that’s potentially a huge amount of garbage getting created.


I noticed that some of the split processors SplitJson, SplitXml, and
SplitAvro all have loops to create a new flow file for each split and it
calls putAttribute more than once (in order to populate the split
attributes FRAGMENT_ID, FRAGMENT_INDEX etc)  for each flow file created.
Would this all suffer from "a huge amount of garbage getting created"
since putAttribute is called multiple times for each iteration of the loop?

On Wed, May 8, 2024 at 5:27 PM Michael Moser <mo...@gmail.com> wrote:

> Oh yeah, I do love this behavior of ProcessSession.  And thanks for the
> tip, it's easy to forget that there are efficiencies to be gained by using
> different parts of an API.
>
> -- Mke
>
>
> On Wed, May 8, 2024 at 11:04 AM Mark Payne <ma...@hotmail.com> wrote:
>
> > Yeah, that was something that kinda flew under the radar. Definitely
> > improved the API.
> >
> > Not really related, but on the note of things you may not realize, with
> > that code snippet :)
> > If you have access to update the processor mentioned here, you should
> > avoid using session.putAttribute many times.
> > Under the hood, in order to maintain object immutability it has to create
> > a new FlowFile object (and a new HashMap of all attributes!)
> > for every call to putAttribute. So if there are 100 attributes that match
> > that’s potentially a huge amount of garbage getting created. Instead,
> > you could just use:
> >
> > ```
> > final Map<String, String> attributes = new HashMap<>();
> > flowFile.getAttributes().forEach( (key, value) -> {
> >   if (key.startsWith(“foo”)) {
> >     attributes.put(“original-“ + key, value);
> >   }
> > }
> >
> > flowFIle = session.putAllAttributes(flowFile, attributes);
> > ```
> >
> > Thanks
> > -Mark
> >
> >
> >
> > > On May 8, 2024, at 10:30 AM, Michael Moser <mo...@gmail.com> wrote:
> > >
> > > Wow, thanks for this information!  Just last week I saw code that
> > modified
> > > attributes in a stream:
> > >
> > > flowFile.getAttributes().entrySet().stream().filter(e ->
> > > e.getKey().startsWith("foo"))
> > >    .forEach(e -> session.putAttribute(flowFile, "original-" + e.getKey,
> > > e.getValue()));
> > >
> > > and I wondered how that could possibly work since the return value of
> > > session.putAttribute is ignored!  Now I know.
> > >
> > > -- Mike
> > >
> > > On Tue, May 7, 2024 at 3:02 PM Russell Bateman <ru...@windofkeltia.com>
> > > wrote:
> > >
> > >> Yes, what you described is what was happening, Mark. I didn't display
> > >> all of the code to the session methods, and I did re-read the
> in-coming
> > >> flowfile for different purposes than I had already read and written
> it.
> > >> So, I wasn't helpful enough. In the end, however, I had forgotten,
> > >> immediately after the call to session.putAllAttributes(), to update
> the
> > >> resulting flowfile for passing to session.transfer(). That solved it
> for
> > >> 1.1.2 which wasn't necessary for 1.13.2 or later versions. Being
> > >> helpful, the newer versions made me a spoiled, entitled child and I
> will
> > >> repent immediately.
> > >>
> > >> Thanks, guys! DevOps are happy they don't have to upgrade the
> customers
> > >> to NiFi 1.13.2. (In a way, I'm unhappy about that, but...).
> > >>
> > >> Best regards,
> > >>
> > >> Russ
> > >>
> > >> On 5/7/24 11:53, Mark Payne wrote:
> > >>> The call to session.putAttribute would throw an Exception because you
> > >>> provided an outdated version of the flowFile (did not capturing the
> > >>> result of calling session.write)
> > >>>
> > >>> Now, as NiFi matured, we found that:
> > >>> (a) for more complex processors that aren’t just a series of
> sequential
> > >>> steps it becomes difficult to manage all of that bookkeeping.
> > >>> (b) it was not intuitive to require this
> > >>> (c) the ProcessSession already had more or less what it needed in
> order
> > >>> to determine what the most up-to-date version of the FlowFile was.
> > >>>
> > >>> So we updated the ProcessSession to automatically grab the latest
> > >>> version of the FlowFile for these methods. But since you’re trying to
> > >>> run an old version, you’ll need to make sure that you capture all of
> > >>> those outputs and always keep track of the most recent version of a
> > >>> FlowFile.
> > >>
> >
> >
>

Re: ...not the most recent version of this FlowFile within this session...

Posted by Michael Moser <mo...@gmail.com>.
Oh yeah, I do love this behavior of ProcessSession.  And thanks for the
tip, it's easy to forget that there are efficiencies to be gained by using
different parts of an API.

-- Mke


On Wed, May 8, 2024 at 11:04 AM Mark Payne <ma...@hotmail.com> wrote:

> Yeah, that was something that kinda flew under the radar. Definitely
> improved the API.
>
> Not really related, but on the note of things you may not realize, with
> that code snippet :)
> If you have access to update the processor mentioned here, you should
> avoid using session.putAttribute many times.
> Under the hood, in order to maintain object immutability it has to create
> a new FlowFile object (and a new HashMap of all attributes!)
> for every call to putAttribute. So if there are 100 attributes that match
> that’s potentially a huge amount of garbage getting created. Instead,
> you could just use:
>
> ```
> final Map<String, String> attributes = new HashMap<>();
> flowFile.getAttributes().forEach( (key, value) -> {
>   if (key.startsWith(“foo”)) {
>     attributes.put(“original-“ + key, value);
>   }
> }
>
> flowFIle = session.putAllAttributes(flowFile, attributes);
> ```
>
> Thanks
> -Mark
>
>
>
> > On May 8, 2024, at 10:30 AM, Michael Moser <mo...@gmail.com> wrote:
> >
> > Wow, thanks for this information!  Just last week I saw code that
> modified
> > attributes in a stream:
> >
> > flowFile.getAttributes().entrySet().stream().filter(e ->
> > e.getKey().startsWith("foo"))
> >    .forEach(e -> session.putAttribute(flowFile, "original-" + e.getKey,
> > e.getValue()));
> >
> > and I wondered how that could possibly work since the return value of
> > session.putAttribute is ignored!  Now I know.
> >
> > -- Mike
> >
> > On Tue, May 7, 2024 at 3:02 PM Russell Bateman <ru...@windofkeltia.com>
> > wrote:
> >
> >> Yes, what you described is what was happening, Mark. I didn't display
> >> all of the code to the session methods, and I did re-read the in-coming
> >> flowfile for different purposes than I had already read and written it.
> >> So, I wasn't helpful enough. In the end, however, I had forgotten,
> >> immediately after the call to session.putAllAttributes(), to update the
> >> resulting flowfile for passing to session.transfer(). That solved it for
> >> 1.1.2 which wasn't necessary for 1.13.2 or later versions. Being
> >> helpful, the newer versions made me a spoiled, entitled child and I will
> >> repent immediately.
> >>
> >> Thanks, guys! DevOps are happy they don't have to upgrade the customers
> >> to NiFi 1.13.2. (In a way, I'm unhappy about that, but...).
> >>
> >> Best regards,
> >>
> >> Russ
> >>
> >> On 5/7/24 11:53, Mark Payne wrote:
> >>> The call to session.putAttribute would throw an Exception because you
> >>> provided an outdated version of the flowFile (did not capturing the
> >>> result of calling session.write)
> >>>
> >>> Now, as NiFi matured, we found that:
> >>> (a) for more complex processors that aren’t just a series of sequential
> >>> steps it becomes difficult to manage all of that bookkeeping.
> >>> (b) it was not intuitive to require this
> >>> (c) the ProcessSession already had more or less what it needed in order
> >>> to determine what the most up-to-date version of the FlowFile was.
> >>>
> >>> So we updated the ProcessSession to automatically grab the latest
> >>> version of the FlowFile for these methods. But since you’re trying to
> >>> run an old version, you’ll need to make sure that you capture all of
> >>> those outputs and always keep track of the most recent version of a
> >>> FlowFile.
> >>
>
>

Re: ...not the most recent version of this FlowFile within this session...

Posted by Mark Payne <ma...@hotmail.com>.
Yeah, that was something that kinda flew under the radar. Definitely improved the API.

Not really related, but on the note of things you may not realize, with that code snippet :)
If you have access to update the processor mentioned here, you should avoid using session.putAttribute many times.
Under the hood, in order to maintain object immutability it has to create a new FlowFile object (and a new HashMap of all attributes!)
for every call to putAttribute. So if there are 100 attributes that match that’s potentially a huge amount of garbage getting created. Instead,
you could just use:

```
final Map<String, String> attributes = new HashMap<>();
flowFile.getAttributes().forEach( (key, value) -> {
  if (key.startsWith(“foo”)) {
    attributes.put(“original-“ + key, value);
  }
}

flowFIle = session.putAllAttributes(flowFile, attributes);
```

Thanks
-Mark



> On May 8, 2024, at 10:30 AM, Michael Moser <mo...@gmail.com> wrote:
> 
> Wow, thanks for this information!  Just last week I saw code that modified
> attributes in a stream:
> 
> flowFile.getAttributes().entrySet().stream().filter(e ->
> e.getKey().startsWith("foo"))
>    .forEach(e -> session.putAttribute(flowFile, "original-" + e.getKey,
> e.getValue()));
> 
> and I wondered how that could possibly work since the return value of
> session.putAttribute is ignored!  Now I know.
> 
> -- Mike
> 
> On Tue, May 7, 2024 at 3:02 PM Russell Bateman <ru...@windofkeltia.com>
> wrote:
> 
>> Yes, what you described is what was happening, Mark. I didn't display
>> all of the code to the session methods, and I did re-read the in-coming
>> flowfile for different purposes than I had already read and written it.
>> So, I wasn't helpful enough. In the end, however, I had forgotten,
>> immediately after the call to session.putAllAttributes(), to update the
>> resulting flowfile for passing to session.transfer(). That solved it for
>> 1.1.2 which wasn't necessary for 1.13.2 or later versions. Being
>> helpful, the newer versions made me a spoiled, entitled child and I will
>> repent immediately.
>> 
>> Thanks, guys! DevOps are happy they don't have to upgrade the customers
>> to NiFi 1.13.2. (In a way, I'm unhappy about that, but...).
>> 
>> Best regards,
>> 
>> Russ
>> 
>> On 5/7/24 11:53, Mark Payne wrote:
>>> The call to session.putAttribute would throw an Exception because you
>>> provided an outdated version of the flowFile (did not capturing the
>>> result of calling session.write)
>>> 
>>> Now, as NiFi matured, we found that:
>>> (a) for more complex processors that aren’t just a series of sequential
>>> steps it becomes difficult to manage all of that bookkeeping.
>>> (b) it was not intuitive to require this
>>> (c) the ProcessSession already had more or less what it needed in order
>>> to determine what the most up-to-date version of the FlowFile was.
>>> 
>>> So we updated the ProcessSession to automatically grab the latest
>>> version of the FlowFile for these methods. But since you’re trying to
>>> run an old version, you’ll need to make sure that you capture all of
>>> those outputs and always keep track of the most recent version of a
>>> FlowFile.
>> 


Re: ...not the most recent version of this FlowFile within this session...

Posted by Michael Moser <mo...@gmail.com>.
Wow, thanks for this information!  Just last week I saw code that modified
attributes in a stream:

flowFile.getAttributes().entrySet().stream().filter(e ->
e.getKey().startsWith("foo"))
    .forEach(e -> session.putAttribute(flowFile, "original-" + e.getKey,
e.getValue()));

and I wondered how that could possibly work since the return value of
session.putAttribute is ignored!  Now I know.

-- Mike

On Tue, May 7, 2024 at 3:02 PM Russell Bateman <ru...@windofkeltia.com>
wrote:

> Yes, what you described is what was happening, Mark. I didn't display
> all of the code to the session methods, and I did re-read the in-coming
> flowfile for different purposes than I had already read and written it.
> So, I wasn't helpful enough. In the end, however, I had forgotten,
> immediately after the call to session.putAllAttributes(), to update the
> resulting flowfile for passing to session.transfer(). That solved it for
> 1.1.2 which wasn't necessary for 1.13.2 or later versions. Being
> helpful, the newer versions made me a spoiled, entitled child and I will
> repent immediately.
>
> Thanks, guys! DevOps are happy they don't have to upgrade the customers
> to NiFi 1.13.2. (In a way, I'm unhappy about that, but...).
>
> Best regards,
>
> Russ
>
> On 5/7/24 11:53, Mark Payne wrote:
> > The call to session.putAttribute would throw an Exception because you
> > provided an outdated version of the flowFile (did not capturing the
> > result of calling session.write)
> >
> > Now, as NiFi matured, we found that:
> > (a) for more complex processors that aren’t just a series of sequential
> > steps it becomes difficult to manage all of that bookkeeping.
> > (b) it was not intuitive to require this
> > (c) the ProcessSession already had more or less what it needed in order
> > to determine what the most up-to-date version of the FlowFile was.
> >
> > So we updated the ProcessSession to automatically grab the latest
> > version of the FlowFile for these methods. But since you’re trying to
> > run an old version, you’ll need to make sure that you capture all of
> > those outputs and always keep track of the most recent version of a
> > FlowFile.
>

Re: ...not the most recent version of this FlowFile within this session...

Posted by Russell Bateman <ru...@windofkeltia.com>.
Yes, what you described is what was happening, Mark. I didn't display 
all of the code to the session methods, and I did re-read the in-coming 
flowfile for different purposes than I had already read and written it. 
So, I wasn't helpful enough. In the end, however, I had forgotten, 
immediately after the call to session.putAllAttributes(), to update the 
resulting flowfile for passing to session.transfer(). That solved it for 
1.1.2 which wasn't necessary for 1.13.2 or later versions. Being 
helpful, the newer versions made me a spoiled, entitled child and I will 
repent immediately.

Thanks, guys! DevOps are happy they don't have to upgrade the customers 
to NiFi 1.13.2. (In a way, I'm unhappy about that, but...).

Best regards,

Russ

On 5/7/24 11:53, Mark Payne wrote:
> The call to session.putAttribute would throw an Exception because you
> provided an outdated version of the flowFile (did not capturing the
> result of calling session.write)
>
> Now, as NiFi matured, we found that:
> (a) for more complex processors that aren’t just a series of sequential
> steps it becomes difficult to manage all of that bookkeeping.
> (b) it was not intuitive to require this
> (c) the ProcessSession already had more or less what it needed in order
> to determine what the most up-to-date version of the FlowFile was.
>
> So we updated the ProcessSession to automatically grab the latest
> version of the FlowFile for these methods. But since you’re trying to
> run an old version, you’ll need to make sure that you capture all of
> those outputs and always keep track of the most recent version of a
> FlowFile.

Re: ...not the most recent version of this FlowFile within this session...

Posted by Mark Payne <ma...@hotmail.com>.
Russell,

May not be too bad to fix :)

Any time you use a method on ProcessSession to modify a FlowFile in some way (such as putAttribute, putAllAttributes, write, removeAttribute, removeAllAttributes, etc.) the ProcessSession returns to you a new FlowFile. So you *should* write code such as:

```
FlowFile flowFile = session.get();
If (flowFile == null) {
  return;
}

flowFile = session.write(flowFile, new OutputStreamCallback() {
…
});

flowFile = session.putAttribute(flowFile, “greeting”, “hello”);

session.transfer(flowFile);
```

Note here that calls to session.write, session.putAttribute are capturing the return value and holding on to that.
That was necessary in older versions. If you instead wrote:

```
flowFile = session.get();
session.write(flowFile, new OutputStreamCallback() {
…
});

session.putAttribute(flowFile, “greeting”, “hello”);
```

The call to session.putAttribute would throw an Exception because you provided an outdated version of the flowFile (did not capturing the result of calling session.write)

Now, as NiFi matured, we found that:
(a) for more complex processors that aren’t just a series of sequential steps it becomes difficult to manage all of that bookkeeping.
(b) it was not intuitive to require this
(c) the ProcessSession already had more or less what it needed in order to determine what the most up-to-date version of the FlowFile was.

So we updated the ProcessSession to automatically grab the latest version of the FlowFile for these methods. But since you’re trying to run an old version, you’ll need to make sure that you capture all of those outputs and always keep track of the most recent version of a FlowFile.

You’ll also need to ensure that you’re not calling any methods, etc. that didn’t exist in the 1.1.2 API :) 

Thanks
-Mark




> On May 7, 2024, at 1:38 PM, Russell Bateman <ru...@windofkeltia.com> wrote:
> 
> In /pom.xml/ I specify using NiFi framework libraries from 1.13.2.
> 
> There are peculiar reasons (that we are trying to fix) that inhibit us from moving forward as all of our customer machines are running 1.1.2. (Don't shoot me, I'm not DevOps, but just the guy who writes custom processors.)
> 
> I have custom processor code that, built as noted using 1.13.2, runs _perfectly_ on any version of NiFi from 1.13.2 to 1.25.0. When loaded on 1.1.2, one processor fails to process a very simple flowfile with the error message below.
> 
> I'm wondering if there's a dance I can do to tweak one or more of what I suspect are the calls where this is happening (in onTrigger()):
> 
>   FlowFile result = session.*write*( flowfile, new StreamCallback()
>                      session.*read*( flowfile, new InputStreamCallback()
>                      session.*removeAllAttributes*( result, keys );
>                      session.*putAllAttributes*( result, attributes );
>                      session.*transfer*( result, SUCCESS );
> 
> 
> Error message:
> 
> 10:32:47 MDTERRORc78b2d48-79a6-39ae-1454-4ee57bbf8928
> ExcerptWarehouseResponse[id=c78b2d48-79a6-39ae-1454-4ee57bbf8928] \
> failed to process session due to
> org.apache.nifi.processor.exception.FlowFileHandlingException: \
> StandardFlowFileRecord[uuid=cfb2a124-4b07-46bd-a70e-3848502b3df1,claim=StandardContentClaim \
> [resourceClaim=StandardResourceClaim[id=1715094869203-1, container=default, section=1], offset=0, length=1077766],offset=0,name=30575811037248611,size=1077766] \
> is not the most recent version of this FlowFile within this session (StandardProcessSession[id=3923]): \
> org.apache.nifi.processor.exception.FlowFileHandlingException: StandardFlowFileRecord[uuid=cfb2a124-4b07-46bd-a70e-3848502b3df1,claim=StandardContentClaim \
> [resourceClaim=StandardResourceClaim[id=1715094869203-1, container=default, section=1], offset=0, length=1077766],offset=0,name=30575811037248611,size=1077766] \
> is not the most recent version of this FlowFile within this session (StandardProcessSession[id=3923])
> 
> 10:32:47 MDTWARNINGc78b2d48-79a6-39ae-1454-4ee57bbf8928
> ExcerptWarehouseResponse[id=c78b2d48-79a6-39ae-1454-4ee57bbf8928] Processor Administratively Yielded for 1 sec due to processing failure
> 
> 
> I know this is old stuff and I'm prepared be told that I'm out of luck or that the solution is unfeasible or would take too much effort to dig through. I'm just hoping that someone remembers something quick off the top of the head and can give advice.
> 
> Thanks,
> 
> Russ
> 
> 
> 
> 
> 


Re: ...not the most recent version of this FlowFile within this session...

Posted by Joe Witt <jo...@gmail.com>.
Russ

FlowFile result = session.*write*( flowfile, new StreamCallback()
                       session.*read*( flowfile, new InputStreamCallback()
                       session.*removeAllAttributes*( result, keys );
                       session.*putAllAttributes*( result, attributes );
                       session.*transfer*( result, SUCCESS );

The session.read passes in the same original flowfile reference that you
send in to the write.  I'm guessing that is not what you intended.  You
probably want

session.read(result, newInputStream....


Thanks
Joe

On Tue, May 7, 2024 at 10:38 AM Russell Bateman <ru...@windofkeltia.com>
wrote:

> In /pom.xml/ I specify using NiFi framework libraries from 1.13.2.
>
> There are peculiar reasons (that we are trying to fix) that inhibit us
> from moving forward as all of our customer machines are running 1.1.2.
> (Don't shoot me, I'm not DevOps, but just the guy who writes custom
> processors.)
>
> I have custom processor code that, built as noted using 1.13.2, runs
> _perfectly_ on any version of NiFi from 1.13.2 to 1.25.0. When loaded on
> 1.1.2, one processor fails to process a very simple flowfile with the
> error message below.
>
> I'm wondering if there's a dance I can do to tweak one or more of what I
> suspect are the calls where this is happening (in onTrigger()):
>
>     FlowFile result = session.*write*( flowfile, new StreamCallback()
>                        session.*read*( flowfile, new InputStreamCallback()
>                        session.*removeAllAttributes*( result, keys );
>                        session.*putAllAttributes*( result, attributes );
>                        session.*transfer*( result, SUCCESS );
>
>
> Error message:
>
> 10:32:47 MDTERRORc78b2d48-79a6-39ae-1454-4ee57bbf8928
> ExcerptWarehouseResponse[id=c78b2d48-79a6-39ae-1454-4ee57bbf8928] \
> failed to process session due to
> org.apache.nifi.processor.exception.FlowFileHandlingException: \
> StandardFlowFileRecord[uuid=cfb2a124-4b07-46bd-a70e-3848502b3df1,claim=StandardContentClaim
> \
> [resourceClaim=StandardResourceClaim[id=1715094869203-1,
> container=default, section=1], offset=0,
> length=1077766],offset=0,name=30575811037248611,size=1077766] \
> is not the most recent version of this FlowFile within this session
> (StandardProcessSession[id=3923]): \
> org.apache.nifi.processor.exception.FlowFileHandlingException:
> StandardFlowFileRecord[uuid=cfb2a124-4b07-46bd-a70e-3848502b3df1,claim=StandardContentClaim
> \
> [resourceClaim=StandardResourceClaim[id=1715094869203-1,
> container=default, section=1], offset=0,
> length=1077766],offset=0,name=30575811037248611,size=1077766] \
> is not the most recent version of this FlowFile within this session
> (StandardProcessSession[id=3923])
>
> 10:32:47 MDTWARNINGc78b2d48-79a6-39ae-1454-4ee57bbf8928
> ExcerptWarehouseResponse[id=c78b2d48-79a6-39ae-1454-4ee57bbf8928]
> Processor Administratively Yielded for 1 sec due to processing failure
>
>
> I know this is old stuff and I'm prepared be told that I'm out of luck
> or that the solution is unfeasible or would take too much effort to dig
> through. I'm just hoping that someone remembers something quick off the
> top of the head and can give advice.
>
> Thanks,
>
> Russ
>
>
>
>
>
>