You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@nifi.apache.org by Joe Gresock <jg...@gmail.com> on 2020/02/10 16:15:40 UTC
NiFi 1.11.1: EOFException in FlowFileAccessInputStream
I don't know if there is something wrong with my Kubernetes-ized deployment
of NiFi, but I'm seeing the following error crop up a lot in my NiFi 1.11.1
deployment. I can see the flow file attributes in "List Queue" for flow
files affected by this problem, but the content claim is missing.
It's not easily reproducible, but I've been running data constantly through
the cluster for a few days, and every 2-4 hours or so, this problem appears
in one of my processors (it's not always the same processors). The error
only goes away when I drain the relevant queue, but obviously that's not an
option in production.
Any thoughts on what might be causing intermittent truncations in the
content_repository?
2020-02-10 10:38:57,205 ERROR [Timer-Driven Process Thread-6]
o.a.nifi.processors.standard.ReplaceText
ReplaceText[id=1abd1d49-0170-1000-ffff-ffffda59ecb2]
ReplaceText[id=1abd1d49-0170-1000-ffff-ffffda59ecb2] fa
iled to process session due to
org.apache.nifi.processor.exception.FlowFileAccessException: Could not read
from
StandardFlowFileRecord[uuid=b0ca5649-ca8f-4a99-88df-70e1143c5f1e,claim=StandardContentClaim
[resourceClaim=StandardResourceClaim[id=1581318452649-212433,
container=default, section=465], offset=0,
length=6401249],offset=2199110,name=90e47e27f806d46b2e190b462db929441993d166-ED,size=318289];
Processor Administratively Yielded for 1 sec:
org.apache.nifi.processor.exception.FlowFileAccessException: Could not read
from
StandardFlowFileRecord[uuid=b0ca5649-ca8f-4a99-88df-70e1143c5f1e,claim=StandardContentClaim
[resourceClaim=StandardResourceClaim[id=1581318452649-212433,
container=default, section=465], offset=0,
length=6401249],offset=2199110,name=90e47e27f806d46b2e190b462db929441993d166-ED,size=318289]
org.apache.nifi.processor.exception.FlowFileAccessException: Could not read
from
StandardFlowFileRecord[uuid=b0ca5649-ca8f-4a99-88df-70e1143c5f1e,claim=StandardContentClaim
[resourceClaim=StandardResourceClaim[id=1581318452649-212433,
container=default, section=465], offset=0,
length=6401249],offset=2199110,name=90e47e27f806d46b2e190b462db929441993d166-ED,size=318289]
at
org.apache.nifi.controller.repository.io.FlowFileAccessInputStream.read(FlowFileAccessInputStream.java:93)
at
org.apache.nifi.controller.repository.io.TaskTerminationInputStream.read(TaskTerminationInputStream.java:68)
at
org.apache.nifi.stream.io.StreamUtils.fillBuffer(StreamUtils.java:89)
at
org.apache.nifi.processors.standard.ReplaceText$RegexReplace$2.process(ReplaceText.java:532)
at
org.apache.nifi.controller.repository.StandardProcessSession.read(StandardProcessSession.java:2324)
at
org.apache.nifi.controller.repository.StandardProcessSession.read(StandardProcessSession.java:2292)
at
org.apache.nifi.controller.repository.BatchingSessionFactory$HighThroughputSession.read(BatchingSessionFactory.java:192)
at
org.apache.nifi.processors.standard.ReplaceText$RegexReplace.replace(ReplaceText.java:529)
at
org.apache.nifi.processors.standard.ReplaceText.onTrigger(ReplaceText.java:316)
at
org.apache.nifi.processor.AbstractProcessor.onTrigger(AbstractProcessor.java:27)
at
org.apache.nifi.controller.StandardProcessorNode.onTrigger(StandardProcessorNode.java:1176)
at
org.apache.nifi.controller.tasks.ConnectableTask.invoke(ConnectableTask.java:213)
at
org.apache.nifi.controller.scheduling.TimerDrivenSchedulingAgent$1.run(TimerDrivenSchedulingAgent.java:117)
at org.apache.nifi.engine.FlowEngine$2.run(FlowEngine.java:110)
at
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Caused by: java.io.EOFException: null
at org.apache.nifi.stream.io.StreamUtils.skip(StreamUtils.java:270)
at
org.apache.nifi.controller.repository.io.ContentClaimInputStream.formDelegate(ContentClaimInputStream.java:155)
at
org.apache.nifi.controller.repository.io.ContentClaimInputStream.getDelegate(ContentClaimInputStream.java:51)
at
org.apache.nifi.controller.repository.io.ContentClaimInputStream.read(ContentClaimInputStream.java:89)
at
org.apache.nifi.controller.repository.io.DisableOnCloseInputStream.read(DisableOnCloseInputStream.java:49)
at
org.apache.nifi.controller.repository.io.LimitedInputStream.read(LimitedInputStream.java:86)
at
org.apache.nifi.controller.repository.io.DisableOnCloseInputStream.read(DisableOnCloseInputStream.java:49)
at
org.apache.nifi.stream.io.ByteCountingInputStream.read(ByteCountingInputStream.java:52)
at java.io.FilterInputStream.read(FilterInputStream.java:133)
at
org.apache.nifi.controller.repository.io.FlowFileAccessInputStream.read(FlowFileAccessInputStream.java:82)
... 20 common frames omitted
Joe Gresock
Re: NiFi 1.11.1: EOFException in FlowFileAccessInputStream
Posted by Joe Gresock <jg...@gmail.com>.
Interestingly, this did not resolve the issue. After another pod restart,
I have the same error at a MergeContent processor. I verified that these
settings are all true.
org.apache.nifi.processor.exception.FlowFileAccessException: Could not read
from
StandardFlowFileRecord[uuid=b38e063f-23a4-47f6-bfff-81ac45277c38,claim=StandardContentClaim
[resourceClaim=StandardResourceClaim[id=1581683172340-190097,
container=default, section=657], offset=0,
length=608],offset=464,name=94952c949967692e3bd9bddda4ea3ba26274ca22-ED,size=40]
Underlying exception is:
Caused by: java.io.EOFException: null
at org.apache.nifi.stream.io.StreamUtils.skip(StreamUtils.java:270)
at
org.apache.nifi.controller.repository.io.ContentClaimInputStream.formDelegate(ContentClaimInputStream.java:155)
at
org.apache.nifi.controller.repository.io.ContentClaimInputStream.getDelegate(ContentClaimInputStream.java:51)
at
org.apache.nifi.controller.repository.io.ContentClaimInputStream.read(ContentClaimInputStream.java:89)
at
org.apache.nifi.controller.repository.io.LimitedInputStream.read(LimitedInputStream.java:86)
at
org.apache.nifi.controller.repository.io.DisableOnCloseInputStream.read(DisableOnCloseInputStream.java:49)
at
org.apache.nifi.stream.io.ByteCountingInputStream.read(ByteCountingInputStream.java:52)
at java.io.FilterInputStream.read(FilterInputStream.java:133)
at
org.apache.nifi.controller.repository.io.FlowFileAccessInputStream.read(FlowFileAccessInputStream.java:82)
... 22 common frames omitted
On Fri, Feb 14, 2020 at 2:14 PM Joe Gresock <jg...@gmail.com> wrote:
> Good idea. I've made the change, and will report back if I see any more
> issues.
>
> On Fri, Feb 14, 2020 at 2:04 PM Joe Witt <jo...@gmail.com> wrote:
>
>> No it should not. But if you want to rule out underlying storage not
>> getting the writes actually written you can use
>>
>> nifi.flowfile.repository.always.sync=true
>> nifi.content.repository.always.sync=true
>> nifi.provenance.repository.always.sync=true
>>
>> That will impact performance as it means we force sync to disk rather than
>> taking advantage of write ahead. But try that and see if that helps in
>> your configuration.
>>
>> Thanks
>
>
--
Be on your guard; stand firm in the faith; be courageous; be strong. Do
everything in love. -*1 Corinthians 16:13-14*
Re: NiFi 1.11.1: EOFException in FlowFileAccessInputStream
Posted by Joe Gresock <jg...@gmail.com>.
Good idea. I've made the change, and will report back if I see any more
issues.
On Fri, Feb 14, 2020 at 2:04 PM Joe Witt <jo...@gmail.com> wrote:
> No it should not. But if you want to rule out underlying storage not
> getting the writes actually written you can use
>
> nifi.flowfile.repository.always.sync=true
> nifi.content.repository.always.sync=true
> nifi.provenance.repository.always.sync=true
>
> That will impact performance as it means we force sync to disk rather than
> taking advantage of write ahead. But try that and see if that helps in
> your configuration.
>
> Thanks
Re: NiFi 1.11.1: EOFException in FlowFileAccessInputStream
Posted by Joe Witt <jo...@gmail.com>.
No it should not. But if you want to rule out underlying storage not
getting the writes actually written you can use
nifi.flowfile.repository.always.sync=true
nifi.content.repository.always.sync=true
nifi.provenance.repository.always.sync=true
That will impact performance as it means we force sync to disk rather than
taking advantage of write ahead. But try that and see if that helps in
your configuration.
Thanks
On Thu, Feb 13, 2020 at 2:33 PM Joe Gresock <jg...@gmail.com> wrote:
> Another data point.. I noticed that the only nodes that seem to have this
> error are ones with recent pod restarts. Perhaps this is just a risk if
> the nifi process suddenly restarts?
>
> In the log, I only see normal log statements, and then suddenly:
> 2020-02-13 18:20:06,810 INFO [Heartbeat Monitor Thread-1]
> o.a.n.c.c.h.AbstractHeartbeatMonito2020-02-13 18:23:50,779 INFO [main]
> org.apache.nifi.NiFi Launching NiFi...
>
> The disk is definitely not filling up, as I have all of my disks
> monitored. Perhaps an OOM could be the culprit, but I don't see anything
> in the logs.
>
> In normal operation, should NiFi be at risk of truncating content claims in
> case of the JVM suddenly stopping?
>
> On Thu, Feb 13, 2020 at 9:18 AM Joe Gresock <jg...@gmail.com> wrote:
>
> > Sure, sorry for the delay.
> >
> > It's an AWS EBS persistent volume. Here are the PV and storage class
> > objects. It's single-zone (us-east-1d):
> >
> > apiVersion: v1
> > kind: PersistentVolume
> > metadata:
> > annotations:
> > kubernetes.io/createdby: aws-ebs-dynamic-provisioner
> > pv.kubernetes.io/bound-by-controller: "yes"
> > pv.kubernetes.io/provisioned-by: kubernetes.io/aws-ebs
> > labels:
> > failure-domain.beta.kubernetes.io/region: us-east-1
> > failure-domain.beta.kubernetes.io/zone: us-east-1d
> > name: pvc-abb1e679-523d-4ef7-a42e-cb42d8b8da0d
> > spec:
> > accessModes:
> > - ReadWriteOnce
> > awsElasticBlockStore:
> > fsType: ext4
> > volumeID: aws://us-east-1d/vol-<instanceId> # Redacting some
> details
> > capacity:
> > storage: 150Gi
> > claimRef:
> > apiVersion: v1
> > kind: PersistentVolumeClaim
> > name: content-repository-nifi-4
> > namespace: default
> > resourceVersion: "7190127"
> > uid: abb1e679-523d-4ef7-a42e-cb42d8b8da0d
> > mountOptions:
> > - debug
> > nodeAffinity:
> > required:
> > nodeSelectorTerms:
> > - matchExpressions:
> > - key: failure-domain.beta.kubernetes.io/region
> > operator: In
> > values:
> > - us-east-1
> > - key: failure-domain.beta.kubernetes.io/zone
> > operator: In
> > values:
> > - us-east-1d
> > persistentVolumeReclaimPolicy: Retain
> > storageClassName: gp2
> > volumeMode: Filesystem
> > ---
> > apiVersion: storage.k8s.io/v1
> > kind: StorageClass
> > metadata:
> > annotations:
> > kubectl.kubernetes.io/last-applied-configuration: |
> > {"allowVolumeExpansion":true,"apiVersion":"storage.k8s.io/v1
> > ","kind":"StorageClass","metadata":{"annotations":{"
> > storageclass.kubernetes.io/is-default-class
> > ":"true"},"labels":{"addonmanager.kubernetes.io
> .mode":"EnsureExists"},"name":"gp2"},"mountOptions":["debug"],"parameters":{"type":"gp2"},"provisioner":"
> > kubernetes.io/aws-ebs
> > ","reclaimPolicy":"Retain","volumeBindingMode":"WaitForFirstConsumer"}
> > storageclass.kubernetes.io/is-default-class: "true"
> > creationTimestamp: "2020-01-03T12:30:49Z"
> > labels:
> > addonmanager.kubernetes.io.mode: EnsureExists
> > name: gp2
> > resourceVersion: "126035"
> > selfLink: /apis/storage.k8s.io/v1/storageclasses/gp2
> > uid: b91b9233-cd1b-4850-b0e1-bb984c4a31c5
> > allowVolumeExpansion: true
> > mountOptions:
> > - debug
> > parameters:
> > type: gp2
> > provisioner: kubernetes.io/aws-ebs
> > reclaimPolicy: Retain
> > volumeBindingMode: WaitForFirstConsumer
> >
> > On Tue, Feb 11, 2020 at 9:20 AM Endre Kovacs
> > <an...@protonmail.com.invalid> wrote:
> >
> >> The more i think about this, the more i believe that the PV had to be
> >> provisioned as some kind of network attached storage, and the exception
> had
> >> to be happening because of intermittent network issue for this NAS.
> >>
> >>
> >>
> >>
> >> Sent with ProtonMail Secure Email.
> >>
> >> ‐‐‐‐‐‐‐ Original Message ‐‐‐‐‐‐‐
> >> On Monday, February 10, 2020 8:41 PM, Endre Kovacs <
> >> andrewsmith87@protonmail.com> wrote:
> >>
> >> > Hi,
> >> >
> >> > What cloud are you running this on?
> >> > Kubernetes version?
> >> >
> >> > Can you share the storageClass you use for creating the repo PVCs?
> >> >
> >> > what is the topology of the cluster? Muli zone?
> >> >
> >> > Best regards,
> >> > Endre
> >> >
> >> > Sent with ProtonMail Secure Email.
> >> >
> >> > ‐‐‐‐‐‐‐ Original Message ‐‐‐‐‐‐‐
> >> > On Monday, February 10, 2020 7:41 PM, Shawn Weeks
> >> sweeks@weeksconsulting.us wrote:
> >> >
> >> > > The more I look at the error the more it looks like the container
> >> couldn't access the persistence volume for some period of time. Can you
> >> share your yaml configuration minus sensitive stuff and what environment
> >> your deploying K8s on, aka Azure, ECS, Rancher, Openshift, etc.
> >> > > Thanks
> >> > > Shawn
> >> > > On 2/10/20, 12:27 PM, "Shawn Weeks" sweeks@weeksconsulting.us
> wrote:
> >> > > I've been testing out NiFi on K8s quite a bit lately and might be
> >> able to help. What are you using for your persistent volume, this kinda
> >> sounds like NiFi can't always access data in the content_repository
> folder?
> >> > > Thanks
> >> > > Shawn
> >> > > On 2/10/20, 11:53 AM, "Joe Witt" joe.witt@gmail.com wrote:
> >> > > ...sent a little too quick. Also seems like if it is related to your
> >> k8s
> >> > > process that the issue would occur on older nifi releases too. So
> that
> >> > > might be an interesting test for you.
> >> > > On Mon, Feb 10, 2020 at 9:52 AM Joe Witt joe.witt@gmail.com wrote:
> >> > >
> >> > > > Joe
> >> > >
> >> > > >
> >> > >
> >> > > > If you can replicate this in a non k8s environment that
> >> would likely make
> >> > >
> >> > > > resolution easier. Far too many variables/questions at
> >> this stage.
> >> > >
> >> > > >
> >> > >
> >> > > > Thanks
> >> > >
> >> > > >
> >> > >
> >> > > > On Mon, Feb 10, 2020 at 8:15 AM Joe Gresock <
> >> jgresock@gmail.com> wrote:
> >> > >
> >> > > >
> >> > >
> >> > > >> I don't know if there is something wrong with my
> >> Kubernetes-ized
> >> > >
> >> > > >> deployment
> >> > >
> >> > > >> of NiFi, but I'm seeing the following error crop up a lot
> >> in my NiFi
> >> > >
> >> > > >> 1.11.1
> >> > >
> >> > > >> deployment. I can see the flow file attributes in "List
> >> Queue" for flow
> >> > >
> >> > > >> files affected by this problem, but the content claim is
> >> missing.
> >> > >
> >> > > >>
> >> > >
> >> > > >> It's not easily reproducible, but I've been running data
> >> constantly
> >> > >
> >> > > >> through
> >> > >
> >> > > >> the cluster for a few days, and every 2-4 hours or so,
> >> this problem
> >> > >
> >> > > >> appears
> >> > >
> >> > > >> in one of my processors (it's not always the same
> >> processors). The error
> >> > >
> >> > > >> only goes away when I drain the relevant queue, but
> >> obviously that's not
> >> > >
> >> > > >> an
> >> > >
> >> > > >> option in production.
> >> > >
> >> > > >>
> >> > >
> >> > > >> Any thoughts on what might be causing intermittent
> >> truncations in the
> >> > >
> >> > > >> content_repository?
> >> > >
> >> > > >>
> >> > >
> >> > > >> 2020-02-10 10:38:57,205 ERROR [Timer-Driven Process
> >> Thread-6]
> >> > >
> >> > > >> o.a.nifi.processors.standard.ReplaceText
> >> > >
> >> > > >> ReplaceText[id=1abd1d49-0170-1000-ffff-ffffda59ecb2]
> >> > >
> >> > > >> ReplaceText[id=1abd1d49-0170-1000-ffff-ffffda59ecb2] fa
> >> > >
> >> > > >> iled to process session due to
> >> > >
> >> > > >>
> >> org.apache.nifi.processor.exception.FlowFileAccessException: Could not
> >> > >
> >> > > >> read
> >> > >
> >> > > >> from
> >> > >
> >> > > >>
> >> > >
> >> > > >>
> >>
> StandardFlowFileRecord[uuid=b0ca5649-ca8f-4a99-88df-70e1143c5f1e,claim=StandardContentClaim
> >> > >
> >> > > >>
> >> [resourceClaim=StandardResourceClaim[id=1581318452649-212433,
> >> > >
> >> > > >> container=default, section=465], offset=0,
> >> > >
> >> > > >>
> >> > >
> >> > > >>
> >>
> length=6401249],offset=2199110,name=90e47e27f806d46b2e190b462db929441993d166-ED,size=318289];
> >> > >
> >> > > >> Processor Administratively Yielded for 1 sec:
> >> > >
> >> > > >>
> >> org.apache.nifi.processor.exception.FlowFileAccessException: Could not
> >> > >
> >> > > >> read
> >> > >
> >> > > >> from
> >> > >
> >> > > >>
> >> > >
> >> > > >>
> >>
> StandardFlowFileRecord[uuid=b0ca5649-ca8f-4a99-88df-70e1143c5f1e,claim=StandardContentClaim
> >> > >
> >> > > >>
> >> [resourceClaim=StandardResourceClaim[id=1581318452649-212433,
> >> > >
> >> > > >> container=default, section=465], offset=0,
> >> > >
> >> > > >>
> >> > >
> >> > > >>
> >>
> length=6401249],offset=2199110,name=90e47e27f806d46b2e190b462db929441993d166-ED,size=318289]
> >> > >
> >> > > >>
> >> org.apache.nifi.processor.exception.FlowFileAccessException: Could not
> >> > >
> >> > > >> read
> >> > >
> >> > > >> from
> >> > >
> >> > > >>
> >> > >
> >> > > >>
> >>
> StandardFlowFileRecord[uuid=b0ca5649-ca8f-4a99-88df-70e1143c5f1e,claim=StandardContentClaim
> >> > >
> >> > > >>
> >> [resourceClaim=StandardResourceClaim[id=1581318452649-212433,
> >> > >
> >> > > >> container=default, section=465], offset=0,
> >> > >
> >> > > >>
> >> > >
> >> > > >>
> >>
> length=6401249],offset=2199110,name=90e47e27f806d46b2e190b462db929441993d166-ED,size=318289]
> >> > >
> >> > > >> at
> >> > >
> >> > > >> org.apache.nifi.controller.repository.io
> >> > >
> >> > > >>
> >> .FlowFileAccessInputStream.read(FlowFileAccessInputStream.java:93)
> >> > >
> >> > > >> at
> >> > >
> >> > > >> org.apache.nifi.controller.repository.io
> >> > >
> >> > > >>
> >> .TaskTerminationInputStream.read(TaskTerminationInputStream.java:68)
> >> > >
> >> > > >> at
> >> > >
> >> > > >> org.apache.nifi.stream.io
> >> .StreamUtils.fillBuffer(StreamUtils.java:89)
> >> > >
> >> > > >> at
> >> > >
> >> > > >>
> >> > >
> >> > > >>
> >>
> org.apache.nifi.processors.standard.ReplaceText$RegexReplace$2.process(ReplaceText.java:532)
> >> > >
> >> > > >> at
> >> > >
> >> > > >>
> >> > >
> >> > > >>
> >>
> org.apache.nifi.controller.repository.StandardProcessSession.read(StandardProcessSession.java:2324)
> >> > >
> >> > > >> at
> >> > >
> >> > > >>
> >> > >
> >> > > >>
> >>
> org.apache.nifi.controller.repository.StandardProcessSession.read(StandardProcessSession.java:2292)
> >> > >
> >> > > >> at
> >> > >
> >> > > >>
> >> > >
> >> > > >>
> >>
> org.apache.nifi.controller.repository.BatchingSessionFactory$HighThroughputSession.read(BatchingSessionFactory.java:192)
> >> > >
> >> > > >> at
> >> > >
> >> > > >>
> >> > >
> >> > > >>
> >>
> org.apache.nifi.processors.standard.ReplaceText$RegexReplace.replace(ReplaceText.java:529)
> >> > >
> >> > > >> at
> >> > >
> >> > > >>
> >> > >
> >> > > >>
> >>
> org.apache.nifi.processors.standard.ReplaceText.onTrigger(ReplaceText.java:316)
> >> > >
> >> > > >> at
> >> > >
> >> > > >>
> >> > >
> >> > > >>
> >>
> org.apache.nifi.processor.AbstractProcessor.onTrigger(AbstractProcessor.java:27)
> >> > >
> >> > > >> at
> >> > >
> >> > > >>
> >> > >
> >> > > >>
> >>
> org.apache.nifi.controller.StandardProcessorNode.onTrigger(StandardProcessorNode.java:1176)
> >> > >
> >> > > >> at
> >> > >
> >> > > >>
> >> > >
> >> > > >>
> >>
> org.apache.nifi.controller.tasks.ConnectableTask.invoke(ConnectableTask.java:213)
> >> > >
> >> > > >> at
> >> > >
> >> > > >>
> >> > >
> >> > > >>
> >>
> org.apache.nifi.controller.scheduling.TimerDrivenSchedulingAgent$1.run(TimerDrivenSchedulingAgent.java:117)
> >> > >
> >> > > >> at
> >> org.apache.nifi.engine.FlowEngine$2.run(FlowEngine.java:110)
> >> > >
> >> > > >> at
> >> > >
> >> > > >>
> >> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
> >> > >
> >> > > >> at
> >> > >
> >> > > >>
> >> java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
> >> > >
> >> > > >> at
> >> > >
> >> > > >>
> >> > >
> >> > > >>
> >>
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
> >> > >
> >> > > >> at
> >> > >
> >> > > >>
> >> > >
> >> > > >>
> >>
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
> >> > >
> >> > > >> at
> >> > >
> >> > > >>
> >> > >
> >> > > >>
> >>
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> >> > >
> >> > > >> at
> >> > >
> >> > > >>
> >> > >
> >> > > >>
> >>
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> >> > >
> >> > > >> at java.lang.Thread.run(Thread.java:748)
> >> > >
> >> > > >> Caused by: java.io.EOFException: null
> >> > >
> >> > > >> at org.apache.nifi.stream.io
> >> > >
> >> > > >> .StreamUtils.skip(StreamUtils.java:270)
> >> > >
> >> > > >> at
> >> > >
> >> > > >> org.apache.nifi.controller.repository.io
> >> > >
> >> > > >>
> >> .ContentClaimInputStream.formDelegate(ContentClaimInputStream.java:155)
> >> > >
> >> > > >> at
> >> > >
> >> > > >> org.apache.nifi.controller.repository.io
> >> > >
> >> > > >>
> >> .ContentClaimInputStream.getDelegate(ContentClaimInputStream.java:51)
> >> > >
> >> > > >> at
> >> > >
> >> > > >> org.apache.nifi.controller.repository.io
> >> > >
> >> > > >>
> >> .ContentClaimInputStream.read(ContentClaimInputStream.java:89)
> >> > >
> >> > > >> at
> >> > >
> >> > > >> org.apache.nifi.controller.repository.io
> >> > >
> >> > > >>
> >> .DisableOnCloseInputStream.read(DisableOnCloseInputStream.java:49)
> >> > >
> >> > > >> at
> >> > >
> >> > > >> org.apache.nifi.controller.repository.io
> >> > >
> >> > > >> .LimitedInputStream.read(LimitedInputStream.java:86)
> >> > >
> >> > > >> at
> >> > >
> >> > > >> org.apache.nifi.controller.repository.io
> >> > >
> >> > > >>
> >> .DisableOnCloseInputStream.read(DisableOnCloseInputStream.java:49)
> >> > >
> >> > > >> at
> >> > >
> >> > > >> org.apache.nifi.stream.io
> >> > >
> >> > > >>
> >> .ByteCountingInputStream.read(ByteCountingInputStream.java:52)
> >> > >
> >> > > >> at
> >> java.io.FilterInputStream.read(FilterInputStream.java:133)
> >> > >
> >> > > >> at
> >> > >
> >> > > >> org.apache.nifi.controller.repository.io
> >> > >
> >> > > >>
> >> .FlowFileAccessInputStream.read(FlowFileAccessInputStream.java:82)
> >> > >
> >> > > >> ... 20 common frames omitted
> >> > >
> >> > > >>
> >> > >
> >> > > >>
> >> > >
> >> > > >> Joe Gresock
> >> > >
> >> > > >>
> >> > >
> >> > > >
> >> > >
> >>
> >>
> >>
> >
> > --
> > Be on your guard; stand firm in the faith; be courageous; be strong. Do
> > everything in love. -*1 Corinthians 16:13-14*
> >
>
>
> --
> Be on your guard; stand firm in the faith; be courageous; be strong. Do
> everything in love. -*1 Corinthians 16:13-14*
>
Re: NiFi 1.11.1: EOFException in FlowFileAccessInputStream
Posted by Joe Gresock <jg...@gmail.com>.
Another data point.. I noticed that the only nodes that seem to have this
error are ones with recent pod restarts. Perhaps this is just a risk if
the nifi process suddenly restarts?
In the log, I only see normal log statements, and then suddenly:
2020-02-13 18:20:06,810 INFO [Heartbeat Monitor Thread-1]
o.a.n.c.c.h.AbstractHeartbeatMonito2020-02-13 18:23:50,779 INFO [main]
org.apache.nifi.NiFi Launching NiFi...
The disk is definitely not filling up, as I have all of my disks
monitored. Perhaps an OOM could be the culprit, but I don't see anything
in the logs.
In normal operation, should NiFi be at risk of truncating content claims in
case of the JVM suddenly stopping?
On Thu, Feb 13, 2020 at 9:18 AM Joe Gresock <jg...@gmail.com> wrote:
> Sure, sorry for the delay.
>
> It's an AWS EBS persistent volume. Here are the PV and storage class
> objects. It's single-zone (us-east-1d):
>
> apiVersion: v1
> kind: PersistentVolume
> metadata:
> annotations:
> kubernetes.io/createdby: aws-ebs-dynamic-provisioner
> pv.kubernetes.io/bound-by-controller: "yes"
> pv.kubernetes.io/provisioned-by: kubernetes.io/aws-ebs
> labels:
> failure-domain.beta.kubernetes.io/region: us-east-1
> failure-domain.beta.kubernetes.io/zone: us-east-1d
> name: pvc-abb1e679-523d-4ef7-a42e-cb42d8b8da0d
> spec:
> accessModes:
> - ReadWriteOnce
> awsElasticBlockStore:
> fsType: ext4
> volumeID: aws://us-east-1d/vol-<instanceId> # Redacting some details
> capacity:
> storage: 150Gi
> claimRef:
> apiVersion: v1
> kind: PersistentVolumeClaim
> name: content-repository-nifi-4
> namespace: default
> resourceVersion: "7190127"
> uid: abb1e679-523d-4ef7-a42e-cb42d8b8da0d
> mountOptions:
> - debug
> nodeAffinity:
> required:
> nodeSelectorTerms:
> - matchExpressions:
> - key: failure-domain.beta.kubernetes.io/region
> operator: In
> values:
> - us-east-1
> - key: failure-domain.beta.kubernetes.io/zone
> operator: In
> values:
> - us-east-1d
> persistentVolumeReclaimPolicy: Retain
> storageClassName: gp2
> volumeMode: Filesystem
> ---
> apiVersion: storage.k8s.io/v1
> kind: StorageClass
> metadata:
> annotations:
> kubectl.kubernetes.io/last-applied-configuration: |
> {"allowVolumeExpansion":true,"apiVersion":"storage.k8s.io/v1
> ","kind":"StorageClass","metadata":{"annotations":{"
> storageclass.kubernetes.io/is-default-class
> ":"true"},"labels":{"addonmanager.kubernetes.io.mode":"EnsureExists"},"name":"gp2"},"mountOptions":["debug"],"parameters":{"type":"gp2"},"provisioner":"
> kubernetes.io/aws-ebs
> ","reclaimPolicy":"Retain","volumeBindingMode":"WaitForFirstConsumer"}
> storageclass.kubernetes.io/is-default-class: "true"
> creationTimestamp: "2020-01-03T12:30:49Z"
> labels:
> addonmanager.kubernetes.io.mode: EnsureExists
> name: gp2
> resourceVersion: "126035"
> selfLink: /apis/storage.k8s.io/v1/storageclasses/gp2
> uid: b91b9233-cd1b-4850-b0e1-bb984c4a31c5
> allowVolumeExpansion: true
> mountOptions:
> - debug
> parameters:
> type: gp2
> provisioner: kubernetes.io/aws-ebs
> reclaimPolicy: Retain
> volumeBindingMode: WaitForFirstConsumer
>
> On Tue, Feb 11, 2020 at 9:20 AM Endre Kovacs
> <an...@protonmail.com.invalid> wrote:
>
>> The more i think about this, the more i believe that the PV had to be
>> provisioned as some kind of network attached storage, and the exception had
>> to be happening because of intermittent network issue for this NAS.
>>
>>
>>
>>
>> Sent with ProtonMail Secure Email.
>>
>> ‐‐‐‐‐‐‐ Original Message ‐‐‐‐‐‐‐
>> On Monday, February 10, 2020 8:41 PM, Endre Kovacs <
>> andrewsmith87@protonmail.com> wrote:
>>
>> > Hi,
>> >
>> > What cloud are you running this on?
>> > Kubernetes version?
>> >
>> > Can you share the storageClass you use for creating the repo PVCs?
>> >
>> > what is the topology of the cluster? Muli zone?
>> >
>> > Best regards,
>> > Endre
>> >
>> > Sent with ProtonMail Secure Email.
>> >
>> > ‐‐‐‐‐‐‐ Original Message ‐‐‐‐‐‐‐
>> > On Monday, February 10, 2020 7:41 PM, Shawn Weeks
>> sweeks@weeksconsulting.us wrote:
>> >
>> > > The more I look at the error the more it looks like the container
>> couldn't access the persistence volume for some period of time. Can you
>> share your yaml configuration minus sensitive stuff and what environment
>> your deploying K8s on, aka Azure, ECS, Rancher, Openshift, etc.
>> > > Thanks
>> > > Shawn
>> > > On 2/10/20, 12:27 PM, "Shawn Weeks" sweeks@weeksconsulting.us wrote:
>> > > I've been testing out NiFi on K8s quite a bit lately and might be
>> able to help. What are you using for your persistent volume, this kinda
>> sounds like NiFi can't always access data in the content_repository folder?
>> > > Thanks
>> > > Shawn
>> > > On 2/10/20, 11:53 AM, "Joe Witt" joe.witt@gmail.com wrote:
>> > > ...sent a little too quick. Also seems like if it is related to your
>> k8s
>> > > process that the issue would occur on older nifi releases too. So that
>> > > might be an interesting test for you.
>> > > On Mon, Feb 10, 2020 at 9:52 AM Joe Witt joe.witt@gmail.com wrote:
>> > >
>> > > > Joe
>> > >
>> > > >
>> > >
>> > > > If you can replicate this in a non k8s environment that
>> would likely make
>> > >
>> > > > resolution easier. Far too many variables/questions at
>> this stage.
>> > >
>> > > >
>> > >
>> > > > Thanks
>> > >
>> > > >
>> > >
>> > > > On Mon, Feb 10, 2020 at 8:15 AM Joe Gresock <
>> jgresock@gmail.com> wrote:
>> > >
>> > > >
>> > >
>> > > >> I don't know if there is something wrong with my
>> Kubernetes-ized
>> > >
>> > > >> deployment
>> > >
>> > > >> of NiFi, but I'm seeing the following error crop up a lot
>> in my NiFi
>> > >
>> > > >> 1.11.1
>> > >
>> > > >> deployment. I can see the flow file attributes in "List
>> Queue" for flow
>> > >
>> > > >> files affected by this problem, but the content claim is
>> missing.
>> > >
>> > > >>
>> > >
>> > > >> It's not easily reproducible, but I've been running data
>> constantly
>> > >
>> > > >> through
>> > >
>> > > >> the cluster for a few days, and every 2-4 hours or so,
>> this problem
>> > >
>> > > >> appears
>> > >
>> > > >> in one of my processors (it's not always the same
>> processors). The error
>> > >
>> > > >> only goes away when I drain the relevant queue, but
>> obviously that's not
>> > >
>> > > >> an
>> > >
>> > > >> option in production.
>> > >
>> > > >>
>> > >
>> > > >> Any thoughts on what might be causing intermittent
>> truncations in the
>> > >
>> > > >> content_repository?
>> > >
>> > > >>
>> > >
>> > > >> 2020-02-10 10:38:57,205 ERROR [Timer-Driven Process
>> Thread-6]
>> > >
>> > > >> o.a.nifi.processors.standard.ReplaceText
>> > >
>> > > >> ReplaceText[id=1abd1d49-0170-1000-ffff-ffffda59ecb2]
>> > >
>> > > >> ReplaceText[id=1abd1d49-0170-1000-ffff-ffffda59ecb2] fa
>> > >
>> > > >> iled to process session due to
>> > >
>> > > >>
>> org.apache.nifi.processor.exception.FlowFileAccessException: Could not
>> > >
>> > > >> read
>> > >
>> > > >> from
>> > >
>> > > >>
>> > >
>> > > >>
>> StandardFlowFileRecord[uuid=b0ca5649-ca8f-4a99-88df-70e1143c5f1e,claim=StandardContentClaim
>> > >
>> > > >>
>> [resourceClaim=StandardResourceClaim[id=1581318452649-212433,
>> > >
>> > > >> container=default, section=465], offset=0,
>> > >
>> > > >>
>> > >
>> > > >>
>> length=6401249],offset=2199110,name=90e47e27f806d46b2e190b462db929441993d166-ED,size=318289];
>> > >
>> > > >> Processor Administratively Yielded for 1 sec:
>> > >
>> > > >>
>> org.apache.nifi.processor.exception.FlowFileAccessException: Could not
>> > >
>> > > >> read
>> > >
>> > > >> from
>> > >
>> > > >>
>> > >
>> > > >>
>> StandardFlowFileRecord[uuid=b0ca5649-ca8f-4a99-88df-70e1143c5f1e,claim=StandardContentClaim
>> > >
>> > > >>
>> [resourceClaim=StandardResourceClaim[id=1581318452649-212433,
>> > >
>> > > >> container=default, section=465], offset=0,
>> > >
>> > > >>
>> > >
>> > > >>
>> length=6401249],offset=2199110,name=90e47e27f806d46b2e190b462db929441993d166-ED,size=318289]
>> > >
>> > > >>
>> org.apache.nifi.processor.exception.FlowFileAccessException: Could not
>> > >
>> > > >> read
>> > >
>> > > >> from
>> > >
>> > > >>
>> > >
>> > > >>
>> StandardFlowFileRecord[uuid=b0ca5649-ca8f-4a99-88df-70e1143c5f1e,claim=StandardContentClaim
>> > >
>> > > >>
>> [resourceClaim=StandardResourceClaim[id=1581318452649-212433,
>> > >
>> > > >> container=default, section=465], offset=0,
>> > >
>> > > >>
>> > >
>> > > >>
>> length=6401249],offset=2199110,name=90e47e27f806d46b2e190b462db929441993d166-ED,size=318289]
>> > >
>> > > >> at
>> > >
>> > > >> org.apache.nifi.controller.repository.io
>> > >
>> > > >>
>> .FlowFileAccessInputStream.read(FlowFileAccessInputStream.java:93)
>> > >
>> > > >> at
>> > >
>> > > >> org.apache.nifi.controller.repository.io
>> > >
>> > > >>
>> .TaskTerminationInputStream.read(TaskTerminationInputStream.java:68)
>> > >
>> > > >> at
>> > >
>> > > >> org.apache.nifi.stream.io
>> .StreamUtils.fillBuffer(StreamUtils.java:89)
>> > >
>> > > >> at
>> > >
>> > > >>
>> > >
>> > > >>
>> org.apache.nifi.processors.standard.ReplaceText$RegexReplace$2.process(ReplaceText.java:532)
>> > >
>> > > >> at
>> > >
>> > > >>
>> > >
>> > > >>
>> org.apache.nifi.controller.repository.StandardProcessSession.read(StandardProcessSession.java:2324)
>> > >
>> > > >> at
>> > >
>> > > >>
>> > >
>> > > >>
>> org.apache.nifi.controller.repository.StandardProcessSession.read(StandardProcessSession.java:2292)
>> > >
>> > > >> at
>> > >
>> > > >>
>> > >
>> > > >>
>> org.apache.nifi.controller.repository.BatchingSessionFactory$HighThroughputSession.read(BatchingSessionFactory.java:192)
>> > >
>> > > >> at
>> > >
>> > > >>
>> > >
>> > > >>
>> org.apache.nifi.processors.standard.ReplaceText$RegexReplace.replace(ReplaceText.java:529)
>> > >
>> > > >> at
>> > >
>> > > >>
>> > >
>> > > >>
>> org.apache.nifi.processors.standard.ReplaceText.onTrigger(ReplaceText.java:316)
>> > >
>> > > >> at
>> > >
>> > > >>
>> > >
>> > > >>
>> org.apache.nifi.processor.AbstractProcessor.onTrigger(AbstractProcessor.java:27)
>> > >
>> > > >> at
>> > >
>> > > >>
>> > >
>> > > >>
>> org.apache.nifi.controller.StandardProcessorNode.onTrigger(StandardProcessorNode.java:1176)
>> > >
>> > > >> at
>> > >
>> > > >>
>> > >
>> > > >>
>> org.apache.nifi.controller.tasks.ConnectableTask.invoke(ConnectableTask.java:213)
>> > >
>> > > >> at
>> > >
>> > > >>
>> > >
>> > > >>
>> org.apache.nifi.controller.scheduling.TimerDrivenSchedulingAgent$1.run(TimerDrivenSchedulingAgent.java:117)
>> > >
>> > > >> at
>> org.apache.nifi.engine.FlowEngine$2.run(FlowEngine.java:110)
>> > >
>> > > >> at
>> > >
>> > > >>
>> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
>> > >
>> > > >> at
>> > >
>> > > >>
>> java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
>> > >
>> > > >> at
>> > >
>> > > >>
>> > >
>> > > >>
>> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
>> > >
>> > > >> at
>> > >
>> > > >>
>> > >
>> > > >>
>> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
>> > >
>> > > >> at
>> > >
>> > > >>
>> > >
>> > > >>
>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
>> > >
>> > > >> at
>> > >
>> > > >>
>> > >
>> > > >>
>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
>> > >
>> > > >> at java.lang.Thread.run(Thread.java:748)
>> > >
>> > > >> Caused by: java.io.EOFException: null
>> > >
>> > > >> at org.apache.nifi.stream.io
>> > >
>> > > >> .StreamUtils.skip(StreamUtils.java:270)
>> > >
>> > > >> at
>> > >
>> > > >> org.apache.nifi.controller.repository.io
>> > >
>> > > >>
>> .ContentClaimInputStream.formDelegate(ContentClaimInputStream.java:155)
>> > >
>> > > >> at
>> > >
>> > > >> org.apache.nifi.controller.repository.io
>> > >
>> > > >>
>> .ContentClaimInputStream.getDelegate(ContentClaimInputStream.java:51)
>> > >
>> > > >> at
>> > >
>> > > >> org.apache.nifi.controller.repository.io
>> > >
>> > > >>
>> .ContentClaimInputStream.read(ContentClaimInputStream.java:89)
>> > >
>> > > >> at
>> > >
>> > > >> org.apache.nifi.controller.repository.io
>> > >
>> > > >>
>> .DisableOnCloseInputStream.read(DisableOnCloseInputStream.java:49)
>> > >
>> > > >> at
>> > >
>> > > >> org.apache.nifi.controller.repository.io
>> > >
>> > > >> .LimitedInputStream.read(LimitedInputStream.java:86)
>> > >
>> > > >> at
>> > >
>> > > >> org.apache.nifi.controller.repository.io
>> > >
>> > > >>
>> .DisableOnCloseInputStream.read(DisableOnCloseInputStream.java:49)
>> > >
>> > > >> at
>> > >
>> > > >> org.apache.nifi.stream.io
>> > >
>> > > >>
>> .ByteCountingInputStream.read(ByteCountingInputStream.java:52)
>> > >
>> > > >> at
>> java.io.FilterInputStream.read(FilterInputStream.java:133)
>> > >
>> > > >> at
>> > >
>> > > >> org.apache.nifi.controller.repository.io
>> > >
>> > > >>
>> .FlowFileAccessInputStream.read(FlowFileAccessInputStream.java:82)
>> > >
>> > > >> ... 20 common frames omitted
>> > >
>> > > >>
>> > >
>> > > >>
>> > >
>> > > >> Joe Gresock
>> > >
>> > > >>
>> > >
>> > > >
>> > >
>>
>>
>>
>
> --
> Be on your guard; stand firm in the faith; be courageous; be strong. Do
> everything in love. -*1 Corinthians 16:13-14*
>
--
Be on your guard; stand firm in the faith; be courageous; be strong. Do
everything in love. -*1 Corinthians 16:13-14*
Re: NiFi 1.11.1: EOFException in FlowFileAccessInputStream
Posted by Joe Gresock <jg...@gmail.com>.
Sure, sorry for the delay.
It's an AWS EBS persistent volume. Here are the PV and storage class
objects. It's single-zone (us-east-1d):
apiVersion: v1
kind: PersistentVolume
metadata:
annotations:
kubernetes.io/createdby: aws-ebs-dynamic-provisioner
pv.kubernetes.io/bound-by-controller: "yes"
pv.kubernetes.io/provisioned-by: kubernetes.io/aws-ebs
labels:
failure-domain.beta.kubernetes.io/region: us-east-1
failure-domain.beta.kubernetes.io/zone: us-east-1d
name: pvc-abb1e679-523d-4ef7-a42e-cb42d8b8da0d
spec:
accessModes:
- ReadWriteOnce
awsElasticBlockStore:
fsType: ext4
volumeID: aws://us-east-1d/vol-<instanceId> # Redacting some details
capacity:
storage: 150Gi
claimRef:
apiVersion: v1
kind: PersistentVolumeClaim
name: content-repository-nifi-4
namespace: default
resourceVersion: "7190127"
uid: abb1e679-523d-4ef7-a42e-cb42d8b8da0d
mountOptions:
- debug
nodeAffinity:
required:
nodeSelectorTerms:
- matchExpressions:
- key: failure-domain.beta.kubernetes.io/region
operator: In
values:
- us-east-1
- key: failure-domain.beta.kubernetes.io/zone
operator: In
values:
- us-east-1d
persistentVolumeReclaimPolicy: Retain
storageClassName: gp2
volumeMode: Filesystem
---
apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
annotations:
kubectl.kubernetes.io/last-applied-configuration: |
{"allowVolumeExpansion":true,"apiVersion":"storage.k8s.io/v1
","kind":"StorageClass","metadata":{"annotations":{"
storageclass.kubernetes.io/is-default-class
":"true"},"labels":{"addonmanager.kubernetes.io.mode":"EnsureExists"},"name":"gp2"},"mountOptions":["debug"],"parameters":{"type":"gp2"},"provisioner":"
kubernetes.io/aws-ebs
","reclaimPolicy":"Retain","volumeBindingMode":"WaitForFirstConsumer"}
storageclass.kubernetes.io/is-default-class: "true"
creationTimestamp: "2020-01-03T12:30:49Z"
labels:
addonmanager.kubernetes.io.mode: EnsureExists
name: gp2
resourceVersion: "126035"
selfLink: /apis/storage.k8s.io/v1/storageclasses/gp2
uid: b91b9233-cd1b-4850-b0e1-bb984c4a31c5
allowVolumeExpansion: true
mountOptions:
- debug
parameters:
type: gp2
provisioner: kubernetes.io/aws-ebs
reclaimPolicy: Retain
volumeBindingMode: WaitForFirstConsumer
On Tue, Feb 11, 2020 at 9:20 AM Endre Kovacs
<an...@protonmail.com.invalid> wrote:
> The more i think about this, the more i believe that the PV had to be
> provisioned as some kind of network attached storage, and the exception had
> to be happening because of intermittent network issue for this NAS.
>
>
>
>
> Sent with ProtonMail Secure Email.
>
> ‐‐‐‐‐‐‐ Original Message ‐‐‐‐‐‐‐
> On Monday, February 10, 2020 8:41 PM, Endre Kovacs <
> andrewsmith87@protonmail.com> wrote:
>
> > Hi,
> >
> > What cloud are you running this on?
> > Kubernetes version?
> >
> > Can you share the storageClass you use for creating the repo PVCs?
> >
> > what is the topology of the cluster? Muli zone?
> >
> > Best regards,
> > Endre
> >
> > Sent with ProtonMail Secure Email.
> >
> > ‐‐‐‐‐‐‐ Original Message ‐‐‐‐‐‐‐
> > On Monday, February 10, 2020 7:41 PM, Shawn Weeks
> sweeks@weeksconsulting.us wrote:
> >
> > > The more I look at the error the more it looks like the container
> couldn't access the persistence volume for some period of time. Can you
> share your yaml configuration minus sensitive stuff and what environment
> your deploying K8s on, aka Azure, ECS, Rancher, Openshift, etc.
> > > Thanks
> > > Shawn
> > > On 2/10/20, 12:27 PM, "Shawn Weeks" sweeks@weeksconsulting.us wrote:
> > > I've been testing out NiFi on K8s quite a bit lately and might be able
> to help. What are you using for your persistent volume, this kinda sounds
> like NiFi can't always access data in the content_repository folder?
> > > Thanks
> > > Shawn
> > > On 2/10/20, 11:53 AM, "Joe Witt" joe.witt@gmail.com wrote:
> > > ...sent a little too quick. Also seems like if it is related to your
> k8s
> > > process that the issue would occur on older nifi releases too. So that
> > > might be an interesting test for you.
> > > On Mon, Feb 10, 2020 at 9:52 AM Joe Witt joe.witt@gmail.com wrote:
> > >
> > > > Joe
> > >
> > > >
> > >
> > > > If you can replicate this in a non k8s environment that
> would likely make
> > >
> > > > resolution easier. Far too many variables/questions at this
> stage.
> > >
> > > >
> > >
> > > > Thanks
> > >
> > > >
> > >
> > > > On Mon, Feb 10, 2020 at 8:15 AM Joe Gresock <
> jgresock@gmail.com> wrote:
> > >
> > > >
> > >
> > > >> I don't know if there is something wrong with my
> Kubernetes-ized
> > >
> > > >> deployment
> > >
> > > >> of NiFi, but I'm seeing the following error crop up a lot
> in my NiFi
> > >
> > > >> 1.11.1
> > >
> > > >> deployment. I can see the flow file attributes in "List
> Queue" for flow
> > >
> > > >> files affected by this problem, but the content claim is
> missing.
> > >
> > > >>
> > >
> > > >> It's not easily reproducible, but I've been running data
> constantly
> > >
> > > >> through
> > >
> > > >> the cluster for a few days, and every 2-4 hours or so, this
> problem
> > >
> > > >> appears
> > >
> > > >> in one of my processors (it's not always the same
> processors). The error
> > >
> > > >> only goes away when I drain the relevant queue, but
> obviously that's not
> > >
> > > >> an
> > >
> > > >> option in production.
> > >
> > > >>
> > >
> > > >> Any thoughts on what might be causing intermittent
> truncations in the
> > >
> > > >> content_repository?
> > >
> > > >>
> > >
> > > >> 2020-02-10 10:38:57,205 ERROR [Timer-Driven Process
> Thread-6]
> > >
> > > >> o.a.nifi.processors.standard.ReplaceText
> > >
> > > >> ReplaceText[id=1abd1d49-0170-1000-ffff-ffffda59ecb2]
> > >
> > > >> ReplaceText[id=1abd1d49-0170-1000-ffff-ffffda59ecb2] fa
> > >
> > > >> iled to process session due to
> > >
> > > >>
> org.apache.nifi.processor.exception.FlowFileAccessException: Could not
> > >
> > > >> read
> > >
> > > >> from
> > >
> > > >>
> > >
> > > >>
> StandardFlowFileRecord[uuid=b0ca5649-ca8f-4a99-88df-70e1143c5f1e,claim=StandardContentClaim
> > >
> > > >>
> [resourceClaim=StandardResourceClaim[id=1581318452649-212433,
> > >
> > > >> container=default, section=465], offset=0,
> > >
> > > >>
> > >
> > > >>
> length=6401249],offset=2199110,name=90e47e27f806d46b2e190b462db929441993d166-ED,size=318289];
> > >
> > > >> Processor Administratively Yielded for 1 sec:
> > >
> > > >>
> org.apache.nifi.processor.exception.FlowFileAccessException: Could not
> > >
> > > >> read
> > >
> > > >> from
> > >
> > > >>
> > >
> > > >>
> StandardFlowFileRecord[uuid=b0ca5649-ca8f-4a99-88df-70e1143c5f1e,claim=StandardContentClaim
> > >
> > > >>
> [resourceClaim=StandardResourceClaim[id=1581318452649-212433,
> > >
> > > >> container=default, section=465], offset=0,
> > >
> > > >>
> > >
> > > >>
> length=6401249],offset=2199110,name=90e47e27f806d46b2e190b462db929441993d166-ED,size=318289]
> > >
> > > >>
> org.apache.nifi.processor.exception.FlowFileAccessException: Could not
> > >
> > > >> read
> > >
> > > >> from
> > >
> > > >>
> > >
> > > >>
> StandardFlowFileRecord[uuid=b0ca5649-ca8f-4a99-88df-70e1143c5f1e,claim=StandardContentClaim
> > >
> > > >>
> [resourceClaim=StandardResourceClaim[id=1581318452649-212433,
> > >
> > > >> container=default, section=465], offset=0,
> > >
> > > >>
> > >
> > > >>
> length=6401249],offset=2199110,name=90e47e27f806d46b2e190b462db929441993d166-ED,size=318289]
> > >
> > > >> at
> > >
> > > >> org.apache.nifi.controller.repository.io
> > >
> > > >>
> .FlowFileAccessInputStream.read(FlowFileAccessInputStream.java:93)
> > >
> > > >> at
> > >
> > > >> org.apache.nifi.controller.repository.io
> > >
> > > >>
> .TaskTerminationInputStream.read(TaskTerminationInputStream.java:68)
> > >
> > > >> at
> > >
> > > >> org.apache.nifi.stream.io
> .StreamUtils.fillBuffer(StreamUtils.java:89)
> > >
> > > >> at
> > >
> > > >>
> > >
> > > >>
> org.apache.nifi.processors.standard.ReplaceText$RegexReplace$2.process(ReplaceText.java:532)
> > >
> > > >> at
> > >
> > > >>
> > >
> > > >>
> org.apache.nifi.controller.repository.StandardProcessSession.read(StandardProcessSession.java:2324)
> > >
> > > >> at
> > >
> > > >>
> > >
> > > >>
> org.apache.nifi.controller.repository.StandardProcessSession.read(StandardProcessSession.java:2292)
> > >
> > > >> at
> > >
> > > >>
> > >
> > > >>
> org.apache.nifi.controller.repository.BatchingSessionFactory$HighThroughputSession.read(BatchingSessionFactory.java:192)
> > >
> > > >> at
> > >
> > > >>
> > >
> > > >>
> org.apache.nifi.processors.standard.ReplaceText$RegexReplace.replace(ReplaceText.java:529)
> > >
> > > >> at
> > >
> > > >>
> > >
> > > >>
> org.apache.nifi.processors.standard.ReplaceText.onTrigger(ReplaceText.java:316)
> > >
> > > >> at
> > >
> > > >>
> > >
> > > >>
> org.apache.nifi.processor.AbstractProcessor.onTrigger(AbstractProcessor.java:27)
> > >
> > > >> at
> > >
> > > >>
> > >
> > > >>
> org.apache.nifi.controller.StandardProcessorNode.onTrigger(StandardProcessorNode.java:1176)
> > >
> > > >> at
> > >
> > > >>
> > >
> > > >>
> org.apache.nifi.controller.tasks.ConnectableTask.invoke(ConnectableTask.java:213)
> > >
> > > >> at
> > >
> > > >>
> > >
> > > >>
> org.apache.nifi.controller.scheduling.TimerDrivenSchedulingAgent$1.run(TimerDrivenSchedulingAgent.java:117)
> > >
> > > >> at
> org.apache.nifi.engine.FlowEngine$2.run(FlowEngine.java:110)
> > >
> > > >> at
> > >
> > > >>
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
> > >
> > > >> at
> > >
> > > >>
> java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
> > >
> > > >> at
> > >
> > > >>
> > >
> > > >>
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
> > >
> > > >> at
> > >
> > > >>
> > >
> > > >>
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
> > >
> > > >> at
> > >
> > > >>
> > >
> > > >>
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> > >
> > > >> at
> > >
> > > >>
> > >
> > > >>
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> > >
> > > >> at java.lang.Thread.run(Thread.java:748)
> > >
> > > >> Caused by: java.io.EOFException: null
> > >
> > > >> at org.apache.nifi.stream.io
> > >
> > > >> .StreamUtils.skip(StreamUtils.java:270)
> > >
> > > >> at
> > >
> > > >> org.apache.nifi.controller.repository.io
> > >
> > > >>
> .ContentClaimInputStream.formDelegate(ContentClaimInputStream.java:155)
> > >
> > > >> at
> > >
> > > >> org.apache.nifi.controller.repository.io
> > >
> > > >>
> .ContentClaimInputStream.getDelegate(ContentClaimInputStream.java:51)
> > >
> > > >> at
> > >
> > > >> org.apache.nifi.controller.repository.io
> > >
> > > >>
> .ContentClaimInputStream.read(ContentClaimInputStream.java:89)
> > >
> > > >> at
> > >
> > > >> org.apache.nifi.controller.repository.io
> > >
> > > >>
> .DisableOnCloseInputStream.read(DisableOnCloseInputStream.java:49)
> > >
> > > >> at
> > >
> > > >> org.apache.nifi.controller.repository.io
> > >
> > > >> .LimitedInputStream.read(LimitedInputStream.java:86)
> > >
> > > >> at
> > >
> > > >> org.apache.nifi.controller.repository.io
> > >
> > > >>
> .DisableOnCloseInputStream.read(DisableOnCloseInputStream.java:49)
> > >
> > > >> at
> > >
> > > >> org.apache.nifi.stream.io
> > >
> > > >>
> .ByteCountingInputStream.read(ByteCountingInputStream.java:52)
> > >
> > > >> at
> java.io.FilterInputStream.read(FilterInputStream.java:133)
> > >
> > > >> at
> > >
> > > >> org.apache.nifi.controller.repository.io
> > >
> > > >>
> .FlowFileAccessInputStream.read(FlowFileAccessInputStream.java:82)
> > >
> > > >> ... 20 common frames omitted
> > >
> > > >>
> > >
> > > >>
> > >
> > > >> Joe Gresock
> > >
> > > >>
> > >
> > > >
> > >
>
>
>
--
Be on your guard; stand firm in the faith; be courageous; be strong. Do
everything in love. -*1 Corinthians 16:13-14*
Re: NiFi 1.11.1: EOFException in FlowFileAccessInputStream
Posted by Endre Kovacs <an...@protonmail.com.INVALID>.
The more i think about this, the more i believe that the PV had to be provisioned as some kind of network attached storage, and the exception had to be happening because of intermittent network issue for this NAS.
Sent with ProtonMail Secure Email.
‐‐‐‐‐‐‐ Original Message ‐‐‐‐‐‐‐
On Monday, February 10, 2020 8:41 PM, Endre Kovacs <an...@protonmail.com> wrote:
> Hi,
>
> What cloud are you running this on?
> Kubernetes version?
>
> Can you share the storageClass you use for creating the repo PVCs?
>
> what is the topology of the cluster? Muli zone?
>
> Best regards,
> Endre
>
> Sent with ProtonMail Secure Email.
>
> ‐‐‐‐‐‐‐ Original Message ‐‐‐‐‐‐‐
> On Monday, February 10, 2020 7:41 PM, Shawn Weeks sweeks@weeksconsulting.us wrote:
>
> > The more I look at the error the more it looks like the container couldn't access the persistence volume for some period of time. Can you share your yaml configuration minus sensitive stuff and what environment your deploying K8s on, aka Azure, ECS, Rancher, Openshift, etc.
> > Thanks
> > Shawn
> > On 2/10/20, 12:27 PM, "Shawn Weeks" sweeks@weeksconsulting.us wrote:
> > I've been testing out NiFi on K8s quite a bit lately and might be able to help. What are you using for your persistent volume, this kinda sounds like NiFi can't always access data in the content_repository folder?
> > Thanks
> > Shawn
> > On 2/10/20, 11:53 AM, "Joe Witt" joe.witt@gmail.com wrote:
> > ...sent a little too quick. Also seems like if it is related to your k8s
> > process that the issue would occur on older nifi releases too. So that
> > might be an interesting test for you.
> > On Mon, Feb 10, 2020 at 9:52 AM Joe Witt joe.witt@gmail.com wrote:
> >
> > > Joe
> >
> > >
> >
> > > If you can replicate this in a non k8s environment that would likely make
> >
> > > resolution easier. Far too many variables/questions at this stage.
> >
> > >
> >
> > > Thanks
> >
> > >
> >
> > > On Mon, Feb 10, 2020 at 8:15 AM Joe Gresock <jg...@gmail.com> wrote:
> >
> > >
> >
> > >> I don't know if there is something wrong with my Kubernetes-ized
> >
> > >> deployment
> >
> > >> of NiFi, but I'm seeing the following error crop up a lot in my NiFi
> >
> > >> 1.11.1
> >
> > >> deployment. I can see the flow file attributes in "List Queue" for flow
> >
> > >> files affected by this problem, but the content claim is missing.
> >
> > >>
> >
> > >> It's not easily reproducible, but I've been running data constantly
> >
> > >> through
> >
> > >> the cluster for a few days, and every 2-4 hours or so, this problem
> >
> > >> appears
> >
> > >> in one of my processors (it's not always the same processors). The error
> >
> > >> only goes away when I drain the relevant queue, but obviously that's not
> >
> > >> an
> >
> > >> option in production.
> >
> > >>
> >
> > >> Any thoughts on what might be causing intermittent truncations in the
> >
> > >> content_repository?
> >
> > >>
> >
> > >> 2020-02-10 10:38:57,205 ERROR [Timer-Driven Process Thread-6]
> >
> > >> o.a.nifi.processors.standard.ReplaceText
> >
> > >> ReplaceText[id=1abd1d49-0170-1000-ffff-ffffda59ecb2]
> >
> > >> ReplaceText[id=1abd1d49-0170-1000-ffff-ffffda59ecb2] fa
> >
> > >> iled to process session due to
> >
> > >> org.apache.nifi.processor.exception.FlowFileAccessException: Could not
> >
> > >> read
> >
> > >> from
> >
> > >>
> >
> > >> StandardFlowFileRecord[uuid=b0ca5649-ca8f-4a99-88df-70e1143c5f1e,claim=StandardContentClaim
> >
> > >> [resourceClaim=StandardResourceClaim[id=1581318452649-212433,
> >
> > >> container=default, section=465], offset=0,
> >
> > >>
> >
> > >> length=6401249],offset=2199110,name=90e47e27f806d46b2e190b462db929441993d166-ED,size=318289];
> >
> > >> Processor Administratively Yielded for 1 sec:
> >
> > >> org.apache.nifi.processor.exception.FlowFileAccessException: Could not
> >
> > >> read
> >
> > >> from
> >
> > >>
> >
> > >> StandardFlowFileRecord[uuid=b0ca5649-ca8f-4a99-88df-70e1143c5f1e,claim=StandardContentClaim
> >
> > >> [resourceClaim=StandardResourceClaim[id=1581318452649-212433,
> >
> > >> container=default, section=465], offset=0,
> >
> > >>
> >
> > >> length=6401249],offset=2199110,name=90e47e27f806d46b2e190b462db929441993d166-ED,size=318289]
> >
> > >> org.apache.nifi.processor.exception.FlowFileAccessException: Could not
> >
> > >> read
> >
> > >> from
> >
> > >>
> >
> > >> StandardFlowFileRecord[uuid=b0ca5649-ca8f-4a99-88df-70e1143c5f1e,claim=StandardContentClaim
> >
> > >> [resourceClaim=StandardResourceClaim[id=1581318452649-212433,
> >
> > >> container=default, section=465], offset=0,
> >
> > >>
> >
> > >> length=6401249],offset=2199110,name=90e47e27f806d46b2e190b462db929441993d166-ED,size=318289]
> >
> > >> at
> >
> > >> org.apache.nifi.controller.repository.io
> >
> > >> .FlowFileAccessInputStream.read(FlowFileAccessInputStream.java:93)
> >
> > >> at
> >
> > >> org.apache.nifi.controller.repository.io
> >
> > >> .TaskTerminationInputStream.read(TaskTerminationInputStream.java:68)
> >
> > >> at
> >
> > >> org.apache.nifi.stream.io.StreamUtils.fillBuffer(StreamUtils.java:89)
> >
> > >> at
> >
> > >>
> >
> > >> org.apache.nifi.processors.standard.ReplaceText$RegexReplace$2.process(ReplaceText.java:532)
> >
> > >> at
> >
> > >>
> >
> > >> org.apache.nifi.controller.repository.StandardProcessSession.read(StandardProcessSession.java:2324)
> >
> > >> at
> >
> > >>
> >
> > >> org.apache.nifi.controller.repository.StandardProcessSession.read(StandardProcessSession.java:2292)
> >
> > >> at
> >
> > >>
> >
> > >> org.apache.nifi.controller.repository.BatchingSessionFactory$HighThroughputSession.read(BatchingSessionFactory.java:192)
> >
> > >> at
> >
> > >>
> >
> > >> org.apache.nifi.processors.standard.ReplaceText$RegexReplace.replace(ReplaceText.java:529)
> >
> > >> at
> >
> > >>
> >
> > >> org.apache.nifi.processors.standard.ReplaceText.onTrigger(ReplaceText.java:316)
> >
> > >> at
> >
> > >>
> >
> > >> org.apache.nifi.processor.AbstractProcessor.onTrigger(AbstractProcessor.java:27)
> >
> > >> at
> >
> > >>
> >
> > >> org.apache.nifi.controller.StandardProcessorNode.onTrigger(StandardProcessorNode.java:1176)
> >
> > >> at
> >
> > >>
> >
> > >> org.apache.nifi.controller.tasks.ConnectableTask.invoke(ConnectableTask.java:213)
> >
> > >> at
> >
> > >>
> >
> > >> org.apache.nifi.controller.scheduling.TimerDrivenSchedulingAgent$1.run(TimerDrivenSchedulingAgent.java:117)
> >
> > >> at org.apache.nifi.engine.FlowEngine$2.run(FlowEngine.java:110)
> >
> > >> at
> >
> > >> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
> >
> > >> at
> >
> > >> java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
> >
> > >> at
> >
> > >>
> >
> > >> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
> >
> > >> at
> >
> > >>
> >
> > >> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
> >
> > >> at
> >
> > >>
> >
> > >> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> >
> > >> at
> >
> > >>
> >
> > >> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> >
> > >> at java.lang.Thread.run(Thread.java:748)
> >
> > >> Caused by: java.io.EOFException: null
> >
> > >> at org.apache.nifi.stream.io
> >
> > >> .StreamUtils.skip(StreamUtils.java:270)
> >
> > >> at
> >
> > >> org.apache.nifi.controller.repository.io
> >
> > >> .ContentClaimInputStream.formDelegate(ContentClaimInputStream.java:155)
> >
> > >> at
> >
> > >> org.apache.nifi.controller.repository.io
> >
> > >> .ContentClaimInputStream.getDelegate(ContentClaimInputStream.java:51)
> >
> > >> at
> >
> > >> org.apache.nifi.controller.repository.io
> >
> > >> .ContentClaimInputStream.read(ContentClaimInputStream.java:89)
> >
> > >> at
> >
> > >> org.apache.nifi.controller.repository.io
> >
> > >> .DisableOnCloseInputStream.read(DisableOnCloseInputStream.java:49)
> >
> > >> at
> >
> > >> org.apache.nifi.controller.repository.io
> >
> > >> .LimitedInputStream.read(LimitedInputStream.java:86)
> >
> > >> at
> >
> > >> org.apache.nifi.controller.repository.io
> >
> > >> .DisableOnCloseInputStream.read(DisableOnCloseInputStream.java:49)
> >
> > >> at
> >
> > >> org.apache.nifi.stream.io
> >
> > >> .ByteCountingInputStream.read(ByteCountingInputStream.java:52)
> >
> > >> at java.io.FilterInputStream.read(FilterInputStream.java:133)
> >
> > >> at
> >
> > >> org.apache.nifi.controller.repository.io
> >
> > >> .FlowFileAccessInputStream.read(FlowFileAccessInputStream.java:82)
> >
> > >> ... 20 common frames omitted
> >
> > >>
> >
> > >>
> >
> > >> Joe Gresock
> >
> > >>
> >
> > >
> >
Re: NiFi 1.11.1: EOFException in FlowFileAccessInputStream
Posted by Endre Kovacs <an...@protonmail.com.INVALID>.
Hi,
What cloud are you running this on?
Kubernetes version?
Can you share the storageClass you use for creating the repo PVCs?
what is the topology of the cluster? Muli zone?
Best regards,
Endre
Sent with ProtonMail Secure Email.
‐‐‐‐‐‐‐ Original Message ‐‐‐‐‐‐‐
On Monday, February 10, 2020 7:41 PM, Shawn Weeks <sw...@weeksconsulting.us> wrote:
> The more I look at the error the more it looks like the container couldn't access the persistence volume for some period of time. Can you share your yaml configuration minus sensitive stuff and what environment your deploying K8s on, aka Azure, ECS, Rancher, Openshift, etc.
>
> Thanks
> Shawn
>
> On 2/10/20, 12:27 PM, "Shawn Weeks" sweeks@weeksconsulting.us wrote:
>
> I've been testing out NiFi on K8s quite a bit lately and might be able to help. What are you using for your persistent volume, this kinda sounds like NiFi can't always access data in the content_repository folder?
>
> Thanks
> Shawn
>
> On 2/10/20, 11:53 AM, "Joe Witt" joe.witt@gmail.com wrote:
>
> ...sent a little too quick. Also seems like if it is related to your k8s
> process that the issue would occur on older nifi releases too. So that
> might be an interesting test for you.
>
> On Mon, Feb 10, 2020 at 9:52 AM Joe Witt joe.witt@gmail.com wrote:
>
> > Joe
>
> >
>
> > If you can replicate this in a non k8s environment that would likely make
>
> > resolution easier. Far too many variables/questions at this stage.
>
> >
>
> > Thanks
>
> >
>
> > On Mon, Feb 10, 2020 at 8:15 AM Joe Gresock <jg...@gmail.com> wrote:
>
> >
>
> >> I don't know if there is something wrong with my Kubernetes-ized
>
> >> deployment
>
> >> of NiFi, but I'm seeing the following error crop up a lot in my NiFi
>
> >> 1.11.1
>
> >> deployment. I can see the flow file attributes in "List Queue" for flow
>
> >> files affected by this problem, but the content claim is missing.
>
> >>
>
> >> It's not easily reproducible, but I've been running data constantly
>
> >> through
>
> >> the cluster for a few days, and every 2-4 hours or so, this problem
>
> >> appears
>
> >> in one of my processors (it's not always the same processors). The error
>
> >> only goes away when I drain the relevant queue, but obviously that's not
>
> >> an
>
> >> option in production.
>
> >>
>
> >> Any thoughts on what might be causing intermittent truncations in the
>
> >> content_repository?
>
> >>
>
> >> 2020-02-10 10:38:57,205 ERROR [Timer-Driven Process Thread-6]
>
> >> o.a.nifi.processors.standard.ReplaceText
>
> >> ReplaceText[id=1abd1d49-0170-1000-ffff-ffffda59ecb2]
>
> >> ReplaceText[id=1abd1d49-0170-1000-ffff-ffffda59ecb2] fa
>
> >> iled to process session due to
>
> >> org.apache.nifi.processor.exception.FlowFileAccessException: Could not
>
> >> read
>
> >> from
>
> >>
>
> >> StandardFlowFileRecord[uuid=b0ca5649-ca8f-4a99-88df-70e1143c5f1e,claim=StandardContentClaim
>
> >> [resourceClaim=StandardResourceClaim[id=1581318452649-212433,
>
> >> container=default, section=465], offset=0,
>
> >>
>
> >> length=6401249],offset=2199110,name=90e47e27f806d46b2e190b462db929441993d166-ED,size=318289];
>
> >> Processor Administratively Yielded for 1 sec:
>
> >> org.apache.nifi.processor.exception.FlowFileAccessException: Could not
>
> >> read
>
> >> from
>
> >>
>
> >> StandardFlowFileRecord[uuid=b0ca5649-ca8f-4a99-88df-70e1143c5f1e,claim=StandardContentClaim
>
> >> [resourceClaim=StandardResourceClaim[id=1581318452649-212433,
>
> >> container=default, section=465], offset=0,
>
> >>
>
> >> length=6401249],offset=2199110,name=90e47e27f806d46b2e190b462db929441993d166-ED,size=318289]
>
> >> org.apache.nifi.processor.exception.FlowFileAccessException: Could not
>
> >> read
>
> >> from
>
> >>
>
> >> StandardFlowFileRecord[uuid=b0ca5649-ca8f-4a99-88df-70e1143c5f1e,claim=StandardContentClaim
>
> >> [resourceClaim=StandardResourceClaim[id=1581318452649-212433,
>
> >> container=default, section=465], offset=0,
>
> >>
>
> >> length=6401249],offset=2199110,name=90e47e27f806d46b2e190b462db929441993d166-ED,size=318289]
>
> >> at
>
> >> org.apache.nifi.controller.repository.io
>
> >> .FlowFileAccessInputStream.read(FlowFileAccessInputStream.java:93)
>
> >> at
>
> >> org.apache.nifi.controller.repository.io
>
> >> .TaskTerminationInputStream.read(TaskTerminationInputStream.java:68)
>
> >> at
>
> >> org.apache.nifi.stream.io.StreamUtils.fillBuffer(StreamUtils.java:89)
>
> >> at
>
> >>
>
> >> org.apache.nifi.processors.standard.ReplaceText$RegexReplace$2.process(ReplaceText.java:532)
>
> >> at
>
> >>
>
> >> org.apache.nifi.controller.repository.StandardProcessSession.read(StandardProcessSession.java:2324)
>
> >> at
>
> >>
>
> >> org.apache.nifi.controller.repository.StandardProcessSession.read(StandardProcessSession.java:2292)
>
> >> at
>
> >>
>
> >> org.apache.nifi.controller.repository.BatchingSessionFactory$HighThroughputSession.read(BatchingSessionFactory.java:192)
>
> >> at
>
> >>
>
> >> org.apache.nifi.processors.standard.ReplaceText$RegexReplace.replace(ReplaceText.java:529)
>
> >> at
>
> >>
>
> >> org.apache.nifi.processors.standard.ReplaceText.onTrigger(ReplaceText.java:316)
>
> >> at
>
> >>
>
> >> org.apache.nifi.processor.AbstractProcessor.onTrigger(AbstractProcessor.java:27)
>
> >> at
>
> >>
>
> >> org.apache.nifi.controller.StandardProcessorNode.onTrigger(StandardProcessorNode.java:1176)
>
> >> at
>
> >>
>
> >> org.apache.nifi.controller.tasks.ConnectableTask.invoke(ConnectableTask.java:213)
>
> >> at
>
> >>
>
> >> org.apache.nifi.controller.scheduling.TimerDrivenSchedulingAgent$1.run(TimerDrivenSchedulingAgent.java:117)
>
> >> at org.apache.nifi.engine.FlowEngine$2.run(FlowEngine.java:110)
>
> >> at
>
> >> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
>
> >> at
>
> >> java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
>
> >> at
>
> >>
>
> >> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
>
> >> at
>
> >>
>
> >> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
>
> >> at
>
> >>
>
> >> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
>
> >> at
>
> >>
>
> >> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
>
> >> at java.lang.Thread.run(Thread.java:748)
>
> >> Caused by: java.io.EOFException: null
>
> >> at org.apache.nifi.stream.io
>
> >> .StreamUtils.skip(StreamUtils.java:270)
>
> >> at
>
> >> org.apache.nifi.controller.repository.io
>
> >> .ContentClaimInputStream.formDelegate(ContentClaimInputStream.java:155)
>
> >> at
>
> >> org.apache.nifi.controller.repository.io
>
> >> .ContentClaimInputStream.getDelegate(ContentClaimInputStream.java:51)
>
> >> at
>
> >> org.apache.nifi.controller.repository.io
>
> >> .ContentClaimInputStream.read(ContentClaimInputStream.java:89)
>
> >> at
>
> >> org.apache.nifi.controller.repository.io
>
> >> .DisableOnCloseInputStream.read(DisableOnCloseInputStream.java:49)
>
> >> at
>
> >> org.apache.nifi.controller.repository.io
>
> >> .LimitedInputStream.read(LimitedInputStream.java:86)
>
> >> at
>
> >> org.apache.nifi.controller.repository.io
>
> >> .DisableOnCloseInputStream.read(DisableOnCloseInputStream.java:49)
>
> >> at
>
> >> org.apache.nifi.stream.io
>
> >> .ByteCountingInputStream.read(ByteCountingInputStream.java:52)
>
> >> at java.io.FilterInputStream.read(FilterInputStream.java:133)
>
> >> at
>
> >> org.apache.nifi.controller.repository.io
>
> >> .FlowFileAccessInputStream.read(FlowFileAccessInputStream.java:82)
>
> >> ... 20 common frames omitted
>
> >>
>
> >>
>
> >> Joe Gresock
>
> >>
>
> >
>
Re: NiFi 1.11.1: EOFException in FlowFileAccessInputStream
Posted by Shawn Weeks <sw...@weeksconsulting.us>.
The more I look at the error the more it looks like the container couldn't access the persistence volume for some period of time. Can you share your yaml configuration minus sensitive stuff and what environment your deploying K8s on, aka Azure, ECS, Rancher, Openshift, etc.
Thanks
Shawn
On 2/10/20, 12:27 PM, "Shawn Weeks" <sw...@weeksconsulting.us> wrote:
I've been testing out NiFi on K8s quite a bit lately and might be able to help. What are you using for your persistent volume, this kinda sounds like NiFi can't always access data in the content_repository folder?
Thanks
Shawn
On 2/10/20, 11:53 AM, "Joe Witt" <jo...@gmail.com> wrote:
...sent a little too quick. Also seems like if it is related to your k8s
process that the issue would occur on older nifi releases too. So that
might be an interesting test for you.
On Mon, Feb 10, 2020 at 9:52 AM Joe Witt <jo...@gmail.com> wrote:
> Joe
>
> If you can replicate this in a non k8s environment that would likely make
> resolution easier. Far too many variables/questions at this stage.
>
> Thanks
>
> On Mon, Feb 10, 2020 at 8:15 AM Joe Gresock <jg...@gmail.com> wrote:
>
>> I don't know if there is something wrong with my Kubernetes-ized
>> deployment
>> of NiFi, but I'm seeing the following error crop up a lot in my NiFi
>> 1.11.1
>> deployment. I can see the flow file attributes in "List Queue" for flow
>> files affected by this problem, but the content claim is missing.
>>
>> It's not easily reproducible, but I've been running data constantly
>> through
>> the cluster for a few days, and every 2-4 hours or so, this problem
>> appears
>> in one of my processors (it's not always the same processors). The error
>> only goes away when I drain the relevant queue, but obviously that's not
>> an
>> option in production.
>>
>> Any thoughts on what might be causing intermittent truncations in the
>> content_repository?
>>
>> 2020-02-10 10:38:57,205 ERROR [Timer-Driven Process Thread-6]
>> o.a.nifi.processors.standard.ReplaceText
>> ReplaceText[id=1abd1d49-0170-1000-ffff-ffffda59ecb2]
>> ReplaceText[id=1abd1d49-0170-1000-ffff-ffffda59ecb2] fa
>> iled to process session due to
>> org.apache.nifi.processor.exception.FlowFileAccessException: Could not
>> read
>> from
>>
>> StandardFlowFileRecord[uuid=b0ca5649-ca8f-4a99-88df-70e1143c5f1e,claim=StandardContentClaim
>> [resourceClaim=StandardResourceClaim[id=1581318452649-212433,
>> container=default, section=465], offset=0,
>>
>> length=6401249],offset=2199110,name=90e47e27f806d46b2e190b462db929441993d166-ED,size=318289];
>> Processor Administratively Yielded for 1 sec:
>> org.apache.nifi.processor.exception.FlowFileAccessException: Could not
>> read
>> from
>>
>> StandardFlowFileRecord[uuid=b0ca5649-ca8f-4a99-88df-70e1143c5f1e,claim=StandardContentClaim
>> [resourceClaim=StandardResourceClaim[id=1581318452649-212433,
>> container=default, section=465], offset=0,
>>
>> length=6401249],offset=2199110,name=90e47e27f806d46b2e190b462db929441993d166-ED,size=318289]
>> org.apache.nifi.processor.exception.FlowFileAccessException: Could not
>> read
>> from
>>
>> StandardFlowFileRecord[uuid=b0ca5649-ca8f-4a99-88df-70e1143c5f1e,claim=StandardContentClaim
>> [resourceClaim=StandardResourceClaim[id=1581318452649-212433,
>> container=default, section=465], offset=0,
>>
>> length=6401249],offset=2199110,name=90e47e27f806d46b2e190b462db929441993d166-ED,size=318289]
>> at
>> org.apache.nifi.controller.repository.io
>> .FlowFileAccessInputStream.read(FlowFileAccessInputStream.java:93)
>> at
>> org.apache.nifi.controller.repository.io
>> .TaskTerminationInputStream.read(TaskTerminationInputStream.java:68)
>> at
>> org.apache.nifi.stream.io.StreamUtils.fillBuffer(StreamUtils.java:89)
>> at
>>
>> org.apache.nifi.processors.standard.ReplaceText$RegexReplace$2.process(ReplaceText.java:532)
>> at
>>
>> org.apache.nifi.controller.repository.StandardProcessSession.read(StandardProcessSession.java:2324)
>> at
>>
>> org.apache.nifi.controller.repository.StandardProcessSession.read(StandardProcessSession.java:2292)
>> at
>>
>> org.apache.nifi.controller.repository.BatchingSessionFactory$HighThroughputSession.read(BatchingSessionFactory.java:192)
>> at
>>
>> org.apache.nifi.processors.standard.ReplaceText$RegexReplace.replace(ReplaceText.java:529)
>> at
>>
>> org.apache.nifi.processors.standard.ReplaceText.onTrigger(ReplaceText.java:316)
>> at
>>
>> org.apache.nifi.processor.AbstractProcessor.onTrigger(AbstractProcessor.java:27)
>> at
>>
>> org.apache.nifi.controller.StandardProcessorNode.onTrigger(StandardProcessorNode.java:1176)
>> at
>>
>> org.apache.nifi.controller.tasks.ConnectableTask.invoke(ConnectableTask.java:213)
>> at
>>
>> org.apache.nifi.controller.scheduling.TimerDrivenSchedulingAgent$1.run(TimerDrivenSchedulingAgent.java:117)
>> at org.apache.nifi.engine.FlowEngine$2.run(FlowEngine.java:110)
>> at
>> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
>> at
>> java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
>> at
>>
>> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
>> at
>>
>> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
>> at
>>
>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
>> at
>>
>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
>> at java.lang.Thread.run(Thread.java:748)
>> Caused by: java.io.EOFException: null
>> at org.apache.nifi.stream.io
>> .StreamUtils.skip(StreamUtils.java:270)
>> at
>> org.apache.nifi.controller.repository.io
>> .ContentClaimInputStream.formDelegate(ContentClaimInputStream.java:155)
>> at
>> org.apache.nifi.controller.repository.io
>> .ContentClaimInputStream.getDelegate(ContentClaimInputStream.java:51)
>> at
>> org.apache.nifi.controller.repository.io
>> .ContentClaimInputStream.read(ContentClaimInputStream.java:89)
>> at
>> org.apache.nifi.controller.repository.io
>> .DisableOnCloseInputStream.read(DisableOnCloseInputStream.java:49)
>> at
>> org.apache.nifi.controller.repository.io
>> .LimitedInputStream.read(LimitedInputStream.java:86)
>> at
>> org.apache.nifi.controller.repository.io
>> .DisableOnCloseInputStream.read(DisableOnCloseInputStream.java:49)
>> at
>> org.apache.nifi.stream.io
>> .ByteCountingInputStream.read(ByteCountingInputStream.java:52)
>> at java.io.FilterInputStream.read(FilterInputStream.java:133)
>> at
>> org.apache.nifi.controller.repository.io
>> .FlowFileAccessInputStream.read(FlowFileAccessInputStream.java:82)
>> ... 20 common frames omitted
>>
>>
>> Joe Gresock
>>
>
Re: NiFi 1.11.1: EOFException in FlowFileAccessInputStream
Posted by Shawn Weeks <sw...@weeksconsulting.us>.
I've been testing out NiFi on K8s quite a bit lately and might be able to help. What are you using for your persistent volume, this kinda sounds like NiFi can't always access data in the content_repository folder?
Thanks
Shawn
On 2/10/20, 11:53 AM, "Joe Witt" <jo...@gmail.com> wrote:
...sent a little too quick. Also seems like if it is related to your k8s
process that the issue would occur on older nifi releases too. So that
might be an interesting test for you.
On Mon, Feb 10, 2020 at 9:52 AM Joe Witt <jo...@gmail.com> wrote:
> Joe
>
> If you can replicate this in a non k8s environment that would likely make
> resolution easier. Far too many variables/questions at this stage.
>
> Thanks
>
> On Mon, Feb 10, 2020 at 8:15 AM Joe Gresock <jg...@gmail.com> wrote:
>
>> I don't know if there is something wrong with my Kubernetes-ized
>> deployment
>> of NiFi, but I'm seeing the following error crop up a lot in my NiFi
>> 1.11.1
>> deployment. I can see the flow file attributes in "List Queue" for flow
>> files affected by this problem, but the content claim is missing.
>>
>> It's not easily reproducible, but I've been running data constantly
>> through
>> the cluster for a few days, and every 2-4 hours or so, this problem
>> appears
>> in one of my processors (it's not always the same processors). The error
>> only goes away when I drain the relevant queue, but obviously that's not
>> an
>> option in production.
>>
>> Any thoughts on what might be causing intermittent truncations in the
>> content_repository?
>>
>> 2020-02-10 10:38:57,205 ERROR [Timer-Driven Process Thread-6]
>> o.a.nifi.processors.standard.ReplaceText
>> ReplaceText[id=1abd1d49-0170-1000-ffff-ffffda59ecb2]
>> ReplaceText[id=1abd1d49-0170-1000-ffff-ffffda59ecb2] fa
>> iled to process session due to
>> org.apache.nifi.processor.exception.FlowFileAccessException: Could not
>> read
>> from
>>
>> StandardFlowFileRecord[uuid=b0ca5649-ca8f-4a99-88df-70e1143c5f1e,claim=StandardContentClaim
>> [resourceClaim=StandardResourceClaim[id=1581318452649-212433,
>> container=default, section=465], offset=0,
>>
>> length=6401249],offset=2199110,name=90e47e27f806d46b2e190b462db929441993d166-ED,size=318289];
>> Processor Administratively Yielded for 1 sec:
>> org.apache.nifi.processor.exception.FlowFileAccessException: Could not
>> read
>> from
>>
>> StandardFlowFileRecord[uuid=b0ca5649-ca8f-4a99-88df-70e1143c5f1e,claim=StandardContentClaim
>> [resourceClaim=StandardResourceClaim[id=1581318452649-212433,
>> container=default, section=465], offset=0,
>>
>> length=6401249],offset=2199110,name=90e47e27f806d46b2e190b462db929441993d166-ED,size=318289]
>> org.apache.nifi.processor.exception.FlowFileAccessException: Could not
>> read
>> from
>>
>> StandardFlowFileRecord[uuid=b0ca5649-ca8f-4a99-88df-70e1143c5f1e,claim=StandardContentClaim
>> [resourceClaim=StandardResourceClaim[id=1581318452649-212433,
>> container=default, section=465], offset=0,
>>
>> length=6401249],offset=2199110,name=90e47e27f806d46b2e190b462db929441993d166-ED,size=318289]
>> at
>> org.apache.nifi.controller.repository.io
>> .FlowFileAccessInputStream.read(FlowFileAccessInputStream.java:93)
>> at
>> org.apache.nifi.controller.repository.io
>> .TaskTerminationInputStream.read(TaskTerminationInputStream.java:68)
>> at
>> org.apache.nifi.stream.io.StreamUtils.fillBuffer(StreamUtils.java:89)
>> at
>>
>> org.apache.nifi.processors.standard.ReplaceText$RegexReplace$2.process(ReplaceText.java:532)
>> at
>>
>> org.apache.nifi.controller.repository.StandardProcessSession.read(StandardProcessSession.java:2324)
>> at
>>
>> org.apache.nifi.controller.repository.StandardProcessSession.read(StandardProcessSession.java:2292)
>> at
>>
>> org.apache.nifi.controller.repository.BatchingSessionFactory$HighThroughputSession.read(BatchingSessionFactory.java:192)
>> at
>>
>> org.apache.nifi.processors.standard.ReplaceText$RegexReplace.replace(ReplaceText.java:529)
>> at
>>
>> org.apache.nifi.processors.standard.ReplaceText.onTrigger(ReplaceText.java:316)
>> at
>>
>> org.apache.nifi.processor.AbstractProcessor.onTrigger(AbstractProcessor.java:27)
>> at
>>
>> org.apache.nifi.controller.StandardProcessorNode.onTrigger(StandardProcessorNode.java:1176)
>> at
>>
>> org.apache.nifi.controller.tasks.ConnectableTask.invoke(ConnectableTask.java:213)
>> at
>>
>> org.apache.nifi.controller.scheduling.TimerDrivenSchedulingAgent$1.run(TimerDrivenSchedulingAgent.java:117)
>> at org.apache.nifi.engine.FlowEngine$2.run(FlowEngine.java:110)
>> at
>> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
>> at
>> java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
>> at
>>
>> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
>> at
>>
>> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
>> at
>>
>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
>> at
>>
>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
>> at java.lang.Thread.run(Thread.java:748)
>> Caused by: java.io.EOFException: null
>> at org.apache.nifi.stream.io
>> .StreamUtils.skip(StreamUtils.java:270)
>> at
>> org.apache.nifi.controller.repository.io
>> .ContentClaimInputStream.formDelegate(ContentClaimInputStream.java:155)
>> at
>> org.apache.nifi.controller.repository.io
>> .ContentClaimInputStream.getDelegate(ContentClaimInputStream.java:51)
>> at
>> org.apache.nifi.controller.repository.io
>> .ContentClaimInputStream.read(ContentClaimInputStream.java:89)
>> at
>> org.apache.nifi.controller.repository.io
>> .DisableOnCloseInputStream.read(DisableOnCloseInputStream.java:49)
>> at
>> org.apache.nifi.controller.repository.io
>> .LimitedInputStream.read(LimitedInputStream.java:86)
>> at
>> org.apache.nifi.controller.repository.io
>> .DisableOnCloseInputStream.read(DisableOnCloseInputStream.java:49)
>> at
>> org.apache.nifi.stream.io
>> .ByteCountingInputStream.read(ByteCountingInputStream.java:52)
>> at java.io.FilterInputStream.read(FilterInputStream.java:133)
>> at
>> org.apache.nifi.controller.repository.io
>> .FlowFileAccessInputStream.read(FlowFileAccessInputStream.java:82)
>> ... 20 common frames omitted
>>
>>
>> Joe Gresock
>>
>
Re: NiFi 1.11.1: EOFException in FlowFileAccessInputStream
Posted by Joe Witt <jo...@gmail.com>.
...sent a little too quick. Also seems like if it is related to your k8s
process that the issue would occur on older nifi releases too. So that
might be an interesting test for you.
On Mon, Feb 10, 2020 at 9:52 AM Joe Witt <jo...@gmail.com> wrote:
> Joe
>
> If you can replicate this in a non k8s environment that would likely make
> resolution easier. Far too many variables/questions at this stage.
>
> Thanks
>
> On Mon, Feb 10, 2020 at 8:15 AM Joe Gresock <jg...@gmail.com> wrote:
>
>> I don't know if there is something wrong with my Kubernetes-ized
>> deployment
>> of NiFi, but I'm seeing the following error crop up a lot in my NiFi
>> 1.11.1
>> deployment. I can see the flow file attributes in "List Queue" for flow
>> files affected by this problem, but the content claim is missing.
>>
>> It's not easily reproducible, but I've been running data constantly
>> through
>> the cluster for a few days, and every 2-4 hours or so, this problem
>> appears
>> in one of my processors (it's not always the same processors). The error
>> only goes away when I drain the relevant queue, but obviously that's not
>> an
>> option in production.
>>
>> Any thoughts on what might be causing intermittent truncations in the
>> content_repository?
>>
>> 2020-02-10 10:38:57,205 ERROR [Timer-Driven Process Thread-6]
>> o.a.nifi.processors.standard.ReplaceText
>> ReplaceText[id=1abd1d49-0170-1000-ffff-ffffda59ecb2]
>> ReplaceText[id=1abd1d49-0170-1000-ffff-ffffda59ecb2] fa
>> iled to process session due to
>> org.apache.nifi.processor.exception.FlowFileAccessException: Could not
>> read
>> from
>>
>> StandardFlowFileRecord[uuid=b0ca5649-ca8f-4a99-88df-70e1143c5f1e,claim=StandardContentClaim
>> [resourceClaim=StandardResourceClaim[id=1581318452649-212433,
>> container=default, section=465], offset=0,
>>
>> length=6401249],offset=2199110,name=90e47e27f806d46b2e190b462db929441993d166-ED,size=318289];
>> Processor Administratively Yielded for 1 sec:
>> org.apache.nifi.processor.exception.FlowFileAccessException: Could not
>> read
>> from
>>
>> StandardFlowFileRecord[uuid=b0ca5649-ca8f-4a99-88df-70e1143c5f1e,claim=StandardContentClaim
>> [resourceClaim=StandardResourceClaim[id=1581318452649-212433,
>> container=default, section=465], offset=0,
>>
>> length=6401249],offset=2199110,name=90e47e27f806d46b2e190b462db929441993d166-ED,size=318289]
>> org.apache.nifi.processor.exception.FlowFileAccessException: Could not
>> read
>> from
>>
>> StandardFlowFileRecord[uuid=b0ca5649-ca8f-4a99-88df-70e1143c5f1e,claim=StandardContentClaim
>> [resourceClaim=StandardResourceClaim[id=1581318452649-212433,
>> container=default, section=465], offset=0,
>>
>> length=6401249],offset=2199110,name=90e47e27f806d46b2e190b462db929441993d166-ED,size=318289]
>> at
>> org.apache.nifi.controller.repository.io
>> .FlowFileAccessInputStream.read(FlowFileAccessInputStream.java:93)
>> at
>> org.apache.nifi.controller.repository.io
>> .TaskTerminationInputStream.read(TaskTerminationInputStream.java:68)
>> at
>> org.apache.nifi.stream.io.StreamUtils.fillBuffer(StreamUtils.java:89)
>> at
>>
>> org.apache.nifi.processors.standard.ReplaceText$RegexReplace$2.process(ReplaceText.java:532)
>> at
>>
>> org.apache.nifi.controller.repository.StandardProcessSession.read(StandardProcessSession.java:2324)
>> at
>>
>> org.apache.nifi.controller.repository.StandardProcessSession.read(StandardProcessSession.java:2292)
>> at
>>
>> org.apache.nifi.controller.repository.BatchingSessionFactory$HighThroughputSession.read(BatchingSessionFactory.java:192)
>> at
>>
>> org.apache.nifi.processors.standard.ReplaceText$RegexReplace.replace(ReplaceText.java:529)
>> at
>>
>> org.apache.nifi.processors.standard.ReplaceText.onTrigger(ReplaceText.java:316)
>> at
>>
>> org.apache.nifi.processor.AbstractProcessor.onTrigger(AbstractProcessor.java:27)
>> at
>>
>> org.apache.nifi.controller.StandardProcessorNode.onTrigger(StandardProcessorNode.java:1176)
>> at
>>
>> org.apache.nifi.controller.tasks.ConnectableTask.invoke(ConnectableTask.java:213)
>> at
>>
>> org.apache.nifi.controller.scheduling.TimerDrivenSchedulingAgent$1.run(TimerDrivenSchedulingAgent.java:117)
>> at org.apache.nifi.engine.FlowEngine$2.run(FlowEngine.java:110)
>> at
>> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
>> at
>> java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
>> at
>>
>> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
>> at
>>
>> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
>> at
>>
>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
>> at
>>
>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
>> at java.lang.Thread.run(Thread.java:748)
>> Caused by: java.io.EOFException: null
>> at org.apache.nifi.stream.io
>> .StreamUtils.skip(StreamUtils.java:270)
>> at
>> org.apache.nifi.controller.repository.io
>> .ContentClaimInputStream.formDelegate(ContentClaimInputStream.java:155)
>> at
>> org.apache.nifi.controller.repository.io
>> .ContentClaimInputStream.getDelegate(ContentClaimInputStream.java:51)
>> at
>> org.apache.nifi.controller.repository.io
>> .ContentClaimInputStream.read(ContentClaimInputStream.java:89)
>> at
>> org.apache.nifi.controller.repository.io
>> .DisableOnCloseInputStream.read(DisableOnCloseInputStream.java:49)
>> at
>> org.apache.nifi.controller.repository.io
>> .LimitedInputStream.read(LimitedInputStream.java:86)
>> at
>> org.apache.nifi.controller.repository.io
>> .DisableOnCloseInputStream.read(DisableOnCloseInputStream.java:49)
>> at
>> org.apache.nifi.stream.io
>> .ByteCountingInputStream.read(ByteCountingInputStream.java:52)
>> at java.io.FilterInputStream.read(FilterInputStream.java:133)
>> at
>> org.apache.nifi.controller.repository.io
>> .FlowFileAccessInputStream.read(FlowFileAccessInputStream.java:82)
>> ... 20 common frames omitted
>>
>>
>> Joe Gresock
>>
>
Re: NiFi 1.11.1: EOFException in FlowFileAccessInputStream
Posted by Joe Witt <jo...@gmail.com>.
Joe
If you can replicate this in a non k8s environment that would likely make
resolution easier. Far too many variables/questions at this stage.
Thanks
On Mon, Feb 10, 2020 at 8:15 AM Joe Gresock <jg...@gmail.com> wrote:
> I don't know if there is something wrong with my Kubernetes-ized deployment
> of NiFi, but I'm seeing the following error crop up a lot in my NiFi 1.11.1
> deployment. I can see the flow file attributes in "List Queue" for flow
> files affected by this problem, but the content claim is missing.
>
> It's not easily reproducible, but I've been running data constantly through
> the cluster for a few days, and every 2-4 hours or so, this problem appears
> in one of my processors (it's not always the same processors). The error
> only goes away when I drain the relevant queue, but obviously that's not an
> option in production.
>
> Any thoughts on what might be causing intermittent truncations in the
> content_repository?
>
> 2020-02-10 10:38:57,205 ERROR [Timer-Driven Process Thread-6]
> o.a.nifi.processors.standard.ReplaceText
> ReplaceText[id=1abd1d49-0170-1000-ffff-ffffda59ecb2]
> ReplaceText[id=1abd1d49-0170-1000-ffff-ffffda59ecb2] fa
> iled to process session due to
> org.apache.nifi.processor.exception.FlowFileAccessException: Could not read
> from
>
> StandardFlowFileRecord[uuid=b0ca5649-ca8f-4a99-88df-70e1143c5f1e,claim=StandardContentClaim
> [resourceClaim=StandardResourceClaim[id=1581318452649-212433,
> container=default, section=465], offset=0,
>
> length=6401249],offset=2199110,name=90e47e27f806d46b2e190b462db929441993d166-ED,size=318289];
> Processor Administratively Yielded for 1 sec:
> org.apache.nifi.processor.exception.FlowFileAccessException: Could not read
> from
>
> StandardFlowFileRecord[uuid=b0ca5649-ca8f-4a99-88df-70e1143c5f1e,claim=StandardContentClaim
> [resourceClaim=StandardResourceClaim[id=1581318452649-212433,
> container=default, section=465], offset=0,
>
> length=6401249],offset=2199110,name=90e47e27f806d46b2e190b462db929441993d166-ED,size=318289]
> org.apache.nifi.processor.exception.FlowFileAccessException: Could not read
> from
>
> StandardFlowFileRecord[uuid=b0ca5649-ca8f-4a99-88df-70e1143c5f1e,claim=StandardContentClaim
> [resourceClaim=StandardResourceClaim[id=1581318452649-212433,
> container=default, section=465], offset=0,
>
> length=6401249],offset=2199110,name=90e47e27f806d46b2e190b462db929441993d166-ED,size=318289]
> at
> org.apache.nifi.controller.repository.io
> .FlowFileAccessInputStream.read(FlowFileAccessInputStream.java:93)
> at
> org.apache.nifi.controller.repository.io
> .TaskTerminationInputStream.read(TaskTerminationInputStream.java:68)
> at
> org.apache.nifi.stream.io.StreamUtils.fillBuffer(StreamUtils.java:89)
> at
>
> org.apache.nifi.processors.standard.ReplaceText$RegexReplace$2.process(ReplaceText.java:532)
> at
>
> org.apache.nifi.controller.repository.StandardProcessSession.read(StandardProcessSession.java:2324)
> at
>
> org.apache.nifi.controller.repository.StandardProcessSession.read(StandardProcessSession.java:2292)
> at
>
> org.apache.nifi.controller.repository.BatchingSessionFactory$HighThroughputSession.read(BatchingSessionFactory.java:192)
> at
>
> org.apache.nifi.processors.standard.ReplaceText$RegexReplace.replace(ReplaceText.java:529)
> at
>
> org.apache.nifi.processors.standard.ReplaceText.onTrigger(ReplaceText.java:316)
> at
>
> org.apache.nifi.processor.AbstractProcessor.onTrigger(AbstractProcessor.java:27)
> at
>
> org.apache.nifi.controller.StandardProcessorNode.onTrigger(StandardProcessorNode.java:1176)
> at
>
> org.apache.nifi.controller.tasks.ConnectableTask.invoke(ConnectableTask.java:213)
> at
>
> org.apache.nifi.controller.scheduling.TimerDrivenSchedulingAgent$1.run(TimerDrivenSchedulingAgent.java:117)
> at org.apache.nifi.engine.FlowEngine$2.run(FlowEngine.java:110)
> at
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
> at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
> at
>
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
> at
>
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
> at
>
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> at
>
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> at java.lang.Thread.run(Thread.java:748)
> Caused by: java.io.EOFException: null
> at org.apache.nifi.stream.io
> .StreamUtils.skip(StreamUtils.java:270)
> at
> org.apache.nifi.controller.repository.io
> .ContentClaimInputStream.formDelegate(ContentClaimInputStream.java:155)
> at
> org.apache.nifi.controller.repository.io
> .ContentClaimInputStream.getDelegate(ContentClaimInputStream.java:51)
> at
> org.apache.nifi.controller.repository.io
> .ContentClaimInputStream.read(ContentClaimInputStream.java:89)
> at
> org.apache.nifi.controller.repository.io
> .DisableOnCloseInputStream.read(DisableOnCloseInputStream.java:49)
> at
> org.apache.nifi.controller.repository.io
> .LimitedInputStream.read(LimitedInputStream.java:86)
> at
> org.apache.nifi.controller.repository.io
> .DisableOnCloseInputStream.read(DisableOnCloseInputStream.java:49)
> at
> org.apache.nifi.stream.io
> .ByteCountingInputStream.read(ByteCountingInputStream.java:52)
> at java.io.FilterInputStream.read(FilterInputStream.java:133)
> at
> org.apache.nifi.controller.repository.io
> .FlowFileAccessInputStream.read(FlowFileAccessInputStream.java:82)
> ... 20 common frames omitted
>
>
> Joe Gresock
>