You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@nifi.apache.org by Martin Mucha <al...@gmail.com> on 2018/01/22 16:06:02 UTC
referencing inexisting file in content repository
Hi,
I'd like to ask what this could mean. ValidateRecord processor from master
branch generates this error for me. I admit, I did some alterations to it
in past, but I checkouted to origin/master, run mvn clean and started
completely from scratch only with data in master branch.
Question: is there something outside of nifi folder which has to be
cleaned? Some /var/whatever or /usr/lib/whatever?
If not -- this message simply means, that in content repository is
referenced file, which does not exist.
Exception below is taken from log and debugger reveals more; actual error
happens here:
sun.nio.fs.UnixException#rethrowAsIOException(sun.nio.fs.UnixPath,
sun.nio.fs.UnixPath)
I googled out, that it can be caused by wrong limits, so I set them
appropriately, but issue did not go away:
ulimit -a
core file size (blocks, -c) unlimited
data seg size (kbytes, -d) unlimited
scheduling priority (-e) 0
file size (blocks, -f) unlimited
pending signals (-i) 29595
max locked memory (kbytes, -l) 64
max memory size (kbytes, -m) unlimited
open files (-n) 50000
pipe size (512 bytes, -p) 8
POSIX message queues (bytes, -q) 819200
real-time priority (-r) 0
stack size (kbytes, -s) 8192
cpu time (seconds, -t) unlimited
max user processes (-u) 10000
virtual memory (kbytes, -v) unlimited
file locks (-x) unlimited
from log:
2018-01-22 15:44:35,416 ERROR [Timer-Driven Process Thread-10]
o.a.n.processors.standard.ValidateRecord
ValidateRecord[id=1e25a850-0161-1000-788b-4f427cfb5631]
Failed to process StandardFlow
FileRecord[uuid=43100221-8ea6-48c4-abe1-d51965ecd8ae,claim=StandardContentClaim
[resourceClaim=StandardResourceClaim[id=1516632234823-1, container=default,
section=1], offset=34, length=17],
offset=0,name=808180285294,size=17]; will route to failure:
java.io.IOException: Not a data file.
java.io.IOException: Not a data file.
at org.apache.avro.file.DataFileStream.initialize(
DataFileStream.java:105)
at org.apache.avro.file.DataFileStream.<init>(
DataFileStream.java:84)
at org.apache.nifi.avro.AvroReaderWithEmbeddedSchema.<init>(
AvroReaderWithEmbeddedSchema.java:38)
at org.apache.nifi.avro.AvroReader.createRecordReader(
AvroReader.java:89)
at org.apache.nifi.serialization.RecordReaderFactory.
createRecordReader(RecordReaderFactory.java:46)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(
NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(
DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.apache.nifi.controller.service.StandardControllerServiceInvoc
ationHandler.invoke(StandardControllerServiceInvocationHandler.java:89)
at com.sun.proxy.$Proxy81.createRecordReader(Unknown Source)
at org.apache.nifi.processors.standard.ValidateRecord.
onTrigger(ValidateRecord.java:295)
at org.apache.nifi.processor.AbstractProcessor.onTrigger(
AbstractProcessor.java:27)
at org.apache.nifi.controller.StandardProcessorNode.onTrigger(
StandardProcessorNode.java:1122)
at org.apache.nifi.controller.tasks.ContinuallyRunProcessorTask.
call(ContinuallyRunProcessorTask.java:147)
at org.apache.nifi.controller.tasks.ContinuallyRunProcessorTask.
call(ContinuallyRunProcessorTask.java:47)
at org.apache.nifi.controller.scheduling.
TimerDrivenSchedulingAgent$1.run(TimerDrivenSchedulingAgent.java:128)
at java.util.concurrent.Executors$RunnableAdapter.
call(Executors.java:511)
at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
at java.util.concurrent.ScheduledThreadPoolExecutor$
ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
at java.util.concurrent.ScheduledThreadPoolExecutor$
ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
at java.util.concurrent.ThreadPoolExecutor.runWorker(
ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(
ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Re: referencing inexisting file in content repository
Posted by Joe Witt <jo...@gmail.com>.
cool. glad youre making progress
On Jan 22, 2018 2:06 PM, "Martin Mucha" <al...@gmail.com> wrote:
> Very good! You are correct, I was clicking too fast when creating
> controllers.
> I thought it must have been something else as it showed that it's trying to
> read inexisting file.
>
> Thanks for pointing me to correct source of problem!
>
> Martin.
>
> 2018-01-22 17:08 GMT+01:00 Joe Witt <jo...@gmail.com>:
>
> > hello
> >
> > just appears to be failing because it is trying to read the data as an
> avro
> > object with embedded schema and it appears to not actually be that.
> >
> > can you share the data it is failing on? and a flow template?
> >
> > thanks
> >
> > On Jan 22, 2018 11:06 AM, "Martin Mucha" <al...@gmail.com> wrote:
> >
> > > Hi,
> > >
> > > I'd like to ask what this could mean. ValidateRecord processor from
> > master
> > > branch generates this error for me. I admit, I did some alterations to
> it
> > > in past, but I checkouted to origin/master, run mvn clean and started
> > > completely from scratch only with data in master branch.
> > >
> > > Question: is there something outside of nifi folder which has to be
> > > cleaned? Some /var/whatever or /usr/lib/whatever?
> > >
> > > If not -- this message simply means, that in content repository is
> > > referenced file, which does not exist.
> > >
> > > Exception below is taken from log and debugger reveals more; actual
> error
> > > happens here:
> > > sun.nio.fs.UnixException#rethrowAsIOException(sun.nio.fs.UnixPath,
> > > sun.nio.fs.UnixPath)
> > >
> > > I googled out, that it can be caused by wrong limits, so I set them
> > > appropriately, but issue did not go away:
> > >
> > > ulimit -a
> > > core file size (blocks, -c) unlimited
> > > data seg size (kbytes, -d) unlimited
> > > scheduling priority (-e) 0
> > > file size (blocks, -f) unlimited
> > > pending signals (-i) 29595
> > > max locked memory (kbytes, -l) 64
> > > max memory size (kbytes, -m) unlimited
> > > open files (-n) 50000
> > > pipe size (512 bytes, -p) 8
> > > POSIX message queues (bytes, -q) 819200
> > > real-time priority (-r) 0
> > > stack size (kbytes, -s) 8192
> > > cpu time (seconds, -t) unlimited
> > > max user processes (-u) 10000
> > > virtual memory (kbytes, -v) unlimited
> > > file locks (-x) unlimited
> > >
> > >
> > > from log:
> > >
> > >
> > > 2018-01-22 15:44:35,416 ERROR [Timer-Driven Process Thread-10]
> > > o.a.n.processors.standard.ValidateRecord
> > > ValidateRecord[id=1e25a850-0161-1000-788b-4f427cfb5631]
> > > Failed to process StandardFlow
> > > FileRecord[uuid=43100221-8ea6-48c4-abe1-d51965ecd8ae,claim=
> > > StandardContentClaim
> > > [resourceClaim=StandardResourceClaim[id=1516632234823-1,
> > > container=default,
> > > section=1], offset=34, length=17],
> > > offset=0,name=808180285294,size=17]; will route to failure:
> > > java.io.IOException: Not a data file.
> > > java.io.IOException: Not a data file.
> > > at org.apache.avro.file.DataFileStream.initialize(
> > > DataFileStream.java:105)
> > > at org.apache.avro.file.DataFileStream.<init>(
> > > DataFileStream.java:84)
> > > at org.apache.nifi.avro.AvroReaderWithEmbeddedSchema.<init>(
> > > AvroReaderWithEmbeddedSchema.java:38)
> > > at org.apache.nifi.avro.AvroReader.createRecordReader(
> > > AvroReader.java:89)
> > > at org.apache.nifi.serialization.RecordReaderFactory.
> > > createRecordReader(RecordReaderFactory.java:46)
> > > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> > > at sun.reflect.NativeMethodAccessorImpl.invoke(
> > > NativeMethodAccessorImpl.java:62)
> > > at sun.reflect.DelegatingMethodAccessorImpl.invoke(
> > > DelegatingMethodAccessorImpl.java:43)
> > > at java.lang.reflect.Method.invoke(Method.java:498)
> > > at org.apache.nifi.controller.service.
> > > StandardControllerServiceInvoc
> > > ationHandler.invoke(StandardControllerServiceInvoc
> ationHandler.java:89)
> > > at com.sun.proxy.$Proxy81.createRecordReader(Unknown Source)
> > > at org.apache.nifi.processors.standard.ValidateRecord.
> > > onTrigger(ValidateRecord.java:295)
> > > at org.apache.nifi.processor.AbstractProcessor.onTrigger(
> > > AbstractProcessor.java:27)
> > > at org.apache.nifi.controller.StandardProcessorNode.onTrigger(
> > > StandardProcessorNode.java:1122)
> > > at org.apache.nifi.controller.tasks.
> ContinuallyRunProcessorTask.
> > > call(ContinuallyRunProcessorTask.java:147)
> > > at org.apache.nifi.controller.tasks.
> ContinuallyRunProcessorTask.
> > > call(ContinuallyRunProcessorTask.java:47)
> > > at org.apache.nifi.controller.scheduling.
> > > TimerDrivenSchedulingAgent$1.run(TimerDrivenSchedulingAgent.java:128)
> > > at java.util.concurrent.Executors$RunnableAdapter.
> > > call(Executors.java:511)
> > > at java.util.concurrent.FutureTask.runAndReset(
> > > FutureTask.java:308)
> > > at java.util.concurrent.ScheduledThreadPoolExecutor$
> > > ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
> > > at java.util.concurrent.ScheduledThreadPoolExecutor$
> > > ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
> > > at java.util.concurrent.ThreadPoolExecutor.runWorker(
> > > ThreadPoolExecutor.java:1149)
> > > at java.util.concurrent.ThreadPoolExecutor$Worker.run(
> > > ThreadPoolExecutor.java:624)
> > > at java.lang.Thread.run(Thread.java:748)
> > >
> >
>
Re: referencing inexisting file in content repository
Posted by Martin Mucha <al...@gmail.com>.
Very good! You are correct, I was clicking too fast when creating
controllers.
I thought it must have been something else as it showed that it's trying to
read inexisting file.
Thanks for pointing me to correct source of problem!
Martin.
2018-01-22 17:08 GMT+01:00 Joe Witt <jo...@gmail.com>:
> hello
>
> just appears to be failing because it is trying to read the data as an avro
> object with embedded schema and it appears to not actually be that.
>
> can you share the data it is failing on? and a flow template?
>
> thanks
>
> On Jan 22, 2018 11:06 AM, "Martin Mucha" <al...@gmail.com> wrote:
>
> > Hi,
> >
> > I'd like to ask what this could mean. ValidateRecord processor from
> master
> > branch generates this error for me. I admit, I did some alterations to it
> > in past, but I checkouted to origin/master, run mvn clean and started
> > completely from scratch only with data in master branch.
> >
> > Question: is there something outside of nifi folder which has to be
> > cleaned? Some /var/whatever or /usr/lib/whatever?
> >
> > If not -- this message simply means, that in content repository is
> > referenced file, which does not exist.
> >
> > Exception below is taken from log and debugger reveals more; actual error
> > happens here:
> > sun.nio.fs.UnixException#rethrowAsIOException(sun.nio.fs.UnixPath,
> > sun.nio.fs.UnixPath)
> >
> > I googled out, that it can be caused by wrong limits, so I set them
> > appropriately, but issue did not go away:
> >
> > ulimit -a
> > core file size (blocks, -c) unlimited
> > data seg size (kbytes, -d) unlimited
> > scheduling priority (-e) 0
> > file size (blocks, -f) unlimited
> > pending signals (-i) 29595
> > max locked memory (kbytes, -l) 64
> > max memory size (kbytes, -m) unlimited
> > open files (-n) 50000
> > pipe size (512 bytes, -p) 8
> > POSIX message queues (bytes, -q) 819200
> > real-time priority (-r) 0
> > stack size (kbytes, -s) 8192
> > cpu time (seconds, -t) unlimited
> > max user processes (-u) 10000
> > virtual memory (kbytes, -v) unlimited
> > file locks (-x) unlimited
> >
> >
> > from log:
> >
> >
> > 2018-01-22 15:44:35,416 ERROR [Timer-Driven Process Thread-10]
> > o.a.n.processors.standard.ValidateRecord
> > ValidateRecord[id=1e25a850-0161-1000-788b-4f427cfb5631]
> > Failed to process StandardFlow
> > FileRecord[uuid=43100221-8ea6-48c4-abe1-d51965ecd8ae,claim=
> > StandardContentClaim
> > [resourceClaim=StandardResourceClaim[id=1516632234823-1,
> > container=default,
> > section=1], offset=34, length=17],
> > offset=0,name=808180285294,size=17]; will route to failure:
> > java.io.IOException: Not a data file.
> > java.io.IOException: Not a data file.
> > at org.apache.avro.file.DataFileStream.initialize(
> > DataFileStream.java:105)
> > at org.apache.avro.file.DataFileStream.<init>(
> > DataFileStream.java:84)
> > at org.apache.nifi.avro.AvroReaderWithEmbeddedSchema.<init>(
> > AvroReaderWithEmbeddedSchema.java:38)
> > at org.apache.nifi.avro.AvroReader.createRecordReader(
> > AvroReader.java:89)
> > at org.apache.nifi.serialization.RecordReaderFactory.
> > createRecordReader(RecordReaderFactory.java:46)
> > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> > at sun.reflect.NativeMethodAccessorImpl.invoke(
> > NativeMethodAccessorImpl.java:62)
> > at sun.reflect.DelegatingMethodAccessorImpl.invoke(
> > DelegatingMethodAccessorImpl.java:43)
> > at java.lang.reflect.Method.invoke(Method.java:498)
> > at org.apache.nifi.controller.service.
> > StandardControllerServiceInvoc
> > ationHandler.invoke(StandardControllerServiceInvocationHandler.java:89)
> > at com.sun.proxy.$Proxy81.createRecordReader(Unknown Source)
> > at org.apache.nifi.processors.standard.ValidateRecord.
> > onTrigger(ValidateRecord.java:295)
> > at org.apache.nifi.processor.AbstractProcessor.onTrigger(
> > AbstractProcessor.java:27)
> > at org.apache.nifi.controller.StandardProcessorNode.onTrigger(
> > StandardProcessorNode.java:1122)
> > at org.apache.nifi.controller.tasks.ContinuallyRunProcessorTask.
> > call(ContinuallyRunProcessorTask.java:147)
> > at org.apache.nifi.controller.tasks.ContinuallyRunProcessorTask.
> > call(ContinuallyRunProcessorTask.java:47)
> > at org.apache.nifi.controller.scheduling.
> > TimerDrivenSchedulingAgent$1.run(TimerDrivenSchedulingAgent.java:128)
> > at java.util.concurrent.Executors$RunnableAdapter.
> > call(Executors.java:511)
> > at java.util.concurrent.FutureTask.runAndReset(
> > FutureTask.java:308)
> > at java.util.concurrent.ScheduledThreadPoolExecutor$
> > ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
> > at java.util.concurrent.ScheduledThreadPoolExecutor$
> > ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
> > at java.util.concurrent.ThreadPoolExecutor.runWorker(
> > ThreadPoolExecutor.java:1149)
> > at java.util.concurrent.ThreadPoolExecutor$Worker.run(
> > ThreadPoolExecutor.java:624)
> > at java.lang.Thread.run(Thread.java:748)
> >
>
Re: referencing inexisting file in content repository
Posted by Joe Witt <jo...@gmail.com>.
hello
just appears to be failing because it is trying to read the data as an avro
object with embedded schema and it appears to not actually be that.
can you share the data it is failing on? and a flow template?
thanks
On Jan 22, 2018 11:06 AM, "Martin Mucha" <al...@gmail.com> wrote:
> Hi,
>
> I'd like to ask what this could mean. ValidateRecord processor from master
> branch generates this error for me. I admit, I did some alterations to it
> in past, but I checkouted to origin/master, run mvn clean and started
> completely from scratch only with data in master branch.
>
> Question: is there something outside of nifi folder which has to be
> cleaned? Some /var/whatever or /usr/lib/whatever?
>
> If not -- this message simply means, that in content repository is
> referenced file, which does not exist.
>
> Exception below is taken from log and debugger reveals more; actual error
> happens here:
> sun.nio.fs.UnixException#rethrowAsIOException(sun.nio.fs.UnixPath,
> sun.nio.fs.UnixPath)
>
> I googled out, that it can be caused by wrong limits, so I set them
> appropriately, but issue did not go away:
>
> ulimit -a
> core file size (blocks, -c) unlimited
> data seg size (kbytes, -d) unlimited
> scheduling priority (-e) 0
> file size (blocks, -f) unlimited
> pending signals (-i) 29595
> max locked memory (kbytes, -l) 64
> max memory size (kbytes, -m) unlimited
> open files (-n) 50000
> pipe size (512 bytes, -p) 8
> POSIX message queues (bytes, -q) 819200
> real-time priority (-r) 0
> stack size (kbytes, -s) 8192
> cpu time (seconds, -t) unlimited
> max user processes (-u) 10000
> virtual memory (kbytes, -v) unlimited
> file locks (-x) unlimited
>
>
> from log:
>
>
> 2018-01-22 15:44:35,416 ERROR [Timer-Driven Process Thread-10]
> o.a.n.processors.standard.ValidateRecord
> ValidateRecord[id=1e25a850-0161-1000-788b-4f427cfb5631]
> Failed to process StandardFlow
> FileRecord[uuid=43100221-8ea6-48c4-abe1-d51965ecd8ae,claim=
> StandardContentClaim
> [resourceClaim=StandardResourceClaim[id=1516632234823-1,
> container=default,
> section=1], offset=34, length=17],
> offset=0,name=808180285294,size=17]; will route to failure:
> java.io.IOException: Not a data file.
> java.io.IOException: Not a data file.
> at org.apache.avro.file.DataFileStream.initialize(
> DataFileStream.java:105)
> at org.apache.avro.file.DataFileStream.<init>(
> DataFileStream.java:84)
> at org.apache.nifi.avro.AvroReaderWithEmbeddedSchema.<init>(
> AvroReaderWithEmbeddedSchema.java:38)
> at org.apache.nifi.avro.AvroReader.createRecordReader(
> AvroReader.java:89)
> at org.apache.nifi.serialization.RecordReaderFactory.
> createRecordReader(RecordReaderFactory.java:46)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at sun.reflect.NativeMethodAccessorImpl.invoke(
> NativeMethodAccessorImpl.java:62)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(
> DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at org.apache.nifi.controller.service.
> StandardControllerServiceInvoc
> ationHandler.invoke(StandardControllerServiceInvocationHandler.java:89)
> at com.sun.proxy.$Proxy81.createRecordReader(Unknown Source)
> at org.apache.nifi.processors.standard.ValidateRecord.
> onTrigger(ValidateRecord.java:295)
> at org.apache.nifi.processor.AbstractProcessor.onTrigger(
> AbstractProcessor.java:27)
> at org.apache.nifi.controller.StandardProcessorNode.onTrigger(
> StandardProcessorNode.java:1122)
> at org.apache.nifi.controller.tasks.ContinuallyRunProcessorTask.
> call(ContinuallyRunProcessorTask.java:147)
> at org.apache.nifi.controller.tasks.ContinuallyRunProcessorTask.
> call(ContinuallyRunProcessorTask.java:47)
> at org.apache.nifi.controller.scheduling.
> TimerDrivenSchedulingAgent$1.run(TimerDrivenSchedulingAgent.java:128)
> at java.util.concurrent.Executors$RunnableAdapter.
> call(Executors.java:511)
> at java.util.concurrent.FutureTask.runAndReset(
> FutureTask.java:308)
> at java.util.concurrent.ScheduledThreadPoolExecutor$
> ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
> at java.util.concurrent.ScheduledThreadPoolExecutor$
> ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
> at java.util.concurrent.ThreadPoolExecutor.runWorker(
> ThreadPoolExecutor.java:1149)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(
> ThreadPoolExecutor.java:624)
> at java.lang.Thread.run(Thread.java:748)
>