You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@nifi.apache.org by Frederik <fr...@qrious.co.nz> on 2017/08/04 03:10:20 UTC
Avro Reader reading from FlowFile throws SchemaNotFoundException
Hi, I created a FlowFile with QueryDataBaseTable and want to convert this
straight to CSV via the ConvertRecord processor. I tried the AvroReader with
the following Schema Access Strategies
1. Use Embedded Avro Schema
2. Use 'Schema Text' Property and Schema Text set to ${avro.schema}
My FlowFile has the following start structure.
Obj^A^B^Vavro.schema<8a>^B{"type":"record","name":"customer_crm_summary","namespace":"any.data","fields":[{"name":"device_tac_code","type":["null","string"]}]}
I would think this seems all ok but the the AvroReader fails with the
Exception below SchemaNotFoundException
I tested this with Nifi 1.2 and 1.3
Any ideas on what this could be? thanks
2017-08-04 14:41:45,236 ERROR [Timer-Driven Process Thread-1]
o.a.n.processors.standard.ConvertRecord
ConvertRecord[id=ab1cb1cb-015d-1000-4818-59412b1f3b2b] Failed to process
records for
StandardFlowFileRecord[uuid=6019524b-c23e-4b5f-977c-8be056aa217f,claim=StandardContentClaim
[resourceClaim=StandardResourceClaim[id=1501814390874-1, container=default,
section=1], offset=0,
length=4090093],offset=0,name=7952490691251418,size=4089924]; will route to
failure: org.apache.nifi.schema.access.SchemaNotFoundException: FlowFile did
not contain appropriate attributes to determine Schema Text
org.apache.nifi.schema.access.SchemaNotFoundException: FlowFile did not
contain appropriate attributes to determine Schema Text
at
org.apache.nifi.schema.access.AvroSchemaTextStrategy.getSchema(AvroSchemaTextStrategy.java:46)
at
org.apache.nifi.serialization.SchemaRegistryService.getSchema(SchemaRegistryService.java:112)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at
org.apache.nifi.controller.service.StandardControllerServiceInvocationHandler.invoke(StandardControllerServiceInvocationHandler.java:89)
at com.sun.proxy.$Proxy121.getSchema(Unknown Source)
at
org.apache.nifi.processors.standard.AbstractRecordProcessor.onTrigger(AbstractRecordProcessor.java:106)
at
org.apache.nifi.processor.AbstractProcessor.onTrigger(AbstractProcessor.java:27)
at
org.apache.nifi.controller.StandardProcessorNode.onTrigger(StandardProcessorNode.java:1120)
at
org.apache.nifi.controller.tasks.ContinuallyRunProcessorTask.call(ContinuallyRunProcessorTask.java:147)
at
org.apache.nifi.controller.tasks.ContinuallyRunProcessorTask.call(ContinuallyRunProcessorTask.java:47)
at
org.apache.nifi.controller.scheduling.TimerDrivenSchedulingAgent$1.run(TimerDrivenSchedulingAgent.java:132)
at
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
2017-08-04 14:41:45,243 ERROR [Timer-Driven Process Thread-1]
o.a.n.processors.standard.ConvertRecord
ConvertRecord[id=ab1cb1cb-015d-1000-4818-59412b1f3b2b] Failed to process
records for
StandardFlowFileRecord[uuid=1414d8a0-c033-4d8d-b2a3-944c6124894f,cl
--
View this message in context: http://apache-nifi-developer-list.39713.n7.nabble.com/Avro-Reader-reading-from-FlowFile-throws-SchemaNotFoundException-tp16574.html
Sent from the Apache NiFi Developer List mailing list archive at Nabble.com.
Re: Avro Reader reading from FlowFile throws SchemaNotFoundException
Posted by Bryan Bende <bb...@gmail.com>.
Hello,
I think I see what the problem is now...
The exception in your second email is coming from the CSV writer which
is set to get the schema from the "Schema Text" property, which is in
turn set to ${avro.schema}.
I believe what you showed in the section ### FLOWFILE ### is the
content of the flow file which is different than a flow file
attribute, so I don't think you actually have a flow file attribute
named avro.schema.
Ideally what you really want here is for the CSV writer to just use
the same schema that the reader used. In the unreleased code which is
currently set for 1.4.0-SNAPSHOT, there is a new Schema Access
Strategy called "Inherit Record Schema" which is what you want to
choose on the CSV writer.
In order to make it work in 1.3.0, you would need to take the schema
generated from QueryDatabaseTable and paste the schema text into the
"Schema Text" property on the CSV writer.
Alternatively you could create an AvroSchemaRegistry and declare the
schema in there, and then reference it by name.
-Bryan
On Mon, Aug 7, 2017 at 1:17 AM, Frederik <fr...@qrious.co.nz> wrote:
> Hey,
>
> no problem. Please find the template file below. Let me know if I can
> provide anything else. thanks
>
> convert_record_template.xml
> <http://apache-nifi-developer-list.39713.n7.nabble.com/file/n16590/convert_record_template.xml>
>
>
>
> --
> View this message in context: http://apache-nifi-developer-list.39713.n7.nabble.com/Avro-Reader-reading-from-FlowFile-throws-SchemaNotFoundException-tp16574p16590.html
> Sent from the Apache NiFi Developer List mailing list archive at Nabble.com.
Re: Avro Reader reading from FlowFile throws
SchemaNotFoundException
Posted by Frederik <fr...@qrious.co.nz>.
Hi guys,
thanks for the hint with the Writer. I build a 1.4 Snapshot and used Inherit
Record Schema. Worked perfectly fine.
thanks
--
View this message in context: http://apache-nifi-developer-list.39713.n7.nabble.com/Avro-Reader-reading-from-FlowFile-throws-SchemaNotFoundException-tp16574p16597.html
Sent from the Apache NiFi Developer List mailing list archive at Nabble.com.
Re: Avro Reader reading from FlowFile throws SchemaNotFoundException
Posted by Mark Payne <ma...@hotmail.com>.
Hey Frederik,
I took a look at the template. The ConvertRecord processor is configured with the CSVRecordSetWriter.
The problem appears to be that the writer is configured with the "Schema Access Strategy" of "Use 'Schema Text' Property"
and the Schema Text property is set to "${avro.schema}". As a result, it is expecting a FlowFile attribute to be present
named "avro.schema" but that attribute is not present. Unfortunately, in version 1.3.0 of NiFi you will need to use a Schema Registry
and provide the schema for the data. We knew that this was a bit of a pain point, so in the master branch, we have provided
a new option on the writers, so that you can set the Schema Access Strategy to "Inherit Record Schema." This would allow you
to use the schema that the Record Reader determines. This works far more seamlessly in this case because the schema is already
present in the data.
Unfortunately, though, I think that for now you would either have to build 'master' and use that, or you would have to provide the schema
via UpdateAttribute (to add the avro.schema attribute) or use a Schema Registry.
Hope this helps!
-Mark
> On Aug 7, 2017, at 1:17 AM, Frederik <fr...@qrious.co.nz> wrote:
>
> Hey,
>
> no problem. Please find the template file below. Let me know if I can
> provide anything else. thanks
>
> convert_record_template.xml
> <http://apache-nifi-developer-list.39713.n7.nabble.com/file/n16590/convert_record_template.xml>
>
>
>
> --
> View this message in context: http://apache-nifi-developer-list.39713.n7.nabble.com/Avro-Reader-reading-from-FlowFile-throws-SchemaNotFoundException-tp16574p16590.html
> Sent from the Apache NiFi Developer List mailing list archive at Nabble.com.
Re: Avro Reader reading from FlowFile throws
SchemaNotFoundException
Posted by Frederik <fr...@qrious.co.nz>.
Hey,
no problem. Please find the template file below. Let me know if I can
provide anything else. thanks
convert_record_template.xml
<http://apache-nifi-developer-list.39713.n7.nabble.com/file/n16590/convert_record_template.xml>
--
View this message in context: http://apache-nifi-developer-list.39713.n7.nabble.com/Avro-Reader-reading-from-FlowFile-throws-SchemaNotFoundException-tp16574p16590.html
Sent from the Apache NiFi Developer List mailing list archive at Nabble.com.
Re: Avro Reader reading from FlowFile throws SchemaNotFoundException
Posted by Joe Witt <jo...@gmail.com>.
Hello
Can you share a template of the flow configuration?
Thanks
On Sat, Aug 5, 2017 at 6:13 PM Frederik <fr...@qrious.co.nz>
wrote:
> Hi,
>
> thanks. I agree in theory this should work without any trouble. But I get
> the same error "SchemaNotFoundException" with
> Embedded Avro Schema (see exception below) and Use 'Schema Text'
>
> Also my Flow file has the avro.schema property see below.
>
> Any ideas how I could overcome this problem or what I could do wrongly.
>
> thanks
> Frederik
>
> ####### FLOW File ####
>
> Obj^A^B^Vavro.schema<8a>^B{"type":"record","name":"customer_crm_summary","namespace":"any.data","fields":[{"name":"device_tac_code","type":["null","string"]}]}
> #######
>
>
> ##### Exception after Use Embedded Avro Schema####
> 2017-08-06 12:26:13,276 ERROR [Timer-Driven Process Thread-3]
> o.a.n.processors.standard.ConvertRecord
> ConvertRecord[id=aa3816d3-015d-1000-957e-603e093ba845] Failed to convert
> records for
>
> StandardFlowFileRecord[uuid=e5aab5ef-f1df-40d5-bb49-1f44b95c9c2b,claim=StandardContentClaim
> [resourceClaim=StandardResourceClaim[id=1501979172410-783,
> container=default, section=783], offset=0,
> length=4091297],offset=0,name=8117272233752224,size=4091128]; will route to
> failure: org.apache.nifi.schema.access.SchemaNotFoundException: FlowFile
> did
> not contain appropriate attributes to determine Schema Text
> org.apache.nifi.schema.access.SchemaNotFoundException: FlowFile did not
> contain appropriate attributes to determine Schema Text
> at
>
> org.apache.nifi.schema.access.AvroSchemaTextStrategy.getSchema(AvroSchemaTextStrategy.java:46)
> at
>
> org.apache.nifi.serialization.SchemaRegistryService.getSchema(SchemaRegistryService.java:106)
> at
>
> org.apache.nifi.csv.CSVRecordSetWriter.createWriter(CSVRecordSetWriter.java:73)
> at sun.reflect.GeneratedMethodAccessor705.invoke(Unknown Source)
> at
>
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at
>
> org.apache.nifi.controller.service.StandardControllerServiceInvocationHandler.invoke(StandardControllerServiceInvocationHandler.java:89)
> at com.sun.proxy.$Proxy76.createWriter(Unknown Source)
> at
>
> org.apache.nifi.processors.standard.ConvertRecord.onTrigger(ConvertRecord.java:128)
> at
>
> org.apache.nifi.processor.AbstractProcessor.onTrigger(AbstractProcessor.java:27)
> at
>
> org.apache.nifi.controller.StandardProcessorNode.onTrigger(StandardProcessorNode.java:1118)
> at
>
> org.apache.nifi.controller.tasks.ContinuallyRunProcessorTask.call(ContinuallyRunProcessorTask.java:144)
> at
>
> org.apache.nifi.controller.tasks.ContinuallyRunProcessorTask.call(ContinuallyRunProcessorTask.java:47)
> at
>
> org.apache.nifi.controller.scheduling.TimerDrivenSchedulingAgent$1.run(TimerDrivenSchedulingAgent.java:132)
> at
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
> at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
> at
>
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
> at
>
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
> at
>
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> at
>
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> at java.lang.Thread.run(Thread.java:745)
>
> #####
>
> thanks
> Frederik
>
>
>
>
> --
> View this message in context:
> http://apache-nifi-developer-list.39713.n7.nabble.com/Avro-Reader-reading-from-FlowFile-throws-SchemaNotFoundException-tp16574p16584.html
> Sent from the Apache NiFi Developer List mailing list archive at
> Nabble.com.
>
Re: Avro Reader reading from FlowFile throws
SchemaNotFoundException
Posted by Frederik <fr...@qrious.co.nz>.
Hi,
thanks. I agree in theory this should work without any trouble. But I get
the same error "SchemaNotFoundException" with
Embedded Avro Schema (see exception below) and Use 'Schema Text'
Also my Flow file has the avro.schema property see below.
Any ideas how I could overcome this problem or what I could do wrongly.
thanks
Frederik
####### FLOW File ####
Obj^A^B^Vavro.schema<8a>^B{"type":"record","name":"customer_crm_summary","namespace":"any.data","fields":[{"name":"device_tac_code","type":["null","string"]}]}
#######
##### Exception after Use Embedded Avro Schema####
2017-08-06 12:26:13,276 ERROR [Timer-Driven Process Thread-3]
o.a.n.processors.standard.ConvertRecord
ConvertRecord[id=aa3816d3-015d-1000-957e-603e093ba845] Failed to convert
records for
StandardFlowFileRecord[uuid=e5aab5ef-f1df-40d5-bb49-1f44b95c9c2b,claim=StandardContentClaim
[resourceClaim=StandardResourceClaim[id=1501979172410-783,
container=default, section=783], offset=0,
length=4091297],offset=0,name=8117272233752224,size=4091128]; will route to
failure: org.apache.nifi.schema.access.SchemaNotFoundException: FlowFile did
not contain appropriate attributes to determine Schema Text
org.apache.nifi.schema.access.SchemaNotFoundException: FlowFile did not
contain appropriate attributes to determine Schema Text
at
org.apache.nifi.schema.access.AvroSchemaTextStrategy.getSchema(AvroSchemaTextStrategy.java:46)
at
org.apache.nifi.serialization.SchemaRegistryService.getSchema(SchemaRegistryService.java:106)
at
org.apache.nifi.csv.CSVRecordSetWriter.createWriter(CSVRecordSetWriter.java:73)
at sun.reflect.GeneratedMethodAccessor705.invoke(Unknown Source)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at
org.apache.nifi.controller.service.StandardControllerServiceInvocationHandler.invoke(StandardControllerServiceInvocationHandler.java:89)
at com.sun.proxy.$Proxy76.createWriter(Unknown Source)
at
org.apache.nifi.processors.standard.ConvertRecord.onTrigger(ConvertRecord.java:128)
at
org.apache.nifi.processor.AbstractProcessor.onTrigger(AbstractProcessor.java:27)
at
org.apache.nifi.controller.StandardProcessorNode.onTrigger(StandardProcessorNode.java:1118)
at
org.apache.nifi.controller.tasks.ContinuallyRunProcessorTask.call(ContinuallyRunProcessorTask.java:144)
at
org.apache.nifi.controller.tasks.ContinuallyRunProcessorTask.call(ContinuallyRunProcessorTask.java:47)
at
org.apache.nifi.controller.scheduling.TimerDrivenSchedulingAgent$1.run(TimerDrivenSchedulingAgent.java:132)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
#####
thanks
Frederik
--
View this message in context: http://apache-nifi-developer-list.39713.n7.nabble.com/Avro-Reader-reading-from-FlowFile-throws-SchemaNotFoundException-tp16574p16584.html
Sent from the Apache NiFi Developer List mailing list archive at Nabble.com.
Re: Avro Reader reading from FlowFile throws SchemaNotFoundException
Posted by Bryan Bende <bb...@gmail.com>.
Hello,
I'm assuming this error came from #2 when you tried to use Schema Text
set to ${avro.schema} ?
The error means your flow file doesn't have an attribute called
avro.schema, which it would need to have if you reference
${avro.schema}.
What were the results using Embedded Avro Schema? That should work.
-Bryan
On Thu, Aug 3, 2017 at 11:10 PM, Frederik <fr...@qrious.co.nz> wrote:
> Hi, I created a FlowFile with QueryDataBaseTable and want to convert this
> straight to CSV via the ConvertRecord processor. I tried the AvroReader with
> the following Schema Access Strategies
>
> 1. Use Embedded Avro Schema
> 2. Use 'Schema Text' Property and Schema Text set to ${avro.schema}
>
> My FlowFile has the following start structure.
> Obj^A^B^Vavro.schema<8a>^B{"type":"record","name":"customer_crm_summary","namespace":"any.data","fields":[{"name":"device_tac_code","type":["null","string"]}]}
> I would think this seems all ok but the the AvroReader fails with the
> Exception below SchemaNotFoundException
> I tested this with Nifi 1.2 and 1.3
> Any ideas on what this could be? thanks
>
> 2017-08-04 14:41:45,236 ERROR [Timer-Driven Process Thread-1]
> o.a.n.processors.standard.ConvertRecord
> ConvertRecord[id=ab1cb1cb-015d-1000-4818-59412b1f3b2b] Failed to process
> records for
> StandardFlowFileRecord[uuid=6019524b-c23e-4b5f-977c-8be056aa217f,claim=StandardContentClaim
> [resourceClaim=StandardResourceClaim[id=1501814390874-1, container=default,
> section=1], offset=0,
> length=4090093],offset=0,name=7952490691251418,size=4089924]; will route to
> failure: org.apache.nifi.schema.access.SchemaNotFoundException: FlowFile did
> not contain appropriate attributes to determine Schema Text
> org.apache.nifi.schema.access.SchemaNotFoundException: FlowFile did not
> contain appropriate attributes to determine Schema Text
> at
> org.apache.nifi.schema.access.AvroSchemaTextStrategy.getSchema(AvroSchemaTextStrategy.java:46)
> at
> org.apache.nifi.serialization.SchemaRegistryService.getSchema(SchemaRegistryService.java:112)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at
> org.apache.nifi.controller.service.StandardControllerServiceInvocationHandler.invoke(StandardControllerServiceInvocationHandler.java:89)
> at com.sun.proxy.$Proxy121.getSchema(Unknown Source)
> at
> org.apache.nifi.processors.standard.AbstractRecordProcessor.onTrigger(AbstractRecordProcessor.java:106)
> at
> org.apache.nifi.processor.AbstractProcessor.onTrigger(AbstractProcessor.java:27)
> at
> org.apache.nifi.controller.StandardProcessorNode.onTrigger(StandardProcessorNode.java:1120)
> at
> org.apache.nifi.controller.tasks.ContinuallyRunProcessorTask.call(ContinuallyRunProcessorTask.java:147)
> at
> org.apache.nifi.controller.tasks.ContinuallyRunProcessorTask.call(ContinuallyRunProcessorTask.java:47)
> at
> org.apache.nifi.controller.scheduling.TimerDrivenSchedulingAgent$1.run(TimerDrivenSchedulingAgent.java:132)
> at
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
> at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
> at
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
> at
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> at java.lang.Thread.run(Thread.java:745)
> 2017-08-04 14:41:45,243 ERROR [Timer-Driven Process Thread-1]
> o.a.n.processors.standard.ConvertRecord
> ConvertRecord[id=ab1cb1cb-015d-1000-4818-59412b1f3b2b] Failed to process
> records for
> StandardFlowFileRecord[uuid=1414d8a0-c033-4d8d-b2a3-944c6124894f,cl
>
>
>
>
>
>
> --
> View this message in context: http://apache-nifi-developer-list.39713.n7.nabble.com/Avro-Reader-reading-from-FlowFile-throws-SchemaNotFoundException-tp16574.html
> Sent from the Apache NiFi Developer List mailing list archive at Nabble.com.