You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@nifi.apache.org by "Mark Ward (Jira)" <ji...@apache.org> on 2022/11/18 16:07:00 UTC
[jira] [Commented] (NIFI-10660) AvroReader issue after upgrading to Nifi 1.18.0
[ https://issues.apache.org/jira/browse/NIFI-10660?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17635936#comment-17635936 ]
Mark Ward commented on NIFI-10660:
----------------------------------
Hi,
I have also faced this issue after updating from 1.15.3 to 1.18.0.
In my case incoming data was CSV, but this data was read correctly and converted to Avro with an embedded schema. The problem arose when subsequent processors read the Avro content using the embedded schema. Furthermore, the alias had to contain a space.
See here for more info: https://apachenifi.slack.com/archives/C0L9VCD47/p1668772654762409
Thanks
Mark
> AvroReader issue after upgrading to Nifi 1.18.0
> -----------------------------------------------
>
> Key: NIFI-10660
> URL: https://issues.apache.org/jira/browse/NIFI-10660
> Project: Apache NiFi
> Issue Type: Bug
> Affects Versions: 1.18.0
> Reporter: Sandro Berger
> Priority: Major
>
> After upgrading from 1.15.3 to 1.18.0 we saw some issue regarding AvroReader when the following conditions are met:
> # The AVRO schema in the flowFile uses the "aliases" attribute
> # There is a record in the flowFile where a field with set alias has a value of "null"
> The use of aliases and null values used to work for us before upgrading Nifi without issues
> Example schema:
> {code:java}
> {
> "type": "record",
> "name": "test",
> "fields": [
> { "name" : "id", "type" : "int", "aliases" : ["F1"] },
> { "name" : "field1", "type" : [ "string", "null" ], "aliases" : ["F2"] },
> { "name" : "field2", "type" : [ "string", "null" ], "aliases" : ["F3"] }
> ]
> }
> {code}
> Example data:
> {code:java}
> [ {
> "id" : 1,
> "field1" : "field1content",
> "field2" : "field2content"
> }, {
> "id" : 2,
> "field1" : null,
> "field2" : "field2content"
> } ]{code}
> Note: If we remove the "aliases" from the schema or make sure that there is never a field with a value of null, we don't have any errors in Nifi 1.18.0.
>
> ERROR log:
> {code:java}
> 2022-10-14 16:30:50,617 ERROR [Timer-Driven Process Thread-6] o.a.n.processors.standard.ConvertRecord ConvertRecord[id=c7ee1f0f-10ac-116e-9e4c-300530853ab8] Failed to process StandardFlow
> FileRecord[uuid=1d201fdb-e507-4d51-87c2-fbc895a450f3,claim=StandardContentClaim [resourceClaim=StandardResourceClaim[id=1665753427357-4670, container=default, section=574], offset=18384,
> length=324],offset=0,name=1d201fdb-e507-4d51-87c2-fbc895a450f3,size=324]; will route to failure
> org.apache.nifi.processor.exception.ProcessException: Could not parse incoming data
> at org.apache.nifi.processors.standard.AbstractRecordProcessor$1.process(AbstractRecordProcessor.java:171)
> at org.apache.nifi.controller.repository.StandardProcessSession.write(StandardProcessSession.java:3434)
> at org.apache.nifi.processors.standard.AbstractRecordProcessor.onTrigger(AbstractRecordProcessor.java:122)
> at org.apache.nifi.processor.AbstractProcessor.onTrigger(AbstractProcessor.java:27)
> at org.apache.nifi.controller.StandardProcessorNode.onTrigger(StandardProcessorNode.java:1354)
> at org.apache.nifi.controller.tasks.ConnectableTask.invoke(ConnectableTask.java:246)
> at org.apache.nifi.controller.scheduling.AbstractTimeBasedSchedulingAgent.lambda$doScheduleOnce$0(AbstractTimeBasedSchedulingAgent.java:59)
> at org.apache.nifi.engine.FlowEngine$2.run(FlowEngine.java:110)
> at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515)
> at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
> at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:304)
> at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
> at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
> at java.base/java.lang.Thread.run(Thread.java:829)
> Caused by: org.apache.nifi.serialization.MalformedRecordException: Error while getting next record
> at org.apache.nifi.avro.AvroRecordReader.nextRecord(AvroRecordReader.java:50)
> at org.apache.nifi.serialization.RecordReader.nextRecord(RecordReader.java:50)
> at jdk.internal.reflect.GeneratedMethodAccessor150.invoke(Unknown Source)
> at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.base/java.lang.reflect.Method.invoke(Method.java:566)
> at org.apache.nifi.controller.service.StandardControllerServiceInvocationHandler.invoke(StandardControllerServiceInvocationHandler.java:254)
> at org.apache.nifi.controller.service.StandardControllerServiceInvocationHandler.access$100(StandardControllerServiceInvocationHandler.java:38)
> at org.apache.nifi.controller.service.StandardControllerServiceInvocationHandler$ProxiedReturnObjectInvocationHandler.invoke(StandardControllerServiceInvocationHandler.java:240)
> at com.sun.proxy.$Proxy277.nextRecord(Unknown Source)
> at org.apache.nifi.processors.standard.AbstractRecordProcessor$1.process(AbstractRecordProcessor.java:157)
> ... 13 common frames omitted
> Caused by: org.apache.avro.AvroRuntimeException: Not a valid schema field: F2
> at org.apache.avro.generic.GenericData$Record.get(GenericData.java:268)
> at org.apache.nifi.avro.AvroTypeUtil.convertAvroRecordToMap(AvroTypeUtil.java:916)
> at org.apache.nifi.avro.AvroTypeUtil.convertAvroRecordToMap(AvroTypeUtil.java:905)
> at org.apache.nifi.avro.AvroRecordReader.nextRecord(AvroRecordReader.java:43)
> ... 22 common frames omitted {code}
> The error mentions the alias of the field containing the null value.
> Could be related to the upgrade of avro introduced with Nifi 1.18.0: https://issues.apache.org/jira/browse/NIFI-10311
--
This message was sent by Atlassian Jira
(v8.20.10#820010)