You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@nifi.apache.org by Arsalan Siddiqi <ar...@live.com> on 2016/10/28 10:15:08 UTC
JSON to AVRO, AVRO to JSON
Hi
I want to get data from the REST server and make it available in NiFi. It
makes sense to have it in avro format so once read i want to convert the
data to avro. The server delivers data in JSON format. I have tested the
JSON and it is in the correct format. Have a look at the image to check the
flow.
I use Apache NiFi version 1.0.0. Firstly the split json processor is
correctly configured to break the incoming data into one record per file. I
do not know why but the correctly split jsons are routed to the failure
relationship.
<http://apache-nifi-developer-list.39713.n7.nabble.com/file/n13769/JSON_AVRO.png>
The data fetched from the get http (1) request is attached below.there are a
total of 10 records in the file.
json_adabas_input.txt
<http://apache-nifi-developer-list.39713.n7.nabble.com/file/n13769/json_adabas_input.txt>
The format is JSON and the file is without any errors. i do a split on
$.Records which correctly splits the input with one record per file and I do
not know why routes it to failure.
The flow file routed to failure has the following content. (2)
{"ISN":200,"AA":"50005900","AB":{"AC":"GASTON","AD":"","AE":"BODIN"},"AF":"S","AG":"M","AH":713043,"AW":[{"AX":19990801,"AY":19990831}]}
which is correct and valid JSON.
At 3 I try to convert the above JSON to avro:
Using the avro schema:
{ "type" : "record",
"name" : "adabas",
"fields" : [
{ "name" : "ISN", "type" : "int"},
{ "name" : "AA", "type" : "string"},
{ "name" : "AB", "type" : { "type" : "record", "name" : "AB", "fields" :
[ { "name" : "AC", "type" : "string"},
{ "name" : "AD", "type" : "string"},
{ "name" : "AE", "type" : "string" } ] } },
{ "name" : "AF", "type" : "string"},
{ "name" : "AG", "type" : "string"},
{ "name" : "AH", "type" : "int" },
{ "name" : "AW", "type" : { "type" : "array", "items" : { "type" :
"record", "name" : "AW", "fields" :
[ { "name" : "AX", "type" : "int" },
{ "name" : "AY", "type" : "int" }]}}} ] }
Here is the resulting avro file which is routed to success:
ConvertJSONtoAvro.ConvertJSONtoAvro
<http://apache-nifi-developer-list.39713.n7.nabble.com/file/n13769/ConvertJSONtoAvro.ConvertJSONtoAvro>
Now i expected if the file was routed to success, it would have been
correctly converted so I can simple get it back in JSON.
In the convert avro to json (4) i configured the avro schema field to the
same as the schema mentioned above and i get the follwoing error:
2016-10-28 12:08:41,439 ERROR [Timer-Driven Process Thread-7]
o.a.n.processors.avro.ConvertAvroToJSON
ConvertAvroToJSON[id=fb761b66-1010-1157-4673-c5198a522367] Failed to convert
StandardFlowFileRecord[uuid=3f234e05-6fa9-4569-9660-c2513f483016,claim=StandardContentClaim
[resourceClaim=StandardResourceClaim[id=1477643564106-1, container=default,
section=1], offset=9094, length=602],offset=0,name=adabas,size=602] from
Avro to JSON due to org.apache.nifi.processor.exception.ProcessException:
IOException thrown from
ConvertAvroToJSON[id=fb761b66-1010-1157-4673-c5198a522367]:
java.io.EOFException; transferring to failure:
org.apache.nifi.processor.exception.ProcessException: IOException thrown
from ConvertAvroToJSON[id=fb761b66-1010-1157-4673-c5198a522367]:
java.io.EOFException
2016-10-28 12:08:41,440 ERROR [Timer-Driven Process Thread-7]
o.a.n.processors.avro.ConvertAvroToJSON
org.apache.nifi.processor.exception.ProcessException: IOException thrown
from ConvertAvroToJSON[id=fb761b66-1010-1157-4673-c5198a522367]:
java.io.EOFException
at
org.apache.nifi.controller.repository.StandardProcessSession.write(StandardProcessSession.java:2388)
~[na:na]
at
org.apache.nifi.processors.avro.ConvertAvroToJSON.onTrigger(ConvertAvroToJSON.java:148)
~[nifi-avro-processors-1.0.0.jar:1.0.0]
at
org.apache.nifi.processor.AbstractProcessor.onTrigger(AbstractProcessor.java:27)
[nifi-api-1.0.0.jar:1.0.0]
at
org.apache.nifi.controller.StandardProcessorNode.onTrigger(StandardProcessorNode.java:1064)
[nifi-framework-core-1.0.0.jar:1.0.0]
at
org.apache.nifi.controller.tasks.ContinuallyRunProcessorTask.call(ContinuallyRunProcessorTask.java:136)
[nifi-framework-core-1.0.0.jar:1.0.0]
at
org.apache.nifi.controller.tasks.ContinuallyRunProcessorTask.call(ContinuallyRunProcessorTask.java:47)
[nifi-framework-core-1.0.0.jar:1.0.0]
at
org.apache.nifi.controller.scheduling.TimerDrivenSchedulingAgent$1.run(TimerDrivenSchedulingAgent.java:132)
[nifi-framework-core-1.0.0.jar:1.0.0]
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
[na:1.8.0_101]
at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
[na:1.8.0_101]
at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
[na:1.8.0_101]
at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
[na:1.8.0_101]
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
[na:1.8.0_101]
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
[na:1.8.0_101]
at java.lang.Thread.run(Thread.java:745) [na:1.8.0_101]
Caused by: java.io.EOFException: null
at org.apache.avro.io.BinaryDecoder.readInt(BinaryDecoder.java:153)
~[avro-1.7.7.jar:1.7.7]
at org.apache.avro.io.ValidatingDecoder.readInt(ValidatingDecoder.java:83)
~[avro-1.7.7.jar:1.7.7]
at
org.apache.avro.generic.GenericDatumReader.readInt(GenericDatumReader.java:444)
~[avro-1.7.7.jar:1.7.7]
at
org.apache.avro.generic.GenericDatumReader.read(GenericDatumReader.java:159)
~[avro-1.7.7.jar:1.7.7]
at
org.apache.avro.generic.GenericDatumReader.readField(GenericDatumReader.java:193)
~[avro-1.7.7.jar:1.7.7]
at
org.apache.avro.generic.GenericDatumReader.readRecord(GenericDatumReader.java:183)
~[avro-1.7.7.jar:1.7.7]
at
org.apache.avro.generic.GenericDatumReader.read(GenericDatumReader.java:151)
~[avro-1.7.7.jar:1.7.7]
at
org.apache.avro.generic.GenericDatumReader.readArray(GenericDatumReader.java:219)
~[avro-1.7.7.jar:1.7.7]
at
org.apache.avro.generic.GenericDatumReader.read(GenericDatumReader.java:153)
~[avro-1.7.7.jar:1.7.7]
at
org.apache.avro.generic.GenericDatumReader.readField(GenericDatumReader.java:193)
~[avro-1.7.7.jar:1.7.7]
at
org.apache.avro.generic.GenericDatumReader.readRecord(GenericDatumReader.java:183)
~[avro-1.7.7.jar:1.7.7]
at
org.apache.avro.generic.GenericDatumReader.read(GenericDatumReader.java:151)
~[avro-1.7.7.jar:1.7.7]
at
org.apache.avro.generic.GenericDatumReader.read(GenericDatumReader.java:142)
~[avro-1.7.7.jar:1.7.7]
at
org.apache.nifi.processors.avro.ConvertAvroToJSON$1.process(ConvertAvroToJSON.java:161)
~[nifi-avro-processors-1.0.0.jar:1.0.0]
at
org.apache.nifi.controller.repository.StandardProcessSession.write(StandardProcessSession.java:2370)
~[na:na]
... 13 common frames omitted
As Nifi has made the conversion to avro I expected it should easily convert
it back without any error.]
In this case the file is transferred to failure and nothing happens to the
file.
If I remove the configured value set at (4) for the field avro schema and
leave it empty i get the following error:
2016-10-28 13:25:42,968 ERROR [Timer-Driven Process Thread-4]
o.a.n.processors.avro.ConvertAvroToJSON
ConvertAvroToJSON[id=fb761b66-1010-1157-4673-c5198a522367] Failed to convert
StandardFlowFileRecord[uuid=3f234e05-6fa9-4569-9660-c2513f483016,claim=StandardContentClaim
[resourceClaim=StandardResourceClaim[id=1477643564106-1, container=default,
section=1], offset=17978, length=157],offset=0,name=adabas,size=157] from
Avro to JSON due to org.apache.nifi.processor.exception.ProcessException:
IOException thrown from
ConvertAvroToJSON[id=fb761b66-1010-1157-4673-c5198a522367]:
java.io.IOException: Not a data file.; transferring to failure:
org.apache.nifi.processor.exception.ProcessException: IOException thrown
from ConvertAvroToJSON[id=fb761b66-1010-1157-4673-c5198a522367]:
java.io.IOException: Not a data file.
2016-10-28 13:25:42,969 ERROR [Timer-Driven Process Thread-4]
o.a.n.processors.avro.ConvertAvroToJSON
org.apache.nifi.processor.exception.ProcessException: IOException thrown
from ConvertAvroToJSON[id=fb761b66-1010-1157-4673-c5198a522367]:
java.io.IOException: Not a data file.
at
org.apache.nifi.controller.repository.StandardProcessSession.write(StandardProcessSession.java:2388)
~[na:na]
at
org.apache.nifi.processors.avro.ConvertAvroToJSON.onTrigger(ConvertAvroToJSON.java:148)
~[nifi-avro-processors-1.0.0.jar:1.0.0]
at
org.apache.nifi.processor.AbstractProcessor.onTrigger(AbstractProcessor.java:27)
[nifi-api-1.0.0.jar:1.0.0]
at
org.apache.nifi.controller.StandardProcessorNode.onTrigger(StandardProcessorNode.java:1064)
[nifi-framework-core-1.0.0.jar:1.0.0]
at
org.apache.nifi.controller.tasks.ContinuallyRunProcessorTask.call(ContinuallyRunProcessorTask.java:136)
[nifi-framework-core-1.0.0.jar:1.0.0]
at
org.apache.nifi.controller.tasks.ContinuallyRunProcessorTask.call(ContinuallyRunProcessorTask.java:47)
[nifi-framework-core-1.0.0.jar:1.0.0]
at
org.apache.nifi.controller.scheduling.TimerDrivenSchedulingAgent$1.run(TimerDrivenSchedulingAgent.java:132)
[nifi-framework-core-1.0.0.jar:1.0.0]
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
[na:1.8.0_101]
at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
[na:1.8.0_101]
at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
[na:1.8.0_101]
at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
[na:1.8.0_101]
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
[na:1.8.0_101]
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
[na:1.8.0_101]
at java.lang.Thread.run(Thread.java:745) [na:1.8.0_101]
Caused by: java.io.IOException: Not a data file.
at org.apache.avro.file.DataFileStream.initialize(DataFileStream.java:105)
~[avro-1.7.7.jar:1.7.7]
at org.apache.avro.file.DataFileStream.<init>(DataFileStream.java:84)
~[avro-1.7.7.jar:1.7.7]
at
org.apache.nifi.processors.avro.ConvertAvroToJSON$1.process(ConvertAvroToJSON.java:179)
~[nifi-avro-processors-1.0.0.jar:1.0.0]
at
org.apache.nifi.controller.repository.StandardProcessSession.write(StandardProcessSession.java:2370)
~[na:na]
... 13 common frames omitted
If i now check the file in the failure relationship it is as follows:
{"ISN": 200, "AA": "50005900", "AB": {"AC": "GASTON", "AD": "", "AE":
"BODIN"}, "AF": "S", "AG": "M", "AH": 713043, "AW": [{"AX": 19990801, "AY":
19990831}]}
which again is correct but why the error and why routed to failure?? this
first happened with the Split json processor and now with convert avro to
json. both work fine but sent the files to failure.
Moving to the other branch where i do not specify the schema but instead use
the infer schema processor, it successfully infers the schema and routes it
to success (5). Then i use (6) to convert JSON to avro and it also transfers
the flow file to success. At (7) if I use the ${inferred.avro.schema} I get
the follwoing error:
2016-10-28 13:34:21,577 INFO [StandardProcessScheduler Thread-2]
o.a.n.c.s.TimerDrivenSchedulingAgent Scheduled
ConvertAvroToJSON[id=000aac50-0158-1000-d9b3-dd8343ed57a7] to run with 1
threads
2016-10-28 13:34:21,579 ERROR [Timer-Driven Process Thread-6]
o.a.n.processors.avro.ConvertAvroToJSON
ConvertAvroToJSON[id=000aac50-0158-1000-d9b3-dd8343ed57a7]
ConvertAvroToJSON[id=000aac50-0158-1000-d9b3-dd8343ed57a7] failed to process
due to org.apache.avro.AvroRuntimeException: Malformed data. Length is
negative: -62; rolling back session: org.apache.avro.AvroRuntimeException:
Malformed data. Length is negative: -62
2016-10-28 13:34:21,581 ERROR [Timer-Driven Process Thread-6]
o.a.n.processors.avro.ConvertAvroToJSON
org.apache.avro.AvroRuntimeException: Malformed data. Length is negative:
-62
at org.apache.avro.io.BinaryDecoder.doReadBytes(BinaryDecoder.java:336)
~[avro-1.7.7.jar:1.7.7]
at org.apache.avro.io.BinaryDecoder.readString(BinaryDecoder.java:263)
~[avro-1.7.7.jar:1.7.7]
at
org.apache.avro.io.ResolvingDecoder.readString(ResolvingDecoder.java:201)
~[avro-1.7.7.jar:1.7.7]
at
org.apache.avro.generic.GenericDatumReader.readString(GenericDatumReader.java:363)
~[avro-1.7.7.jar:1.7.7]
at
org.apache.avro.generic.GenericDatumReader.readString(GenericDatumReader.java:355)
~[avro-1.7.7.jar:1.7.7]
at
org.apache.avro.generic.GenericDatumReader.read(GenericDatumReader.java:157)
~[avro-1.7.7.jar:1.7.7]
at
org.apache.avro.generic.GenericDatumReader.readField(GenericDatumReader.java:193)
~[avro-1.7.7.jar:1.7.7]
at
org.apache.avro.generic.GenericDatumReader.readRecord(GenericDatumReader.java:183)
~[avro-1.7.7.jar:1.7.7]
at
org.apache.avro.generic.GenericDatumReader.read(GenericDatumReader.java:151)
~[avro-1.7.7.jar:1.7.7]
at
org.apache.avro.generic.GenericDatumReader.readField(GenericDatumReader.java:193)
~[avro-1.7.7.jar:1.7.7]
at
org.apache.avro.generic.GenericDatumReader.readRecord(GenericDatumReader.java:183)
~[avro-1.7.7.jar:1.7.7]
at
org.apache.avro.generic.GenericDatumReader.read(GenericDatumReader.java:151)
~[avro-1.7.7.jar:1.7.7]
at
org.apache.avro.generic.GenericDatumReader.read(GenericDatumReader.java:142)
~[avro-1.7.7.jar:1.7.7]
at
org.apache.nifi.processors.avro.ConvertAvroToJSON$1.process(ConvertAvroToJSON.java:161)
~[nifi-avro-processors-1.0.0.jar:1.0.0]
at
org.apache.nifi.controller.repository.StandardProcessSession.write(StandardProcessSession.java:2370)
~[nifi-framework-core-1.0.0.jar:1.0.0]
at
org.apache.nifi.processors.avro.ConvertAvroToJSON.onTrigger(ConvertAvroToJSON.java:148)
~[nifi-avro-processors-1.0.0.jar:1.0.0]
at
org.apache.nifi.processor.AbstractProcessor.onTrigger(AbstractProcessor.java:27)
~[nifi-api-1.0.0.jar:1.0.0]
at
org.apache.nifi.controller.StandardProcessorNode.onTrigger(StandardProcessorNode.java:1064)
[nifi-framework-core-1.0.0.jar:1.0.0]
at
org.apache.nifi.controller.tasks.ContinuallyRunProcessorTask.call(ContinuallyRunProcessorTask.java:136)
[nifi-framework-core-1.0.0.jar:1.0.0]
at
org.apache.nifi.controller.tasks.ContinuallyRunProcessorTask.call(ContinuallyRunProcessorTask.java:47)
[nifi-framework-core-1.0.0.jar:1.0.0]
at
org.apache.nifi.controller.scheduling.TimerDrivenSchedulingAgent$1.run(TimerDrivenSchedulingAgent.java:132)
[nifi-framework-core-1.0.0.jar:1.0.0]
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
[na:1.8.0_101]
at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
[na:1.8.0_101]
at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
[na:1.8.0_101]
at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
[na:1.8.0_101]
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
[na:1.8.0_101]
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
[na:1.8.0_101]
at java.lang.Thread.run(Thread.java:745) [na:1.8.0_101]
In this case the file stays in the incoming connection. It is the same
situation if i enter my schema definition.
Lasty if i remove the value for the schema field i get the same result as in
(4). The record is correctly converted to JSON but routed to failure.
I am very confused. Can anyone clarify what is going on??????
Refards
Arsalan Siddiqi
--
View this message in context: http://apache-nifi-developer-list.39713.n7.nabble.com/JSON-to-AVRO-AVRO-to-JSON-tp13769.html
Sent from the Apache NiFi Developer List mailing list archive at Nabble.com.
Re: JSON to AVRO, AVRO to JSON
Posted by Michal Klempa <mi...@gmail.com>.
Hi, to me, it seems from the first picture, that you are routing the
"split" connection out of SpliJson back to SplitJson again. And the
failure on splitting $.Records happens when already splitted files -
which are routed to split connection from SplitJson are tried to be
splitted again on $.Records, which, of course, is not present at the
moment.
On Fri, Oct 28, 2016 at 12:15 PM, Arsalan Siddiqi
<ar...@live.com> wrote:
> Hi
> I want to get data from the REST server and make it available in NiFi. It
> makes sense to have it in avro format so once read i want to convert the
> data to avro. The server delivers data in JSON format. I have tested the
> JSON and it is in the correct format. Have a look at the image to check the
> flow.
> I use Apache NiFi version 1.0.0. Firstly the split json processor is
> correctly configured to break the incoming data into one record per file. I
> do not know why but the correctly split jsons are routed to the failure
> relationship.
> <http://apache-nifi-developer-list.39713.n7.nabble.com/file/n13769/JSON_AVRO.png>
> The data fetched from the get http (1) request is attached below.there are a
> total of 10 records in the file.
> json_adabas_input.txt
> <http://apache-nifi-developer-list.39713.n7.nabble.com/file/n13769/json_adabas_input.txt>
>
> The format is JSON and the file is without any errors. i do a split on
> $.Records which correctly splits the input with one record per file and I do
> not know why routes it to failure.
> The flow file routed to failure has the following content. (2)
> {"ISN":200,"AA":"50005900","AB":{"AC":"GASTON","AD":"","AE":"BODIN"},"AF":"S","AG":"M","AH":713043,"AW":[{"AX":19990801,"AY":19990831}]}
>
> which is correct and valid JSON.
> At 3 I try to convert the above JSON to avro:
> Using the avro schema:
>
> { "type" : "record",
> "name" : "adabas",
> "fields" : [
> { "name" : "ISN", "type" : "int"},
> { "name" : "AA", "type" : "string"},
> { "name" : "AB", "type" : { "type" : "record", "name" : "AB", "fields" :
> [ { "name" : "AC", "type" : "string"},
> { "name" : "AD", "type" : "string"},
> { "name" : "AE", "type" : "string" } ] } },
> { "name" : "AF", "type" : "string"},
> { "name" : "AG", "type" : "string"},
> { "name" : "AH", "type" : "int" },
> { "name" : "AW", "type" : { "type" : "array", "items" : { "type" :
> "record", "name" : "AW", "fields" :
> [ { "name" : "AX", "type" : "int" },
> { "name" : "AY", "type" : "int" }]}}} ] }
> Here is the resulting avro file which is routed to success:
> ConvertJSONtoAvro.ConvertJSONtoAvro
> <http://apache-nifi-developer-list.39713.n7.nabble.com/file/n13769/ConvertJSONtoAvro.ConvertJSONtoAvro>
>
> Now i expected if the file was routed to success, it would have been
> correctly converted so I can simple get it back in JSON.
> In the convert avro to json (4) i configured the avro schema field to the
> same as the schema mentioned above and i get the follwoing error:
>
> 2016-10-28 12:08:41,439 ERROR [Timer-Driven Process Thread-7]
> o.a.n.processors.avro.ConvertAvroToJSON
> ConvertAvroToJSON[id=fb761b66-1010-1157-4673-c5198a522367] Failed to convert
> StandardFlowFileRecord[uuid=3f234e05-6fa9-4569-9660-c2513f483016,claim=StandardContentClaim
> [resourceClaim=StandardResourceClaim[id=1477643564106-1, container=default,
> section=1], offset=9094, length=602],offset=0,name=adabas,size=602] from
> Avro to JSON due to org.apache.nifi.processor.exception.ProcessException:
> IOException thrown from
> ConvertAvroToJSON[id=fb761b66-1010-1157-4673-c5198a522367]:
> java.io.EOFException; transferring to failure:
> org.apache.nifi.processor.exception.ProcessException: IOException thrown
> from ConvertAvroToJSON[id=fb761b66-1010-1157-4673-c5198a522367]:
> java.io.EOFException
> 2016-10-28 12:08:41,440 ERROR [Timer-Driven Process Thread-7]
> o.a.n.processors.avro.ConvertAvroToJSON
> org.apache.nifi.processor.exception.ProcessException: IOException thrown
> from ConvertAvroToJSON[id=fb761b66-1010-1157-4673-c5198a522367]:
> java.io.EOFException
> at
> org.apache.nifi.controller.repository.StandardProcessSession.write(StandardProcessSession.java:2388)
> ~[na:na]
> at
> org.apache.nifi.processors.avro.ConvertAvroToJSON.onTrigger(ConvertAvroToJSON.java:148)
> ~[nifi-avro-processors-1.0.0.jar:1.0.0]
> at
> org.apache.nifi.processor.AbstractProcessor.onTrigger(AbstractProcessor.java:27)
> [nifi-api-1.0.0.jar:1.0.0]
> at
> org.apache.nifi.controller.StandardProcessorNode.onTrigger(StandardProcessorNode.java:1064)
> [nifi-framework-core-1.0.0.jar:1.0.0]
> at
> org.apache.nifi.controller.tasks.ContinuallyRunProcessorTask.call(ContinuallyRunProcessorTask.java:136)
> [nifi-framework-core-1.0.0.jar:1.0.0]
> at
> org.apache.nifi.controller.tasks.ContinuallyRunProcessorTask.call(ContinuallyRunProcessorTask.java:47)
> [nifi-framework-core-1.0.0.jar:1.0.0]
> at
> org.apache.nifi.controller.scheduling.TimerDrivenSchedulingAgent$1.run(TimerDrivenSchedulingAgent.java:132)
> [nifi-framework-core-1.0.0.jar:1.0.0]
> at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
> [na:1.8.0_101]
> at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
> [na:1.8.0_101]
> at
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
> [na:1.8.0_101]
> at
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
> [na:1.8.0_101]
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> [na:1.8.0_101]
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> [na:1.8.0_101]
> at java.lang.Thread.run(Thread.java:745) [na:1.8.0_101]
> Caused by: java.io.EOFException: null
> at org.apache.avro.io.BinaryDecoder.readInt(BinaryDecoder.java:153)
> ~[avro-1.7.7.jar:1.7.7]
> at org.apache.avro.io.ValidatingDecoder.readInt(ValidatingDecoder.java:83)
> ~[avro-1.7.7.jar:1.7.7]
> at
> org.apache.avro.generic.GenericDatumReader.readInt(GenericDatumReader.java:444)
> ~[avro-1.7.7.jar:1.7.7]
> at
> org.apache.avro.generic.GenericDatumReader.read(GenericDatumReader.java:159)
> ~[avro-1.7.7.jar:1.7.7]
> at
> org.apache.avro.generic.GenericDatumReader.readField(GenericDatumReader.java:193)
> ~[avro-1.7.7.jar:1.7.7]
> at
> org.apache.avro.generic.GenericDatumReader.readRecord(GenericDatumReader.java:183)
> ~[avro-1.7.7.jar:1.7.7]
> at
> org.apache.avro.generic.GenericDatumReader.read(GenericDatumReader.java:151)
> ~[avro-1.7.7.jar:1.7.7]
> at
> org.apache.avro.generic.GenericDatumReader.readArray(GenericDatumReader.java:219)
> ~[avro-1.7.7.jar:1.7.7]
> at
> org.apache.avro.generic.GenericDatumReader.read(GenericDatumReader.java:153)
> ~[avro-1.7.7.jar:1.7.7]
> at
> org.apache.avro.generic.GenericDatumReader.readField(GenericDatumReader.java:193)
> ~[avro-1.7.7.jar:1.7.7]
> at
> org.apache.avro.generic.GenericDatumReader.readRecord(GenericDatumReader.java:183)
> ~[avro-1.7.7.jar:1.7.7]
> at
> org.apache.avro.generic.GenericDatumReader.read(GenericDatumReader.java:151)
> ~[avro-1.7.7.jar:1.7.7]
> at
> org.apache.avro.generic.GenericDatumReader.read(GenericDatumReader.java:142)
> ~[avro-1.7.7.jar:1.7.7]
> at
> org.apache.nifi.processors.avro.ConvertAvroToJSON$1.process(ConvertAvroToJSON.java:161)
> ~[nifi-avro-processors-1.0.0.jar:1.0.0]
> at
> org.apache.nifi.controller.repository.StandardProcessSession.write(StandardProcessSession.java:2370)
> ~[na:na]
> ... 13 common frames omitted
>
>
> As Nifi has made the conversion to avro I expected it should easily convert
> it back without any error.]
> In this case the file is transferred to failure and nothing happens to the
> file.
>
>
>
> If I remove the configured value set at (4) for the field avro schema and
> leave it empty i get the following error:
>
> 2016-10-28 13:25:42,968 ERROR [Timer-Driven Process Thread-4]
> o.a.n.processors.avro.ConvertAvroToJSON
> ConvertAvroToJSON[id=fb761b66-1010-1157-4673-c5198a522367] Failed to convert
> StandardFlowFileRecord[uuid=3f234e05-6fa9-4569-9660-c2513f483016,claim=StandardContentClaim
> [resourceClaim=StandardResourceClaim[id=1477643564106-1, container=default,
> section=1], offset=17978, length=157],offset=0,name=adabas,size=157] from
> Avro to JSON due to org.apache.nifi.processor.exception.ProcessException:
> IOException thrown from
> ConvertAvroToJSON[id=fb761b66-1010-1157-4673-c5198a522367]:
> java.io.IOException: Not a data file.; transferring to failure:
> org.apache.nifi.processor.exception.ProcessException: IOException thrown
> from ConvertAvroToJSON[id=fb761b66-1010-1157-4673-c5198a522367]:
> java.io.IOException: Not a data file.
> 2016-10-28 13:25:42,969 ERROR [Timer-Driven Process Thread-4]
> o.a.n.processors.avro.ConvertAvroToJSON
> org.apache.nifi.processor.exception.ProcessException: IOException thrown
> from ConvertAvroToJSON[id=fb761b66-1010-1157-4673-c5198a522367]:
> java.io.IOException: Not a data file.
> at
> org.apache.nifi.controller.repository.StandardProcessSession.write(StandardProcessSession.java:2388)
> ~[na:na]
> at
> org.apache.nifi.processors.avro.ConvertAvroToJSON.onTrigger(ConvertAvroToJSON.java:148)
> ~[nifi-avro-processors-1.0.0.jar:1.0.0]
> at
> org.apache.nifi.processor.AbstractProcessor.onTrigger(AbstractProcessor.java:27)
> [nifi-api-1.0.0.jar:1.0.0]
> at
> org.apache.nifi.controller.StandardProcessorNode.onTrigger(StandardProcessorNode.java:1064)
> [nifi-framework-core-1.0.0.jar:1.0.0]
> at
> org.apache.nifi.controller.tasks.ContinuallyRunProcessorTask.call(ContinuallyRunProcessorTask.java:136)
> [nifi-framework-core-1.0.0.jar:1.0.0]
> at
> org.apache.nifi.controller.tasks.ContinuallyRunProcessorTask.call(ContinuallyRunProcessorTask.java:47)
> [nifi-framework-core-1.0.0.jar:1.0.0]
> at
> org.apache.nifi.controller.scheduling.TimerDrivenSchedulingAgent$1.run(TimerDrivenSchedulingAgent.java:132)
> [nifi-framework-core-1.0.0.jar:1.0.0]
> at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
> [na:1.8.0_101]
> at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
> [na:1.8.0_101]
> at
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
> [na:1.8.0_101]
> at
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
> [na:1.8.0_101]
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> [na:1.8.0_101]
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> [na:1.8.0_101]
> at java.lang.Thread.run(Thread.java:745) [na:1.8.0_101]
> Caused by: java.io.IOException: Not a data file.
> at org.apache.avro.file.DataFileStream.initialize(DataFileStream.java:105)
> ~[avro-1.7.7.jar:1.7.7]
> at org.apache.avro.file.DataFileStream.<init>(DataFileStream.java:84)
> ~[avro-1.7.7.jar:1.7.7]
> at
> org.apache.nifi.processors.avro.ConvertAvroToJSON$1.process(ConvertAvroToJSON.java:179)
> ~[nifi-avro-processors-1.0.0.jar:1.0.0]
> at
> org.apache.nifi.controller.repository.StandardProcessSession.write(StandardProcessSession.java:2370)
> ~[na:na]
> ... 13 common frames omitted
>
> If i now check the file in the failure relationship it is as follows:
> {"ISN": 200, "AA": "50005900", "AB": {"AC": "GASTON", "AD": "", "AE":
> "BODIN"}, "AF": "S", "AG": "M", "AH": 713043, "AW": [{"AX": 19990801, "AY":
> 19990831}]}
>
> which again is correct but why the error and why routed to failure?? this
> first happened with the Split json processor and now with convert avro to
> json. both work fine but sent the files to failure.
>
>
> Moving to the other branch where i do not specify the schema but instead use
> the infer schema processor, it successfully infers the schema and routes it
> to success (5). Then i use (6) to convert JSON to avro and it also transfers
> the flow file to success. At (7) if I use the ${inferred.avro.schema} I get
> the follwoing error:
>
> 2016-10-28 13:34:21,577 INFO [StandardProcessScheduler Thread-2]
> o.a.n.c.s.TimerDrivenSchedulingAgent Scheduled
> ConvertAvroToJSON[id=000aac50-0158-1000-d9b3-dd8343ed57a7] to run with 1
> threads
> 2016-10-28 13:34:21,579 ERROR [Timer-Driven Process Thread-6]
> o.a.n.processors.avro.ConvertAvroToJSON
> ConvertAvroToJSON[id=000aac50-0158-1000-d9b3-dd8343ed57a7]
> ConvertAvroToJSON[id=000aac50-0158-1000-d9b3-dd8343ed57a7] failed to process
> due to org.apache.avro.AvroRuntimeException: Malformed data. Length is
> negative: -62; rolling back session: org.apache.avro.AvroRuntimeException:
> Malformed data. Length is negative: -62
> 2016-10-28 13:34:21,581 ERROR [Timer-Driven Process Thread-6]
> o.a.n.processors.avro.ConvertAvroToJSON
> org.apache.avro.AvroRuntimeException: Malformed data. Length is negative:
> -62
> at org.apache.avro.io.BinaryDecoder.doReadBytes(BinaryDecoder.java:336)
> ~[avro-1.7.7.jar:1.7.7]
> at org.apache.avro.io.BinaryDecoder.readString(BinaryDecoder.java:263)
> ~[avro-1.7.7.jar:1.7.7]
> at
> org.apache.avro.io.ResolvingDecoder.readString(ResolvingDecoder.java:201)
> ~[avro-1.7.7.jar:1.7.7]
> at
> org.apache.avro.generic.GenericDatumReader.readString(GenericDatumReader.java:363)
> ~[avro-1.7.7.jar:1.7.7]
> at
> org.apache.avro.generic.GenericDatumReader.readString(GenericDatumReader.java:355)
> ~[avro-1.7.7.jar:1.7.7]
> at
> org.apache.avro.generic.GenericDatumReader.read(GenericDatumReader.java:157)
> ~[avro-1.7.7.jar:1.7.7]
> at
> org.apache.avro.generic.GenericDatumReader.readField(GenericDatumReader.java:193)
> ~[avro-1.7.7.jar:1.7.7]
> at
> org.apache.avro.generic.GenericDatumReader.readRecord(GenericDatumReader.java:183)
> ~[avro-1.7.7.jar:1.7.7]
> at
> org.apache.avro.generic.GenericDatumReader.read(GenericDatumReader.java:151)
> ~[avro-1.7.7.jar:1.7.7]
> at
> org.apache.avro.generic.GenericDatumReader.readField(GenericDatumReader.java:193)
> ~[avro-1.7.7.jar:1.7.7]
> at
> org.apache.avro.generic.GenericDatumReader.readRecord(GenericDatumReader.java:183)
> ~[avro-1.7.7.jar:1.7.7]
> at
> org.apache.avro.generic.GenericDatumReader.read(GenericDatumReader.java:151)
> ~[avro-1.7.7.jar:1.7.7]
> at
> org.apache.avro.generic.GenericDatumReader.read(GenericDatumReader.java:142)
> ~[avro-1.7.7.jar:1.7.7]
> at
> org.apache.nifi.processors.avro.ConvertAvroToJSON$1.process(ConvertAvroToJSON.java:161)
> ~[nifi-avro-processors-1.0.0.jar:1.0.0]
> at
> org.apache.nifi.controller.repository.StandardProcessSession.write(StandardProcessSession.java:2370)
> ~[nifi-framework-core-1.0.0.jar:1.0.0]
> at
> org.apache.nifi.processors.avro.ConvertAvroToJSON.onTrigger(ConvertAvroToJSON.java:148)
> ~[nifi-avro-processors-1.0.0.jar:1.0.0]
> at
> org.apache.nifi.processor.AbstractProcessor.onTrigger(AbstractProcessor.java:27)
> ~[nifi-api-1.0.0.jar:1.0.0]
> at
> org.apache.nifi.controller.StandardProcessorNode.onTrigger(StandardProcessorNode.java:1064)
> [nifi-framework-core-1.0.0.jar:1.0.0]
> at
> org.apache.nifi.controller.tasks.ContinuallyRunProcessorTask.call(ContinuallyRunProcessorTask.java:136)
> [nifi-framework-core-1.0.0.jar:1.0.0]
> at
> org.apache.nifi.controller.tasks.ContinuallyRunProcessorTask.call(ContinuallyRunProcessorTask.java:47)
> [nifi-framework-core-1.0.0.jar:1.0.0]
> at
> org.apache.nifi.controller.scheduling.TimerDrivenSchedulingAgent$1.run(TimerDrivenSchedulingAgent.java:132)
> [nifi-framework-core-1.0.0.jar:1.0.0]
> at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
> [na:1.8.0_101]
> at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
> [na:1.8.0_101]
> at
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
> [na:1.8.0_101]
> at
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
> [na:1.8.0_101]
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> [na:1.8.0_101]
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> [na:1.8.0_101]
> at java.lang.Thread.run(Thread.java:745) [na:1.8.0_101]
>
>
> In this case the file stays in the incoming connection. It is the same
> situation if i enter my schema definition.
>
> Lasty if i remove the value for the schema field i get the same result as in
> (4). The record is correctly converted to JSON but routed to failure.
>
>
> I am very confused. Can anyone clarify what is going on??????
>
> Refards
> Arsalan Siddiqi
>
>
>
>
> --
> View this message in context: http://apache-nifi-developer-list.39713.n7.nabble.com/JSON-to-AVRO-AVRO-to-JSON-tp13769.html
> Sent from the Apache NiFi Developer List mailing list archive at Nabble.com.