ConvertJSONToAvro converts only last valid JASON record from an Input file with many JSON records

[Anshuman Ghosh <an...@gmail.com>]

Hello Group/ Bryan,

Anshu here.
While trying the "*ConvertJSONToAvro*" processor, I came across the
following issue. I am not sure whether it is correct or not though.
So I have this input JSON file which might contain one or more valid/
invalid JSON records. Need to use this processor to convert them into Avro
format before I write and process them onto HDFS.
However strangely in the output FlowFile, I am getting only 1 record
converted (last valid JSON record is only getting converted).

Following "*Record Schem*a" I am using for this purpose.

*{"type": "record",*
* "name": "ClickData",*
* "fields": [*
*     {"name": "SuperShopId", "type": ["string", "null"]},*
*     {"name": "LinkId",  "type": ["string", "null"]},*
*     {"name": "ClickId", "type": ["string", "null"]},  *
* {"name": "VisitorId", "type": ["string", "null"]},*
* {"name": "RawHTTPRequest", "type": ["string", "null"]},*
* {"name": "RequestTimestamp", "type": "long"},*
* {"name": "RedirectURL", "type": ["string", "null"]},*
* {"name": "ResponseTimestamp", "type": "long"}*
* ]*
*}*

Please let me know if I am doing something wrong!
Thanking you in advance!

​
______________________

*Kind Regards,*
*Anshuman Ghosh*
*Contact - +49 179 9090964*

Re: ConvertJSONToAvro converts only last valid JASON record from an Input file with many JSON records

[Anshuman Ghosh <an...@gmail.com>]

Hello James,

Thank you for your reply!

The problem was that the JSON rows weren't new line delimited and hence it
was only considering the last matched valid record and discarding
everything else!
Now that we are passing new line delimited records, it is able to convert
the correct ones to Avro format and discarding the incompatible ones.
Why should we split the JSON, won't it be an overkill of processing each
record separately?
Thanks you for the reply again.

​
______________________

*Kind Regards,*
*Anshuman Ghosh*
*Contact - +49 179 9090964*


On Tue, Mar 14, 2017 at 9:15 PM, James Wing <jv...@gmail.com> wrote:

> Anshuman,
>
> What kind of error messages are you seeing? If the input is
> newline-delimited JSON records, I believe ConvertJSONToAvro should transfer
> all valid records to the "success" relationship as a single Avro, and the
> entire input flowfile to the "incompatible" output if any are invalid.
> Does your "success" Avro contain more than one record?
>
> One thing you could do is split the JSON records before conversion, so they
> will succeed or fail individually.
>
> Thanks,
>
> James
>
> On Tue, Mar 14, 2017 at 8:59 AM, Anshuman Ghosh <
> anshuman.ghosh2009@gmail.com> wrote:
>
> > Hello Group/ Bryan,
> >
> > Anshu here.
> > While trying the "*ConvertJSONToAvro*" processor, I came across the
> > following issue. I am not sure whether it is correct or not though.
> > So I have this input JSON file which might contain one or more valid/
> > invalid JSON records. Need to use this processor to convert them into
> Avro
> > format before I write and process them onto HDFS.
> > However strangely in the output FlowFile, I am getting only 1 record
> > converted (last valid JSON record is only getting converted).
> >
> > Following "*Record Schem*a" I am using for this purpose.
> >
> > *{"type": "record",*
> > * "name": "ClickData",*
> > * "fields": [*
> > *     {"name": "SuperShopId", "type": ["string", "null"]},*
> > *     {"name": "LinkId",  "type": ["string", "null"]},*
> > *     {"name": "ClickId", "type": ["string", "null"]},  *
> > * {"name": "VisitorId", "type": ["string", "null"]},*
> > * {"name": "RawHTTPRequest", "type": ["string", "null"]},*
> > * {"name": "RequestTimestamp", "type": "long"},*
> > * {"name": "RedirectURL", "type": ["string", "null"]},*
> > * {"name": "ResponseTimestamp", "type": "long"}*
> > * ]*
> > *}*
> >
> > Please let me know if I am doing something wrong!
> > Thanking you in advance!
> >
> > ​
> > ______________________
> >
> > *Kind Regards,*
> > *Anshuman Ghosh*
> > *Contact - +49 179 9090964*
> >
>

Re: ConvertJSONToAvro converts only last valid JASON record from an Input file with many JSON records

[James Wing <jv...@gmail.com>]

Anshuman,

What kind of error messages are you seeing? If the input is
newline-delimited JSON records, I believe ConvertJSONToAvro should transfer
all valid records to the "success" relationship as a single Avro, and the
entire input flowfile to the "incompatible" output if any are invalid.
Does your "success" Avro contain more than one record?

One thing you could do is split the JSON records before conversion, so they
will succeed or fail individually.

Thanks,

James

On Tue, Mar 14, 2017 at 8:59 AM, Anshuman Ghosh <
anshuman.ghosh2009@gmail.com> wrote:

> Hello Group/ Bryan,
>
> Anshu here.
> While trying the "*ConvertJSONToAvro*" processor, I came across the
> following issue. I am not sure whether it is correct or not though.
> So I have this input JSON file which might contain one or more valid/
> invalid JSON records. Need to use this processor to convert them into Avro
> format before I write and process them onto HDFS.
> However strangely in the output FlowFile, I am getting only 1 record
> converted (last valid JSON record is only getting converted).
>
> Following "*Record Schem*a" I am using for this purpose.
>
> *{"type": "record",*
> * "name": "ClickData",*
> * "fields": [*
> *     {"name": "SuperShopId", "type": ["string", "null"]},*
> *     {"name": "LinkId",  "type": ["string", "null"]},*
> *     {"name": "ClickId", "type": ["string", "null"]},  *
> * {"name": "VisitorId", "type": ["string", "null"]},*
> * {"name": "RawHTTPRequest", "type": ["string", "null"]},*
> * {"name": "RequestTimestamp", "type": "long"},*
> * {"name": "RedirectURL", "type": ["string", "null"]},*
> * {"name": "ResponseTimestamp", "type": "long"}*
> * ]*
> *}*
>
> Please let me know if I am doing something wrong!
> Thanking you in advance!
>
> ​
> ______________________
>
> *Kind Regards,*
> *Anshuman Ghosh*
> *Contact - +49 179 9090964*
>