[jira] Updated: (AVRO-672) Convert JSON Text Input to Avro Tool

["Doug Cutting (JIRA)" <ji...@apache.org>]

     [ https://issues.apache.org/jira/browse/AVRO-672?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Doug Cutting updated AVRO-672:
------------------------------

    Attachment: AVRO-672.patch

It might be confusing to provide two different JSON encodings for Avro data.  Also, the encoding in your patch is indeed simpler, but can lose information.  For example, a string that looks like base64-encoded binary data would be assumed by Jackson to be binary data, which might not always be the case.  Schemas that include fixed or enum values are not supported by this encoding, nor are many unions.

If reading and writing arbitrary JSON is a priority, then the approach taken in AVRO-251 might be of interest.  Here's a patch that provides a DatumReader and DatumWriter for Jackson's JsonNode.  This uses a schema that permits arbitrary JSON data.  Would this be useful to you?  If so, we could provide it as a tool.

> Convert JSON Text Input to Avro Tool
> ------------------------------------
>
>                 Key: AVRO-672
>                 URL: https://issues.apache.org/jira/browse/AVRO-672
>             Project: Avro
>          Issue Type: New Feature
>            Reporter: Ron Bodkin
>         Attachments: AVRO-672.patch, AVRO-672.patch
>
>
> The attached patch allows reading a JSON-formatted text file in, converting to a conforming Avro text file, emitting one record per line, e.g., it can read this input file:
> {"intval":12}
> {"intval":-73,"strval":"hello, there!!"}
> with this schema:
> { "type":"record", "name":"TestRecord", "fields": [ {"name":"intval","type":"int"}, {"name":"strval","type":["string", "null"]}]}
> returning valid Avro. This is different than the DataFileWriteTool, which would read in the following internal encoding:
> {"intval":12,"strval":null}
> {"intval":-73,"strval":{"string":"hello, there!!"}}
> In general, the internal encodings used by Avro aren't natural when reading in JSON text that appears in the wild. Likewise, this utility allows changing invalid Avro identifier characters into an underscore, again to tolerate JSON that wasn't designed to be readable by Avro.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.