[jira] [Updated] (AVRO-1402) Support for DECIMAL primitive type

["Tom White (JIRA)" <ji...@apache.org>]

     [ https://issues.apache.org/jira/browse/AVRO-1402?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Tom White updated AVRO-1402:
----------------------------

    Attachment: AVRO-1402.patch

Following Doug's [subType suggestion|https://issues.apache.org/jira/browse/AVRO-739?focusedCommentId=13933465&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-13933465] on AVRO-739 I wrote a patch to support a decimal type.

The schema declaration looks like this: {"type":"bytes", "subType":"decimal"}

The encoding is an int to represent the scale followed by a byte array containing the unscaled integer, in the (language-neutral) format described here: http://docs.oracle.com/javase/6/docs/api/java/math/BigInteger.html#toByteArray%28%29.

One thing to notice is that in this patch the type does not define the precision and scale as a part of the type. This means that there is no restriction in Avro on the decimal that may be written. Instead, the burden of limiting the precision and scale falls on the application. Hive, for example, already has logic for ensuring that the precision and scale of a decimal value are consistent with the precision and scale values set as a part of the type definition for that decimal column. (There is more discussion on this point on HIVE-3976, and in particular in the [functional spec|https://cwiki.apache.org/confluence/download/attachments/27362075/Hive_Decimal_Precision_Scale_Support.pdf].)

It would be good if we could get some agreement on i) whether this is a good approach for adding new (optional) types to Avro and ii) what the binary encoding should look like. Thoughts?

In this initial patch I added the decimal subtype to GenericDatumWriter/Reader. I did this since Hive uses generic, but there is a potential compatibility issue where code using a GenericDatumReader receives a BigDecimal instead of a ByteBuffer when reading a new schema with a decimal subtype. Any thoughts on how to tackle this would be gratefully received too.

> Support for DECIMAL primitive type
> ----------------------------------
>
>                 Key: AVRO-1402
>                 URL: https://issues.apache.org/jira/browse/AVRO-1402
>             Project: Avro
>          Issue Type: New Feature
>    Affects Versions: 1.7.5
>            Reporter: Mariano Dominguez
>            Priority: Minor
>              Labels: Hive
>         Attachments: AVRO-1402.patch
>
>
> Currently, Avro does not seem to support a DECIMAL type or equivalent.
> http://avro.apache.org/docs/1.7.5/spec.html#schema_primitive
> Adding DECIMAL support would be particularly interesting when converting types from Avro to Hive, since DECIMAL is already a supported data type in Hive (0.11.0).



--
This message was sent by Atlassian JIRA
(v6.2#6252)