You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@avro.apache.org by Indhumathi M <in...@gmail.com> on 2018/09/04 15:55:36 UTC

Avro Logical type

Hello all,
I am working on avro logical type decimal, but was not sure what data has
to be provided.
As mentioned in the Avro document,
A decimal logical type annotates Avro bytes or fixed types. The byte array
must contain the two's-complement representation of the unscaled integer
value in big-endian byte order. The scale is fixed, and is specified using
an attribute.

But in code, in Conversions class, I can see that, from a decimal value, it
is forming a ByteArray.

@Override
public BigDecimal fromBytes(ByteBuffer value, Schema schema, LogicalType type) {
  int scale = ((LogicalTypes.Decimal) type).getScale();
  // always copy the bytes out because BigInteger has no offset/length ctor
  byte[] bytes = value.get(new byte[value.remaining()]).array();
  return new BigDecimal(new BigInteger(bytes), scale);
}

@Override
public ByteBuffer toBytes(BigDecimal value, Schema schema, LogicalType type) {
  int scale = ((LogicalTypes.Decimal) type).getScale();
  if (scale != value.scale()) {
    throw new AvroTypeException("Cannot encode decimal with scale " +
        value.scale() + " as scale " + scale);
  }
  return ByteBuffer.wrap(value.unscaledValue().toByteArray());
}

but, I find that, nowhere two's-complement representation of the
unscaled integer
value in big-endian byte order is stored in byte array.

Can anyone help me with this?

Thanks,

Indhumathi M

Re: Avro Logical type

Posted by Indhumathi M <in...@gmail.com>.
Hi Ryan,
Thanks for your reply.

If i am creating a byte avro schema of logical type decimal and my data is
a json record.
For example,
val json = {"bytes":"10.2"}
and i am using the following converter provided by Avro.
[image: image.png]
But this returns a bytearray, which is a direct conversion of json string
to bytearray and not a 2's complement representation.
so, what will be the case if record is json for avro logical decimal type?

Thanks,
Indhumathi M

On Tue, Sep 4, 2018 at 9:39 PM Ryan Blue <rb...@netflix.com.invalid> wrote:

> Indhumathi,
>
> BigInteger provides the values as a two's-complement big endian byte array
> when toByteArray is called. It also accepts byte arrays with that format in
> its constructor. See
>
> https://docs.oracle.com/javase/7/docs/api/java/math/BigInteger.html#toByteArray()
>
> The conversion you're looking at is for bytes, so this also doesn't need to
> pad the values to a certain length. If you want to see how that is done,
> look at the fixed conversion.
>
> rb
>
> On Tue, Sep 4, 2018 at 8:55 AM Indhumathi M <in...@gmail.com>
> wrote:
>
> > Hello all,
> > I am working on avro logical type decimal, but was not sure what data has
> > to be provided.
> > As mentioned in the Avro document,
> > A decimal logical type annotates Avro bytes or fixed types. The byte
> array
> > must contain the two's-complement representation of the unscaled integer
> > value in big-endian byte order. The scale is fixed, and is specified
> using
> > an attribute.
> >
> > But in code, in Conversions class, I can see that, from a decimal value,
> it
> > is forming a ByteArray.
> >
> > @Override
> > public BigDecimal fromBytes(ByteBuffer value, Schema schema, LogicalType
> > type) {
> >   int scale = ((LogicalTypes.Decimal) type).getScale();
> >   // always copy the bytes out because BigInteger has no offset/length
> ctor
> >   byte[] bytes = value.get(new byte[value.remaining()]).array();
> >   return new BigDecimal(new BigInteger(bytes), scale);
> > }
> >
> > @Override
> > public ByteBuffer toBytes(BigDecimal value, Schema schema, LogicalType
> > type) {
> >   int scale = ((LogicalTypes.Decimal) type).getScale();
> >   if (scale != value.scale()) {
> >     throw new AvroTypeException("Cannot encode decimal with scale " +
> >         value.scale() + " as scale " + scale);
> >   }
> >   return ByteBuffer.wrap(value.unscaledValue().toByteArray());
> > }
> >
> > but, I find that, nowhere two's-complement representation of the
> > unscaled integer
> > value in big-endian byte order is stored in byte array.
> >
> > Can anyone help me with this?
> >
> > Thanks,
> >
> > Indhumathi M
> >
>
>
> --
> Ryan Blue
> Software Engineer
> Netflix
>

Re: Avro Logical type

Posted by Ryan Blue <rb...@netflix.com.INVALID>.
Indhumathi,

BigInteger provides the values as a two's-complement big endian byte array
when toByteArray is called. It also accepts byte arrays with that format in
its constructor. See
https://docs.oracle.com/javase/7/docs/api/java/math/BigInteger.html#toByteArray()

The conversion you're looking at is for bytes, so this also doesn't need to
pad the values to a certain length. If you want to see how that is done,
look at the fixed conversion.

rb

On Tue, Sep 4, 2018 at 8:55 AM Indhumathi M <in...@gmail.com> wrote:

> Hello all,
> I am working on avro logical type decimal, but was not sure what data has
> to be provided.
> As mentioned in the Avro document,
> A decimal logical type annotates Avro bytes or fixed types. The byte array
> must contain the two's-complement representation of the unscaled integer
> value in big-endian byte order. The scale is fixed, and is specified using
> an attribute.
>
> But in code, in Conversions class, I can see that, from a decimal value, it
> is forming a ByteArray.
>
> @Override
> public BigDecimal fromBytes(ByteBuffer value, Schema schema, LogicalType
> type) {
>   int scale = ((LogicalTypes.Decimal) type).getScale();
>   // always copy the bytes out because BigInteger has no offset/length ctor
>   byte[] bytes = value.get(new byte[value.remaining()]).array();
>   return new BigDecimal(new BigInteger(bytes), scale);
> }
>
> @Override
> public ByteBuffer toBytes(BigDecimal value, Schema schema, LogicalType
> type) {
>   int scale = ((LogicalTypes.Decimal) type).getScale();
>   if (scale != value.scale()) {
>     throw new AvroTypeException("Cannot encode decimal with scale " +
>         value.scale() + " as scale " + scale);
>   }
>   return ByteBuffer.wrap(value.unscaledValue().toByteArray());
> }
>
> but, I find that, nowhere two's-complement representation of the
> unscaled integer
> value in big-endian byte order is stored in byte array.
>
> Can anyone help me with this?
>
> Thanks,
>
> Indhumathi M
>


-- 
Ryan Blue
Software Engineer
Netflix