[jira] [Comment Edited] (NIFI-2624) JDBC-to-Avro processors handle BigDecimals as Strings

["Toivo Adams (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/NIFI-2624?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15460710#comment-15460710 ] 

Toivo Adams edited comment on NIFI-2624 at 9/3/16 8:49 AM:
-----------------------------------------------------------

Note!! BigInteger handling is not yet changed.

I am not sure how Avro supports DECIMAL types.
Current solution looks clumsy.
You can’t put BigDecimal value directly to Avro record, like
   rec.put(i - 1, bigDecimalValue);
Avro will throw 
org.apache.avro.AvroRuntimeException: Unknown datum type java.math.BigDecimal: 38

Instead you need to convert value to bytes, like
   Schema decimalSchema = getDecimalSchema(schema, meta.getColumnName(i));
   LogicalType logicalType = LogicalTypes.fromSchema(decimalSchema);
   ByteBuffer byteBuffer = decimalConversion.toBytes((BigDecimal) value, decimalSchema, logicalType);
   rec.put(i - 1, byteBuffer);

And getting BigDecimal value from record is equally weird:

    record = dataFileReader.next(record);
    DecimalConversion decimalConversion = new DecimalConversion();
    Schema schema = record.getSchema();
    Schema decimalSchema = getDecimalSchema(schema, "Chairman");
    LogicalType logicalType = LogicalTypes.fromSchema(decimalSchema);
    ByteBuffer buffer = (ByteBuffer) record.get("Chairman");

    BigDecimal resultBD = decimalConversion.fromBytes(buffer, schema, logicalType);

I am not sure how to handle BigInteger, because its seems DecimalConversion supports only BigDecimal.

Thanks
Toivo




was (Author: toivo adams):
Note!! BigInteger handling is not yet changed.

I am not sure how Avro supports DECIMAL types.
Current solution looks clumsy.
You can’ put BigDecimal value directly to Avro record, like
   rec.put(i - 1, bigDecimalValue);
Avro will throw 
org.apache.avro.AvroRuntimeException: Unknown datum type java.math.BigDecimal: 38

Instead you need to convert value to bytes, like
   Schema decimalSchema = getDecimalSchema(schema, meta.getColumnName(i));
   LogicalType logicalType = LogicalTypes.fromSchema(decimalSchema);
   ByteBuffer byteBuffer = decimalConversion.toBytes((BigDecimal) value, decimalSchema, logicalType);
   rec.put(i - 1, byteBuffer);

And getting BigDecimal value from record is equally weird:

    record = dataFileReader.next(record);
    DecimalConversion decimalConversion = new DecimalConversion();
    Schema schema = record.getSchema();
    Schema decimalSchema = getDecimalSchema(schema, "Chairman");
    LogicalType logicalType = LogicalTypes.fromSchema(decimalSchema);
    ByteBuffer buffer = (ByteBuffer) record.get("Chairman");

    BigDecimal resultBD = decimalConversion.fromBytes(buffer, schema, logicalType);

I am not sure how to handle BigInteger, because its seems DecimalConversion supports only BigDecimal.

Thanks
Toivo



> JDBC-to-Avro processors handle BigDecimals as Strings
> -----------------------------------------------------
>
>                 Key: NIFI-2624
>                 URL: https://issues.apache.org/jira/browse/NIFI-2624
>             Project: Apache NiFi
>          Issue Type: Improvement
>          Components: Extensions
>            Reporter: Matt Burgess
>            Assignee: Toivo Adams
>
> The original SQL processors implemented BigDecimal values as Strings for 
> Avro, as the version of Avro it used (1.7.6) did not support DECIMAL type.
> As of Avro 1.7.7 (AVRO-1402), this type is supported and so the SQL/HiveQL processors should be updated to handle BigDecimals correctly if possible.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)