You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@hudi.apache.org by "subinsonynaukri (via GitHub)" <gi...@apache.org> on 2023/01/23 10:11:12 UTC

[GitHub] [hudi] subinsonynaukri opened a new issue, #7732: org.apache.avro.AvroTypeException: Cannot encode decimal with precision 33 as max precision 30

subinsonynaukri opened a new issue, #7732:
URL: https://github.com/apache/hudi/issues/7732

   When below hudi configurations are enabled, if the data contains input number column which has a number of length 18 digits, it's throwing error "org.apache.avro.AvroTypeException: Cannot encode decimal with precision 33 as max precision 30". When below configurations are disabled, data ingestion is working fine. Also we have the requirement to cast this input column schema to DecimalType(38,0)
   
   hoodie.metadata.enable=true
   hoodie.metadata.index.column.stats.enable=true
   hoodie.enable.data.skipping=true


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [hudi] alexeykudinkin closed issue #7732: [BUG] org.apache.avro.AvroTypeException: Cannot encode decimal with precision 33 as max precision 30

Posted by "alexeykudinkin (via GitHub)" <gi...@apache.org>.
alexeykudinkin closed issue #7732: [BUG] org.apache.avro.AvroTypeException: Cannot encode decimal with precision 33 as max precision 30
URL: https://github.com/apache/hudi/issues/7732


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [hudi] subinsonynaukri commented on issue #7732: [BUG] org.apache.avro.AvroTypeException: Cannot encode decimal with precision 33 as max precision 30

Posted by "subinsonynaukri (via GitHub)" <gi...@apache.org>.
subinsonynaukri commented on issue #7732:
URL: https://github.com/apache/hudi/issues/7732#issuecomment-1403157805

   Thanks @alexeykudinkin for the reply. Even I tried it with DecimalType(18,0) and still getting this exception. Somehow if the number of digits is more than 15, this exception occurs.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [I] [BUG] org.apache.avro.AvroTypeException: Cannot encode decimal with precision 33 as max precision 30 [hudi]

Posted by "rajat1401 (via GitHub)" <gi...@apache.org>.
rajat1401 commented on issue #7732:
URL: https://github.com/apache/hudi/issues/7732#issuecomment-2582799585

   @subinsonynaukri you can try with keeping `hoodie.metadata.index.column.stats.enable` as `false` since Hudi tries to index the column ranges of your data in the metadata table otherwise which might be causing the issue as one of your columns would be of Decimal(38,0) type. Otherwise, you can also cast the datatype to any other type such as string for now. 


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [hudi] alexeykudinkin commented on issue #7732: [BUG] org.apache.avro.AvroTypeException: Cannot encode decimal with precision 33 as max precision 30

Posted by "alexeykudinkin (via GitHub)" <gi...@apache.org>.
alexeykudinkin commented on issue #7732:
URL: https://github.com/apache/hudi/issues/7732#issuecomment-1402420562

   Yeah, this is a known limitation -- we're trying to balance capacity to support arbitrary Decimals w/ the amount of space we waste to store the decimals which are much smaller (we're encoding records using Avro, which doesn't have the same efficiency as Parquet so we have to control this very tightly)
   
   Created HUDI-5608 to follow-up on it


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [I] [BUG] org.apache.avro.AvroTypeException: Cannot encode decimal with precision 33 as max precision 30 [hudi]

Posted by "prabhu-kadode (via GitHub)" <gi...@apache.org>.
prabhu-kadode commented on issue #7732:
URL: https://github.com/apache/hudi/issues/7732#issuecomment-2340759243

   > Thanks @alexeykudinkin for the reply. Even I tried it with DecimalType(18,0) and still getting this exception. Somehow if the number of digits is more than 15, this exception occurs.
   
   Hi bro , any solution for this. When we have number of digits is more than 15 we are getting issue. How to solve this. 


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org