You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@hudi.apache.org by "subinsonynaukri (via GitHub)" <gi...@apache.org> on 2023/01/23 10:11:12 UTC
[GitHub] [hudi] subinsonynaukri opened a new issue, #7732: org.apache.avro.AvroTypeException: Cannot encode decimal with precision 33 as max precision 30
subinsonynaukri opened a new issue, #7732:
URL: https://github.com/apache/hudi/issues/7732
When below hudi configurations are enabled, if the data contains input number column which has a number of length 18 digits, it's throwing error "org.apache.avro.AvroTypeException: Cannot encode decimal with precision 33 as max precision 30". When below configurations are disabled, data ingestion is working fine. Also we have the requirement to cast this input column schema to DecimalType(38,0)
hoodie.metadata.enable=true
hoodie.metadata.index.column.stats.enable=true
hoodie.enable.data.skipping=true
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] alexeykudinkin closed issue #7732: [BUG] org.apache.avro.AvroTypeException: Cannot encode decimal with precision 33 as max precision 30
Posted by "alexeykudinkin (via GitHub)" <gi...@apache.org>.
alexeykudinkin closed issue #7732: [BUG] org.apache.avro.AvroTypeException: Cannot encode decimal with precision 33 as max precision 30
URL: https://github.com/apache/hudi/issues/7732
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] subinsonynaukri commented on issue #7732: [BUG] org.apache.avro.AvroTypeException: Cannot encode decimal with precision 33 as max precision 30
Posted by "subinsonynaukri (via GitHub)" <gi...@apache.org>.
subinsonynaukri commented on issue #7732:
URL: https://github.com/apache/hudi/issues/7732#issuecomment-1403157805
Thanks @alexeykudinkin for the reply. Even I tried it with DecimalType(18,0) and still getting this exception. Somehow if the number of digits is more than 15, this exception occurs.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [BUG] org.apache.avro.AvroTypeException: Cannot encode decimal with precision 33 as max precision 30 [hudi]
Posted by "rajat1401 (via GitHub)" <gi...@apache.org>.
rajat1401 commented on issue #7732:
URL: https://github.com/apache/hudi/issues/7732#issuecomment-2582799585
@subinsonynaukri you can try with keeping `hoodie.metadata.index.column.stats.enable` as `false` since Hudi tries to index the column ranges of your data in the metadata table otherwise which might be causing the issue as one of your columns would be of Decimal(38,0) type. Otherwise, you can also cast the datatype to any other type such as string for now.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] alexeykudinkin commented on issue #7732: [BUG] org.apache.avro.AvroTypeException: Cannot encode decimal with precision 33 as max precision 30
Posted by "alexeykudinkin (via GitHub)" <gi...@apache.org>.
alexeykudinkin commented on issue #7732:
URL: https://github.com/apache/hudi/issues/7732#issuecomment-1402420562
Yeah, this is a known limitation -- we're trying to balance capacity to support arbitrary Decimals w/ the amount of space we waste to store the decimals which are much smaller (we're encoding records using Avro, which doesn't have the same efficiency as Parquet so we have to control this very tightly)
Created HUDI-5608 to follow-up on it
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [BUG] org.apache.avro.AvroTypeException: Cannot encode decimal with precision 33 as max precision 30 [hudi]
Posted by "prabhu-kadode (via GitHub)" <gi...@apache.org>.
prabhu-kadode commented on issue #7732:
URL: https://github.com/apache/hudi/issues/7732#issuecomment-2340759243
> Thanks @alexeykudinkin for the reply. Even I tried it with DecimalType(18,0) and still getting this exception. Somehow if the number of digits is more than 15, this exception occurs.
Hi bro , any solution for this. When we have number of digits is more than 15 we are getting issue. How to solve this.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org