You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@hudi.apache.org by "junkri (via GitHub)" <gi...@apache.org> on 2024/04/09 14:36:22 UTC
[I] [SUPPORT] org.apache.avro.SchemaParseException: Can't redefine decimal field [hudi]
junkri opened a new issue, #10983:
URL: https://github.com/apache/hudi/issues/10983
**Describe the problem you faced**
When using Decimal types, I ran into a problem where Hudi cannot write into a non-empty table, getting exception as `Caused by: org.apache.hudi.exception.HoodieException: org.apache.avro.SchemaParseException: Can't redefine: <field>`
I can trigger this error in 2 ways:
- I have 2 `decimal` fields with the same name, but with different precision/scale and in different `struct` fields (so that I can use the same fieldname)
- I have a `decimal` field and a `struct` with the same name
**To Reproduce**
I created a small runnable github project with 2 small examples to trigger this error:
https://github.com/junkri/hudi-cant-redefine-field-demo
You can run the examples with maven or from any IDE as well.
**Expected behavior**
I expect that I can use `decimal` fields in different structures without an issue
**Environment Description**
I use AWS EMR serverless mainly, so I chose the versions from the last EMR 6 environment
* Hudi version : 0.14.1
* Spark version : 3.4.1
* Hive version : --
* Hadoop version : --
* Storage (HDFS/S3/GCS..) : local filesystem, but happens with S3 as well
* Running on Docker? (yes/no) : no
**Additional context**
I am aware of https://github.com/apache/hudi/issues/7717, but here I don't use very complex structures, and in my case `decimal` fields cause the issue. I tried to force to update the `parquet-avro` library in my project, but it didn't help.
I tried to debug into Hudi, and I saw that when it reads back from Parquet and converts things into Avro, the decimal fields are created as `fixed` Avro type, which has an empty `namespace` attribute! I guess that means the decimal fields can be defined once in the whole avro schema, and re-used later, but because of the different precision/scale settings for my decimal fields (having the same name), the `size` attribute of `fixed` field has to be different, so that is when we can't re-define a field.
In our projects we use Kafka as input source, so we define `decimal` fields with `bytes` Avro type (and not the `fixed` one), so we use something like `{"type": "bytes", "logicalType": "decimal", "precision": 19, "scale": 6}`. Maybe the `parquet-avro` library should use it as well?
**Stacktrace**
```
Caused by: org.apache.hudi.exception.HoodieException: org.apache.avro.SchemaParseException: Can't redefine: trick
at org.apache.hudi.table.action.commit.HoodieMergeHelper.runMerge(HoodieMergeHelper.java:149)
at org.apache.hudi.table.action.commit.BaseSparkCommitActionExecutor.handleUpdateInternal(BaseSparkCommitActionExecutor.java:387)
at org.apache.hudi.table.action.commit.BaseSparkCommitActionExecutor.handleUpdate(BaseSparkCommitActionExecutor.java:369)
at org.apache.hudi.table.action.commit.BaseSparkCommitActionExecutor.handleUpsertPartition(BaseSparkCommitActionExecutor.java:335)
... 29 more
Caused by: org.apache.avro.SchemaParseException: Can't redefine: trick
at org.apache.avro.Schema$Names.put(Schema.java:1586)
at org.apache.avro.Schema$NamedSchema.writeNameRef(Schema.java:844)
at org.apache.avro.Schema$RecordSchema.toJson(Schema.java:1011)
at org.apache.avro.Schema$UnionSchema.toJson(Schema.java:1278)
at org.apache.avro.Schema$RecordSchema.fieldsToJson(Schema.java:1039)
at org.apache.avro.Schema$RecordSchema.toJson(Schema.java:1023)
at org.apache.avro.Schema.toString(Schema.java:433)
at org.apache.avro.Schema.toString(Schema.java:405)
at org.apache.avro.Schema.toString(Schema.java:396)
at org.apache.parquet.avro.AvroReadSupport.setAvroReadSchema(AvroReadSupport.java:77)
at org.apache.hudi.io.storage.HoodieAvroParquetReader.getIndexedRecordIteratorInternal(HoodieAvroParquetReader.java:162)
at org.apache.hudi.io.storage.HoodieAvroParquetReader.getIndexedRecordIterator(HoodieAvroParquetReader.java:94)
at org.apache.hudi.io.storage.HoodieAvroParquetReader.getRecordIterator(HoodieAvroParquetReader.java:73)
at org.apache.hudi.table.action.commit.HoodieMergeHelper.runMerge(HoodieMergeHelper.java:126)
... 32 more
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [SUPPORT] org.apache.avro.SchemaParseException: Can't redefine decimal field [hudi]
Posted by "xicm (via GitHub)" <gi...@apache.org>.
xicm commented on issue #10983:
URL: https://github.com/apache/hudi/issues/10983#issuecomment-2373542115
This seems a bug of parquet-avro.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [SUPPORT] org.apache.avro.SchemaParseException: Can't redefine decimal field [hudi]
Posted by "ad1happy2go (via GitHub)" <gi...@apache.org>.
ad1happy2go commented on issue #10983:
URL: https://github.com/apache/hudi/issues/10983#issuecomment-2046872506
Thanks @junkri for raising this . We will look into this.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [SUPPORT] org.apache.avro.SchemaParseException: Can't redefine decimal field [hudi]
Posted by "deepakpanda93 (via GitHub)" <gi...@apache.org>.
deepakpanda93 closed issue #10983: [SUPPORT] org.apache.avro.SchemaParseException: Can't redefine decimal field
URL: https://github.com/apache/hudi/issues/10983
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [SUPPORT] org.apache.avro.SchemaParseException: Can't redefine decimal field [hudi]
Posted by "imonteroq (via GitHub)" <gi...@apache.org>.
imonteroq commented on issue #10983:
URL: https://github.com/apache/hudi/issues/10983#issuecomment-2912501684
@ad1happy2go Would you mind adding the above to the [JIRA](https://issues.apache.org/jira/browse/HUDI-7602) you created?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [SUPPORT] org.apache.avro.SchemaParseException: Can't redefine decimal field [hudi]
Posted by "junkri (via GitHub)" <gi...@apache.org>.
junkri commented on issue #10983:
URL: https://github.com/apache/hudi/issues/10983#issuecomment-2050076041
thank you very much for creating a Jira issue on this.
I also found out that the same error is triggered when we have a `decimal` and a `struct` field with the same name, so this also causes issues:
```scala
spark.sql(s"""
create table trick(
tricky struct<tricky decimal(10,2)>
)
using hudi
location '$location'
""")
spark.sql("""
insert into trick
values (
named_struct('tricky', 1.2)
)
""") // works fine
spark.sql("""
insert into trick
values (
named_struct('tricky', 3.4)
)
""") // org.apache.avro.SchemaParseException: Can't redefine: tricky
```
I suspect this also happens as the `decimal` is represented as a `fixed` type with an empty `namespace` during parquet->avro conversion
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [SUPPORT] org.apache.avro.SchemaParseException: Can't redefine decimal field [hudi]
Posted by "imonteroq (via GitHub)" <gi...@apache.org>.
imonteroq commented on issue #10983:
URL: https://github.com/apache/hudi/issues/10983#issuecomment-2912338028
It looks like this was [fixed](https://issues.apache.org/jira/browse/PARQUET-2456) in parquet a few years ago. Problem is Hudi still uses a fairly [old](https://github.com/apache/hudi/blob/master/pom.xml#L113) parquet dependency.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [SUPPORT] org.apache.avro.SchemaParseException: Can't redefine decimal field [hudi]
Posted by "ad1happy2go (via GitHub)" <gi...@apache.org>.
ad1happy2go commented on issue #10983:
URL: https://github.com/apache/hudi/issues/10983#issuecomment-2049410775
@junkri This looks different than https://github.com/apache/hudi/issues/7717 as that is fixed with later version of spark which included this fix (https://issues.apache.org/jira/browse/PARQUET-1441) .
It is still failing with spark 3.4 and hudi 0.14.1. Created a JIRA to track a fix on this. https://issues.apache.org/jira/browse/HUDI-7602
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org