You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@hudi.apache.org by GitBox <gi...@apache.org> on 2022/07/25 10:56:19 UTC
[GitHub] [hudi] lz1984sh opened a new issue, #6209: [SUPPORT] hudi 0.11 not support decimal field precision increase
lz1984sh opened a new issue, #6209:
URL: https://github.com/apache/hudi/issues/6209
**Describe the problem you faced**
It seems like hudi doesn't support decimal field precision increase when size of underlying fixed byte length array increases. E.g., I insert one row
id1 name1 111.11
with schema below
id - string, name - string, amount decimal(**10,2**)
and then update the row with same data but increased precision decimal schema
id - string, name - string, amount decimal(**20,4**)
Then java.lang.ArrayIndexOutOfBoundsException would be thrown, look details in exception stack trace below.
**To Reproduce**
Steps to reproduce the behavior:
1. Download attached [pom.xml.txt](https://github.com/apache/hudi/files/9180429/pom.xml.txt) and rename as pom.xml and create project in your IDE workspace.
2. Download attached java program [HudiTest.java.txt](https://github.com/apache/hudi/files/9180407/HudiTest.java.txt) as HudiTest.java and run it in the project created in step 1.
**Environment Description**
* Hudi version : 0.11.0
* Spark version : 3.1.1
* Hive version : N/A
* Hadoop version : 3.3.1
* Storage (HDFS/S3/GCS..) : Local file system
* Running on Docker? (yes/no) : no
**Stacktrace**
```
Caused by: org.apache.parquet.io.ParquetDecodingException: Can not read value at 1 in block 0 in file file:/xxx/e6043078-cadc-43cc-8412-50ecf3dc9a5b-0_0-34-1617_20220725183004104.parquet
at org.apache.parquet.hadoop.InternalParquetRecordReader.nextKeyValue(InternalParquetRecordReader.java:251)
at org.apache.parquet.hadoop.ParquetReader.read(ParquetReader.java:132)
at org.apache.parquet.hadoop.ParquetReader.read(ParquetReader.java:136)
at org.apache.hudi.common.util.ParquetReaderIterator.hasNext(ParquetReaderIterator.java:48)
... 10 more
Caused by: java.lang.ArrayIndexOutOfBoundsException
at java.lang.System.arraycopy(Native Method)
at org.apache.avro.generic.GenericData.createFixed(GenericData.java:1168)
at org.apache.parquet.avro.AvroConverters$FieldFixedConverter.convert(AvroConverters.java:310)
at org.apache.parquet.avro.AvroConverters$BinaryConverter.addBinary(AvroConverters.java:62)
at org.apache.parquet.column.impl.ColumnReaderImpl$2$6.writeValue(ColumnReaderImpl.java:317)
at org.apache.parquet.column.impl.ColumnReaderImpl.writeCurrentValueToConverter(ColumnReaderImpl.java:367)
at org.apache.parquet.io.RecordReaderImplementation.read(RecordReaderImplementation.java:406)
at org.apache.parquet.hadoop.InternalParquetRecordReader.nextKeyValue(InternalParquetRecordReader.java:226)
... 13 more
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] xiarixiaoyao commented on issue #6209: [SUPPORT] hudi 0.11 not support decimal field precision increase
Posted by GitBox <gi...@apache.org>.
xiarixiaoyao commented on issue #6209:
URL: https://github.com/apache/hudi/issues/6209#issuecomment-1198804788
@yihua @lz1984sh
yes, schema evolution can support that case;
```
spark.sql("set hoodie.schema.on.read.enable=true")
// NOTE: This is required since as this tests use type coercions which were only permitted in Spark 2.x
// and are disallowed now by default in Spark 3.x
spark.sql("set spark.sql.storeAssignmentPolicy=legacy")
spark.sql(
s"""
|create table $tableName (
| id int, comb int, col0 int, col1 bigint, col2 float, col3 double, col4 decimal(10,4), par date
|) using hudi
| location '$tablePath'
| options (
| type = '$tableType',
| primaryKey = 'id',
| preCombineField = 'comb'
| )
| partitioned by (par)
""".stripMargin)
spark.sql(
s"""
| insert into $tableName values
| (1,1,11,100001,101.01,1001.0001,100001.0001,TIMESTAMP'2021-12-25'),
| (2,2,12,100002,102.02,1002.0002,100002.0002,TIMESTAMP'2021-12-25')
|""".stripMargin)
// alter table change
spark.sql(s"alter table $tableName alter column col4 type decimal(18, 6)")
spark.sql(
s"""
| insert into $tableName values
| (1,1,11,100001,101.01,1001.0001,200001.000001,TIMESTAMP'2021-12-25'),
| (2,2,12,100002,102.02,1002.0002,300002.000002,TIMESTAMP'2021-12-25')
|""".stripMargin)
spark.sql(s"select col4 from $tableName").show(false)
spark.sql(s"select * from $tableName").printSchema()
```
+-------------+
|col4 |
+-------------+
|300002.000002|
|200001.000001|
+-------------+
root
|-- _hoodie_commit_time: string (nullable = true)
|-- _hoodie_commit_seqno: string (nullable = true)
|-- _hoodie_record_key: string (nullable = true)
|-- _hoodie_partition_path: string (nullable = true)
|-- _hoodie_file_name: string (nullable = true)
|-- id: integer (nullable = true)
|-- comb: integer (nullable = true)
|-- col0: integer (nullable = true)
|-- col1: long (nullable = true)
|-- col2: float (nullable = true)
|-- col3: double (nullable = true)
|-- col4: decimal(18,6) (nullable = true)
|-- par: date (nullable = true)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] xushiyan closed issue #6209: [SUPPORT] hudi 0.11 not support decimal field precision increase
Posted by GitBox <gi...@apache.org>.
xushiyan closed issue #6209: [SUPPORT] hudi 0.11 not support decimal field precision increase
URL: https://github.com/apache/hudi/issues/6209
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] xushiyan commented on issue #6209: [SUPPORT] hudi 0.11 not support decimal field precision increase
Posted by GitBox <gi...@apache.org>.
xushiyan commented on issue #6209:
URL: https://github.com/apache/hudi/issues/6209#issuecomment-1296227206
Thanks @xiarixiaoyao for showing the right configs and the example. will close this now.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] yihua commented on issue #6209: [SUPPORT] hudi 0.11 not support decimal field precision increase
Posted by GitBox <gi...@apache.org>.
yihua commented on issue #6209:
URL: https://github.com/apache/hudi/issues/6209#issuecomment-1198758964
@xiarixiaoyao do you know if schema evolution support such a use case?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org