You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@hudi.apache.org by GitBox <gi...@apache.org> on 2022/10/23 13:05:38 UTC
[GitHub] [hudi] MihawkZoro opened a new issue, #7040: [SUPPORT] spark-sql schema_evolution
MihawkZoro opened a new issue, #7040:
URL: https://github.com/apache/hudi/issues/7040
**Environment Description**
* Hudi version :
0.11.1
* Spark version :
3.2.2
* Hive version :
2.3.9
* Hadoop version :
2.7.3
* Storage
hdfs
**Describe the problem you faced**
I have a hudi table
```
create table ddl_test_t2 (
col1 string,
col2 string,
col3 string,
ts bigint
) using hudi
tblproperties (
type = 'mor',
primaryKey = 'col1',
preCombineField = 'ts'
);
```
I executed some DML and DDL for test about schema evolution
```
insert into ddl_test_t2 values('1','col2','col3',1),('2','col2','col3',2),('3','col2','col3',3);
ALTER TABLE ddl_test_t2 DROP COLUMN col3;
ALTER TABLE ddl_test_t2 RENAME COLUMN col2 to col3;
insert into ddl_test_t2 values('4','col2',4);
```
then I searched column col3 from table ddl_test_t2
```
select col3 from ddl_test_t2;
```
the result I expect is
```
col2
col2
col2
col2
```
the actual result was
```
col3
col3
col3
col2
```
<img width="1218" alt="image" src="https://user-images.githubusercontent.com/32875366/197393752-8b690175-2c3c-4f0a-bef5-d6e604eca03a.png">
I want know what is the problem and if this is a bug
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] xiarixiaoyao commented on issue #7040: [SUPPORT] spark-sql schema_evolution
Posted by GitBox <gi...@apache.org>.
xiarixiaoyao commented on issue #7040:
URL: https://github.com/apache/hudi/issues/7040#issuecomment-1288365585
@MihawkZoro schema evolution for hive and presto（mor table） can be found https://github.com/apache/hudi/pull/6989
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] xiarixiaoyao commented on issue #7040: [SUPPORT] spark-sql schema_evolution
Posted by GitBox <gi...@apache.org>.
xiarixiaoyao commented on issue #7040:
URL: https://github.com/apache/hudi/issues/7040#issuecomment-1290252404
> @xiarixiaoyao Thank you. When will the repaired official spark bundle jar be released?
expect 0.13.0
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] xiarixiaoyao commented on issue #7040: [SUPPORT] spark-sql schema_evolution
Posted by GitBox <gi...@apache.org>.
xiarixiaoyao commented on issue #7040:
URL: https://github.com/apache/hudi/issues/7040#issuecomment-1288348142
rewriteRecordWithNewSchema deal with rename failed，it should deal with rename first
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] MihawkZoro commented on issue #7040: [SUPPORT] spark-sql schema_evolution
Posted by GitBox <gi...@apache.org>.
MihawkZoro commented on issue #7040:
URL: https://github.com/apache/hudi/issues/7040#issuecomment-1288357272
@xiarixiaoyao When will this bug be fixed, we are using this feature, it is urgent
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] MihawkZoro commented on issue #7040: [SUPPORT] spark-sql schema_evolution
Posted by GitBox <gi...@apache.org>.
MihawkZoro commented on issue #7040:
URL: https://github.com/apache/hudi/issues/7040#issuecomment-1288368315
@xiarixiaoyao Thank you. When will the repaired official spark bundle jar be released?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] xiarixiaoyao commented on issue #7040: [SUPPORT] spark-sql schema_evolution
Posted by GitBox <gi...@apache.org>.
xiarixiaoyao commented on issue #7040:
URL: https://github.com/apache/hudi/issues/7040#issuecomment-1288359886
already fix local， let me raise a pr
spark.sql("set hoodie.schema.on.read.enable=true")
spark.sql("""create table ddl_test_t2 (
| col1 string,
| col2 string,
| col3 string,
| ts bigint
|) using hudi
|tblproperties (
| type = 'mor',
| primaryKey = 'col1',
| preCombineField = 'ts'
|)""".stripMargin)
spark.sql("insert into ddl_test_t2 values('1','col2','col3',1),('2','col2','col3',2),('3','col2','col3',3)")
spark.sql("""ALTER TABLE ddl_test_t2 DROP COLUMN col3""")
spark.sql("ALTER TABLE ddl_test_t2 RENAME COLUMN col2 to col3")
spark.sql("insert into ddl_test_t2 values('4','col2',4)")
spark.sql("select col3 from ddl_test_t2").show(false)
+----+
|col3|
+----+
|col2|
|col2|
|col2|
|col2|
+----+
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] xiarixiaoyao commented on issue #7040: [SUPPORT] spark-sql schema_evolution
Posted by GitBox <gi...@apache.org>.
xiarixiaoyao commented on issue #7040:
URL: https://github.com/apache/hudi/issues/7040#issuecomment-1288334741
@MihawkZoro
Thank you for your test，
This is really a bug， the final write ‘insert into ddl_test_t2 values('4','col2',4);’ trigger is bug，Fix this bug as soon as possible
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] xiarixiaoyao closed issue #7040: [SUPPORT] spark-sql schema_evolution
Posted by GitBox <gi...@apache.org>.
xiarixiaoyao closed issue #7040: [SUPPORT] spark-sql schema_evolution
URL: https://github.com/apache/hudi/issues/7040
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org