You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@hudi.apache.org by "JoshuaZhuCN (via GitHub)" <gi...@apache.org> on 2023/05/12 06:04:48 UTC
[GitHub] [hudi] JoshuaZhuCN opened a new issue, #8695: [SUPPORT] Failed to add fields in BUCKET index table
JoshuaZhuCN opened a new issue, #8695:
URL: https://github.com/apache/hudi/issues/8695
Failed to add fields in BUCKET index table
**To Reproduce**
Steps to reproduce the behavior:
1. create a bucket index table
```
create table if not exists `default`.hudi_table0 (
id int,
name string,
price double,
ts timestamp
) using hudi
options (
type = 'cow',
primaryKey = 'id',
preCombineField = 'ts',
`hoodie.index.type`='BUCKET',
`hoodie.index.bucket.engine`='SIMPLE',
`hoodie.bucket.index.num.buckets`='2',
`hoodie.bucket.index.hash.field`='id',
`hoodie.storage.layout.type`='BUCKET',
`hoodie.storage.layout.partitioner.class`='org.apache.hudi.table.action.commit.SparkBucketIndexPartitioner'
);
```
2. try to add a new column
```ALTER TABLE `default`.hudi_table0 ADD COLUMNS(`column1_added` string comment 'first add column');```
**Expected behavior**
A clear and concise description of what you expected to happen.
**Environment Description**
* Hudi version : 0.12.1
* Spark version : 3.1.3
* Hive version : 3.1.0
* Hadoop version : 3.1.1
* Storage (HDFS/S3/GCS..) : HDFS
* Running on Docker? (yes/no) : no
**Stacktrace**
```
ERROR org.apache.spark.sql.hive.thriftserver.SparkSQLDriver - Failed in [ALTER TABLE `default`.hudi_table0 ADD COLUMNS(`column1_added` string comment 'first add column')]
org.apache.hudi.exception.HoodieIndexException: Bucket index key (if configured) must be subset of record key.
at org.apache.hudi.config.HoodieIndexConfig$Builder.validateBucketIndexConfig(HoodieIndexConfig.java:647) ~[hudi-spark3.1.3-bundle_2.12-0.12.1.jar:0.12.1]
at org.apache.hudi.config.HoodieIndexConfig$Builder.build(HoodieIndexConfig.java:615) ~[hudi-spark3.1.3-bundle_2.12-0.12.1.jar:0.12.1]
at org.apache.hudi.config.HoodieWriteConfig$Builder.setDefaults(HoodieWriteConfig.java:2554) ~[hudi-spark3.1.3-bundle_2.12-0.12.1.jar:0.12.1]
at org.apache.hudi.config.HoodieWriteConfig$Builder.build(HoodieWriteConfig.java:2679) ~[hudi-spark3.1.3-bundle_2.12-0.12.1.jar:0.12.1]
at org.apache.hudi.DataSourceUtils.createHoodieConfig(DataSourceUtils.java:188) ~[hudi-spark3.1.3-bundle_2.12-0.12.1.jar:0.12.1]
at org.apache.hudi.DataSourceUtils.createHoodieClient(DataSourceUtils.java:193) ~[hudi-spark3.1.3-bundle_2.12-0.12.1.jar:0.12.1]
at org.apache.spark.sql.hudi.command.AlterHoodieTableAddColumnsCommand$.commitWithSchema(AlterHoodieTableAddColumnsCommand.scala:110) ~[hudi-spark3.1.3-bundle_2.12-0.12.1.jar:0.12.1]
at org.apache.spark.sql.hudi.command.AlterHoodieTableAddColumnsCommand.run(AlterHoodieTableAddColumnsCommand.scala:66) ~[hudi-spark3.1.3-bundle_2.12-0.12.1.jar:0.12.1]
at org.apache.spark.sql.execution.command.ExecutedCommandExec.sideEffectResult$lzycompute(commands.scala:70) ~[spark-sql_2.12-3.1.3.jar:3.1.3]
at org.apache.spark.sql.execution.command.ExecutedCommandExec.sideEffectResult(commands.scala:68) ~[spark-sql_2.12-3.1.3.jar:3.1.3]
at org.apache.spark.sql.execution.command.ExecutedCommandExec.executeCollect(commands.scala:79) ~[spark-sql_2.12-3.1.3.jar:3.1.3]
at org.apache.spark.sql.Dataset.$anonfun$logicalPlan$1(Dataset.scala:228) ~[spark-sql_2.12-3.1.3.jar:3.1.3]
at org.apache.spark.sql.Dataset.$anonfun$withAction$1(Dataset.scala:3700) ~[spark-sql_2.12-3.1.3.jar:3.1.3]
at org.apache.spark.sql.execution.SQLExecution$.$anonfun$withNewExecutionId$5(SQLExecution.scala:103) ~[spark-sql_2.12-3.1.3.jar:3.1.3]
at org.apache.spark.sql.execution.SQLExecution$.withSQLConfPropagated(SQLExecution.scala:163) ~[spark-sql_2.12-3.1.3.jar:3.1.3]
at org.apache.spark.sql.execution.SQLExecution$.$anonfun$withNewExecutionId$1(SQLExecution.scala:90) ~[spark-sql_2.12-3.1.3.jar:3.1.3]
at org.apache.spark.sql.SparkSession.withActive(SparkSession.scala:775) ~[spark-sql_2.12-3.1.3.jar:3.1.3]
at org.apache.spark.sql.execution.SQLExecution$.withNewExecutionId(SQLExecution.scala:64) ~[spark-sql_2.12-3.1.3.jar:3.1.3]
at org.apache.spark.sql.Dataset.withAction(Dataset.scala:3698) ~[spark-sql_2.12-3.1.3.jar:3.1.3]
at org.apache.spark.sql.Dataset.<init>(Dataset.scala:228) ~[spark-sql_2.12-3.1.3.jar:3.1.3]
at org.apache.spark.sql.Dataset$.$anonfun$ofRows$2(Dataset.scala:99) ~[spark-sql_2.12-3.1.3.jar:3.1.3]
at org.apache.spark.sql.SparkSession.withActive(SparkSession.scala:775) ~[spark-sql_2.12-3.1.3.jar:3.1.3]
at org.apache.spark.sql.Dataset$.ofRows(Dataset.scala:96) ~[spark-sql_2.12-3.1.3.jar:3.1.3]
at org.apache.spark.sql.SparkSession.$anonfun$sql$1(SparkSession.scala:618) ~[spark-sql_2.12-3.1.3.jar:3.1.3]
at org.apache.spark.sql.SparkSession.withActive(SparkSession.scala:775) ~[spark-sql_2.12-3.1.3.jar:3.1.3]
at org.apache.spark.sql.SparkSession.sql(SparkSession.scala:613) ~[spark-sql_2.12-3.1.3.jar:3.1.3]
at org.apache.spark.sql.SQLContext.sql(SQLContext.scala:650) ~[spark-sql_2.12-3.1.3.jar:3.1.3]
at org.apache.spark.sql.hive.thriftserver.SparkSQLDriver.run(SparkSQLDriver.scala:67) ~[spark-hive-thriftserver_2.12-3.1.3.jar:3.1.3]
at org.apache.spark.sql.hive.thriftserver.SparkSQLCLIDriver.processCmd(SparkSQLCLIDriver.scala:381) ~[spark-hive-thriftserver_2.12-3.1.3.jar:3.1.3]
at org.apache.spark.sql.hive.thriftserver.SparkSQLCLIDriver.$anonfun$processLine$1(SparkSQLCLIDriver.scala:500) ~[spark-hive-thriftserver_2.12-3.1.3.jar:3.1.3]
at org.apache.spark.sql.hive.thriftserver.SparkSQLCLIDriver.$anonfun$processLine$1$adapted(SparkSQLCLIDriver.scala:494) ~[spark-hive-thriftserver_2.12-3.1.3.jar:3.1.3]
at scala.collection.Iterator.foreach(Iterator.scala:941) [scala-library-2.12.10.jar:?]
at scala.collection.Iterator.foreach$(Iterator.scala:941) [scala-library-2.12.10.jar:?]
at scala.collection.AbstractIterator.foreach(Iterator.scala:1429) [scala-library-2.12.10.jar:?]
at scala.collection.IterableLike.foreach(IterableLike.scala:74) [scala-library-2.12.10.jar:?]
at scala.collection.IterableLike.foreach$(IterableLike.scala:73) [scala-library-2.12.10.jar:?]
at scala.collection.AbstractIterable.foreach(Iterable.scala:56) [scala-library-2.12.10.jar:?]
at org.apache.spark.sql.hive.thriftserver.SparkSQLCLIDriver.processLine(SparkSQLCLIDriver.scala:494) [spark-hive-thriftserver_2.12-3.1.3.jar:3.1.3]
at org.apache.spark.sql.hive.thriftserver.SparkSQLCLIDriver$.main(SparkSQLCLIDriver.scala:284) [spark-hive-thriftserver_2.12-3.1.3.jar:3.1.3]
at org.apache.spark.sql.hive.thriftserver.SparkSQLCLIDriver.main(SparkSQLCLIDriver.scala) [spark-hive-thriftserver_2.12-3.1.3.jar:3.1.3]
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[?:1.8.0_271]
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) ~[?:1.8.0_271]
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[?:1.8.0_271]
at java.lang.reflect.Method.invoke(Method.java:498) ~[?:1.8.0_271]
at org.apache.spark.deploy.JavaMainApplication.start(SparkApplication.scala:52) [spark-core_2.12-3.1.3.jar:3.1.3]
at org.apache.spark.deploy.SparkSubmit.org$apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:951) [spark-core_2.12-3.1.3.jar:3.1.3]
at org.apache.spark.deploy.SparkSubmit.doRunMain$1(SparkSubmit.scala:180) [spark-core_2.12-3.1.3.jar:3.1.3]
at org.apache.spark.deploy.SparkSubmit.submit(SparkSubmit.scala:203) [spark-core_2.12-3.1.3.jar:3.1.3]
at org.apache.spark.deploy.SparkSubmit.doSubmit(SparkSubmit.scala:90) [spark-core_2.12-3.1.3.jar:3.1.3]
at org.apache.spark.deploy.SparkSubmit$$anon$2.doSubmit(SparkSubmit.scala:1039) [spark-core_2.12-3.1.3.jar:3.1.3]
at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:1048) [spark-core_2.12-3.1.3.jar:3.1.3]
at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala) [spark-core_2.12-3.1.3.jar:3.1.3]
org.apache.hudi.exception.HoodieIndexException: Bucket index key (if configured) must be subset of record key.
at org.apache.hudi.config.HoodieIndexConfig$Builder.validateBucketIndexConfig(HoodieIndexConfig.java:647)
at org.apache.hudi.config.HoodieIndexConfig$Builder.build(HoodieIndexConfig.java:615)
at org.apache.hudi.config.HoodieWriteConfig$Builder.setDefaults(HoodieWriteConfig.java:2554)
at org.apache.hudi.config.HoodieWriteConfig$Builder.build(HoodieWriteConfig.java:2679)
at org.apache.hudi.DataSourceUtils.createHoodieConfig(DataSourceUtils.java:188)
at org.apache.hudi.DataSourceUtils.createHoodieClient(DataSourceUtils.java:193)
at org.apache.spark.sql.hudi.command.AlterHoodieTableAddColumnsCommand$.commitWithSchema(AlterHoodieTableAddColumnsCommand.scala:110)
at org.apache.spark.sql.hudi.command.AlterHoodieTableAddColumnsCommand.run(AlterHoodieTableAddColumnsCommand.scala:66)
at org.apache.spark.sql.execution.command.ExecutedCommandExec.sideEffectResult$lzycompute(commands.scala:70)
at org.apache.spark.sql.execution.command.ExecutedCommandExec.sideEffectResult(commands.scala:68)
at org.apache.spark.sql.execution.command.ExecutedCommandExec.executeCollect(commands.scala:79)
at org.apache.spark.sql.Dataset.$anonfun$logicalPlan$1(Dataset.scala:228)
at org.apache.spark.sql.Dataset.$anonfun$withAction$1(Dataset.scala:3700)
at org.apache.spark.sql.execution.SQLExecution$.$anonfun$withNewExecutionId$5(SQLExecution.scala:103)
at org.apache.spark.sql.execution.SQLExecution$.withSQLConfPropagated(SQLExecution.scala:163)
at org.apache.spark.sql.execution.SQLExecution$.$anonfun$withNewExecutionId$1(SQLExecution.scala:90)
at org.apache.spark.sql.SparkSession.withActive(SparkSession.scala:775)
at org.apache.spark.sql.execution.SQLExecution$.withNewExecutionId(SQLExecution.scala:64)
at org.apache.spark.sql.Dataset.withAction(Dataset.scala:3698)
at org.apache.spark.sql.Dataset.<init>(Dataset.scala:228)
at org.apache.spark.sql.Dataset$.$anonfun$ofRows$2(Dataset.scala:99)
at org.apache.spark.sql.SparkSession.withActive(SparkSession.scala:775)
at org.apache.spark.sql.Dataset$.ofRows(Dataset.scala:96)
at org.apache.spark.sql.SparkSession.$anonfun$sql$1(SparkSession.scala:618)
at org.apache.spark.sql.SparkSession.withActive(SparkSession.scala:775)
at org.apache.spark.sql.SparkSession.sql(SparkSession.scala:613)
at org.apache.spark.sql.SQLContext.sql(SQLContext.scala:650)
at org.apache.spark.sql.hive.thriftserver.SparkSQLDriver.run(SparkSQLDriver.scala:67)
at org.apache.spark.sql.hive.thriftserver.SparkSQLCLIDriver.processCmd(SparkSQLCLIDriver.scala:381)
at org.apache.spark.sql.hive.thriftserver.SparkSQLCLIDriver.$anonfun$processLine$1(SparkSQLCLIDriver.scala:500)
at org.apache.spark.sql.hive.thriftserver.SparkSQLCLIDriver.$anonfun$processLine$1$adapted(SparkSQLCLIDriver.scala:494)
at scala.collection.Iterator.foreach(Iterator.scala:941)
at scala.collection.Iterator.foreach$(Iterator.scala:941)
at scala.collection.AbstractIterator.foreach(Iterator.scala:1429)
at scala.collection.IterableLike.foreach(IterableLike.scala:74)
at scala.collection.IterableLike.foreach$(IterableLike.scala:73)
at scala.collection.AbstractIterable.foreach(Iterable.scala:56)
at org.apache.spark.sql.hive.thriftserver.SparkSQLCLIDriver.processLine(SparkSQLCLIDriver.scala:494)
at org.apache.spark.sql.hive.thriftserver.SparkSQLCLIDriver$.main(SparkSQLCLIDriver.scala:284)
at org.apache.spark.sql.hive.thriftserver.SparkSQLCLIDriver.main(SparkSQLCLIDriver.scala)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.apache.spark.deploy.JavaMainApplication.start(SparkApplication.scala:52)
at org.apache.spark.deploy.SparkSubmit.org$apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:951)
at org.apache.spark.deploy.SparkSubmit.doRunMain$1(SparkSubmit.scala:180)
at org.apache.spark.deploy.SparkSubmit.submit(SparkSubmit.scala:203)
at org.apache.spark.deploy.SparkSubmit.doSubmit(SparkSubmit.scala:90)
at org.apache.spark.deploy.SparkSubmit$$anon$2.doSubmit(SparkSubmit.scala:1039)
at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:1048)
at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] danny0405 closed issue #8695: [SUPPORT] Failed to add fields in BUCKET index table
Posted by "danny0405 (via GitHub)" <gi...@apache.org>.
danny0405 closed issue #8695: [SUPPORT] Failed to add fields in BUCKET index table
URL: https://github.com/apache/hudi/issues/8695
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] JoshuaZhuCN commented on issue #8695: [SUPPORT] Failed to add fields in BUCKET index table
Posted by "JoshuaZhuCN (via GitHub)" <gi...@apache.org>.
JoshuaZhuCN commented on issue #8695:
URL: https://github.com/apache/hudi/issues/8695#issuecomment-1547845752
> Thanks for the report @JoshuaZhuCN , I can not re-produce the same error with the SQL statements you provided, but there indeed exists some mistakes for the sql options, have fired a PR to fix it: #8710
@danny0405 maybe it's related to this PR (https://github.com/apache/hudi/pull/8243) ，i merged this PR into the hudi version of my cluster
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] danny0405 commented on issue #8695: [SUPPORT] Failed to add fields in BUCKET index table
Posted by "danny0405 (via GitHub)" <gi...@apache.org>.
danny0405 commented on issue #8695:
URL: https://github.com/apache/hudi/issues/8695#issuecomment-1548058598
> > Thanks for the report @JoshuaZhuCN , I can not re-produce the same error with the SQL statements you provided, but there indeed exists some mistakes for the sql options, have fired a PR to fix it: #8710
>
> @danny0405 maybe it's related to this PR (#8243) ，i merged this PR into the hudi version of my cluster
Maybe, but the code in master is also problematic. The spark sql option key is not translated into write options correctly.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] danny0405 commented on issue #8695: [SUPPORT] Failed to add fields in BUCKET index table
Posted by "danny0405 (via GitHub)" <gi...@apache.org>.
danny0405 commented on issue #8695:
URL: https://github.com/apache/hudi/issues/8695#issuecomment-1547294067
Thanks for the report @JoshuaZhuCN , I can not re-produce the same error with the SQL statements you provided, but there indeed exists some mistakes for the sql options, have fired a PR to fix it: https://github.com/apache/hudi/pull/8710
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org