You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@hudi.apache.org by GitBox <gi...@apache.org> on 2022/05/24 13:54:08 UTC
[GitHub] [hudi] sunke38 opened a new issue, #5673: [SUPPORT]
sunke38 opened a new issue, #5673:
URL: https://github.com/apache/hudi/issues/5673
I am try to rename a colum by spark-sql.but it show scala.MatchError
spark.sql("ALTER TABLE hudi_cow_pt_tbl RENAME COLUMN ts TO tts")
Environment Description
Hudi version : 0.11
Spark version : 3.2.1
Hadoop version : 3.2.2
Storage (HDFS/S3/GCS..) : HDFS
Running on Docker? (yes/no) : no
spark.sql("ALTER TABLE hudi_cow_pt_tbl RENAME COLUMN ts TO tts")
ANTLR Tool version 4.7 used for code generation does not match the current runtime version 4.8ANTLR Tool version 4.7 used for code generation does not match the current runtime version 4.8ANTLR Tool version 4.7 used for code generation does not match the current runtime version 4.8ANTLR Tool version 4.7 used for code generation does not match the current runtime version 4.822/05/24 13:45:03 WARN HiveConf: HiveConf of name hive.metastore.event.db.notification.api.auth does not exist
scala.MatchError: org.apache.spark.sql.connector.catalog.TableChange$RenameColumn@368ecc (of class org.apache.spark.sql.connector.catalog.TableChange$RenameColumn)
at org.apache.spark.sql.hudi.catalog.HoodieCatalog.$anonfun$alterTable$4(HoodieCatalog.scala:169)
at scala.collection.IndexedSeqOptimized.foreach(IndexedSeqOptimized.scala:36)
at scala.collection.IndexedSeqOptimized.foreach$(IndexedSeqOptimized.scala:33)
at scala.collection.mutable.WrappedArray.foreach(WrappedArray.scala:38)
at org.apache.spark.sql.hudi.catalog.HoodieCatalog.$anonfun$alterTable$2(HoodieCatalog.scala:169)
at scala.collection.immutable.Map$Map1.foreach(Map.scala:193)
at org.apache.spark.sql.hudi.catalog.HoodieCatalog.alterTable(HoodieCatalog.scala:158)
at org.apache.spark.sql.hudi.catalog.HoodieCatalog.alterTable(HoodieCatalog.scala:43)
at org.apache.spark.sql.execution.datasources.v2.AlterTableExec.run(AlterTableExec.scala:37)
at org.apache.spark.sql.execution.datasources.v2.V2CommandExec.result$lzycompute(V2CommandExec.scala:43)
at org.apache.spark.sql.execution.datasources.v2.V2CommandExec.result(V2CommandExec.scala:43)
at org.apache.spark.sql.execution.datasources.v2.V2CommandExec.executeCollect(V2CommandExec.scala:49)
at org.apache.spark.sql.execution.QueryExecution$$anonfun$eagerlyExecuteCommands$1.$anonfun$applyOrElse$1(QueryExecution.scala:110)
at org.apache.spark.sql.execution.SQLExecution$.$anonfun$withNewExecutionId$5(SQLExecution.scala:103)
at org.apache.spark.sql.execution.SQLExecution$.withSQLConfPropagated(SQLExecution.scala:163)
at org.apache.spark.sql.execution.SQLExecution$.$anonfun$withNewExecutionId$1(SQLExecution.scala:90)
at org.apache.spark.sql.SparkSession.withActive(SparkSession.scala:775)
at org.apache.spark.sql.execution.SQLExecution$.withNewExecutionId(SQLExecution.scala:64)
at org.apache.spark.sql.execution.QueryExecution$$anonfun$eagerlyExecuteCommands$1.applyOrElse(QueryExecution.scala:110)
at org.apache.spark.sql.execution.QueryExecution$$anonfun$eagerlyExecuteCommands$1.applyOrElse(QueryExecution.scala:106)
at org.apache.spark.sql.catalyst.trees.TreeNode.$anonfun$transformDownWithPruning$1(TreeNode.scala:481)
at org.apache.spark.sql.catalyst.trees.CurrentOrigin$.withOrigin(TreeNode.scala:82)
at org.apache.spark.sql.catalyst.trees.TreeNode.transformDownWithPruning(TreeNode.scala:481)
at org.apache.spark.sql.catalyst.plans.logical.LogicalPlan.org$apache$spark$sql$catalyst$plans$logical$AnalysisHelper$$super$transformDownWithPruning(LogicalPlan.scala:30)
at org.apache.spark.sql.catalyst.plans.logical.AnalysisHelper.transformDownWithPruning(AnalysisHelper.scala:267)
at org.apache.spark.sql.catalyst.plans.logical.AnalysisHelper.transformDownWithPruning$(AnalysisHelper.scala:263)
at org.apache.spark.sql.catalyst.plans.logical.LogicalPlan.transformDownWithPruning(LogicalPlan.scala:30)
at org.apache.spark.sql.catalyst.plans.logical.LogicalPlan.transformDownWithPruning(LogicalPlan.scala:30)
at org.apache.spark.sql.catalyst.trees.TreeNode.transformDown(TreeNode.scala:457)
at org.apache.spark.sql.execution.QueryExecution.eagerlyExecuteCommands(QueryExecution.scala:106)
at org.apache.spark.sql.execution.QueryExecution.commandExecuted$lzycompute(QueryExecution.scala:93)
at org.apache.spark.sql.execution.QueryExecution.commandExecuted(QueryExecution.scala:91)
at org.apache.spark.sql.Dataset.<init>(Dataset.scala:219)
at org.apache.spark.sql.Dataset$.$anonfun$ofRows$2(Dataset.scala:99)
at org.apache.spark.sql.SparkSession.withActive(SparkSession.scala:775)
at org.apache.spark.sql.Dataset$.ofRows(Dataset.scala:96)
at org.apache.spark.sql.SparkSession.$anonfun$sql$1(SparkSession.scala:618)
at org.apache.spark.sql.SparkSession.withActive(SparkSession.scala:775)
at org.apache.spark.sql.SparkSession.sql(SparkSession.scala:613)
... 47 elided
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] sunke38 commented on issue #5673: [SUPPORT]scala.MatchError for rename colum
Posted by GitBox <gi...@apache.org>.
sunke38 commented on issue #5673:
URL: https://github.com/apache/hudi/issues/5673#issuecomment-1138047204
@xushiyan I did this in spark shell ,so only code is
'''
spark.sql("ALTER TABLE hudi_cow_pt_tbl RENAME COLUMN ts TO tts")
'''
here is the command for spark shell
'''
spark-shell
--jars /home/kadm/module/hudi-0.11/packaging/hudi-spark-bundle/target/hudi-spark3.2-bundle_2.12-0.11.0.jar
--packages org.apache.spark:spark-avro_2.12:3.2.1 --conf 'spark.serializer=org.apache.spark.serializer.KryoSerializer'
--conf 'spark.sql.extensions=org.apache.spark.sql.hudi.HoodieSparkSessionExtension'
--conf 'spark.sql.catalog.spark_catalog=org.apache.spark.sql.hudi.catalog.HoodieCatalog'
'''
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] feiwei8586 commented on issue #5673: [SUPPORT]scala.MatchError for rename colum
Posted by GitBox <gi...@apache.org>.
feiwei8586 commented on issue #5673:
URL: https://github.com/apache/hudi/issues/5673#issuecomment-1200959361
in hudi-spark-datasource/hudi-spark3.2.x/src/main/scala/org/apache/spark/sql/hudi/catalog)/HoodieCatalog.scala
spark change table column name is not support . @XuQianJin-Stars @YannByron
override def alterTable(ident: Identifier, changes: TableChange*): Table = {
loadTable(ident) match {
case HoodieV1OrV2Table(table) => {
val tableIdent = TableIdentifier(ident.name(), ident.namespace().lastOption)
changes.groupBy(c => c.getClass).foreach {
case (t, newColumns) if t == classOf[AddColumn] =>
AlterHoodieTableAddColumnsCommand(
tableIdent,
newColumns.asInstanceOf[Seq[AddColumn]].map { col =>
StructField(
col.fieldNames()(0),
col.dataType(),
col.isNullable)
}).run(spark)
case (t, columnChanges) if classOf[ColumnChange].isAssignableFrom(t) =>
columnChanges.foreach {
case dataType: UpdateColumnType =>
val colName = UnresolvedAttribute(dataType.fieldNames()).name
val newDataType = dataType.newDataType()
val structField = StructField(colName, newDataType)
AlterHoodieTableChangeColumnCommand(tableIdent, colName, structField).run(spark)
case dataType: UpdateColumnComment =>
val newComment = dataType.newComment()
val colName = UnresolvedAttribute(dataType.fieldNames()).name
val fieldOpt = table.schema.findNestedField(dataType.fieldNames(), includeCollections = true,
spark.sessionState.conf.resolver).map(_._2)
val field = fieldOpt.getOrElse {
throw new AnalysisException(
s"Couldn't find column $colName in:\n${table.schema.treeString}")
}
AlterHoodieTableChangeColumnCommand(tableIdent, colName, field.withComment(newComment)).run(spark)
}
case (t, _) =>
throw new UnsupportedOperationException(s"not supported table change: ${t.getClass}")
}
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] nsivabalan commented on issue #5673: [SUPPORT]scala.MatchError for rename colum
Posted by GitBox <gi...@apache.org>.
nsivabalan commented on issue #5673:
URL: https://github.com/apache/hudi/issues/5673#issuecomment-1229359370
you can enable schema on read and give it a try
https://hudi.apache.org/docs/schema_evolution/#sparksql-schema-evolution-and-syntax-description
hoodie.schema.on.read.enable=true
CC @xiarixiaoyao
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] nsivabalan commented on issue #5673: [SUPPORT]scala.MatchError for rename colum
Posted by GitBox <gi...@apache.org>.
nsivabalan commented on issue #5673:
URL: https://github.com/apache/hudi/issues/5673#issuecomment-1149280613
@XuQianJin-Stars @YannByron : Can either of you assist here please.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] codope closed issue #5673: [SUPPORT]scala.MatchError for rename colum
Posted by "codope (via GitHub)" <gi...@apache.org>.
codope closed issue #5673: [SUPPORT]scala.MatchError for rename colum
URL: https://github.com/apache/hudi/issues/5673
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] nsivabalan commented on issue #5673: [SUPPORT]scala.MatchError for rename colum
Posted by GitBox <gi...@apache.org>.
nsivabalan commented on issue #5673:
URL: https://github.com/apache/hudi/issues/5673#issuecomment-1302884770
@sunke38 : did you get a chance to give it a try. are we good to close it out. or is there anything you need more assistance.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] codope commented on issue #5673: [SUPPORT]scala.MatchError for rename colum
Posted by GitBox <gi...@apache.org>.
codope commented on issue #5673:
URL: https://github.com/apache/hudi/issues/5673#issuecomment-1348758030
@sunke38 Gentle reminder to try out the suggestion and close if it works.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] xushiyan commented on issue #5673: [SUPPORT]scala.MatchError for rename colum
Posted by GitBox <gi...@apache.org>.
xushiyan commented on issue #5673:
URL: https://github.com/apache/hudi/issues/5673#issuecomment-1137951296
@sunke38 can you share your complete command to start the spark session
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] codope commented on issue #5673: [SUPPORT]scala.MatchError for rename colum
Posted by "codope (via GitHub)" <gi...@apache.org>.
codope commented on issue #5673:
URL: https://github.com/apache/hudi/issues/5673#issuecomment-1412104529
Closing due to inactivity.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org