You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@hudi.apache.org by "bigdata-spec (via GitHub)" <gi...@apache.org> on 2023/04/07 08:01:38 UTC
[GitHub] [hudi] bigdata-spec opened a new issue, #8403: [SUPPORT]Caused by: java.lang.IllegalArgumentException: Partition path is not in the form partition_key=partition_value.
bigdata-spec opened a new issue, #8403:
URL: https://github.com/apache/hudi/issues/8403
Hello,
When I delete mor hudi table .it doen't work well.
Spark Version: spark-3.2.3-bin-hadoop2.7 download from spark web.
Hive Version: 2.1.1-cdh6.3.2
Hudi Version: modify pom.xml of spark.version and run
mvn clean package -DskipTests -Drat.skip=true -Dcheckstyle.skip=true -Dspark3.2 -Dscala-2.12
```
[root@dataplatform-node-01 bin]$ ./spark-sql --conf 'spark.serializer=org.apache.spark.serializer.KryoSerializer' --conf 'spark.sql.extensions=org.apache.spark.sql.hudi.HoodieSparkSessionExtension' --conf 'spark.sql.catalog.spark_catalog=org.apache.spark.sql.hudi.catalog.HoodieCatalog'
spark-sql> create table hudi_mor_tbl (
> id int,
> name string,
> price double,
> ts bigint
> ) using hudi
> tblproperties (
> type = 'mor',
> primaryKey = 'id',
> preCombineField = 'ts'
> );
57964 [main] WARN org.apache.hudi.common.config.DFSPropertiesConfiguration - Cannot find HUDI_CONF_DIR, please set it as the dir of hudi-defaults.conf
57986 [main] WARN org.apache.hudi.common.config.DFSPropertiesConfiguration - Properties file file:/etc/hudi/conf/hudi-defaults.conf not found. Ignoring to load props file
63326 [main] WARN org.apache.hadoop.hive.ql.session.SessionState - METASTORE_FILTER_HOOK will be ignored, since hive.security.authorization.manager is set to instance of HiveAuthorizerFactory.
Time taken: 5.795 seconds
spark-sql> **insert into hudi_mor_tbl select 1, 'a1', 20, 1000;**
75872 [main] WARN org.apache.hudi.metadata.HoodieBackedTableMetadata - Metadata table was not found at path hdfs://nameservice1/user/hive/warehouse/hudi_demo.db/hudi_mor_tbl/.hoodie/metadata
94603 [Executor task launch worker for task 0.0 in stage 6.0 (TID 6)] WARN org.apache.hadoop.metrics2.impl.MetricsConfig - Cannot locate configuration: tried hadoop-metrics2-hbase.properties,hadoop-metrics2.properties
00:44 WARN: Timeline-server-based markers are not supported for HDFS: base path hdfs://nameservice1/user/hive/warehouse/hudi_demo.db/hudi_mor_tbl. Falling back to direct markers.
00:46 WARN: Timeline-server-based markers are not supported for HDFS: base path hdfs://nameservice1/user/hive/warehouse/hudi_demo.db/hudi_mor_tbl. Falling back to direct markers.
01:05 WARN: Timeline-server-based markers are not supported for HDFS: base path hdfs://nameservice1/user/hive/warehouse/hudi_demo.db/hudi_mor_tbl. Falling back to direct markers.
149496 [main] WARN org.apache.hadoop.hive.conf.HiveConf - HiveConf of name hive.vectorized.use.checked.expressions does not exist
149496 [main] WARN org.apache.hadoop.hive.conf.HiveConf - HiveConf of name hive.strict.checks.no.partition.filter does not exist
149496 [main] WARN org.apache.hadoop.hive.conf.HiveConf - HiveConf of name hive.internal.ss.authz.settings.applied.marker does not exist
149496 [main] WARN org.apache.hadoop.hive.conf.HiveConf - HiveConf of name hive.stats.jdbc.timeout does not exist
149496 [main] WARN org.apache.hadoop.hive.conf.HiveConf - HiveConf of name hive.strict.checks.orderby.no.limit does not exist
149496 [main] WARN org.apache.hadoop.hive.conf.HiveConf - HiveConf of name hive.vectorized.input.format.excludes does not exist
149497 [main] WARN org.apache.hadoop.hive.conf.HiveConf - HiveConf of name hive.stats.retries.wait does not exist
Time taken: 77.571 seconds
spark-sql> **select * from hudi_mor_tbl;**
20230407155323663 20230407155323663_0_0 id:1 172d0648-4dea-4aaf-b5b5-94bd2ae50f84-0_0-27-1215_20230407155323663.parquet 1 a1 20.0 1000
Time taken: 0.573 seconds, Fetched 1 row(s)
spark-sql> **delete from hudi_mor_tbl where id=1;**
01:48 WARN: Timeline-server-based markers are not supported for HDFS: base path hdfs://nameservice1/user/hive/warehouse/hudi_demo.db/hudi_mor_tbl. Falling back to direct markers.
01:54 WARN: Timeline-server-based markers are not supported for HDFS: base path hdfs://nameservice1/user/hive/warehouse/hudi_demo.db/hudi_mor_tbl. Falling back to direct markers.
02:08 WARN: Timeline-server-based markers are not supported for HDFS: base path hdfs://nameservice1/user/hive/warehouse/hudi_demo.db/hudi_mor_tbl. Falling back to direct markers.
211041 [main] WARN org.apache.hadoop.hive.conf.HiveConf - HiveConf of name hive.vectorized.use.checked.expressions does not exist
211041 [main] WARN org.apache.hadoop.hive.conf.HiveConf - HiveConf of name hive.strict.checks.no.partition.filter does not exist
211041 [main] WARN org.apache.hadoop.hive.conf.HiveConf - HiveConf of name hive.internal.ss.authz.settings.applied.marker does not exist
211041 [main] WARN org.apache.hadoop.hive.conf.HiveConf - HiveConf of name hive.stats.jdbc.timeout does not exist
211041 [main] WARN org.apache.hadoop.hive.conf.HiveConf - HiveConf of name hive.strict.checks.orderby.no.limit does not exist
211041 [main] WARN org.apache.hadoop.hive.conf.HiveConf - HiveConf of name hive.vectorized.input.format.excludes does not exist
211042 [main] WARN org.apache.hadoop.hive.conf.HiveConf - HiveConf of name hive.stats.retries.wait does not exist
211897 [main] ERROR org.apache.spark.sql.hive.thriftserver.SparkSQLDriver - Failed in [delete from hudi_mor_tbl where id=1]
org.apache.hudi.exception.HoodieException: Could not sync using the meta sync class org.apache.hudi.hive.HiveSyncTool
at org.apache.hudi.sync.common.util.SyncUtilHelpers.runHoodieMetaSync(SyncUtilHelpers.java:61)
at org.apache.hudi.HoodieSparkSqlWriter$.$anonfun$metaSync$2(HoodieSparkSqlWriter.scala:623)
at org.apache.hudi.HoodieSparkSqlWriter$.$anonfun$metaSync$2$adapted(HoodieSparkSqlWriter.scala:622)
at scala.collection.mutable.HashSet.foreach(HashSet.scala:79)
at org.apache.hudi.HoodieSparkSqlWriter$.metaSync(HoodieSparkSqlWriter.scala:622)
at org.apache.hudi.HoodieSparkSqlWriter$.commitAndPerformPostOperations(HoodieSparkSqlWriter.scala:681)
at org.apache.hudi.HoodieSparkSqlWriter$.write(HoodieSparkSqlWriter.scala:315)
at org.apache.hudi.DefaultSource.createRelation(DefaultSource.scala:171)
at org.apache.spark.sql.execution.datasources.SaveIntoDataSourceCommand.run(SaveIntoDataSourceCommand.scala:47)
at org.apache.spark.sql.execution.command.ExecutedCommandExec.sideEffectResult$lzycompute(commands.scala:75)
at org.apache.spark.sql.execution.command.ExecutedCommandExec.sideEffectResult(commands.scala:73)
at org.apache.spark.sql.execution.command.ExecutedCommandExec.executeCollect(commands.scala:84)
at org.apache.spark.sql.execution.QueryExecution$$anonfun$eagerlyExecuteCommands$1.$anonfun$applyOrElse$1(QueryExecution.scala:97)
at org.apache.spark.sql.execution.SQLExecution$.$anonfun$withNewExecutionId$5(SQLExecution.scala:103)
at org.apache.spark.sql.execution.SQLExecution$.withSQLConfPropagated(SQLExecution.scala:163)
at org.apache.spark.sql.execution.SQLExecution$.$anonfun$withNewExecutionId$1(SQLExecution.scala:90)
at org.apache.spark.sql.SparkSession.withActive(SparkSession.scala:775)
at org.apache.spark.sql.execution.SQLExecution$.withNewExecutionId(SQLExecution.scala:64)
at org.apache.spark.sql.execution.QueryExecution$$anonfun$eagerlyExecuteCommands$1.applyOrElse(QueryExecution.scala:97)
at org.apache.spark.sql.execution.QueryExecution$$anonfun$eagerlyExecuteCommands$1.applyOrElse(QueryExecution.scala:93)
at org.apache.spark.sql.catalyst.trees.TreeNode.$anonfun$transformDownWithPruning$1(TreeNode.scala:481)
at org.apache.spark.sql.catalyst.trees.CurrentOrigin$.withOrigin(TreeNode.scala:82)
at org.apache.spark.sql.catalyst.trees.TreeNode.transformDownWithPruning(TreeNode.scala:481)
at org.apache.spark.sql.catalyst.plans.logical.LogicalPlan.org$apache$spark$sql$catalyst$plans$logical$AnalysisHelper$$super$transformDownWithPruning(LogicalPlan.scala:30)
at org.apache.spark.sql.catalyst.plans.logical.AnalysisHelper.transformDownWithPruning(AnalysisHelper.scala:267)
at org.apache.spark.sql.catalyst.plans.logical.AnalysisHelper.transformDownWithPruning$(AnalysisHelper.scala:263)
at org.apache.spark.sql.catalyst.plans.logical.LogicalPlan.transformDownWithPruning(LogicalPlan.scala:30)
at org.apache.spark.sql.catalyst.plans.logical.LogicalPlan.transformDownWithPruning(LogicalPlan.scala:30)
at org.apache.spark.sql.catalyst.trees.TreeNode.transformDown(TreeNode.scala:457)
at org.apache.spark.sql.execution.QueryExecution.eagerlyExecuteCommands(QueryExecution.scala:93)
at org.apache.spark.sql.execution.QueryExecution.commandExecuted$lzycompute(QueryExecution.scala:80)
at org.apache.spark.sql.execution.QueryExecution.commandExecuted(QueryExecution.scala:78)
at org.apache.spark.sql.execution.QueryExecution.assertCommandExecuted(QueryExecution.scala:115)
at org.apache.spark.sql.DataFrameWriter.runCommand(DataFrameWriter.scala:848)
at org.apache.spark.sql.DataFrameWriter.saveToV1Source(DataFrameWriter.scala:382)
at org.apache.spark.sql.DataFrameWriter.saveInternal(DataFrameWriter.scala:355)
at org.apache.spark.sql.DataFrameWriter.save(DataFrameWriter.scala:247)
at org.apache.spark.sql.hudi.command.DeleteHoodieTableCommand.run(DeleteHoodieTableCommand.scala:48)
at org.apache.spark.sql.execution.command.ExecutedCommandExec.sideEffectResult$lzycompute(commands.scala:75)
at org.apache.spark.sql.execution.command.ExecutedCommandExec.sideEffectResult(commands.scala:73)
at org.apache.spark.sql.execution.command.ExecutedCommandExec.executeCollect(commands.scala:84)
at org.apache.spark.sql.execution.QueryExecution$$anonfun$eagerlyExecuteCommands$1.$anonfun$applyOrElse$1(QueryExecution.scala:97)
at org.apache.spark.sql.execution.SQLExecution$.$anonfun$withNewExecutionId$5(SQLExecution.scala:103)
at org.apache.spark.sql.execution.SQLExecution$.withSQLConfPropagated(SQLExecution.scala:163)
at org.apache.spark.sql.execution.SQLExecution$.$anonfun$withNewExecutionId$1(SQLExecution.scala:90)
at org.apache.spark.sql.SparkSession.withActive(SparkSession.scala:775)
at org.apache.spark.sql.execution.SQLExecution$.withNewExecutionId(SQLExecution.scala:64)
at org.apache.spark.sql.execution.QueryExecution$$anonfun$eagerlyExecuteCommands$1.applyOrElse(QueryExecution.scala:97)
at org.apache.spark.sql.execution.QueryExecution$$anonfun$eagerlyExecuteCommands$1.applyOrElse(QueryExecution.scala:93)
at org.apache.spark.sql.catalyst.trees.TreeNode.$anonfun$transformDownWithPruning$1(TreeNode.scala:481)
at org.apache.spark.sql.catalyst.trees.CurrentOrigin$.withOrigin(TreeNode.scala:82)
at org.apache.spark.sql.catalyst.trees.TreeNode.transformDownWithPruning(TreeNode.scala:481)
at org.apache.spark.sql.catalyst.plans.logical.LogicalPlan.org$apache$spark$sql$catalyst$plans$logical$AnalysisHelper$$super$transformDownWithPruning(LogicalPlan.scala:30)
at org.apache.spark.sql.catalyst.plans.logical.AnalysisHelper.transformDownWithPruning(AnalysisHelper.scala:267)
at org.apache.spark.sql.catalyst.plans.logical.AnalysisHelper.transformDownWithPruning$(AnalysisHelper.scala:263)
at org.apache.spark.sql.catalyst.plans.logical.LogicalPlan.transformDownWithPruning(LogicalPlan.scala:30)
at org.apache.spark.sql.catalyst.plans.logical.LogicalPlan.transformDownWithPruning(LogicalPlan.scala:30)
at org.apache.spark.sql.catalyst.trees.TreeNode.transformDown(TreeNode.scala:457)
at org.apache.spark.sql.execution.QueryExecution.eagerlyExecuteCommands(QueryExecution.scala:93)
at org.apache.spark.sql.execution.QueryExecution.commandExecuted$lzycompute(QueryExecution.scala:80)
at org.apache.spark.sql.execution.QueryExecution.commandExecuted(QueryExecution.scala:78)
at org.apache.spark.sql.Dataset.<init>(Dataset.scala:219)
at org.apache.spark.sql.Dataset$.$anonfun$ofRows$2(Dataset.scala:99)
at org.apache.spark.sql.SparkSession.withActive(SparkSession.scala:775)
at org.apache.spark.sql.Dataset$.ofRows(Dataset.scala:96)
at org.apache.spark.sql.SparkSession.$anonfun$sql$1(SparkSession.scala:618)
at org.apache.spark.sql.SparkSession.withActive(SparkSession.scala:775)
at org.apache.spark.sql.SparkSession.sql(SparkSession.scala:613)
at org.apache.spark.sql.SQLContext.sql(SQLContext.scala:651)
at org.apache.spark.sql.hive.thriftserver.SparkSQLDriver.run(SparkSQLDriver.scala:67)
at org.apache.spark.sql.hive.thriftserver.SparkSQLCLIDriver.processCmd(SparkSQLCLIDriver.scala:384)
at org.apache.spark.sql.hive.thriftserver.SparkSQLCLIDriver.$anonfun$processLine$1(SparkSQLCLIDriver.scala:504)
at org.apache.spark.sql.hive.thriftserver.SparkSQLCLIDriver.$anonfun$processLine$1$adapted(SparkSQLCLIDriver.scala:498)
at scala.collection.Iterator.foreach(Iterator.scala:943)
at scala.collection.Iterator.foreach$(Iterator.scala:943)
at scala.collection.AbstractIterator.foreach(Iterator.scala:1431)
at scala.collection.IterableLike.foreach(IterableLike.scala:74)
at scala.collection.IterableLike.foreach$(IterableLike.scala:73)
at scala.collection.AbstractIterable.foreach(Iterable.scala:56)
at org.apache.spark.sql.hive.thriftserver.SparkSQLCLIDriver.processLine(SparkSQLCLIDriver.scala:498)
at org.apache.spark.sql.hive.thriftserver.SparkSQLCLIDriver$.main(SparkSQLCLIDriver.scala:287)
at org.apache.spark.sql.hive.thriftserver.SparkSQLCLIDriver.main(SparkSQLCLIDriver.scala)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.apache.spark.deploy.JavaMainApplication.start(SparkApplication.scala:52)
at org.apache.spark.deploy.SparkSubmit.org$apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:955)
at org.apache.spark.deploy.SparkSubmit.doRunMain$1(SparkSubmit.scala:180)
at org.apache.spark.deploy.SparkSubmit.submit(SparkSubmit.scala:203)
at org.apache.spark.deploy.SparkSubmit.doSubmit(SparkSubmit.scala:90)
at org.apache.spark.deploy.SparkSubmit$$anon$2.doSubmit(SparkSubmit.scala:1043)
at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:1052)
at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)
Caused by: org.apache.hudi.exception.HoodieException: Got runtime exception when hive syncing hudi_mor_tbl
at org.apache.hudi.hive.HiveSyncTool.syncHoodieTable(HiveSyncTool.java:143)
at org.apache.hudi.sync.common.util.SyncUtilHelpers.runHoodieMetaSync(SyncUtilHelpers.java:59)
... 93 more
Caused by: org.apache.hudi.hive.HoodieHiveSyncException: Failed to sync partitions for table hudi_mor_tbl_ro
at org.apache.hudi.hive.HiveSyncTool.syncPartitions(HiveSyncTool.java:414)
at org.apache.hudi.hive.HiveSyncTool.syncHoodieTable(HiveSyncTool.java:232)
at org.apache.hudi.hive.HiveSyncTool.doSync(HiveSyncTool.java:156)
at org.apache.hudi.hive.HiveSyncTool.syncHoodieTable(HiveSyncTool.java:140)
... 94 more
Caused by: java.lang.IllegalArgumentException: **Partition path is not in the form partition_key=partition_value.**
at org.apache.hudi.hive.HiveStylePartitionValueExtractor.extractPartitionValuesInPath(HiveStylePartitionValueExtractor.java:37)
at org.apache.hudi.hive.AbstractHiveSyncHoodieClient.getPartitionEvents(AbstractHiveSyncHoodieClient.java:81)
at org.apache.hudi.hive.HiveSyncTool.syncPartitions(HiveSyncTool.java:392)
... 97 more
org.apache.hudi.exception.HoodieException: Could not sync using the meta sync class org.apache.hudi.hive.HiveSyncTool
at org.apache.hudi.sync.common.util.SyncUtilHelpers.runHoodieMetaSync(SyncUtilHelpers.java:61)
at org.apache.hudi.HoodieSparkSqlWriter$.$anonfun$metaSync$2(HoodieSparkSqlWriter.scala:623)
at org.apache.hudi.HoodieSparkSqlWriter$.$anonfun$metaSync$2$adapted(HoodieSparkSqlWriter.scala:622)
at scala.collection.mutable.HashSet.foreach(HashSet.scala:79)
at org.apache.hudi.HoodieSparkSqlWriter$.metaSync(HoodieSparkSqlWriter.scala:622)
at org.apache.hudi.HoodieSparkSqlWriter$.commitAndPerformPostOperations(HoodieSparkSqlWriter.scala:681)
at org.apache.hudi.HoodieSparkSqlWriter$.write(HoodieSparkSqlWriter.scala:315)
at org.apache.hudi.DefaultSource.createRelation(DefaultSource.scala:171)
at org.apache.spark.sql.execution.datasources.SaveIntoDataSourceCommand.run(SaveIntoDataSourceCommand.scala:47)
at org.apache.spark.sql.execution.command.ExecutedCommandExec.sideEffectResult$lzycompute(commands.scala:75)
at org.apache.spark.sql.execution.command.ExecutedCommandExec.sideEffectResult(commands.scala:73)
at org.apache.spark.sql.execution.command.ExecutedCommandExec.executeCollect(commands.scala:84)
at org.apache.spark.sql.execution.QueryExecution$$anonfun$eagerlyExecuteCommands$1.$anonfun$applyOrElse$1(QueryExecution.scala:97)
at org.apache.spark.sql.execution.SQLExecution$.$anonfun$withNewExecutionId$5(SQLExecution.scala:103)
at org.apache.spark.sql.execution.SQLExecution$.withSQLConfPropagated(SQLExecution.scala:163)
at org.apache.spark.sql.execution.SQLExecution$.$anonfun$withNewExecutionId$1(SQLExecution.scala:90)
at org.apache.spark.sql.SparkSession.withActive(SparkSession.scala:775)
at org.apache.spark.sql.execution.SQLExecution$.withNewExecutionId(SQLExecution.scala:64)
at org.apache.spark.sql.execution.QueryExecution$$anonfun$eagerlyExecuteCommands$1.applyOrElse(QueryExecution.scala:97)
at org.apache.spark.sql.execution.QueryExecution$$anonfun$eagerlyExecuteCommands$1.applyOrElse(QueryExecution.scala:93)
at org.apache.spark.sql.catalyst.trees.TreeNode.$anonfun$transformDownWithPruning$1(TreeNode.scala:481)
at org.apache.spark.sql.catalyst.trees.CurrentOrigin$.withOrigin(TreeNode.scala:82)
at org.apache.spark.sql.catalyst.trees.TreeNode.transformDownWithPruning(TreeNode.scala:481)
at org.apache.spark.sql.catalyst.plans.logical.LogicalPlan.org$apache$spark$sql$catalyst$plans$logical$AnalysisHelper$$super$transformDownWithPruning(LogicalPlan.scala:30)
at org.apache.spark.sql.catalyst.plans.logical.AnalysisHelper.transformDownWithPruning(AnalysisHelper.scala:267)
at org.apache.spark.sql.catalyst.plans.logical.AnalysisHelper.transformDownWithPruning$(AnalysisHelper.scala:263)
at org.apache.spark.sql.catalyst.plans.logical.LogicalPlan.transformDownWithPruning(LogicalPlan.scala:30)
at org.apache.spark.sql.catalyst.plans.logical.LogicalPlan.transformDownWithPruning(LogicalPlan.scala:30)
at org.apache.spark.sql.catalyst.trees.TreeNode.transformDown(TreeNode.scala:457)
at org.apache.spark.sql.execution.QueryExecution.eagerlyExecuteCommands(QueryExecution.scala:93)
at org.apache.spark.sql.execution.QueryExecution.commandExecuted$lzycompute(QueryExecution.scala:80)
at org.apache.spark.sql.execution.QueryExecution.commandExecuted(QueryExecution.scala:78)
at org.apache.spark.sql.execution.QueryExecution.assertCommandExecuted(QueryExecution.scala:115)
at org.apache.spark.sql.DataFrameWriter.runCommand(DataFrameWriter.scala:848)
at org.apache.spark.sql.DataFrameWriter.saveToV1Source(DataFrameWriter.scala:382)
at org.apache.spark.sql.DataFrameWriter.saveInternal(DataFrameWriter.scala:355)
at org.apache.spark.sql.DataFrameWriter.save(DataFrameWriter.scala:247)
at org.apache.spark.sql.hudi.command.DeleteHoodieTableCommand.run(DeleteHoodieTableCommand.scala:48)
at org.apache.spark.sql.execution.command.ExecutedCommandExec.sideEffectResult$lzycompute(commands.scala:75)
at org.apache.spark.sql.execution.command.ExecutedCommandExec.sideEffectResult(commands.scala:73)
at org.apache.spark.sql.execution.command.ExecutedCommandExec.executeCollect(commands.scala:84)
at org.apache.spark.sql.execution.QueryExecution$$anonfun$eagerlyExecuteCommands$1.$anonfun$applyOrElse$1(QueryExecution.scala:97)
at org.apache.spark.sql.execution.SQLExecution$.$anonfun$withNewExecutionId$5(SQLExecution.scala:103)
at org.apache.spark.sql.execution.SQLExecution$.withSQLConfPropagated(SQLExecution.scala:163)
at org.apache.spark.sql.execution.SQLExecution$.$anonfun$withNewExecutionId$1(SQLExecution.scala:90)
at org.apache.spark.sql.SparkSession.withActive(SparkSession.scala:775)
at org.apache.spark.sql.execution.SQLExecution$.withNewExecutionId(SQLExecution.scala:64)
at org.apache.spark.sql.execution.QueryExecution$$anonfun$eagerlyExecuteCommands$1.applyOrElse(QueryExecution.scala:97)
at org.apache.spark.sql.execution.QueryExecution$$anonfun$eagerlyExecuteCommands$1.applyOrElse(QueryExecution.scala:93)
at org.apache.spark.sql.catalyst.trees.TreeNode.$anonfun$transformDownWithPruning$1(TreeNode.scala:481)
at org.apache.spark.sql.catalyst.trees.CurrentOrigin$.withOrigin(TreeNode.scala:82)
at org.apache.spark.sql.catalyst.trees.TreeNode.transformDownWithPruning(TreeNode.scala:481)
at org.apache.spark.sql.catalyst.plans.logical.LogicalPlan.org$apache$spark$sql$catalyst$plans$logical$AnalysisHelper$$super$transformDownWithPruning(LogicalPlan.scala:30)
at org.apache.spark.sql.catalyst.plans.logical.AnalysisHelper.transformDownWithPruning(AnalysisHelper.scala:267)
at org.apache.spark.sql.catalyst.plans.logical.AnalysisHelper.transformDownWithPruning$(AnalysisHelper.scala:263)
at org.apache.spark.sql.catalyst.plans.logical.LogicalPlan.transformDownWithPruning(LogicalPlan.scala:30)
at org.apache.spark.sql.catalyst.plans.logical.LogicalPlan.transformDownWithPruning(LogicalPlan.scala:30)
at org.apache.spark.sql.catalyst.trees.TreeNode.transformDown(TreeNode.scala:457)
at org.apache.spark.sql.execution.QueryExecution.eagerlyExecuteCommands(QueryExecution.scala:93)
at org.apache.spark.sql.execution.QueryExecution.commandExecuted$lzycompute(QueryExecution.scala:80)
at org.apache.spark.sql.execution.QueryExecution.commandExecuted(QueryExecution.scala:78)
at org.apache.spark.sql.Dataset.<init>(Dataset.scala:219)
at org.apache.spark.sql.Dataset$.$anonfun$ofRows$2(Dataset.scala:99)
at org.apache.spark.sql.SparkSession.withActive(SparkSession.scala:775)
at org.apache.spark.sql.Dataset$.ofRows(Dataset.scala:96)
at org.apache.spark.sql.SparkSession.$anonfun$sql$1(SparkSession.scala:618)
at org.apache.spark.sql.SparkSession.withActive(SparkSession.scala:775)
at org.apache.spark.sql.SparkSession.sql(SparkSession.scala:613)
at org.apache.spark.sql.SQLContext.sql(SQLContext.scala:651)
at org.apache.spark.sql.hive.thriftserver.SparkSQLDriver.run(SparkSQLDriver.scala:67)
at org.apache.spark.sql.hive.thriftserver.SparkSQLCLIDriver.processCmd(SparkSQLCLIDriver.scala:384)
at org.apache.spark.sql.hive.thriftserver.SparkSQLCLIDriver.$anonfun$processLine$1(SparkSQLCLIDriver.scala:504)
at org.apache.spark.sql.hive.thriftserver.SparkSQLCLIDriver.$anonfun$processLine$1$adapted(SparkSQLCLIDriver.scala:498)
at scala.collection.Iterator.foreach(Iterator.scala:943)
at scala.collection.Iterator.foreach$(Iterator.scala:943)
at scala.collection.AbstractIterator.foreach(Iterator.scala:1431)
at scala.collection.IterableLike.foreach(IterableLike.scala:74)
at scala.collection.IterableLike.foreach$(IterableLike.scala:73)
at scala.collection.AbstractIterable.foreach(Iterable.scala:56)
at org.apache.spark.sql.hive.thriftserver.SparkSQLCLIDriver.processLine(SparkSQLCLIDriver.scala:498)
at org.apache.spark.sql.hive.thriftserver.SparkSQLCLIDriver$.main(SparkSQLCLIDriver.scala:287)
at org.apache.spark.sql.hive.thriftserver.SparkSQLCLIDriver.main(SparkSQLCLIDriver.scala)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.apache.spark.deploy.JavaMainApplication.start(SparkApplication.scala:52)
at org.apache.spark.deploy.SparkSubmit.org$apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:955)
at org.apache.spark.deploy.SparkSubmit.doRunMain$1(SparkSubmit.scala:180)
at org.apache.spark.deploy.SparkSubmit.submit(SparkSubmit.scala:203)
at org.apache.spark.deploy.SparkSubmit.doSubmit(SparkSubmit.scala:90)
at org.apache.spark.deploy.SparkSubmit$$anon$2.doSubmit(SparkSubmit.scala:1043)
at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:1052)
at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)
Caused by: org.apache.hudi.exception.HoodieException: Got runtime exception when hive syncing hudi_mor_tbl
at org.apache.hudi.hive.HiveSyncTool.syncHoodieTable(HiveSyncTool.java:143)
at org.apache.hudi.sync.common.util.SyncUtilHelpers.runHoodieMetaSync(SyncUtilHelpers.java:59)
... 93 more
Caused by: org.apache.hudi.hive.HoodieHiveSyncException: Failed to sync partitions for table hudi_mor_tbl_ro
at org.apache.hudi.hive.HiveSyncTool.syncPartitions(HiveSyncTool.java:414)
at org.apache.hudi.hive.HiveSyncTool.syncHoodieTable(HiveSyncTool.java:232)
at org.apache.hudi.hive.HiveSyncTool.doSync(HiveSyncTool.java:156)
at org.apache.hudi.hive.HiveSyncTool.syncHoodieTable(HiveSyncTool.java:140)
... 94 more
Caused by: java.lang.IllegalArgumentException: Partition path is not in the form partition_key=partition_value.
at org.apache.hudi.hive.HiveStylePartitionValueExtractor.extractPartitionValuesInPath(HiveStylePartitionValueExtractor.java:37)
at org.apache.hudi.hive.AbstractHiveSyncHoodieClient.getPartitionEvents(AbstractHiveSyncHoodieClient.java:81)
at org.apache.hudi.hive.HiveSyncTool.syncPartitions(HiveSyncTool.java:392)
... 97 more
spark-sql>
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] bigdata-spec commented on issue #8403: [SUPPORT]Caused by: java.lang.IllegalArgumentException: Partition path is not in the form partition_key=partition_value.
Posted by "bigdata-spec (via GitHub)" <gi...@apache.org>.
bigdata-spec commented on issue #8403:
URL: https://github.com/apache/hudi/issues/8403#issuecomment-1501462828
> The newer hudi should not need this conf, the class can be inferred automatically.
My Hudi version is 0.11.1
I used Spark2.4 before，hoodie.datasource.hive_sync.partition_extractor_class it not need to use.
But Spark3.2 it need` hoodie.datasource.hive_sync.partition_extractor_class = org.apache.hudi.hive.NonPartitionedExtractor`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] xicm commented on issue #8403: [SUPPORT]Caused by: java.lang.IllegalArgumentException: Partition path is not in the form partition_key=partition_value.
Posted by "xicm (via GitHub)" <gi...@apache.org>.
xicm commented on issue #8403:
URL: https://github.com/apache/hudi/issues/8403#issuecomment-1501438099
The newer hudi should not need this conf, the class can be inferred automatically.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] bigdata-spec commented on issue #8403: [SUPPORT]Caused by: java.lang.IllegalArgumentException: Partition path is not in the form partition_key=partition_value.
Posted by "bigdata-spec (via GitHub)" <gi...@apache.org>.
bigdata-spec commented on issue #8403:
URL: https://github.com/apache/hudi/issues/8403#issuecomment-1500083165
But if I create a partition hudi table ,delete from table where xxx is ok,
So, does mor table doen't support delete no partition table?
```
spark-sql> create table if not exists zone_hudi.hudi_spark_table0725_0111_mor
> (
> id string,
> brand_id int,
> brand_name string,
> vehicle_model_id int,
> vehicle_model_name string,
> etl_update_time string,
> dt string,
> hh string
> ) using hudi
> options (
> type = 'mor',
> primaryKey = 'brand_id,vehicle_model_id',
> preCombineField = 'etl_update_time',
> hoodie.cleaner.commits.retained = '2',
> hoodie.table.name='hudi_spark_table0725_0111_mor'
> )
> partitioned by (dt,hh);
64856 [main] WARN org.apache.hudi.common.config.DFSPropertiesConfiguration - Cannot find HUDI_CONF_DIR, please set it as the dir of hudi-defaults.conf
64877 [main] WARN org.apache.hudi.common.config.DFSPropertiesConfiguration - Properties file file:/etc/hudi/conf/hudi-defaults.conf not found. Ignoring to load props file
67599 [main] WARN org.apache.hadoop.hive.ql.session.SessionState - METASTORE_FILTER_HOOK will be ignored, since hive.security.authorization.manager is set to instance of HiveAuthorizerFactory.
Time taken: 3.291 seconds
spark-sql> insert into zone_hudi.
> partition (dt,hh)
> select 'id',
> 100,
> 'rand_name',
> 1000,
> 'ehicle_model_name',
> CAST(current_timestamp AS string) as etl_update_time,
> '20220712',
> '09'
> ;
Error in query: Table or view 'partition' not found in database 'zone_hudi'
spark-sql> insert into zone_hudi.hudi_spark_table0725_0111_mor
> partition (dt,hh)
> select 'id',
> 100,
> 'rand_name',
> 1000,
> 'ehicle_model_name',
> CAST(current_timestamp AS string) as etl_update_time,
> '20220712',
> '09'
> ;
90584 [main] WARN org.apache.hudi.metadata.HoodieBackedTableMetadata - Metadata table was not found at path hdfs://nameservice1/user/hive/warehouse/zone_hudi.db/hudi_spark_table0725_0111_mor/.hoodie/metadata
109989 [Executor task launch worker for task 0.0 in stage 6.0 (TID 6)] WARN org.apache.hadoop.metrics2.impl.MetricsConfig - Cannot locate configuration: tried hadoop-metrics2-hbase.properties,hadoop-metrics2.properties
00:40 WARN: Timeline-server-based markers are not supported for HDFS: base path hdfs://nameservice1/user/hive/warehouse/zone_hudi.db/hudi_spark_table0725_0111_mor. Falling back to direct markers.
00:44 WARN: Timeline-server-based markers are not supported for HDFS: base path hdfs://nameservice1/user/hive/warehouse/zone_hudi.db/hudi_spark_table0725_0111_mor. Falling back to direct markers.
00:59 WARN: Timeline-server-based markers are not supported for HDFS: base path hdfs://nameservice1/user/hive/warehouse/zone_hudi.db/hudi_spark_table0725_0111_mor. Falling back to direct markers.
156792 [main] WARN org.apache.hadoop.hive.conf.HiveConf - HiveConf of name hive.vectorized.use.checked.expressions does not exist
156792 [main] WARN org.apache.hadoop.hive.conf.HiveConf - HiveConf of name hive.strict.checks.no.partition.filter does not exist
156792 [main] WARN org.apache.hadoop.hive.conf.HiveConf - HiveConf of name hive.internal.ss.authz.settings.applied.marker does not exist
156792 [main] WARN org.apache.hadoop.hive.conf.HiveConf - HiveConf of name hive.stats.jdbc.timeout does not exist
156792 [main] WARN org.apache.hadoop.hive.conf.HiveConf - HiveConf of name hive.strict.checks.orderby.no.limit does not exist
156792 [main] WARN org.apache.hadoop.hive.conf.HiveConf - HiveConf of name hive.vectorized.input.format.excludes does not exist
156793 [main] WARN org.apache.hadoop.hive.conf.HiveConf - HiveConf of name hive.stats.retries.wait does not exist
Time taken: 71.445 seconds
spark-sql> insert into zone_hudi.hudi_spark_table0725_0111_mor partition (dt,hh)
> select 'id1',
> 100,
> 'rand_name',
> 1010,
> 'ehicle_model_name',
> CAST(current_timestamp AS string) as etl_update_time,
> '20220712',
> '09'
> ;
01:26 WARN: Timeline-server-based markers are not supported for HDFS: base path hdfs://nameservice1/user/hive/warehouse/zone_hudi.db/hudi_spark_table0725_0111_mor. Falling back to direct markers.
01:30 WARN: Timeline-server-based markers are not supported for HDFS: base path hdfs://nameservice1/user/hive/warehouse/zone_hudi.db/hudi_spark_table0725_0111_mor. Falling back to direct markers.
01:39 WARN: Timeline-server-based markers are not supported for HDFS: base path hdfs://nameservice1/user/hive/warehouse/zone_hudi.db/hudi_spark_table0725_0111_mor. Falling back to direct markers.
196295 [main] WARN org.apache.hadoop.hive.conf.HiveConf - HiveConf of name hive.vectorized.use.checked.expressions does not exist
196295 [main] WARN org.apache.hadoop.hive.conf.HiveConf - HiveConf of name hive.strict.checks.no.partition.filter does not exist
196295 [main] WARN org.apache.hadoop.hive.conf.HiveConf - HiveConf of name hive.internal.ss.authz.settings.applied.marker does not exist
196295 [main] WARN org.apache.hadoop.hive.conf.HiveConf - HiveConf of name hive.stats.jdbc.timeout does not exist
196295 [main] WARN org.apache.hadoop.hive.conf.HiveConf - HiveConf of name hive.strict.checks.orderby.no.limit does not exist
196295 [main] WARN org.apache.hadoop.hive.conf.HiveConf - HiveConf of name hive.vectorized.input.format.excludes does not exist
196295 [main] WARN org.apache.hadoop.hive.conf.HiveConf - HiveConf of name hive.stats.retries.wait does not exist
Time taken: 29.951 seconds
spark-sql> select * from zone_hudi.hudi_spark_table0725_0111_mor;
20230407163943734 20230407163943734_0_0 brand_id:100,vehicle_model_id:1000 dt=20220712/hh=09 65d6968a-1831-4d73-8cba-968ea6e2c55f-0_0-60-2432_20230407164100742.parquet id 100 rand_name 1000 ehicle_model_name 2023-04-07 16:39:43.438 20220712 09
20230407164100742 20230407164100742_0_1 brand_id:100,vehicle_model_id:1010 dt=20220712/hh=09 65d6968a-1831-4d73-8cba-968ea6e2c55f-0_0-60-2432_20230407164100742.parquet id1 100 rand_name 1010 ehicle_model_name 2023-04-07 16:41:00.702 20220712 09
Time taken: 0.661 seconds, Fetched 2 row(s)
spark-sql> delete from zone_hudi.hudi_spark_table0725_0111_mor where id='id';
02:35 WARN: Timeline-server-based markers are not supported for HDFS: base path hdfs://nameservice1/user/hive/warehouse/zone_hudi.db/hudi_spark_table0725_0111_mor. Falling back to direct markers.
02:39 WARN: Timeline-server-based markers are not supported for HDFS: base path hdfs://nameservice1/user/hive/warehouse/zone_hudi.db/hudi_spark_table0725_0111_mor. Falling back to direct markers.
02:50 WARN: Timeline-server-based markers are not supported for HDFS: base path hdfs://nameservice1/user/hive/warehouse/zone_hudi.db/hudi_spark_table0725_0111_mor. Falling back to direct markers.
267474 [main] WARN org.apache.hadoop.hive.conf.HiveConf - HiveConf of name hive.vectorized.use.checked.expressions does not exist
267474 [main] WARN org.apache.hadoop.hive.conf.HiveConf - HiveConf of name hive.strict.checks.no.partition.filter does not exist
267474 [main] WARN org.apache.hadoop.hive.conf.HiveConf - HiveConf of name hive.internal.ss.authz.settings.applied.marker does not exist
267474 [main] WARN org.apache.hadoop.hive.conf.HiveConf - HiveConf of name hive.stats.jdbc.timeout does not exist
267474 [main] WARN org.apache.hadoop.hive.conf.HiveConf - HiveConf of name hive.strict.checks.orderby.no.limit does not exist
267474 [main] WARN org.apache.hadoop.hive.conf.HiveConf - HiveConf of name hive.vectorized.input.format.excludes does not exist
267474 [main] WARN org.apache.hadoop.hive.conf.HiveConf - HiveConf of name hive.stats.retries.wait does not exist
Time taken: 29.054 seconds
spark-sql> select * from zone_hudi.hudi_spark_table0725_0111_mor;
20230407164100742 20230407164100742_0_1 brand_id:100,vehicle_model_id:1010 dt=20220712/hh=09 65d6968a-1831-4d73-8cba-968ea6e2c55f-0_0-60-2432_20230407164100742.parquet id1 100 rand_name 1010 ehicle_model_name 2023-04-07 16:41:00.702 20220712 09
Time taken: 1.104 seconds, Fetched 1 row(s)
spark-sql>
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] xicm commented on issue #8403: [SUPPORT]Caused by: java.lang.IllegalArgumentException: Partition path is not in the form partition_key=partition_value.
Posted by "xicm (via GitHub)" <gi...@apache.org>.
xicm commented on issue #8403:
URL: https://github.com/apache/hudi/issues/8403#issuecomment-1501287817
If the table is none partitioned table set `hoodie.datasource.hive_sync.partition_extractor_class = org.apache.hudi.hive.NonPartitionedExtractor`. The extractor class should be inferred automatically, do you set the class in conf?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] bigdata-spec commented on issue #8403: [SUPPORT]Caused by: java.lang.IllegalArgumentException: Partition path is not in the form partition_key=partition_value.
Posted by "bigdata-spec (via GitHub)" <gi...@apache.org>.
bigdata-spec commented on issue #8403:
URL: https://github.com/apache/hudi/issues/8403#issuecomment-1500082955
But if I create a partition hudi table ,delete from table where xxx is ok,
So, does mor table doen't support delete no partition table?
```
spark-sql> create table if not exists zone_hudi.hudi_spark_table0725_0111_mor
> (
> id string,
> brand_id int,
> brand_name string,
> vehicle_model_id int,
> vehicle_model_name string,
> etl_update_time string,
> dt string,
> hh string
> ) using hudi
> options (
> type = 'mor',
> primaryKey = 'brand_id,vehicle_model_id',
> preCombineField = 'etl_update_time',
> hoodie.cleaner.commits.retained = '2',
> hoodie.table.name='hudi_spark_table0725_0111_mor'
> )
> partitioned by (dt,hh);
64856 [main] WARN org.apache.hudi.common.config.DFSPropertiesConfiguration - Cannot find HUDI_CONF_DIR, please set it as the dir of hudi-defaults.conf
64877 [main] WARN org.apache.hudi.common.config.DFSPropertiesConfiguration - Properties file file:/etc/hudi/conf/hudi-defaults.conf not found. Ignoring to load props file
67599 [main] WARN org.apache.hadoop.hive.ql.session.SessionState - METASTORE_FILTER_HOOK will be ignored, since hive.security.authorization.manager is set to instance of HiveAuthorizerFactory.
Time taken: 3.291 seconds
spark-sql> insert into zone_hudi.
> partition (dt,hh)
> select 'id',
> 100,
> 'rand_name',
> 1000,
> 'ehicle_model_name',
> CAST(current_timestamp AS string) as etl_update_time,
> '20220712',
> '09'
> ;
Error in query: Table or view 'partition' not found in database 'zone_hudi'
spark-sql> insert into zone_hudi.hudi_spark_table0725_0111_mor
> partition (dt,hh)
> select 'id',
> 100,
> 'rand_name',
> 1000,
> 'ehicle_model_name',
> CAST(current_timestamp AS string) as etl_update_time,
> '20220712',
> '09'
> ;
90584 [main] WARN org.apache.hudi.metadata.HoodieBackedTableMetadata - Metadata table was not found at path hdfs://nameservice1/user/hive/warehouse/zone_hudi.db/hudi_spark_table0725_0111_mor/.hoodie/metadata
109989 [Executor task launch worker for task 0.0 in stage 6.0 (TID 6)] WARN org.apache.hadoop.metrics2.impl.MetricsConfig - Cannot locate configuration: tried hadoop-metrics2-hbase.properties,hadoop-metrics2.properties
00:40 WARN: Timeline-server-based markers are not supported for HDFS: base path hdfs://nameservice1/user/hive/warehouse/zone_hudi.db/hudi_spark_table0725_0111_mor. Falling back to direct markers.
00:44 WARN: Timeline-server-based markers are not supported for HDFS: base path hdfs://nameservice1/user/hive/warehouse/zone_hudi.db/hudi_spark_table0725_0111_mor. Falling back to direct markers.
00:59 WARN: Timeline-server-based markers are not supported for HDFS: base path hdfs://nameservice1/user/hive/warehouse/zone_hudi.db/hudi_spark_table0725_0111_mor. Falling back to direct markers.
156792 [main] WARN org.apache.hadoop.hive.conf.HiveConf - HiveConf of name hive.vectorized.use.checked.expressions does not exist
156792 [main] WARN org.apache.hadoop.hive.conf.HiveConf - HiveConf of name hive.strict.checks.no.partition.filter does not exist
156792 [main] WARN org.apache.hadoop.hive.conf.HiveConf - HiveConf of name hive.internal.ss.authz.settings.applied.marker does not exist
156792 [main] WARN org.apache.hadoop.hive.conf.HiveConf - HiveConf of name hive.stats.jdbc.timeout does not exist
156792 [main] WARN org.apache.hadoop.hive.conf.HiveConf - HiveConf of name hive.strict.checks.orderby.no.limit does not exist
156792 [main] WARN org.apache.hadoop.hive.conf.HiveConf - HiveConf of name hive.vectorized.input.format.excludes does not exist
156793 [main] WARN org.apache.hadoop.hive.conf.HiveConf - HiveConf of name hive.stats.retries.wait does not exist
Time taken: 71.445 seconds
spark-sql> insert into zone_hudi.hudi_spark_table0725_0111_mor partition (dt,hh)
> select 'id1',
> 100,
> 'rand_name',
> 1010,
> 'ehicle_model_name',
> CAST(current_timestamp AS string) as etl_update_time,
> '20220712',
> '09'
> ;
01:26 WARN: Timeline-server-based markers are not supported for HDFS: base path hdfs://nameservice1/user/hive/warehouse/zone_hudi.db/hudi_spark_table0725_0111_mor. Falling back to direct markers.
01:30 WARN: Timeline-server-based markers are not supported for HDFS: base path hdfs://nameservice1/user/hive/warehouse/zone_hudi.db/hudi_spark_table0725_0111_mor. Falling back to direct markers.
01:39 WARN: Timeline-server-based markers are not supported for HDFS: base path hdfs://nameservice1/user/hive/warehouse/zone_hudi.db/hudi_spark_table0725_0111_mor. Falling back to direct markers.
196295 [main] WARN org.apache.hadoop.hive.conf.HiveConf - HiveConf of name hive.vectorized.use.checked.expressions does not exist
196295 [main] WARN org.apache.hadoop.hive.conf.HiveConf - HiveConf of name hive.strict.checks.no.partition.filter does not exist
196295 [main] WARN org.apache.hadoop.hive.conf.HiveConf - HiveConf of name hive.internal.ss.authz.settings.applied.marker does not exist
196295 [main] WARN org.apache.hadoop.hive.conf.HiveConf - HiveConf of name hive.stats.jdbc.timeout does not exist
196295 [main] WARN org.apache.hadoop.hive.conf.HiveConf - HiveConf of name hive.strict.checks.orderby.no.limit does not exist
196295 [main] WARN org.apache.hadoop.hive.conf.HiveConf - HiveConf of name hive.vectorized.input.format.excludes does not exist
196295 [main] WARN org.apache.hadoop.hive.conf.HiveConf - HiveConf of name hive.stats.retries.wait does not exist
Time taken: 29.951 seconds
spark-sql> select * from zone_hudi.hudi_spark_table0725_0111_mor;
20230407163943734 20230407163943734_0_0 brand_id:100,vehicle_model_id:1000 dt=20220712/hh=09 65d6968a-1831-4d73-8cba-968ea6e2c55f-0_0-60-2432_20230407164100742.parquet id 100 rand_name 1000 ehicle_model_name 2023-04-07 16:39:43.438 20220712 09
20230407164100742 20230407164100742_0_1 brand_id:100,vehicle_model_id:1010 dt=20220712/hh=09 65d6968a-1831-4d73-8cba-968ea6e2c55f-0_0-60-2432_20230407164100742.parquet id1 100 rand_name 1010 ehicle_model_name 2023-04-07 16:41:00.702 20220712 09
Time taken: 0.661 seconds, Fetched 2 row(s)
spark-sql> delete from zone_hudi.hudi_spark_table0725_0111_mor where id='id';
02:35 WARN: Timeline-server-based markers are not supported for HDFS: base path hdfs://nameservice1/user/hive/warehouse/zone_hudi.db/hudi_spark_table0725_0111_mor. Falling back to direct markers.
02:39 WARN: Timeline-server-based markers are not supported for HDFS: base path hdfs://nameservice1/user/hive/warehouse/zone_hudi.db/hudi_spark_table0725_0111_mor. Falling back to direct markers.
02:50 WARN: Timeline-server-based markers are not supported for HDFS: base path hdfs://nameservice1/user/hive/warehouse/zone_hudi.db/hudi_spark_table0725_0111_mor. Falling back to direct markers.
267474 [main] WARN org.apache.hadoop.hive.conf.HiveConf - HiveConf of name hive.vectorized.use.checked.expressions does not exist
267474 [main] WARN org.apache.hadoop.hive.conf.HiveConf - HiveConf of name hive.strict.checks.no.partition.filter does not exist
267474 [main] WARN org.apache.hadoop.hive.conf.HiveConf - HiveConf of name hive.internal.ss.authz.settings.applied.marker does not exist
267474 [main] WARN org.apache.hadoop.hive.conf.HiveConf - HiveConf of name hive.stats.jdbc.timeout does not exist
267474 [main] WARN org.apache.hadoop.hive.conf.HiveConf - HiveConf of name hive.strict.checks.orderby.no.limit does not exist
267474 [main] WARN org.apache.hadoop.hive.conf.HiveConf - HiveConf of name hive.vectorized.input.format.excludes does not exist
267474 [main] WARN org.apache.hadoop.hive.conf.HiveConf - HiveConf of name hive.stats.retries.wait does not exist
Time taken: 29.054 seconds
spark-sql> select * from zone_hudi.hudi_spark_table0725_0111_mor;
20230407164100742 20230407164100742_0_1 brand_id:100,vehicle_model_id:1010 dt=20220712/hh=09 65d6968a-1831-4d73-8cba-968ea6e2c55f-0_0-60-2432_20230407164100742.parquet id1 100 rand_name 1010 ehicle_model_name 2023-04-07 16:41:00.702 20220712 09
Time taken: 1.104 seconds, Fetched 1 row(s)
spark-sql>
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] ad1happy2go commented on issue #8403: [SUPPORT]Caused by: java.lang.IllegalArgumentException: Partition path is not in the form partition_key=partition_value.
Posted by "ad1happy2go (via GitHub)" <gi...@apache.org>.
ad1happy2go commented on issue #8403:
URL: https://github.com/apache/hudi/issues/8403#issuecomment-1569645236
@bigdata-spec I confirmed with spark 3.2.3 and hudi 0.12.1 and also with spark 3.2.3 and hudi 0.11.1. this is working fine without specifying partitioner class.
<img width="1194" alt="image" src="https://github.com/apache/hudi/assets/63430370/bee5604e-0645-46e7-a791-54ca3b48f61a">
Is this still a blocker for you.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] bigdata-spec commented on issue #8403: [SUPPORT]Caused by: java.lang.IllegalArgumentException: Partition path is not in the form partition_key=partition_value.
Posted by "bigdata-spec (via GitHub)" <gi...@apache.org>.
bigdata-spec commented on issue #8403:
URL: https://github.com/apache/hudi/issues/8403#issuecomment-1501408150
> If the table is none partitioned table set `hoodie.datasource.hive_sync.partition_extractor_class = org.apache.hudi.hive.NonPartitionedExtractor`. The extractor class should be inferred automatically, do you set the class in conf?
@xicm Thank you very much,
I used spark2.4 before，hoodie.datasource.hive_sync.partition_extractor_class it not need to use.
Does spark3 need this conf?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org