You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@hudi.apache.org by "bigdata-spec (via GitHub)" <gi...@apache.org> on 2023/04/07 08:49:34 UTC
[GitHub] [hudi] bigdata-spec commented on issue #8403: [SUPPORT]Caused by: java.lang.IllegalArgumentException: Partition path is not in the form partition_key=partition_value.
bigdata-spec commented on issue #8403:
URL: https://github.com/apache/hudi/issues/8403#issuecomment-1500082955
But if I create a partition hudi table ,delete from table where xxx is ok,
So, does mor table doen't support delete no partition table?
```
spark-sql> create table if not exists zone_hudi.hudi_spark_table0725_0111_mor
> (
> id string,
> brand_id int,
> brand_name string,
> vehicle_model_id int,
> vehicle_model_name string,
> etl_update_time string,
> dt string,
> hh string
> ) using hudi
> options (
> type = 'mor',
> primaryKey = 'brand_id,vehicle_model_id',
> preCombineField = 'etl_update_time',
> hoodie.cleaner.commits.retained = '2',
> hoodie.table.name='hudi_spark_table0725_0111_mor'
> )
> partitioned by (dt,hh);
64856 [main] WARN org.apache.hudi.common.config.DFSPropertiesConfiguration - Cannot find HUDI_CONF_DIR, please set it as the dir of hudi-defaults.conf
64877 [main] WARN org.apache.hudi.common.config.DFSPropertiesConfiguration - Properties file file:/etc/hudi/conf/hudi-defaults.conf not found. Ignoring to load props file
67599 [main] WARN org.apache.hadoop.hive.ql.session.SessionState - METASTORE_FILTER_HOOK will be ignored, since hive.security.authorization.manager is set to instance of HiveAuthorizerFactory.
Time taken: 3.291 seconds
spark-sql> insert into zone_hudi.
> partition (dt,hh)
> select 'id',
> 100,
> 'rand_name',
> 1000,
> 'ehicle_model_name',
> CAST(current_timestamp AS string) as etl_update_time,
> '20220712',
> '09'
> ;
Error in query: Table or view 'partition' not found in database 'zone_hudi'
spark-sql> insert into zone_hudi.hudi_spark_table0725_0111_mor
> partition (dt,hh)
> select 'id',
> 100,
> 'rand_name',
> 1000,
> 'ehicle_model_name',
> CAST(current_timestamp AS string) as etl_update_time,
> '20220712',
> '09'
> ;
90584 [main] WARN org.apache.hudi.metadata.HoodieBackedTableMetadata - Metadata table was not found at path hdfs://nameservice1/user/hive/warehouse/zone_hudi.db/hudi_spark_table0725_0111_mor/.hoodie/metadata
109989 [Executor task launch worker for task 0.0 in stage 6.0 (TID 6)] WARN org.apache.hadoop.metrics2.impl.MetricsConfig - Cannot locate configuration: tried hadoop-metrics2-hbase.properties,hadoop-metrics2.properties
00:40 WARN: Timeline-server-based markers are not supported for HDFS: base path hdfs://nameservice1/user/hive/warehouse/zone_hudi.db/hudi_spark_table0725_0111_mor. Falling back to direct markers.
00:44 WARN: Timeline-server-based markers are not supported for HDFS: base path hdfs://nameservice1/user/hive/warehouse/zone_hudi.db/hudi_spark_table0725_0111_mor. Falling back to direct markers.
00:59 WARN: Timeline-server-based markers are not supported for HDFS: base path hdfs://nameservice1/user/hive/warehouse/zone_hudi.db/hudi_spark_table0725_0111_mor. Falling back to direct markers.
156792 [main] WARN org.apache.hadoop.hive.conf.HiveConf - HiveConf of name hive.vectorized.use.checked.expressions does not exist
156792 [main] WARN org.apache.hadoop.hive.conf.HiveConf - HiveConf of name hive.strict.checks.no.partition.filter does not exist
156792 [main] WARN org.apache.hadoop.hive.conf.HiveConf - HiveConf of name hive.internal.ss.authz.settings.applied.marker does not exist
156792 [main] WARN org.apache.hadoop.hive.conf.HiveConf - HiveConf of name hive.stats.jdbc.timeout does not exist
156792 [main] WARN org.apache.hadoop.hive.conf.HiveConf - HiveConf of name hive.strict.checks.orderby.no.limit does not exist
156792 [main] WARN org.apache.hadoop.hive.conf.HiveConf - HiveConf of name hive.vectorized.input.format.excludes does not exist
156793 [main] WARN org.apache.hadoop.hive.conf.HiveConf - HiveConf of name hive.stats.retries.wait does not exist
Time taken: 71.445 seconds
spark-sql> insert into zone_hudi.hudi_spark_table0725_0111_mor partition (dt,hh)
> select 'id1',
> 100,
> 'rand_name',
> 1010,
> 'ehicle_model_name',
> CAST(current_timestamp AS string) as etl_update_time,
> '20220712',
> '09'
> ;
01:26 WARN: Timeline-server-based markers are not supported for HDFS: base path hdfs://nameservice1/user/hive/warehouse/zone_hudi.db/hudi_spark_table0725_0111_mor. Falling back to direct markers.
01:30 WARN: Timeline-server-based markers are not supported for HDFS: base path hdfs://nameservice1/user/hive/warehouse/zone_hudi.db/hudi_spark_table0725_0111_mor. Falling back to direct markers.
01:39 WARN: Timeline-server-based markers are not supported for HDFS: base path hdfs://nameservice1/user/hive/warehouse/zone_hudi.db/hudi_spark_table0725_0111_mor. Falling back to direct markers.
196295 [main] WARN org.apache.hadoop.hive.conf.HiveConf - HiveConf of name hive.vectorized.use.checked.expressions does not exist
196295 [main] WARN org.apache.hadoop.hive.conf.HiveConf - HiveConf of name hive.strict.checks.no.partition.filter does not exist
196295 [main] WARN org.apache.hadoop.hive.conf.HiveConf - HiveConf of name hive.internal.ss.authz.settings.applied.marker does not exist
196295 [main] WARN org.apache.hadoop.hive.conf.HiveConf - HiveConf of name hive.stats.jdbc.timeout does not exist
196295 [main] WARN org.apache.hadoop.hive.conf.HiveConf - HiveConf of name hive.strict.checks.orderby.no.limit does not exist
196295 [main] WARN org.apache.hadoop.hive.conf.HiveConf - HiveConf of name hive.vectorized.input.format.excludes does not exist
196295 [main] WARN org.apache.hadoop.hive.conf.HiveConf - HiveConf of name hive.stats.retries.wait does not exist
Time taken: 29.951 seconds
spark-sql> select * from zone_hudi.hudi_spark_table0725_0111_mor;
20230407163943734 20230407163943734_0_0 brand_id:100,vehicle_model_id:1000 dt=20220712/hh=09 65d6968a-1831-4d73-8cba-968ea6e2c55f-0_0-60-2432_20230407164100742.parquet id 100 rand_name 1000 ehicle_model_name 2023-04-07 16:39:43.438 20220712 09
20230407164100742 20230407164100742_0_1 brand_id:100,vehicle_model_id:1010 dt=20220712/hh=09 65d6968a-1831-4d73-8cba-968ea6e2c55f-0_0-60-2432_20230407164100742.parquet id1 100 rand_name 1010 ehicle_model_name 2023-04-07 16:41:00.702 20220712 09
Time taken: 0.661 seconds, Fetched 2 row(s)
spark-sql> delete from zone_hudi.hudi_spark_table0725_0111_mor where id='id';
02:35 WARN: Timeline-server-based markers are not supported for HDFS: base path hdfs://nameservice1/user/hive/warehouse/zone_hudi.db/hudi_spark_table0725_0111_mor. Falling back to direct markers.
02:39 WARN: Timeline-server-based markers are not supported for HDFS: base path hdfs://nameservice1/user/hive/warehouse/zone_hudi.db/hudi_spark_table0725_0111_mor. Falling back to direct markers.
02:50 WARN: Timeline-server-based markers are not supported for HDFS: base path hdfs://nameservice1/user/hive/warehouse/zone_hudi.db/hudi_spark_table0725_0111_mor. Falling back to direct markers.
267474 [main] WARN org.apache.hadoop.hive.conf.HiveConf - HiveConf of name hive.vectorized.use.checked.expressions does not exist
267474 [main] WARN org.apache.hadoop.hive.conf.HiveConf - HiveConf of name hive.strict.checks.no.partition.filter does not exist
267474 [main] WARN org.apache.hadoop.hive.conf.HiveConf - HiveConf of name hive.internal.ss.authz.settings.applied.marker does not exist
267474 [main] WARN org.apache.hadoop.hive.conf.HiveConf - HiveConf of name hive.stats.jdbc.timeout does not exist
267474 [main] WARN org.apache.hadoop.hive.conf.HiveConf - HiveConf of name hive.strict.checks.orderby.no.limit does not exist
267474 [main] WARN org.apache.hadoop.hive.conf.HiveConf - HiveConf of name hive.vectorized.input.format.excludes does not exist
267474 [main] WARN org.apache.hadoop.hive.conf.HiveConf - HiveConf of name hive.stats.retries.wait does not exist
Time taken: 29.054 seconds
spark-sql> select * from zone_hudi.hudi_spark_table0725_0111_mor;
20230407164100742 20230407164100742_0_1 brand_id:100,vehicle_model_id:1010 dt=20220712/hh=09 65d6968a-1831-4d73-8cba-968ea6e2c55f-0_0-60-2432_20230407164100742.parquet id1 100 rand_name 1010 ehicle_model_name 2023-04-07 16:41:00.702 20220712 09
Time taken: 1.104 seconds, Fetched 1 row(s)
spark-sql>
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org