dujl opened a new issue, #5836:
URL: https://github.com/apache/hudi/issues/5836
Create a non-partition hudi table in Spark，it will store hoodie.table.partition.fields with an empty value in hiveMetastore.
This is unexpected behavior.
**To Reproduce**
Steps to reproduce the behavior:
1. Create a non-partition hudi table in Spark
```
create table hudi_mor_tbl (
id int,
name string,
price double,
ts bigint
) using hudi
tblproperties (
type = 'mor',
primaryKey = 'id',
preCombineField = 'ts'
）
```
2. insert data one row to it.
```
insert into hudi_mor_tbl select 1, 'a1', 20, 1000;
```
3. cat hoodie.properties in table's base path, it include a none
hoodie.table.partition.fields=
4. and in hive metastore, it will store the key hoodie.table.partition.fields with an empty value in table TABLE_PARAMS.
**Expected behavior**
this is no hoodie.table.partition.fields in hoodie.properties and in HiveMetastore
**Environment Description**
* Hudi version : 0.10.0
* Spark version : 3.2.1
* Hive version : 3.1.2
* Hadoop version : 3.3.1
* Storage (HDFS/S3/GCS..) : HDFS
* Running on Docker? (yes/no) : no
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
dujl commented on issue #5836: URL: https://github.com/apache/hudi/issues/5836#issuecomment-1229958784 i have raised a pr https://github.com/apache/hudi/pull/6525 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org For queries about this service, please contact Infrastructure at: users@infra.apache.org
dujl commented on issue #5836:
URL: https://github.com/apache/hudi/issues/5836#issuecomment-1153039929
when resolve hudi table from hivemetastore，hive table's partition keys are not empty ，it should be empty<br/><br/><br/><html>
<head>
<meta http-equiv="Content-Type" content="text/html; charset=UTF-8">
</head>
<body>
<div class="ntes-mailmaster-quote" style="padding-top: 1px; padding-bottom: 1px" >
<div style=" margin-top: 20px; margin-bottom: 12px; font-size: 14px; line-height: 1.25; color: #89898c; " >---- Replied Message ----</div>
<div style=" margin-bottom: 12px; font-size: 13px; line-height: 1.25; color: #2c2c2e; padding: 2px; border-radius: 8px; background-color: #f0f0f0; " >
<table width="100%" cellpadding="0" cellspacing="10" border="0">
<tr>
<td valign="top" style=" width: 4em; font-size: 13px; line-height: 1.25; color: #89898c; white-space: nowrap; " >From</td>
<td valign="top" style=" font-size: 13px; line-height: 1.25; color: #2c2c2e; word-break: break-all; " ><a class="mail-from" style="color: #1c83eb; text-decoration: none" ***@***.***" ***@***.***></a></td>
</tr>
<tr>
<td valign="top" style=" width: 4em; font-size: 13px; line-height: 1.25; color: #89898c; white-space: nowrap; " >Date</td>
<td class="mail-date" valign="top" style=" font-size: 13px; line-height: 1.25; color: #2c2c2e; word-break: break-all; " >06/12/2022 05:30</td>
</tr>
<tr style="">
<td valign="top" style=" width: 4em; font-size: 13px; line-height: 1.25; color: #89898c; " >To</td>
<td valign="top" style=" font-size: 13px; line-height: 1.25; color: #2c2c2e; word-break: break-all; " ><a class="mail-to" style="color: #1c83eb; text-decoration: none" ***@***.***" ***@***.***></a></td>
</tr>
<tr style="">
<td valign="top" style=" width: 4em; font-size: 13px; line-height: 1.25; color: #89898c; " >Cc</td>
<td valign="top" style=" font-size: 13px; line-height: 1.25; color: #2c2c2e; word-break: break-all; " ><a class="mail-cc" style="color: #1c83eb; text-decoration: none" ***@***.***" ***@***.***></a>、<a class="mail-cc" style="color: #1c83eb; text-decoration: none" ***@***.***" ***@***.***></a></td>
</tr>
<tr>
<td valign="top" style=" width: 4em; font-size: 13px; line-height: 1.25; color: #89898c; " >Subject</td>
<td class="mail-subject" valign="top" style=" font-size: 13px; line-height: 1.25; color: #2c2c2e; word-break: break-all; " >Re: [apache/hudi] [SUPPORT] key hoodie.table.partition.fields is stored in HiveMetaStore when create non-partition table in Spark (Issue #5836)</td>
</tr>
</table>
</div>
<div><p></p>
<p dir="auto"><a class="user-mention notranslate" data-hovercard-type="user" data-hovercard-url="/users/dujl/hovercard" data-octo-click="hovercard-link-click" data-octo-dimensions="link_type:self" ***@***.***</a> what side affects are you seeing with <code class="notranslate">hoodie.table.partition.fields</code> being set to an empty field in hoodie.properties.</p>
<p dir="auto">I also see the following extra configs in hoodie.properties that might need cleanup for non-partitioned tables.</p>
<p dir="auto">hoodie.datasource.write.hive_style_partitioning=true<br>
hoodie.datasource.write.drop.partition.columns=false<br>
hoodie.datasource.write.partitionpath.urlencode=false</p>
<p style="font-size:small;-webkit-text-size-adjust:none;color:#666;">—<br />Reply to this email directly, <a href="https://github.com/apache/hudi/issues/5836#issuecomment-1153001072">view it on GitHub</a>, or <a href="https://github.com/notifications/unsubscribe-auth/ABBN7FPXZYRFUIJK4UBVNQLVOUAQBANCNFSM5YNGPLZQ">unsubscribe</a>.<br />You are receiving this because you were mentioned.<img src="https://github.com/notifications/beacon/ABBN7FMSU3RYCKTDGP2XJD3VOUAQBA5CNFSM5YNGPLZ2YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOIS4WM4A.gif" height="1" width="1" alt="" /><span style="color: transparent; font-size: 0; display: none; visibility: hidden; overflow: hidden; opacity: 0; width: 0; height: 0; max-width: 0; max-height: 0; mso-hide: all">Message ID: <span><apache/hudi/issues/5836/1153001072</span><span>@</span><span>github</span><span>.</span><span>com></span></span></p>
<script type="application/ld+json">[
{
***@***.***": "http://schema.org",
***@***.***": "EmailMessage",
"potentialAction": {
***@***.***": "ViewAction",
"target": "https://github.com/apache/hudi/issues/5836#issuecomment-1153001072",
"url": "https://github.com/apache/hudi/issues/5836#issuecomment-1153001072",
"name": "View Issue"
},
"description": "View this Issue on GitHub",
"publisher": {
***@***.***": "Organization",
"name": "GitHub",
"url": "https://github.com"
}
}
]</script></div>
</div>
</body>
</html>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
nsivabalan commented on issue #5836: URL: https://github.com/apache/hudi/issues/5836#issuecomment-1289934444 https://github.com/apache/hudi/pull/6821 closing the issue as the fix is already landed. feel free to raise a new issue if you are looking for further assistance. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org For queries about this service, please contact Infrastructure at: users@infra.apache.org
nsivabalan commented on issue #5836: URL: https://github.com/apache/hudi/issues/5836#issuecomment-1229355315 @dujl : can you respond to Sagar's comment above. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org For queries about this service, please contact Infrastructure at: users@infra.apache.org
rkkalluri commented on issue #5836: URL: https://github.com/apache/hudi/issues/5836#issuecomment-1153001072 @dujl what side affects are you seeing with `hoodie.table.partition.fields` being set to an empty field in hoodie.properties. I also see the following extra configs in hoodie.properties that might need cleanup for non-partitioned tables. hoodie.datasource.write.hive_style_partitioning=true hoodie.datasource.write.drop.partition.columns=false hoodie.datasource.write.partitionpath.urlencode=false -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org For queries about this service, please contact Infrastructure at: users@infra.apache.org
codope commented on issue #5836:
URL: https://github.com/apache/hudi/issues/5836#issuecomment-1156348577
@dujl I don't see that config being present in hive metastore. I created the table and below is what my hoodie.properties and `show create table` looks like:
```
#Properties saved on 2022-06-15T10:53:55.329Z
#Wed Jun 15 10:53:55 UTC 2022
hoodie.table.precombine.field=ts
hoodie.datasource.write.drop.partition.columns=false
hoodie.table.partition.fields=
hoodie.table.type=MERGE_ON_READ
hoodie.archivelog.folder=archived
hoodie.compaction.payload.class=org.apache.hudi.common.model.DefaultHoodieRecordPayload
hoodie.timeline.layout.version=1
hoodie.table.version=4
hoodie.table.recordkey.fields=id
hoodie.database.name=default
hoodie.datasource.write.partitionpath.urlencode=false
hoodie.table.name=hudi_mor_tbl
hoodie.table.keygenerator.class=org.apache.hudi.keygen.ComplexKeyGenerator
hoodie.datasource.write.hive_style_partitioning=true
hoodie.table.create.schema={"type"\:"record","name"\:"hudi_mor_tbl_record","namespace"\:"hoodie.hudi_mor_tbl","fields"\:[{"name"\:"_hoodie_commit_time","type"\:["string","null"]},{"name"\:"_hoodie_commit_seqno","type"\:["string","null"]},{"name"\:"_hoodie_record_key","type"\:["string","null"]},{"name"\:"_hoodie_partition_path","type"\:["string","null"]},{"name"\:"_hoodie_file_name","type"\:["string","null"]},{"name"\:"id","type"\:["int","null"]},{"name"\:"name","type"\:["string","null"]},{"name"\:"price","type"\:["double","null"]},{"name"\:"ts","type"\:["long","null"]}]}
hoodie.table.checksum=2013841463
```
output of `show create table`
```
show create table hudi_mor_tbl;
+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+--+
| createtab_stmt |
+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+--+
| CREATE TABLE `hudi_mor_tbl`( |
| `_hoodie_commit_time` string, |
| `_hoodie_commit_seqno` string, |
| `_hoodie_record_key` string, |
| `_hoodie_partition_path` string, |
| `_hoodie_file_name` string, |
| `id` int, |
| `name` string, |
| `price` double, |
| `ts` bigint) |
| ROW FORMAT SERDE |
| 'org.apache.hadoop.hive.ql.io.parquet.serde.ParquetHiveSerDe' |
| WITH SERDEPROPERTIES ( |
| 'path'='hdfs://namenode:8020/user/hive/warehouse/hudi_mor_tbl') |
| STORED AS INPUTFORMAT |
| 'org.apache.hudi.hadoop.realtime.HoodieParquetRealtimeInputFormat' |
| OUTPUTFORMAT |
| 'org.apache.hadoop.hive.ql.io.parquet.MapredParquetOutputFormat' |
| LOCATION |
| 'hdfs://namenode:8020/user/hive/warehouse/hudi_mor_tbl' |
| TBLPROPERTIES ( |
| 'preCombineField'='ts', |
| 'primaryKey'='id', |
| 'spark.sql.create.version'='2.4.4', |
| 'spark.sql.sources.provider'='hudi', |
| 'spark.sql.sources.schema.numParts'='1', |
| 'spark.sql.sources.schema.part.0'='{"type":"struct","fields":[{"name":"_hoodie_commit_time","type":"string","nullable":true,"metadata":{}},{"name":"_hoodie_commit_seqno","type":"string","nullable":true,"metadata":{}},{"name":"_hoodie_record_key","type":"string","nullable":true,"metadata":{}},{"name":"_hoodie_partition_path","type":"string","nullable":true,"metadata":{}},{"name":"_hoodie_file_name","type":"string","nullable":true,"metadata":{}},{"name":"id","type":"integer","nullable":true,"metadata":{}},{"name":"name","type":"string","nullable":true,"metadata":{}},{"name":"price","type":"double","nullable":true,"metadata":{}},{"name":"ts","type":"long","nullable":true,"metadata":{}}]}', |
| 'transient_lastDdlTime'='1655290436', |
| 'type'='mor') |
+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+--+
29 rows selected (0.368 seconds)
0: jdbc:hive2://hiveserver:10000> show create table hudi_mor_tbl;
+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+--+
| createtab_stmt |
+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+--+
| CREATE TABLE `hudi_mor_tbl`( |
| `_hoodie_commit_time` string, |
| `_hoodie_commit_seqno` string, |
| `_hoodie_record_key` string, |
| `_hoodie_partition_path` string, |
| `_hoodie_file_name` string, |
| `id` int, |
| `name` string, |
| `price` double, |
| `ts` bigint) |
| ROW FORMAT SERDE |
| 'org.apache.hadoop.hive.ql.io.parquet.serde.ParquetHiveSerDe' |
| WITH SERDEPROPERTIES ( |
| 'path'='hdfs://namenode:8020/user/hive/warehouse/hudi_mor_tbl') |
| STORED AS INPUTFORMAT |
| 'org.apache.hudi.hadoop.realtime.HoodieParquetRealtimeInputFormat' |
| OUTPUTFORMAT |
| 'org.apache.hadoop.hive.ql.io.parquet.MapredParquetOutputFormat' |
| LOCATION |
| 'hdfs://namenode:8020/user/hive/warehouse/hudi_mor_tbl' |
| TBLPROPERTIES ( |
| 'preCombineField'='ts', |
| 'primaryKey'='id', |
| 'spark.sql.create.version'='2.4.4', |
| 'spark.sql.sources.provider'='hudi', |
| 'spark.sql.sources.schema.numParts'='1', |
| 'spark.sql.sources.schema.part.0'='{"type":"struct","fields":[{"name":"_hoodie_commit_time","type":"string","nullable":true,"metadata":{}},{"name":"_hoodie_commit_seqno","type":"string","nullable":true,"metadata":{}},{"name":"_hoodie_record_key","type":"string","nullable":true,"metadata":{}},{"name":"_hoodie_partition_path","type":"string","nullable":true,"metadata":{}},{"name":"_hoodie_file_name","type":"string","nullable":true,"metadata":{}},{"name":"id","type":"integer","nullable":true,"metadata":{}},{"name":"name","type":"string","nullable":true,"metadata":{}},{"name":"price","type":"double","nullable":true,"metadata":{}},{"name":"ts","type":"long","nullable":true,"metadata":{}}]}', |
| 'transient_lastDdlTime'='1655290436', |
| 'type'='mor')```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
dujl commented on issue #5836: URL: https://github.com/apache/hudi/issues/5836#issuecomment-1229872984 @codope in your hoodie.properties there is a ``` hoodie.table.partition.fields= ``` and you can query table_params in hive metastore mysql， you will see there is a table parameters like ``` | 50 | spark.sql.sources.schema.partCol.0 | ``` show create table will filter the empty key when display the result。 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org For queries about this service, please contact Infrastructure at: users@infra.apache.org
nsivabalan closed issue #5836: [SUPPORT] key hoodie.table.partition.fields is stored in HiveMetaStore when create non-partition table in Spark URL: https://github.com/apache/hudi/issues/5836 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org For queries about this service, please contact Infrastructure at: users@infra.apache.org