You are viewing a plain text version of this content. The canonical link for it is here.
Posted to notifications@kyuubi.apache.org by "maomaodev (via GitHub)" <gi...@apache.org> on 2026/04/11 12:33:15 UTC
[PR] [KYUUBI #7398] SERDEPROPERTIES are missing when KSHC create table [kyuubi]
maomaodev opened a new pull request, #7399:
URL: https://github.com/apache/kyuubi/pull/7399
<!--
Thanks for sending a pull request!
Here are some tips for you:
1. If this is your first time, please read our contributor guidelines: https://kyuubi.readthedocs.io/en/master/contributing/code/index.html
2. If the PR is related to an issue in https://github.com/apache/kyuubi/issues, add '[KYUUBI #XXXX]' in your PR title, e.g., '[KYUUBI #XXXX] Your PR title ...'.
3. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP][KYUUBI #XXXX] Your PR title ...'.
-->
### Why are the changes needed?
<!--
Please clarify why the changes are needed. For instance,
1. If you propose a new API, clarify the use case for a new API.
2. If you fix a bug, you can clarify why it is a bug, and what versions are affected.
-->
1. Execute the following SQL to create table and insert.
```
CREATE TABLE test_table (name STRING, age INT)
ROW FORMAT SERDE 'org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe'
WITH SERDEPROPERTIES ('field.delim' = ',')
STORED AS TEXTFILE;
INSERT INTO TABLE test_table VALUES ('alice', 30);
```
2. Check the underlying HDFS data. Output is `alice30`, Expected is `alice,30`
```
hdfs dfs -cat /usr/hive/warehouse/test_table/part-00000-08cdc3f0-15af-413a-a709-46e24f1ace91-c000
```
3. Cause: SERDEPROPERTIES are missing when KSHC create table, it did not strip the option. prefix. See: https://github.com/apache/spark/pull/28026
### How was this patch tested?
<!--
If tests were added, say they were added here. Please make sure to add some test cases that check the changes thoroughly including negative and positive cases if possible.
If it was tested in a way different from regular unit tests, please clarify how you tested step by step, ideally copy and paste-able, so that other reviewers can test and check, and descendants can verify in the future.
If tests were not added, please describe why they were not added and/or why it was difficult to add.
-->
UT
### Was this patch authored or co-authored using generative AI tooling?
<!--
If a generative AI tooling has been used in the process of authoring this patch, please include
phrase 'Generated-by: ' followed by the name of the tool and its version.
If no, write 'No'.
Please refer to the [ASF Generative Tooling Guidance](https://www.apache.org/legal/generative-tooling.html) for details.
-->
No
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For additional commands, e-mail: notifications-help@kyuubi.apache.org
Re: [PR] [KYUUBI #7398] SERDEPROPERTIES are missing when KSHC create table [kyuubi]
Posted by "maomaodev (via GitHub)" <gi...@apache.org>.
maomaodev commented on code in PR #7399:
URL: https://github.com/apache/kyuubi/pull/7399#discussion_r3070701683
##########
extensions/spark/kyuubi-spark-connector-hive/src/main/scala/org/apache/kyuubi/spark/connector/hive/HiveTableCatalog.scala:
##########
@@ -431,10 +434,25 @@ class HiveTableCatalog(sparkSession: SparkSession)
catalog.renameTable(oldIdent.asTableIdentifier, newIdent.asTableIdentifier)
}
- private def toOptions(properties: Map[String, String]): Map[String, String] = {
- properties.filterKeys(_.startsWith(TableCatalog.OPTION_PREFIX)).map {
- case (key, value) => key.drop(TableCatalog.OPTION_PREFIX.length) -> value
- }.toMap
+ /**
+ * Splits properties into optionsProps and serdeProps based on the `options.` prefix.
+ *
+ * - optionsProps: keys with "options." prefix whose stripped key ALREADY exist in properties,
+ * indicating they were originally specified via OPTIONS clause.
+ * - serdeProps: keys with "options." prefix whose stripped key does NOT exists in properties,
+ * indicating they were originally specified via SERDEPROPERTIES clause
+ *
+ * @param properties the full properties map
+ * @return a tuple of (optionsProps, serdeProps), both with the "options." prefix stripped
+ */
+ private def toOptionsAndSerdeProps(
Review Comment:
done
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For additional commands, e-mail: notifications-help@kyuubi.apache.org
Re: [PR] [KYUUBI #7398] SERDEPROPERTIES are missing when KSHC create table [kyuubi]
Posted by "pan3793 (via GitHub)" <gi...@apache.org>.
pan3793 commented on code in PR #7399:
URL: https://github.com/apache/kyuubi/pull/7399#discussion_r3070591115
##########
extensions/spark/kyuubi-spark-connector-hive/src/main/scala/org/apache/kyuubi/spark/connector/hive/HiveTableCatalog.scala:
##########
@@ -431,10 +434,25 @@ class HiveTableCatalog(sparkSession: SparkSession)
catalog.renameTable(oldIdent.asTableIdentifier, newIdent.asTableIdentifier)
}
- private def toOptions(properties: Map[String, String]): Map[String, String] = {
- properties.filterKeys(_.startsWith(TableCatalog.OPTION_PREFIX)).map {
- case (key, value) => key.drop(TableCatalog.OPTION_PREFIX.length) -> value
- }.toMap
+ /**
+ * Splits properties into optionsProps and serdeProps based on the `options.` prefix.
+ *
+ * - optionsProps: keys with "options." prefix whose stripped key ALREADY exist in properties,
+ * indicating they were originally specified via OPTIONS clause.
+ * - serdeProps: keys with "options." prefix whose stripped key does NOT exists in properties,
+ * indicating they were originally specified via SERDEPROPERTIES clause
+ *
+ * @param properties the full properties map
+ * @return a tuple of (optionsProps, serdeProps), both with the "options." prefix stripped
+ */
+ private def toOptionsAndSerdeProps(
Review Comment:
let's add a UT for this method
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For additional commands, e-mail: notifications-help@kyuubi.apache.org
Re: [PR] [KYUUBI #7398] SERDEPROPERTIES are missing when KSHC create table [kyuubi]
Posted by "pan3793 (via GitHub)" <gi...@apache.org>.
pan3793 closed pull request #7399: [KYUUBI #7398] SERDEPROPERTIES are missing when KSHC create table
URL: https://github.com/apache/kyuubi/pull/7399
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For additional commands, e-mail: notifications-help@kyuubi.apache.org
Re: [PR] [KYUUBI #7398] SERDEPROPERTIES are missing when KSHC create table [kyuubi]
Posted by "maomaodev (via GitHub)" <gi...@apache.org>.
maomaodev commented on PR #7399:
URL: https://github.com/apache/kyuubi/pull/7399#issuecomment-4235931329
Gentle ping @pan3793 . Could you please review it again?
One of the checks failed, but it seems unrelated to this PR.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For additional commands, e-mail: notifications-help@kyuubi.apache.org
Re: [PR] [KYUUBI #7398] SERDEPROPERTIES are missing when KSHC create table [kyuubi]
Posted by "pan3793 (via GitHub)" <gi...@apache.org>.
pan3793 commented on PR #7399:
URL: https://github.com/apache/kyuubi/pull/7399#issuecomment-4235946625
thanks, merged to master
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For additional commands, e-mail: notifications-help@kyuubi.apache.org