[GitHub] [incubator-paimon] tsreaper opened a new pull request, #1545: [core][hive] Support syncing partition into Hive metastore when using Hive catalog

["tsreaper (via GitHub)" <gi...@apache.org>]

tsreaper opened a new pull request, #1545:
URL: https://github.com/apache/incubator-paimon/pull/1545

   ### Purpose
   
   Currently we cannot create partitioned Paimon table in Hive catalog. In Hive, all fields are considered as normal fields and there is no partition key. It brings us at least two downsides:
   
   1. Users cannot list all partitions through the SHOW PARTITIONS command.
   2. Hive's native partition pushdown cannot be performed. Although currently we approach partition pushdown through Hive's filter pushdown, some Hive compatible system may only support partition pushdown and not filter pushdown, making Paimon nearly unusable in those systems.
   
   This PR introduces a feature to synchronize partitions into Hive metastore when using Hive catalog. In this PR, users can create partitioned Hive table through Flink and Spark using Hive catalog. Creating partitioned table directly in Hive is currently not supported and will be introduced in the next PR.
   
   ### Tests
   
   * `HiveCatalogITCaseBase#testAddPartitionsToMetastore`
   
   ### API and Format
   
   No.
   
   ### Documentation
   
   Yes.
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [incubator-paimon] JingsongLi merged pull request #1545: [core][hive] Support syncing partition into Hive metastore when using Hive catalog

["JingsongLi (via GitHub)" <gi...@apache.org>]

JingsongLi merged PR #1545:
URL: https://github.com/apache/incubator-paimon/pull/1545


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [incubator-paimon] JingsongLi commented on a diff in pull request #1545: [core][hive] Support syncing partition into Hive metastore when using Hive catalog

["JingsongLi (via GitHub)" <gi...@apache.org>]

JingsongLi commented on code in PR #1545:
URL: https://github.com/apache/incubator-paimon/pull/1545#discussion_r1260613763


##########
paimon-core/src/main/java/org/apache/paimon/CoreOptions.java:
##########
@@ -755,6 +755,15 @@ public class CoreOptions implements Serializable {
                             "Parameter string for the constructor of class #. "
                                     + "Callback class should parse the parameter by itself.");
 
+    public static final ConfigOption<Boolean> PARTITION_ADD_TO_METASTORE =
+            key("partition.add-to-metastore")

Review Comment:
   I have a feeling that what is actually being said here is that we will synchronize paimon's partition table into Hive's partition table, and after synchronization, of course, we need to addPartition.
   
   So this key better say create hive partitioned table?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org