You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@paimon.apache.org by "JingsongLi (via GitHub)" <gi...@apache.org> on 2024/12/17 02:56:50 UTC

[PR] [core] Refactor MetastoreClient methods to simplify catalog [paimon]

JingsongLi opened a new pull request, #4726:
URL: https://github.com/apache/paimon/pull/4726

   <!-- Please specify the module before the PR name: [core] ... or [flink] ... -->
   
   ### Purpose
   
   <!-- Linking this pull request to the issue -->
   New MetastoreClient API:
   ```java
   /**
    * A metastore client related to a table. All methods of this interface operate on the same specific
    * table.
    */
   public interface MetastoreClient extends AutoCloseable {
   
       void addPartition(LinkedHashMap<String, String> partition) throws Exception;
   
       void addPartitions(List<LinkedHashMap<String, String>> partitions) throws Exception;
   
       void dropPartition(LinkedHashMap<String, String> partition) throws Exception;
   
       void dropPartitions(List<LinkedHashMap<String, String>> partitions) throws Exception;
   
       void markPartitionDone(LinkedHashMap<String, String> partition) throws Exception;
   
       default void alterPartition(
               LinkedHashMap<String, String> partition, PartitionStats partitionStats)
               throws Exception {
           throw new UnsupportedOperationException();
       }
   
       /** Factory to create {@link MetastoreClient}. */
       interface Factory extends Serializable {
   
           MetastoreClient create();
       }
   }
   ```
   
   <!-- What is the purpose of the change -->
   
   ### Tests
   
   <!-- List UT and IT cases to verify this change -->
   
   ### API and Format
   
   <!-- Does this change affect API or storage format -->
   
   ### Documentation
   
   <!-- Does this change introduce a new feature -->
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [core] Refactor MetastoreClient methods to simplify catalog [paimon]

Posted by "JingsongLi (via GitHub)" <gi...@apache.org>.
JingsongLi merged PR #4726:
URL: https://github.com/apache/paimon/pull/4726


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [core] Refactor MetastoreClient methods to simplify catalog [paimon]

Posted by "JingsongLi (via GitHub)" <gi...@apache.org>.
JingsongLi commented on code in PR #4726:
URL: https://github.com/apache/paimon/pull/4726#discussion_r1887857622


##########
paimon-flink/paimon-flink-common/src/main/java/org/apache/paimon/flink/sink/partition/PartitionStatisticsReporter.java:
##########
@@ -64,7 +57,7 @@ public PartitionStatisticsReporter(FileStoreTable table, MetastoreClient client)
         this.snapshotManager = table.snapshotManager();
     }
 
-    public void report(String partition, long modifyTime) throws Exception {
+    public void report(String partition, long modifyTimeMillis) throws Exception {

Review Comment:
   See `System.currentTimeMillis`



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [core] Refactor MetastoreClient methods to simplify catalog [paimon]

Posted by "wwj6591812 (via GitHub)" <gi...@apache.org>.
wwj6591812 commented on code in PR #4726:
URL: https://github.com/apache/paimon/pull/4726#discussion_r1887847765


##########
paimon-flink/paimon-flink-common/src/main/java/org/apache/paimon/flink/sink/partition/PartitionStatisticsReporter.java:
##########
@@ -64,7 +57,7 @@ public PartitionStatisticsReporter(FileStoreTable table, MetastoreClient client)
         this.snapshotManager = table.snapshotManager();
     }
 
-    public void report(String partition, long modifyTime) throws Exception {
+    public void report(String partition, long modifyTimeMillis) throws Exception {

Review Comment:
   modifyTimeInMillis



##########
paimon-hive/paimon-hive-catalog/src/main/java/org/apache/paimon/hive/HiveMetastoreClient.java:
##########
@@ -105,55 +85,56 @@ public void addPartition(LinkedHashMap<String, String> partitionSpec) throws Exc
     }
 
     @Override
-    public void addPartitionsSpec(List<LinkedHashMap<String, String>> partitionSpecsList)
-            throws Exception {
+    public void addPartitions(List<LinkedHashMap<String, String>> partitions) throws Exception {
         int currentTime = (int) (System.currentTimeMillis() / 1000);
         List<Partition> hivePartitions =
-                partitionSpecsList.stream()
+                partitions.stream()
                         .map(partitionSpec -> toHivePartition(partitionSpec, currentTime))
                         .collect(Collectors.toList());
         clients.execute(client -> client.add_partitions(hivePartitions, true, false));
     }
 
     @Override
     public void alterPartition(
-            LinkedHashMap<String, String> partitionSpec,
-            Map<String, String> parameters,
-            long modifyTime,
-            boolean ignoreIfNotExist)
+            LinkedHashMap<String, String> partition, PartitionStats partitionStats)
             throws Exception {
-        List<String> partitionValues = new ArrayList<>(partitionSpec.values());
-        int currentTime = (int) (modifyTime / 1000);
-        Partition hivePartition;
+        List<String> partitionValues = new ArrayList<>(partition.values());
+
+        Map<String, String> statistic = new HashMap<>();

Review Comment:
   Map<String, String> statistic = new HashMap<>(5);



##########
paimon-core/src/main/java/org/apache/paimon/operation/PartitionExpire.java:
##########
@@ -167,15 +167,13 @@ private List<Map<String, String>> doExpire(
     }
 
     private void deleteMetastorePartitions(List<Map<String, String>> partitions) {
-        if (metastoreClient != null) {
-            partitions.forEach(
-                    partition -> {
-                        try {
-                            metastoreClient.deletePartition(new LinkedHashMap<>(partition));
-                        } catch (Exception e) {
-                            throw new RuntimeException(e);
-                        }
-                    });
+        if (metastoreClient != null && partitions.size() > 0) {
+            try {
+                metastoreClient.addPartitions(

Review Comment:
   addPartitions?not dropPartitions?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [core] Refactor MetastoreClient methods to simplify catalog [paimon]

Posted by "JingsongLi (via GitHub)" <gi...@apache.org>.
JingsongLi commented on code in PR #4726:
URL: https://github.com/apache/paimon/pull/4726#discussion_r1887858000


##########
paimon-hive/paimon-hive-catalog/src/main/java/org/apache/paimon/hive/HiveMetastoreClient.java:
##########
@@ -105,55 +85,56 @@ public void addPartition(LinkedHashMap<String, String> partitionSpec) throws Exc
     }
 
     @Override
-    public void addPartitionsSpec(List<LinkedHashMap<String, String>> partitionSpecsList)
-            throws Exception {
+    public void addPartitions(List<LinkedHashMap<String, String>> partitions) throws Exception {
         int currentTime = (int) (System.currentTimeMillis() / 1000);
         List<Partition> hivePartitions =
-                partitionSpecsList.stream()
+                partitions.stream()
                         .map(partitionSpec -> toHivePartition(partitionSpec, currentTime))
                         .collect(Collectors.toList());
         clients.execute(client -> client.add_partitions(hivePartitions, true, false));
     }
 
     @Override
     public void alterPartition(
-            LinkedHashMap<String, String> partitionSpec,
-            Map<String, String> parameters,
-            long modifyTime,
-            boolean ignoreIfNotExist)
+            LinkedHashMap<String, String> partition, PartitionStats partitionStats)
             throws Exception {
-        List<String> partitionValues = new ArrayList<>(partitionSpec.values());
-        int currentTime = (int) (modifyTime / 1000);
-        Partition hivePartition;
+        List<String> partitionValues = new ArrayList<>(partition.values());
+
+        Map<String, String> statistic = new HashMap<>();

Review Comment:
   For these small collection, we don't need to specific the init size.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org