You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@amoro.apache.org by "pallavithegod (via GitHub)" <gi...@apache.org> on 2026/10/15 19:23:59 UTC

[PR] [AMORO-1503][Common] Log successful Hive partition mutations [amoro]

pallavithegod opened a new pull request, #4444:
URL: https://github.com/apache/amoro/pull/4444

   ## Why are the changes needed?
   Close #1503.
   
   `HMSClientImpl` delegates partition additions, drops, and alterations without recording the affected table, partition values, or new location. This makes it difficult to trace mixed-Hive partition changes when diagnosing metadata or data-location problems.
   
   Log completed mutations at INFO after the metastore call returns. Failed operations do not emit success messages; a false drop result or a zero-count batch addition is also not reported as a successful change. Existing return values and exception behavior are preserved.
   
   ## Brief change log
   - Record single and batch partition additions and alterations, including table identity, values, and requested locations.
   - Record successful drops with table identity and partition values.
   - Report the actual batch-add count separately from requested partition details.
   - Add coverage for successful operations, failed additions, and no-op results.
   
   ## How was this patch tested?
   - [x] Added new unit test cases that verify the changes (`TestHMSPartitionLogging`).
   - [x] Verified existing unit tests pass locally (`TestHiveTableTypeUtil`); 10 selected tests passed in total.
   - [ ] Tested manually in local environment.
   
   Using Java 11:
   - `./mvnw spotless:apply checkstyle:check` passed.
   - `./mvnw test -pl amoro-common -am -Dtest=TestHMSPartitionLogging,TestHiveTableTypeUtil -Dsurefire.failIfNoSpecifiedTests=false` passed.
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@amoro.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [AMORO-1503][Common] Log successful Hive partition mutations [amoro]

Posted by "github-actions[bot] (via GitHub) via commits" <co...@amoro.apache.org>.
github-actions[bot] commented on PR #4444:
URL: https://github.com/apache/amoro/pull/4444#issuecomment-6886303861

   This pull request has been marked as stale due to 30 days of inactivity. It will be closed in 1 week if no further activity occurs. If you think that’s incorrect or this pull request requires a review, please simply write any comment. If closed, you can revive the PR at any time and @mention a reviewer or discuss it on the dev@amoro.apache.org list. Thank you for your contributions.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@amoro.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [AMORO-1503][Common] Log successful Hive partition mutations [amoro]

Posted by "czy006 (via GitHub) via commits" <co...@amoro.apache.org>.
czy006 commented on code in PR #4444:
URL: https://github.com/apache/amoro/pull/4444#discussion_r4587390182


##########
amoro-common/src/main/java/org/apache/amoro/hive/HMSClientImpl.java:
##########
@@ -128,19 +133,32 @@ public Database getDatabase(String databaseName) throws TException {
 
   @Override
   public Partition addPartition(Partition partition) throws TException {
-    return getClient().add_partition(partition);
+    Partition added = getClient().add_partition(partition);
+    LOG.info("Added Hive partition {}", describePartition(partition));
+    return added;
   }
 
   @Override
   public boolean dropPartition(
       String dbName, String tblName, List<String> partVals, PartitionDropOptions options)
       throws TException {
-    return getClient().dropPartition(dbName, tblName, partVals, options);
+    boolean dropped = getClient().dropPartition(dbName, tblName, partVals, options);
+    if (dropped) {
+      LOG.info("Dropped Hive partition {}.{} values={}", dbName, tblName, partVals);
+    }
+    return dropped;
   }
 
   @Override
   public int addPartitions(List<Partition> partitions) throws TException {
-    return getClient().add_partitions(partitions);
+    int added = getClient().add_partitions(partitions);
+    if (LOG.isInfoEnabled() && added > 0) {
+      LOG.info(
+          "Added {} Hive partitions; requested partitions: {}",
+          added,
+          describePartitions(partitions));

Review Comment:
   **[Minor] Consider capping how many partition descriptors go into a single batch log line.**
   
   `UpdateHiveFiles#commitPartitionedTable()` and `ReplaceHivePartitions#commitPartitionedTable()` pass *all* partitions touched in one commit to `addPartitions`/`alterPartitions` in a single call. For a Flink overwrite on a table with thousands of partitions, `describePartitions` builds one INFO line of hundreds of KB (each descriptor is roughly 60–150 chars).
   
   Suggestion: keep the actual count and log only the first N descriptors, e.g.
   
   ```java
   String details =
       partitions.stream()
           .limit(20)
           .map(HMSClientImpl::describePartition)
           .collect(Collectors.joining(", "));
   LOG.info("Added {} Hive partitions; requested partitions: {}", added, details);
   ```
   
   The same applies to `alterPartitions` below. Happy either way — feel free to address here or as a follow-up.



##########
amoro-common/src/main/java/org/apache/amoro/hive/HMSClientImpl.java:
##########
@@ -128,19 +133,32 @@ public Database getDatabase(String databaseName) throws TException {
 
   @Override
   public Partition addPartition(Partition partition) throws TException {
-    return getClient().add_partition(partition);
+    Partition added = getClient().add_partition(partition);
+    LOG.info("Added Hive partition {}", describePartition(partition));

Review Comment:
   **[Nit] Two small things:**
   
   1. `addPartition`/`alterPartition` build the descriptor eagerly, while `addPartitions`/`alterPartitions` guard with `LOG.isInfoEnabled()`. The cost is negligible for a single partition, so this is just an inconsistency — either drop the guards everywhere or keep them everywhere.
   2. This logs the *requested* partition, so when the caller doesn't set a location it prints `location=null` even though HMS assigns one at creation. That's acceptable for tracing (the request is what the caller did); just flagging it in case you'd prefer logging the returned partition.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@amoro.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org