You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@hudi.apache.org by "lokeshj1703 (via GitHub)" <gi...@apache.org> on 2023/08/17 12:58:12 UTC

[GitHub] [hudi] lokeshj1703 opened a new pull request, #9467: [HUDI-6717] Fix downgrade handler for 0.14.0

lokeshj1703 opened a new pull request, #9467:
URL: https://github.com/apache/hudi/pull/9467

   ### Change Logs
   
   Since the log block version (due to delete block change) has been upgraded in 0.14.0, the delete blocks can not be read in 0.13.0 or earlier.
   Similarly the addition of record level index field in metadata table leads to column drop error on downgrade. The Jira aims to fix the downgrade handler to trigger compaction and delete metadata table if user wishes to downgrade from version six (0.14.0) to version 5 (0.13.0).
   
   ### Impact
   
   NA
   
   ### Risk level (write none, low medium or high below)
   
   low
   
   ### Documentation Update
   
   NA
   
   ### Contributor's checklist
   
   - [ ] Read through [contributor's guide](https://hudi.apache.org/contribute/how-to-contribute)
   - [ ] Change Logs and Impact were stated clearly
   - [ ] Adequate tests were added if applicable
   - [ ] CI passed
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [hudi] nsivabalan commented on a diff in pull request #9467: [HUDI-6717] Fix downgrade handler for 0.14.0

Posted by "nsivabalan (via GitHub)" <gi...@apache.org>.
nsivabalan commented on code in PR #9467:
URL: https://github.com/apache/hudi/pull/9467#discussion_r1298605259


##########
hudi-client/hudi-client-common/src/main/java/org/apache/hudi/table/upgrade/SixToFiveDowngradeHandler.java:
##########
@@ -39,20 +47,26 @@
 
 import static org.apache.hudi.common.table.HoodieTableConfig.TABLE_METADATA_PARTITIONS;
 import static org.apache.hudi.common.table.HoodieTableConfig.TABLE_METADATA_PARTITIONS_INFLIGHT;
-import static org.apache.hudi.metadata.HoodieTableMetadataUtil.deleteMetadataTablePartition;
 
 /**
  * Downgrade handle to assist in downgrading hoodie table from version 6 to 5.
  * To ensure compatibility, we need recreate the compaction requested file to
  * .aux folder.
+ * Since version 6 includes a new schema field for metadata table(MDT),
+ * the MDT needs to be deleted during downgrade to avoid column drop error.
+ * Also log block version was upgraded in version 6, therefore full compaction needs
+ * to be completed during downgrade to avoid write failures.
  */
 public class SixToFiveDowngradeHandler implements DowngradeHandler {
 
   @Override
   public Map<ConfigProperty, String> downgrade(HoodieWriteConfig config, HoodieEngineContext context, String instantTime, SupportsUpgradeDowngrade upgradeDowngradeHelper) {
     final HoodieTable table = upgradeDowngradeHelper.getTable(config, context);
 
-    removeRecordIndexIfNeeded(table, context);
+    // Since version 6 includes a new schema field for metadata table(MDT), the MDT needs to be deleted during downgrade to avoid column drop error.
+    HoodieTableMetadataUtil.deleteMetadataTable(config.getBasePath(), context);
+    runCompaction(table, context, config, upgradeDowngradeHelper);

Review Comment:
   java docs as to why we do compaction



##########
hudi-client/hudi-client-common/src/main/java/org/apache/hudi/table/upgrade/SixToFiveDowngradeHandler.java:
##########
@@ -39,20 +47,26 @@
 
 import static org.apache.hudi.common.table.HoodieTableConfig.TABLE_METADATA_PARTITIONS;
 import static org.apache.hudi.common.table.HoodieTableConfig.TABLE_METADATA_PARTITIONS_INFLIGHT;
-import static org.apache.hudi.metadata.HoodieTableMetadataUtil.deleteMetadataTablePartition;
 
 /**
  * Downgrade handle to assist in downgrading hoodie table from version 6 to 5.
  * To ensure compatibility, we need recreate the compaction requested file to
  * .aux folder.
+ * Since version 6 includes a new schema field for metadata table(MDT),
+ * the MDT needs to be deleted during downgrade to avoid column drop error.
+ * Also log block version was upgraded in version 6, therefore full compaction needs
+ * to be completed during downgrade to avoid write failures.

Review Comment:
   minor: ..... to avoid both read and future compaction failures. 



##########
hudi-client/hudi-client-common/src/main/java/org/apache/hudi/table/upgrade/SixToFiveDowngradeHandler.java:
##########
@@ -65,13 +79,31 @@ public Map<ConfigProperty, String> downgrade(HoodieWriteConfig config, HoodieEng
   }
 
   /**
-   * Record-level index, a new partition in metadata table, was first added in
-   * 0.14.0 ({@link HoodieTableVersion#SIX}. Any downgrade from this version
-   * should remove this partition.
+   * Utility method to run compaction for MOR table as part of downgrade step.
    */
-  private static void removeRecordIndexIfNeeded(HoodieTable table, HoodieEngineContext context) {
-    HoodieTableMetaClient metaClient = table.getMetaClient();
-    deleteMetadataTablePartition(metaClient, context, MetadataPartitionType.RECORD_INDEX, false);
+  private void runCompaction(HoodieTable table, HoodieEngineContext context, HoodieWriteConfig config,
+                             SupportsUpgradeDowngrade upgradeDowngradeHelper) {
+    try {
+      if (table.getMetaClient().getTableType() == HoodieTableType.MERGE_ON_READ) {
+        // The log block version has been upgraded in version six so compaction is required for downgrade.
+        // set required configs for scheduling compaction.
+        HoodieInstantTimeGenerator.setCommitTimeZone(table.getMetaClient().getTableConfig().getTimelineTimezone());
+        HoodieWriteConfig compactionConfig = HoodieWriteConfig.newBuilder().withProps(config.getProps()).build();
+        compactionConfig.setValue(HoodieCompactionConfig.INLINE_COMPACT.key(), "true");
+        compactionConfig.setValue(HoodieCompactionConfig.INLINE_COMPACT_NUM_DELTA_COMMITS.key(), "1");
+        compactionConfig.setValue(HoodieCompactionConfig.INLINE_COMPACT_TRIGGER_STRATEGY.key(), CompactionTriggerStrategy.NUM_COMMITS.name());
+        compactionConfig.setValue(HoodieCompactionConfig.COMPACTION_STRATEGY.key(), UnBoundedCompactionStrategy.class.getName());
+        compactionConfig.setValue(HoodieMetadataConfig.ENABLE.key(), "false");
+        EmbeddedTimelineServerHelper.createEmbeddedTimelineService(context, config);

Review Comment:
   why create timeline server outside. It will create one internally ? 



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [hudi] nsivabalan merged pull request #9467: [HUDI-6621] Fix downgrade handler for 0.14.0

Posted by "nsivabalan (via GitHub)" <gi...@apache.org>.
nsivabalan merged PR #9467:
URL: https://github.com/apache/hudi/pull/9467


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [hudi] hudi-bot commented on pull request #9467: [HUDI-6717] Fix downgrade handler for 0.14.0

Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #9467:
URL: https://github.com/apache/hudi/pull/9467#issuecomment-1683697594

   <!--
   Meta data
   {
     "version" : 1,
     "metaDataEntries" : [ {
       "hash" : "2ade66c64355778bea62ef8ef81c80b929f50b3f",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19339",
       "triggerID" : "2ade66c64355778bea62ef8ef81c80b929f50b3f",
       "triggerType" : "PUSH"
     }, {
       "hash" : "50d4aea5e545b5094368e3a192ffb5fd2008c481",
       "status" : "FAILURE",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19349",
       "triggerID" : "50d4aea5e545b5094368e3a192ffb5fd2008c481",
       "triggerType" : "PUSH"
     } ]
   }-->
   ## CI report:
   
   * 50d4aea5e545b5094368e3a192ffb5fd2008c481 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19349) 
   
   <details>
   <summary>Bot commands</summary>
     @hudi-bot supports the following commands:
   
    - `@hudi-bot run azure` re-run the last Azure build
   </details>


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [hudi] hudi-bot commented on pull request #9467: [HUDI-6717] Fix downgrade handler for 0.14.0

Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #9467:
URL: https://github.com/apache/hudi/pull/9467#issuecomment-1682297929

   <!--
   Meta data
   {
     "version" : 1,
     "metaDataEntries" : [ {
       "hash" : "2ade66c64355778bea62ef8ef81c80b929f50b3f",
       "status" : "PENDING",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19339",
       "triggerID" : "2ade66c64355778bea62ef8ef81c80b929f50b3f",
       "triggerType" : "PUSH"
     } ]
   }-->
   ## CI report:
   
   * 2ade66c64355778bea62ef8ef81c80b929f50b3f Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19339) 
   
   <details>
   <summary>Bot commands</summary>
     @hudi-bot supports the following commands:
   
    - `@hudi-bot run azure` re-run the last Azure build
   </details>


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [hudi] yihua commented on pull request #9467: [HUDI-6717] Fix downgrade handler for 0.14.0

Posted by "yihua (via GitHub)" <gi...@apache.org>.
yihua commented on PR #9467:
URL: https://github.com/apache/hudi/pull/9467#issuecomment-1684167213

   Could you reuse the same JIRA ticket already filed instead of creating a new one: HUDI-6621?


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [hudi] hudi-bot commented on pull request #9467: [HUDI-6621] Fix downgrade handler for 0.14.0

Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #9467:
URL: https://github.com/apache/hudi/pull/9467#issuecomment-1684942768

   <!--
   Meta data
   {
     "version" : 1,
     "metaDataEntries" : [ {
       "hash" : "2ade66c64355778bea62ef8ef81c80b929f50b3f",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19339",
       "triggerID" : "2ade66c64355778bea62ef8ef81c80b929f50b3f",
       "triggerType" : "PUSH"
     }, {
       "hash" : "50d4aea5e545b5094368e3a192ffb5fd2008c481",
       "status" : "FAILURE",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19349",
       "triggerID" : "50d4aea5e545b5094368e3a192ffb5fd2008c481",
       "triggerType" : "PUSH"
     }, {
       "hash" : "d20d5b2e45e0eccf8f3ec40077696eecf9dfc4bb",
       "status" : "PENDING",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19368",
       "triggerID" : "d20d5b2e45e0eccf8f3ec40077696eecf9dfc4bb",
       "triggerType" : "PUSH"
     } ]
   }-->
   ## CI report:
   
   * 50d4aea5e545b5094368e3a192ffb5fd2008c481 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19349) 
   * d20d5b2e45e0eccf8f3ec40077696eecf9dfc4bb Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19368) 
   
   <details>
   <summary>Bot commands</summary>
     @hudi-bot supports the following commands:
   
    - `@hudi-bot run azure` re-run the last Azure build
   </details>


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [hudi] hudi-bot commented on pull request #9467: [HUDI-6621] Fix downgrade handler for 0.14.0

Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #9467:
URL: https://github.com/apache/hudi/pull/9467#issuecomment-1685042082

   <!--
   Meta data
   {
     "version" : 1,
     "metaDataEntries" : [ {
       "hash" : "2ade66c64355778bea62ef8ef81c80b929f50b3f",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19339",
       "triggerID" : "2ade66c64355778bea62ef8ef81c80b929f50b3f",
       "triggerType" : "PUSH"
     }, {
       "hash" : "50d4aea5e545b5094368e3a192ffb5fd2008c481",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19349",
       "triggerID" : "50d4aea5e545b5094368e3a192ffb5fd2008c481",
       "triggerType" : "PUSH"
     }, {
       "hash" : "d20d5b2e45e0eccf8f3ec40077696eecf9dfc4bb",
       "status" : "FAILURE",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19368",
       "triggerID" : "d20d5b2e45e0eccf8f3ec40077696eecf9dfc4bb",
       "triggerType" : "PUSH"
     } ]
   }-->
   ## CI report:
   
   * d20d5b2e45e0eccf8f3ec40077696eecf9dfc4bb Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19368) 
   
   <details>
   <summary>Bot commands</summary>
     @hudi-bot supports the following commands:
   
    - `@hudi-bot run azure` re-run the last Azure build
   </details>


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [hudi] hudi-bot commented on pull request #9467: [HUDI-6717] Fix downgrade handler for 0.14.0

Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #9467:
URL: https://github.com/apache/hudi/pull/9467#issuecomment-1683481345

   <!--
   Meta data
   {
     "version" : 1,
     "metaDataEntries" : [ {
       "hash" : "2ade66c64355778bea62ef8ef81c80b929f50b3f",
       "status" : "FAILURE",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19339",
       "triggerID" : "2ade66c64355778bea62ef8ef81c80b929f50b3f",
       "triggerType" : "PUSH"
     }, {
       "hash" : "50d4aea5e545b5094368e3a192ffb5fd2008c481",
       "status" : "PENDING",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19349",
       "triggerID" : "50d4aea5e545b5094368e3a192ffb5fd2008c481",
       "triggerType" : "PUSH"
     } ]
   }-->
   ## CI report:
   
   * 2ade66c64355778bea62ef8ef81c80b929f50b3f Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19339) 
   * 50d4aea5e545b5094368e3a192ffb5fd2008c481 Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19349) 
   
   <details>
   <summary>Bot commands</summary>
     @hudi-bot supports the following commands:
   
    - `@hudi-bot run azure` re-run the last Azure build
   </details>


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [hudi] lokeshj1703 commented on a diff in pull request #9467: [HUDI-6717] Fix downgrade handler for 0.14.0

Posted by "lokeshj1703 (via GitHub)" <gi...@apache.org>.
lokeshj1703 commented on code in PR #9467:
URL: https://github.com/apache/hudi/pull/9467#discussion_r1299179473


##########
hudi-client/hudi-client-common/src/main/java/org/apache/hudi/table/upgrade/SixToFiveDowngradeHandler.java:
##########
@@ -39,20 +47,26 @@
 
 import static org.apache.hudi.common.table.HoodieTableConfig.TABLE_METADATA_PARTITIONS;
 import static org.apache.hudi.common.table.HoodieTableConfig.TABLE_METADATA_PARTITIONS_INFLIGHT;
-import static org.apache.hudi.metadata.HoodieTableMetadataUtil.deleteMetadataTablePartition;
 
 /**
  * Downgrade handle to assist in downgrading hoodie table from version 6 to 5.
  * To ensure compatibility, we need recreate the compaction requested file to
  * .aux folder.
+ * Since version 6 includes a new schema field for metadata table(MDT),
+ * the MDT needs to be deleted during downgrade to avoid column drop error.
+ * Also log block version was upgraded in version 6, therefore full compaction needs
+ * to be completed during downgrade to avoid write failures.
  */
 public class SixToFiveDowngradeHandler implements DowngradeHandler {
 
   @Override
   public Map<ConfigProperty, String> downgrade(HoodieWriteConfig config, HoodieEngineContext context, String instantTime, SupportsUpgradeDowngrade upgradeDowngradeHelper) {
     final HoodieTable table = upgradeDowngradeHelper.getTable(config, context);
 
-    removeRecordIndexIfNeeded(table, context);
+    // Since version 6 includes a new schema field for metadata table(MDT), the MDT needs to be deleted during downgrade to avoid column drop error.
+    HoodieTableMetadataUtil.deleteMetadataTable(config.getBasePath(), context);
+    runCompaction(table, context, config, upgradeDowngradeHelper);

Review Comment:
   Addressed



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [hudi] hudi-bot commented on pull request #9467: [HUDI-6621] Fix downgrade handler for 0.14.0

Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #9467:
URL: https://github.com/apache/hudi/pull/9467#issuecomment-1686918146

   <!--
   Meta data
   {
     "version" : 1,
     "metaDataEntries" : [ {
       "hash" : "2ade66c64355778bea62ef8ef81c80b929f50b3f",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19339",
       "triggerID" : "2ade66c64355778bea62ef8ef81c80b929f50b3f",
       "triggerType" : "PUSH"
     }, {
       "hash" : "50d4aea5e545b5094368e3a192ffb5fd2008c481",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19349",
       "triggerID" : "50d4aea5e545b5094368e3a192ffb5fd2008c481",
       "triggerType" : "PUSH"
     }, {
       "hash" : "d20d5b2e45e0eccf8f3ec40077696eecf9dfc4bb",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19368",
       "triggerID" : "d20d5b2e45e0eccf8f3ec40077696eecf9dfc4bb",
       "triggerType" : "PUSH"
     }, {
       "hash" : "6cba53453621aa518273a8879eeea3081f4c94cb",
       "status" : "FAILURE",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19384",
       "triggerID" : "6cba53453621aa518273a8879eeea3081f4c94cb",
       "triggerType" : "PUSH"
     } ]
   }-->
   ## CI report:
   
   * 6cba53453621aa518273a8879eeea3081f4c94cb Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19384) 
   
   <details>
   <summary>Bot commands</summary>
     @hudi-bot supports the following commands:
   
    - `@hudi-bot run azure` re-run the last Azure build
   </details>


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [hudi] lokeshj1703 commented on pull request #9467: [HUDI-6717] Fix downgrade handler for 0.14.0

Posted by "lokeshj1703 (via GitHub)" <gi...@apache.org>.
lokeshj1703 commented on PR #9467:
URL: https://github.com/apache/hudi/pull/9467#issuecomment-1682243995

   @nsivabalan @codope Please review.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [hudi] hudi-bot commented on pull request #9467: [HUDI-6621] Fix downgrade handler for 0.14.0

Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #9467:
URL: https://github.com/apache/hudi/pull/9467#issuecomment-1685814810

   <!--
   Meta data
   {
     "version" : 1,
     "metaDataEntries" : [ {
       "hash" : "2ade66c64355778bea62ef8ef81c80b929f50b3f",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19339",
       "triggerID" : "2ade66c64355778bea62ef8ef81c80b929f50b3f",
       "triggerType" : "PUSH"
     }, {
       "hash" : "50d4aea5e545b5094368e3a192ffb5fd2008c481",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19349",
       "triggerID" : "50d4aea5e545b5094368e3a192ffb5fd2008c481",
       "triggerType" : "PUSH"
     }, {
       "hash" : "d20d5b2e45e0eccf8f3ec40077696eecf9dfc4bb",
       "status" : "FAILURE",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19368",
       "triggerID" : "d20d5b2e45e0eccf8f3ec40077696eecf9dfc4bb",
       "triggerType" : "PUSH"
     }, {
       "hash" : "6cba53453621aa518273a8879eeea3081f4c94cb",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "6cba53453621aa518273a8879eeea3081f4c94cb",
       "triggerType" : "PUSH"
     } ]
   }-->
   ## CI report:
   
   * d20d5b2e45e0eccf8f3ec40077696eecf9dfc4bb Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19368) 
   * 6cba53453621aa518273a8879eeea3081f4c94cb UNKNOWN
   
   <details>
   <summary>Bot commands</summary>
     @hudi-bot supports the following commands:
   
    - `@hudi-bot run azure` re-run the last Azure build
   </details>


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [hudi] hudi-bot commented on pull request #9467: [HUDI-6717] Fix downgrade handler for 0.14.0

Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #9467:
URL: https://github.com/apache/hudi/pull/9467#issuecomment-1682688492

   <!--
   Meta data
   {
     "version" : 1,
     "metaDataEntries" : [ {
       "hash" : "2ade66c64355778bea62ef8ef81c80b929f50b3f",
       "status" : "FAILURE",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19339",
       "triggerID" : "2ade66c64355778bea62ef8ef81c80b929f50b3f",
       "triggerType" : "PUSH"
     } ]
   }-->
   ## CI report:
   
   * 2ade66c64355778bea62ef8ef81c80b929f50b3f Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19339) 
   
   <details>
   <summary>Bot commands</summary>
     @hudi-bot supports the following commands:
   
    - `@hudi-bot run azure` re-run the last Azure build
   </details>


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [hudi] hudi-bot commented on pull request #9467: [HUDI-6717] Fix downgrade handler for 0.14.0

Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #9467:
URL: https://github.com/apache/hudi/pull/9467#issuecomment-1682284936

   <!--
   Meta data
   {
     "version" : 1,
     "metaDataEntries" : [ {
       "hash" : "2ade66c64355778bea62ef8ef81c80b929f50b3f",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "2ade66c64355778bea62ef8ef81c80b929f50b3f",
       "triggerType" : "PUSH"
     } ]
   }-->
   ## CI report:
   
   * 2ade66c64355778bea62ef8ef81c80b929f50b3f UNKNOWN
   
   <details>
   <summary>Bot commands</summary>
     @hudi-bot supports the following commands:
   
    - `@hudi-bot run azure` re-run the last Azure build
   </details>


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [hudi] lokeshj1703 commented on pull request #9467: [HUDI-6621] Fix downgrade handler for 0.14.0

Posted by "lokeshj1703 (via GitHub)" <gi...@apache.org>.
lokeshj1703 commented on PR #9467:
URL: https://github.com/apache/hudi/pull/9467#issuecomment-1689684602

   <img width="984" alt="image" src="https://github.com/apache/hudi/assets/9255455/56fff9c4-70d3-4ce1-941d-e25a1e7539c2">
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [hudi] lokeshj1703 commented on pull request #9467: [HUDI-6621] Fix downgrade handler for 0.14.0

Posted by "lokeshj1703 (via GitHub)" <gi...@apache.org>.
lokeshj1703 commented on PR #9467:
URL: https://github.com/apache/hudi/pull/9467#issuecomment-1689683472

   ![Uploading Screenshot 2023-08-23 at 3.40.17 PM.png…]()
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [hudi] lokeshj1703 commented on a diff in pull request #9467: [HUDI-6717] Fix downgrade handler for 0.14.0

Posted by "lokeshj1703 (via GitHub)" <gi...@apache.org>.
lokeshj1703 commented on code in PR #9467:
URL: https://github.com/apache/hudi/pull/9467#discussion_r1299179444


##########
hudi-client/hudi-client-common/src/main/java/org/apache/hudi/table/upgrade/SixToFiveDowngradeHandler.java:
##########
@@ -65,13 +79,31 @@ public Map<ConfigProperty, String> downgrade(HoodieWriteConfig config, HoodieEng
   }
 
   /**
-   * Record-level index, a new partition in metadata table, was first added in
-   * 0.14.0 ({@link HoodieTableVersion#SIX}. Any downgrade from this version
-   * should remove this partition.
+   * Utility method to run compaction for MOR table as part of downgrade step.
    */
-  private static void removeRecordIndexIfNeeded(HoodieTable table, HoodieEngineContext context) {
-    HoodieTableMetaClient metaClient = table.getMetaClient();
-    deleteMetadataTablePartition(metaClient, context, MetadataPartitionType.RECORD_INDEX, false);
+  private void runCompaction(HoodieTable table, HoodieEngineContext context, HoodieWriteConfig config,
+                             SupportsUpgradeDowngrade upgradeDowngradeHelper) {
+    try {
+      if (table.getMetaClient().getTableType() == HoodieTableType.MERGE_ON_READ) {
+        // The log block version has been upgraded in version six so compaction is required for downgrade.
+        // set required configs for scheduling compaction.
+        HoodieInstantTimeGenerator.setCommitTimeZone(table.getMetaClient().getTableConfig().getTimelineTimezone());
+        HoodieWriteConfig compactionConfig = HoodieWriteConfig.newBuilder().withProps(config.getProps()).build();
+        compactionConfig.setValue(HoodieCompactionConfig.INLINE_COMPACT.key(), "true");
+        compactionConfig.setValue(HoodieCompactionConfig.INLINE_COMPACT_NUM_DELTA_COMMITS.key(), "1");
+        compactionConfig.setValue(HoodieCompactionConfig.INLINE_COMPACT_TRIGGER_STRATEGY.key(), CompactionTriggerStrategy.NUM_COMMITS.name());
+        compactionConfig.setValue(HoodieCompactionConfig.COMPACTION_STRATEGY.key(), UnBoundedCompactionStrategy.class.getName());
+        compactionConfig.setValue(HoodieMetadataConfig.ENABLE.key(), "false");
+        EmbeddedTimelineServerHelper.createEmbeddedTimelineService(context, config);

Review Comment:
   Addressed. This was required earlier but not needed any more.



##########
hudi-client/hudi-client-common/src/main/java/org/apache/hudi/table/upgrade/SixToFiveDowngradeHandler.java:
##########
@@ -39,20 +47,26 @@
 
 import static org.apache.hudi.common.table.HoodieTableConfig.TABLE_METADATA_PARTITIONS;
 import static org.apache.hudi.common.table.HoodieTableConfig.TABLE_METADATA_PARTITIONS_INFLIGHT;
-import static org.apache.hudi.metadata.HoodieTableMetadataUtil.deleteMetadataTablePartition;
 
 /**
  * Downgrade handle to assist in downgrading hoodie table from version 6 to 5.
  * To ensure compatibility, we need recreate the compaction requested file to
  * .aux folder.
+ * Since version 6 includes a new schema field for metadata table(MDT),
+ * the MDT needs to be deleted during downgrade to avoid column drop error.
+ * Also log block version was upgraded in version 6, therefore full compaction needs
+ * to be completed during downgrade to avoid write failures.

Review Comment:
   Addressed
   



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [hudi] hudi-bot commented on pull request #9467: [HUDI-6717] Fix downgrade handler for 0.14.0

Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #9467:
URL: https://github.com/apache/hudi/pull/9467#issuecomment-1683464037

   <!--
   Meta data
   {
     "version" : 1,
     "metaDataEntries" : [ {
       "hash" : "2ade66c64355778bea62ef8ef81c80b929f50b3f",
       "status" : "FAILURE",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19339",
       "triggerID" : "2ade66c64355778bea62ef8ef81c80b929f50b3f",
       "triggerType" : "PUSH"
     }, {
       "hash" : "50d4aea5e545b5094368e3a192ffb5fd2008c481",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "50d4aea5e545b5094368e3a192ffb5fd2008c481",
       "triggerType" : "PUSH"
     } ]
   }-->
   ## CI report:
   
   * 2ade66c64355778bea62ef8ef81c80b929f50b3f Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19339) 
   * 50d4aea5e545b5094368e3a192ffb5fd2008c481 UNKNOWN
   
   <details>
   <summary>Bot commands</summary>
     @hudi-bot supports the following commands:
   
    - `@hudi-bot run azure` re-run the last Azure build
   </details>


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [hudi] hudi-bot commented on pull request #9467: [HUDI-6621] Fix downgrade handler for 0.14.0

Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #9467:
URL: https://github.com/apache/hudi/pull/9467#issuecomment-1684934537

   <!--
   Meta data
   {
     "version" : 1,
     "metaDataEntries" : [ {
       "hash" : "2ade66c64355778bea62ef8ef81c80b929f50b3f",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19339",
       "triggerID" : "2ade66c64355778bea62ef8ef81c80b929f50b3f",
       "triggerType" : "PUSH"
     }, {
       "hash" : "50d4aea5e545b5094368e3a192ffb5fd2008c481",
       "status" : "FAILURE",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19349",
       "triggerID" : "50d4aea5e545b5094368e3a192ffb5fd2008c481",
       "triggerType" : "PUSH"
     }, {
       "hash" : "d20d5b2e45e0eccf8f3ec40077696eecf9dfc4bb",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "d20d5b2e45e0eccf8f3ec40077696eecf9dfc4bb",
       "triggerType" : "PUSH"
     } ]
   }-->
   ## CI report:
   
   * 50d4aea5e545b5094368e3a192ffb5fd2008c481 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19349) 
   * d20d5b2e45e0eccf8f3ec40077696eecf9dfc4bb UNKNOWN
   
   <details>
   <summary>Bot commands</summary>
     @hudi-bot supports the following commands:
   
    - `@hudi-bot run azure` re-run the last Azure build
   </details>


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [hudi] hudi-bot commented on pull request #9467: [HUDI-6621] Fix downgrade handler for 0.14.0

Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #9467:
URL: https://github.com/apache/hudi/pull/9467#issuecomment-1685883384

   <!--
   Meta data
   {
     "version" : 1,
     "metaDataEntries" : [ {
       "hash" : "2ade66c64355778bea62ef8ef81c80b929f50b3f",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19339",
       "triggerID" : "2ade66c64355778bea62ef8ef81c80b929f50b3f",
       "triggerType" : "PUSH"
     }, {
       "hash" : "50d4aea5e545b5094368e3a192ffb5fd2008c481",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19349",
       "triggerID" : "50d4aea5e545b5094368e3a192ffb5fd2008c481",
       "triggerType" : "PUSH"
     }, {
       "hash" : "d20d5b2e45e0eccf8f3ec40077696eecf9dfc4bb",
       "status" : "FAILURE",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19368",
       "triggerID" : "d20d5b2e45e0eccf8f3ec40077696eecf9dfc4bb",
       "triggerType" : "PUSH"
     }, {
       "hash" : "6cba53453621aa518273a8879eeea3081f4c94cb",
       "status" : "PENDING",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19384",
       "triggerID" : "6cba53453621aa518273a8879eeea3081f4c94cb",
       "triggerType" : "PUSH"
     } ]
   }-->
   ## CI report:
   
   * d20d5b2e45e0eccf8f3ec40077696eecf9dfc4bb Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19368) 
   * 6cba53453621aa518273a8879eeea3081f4c94cb Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19384) 
   
   <details>
   <summary>Bot commands</summary>
     @hudi-bot supports the following commands:
   
    - `@hudi-bot run azure` re-run the last Azure build
   </details>


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org