You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@hudi.apache.org by GitBox <gi...@apache.org> on 2022/03/31 09:48:52 UTC

[GitHub] [hudi] zhangyue19921010 opened a new pull request #5186: [HUDI-3451] Add checks for metadata table init to avoid possible out-of-sync

zhangyue19921010 opened a new pull request #5186:
URL: https://github.com/apache/hudi/pull/5186


   please refer https://issues.apache.org/jira/browse/HUDI-3451 for details
   
   After this patch we will add a new config named `hoodie.populate.meta.fields` in hoodie.properties file.
   ```
   yuezhang-mac:.hoodie yuezhang$ cat hoodie.properties 
   #Updated at 2022-03-31T09:18:10.685Z
   #Thu Mar 31 17:18:10 CST 2022
   hoodie.table.precombine.field=ts
   hoodie.table.partition.fields=forecast_predict_date
   hoodie.table.type=COPY_ON_WRITE
   hoodie.archivelog.folder=archived
   hoodie.populate.meta.fields=true
   hoodie.timeline.layout.version=1
   hoodie.table.version=4
   hoodie.table.metadata.enable=true
   hoodie.table.recordkey.fields=hash
   hoodie.table.base.file.format=PARQUET
   hoodie.datasource.write.partitionpath.urlencode=false
   hoodie.database.name=
   hoodie.table.timeline.timezone=LOCAL
   hoodie.table.keygenerator.class=org.apache.hudi.keygen.ComplexKeyGenerator
   hoodie.table.name=forecast_agg
   hoodie.datasource.write.hive_style_partitioning=false
   hoodie.table.checksum=2008503694
   ```
   
   ## What is the purpose of the pull request
   
   *(For example: This pull request adds quick-start document.)*
   
   ## Brief change log
   
   *(for example:)*
     - *Modify AnnotationLocation checkstyle rule in checkstyle.xml*
   
   ## Verify this pull request
   
   *(Please pick either of the following options)*
   
   This pull request is a trivial rework / code cleanup without any test coverage.
   
   *(or)*
   
   This pull request is already covered by existing tests, such as *(please describe tests)*.
   
   (or)
   
   This change added tests and can be verified as follows:
   
   *(example:)*
   
     - *Added integration tests for end-to-end.*
     - *Added HoodieClientWriteTest to verify the change.*
     - *Manually verified the change by running a job locally.*
   
   ## Committer checklist
   
    - [ ] Has a corresponding JIRA in PR title & commit
    
    - [ ] Commit message is descriptive of the change
    
    - [ ] CI is green
   
    - [ ] Necessary doc changes done or have another open PR
          
    - [ ] For large changes, please consider breaking it into sub-tasks under an umbrella JIRA.
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [hudi] hudi-bot removed a comment on pull request #5186: [HUDI-3451] Add checks for metadata table init to avoid possible out-of-sync

Posted by GitBox <gi...@apache.org>.
hudi-bot removed a comment on pull request #5186:
URL: https://github.com/apache/hudi/pull/5186#issuecomment-1084350247


   <!--
   Meta data
   {
     "version" : 1,
     "metaDataEntries" : [ {
       "hash" : "5a1c9f13870f52750e25f8658776c0e9c27a4426",
       "status" : "PENDING",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=7632",
       "triggerID" : "5a1c9f13870f52750e25f8658776c0e9c27a4426",
       "triggerType" : "PUSH"
     } ]
   }-->
   ## CI report:
   
   * 5a1c9f13870f52750e25f8658776c0e9c27a4426 Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=7632) 
   
   <details>
   <summary>Bot commands</summary>
     @hudi-bot supports the following commands:
   
    - `@hudi-bot run azure` re-run the last Azure build
   </details>


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [hudi] hudi-bot removed a comment on pull request #5186: [HUDI-3451] Add checks for metadata table init to avoid possible out-of-sync

Posted by GitBox <gi...@apache.org>.
hudi-bot removed a comment on pull request #5186:
URL: https://github.com/apache/hudi/pull/5186#issuecomment-1084346921


   <!--
   Meta data
   {
     "version" : 1,
     "metaDataEntries" : [ {
       "hash" : "5a1c9f13870f52750e25f8658776c0e9c27a4426",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "5a1c9f13870f52750e25f8658776c0e9c27a4426",
       "triggerType" : "PUSH"
     } ]
   }-->
   ## CI report:
   
   * 5a1c9f13870f52750e25f8658776c0e9c27a4426 UNKNOWN
   
   <details>
   <summary>Bot commands</summary>
     @hudi-bot supports the following commands:
   
    - `@hudi-bot run azure` re-run the last Azure build
   </details>


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [hudi] hudi-bot removed a comment on pull request #5186: [HUDI-3451] Add checks for metadata table init to avoid possible out-of-sync

Posted by GitBox <gi...@apache.org>.
hudi-bot removed a comment on pull request #5186:
URL: https://github.com/apache/hudi/pull/5186#issuecomment-1085305276


   <!--
   Meta data
   {
     "version" : 1,
     "metaDataEntries" : [ {
       "hash" : "5a1c9f13870f52750e25f8658776c0e9c27a4426",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=7632",
       "triggerID" : "5a1c9f13870f52750e25f8658776c0e9c27a4426",
       "triggerType" : "PUSH"
     }, {
       "hash" : "82c4b1422058fd68b1d5843b1ab1386a96adbc64",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=7646",
       "triggerID" : "82c4b1422058fd68b1d5843b1ab1386a96adbc64",
       "triggerType" : "PUSH"
     }, {
       "hash" : "f3a50b13c1802ffd1c33f9469d2f7dcafde47bc2",
       "status" : "CANCELED",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=7647",
       "triggerID" : "f3a50b13c1802ffd1c33f9469d2f7dcafde47bc2",
       "triggerType" : "PUSH"
     }, {
       "hash" : "a7c6cd3eb6c08cf4d6a97e2c1981025493dac833",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "a7c6cd3eb6c08cf4d6a97e2c1981025493dac833",
       "triggerType" : "PUSH"
     } ]
   }-->
   ## CI report:
   
   * f3a50b13c1802ffd1c33f9469d2f7dcafde47bc2 Azure: [CANCELED](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=7647) 
   * a7c6cd3eb6c08cf4d6a97e2c1981025493dac833 UNKNOWN
   
   <details>
   <summary>Bot commands</summary>
     @hudi-bot supports the following commands:
   
    - `@hudi-bot run azure` re-run the last Azure build
   </details>


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [hudi] zhangyue19921010 commented on a change in pull request #5186: [HUDI-3451] Add checks for metadata table init to avoid possible out-of-sync

Posted by GitBox <gi...@apache.org>.
zhangyue19921010 commented on a change in pull request #5186:
URL: https://github.com/apache/hudi/pull/5186#discussion_r839721790



##########
File path: hudi-common/src/main/java/org/apache/hudi/common/table/HoodieTableConfig.java
##########
@@ -208,6 +209,11 @@
       .sinceVersion("0.11.0")
       .withDocumentation("Table checksum is used to guard against partial writes in HDFS. It is added as the last entry in hoodie.properties and then used to validate while reading table config.");
 
+  public static final ConfigProperty<Boolean> METADATA_TABLE_ENABLE = ConfigProperty
+      .key("hoodie.table.metadata.enable")
+      .defaultValue(HoodieMetadataConfig.ENABLE.defaultValue())
+      .withDocumentation("Enable the internal metadata table which serves table metadata like level file listings.");

Review comment:
       Sure, we need to let users know it. Just thinking maybe we can  let reader side to notice the config in hoodie.properties:
   1. if `METADATA_TABLE_ENABLE` false in hoodie.properties, no matter users set on reader, we will use FileSystemBased
   2. if `METADATA_TABLE_ENABLE` true in hoodie.properties, and users set MDT true, then we use MDT Based action?




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [hudi] hudi-bot commented on pull request #5186: [HUDI-3451] Add checks for metadata table init to avoid possible out-of-sync

Posted by GitBox <gi...@apache.org>.
hudi-bot commented on pull request #5186:
URL: https://github.com/apache/hudi/pull/5186#issuecomment-1086568410


   <!--
   Meta data
   {
     "version" : 1,
     "metaDataEntries" : [ {
       "hash" : "5a1c9f13870f52750e25f8658776c0e9c27a4426",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=7632",
       "triggerID" : "5a1c9f13870f52750e25f8658776c0e9c27a4426",
       "triggerType" : "PUSH"
     }, {
       "hash" : "82c4b1422058fd68b1d5843b1ab1386a96adbc64",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=7646",
       "triggerID" : "82c4b1422058fd68b1d5843b1ab1386a96adbc64",
       "triggerType" : "PUSH"
     }, {
       "hash" : "f3a50b13c1802ffd1c33f9469d2f7dcafde47bc2",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=7647",
       "triggerID" : "f3a50b13c1802ffd1c33f9469d2f7dcafde47bc2",
       "triggerType" : "PUSH"
     }, {
       "hash" : "a7c6cd3eb6c08cf4d6a97e2c1981025493dac833",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=7668",
       "triggerID" : "a7c6cd3eb6c08cf4d6a97e2c1981025493dac833",
       "triggerType" : "PUSH"
     }, {
       "hash" : "93866c7fb3fa7ff2b9bb978fe2bed922c28ad381",
       "status" : "SUCCESS",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=7675",
       "triggerID" : "93866c7fb3fa7ff2b9bb978fe2bed922c28ad381",
       "triggerType" : "PUSH"
     }, {
       "hash" : "4f6001bbde0a330c9cecb7783cf8999276701039",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "4f6001bbde0a330c9cecb7783cf8999276701039",
       "triggerType" : "PUSH"
     } ]
   }-->
   ## CI report:
   
   * 93866c7fb3fa7ff2b9bb978fe2bed922c28ad381 Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=7675) 
   * 4f6001bbde0a330c9cecb7783cf8999276701039 UNKNOWN
   
   <details>
   <summary>Bot commands</summary>
     @hudi-bot supports the following commands:
   
    - `@hudi-bot run azure` re-run the last Azure build
   </details>


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [hudi] hudi-bot commented on pull request #5186: [HUDI-3451] Delete metadata table when the write client disables MDT

Posted by GitBox <gi...@apache.org>.
hudi-bot commented on pull request #5186:
URL: https://github.com/apache/hudi/pull/5186#issuecomment-1086569967


   <!--
   Meta data
   {
     "version" : 1,
     "metaDataEntries" : [ {
       "hash" : "5a1c9f13870f52750e25f8658776c0e9c27a4426",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=7632",
       "triggerID" : "5a1c9f13870f52750e25f8658776c0e9c27a4426",
       "triggerType" : "PUSH"
     }, {
       "hash" : "82c4b1422058fd68b1d5843b1ab1386a96adbc64",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=7646",
       "triggerID" : "82c4b1422058fd68b1d5843b1ab1386a96adbc64",
       "triggerType" : "PUSH"
     }, {
       "hash" : "f3a50b13c1802ffd1c33f9469d2f7dcafde47bc2",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=7647",
       "triggerID" : "f3a50b13c1802ffd1c33f9469d2f7dcafde47bc2",
       "triggerType" : "PUSH"
     }, {
       "hash" : "a7c6cd3eb6c08cf4d6a97e2c1981025493dac833",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=7668",
       "triggerID" : "a7c6cd3eb6c08cf4d6a97e2c1981025493dac833",
       "triggerType" : "PUSH"
     }, {
       "hash" : "93866c7fb3fa7ff2b9bb978fe2bed922c28ad381",
       "status" : "SUCCESS",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=7675",
       "triggerID" : "93866c7fb3fa7ff2b9bb978fe2bed922c28ad381",
       "triggerType" : "PUSH"
     }, {
       "hash" : "4f6001bbde0a330c9cecb7783cf8999276701039",
       "status" : "PENDING",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=7744",
       "triggerID" : "4f6001bbde0a330c9cecb7783cf8999276701039",
       "triggerType" : "PUSH"
     }, {
       "hash" : "9aa5907f19bc0a023ee881f013c03f7abcf3bcf7",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "9aa5907f19bc0a023ee881f013c03f7abcf3bcf7",
       "triggerType" : "PUSH"
     } ]
   }-->
   ## CI report:
   
   * 93866c7fb3fa7ff2b9bb978fe2bed922c28ad381 Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=7675) 
   * 4f6001bbde0a330c9cecb7783cf8999276701039 Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=7744) 
   * 9aa5907f19bc0a023ee881f013c03f7abcf3bcf7 UNKNOWN
   
   <details>
   <summary>Bot commands</summary>
     @hudi-bot supports the following commands:
   
    - `@hudi-bot run azure` re-run the last Azure build
   </details>


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [hudi] yihua commented on pull request #5186: [HUDI-3451] Delete metadata table when the write client disables MDT

Posted by GitBox <gi...@apache.org>.
yihua commented on pull request #5186:
URL: https://github.com/apache/hudi/pull/5186#issuecomment-1086571780


   @zhangyue19921010 @codope I revised the PR.  PTAL.  Per discussion, we should keep the design simple here, and advise users not to flip the `hoodie.metadata.enable` on and off frequently, making sure they understand that the metadata table must be fully deleted before re-enabling MDT again.  The docs update to advise users on this is tracked in HUDI-3779.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [hudi] codope commented on a change in pull request #5186: [HUDI-3451] Add checks for metadata table init to avoid possible out-of-sync

Posted by GitBox <gi...@apache.org>.
codope commented on a change in pull request #5186:
URL: https://github.com/apache/hudi/pull/5186#discussion_r839909753



##########
File path: hudi-common/src/main/java/org/apache/hudi/common/table/HoodieTableConfig.java
##########
@@ -208,6 +209,11 @@
       .sinceVersion("0.11.0")
       .withDocumentation("Table checksum is used to guard against partial writes in HDFS. It is added as the last entry in hoodie.properties and then used to validate while reading table config.");
 
+  public static final ConfigProperty<Boolean> METADATA_TABLE_ENABLE = ConfigProperty
+      .key("hoodie.table.metadata.enable")
+      .defaultValue(HoodieMetadataConfig.ENABLE.defaultValue())
+      .withDocumentation("Enable the internal metadata table which serves table metadata like level file listings.");

Review comment:
       And what if  METADATA_TABLE_ENABLE is not present in the config?




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [hudi] zhangyue19921010 commented on a change in pull request #5186: [HUDI-3451] Add checks for metadata table init to avoid possible out-of-sync

Posted by GitBox <gi...@apache.org>.
zhangyue19921010 commented on a change in pull request #5186:
URL: https://github.com/apache/hudi/pull/5186#discussion_r839712993



##########
File path: hudi-client/hudi-spark-client/src/test/java/org/apache/hudi/client/functional/TestHoodieBackedMetadata.java
##########
@@ -197,6 +197,41 @@ public void testMetadataTableBootstrap(HoodieTableType tableType, boolean addRol
     validateMetadata(testTable, true);
   }
 
+  @Test
+  public void testTurnOffMetadataTableAfterEnable() throws Exception {
+    init(COPY_ON_WRITE, true);
+    String instant1 = "0000001";
+    HoodieCommitMetadata hoodieCommitMetadata = doWriteOperationWithMeta(testTable, instant1, INSERT);
+
+    // sync to metadata table
+    HoodieTable table = HoodieSparkTable.create(writeConfig, context, metaClient);
+    Option metadataWriter = table.getMetadataWriter(instant1, Option.of(hoodieCommitMetadata));
+    validateMetadata(testTable, true);
+
+    assertTrue(metadataWriter.isPresent());
+    HoodieTableConfig hoodieTableConfig = new HoodieTableConfig(fs, metaClient.getMetaPath(), writeConfig.getPayloadClass());
+    assertTrue(hoodieTableConfig.getMetadataTableEnable());
+
+    // Turn off  enableMetadata Table
+    HoodieWriteConfig writeConfig2 = HoodieWriteConfig.newBuilder()
+        .withProperties(this.writeConfig.getProps())
+        .withMetadataConfig(HoodieMetadataConfig.newBuilder().enable(false).build())
+        .build();
+
+    metaClient.reloadActiveTimeline();
+    HoodieTable table2 = HoodieSparkTable.create(writeConfig2, context, metaClient);
+    String instant2 = "0000002";
+    HoodieCommitMetadata hoodieCommitMetadata2 = doWriteOperationWithMeta(testTable, instant2, INSERT);
+    Option metadataWriter2 = table2.getMetadataWriter(instant2, Option.of(hoodieCommitMetadata2));
+    assertFalse(metadataWriter2.isPresent());
+    HoodieTableConfig hoodieTableConfig2 = new HoodieTableConfig(fs, metaClient.getMetaPath(), writeConfig2.getPayloadClass());
+
+    // assert hoodie.table.metadata.enable=false
+    assertFalse(hoodieTableConfig2.getMetadataTableEnable());
+    // assert MDT is deleted.
+    assertFalse(metaClient.getFs().exists(new Path(HoodieTableMetadata.getMetadataTableBasePath(writeConfig2.getBasePath()))));

Review comment:
       Sure. Changed.




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [hudi] zhangyue19921010 commented on a change in pull request #5186: [HUDI-3451] Add checks for metadata table init to avoid possible out-of-sync

Posted by GitBox <gi...@apache.org>.
zhangyue19921010 commented on a change in pull request #5186:
URL: https://github.com/apache/hudi/pull/5186#discussion_r840154030



##########
File path: hudi-common/src/main/java/org/apache/hudi/common/table/HoodieTableConfig.java
##########
@@ -208,6 +209,11 @@
       .sinceVersion("0.11.0")
       .withDocumentation("Table checksum is used to guard against partial writes in HDFS. It is added as the last entry in hoodie.properties and then used to validate while reading table config.");
 
+  public static final ConfigProperty<Boolean> METADATA_TABLE_ENABLE = ConfigProperty
+      .key("hoodie.table.metadata.enable")
+      .defaultValue(HoodieMetadataConfig.ENABLE.defaultValue())
+      .withDocumentation("Enable the internal metadata table which serves table metadata like level file listings.");

Review comment:
       3. if `METADATA_TABLE_ENABLE` is absent, we could read based on user's option.




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [hudi] hudi-bot removed a comment on pull request #5186: [HUDI-3451] Add checks for metadata table init to avoid possible out-of-sync

Posted by GitBox <gi...@apache.org>.
hudi-bot removed a comment on pull request #5186:
URL: https://github.com/apache/hudi/pull/5186#issuecomment-1086568410


   <!--
   Meta data
   {
     "version" : 1,
     "metaDataEntries" : [ {
       "hash" : "5a1c9f13870f52750e25f8658776c0e9c27a4426",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=7632",
       "triggerID" : "5a1c9f13870f52750e25f8658776c0e9c27a4426",
       "triggerType" : "PUSH"
     }, {
       "hash" : "82c4b1422058fd68b1d5843b1ab1386a96adbc64",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=7646",
       "triggerID" : "82c4b1422058fd68b1d5843b1ab1386a96adbc64",
       "triggerType" : "PUSH"
     }, {
       "hash" : "f3a50b13c1802ffd1c33f9469d2f7dcafde47bc2",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=7647",
       "triggerID" : "f3a50b13c1802ffd1c33f9469d2f7dcafde47bc2",
       "triggerType" : "PUSH"
     }, {
       "hash" : "a7c6cd3eb6c08cf4d6a97e2c1981025493dac833",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=7668",
       "triggerID" : "a7c6cd3eb6c08cf4d6a97e2c1981025493dac833",
       "triggerType" : "PUSH"
     }, {
       "hash" : "93866c7fb3fa7ff2b9bb978fe2bed922c28ad381",
       "status" : "SUCCESS",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=7675",
       "triggerID" : "93866c7fb3fa7ff2b9bb978fe2bed922c28ad381",
       "triggerType" : "PUSH"
     }, {
       "hash" : "4f6001bbde0a330c9cecb7783cf8999276701039",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "4f6001bbde0a330c9cecb7783cf8999276701039",
       "triggerType" : "PUSH"
     } ]
   }-->
   ## CI report:
   
   * 93866c7fb3fa7ff2b9bb978fe2bed922c28ad381 Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=7675) 
   * 4f6001bbde0a330c9cecb7783cf8999276701039 UNKNOWN
   
   <details>
   <summary>Bot commands</summary>
     @hudi-bot supports the following commands:
   
    - `@hudi-bot run azure` re-run the last Azure build
   </details>


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [hudi] hudi-bot commented on pull request #5186: [HUDI-3451] Delete metadata table when the write client disables MDT

Posted by GitBox <gi...@apache.org>.
hudi-bot commented on pull request #5186:
URL: https://github.com/apache/hudi/pull/5186#issuecomment-1086581799


   <!--
   Meta data
   {
     "version" : 1,
     "metaDataEntries" : [ {
       "hash" : "5a1c9f13870f52750e25f8658776c0e9c27a4426",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=7632",
       "triggerID" : "5a1c9f13870f52750e25f8658776c0e9c27a4426",
       "triggerType" : "PUSH"
     }, {
       "hash" : "82c4b1422058fd68b1d5843b1ab1386a96adbc64",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=7646",
       "triggerID" : "82c4b1422058fd68b1d5843b1ab1386a96adbc64",
       "triggerType" : "PUSH"
     }, {
       "hash" : "f3a50b13c1802ffd1c33f9469d2f7dcafde47bc2",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=7647",
       "triggerID" : "f3a50b13c1802ffd1c33f9469d2f7dcafde47bc2",
       "triggerType" : "PUSH"
     }, {
       "hash" : "a7c6cd3eb6c08cf4d6a97e2c1981025493dac833",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=7668",
       "triggerID" : "a7c6cd3eb6c08cf4d6a97e2c1981025493dac833",
       "triggerType" : "PUSH"
     }, {
       "hash" : "93866c7fb3fa7ff2b9bb978fe2bed922c28ad381",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=7675",
       "triggerID" : "93866c7fb3fa7ff2b9bb978fe2bed922c28ad381",
       "triggerType" : "PUSH"
     }, {
       "hash" : "4f6001bbde0a330c9cecb7783cf8999276701039",
       "status" : "CANCELED",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=7744",
       "triggerID" : "4f6001bbde0a330c9cecb7783cf8999276701039",
       "triggerType" : "PUSH"
     }, {
       "hash" : "9aa5907f19bc0a023ee881f013c03f7abcf3bcf7",
       "status" : "PENDING",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=7747",
       "triggerID" : "9aa5907f19bc0a023ee881f013c03f7abcf3bcf7",
       "triggerType" : "PUSH"
     } ]
   }-->
   ## CI report:
   
   * 4f6001bbde0a330c9cecb7783cf8999276701039 Azure: [CANCELED](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=7744) 
   * 9aa5907f19bc0a023ee881f013c03f7abcf3bcf7 Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=7747) 
   
   <details>
   <summary>Bot commands</summary>
     @hudi-bot supports the following commands:
   
    - `@hudi-bot run azure` re-run the last Azure build
   </details>


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [hudi] zhangyue19921010 commented on a change in pull request #5186: [HUDI-3451] Add checks for metadata table init to avoid possible out-of-sync

Posted by GitBox <gi...@apache.org>.
zhangyue19921010 commented on a change in pull request #5186:
URL: https://github.com/apache/hudi/pull/5186#discussion_r839701369



##########
File path: hudi-client/hudi-client-common/src/main/java/org/apache/hudi/table/HoodieTable.java
##########
@@ -781,6 +782,53 @@ public HoodieEngineContext getContext() {
     return Option.empty();
   }
 
+  /**
+   * Initialize hoodie.table.metadata.enable in hoodie.proterties;

Review comment:
       Changed.




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [hudi] hudi-bot commented on pull request #5186: [HUDI-3451] Add checks for metadata table init to avoid possible out-of-sync

Posted by GitBox <gi...@apache.org>.
hudi-bot commented on pull request #5186:
URL: https://github.com/apache/hudi/pull/5186#issuecomment-1084346921


   <!--
   Meta data
   {
     "version" : 1,
     "metaDataEntries" : [ {
       "hash" : "5a1c9f13870f52750e25f8658776c0e9c27a4426",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "5a1c9f13870f52750e25f8658776c0e9c27a4426",
       "triggerType" : "PUSH"
     } ]
   }-->
   ## CI report:
   
   * 5a1c9f13870f52750e25f8658776c0e9c27a4426 UNKNOWN
   
   <details>
   <summary>Bot commands</summary>
     @hudi-bot supports the following commands:
   
    - `@hudi-bot run azure` re-run the last Azure build
   </details>


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [hudi] hudi-bot removed a comment on pull request #5186: [HUDI-3451] Add checks for metadata table init to avoid possible out-of-sync

Posted by GitBox <gi...@apache.org>.
hudi-bot removed a comment on pull request #5186:
URL: https://github.com/apache/hudi/pull/5186#issuecomment-1084728869


   <!--
   Meta data
   {
     "version" : 1,
     "metaDataEntries" : [ {
       "hash" : "5a1c9f13870f52750e25f8658776c0e9c27a4426",
       "status" : "SUCCESS",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=7632",
       "triggerID" : "5a1c9f13870f52750e25f8658776c0e9c27a4426",
       "triggerType" : "PUSH"
     }, {
       "hash" : "82c4b1422058fd68b1d5843b1ab1386a96adbc64",
       "status" : "PENDING",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=7646",
       "triggerID" : "82c4b1422058fd68b1d5843b1ab1386a96adbc64",
       "triggerType" : "PUSH"
     }, {
       "hash" : "f3a50b13c1802ffd1c33f9469d2f7dcafde47bc2",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "f3a50b13c1802ffd1c33f9469d2f7dcafde47bc2",
       "triggerType" : "PUSH"
     } ]
   }-->
   ## CI report:
   
   * 5a1c9f13870f52750e25f8658776c0e9c27a4426 Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=7632) 
   * 82c4b1422058fd68b1d5843b1ab1386a96adbc64 Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=7646) 
   * f3a50b13c1802ffd1c33f9469d2f7dcafde47bc2 UNKNOWN
   
   <details>
   <summary>Bot commands</summary>
     @hudi-bot supports the following commands:
   
    - `@hudi-bot run azure` re-run the last Azure build
   </details>


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [hudi] codope commented on a change in pull request #5186: [HUDI-3451] Add checks for metadata table init to avoid possible out-of-sync

Posted by GitBox <gi...@apache.org>.
codope commented on a change in pull request #5186:
URL: https://github.com/apache/hudi/pull/5186#discussion_r839513188



##########
File path: hudi-common/src/main/java/org/apache/hudi/common/table/HoodieTableConfig.java
##########
@@ -208,6 +209,11 @@
       .sinceVersion("0.11.0")
       .withDocumentation("Table checksum is used to guard against partial writes in HDFS. It is added as the last entry in hoodie.properties and then used to validate while reading table config.");
 
+  public static final ConfigProperty<Boolean> METADATA_TABLE_ENABLE = ConfigProperty

Review comment:
       Are we sure there is no need to handle this config in upgrade step? Would it be better to add this config in hoodie.properties when users upgrade with an existing metadata table?

##########
File path: hudi-client/hudi-spark-client/src/test/java/org/apache/hudi/client/functional/TestHoodieBackedMetadata.java
##########
@@ -197,6 +197,41 @@ public void testMetadataTableBootstrap(HoodieTableType tableType, boolean addRol
     validateMetadata(testTable, true);
   }
 
+  @Test
+  public void testTurnOffMetadataTableAfterEnable() throws Exception {
+    init(COPY_ON_WRITE, true);
+    String instant1 = "0000001";
+    HoodieCommitMetadata hoodieCommitMetadata = doWriteOperationWithMeta(testTable, instant1, INSERT);
+
+    // sync to metadata table
+    HoodieTable table = HoodieSparkTable.create(writeConfig, context, metaClient);
+    Option metadataWriter = table.getMetadataWriter(instant1, Option.of(hoodieCommitMetadata));
+    validateMetadata(testTable, true);
+
+    assertTrue(metadataWriter.isPresent());
+    HoodieTableConfig hoodieTableConfig = new HoodieTableConfig(fs, metaClient.getMetaPath(), writeConfig.getPayloadClass());
+    assertTrue(hoodieTableConfig.getMetadataTableEnable());
+
+    // Turn off  enableMetadata Table
+    HoodieWriteConfig writeConfig2 = HoodieWriteConfig.newBuilder()
+        .withProperties(this.writeConfig.getProps())
+        .withMetadataConfig(HoodieMetadataConfig.newBuilder().enable(false).build())
+        .build();
+
+    metaClient.reloadActiveTimeline();
+    HoodieTable table2 = HoodieSparkTable.create(writeConfig2, context, metaClient);
+    String instant2 = "0000002";
+    HoodieCommitMetadata hoodieCommitMetadata2 = doWriteOperationWithMeta(testTable, instant2, INSERT);
+    Option metadataWriter2 = table2.getMetadataWriter(instant2, Option.of(hoodieCommitMetadata2));
+    assertFalse(metadataWriter2.isPresent());
+    HoodieTableConfig hoodieTableConfig2 = new HoodieTableConfig(fs, metaClient.getMetaPath(), writeConfig2.getPayloadClass());
+
+    // assert hoodie.table.metadata.enable=false
+    assertFalse(hoodieTableConfig2.getMetadataTableEnable());
+    // assert MDT is deleted.
+    assertFalse(metaClient.getFs().exists(new Path(HoodieTableMetadata.getMetadataTableBasePath(writeConfig2.getBasePath()))));

Review comment:
       let's enable again after disable and validate that confg gets updated.

##########
File path: hudi-client/hudi-client-common/src/main/java/org/apache/hudi/table/HoodieTable.java
##########
@@ -781,6 +782,53 @@ public HoodieEngineContext getContext() {
     return Option.empty();
   }
 
+  /**
+   * Initialize hoodie.table.metadata.enable in hoodie.proterties;
+   * Check if current metadata table flag in hoodieWriteConfig is the same as recorded in hoodie.proterties:
+   *  1. If the flag in hoodie.proterties is true but it is false in hoodieWriteConfig, It means users turn off MDT and we need to clean up MDT.
+   *  2. Update hoodie.table.metadata.enable in hoodie.proterties based on HoodieWriteConfig if the value is different.
+   */
+  protected void verifyMetadataTableInTableConfig() {
+    // ignore the hoodie.properties in MDT
+    if (metaClient.getBasePath().contains(HoodieTableMetaClient.METADATA_TABLE_FOLDER_PATH)) {
+      return;
+    }
+
+    try {
+      if (metaClient.getTableConfig().contains(HoodieTableConfig.METADATA_TABLE_ENABLE)) {
+        Boolean enableMDTInTableConfig = metaClient.getTableConfig().getBoolean(HoodieTableConfig.METADATA_TABLE_ENABLE);
+        Path mdtBasePath = new Path(HoodieTableMetadata.getMetadataTableBasePath(config.getBasePath()));
+        // MDT flag in TableConfig is true;
+        // MDT flag in write config is false;
+        // ===> Users turn off MDT, so that we need to clean up this metadata table in case out-of-sync issue.
+        // the condition order in if(xx) is important.
+        if (enableMDTInTableConfig && !config.isMetadataTableEnabled() && metaClient.getFs().exists(mdtBasePath)) {
+          LOG.info("Deleting metadata table because of disabled in writer.");
+          metaClient.getFs().delete(mdtBasePath, true);
+        }
+
+        // update table config if necessary
+        if (Boolean.compare(enableMDTInTableConfig, config.isMetadataTableEnabled()) != 0) {
+          updateMetadataTableEnableInTableConfig();
+        }
+      } else {
+        // initial METADATA_TABLE_ENABLE.key in table config
+        updateMetadataTableEnableInTableConfig();

Review comment:
       When: 
   a) table config does not contain `METADATA_TABLE_ENABLE`,
   b) MDT is disabled in write config,
   c) but MDT exists on the filesystem, 
   then shouldn't we delete it? 
   This can happen users with initially MDT enabled upgrade to the current version with MDT disabled. Isn't it?

##########
File path: hudi-client/hudi-client-common/src/main/java/org/apache/hudi/table/HoodieTable.java
##########
@@ -781,6 +782,53 @@ public HoodieEngineContext getContext() {
     return Option.empty();
   }
 
+  /**
+   * Initialize hoodie.table.metadata.enable in hoodie.proterties;

Review comment:
       typo: hoodie.pro**p**erties

##########
File path: hudi-common/src/main/java/org/apache/hudi/common/table/HoodieTableConfig.java
##########
@@ -208,6 +209,11 @@
       .sinceVersion("0.11.0")
       .withDocumentation("Table checksum is used to guard against partial writes in HDFS. It is added as the last entry in hoodie.properties and then used to validate while reading table config.");
 
+  public static final ConfigProperty<Boolean> METADATA_TABLE_ENABLE = ConfigProperty
+      .key("hoodie.table.metadata.enable")
+      .defaultValue(HoodieMetadataConfig.ENABLE.defaultValue())
+      .withDocumentation("Enable the internal metadata table which serves table metadata like level file listings.");

Review comment:
       How does the reader path look like? What if MDT is disabled in write config but it was true for reader? Do we need to call out something in the doc that if MDT is disabled in write config then queries will not make use of MDT irrespective of reader MDT config is enabled or not?




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [hudi] hudi-bot removed a comment on pull request #5186: [HUDI-3451] Add checks for metadata table init to avoid possible out-of-sync

Posted by GitBox <gi...@apache.org>.
hudi-bot removed a comment on pull request #5186:
URL: https://github.com/apache/hudi/pull/5186#issuecomment-1085442822


   <!--
   Meta data
   {
     "version" : 1,
     "metaDataEntries" : [ {
       "hash" : "5a1c9f13870f52750e25f8658776c0e9c27a4426",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=7632",
       "triggerID" : "5a1c9f13870f52750e25f8658776c0e9c27a4426",
       "triggerType" : "PUSH"
     }, {
       "hash" : "82c4b1422058fd68b1d5843b1ab1386a96adbc64",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=7646",
       "triggerID" : "82c4b1422058fd68b1d5843b1ab1386a96adbc64",
       "triggerType" : "PUSH"
     }, {
       "hash" : "f3a50b13c1802ffd1c33f9469d2f7dcafde47bc2",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=7647",
       "triggerID" : "f3a50b13c1802ffd1c33f9469d2f7dcafde47bc2",
       "triggerType" : "PUSH"
     }, {
       "hash" : "a7c6cd3eb6c08cf4d6a97e2c1981025493dac833",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=7668",
       "triggerID" : "a7c6cd3eb6c08cf4d6a97e2c1981025493dac833",
       "triggerType" : "PUSH"
     }, {
       "hash" : "93866c7fb3fa7ff2b9bb978fe2bed922c28ad381",
       "status" : "SUCCESS",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=7675",
       "triggerID" : "93866c7fb3fa7ff2b9bb978fe2bed922c28ad381",
       "triggerType" : "PUSH"
     } ]
   }-->
   ## CI report:
   
   * 93866c7fb3fa7ff2b9bb978fe2bed922c28ad381 Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=7675) 
   
   <details>
   <summary>Bot commands</summary>
     @hudi-bot supports the following commands:
   
    - `@hudi-bot run azure` re-run the last Azure build
   </details>


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [hudi] hudi-bot commented on pull request #5186: [HUDI-3451] Add checks for metadata table init to avoid possible out-of-sync

Posted by GitBox <gi...@apache.org>.
hudi-bot commented on pull request #5186:
URL: https://github.com/apache/hudi/pull/5186#issuecomment-1084780589


   <!--
   Meta data
   {
     "version" : 1,
     "metaDataEntries" : [ {
       "hash" : "5a1c9f13870f52750e25f8658776c0e9c27a4426",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=7632",
       "triggerID" : "5a1c9f13870f52750e25f8658776c0e9c27a4426",
       "triggerType" : "PUSH"
     }, {
       "hash" : "82c4b1422058fd68b1d5843b1ab1386a96adbc64",
       "status" : "CANCELED",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=7646",
       "triggerID" : "82c4b1422058fd68b1d5843b1ab1386a96adbc64",
       "triggerType" : "PUSH"
     }, {
       "hash" : "f3a50b13c1802ffd1c33f9469d2f7dcafde47bc2",
       "status" : "PENDING",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=7647",
       "triggerID" : "f3a50b13c1802ffd1c33f9469d2f7dcafde47bc2",
       "triggerType" : "PUSH"
     } ]
   }-->
   ## CI report:
   
   * 82c4b1422058fd68b1d5843b1ab1386a96adbc64 Azure: [CANCELED](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=7646) 
   * f3a50b13c1802ffd1c33f9469d2f7dcafde47bc2 Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=7647) 
   
   <details>
   <summary>Bot commands</summary>
     @hudi-bot supports the following commands:
   
    - `@hudi-bot run azure` re-run the last Azure build
   </details>


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [hudi] hudi-bot removed a comment on pull request #5186: [HUDI-3451] Add checks for metadata table init to avoid possible out-of-sync

Posted by GitBox <gi...@apache.org>.
hudi-bot removed a comment on pull request #5186:
URL: https://github.com/apache/hudi/pull/5186#issuecomment-1084732506


   <!--
   Meta data
   {
     "version" : 1,
     "metaDataEntries" : [ {
       "hash" : "5a1c9f13870f52750e25f8658776c0e9c27a4426",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=7632",
       "triggerID" : "5a1c9f13870f52750e25f8658776c0e9c27a4426",
       "triggerType" : "PUSH"
     }, {
       "hash" : "82c4b1422058fd68b1d5843b1ab1386a96adbc64",
       "status" : "CANCELED",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=7646",
       "triggerID" : "82c4b1422058fd68b1d5843b1ab1386a96adbc64",
       "triggerType" : "PUSH"
     }, {
       "hash" : "f3a50b13c1802ffd1c33f9469d2f7dcafde47bc2",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "f3a50b13c1802ffd1c33f9469d2f7dcafde47bc2",
       "triggerType" : "PUSH"
     } ]
   }-->
   ## CI report:
   
   * 82c4b1422058fd68b1d5843b1ab1386a96adbc64 Azure: [CANCELED](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=7646) 
   * f3a50b13c1802ffd1c33f9469d2f7dcafde47bc2 UNKNOWN
   
   <details>
   <summary>Bot commands</summary>
     @hudi-bot supports the following commands:
   
    - `@hudi-bot run azure` re-run the last Azure build
   </details>


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [hudi] hudi-bot removed a comment on pull request #5186: [HUDI-3451] Add checks for metadata table init to avoid possible out-of-sync

Posted by GitBox <gi...@apache.org>.
hudi-bot removed a comment on pull request #5186:
URL: https://github.com/apache/hudi/pull/5186#issuecomment-1084636242


   <!--
   Meta data
   {
     "version" : 1,
     "metaDataEntries" : [ {
       "hash" : "5a1c9f13870f52750e25f8658776c0e9c27a4426",
       "status" : "SUCCESS",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=7632",
       "triggerID" : "5a1c9f13870f52750e25f8658776c0e9c27a4426",
       "triggerType" : "PUSH"
     } ]
   }-->
   ## CI report:
   
   * 5a1c9f13870f52750e25f8658776c0e9c27a4426 Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=7632) 
   
   <details>
   <summary>Bot commands</summary>
     @hudi-bot supports the following commands:
   
    - `@hudi-bot run azure` re-run the last Azure build
   </details>


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [hudi] hudi-bot removed a comment on pull request #5186: [HUDI-3451] Add checks for metadata table init to avoid possible out-of-sync

Posted by GitBox <gi...@apache.org>.
hudi-bot removed a comment on pull request #5186:
URL: https://github.com/apache/hudi/pull/5186#issuecomment-1084780589


   <!--
   Meta data
   {
     "version" : 1,
     "metaDataEntries" : [ {
       "hash" : "5a1c9f13870f52750e25f8658776c0e9c27a4426",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=7632",
       "triggerID" : "5a1c9f13870f52750e25f8658776c0e9c27a4426",
       "triggerType" : "PUSH"
     }, {
       "hash" : "82c4b1422058fd68b1d5843b1ab1386a96adbc64",
       "status" : "CANCELED",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=7646",
       "triggerID" : "82c4b1422058fd68b1d5843b1ab1386a96adbc64",
       "triggerType" : "PUSH"
     }, {
       "hash" : "f3a50b13c1802ffd1c33f9469d2f7dcafde47bc2",
       "status" : "PENDING",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=7647",
       "triggerID" : "f3a50b13c1802ffd1c33f9469d2f7dcafde47bc2",
       "triggerType" : "PUSH"
     } ]
   }-->
   ## CI report:
   
   * 82c4b1422058fd68b1d5843b1ab1386a96adbc64 Azure: [CANCELED](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=7646) 
   * f3a50b13c1802ffd1c33f9469d2f7dcafde47bc2 Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=7647) 
   
   <details>
   <summary>Bot commands</summary>
     @hudi-bot supports the following commands:
   
    - `@hudi-bot run azure` re-run the last Azure build
   </details>


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [hudi] zhangyue19921010 commented on a change in pull request #5186: [HUDI-3451] Add checks for metadata table init to avoid possible out-of-sync

Posted by GitBox <gi...@apache.org>.
zhangyue19921010 commented on a change in pull request #5186:
URL: https://github.com/apache/hudi/pull/5186#discussion_r840154030



##########
File path: hudi-common/src/main/java/org/apache/hudi/common/table/HoodieTableConfig.java
##########
@@ -208,6 +209,11 @@
       .sinceVersion("0.11.0")
       .withDocumentation("Table checksum is used to guard against partial writes in HDFS. It is added as the last entry in hoodie.properties and then used to validate while reading table config.");
 
+  public static final ConfigProperty<Boolean> METADATA_TABLE_ENABLE = ConfigProperty
+      .key("hoodie.table.metadata.enable")
+      .defaultValue(HoodieMetadataConfig.ENABLE.defaultValue())
+      .withDocumentation("Enable the internal metadata table which serves table metadata like level file listings.");

Review comment:
       3. if `METADATA_TABLE_ENABLE` is absent, maybe we could read based on user's option?




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [hudi] hudi-bot removed a comment on pull request #5186: [HUDI-3451] Add checks for metadata table init to avoid possible out-of-sync

Posted by GitBox <gi...@apache.org>.
hudi-bot removed a comment on pull request #5186:
URL: https://github.com/apache/hudi/pull/5186#issuecomment-1085387394


   <!--
   Meta data
   {
     "version" : 1,
     "metaDataEntries" : [ {
       "hash" : "5a1c9f13870f52750e25f8658776c0e9c27a4426",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=7632",
       "triggerID" : "5a1c9f13870f52750e25f8658776c0e9c27a4426",
       "triggerType" : "PUSH"
     }, {
       "hash" : "82c4b1422058fd68b1d5843b1ab1386a96adbc64",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=7646",
       "triggerID" : "82c4b1422058fd68b1d5843b1ab1386a96adbc64",
       "triggerType" : "PUSH"
     }, {
       "hash" : "f3a50b13c1802ffd1c33f9469d2f7dcafde47bc2",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=7647",
       "triggerID" : "f3a50b13c1802ffd1c33f9469d2f7dcafde47bc2",
       "triggerType" : "PUSH"
     }, {
       "hash" : "a7c6cd3eb6c08cf4d6a97e2c1981025493dac833",
       "status" : "SUCCESS",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=7668",
       "triggerID" : "a7c6cd3eb6c08cf4d6a97e2c1981025493dac833",
       "triggerType" : "PUSH"
     }, {
       "hash" : "93866c7fb3fa7ff2b9bb978fe2bed922c28ad381",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "93866c7fb3fa7ff2b9bb978fe2bed922c28ad381",
       "triggerType" : "PUSH"
     } ]
   }-->
   ## CI report:
   
   * a7c6cd3eb6c08cf4d6a97e2c1981025493dac833 Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=7668) 
   * 93866c7fb3fa7ff2b9bb978fe2bed922c28ad381 UNKNOWN
   
   <details>
   <summary>Bot commands</summary>
     @hudi-bot supports the following commands:
   
    - `@hudi-bot run azure` re-run the last Azure build
   </details>


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [hudi] hudi-bot commented on pull request #5186: [HUDI-3451] Delete metadata table when the write client disables MDT

Posted by GitBox <gi...@apache.org>.
hudi-bot commented on pull request #5186:
URL: https://github.com/apache/hudi/pull/5186#issuecomment-1086614418


   <!--
   Meta data
   {
     "version" : 1,
     "metaDataEntries" : [ {
       "hash" : "5a1c9f13870f52750e25f8658776c0e9c27a4426",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=7632",
       "triggerID" : "5a1c9f13870f52750e25f8658776c0e9c27a4426",
       "triggerType" : "PUSH"
     }, {
       "hash" : "82c4b1422058fd68b1d5843b1ab1386a96adbc64",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=7646",
       "triggerID" : "82c4b1422058fd68b1d5843b1ab1386a96adbc64",
       "triggerType" : "PUSH"
     }, {
       "hash" : "f3a50b13c1802ffd1c33f9469d2f7dcafde47bc2",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=7647",
       "triggerID" : "f3a50b13c1802ffd1c33f9469d2f7dcafde47bc2",
       "triggerType" : "PUSH"
     }, {
       "hash" : "a7c6cd3eb6c08cf4d6a97e2c1981025493dac833",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=7668",
       "triggerID" : "a7c6cd3eb6c08cf4d6a97e2c1981025493dac833",
       "triggerType" : "PUSH"
     }, {
       "hash" : "93866c7fb3fa7ff2b9bb978fe2bed922c28ad381",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=7675",
       "triggerID" : "93866c7fb3fa7ff2b9bb978fe2bed922c28ad381",
       "triggerType" : "PUSH"
     }, {
       "hash" : "4f6001bbde0a330c9cecb7783cf8999276701039",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=7744",
       "triggerID" : "4f6001bbde0a330c9cecb7783cf8999276701039",
       "triggerType" : "PUSH"
     }, {
       "hash" : "9aa5907f19bc0a023ee881f013c03f7abcf3bcf7",
       "status" : "SUCCESS",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=7747",
       "triggerID" : "9aa5907f19bc0a023ee881f013c03f7abcf3bcf7",
       "triggerType" : "PUSH"
     } ]
   }-->
   ## CI report:
   
   * 9aa5907f19bc0a023ee881f013c03f7abcf3bcf7 Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=7747) 
   
   <details>
   <summary>Bot commands</summary>
     @hudi-bot supports the following commands:
   
    - `@hudi-bot run azure` re-run the last Azure build
   </details>


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [hudi] hudi-bot commented on pull request #5186: [HUDI-3451] Add checks for metadata table init to avoid possible out-of-sync

Posted by GitBox <gi...@apache.org>.
hudi-bot commented on pull request #5186:
URL: https://github.com/apache/hudi/pull/5186#issuecomment-1085387394


   <!--
   Meta data
   {
     "version" : 1,
     "metaDataEntries" : [ {
       "hash" : "5a1c9f13870f52750e25f8658776c0e9c27a4426",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=7632",
       "triggerID" : "5a1c9f13870f52750e25f8658776c0e9c27a4426",
       "triggerType" : "PUSH"
     }, {
       "hash" : "82c4b1422058fd68b1d5843b1ab1386a96adbc64",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=7646",
       "triggerID" : "82c4b1422058fd68b1d5843b1ab1386a96adbc64",
       "triggerType" : "PUSH"
     }, {
       "hash" : "f3a50b13c1802ffd1c33f9469d2f7dcafde47bc2",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=7647",
       "triggerID" : "f3a50b13c1802ffd1c33f9469d2f7dcafde47bc2",
       "triggerType" : "PUSH"
     }, {
       "hash" : "a7c6cd3eb6c08cf4d6a97e2c1981025493dac833",
       "status" : "SUCCESS",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=7668",
       "triggerID" : "a7c6cd3eb6c08cf4d6a97e2c1981025493dac833",
       "triggerType" : "PUSH"
     }, {
       "hash" : "93866c7fb3fa7ff2b9bb978fe2bed922c28ad381",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "93866c7fb3fa7ff2b9bb978fe2bed922c28ad381",
       "triggerType" : "PUSH"
     } ]
   }-->
   ## CI report:
   
   * a7c6cd3eb6c08cf4d6a97e2c1981025493dac833 Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=7668) 
   * 93866c7fb3fa7ff2b9bb978fe2bed922c28ad381 UNKNOWN
   
   <details>
   <summary>Bot commands</summary>
     @hudi-bot supports the following commands:
   
    - `@hudi-bot run azure` re-run the last Azure build
   </details>


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [hudi] hudi-bot removed a comment on pull request #5186: [HUDI-3451] Add checks for metadata table init to avoid possible out-of-sync

Posted by GitBox <gi...@apache.org>.
hudi-bot removed a comment on pull request #5186:
URL: https://github.com/apache/hudi/pull/5186#issuecomment-1084696205


   <!--
   Meta data
   {
     "version" : 1,
     "metaDataEntries" : [ {
       "hash" : "5a1c9f13870f52750e25f8658776c0e9c27a4426",
       "status" : "SUCCESS",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=7632",
       "triggerID" : "5a1c9f13870f52750e25f8658776c0e9c27a4426",
       "triggerType" : "PUSH"
     }, {
       "hash" : "82c4b1422058fd68b1d5843b1ab1386a96adbc64",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "82c4b1422058fd68b1d5843b1ab1386a96adbc64",
       "triggerType" : "PUSH"
     } ]
   }-->
   ## CI report:
   
   * 5a1c9f13870f52750e25f8658776c0e9c27a4426 Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=7632) 
   * 82c4b1422058fd68b1d5843b1ab1386a96adbc64 UNKNOWN
   
   <details>
   <summary>Bot commands</summary>
     @hudi-bot supports the following commands:
   
    - `@hudi-bot run azure` re-run the last Azure build
   </details>


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [hudi] hudi-bot removed a comment on pull request #5186: [HUDI-3451] Add checks for metadata table init to avoid possible out-of-sync

Posted by GitBox <gi...@apache.org>.
hudi-bot removed a comment on pull request #5186:
URL: https://github.com/apache/hudi/pull/5186#issuecomment-1085306569


   <!--
   Meta data
   {
     "version" : 1,
     "metaDataEntries" : [ {
       "hash" : "5a1c9f13870f52750e25f8658776c0e9c27a4426",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=7632",
       "triggerID" : "5a1c9f13870f52750e25f8658776c0e9c27a4426",
       "triggerType" : "PUSH"
     }, {
       "hash" : "82c4b1422058fd68b1d5843b1ab1386a96adbc64",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=7646",
       "triggerID" : "82c4b1422058fd68b1d5843b1ab1386a96adbc64",
       "triggerType" : "PUSH"
     }, {
       "hash" : "f3a50b13c1802ffd1c33f9469d2f7dcafde47bc2",
       "status" : "CANCELED",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=7647",
       "triggerID" : "f3a50b13c1802ffd1c33f9469d2f7dcafde47bc2",
       "triggerType" : "PUSH"
     }, {
       "hash" : "a7c6cd3eb6c08cf4d6a97e2c1981025493dac833",
       "status" : "PENDING",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=7668",
       "triggerID" : "a7c6cd3eb6c08cf4d6a97e2c1981025493dac833",
       "triggerType" : "PUSH"
     } ]
   }-->
   ## CI report:
   
   * f3a50b13c1802ffd1c33f9469d2f7dcafde47bc2 Azure: [CANCELED](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=7647) 
   * a7c6cd3eb6c08cf4d6a97e2c1981025493dac833 Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=7668) 
   
   <details>
   <summary>Bot commands</summary>
     @hudi-bot supports the following commands:
   
    - `@hudi-bot run azure` re-run the last Azure build
   </details>


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [hudi] hudi-bot removed a comment on pull request #5186: [HUDI-3451] Add checks for metadata table init to avoid possible out-of-sync

Posted by GitBox <gi...@apache.org>.
hudi-bot removed a comment on pull request #5186:
URL: https://github.com/apache/hudi/pull/5186#issuecomment-1084705101


   <!--
   Meta data
   {
     "version" : 1,
     "metaDataEntries" : [ {
       "hash" : "5a1c9f13870f52750e25f8658776c0e9c27a4426",
       "status" : "SUCCESS",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=7632",
       "triggerID" : "5a1c9f13870f52750e25f8658776c0e9c27a4426",
       "triggerType" : "PUSH"
     }, {
       "hash" : "82c4b1422058fd68b1d5843b1ab1386a96adbc64",
       "status" : "PENDING",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=7646",
       "triggerID" : "82c4b1422058fd68b1d5843b1ab1386a96adbc64",
       "triggerType" : "PUSH"
     } ]
   }-->
   ## CI report:
   
   * 5a1c9f13870f52750e25f8658776c0e9c27a4426 Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=7632) 
   * 82c4b1422058fd68b1d5843b1ab1386a96adbc64 Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=7646) 
   
   <details>
   <summary>Bot commands</summary>
     @hudi-bot supports the following commands:
   
    - `@hudi-bot run azure` re-run the last Azure build
   </details>


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [hudi] hudi-bot commented on pull request #5186: [HUDI-3451] Add checks for metadata table init to avoid possible out-of-sync

Posted by GitBox <gi...@apache.org>.
hudi-bot commented on pull request #5186:
URL: https://github.com/apache/hudi/pull/5186#issuecomment-1084811997


   <!--
   Meta data
   {
     "version" : 1,
     "metaDataEntries" : [ {
       "hash" : "5a1c9f13870f52750e25f8658776c0e9c27a4426",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=7632",
       "triggerID" : "5a1c9f13870f52750e25f8658776c0e9c27a4426",
       "triggerType" : "PUSH"
     }, {
       "hash" : "82c4b1422058fd68b1d5843b1ab1386a96adbc64",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=7646",
       "triggerID" : "82c4b1422058fd68b1d5843b1ab1386a96adbc64",
       "triggerType" : "PUSH"
     }, {
       "hash" : "f3a50b13c1802ffd1c33f9469d2f7dcafde47bc2",
       "status" : "CANCELED",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=7647",
       "triggerID" : "f3a50b13c1802ffd1c33f9469d2f7dcafde47bc2",
       "triggerType" : "PUSH"
     } ]
   }-->
   ## CI report:
   
   * f3a50b13c1802ffd1c33f9469d2f7dcafde47bc2 Azure: [CANCELED](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=7647) 
   
   <details>
   <summary>Bot commands</summary>
     @hudi-bot supports the following commands:
   
    - `@hudi-bot run azure` re-run the last Azure build
   </details>


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [hudi] hudi-bot removed a comment on pull request #5186: [HUDI-3451] Delete metadata table when the write client disables MDT

Posted by GitBox <gi...@apache.org>.
hudi-bot removed a comment on pull request #5186:
URL: https://github.com/apache/hudi/pull/5186#issuecomment-1086570469


   <!--
   Meta data
   {
     "version" : 1,
     "metaDataEntries" : [ {
       "hash" : "5a1c9f13870f52750e25f8658776c0e9c27a4426",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=7632",
       "triggerID" : "5a1c9f13870f52750e25f8658776c0e9c27a4426",
       "triggerType" : "PUSH"
     }, {
       "hash" : "82c4b1422058fd68b1d5843b1ab1386a96adbc64",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=7646",
       "triggerID" : "82c4b1422058fd68b1d5843b1ab1386a96adbc64",
       "triggerType" : "PUSH"
     }, {
       "hash" : "f3a50b13c1802ffd1c33f9469d2f7dcafde47bc2",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=7647",
       "triggerID" : "f3a50b13c1802ffd1c33f9469d2f7dcafde47bc2",
       "triggerType" : "PUSH"
     }, {
       "hash" : "a7c6cd3eb6c08cf4d6a97e2c1981025493dac833",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=7668",
       "triggerID" : "a7c6cd3eb6c08cf4d6a97e2c1981025493dac833",
       "triggerType" : "PUSH"
     }, {
       "hash" : "93866c7fb3fa7ff2b9bb978fe2bed922c28ad381",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=7675",
       "triggerID" : "93866c7fb3fa7ff2b9bb978fe2bed922c28ad381",
       "triggerType" : "PUSH"
     }, {
       "hash" : "4f6001bbde0a330c9cecb7783cf8999276701039",
       "status" : "CANCELED",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=7744",
       "triggerID" : "4f6001bbde0a330c9cecb7783cf8999276701039",
       "triggerType" : "PUSH"
     }, {
       "hash" : "9aa5907f19bc0a023ee881f013c03f7abcf3bcf7",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "9aa5907f19bc0a023ee881f013c03f7abcf3bcf7",
       "triggerType" : "PUSH"
     } ]
   }-->
   ## CI report:
   
   * 4f6001bbde0a330c9cecb7783cf8999276701039 Azure: [CANCELED](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=7744) 
   * 9aa5907f19bc0a023ee881f013c03f7abcf3bcf7 UNKNOWN
   
   <details>
   <summary>Bot commands</summary>
     @hudi-bot supports the following commands:
   
    - `@hudi-bot run azure` re-run the last Azure build
   </details>


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [hudi] hudi-bot commented on pull request #5186: [HUDI-3451] Add checks for metadata table init to avoid possible out-of-sync

Posted by GitBox <gi...@apache.org>.
hudi-bot commented on pull request #5186:
URL: https://github.com/apache/hudi/pull/5186#issuecomment-1084728869


   <!--
   Meta data
   {
     "version" : 1,
     "metaDataEntries" : [ {
       "hash" : "5a1c9f13870f52750e25f8658776c0e9c27a4426",
       "status" : "SUCCESS",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=7632",
       "triggerID" : "5a1c9f13870f52750e25f8658776c0e9c27a4426",
       "triggerType" : "PUSH"
     }, {
       "hash" : "82c4b1422058fd68b1d5843b1ab1386a96adbc64",
       "status" : "PENDING",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=7646",
       "triggerID" : "82c4b1422058fd68b1d5843b1ab1386a96adbc64",
       "triggerType" : "PUSH"
     }, {
       "hash" : "f3a50b13c1802ffd1c33f9469d2f7dcafde47bc2",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "f3a50b13c1802ffd1c33f9469d2f7dcafde47bc2",
       "triggerType" : "PUSH"
     } ]
   }-->
   ## CI report:
   
   * 5a1c9f13870f52750e25f8658776c0e9c27a4426 Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=7632) 
   * 82c4b1422058fd68b1d5843b1ab1386a96adbc64 Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=7646) 
   * f3a50b13c1802ffd1c33f9469d2f7dcafde47bc2 UNKNOWN
   
   <details>
   <summary>Bot commands</summary>
     @hudi-bot supports the following commands:
   
    - `@hudi-bot run azure` re-run the last Azure build
   </details>


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [hudi] hudi-bot commented on pull request #5186: [HUDI-3451] Add checks for metadata table init to avoid possible out-of-sync

Posted by GitBox <gi...@apache.org>.
hudi-bot commented on pull request #5186:
URL: https://github.com/apache/hudi/pull/5186#issuecomment-1085306569


   <!--
   Meta data
   {
     "version" : 1,
     "metaDataEntries" : [ {
       "hash" : "5a1c9f13870f52750e25f8658776c0e9c27a4426",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=7632",
       "triggerID" : "5a1c9f13870f52750e25f8658776c0e9c27a4426",
       "triggerType" : "PUSH"
     }, {
       "hash" : "82c4b1422058fd68b1d5843b1ab1386a96adbc64",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=7646",
       "triggerID" : "82c4b1422058fd68b1d5843b1ab1386a96adbc64",
       "triggerType" : "PUSH"
     }, {
       "hash" : "f3a50b13c1802ffd1c33f9469d2f7dcafde47bc2",
       "status" : "CANCELED",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=7647",
       "triggerID" : "f3a50b13c1802ffd1c33f9469d2f7dcafde47bc2",
       "triggerType" : "PUSH"
     }, {
       "hash" : "a7c6cd3eb6c08cf4d6a97e2c1981025493dac833",
       "status" : "PENDING",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=7668",
       "triggerID" : "a7c6cd3eb6c08cf4d6a97e2c1981025493dac833",
       "triggerType" : "PUSH"
     } ]
   }-->
   ## CI report:
   
   * f3a50b13c1802ffd1c33f9469d2f7dcafde47bc2 Azure: [CANCELED](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=7647) 
   * a7c6cd3eb6c08cf4d6a97e2c1981025493dac833 Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=7668) 
   
   <details>
   <summary>Bot commands</summary>
     @hudi-bot supports the following commands:
   
    - `@hudi-bot run azure` re-run the last Azure build
   </details>


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [hudi] hudi-bot commented on pull request #5186: [HUDI-3451] Add checks for metadata table init to avoid possible out-of-sync

Posted by GitBox <gi...@apache.org>.
hudi-bot commented on pull request #5186:
URL: https://github.com/apache/hudi/pull/5186#issuecomment-1085305276


   <!--
   Meta data
   {
     "version" : 1,
     "metaDataEntries" : [ {
       "hash" : "5a1c9f13870f52750e25f8658776c0e9c27a4426",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=7632",
       "triggerID" : "5a1c9f13870f52750e25f8658776c0e9c27a4426",
       "triggerType" : "PUSH"
     }, {
       "hash" : "82c4b1422058fd68b1d5843b1ab1386a96adbc64",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=7646",
       "triggerID" : "82c4b1422058fd68b1d5843b1ab1386a96adbc64",
       "triggerType" : "PUSH"
     }, {
       "hash" : "f3a50b13c1802ffd1c33f9469d2f7dcafde47bc2",
       "status" : "CANCELED",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=7647",
       "triggerID" : "f3a50b13c1802ffd1c33f9469d2f7dcafde47bc2",
       "triggerType" : "PUSH"
     }, {
       "hash" : "a7c6cd3eb6c08cf4d6a97e2c1981025493dac833",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "a7c6cd3eb6c08cf4d6a97e2c1981025493dac833",
       "triggerType" : "PUSH"
     } ]
   }-->
   ## CI report:
   
   * f3a50b13c1802ffd1c33f9469d2f7dcafde47bc2 Azure: [CANCELED](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=7647) 
   * a7c6cd3eb6c08cf4d6a97e2c1981025493dac833 UNKNOWN
   
   <details>
   <summary>Bot commands</summary>
     @hudi-bot supports the following commands:
   
    - `@hudi-bot run azure` re-run the last Azure build
   </details>


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [hudi] hudi-bot removed a comment on pull request #5186: [HUDI-3451] Add checks for metadata table init to avoid possible out-of-sync

Posted by GitBox <gi...@apache.org>.
hudi-bot removed a comment on pull request #5186:
URL: https://github.com/apache/hudi/pull/5186#issuecomment-1084811997


   <!--
   Meta data
   {
     "version" : 1,
     "metaDataEntries" : [ {
       "hash" : "5a1c9f13870f52750e25f8658776c0e9c27a4426",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=7632",
       "triggerID" : "5a1c9f13870f52750e25f8658776c0e9c27a4426",
       "triggerType" : "PUSH"
     }, {
       "hash" : "82c4b1422058fd68b1d5843b1ab1386a96adbc64",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=7646",
       "triggerID" : "82c4b1422058fd68b1d5843b1ab1386a96adbc64",
       "triggerType" : "PUSH"
     }, {
       "hash" : "f3a50b13c1802ffd1c33f9469d2f7dcafde47bc2",
       "status" : "CANCELED",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=7647",
       "triggerID" : "f3a50b13c1802ffd1c33f9469d2f7dcafde47bc2",
       "triggerType" : "PUSH"
     } ]
   }-->
   ## CI report:
   
   * f3a50b13c1802ffd1c33f9469d2f7dcafde47bc2 Azure: [CANCELED](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=7647) 
   
   <details>
   <summary>Bot commands</summary>
     @hudi-bot supports the following commands:
   
    - `@hudi-bot run azure` re-run the last Azure build
   </details>


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [hudi] hudi-bot commented on pull request #5186: [HUDI-3451] Add checks for metadata table init to avoid possible out-of-sync

Posted by GitBox <gi...@apache.org>.
hudi-bot commented on pull request #5186:
URL: https://github.com/apache/hudi/pull/5186#issuecomment-1084350247


   <!--
   Meta data
   {
     "version" : 1,
     "metaDataEntries" : [ {
       "hash" : "5a1c9f13870f52750e25f8658776c0e9c27a4426",
       "status" : "PENDING",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=7632",
       "triggerID" : "5a1c9f13870f52750e25f8658776c0e9c27a4426",
       "triggerType" : "PUSH"
     } ]
   }-->
   ## CI report:
   
   * 5a1c9f13870f52750e25f8658776c0e9c27a4426 Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=7632) 
   
   <details>
   <summary>Bot commands</summary>
     @hudi-bot supports the following commands:
   
    - `@hudi-bot run azure` re-run the last Azure build
   </details>


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [hudi] hudi-bot commented on pull request #5186: [HUDI-3451] Add checks for metadata table init to avoid possible out-of-sync

Posted by GitBox <gi...@apache.org>.
hudi-bot commented on pull request #5186:
URL: https://github.com/apache/hudi/pull/5186#issuecomment-1084696205


   <!--
   Meta data
   {
     "version" : 1,
     "metaDataEntries" : [ {
       "hash" : "5a1c9f13870f52750e25f8658776c0e9c27a4426",
       "status" : "SUCCESS",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=7632",
       "triggerID" : "5a1c9f13870f52750e25f8658776c0e9c27a4426",
       "triggerType" : "PUSH"
     }, {
       "hash" : "82c4b1422058fd68b1d5843b1ab1386a96adbc64",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "82c4b1422058fd68b1d5843b1ab1386a96adbc64",
       "triggerType" : "PUSH"
     } ]
   }-->
   ## CI report:
   
   * 5a1c9f13870f52750e25f8658776c0e9c27a4426 Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=7632) 
   * 82c4b1422058fd68b1d5843b1ab1386a96adbc64 UNKNOWN
   
   <details>
   <summary>Bot commands</summary>
     @hudi-bot supports the following commands:
   
    - `@hudi-bot run azure` re-run the last Azure build
   </details>


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [hudi] hudi-bot removed a comment on pull request #5186: [HUDI-3451] Add checks for metadata table init to avoid possible out-of-sync

Posted by GitBox <gi...@apache.org>.
hudi-bot removed a comment on pull request #5186:
URL: https://github.com/apache/hudi/pull/5186#issuecomment-1085384880


   <!--
   Meta data
   {
     "version" : 1,
     "metaDataEntries" : [ {
       "hash" : "5a1c9f13870f52750e25f8658776c0e9c27a4426",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=7632",
       "triggerID" : "5a1c9f13870f52750e25f8658776c0e9c27a4426",
       "triggerType" : "PUSH"
     }, {
       "hash" : "82c4b1422058fd68b1d5843b1ab1386a96adbc64",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=7646",
       "triggerID" : "82c4b1422058fd68b1d5843b1ab1386a96adbc64",
       "triggerType" : "PUSH"
     }, {
       "hash" : "f3a50b13c1802ffd1c33f9469d2f7dcafde47bc2",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=7647",
       "triggerID" : "f3a50b13c1802ffd1c33f9469d2f7dcafde47bc2",
       "triggerType" : "PUSH"
     }, {
       "hash" : "a7c6cd3eb6c08cf4d6a97e2c1981025493dac833",
       "status" : "SUCCESS",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=7668",
       "triggerID" : "a7c6cd3eb6c08cf4d6a97e2c1981025493dac833",
       "triggerType" : "PUSH"
     } ]
   }-->
   ## CI report:
   
   * a7c6cd3eb6c08cf4d6a97e2c1981025493dac833 Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=7668) 
   
   <details>
   <summary>Bot commands</summary>
     @hudi-bot supports the following commands:
   
    - `@hudi-bot run azure` re-run the last Azure build
   </details>


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [hudi] hudi-bot commented on pull request #5186: [HUDI-3451] Add checks for metadata table init to avoid possible out-of-sync

Posted by GitBox <gi...@apache.org>.
hudi-bot commented on pull request #5186:
URL: https://github.com/apache/hudi/pull/5186#issuecomment-1085384880


   <!--
   Meta data
   {
     "version" : 1,
     "metaDataEntries" : [ {
       "hash" : "5a1c9f13870f52750e25f8658776c0e9c27a4426",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=7632",
       "triggerID" : "5a1c9f13870f52750e25f8658776c0e9c27a4426",
       "triggerType" : "PUSH"
     }, {
       "hash" : "82c4b1422058fd68b1d5843b1ab1386a96adbc64",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=7646",
       "triggerID" : "82c4b1422058fd68b1d5843b1ab1386a96adbc64",
       "triggerType" : "PUSH"
     }, {
       "hash" : "f3a50b13c1802ffd1c33f9469d2f7dcafde47bc2",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=7647",
       "triggerID" : "f3a50b13c1802ffd1c33f9469d2f7dcafde47bc2",
       "triggerType" : "PUSH"
     }, {
       "hash" : "a7c6cd3eb6c08cf4d6a97e2c1981025493dac833",
       "status" : "SUCCESS",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=7668",
       "triggerID" : "a7c6cd3eb6c08cf4d6a97e2c1981025493dac833",
       "triggerType" : "PUSH"
     } ]
   }-->
   ## CI report:
   
   * a7c6cd3eb6c08cf4d6a97e2c1981025493dac833 Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=7668) 
   
   <details>
   <summary>Bot commands</summary>
     @hudi-bot supports the following commands:
   
    - `@hudi-bot run azure` re-run the last Azure build
   </details>


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [hudi] hudi-bot removed a comment on pull request #5186: [HUDI-3451] Delete metadata table when the write client disables MDT

Posted by GitBox <gi...@apache.org>.
hudi-bot removed a comment on pull request #5186:
URL: https://github.com/apache/hudi/pull/5186#issuecomment-1086569967


   <!--
   Meta data
   {
     "version" : 1,
     "metaDataEntries" : [ {
       "hash" : "5a1c9f13870f52750e25f8658776c0e9c27a4426",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=7632",
       "triggerID" : "5a1c9f13870f52750e25f8658776c0e9c27a4426",
       "triggerType" : "PUSH"
     }, {
       "hash" : "82c4b1422058fd68b1d5843b1ab1386a96adbc64",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=7646",
       "triggerID" : "82c4b1422058fd68b1d5843b1ab1386a96adbc64",
       "triggerType" : "PUSH"
     }, {
       "hash" : "f3a50b13c1802ffd1c33f9469d2f7dcafde47bc2",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=7647",
       "triggerID" : "f3a50b13c1802ffd1c33f9469d2f7dcafde47bc2",
       "triggerType" : "PUSH"
     }, {
       "hash" : "a7c6cd3eb6c08cf4d6a97e2c1981025493dac833",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=7668",
       "triggerID" : "a7c6cd3eb6c08cf4d6a97e2c1981025493dac833",
       "triggerType" : "PUSH"
     }, {
       "hash" : "93866c7fb3fa7ff2b9bb978fe2bed922c28ad381",
       "status" : "SUCCESS",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=7675",
       "triggerID" : "93866c7fb3fa7ff2b9bb978fe2bed922c28ad381",
       "triggerType" : "PUSH"
     }, {
       "hash" : "4f6001bbde0a330c9cecb7783cf8999276701039",
       "status" : "PENDING",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=7744",
       "triggerID" : "4f6001bbde0a330c9cecb7783cf8999276701039",
       "triggerType" : "PUSH"
     }, {
       "hash" : "9aa5907f19bc0a023ee881f013c03f7abcf3bcf7",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "9aa5907f19bc0a023ee881f013c03f7abcf3bcf7",
       "triggerType" : "PUSH"
     } ]
   }-->
   ## CI report:
   
   * 93866c7fb3fa7ff2b9bb978fe2bed922c28ad381 Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=7675) 
   * 4f6001bbde0a330c9cecb7783cf8999276701039 Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=7744) 
   * 9aa5907f19bc0a023ee881f013c03f7abcf3bcf7 UNKNOWN
   
   <details>
   <summary>Bot commands</summary>
     @hudi-bot supports the following commands:
   
    - `@hudi-bot run azure` re-run the last Azure build
   </details>


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [hudi] codope merged pull request #5186: [HUDI-3451] Delete metadata table when the write client disables MDT

Posted by GitBox <gi...@apache.org>.
codope merged pull request #5186:
URL: https://github.com/apache/hudi/pull/5186


   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [hudi] hudi-bot commented on pull request #5186: [HUDI-3451] Add checks for metadata table init to avoid possible out-of-sync

Posted by GitBox <gi...@apache.org>.
hudi-bot commented on pull request #5186:
URL: https://github.com/apache/hudi/pull/5186#issuecomment-1085388601


   <!--
   Meta data
   {
     "version" : 1,
     "metaDataEntries" : [ {
       "hash" : "5a1c9f13870f52750e25f8658776c0e9c27a4426",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=7632",
       "triggerID" : "5a1c9f13870f52750e25f8658776c0e9c27a4426",
       "triggerType" : "PUSH"
     }, {
       "hash" : "82c4b1422058fd68b1d5843b1ab1386a96adbc64",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=7646",
       "triggerID" : "82c4b1422058fd68b1d5843b1ab1386a96adbc64",
       "triggerType" : "PUSH"
     }, {
       "hash" : "f3a50b13c1802ffd1c33f9469d2f7dcafde47bc2",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=7647",
       "triggerID" : "f3a50b13c1802ffd1c33f9469d2f7dcafde47bc2",
       "triggerType" : "PUSH"
     }, {
       "hash" : "a7c6cd3eb6c08cf4d6a97e2c1981025493dac833",
       "status" : "SUCCESS",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=7668",
       "triggerID" : "a7c6cd3eb6c08cf4d6a97e2c1981025493dac833",
       "triggerType" : "PUSH"
     }, {
       "hash" : "93866c7fb3fa7ff2b9bb978fe2bed922c28ad381",
       "status" : "PENDING",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=7675",
       "triggerID" : "93866c7fb3fa7ff2b9bb978fe2bed922c28ad381",
       "triggerType" : "PUSH"
     } ]
   }-->
   ## CI report:
   
   * a7c6cd3eb6c08cf4d6a97e2c1981025493dac833 Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=7668) 
   * 93866c7fb3fa7ff2b9bb978fe2bed922c28ad381 Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=7675) 
   
   <details>
   <summary>Bot commands</summary>
     @hudi-bot supports the following commands:
   
    - `@hudi-bot run azure` re-run the last Azure build
   </details>


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [hudi] zhangyue19921010 commented on a change in pull request #5186: [HUDI-3451] Add checks for metadata table init to avoid possible out-of-sync

Posted by GitBox <gi...@apache.org>.
zhangyue19921010 commented on a change in pull request #5186:
URL: https://github.com/apache/hudi/pull/5186#discussion_r840256140



##########
File path: hudi-common/src/main/java/org/apache/hudi/common/table/HoodieTableConfig.java
##########
@@ -208,6 +209,11 @@
       .sinceVersion("0.11.0")
       .withDocumentation("Table checksum is used to guard against partial writes in HDFS. It is added as the last entry in hoodie.properties and then used to validate while reading table config.");
 
+  public static final ConfigProperty<Boolean> METADATA_TABLE_ENABLE = ConfigProperty
+      .key("hoodie.table.metadata.enable")
+      .defaultValue(HoodieMetadataConfig.ENABLE.defaultValue())
+      .withDocumentation("Enable the internal metadata table which serves table metadata like level file listings.");

Review comment:
       Hi @codope just add a new ticket https://issues.apache.org/jira/browse/HUDI-3766 here.
   Maybe we can handle this flag for reader side for another PR?
   Take this patch as a first step solving the possible out-of-sync issue :)
   




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [hudi] codope commented on a change in pull request #5186: [HUDI-3451] Add checks for metadata table init to avoid possible out-of-sync

Posted by GitBox <gi...@apache.org>.
codope commented on a change in pull request #5186:
URL: https://github.com/apache/hudi/pull/5186#discussion_r839908036



##########
File path: hudi-client/hudi-client-common/src/main/java/org/apache/hudi/table/HoodieTable.java
##########
@@ -781,6 +782,53 @@ public HoodieEngineContext getContext() {
     return Option.empty();
   }
 
+  /**
+   * Initialize hoodie.table.metadata.enable in hoodie.proterties;
+   * Check if current metadata table flag in hoodieWriteConfig is the same as recorded in hoodie.proterties:
+   *  1. If the flag in hoodie.proterties is true but it is false in hoodieWriteConfig, It means users turn off MDT and we need to clean up MDT.
+   *  2. Update hoodie.table.metadata.enable in hoodie.proterties based on HoodieWriteConfig if the value is different.
+   */
+  protected void verifyMetadataTableInTableConfig() {
+    // ignore the hoodie.properties in MDT
+    if (metaClient.getBasePath().contains(HoodieTableMetaClient.METADATA_TABLE_FOLDER_PATH)) {
+      return;
+    }
+
+    try {
+      if (metaClient.getTableConfig().contains(HoodieTableConfig.METADATA_TABLE_ENABLE)) {
+        Boolean enableMDTInTableConfig = metaClient.getTableConfig().getBoolean(HoodieTableConfig.METADATA_TABLE_ENABLE);
+        Path mdtBasePath = new Path(HoodieTableMetadata.getMetadataTableBasePath(config.getBasePath()));
+        // MDT flag in TableConfig is true;
+        // MDT flag in write config is false;
+        // ===> Users turn off MDT, so that we need to clean up this metadata table in case out-of-sync issue.
+        // the condition order in if(xx) is important.
+        if (enableMDTInTableConfig && !config.isMetadataTableEnabled() && metaClient.getFs().exists(mdtBasePath)) {
+          LOG.info("Deleting metadata table because of disabled in writer.");
+          metaClient.getFs().delete(mdtBasePath, true);
+        }
+
+        // update table config if necessary
+        if (Boolean.compare(enableMDTInTableConfig, config.isMetadataTableEnabled()) != 0) {
+          updateMetadataTableEnableInTableConfig();
+        }
+      } else {
+        // initial METADATA_TABLE_ENABLE.key in table config
+        updateMetadataTableEnableInTableConfig();

Review comment:
       yes this looks right.




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [hudi] zhangyue19921010 commented on pull request #5186: [HUDI-3451] Add checks for metadata table init to avoid possible out-of-sync

Posted by GitBox <gi...@apache.org>.
zhangyue19921010 commented on pull request #5186:
URL: https://github.com/apache/hudi/pull/5186#issuecomment-1085386012


   azure passed, trigger another Java packaging test because of
   ```
   Error:  Plugin org.scalatest:scalatest-maven-plugin:1.0 or one of its dependencies could not be resolved: Failed to read artifact descriptor for org.scalatest:scalatest-maven-plugin:jar:1.0: Could not transfer artifact org.scalatest:scalatest-maven-plugin:pom:1.0 from/to central (https://repo.maven.apache.org/maven2): transfer failed for https://repo.maven.apache.org/maven2/org/scalatest/scalatest-maven-plugin/1.0/scalatest-maven-plugin-1.0.pom: Connection timed out (Read failed) -> [Help 1]
   ```


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [hudi] zhangyue19921010 commented on a change in pull request #5186: [HUDI-3451] Add checks for metadata table init to avoid possible out-of-sync

Posted by GitBox <gi...@apache.org>.
zhangyue19921010 commented on a change in pull request #5186:
URL: https://github.com/apache/hudi/pull/5186#discussion_r839712711



##########
File path: hudi-client/hudi-client-common/src/main/java/org/apache/hudi/table/HoodieTable.java
##########
@@ -781,6 +782,53 @@ public HoodieEngineContext getContext() {
     return Option.empty();
   }
 
+  /**
+   * Initialize hoodie.table.metadata.enable in hoodie.proterties;
+   * Check if current metadata table flag in hoodieWriteConfig is the same as recorded in hoodie.proterties:
+   *  1. If the flag in hoodie.proterties is true but it is false in hoodieWriteConfig, It means users turn off MDT and we need to clean up MDT.
+   *  2. Update hoodie.table.metadata.enable in hoodie.proterties based on HoodieWriteConfig if the value is different.
+   */
+  protected void verifyMetadataTableInTableConfig() {
+    // ignore the hoodie.properties in MDT
+    if (metaClient.getBasePath().contains(HoodieTableMetaClient.METADATA_TABLE_FOLDER_PATH)) {
+      return;
+    }
+
+    try {
+      if (metaClient.getTableConfig().contains(HoodieTableConfig.METADATA_TABLE_ENABLE)) {
+        Boolean enableMDTInTableConfig = metaClient.getTableConfig().getBoolean(HoodieTableConfig.METADATA_TABLE_ENABLE);
+        Path mdtBasePath = new Path(HoodieTableMetadata.getMetadataTableBasePath(config.getBasePath()));
+        // MDT flag in TableConfig is true;
+        // MDT flag in write config is false;
+        // ===> Users turn off MDT, so that we need to clean up this metadata table in case out-of-sync issue.
+        // the condition order in if(xx) is important.
+        if (enableMDTInTableConfig && !config.isMetadataTableEnabled() && metaClient.getFs().exists(mdtBasePath)) {
+          LOG.info("Deleting metadata table because of disabled in writer.");
+          metaClient.getFs().delete(mdtBasePath, true);
+        }
+
+        // update table config if necessary
+        if (Boolean.compare(enableMDTInTableConfig, config.isMetadataTableEnabled()) != 0) {
+          updateMetadataTableEnableInTableConfig();
+        }
+      } else {
+        // initial METADATA_TABLE_ENABLE.key in table config
+        updateMetadataTableEnableInTableConfig();

Review comment:
       Nice catching! Sure we do need to take care of upgrade.
   Maybe we can add `METADATA_TABLE_ENABLE` in ThreeToFourUpgradeHandler. Also clean up MDT during upgrade when users disable MDT.
   
   So that :
   1. All old version MDT upgrade to master has `METADATA_TABLE_ENABLE` config in hoodie.properties.
   2. When `METADATA_TABLE_ENABLE` are false, there is no $basePath/.hoodie/metadata left on storage anymore.
   3. We could reduce the total FS call(fs.exist or fs.delete) as much as possible.




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [hudi] hudi-bot commented on pull request #5186: [HUDI-3451] Add checks for metadata table init to avoid possible out-of-sync

Posted by GitBox <gi...@apache.org>.
hudi-bot commented on pull request #5186:
URL: https://github.com/apache/hudi/pull/5186#issuecomment-1084705101


   <!--
   Meta data
   {
     "version" : 1,
     "metaDataEntries" : [ {
       "hash" : "5a1c9f13870f52750e25f8658776c0e9c27a4426",
       "status" : "SUCCESS",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=7632",
       "triggerID" : "5a1c9f13870f52750e25f8658776c0e9c27a4426",
       "triggerType" : "PUSH"
     }, {
       "hash" : "82c4b1422058fd68b1d5843b1ab1386a96adbc64",
       "status" : "PENDING",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=7646",
       "triggerID" : "82c4b1422058fd68b1d5843b1ab1386a96adbc64",
       "triggerType" : "PUSH"
     } ]
   }-->
   ## CI report:
   
   * 5a1c9f13870f52750e25f8658776c0e9c27a4426 Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=7632) 
   * 82c4b1422058fd68b1d5843b1ab1386a96adbc64 Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=7646) 
   
   <details>
   <summary>Bot commands</summary>
     @hudi-bot supports the following commands:
   
    - `@hudi-bot run azure` re-run the last Azure build
   </details>


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [hudi] zhangyue19921010 commented on a change in pull request #5186: [HUDI-3451] Add checks for metadata table init to avoid possible out-of-sync

Posted by GitBox <gi...@apache.org>.
zhangyue19921010 commented on a change in pull request #5186:
URL: https://github.com/apache/hudi/pull/5186#discussion_r839713414



##########
File path: hudi-common/src/main/java/org/apache/hudi/common/table/HoodieTableConfig.java
##########
@@ -208,6 +209,11 @@
       .sinceVersion("0.11.0")
       .withDocumentation("Table checksum is used to guard against partial writes in HDFS. It is added as the last entry in hoodie.properties and then used to validate while reading table config.");
 
+  public static final ConfigProperty<Boolean> METADATA_TABLE_ENABLE = ConfigProperty

Review comment:
       Mentioned before. Thanks a lot for your review!




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [hudi] hudi-bot commented on pull request #5186: [HUDI-3451] Delete metadata table when the write client disables MDT

Posted by GitBox <gi...@apache.org>.
hudi-bot commented on pull request #5186:
URL: https://github.com/apache/hudi/pull/5186#issuecomment-1086570469


   <!--
   Meta data
   {
     "version" : 1,
     "metaDataEntries" : [ {
       "hash" : "5a1c9f13870f52750e25f8658776c0e9c27a4426",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=7632",
       "triggerID" : "5a1c9f13870f52750e25f8658776c0e9c27a4426",
       "triggerType" : "PUSH"
     }, {
       "hash" : "82c4b1422058fd68b1d5843b1ab1386a96adbc64",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=7646",
       "triggerID" : "82c4b1422058fd68b1d5843b1ab1386a96adbc64",
       "triggerType" : "PUSH"
     }, {
       "hash" : "f3a50b13c1802ffd1c33f9469d2f7dcafde47bc2",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=7647",
       "triggerID" : "f3a50b13c1802ffd1c33f9469d2f7dcafde47bc2",
       "triggerType" : "PUSH"
     }, {
       "hash" : "a7c6cd3eb6c08cf4d6a97e2c1981025493dac833",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=7668",
       "triggerID" : "a7c6cd3eb6c08cf4d6a97e2c1981025493dac833",
       "triggerType" : "PUSH"
     }, {
       "hash" : "93866c7fb3fa7ff2b9bb978fe2bed922c28ad381",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=7675",
       "triggerID" : "93866c7fb3fa7ff2b9bb978fe2bed922c28ad381",
       "triggerType" : "PUSH"
     }, {
       "hash" : "4f6001bbde0a330c9cecb7783cf8999276701039",
       "status" : "CANCELED",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=7744",
       "triggerID" : "4f6001bbde0a330c9cecb7783cf8999276701039",
       "triggerType" : "PUSH"
     }, {
       "hash" : "9aa5907f19bc0a023ee881f013c03f7abcf3bcf7",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "9aa5907f19bc0a023ee881f013c03f7abcf3bcf7",
       "triggerType" : "PUSH"
     } ]
   }-->
   ## CI report:
   
   * 4f6001bbde0a330c9cecb7783cf8999276701039 Azure: [CANCELED](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=7744) 
   * 9aa5907f19bc0a023ee881f013c03f7abcf3bcf7 UNKNOWN
   
   <details>
   <summary>Bot commands</summary>
     @hudi-bot supports the following commands:
   
    - `@hudi-bot run azure` re-run the last Azure build
   </details>


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [hudi] hudi-bot removed a comment on pull request #5186: [HUDI-3451] Delete metadata table when the write client disables MDT

Posted by GitBox <gi...@apache.org>.
hudi-bot removed a comment on pull request #5186:
URL: https://github.com/apache/hudi/pull/5186#issuecomment-1086581799


   <!--
   Meta data
   {
     "version" : 1,
     "metaDataEntries" : [ {
       "hash" : "5a1c9f13870f52750e25f8658776c0e9c27a4426",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=7632",
       "triggerID" : "5a1c9f13870f52750e25f8658776c0e9c27a4426",
       "triggerType" : "PUSH"
     }, {
       "hash" : "82c4b1422058fd68b1d5843b1ab1386a96adbc64",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=7646",
       "triggerID" : "82c4b1422058fd68b1d5843b1ab1386a96adbc64",
       "triggerType" : "PUSH"
     }, {
       "hash" : "f3a50b13c1802ffd1c33f9469d2f7dcafde47bc2",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=7647",
       "triggerID" : "f3a50b13c1802ffd1c33f9469d2f7dcafde47bc2",
       "triggerType" : "PUSH"
     }, {
       "hash" : "a7c6cd3eb6c08cf4d6a97e2c1981025493dac833",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=7668",
       "triggerID" : "a7c6cd3eb6c08cf4d6a97e2c1981025493dac833",
       "triggerType" : "PUSH"
     }, {
       "hash" : "93866c7fb3fa7ff2b9bb978fe2bed922c28ad381",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=7675",
       "triggerID" : "93866c7fb3fa7ff2b9bb978fe2bed922c28ad381",
       "triggerType" : "PUSH"
     }, {
       "hash" : "4f6001bbde0a330c9cecb7783cf8999276701039",
       "status" : "CANCELED",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=7744",
       "triggerID" : "4f6001bbde0a330c9cecb7783cf8999276701039",
       "triggerType" : "PUSH"
     }, {
       "hash" : "9aa5907f19bc0a023ee881f013c03f7abcf3bcf7",
       "status" : "PENDING",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=7747",
       "triggerID" : "9aa5907f19bc0a023ee881f013c03f7abcf3bcf7",
       "triggerType" : "PUSH"
     } ]
   }-->
   ## CI report:
   
   * 4f6001bbde0a330c9cecb7783cf8999276701039 Azure: [CANCELED](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=7744) 
   * 9aa5907f19bc0a023ee881f013c03f7abcf3bcf7 Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=7747) 
   
   <details>
   <summary>Bot commands</summary>
     @hudi-bot supports the following commands:
   
    - `@hudi-bot run azure` re-run the last Azure build
   </details>


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [hudi] hudi-bot removed a comment on pull request #5186: [HUDI-3451] Add checks for metadata table init to avoid possible out-of-sync

Posted by GitBox <gi...@apache.org>.
hudi-bot removed a comment on pull request #5186:
URL: https://github.com/apache/hudi/pull/5186#issuecomment-1085388601


   <!--
   Meta data
   {
     "version" : 1,
     "metaDataEntries" : [ {
       "hash" : "5a1c9f13870f52750e25f8658776c0e9c27a4426",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=7632",
       "triggerID" : "5a1c9f13870f52750e25f8658776c0e9c27a4426",
       "triggerType" : "PUSH"
     }, {
       "hash" : "82c4b1422058fd68b1d5843b1ab1386a96adbc64",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=7646",
       "triggerID" : "82c4b1422058fd68b1d5843b1ab1386a96adbc64",
       "triggerType" : "PUSH"
     }, {
       "hash" : "f3a50b13c1802ffd1c33f9469d2f7dcafde47bc2",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=7647",
       "triggerID" : "f3a50b13c1802ffd1c33f9469d2f7dcafde47bc2",
       "triggerType" : "PUSH"
     }, {
       "hash" : "a7c6cd3eb6c08cf4d6a97e2c1981025493dac833",
       "status" : "SUCCESS",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=7668",
       "triggerID" : "a7c6cd3eb6c08cf4d6a97e2c1981025493dac833",
       "triggerType" : "PUSH"
     }, {
       "hash" : "93866c7fb3fa7ff2b9bb978fe2bed922c28ad381",
       "status" : "PENDING",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=7675",
       "triggerID" : "93866c7fb3fa7ff2b9bb978fe2bed922c28ad381",
       "triggerType" : "PUSH"
     } ]
   }-->
   ## CI report:
   
   * a7c6cd3eb6c08cf4d6a97e2c1981025493dac833 Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=7668) 
   * 93866c7fb3fa7ff2b9bb978fe2bed922c28ad381 Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=7675) 
   
   <details>
   <summary>Bot commands</summary>
     @hudi-bot supports the following commands:
   
    - `@hudi-bot run azure` re-run the last Azure build
   </details>


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [hudi] hudi-bot commented on pull request #5186: [HUDI-3451] Add checks for metadata table init to avoid possible out-of-sync

Posted by GitBox <gi...@apache.org>.
hudi-bot commented on pull request #5186:
URL: https://github.com/apache/hudi/pull/5186#issuecomment-1085442822


   <!--
   Meta data
   {
     "version" : 1,
     "metaDataEntries" : [ {
       "hash" : "5a1c9f13870f52750e25f8658776c0e9c27a4426",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=7632",
       "triggerID" : "5a1c9f13870f52750e25f8658776c0e9c27a4426",
       "triggerType" : "PUSH"
     }, {
       "hash" : "82c4b1422058fd68b1d5843b1ab1386a96adbc64",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=7646",
       "triggerID" : "82c4b1422058fd68b1d5843b1ab1386a96adbc64",
       "triggerType" : "PUSH"
     }, {
       "hash" : "f3a50b13c1802ffd1c33f9469d2f7dcafde47bc2",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=7647",
       "triggerID" : "f3a50b13c1802ffd1c33f9469d2f7dcafde47bc2",
       "triggerType" : "PUSH"
     }, {
       "hash" : "a7c6cd3eb6c08cf4d6a97e2c1981025493dac833",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=7668",
       "triggerID" : "a7c6cd3eb6c08cf4d6a97e2c1981025493dac833",
       "triggerType" : "PUSH"
     }, {
       "hash" : "93866c7fb3fa7ff2b9bb978fe2bed922c28ad381",
       "status" : "SUCCESS",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=7675",
       "triggerID" : "93866c7fb3fa7ff2b9bb978fe2bed922c28ad381",
       "triggerType" : "PUSH"
     } ]
   }-->
   ## CI report:
   
   * 93866c7fb3fa7ff2b9bb978fe2bed922c28ad381 Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=7675) 
   
   <details>
   <summary>Bot commands</summary>
     @hudi-bot supports the following commands:
   
    - `@hudi-bot run azure` re-run the last Azure build
   </details>


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [hudi] hudi-bot removed a comment on pull request #5186: [HUDI-3451] Delete metadata table when the write client disables MDT

Posted by GitBox <gi...@apache.org>.
hudi-bot removed a comment on pull request #5186:
URL: https://github.com/apache/hudi/pull/5186#issuecomment-1086568904


   <!--
   Meta data
   {
     "version" : 1,
     "metaDataEntries" : [ {
       "hash" : "5a1c9f13870f52750e25f8658776c0e9c27a4426",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=7632",
       "triggerID" : "5a1c9f13870f52750e25f8658776c0e9c27a4426",
       "triggerType" : "PUSH"
     }, {
       "hash" : "82c4b1422058fd68b1d5843b1ab1386a96adbc64",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=7646",
       "triggerID" : "82c4b1422058fd68b1d5843b1ab1386a96adbc64",
       "triggerType" : "PUSH"
     }, {
       "hash" : "f3a50b13c1802ffd1c33f9469d2f7dcafde47bc2",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=7647",
       "triggerID" : "f3a50b13c1802ffd1c33f9469d2f7dcafde47bc2",
       "triggerType" : "PUSH"
     }, {
       "hash" : "a7c6cd3eb6c08cf4d6a97e2c1981025493dac833",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=7668",
       "triggerID" : "a7c6cd3eb6c08cf4d6a97e2c1981025493dac833",
       "triggerType" : "PUSH"
     }, {
       "hash" : "93866c7fb3fa7ff2b9bb978fe2bed922c28ad381",
       "status" : "SUCCESS",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=7675",
       "triggerID" : "93866c7fb3fa7ff2b9bb978fe2bed922c28ad381",
       "triggerType" : "PUSH"
     }, {
       "hash" : "4f6001bbde0a330c9cecb7783cf8999276701039",
       "status" : "PENDING",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=7744",
       "triggerID" : "4f6001bbde0a330c9cecb7783cf8999276701039",
       "triggerType" : "PUSH"
     } ]
   }-->
   ## CI report:
   
   * 93866c7fb3fa7ff2b9bb978fe2bed922c28ad381 Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=7675) 
   * 4f6001bbde0a330c9cecb7783cf8999276701039 Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=7744) 
   
   <details>
   <summary>Bot commands</summary>
     @hudi-bot supports the following commands:
   
    - `@hudi-bot run azure` re-run the last Azure build
   </details>


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [hudi] hudi-bot commented on pull request #5186: [HUDI-3451] Add checks for metadata table init to avoid possible out-of-sync

Posted by GitBox <gi...@apache.org>.
hudi-bot commented on pull request #5186:
URL: https://github.com/apache/hudi/pull/5186#issuecomment-1086568904


   <!--
   Meta data
   {
     "version" : 1,
     "metaDataEntries" : [ {
       "hash" : "5a1c9f13870f52750e25f8658776c0e9c27a4426",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=7632",
       "triggerID" : "5a1c9f13870f52750e25f8658776c0e9c27a4426",
       "triggerType" : "PUSH"
     }, {
       "hash" : "82c4b1422058fd68b1d5843b1ab1386a96adbc64",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=7646",
       "triggerID" : "82c4b1422058fd68b1d5843b1ab1386a96adbc64",
       "triggerType" : "PUSH"
     }, {
       "hash" : "f3a50b13c1802ffd1c33f9469d2f7dcafde47bc2",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=7647",
       "triggerID" : "f3a50b13c1802ffd1c33f9469d2f7dcafde47bc2",
       "triggerType" : "PUSH"
     }, {
       "hash" : "a7c6cd3eb6c08cf4d6a97e2c1981025493dac833",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=7668",
       "triggerID" : "a7c6cd3eb6c08cf4d6a97e2c1981025493dac833",
       "triggerType" : "PUSH"
     }, {
       "hash" : "93866c7fb3fa7ff2b9bb978fe2bed922c28ad381",
       "status" : "SUCCESS",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=7675",
       "triggerID" : "93866c7fb3fa7ff2b9bb978fe2bed922c28ad381",
       "triggerType" : "PUSH"
     }, {
       "hash" : "4f6001bbde0a330c9cecb7783cf8999276701039",
       "status" : "PENDING",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=7744",
       "triggerID" : "4f6001bbde0a330c9cecb7783cf8999276701039",
       "triggerType" : "PUSH"
     } ]
   }-->
   ## CI report:
   
   * 93866c7fb3fa7ff2b9bb978fe2bed922c28ad381 Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=7675) 
   * 4f6001bbde0a330c9cecb7783cf8999276701039 Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=7744) 
   
   <details>
   <summary>Bot commands</summary>
     @hudi-bot supports the following commands:
   
    - `@hudi-bot run azure` re-run the last Azure build
   </details>


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [hudi] codope commented on a change in pull request #5186: [HUDI-3451] Delete metadata table when the write client disables MDT

Posted by GitBox <gi...@apache.org>.
codope commented on a change in pull request #5186:
URL: https://github.com/apache/hudi/pull/5186#discussion_r841047175



##########
File path: hudi-client/hudi-client-common/src/main/java/org/apache/hudi/table/HoodieTable.java
##########
@@ -802,6 +805,48 @@ public HoodieEngineContext getContext() {
     return Option.empty();
   }
 
+  /**
+   * Deletes the metadata table if the writer disables metadata table with hoodie.metadata.enable=false
+   */
+  public void maybeDeleteMetadataTable() {
+    if (shouldExecuteMetadataTableDeletion()) {
+      try {
+        Path mdtBasePath = new Path(HoodieTableMetadata.getMetadataTableBasePath(config.getBasePath()));
+        FileSystem fileSystem = metaClient.getFs();
+        if (fileSystem.exists(mdtBasePath)) {
+          LOG.info("Deleting metadata table because it is disabled in writer.");
+          fileSystem.delete(mdtBasePath, true);
+        }

Review comment:
       Can we reuse `HoodieTableMetadataUtil#deleteMetadataTable` or refactor existing/add another method in util?




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [hudi] hudi-bot commented on pull request #5186: [HUDI-3451] Add checks for metadata table init to avoid possible out-of-sync

Posted by GitBox <gi...@apache.org>.
hudi-bot commented on pull request #5186:
URL: https://github.com/apache/hudi/pull/5186#issuecomment-1084732506


   <!--
   Meta data
   {
     "version" : 1,
     "metaDataEntries" : [ {
       "hash" : "5a1c9f13870f52750e25f8658776c0e9c27a4426",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=7632",
       "triggerID" : "5a1c9f13870f52750e25f8658776c0e9c27a4426",
       "triggerType" : "PUSH"
     }, {
       "hash" : "82c4b1422058fd68b1d5843b1ab1386a96adbc64",
       "status" : "CANCELED",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=7646",
       "triggerID" : "82c4b1422058fd68b1d5843b1ab1386a96adbc64",
       "triggerType" : "PUSH"
     }, {
       "hash" : "f3a50b13c1802ffd1c33f9469d2f7dcafde47bc2",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "f3a50b13c1802ffd1c33f9469d2f7dcafde47bc2",
       "triggerType" : "PUSH"
     } ]
   }-->
   ## CI report:
   
   * 82c4b1422058fd68b1d5843b1ab1386a96adbc64 Azure: [CANCELED](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=7646) 
   * f3a50b13c1802ffd1c33f9469d2f7dcafde47bc2 UNKNOWN
   
   <details>
   <summary>Bot commands</summary>
     @hudi-bot supports the following commands:
   
    - `@hudi-bot run azure` re-run the last Azure build
   </details>


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [hudi] hudi-bot commented on pull request #5186: [HUDI-3451] Add checks for metadata table init to avoid possible out-of-sync

Posted by GitBox <gi...@apache.org>.
hudi-bot commented on pull request #5186:
URL: https://github.com/apache/hudi/pull/5186#issuecomment-1084636242


   <!--
   Meta data
   {
     "version" : 1,
     "metaDataEntries" : [ {
       "hash" : "5a1c9f13870f52750e25f8658776c0e9c27a4426",
       "status" : "SUCCESS",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=7632",
       "triggerID" : "5a1c9f13870f52750e25f8658776c0e9c27a4426",
       "triggerType" : "PUSH"
     } ]
   }-->
   ## CI report:
   
   * 5a1c9f13870f52750e25f8658776c0e9c27a4426 Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=7632) 
   
   <details>
   <summary>Bot commands</summary>
     @hudi-bot supports the following commands:
   
    - `@hudi-bot run azure` re-run the last Azure build
   </details>


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [hudi] yihua commented on pull request #5186: [HUDI-3451] Add checks for metadata table init to avoid possible out-of-sync

Posted by GitBox <gi...@apache.org>.
yihua commented on pull request #5186:
URL: https://github.com/apache/hudi/pull/5186#issuecomment-1085112734


   @zhangyue19921010 Thanks for your contribution!  As discussed, given the release timeline, I'm going to take over this PR and get it to the finish line today.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [hudi] yihua commented on pull request #5186: [HUDI-3451] Add checks for metadata table init to avoid possible out-of-sync

Posted by GitBox <gi...@apache.org>.
yihua commented on pull request #5186:
URL: https://github.com/apache/hudi/pull/5186#issuecomment-1086039243


   After a discussion, simply using `hoodie.table.metadata.enable` with two values (true/false) may not guarantee atomic deletion of metadata table in `.hoodie/metadata`.
   
   Let's go through the following failure scenario based on the current logic:
   1. Metadata table (MDT) is enabled in HoodieWriteConfig (HWC) for the writer.  No failure, MDT is initialized and commit succeeds.  `hoodie.table.metadata.enable` is set to true in the table config.
   2. Before the next commit, MDT is disabled in HWC.  No failure, MDT is removed.  `hoodie.table.metadata.enable` is set to false in the table config.
   3. Before the next commit, MDT is enabled in HWC.  MDT is created and the job fails before updating `hoodie.table.metadata.enable`.  Now, `hoodie.table.metadata.enable` is still false but there is MDT in `.hoodie/metadata`.
   4. Before the next commit, MDT is disabled in HWC.  The writer does not clean up MDT because `hoodie.table.metadata.enable` is false.  In such a case, the detention of MDT is not complete.  The lingering MDT can cause correctness issue later on if MDT is enabled again in HWC.
   
   We need to have three states like Hudi instant in the timeline: requested, inflight, and completed, to identify the failure scenario like above.  Given the complexity of the potential new design, this may not be able to make to 0.11.0 release.  Removing the blocker label.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org