You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@hudi.apache.org by "prashantwason (via GitHub)" <gi...@apache.org> on 2023/05/10 22:54:17 UTC
[GitHub] [hudi] prashantwason opened a new pull request, #8684: [HUDI-6200] Enhancements to the MDT for improving performance of larger indexes.
prashantwason opened a new pull request, #8684:
URL: https://github.com/apache/hudi/pull/8684
[HUDI-6200] Enhancements to the MDT for improving performance of larger indexes.
### Change Logs
TBD
### Impact
TBD
### Risk level (write none, low medium or high below)
TBD
### Documentation Update
None
### Contributor's checklist
- [ ] Read through [contributor's guide](https://hudi.apache.org/contribute/how-to-contribute)
- [ ] Change Logs and Impact were stated clearly
- [ ] Adequate tests were added if applicable
- [ ] CI passed
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #8684: [HUDI-6200] Enhancements to the MDT for improving performance of larger indexes.
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #8684:
URL: https://github.com/apache/hudi/pull/8684#issuecomment-1578065217
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "35148aeb4ba78eb6f3316c75f8a0a7e4c6d6df87",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17007",
"triggerID" : "35148aeb4ba78eb6f3316c75f8a0a7e4c6d6df87",
"triggerType" : "PUSH"
}, {
"hash" : "e2785f4675ddf74582ff34590608a5d71c5e9a2d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17008",
"triggerID" : "e2785f4675ddf74582ff34590608a5d71c5e9a2d",
"triggerType" : "PUSH"
}, {
"hash" : "364b1fe0dde303750ca0f8eebe5cd530c2a66a3a",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17042",
"triggerID" : "364b1fe0dde303750ca0f8eebe5cd530c2a66a3a",
"triggerType" : "PUSH"
}, {
"hash" : "9d73b9664eed48ada68721a92526640cfa06a021",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17098",
"triggerID" : "9d73b9664eed48ada68721a92526640cfa06a021",
"triggerType" : "PUSH"
}, {
"hash" : "7be538f4045a42ba33ab8fe62594178e2db75bbb",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17100",
"triggerID" : "7be538f4045a42ba33ab8fe62594178e2db75bbb",
"triggerType" : "PUSH"
}, {
"hash" : "cc0da2372d50d99c98c2ce4bcbe5a60303bde938",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17195",
"triggerID" : "cc0da2372d50d99c98c2ce4bcbe5a60303bde938",
"triggerType" : "PUSH"
}, {
"hash" : "da0e00e3655e06b064d11e20500317e2a1d2229c",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "da0e00e3655e06b064d11e20500317e2a1d2229c",
"triggerType" : "PUSH"
}, {
"hash" : "589c8cd7ffc5df7984c099d6b30fa117463db2b9",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17335",
"triggerID" : "589c8cd7ffc5df7984c099d6b30fa117463db2b9",
"triggerType" : "PUSH"
}, {
"hash" : "226c96ce48b33d7b1650d00e93bc5ea42ad276f6",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "226c96ce48b33d7b1650d00e93bc5ea42ad276f6",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* da0e00e3655e06b064d11e20500317e2a1d2229c UNKNOWN
* 589c8cd7ffc5df7984c099d6b30fa117463db2b9 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17335)
* 226c96ce48b33d7b1650d00e93bc5ea42ad276f6 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #8684: [HUDI-6200] Enhancements to the MDT for improving performance of larger indexes.
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #8684:
URL: https://github.com/apache/hudi/pull/8684#issuecomment-1582590820
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "35148aeb4ba78eb6f3316c75f8a0a7e4c6d6df87",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17007",
"triggerID" : "35148aeb4ba78eb6f3316c75f8a0a7e4c6d6df87",
"triggerType" : "PUSH"
}, {
"hash" : "e2785f4675ddf74582ff34590608a5d71c5e9a2d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17008",
"triggerID" : "e2785f4675ddf74582ff34590608a5d71c5e9a2d",
"triggerType" : "PUSH"
}, {
"hash" : "364b1fe0dde303750ca0f8eebe5cd530c2a66a3a",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17042",
"triggerID" : "364b1fe0dde303750ca0f8eebe5cd530c2a66a3a",
"triggerType" : "PUSH"
}, {
"hash" : "9d73b9664eed48ada68721a92526640cfa06a021",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17098",
"triggerID" : "9d73b9664eed48ada68721a92526640cfa06a021",
"triggerType" : "PUSH"
}, {
"hash" : "7be538f4045a42ba33ab8fe62594178e2db75bbb",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17100",
"triggerID" : "7be538f4045a42ba33ab8fe62594178e2db75bbb",
"triggerType" : "PUSH"
}, {
"hash" : "cc0da2372d50d99c98c2ce4bcbe5a60303bde938",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17195",
"triggerID" : "cc0da2372d50d99c98c2ce4bcbe5a60303bde938",
"triggerType" : "PUSH"
}, {
"hash" : "da0e00e3655e06b064d11e20500317e2a1d2229c",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "da0e00e3655e06b064d11e20500317e2a1d2229c",
"triggerType" : "PUSH"
}, {
"hash" : "589c8cd7ffc5df7984c099d6b30fa117463db2b9",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17335",
"triggerID" : "589c8cd7ffc5df7984c099d6b30fa117463db2b9",
"triggerType" : "PUSH"
}, {
"hash" : "226c96ce48b33d7b1650d00e93bc5ea42ad276f6",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17624",
"triggerID" : "226c96ce48b33d7b1650d00e93bc5ea42ad276f6",
"triggerType" : "PUSH"
}, {
"hash" : "ad81a08799a98cd6ee63b8bf792d42f16a98f775",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17634",
"triggerID" : "ad81a08799a98cd6ee63b8bf792d42f16a98f775",
"triggerType" : "PUSH"
}, {
"hash" : "fd2ec462e554190843ddcc8d946317ae76cb81e1",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "fd2ec462e554190843ddcc8d946317ae76cb81e1",
"triggerType" : "PUSH"
}, {
"hash" : "cae57c2b03a3131cfa7774fa4335f4d153f85d61",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17644",
"triggerID" : "cae57c2b03a3131cfa7774fa4335f4d153f85d61",
"triggerType" : "PUSH"
}, {
"hash" : "a21fb8753ed206fefcf598c9e134c280e90d3dba",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17654",
"triggerID" : "a21fb8753ed206fefcf598c9e134c280e90d3dba",
"triggerType" : "PUSH"
}, {
"hash" : "4dfaefde334b712b3ba272bb330242ddb7f4c0ac",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17655",
"triggerID" : "4dfaefde334b712b3ba272bb330242ddb7f4c0ac",
"triggerType" : "PUSH"
}, {
"hash" : "ca59a379b830f88fd72c175cb9b96a14b33550c0",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17660",
"triggerID" : "ca59a379b830f88fd72c175cb9b96a14b33550c0",
"triggerType" : "PUSH"
}, {
"hash" : "1d49e658315a2ae22a2c29e3d388e82f869b146d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17664",
"triggerID" : "1d49e658315a2ae22a2c29e3d388e82f869b146d",
"triggerType" : "PUSH"
}, {
"hash" : "df3a5b57a375fd344814b1f5a2451cfb79fea2c3",
"status" : "CANCELED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17669",
"triggerID" : "df3a5b57a375fd344814b1f5a2451cfb79fea2c3",
"triggerType" : "PUSH"
}, {
"hash" : "da92c6c310fccdb4f1e797b481eaf6bb027e551f",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17673",
"triggerID" : "da92c6c310fccdb4f1e797b481eaf6bb027e551f",
"triggerType" : "PUSH"
}, {
"hash" : "86902aa16075d75d61347a93819295d78cda0d2d",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "86902aa16075d75d61347a93819295d78cda0d2d",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* da0e00e3655e06b064d11e20500317e2a1d2229c UNKNOWN
* fd2ec462e554190843ddcc8d946317ae76cb81e1 UNKNOWN
* df3a5b57a375fd344814b1f5a2451cfb79fea2c3 Azure: [CANCELED](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17669)
* da92c6c310fccdb4f1e797b481eaf6bb027e551f Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17673)
* 86902aa16075d75d61347a93819295d78cda0d2d UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #8684: [HUDI-6200] Enhancements to the MDT for improving performance of larger indexes.
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #8684:
URL: https://github.com/apache/hudi/pull/8684#issuecomment-1579808002
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "35148aeb4ba78eb6f3316c75f8a0a7e4c6d6df87",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17007",
"triggerID" : "35148aeb4ba78eb6f3316c75f8a0a7e4c6d6df87",
"triggerType" : "PUSH"
}, {
"hash" : "e2785f4675ddf74582ff34590608a5d71c5e9a2d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17008",
"triggerID" : "e2785f4675ddf74582ff34590608a5d71c5e9a2d",
"triggerType" : "PUSH"
}, {
"hash" : "364b1fe0dde303750ca0f8eebe5cd530c2a66a3a",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17042",
"triggerID" : "364b1fe0dde303750ca0f8eebe5cd530c2a66a3a",
"triggerType" : "PUSH"
}, {
"hash" : "9d73b9664eed48ada68721a92526640cfa06a021",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17098",
"triggerID" : "9d73b9664eed48ada68721a92526640cfa06a021",
"triggerType" : "PUSH"
}, {
"hash" : "7be538f4045a42ba33ab8fe62594178e2db75bbb",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17100",
"triggerID" : "7be538f4045a42ba33ab8fe62594178e2db75bbb",
"triggerType" : "PUSH"
}, {
"hash" : "cc0da2372d50d99c98c2ce4bcbe5a60303bde938",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17195",
"triggerID" : "cc0da2372d50d99c98c2ce4bcbe5a60303bde938",
"triggerType" : "PUSH"
}, {
"hash" : "da0e00e3655e06b064d11e20500317e2a1d2229c",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "da0e00e3655e06b064d11e20500317e2a1d2229c",
"triggerType" : "PUSH"
}, {
"hash" : "589c8cd7ffc5df7984c099d6b30fa117463db2b9",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17335",
"triggerID" : "589c8cd7ffc5df7984c099d6b30fa117463db2b9",
"triggerType" : "PUSH"
}, {
"hash" : "226c96ce48b33d7b1650d00e93bc5ea42ad276f6",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17624",
"triggerID" : "226c96ce48b33d7b1650d00e93bc5ea42ad276f6",
"triggerType" : "PUSH"
}, {
"hash" : "ad81a08799a98cd6ee63b8bf792d42f16a98f775",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17634",
"triggerID" : "ad81a08799a98cd6ee63b8bf792d42f16a98f775",
"triggerType" : "PUSH"
}, {
"hash" : "fd2ec462e554190843ddcc8d946317ae76cb81e1",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "fd2ec462e554190843ddcc8d946317ae76cb81e1",
"triggerType" : "PUSH"
}, {
"hash" : "cae57c2b03a3131cfa7774fa4335f4d153f85d61",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "cae57c2b03a3131cfa7774fa4335f4d153f85d61",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* da0e00e3655e06b064d11e20500317e2a1d2229c UNKNOWN
* ad81a08799a98cd6ee63b8bf792d42f16a98f775 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17634)
* fd2ec462e554190843ddcc8d946317ae76cb81e1 UNKNOWN
* cae57c2b03a3131cfa7774fa4335f4d153f85d61 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] prashantwason commented on a diff in pull request #8684: [HUDI-6200] Enhancements to the MDT for improving performance of larger indexes.
Posted by "prashantwason (via GitHub)" <gi...@apache.org>.
prashantwason commented on code in PR #8684:
URL: https://github.com/apache/hudi/pull/8684#discussion_r1192138716
##########
hudi-common/src/main/java/org/apache/hudi/common/table/HoodieTableConfig.java:
##########
@@ -670,17 +670,75 @@ private Long getTableChecksum() {
return getLong(TABLE_CHECKSUM);
}
- public List<String> getMetadataPartitionsInflight() {
- return StringUtils.split(
- getStringOrDefault(TABLE_METADATA_PARTITIONS_INFLIGHT, StringUtils.EMPTY_STRING),
- CONFIG_VALUES_DELIMITER
- );
+ public Set<String> getMetadataPartitionsInflight() {
+ return new HashSet<>(StringUtils.split(
+ getStringOrDefault(TABLE_METADATA_PARTITIONS_INFLIGHT, StringUtils.EMPTY_STRING),
+ CONFIG_VALUES_DELIMITER));
}
public Set<String> getMetadataPartitions() {
return new HashSet<>(
- StringUtils.split(getStringOrDefault(TABLE_METADATA_PARTITIONS, StringUtils.EMPTY_STRING),
- CONFIG_VALUES_DELIMITER));
+ StringUtils.split(getStringOrDefault(TABLE_METADATA_PARTITIONS, StringUtils.EMPTY_STRING),
+ CONFIG_VALUES_DELIMITER));
+ }
+
+ /**
+ * @returns true if metadata table has been created and is being used for this dataset, else returns false.
+ */
+ public boolean isMetadataTableEnabled() {
+ return isMetadataPartitionEnabled(MetadataPartitionType.FILES);
+ }
+
+ /**
+ * Checks if metadata table is enabled and the specified partition has been initialized.
+ *
+ * @param partition The partition to check
+ * @returns true if the specific partition has been initialized, else returns false.
+ */
+ public boolean isMetadataPartitionEnabled(MetadataPartitionType partition) {
+ return getMetadataPartitions().contains(partition.getPartitionPath());
+ }
+
+ /**
+ * Enables or disables the specified metadata table partition.
+ *
+ * @param partition The partition
+ * @param enabled If true, the partition is enabled, else disabled
+ */
+ public void setMetadataPartitionState(MetadataPartitionType partition, boolean enabled) {
+ ValidationUtils.checkArgument(!partition.getPartitionPath().contains(CONFIG_VALUES_DELIMITER),
+ "Metadata Table partition path cannot contain a comma: " + partition.getPartitionPath());
+ Set<String> partitions = getMetadataPartitions();
+ Set<String> partitionsInflight = getMetadataPartitionsInflight();
+ if (enabled) {
+ partitions.add(partition.getPartitionPath());
+ partitionsInflight.remove(partition.getPartitionPath());
+ } else if (partition.equals(MetadataPartitionType.FILES)) {
+ // file listing partition is required for all other partitions to work
+ // Disabling file partition will also disable all partitions
+ partitions.clear();
+ partitionsInflight.clear();
+ } else {
+ partitions.remove(partition.getPartitionPath());
+ partitionsInflight.remove(partition.getPartitionPath());
+ }
+ setValue(TABLE_METADATA_PARTITIONS, partitions.stream().sorted().collect(Collectors.joining(CONFIG_VALUES_DELIMITER)));
+ setValue(TABLE_METADATA_PARTITIONS_INFLIGHT, partitionsInflight.stream().sorted().collect(Collectors.joining(CONFIG_VALUES_DELIMITER)));
+ }
+
+ /**
+ * Enables the specified metadata table partition as inflight.
+ *
+ * @param partition The list of partitions to enable as inflight.
+ */
+ public void setMetadataPartitionsAsInflight(List<MetadataPartitionType> partitionTypes) {
+ Set<String> partitions = getMetadataPartitionsInflight();
Review Comment:
Renamed
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] prashantwason commented on a diff in pull request #8684: [HUDI-6200] Enhancements to the MDT for improving performance of larger indexes.
Posted by "prashantwason (via GitHub)" <gi...@apache.org>.
prashantwason commented on code in PR #8684:
URL: https://github.com/apache/hudi/pull/8684#discussion_r1192139220
##########
hudi-common/src/main/java/org/apache/hudi/metadata/MetadataPartitionType.java:
##########
@@ -33,10 +33,6 @@ public enum MetadataPartitionType {
private final String partitionPath;
// FileId prefix used for all file groups in this partition.
private final String fileIdPrefix;
- // Total file groups
- // TODO fix: enum should not have any mutable aspect as this compromises whole idea
- // of the enum being static, immutable entity
- private int fileGroupCount = 1;
Review Comment:
Didnt see any code serializing and saving this.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #8684: [HUDI-6200] Enhancements to the MDT for improving performance of larger indexes.
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #8684:
URL: https://github.com/apache/hudi/pull/8684#issuecomment-1548542471
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "35148aeb4ba78eb6f3316c75f8a0a7e4c6d6df87",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17007",
"triggerID" : "35148aeb4ba78eb6f3316c75f8a0a7e4c6d6df87",
"triggerType" : "PUSH"
}, {
"hash" : "e2785f4675ddf74582ff34590608a5d71c5e9a2d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17008",
"triggerID" : "e2785f4675ddf74582ff34590608a5d71c5e9a2d",
"triggerType" : "PUSH"
}, {
"hash" : "364b1fe0dde303750ca0f8eebe5cd530c2a66a3a",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17042",
"triggerID" : "364b1fe0dde303750ca0f8eebe5cd530c2a66a3a",
"triggerType" : "PUSH"
}, {
"hash" : "9d73b9664eed48ada68721a92526640cfa06a021",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17098",
"triggerID" : "9d73b9664eed48ada68721a92526640cfa06a021",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 364b1fe0dde303750ca0f8eebe5cd530c2a66a3a Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17042)
* 9d73b9664eed48ada68721a92526640cfa06a021 Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17098)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] danny0405 commented on a diff in pull request #8684: [HUDI-6200] Enhancements to the MDT for improving performance of larger indexes.
Posted by "danny0405 (via GitHub)" <gi...@apache.org>.
danny0405 commented on code in PR #8684:
URL: https://github.com/apache/hudi/pull/8684#discussion_r1225695785
##########
hudi-client/hudi-spark-client/src/main/java/org/apache/hudi/metadata/SparkHoodieMetadataBulkInsertPartitioner.java:
##########
@@ -0,0 +1,111 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hudi.metadata;
+
+import java.io.Serializable;
+import java.util.ArrayList;
+import java.util.Comparator;
+import java.util.List;
+
+import org.apache.hudi.common.model.HoodieRecord;
+import org.apache.hudi.common.util.ValidationUtils;
+import org.apache.hudi.table.BulkInsertPartitioner;
+import org.apache.spark.Partitioner;
+import org.apache.spark.api.java.JavaRDD;
+
+import scala.Tuple2;
+
+/**
+ * A {@code BulkInsertPartitioner} implementation for Metadata Table to improve performance of initialization of metadata
+ * table partition when a very large number of records are inserted.
+ *
+ * This partitioner requires the records to be already tagged with location.
+ */
+public class SparkHoodieMetadataBulkInsertPartitioner implements BulkInsertPartitioner<JavaRDD<HoodieRecord>> {
+ final int numPartitions;
+ public SparkHoodieMetadataBulkInsertPartitioner(int numPartitions) {
+ this.numPartitions = numPartitions;
+ }
+
+ private class FileGroupPartitioner extends Partitioner {
+
+ @Override
+ public int getPartition(Object key) {
+ return ((Tuple2<Integer, String>)key)._1;
+ }
+
+ @Override
+ public int numPartitions() {
+ return numPartitions;
+ }
+ }
+
+ // FileIDs for the various partitions
+ private List<String> fileIDPfxs;
+
+ /**
+ * Partition the records by their location. The number of partitions is determined by the number of MDT fileGroups being udpated rather than the
+ * specific value of outputSparkPartitions.
+ */
+ @Override
+ public JavaRDD<HoodieRecord> repartitionRecords(JavaRDD<HoodieRecord> records, int outputSparkPartitions) {
+ Comparator<Tuple2<Integer, String>> keyComparator =
+ (Comparator<Tuple2<Integer, String>> & Serializable)(t1, t2) -> t1._2.compareTo(t2._2);
+
+ // Partition the records by their file group
+ JavaRDD<HoodieRecord> partitionedRDD = records
+ // key by <file group index, record key>. The file group index is used to partition and the record key is used to sort within the partition.
+ .keyBy(r -> {
+ int fileGroupIndex = HoodieTableMetadataUtil.getFileGroupIndexFromFileId(r.getCurrentLocation().getFileId());
+ return new Tuple2<>(fileGroupIndex, r.getRecordKey());
+ })
+ .repartitionAndSortWithinPartitions(new FileGroupPartitioner(), keyComparator)
+ .map(t -> t._2);
+
+ fileIDPfxs = partitionedRDD.mapPartitions(recordItr -> {
+ // Due to partitioning, all record in the partition should have same fileID. So we only can get the fileID prefix from the first record.
+ List<String> fileIds = new ArrayList<>(1);
+ if (recordItr.hasNext()) {
+ HoodieRecord record = recordItr.next();
+ final String fileID = HoodieTableMetadataUtil.getFileGroupPrefix(record.getCurrentLocation().getFileId());
+ fileIds.add(fileID);
+ } else {
+ // FileGroupPartitioner returns a fixed number of partition as part of numPartitions(). In the special case that recordsRDD has fewer
+ // records than fileGroupCount, some of these partitions (corresponding to fileGroups) will not have any data.
+ // But we still need to return a fileID for use within {@code BulkInsertMapFunction}
+ fileIds.add("");
+ }
+ return fileIds.iterator();
+ }, true).collect();
+ ValidationUtils.checkArgument(partitionedRDD.getNumPartitions() == fileIDPfxs.size(),
+ String.format("Generated fileIDPfxs (%d) are lesser in size than the partitions %d", fileIDPfxs.size(), partitionedRDD.getNumPartitions()));
+
+ return partitionedRDD;
+ }
+
+ @Override
+ public boolean arePartitionRecordsSorted() {
+ return true;
+ }
+
+ @Override
+ public String getFileIdPfx(int partitionId) {
+ return fileIDPfxs.get(partitionId);
Review Comment:
This may not be right when one Hfile in one file group is too large to write with, there is a upper threshold for each base file handle, `HoodieAvroHFileWrite#canWrite` may return false, then the write handle factory would suffix the file group id with another auto inc number like `-1` which is not correct any more.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] prashantwason commented on a diff in pull request #8684: [HUDI-6200] Enhancements to the MDT for improving performance of larger indexes.
Posted by "prashantwason (via GitHub)" <gi...@apache.org>.
prashantwason commented on code in PR #8684:
URL: https://github.com/apache/hudi/pull/8684#discussion_r1203518705
##########
hudi-client/hudi-spark-client/src/main/java/org/apache/hudi/client/SparkRDDWriteClient.java:
##########
@@ -337,15 +337,33 @@ protected void initMetadataTable(Option<String> instantTime) {
*
* @param inFlightInstantTimestamp - The in-flight action responsible for the metadata table initialization
*/
- private void initializeMetadataTable(Option<String> inFlightInstantTimestamp) {
- if (config.isMetadataTableEnabled()) {
- HoodieTableMetadataWriter writer = SparkHoodieBackedTableMetadataWriter.create(context.getHadoopConf().get(), config,
- context, Option.empty(), inFlightInstantTimestamp);
- try {
- writer.close();
- } catch (Exception e) {
- throw new HoodieException("Failed to instantiate Metadata table ", e);
+ private void initializeMetadataTable(WriteOperationType operationType, Option<String> inFlightInstantTimestamp) {
+ if (!config.isMetadataTableEnabled()) {
+ return;
+ }
+
+ try (HoodieTableMetadataWriter writer = SparkHoodieBackedTableMetadataWriter.create(context.getHadoopConf().get(), config,
+ context, Option.empty(), inFlightInstantTimestamp)) {
+ if (writer.isInitialized()) {
+ // Optimize the metadata table which involves compacton. cleaning, etc. This should only be called from writers.
+ switch (operationType) {
+ case INSERT:
+ case INSERT_PREPPED:
+ case UPSERT:
+ case UPSERT_PREPPED:
+ case BULK_INSERT:
+ case BULK_INSERT_PREPPED:
+ case DELETE:
Review Comment:
On second thoughts, the switch is not necessary. The above code is within a transaction lock so there should not be any conflicts of multiple writers optimizing MDT together. The checks within performTableServices should be light enough or we can optimize them.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #8684: [HUDI-6200] Enhancements to the MDT for improving performance of larger indexes.
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #8684:
URL: https://github.com/apache/hudi/pull/8684#issuecomment-1579802051
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "35148aeb4ba78eb6f3316c75f8a0a7e4c6d6df87",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17007",
"triggerID" : "35148aeb4ba78eb6f3316c75f8a0a7e4c6d6df87",
"triggerType" : "PUSH"
}, {
"hash" : "e2785f4675ddf74582ff34590608a5d71c5e9a2d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17008",
"triggerID" : "e2785f4675ddf74582ff34590608a5d71c5e9a2d",
"triggerType" : "PUSH"
}, {
"hash" : "364b1fe0dde303750ca0f8eebe5cd530c2a66a3a",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17042",
"triggerID" : "364b1fe0dde303750ca0f8eebe5cd530c2a66a3a",
"triggerType" : "PUSH"
}, {
"hash" : "9d73b9664eed48ada68721a92526640cfa06a021",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17098",
"triggerID" : "9d73b9664eed48ada68721a92526640cfa06a021",
"triggerType" : "PUSH"
}, {
"hash" : "7be538f4045a42ba33ab8fe62594178e2db75bbb",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17100",
"triggerID" : "7be538f4045a42ba33ab8fe62594178e2db75bbb",
"triggerType" : "PUSH"
}, {
"hash" : "cc0da2372d50d99c98c2ce4bcbe5a60303bde938",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17195",
"triggerID" : "cc0da2372d50d99c98c2ce4bcbe5a60303bde938",
"triggerType" : "PUSH"
}, {
"hash" : "da0e00e3655e06b064d11e20500317e2a1d2229c",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "da0e00e3655e06b064d11e20500317e2a1d2229c",
"triggerType" : "PUSH"
}, {
"hash" : "589c8cd7ffc5df7984c099d6b30fa117463db2b9",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17335",
"triggerID" : "589c8cd7ffc5df7984c099d6b30fa117463db2b9",
"triggerType" : "PUSH"
}, {
"hash" : "226c96ce48b33d7b1650d00e93bc5ea42ad276f6",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17624",
"triggerID" : "226c96ce48b33d7b1650d00e93bc5ea42ad276f6",
"triggerType" : "PUSH"
}, {
"hash" : "ad81a08799a98cd6ee63b8bf792d42f16a98f775",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17634",
"triggerID" : "ad81a08799a98cd6ee63b8bf792d42f16a98f775",
"triggerType" : "PUSH"
}, {
"hash" : "fd2ec462e554190843ddcc8d946317ae76cb81e1",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "fd2ec462e554190843ddcc8d946317ae76cb81e1",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* da0e00e3655e06b064d11e20500317e2a1d2229c UNKNOWN
* ad81a08799a98cd6ee63b8bf792d42f16a98f775 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17634)
* fd2ec462e554190843ddcc8d946317ae76cb81e1 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #8684: [HUDI-6200] Enhancements to the MDT for improving performance of larger indexes.
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #8684:
URL: https://github.com/apache/hudi/pull/8684#issuecomment-1581051062
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "35148aeb4ba78eb6f3316c75f8a0a7e4c6d6df87",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17007",
"triggerID" : "35148aeb4ba78eb6f3316c75f8a0a7e4c6d6df87",
"triggerType" : "PUSH"
}, {
"hash" : "e2785f4675ddf74582ff34590608a5d71c5e9a2d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17008",
"triggerID" : "e2785f4675ddf74582ff34590608a5d71c5e9a2d",
"triggerType" : "PUSH"
}, {
"hash" : "364b1fe0dde303750ca0f8eebe5cd530c2a66a3a",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17042",
"triggerID" : "364b1fe0dde303750ca0f8eebe5cd530c2a66a3a",
"triggerType" : "PUSH"
}, {
"hash" : "9d73b9664eed48ada68721a92526640cfa06a021",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17098",
"triggerID" : "9d73b9664eed48ada68721a92526640cfa06a021",
"triggerType" : "PUSH"
}, {
"hash" : "7be538f4045a42ba33ab8fe62594178e2db75bbb",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17100",
"triggerID" : "7be538f4045a42ba33ab8fe62594178e2db75bbb",
"triggerType" : "PUSH"
}, {
"hash" : "cc0da2372d50d99c98c2ce4bcbe5a60303bde938",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17195",
"triggerID" : "cc0da2372d50d99c98c2ce4bcbe5a60303bde938",
"triggerType" : "PUSH"
}, {
"hash" : "da0e00e3655e06b064d11e20500317e2a1d2229c",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "da0e00e3655e06b064d11e20500317e2a1d2229c",
"triggerType" : "PUSH"
}, {
"hash" : "589c8cd7ffc5df7984c099d6b30fa117463db2b9",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17335",
"triggerID" : "589c8cd7ffc5df7984c099d6b30fa117463db2b9",
"triggerType" : "PUSH"
}, {
"hash" : "226c96ce48b33d7b1650d00e93bc5ea42ad276f6",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17624",
"triggerID" : "226c96ce48b33d7b1650d00e93bc5ea42ad276f6",
"triggerType" : "PUSH"
}, {
"hash" : "ad81a08799a98cd6ee63b8bf792d42f16a98f775",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17634",
"triggerID" : "ad81a08799a98cd6ee63b8bf792d42f16a98f775",
"triggerType" : "PUSH"
}, {
"hash" : "fd2ec462e554190843ddcc8d946317ae76cb81e1",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "fd2ec462e554190843ddcc8d946317ae76cb81e1",
"triggerType" : "PUSH"
}, {
"hash" : "cae57c2b03a3131cfa7774fa4335f4d153f85d61",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17644",
"triggerID" : "cae57c2b03a3131cfa7774fa4335f4d153f85d61",
"triggerType" : "PUSH"
}, {
"hash" : "a21fb8753ed206fefcf598c9e134c280e90d3dba",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17654",
"triggerID" : "a21fb8753ed206fefcf598c9e134c280e90d3dba",
"triggerType" : "PUSH"
}, {
"hash" : "4dfaefde334b712b3ba272bb330242ddb7f4c0ac",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "4dfaefde334b712b3ba272bb330242ddb7f4c0ac",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* da0e00e3655e06b064d11e20500317e2a1d2229c UNKNOWN
* fd2ec462e554190843ddcc8d946317ae76cb81e1 UNKNOWN
* cae57c2b03a3131cfa7774fa4335f4d153f85d61 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17644)
* a21fb8753ed206fefcf598c9e134c280e90d3dba Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17654)
* 4dfaefde334b712b3ba272bb330242ddb7f4c0ac UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #8684: [HUDI-6200] Enhancements to the MDT for improving performance of larger indexes.
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #8684:
URL: https://github.com/apache/hudi/pull/8684#issuecomment-1578357497
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "35148aeb4ba78eb6f3316c75f8a0a7e4c6d6df87",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17007",
"triggerID" : "35148aeb4ba78eb6f3316c75f8a0a7e4c6d6df87",
"triggerType" : "PUSH"
}, {
"hash" : "e2785f4675ddf74582ff34590608a5d71c5e9a2d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17008",
"triggerID" : "e2785f4675ddf74582ff34590608a5d71c5e9a2d",
"triggerType" : "PUSH"
}, {
"hash" : "364b1fe0dde303750ca0f8eebe5cd530c2a66a3a",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17042",
"triggerID" : "364b1fe0dde303750ca0f8eebe5cd530c2a66a3a",
"triggerType" : "PUSH"
}, {
"hash" : "9d73b9664eed48ada68721a92526640cfa06a021",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17098",
"triggerID" : "9d73b9664eed48ada68721a92526640cfa06a021",
"triggerType" : "PUSH"
}, {
"hash" : "7be538f4045a42ba33ab8fe62594178e2db75bbb",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17100",
"triggerID" : "7be538f4045a42ba33ab8fe62594178e2db75bbb",
"triggerType" : "PUSH"
}, {
"hash" : "cc0da2372d50d99c98c2ce4bcbe5a60303bde938",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17195",
"triggerID" : "cc0da2372d50d99c98c2ce4bcbe5a60303bde938",
"triggerType" : "PUSH"
}, {
"hash" : "da0e00e3655e06b064d11e20500317e2a1d2229c",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "da0e00e3655e06b064d11e20500317e2a1d2229c",
"triggerType" : "PUSH"
}, {
"hash" : "589c8cd7ffc5df7984c099d6b30fa117463db2b9",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17335",
"triggerID" : "589c8cd7ffc5df7984c099d6b30fa117463db2b9",
"triggerType" : "PUSH"
}, {
"hash" : "226c96ce48b33d7b1650d00e93bc5ea42ad276f6",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17624",
"triggerID" : "226c96ce48b33d7b1650d00e93bc5ea42ad276f6",
"triggerType" : "PUSH"
}, {
"hash" : "ad81a08799a98cd6ee63b8bf792d42f16a98f775",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "ad81a08799a98cd6ee63b8bf792d42f16a98f775",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* da0e00e3655e06b064d11e20500317e2a1d2229c UNKNOWN
* 589c8cd7ffc5df7984c099d6b30fa117463db2b9 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17335)
* 226c96ce48b33d7b1650d00e93bc5ea42ad276f6 Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17624)
* ad81a08799a98cd6ee63b8bf792d42f16a98f775 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #8684: [HUDI-6200] Enhancements to the MDT for improving performance of larger indexes.
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #8684:
URL: https://github.com/apache/hudi/pull/8684#issuecomment-1580949056
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "35148aeb4ba78eb6f3316c75f8a0a7e4c6d6df87",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17007",
"triggerID" : "35148aeb4ba78eb6f3316c75f8a0a7e4c6d6df87",
"triggerType" : "PUSH"
}, {
"hash" : "e2785f4675ddf74582ff34590608a5d71c5e9a2d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17008",
"triggerID" : "e2785f4675ddf74582ff34590608a5d71c5e9a2d",
"triggerType" : "PUSH"
}, {
"hash" : "364b1fe0dde303750ca0f8eebe5cd530c2a66a3a",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17042",
"triggerID" : "364b1fe0dde303750ca0f8eebe5cd530c2a66a3a",
"triggerType" : "PUSH"
}, {
"hash" : "9d73b9664eed48ada68721a92526640cfa06a021",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17098",
"triggerID" : "9d73b9664eed48ada68721a92526640cfa06a021",
"triggerType" : "PUSH"
}, {
"hash" : "7be538f4045a42ba33ab8fe62594178e2db75bbb",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17100",
"triggerID" : "7be538f4045a42ba33ab8fe62594178e2db75bbb",
"triggerType" : "PUSH"
}, {
"hash" : "cc0da2372d50d99c98c2ce4bcbe5a60303bde938",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17195",
"triggerID" : "cc0da2372d50d99c98c2ce4bcbe5a60303bde938",
"triggerType" : "PUSH"
}, {
"hash" : "da0e00e3655e06b064d11e20500317e2a1d2229c",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "da0e00e3655e06b064d11e20500317e2a1d2229c",
"triggerType" : "PUSH"
}, {
"hash" : "589c8cd7ffc5df7984c099d6b30fa117463db2b9",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17335",
"triggerID" : "589c8cd7ffc5df7984c099d6b30fa117463db2b9",
"triggerType" : "PUSH"
}, {
"hash" : "226c96ce48b33d7b1650d00e93bc5ea42ad276f6",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17624",
"triggerID" : "226c96ce48b33d7b1650d00e93bc5ea42ad276f6",
"triggerType" : "PUSH"
}, {
"hash" : "ad81a08799a98cd6ee63b8bf792d42f16a98f775",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17634",
"triggerID" : "ad81a08799a98cd6ee63b8bf792d42f16a98f775",
"triggerType" : "PUSH"
}, {
"hash" : "fd2ec462e554190843ddcc8d946317ae76cb81e1",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "fd2ec462e554190843ddcc8d946317ae76cb81e1",
"triggerType" : "PUSH"
}, {
"hash" : "cae57c2b03a3131cfa7774fa4335f4d153f85d61",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17644",
"triggerID" : "cae57c2b03a3131cfa7774fa4335f4d153f85d61",
"triggerType" : "PUSH"
}, {
"hash" : "a21fb8753ed206fefcf598c9e134c280e90d3dba",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17654",
"triggerID" : "a21fb8753ed206fefcf598c9e134c280e90d3dba",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* da0e00e3655e06b064d11e20500317e2a1d2229c UNKNOWN
* fd2ec462e554190843ddcc8d946317ae76cb81e1 UNKNOWN
* cae57c2b03a3131cfa7774fa4335f4d153f85d61 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17644)
* a21fb8753ed206fefcf598c9e134c280e90d3dba Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17654)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #8684: [HUDI-6200] Enhancements to the MDT for improving performance of larger indexes.
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #8684:
URL: https://github.com/apache/hudi/pull/8684#issuecomment-1581951617
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "35148aeb4ba78eb6f3316c75f8a0a7e4c6d6df87",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17007",
"triggerID" : "35148aeb4ba78eb6f3316c75f8a0a7e4c6d6df87",
"triggerType" : "PUSH"
}, {
"hash" : "e2785f4675ddf74582ff34590608a5d71c5e9a2d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17008",
"triggerID" : "e2785f4675ddf74582ff34590608a5d71c5e9a2d",
"triggerType" : "PUSH"
}, {
"hash" : "364b1fe0dde303750ca0f8eebe5cd530c2a66a3a",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17042",
"triggerID" : "364b1fe0dde303750ca0f8eebe5cd530c2a66a3a",
"triggerType" : "PUSH"
}, {
"hash" : "9d73b9664eed48ada68721a92526640cfa06a021",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17098",
"triggerID" : "9d73b9664eed48ada68721a92526640cfa06a021",
"triggerType" : "PUSH"
}, {
"hash" : "7be538f4045a42ba33ab8fe62594178e2db75bbb",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17100",
"triggerID" : "7be538f4045a42ba33ab8fe62594178e2db75bbb",
"triggerType" : "PUSH"
}, {
"hash" : "cc0da2372d50d99c98c2ce4bcbe5a60303bde938",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17195",
"triggerID" : "cc0da2372d50d99c98c2ce4bcbe5a60303bde938",
"triggerType" : "PUSH"
}, {
"hash" : "da0e00e3655e06b064d11e20500317e2a1d2229c",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "da0e00e3655e06b064d11e20500317e2a1d2229c",
"triggerType" : "PUSH"
}, {
"hash" : "589c8cd7ffc5df7984c099d6b30fa117463db2b9",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17335",
"triggerID" : "589c8cd7ffc5df7984c099d6b30fa117463db2b9",
"triggerType" : "PUSH"
}, {
"hash" : "226c96ce48b33d7b1650d00e93bc5ea42ad276f6",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17624",
"triggerID" : "226c96ce48b33d7b1650d00e93bc5ea42ad276f6",
"triggerType" : "PUSH"
}, {
"hash" : "ad81a08799a98cd6ee63b8bf792d42f16a98f775",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17634",
"triggerID" : "ad81a08799a98cd6ee63b8bf792d42f16a98f775",
"triggerType" : "PUSH"
}, {
"hash" : "fd2ec462e554190843ddcc8d946317ae76cb81e1",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "fd2ec462e554190843ddcc8d946317ae76cb81e1",
"triggerType" : "PUSH"
}, {
"hash" : "cae57c2b03a3131cfa7774fa4335f4d153f85d61",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17644",
"triggerID" : "cae57c2b03a3131cfa7774fa4335f4d153f85d61",
"triggerType" : "PUSH"
}, {
"hash" : "a21fb8753ed206fefcf598c9e134c280e90d3dba",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17654",
"triggerID" : "a21fb8753ed206fefcf598c9e134c280e90d3dba",
"triggerType" : "PUSH"
}, {
"hash" : "4dfaefde334b712b3ba272bb330242ddb7f4c0ac",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17655",
"triggerID" : "4dfaefde334b712b3ba272bb330242ddb7f4c0ac",
"triggerType" : "PUSH"
}, {
"hash" : "ca59a379b830f88fd72c175cb9b96a14b33550c0",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17660",
"triggerID" : "ca59a379b830f88fd72c175cb9b96a14b33550c0",
"triggerType" : "PUSH"
}, {
"hash" : "1d49e658315a2ae22a2c29e3d388e82f869b146d",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17664",
"triggerID" : "1d49e658315a2ae22a2c29e3d388e82f869b146d",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* da0e00e3655e06b064d11e20500317e2a1d2229c UNKNOWN
* fd2ec462e554190843ddcc8d946317ae76cb81e1 UNKNOWN
* 1d49e658315a2ae22a2c29e3d388e82f869b146d Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17664)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] nsivabalan merged pull request #8684: [HUDI-6200] Enhancements to the MDT for improving performance of larger indexes.
Posted by "nsivabalan (via GitHub)" <gi...@apache.org>.
nsivabalan merged PR #8684:
URL: https://github.com/apache/hudi/pull/8684
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] danny0405 commented on a diff in pull request #8684: [HUDI-6200] Enhancements to the MDT for improving performance of larger indexes.
Posted by "danny0405 (via GitHub)" <gi...@apache.org>.
danny0405 commented on code in PR #8684:
URL: https://github.com/apache/hudi/pull/8684#discussion_r1223719557
##########
hudi-client/hudi-client-common/src/main/java/org/apache/hudi/metadata/HoodieBackedTableMetadataWriter.java:
##########
@@ -1052,51 +1091,81 @@ protected HoodieData<HoodieRecord> prepRecords(Map<MetadataPartitionType,
}
/**
- * Perform a compaction on the Metadata Table.
- *
- * Cases to be handled:
- * 1. We cannot perform compaction if there are previous inflight operations on the dataset. This is because
- * a compacted metadata base file at time Tx should represent all the actions on the dataset till time Tx.
- *
- * 2. In multi-writer scenario, a parallel operation with a greater instantTime may have completed creating a
- * deltacommit.
+ * Optimize the metadata table by running compaction, clean and archive as required.
+ * <p>
+ * Don't perform optimization if there are inflight operations on the dataset. This is for two reasons:
+ * - The compaction will contain the correct data as all failed operations have been rolled back.
+ * - Clean/compaction etc. will have the highest timestamp on the MDT and we won't be adding new operations
+ * with smaller timestamps to metadata table (makes for easier debugging)
+ * <p>
+ * This adds the limitations that long-running async operations (clustering, etc.) may cause delay in such MDT
+ * optimizations. We will relax this after MDT code has been hardened.
*/
- protected void compactIfNecessary(BaseHoodieWriteClient writeClient, String instantTime) {
- // finish off any pending compactions if any from previous attempt.
- writeClient.runAnyPendingCompactions();
-
- String latestDeltaCommitTimeInMetadataTable = metadataMetaClient.reloadActiveTimeline()
- .getDeltaCommitTimeline()
- .filterCompletedInstants()
- .lastInstant().orElseThrow(() -> new HoodieMetadataException("No completed deltacommit in metadata table"))
- .getTimestamp();
- // we need to find if there are any inflights in data table timeline before or equal to the latest delta commit in metadata table.
- // Whenever you want to change this logic, please ensure all below scenarios are considered.
- // a. There could be a chance that latest delta commit in MDT is committed in MDT, but failed in DT. And so findInstantsBeforeOrEquals() should be employed
- // b. There could be DT inflights after latest delta commit in MDT and we are ok with it. bcoz, the contract is, latest compaction instant time in MDT represents
- // any instants before that is already synced with metadata table.
- // c. Do consider out of order commits. For eg, c4 from DT could complete before c3. and we can't trigger compaction in MDT with c4 as base instant time, until every
- // instant before c4 is synced with metadata table.
- List<HoodieInstant> pendingInstants = dataMetaClient.reloadActiveTimeline().filterInflightsAndRequested()
- .findInstantsBeforeOrEquals(latestDeltaCommitTimeInMetadataTable).getInstants();
+ @Override
+ public void performTableServices(Option<String> inFlightInstantTimestamp) {
+ HoodieTimer metadataTableServicesTimer = HoodieTimer.start();
+ boolean allTableServicesExecutedSuccessfullyOrSkipped = true;
+ try {
+ BaseHoodieWriteClient writeClient = getWriteClient();
+ // Run any pending table services operations.
+ runPendingTableServicesOperations(writeClient);
+
+ // Check and run clean operations.
+ String latestDeltacommitTime = metadataMetaClient.reloadActiveTimeline().getDeltaCommitTimeline()
+ .filterCompletedInstants()
+ .lastInstant().get()
+ .getTimestamp();
+ LOG.info("Latest deltacommit time found is " + latestDeltacommitTime + ", running clean operations.");
+ cleanIfNecessary(writeClient, latestDeltacommitTime);
+
+ // Do timeline validation before scheduling compaction/logcompaction operations.
+ if (!validateTimelineBeforeSchedulingCompaction(inFlightInstantTimestamp, latestDeltacommitTime)) {
+ return;
Review Comment:
> unless compaction in MDT kicks in, archival might not have anything to do after last time it was able to archive something.
Then archiving will always be blocked by the compaction.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] prashantwason commented on a diff in pull request #8684: [HUDI-6200] Enhancements to the MDT for improving performance of larger indexes.
Posted by "prashantwason (via GitHub)" <gi...@apache.org>.
prashantwason commented on code in PR #8684:
URL: https://github.com/apache/hudi/pull/8684#discussion_r1194213033
##########
hudi-client/hudi-spark-client/src/main/java/org/apache/hudi/metadata/SparkHoodieMetadataBulkInsertPartitioner.java:
##########
@@ -0,0 +1,111 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hudi.metadata;
+
+import java.io.Serializable;
+import java.util.ArrayList;
+import java.util.Comparator;
+import java.util.List;
+
+import org.apache.hudi.common.model.HoodieRecord;
+import org.apache.hudi.common.util.ValidationUtils;
+import org.apache.hudi.table.BulkInsertPartitioner;
+import org.apache.spark.Partitioner;
+import org.apache.spark.api.java.JavaRDD;
+
+import scala.Tuple2;
+
+/**
+ * A {@code BulkInsertPartitioner} implementation for Metadata Table to improve performance of initialization of metadata
+ * table partition when a very large number of records are inserted.
+ *
+ * This partitioner requires the records to be already tagged with location.
+ */
+public class SparkHoodieMetadataBulkInsertPartitioner implements BulkInsertPartitioner<JavaRDD<HoodieRecord>> {
Review Comment:
Yes, added TestSparkHoodieMetadataBulkInsertPartitioner
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] prashantwason commented on a diff in pull request #8684: [HUDI-6200] Enhancements to the MDT for improving performance of larger indexes.
Posted by "prashantwason (via GitHub)" <gi...@apache.org>.
prashantwason commented on code in PR #8684:
URL: https://github.com/apache/hudi/pull/8684#discussion_r1194268603
##########
hudi-client/hudi-client-common/src/main/java/org/apache/hudi/metadata/HoodieBackedTableMetadataWriter.java:
##########
@@ -1097,87 +1165,76 @@ protected void cleanIfNecessary(BaseHoodieWriteClient writeClient, String instan
// Trigger cleaning with suffixes based on the same instant time. This ensures that any future
// delta commits synced over will not have an instant time lesser than the last completed instant on the
// metadata table.
- writeClient.clean(instantTime + "002");
+ writeClient.clean(HoodieTableMetadataUtil.createCleanTimestamp(instantTime));
writeClient.lazyRollbackFailedIndexing();
}
/**
- * This is invoked to initialize metadata table for a dataset.
- * Initial commit has special handling mechanism due to its scale compared to other regular commits.
- * During cold startup, the list of files to be committed can be huge.
- * So creating a HoodieCommitMetadata out of these large number of files,
- * and calling the existing update(HoodieCommitMetadata) function does not scale well.
- * Hence, we have a special commit just for the initialization scenario.
+ * Validates the timeline for both main and metadata tables.
*/
- private void initialCommit(String createInstantTime, List<MetadataPartitionType> partitionTypes) {
- // List all partitions in the basePath of the containing dataset
- LOG.info("Initializing metadata table by using file listings in " + dataWriteConfig.getBasePath());
- engineContext.setJobStatus(this.getClass().getSimpleName(), "Initializing metadata table by listing files and partitions: " + dataWriteConfig.getTableName());
-
- Map<MetadataPartitionType, HoodieData<HoodieRecord>> partitionToRecordsMap = new HashMap<>();
-
- // skip file system listing to populate metadata records if it's a fresh table.
- // this is applicable only if the table already has N commits and metadata is enabled at a later point in time.
- if (createInstantTime.equals(SOLO_COMMIT_TIMESTAMP)) { // SOLO_COMMIT_TIMESTAMP will be the initial commit time in MDT for a fresh table.
- // If not, last completed commit in data table will be chosen as the initial commit time.
- LOG.info("Triggering empty Commit to metadata to initialize");
- } else {
- List<DirectoryInfo> partitionInfoList = listAllPartitions(dataMetaClient);
- Map<String, Map<String, Long>> partitionToFilesMap = partitionInfoList.stream()
- .map(p -> {
- String partitionName = HoodieTableMetadataUtil.getPartitionIdentifier(p.getRelativePath());
- return Pair.of(partitionName, p.getFileNameToSizeMap());
- })
- .collect(Collectors.toMap(Pair::getKey, Pair::getValue));
-
- int totalDataFilesCount = partitionToFilesMap.values().stream().mapToInt(Map::size).sum();
- List<String> partitions = new ArrayList<>(partitionToFilesMap.keySet());
-
- if (partitionTypes.contains(MetadataPartitionType.FILES)) {
- // Record which saves the list of all partitions
- HoodieRecord allPartitionRecord = HoodieMetadataPayload.createPartitionListRecord(partitions);
- HoodieData<HoodieRecord> filesPartitionRecords = getFilesPartitionRecords(createInstantTime, partitionInfoList, allPartitionRecord);
- ValidationUtils.checkState(filesPartitionRecords.count() == (partitions.size() + 1));
- partitionToRecordsMap.put(MetadataPartitionType.FILES, filesPartitionRecords);
- }
-
- if (partitionTypes.contains(MetadataPartitionType.BLOOM_FILTERS) && totalDataFilesCount > 0) {
- final HoodieData<HoodieRecord> recordsRDD = HoodieTableMetadataUtil.convertFilesToBloomFilterRecords(
- engineContext, Collections.emptyMap(), partitionToFilesMap, getRecordsGenerationParams(), createInstantTime);
- partitionToRecordsMap.put(MetadataPartitionType.BLOOM_FILTERS, recordsRDD);
- }
-
- if (partitionTypes.contains(MetadataPartitionType.COLUMN_STATS) && totalDataFilesCount > 0) {
- final HoodieData<HoodieRecord> recordsRDD = HoodieTableMetadataUtil.convertFilesToColumnStatsRecords(
- engineContext, Collections.emptyMap(), partitionToFilesMap, getRecordsGenerationParams());
- partitionToRecordsMap.put(MetadataPartitionType.COLUMN_STATS, recordsRDD);
- }
- LOG.info("Committing " + partitions.size() + " partitions and " + totalDataFilesCount + " files to metadata");
+ private boolean validateTimelineBeforeSchedulingCompaction(Option<String> inFlightInstantTimestamp, String latestDeltacommitTime) {
+ // There should not be any incomplete instants on MDT
+ HoodieActiveTimeline metadataTimeline = metadataMetaClient.reloadActiveTimeline();
+ List<HoodieInstant> pendingInstantsOnMetadataTable = metadataTimeline.filterInflightsAndRequested().getInstants();
+ if (!pendingInstantsOnMetadataTable.isEmpty()) {
+ LOG.info(String.format(
+ "Cannot compact MDT as there are %d inflight instants: %s",
+ pendingInstantsOnMetadataTable.size(), Arrays.toString(pendingInstantsOnMetadataTable.toArray())));
+ metrics.ifPresent(m -> m.incrementMetric(HoodieMetadataMetrics.SKIP_TABLE_SERVICES, 1));
+ return false;
}
- commit(createInstantTime, partitionToRecordsMap, false);
- }
+ // There should not be any incomplete instants on dataset
+ HoodieActiveTimeline datasetTimeline = dataMetaClient.reloadActiveTimeline();
+ List<HoodieInstant> pendingInstantsOnDataset = datasetTimeline.filterInflightsAndRequested().getInstantsAsStream()
+ .filter(i -> !inFlightInstantTimestamp.isPresent() || !i.getTimestamp().equals(inFlightInstantTimestamp.get()))
+ .collect(Collectors.toList());
+ if (!pendingInstantsOnDataset.isEmpty()) {
+ LOG.info(String.format(
+ "Cannot compact MDT as there are %d inflight instants on dataset before latest deltacommit %s: %s",
+ pendingInstantsOnDataset.size(), latestDeltacommitTime, Arrays.toString(pendingInstantsOnDataset.toArray())));
+ metrics.ifPresent(m -> m.incrementMetric(HoodieMetadataMetrics.SKIP_TABLE_SERVICES, 1));
+ return false;
+ }
- private HoodieData<HoodieRecord> getFilesPartitionRecords(String createInstantTime, List<DirectoryInfo> partitionInfoList, HoodieRecord allPartitionRecord) {
- HoodieData<HoodieRecord> filesPartitionRecords = engineContext.parallelize(Arrays.asList(allPartitionRecord), 1);
- if (partitionInfoList.isEmpty()) {
- return filesPartitionRecords;
+ // Check if the inflight commit is greater than all the completed commits.
+ Option<HoodieInstant> lastCompletedInstant = dataMetaClient.getActiveTimeline().filterCompletedInstants().lastInstant();
+ if (!lastCompletedInstant.isPresent()) {
+ LOG.info("Last completed commit is not present.");
+ return false;
+ }
+ if (HoodieTimeline.compareTimestamps(lastCompletedInstant.get().getTimestamp(),
+ HoodieTimeline.GREATER_THAN, inFlightInstantTimestamp.get())) {
+ // Completed commits validation failed.
+ LOG.info(String.format(
+ "Cannot compact MDT as there is %s that is greater than inflight instant: %s",
+ lastCompletedInstant.get(), inFlightInstantTimestamp.get()));
+ return false;
}
- HoodieData<HoodieRecord> fileListRecords = engineContext.parallelize(partitionInfoList, partitionInfoList.size()).map(partitionInfo -> {
- Map<String, Long> fileNameToSizeMap = partitionInfo.getFileNameToSizeMap();
- // filter for files that are part of the completed commits
- Map<String, Long> validFileNameToSizeMap = fileNameToSizeMap.entrySet().stream().filter(fileSizePair -> {
- String commitTime = FSUtils.getCommitTime(fileSizePair.getKey());
- return HoodieTimeline.compareTimestamps(commitTime, HoodieTimeline.LESSER_THAN_OR_EQUALS, createInstantTime);
- }).collect(Collectors.toMap(Map.Entry::getKey, Map.Entry::getValue));
+ String latestDeltaCommitTimeInMetadataTable = metadataMetaClient.reloadActiveTimeline()
Review Comment:
Removed the reload as the latestDeltaCommitTimeInMetadataTable argument is created after reloading in the caller.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] nsivabalan commented on a diff in pull request #8684: [HUDI-6200] Enhancements to the MDT for improving performance of larger indexes.
Posted by "nsivabalan (via GitHub)" <gi...@apache.org>.
nsivabalan commented on code in PR #8684:
URL: https://github.com/apache/hudi/pull/8684#discussion_r1192717879
##########
hudi-client/hudi-client-common/src/main/java/org/apache/hudi/metadata/HoodieBackedTableMetadataWriter.java:
##########
@@ -562,53 +532,144 @@ private <T extends SpecificRecordBase> boolean isCommitRevertedByInFlightAction(
/**
* Initialize the Metadata Table by listing files and partitions from the file system.
*
- * @param dataMetaClient - {@code HoodieTableMetaClient} for the dataset.
+ * @param initializationTime - Timestamp to use for the commit
+ * @param partitionsToInit - List of MDT partitions to initialize
* @param inflightInstantTimestamp - Current action instant responsible for this initialization
*/
- private boolean initializeFromFilesystem(HoodieTableMetaClient dataMetaClient,
+ private boolean initializeFromFilesystem(String initializationTime, List<MetadataPartitionType> partitionsToInit,
Option<String> inflightInstantTimestamp) throws IOException {
if (anyPendingDataInstant(dataMetaClient, inflightInstantTimestamp)) {
return false;
}
- String createInstantTime = getInitialCommitInstantTime(dataMetaClient);
-
- initializeMetaClient(DEFAULT_METADATA_POPULATE_META_FIELDS);
- initTableMetadata();
- // if async metadata indexing is enabled,
- // then only initialize files partition as other partitions will be built using HoodieIndexer
- List<MetadataPartitionType> enabledPartitionTypes = new ArrayList<>();
- if (dataWriteConfig.isMetadataAsyncIndex()) {
- enabledPartitionTypes.add(MetadataPartitionType.FILES);
- } else {
- // all enabled ones should be initialized
- enabledPartitionTypes = this.enabledPartitionTypes;
+ // FILES partition is always initialized first
+ ValidationUtils.checkArgument(!partitionsToInit.contains(MetadataPartitionType.FILES)
+ || partitionsToInit.get(0).equals(MetadataPartitionType.FILES), "FILES partition should be initialized first: " + partitionsToInit);
+
+ metadataMetaClient = initializeMetaClient();
+
+ // Get a complete list of files and partitions from the file system or from already initialized FILES partition of MDT
+ boolean filesPartitionAvailable = dataMetaClient.getTableConfig().isMetadataPartitionEnabled(MetadataPartitionType.FILES);
+ List<DirectoryInfo> partitionInfoList = filesPartitionAvailable ? listAllPartitionsFromMDT(initializationTime) : listAllPartitionsFromFilesystem(initializationTime);
+ Map<String, Map<String, Long>> partitionToFilesMap = partitionInfoList.stream()
+ .map(p -> {
+ String partitionName = HoodieTableMetadataUtil.getPartitionIdentifier(p.getRelativePath());
+ return Pair.of(partitionName, p.getFileNameToSizeMap());
+ })
+ .collect(Collectors.toMap(Pair::getKey, Pair::getValue));
+
+ for (MetadataPartitionType partitionType : partitionsToInit) {
+ // Find the commit timestamp to use for this partition. Each initialization should use its own unique commit time.
+ String commitTimeForPartition = generateUniqueCommitInstantTime(initializationTime);
+
+ LOG.info("Initializing MDT partition " + partitionType + " at instant " + commitTimeForPartition);
+
+ Pair<Integer, HoodieData<HoodieRecord>> fileGroupCountAndRecordsPair;
+ switch (partitionType) {
+ case FILES:
+ fileGroupCountAndRecordsPair = initializeFilesPartition(initializationTime, partitionInfoList);
+ break;
+ case BLOOM_FILTERS:
+ fileGroupCountAndRecordsPair = initializeBloomFiltersPartition(initializationTime, partitionToFilesMap);
+ break;
+ case COLUMN_STATS:
+ fileGroupCountAndRecordsPair = initializeColumnStatsPartition(partitionToFilesMap);
+ break;
+ default:
+ throw new HoodieMetadataException("Unsupported MDT partition type: " + partitionType);
+ }
+
+ // Generate the file groups
+ final int fileGroupCount = fileGroupCountAndRecordsPair.getKey();
+ ValidationUtils.checkArgument(fileGroupCount > 0, "FileGroup count for MDT partition " + partitionType + " should be > 0");
+ initializeFileGroups(dataMetaClient, partitionType, commitTimeForPartition, fileGroupCount);
+
+ // Perform the commit using bulkCommit
+ HoodieData<HoodieRecord> records = fileGroupCountAndRecordsPair.getValue();
+ bulkCommit(commitTimeForPartition, partitionType, records, fileGroupCount);
+ metadataMetaClient.reloadActiveTimeline();
+ dataMetaClient = HoodieTableMetadataUtil.setMetadataPartitionState(dataMetaClient, partitionType, true);
}
- initializeEnabledFileGroups(dataMetaClient, createInstantTime, enabledPartitionTypes);
- initialCommit(createInstantTime, enabledPartitionTypes);
- updateInitializedPartitionsInTableConfig(enabledPartitionTypes);
+
return true;
}
- private String getInitialCommitInstantTime(HoodieTableMetaClient dataMetaClient) {
- // If there is no commit on the dataset yet, use the SOLO_COMMIT_TIMESTAMP as the instant time for initial commit
- // Otherwise, we use the timestamp of the latest completed action.
- String createInstantTime = dataMetaClient.getActiveTimeline().filterCompletedInstants()
- .getReverseOrderedInstants().findFirst().map(HoodieInstant::getTimestamp).orElse(SOLO_COMMIT_TIMESTAMP);
- LOG.info("Creating a new metadata table in " + metadataWriteConfig.getBasePath() + " at instant " + createInstantTime);
- return createInstantTime;
+ /**
+ * Returns a unique timestamp to use for initializing a MDT partition.
+ * <p>
+ * Since commits are immutable, we should use unique timestamps to initialize each partition. For this, we will add a suffix to the given initializationTime
+ * until we find a unique timestamp.
+ *
+ * @param initializationTime Timestamp from dataset to use for initialization
+ * @return a unique timestamp for MDT
+ */
+ private String generateUniqueCommitInstantTime(String initializationTime) {
+ for (int offset = 0; ; ++offset) {
+ final String commitInstantTime = HoodieTableMetadataUtil.createIndexInitTimestamp(initializationTime, offset);
+ if (!metadataMetaClient.getCommitTimeline().containsInstant(commitInstantTime)) {
+ return commitInstantTime;
+ }
+ }
}
- private boolean anyPendingDataInstant(HoodieTableMetaClient dataMetaClient, Option<String> inflightInstantTimestamp) {
- ValidationUtils.checkState(enabled, "Metadata table cannot be initialized as it is not enabled");
+ private Pair<Integer, HoodieData<HoodieRecord>> initializeColumnStatsPartition(Map<String, Map<String, Long>> partitionToFilesMap) {
Review Comment:
we do have some tests in TestHoodieBackedMetadata
for eg: testTableOperationsWithMetadataIndex. But there is no actual validation for col stats nor bloom filter records. So, I have added https://issues.apache.org/jira/browse/HUDI-6206
we should def enhance tests.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] prashantwason commented on a diff in pull request #8684: [HUDI-6200] Enhancements to the MDT for improving performance of larger indexes.
Posted by "prashantwason (via GitHub)" <gi...@apache.org>.
prashantwason commented on code in PR #8684:
URL: https://github.com/apache/hudi/pull/8684#discussion_r1192156057
##########
hudi-client/hudi-client-common/src/main/java/org/apache/hudi/metadata/HoodieBackedTableMetadataWriter.java:
##########
@@ -562,53 +532,144 @@ private <T extends SpecificRecordBase> boolean isCommitRevertedByInFlightAction(
/**
* Initialize the Metadata Table by listing files and partitions from the file system.
*
- * @param dataMetaClient - {@code HoodieTableMetaClient} for the dataset.
+ * @param initializationTime - Timestamp to use for the commit
+ * @param partitionsToInit - List of MDT partitions to initialize
* @param inflightInstantTimestamp - Current action instant responsible for this initialization
*/
- private boolean initializeFromFilesystem(HoodieTableMetaClient dataMetaClient,
+ private boolean initializeFromFilesystem(String initializationTime, List<MetadataPartitionType> partitionsToInit,
Option<String> inflightInstantTimestamp) throws IOException {
if (anyPendingDataInstant(dataMetaClient, inflightInstantTimestamp)) {
return false;
}
- String createInstantTime = getInitialCommitInstantTime(dataMetaClient);
-
- initializeMetaClient(DEFAULT_METADATA_POPULATE_META_FIELDS);
- initTableMetadata();
- // if async metadata indexing is enabled,
- // then only initialize files partition as other partitions will be built using HoodieIndexer
- List<MetadataPartitionType> enabledPartitionTypes = new ArrayList<>();
- if (dataWriteConfig.isMetadataAsyncIndex()) {
- enabledPartitionTypes.add(MetadataPartitionType.FILES);
- } else {
- // all enabled ones should be initialized
- enabledPartitionTypes = this.enabledPartitionTypes;
+ // FILES partition is always initialized first
+ ValidationUtils.checkArgument(!partitionsToInit.contains(MetadataPartitionType.FILES)
+ || partitionsToInit.get(0).equals(MetadataPartitionType.FILES), "FILES partition should be initialized first: " + partitionsToInit);
+
+ metadataMetaClient = initializeMetaClient();
+
+ // Get a complete list of files and partitions from the file system or from already initialized FILES partition of MDT
+ boolean filesPartitionAvailable = dataMetaClient.getTableConfig().isMetadataPartitionEnabled(MetadataPartitionType.FILES);
+ List<DirectoryInfo> partitionInfoList = filesPartitionAvailable ? listAllPartitionsFromMDT(initializationTime) : listAllPartitionsFromFilesystem(initializationTime);
+ Map<String, Map<String, Long>> partitionToFilesMap = partitionInfoList.stream()
+ .map(p -> {
+ String partitionName = HoodieTableMetadataUtil.getPartitionIdentifier(p.getRelativePath());
+ return Pair.of(partitionName, p.getFileNameToSizeMap());
+ })
+ .collect(Collectors.toMap(Pair::getKey, Pair::getValue));
+
+ for (MetadataPartitionType partitionType : partitionsToInit) {
+ // Find the commit timestamp to use for this partition. Each initialization should use its own unique commit time.
+ String commitTimeForPartition = generateUniqueCommitInstantTime(initializationTime);
+
+ LOG.info("Initializing MDT partition " + partitionType + " at instant " + commitTimeForPartition);
+
+ Pair<Integer, HoodieData<HoodieRecord>> fileGroupCountAndRecordsPair;
+ switch (partitionType) {
+ case FILES:
+ fileGroupCountAndRecordsPair = initializeFilesPartition(initializationTime, partitionInfoList);
+ break;
+ case BLOOM_FILTERS:
+ fileGroupCountAndRecordsPair = initializeBloomFiltersPartition(initializationTime, partitionToFilesMap);
+ break;
+ case COLUMN_STATS:
+ fileGroupCountAndRecordsPair = initializeColumnStatsPartition(partitionToFilesMap);
+ break;
+ default:
+ throw new HoodieMetadataException("Unsupported MDT partition type: " + partitionType);
+ }
+
+ // Generate the file groups
+ final int fileGroupCount = fileGroupCountAndRecordsPair.getKey();
+ ValidationUtils.checkArgument(fileGroupCount > 0, "FileGroup count for MDT partition " + partitionType + " should be > 0");
+ initializeFileGroups(dataMetaClient, partitionType, commitTimeForPartition, fileGroupCount);
+
+ // Perform the commit using bulkCommit
+ HoodieData<HoodieRecord> records = fileGroupCountAndRecordsPair.getValue();
+ bulkCommit(commitTimeForPartition, partitionType, records, fileGroupCount);
+ metadataMetaClient.reloadActiveTimeline();
+ dataMetaClient = HoodieTableMetadataUtil.setMetadataPartitionState(dataMetaClient, partitionType, true);
}
- initializeEnabledFileGroups(dataMetaClient, createInstantTime, enabledPartitionTypes);
- initialCommit(createInstantTime, enabledPartitionTypes);
- updateInitializedPartitionsInTableConfig(enabledPartitionTypes);
+
return true;
}
- private String getInitialCommitInstantTime(HoodieTableMetaClient dataMetaClient) {
- // If there is no commit on the dataset yet, use the SOLO_COMMIT_TIMESTAMP as the instant time for initial commit
- // Otherwise, we use the timestamp of the latest completed action.
- String createInstantTime = dataMetaClient.getActiveTimeline().filterCompletedInstants()
- .getReverseOrderedInstants().findFirst().map(HoodieInstant::getTimestamp).orElse(SOLO_COMMIT_TIMESTAMP);
- LOG.info("Creating a new metadata table in " + metadataWriteConfig.getBasePath() + " at instant " + createInstantTime);
- return createInstantTime;
+ /**
+ * Returns a unique timestamp to use for initializing a MDT partition.
+ * <p>
+ * Since commits are immutable, we should use unique timestamps to initialize each partition. For this, we will add a suffix to the given initializationTime
+ * until we find a unique timestamp.
+ *
+ * @param initializationTime Timestamp from dataset to use for initialization
+ * @return a unique timestamp for MDT
+ */
+ private String generateUniqueCommitInstantTime(String initializationTime) {
+ for (int offset = 0; ; ++offset) {
+ final String commitInstantTime = HoodieTableMetadataUtil.createIndexInitTimestamp(initializationTime, offset);
+ if (!metadataMetaClient.getCommitTimeline().containsInstant(commitInstantTime)) {
+ return commitInstantTime;
+ }
+ }
}
- private boolean anyPendingDataInstant(HoodieTableMetaClient dataMetaClient, Option<String> inflightInstantTimestamp) {
- ValidationUtils.checkState(enabled, "Metadata table cannot be initialized as it is not enabled");
+ private Pair<Integer, HoodieData<HoodieRecord>> initializeColumnStatsPartition(Map<String, Map<String, Long>> partitionToFilesMap) {
Review Comment:
We will need to check if there are existing unit tests that cover multi index initialization. I haven't added any new ones.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] prashantwason commented on a diff in pull request #8684: [HUDI-6200] Enhancements to the MDT for improving performance of larger indexes.
Posted by "prashantwason (via GitHub)" <gi...@apache.org>.
prashantwason commented on code in PR #8684:
URL: https://github.com/apache/hudi/pull/8684#discussion_r1192161017
##########
hudi-client/hudi-client-common/src/main/java/org/apache/hudi/metadata/HoodieBackedTableMetadataWriter.java:
##########
@@ -619,41 +680,34 @@ private boolean anyPendingDataInstant(HoodieTableMetaClient dataMetaClient, Opti
return false;
}
- private void updateInitializedPartitionsInTableConfig(List<MetadataPartitionType> partitionTypes) {
- Set<String> completedPartitions = dataMetaClient.getTableConfig().getMetadataPartitions();
- completedPartitions.addAll(partitionTypes.stream().map(MetadataPartitionType::getPartitionPath).collect(Collectors.toSet()));
- dataMetaClient.getTableConfig().setValue(HoodieTableConfig.TABLE_METADATA_PARTITIONS.key(), String.join(",", completedPartitions));
- HoodieTableConfig.update(dataMetaClient.getFs(), new Path(dataMetaClient.getMetaPath()), dataMetaClient.getTableConfig().getProps());
- }
-
- private HoodieTableMetaClient initializeMetaClient(boolean populateMetaFields) throws IOException {
+ private HoodieTableMetaClient initializeMetaClient() throws IOException {
return HoodieTableMetaClient.withPropertyBuilder()
.setTableType(HoodieTableType.MERGE_ON_READ)
.setTableName(tableName)
.setArchiveLogFolder(ARCHIVELOG_FOLDER.defaultValue())
.setPayloadClassName(HoodieMetadataPayload.class.getName())
.setBaseFileFormat(HoodieFileFormat.HFILE.toString())
.setRecordKeyFields(RECORD_KEY_FIELD_NAME)
- .setPopulateMetaFields(populateMetaFields)
- .setKeyGeneratorClassProp(HoodieTableMetadataKeyGenerator.class.getCanonicalName())
- .initTable(hadoopConf.get(), metadataWriteConfig.getBasePath());
+ .setPopulateMetaFields(DEFAULT_METADATA_POPULATE_META_FIELDS)
+ .setKeyGeneratorClassProp(HoodieTableMetadataKeyGenerator.class.getCanonicalName())
+ .initTable(hadoopConf.get(), metadataWriteConfig.getBasePath());
Review Comment:
fixed
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] nsivabalan commented on a diff in pull request #8684: [HUDI-6200] Enhancements to the MDT for improving performance of larger indexes.
Posted by "nsivabalan (via GitHub)" <gi...@apache.org>.
nsivabalan commented on code in PR #8684:
URL: https://github.com/apache/hudi/pull/8684#discussion_r1192728731
##########
hudi-common/src/main/java/org/apache/hudi/common/table/HoodieTableConfig.java:
##########
@@ -670,17 +670,75 @@ private Long getTableChecksum() {
return getLong(TABLE_CHECKSUM);
}
- public List<String> getMetadataPartitionsInflight() {
- return StringUtils.split(
- getStringOrDefault(TABLE_METADATA_PARTITIONS_INFLIGHT, StringUtils.EMPTY_STRING),
- CONFIG_VALUES_DELIMITER
- );
+ public Set<String> getMetadataPartitionsInflight() {
+ return new HashSet<>(StringUtils.split(
+ getStringOrDefault(TABLE_METADATA_PARTITIONS_INFLIGHT, StringUtils.EMPTY_STRING),
+ CONFIG_VALUES_DELIMITER));
}
public Set<String> getMetadataPartitions() {
return new HashSet<>(
- StringUtils.split(getStringOrDefault(TABLE_METADATA_PARTITIONS, StringUtils.EMPTY_STRING),
- CONFIG_VALUES_DELIMITER));
+ StringUtils.split(getStringOrDefault(TABLE_METADATA_PARTITIONS, StringUtils.EMPTY_STRING),
+ CONFIG_VALUES_DELIMITER));
+ }
+
+ /**
+ * @returns true if metadata table has been created and is being used for this dataset, else returns false.
+ */
+ public boolean isMetadataTableEnabled() {
+ return isMetadataPartitionEnabled(MetadataPartitionType.FILES);
+ }
+
+ /**
+ * Checks if metadata table is enabled and the specified partition has been initialized.
+ *
+ * @param partition The partition to check
+ * @returns true if the specific partition has been initialized, else returns false.
+ */
+ public boolean isMetadataPartitionEnabled(MetadataPartitionType partition) {
+ return getMetadataPartitions().contains(partition.getPartitionPath());
+ }
+
+ /**
+ * Enables or disables the specified metadata table partition.
+ *
+ * @param partition The partition
+ * @param enabled If true, the partition is enabled, else disabled
+ */
+ public void setMetadataPartitionState(MetadataPartitionType partition, boolean enabled) {
+ ValidationUtils.checkArgument(!partition.getPartitionPath().contains(CONFIG_VALUES_DELIMITER),
+ "Metadata Table partition path cannot contain a comma: " + partition.getPartitionPath());
+ Set<String> partitions = getMetadataPartitions();
+ Set<String> partitionsInflight = getMetadataPartitionsInflight();
+ if (enabled) {
+ partitions.add(partition.getPartitionPath());
+ partitionsInflight.remove(partition.getPartitionPath());
+ } else if (partition.equals(MetadataPartitionType.FILES)) {
+ // file listing partition is required for all other partitions to work
+ // Disabling file partition will also disable all partitions
+ partitions.clear();
+ partitionsInflight.clear();
+ } else {
+ partitions.remove(partition.getPartitionPath());
+ partitionsInflight.remove(partition.getPartitionPath());
+ }
+ setValue(TABLE_METADATA_PARTITIONS, partitions.stream().sorted().collect(Collectors.joining(CONFIG_VALUES_DELIMITER)));
+ setValue(TABLE_METADATA_PARTITIONS_INFLIGHT, partitionsInflight.stream().sorted().collect(Collectors.joining(CONFIG_VALUES_DELIMITER)));
+ }
+
+ /**
+ * Enables the specified metadata table partition as inflight.
+ *
+ * @param partition The list of partitions to enable as inflight.
+ */
+ public void setMetadataPartitionsAsInflight(List<MetadataPartitionType> partitionTypes) {
+ Set<String> partitions = getMetadataPartitionsInflight();
Review Comment:
oh, I meant
```
Set<String> partitions
```
```
Set<String> partitionsInflight
```
##########
hudi-common/src/main/java/org/apache/hudi/common/table/HoodieTableConfig.java:
##########
@@ -670,17 +670,75 @@ private Long getTableChecksum() {
return getLong(TABLE_CHECKSUM);
}
- public List<String> getMetadataPartitionsInflight() {
- return StringUtils.split(
- getStringOrDefault(TABLE_METADATA_PARTITIONS_INFLIGHT, StringUtils.EMPTY_STRING),
- CONFIG_VALUES_DELIMITER
- );
+ public Set<String> getMetadataPartitionsInflight() {
+ return new HashSet<>(StringUtils.split(
+ getStringOrDefault(TABLE_METADATA_PARTITIONS_INFLIGHT, StringUtils.EMPTY_STRING),
+ CONFIG_VALUES_DELIMITER));
}
public Set<String> getMetadataPartitions() {
return new HashSet<>(
- StringUtils.split(getStringOrDefault(TABLE_METADATA_PARTITIONS, StringUtils.EMPTY_STRING),
- CONFIG_VALUES_DELIMITER));
+ StringUtils.split(getStringOrDefault(TABLE_METADATA_PARTITIONS, StringUtils.EMPTY_STRING),
+ CONFIG_VALUES_DELIMITER));
+ }
+
+ /**
+ * @returns true if metadata table has been created and is being used for this dataset, else returns false.
+ */
+ public boolean isMetadataTableEnabled() {
+ return isMetadataPartitionEnabled(MetadataPartitionType.FILES);
+ }
+
+ /**
+ * Checks if metadata table is enabled and the specified partition has been initialized.
+ *
+ * @param partition The partition to check
+ * @returns true if the specific partition has been initialized, else returns false.
+ */
+ public boolean isMetadataPartitionEnabled(MetadataPartitionType partition) {
+ return getMetadataPartitions().contains(partition.getPartitionPath());
+ }
+
+ /**
+ * Enables or disables the specified metadata table partition.
+ *
+ * @param partition The partition
+ * @param enabled If true, the partition is enabled, else disabled
+ */
+ public void setMetadataPartitionState(MetadataPartitionType partition, boolean enabled) {
+ ValidationUtils.checkArgument(!partition.getPartitionPath().contains(CONFIG_VALUES_DELIMITER),
+ "Metadata Table partition path cannot contain a comma: " + partition.getPartitionPath());
+ Set<String> partitions = getMetadataPartitions();
+ Set<String> partitionsInflight = getMetadataPartitionsInflight();
+ if (enabled) {
+ partitions.add(partition.getPartitionPath());
+ partitionsInflight.remove(partition.getPartitionPath());
+ } else if (partition.equals(MetadataPartitionType.FILES)) {
+ // file listing partition is required for all other partitions to work
+ // Disabling file partition will also disable all partitions
+ partitions.clear();
+ partitionsInflight.clear();
+ } else {
+ partitions.remove(partition.getPartitionPath());
+ partitionsInflight.remove(partition.getPartitionPath());
+ }
+ setValue(TABLE_METADATA_PARTITIONS, partitions.stream().sorted().collect(Collectors.joining(CONFIG_VALUES_DELIMITER)));
+ setValue(TABLE_METADATA_PARTITIONS_INFLIGHT, partitionsInflight.stream().sorted().collect(Collectors.joining(CONFIG_VALUES_DELIMITER)));
+ }
+
+ /**
+ * Enables the specified metadata table partition as inflight.
+ *
+ * @param partition The list of partitions to enable as inflight.
+ */
+ public void setMetadataPartitionsAsInflight(List<MetadataPartitionType> partitionTypes) {
+ Set<String> partitions = getMetadataPartitionsInflight();
Review Comment:
oh, I meant
```
Set<String> partitions
```
->
```
Set<String> partitionsInflight
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] prashantwason commented on a diff in pull request #8684: [HUDI-6200] Enhancements to the MDT for improving performance of larger indexes.
Posted by "prashantwason (via GitHub)" <gi...@apache.org>.
prashantwason commented on code in PR #8684:
URL: https://github.com/apache/hudi/pull/8684#discussion_r1192164647
##########
hudi-client/hudi-spark-client/src/main/java/org/apache/hudi/metadata/SparkHoodieBackedTableMetadataWriter.java:
##########
@@ -118,46 +123,32 @@ protected void initRegistry() {
}
@Override
- protected <T extends SpecificRecordBase> void initialize(HoodieEngineContext engineContext,
- Option<T> actionMetadata,
- Option<String> inflightInstantTimestamp) {
- try {
- metrics.map(HoodieMetadataMetrics::registry).ifPresent(registry -> {
- if (registry instanceof DistributedRegistry) {
- HoodieSparkEngineContext sparkEngineContext = (HoodieSparkEngineContext) engineContext;
- ((DistributedRegistry) registry).register(sparkEngineContext.getJavaSparkContext());
- }
- });
+ protected void commit(String instantTime, Map<MetadataPartitionType, HoodieData<HoodieRecord>> partitionRecordsMap) {
+ commitInternal(instantTime, partitionRecordsMap, Option.empty());
+ }
- if (enabled) {
- initializeIfNeeded(dataMetaClient, actionMetadata, inflightInstantTimestamp);
- }
- } catch (IOException e) {
- LOG.error("Failed to initialize metadata table. Disabling the writer.", e);
- enabled = false;
- }
+ protected void bulkCommit(
+ String instantTime, MetadataPartitionType partitionType, HoodieData<HoodieRecord> records,
+ int fileGroupCount) {
+ Map<MetadataPartitionType, HoodieData<HoodieRecord>> partitionRecordsMap = new HashMap<>();
+ partitionRecordsMap.put(partitionType, records);
+ SparkHoodieMetadataBulkInsertPartitioner partitioner = new SparkHoodieMetadataBulkInsertPartitioner(fileGroupCount);
Review Comment:
It will write HFile since the base file format of the MDT is HFile.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #8684: [HUDI-6200] Enhancements to the MDT for improving performance of larger indexes.
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #8684:
URL: https://github.com/apache/hudi/pull/8684#issuecomment-1582053742
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "35148aeb4ba78eb6f3316c75f8a0a7e4c6d6df87",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17007",
"triggerID" : "35148aeb4ba78eb6f3316c75f8a0a7e4c6d6df87",
"triggerType" : "PUSH"
}, {
"hash" : "e2785f4675ddf74582ff34590608a5d71c5e9a2d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17008",
"triggerID" : "e2785f4675ddf74582ff34590608a5d71c5e9a2d",
"triggerType" : "PUSH"
}, {
"hash" : "364b1fe0dde303750ca0f8eebe5cd530c2a66a3a",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17042",
"triggerID" : "364b1fe0dde303750ca0f8eebe5cd530c2a66a3a",
"triggerType" : "PUSH"
}, {
"hash" : "9d73b9664eed48ada68721a92526640cfa06a021",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17098",
"triggerID" : "9d73b9664eed48ada68721a92526640cfa06a021",
"triggerType" : "PUSH"
}, {
"hash" : "7be538f4045a42ba33ab8fe62594178e2db75bbb",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17100",
"triggerID" : "7be538f4045a42ba33ab8fe62594178e2db75bbb",
"triggerType" : "PUSH"
}, {
"hash" : "cc0da2372d50d99c98c2ce4bcbe5a60303bde938",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17195",
"triggerID" : "cc0da2372d50d99c98c2ce4bcbe5a60303bde938",
"triggerType" : "PUSH"
}, {
"hash" : "da0e00e3655e06b064d11e20500317e2a1d2229c",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "da0e00e3655e06b064d11e20500317e2a1d2229c",
"triggerType" : "PUSH"
}, {
"hash" : "589c8cd7ffc5df7984c099d6b30fa117463db2b9",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17335",
"triggerID" : "589c8cd7ffc5df7984c099d6b30fa117463db2b9",
"triggerType" : "PUSH"
}, {
"hash" : "226c96ce48b33d7b1650d00e93bc5ea42ad276f6",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17624",
"triggerID" : "226c96ce48b33d7b1650d00e93bc5ea42ad276f6",
"triggerType" : "PUSH"
}, {
"hash" : "ad81a08799a98cd6ee63b8bf792d42f16a98f775",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17634",
"triggerID" : "ad81a08799a98cd6ee63b8bf792d42f16a98f775",
"triggerType" : "PUSH"
}, {
"hash" : "fd2ec462e554190843ddcc8d946317ae76cb81e1",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "fd2ec462e554190843ddcc8d946317ae76cb81e1",
"triggerType" : "PUSH"
}, {
"hash" : "cae57c2b03a3131cfa7774fa4335f4d153f85d61",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17644",
"triggerID" : "cae57c2b03a3131cfa7774fa4335f4d153f85d61",
"triggerType" : "PUSH"
}, {
"hash" : "a21fb8753ed206fefcf598c9e134c280e90d3dba",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17654",
"triggerID" : "a21fb8753ed206fefcf598c9e134c280e90d3dba",
"triggerType" : "PUSH"
}, {
"hash" : "4dfaefde334b712b3ba272bb330242ddb7f4c0ac",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17655",
"triggerID" : "4dfaefde334b712b3ba272bb330242ddb7f4c0ac",
"triggerType" : "PUSH"
}, {
"hash" : "ca59a379b830f88fd72c175cb9b96a14b33550c0",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17660",
"triggerID" : "ca59a379b830f88fd72c175cb9b96a14b33550c0",
"triggerType" : "PUSH"
}, {
"hash" : "1d49e658315a2ae22a2c29e3d388e82f869b146d",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17664",
"triggerID" : "1d49e658315a2ae22a2c29e3d388e82f869b146d",
"triggerType" : "PUSH"
}, {
"hash" : "df3a5b57a375fd344814b1f5a2451cfb79fea2c3",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "df3a5b57a375fd344814b1f5a2451cfb79fea2c3",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* da0e00e3655e06b064d11e20500317e2a1d2229c UNKNOWN
* fd2ec462e554190843ddcc8d946317ae76cb81e1 UNKNOWN
* 1d49e658315a2ae22a2c29e3d388e82f869b146d Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17664)
* df3a5b57a375fd344814b1f5a2451cfb79fea2c3 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #8684: [HUDI-6200] Enhancements to the MDT for improving performance of larger indexes.
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #8684:
URL: https://github.com/apache/hudi/pull/8684#issuecomment-1581832380
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "35148aeb4ba78eb6f3316c75f8a0a7e4c6d6df87",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17007",
"triggerID" : "35148aeb4ba78eb6f3316c75f8a0a7e4c6d6df87",
"triggerType" : "PUSH"
}, {
"hash" : "e2785f4675ddf74582ff34590608a5d71c5e9a2d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17008",
"triggerID" : "e2785f4675ddf74582ff34590608a5d71c5e9a2d",
"triggerType" : "PUSH"
}, {
"hash" : "364b1fe0dde303750ca0f8eebe5cd530c2a66a3a",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17042",
"triggerID" : "364b1fe0dde303750ca0f8eebe5cd530c2a66a3a",
"triggerType" : "PUSH"
}, {
"hash" : "9d73b9664eed48ada68721a92526640cfa06a021",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17098",
"triggerID" : "9d73b9664eed48ada68721a92526640cfa06a021",
"triggerType" : "PUSH"
}, {
"hash" : "7be538f4045a42ba33ab8fe62594178e2db75bbb",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17100",
"triggerID" : "7be538f4045a42ba33ab8fe62594178e2db75bbb",
"triggerType" : "PUSH"
}, {
"hash" : "cc0da2372d50d99c98c2ce4bcbe5a60303bde938",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17195",
"triggerID" : "cc0da2372d50d99c98c2ce4bcbe5a60303bde938",
"triggerType" : "PUSH"
}, {
"hash" : "da0e00e3655e06b064d11e20500317e2a1d2229c",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "da0e00e3655e06b064d11e20500317e2a1d2229c",
"triggerType" : "PUSH"
}, {
"hash" : "589c8cd7ffc5df7984c099d6b30fa117463db2b9",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17335",
"triggerID" : "589c8cd7ffc5df7984c099d6b30fa117463db2b9",
"triggerType" : "PUSH"
}, {
"hash" : "226c96ce48b33d7b1650d00e93bc5ea42ad276f6",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17624",
"triggerID" : "226c96ce48b33d7b1650d00e93bc5ea42ad276f6",
"triggerType" : "PUSH"
}, {
"hash" : "ad81a08799a98cd6ee63b8bf792d42f16a98f775",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17634",
"triggerID" : "ad81a08799a98cd6ee63b8bf792d42f16a98f775",
"triggerType" : "PUSH"
}, {
"hash" : "fd2ec462e554190843ddcc8d946317ae76cb81e1",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "fd2ec462e554190843ddcc8d946317ae76cb81e1",
"triggerType" : "PUSH"
}, {
"hash" : "cae57c2b03a3131cfa7774fa4335f4d153f85d61",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17644",
"triggerID" : "cae57c2b03a3131cfa7774fa4335f4d153f85d61",
"triggerType" : "PUSH"
}, {
"hash" : "a21fb8753ed206fefcf598c9e134c280e90d3dba",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17654",
"triggerID" : "a21fb8753ed206fefcf598c9e134c280e90d3dba",
"triggerType" : "PUSH"
}, {
"hash" : "4dfaefde334b712b3ba272bb330242ddb7f4c0ac",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17655",
"triggerID" : "4dfaefde334b712b3ba272bb330242ddb7f4c0ac",
"triggerType" : "PUSH"
}, {
"hash" : "ca59a379b830f88fd72c175cb9b96a14b33550c0",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17660",
"triggerID" : "ca59a379b830f88fd72c175cb9b96a14b33550c0",
"triggerType" : "PUSH"
}, {
"hash" : "1d49e658315a2ae22a2c29e3d388e82f869b146d",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "1d49e658315a2ae22a2c29e3d388e82f869b146d",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* da0e00e3655e06b064d11e20500317e2a1d2229c UNKNOWN
* fd2ec462e554190843ddcc8d946317ae76cb81e1 UNKNOWN
* ca59a379b830f88fd72c175cb9b96a14b33550c0 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17660)
* 1d49e658315a2ae22a2c29e3d388e82f869b146d UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #8684: [HUDI-6200] Enhancements to the MDT for improving performance of larger indexes.
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #8684:
URL: https://github.com/apache/hudi/pull/8684#issuecomment-1581294345
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "35148aeb4ba78eb6f3316c75f8a0a7e4c6d6df87",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17007",
"triggerID" : "35148aeb4ba78eb6f3316c75f8a0a7e4c6d6df87",
"triggerType" : "PUSH"
}, {
"hash" : "e2785f4675ddf74582ff34590608a5d71c5e9a2d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17008",
"triggerID" : "e2785f4675ddf74582ff34590608a5d71c5e9a2d",
"triggerType" : "PUSH"
}, {
"hash" : "364b1fe0dde303750ca0f8eebe5cd530c2a66a3a",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17042",
"triggerID" : "364b1fe0dde303750ca0f8eebe5cd530c2a66a3a",
"triggerType" : "PUSH"
}, {
"hash" : "9d73b9664eed48ada68721a92526640cfa06a021",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17098",
"triggerID" : "9d73b9664eed48ada68721a92526640cfa06a021",
"triggerType" : "PUSH"
}, {
"hash" : "7be538f4045a42ba33ab8fe62594178e2db75bbb",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17100",
"triggerID" : "7be538f4045a42ba33ab8fe62594178e2db75bbb",
"triggerType" : "PUSH"
}, {
"hash" : "cc0da2372d50d99c98c2ce4bcbe5a60303bde938",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17195",
"triggerID" : "cc0da2372d50d99c98c2ce4bcbe5a60303bde938",
"triggerType" : "PUSH"
}, {
"hash" : "da0e00e3655e06b064d11e20500317e2a1d2229c",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "da0e00e3655e06b064d11e20500317e2a1d2229c",
"triggerType" : "PUSH"
}, {
"hash" : "589c8cd7ffc5df7984c099d6b30fa117463db2b9",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17335",
"triggerID" : "589c8cd7ffc5df7984c099d6b30fa117463db2b9",
"triggerType" : "PUSH"
}, {
"hash" : "226c96ce48b33d7b1650d00e93bc5ea42ad276f6",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17624",
"triggerID" : "226c96ce48b33d7b1650d00e93bc5ea42ad276f6",
"triggerType" : "PUSH"
}, {
"hash" : "ad81a08799a98cd6ee63b8bf792d42f16a98f775",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17634",
"triggerID" : "ad81a08799a98cd6ee63b8bf792d42f16a98f775",
"triggerType" : "PUSH"
}, {
"hash" : "fd2ec462e554190843ddcc8d946317ae76cb81e1",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "fd2ec462e554190843ddcc8d946317ae76cb81e1",
"triggerType" : "PUSH"
}, {
"hash" : "cae57c2b03a3131cfa7774fa4335f4d153f85d61",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17644",
"triggerID" : "cae57c2b03a3131cfa7774fa4335f4d153f85d61",
"triggerType" : "PUSH"
}, {
"hash" : "a21fb8753ed206fefcf598c9e134c280e90d3dba",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17654",
"triggerID" : "a21fb8753ed206fefcf598c9e134c280e90d3dba",
"triggerType" : "PUSH"
}, {
"hash" : "4dfaefde334b712b3ba272bb330242ddb7f4c0ac",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17655",
"triggerID" : "4dfaefde334b712b3ba272bb330242ddb7f4c0ac",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* da0e00e3655e06b064d11e20500317e2a1d2229c UNKNOWN
* fd2ec462e554190843ddcc8d946317ae76cb81e1 UNKNOWN
* 4dfaefde334b712b3ba272bb330242ddb7f4c0ac Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17655)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #8684: [HUDI-6200] Enhancements to the MDT for improving performance of larger indexes.
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #8684:
URL: https://github.com/apache/hudi/pull/8684#issuecomment-1546072246
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "35148aeb4ba78eb6f3316c75f8a0a7e4c6d6df87",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17007",
"triggerID" : "35148aeb4ba78eb6f3316c75f8a0a7e4c6d6df87",
"triggerType" : "PUSH"
}, {
"hash" : "e2785f4675ddf74582ff34590608a5d71c5e9a2d",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17008",
"triggerID" : "e2785f4675ddf74582ff34590608a5d71c5e9a2d",
"triggerType" : "PUSH"
}, {
"hash" : "364b1fe0dde303750ca0f8eebe5cd530c2a66a3a",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "364b1fe0dde303750ca0f8eebe5cd530c2a66a3a",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* e2785f4675ddf74582ff34590608a5d71c5e9a2d Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17008)
* 364b1fe0dde303750ca0f8eebe5cd530c2a66a3a UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] nsivabalan commented on a diff in pull request #8684: [HUDI-6200] Enhancements to the MDT for improving performance of larger indexes.
Posted by "nsivabalan (via GitHub)" <gi...@apache.org>.
nsivabalan commented on code in PR #8684:
URL: https://github.com/apache/hudi/pull/8684#discussion_r1190732568
##########
hudi-client/hudi-client-common/src/main/java/org/apache/hudi/metadata/HoodieBackedTableMetadataWriter.java:
##########
@@ -892,25 +917,24 @@ public void buildMetadataPartitions(HoodieEngineContext engineContext, List<Hood
}
partitionTypes.add(partitionType);
});
- // before initial commit update inflight indexes in table config
- Set<String> inflightIndexes = getInflightMetadataPartitions(dataMetaClient.getTableConfig());
- inflightIndexes.addAll(indexPartitionInfos.stream().map(HoodieIndexPartitionInfo::getMetadataPartitionPath).collect(Collectors.toSet()));
- dataMetaClient.getTableConfig().setValue(HoodieTableConfig.TABLE_METADATA_PARTITIONS_INFLIGHT.key(), String.join(",", inflightIndexes));
- HoodieTableConfig.update(dataMetaClient.getFs(), new Path(dataMetaClient.getMetaPath()), dataMetaClient.getTableConfig().getProps());
- initialCommit(indexUptoInstantTime + METADATA_INDEXER_TIME_SUFFIX, partitionTypes);
+
+ // before initialization set these partitions as inflight in table config
+ HoodieTableMetadataUtil.setMetadataInflightPartitions(dataMetaClient, partitionTypes);
Review Comment:
wrt 2nd flight. we should be cautious in updating table config on the fly.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] prashantwason commented on a diff in pull request #8684: [HUDI-6200] Enhancements to the MDT for improving performance of larger indexes.
Posted by "prashantwason (via GitHub)" <gi...@apache.org>.
prashantwason commented on code in PR #8684:
URL: https://github.com/apache/hudi/pull/8684#discussion_r1192162585
##########
hudi-client/hudi-client-common/src/main/java/org/apache/hudi/metadata/HoodieBackedTableMetadataWriter.java:
##########
@@ -824,25 +863,22 @@ private interface ConvertMetadataFunction {
/**
* Processes commit metadata from data table and commits to metadata table.
*
- * @param instantTime instant time of interest.
+ * @param instantTime instant time of interest.
* @param convertMetadataFunction converter function to convert the respective metadata to List of HoodieRecords to be written to metadata table.
- * @param <T> type of commit metadata.
- * @param canTriggerTableService true if table services can be triggered. false otherwise.
*/
- private <T> void processAndCommit(String instantTime, ConvertMetadataFunction convertMetadataFunction, boolean canTriggerTableService) {
- if (!dataWriteConfig.isMetadataTableEnabled()) {
- return;
- }
+ private void processAndCommit(String instantTime, ConvertMetadataFunction convertMetadataFunction) {
+ ValidationUtils.checkArgument(dataWriteConfig.isMetadataTableEnabled());
+
Set<String> partitionsToUpdate = getMetadataPartitionsToUpdate();
Set<String> inflightIndexes = getInflightMetadataPartitions(dataMetaClient.getTableConfig());
// if indexing is inflight then do not trigger table service
boolean doNotTriggerTableService = partitionsToUpdate.stream().anyMatch(inflightIndexes::contains);
Review Comment:
If you mean the function processAndCommit, then yes. This is used for non-initial commits into MDT. Only the initial commit is a bulkInsert.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] danny0405 commented on a diff in pull request #8684: [HUDI-6200] Enhancements to the MDT for improving performance of larger indexes.
Posted by "danny0405 (via GitHub)" <gi...@apache.org>.
danny0405 commented on code in PR #8684:
URL: https://github.com/apache/hudi/pull/8684#discussion_r1190914223
##########
hudi-client/hudi-client-common/src/main/java/org/apache/hudi/metadata/HoodieBackedTableMetadataWriter.java:
##########
@@ -619,41 +680,34 @@ private boolean anyPendingDataInstant(HoodieTableMetaClient dataMetaClient, Opti
return false;
}
- private void updateInitializedPartitionsInTableConfig(List<MetadataPartitionType> partitionTypes) {
- Set<String> completedPartitions = dataMetaClient.getTableConfig().getMetadataPartitions();
- completedPartitions.addAll(partitionTypes.stream().map(MetadataPartitionType::getPartitionPath).collect(Collectors.toSet()));
- dataMetaClient.getTableConfig().setValue(HoodieTableConfig.TABLE_METADATA_PARTITIONS.key(), String.join(",", completedPartitions));
- HoodieTableConfig.update(dataMetaClient.getFs(), new Path(dataMetaClient.getMetaPath()), dataMetaClient.getTableConfig().getProps());
- }
-
- private HoodieTableMetaClient initializeMetaClient(boolean populateMetaFields) throws IOException {
+ private HoodieTableMetaClient initializeMetaClient() throws IOException {
return HoodieTableMetaClient.withPropertyBuilder()
.setTableType(HoodieTableType.MERGE_ON_READ)
.setTableName(tableName)
.setArchiveLogFolder(ARCHIVELOG_FOLDER.defaultValue())
.setPayloadClassName(HoodieMetadataPayload.class.getName())
.setBaseFileFormat(HoodieFileFormat.HFILE.toString())
.setRecordKeyFields(RECORD_KEY_FIELD_NAME)
- .setPopulateMetaFields(populateMetaFields)
- .setKeyGeneratorClassProp(HoodieTableMetadataKeyGenerator.class.getCanonicalName())
- .initTable(hadoopConf.get(), metadataWriteConfig.getBasePath());
+ .setPopulateMetaFields(DEFAULT_METADATA_POPULATE_META_FIELDS)
+ .setKeyGeneratorClassProp(HoodieTableMetadataKeyGenerator.class.getCanonicalName())
+ .initTable(hadoopConf.get(), metadataWriteConfig.getBasePath());
Review Comment:
Fix the indentation.
##########
hudi-client/hudi-client-common/src/main/java/org/apache/hudi/metadata/HoodieBackedTableMetadataWriter.java:
##########
@@ -824,25 +863,22 @@ private interface ConvertMetadataFunction {
/**
* Processes commit metadata from data table and commits to metadata table.
*
- * @param instantTime instant time of interest.
+ * @param instantTime instant time of interest.
* @param convertMetadataFunction converter function to convert the respective metadata to List of HoodieRecords to be written to metadata table.
- * @param <T> type of commit metadata.
- * @param canTriggerTableService true if table services can be triggered. false otherwise.
*/
- private <T> void processAndCommit(String instantTime, ConvertMetadataFunction convertMetadataFunction, boolean canTriggerTableService) {
- if (!dataWriteConfig.isMetadataTableEnabled()) {
- return;
- }
+ private void processAndCommit(String instantTime, ConvertMetadataFunction convertMetadataFunction) {
+ ValidationUtils.checkArgument(dataWriteConfig.isMetadataTableEnabled());
+
Set<String> partitionsToUpdate = getMetadataPartitionsToUpdate();
Set<String> inflightIndexes = getInflightMetadataPartitions(dataMetaClient.getTableConfig());
// if indexing is inflight then do not trigger table service
boolean doNotTriggerTableService = partitionsToUpdate.stream().anyMatch(inflightIndexes::contains);
Review Comment:
Do we still need this?
##########
hudi-client/hudi-spark-client/src/main/java/org/apache/hudi/metadata/SparkHoodieBackedTableMetadataWriter.java:
##########
@@ -118,46 +123,32 @@ protected void initRegistry() {
}
@Override
- protected <T extends SpecificRecordBase> void initialize(HoodieEngineContext engineContext,
- Option<T> actionMetadata,
- Option<String> inflightInstantTimestamp) {
- try {
- metrics.map(HoodieMetadataMetrics::registry).ifPresent(registry -> {
- if (registry instanceof DistributedRegistry) {
- HoodieSparkEngineContext sparkEngineContext = (HoodieSparkEngineContext) engineContext;
- ((DistributedRegistry) registry).register(sparkEngineContext.getJavaSparkContext());
- }
- });
+ protected void commit(String instantTime, Map<MetadataPartitionType, HoodieData<HoodieRecord>> partitionRecordsMap) {
+ commitInternal(instantTime, partitionRecordsMap, Option.empty());
+ }
- if (enabled) {
- initializeIfNeeded(dataMetaClient, actionMetadata, inflightInstantTimestamp);
- }
- } catch (IOException e) {
- LOG.error("Failed to initialize metadata table. Disabling the writer.", e);
- enabled = false;
- }
+ protected void bulkCommit(
+ String instantTime, MetadataPartitionType partitionType, HoodieData<HoodieRecord> records,
+ int fileGroupCount) {
+ Map<MetadataPartitionType, HoodieData<HoodieRecord>> partitionRecordsMap = new HashMap<>();
+ partitionRecordsMap.put(partitionType, records);
+ SparkHoodieMetadataBulkInsertPartitioner partitioner = new SparkHoodieMetadataBulkInsertPartitioner(fileGroupCount);
Review Comment:
Can bulk_insert supporting writing HFile directly ? It seems it still write parquets, then the k-v query performance should not be good ?
##########
hudi-client/hudi-client-common/src/main/java/org/apache/hudi/metadata/HoodieBackedTableMetadataWriter.java:
##########
@@ -373,105 +356,92 @@ public List<MetadataPartitionType> getEnabledPartitionTypes() {
return this.enabledPartitionTypes;
}
- /**
- * Initialize the metadata table if it does not exist.
- * <p>
- * If the metadata table does not exist, then file and partition listing is used to initialize the table.
- *
- * @param engineContext
- * @param actionMetadata Action metadata types extending Avro generated SpecificRecordBase
- * @param inflightInstantTimestamp Timestamp of an instant in progress on the dataset. This instant is ignored
- * while deciding to initialize the metadata table.
- */
- protected abstract <T extends SpecificRecordBase> void initialize(HoodieEngineContext engineContext,
- Option<T> actionMetadata,
- Option<String> inflightInstantTimestamp);
-
- public void initTableMetadata() {
- try {
- if (this.metadata != null) {
- this.metadata.close();
- }
- this.metadata = new HoodieBackedTableMetadata(engineContext, dataWriteConfig.getMetadataConfig(),
- dataWriteConfig.getBasePath(), dataWriteConfig.getSpillableMapBasePath());
- this.metadataMetaClient = metadata.getMetadataMetaClient();
- } catch (Exception e) {
- throw new HoodieException("Error initializing metadata table for reads", e);
- }
- }
-
/**
* Initialize the metadata table if needed.
*
* @param dataMetaClient - meta client for the data table
* @param actionMetadata - optional action metadata
* @param inflightInstantTimestamp - timestamp of an instant in progress on the dataset
* @param <T> - action metadata types extending Avro generated SpecificRecordBase
- * @throws IOException
+ * @throws IOException on errors
*/
- protected <T extends SpecificRecordBase> void initializeIfNeeded(HoodieTableMetaClient dataMetaClient,
- Option<T> actionMetadata,
- Option<String> inflightInstantTimestamp) throws IOException {
+ protected <T extends SpecificRecordBase> boolean initializeIfNeeded(HoodieTableMetaClient dataMetaClient,
+ Option<T> actionMetadata,
+ Option<String> inflightInstantTimestamp) throws IOException {
HoodieTimer timer = HoodieTimer.start();
+ List<MetadataPartitionType> partitionsToInit = new ArrayList<>(MetadataPartitionType.values().length);
- boolean exists = metadataTableExists(dataMetaClient, actionMetadata);
+ try {
+ boolean exists = metadataTableExists(dataMetaClient, actionMetadata);
+ if (!exists) {
+ // FILES partition is always required
+ partitionsToInit.add(MetadataPartitionType.FILES);
+ }
- if (!exists) {
- // Initialize for the first time by listing partitions and files directly from the file system
- if (initializeFromFilesystem(dataMetaClient, inflightInstantTimestamp)) {
- metrics.ifPresent(m -> m.updateMetrics(HoodieMetadataMetrics.INITIALIZE_STR, timer.endTimer()));
+ // check if any of the enabled partition types needs to be initialized
+ // NOTE: It needs to be guarded by async index config because if that is enabled then initialization happens through the index scheduler.
+ if (!dataWriteConfig.isMetadataAsyncIndex()) {
+ Set<String> inflightAndCompletedPartitions = getInflightAndCompletedMetadataPartitions(dataMetaClient.getTableConfig());
+ LOG.info("Async metadata indexing disabled and following partitions already initialized: " + inflightAndCompletedPartitions);
+ this.enabledPartitionTypes.stream()
+ .filter(p -> !inflightAndCompletedPartitions.contains(p.getPartitionPath()) && !MetadataPartitionType.FILES.equals(p))
+ .forEach(partitionsToInit::add);
}
- return;
- }
- // if metadata table exists, then check if any of the enabled partition types needs to be initialized
- // NOTE: It needs to be guarded by async index config because if that is enabled then initialization happens through the index scheduler.
- if (!dataWriteConfig.isMetadataAsyncIndex()) {
- Set<String> inflightAndCompletedPartitions = getInflightAndCompletedMetadataPartitions(dataMetaClient.getTableConfig());
- LOG.info("Async metadata indexing enabled and following partitions already initialized: " + inflightAndCompletedPartitions);
- List<MetadataPartitionType> partitionsToInit = this.enabledPartitionTypes.stream()
- .filter(p -> !inflightAndCompletedPartitions.contains(p.getPartitionPath()) && !MetadataPartitionType.FILES.equals(p))
- .collect(Collectors.toList());
- // if there are no partitions to initialize or there is a pending operation, then don't initialize in this round
- if (partitionsToInit.isEmpty() || anyPendingDataInstant(dataMetaClient, inflightInstantTimestamp)) {
- return;
+ if (partitionsToInit.isEmpty()) {
+ // No partitions to initialize
+ return true;
}
- String createInstantTime = getInitialCommitInstantTime(dataMetaClient);
- initTableMetadata(); // re-init certain flags in BaseTableMetadata
- initializeEnabledFileGroups(dataMetaClient, createInstantTime, partitionsToInit);
- initialCommit(createInstantTime, partitionsToInit);
- updateInitializedPartitionsInTableConfig(partitionsToInit);
+ // If there is no commit on the dataset yet, use the SOLO_COMMIT_TIMESTAMP as the instant time for initial commit
+ // Otherwise, we use the timestamp of the latest completed action.
+ String initializationTime = dataMetaClient.getActiveTimeline().filterCompletedInstants()
+ .getReverseOrderedInstants().findFirst().map(HoodieInstant::getTimestamp).orElse(SOLO_COMMIT_TIMESTAMP);
Review Comment:
`.getReverseOrderedInstants().findFirst()` -> `lastInstant` ?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] prashantwason commented on a diff in pull request #8684: [HUDI-6200] Enhancements to the MDT for improving performance of larger indexes.
Posted by "prashantwason (via GitHub)" <gi...@apache.org>.
prashantwason commented on code in PR #8684:
URL: https://github.com/apache/hudi/pull/8684#discussion_r1192180200
##########
hudi-common/src/main/java/org/apache/hudi/metadata/HoodieTableMetadataUtil.java:
##########
@@ -1378,6 +1339,206 @@ public static Set<String> getInflightAndCompletedMetadataPartitions(HoodieTableC
*/
public static boolean isIndexingCommit(String instantTime) {
return instantTime.length() == MILLIS_INSTANT_ID_LENGTH + METADATA_INDEXER_TIME_SUFFIX.length()
- && instantTime.endsWith(METADATA_INDEXER_TIME_SUFFIX);
+ && instantTime.endsWith(METADATA_INDEXER_TIME_SUFFIX);
+ }
+
+ /**
+ * Delete the metadata table for the dataset and backup if required.
+ *
+ * @param dataMetaClient {@code HoodieTableMetaClient} of the dataset for which metadata table is to be deleted
+ * @param context instance of {@link HoodieEngineContext}.
+ * @param backup Whether metadata table should be backed up before deletion. If true, the table is backed up to the
+ * directory with name metadata_<current_timestamp>.
+ * @return The backup directory if backup was requested
+ */
+ public static String deleteMetadataTable(HoodieTableMetaClient dataMetaClient, HoodieEngineContext context, boolean backup) {
+ final Path metadataTablePath = HoodieTableMetadata.getMetadataTableBasePath(dataMetaClient.getBasePathV2());
+ FileSystem fs = FSUtils.getFs(metadataTablePath.toString(), context.getHadoopConf().get());
+ setMetadataPartitionState(dataMetaClient, MetadataPartitionType.FILES, false);
+ try {
+ if (!fs.exists(metadataTablePath)) {
+ return null;
+ }
+ } catch (FileNotFoundException e) {
+ // Ignoring exception as metadata table already does not exist
+ return null;
+ } catch (IOException e) {
+ throw new HoodieMetadataException("Failed to check metadata table existence", e);
+ }
+
+ if (backup) {
+ final Path metadataBackupPath = new Path(metadataTablePath.getParent(), ".metadata_" + HoodieActiveTimeline.createNewInstantTime());
+ LOG.info("Backing up metadata directory to " + metadataBackupPath + " before deletion");
+ try {
+ if (fs.rename(metadataTablePath, metadataBackupPath)) {
+ return metadataBackupPath.toString();
+ }
+ } catch (Exception e) {
+ // If rename fails, we will ignore the backup and still delete the MDT
+ LOG.error("Failed to backup metadata table using rename", e);
+ }
+ }
+
+ LOG.info("Deleting metadata table from " + metadataTablePath);
+ try {
+ fs.delete(metadataTablePath, true);
+ } catch (Exception e) {
+ throw new HoodieMetadataException("Failed to delete metadata table from path " + metadataTablePath, e);
+ }
+
+ return null;
+ }
+
+ public static HoodieTableMetaClient setMetadataPartitionState(HoodieTableMetaClient dataMetaClient, MetadataPartitionType partitionType, boolean enabled) {
+ dataMetaClient.getTableConfig().setMetadataPartitionState(partitionType, enabled);
+ HoodieTableConfig.update(dataMetaClient.getFs(), new Path(dataMetaClient.getMetaPath()), dataMetaClient.getTableConfig().getProps());
+ dataMetaClient = HoodieTableMetaClient.reload(dataMetaClient);
+ ValidationUtils.checkState(dataMetaClient.getTableConfig().isMetadataPartitionEnabled(partitionType) == enabled,
+ "Metadata table state change should be persisted");
+
+ LOG.info(String.format("Metadata table %s partition %s has been %s", dataMetaClient.getBasePathV2(), partitionType,
+ enabled ? "enabled" : "disabled"));
+ return dataMetaClient;
+ }
+
+ /**
+ * Delete a partition within the metadata table.
+ * <p>
+ * This can be used to delete a partition so that it can be re-bootstrapped.
+ *
+ * @param dataMetaClient {@code HoodieTableMetaClient} of the dataset for which metadata table is to be deleted
+ * @param context instance of {@code HoodieEngineContext}.
+ * @param backup Whether metadata table should be backed up before deletion. If true, the table is backed up to the
+ * directory with name metadata_<current_timestamp>.
+ * @param partitionType The partition to delete
+ * @return The backup directory if backup was requested, null otherwise
+ */
+ public static String deleteMetadataTablePartition(HoodieTableMetaClient dataMetaClient, HoodieEngineContext context,
+ MetadataPartitionType partitionType, boolean backup) {
+ if (partitionType.equals(MetadataPartitionType.FILES)) {
+ return deleteMetadataTable(dataMetaClient, context, backup);
+ }
+
+ final Path metadataTablePartitionPath = new Path(HoodieTableMetadata.getMetadataTableBasePath(dataMetaClient.getBasePath()), partitionType.getPartitionPath());
+ FileSystem fs = FSUtils.getFs(metadataTablePartitionPath.toString(), context.getHadoopConf().get());
+ setMetadataPartitionState(dataMetaClient, partitionType, false);
+ try {
+ if (!fs.exists(metadataTablePartitionPath)) {
+ return null;
+ }
+ } catch (FileNotFoundException e) {
+ // Ignoring exception as metadata table already does not exist
+ LOG.debug("Metadata table partition " + partitionType + " not found at path " + metadataTablePartitionPath);
+ return null;
+ } catch (Exception e) {
+ throw new HoodieMetadataException("Failed to check metadata table partition existence", e);
+ }
+
+ if (backup) {
+ final Path metadataPartitionBackupPath = new Path(metadataTablePartitionPath.getParent().getParent(),
+ String.format(".metadata_%s_%s", partitionType.getPartitionPath(), HoodieActiveTimeline.createNewInstantTime()));
+ LOG.info(String.format("Backing up MDT partition %s to %s before deletion", partitionType, metadataPartitionBackupPath));
+ try {
+ if (fs.rename(metadataTablePartitionPath, metadataPartitionBackupPath)) {
+ return metadataPartitionBackupPath.toString();
+ }
+ } catch (Exception e) {
+ // If rename fails, we will try to delete the table instead
+ LOG.error(String.format("Failed to backup MDT partition %s using rename", partitionType), e);
+ }
+ } else {
+ LOG.info("Deleting metadata table partition from " + metadataTablePartitionPath);
+ try {
+ fs.delete(metadataTablePartitionPath, true);
+ } catch (Exception e) {
+ throw new HoodieMetadataException("Failed to delete metadata table partition from path " + metadataTablePartitionPath, e);
+ }
+ }
+
+ return null;
+ }
+
+ public static HoodieTableMetaClient setMetadataInflightPartitions(HoodieTableMetaClient dataMetaClient, List<MetadataPartitionType> partitionTypes) {
+ dataMetaClient.getTableConfig().setMetadataPartitionsAsInflight(partitionTypes);
+ HoodieTableConfig.update(dataMetaClient.getFs(), new Path(dataMetaClient.getMetaPath()), dataMetaClient.getTableConfig().getProps());
+ dataMetaClient = HoodieTableMetaClient.reload(dataMetaClient);
+
+ LOG.info(String.format("Metadata table %s partitions %s hasvebeen set to inflight", dataMetaClient.getBasePathV2(), partitionTypes));
+ return dataMetaClient;
+ }
+
+ /**
+ * Return the complete fileID for a file group within a MDT partition.
+ * <p>
+ * MDT fileGroups have the format <fileIDPrefix>-<index>. The fileIDPrefix is hardcoded for each MDT partition and index is an integer.
+ *
+ * @param partitionType The type of the MDT partition
+ * @param index Index of the file group within the partition
+ * @return The fileID
+ */
+ public static String getFileIDForFileGroup(MetadataPartitionType partitionType, int index) {
+ return String.format("%s%04d", partitionType.getFileIdPrefix(), index);
+ }
+
+ /**
+ * Extract the index from the fileID of a file group in the MDT partition. See {@code getFileIDForFileGroup} for the format of the fileID.
+ *
+ * @param fileId fileID of a file group.
+ * @return The index of file group
+ */
+ public static int getFileGroupIndexFromFileId(String fileId) {
Review Comment:
Added
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #8684: [HUDI-6200] Enhancements to the MDT for improving performance of larger indexes.
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #8684:
URL: https://github.com/apache/hudi/pull/8684#issuecomment-1560756525
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "35148aeb4ba78eb6f3316c75f8a0a7e4c6d6df87",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17007",
"triggerID" : "35148aeb4ba78eb6f3316c75f8a0a7e4c6d6df87",
"triggerType" : "PUSH"
}, {
"hash" : "e2785f4675ddf74582ff34590608a5d71c5e9a2d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17008",
"triggerID" : "e2785f4675ddf74582ff34590608a5d71c5e9a2d",
"triggerType" : "PUSH"
}, {
"hash" : "364b1fe0dde303750ca0f8eebe5cd530c2a66a3a",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17042",
"triggerID" : "364b1fe0dde303750ca0f8eebe5cd530c2a66a3a",
"triggerType" : "PUSH"
}, {
"hash" : "9d73b9664eed48ada68721a92526640cfa06a021",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17098",
"triggerID" : "9d73b9664eed48ada68721a92526640cfa06a021",
"triggerType" : "PUSH"
}, {
"hash" : "7be538f4045a42ba33ab8fe62594178e2db75bbb",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17100",
"triggerID" : "7be538f4045a42ba33ab8fe62594178e2db75bbb",
"triggerType" : "PUSH"
}, {
"hash" : "cc0da2372d50d99c98c2ce4bcbe5a60303bde938",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17195",
"triggerID" : "cc0da2372d50d99c98c2ce4bcbe5a60303bde938",
"triggerType" : "PUSH"
}, {
"hash" : "da0e00e3655e06b064d11e20500317e2a1d2229c",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "da0e00e3655e06b064d11e20500317e2a1d2229c",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* cc0da2372d50d99c98c2ce4bcbe5a60303bde938 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17195)
* da0e00e3655e06b064d11e20500317e2a1d2229c UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #8684: [HUDI-6200] Enhancements to the MDT for improving performance of larger indexes.
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #8684:
URL: https://github.com/apache/hudi/pull/8684#issuecomment-1560839163
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "35148aeb4ba78eb6f3316c75f8a0a7e4c6d6df87",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17007",
"triggerID" : "35148aeb4ba78eb6f3316c75f8a0a7e4c6d6df87",
"triggerType" : "PUSH"
}, {
"hash" : "e2785f4675ddf74582ff34590608a5d71c5e9a2d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17008",
"triggerID" : "e2785f4675ddf74582ff34590608a5d71c5e9a2d",
"triggerType" : "PUSH"
}, {
"hash" : "364b1fe0dde303750ca0f8eebe5cd530c2a66a3a",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17042",
"triggerID" : "364b1fe0dde303750ca0f8eebe5cd530c2a66a3a",
"triggerType" : "PUSH"
}, {
"hash" : "9d73b9664eed48ada68721a92526640cfa06a021",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17098",
"triggerID" : "9d73b9664eed48ada68721a92526640cfa06a021",
"triggerType" : "PUSH"
}, {
"hash" : "7be538f4045a42ba33ab8fe62594178e2db75bbb",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17100",
"triggerID" : "7be538f4045a42ba33ab8fe62594178e2db75bbb",
"triggerType" : "PUSH"
}, {
"hash" : "cc0da2372d50d99c98c2ce4bcbe5a60303bde938",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17195",
"triggerID" : "cc0da2372d50d99c98c2ce4bcbe5a60303bde938",
"triggerType" : "PUSH"
}, {
"hash" : "da0e00e3655e06b064d11e20500317e2a1d2229c",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "da0e00e3655e06b064d11e20500317e2a1d2229c",
"triggerType" : "PUSH"
}, {
"hash" : "589c8cd7ffc5df7984c099d6b30fa117463db2b9",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17335",
"triggerID" : "589c8cd7ffc5df7984c099d6b30fa117463db2b9",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* cc0da2372d50d99c98c2ce4bcbe5a60303bde938 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17195)
* da0e00e3655e06b064d11e20500317e2a1d2229c UNKNOWN
* 589c8cd7ffc5df7984c099d6b30fa117463db2b9 Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17335)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] prashantwason commented on a diff in pull request #8684: [HUDI-6200] Enhancements to the MDT for improving performance of larger indexes.
Posted by "prashantwason (via GitHub)" <gi...@apache.org>.
prashantwason commented on code in PR #8684:
URL: https://github.com/apache/hudi/pull/8684#discussion_r1203503606
##########
hudi-common/src/main/java/org/apache/hudi/common/table/HoodieTableConfig.java:
##########
@@ -694,17 +695,75 @@ private Long getTableChecksum() {
return getLong(TABLE_CHECKSUM);
}
- public List<String> getMetadataPartitionsInflight() {
- return StringUtils.split(
- getStringOrDefault(TABLE_METADATA_PARTITIONS_INFLIGHT, StringUtils.EMPTY_STRING),
- CONFIG_VALUES_DELIMITER
- );
+ public Set<String> getMetadataPartitionsInflight() {
+ return new HashSet<>(StringUtils.split(
+ getStringOrDefault(TABLE_METADATA_PARTITIONS_INFLIGHT, StringUtils.EMPTY_STRING),
+ CONFIG_VALUES_DELIMITER));
}
public Set<String> getMetadataPartitions() {
return new HashSet<>(
- StringUtils.split(getStringOrDefault(TABLE_METADATA_PARTITIONS, StringUtils.EMPTY_STRING),
- CONFIG_VALUES_DELIMITER));
+ StringUtils.split(getStringOrDefault(TABLE_METADATA_PARTITIONS, StringUtils.EMPTY_STRING),
+ CONFIG_VALUES_DELIMITER));
+ }
+
+ /**
+ * @returns true if metadata table has been created and is being used for this dataset, else returns false.
+ */
+ public boolean isMetadataTableEnabled() {
+ return isMetadataPartitionEnabled(MetadataPartitionType.FILES);
+ }
+
+ /**
+ * Checks if metadata table is enabled and the specified partition has been initialized.
+ *
+ * @param partition The partition to check
+ * @returns true if the specific partition has been initialized, else returns false.
+ */
+ public boolean isMetadataPartitionEnabled(MetadataPartitionType partition) {
+ return getMetadataPartitions().contains(partition.getPartitionPath());
+ }
+
+ /**
+ * Enables or disables the specified metadata table partition.
+ *
+ * @param partition The partition
+ * @param enabled If true, the partition is enabled, else disabled
+ */
+ public void setMetadataPartitionState(MetadataPartitionType partition, boolean enabled) {
+ ValidationUtils.checkArgument(!partition.getPartitionPath().contains(CONFIG_VALUES_DELIMITER),
+ "Metadata Table partition path cannot contain a comma: " + partition.getPartitionPath());
+ Set<String> partitions = getMetadataPartitions();
+ Set<String> partitionsInflight = getMetadataPartitionsInflight();
+ if (enabled) {
+ partitions.add(partition.getPartitionPath());
+ partitionsInflight.remove(partition.getPartitionPath());
+ } else if (partition.equals(MetadataPartitionType.FILES)) {
+ // file listing partition is required for all other partitions to work
+ // Disabling file partition will also disable all partitions
+ partitions.clear();
+ partitionsInflight.clear();
+ } else {
+ partitions.remove(partition.getPartitionPath());
+ partitionsInflight.remove(partition.getPartitionPath());
+ }
+ setValue(TABLE_METADATA_PARTITIONS, partitions.stream().sorted().collect(Collectors.joining(CONFIG_VALUES_DELIMITER)));
+ setValue(TABLE_METADATA_PARTITIONS_INFLIGHT, partitionsInflight.stream().sorted().collect(Collectors.joining(CONFIG_VALUES_DELIMITER)));
Review Comment:
It is persisted from the caller side - HoodieTableMetadataUtil.setMetadataPartitionState
I have removed the HoodieTableMetadataUtil.setMetadataPartitionState as it was unnecessary.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] danny0405 commented on a diff in pull request #8684: [HUDI-6200] Enhancements to the MDT for improving performance of larger indexes.
Posted by "danny0405 (via GitHub)" <gi...@apache.org>.
danny0405 commented on code in PR #8684:
URL: https://github.com/apache/hudi/pull/8684#discussion_r1222548105
##########
hudi-common/src/main/java/org/apache/hudi/metadata/HoodieTableMetadataUtil.java:
##########
@@ -1453,7 +1453,11 @@ public static String deleteMetadataTablePartition(HoodieTableMetaClient dataMeta
* @return The fileID
*/
public static String getFileIDForFileGroup(MetadataPartitionType partitionType, int index) {
- return String.format("%s%04d", partitionType.getFileIdPrefix(), index);
+ if (partitionType == MetadataPartitionType.FILES) {
+ return String.format("%s%04d-%d", partitionType.getFileIdPrefix(), index, 0);
+ } else {
Review Comment:
Should we fix all the partitions or just the `FILES` partition?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #8684: [HUDI-6200] Enhancements to the MDT for improving performance of larger indexes.
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #8684:
URL: https://github.com/apache/hudi/pull/8684#issuecomment-1583054671
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "35148aeb4ba78eb6f3316c75f8a0a7e4c6d6df87",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17007",
"triggerID" : "35148aeb4ba78eb6f3316c75f8a0a7e4c6d6df87",
"triggerType" : "PUSH"
}, {
"hash" : "e2785f4675ddf74582ff34590608a5d71c5e9a2d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17008",
"triggerID" : "e2785f4675ddf74582ff34590608a5d71c5e9a2d",
"triggerType" : "PUSH"
}, {
"hash" : "364b1fe0dde303750ca0f8eebe5cd530c2a66a3a",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17042",
"triggerID" : "364b1fe0dde303750ca0f8eebe5cd530c2a66a3a",
"triggerType" : "PUSH"
}, {
"hash" : "9d73b9664eed48ada68721a92526640cfa06a021",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17098",
"triggerID" : "9d73b9664eed48ada68721a92526640cfa06a021",
"triggerType" : "PUSH"
}, {
"hash" : "7be538f4045a42ba33ab8fe62594178e2db75bbb",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17100",
"triggerID" : "7be538f4045a42ba33ab8fe62594178e2db75bbb",
"triggerType" : "PUSH"
}, {
"hash" : "cc0da2372d50d99c98c2ce4bcbe5a60303bde938",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17195",
"triggerID" : "cc0da2372d50d99c98c2ce4bcbe5a60303bde938",
"triggerType" : "PUSH"
}, {
"hash" : "da0e00e3655e06b064d11e20500317e2a1d2229c",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "da0e00e3655e06b064d11e20500317e2a1d2229c",
"triggerType" : "PUSH"
}, {
"hash" : "589c8cd7ffc5df7984c099d6b30fa117463db2b9",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17335",
"triggerID" : "589c8cd7ffc5df7984c099d6b30fa117463db2b9",
"triggerType" : "PUSH"
}, {
"hash" : "226c96ce48b33d7b1650d00e93bc5ea42ad276f6",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17624",
"triggerID" : "226c96ce48b33d7b1650d00e93bc5ea42ad276f6",
"triggerType" : "PUSH"
}, {
"hash" : "ad81a08799a98cd6ee63b8bf792d42f16a98f775",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17634",
"triggerID" : "ad81a08799a98cd6ee63b8bf792d42f16a98f775",
"triggerType" : "PUSH"
}, {
"hash" : "fd2ec462e554190843ddcc8d946317ae76cb81e1",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "fd2ec462e554190843ddcc8d946317ae76cb81e1",
"triggerType" : "PUSH"
}, {
"hash" : "cae57c2b03a3131cfa7774fa4335f4d153f85d61",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17644",
"triggerID" : "cae57c2b03a3131cfa7774fa4335f4d153f85d61",
"triggerType" : "PUSH"
}, {
"hash" : "a21fb8753ed206fefcf598c9e134c280e90d3dba",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17654",
"triggerID" : "a21fb8753ed206fefcf598c9e134c280e90d3dba",
"triggerType" : "PUSH"
}, {
"hash" : "4dfaefde334b712b3ba272bb330242ddb7f4c0ac",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17655",
"triggerID" : "4dfaefde334b712b3ba272bb330242ddb7f4c0ac",
"triggerType" : "PUSH"
}, {
"hash" : "ca59a379b830f88fd72c175cb9b96a14b33550c0",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17660",
"triggerID" : "ca59a379b830f88fd72c175cb9b96a14b33550c0",
"triggerType" : "PUSH"
}, {
"hash" : "1d49e658315a2ae22a2c29e3d388e82f869b146d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17664",
"triggerID" : "1d49e658315a2ae22a2c29e3d388e82f869b146d",
"triggerType" : "PUSH"
}, {
"hash" : "df3a5b57a375fd344814b1f5a2451cfb79fea2c3",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17669",
"triggerID" : "df3a5b57a375fd344814b1f5a2451cfb79fea2c3",
"triggerType" : "PUSH"
}, {
"hash" : "da92c6c310fccdb4f1e797b481eaf6bb027e551f",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17673",
"triggerID" : "da92c6c310fccdb4f1e797b481eaf6bb027e551f",
"triggerType" : "PUSH"
}, {
"hash" : "86902aa16075d75d61347a93819295d78cda0d2d",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17675",
"triggerID" : "86902aa16075d75d61347a93819295d78cda0d2d",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* da0e00e3655e06b064d11e20500317e2a1d2229c UNKNOWN
* fd2ec462e554190843ddcc8d946317ae76cb81e1 UNKNOWN
* 86902aa16075d75d61347a93819295d78cda0d2d Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17675)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #8684: [HUDI-6200] Enhancements to the MDT for improving performance of larger indexes.
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #8684:
URL: https://github.com/apache/hudi/pull/8684#issuecomment-1578972294
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "35148aeb4ba78eb6f3316c75f8a0a7e4c6d6df87",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17007",
"triggerID" : "35148aeb4ba78eb6f3316c75f8a0a7e4c6d6df87",
"triggerType" : "PUSH"
}, {
"hash" : "e2785f4675ddf74582ff34590608a5d71c5e9a2d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17008",
"triggerID" : "e2785f4675ddf74582ff34590608a5d71c5e9a2d",
"triggerType" : "PUSH"
}, {
"hash" : "364b1fe0dde303750ca0f8eebe5cd530c2a66a3a",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17042",
"triggerID" : "364b1fe0dde303750ca0f8eebe5cd530c2a66a3a",
"triggerType" : "PUSH"
}, {
"hash" : "9d73b9664eed48ada68721a92526640cfa06a021",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17098",
"triggerID" : "9d73b9664eed48ada68721a92526640cfa06a021",
"triggerType" : "PUSH"
}, {
"hash" : "7be538f4045a42ba33ab8fe62594178e2db75bbb",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17100",
"triggerID" : "7be538f4045a42ba33ab8fe62594178e2db75bbb",
"triggerType" : "PUSH"
}, {
"hash" : "cc0da2372d50d99c98c2ce4bcbe5a60303bde938",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17195",
"triggerID" : "cc0da2372d50d99c98c2ce4bcbe5a60303bde938",
"triggerType" : "PUSH"
}, {
"hash" : "da0e00e3655e06b064d11e20500317e2a1d2229c",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "da0e00e3655e06b064d11e20500317e2a1d2229c",
"triggerType" : "PUSH"
}, {
"hash" : "589c8cd7ffc5df7984c099d6b30fa117463db2b9",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17335",
"triggerID" : "589c8cd7ffc5df7984c099d6b30fa117463db2b9",
"triggerType" : "PUSH"
}, {
"hash" : "226c96ce48b33d7b1650d00e93bc5ea42ad276f6",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17624",
"triggerID" : "226c96ce48b33d7b1650d00e93bc5ea42ad276f6",
"triggerType" : "PUSH"
}, {
"hash" : "ad81a08799a98cd6ee63b8bf792d42f16a98f775",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17634",
"triggerID" : "ad81a08799a98cd6ee63b8bf792d42f16a98f775",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* da0e00e3655e06b064d11e20500317e2a1d2229c UNKNOWN
* ad81a08799a98cd6ee63b8bf792d42f16a98f775 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17634)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #8684: [HUDI-6200] Enhancements to the MDT for improving performance of larger indexes.
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #8684:
URL: https://github.com/apache/hudi/pull/8684#issuecomment-1578369721
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "35148aeb4ba78eb6f3316c75f8a0a7e4c6d6df87",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17007",
"triggerID" : "35148aeb4ba78eb6f3316c75f8a0a7e4c6d6df87",
"triggerType" : "PUSH"
}, {
"hash" : "e2785f4675ddf74582ff34590608a5d71c5e9a2d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17008",
"triggerID" : "e2785f4675ddf74582ff34590608a5d71c5e9a2d",
"triggerType" : "PUSH"
}, {
"hash" : "364b1fe0dde303750ca0f8eebe5cd530c2a66a3a",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17042",
"triggerID" : "364b1fe0dde303750ca0f8eebe5cd530c2a66a3a",
"triggerType" : "PUSH"
}, {
"hash" : "9d73b9664eed48ada68721a92526640cfa06a021",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17098",
"triggerID" : "9d73b9664eed48ada68721a92526640cfa06a021",
"triggerType" : "PUSH"
}, {
"hash" : "7be538f4045a42ba33ab8fe62594178e2db75bbb",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17100",
"triggerID" : "7be538f4045a42ba33ab8fe62594178e2db75bbb",
"triggerType" : "PUSH"
}, {
"hash" : "cc0da2372d50d99c98c2ce4bcbe5a60303bde938",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17195",
"triggerID" : "cc0da2372d50d99c98c2ce4bcbe5a60303bde938",
"triggerType" : "PUSH"
}, {
"hash" : "da0e00e3655e06b064d11e20500317e2a1d2229c",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "da0e00e3655e06b064d11e20500317e2a1d2229c",
"triggerType" : "PUSH"
}, {
"hash" : "589c8cd7ffc5df7984c099d6b30fa117463db2b9",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17335",
"triggerID" : "589c8cd7ffc5df7984c099d6b30fa117463db2b9",
"triggerType" : "PUSH"
}, {
"hash" : "226c96ce48b33d7b1650d00e93bc5ea42ad276f6",
"status" : "CANCELED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17624",
"triggerID" : "226c96ce48b33d7b1650d00e93bc5ea42ad276f6",
"triggerType" : "PUSH"
}, {
"hash" : "ad81a08799a98cd6ee63b8bf792d42f16a98f775",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17634",
"triggerID" : "ad81a08799a98cd6ee63b8bf792d42f16a98f775",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* da0e00e3655e06b064d11e20500317e2a1d2229c UNKNOWN
* 226c96ce48b33d7b1650d00e93bc5ea42ad276f6 Azure: [CANCELED](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17624)
* ad81a08799a98cd6ee63b8bf792d42f16a98f775 Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17634)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #8684: [HUDI-6200] Enhancements to the MDT for improving performance of larger indexes.
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #8684:
URL: https://github.com/apache/hudi/pull/8684#issuecomment-1579965886
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "35148aeb4ba78eb6f3316c75f8a0a7e4c6d6df87",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17007",
"triggerID" : "35148aeb4ba78eb6f3316c75f8a0a7e4c6d6df87",
"triggerType" : "PUSH"
}, {
"hash" : "e2785f4675ddf74582ff34590608a5d71c5e9a2d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17008",
"triggerID" : "e2785f4675ddf74582ff34590608a5d71c5e9a2d",
"triggerType" : "PUSH"
}, {
"hash" : "364b1fe0dde303750ca0f8eebe5cd530c2a66a3a",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17042",
"triggerID" : "364b1fe0dde303750ca0f8eebe5cd530c2a66a3a",
"triggerType" : "PUSH"
}, {
"hash" : "9d73b9664eed48ada68721a92526640cfa06a021",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17098",
"triggerID" : "9d73b9664eed48ada68721a92526640cfa06a021",
"triggerType" : "PUSH"
}, {
"hash" : "7be538f4045a42ba33ab8fe62594178e2db75bbb",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17100",
"triggerID" : "7be538f4045a42ba33ab8fe62594178e2db75bbb",
"triggerType" : "PUSH"
}, {
"hash" : "cc0da2372d50d99c98c2ce4bcbe5a60303bde938",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17195",
"triggerID" : "cc0da2372d50d99c98c2ce4bcbe5a60303bde938",
"triggerType" : "PUSH"
}, {
"hash" : "da0e00e3655e06b064d11e20500317e2a1d2229c",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "da0e00e3655e06b064d11e20500317e2a1d2229c",
"triggerType" : "PUSH"
}, {
"hash" : "589c8cd7ffc5df7984c099d6b30fa117463db2b9",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17335",
"triggerID" : "589c8cd7ffc5df7984c099d6b30fa117463db2b9",
"triggerType" : "PUSH"
}, {
"hash" : "226c96ce48b33d7b1650d00e93bc5ea42ad276f6",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17624",
"triggerID" : "226c96ce48b33d7b1650d00e93bc5ea42ad276f6",
"triggerType" : "PUSH"
}, {
"hash" : "ad81a08799a98cd6ee63b8bf792d42f16a98f775",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17634",
"triggerID" : "ad81a08799a98cd6ee63b8bf792d42f16a98f775",
"triggerType" : "PUSH"
}, {
"hash" : "fd2ec462e554190843ddcc8d946317ae76cb81e1",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "fd2ec462e554190843ddcc8d946317ae76cb81e1",
"triggerType" : "PUSH"
}, {
"hash" : "cae57c2b03a3131cfa7774fa4335f4d153f85d61",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17644",
"triggerID" : "cae57c2b03a3131cfa7774fa4335f4d153f85d61",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* da0e00e3655e06b064d11e20500317e2a1d2229c UNKNOWN
* fd2ec462e554190843ddcc8d946317ae76cb81e1 UNKNOWN
* cae57c2b03a3131cfa7774fa4335f4d153f85d61 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17644)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #8684: [HUDI-6200] Enhancements to the MDT for improving performance of larger indexes.
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #8684:
URL: https://github.com/apache/hudi/pull/8684#issuecomment-1581066382
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "35148aeb4ba78eb6f3316c75f8a0a7e4c6d6df87",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17007",
"triggerID" : "35148aeb4ba78eb6f3316c75f8a0a7e4c6d6df87",
"triggerType" : "PUSH"
}, {
"hash" : "e2785f4675ddf74582ff34590608a5d71c5e9a2d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17008",
"triggerID" : "e2785f4675ddf74582ff34590608a5d71c5e9a2d",
"triggerType" : "PUSH"
}, {
"hash" : "364b1fe0dde303750ca0f8eebe5cd530c2a66a3a",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17042",
"triggerID" : "364b1fe0dde303750ca0f8eebe5cd530c2a66a3a",
"triggerType" : "PUSH"
}, {
"hash" : "9d73b9664eed48ada68721a92526640cfa06a021",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17098",
"triggerID" : "9d73b9664eed48ada68721a92526640cfa06a021",
"triggerType" : "PUSH"
}, {
"hash" : "7be538f4045a42ba33ab8fe62594178e2db75bbb",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17100",
"triggerID" : "7be538f4045a42ba33ab8fe62594178e2db75bbb",
"triggerType" : "PUSH"
}, {
"hash" : "cc0da2372d50d99c98c2ce4bcbe5a60303bde938",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17195",
"triggerID" : "cc0da2372d50d99c98c2ce4bcbe5a60303bde938",
"triggerType" : "PUSH"
}, {
"hash" : "da0e00e3655e06b064d11e20500317e2a1d2229c",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "da0e00e3655e06b064d11e20500317e2a1d2229c",
"triggerType" : "PUSH"
}, {
"hash" : "589c8cd7ffc5df7984c099d6b30fa117463db2b9",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17335",
"triggerID" : "589c8cd7ffc5df7984c099d6b30fa117463db2b9",
"triggerType" : "PUSH"
}, {
"hash" : "226c96ce48b33d7b1650d00e93bc5ea42ad276f6",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17624",
"triggerID" : "226c96ce48b33d7b1650d00e93bc5ea42ad276f6",
"triggerType" : "PUSH"
}, {
"hash" : "ad81a08799a98cd6ee63b8bf792d42f16a98f775",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17634",
"triggerID" : "ad81a08799a98cd6ee63b8bf792d42f16a98f775",
"triggerType" : "PUSH"
}, {
"hash" : "fd2ec462e554190843ddcc8d946317ae76cb81e1",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "fd2ec462e554190843ddcc8d946317ae76cb81e1",
"triggerType" : "PUSH"
}, {
"hash" : "cae57c2b03a3131cfa7774fa4335f4d153f85d61",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17644",
"triggerID" : "cae57c2b03a3131cfa7774fa4335f4d153f85d61",
"triggerType" : "PUSH"
}, {
"hash" : "a21fb8753ed206fefcf598c9e134c280e90d3dba",
"status" : "CANCELED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17654",
"triggerID" : "a21fb8753ed206fefcf598c9e134c280e90d3dba",
"triggerType" : "PUSH"
}, {
"hash" : "4dfaefde334b712b3ba272bb330242ddb7f4c0ac",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "4dfaefde334b712b3ba272bb330242ddb7f4c0ac",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* da0e00e3655e06b064d11e20500317e2a1d2229c UNKNOWN
* fd2ec462e554190843ddcc8d946317ae76cb81e1 UNKNOWN
* a21fb8753ed206fefcf598c9e134c280e90d3dba Azure: [CANCELED](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17654)
* 4dfaefde334b712b3ba272bb330242ddb7f4c0ac UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #8684: [HUDI-6200] Enhancements to the MDT for improving performance of larger indexes.
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #8684:
URL: https://github.com/apache/hudi/pull/8684#issuecomment-1581141374
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "35148aeb4ba78eb6f3316c75f8a0a7e4c6d6df87",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17007",
"triggerID" : "35148aeb4ba78eb6f3316c75f8a0a7e4c6d6df87",
"triggerType" : "PUSH"
}, {
"hash" : "e2785f4675ddf74582ff34590608a5d71c5e9a2d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17008",
"triggerID" : "e2785f4675ddf74582ff34590608a5d71c5e9a2d",
"triggerType" : "PUSH"
}, {
"hash" : "364b1fe0dde303750ca0f8eebe5cd530c2a66a3a",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17042",
"triggerID" : "364b1fe0dde303750ca0f8eebe5cd530c2a66a3a",
"triggerType" : "PUSH"
}, {
"hash" : "9d73b9664eed48ada68721a92526640cfa06a021",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17098",
"triggerID" : "9d73b9664eed48ada68721a92526640cfa06a021",
"triggerType" : "PUSH"
}, {
"hash" : "7be538f4045a42ba33ab8fe62594178e2db75bbb",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17100",
"triggerID" : "7be538f4045a42ba33ab8fe62594178e2db75bbb",
"triggerType" : "PUSH"
}, {
"hash" : "cc0da2372d50d99c98c2ce4bcbe5a60303bde938",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17195",
"triggerID" : "cc0da2372d50d99c98c2ce4bcbe5a60303bde938",
"triggerType" : "PUSH"
}, {
"hash" : "da0e00e3655e06b064d11e20500317e2a1d2229c",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "da0e00e3655e06b064d11e20500317e2a1d2229c",
"triggerType" : "PUSH"
}, {
"hash" : "589c8cd7ffc5df7984c099d6b30fa117463db2b9",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17335",
"triggerID" : "589c8cd7ffc5df7984c099d6b30fa117463db2b9",
"triggerType" : "PUSH"
}, {
"hash" : "226c96ce48b33d7b1650d00e93bc5ea42ad276f6",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17624",
"triggerID" : "226c96ce48b33d7b1650d00e93bc5ea42ad276f6",
"triggerType" : "PUSH"
}, {
"hash" : "ad81a08799a98cd6ee63b8bf792d42f16a98f775",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17634",
"triggerID" : "ad81a08799a98cd6ee63b8bf792d42f16a98f775",
"triggerType" : "PUSH"
}, {
"hash" : "fd2ec462e554190843ddcc8d946317ae76cb81e1",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "fd2ec462e554190843ddcc8d946317ae76cb81e1",
"triggerType" : "PUSH"
}, {
"hash" : "cae57c2b03a3131cfa7774fa4335f4d153f85d61",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17644",
"triggerID" : "cae57c2b03a3131cfa7774fa4335f4d153f85d61",
"triggerType" : "PUSH"
}, {
"hash" : "a21fb8753ed206fefcf598c9e134c280e90d3dba",
"status" : "CANCELED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17654",
"triggerID" : "a21fb8753ed206fefcf598c9e134c280e90d3dba",
"triggerType" : "PUSH"
}, {
"hash" : "4dfaefde334b712b3ba272bb330242ddb7f4c0ac",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17655",
"triggerID" : "4dfaefde334b712b3ba272bb330242ddb7f4c0ac",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* da0e00e3655e06b064d11e20500317e2a1d2229c UNKNOWN
* fd2ec462e554190843ddcc8d946317ae76cb81e1 UNKNOWN
* a21fb8753ed206fefcf598c9e134c280e90d3dba Azure: [CANCELED](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17654)
* 4dfaefde334b712b3ba272bb330242ddb7f4c0ac Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17655)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #8684: [HUDI-6200] Enhancements to the MDT for improving performance of larger indexes.
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #8684:
URL: https://github.com/apache/hudi/pull/8684#issuecomment-1582340844
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "35148aeb4ba78eb6f3316c75f8a0a7e4c6d6df87",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17007",
"triggerID" : "35148aeb4ba78eb6f3316c75f8a0a7e4c6d6df87",
"triggerType" : "PUSH"
}, {
"hash" : "e2785f4675ddf74582ff34590608a5d71c5e9a2d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17008",
"triggerID" : "e2785f4675ddf74582ff34590608a5d71c5e9a2d",
"triggerType" : "PUSH"
}, {
"hash" : "364b1fe0dde303750ca0f8eebe5cd530c2a66a3a",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17042",
"triggerID" : "364b1fe0dde303750ca0f8eebe5cd530c2a66a3a",
"triggerType" : "PUSH"
}, {
"hash" : "9d73b9664eed48ada68721a92526640cfa06a021",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17098",
"triggerID" : "9d73b9664eed48ada68721a92526640cfa06a021",
"triggerType" : "PUSH"
}, {
"hash" : "7be538f4045a42ba33ab8fe62594178e2db75bbb",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17100",
"triggerID" : "7be538f4045a42ba33ab8fe62594178e2db75bbb",
"triggerType" : "PUSH"
}, {
"hash" : "cc0da2372d50d99c98c2ce4bcbe5a60303bde938",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17195",
"triggerID" : "cc0da2372d50d99c98c2ce4bcbe5a60303bde938",
"triggerType" : "PUSH"
}, {
"hash" : "da0e00e3655e06b064d11e20500317e2a1d2229c",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "da0e00e3655e06b064d11e20500317e2a1d2229c",
"triggerType" : "PUSH"
}, {
"hash" : "589c8cd7ffc5df7984c099d6b30fa117463db2b9",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17335",
"triggerID" : "589c8cd7ffc5df7984c099d6b30fa117463db2b9",
"triggerType" : "PUSH"
}, {
"hash" : "226c96ce48b33d7b1650d00e93bc5ea42ad276f6",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17624",
"triggerID" : "226c96ce48b33d7b1650d00e93bc5ea42ad276f6",
"triggerType" : "PUSH"
}, {
"hash" : "ad81a08799a98cd6ee63b8bf792d42f16a98f775",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17634",
"triggerID" : "ad81a08799a98cd6ee63b8bf792d42f16a98f775",
"triggerType" : "PUSH"
}, {
"hash" : "fd2ec462e554190843ddcc8d946317ae76cb81e1",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "fd2ec462e554190843ddcc8d946317ae76cb81e1",
"triggerType" : "PUSH"
}, {
"hash" : "cae57c2b03a3131cfa7774fa4335f4d153f85d61",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17644",
"triggerID" : "cae57c2b03a3131cfa7774fa4335f4d153f85d61",
"triggerType" : "PUSH"
}, {
"hash" : "a21fb8753ed206fefcf598c9e134c280e90d3dba",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17654",
"triggerID" : "a21fb8753ed206fefcf598c9e134c280e90d3dba",
"triggerType" : "PUSH"
}, {
"hash" : "4dfaefde334b712b3ba272bb330242ddb7f4c0ac",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17655",
"triggerID" : "4dfaefde334b712b3ba272bb330242ddb7f4c0ac",
"triggerType" : "PUSH"
}, {
"hash" : "ca59a379b830f88fd72c175cb9b96a14b33550c0",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17660",
"triggerID" : "ca59a379b830f88fd72c175cb9b96a14b33550c0",
"triggerType" : "PUSH"
}, {
"hash" : "1d49e658315a2ae22a2c29e3d388e82f869b146d",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17664",
"triggerID" : "1d49e658315a2ae22a2c29e3d388e82f869b146d",
"triggerType" : "PUSH"
}, {
"hash" : "df3a5b57a375fd344814b1f5a2451cfb79fea2c3",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17669",
"triggerID" : "df3a5b57a375fd344814b1f5a2451cfb79fea2c3",
"triggerType" : "PUSH"
}, {
"hash" : "da92c6c310fccdb4f1e797b481eaf6bb027e551f",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "da92c6c310fccdb4f1e797b481eaf6bb027e551f",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* da0e00e3655e06b064d11e20500317e2a1d2229c UNKNOWN
* fd2ec462e554190843ddcc8d946317ae76cb81e1 UNKNOWN
* 1d49e658315a2ae22a2c29e3d388e82f869b146d Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17664)
* df3a5b57a375fd344814b1f5a2451cfb79fea2c3 Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17669)
* da92c6c310fccdb4f1e797b481eaf6bb027e551f UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] prashantwason commented on a diff in pull request #8684: [HUDI-6200] Enhancements to the MDT for improving performance of larger indexes.
Posted by "prashantwason (via GitHub)" <gi...@apache.org>.
prashantwason commented on code in PR #8684:
URL: https://github.com/apache/hudi/pull/8684#discussion_r1192159380
##########
hudi-client/hudi-client-common/src/main/java/org/apache/hudi/metadata/HoodieBackedTableMetadataWriter.java:
##########
@@ -562,53 +532,144 @@ private <T extends SpecificRecordBase> boolean isCommitRevertedByInFlightAction(
/**
* Initialize the Metadata Table by listing files and partitions from the file system.
*
- * @param dataMetaClient - {@code HoodieTableMetaClient} for the dataset.
+ * @param initializationTime - Timestamp to use for the commit
+ * @param partitionsToInit - List of MDT partitions to initialize
* @param inflightInstantTimestamp - Current action instant responsible for this initialization
*/
- private boolean initializeFromFilesystem(HoodieTableMetaClient dataMetaClient,
+ private boolean initializeFromFilesystem(String initializationTime, List<MetadataPartitionType> partitionsToInit,
Option<String> inflightInstantTimestamp) throws IOException {
if (anyPendingDataInstant(dataMetaClient, inflightInstantTimestamp)) {
return false;
}
- String createInstantTime = getInitialCommitInstantTime(dataMetaClient);
-
- initializeMetaClient(DEFAULT_METADATA_POPULATE_META_FIELDS);
- initTableMetadata();
- // if async metadata indexing is enabled,
- // then only initialize files partition as other partitions will be built using HoodieIndexer
- List<MetadataPartitionType> enabledPartitionTypes = new ArrayList<>();
- if (dataWriteConfig.isMetadataAsyncIndex()) {
- enabledPartitionTypes.add(MetadataPartitionType.FILES);
- } else {
- // all enabled ones should be initialized
- enabledPartitionTypes = this.enabledPartitionTypes;
+ // FILES partition is always initialized first
+ ValidationUtils.checkArgument(!partitionsToInit.contains(MetadataPartitionType.FILES)
+ || partitionsToInit.get(0).equals(MetadataPartitionType.FILES), "FILES partition should be initialized first: " + partitionsToInit);
+
+ metadataMetaClient = initializeMetaClient();
+
+ // Get a complete list of files and partitions from the file system or from already initialized FILES partition of MDT
+ boolean filesPartitionAvailable = dataMetaClient.getTableConfig().isMetadataPartitionEnabled(MetadataPartitionType.FILES);
+ List<DirectoryInfo> partitionInfoList = filesPartitionAvailable ? listAllPartitionsFromMDT(initializationTime) : listAllPartitionsFromFilesystem(initializationTime);
+ Map<String, Map<String, Long>> partitionToFilesMap = partitionInfoList.stream()
+ .map(p -> {
+ String partitionName = HoodieTableMetadataUtil.getPartitionIdentifier(p.getRelativePath());
+ return Pair.of(partitionName, p.getFileNameToSizeMap());
+ })
+ .collect(Collectors.toMap(Pair::getKey, Pair::getValue));
+
+ for (MetadataPartitionType partitionType : partitionsToInit) {
+ // Find the commit timestamp to use for this partition. Each initialization should use its own unique commit time.
+ String commitTimeForPartition = generateUniqueCommitInstantTime(initializationTime);
+
+ LOG.info("Initializing MDT partition " + partitionType + " at instant " + commitTimeForPartition);
+
+ Pair<Integer, HoodieData<HoodieRecord>> fileGroupCountAndRecordsPair;
+ switch (partitionType) {
+ case FILES:
+ fileGroupCountAndRecordsPair = initializeFilesPartition(initializationTime, partitionInfoList);
+ break;
+ case BLOOM_FILTERS:
+ fileGroupCountAndRecordsPair = initializeBloomFiltersPartition(initializationTime, partitionToFilesMap);
+ break;
+ case COLUMN_STATS:
+ fileGroupCountAndRecordsPair = initializeColumnStatsPartition(partitionToFilesMap);
+ break;
+ default:
+ throw new HoodieMetadataException("Unsupported MDT partition type: " + partitionType);
+ }
+
+ // Generate the file groups
+ final int fileGroupCount = fileGroupCountAndRecordsPair.getKey();
+ ValidationUtils.checkArgument(fileGroupCount > 0, "FileGroup count for MDT partition " + partitionType + " should be > 0");
+ initializeFileGroups(dataMetaClient, partitionType, commitTimeForPartition, fileGroupCount);
Review Comment:
Added doc
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #8684: [HUDI-6200] Enhancements to the MDT for improving performance of larger indexes.
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #8684:
URL: https://github.com/apache/hudi/pull/8684#issuecomment-1546079839
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "35148aeb4ba78eb6f3316c75f8a0a7e4c6d6df87",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17007",
"triggerID" : "35148aeb4ba78eb6f3316c75f8a0a7e4c6d6df87",
"triggerType" : "PUSH"
}, {
"hash" : "e2785f4675ddf74582ff34590608a5d71c5e9a2d",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17008",
"triggerID" : "e2785f4675ddf74582ff34590608a5d71c5e9a2d",
"triggerType" : "PUSH"
}, {
"hash" : "364b1fe0dde303750ca0f8eebe5cd530c2a66a3a",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17042",
"triggerID" : "364b1fe0dde303750ca0f8eebe5cd530c2a66a3a",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* e2785f4675ddf74582ff34590608a5d71c5e9a2d Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17008)
* 364b1fe0dde303750ca0f8eebe5cd530c2a66a3a Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17042)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] nsivabalan commented on a diff in pull request #8684: [HUDI-6200] Enhancements to the MDT for improving performance of larger indexes.
Posted by "nsivabalan (via GitHub)" <gi...@apache.org>.
nsivabalan commented on code in PR #8684:
URL: https://github.com/apache/hudi/pull/8684#discussion_r1192689257
##########
hudi-client/hudi-client-common/src/main/java/org/apache/hudi/metadata/HoodieBackedTableMetadataWriter.java:
##########
@@ -373,105 +356,92 @@ public List<MetadataPartitionType> getEnabledPartitionTypes() {
return this.enabledPartitionTypes;
}
- /**
- * Initialize the metadata table if it does not exist.
- * <p>
- * If the metadata table does not exist, then file and partition listing is used to initialize the table.
- *
- * @param engineContext
- * @param actionMetadata Action metadata types extending Avro generated SpecificRecordBase
- * @param inflightInstantTimestamp Timestamp of an instant in progress on the dataset. This instant is ignored
- * while deciding to initialize the metadata table.
- */
- protected abstract <T extends SpecificRecordBase> void initialize(HoodieEngineContext engineContext,
- Option<T> actionMetadata,
- Option<String> inflightInstantTimestamp);
-
- public void initTableMetadata() {
- try {
- if (this.metadata != null) {
- this.metadata.close();
- }
- this.metadata = new HoodieBackedTableMetadata(engineContext, dataWriteConfig.getMetadataConfig(),
- dataWriteConfig.getBasePath(), dataWriteConfig.getSpillableMapBasePath());
- this.metadataMetaClient = metadata.getMetadataMetaClient();
- } catch (Exception e) {
- throw new HoodieException("Error initializing metadata table for reads", e);
- }
- }
-
/**
* Initialize the metadata table if needed.
*
* @param dataMetaClient - meta client for the data table
* @param actionMetadata - optional action metadata
* @param inflightInstantTimestamp - timestamp of an instant in progress on the dataset
* @param <T> - action metadata types extending Avro generated SpecificRecordBase
- * @throws IOException
+ * @throws IOException on errors
*/
- protected <T extends SpecificRecordBase> void initializeIfNeeded(HoodieTableMetaClient dataMetaClient,
- Option<T> actionMetadata,
- Option<String> inflightInstantTimestamp) throws IOException {
+ protected <T extends SpecificRecordBase> boolean initializeIfNeeded(HoodieTableMetaClient dataMetaClient,
+ Option<T> actionMetadata,
+ Option<String> inflightInstantTimestamp) throws IOException {
HoodieTimer timer = HoodieTimer.start();
+ List<MetadataPartitionType> partitionsToInit = new ArrayList<>(MetadataPartitionType.values().length);
- boolean exists = metadataTableExists(dataMetaClient, actionMetadata);
+ try {
+ boolean exists = metadataTableExists(dataMetaClient, actionMetadata);
+ if (!exists) {
+ // FILES partition is always required
+ partitionsToInit.add(MetadataPartitionType.FILES);
+ }
- if (!exists) {
- // Initialize for the first time by listing partitions and files directly from the file system
- if (initializeFromFilesystem(dataMetaClient, inflightInstantTimestamp)) {
- metrics.ifPresent(m -> m.updateMetrics(HoodieMetadataMetrics.INITIALIZE_STR, timer.endTimer()));
+ // check if any of the enabled partition types needs to be initialized
+ // NOTE: It needs to be guarded by async index config because if that is enabled then initialization happens through the index scheduler.
+ if (!dataWriteConfig.isMetadataAsyncIndex()) {
Review Comment:
https://issues.apache.org/jira/browse/HUDI-6205
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] nsivabalan commented on a diff in pull request #8684: [HUDI-6200] Enhancements to the MDT for improving performance of larger indexes.
Posted by "nsivabalan (via GitHub)" <gi...@apache.org>.
nsivabalan commented on code in PR #8684:
URL: https://github.com/apache/hudi/pull/8684#discussion_r1191243539
##########
hudi-client/hudi-client-common/src/main/java/org/apache/hudi/metadata/HoodieBackedTableMetadataWriter.java:
##########
@@ -873,17 +908,7 @@ public void buildMetadataPartitions(HoodieEngineContext engineContext, List<Hood
indexPartitionInfos.forEach(indexPartitionInfo -> {
String relativePartitionPath = indexPartitionInfo.getMetadataPartitionPath();
LOG.info(String.format("Creating a new metadata index for partition '%s' under path %s upto instant %s",
- relativePartitionPath, metadataWriteConfig.getBasePath(), indexUptoInstantTime));
- try {
- // file group should have already been initialized while scheduling index for this partition
- if (!dataMetaClient.getFs().exists(new Path(metadataWriteConfig.getBasePath(), relativePartitionPath))) {
Review Comment:
can you point me to the code where we handle partial initialization failure. Guess this code is handling that. I assume we handle it elsewhere in this patch and hence have removed this .
##########
hudi-client/hudi-client-common/src/main/java/org/apache/hudi/metadata/HoodieBackedTableMetadataWriter.java:
##########
@@ -1097,87 +1165,76 @@ protected void cleanIfNecessary(BaseHoodieWriteClient writeClient, String instan
// Trigger cleaning with suffixes based on the same instant time. This ensures that any future
// delta commits synced over will not have an instant time lesser than the last completed instant on the
// metadata table.
- writeClient.clean(instantTime + "002");
+ writeClient.clean(HoodieTableMetadataUtil.createCleanTimestamp(instantTime));
writeClient.lazyRollbackFailedIndexing();
}
/**
- * This is invoked to initialize metadata table for a dataset.
- * Initial commit has special handling mechanism due to its scale compared to other regular commits.
- * During cold startup, the list of files to be committed can be huge.
- * So creating a HoodieCommitMetadata out of these large number of files,
- * and calling the existing update(HoodieCommitMetadata) function does not scale well.
- * Hence, we have a special commit just for the initialization scenario.
+ * Validates the timeline for both main and metadata tables.
*/
- private void initialCommit(String createInstantTime, List<MetadataPartitionType> partitionTypes) {
- // List all partitions in the basePath of the containing dataset
- LOG.info("Initializing metadata table by using file listings in " + dataWriteConfig.getBasePath());
- engineContext.setJobStatus(this.getClass().getSimpleName(), "Initializing metadata table by listing files and partitions: " + dataWriteConfig.getTableName());
-
- Map<MetadataPartitionType, HoodieData<HoodieRecord>> partitionToRecordsMap = new HashMap<>();
-
- // skip file system listing to populate metadata records if it's a fresh table.
- // this is applicable only if the table already has N commits and metadata is enabled at a later point in time.
- if (createInstantTime.equals(SOLO_COMMIT_TIMESTAMP)) { // SOLO_COMMIT_TIMESTAMP will be the initial commit time in MDT for a fresh table.
- // If not, last completed commit in data table will be chosen as the initial commit time.
- LOG.info("Triggering empty Commit to metadata to initialize");
- } else {
- List<DirectoryInfo> partitionInfoList = listAllPartitions(dataMetaClient);
- Map<String, Map<String, Long>> partitionToFilesMap = partitionInfoList.stream()
- .map(p -> {
- String partitionName = HoodieTableMetadataUtil.getPartitionIdentifier(p.getRelativePath());
- return Pair.of(partitionName, p.getFileNameToSizeMap());
- })
- .collect(Collectors.toMap(Pair::getKey, Pair::getValue));
-
- int totalDataFilesCount = partitionToFilesMap.values().stream().mapToInt(Map::size).sum();
- List<String> partitions = new ArrayList<>(partitionToFilesMap.keySet());
-
- if (partitionTypes.contains(MetadataPartitionType.FILES)) {
- // Record which saves the list of all partitions
- HoodieRecord allPartitionRecord = HoodieMetadataPayload.createPartitionListRecord(partitions);
- HoodieData<HoodieRecord> filesPartitionRecords = getFilesPartitionRecords(createInstantTime, partitionInfoList, allPartitionRecord);
- ValidationUtils.checkState(filesPartitionRecords.count() == (partitions.size() + 1));
- partitionToRecordsMap.put(MetadataPartitionType.FILES, filesPartitionRecords);
- }
-
- if (partitionTypes.contains(MetadataPartitionType.BLOOM_FILTERS) && totalDataFilesCount > 0) {
- final HoodieData<HoodieRecord> recordsRDD = HoodieTableMetadataUtil.convertFilesToBloomFilterRecords(
- engineContext, Collections.emptyMap(), partitionToFilesMap, getRecordsGenerationParams(), createInstantTime);
- partitionToRecordsMap.put(MetadataPartitionType.BLOOM_FILTERS, recordsRDD);
- }
-
- if (partitionTypes.contains(MetadataPartitionType.COLUMN_STATS) && totalDataFilesCount > 0) {
- final HoodieData<HoodieRecord> recordsRDD = HoodieTableMetadataUtil.convertFilesToColumnStatsRecords(
- engineContext, Collections.emptyMap(), partitionToFilesMap, getRecordsGenerationParams());
- partitionToRecordsMap.put(MetadataPartitionType.COLUMN_STATS, recordsRDD);
- }
- LOG.info("Committing " + partitions.size() + " partitions and " + totalDataFilesCount + " files to metadata");
+ private boolean validateTimelineBeforeSchedulingCompaction(Option<String> inFlightInstantTimestamp, String latestDeltacommitTime) {
+ // There should not be any incomplete instants on MDT
+ HoodieActiveTimeline metadataTimeline = metadataMetaClient.reloadActiveTimeline();
+ List<HoodieInstant> pendingInstantsOnMetadataTable = metadataTimeline.filterInflightsAndRequested().getInstants();
+ if (!pendingInstantsOnMetadataTable.isEmpty()) {
+ LOG.info(String.format(
+ "Cannot compact MDT as there are %d inflight instants: %s",
+ pendingInstantsOnMetadataTable.size(), Arrays.toString(pendingInstantsOnMetadataTable.toArray())));
+ metrics.ifPresent(m -> m.incrementMetric(HoodieMetadataMetrics.SKIP_TABLE_SERVICES, 1));
+ return false;
}
- commit(createInstantTime, partitionToRecordsMap, false);
- }
+ // There should not be any incomplete instants on dataset
+ HoodieActiveTimeline datasetTimeline = dataMetaClient.reloadActiveTimeline();
+ List<HoodieInstant> pendingInstantsOnDataset = datasetTimeline.filterInflightsAndRequested().getInstantsAsStream()
+ .filter(i -> !inFlightInstantTimestamp.isPresent() || !i.getTimestamp().equals(inFlightInstantTimestamp.get()))
+ .collect(Collectors.toList());
+ if (!pendingInstantsOnDataset.isEmpty()) {
+ LOG.info(String.format(
+ "Cannot compact MDT as there are %d inflight instants on dataset before latest deltacommit %s: %s",
+ pendingInstantsOnDataset.size(), latestDeltacommitTime, Arrays.toString(pendingInstantsOnDataset.toArray())));
+ metrics.ifPresent(m -> m.incrementMetric(HoodieMetadataMetrics.SKIP_TABLE_SERVICES, 1));
+ return false;
+ }
- private HoodieData<HoodieRecord> getFilesPartitionRecords(String createInstantTime, List<DirectoryInfo> partitionInfoList, HoodieRecord allPartitionRecord) {
- HoodieData<HoodieRecord> filesPartitionRecords = engineContext.parallelize(Arrays.asList(allPartitionRecord), 1);
- if (partitionInfoList.isEmpty()) {
- return filesPartitionRecords;
+ // Check if the inflight commit is greater than all the completed commits.
Review Comment:
do we really need to add this constraint? again, can you check if L1215 - 1234 would suffice. I am inclining to favor the liveness of MDT compaction. if these are added just to be very cautious and we don't really need it, we should remove it.
##########
hudi-client/hudi-client-common/src/main/java/org/apache/hudi/metadata/HoodieBackedTableMetadataWriter.java:
##########
@@ -1097,87 +1165,76 @@ protected void cleanIfNecessary(BaseHoodieWriteClient writeClient, String instan
// Trigger cleaning with suffixes based on the same instant time. This ensures that any future
// delta commits synced over will not have an instant time lesser than the last completed instant on the
// metadata table.
- writeClient.clean(instantTime + "002");
+ writeClient.clean(HoodieTableMetadataUtil.createCleanTimestamp(instantTime));
writeClient.lazyRollbackFailedIndexing();
}
/**
- * This is invoked to initialize metadata table for a dataset.
- * Initial commit has special handling mechanism due to its scale compared to other regular commits.
- * During cold startup, the list of files to be committed can be huge.
- * So creating a HoodieCommitMetadata out of these large number of files,
- * and calling the existing update(HoodieCommitMetadata) function does not scale well.
- * Hence, we have a special commit just for the initialization scenario.
+ * Validates the timeline for both main and metadata tables.
*/
- private void initialCommit(String createInstantTime, List<MetadataPartitionType> partitionTypes) {
- // List all partitions in the basePath of the containing dataset
- LOG.info("Initializing metadata table by using file listings in " + dataWriteConfig.getBasePath());
- engineContext.setJobStatus(this.getClass().getSimpleName(), "Initializing metadata table by listing files and partitions: " + dataWriteConfig.getTableName());
-
- Map<MetadataPartitionType, HoodieData<HoodieRecord>> partitionToRecordsMap = new HashMap<>();
-
- // skip file system listing to populate metadata records if it's a fresh table.
- // this is applicable only if the table already has N commits and metadata is enabled at a later point in time.
- if (createInstantTime.equals(SOLO_COMMIT_TIMESTAMP)) { // SOLO_COMMIT_TIMESTAMP will be the initial commit time in MDT for a fresh table.
- // If not, last completed commit in data table will be chosen as the initial commit time.
- LOG.info("Triggering empty Commit to metadata to initialize");
- } else {
- List<DirectoryInfo> partitionInfoList = listAllPartitions(dataMetaClient);
- Map<String, Map<String, Long>> partitionToFilesMap = partitionInfoList.stream()
- .map(p -> {
- String partitionName = HoodieTableMetadataUtil.getPartitionIdentifier(p.getRelativePath());
- return Pair.of(partitionName, p.getFileNameToSizeMap());
- })
- .collect(Collectors.toMap(Pair::getKey, Pair::getValue));
-
- int totalDataFilesCount = partitionToFilesMap.values().stream().mapToInt(Map::size).sum();
- List<String> partitions = new ArrayList<>(partitionToFilesMap.keySet());
-
- if (partitionTypes.contains(MetadataPartitionType.FILES)) {
- // Record which saves the list of all partitions
- HoodieRecord allPartitionRecord = HoodieMetadataPayload.createPartitionListRecord(partitions);
- HoodieData<HoodieRecord> filesPartitionRecords = getFilesPartitionRecords(createInstantTime, partitionInfoList, allPartitionRecord);
- ValidationUtils.checkState(filesPartitionRecords.count() == (partitions.size() + 1));
- partitionToRecordsMap.put(MetadataPartitionType.FILES, filesPartitionRecords);
- }
-
- if (partitionTypes.contains(MetadataPartitionType.BLOOM_FILTERS) && totalDataFilesCount > 0) {
- final HoodieData<HoodieRecord> recordsRDD = HoodieTableMetadataUtil.convertFilesToBloomFilterRecords(
- engineContext, Collections.emptyMap(), partitionToFilesMap, getRecordsGenerationParams(), createInstantTime);
- partitionToRecordsMap.put(MetadataPartitionType.BLOOM_FILTERS, recordsRDD);
- }
-
- if (partitionTypes.contains(MetadataPartitionType.COLUMN_STATS) && totalDataFilesCount > 0) {
- final HoodieData<HoodieRecord> recordsRDD = HoodieTableMetadataUtil.convertFilesToColumnStatsRecords(
- engineContext, Collections.emptyMap(), partitionToFilesMap, getRecordsGenerationParams());
- partitionToRecordsMap.put(MetadataPartitionType.COLUMN_STATS, recordsRDD);
- }
- LOG.info("Committing " + partitions.size() + " partitions and " + totalDataFilesCount + " files to metadata");
+ private boolean validateTimelineBeforeSchedulingCompaction(Option<String> inFlightInstantTimestamp, String latestDeltacommitTime) {
+ // There should not be any incomplete instants on MDT
+ HoodieActiveTimeline metadataTimeline = metadataMetaClient.reloadActiveTimeline();
+ List<HoodieInstant> pendingInstantsOnMetadataTable = metadataTimeline.filterInflightsAndRequested().getInstants();
+ if (!pendingInstantsOnMetadataTable.isEmpty()) {
+ LOG.info(String.format(
+ "Cannot compact MDT as there are %d inflight instants: %s",
+ pendingInstantsOnMetadataTable.size(), Arrays.toString(pendingInstantsOnMetadataTable.toArray())));
+ metrics.ifPresent(m -> m.incrementMetric(HoodieMetadataMetrics.SKIP_TABLE_SERVICES, 1));
+ return false;
}
- commit(createInstantTime, partitionToRecordsMap, false);
- }
+ // There should not be any incomplete instants on dataset
+ HoodieActiveTimeline datasetTimeline = dataMetaClient.reloadActiveTimeline();
+ List<HoodieInstant> pendingInstantsOnDataset = datasetTimeline.filterInflightsAndRequested().getInstantsAsStream()
+ .filter(i -> !inFlightInstantTimestamp.isPresent() || !i.getTimestamp().equals(inFlightInstantTimestamp.get()))
+ .collect(Collectors.toList());
+ if (!pendingInstantsOnDataset.isEmpty()) {
+ LOG.info(String.format(
+ "Cannot compact MDT as there are %d inflight instants on dataset before latest deltacommit %s: %s",
+ pendingInstantsOnDataset.size(), latestDeltacommitTime, Arrays.toString(pendingInstantsOnDataset.toArray())));
+ metrics.ifPresent(m -> m.incrementMetric(HoodieMetadataMetrics.SKIP_TABLE_SERVICES, 1));
+ return false;
+ }
- private HoodieData<HoodieRecord> getFilesPartitionRecords(String createInstantTime, List<DirectoryInfo> partitionInfoList, HoodieRecord allPartitionRecord) {
- HoodieData<HoodieRecord> filesPartitionRecords = engineContext.parallelize(Arrays.asList(allPartitionRecord), 1);
- if (partitionInfoList.isEmpty()) {
- return filesPartitionRecords;
+ // Check if the inflight commit is greater than all the completed commits.
+ Option<HoodieInstant> lastCompletedInstant = dataMetaClient.getActiveTimeline().filterCompletedInstants().lastInstant();
+ if (!lastCompletedInstant.isPresent()) {
+ LOG.info("Last completed commit is not present.");
+ return false;
+ }
+ if (HoodieTimeline.compareTimestamps(lastCompletedInstant.get().getTimestamp(),
+ HoodieTimeline.GREATER_THAN, inFlightInstantTimestamp.get())) {
+ // Completed commits validation failed.
+ LOG.info(String.format(
+ "Cannot compact MDT as there is %s that is greater than inflight instant: %s",
+ lastCompletedInstant.get(), inFlightInstantTimestamp.get()));
+ return false;
}
- HoodieData<HoodieRecord> fileListRecords = engineContext.parallelize(partitionInfoList, partitionInfoList.size()).map(partitionInfo -> {
- Map<String, Long> fileNameToSizeMap = partitionInfo.getFileNameToSizeMap();
- // filter for files that are part of the completed commits
- Map<String, Long> validFileNameToSizeMap = fileNameToSizeMap.entrySet().stream().filter(fileSizePair -> {
- String commitTime = FSUtils.getCommitTime(fileSizePair.getKey());
- return HoodieTimeline.compareTimestamps(commitTime, HoodieTimeline.LESSER_THAN_OR_EQUALS, createInstantTime);
- }).collect(Collectors.toMap(Map.Entry::getKey, Map.Entry::getValue));
+ String latestDeltaCommitTimeInMetadataTable = metadataMetaClient.reloadActiveTimeline()
Review Comment:
I see that we are reloading more than once for datatable and could possibly w/ metadata table as well even within validateTimelineBeforeSchedulingCompaction. Can you revisit all such reloads and ensure we do it just once. these might turn out to be costly.
##########
hudi-client/hudi-client-common/src/main/java/org/apache/hudi/metadata/HoodieBackedTableMetadataWriter.java:
##########
@@ -1097,87 +1165,76 @@ protected void cleanIfNecessary(BaseHoodieWriteClient writeClient, String instan
// Trigger cleaning with suffixes based on the same instant time. This ensures that any future
// delta commits synced over will not have an instant time lesser than the last completed instant on the
// metadata table.
- writeClient.clean(instantTime + "002");
+ writeClient.clean(HoodieTableMetadataUtil.createCleanTimestamp(instantTime));
writeClient.lazyRollbackFailedIndexing();
}
/**
- * This is invoked to initialize metadata table for a dataset.
- * Initial commit has special handling mechanism due to its scale compared to other regular commits.
- * During cold startup, the list of files to be committed can be huge.
- * So creating a HoodieCommitMetadata out of these large number of files,
- * and calling the existing update(HoodieCommitMetadata) function does not scale well.
- * Hence, we have a special commit just for the initialization scenario.
+ * Validates the timeline for both main and metadata tables.
*/
- private void initialCommit(String createInstantTime, List<MetadataPartitionType> partitionTypes) {
- // List all partitions in the basePath of the containing dataset
- LOG.info("Initializing metadata table by using file listings in " + dataWriteConfig.getBasePath());
- engineContext.setJobStatus(this.getClass().getSimpleName(), "Initializing metadata table by listing files and partitions: " + dataWriteConfig.getTableName());
-
- Map<MetadataPartitionType, HoodieData<HoodieRecord>> partitionToRecordsMap = new HashMap<>();
-
- // skip file system listing to populate metadata records if it's a fresh table.
- // this is applicable only if the table already has N commits and metadata is enabled at a later point in time.
- if (createInstantTime.equals(SOLO_COMMIT_TIMESTAMP)) { // SOLO_COMMIT_TIMESTAMP will be the initial commit time in MDT for a fresh table.
- // If not, last completed commit in data table will be chosen as the initial commit time.
- LOG.info("Triggering empty Commit to metadata to initialize");
- } else {
- List<DirectoryInfo> partitionInfoList = listAllPartitions(dataMetaClient);
- Map<String, Map<String, Long>> partitionToFilesMap = partitionInfoList.stream()
- .map(p -> {
- String partitionName = HoodieTableMetadataUtil.getPartitionIdentifier(p.getRelativePath());
- return Pair.of(partitionName, p.getFileNameToSizeMap());
- })
- .collect(Collectors.toMap(Pair::getKey, Pair::getValue));
-
- int totalDataFilesCount = partitionToFilesMap.values().stream().mapToInt(Map::size).sum();
- List<String> partitions = new ArrayList<>(partitionToFilesMap.keySet());
-
- if (partitionTypes.contains(MetadataPartitionType.FILES)) {
- // Record which saves the list of all partitions
- HoodieRecord allPartitionRecord = HoodieMetadataPayload.createPartitionListRecord(partitions);
- HoodieData<HoodieRecord> filesPartitionRecords = getFilesPartitionRecords(createInstantTime, partitionInfoList, allPartitionRecord);
- ValidationUtils.checkState(filesPartitionRecords.count() == (partitions.size() + 1));
- partitionToRecordsMap.put(MetadataPartitionType.FILES, filesPartitionRecords);
- }
-
- if (partitionTypes.contains(MetadataPartitionType.BLOOM_FILTERS) && totalDataFilesCount > 0) {
- final HoodieData<HoodieRecord> recordsRDD = HoodieTableMetadataUtil.convertFilesToBloomFilterRecords(
- engineContext, Collections.emptyMap(), partitionToFilesMap, getRecordsGenerationParams(), createInstantTime);
- partitionToRecordsMap.put(MetadataPartitionType.BLOOM_FILTERS, recordsRDD);
- }
-
- if (partitionTypes.contains(MetadataPartitionType.COLUMN_STATS) && totalDataFilesCount > 0) {
- final HoodieData<HoodieRecord> recordsRDD = HoodieTableMetadataUtil.convertFilesToColumnStatsRecords(
- engineContext, Collections.emptyMap(), partitionToFilesMap, getRecordsGenerationParams());
- partitionToRecordsMap.put(MetadataPartitionType.COLUMN_STATS, recordsRDD);
- }
- LOG.info("Committing " + partitions.size() + " partitions and " + totalDataFilesCount + " files to metadata");
+ private boolean validateTimelineBeforeSchedulingCompaction(Option<String> inFlightInstantTimestamp, String latestDeltacommitTime) {
+ // There should not be any incomplete instants on MDT
+ HoodieActiveTimeline metadataTimeline = metadataMetaClient.reloadActiveTimeline();
+ List<HoodieInstant> pendingInstantsOnMetadataTable = metadataTimeline.filterInflightsAndRequested().getInstants();
+ if (!pendingInstantsOnMetadataTable.isEmpty()) {
+ LOG.info(String.format(
+ "Cannot compact MDT as there are %d inflight instants: %s",
+ pendingInstantsOnMetadataTable.size(), Arrays.toString(pendingInstantsOnMetadataTable.toArray())));
+ metrics.ifPresent(m -> m.incrementMetric(HoodieMetadataMetrics.SKIP_TABLE_SERVICES, 1));
+ return false;
}
- commit(createInstantTime, partitionToRecordsMap, false);
- }
+ // There should not be any incomplete instants on dataset
Review Comment:
I guess the conditions in L1215 - 1234 should subsume this condition (L 1187 to 1197). infact we had it this way probably 2 releases back and evolved it.
we can probably remove L1215 to 1234.
##########
hudi-common/src/main/java/org/apache/hudi/metadata/HoodieTableMetadataUtil.java:
##########
@@ -1378,6 +1339,206 @@ public static Set<String> getInflightAndCompletedMetadataPartitions(HoodieTableC
*/
public static boolean isIndexingCommit(String instantTime) {
return instantTime.length() == MILLIS_INSTANT_ID_LENGTH + METADATA_INDEXER_TIME_SUFFIX.length()
- && instantTime.endsWith(METADATA_INDEXER_TIME_SUFFIX);
+ && instantTime.endsWith(METADATA_INDEXER_TIME_SUFFIX);
+ }
+
+ /**
+ * Delete the metadata table for the dataset and backup if required.
+ *
+ * @param dataMetaClient {@code HoodieTableMetaClient} of the dataset for which metadata table is to be deleted
+ * @param context instance of {@link HoodieEngineContext}.
+ * @param backup Whether metadata table should be backed up before deletion. If true, the table is backed up to the
+ * directory with name metadata_<current_timestamp>.
+ * @return The backup directory if backup was requested
+ */
+ public static String deleteMetadataTable(HoodieTableMetaClient dataMetaClient, HoodieEngineContext context, boolean backup) {
+ final Path metadataTablePath = HoodieTableMetadata.getMetadataTableBasePath(dataMetaClient.getBasePathV2());
+ FileSystem fs = FSUtils.getFs(metadataTablePath.toString(), context.getHadoopConf().get());
+ setMetadataPartitionState(dataMetaClient, MetadataPartitionType.FILES, false);
Review Comment:
shouldn't we clear all partitions ?
##########
hudi-common/src/main/java/org/apache/hudi/metadata/HoodieTableMetadataUtil.java:
##########
@@ -1378,6 +1339,206 @@ public static Set<String> getInflightAndCompletedMetadataPartitions(HoodieTableC
*/
public static boolean isIndexingCommit(String instantTime) {
return instantTime.length() == MILLIS_INSTANT_ID_LENGTH + METADATA_INDEXER_TIME_SUFFIX.length()
- && instantTime.endsWith(METADATA_INDEXER_TIME_SUFFIX);
+ && instantTime.endsWith(METADATA_INDEXER_TIME_SUFFIX);
+ }
+
+ /**
+ * Delete the metadata table for the dataset and backup if required.
+ *
+ * @param dataMetaClient {@code HoodieTableMetaClient} of the dataset for which metadata table is to be deleted
+ * @param context instance of {@link HoodieEngineContext}.
+ * @param backup Whether metadata table should be backed up before deletion. If true, the table is backed up to the
+ * directory with name metadata_<current_timestamp>.
+ * @return The backup directory if backup was requested
+ */
+ public static String deleteMetadataTable(HoodieTableMetaClient dataMetaClient, HoodieEngineContext context, boolean backup) {
+ final Path metadataTablePath = HoodieTableMetadata.getMetadataTableBasePath(dataMetaClient.getBasePathV2());
+ FileSystem fs = FSUtils.getFs(metadataTablePath.toString(), context.getHadoopConf().get());
+ setMetadataPartitionState(dataMetaClient, MetadataPartitionType.FILES, false);
+ try {
+ if (!fs.exists(metadataTablePath)) {
+ return null;
+ }
+ } catch (FileNotFoundException e) {
+ // Ignoring exception as metadata table already does not exist
+ return null;
+ } catch (IOException e) {
+ throw new HoodieMetadataException("Failed to check metadata table existence", e);
+ }
+
+ if (backup) {
+ final Path metadataBackupPath = new Path(metadataTablePath.getParent(), ".metadata_" + HoodieActiveTimeline.createNewInstantTime());
+ LOG.info("Backing up metadata directory to " + metadataBackupPath + " before deletion");
+ try {
+ if (fs.rename(metadataTablePath, metadataBackupPath)) {
Review Comment:
also, who is responsible to clean up the backup metadata table? I assume its manual for now. and if no one intervenes, it might be left as is.
##########
hudi-common/src/main/java/org/apache/hudi/metadata/HoodieTableMetadataUtil.java:
##########
@@ -1378,6 +1339,206 @@ public static Set<String> getInflightAndCompletedMetadataPartitions(HoodieTableC
*/
public static boolean isIndexingCommit(String instantTime) {
return instantTime.length() == MILLIS_INSTANT_ID_LENGTH + METADATA_INDEXER_TIME_SUFFIX.length()
- && instantTime.endsWith(METADATA_INDEXER_TIME_SUFFIX);
+ && instantTime.endsWith(METADATA_INDEXER_TIME_SUFFIX);
+ }
+
+ /**
+ * Delete the metadata table for the dataset and backup if required.
+ *
+ * @param dataMetaClient {@code HoodieTableMetaClient} of the dataset for which metadata table is to be deleted
+ * @param context instance of {@link HoodieEngineContext}.
+ * @param backup Whether metadata table should be backed up before deletion. If true, the table is backed up to the
+ * directory with name metadata_<current_timestamp>.
+ * @return The backup directory if backup was requested
+ */
+ public static String deleteMetadataTable(HoodieTableMetaClient dataMetaClient, HoodieEngineContext context, boolean backup) {
+ final Path metadataTablePath = HoodieTableMetadata.getMetadataTableBasePath(dataMetaClient.getBasePathV2());
+ FileSystem fs = FSUtils.getFs(metadataTablePath.toString(), context.getHadoopConf().get());
+ setMetadataPartitionState(dataMetaClient, MetadataPartitionType.FILES, false);
+ try {
+ if (!fs.exists(metadataTablePath)) {
+ return null;
+ }
+ } catch (FileNotFoundException e) {
+ // Ignoring exception as metadata table already does not exist
+ return null;
+ } catch (IOException e) {
+ throw new HoodieMetadataException("Failed to check metadata table existence", e);
+ }
+
+ if (backup) {
+ final Path metadataBackupPath = new Path(metadataTablePath.getParent(), ".metadata_" + HoodieActiveTimeline.createNewInstantTime());
+ LOG.info("Backing up metadata directory to " + metadataBackupPath + " before deletion");
+ try {
+ if (fs.rename(metadataTablePath, metadataBackupPath)) {
+ return metadataBackupPath.toString();
+ }
+ } catch (Exception e) {
+ // If rename fails, we will ignore the backup and still delete the MDT
+ LOG.error("Failed to backup metadata table using rename", e);
+ }
+ }
+
+ LOG.info("Deleting metadata table from " + metadataTablePath);
+ try {
+ fs.delete(metadataTablePath, true);
+ } catch (Exception e) {
+ throw new HoodieMetadataException("Failed to delete metadata table from path " + metadataTablePath, e);
+ }
+
+ return null;
+ }
+
+ public static HoodieTableMetaClient setMetadataPartitionState(HoodieTableMetaClient dataMetaClient, MetadataPartitionType partitionType, boolean enabled) {
+ dataMetaClient.getTableConfig().setMetadataPartitionState(partitionType, enabled);
+ HoodieTableConfig.update(dataMetaClient.getFs(), new Path(dataMetaClient.getMetaPath()), dataMetaClient.getTableConfig().getProps());
+ dataMetaClient = HoodieTableMetaClient.reload(dataMetaClient);
+ ValidationUtils.checkState(dataMetaClient.getTableConfig().isMetadataPartitionEnabled(partitionType) == enabled,
+ "Metadata table state change should be persisted");
+
+ LOG.info(String.format("Metadata table %s partition %s has been %s", dataMetaClient.getBasePathV2(), partitionType,
+ enabled ? "enabled" : "disabled"));
+ return dataMetaClient;
+ }
+
+ /**
+ * Delete a partition within the metadata table.
+ * <p>
+ * This can be used to delete a partition so that it can be re-bootstrapped.
+ *
+ * @param dataMetaClient {@code HoodieTableMetaClient} of the dataset for which metadata table is to be deleted
+ * @param context instance of {@code HoodieEngineContext}.
+ * @param backup Whether metadata table should be backed up before deletion. If true, the table is backed up to the
+ * directory with name metadata_<current_timestamp>.
+ * @param partitionType The partition to delete
+ * @return The backup directory if backup was requested, null otherwise
+ */
+ public static String deleteMetadataTablePartition(HoodieTableMetaClient dataMetaClient, HoodieEngineContext context,
+ MetadataPartitionType partitionType, boolean backup) {
+ if (partitionType.equals(MetadataPartitionType.FILES)) {
+ return deleteMetadataTable(dataMetaClient, context, backup);
+ }
+
+ final Path metadataTablePartitionPath = new Path(HoodieTableMetadata.getMetadataTableBasePath(dataMetaClient.getBasePath()), partitionType.getPartitionPath());
+ FileSystem fs = FSUtils.getFs(metadataTablePartitionPath.toString(), context.getHadoopConf().get());
+ setMetadataPartitionState(dataMetaClient, partitionType, false);
+ try {
+ if (!fs.exists(metadataTablePartitionPath)) {
+ return null;
+ }
+ } catch (FileNotFoundException e) {
+ // Ignoring exception as metadata table already does not exist
+ LOG.debug("Metadata table partition " + partitionType + " not found at path " + metadataTablePartitionPath);
+ return null;
+ } catch (Exception e) {
+ throw new HoodieMetadataException("Failed to check metadata table partition existence", e);
Review Comment:
can we print the partition of interest along w/ full path.
##########
hudi-common/src/main/java/org/apache/hudi/metadata/HoodieTableMetadataUtil.java:
##########
@@ -1378,6 +1339,206 @@ public static Set<String> getInflightAndCompletedMetadataPartitions(HoodieTableC
*/
public static boolean isIndexingCommit(String instantTime) {
return instantTime.length() == MILLIS_INSTANT_ID_LENGTH + METADATA_INDEXER_TIME_SUFFIX.length()
- && instantTime.endsWith(METADATA_INDEXER_TIME_SUFFIX);
+ && instantTime.endsWith(METADATA_INDEXER_TIME_SUFFIX);
+ }
+
+ /**
+ * Delete the metadata table for the dataset and backup if required.
+ *
+ * @param dataMetaClient {@code HoodieTableMetaClient} of the dataset for which metadata table is to be deleted
+ * @param context instance of {@link HoodieEngineContext}.
+ * @param backup Whether metadata table should be backed up before deletion. If true, the table is backed up to the
+ * directory with name metadata_<current_timestamp>.
+ * @return The backup directory if backup was requested
+ */
+ public static String deleteMetadataTable(HoodieTableMetaClient dataMetaClient, HoodieEngineContext context, boolean backup) {
+ final Path metadataTablePath = HoodieTableMetadata.getMetadataTableBasePath(dataMetaClient.getBasePathV2());
+ FileSystem fs = FSUtils.getFs(metadataTablePath.toString(), context.getHadoopConf().get());
+ setMetadataPartitionState(dataMetaClient, MetadataPartitionType.FILES, false);
+ try {
+ if (!fs.exists(metadataTablePath)) {
+ return null;
+ }
+ } catch (FileNotFoundException e) {
+ // Ignoring exception as metadata table already does not exist
+ return null;
+ } catch (IOException e) {
+ throw new HoodieMetadataException("Failed to check metadata table existence", e);
+ }
+
+ if (backup) {
+ final Path metadataBackupPath = new Path(metadataTablePath.getParent(), ".metadata_" + HoodieActiveTimeline.createNewInstantTime());
+ LOG.info("Backing up metadata directory to " + metadataBackupPath + " before deletion");
+ try {
+ if (fs.rename(metadataTablePath, metadataBackupPath)) {
+ return metadataBackupPath.toString();
+ }
+ } catch (Exception e) {
+ // If rename fails, we will ignore the backup and still delete the MDT
+ LOG.error("Failed to backup metadata table using rename", e);
+ }
+ }
+
+ LOG.info("Deleting metadata table from " + metadataTablePath);
+ try {
+ fs.delete(metadataTablePath, true);
+ } catch (Exception e) {
+ throw new HoodieMetadataException("Failed to delete metadata table from path " + metadataTablePath, e);
+ }
+
+ return null;
+ }
+
+ public static HoodieTableMetaClient setMetadataPartitionState(HoodieTableMetaClient dataMetaClient, MetadataPartitionType partitionType, boolean enabled) {
+ dataMetaClient.getTableConfig().setMetadataPartitionState(partitionType, enabled);
+ HoodieTableConfig.update(dataMetaClient.getFs(), new Path(dataMetaClient.getMetaPath()), dataMetaClient.getTableConfig().getProps());
+ dataMetaClient = HoodieTableMetaClient.reload(dataMetaClient);
+ ValidationUtils.checkState(dataMetaClient.getTableConfig().isMetadataPartitionEnabled(partitionType) == enabled,
+ "Metadata table state change should be persisted");
+
+ LOG.info(String.format("Metadata table %s partition %s has been %s", dataMetaClient.getBasePathV2(), partitionType,
+ enabled ? "enabled" : "disabled"));
+ return dataMetaClient;
+ }
+
+ /**
+ * Delete a partition within the metadata table.
+ * <p>
+ * This can be used to delete a partition so that it can be re-bootstrapped.
+ *
+ * @param dataMetaClient {@code HoodieTableMetaClient} of the dataset for which metadata table is to be deleted
+ * @param context instance of {@code HoodieEngineContext}.
+ * @param backup Whether metadata table should be backed up before deletion. If true, the table is backed up to the
+ * directory with name metadata_<current_timestamp>.
+ * @param partitionType The partition to delete
+ * @return The backup directory if backup was requested, null otherwise
+ */
+ public static String deleteMetadataTablePartition(HoodieTableMetaClient dataMetaClient, HoodieEngineContext context,
+ MetadataPartitionType partitionType, boolean backup) {
+ if (partitionType.equals(MetadataPartitionType.FILES)) {
+ return deleteMetadataTable(dataMetaClient, context, backup);
+ }
+
+ final Path metadataTablePartitionPath = new Path(HoodieTableMetadata.getMetadataTableBasePath(dataMetaClient.getBasePath()), partitionType.getPartitionPath());
+ FileSystem fs = FSUtils.getFs(metadataTablePartitionPath.toString(), context.getHadoopConf().get());
+ setMetadataPartitionState(dataMetaClient, partitionType, false);
+ try {
+ if (!fs.exists(metadataTablePartitionPath)) {
+ return null;
+ }
+ } catch (FileNotFoundException e) {
+ // Ignoring exception as metadata table already does not exist
+ LOG.debug("Metadata table partition " + partitionType + " not found at path " + metadataTablePartitionPath);
+ return null;
+ } catch (Exception e) {
+ throw new HoodieMetadataException("Failed to check metadata table partition existence", e);
+ }
+
+ if (backup) {
+ final Path metadataPartitionBackupPath = new Path(metadataTablePartitionPath.getParent().getParent(),
+ String.format(".metadata_%s_%s", partitionType.getPartitionPath(), HoodieActiveTimeline.createNewInstantTime()));
+ LOG.info(String.format("Backing up MDT partition %s to %s before deletion", partitionType, metadataPartitionBackupPath));
+ try {
+ if (fs.rename(metadataTablePartitionPath, metadataPartitionBackupPath)) {
+ return metadataPartitionBackupPath.toString();
+ }
+ } catch (Exception e) {
+ // If rename fails, we will try to delete the table instead
+ LOG.error(String.format("Failed to backup MDT partition %s using rename", partitionType), e);
+ }
+ } else {
+ LOG.info("Deleting metadata table partition from " + metadataTablePartitionPath);
+ try {
+ fs.delete(metadataTablePartitionPath, true);
+ } catch (Exception e) {
+ throw new HoodieMetadataException("Failed to delete metadata table partition from path " + metadataTablePartitionPath, e);
+ }
+ }
+
+ return null;
+ }
+
+ public static HoodieTableMetaClient setMetadataInflightPartitions(HoodieTableMetaClient dataMetaClient, List<MetadataPartitionType> partitionTypes) {
+ dataMetaClient.getTableConfig().setMetadataPartitionsAsInflight(partitionTypes);
+ HoodieTableConfig.update(dataMetaClient.getFs(), new Path(dataMetaClient.getMetaPath()), dataMetaClient.getTableConfig().getProps());
+ dataMetaClient = HoodieTableMetaClient.reload(dataMetaClient);
+
+ LOG.info(String.format("Metadata table %s partitions %s hasvebeen set to inflight", dataMetaClient.getBasePathV2(), partitionTypes));
+ return dataMetaClient;
+ }
+
+ /**
+ * Return the complete fileID for a file group within a MDT partition.
+ * <p>
+ * MDT fileGroups have the format <fileIDPrefix>-<index>. The fileIDPrefix is hardcoded for each MDT partition and index is an integer.
+ *
+ * @param partitionType The type of the MDT partition
+ * @param index Index of the file group within the partition
+ * @return The fileID
+ */
+ public static String getFileIDForFileGroup(MetadataPartitionType partitionType, int index) {
+ return String.format("%s%04d", partitionType.getFileIdPrefix(), index);
+ }
+
+ /**
+ * Extract the index from the fileID of a file group in the MDT partition. See {@code getFileIDForFileGroup} for the format of the fileID.
+ *
+ * @param fileId fileID of a file group.
+ * @return The index of file group
+ */
+ public static int getFileGroupIndexFromFileId(String fileId) {
Review Comment:
can we add UTs for these static methods
##########
hudi-common/src/main/java/org/apache/hudi/metadata/HoodieTableMetadataUtil.java:
##########
@@ -1378,6 +1339,206 @@ public static Set<String> getInflightAndCompletedMetadataPartitions(HoodieTableC
*/
public static boolean isIndexingCommit(String instantTime) {
return instantTime.length() == MILLIS_INSTANT_ID_LENGTH + METADATA_INDEXER_TIME_SUFFIX.length()
- && instantTime.endsWith(METADATA_INDEXER_TIME_SUFFIX);
+ && instantTime.endsWith(METADATA_INDEXER_TIME_SUFFIX);
+ }
+
+ /**
+ * Delete the metadata table for the dataset and backup if required.
+ *
+ * @param dataMetaClient {@code HoodieTableMetaClient} of the dataset for which metadata table is to be deleted
+ * @param context instance of {@link HoodieEngineContext}.
+ * @param backup Whether metadata table should be backed up before deletion. If true, the table is backed up to the
+ * directory with name metadata_<current_timestamp>.
+ * @return The backup directory if backup was requested
+ */
+ public static String deleteMetadataTable(HoodieTableMetaClient dataMetaClient, HoodieEngineContext context, boolean backup) {
+ final Path metadataTablePath = HoodieTableMetadata.getMetadataTableBasePath(dataMetaClient.getBasePathV2());
+ FileSystem fs = FSUtils.getFs(metadataTablePath.toString(), context.getHadoopConf().get());
+ setMetadataPartitionState(dataMetaClient, MetadataPartitionType.FILES, false);
+ try {
+ if (!fs.exists(metadataTablePath)) {
+ return null;
+ }
+ } catch (FileNotFoundException e) {
+ // Ignoring exception as metadata table already does not exist
+ return null;
+ } catch (IOException e) {
+ throw new HoodieMetadataException("Failed to check metadata table existence", e);
+ }
+
+ if (backup) {
+ final Path metadataBackupPath = new Path(metadataTablePath.getParent(), ".metadata_" + HoodieActiveTimeline.createNewInstantTime());
+ LOG.info("Backing up metadata directory to " + metadataBackupPath + " before deletion");
+ try {
+ if (fs.rename(metadataTablePath, metadataBackupPath)) {
+ return metadataBackupPath.toString();
+ }
+ } catch (Exception e) {
+ // If rename fails, we will ignore the backup and still delete the MDT
+ LOG.error("Failed to backup metadata table using rename", e);
+ }
+ }
+
+ LOG.info("Deleting metadata table from " + metadataTablePath);
+ try {
+ fs.delete(metadataTablePath, true);
+ } catch (Exception e) {
+ throw new HoodieMetadataException("Failed to delete metadata table from path " + metadataTablePath, e);
+ }
+
+ return null;
+ }
+
+ public static HoodieTableMetaClient setMetadataPartitionState(HoodieTableMetaClient dataMetaClient, MetadataPartitionType partitionType, boolean enabled) {
+ dataMetaClient.getTableConfig().setMetadataPartitionState(partitionType, enabled);
+ HoodieTableConfig.update(dataMetaClient.getFs(), new Path(dataMetaClient.getMetaPath()), dataMetaClient.getTableConfig().getProps());
+ dataMetaClient = HoodieTableMetaClient.reload(dataMetaClient);
+ ValidationUtils.checkState(dataMetaClient.getTableConfig().isMetadataPartitionEnabled(partitionType) == enabled,
+ "Metadata table state change should be persisted");
+
+ LOG.info(String.format("Metadata table %s partition %s has been %s", dataMetaClient.getBasePathV2(), partitionType,
+ enabled ? "enabled" : "disabled"));
+ return dataMetaClient;
+ }
+
+ /**
+ * Delete a partition within the metadata table.
+ * <p>
+ * This can be used to delete a partition so that it can be re-bootstrapped.
+ *
+ * @param dataMetaClient {@code HoodieTableMetaClient} of the dataset for which metadata table is to be deleted
+ * @param context instance of {@code HoodieEngineContext}.
+ * @param backup Whether metadata table should be backed up before deletion. If true, the table is backed up to the
+ * directory with name metadata_<current_timestamp>.
+ * @param partitionType The partition to delete
+ * @return The backup directory if backup was requested, null otherwise
+ */
+ public static String deleteMetadataTablePartition(HoodieTableMetaClient dataMetaClient, HoodieEngineContext context,
Review Comment:
may I know in which code path the backup is set to true. I could not triage it. every caller is setting it to false.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] prashantwason commented on a diff in pull request #8684: [HUDI-6200] Enhancements to the MDT for improving performance of larger indexes.
Posted by "prashantwason (via GitHub)" <gi...@apache.org>.
prashantwason commented on code in PR #8684:
URL: https://github.com/apache/hudi/pull/8684#discussion_r1203530124
##########
hudi-client/hudi-client-common/src/main/java/org/apache/hudi/metadata/HoodieBackedTableMetadataWriter.java:
##########
@@ -564,53 +532,147 @@ private <T extends SpecificRecordBase> boolean isCommitRevertedByInFlightAction(
/**
* Initialize the Metadata Table by listing files and partitions from the file system.
*
- * @param dataMetaClient - {@code HoodieTableMetaClient} for the dataset.
+ * @param initializationTime - Timestamp to use for the commit
+ * @param partitionsToInit - List of MDT partitions to initialize
* @param inflightInstantTimestamp - Current action instant responsible for this initialization
*/
- private boolean initializeFromFilesystem(HoodieTableMetaClient dataMetaClient,
+ private boolean initializeFromFilesystem(String initializationTime, List<MetadataPartitionType> partitionsToInit,
Option<String> inflightInstantTimestamp) throws IOException {
if (anyPendingDataInstant(dataMetaClient, inflightInstantTimestamp)) {
return false;
}
- String createInstantTime = getInitialCommitInstantTime(dataMetaClient);
-
- initializeMetaClient(DEFAULT_METADATA_POPULATE_META_FIELDS);
- initTableMetadata();
- // if async metadata indexing is enabled,
- // then only initialize files partition as other partitions will be built using HoodieIndexer
- List<MetadataPartitionType> enabledPartitionTypes = new ArrayList<>();
- if (dataWriteConfig.isMetadataAsyncIndex()) {
- enabledPartitionTypes.add(MetadataPartitionType.FILES);
- } else {
- // all enabled ones should be initialized
- enabledPartitionTypes = this.enabledPartitionTypes;
+ // FILES partition is always initialized first
+ ValidationUtils.checkArgument(!partitionsToInit.contains(MetadataPartitionType.FILES)
+ || partitionsToInit.get(0).equals(MetadataPartitionType.FILES), "FILES partition should be initialized first: " + partitionsToInit);
+
+ metadataMetaClient = initializeMetaClient();
+
+ // Get a complete list of files and partitions from the file system or from already initialized FILES partition of MDT
+ boolean filesPartitionAvailable = dataMetaClient.getTableConfig().isMetadataPartitionEnabled(MetadataPartitionType.FILES);
+ List<DirectoryInfo> partitionInfoList = filesPartitionAvailable ? listAllPartitionsFromMDT(initializationTime) : listAllPartitionsFromFilesystem(initializationTime);
+ Map<String, Map<String, Long>> partitionToFilesMap = partitionInfoList.stream()
Review Comment:
If the MDT already exists then we are here initializing other partitions. For that we would need the list of all files and partitions which we can load from MDT files partition itself.
MDT files partitions is pre-requisite for all other partitions so it should either exist or be created first.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] prashantwason commented on a diff in pull request #8684: [HUDI-6200] Enhancements to the MDT for improving performance of larger indexes.
Posted by "prashantwason (via GitHub)" <gi...@apache.org>.
prashantwason commented on code in PR #8684:
URL: https://github.com/apache/hudi/pull/8684#discussion_r1192160408
##########
hudi-client/hudi-client-common/src/main/java/org/apache/hudi/metadata/HoodieBackedTableMetadataWriter.java:
##########
@@ -373,105 +356,92 @@ public List<MetadataPartitionType> getEnabledPartitionTypes() {
return this.enabledPartitionTypes;
}
- /**
- * Initialize the metadata table if it does not exist.
- * <p>
- * If the metadata table does not exist, then file and partition listing is used to initialize the table.
- *
- * @param engineContext
- * @param actionMetadata Action metadata types extending Avro generated SpecificRecordBase
- * @param inflightInstantTimestamp Timestamp of an instant in progress on the dataset. This instant is ignored
- * while deciding to initialize the metadata table.
- */
- protected abstract <T extends SpecificRecordBase> void initialize(HoodieEngineContext engineContext,
- Option<T> actionMetadata,
- Option<String> inflightInstantTimestamp);
-
- public void initTableMetadata() {
- try {
- if (this.metadata != null) {
- this.metadata.close();
- }
- this.metadata = new HoodieBackedTableMetadata(engineContext, dataWriteConfig.getMetadataConfig(),
- dataWriteConfig.getBasePath(), dataWriteConfig.getSpillableMapBasePath());
- this.metadataMetaClient = metadata.getMetadataMetaClient();
- } catch (Exception e) {
- throw new HoodieException("Error initializing metadata table for reads", e);
- }
- }
-
/**
* Initialize the metadata table if needed.
*
* @param dataMetaClient - meta client for the data table
* @param actionMetadata - optional action metadata
* @param inflightInstantTimestamp - timestamp of an instant in progress on the dataset
* @param <T> - action metadata types extending Avro generated SpecificRecordBase
- * @throws IOException
+ * @throws IOException on errors
*/
- protected <T extends SpecificRecordBase> void initializeIfNeeded(HoodieTableMetaClient dataMetaClient,
- Option<T> actionMetadata,
- Option<String> inflightInstantTimestamp) throws IOException {
+ protected <T extends SpecificRecordBase> boolean initializeIfNeeded(HoodieTableMetaClient dataMetaClient,
+ Option<T> actionMetadata,
+ Option<String> inflightInstantTimestamp) throws IOException {
HoodieTimer timer = HoodieTimer.start();
+ List<MetadataPartitionType> partitionsToInit = new ArrayList<>(MetadataPartitionType.values().length);
- boolean exists = metadataTableExists(dataMetaClient, actionMetadata);
+ try {
+ boolean exists = metadataTableExists(dataMetaClient, actionMetadata);
+ if (!exists) {
+ // FILES partition is always required
+ partitionsToInit.add(MetadataPartitionType.FILES);
+ }
- if (!exists) {
- // Initialize for the first time by listing partitions and files directly from the file system
- if (initializeFromFilesystem(dataMetaClient, inflightInstantTimestamp)) {
- metrics.ifPresent(m -> m.updateMetrics(HoodieMetadataMetrics.INITIALIZE_STR, timer.endTimer()));
+ // check if any of the enabled partition types needs to be initialized
+ // NOTE: It needs to be guarded by async index config because if that is enabled then initialization happens through the index scheduler.
+ if (!dataWriteConfig.isMetadataAsyncIndex()) {
+ Set<String> inflightAndCompletedPartitions = getInflightAndCompletedMetadataPartitions(dataMetaClient.getTableConfig());
+ LOG.info("Async metadata indexing disabled and following partitions already initialized: " + inflightAndCompletedPartitions);
+ this.enabledPartitionTypes.stream()
+ .filter(p -> !inflightAndCompletedPartitions.contains(p.getPartitionPath()) && !MetadataPartitionType.FILES.equals(p))
+ .forEach(partitionsToInit::add);
}
- return;
- }
- // if metadata table exists, then check if any of the enabled partition types needs to be initialized
- // NOTE: It needs to be guarded by async index config because if that is enabled then initialization happens through the index scheduler.
- if (!dataWriteConfig.isMetadataAsyncIndex()) {
- Set<String> inflightAndCompletedPartitions = getInflightAndCompletedMetadataPartitions(dataMetaClient.getTableConfig());
- LOG.info("Async metadata indexing enabled and following partitions already initialized: " + inflightAndCompletedPartitions);
- List<MetadataPartitionType> partitionsToInit = this.enabledPartitionTypes.stream()
- .filter(p -> !inflightAndCompletedPartitions.contains(p.getPartitionPath()) && !MetadataPartitionType.FILES.equals(p))
- .collect(Collectors.toList());
- // if there are no partitions to initialize or there is a pending operation, then don't initialize in this round
- if (partitionsToInit.isEmpty() || anyPendingDataInstant(dataMetaClient, inflightInstantTimestamp)) {
- return;
+ if (partitionsToInit.isEmpty()) {
+ // No partitions to initialize
+ return true;
}
- String createInstantTime = getInitialCommitInstantTime(dataMetaClient);
- initTableMetadata(); // re-init certain flags in BaseTableMetadata
- initializeEnabledFileGroups(dataMetaClient, createInstantTime, partitionsToInit);
- initialCommit(createInstantTime, partitionsToInit);
- updateInitializedPartitionsInTableConfig(partitionsToInit);
+ // If there is no commit on the dataset yet, use the SOLO_COMMIT_TIMESTAMP as the instant time for initial commit
+ // Otherwise, we use the timestamp of the latest completed action.
+ String initializationTime = dataMetaClient.getActiveTimeline().filterCompletedInstants()
+ .getReverseOrderedInstants().findFirst().map(HoodieInstant::getTimestamp).orElse(SOLO_COMMIT_TIMESTAMP);
Review Comment:
Done
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #8684: [HUDI-6200] Enhancements to the MDT for improving performance of larger indexes.
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #8684:
URL: https://github.com/apache/hudi/pull/8684#issuecomment-1542969346
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "35148aeb4ba78eb6f3316c75f8a0a7e4c6d6df87",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17007",
"triggerID" : "35148aeb4ba78eb6f3316c75f8a0a7e4c6d6df87",
"triggerType" : "PUSH"
}, {
"hash" : "e2785f4675ddf74582ff34590608a5d71c5e9a2d",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "e2785f4675ddf74582ff34590608a5d71c5e9a2d",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 35148aeb4ba78eb6f3316c75f8a0a7e4c6d6df87 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17007)
* e2785f4675ddf74582ff34590608a5d71c5e9a2d UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] prashantwason commented on a diff in pull request #8684: [HUDI-6200] Enhancements to the MDT for improving performance of larger indexes.
Posted by "prashantwason (via GitHub)" <gi...@apache.org>.
prashantwason commented on code in PR #8684:
URL: https://github.com/apache/hudi/pull/8684#discussion_r1192161869
##########
hudi-client/hudi-client-common/src/main/java/org/apache/hudi/metadata/HoodieBackedTableMetadataWriter.java:
##########
@@ -824,25 +863,22 @@ private interface ConvertMetadataFunction {
/**
* Processes commit metadata from data table and commits to metadata table.
*
- * @param instantTime instant time of interest.
+ * @param instantTime instant time of interest.
* @param convertMetadataFunction converter function to convert the respective metadata to List of HoodieRecords to be written to metadata table.
- * @param <T> type of commit metadata.
- * @param canTriggerTableService true if table services can be triggered. false otherwise.
*/
- private <T> void processAndCommit(String instantTime, ConvertMetadataFunction convertMetadataFunction, boolean canTriggerTableService) {
- if (!dataWriteConfig.isMetadataTableEnabled()) {
- return;
- }
+ private void processAndCommit(String instantTime, ConvertMetadataFunction convertMetadataFunction) {
+ ValidationUtils.checkArgument(dataWriteConfig.isMetadataTableEnabled());
+
Set<String> partitionsToUpdate = getMetadataPartitionsToUpdate();
Set<String> inflightIndexes = getInflightMetadataPartitions(dataMetaClient.getTableConfig());
// if indexing is inflight then do not trigger table service
boolean doNotTriggerTableService = partitionsToUpdate.stream().anyMatch(inflightIndexes::contains);
Review Comment:
You mean the ValidationUtils.checkArgument? Seems redundant as this is already checked in constructor.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #8684: [HUDI-6200] Enhancements to the MDT for improving performance of larger indexes.
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #8684:
URL: https://github.com/apache/hudi/pull/8684#issuecomment-1542935837
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "35148aeb4ba78eb6f3316c75f8a0a7e4c6d6df87",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17007",
"triggerID" : "35148aeb4ba78eb6f3316c75f8a0a7e4c6d6df87",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 35148aeb4ba78eb6f3316c75f8a0a7e4c6d6df87 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17007)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #8684: [HUDI-6200] Enhancements to the MDT for improving performance of larger indexes.
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #8684:
URL: https://github.com/apache/hudi/pull/8684#issuecomment-1542931783
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "35148aeb4ba78eb6f3316c75f8a0a7e4c6d6df87",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "35148aeb4ba78eb6f3316c75f8a0a7e4c6d6df87",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 35148aeb4ba78eb6f3316c75f8a0a7e4c6d6df87 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] nsivabalan commented on a diff in pull request #8684: [HUDI-6200] Enhancements to the MDT for improving performance of larger indexes.
Posted by "nsivabalan (via GitHub)" <gi...@apache.org>.
nsivabalan commented on code in PR #8684:
URL: https://github.com/apache/hudi/pull/8684#discussion_r1192675584
##########
hudi-client/hudi-client-common/src/main/java/org/apache/hudi/table/action/index/ScheduleIndexActionExecutor.java:
##########
@@ -100,15 +99,6 @@ public Option<HoodieIndexPlan> execute() {
// get last completed instant
Option<HoodieInstant> indexUptoInstant = table.getActiveTimeline().getContiguousCompletedWriteTimeline().lastInstant();
if (indexUptoInstant.isPresent()) {
- // start initializing file groups
- // in case FILES partition itself was not initialized before (i.e. metadata was never enabled), this will initialize synchronously
- HoodieTableMetadataWriter metadataWriter = table.getMetadataWriter(instantTime)
- .orElseThrow(() -> new HoodieIndexException(String.format("Could not get metadata writer to initialize filegroups for indexing for instant: %s", instantTime)));
- if (!finalPartitionsToIndex.get(0).getPartitionPath().equals(MetadataPartitionType.FILES.getPartitionPath())) {
- // initialize metadata partition only if not for FILES partition.
- metadataWriter.initializeMetadataPartitions(table.getMetaClient(), finalPartitionsToIndex, indexUptoInstant.get().getTimestamp());
Review Comment:
So, prior to this patch, during schedule indexing, we will initialize the partition of interest w/ empty delete log block. and when executing the indexing action, we will fully populate the valid records in MDT for the resp partition.
but after this patch, during schedule, we don't do such initialization. Only when we are executing the indexing action, we will initialize the partition of interest and populate the records as well
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #8684: [HUDI-6200] Enhancements to the MDT for improving performance of larger indexes.
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #8684:
URL: https://github.com/apache/hudi/pull/8684#issuecomment-1561936210
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "35148aeb4ba78eb6f3316c75f8a0a7e4c6d6df87",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17007",
"triggerID" : "35148aeb4ba78eb6f3316c75f8a0a7e4c6d6df87",
"triggerType" : "PUSH"
}, {
"hash" : "e2785f4675ddf74582ff34590608a5d71c5e9a2d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17008",
"triggerID" : "e2785f4675ddf74582ff34590608a5d71c5e9a2d",
"triggerType" : "PUSH"
}, {
"hash" : "364b1fe0dde303750ca0f8eebe5cd530c2a66a3a",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17042",
"triggerID" : "364b1fe0dde303750ca0f8eebe5cd530c2a66a3a",
"triggerType" : "PUSH"
}, {
"hash" : "9d73b9664eed48ada68721a92526640cfa06a021",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17098",
"triggerID" : "9d73b9664eed48ada68721a92526640cfa06a021",
"triggerType" : "PUSH"
}, {
"hash" : "7be538f4045a42ba33ab8fe62594178e2db75bbb",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17100",
"triggerID" : "7be538f4045a42ba33ab8fe62594178e2db75bbb",
"triggerType" : "PUSH"
}, {
"hash" : "cc0da2372d50d99c98c2ce4bcbe5a60303bde938",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17195",
"triggerID" : "cc0da2372d50d99c98c2ce4bcbe5a60303bde938",
"triggerType" : "PUSH"
}, {
"hash" : "da0e00e3655e06b064d11e20500317e2a1d2229c",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "da0e00e3655e06b064d11e20500317e2a1d2229c",
"triggerType" : "PUSH"
}, {
"hash" : "589c8cd7ffc5df7984c099d6b30fa117463db2b9",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17335",
"triggerID" : "589c8cd7ffc5df7984c099d6b30fa117463db2b9",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* da0e00e3655e06b064d11e20500317e2a1d2229c UNKNOWN
* 589c8cd7ffc5df7984c099d6b30fa117463db2b9 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17335)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] prashantwason commented on a diff in pull request #8684: [HUDI-6200] Enhancements to the MDT for improving performance of larger indexes.
Posted by "prashantwason (via GitHub)" <gi...@apache.org>.
prashantwason commented on code in PR #8684:
URL: https://github.com/apache/hudi/pull/8684#discussion_r1203532474
##########
hudi-client/hudi-client-common/src/main/java/org/apache/hudi/metadata/HoodieBackedTableMetadataWriter.java:
##########
@@ -1052,51 +1091,81 @@ protected HoodieData<HoodieRecord> prepRecords(Map<MetadataPartitionType,
}
/**
- * Perform a compaction on the Metadata Table.
- *
- * Cases to be handled:
- * 1. We cannot perform compaction if there are previous inflight operations on the dataset. This is because
- * a compacted metadata base file at time Tx should represent all the actions on the dataset till time Tx.
- *
- * 2. In multi-writer scenario, a parallel operation with a greater instantTime may have completed creating a
- * deltacommit.
+ * Optimize the metadata table by running compaction, clean and archive as required.
+ * <p>
+ * Don't perform optimization if there are inflight operations on the dataset. This is for two reasons:
+ * - The compaction will contain the correct data as all failed operations have been rolled back.
+ * - Clean/compaction etc. will have the highest timestamp on the MDT and we won't be adding new operations
+ * with smaller timestamps to metadata table (makes for easier debugging)
+ * <p>
+ * This adds the limitations that long-running async operations (clustering, etc.) may cause delay in such MDT
+ * optimizations. We will relax this after MDT code has been hardened.
*/
- protected void compactIfNecessary(BaseHoodieWriteClient writeClient, String instantTime) {
- // finish off any pending compactions if any from previous attempt.
- writeClient.runAnyPendingCompactions();
-
- String latestDeltaCommitTimeInMetadataTable = metadataMetaClient.reloadActiveTimeline()
- .getDeltaCommitTimeline()
- .filterCompletedInstants()
- .lastInstant().orElseThrow(() -> new HoodieMetadataException("No completed deltacommit in metadata table"))
- .getTimestamp();
- // we need to find if there are any inflights in data table timeline before or equal to the latest delta commit in metadata table.
- // Whenever you want to change this logic, please ensure all below scenarios are considered.
- // a. There could be a chance that latest delta commit in MDT is committed in MDT, but failed in DT. And so findInstantsBeforeOrEquals() should be employed
- // b. There could be DT inflights after latest delta commit in MDT and we are ok with it. bcoz, the contract is, latest compaction instant time in MDT represents
- // any instants before that is already synced with metadata table.
- // c. Do consider out of order commits. For eg, c4 from DT could complete before c3. and we can't trigger compaction in MDT with c4 as base instant time, until every
- // instant before c4 is synced with metadata table.
- List<HoodieInstant> pendingInstants = dataMetaClient.reloadActiveTimeline().filterInflightsAndRequested()
- .findInstantsBeforeOrEquals(latestDeltaCommitTimeInMetadataTable).getInstants();
+ @Override
+ public void performTableServices(Option<String> inFlightInstantTimestamp) {
+ HoodieTimer metadataTableServicesTimer = HoodieTimer.start();
+ boolean allTableServicesExecutedSuccessfullyOrSkipped = true;
+ try {
+ BaseHoodieWriteClient writeClient = getWriteClient();
+ // Run any pending table services operations.
+ runPendingTableServicesOperations(writeClient);
+
+ // Check and run clean operations.
+ String latestDeltacommitTime = metadataMetaClient.reloadActiveTimeline().getDeltaCommitTimeline()
+ .filterCompletedInstants()
+ .lastInstant().get()
+ .getTimestamp();
+ LOG.info("Latest deltacommit time found is " + latestDeltacommitTime + ", running clean operations.");
+ cleanIfNecessary(writeClient, latestDeltacommitTime);
+
+ // Do timeline validation before scheduling compaction/logcompaction operations.
+ if (!validateTimelineBeforeSchedulingCompaction(inFlightInstantTimestamp, latestDeltacommitTime)) {
+ return;
Review Comment:
Done
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] prashantwason commented on a diff in pull request #8684: [HUDI-6200] Enhancements to the MDT for improving performance of larger indexes.
Posted by "prashantwason (via GitHub)" <gi...@apache.org>.
prashantwason commented on code in PR #8684:
URL: https://github.com/apache/hudi/pull/8684#discussion_r1194226408
##########
hudi-client/hudi-spark-client/src/test/java/org/apache/hudi/client/functional/TestHoodieBackedMetadata.java:
##########
@@ -2482,6 +2483,14 @@ public void testMetadataMetrics() throws Exception {
}
}
+ @Test
+ public void testGetFileGroupIndexFromFileId() {
Review Comment:
Added test for first 3.
getFileIdLengthWithoutFileIndex is private and is called from other functions so is indirectly tested.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] prashantwason commented on a diff in pull request #8684: [HUDI-6200] Enhancements to the MDT for improving performance of larger indexes.
Posted by "prashantwason (via GitHub)" <gi...@apache.org>.
prashantwason commented on code in PR #8684:
URL: https://github.com/apache/hudi/pull/8684#discussion_r1194208983
##########
hudi-client/hudi-client-common/src/main/java/org/apache/hudi/metadata/HoodieBackedTableMetadataWriter.java:
##########
@@ -562,53 +532,144 @@ private <T extends SpecificRecordBase> boolean isCommitRevertedByInFlightAction(
/**
* Initialize the Metadata Table by listing files and partitions from the file system.
*
- * @param dataMetaClient - {@code HoodieTableMetaClient} for the dataset.
+ * @param initializationTime - Timestamp to use for the commit
+ * @param partitionsToInit - List of MDT partitions to initialize
* @param inflightInstantTimestamp - Current action instant responsible for this initialization
*/
- private boolean initializeFromFilesystem(HoodieTableMetaClient dataMetaClient,
+ private boolean initializeFromFilesystem(String initializationTime, List<MetadataPartitionType> partitionsToInit,
Option<String> inflightInstantTimestamp) throws IOException {
if (anyPendingDataInstant(dataMetaClient, inflightInstantTimestamp)) {
return false;
}
- String createInstantTime = getInitialCommitInstantTime(dataMetaClient);
-
- initializeMetaClient(DEFAULT_METADATA_POPULATE_META_FIELDS);
- initTableMetadata();
- // if async metadata indexing is enabled,
- // then only initialize files partition as other partitions will be built using HoodieIndexer
- List<MetadataPartitionType> enabledPartitionTypes = new ArrayList<>();
- if (dataWriteConfig.isMetadataAsyncIndex()) {
- enabledPartitionTypes.add(MetadataPartitionType.FILES);
- } else {
- // all enabled ones should be initialized
- enabledPartitionTypes = this.enabledPartitionTypes;
+ // FILES partition is always initialized first
+ ValidationUtils.checkArgument(!partitionsToInit.contains(MetadataPartitionType.FILES)
+ || partitionsToInit.get(0).equals(MetadataPartitionType.FILES), "FILES partition should be initialized first: " + partitionsToInit);
+
+ metadataMetaClient = initializeMetaClient();
+
+ // Get a complete list of files and partitions from the file system or from already initialized FILES partition of MDT
+ boolean filesPartitionAvailable = dataMetaClient.getTableConfig().isMetadataPartitionEnabled(MetadataPartitionType.FILES);
+ List<DirectoryInfo> partitionInfoList = filesPartitionAvailable ? listAllPartitionsFromMDT(initializationTime) : listAllPartitionsFromFilesystem(initializationTime);
+ Map<String, Map<String, Long>> partitionToFilesMap = partitionInfoList.stream()
+ .map(p -> {
+ String partitionName = HoodieTableMetadataUtil.getPartitionIdentifier(p.getRelativePath());
+ return Pair.of(partitionName, p.getFileNameToSizeMap());
+ })
+ .collect(Collectors.toMap(Pair::getKey, Pair::getValue));
+
+ for (MetadataPartitionType partitionType : partitionsToInit) {
+ // Find the commit timestamp to use for this partition. Each initialization should use its own unique commit time.
+ String commitTimeForPartition = generateUniqueCommitInstantTime(initializationTime);
+
+ LOG.info("Initializing MDT partition " + partitionType + " at instant " + commitTimeForPartition);
+
+ Pair<Integer, HoodieData<HoodieRecord>> fileGroupCountAndRecordsPair;
+ switch (partitionType) {
+ case FILES:
+ fileGroupCountAndRecordsPair = initializeFilesPartition(initializationTime, partitionInfoList);
+ break;
+ case BLOOM_FILTERS:
+ fileGroupCountAndRecordsPair = initializeBloomFiltersPartition(initializationTime, partitionToFilesMap);
+ break;
+ case COLUMN_STATS:
+ fileGroupCountAndRecordsPair = initializeColumnStatsPartition(partitionToFilesMap);
+ break;
+ default:
+ throw new HoodieMetadataException("Unsupported MDT partition type: " + partitionType);
+ }
+
+ // Generate the file groups
+ final int fileGroupCount = fileGroupCountAndRecordsPair.getKey();
+ ValidationUtils.checkArgument(fileGroupCount > 0, "FileGroup count for MDT partition " + partitionType + " should be > 0");
+ initializeFileGroups(dataMetaClient, partitionType, commitTimeForPartition, fileGroupCount);
+
+ // Perform the commit using bulkCommit
+ HoodieData<HoodieRecord> records = fileGroupCountAndRecordsPair.getValue();
+ bulkCommit(commitTimeForPartition, partitionType, records, fileGroupCount);
+ metadataMetaClient.reloadActiveTimeline();
+ dataMetaClient = HoodieTableMetadataUtil.setMetadataPartitionState(dataMetaClient, partitionType, true);
}
- initializeEnabledFileGroups(dataMetaClient, createInstantTime, enabledPartitionTypes);
- initialCommit(createInstantTime, enabledPartitionTypes);
- updateInitializedPartitionsInTableConfig(enabledPartitionTypes);
+
return true;
}
- private String getInitialCommitInstantTime(HoodieTableMetaClient dataMetaClient) {
- // If there is no commit on the dataset yet, use the SOLO_COMMIT_TIMESTAMP as the instant time for initial commit
- // Otherwise, we use the timestamp of the latest completed action.
- String createInstantTime = dataMetaClient.getActiveTimeline().filterCompletedInstants()
- .getReverseOrderedInstants().findFirst().map(HoodieInstant::getTimestamp).orElse(SOLO_COMMIT_TIMESTAMP);
- LOG.info("Creating a new metadata table in " + metadataWriteConfig.getBasePath() + " at instant " + createInstantTime);
- return createInstantTime;
+ /**
+ * Returns a unique timestamp to use for initializing a MDT partition.
+ * <p>
+ * Since commits are immutable, we should use unique timestamps to initialize each partition. For this, we will add a suffix to the given initializationTime
+ * until we find a unique timestamp.
+ *
+ * @param initializationTime Timestamp from dataset to use for initialization
+ * @return a unique timestamp for MDT
+ */
+ private String generateUniqueCommitInstantTime(String initializationTime) {
+ for (int offset = 0; ; ++offset) {
Review Comment:
Added
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] danny0405 commented on a diff in pull request #8684: [HUDI-6200] Enhancements to the MDT for improving performance of larger indexes.
Posted by "danny0405 (via GitHub)" <gi...@apache.org>.
danny0405 commented on code in PR #8684:
URL: https://github.com/apache/hudi/pull/8684#discussion_r1222772904
##########
hudi-client/hudi-flink-client/src/main/java/org/apache/hudi/metadata/FlinkHoodieBackedTableMetadataWriter.java:
##########
@@ -104,40 +106,19 @@ protected void initRegistry() {
}
@Override
- protected <T extends SpecificRecordBase> void initialize(HoodieEngineContext engineContext,
- Option<T> actionMetadata,
- Option<String> inflightInstantTimestamp) {
- try {
- if (enabled) {
- initializeIfNeeded(dataMetaClient, actionMetadata, inflightInstantTimestamp);
- }
- } catch (IOException e) {
- LOG.error("Failed to initialize metadata table. Disabling the writer.", e);
- enabled = false;
- }
- }
-
- @Override
- protected void commit(String instantTime, Map<MetadataPartitionType, HoodieData<HoodieRecord>> partitionRecordsMap,
- boolean canTriggerTableService) {
- ValidationUtils.checkState(enabled, "Metadata table cannot be committed to as it is not enabled");
+ protected void commit(String instantTime, Map<MetadataPartitionType, HoodieData<HoodieRecord>> partitionRecordsMap) {
ValidationUtils.checkState(metadataMetaClient != null, "Metadata table is not fully initialized yet.");
HoodieData<HoodieRecord> preppedRecords = prepRecords(partitionRecordsMap);
List<HoodieRecord> preppedRecordList = preppedRecords.collectAsList();
- try (HoodieFlinkWriteClient writeClient = new HoodieFlinkWriteClient(engineContext, metadataWriteConfig)) {
+ // Flink engine does not optimize initialCommit to MDT as bulk insert is not yet supported
+
+ try (HoodieFlinkWriteClient writeClient = (HoodieFlinkWriteClient) getWriteClient()) {
// rollback partially failed writes if any.
if (writeClient.rollbackFailedWrites()) {
metadataMetaClient = HoodieTableMetaClient.reload(metadataMetaClient);
}
- if (canTriggerTableService) {
- // trigger compaction before doing the delta commit. this is to ensure, if this delta commit succeeds in metadata table, but failed in data table,
- // we would have compacted metadata table and so could have included uncommitted data which will never be ignored while reading from metadata
- // table (since reader will filter out only from delta commits)
- compactIfNecessary(writeClient, instantTime);
- }
Review Comment:
You are right, we should trigger the metadata table compaction on each commit.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] danny0405 commented on a diff in pull request #8684: [HUDI-6200] Enhancements to the MDT for improving performance of larger indexes.
Posted by "danny0405 (via GitHub)" <gi...@apache.org>.
danny0405 commented on code in PR #8684:
URL: https://github.com/apache/hudi/pull/8684#discussion_r1220774838
##########
hudi-common/src/main/java/org/apache/hudi/metadata/HoodieTableMetadataUtil.java:
##########
@@ -1453,7 +1453,11 @@ public static String deleteMetadataTablePartition(HoodieTableMetaClient dataMeta
* @return The fileID
*/
public static String getFileIDForFileGroup(MetadataPartitionType partitionType, int index) {
- return String.format("%s%04d", partitionType.getFileIdPrefix(), index);
+ if (partitionType == MetadataPartitionType.FILES) {
+ return String.format("%s%04d-%d", partitionType.getFileIdPrefix(), index, 0);
+ } else {
Review Comment:
What is the purpos of extra suffix `-0` for `FILES` file group?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #8684: [HUDI-6200] Enhancements to the MDT for improving performance of larger indexes.
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #8684:
URL: https://github.com/apache/hudi/pull/8684#issuecomment-1581743087
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "35148aeb4ba78eb6f3316c75f8a0a7e4c6d6df87",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17007",
"triggerID" : "35148aeb4ba78eb6f3316c75f8a0a7e4c6d6df87",
"triggerType" : "PUSH"
}, {
"hash" : "e2785f4675ddf74582ff34590608a5d71c5e9a2d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17008",
"triggerID" : "e2785f4675ddf74582ff34590608a5d71c5e9a2d",
"triggerType" : "PUSH"
}, {
"hash" : "364b1fe0dde303750ca0f8eebe5cd530c2a66a3a",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17042",
"triggerID" : "364b1fe0dde303750ca0f8eebe5cd530c2a66a3a",
"triggerType" : "PUSH"
}, {
"hash" : "9d73b9664eed48ada68721a92526640cfa06a021",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17098",
"triggerID" : "9d73b9664eed48ada68721a92526640cfa06a021",
"triggerType" : "PUSH"
}, {
"hash" : "7be538f4045a42ba33ab8fe62594178e2db75bbb",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17100",
"triggerID" : "7be538f4045a42ba33ab8fe62594178e2db75bbb",
"triggerType" : "PUSH"
}, {
"hash" : "cc0da2372d50d99c98c2ce4bcbe5a60303bde938",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17195",
"triggerID" : "cc0da2372d50d99c98c2ce4bcbe5a60303bde938",
"triggerType" : "PUSH"
}, {
"hash" : "da0e00e3655e06b064d11e20500317e2a1d2229c",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "da0e00e3655e06b064d11e20500317e2a1d2229c",
"triggerType" : "PUSH"
}, {
"hash" : "589c8cd7ffc5df7984c099d6b30fa117463db2b9",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17335",
"triggerID" : "589c8cd7ffc5df7984c099d6b30fa117463db2b9",
"triggerType" : "PUSH"
}, {
"hash" : "226c96ce48b33d7b1650d00e93bc5ea42ad276f6",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17624",
"triggerID" : "226c96ce48b33d7b1650d00e93bc5ea42ad276f6",
"triggerType" : "PUSH"
}, {
"hash" : "ad81a08799a98cd6ee63b8bf792d42f16a98f775",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17634",
"triggerID" : "ad81a08799a98cd6ee63b8bf792d42f16a98f775",
"triggerType" : "PUSH"
}, {
"hash" : "fd2ec462e554190843ddcc8d946317ae76cb81e1",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "fd2ec462e554190843ddcc8d946317ae76cb81e1",
"triggerType" : "PUSH"
}, {
"hash" : "cae57c2b03a3131cfa7774fa4335f4d153f85d61",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17644",
"triggerID" : "cae57c2b03a3131cfa7774fa4335f4d153f85d61",
"triggerType" : "PUSH"
}, {
"hash" : "a21fb8753ed206fefcf598c9e134c280e90d3dba",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17654",
"triggerID" : "a21fb8753ed206fefcf598c9e134c280e90d3dba",
"triggerType" : "PUSH"
}, {
"hash" : "4dfaefde334b712b3ba272bb330242ddb7f4c0ac",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17655",
"triggerID" : "4dfaefde334b712b3ba272bb330242ddb7f4c0ac",
"triggerType" : "PUSH"
}, {
"hash" : "ca59a379b830f88fd72c175cb9b96a14b33550c0",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17660",
"triggerID" : "ca59a379b830f88fd72c175cb9b96a14b33550c0",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* da0e00e3655e06b064d11e20500317e2a1d2229c UNKNOWN
* fd2ec462e554190843ddcc8d946317ae76cb81e1 UNKNOWN
* ca59a379b830f88fd72c175cb9b96a14b33550c0 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17660)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #8684: [HUDI-6200] Enhancements to the MDT for improving performance of larger indexes.
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #8684:
URL: https://github.com/apache/hudi/pull/8684#issuecomment-1546124497
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "35148aeb4ba78eb6f3316c75f8a0a7e4c6d6df87",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17007",
"triggerID" : "35148aeb4ba78eb6f3316c75f8a0a7e4c6d6df87",
"triggerType" : "PUSH"
}, {
"hash" : "e2785f4675ddf74582ff34590608a5d71c5e9a2d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17008",
"triggerID" : "e2785f4675ddf74582ff34590608a5d71c5e9a2d",
"triggerType" : "PUSH"
}, {
"hash" : "364b1fe0dde303750ca0f8eebe5cd530c2a66a3a",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17042",
"triggerID" : "364b1fe0dde303750ca0f8eebe5cd530c2a66a3a",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 364b1fe0dde303750ca0f8eebe5cd530c2a66a3a Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17042)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] danny0405 commented on a diff in pull request #8684: [HUDI-6200] Enhancements to the MDT for improving performance of larger indexes.
Posted by "danny0405 (via GitHub)" <gi...@apache.org>.
danny0405 commented on code in PR #8684:
URL: https://github.com/apache/hudi/pull/8684#discussion_r1192914114
##########
hudi-client/hudi-client-common/src/main/java/org/apache/hudi/metadata/HoodieBackedTableMetadataWriter.java:
##########
@@ -824,25 +863,22 @@ private interface ConvertMetadataFunction {
/**
* Processes commit metadata from data table and commits to metadata table.
*
- * @param instantTime instant time of interest.
+ * @param instantTime instant time of interest.
* @param convertMetadataFunction converter function to convert the respective metadata to List of HoodieRecords to be written to metadata table.
- * @param <T> type of commit metadata.
- * @param canTriggerTableService true if table services can be triggered. false otherwise.
*/
- private <T> void processAndCommit(String instantTime, ConvertMetadataFunction convertMetadataFunction, boolean canTriggerTableService) {
- if (!dataWriteConfig.isMetadataTableEnabled()) {
- return;
- }
+ private void processAndCommit(String instantTime, ConvertMetadataFunction convertMetadataFunction) {
+ ValidationUtils.checkArgument(dataWriteConfig.isMetadataTableEnabled());
+
Set<String> partitionsToUpdate = getMetadataPartitionsToUpdate();
Set<String> inflightIndexes = getInflightMetadataPartitions(dataMetaClient.getTableConfig());
// if indexing is inflight then do not trigger table service
boolean doNotTriggerTableService = partitionsToUpdate.stream().anyMatch(inflightIndexes::contains);
Review Comment:
I mean `doNotTriggerTableService`, it seems not used in anywhere.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] prashantwason commented on a diff in pull request #8684: [HUDI-6200] Enhancements to the MDT for improving performance of larger indexes.
Posted by "prashantwason (via GitHub)" <gi...@apache.org>.
prashantwason commented on code in PR #8684:
URL: https://github.com/apache/hudi/pull/8684#discussion_r1203525353
##########
hudi-client/hudi-client-common/src/main/java/org/apache/hudi/metadata/HoodieBackedTableMetadataWriter.java:
##########
@@ -375,105 +357,91 @@ public List<MetadataPartitionType> getEnabledPartitionTypes() {
return this.enabledPartitionTypes;
}
- /**
- * Initialize the metadata table if it does not exist.
- * <p>
- * If the metadata table does not exist, then file and partition listing is used to initialize the table.
- *
- * @param engineContext
- * @param actionMetadata Action metadata types extending Avro generated SpecificRecordBase
- * @param inflightInstantTimestamp Timestamp of an instant in progress on the dataset. This instant is ignored
- * while deciding to initialize the metadata table.
- */
- protected abstract <T extends SpecificRecordBase> void initialize(HoodieEngineContext engineContext,
- Option<T> actionMetadata,
- Option<String> inflightInstantTimestamp);
-
- public void initTableMetadata() {
- try {
- if (this.metadata != null) {
- this.metadata.close();
- }
- this.metadata = new HoodieBackedTableMetadata(engineContext, dataWriteConfig.getMetadataConfig(),
- dataWriteConfig.getBasePath(), dataWriteConfig.getSpillableMapBasePath());
- this.metadataMetaClient = metadata.getMetadataMetaClient();
- } catch (Exception e) {
- throw new HoodieException("Error initializing metadata table for reads", e);
- }
- }
-
/**
* Initialize the metadata table if needed.
*
* @param dataMetaClient - meta client for the data table
* @param actionMetadata - optional action metadata
* @param inflightInstantTimestamp - timestamp of an instant in progress on the dataset
* @param <T> - action metadata types extending Avro generated SpecificRecordBase
- * @throws IOException
+ * @throws IOException on errors
*/
- protected <T extends SpecificRecordBase> void initializeIfNeeded(HoodieTableMetaClient dataMetaClient,
- Option<T> actionMetadata,
- Option<String> inflightInstantTimestamp) throws IOException {
+ protected <T extends SpecificRecordBase> boolean initializeIfNeeded(HoodieTableMetaClient dataMetaClient,
+ Option<T> actionMetadata,
+ Option<String> inflightInstantTimestamp) throws IOException {
HoodieTimer timer = HoodieTimer.start();
+ List<MetadataPartitionType> partitionsToInit = new ArrayList<>(MetadataPartitionType.values().length);
- boolean exists = metadataTableExists(dataMetaClient, actionMetadata);
+ try {
+ boolean exists = metadataTableExists(dataMetaClient, actionMetadata);
+ if (!exists) {
+ // FILES partition is always required
+ partitionsToInit.add(MetadataPartitionType.FILES);
+ }
- if (!exists) {
- // Initialize for the first time by listing partitions and files directly from the file system
- if (initializeFromFilesystem(dataMetaClient, inflightInstantTimestamp)) {
- metrics.ifPresent(m -> m.updateMetrics(HoodieMetadataMetrics.INITIALIZE_STR, timer.endTimer()));
+ // check if any of the enabled partition types needs to be initialized
+ // NOTE: It needs to be guarded by async index config because if that is enabled then initialization happens through the index scheduler.
+ if (!dataWriteConfig.isMetadataAsyncIndex()) {
+ Set<String> inflightAndCompletedPartitions = getInflightAndCompletedMetadataPartitions(dataMetaClient.getTableConfig());
+ LOG.info("Async metadata indexing disabled and following partitions already initialized: " + inflightAndCompletedPartitions);
+ this.enabledPartitionTypes.stream()
+ .filter(p -> !inflightAndCompletedPartitions.contains(p.getPartitionPath()) && !MetadataPartitionType.FILES.equals(p))
+ .forEach(partitionsToInit::add);
}
- return;
- }
- // if metadata table exists, then check if any of the enabled partition types needs to be initialized
- // NOTE: It needs to be guarded by async index config because if that is enabled then initialization happens through the index scheduler.
- if (!dataWriteConfig.isMetadataAsyncIndex()) {
- Set<String> inflightAndCompletedPartitions = getInflightAndCompletedMetadataPartitions(dataMetaClient.getTableConfig());
- LOG.info("Async metadata indexing enabled and following partitions already initialized: " + inflightAndCompletedPartitions);
- List<MetadataPartitionType> partitionsToInit = this.enabledPartitionTypes.stream()
- .filter(p -> !inflightAndCompletedPartitions.contains(p.getPartitionPath()) && !MetadataPartitionType.FILES.equals(p))
- .collect(Collectors.toList());
- // if there are no partitions to initialize or there is a pending operation, then don't initialize in this round
- if (partitionsToInit.isEmpty() || anyPendingDataInstant(dataMetaClient, inflightInstantTimestamp)) {
- return;
+ if (partitionsToInit.isEmpty()) {
+ // No partitions to initialize
+ return true;
+ }
+
+ // If there is no commit on the dataset yet, use the SOLO_COMMIT_TIMESTAMP as the instant time for initial commit
+ // Otherwise, we use the timestamp of the latest completed action.
+ String initializationTime = dataMetaClient.getActiveTimeline().filterCompletedInstants().lastInstant().map(HoodieInstant::getTimestamp).orElse(SOLO_COMMIT_TIMESTAMP);
+
+ // Initialize partitions for the first time using data from the files on the file system
+ if (!initializeFromFilesystem(initializationTime, partitionsToInit, inflightInstantTimestamp)) {
+ LOG.error("Failed to initialize MDT from filesystem");
+ return false;
}
- String createInstantTime = getInitialCommitInstantTime(dataMetaClient);
- initTableMetadata(); // re-init certain flags in BaseTableMetadata
- initializeEnabledFileGroups(dataMetaClient, createInstantTime, partitionsToInit);
- initialCommit(createInstantTime, partitionsToInit);
- updateInitializedPartitionsInTableConfig(partitionsToInit);
+ metrics.ifPresent(m -> m.updateMetrics(HoodieMetadataMetrics.INITIALIZE_STR, timer.endTimer()));
+ return true;
+ } catch (IOException e) {
+ LOG.error("Failed to initialize metadata table. Disabling the writer.", e);
+ return false;
}
}
private <T extends SpecificRecordBase> boolean metadataTableExists(HoodieTableMetaClient dataMetaClient,
Option<T> actionMetadata) throws IOException {
- boolean exists = dataMetaClient.getFs().exists(new Path(metadataWriteConfig.getBasePath(),
- HoodieTableMetaClient.METAFOLDER_NAME));
+ boolean exists = dataMetaClient.getTableConfig().isMetadataTableEnabled();
boolean reInitialize = false;
// If the un-synced instants have been archived, then
// the metadata table will need to be initialized again.
if (exists) {
- HoodieTableMetaClient metadataMetaClient = HoodieTableMetaClient.builder().setConf(hadoopConf.get())
- .setBasePath(metadataWriteConfig.getBasePath()).build();
+ try {
+ metadataMetaClient = HoodieTableMetaClient.builder().setConf(hadoopConf.get()).setBasePath(metadataWriteConfig.getBasePath()).build();
+ } catch (TableNotFoundException e) {
+ // Table not found, initialize the metadata table.
+ metadataMetaClient = initializeMetaClient();
+ }
if (DEFAULT_METADATA_POPULATE_META_FIELDS != metadataMetaClient.getTableConfig().populateMetaFields()) {
LOG.info("Re-initiating metadata table properties since populate meta fields have changed");
- metadataMetaClient = initializeMetaClient(DEFAULT_METADATA_POPULATE_META_FIELDS);
+ metadataMetaClient = initializeMetaClient();
Review Comment:
Good catch. Moved this block to within the try-catch
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] prashantwason commented on a diff in pull request #8684: [HUDI-6200] Enhancements to the MDT for improving performance of larger indexes.
Posted by "prashantwason (via GitHub)" <gi...@apache.org>.
prashantwason commented on code in PR #8684:
URL: https://github.com/apache/hudi/pull/8684#discussion_r1203521583
##########
hudi-client/hudi-client-common/src/main/java/org/apache/hudi/metadata/HoodieBackedTableMetadataWriter.java:
##########
@@ -489,7 +457,7 @@ private <T extends SpecificRecordBase> boolean metadataTableExists(HoodieTableMe
* TODO: Revisit this logic and validate that filtering for all
* commits timeline is the right thing to do
*
- * @return True if the initialize is not needed, False otherwise
+ * @return True if the initialization is not needed, False otherwise
Review Comment:
Done
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] prashantwason commented on a diff in pull request #8684: [HUDI-6200] Enhancements to the MDT for improving performance of larger indexes.
Posted by "prashantwason (via GitHub)" <gi...@apache.org>.
prashantwason commented on code in PR #8684:
URL: https://github.com/apache/hudi/pull/8684#discussion_r1203530662
##########
hudi-client/hudi-client-common/src/main/java/org/apache/hudi/metadata/HoodieBackedTableMetadataWriter.java:
##########
@@ -760,15 +815,14 @@ private void initializeFileGroups(HoodieTableMetaClient dataMetaClient, Metadata
HoodieLogFormat.Writer writer = HoodieLogFormat.newWriterBuilder()
.onParentPath(FSUtils.getPartitionPath(metadataWriteConfig.getBasePath(), metadataPartition.getPartitionPath()))
- .withFileId(fileGroupFileId)
- .overBaseCommit(instantTime)
+ .withFileId(fileGroupFileId).overBaseCommit(instantTime)
.withLogVersion(HoodieLogFile.LOGFILE_BASE_VERSION)
.withFileSize(0L)
- .withSizeThreshold(metadataWriteConfig.getLogFileMaxSize())
- .withFs(dataMetaClient.getFs())
- .withRolloverLogWriteToken(HoodieLogFormat.DEFAULT_WRITE_TOKEN)
- .withLogWriteToken(HoodieLogFormat.DEFAULT_WRITE_TOKEN)
- .withFileExtension(HoodieLogFile.DELTA_EXTENSION).build();
+ .withSizeThreshold(metadataWriteConfig.getLogFileMaxSize())
+ .withFs(dataMetaClient.getFs())
+ .withRolloverLogWriteToken(HoodieLogFormat.DEFAULT_WRITE_TOKEN)
Review Comment:
Fixed
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] danny0405 commented on a diff in pull request #8684: [HUDI-6200] Enhancements to the MDT for improving performance of larger indexes.
Posted by "danny0405 (via GitHub)" <gi...@apache.org>.
danny0405 commented on code in PR #8684:
URL: https://github.com/apache/hudi/pull/8684#discussion_r1203914496
##########
hudi-client/hudi-spark-client/src/main/java/org/apache/hudi/client/SparkRDDWriteClient.java:
##########
@@ -338,14 +338,20 @@ protected void initMetadataTable(Option<String> instantTime) {
* @param inFlightInstantTimestamp - The in-flight action responsible for the metadata table initialization
*/
private void initializeMetadataTable(Option<String> inFlightInstantTimestamp) {
- if (config.isMetadataTableEnabled()) {
- HoodieTableMetadataWriter writer = SparkHoodieBackedTableMetadataWriter.create(context.getHadoopConf().get(), config,
- context, Option.empty(), inFlightInstantTimestamp);
- try {
- writer.close();
- } catch (Exception e) {
- throw new HoodieException("Failed to instantiate Metadata table ", e);
+ if (!config.isMetadataTableEnabled()) {
+ LOG.error("================================111111111111111111");
+ return;
+ }
+
+ try (HoodieTableMetadataWriter writer = SparkHoodieBackedTableMetadataWriter.create(context.getHadoopConf().get(), config,
+ context, Option.empty(), inFlightInstantTimestamp)) {
+ if (writer.isInitialized()) {
+ writer.performTableServices(inFlightInstantTimestamp);
+ } else {
+ throw new HoodieException((".................2222222222"));
Review Comment:
Ditto: a meaningless exception msg.
##########
hudi-client/hudi-client-common/src/main/java/org/apache/hudi/metadata/HoodieBackedTableMetadataWriter.java:
##########
@@ -626,8 +657,7 @@ private void initializeFileGroups(HoodieTableMetaClient dataMetaClient, Metadata
HoodieLogFormat.Writer writer = HoodieLogFormat.newWriterBuilder()
.onParentPath(FSUtils.getPartitionPath(metadataWriteConfig.getBasePath(), metadataPartition.getPartitionPath()))
- .withFileId(fileGroupFileId)
- .overBaseCommit(instantTime)
+ .withFileId(fileGroupFileId).overBaseCommit(instantTime)
.withLogVersion(HoodieLogFile.LOGFILE_BASE_VERSION)
Review Comment:
Unnecessary change.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] nsivabalan commented on a diff in pull request #8684: [HUDI-6200] Enhancements to the MDT for improving performance of larger indexes.
Posted by "nsivabalan (via GitHub)" <gi...@apache.org>.
nsivabalan commented on code in PR #8684:
URL: https://github.com/apache/hudi/pull/8684#discussion_r1192729597
##########
hudi-client/hudi-spark-client/src/main/java/org/apache/hudi/metadata/SparkHoodieMetadataBulkInsertPartitioner.java:
##########
@@ -0,0 +1,111 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hudi.metadata;
+
+import java.io.Serializable;
+import java.util.ArrayList;
+import java.util.Comparator;
+import java.util.List;
+
+import org.apache.hudi.common.model.HoodieRecord;
+import org.apache.hudi.common.util.ValidationUtils;
+import org.apache.hudi.table.BulkInsertPartitioner;
+import org.apache.spark.Partitioner;
+import org.apache.spark.api.java.JavaRDD;
+
+import scala.Tuple2;
+
+/**
+ * A {@code BulkInsertPartitioner} implementation for Metadata Table to improve performance of initialization of metadata
+ * table partition when a very large number of records are inserted.
+ *
+ * This partitioner requires the records to be already tagged with location.
+ */
+public class SparkHoodieMetadataBulkInsertPartitioner implements BulkInsertPartitioner<JavaRDD<HoodieRecord>> {
Review Comment:
do we have tests for this class
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] danny0405 commented on a diff in pull request #8684: [HUDI-6200] Enhancements to the MDT for improving performance of larger indexes.
Posted by "danny0405 (via GitHub)" <gi...@apache.org>.
danny0405 commented on code in PR #8684:
URL: https://github.com/apache/hudi/pull/8684#discussion_r1194731645
##########
hudi-common/src/main/java/org/apache/hudi/common/table/HoodieTableConfig.java:
##########
@@ -694,17 +695,75 @@ private Long getTableChecksum() {
return getLong(TABLE_CHECKSUM);
}
- public List<String> getMetadataPartitionsInflight() {
- return StringUtils.split(
- getStringOrDefault(TABLE_METADATA_PARTITIONS_INFLIGHT, StringUtils.EMPTY_STRING),
- CONFIG_VALUES_DELIMITER
- );
+ public Set<String> getMetadataPartitionsInflight() {
+ return new HashSet<>(StringUtils.split(
+ getStringOrDefault(TABLE_METADATA_PARTITIONS_INFLIGHT, StringUtils.EMPTY_STRING),
+ CONFIG_VALUES_DELIMITER));
}
public Set<String> getMetadataPartitions() {
return new HashSet<>(
- StringUtils.split(getStringOrDefault(TABLE_METADATA_PARTITIONS, StringUtils.EMPTY_STRING),
- CONFIG_VALUES_DELIMITER));
+ StringUtils.split(getStringOrDefault(TABLE_METADATA_PARTITIONS, StringUtils.EMPTY_STRING),
+ CONFIG_VALUES_DELIMITER));
+ }
+
+ /**
+ * @returns true if metadata table has been created and is being used for this dataset, else returns false.
+ */
+ public boolean isMetadataTableEnabled() {
+ return isMetadataPartitionEnabled(MetadataPartitionType.FILES);
+ }
+
+ /**
+ * Checks if metadata table is enabled and the specified partition has been initialized.
+ *
+ * @param partition The partition to check
+ * @returns true if the specific partition has been initialized, else returns false.
+ */
+ public boolean isMetadataPartitionEnabled(MetadataPartitionType partition) {
+ return getMetadataPartitions().contains(partition.getPartitionPath());
+ }
+
+ /**
+ * Enables or disables the specified metadata table partition.
+ *
+ * @param partition The partition
+ * @param enabled If true, the partition is enabled, else disabled
+ */
+ public void setMetadataPartitionState(MetadataPartitionType partition, boolean enabled) {
+ ValidationUtils.checkArgument(!partition.getPartitionPath().contains(CONFIG_VALUES_DELIMITER),
+ "Metadata Table partition path cannot contain a comma: " + partition.getPartitionPath());
+ Set<String> partitions = getMetadataPartitions();
+ Set<String> partitionsInflight = getMetadataPartitionsInflight();
+ if (enabled) {
+ partitions.add(partition.getPartitionPath());
+ partitionsInflight.remove(partition.getPartitionPath());
+ } else if (partition.equals(MetadataPartitionType.FILES)) {
+ // file listing partition is required for all other partitions to work
+ // Disabling file partition will also disable all partitions
+ partitions.clear();
+ partitionsInflight.clear();
+ } else {
+ partitions.remove(partition.getPartitionPath());
+ partitionsInflight.remove(partition.getPartitionPath());
+ }
+ setValue(TABLE_METADATA_PARTITIONS, partitions.stream().sorted().collect(Collectors.joining(CONFIG_VALUES_DELIMITER)));
+ setValue(TABLE_METADATA_PARTITIONS_INFLIGHT, partitionsInflight.stream().sorted().collect(Collectors.joining(CONFIG_VALUES_DELIMITER)));
Review Comment:
Do we need to persist these options?
##########
hudi-client/hudi-spark-client/src/main/java/org/apache/hudi/client/SparkRDDWriteClient.java:
##########
@@ -337,15 +337,33 @@ protected void initMetadataTable(Option<String> instantTime) {
*
* @param inFlightInstantTimestamp - The in-flight action responsible for the metadata table initialization
*/
- private void initializeMetadataTable(Option<String> inFlightInstantTimestamp) {
- if (config.isMetadataTableEnabled()) {
- HoodieTableMetadataWriter writer = SparkHoodieBackedTableMetadataWriter.create(context.getHadoopConf().get(), config,
- context, Option.empty(), inFlightInstantTimestamp);
- try {
- writer.close();
- } catch (Exception e) {
- throw new HoodieException("Failed to instantiate Metadata table ", e);
+ private void initializeMetadataTable(WriteOperationType operationType, Option<String> inFlightInstantTimestamp) {
+ if (!config.isMetadataTableEnabled()) {
+ return;
+ }
+
+ try (HoodieTableMetadataWriter writer = SparkHoodieBackedTableMetadataWriter.create(context.getHadoopConf().get(), config,
+ context, Option.empty(), inFlightInstantTimestamp)) {
+ if (writer.isInitialized()) {
+ // Optimize the metadata table which involves compacton. cleaning, etc. This should only be called from writers.
+ switch (operationType) {
+ case INSERT:
+ case INSERT_PREPPED:
+ case UPSERT:
+ case UPSERT_PREPPED:
+ case BULK_INSERT:
+ case BULK_INSERT_PREPPED:
+ case DELETE:
Review Comment:
Enum the write operation is really hard to maintain, can we triggers the table sercive whatever the operation is ?
##########
hudi-client/hudi-client-common/src/main/java/org/apache/hudi/metadata/HoodieBackedTableMetadataWriter.java:
##########
@@ -760,15 +815,14 @@ private void initializeFileGroups(HoodieTableMetaClient dataMetaClient, Metadata
HoodieLogFormat.Writer writer = HoodieLogFormat.newWriterBuilder()
.onParentPath(FSUtils.getPartitionPath(metadataWriteConfig.getBasePath(), metadataPartition.getPartitionPath()))
- .withFileId(fileGroupFileId)
- .overBaseCommit(instantTime)
+ .withFileId(fileGroupFileId).overBaseCommit(instantTime)
.withLogVersion(HoodieLogFile.LOGFILE_BASE_VERSION)
.withFileSize(0L)
- .withSizeThreshold(metadataWriteConfig.getLogFileMaxSize())
- .withFs(dataMetaClient.getFs())
- .withRolloverLogWriteToken(HoodieLogFormat.DEFAULT_WRITE_TOKEN)
- .withLogWriteToken(HoodieLogFormat.DEFAULT_WRITE_TOKEN)
- .withFileExtension(HoodieLogFile.DELTA_EXTENSION).build();
+ .withSizeThreshold(metadataWriteConfig.getLogFileMaxSize())
+ .withFs(dataMetaClient.getFs())
+ .withRolloverLogWriteToken(HoodieLogFormat.DEFAULT_WRITE_TOKEN)
Review Comment:
Fix the indentation.
##########
hudi-client/hudi-client-common/src/main/java/org/apache/hudi/metadata/HoodieBackedTableMetadataWriter.java:
##########
@@ -375,105 +357,91 @@ public List<MetadataPartitionType> getEnabledPartitionTypes() {
return this.enabledPartitionTypes;
}
- /**
- * Initialize the metadata table if it does not exist.
- * <p>
- * If the metadata table does not exist, then file and partition listing is used to initialize the table.
- *
- * @param engineContext
- * @param actionMetadata Action metadata types extending Avro generated SpecificRecordBase
- * @param inflightInstantTimestamp Timestamp of an instant in progress on the dataset. This instant is ignored
- * while deciding to initialize the metadata table.
- */
- protected abstract <T extends SpecificRecordBase> void initialize(HoodieEngineContext engineContext,
- Option<T> actionMetadata,
- Option<String> inflightInstantTimestamp);
-
- public void initTableMetadata() {
- try {
- if (this.metadata != null) {
- this.metadata.close();
- }
- this.metadata = new HoodieBackedTableMetadata(engineContext, dataWriteConfig.getMetadataConfig(),
- dataWriteConfig.getBasePath(), dataWriteConfig.getSpillableMapBasePath());
- this.metadataMetaClient = metadata.getMetadataMetaClient();
- } catch (Exception e) {
- throw new HoodieException("Error initializing metadata table for reads", e);
- }
- }
-
/**
* Initialize the metadata table if needed.
*
* @param dataMetaClient - meta client for the data table
* @param actionMetadata - optional action metadata
* @param inflightInstantTimestamp - timestamp of an instant in progress on the dataset
* @param <T> - action metadata types extending Avro generated SpecificRecordBase
- * @throws IOException
+ * @throws IOException on errors
*/
- protected <T extends SpecificRecordBase> void initializeIfNeeded(HoodieTableMetaClient dataMetaClient,
- Option<T> actionMetadata,
- Option<String> inflightInstantTimestamp) throws IOException {
+ protected <T extends SpecificRecordBase> boolean initializeIfNeeded(HoodieTableMetaClient dataMetaClient,
+ Option<T> actionMetadata,
+ Option<String> inflightInstantTimestamp) throws IOException {
HoodieTimer timer = HoodieTimer.start();
+ List<MetadataPartitionType> partitionsToInit = new ArrayList<>(MetadataPartitionType.values().length);
- boolean exists = metadataTableExists(dataMetaClient, actionMetadata);
+ try {
+ boolean exists = metadataTableExists(dataMetaClient, actionMetadata);
+ if (!exists) {
+ // FILES partition is always required
+ partitionsToInit.add(MetadataPartitionType.FILES);
+ }
- if (!exists) {
- // Initialize for the first time by listing partitions and files directly from the file system
- if (initializeFromFilesystem(dataMetaClient, inflightInstantTimestamp)) {
- metrics.ifPresent(m -> m.updateMetrics(HoodieMetadataMetrics.INITIALIZE_STR, timer.endTimer()));
+ // check if any of the enabled partition types needs to be initialized
+ // NOTE: It needs to be guarded by async index config because if that is enabled then initialization happens through the index scheduler.
+ if (!dataWriteConfig.isMetadataAsyncIndex()) {
+ Set<String> inflightAndCompletedPartitions = getInflightAndCompletedMetadataPartitions(dataMetaClient.getTableConfig());
+ LOG.info("Async metadata indexing disabled and following partitions already initialized: " + inflightAndCompletedPartitions);
+ this.enabledPartitionTypes.stream()
+ .filter(p -> !inflightAndCompletedPartitions.contains(p.getPartitionPath()) && !MetadataPartitionType.FILES.equals(p))
+ .forEach(partitionsToInit::add);
}
- return;
- }
- // if metadata table exists, then check if any of the enabled partition types needs to be initialized
- // NOTE: It needs to be guarded by async index config because if that is enabled then initialization happens through the index scheduler.
- if (!dataWriteConfig.isMetadataAsyncIndex()) {
- Set<String> inflightAndCompletedPartitions = getInflightAndCompletedMetadataPartitions(dataMetaClient.getTableConfig());
- LOG.info("Async metadata indexing enabled and following partitions already initialized: " + inflightAndCompletedPartitions);
- List<MetadataPartitionType> partitionsToInit = this.enabledPartitionTypes.stream()
- .filter(p -> !inflightAndCompletedPartitions.contains(p.getPartitionPath()) && !MetadataPartitionType.FILES.equals(p))
- .collect(Collectors.toList());
- // if there are no partitions to initialize or there is a pending operation, then don't initialize in this round
- if (partitionsToInit.isEmpty() || anyPendingDataInstant(dataMetaClient, inflightInstantTimestamp)) {
- return;
+ if (partitionsToInit.isEmpty()) {
+ // No partitions to initialize
+ return true;
+ }
+
+ // If there is no commit on the dataset yet, use the SOLO_COMMIT_TIMESTAMP as the instant time for initial commit
+ // Otherwise, we use the timestamp of the latest completed action.
+ String initializationTime = dataMetaClient.getActiveTimeline().filterCompletedInstants().lastInstant().map(HoodieInstant::getTimestamp).orElse(SOLO_COMMIT_TIMESTAMP);
+
+ // Initialize partitions for the first time using data from the files on the file system
+ if (!initializeFromFilesystem(initializationTime, partitionsToInit, inflightInstantTimestamp)) {
+ LOG.error("Failed to initialize MDT from filesystem");
+ return false;
}
- String createInstantTime = getInitialCommitInstantTime(dataMetaClient);
- initTableMetadata(); // re-init certain flags in BaseTableMetadata
- initializeEnabledFileGroups(dataMetaClient, createInstantTime, partitionsToInit);
- initialCommit(createInstantTime, partitionsToInit);
- updateInitializedPartitionsInTableConfig(partitionsToInit);
+ metrics.ifPresent(m -> m.updateMetrics(HoodieMetadataMetrics.INITIALIZE_STR, timer.endTimer()));
+ return true;
+ } catch (IOException e) {
+ LOG.error("Failed to initialize metadata table. Disabling the writer.", e);
+ return false;
}
}
private <T extends SpecificRecordBase> boolean metadataTableExists(HoodieTableMetaClient dataMetaClient,
Option<T> actionMetadata) throws IOException {
- boolean exists = dataMetaClient.getFs().exists(new Path(metadataWriteConfig.getBasePath(),
- HoodieTableMetaClient.METAFOLDER_NAME));
+ boolean exists = dataMetaClient.getTableConfig().isMetadataTableEnabled();
boolean reInitialize = false;
// If the un-synced instants have been archived, then
// the metadata table will need to be initialized again.
if (exists) {
- HoodieTableMetaClient metadataMetaClient = HoodieTableMetaClient.builder().setConf(hadoopConf.get())
- .setBasePath(metadataWriteConfig.getBasePath()).build();
+ try {
+ metadataMetaClient = HoodieTableMetaClient.builder().setConf(hadoopConf.get()).setBasePath(metadataWriteConfig.getBasePath()).build();
+ } catch (TableNotFoundException e) {
+ // Table not found, initialize the metadata table.
+ metadataMetaClient = initializeMetaClient();
+ }
if (DEFAULT_METADATA_POPULATE_META_FIELDS != metadataMetaClient.getTableConfig().populateMetaFields()) {
LOG.info("Re-initiating metadata table properties since populate meta fields have changed");
- metadataMetaClient = initializeMetaClient(DEFAULT_METADATA_POPULATE_META_FIELDS);
+ metadataMetaClient = initializeMetaClient();
Review Comment:
If MDT does not exist and `metadataMetaClient.getTableConfig().populateMetaFields()` is true, the `initializeMetaClient()` could be invoked 2 times, which could incur exception.
##########
hudi-client/hudi-client-common/src/main/java/org/apache/hudi/metadata/HoodieBackedTableMetadataWriter.java:
##########
@@ -1052,51 +1091,81 @@ protected HoodieData<HoodieRecord> prepRecords(Map<MetadataPartitionType,
}
/**
- * Perform a compaction on the Metadata Table.
- *
- * Cases to be handled:
- * 1. We cannot perform compaction if there are previous inflight operations on the dataset. This is because
- * a compacted metadata base file at time Tx should represent all the actions on the dataset till time Tx.
- *
- * 2. In multi-writer scenario, a parallel operation with a greater instantTime may have completed creating a
- * deltacommit.
+ * Optimize the metadata table by running compaction, clean and archive as required.
+ * <p>
+ * Don't perform optimization if there are inflight operations on the dataset. This is for two reasons:
+ * - The compaction will contain the correct data as all failed operations have been rolled back.
+ * - Clean/compaction etc. will have the highest timestamp on the MDT and we won't be adding new operations
+ * with smaller timestamps to metadata table (makes for easier debugging)
+ * <p>
+ * This adds the limitations that long-running async operations (clustering, etc.) may cause delay in such MDT
+ * optimizations. We will relax this after MDT code has been hardened.
*/
- protected void compactIfNecessary(BaseHoodieWriteClient writeClient, String instantTime) {
- // finish off any pending compactions if any from previous attempt.
- writeClient.runAnyPendingCompactions();
-
- String latestDeltaCommitTimeInMetadataTable = metadataMetaClient.reloadActiveTimeline()
- .getDeltaCommitTimeline()
- .filterCompletedInstants()
- .lastInstant().orElseThrow(() -> new HoodieMetadataException("No completed deltacommit in metadata table"))
- .getTimestamp();
- // we need to find if there are any inflights in data table timeline before or equal to the latest delta commit in metadata table.
- // Whenever you want to change this logic, please ensure all below scenarios are considered.
- // a. There could be a chance that latest delta commit in MDT is committed in MDT, but failed in DT. And so findInstantsBeforeOrEquals() should be employed
- // b. There could be DT inflights after latest delta commit in MDT and we are ok with it. bcoz, the contract is, latest compaction instant time in MDT represents
- // any instants before that is already synced with metadata table.
- // c. Do consider out of order commits. For eg, c4 from DT could complete before c3. and we can't trigger compaction in MDT with c4 as base instant time, until every
- // instant before c4 is synced with metadata table.
- List<HoodieInstant> pendingInstants = dataMetaClient.reloadActiveTimeline().filterInflightsAndRequested()
- .findInstantsBeforeOrEquals(latestDeltaCommitTimeInMetadataTable).getInstants();
+ @Override
+ public void performTableServices(Option<String> inFlightInstantTimestamp) {
+ HoodieTimer metadataTableServicesTimer = HoodieTimer.start();
+ boolean allTableServicesExecutedSuccessfullyOrSkipped = true;
+ try {
+ BaseHoodieWriteClient writeClient = getWriteClient();
+ // Run any pending table services operations.
+ runPendingTableServicesOperations(writeClient);
+
+ // Check and run clean operations.
+ String latestDeltacommitTime = metadataMetaClient.reloadActiveTimeline().getDeltaCommitTimeline()
+ .filterCompletedInstants()
+ .lastInstant().get()
+ .getTimestamp();
+ LOG.info("Latest deltacommit time found is " + latestDeltacommitTime + ", running clean operations.");
+ cleanIfNecessary(writeClient, latestDeltacommitTime);
+
+ // Do timeline validation before scheduling compaction/logcompaction operations.
+ if (!validateTimelineBeforeSchedulingCompaction(inFlightInstantTimestamp, latestDeltacommitTime)) {
+ return;
Review Comment:
We should not return directly because the archiving is also blocked, if no compaction plan should be scheduled, the archiving should also be triggered.
##########
hudi-client/hudi-client-common/src/main/java/org/apache/hudi/metadata/HoodieBackedTableMetadataWriter.java:
##########
@@ -564,53 +532,147 @@ private <T extends SpecificRecordBase> boolean isCommitRevertedByInFlightAction(
/**
* Initialize the Metadata Table by listing files and partitions from the file system.
*
- * @param dataMetaClient - {@code HoodieTableMetaClient} for the dataset.
+ * @param initializationTime - Timestamp to use for the commit
+ * @param partitionsToInit - List of MDT partitions to initialize
* @param inflightInstantTimestamp - Current action instant responsible for this initialization
*/
- private boolean initializeFromFilesystem(HoodieTableMetaClient dataMetaClient,
+ private boolean initializeFromFilesystem(String initializationTime, List<MetadataPartitionType> partitionsToInit,
Option<String> inflightInstantTimestamp) throws IOException {
if (anyPendingDataInstant(dataMetaClient, inflightInstantTimestamp)) {
return false;
}
- String createInstantTime = getInitialCommitInstantTime(dataMetaClient);
-
- initializeMetaClient(DEFAULT_METADATA_POPULATE_META_FIELDS);
- initTableMetadata();
- // if async metadata indexing is enabled,
- // then only initialize files partition as other partitions will be built using HoodieIndexer
- List<MetadataPartitionType> enabledPartitionTypes = new ArrayList<>();
- if (dataWriteConfig.isMetadataAsyncIndex()) {
- enabledPartitionTypes.add(MetadataPartitionType.FILES);
- } else {
- // all enabled ones should be initialized
- enabledPartitionTypes = this.enabledPartitionTypes;
+ // FILES partition is always initialized first
+ ValidationUtils.checkArgument(!partitionsToInit.contains(MetadataPartitionType.FILES)
+ || partitionsToInit.get(0).equals(MetadataPartitionType.FILES), "FILES partition should be initialized first: " + partitionsToInit);
+
+ metadataMetaClient = initializeMetaClient();
+
+ // Get a complete list of files and partitions from the file system or from already initialized FILES partition of MDT
+ boolean filesPartitionAvailable = dataMetaClient.getTableConfig().isMetadataPartitionEnabled(MetadataPartitionType.FILES);
+ List<DirectoryInfo> partitionInfoList = filesPartitionAvailable ? listAllPartitionsFromMDT(initializationTime) : listAllPartitionsFromFilesystem(initializationTime);
+ Map<String, Map<String, Long>> partitionToFilesMap = partitionInfoList.stream()
Review Comment:
Can you elaborate why we still load file info from MDT if it is enabled before? Could this cause metadata in-consistency?
##########
hudi-client/hudi-client-common/src/main/java/org/apache/hudi/metadata/HoodieBackedTableMetadataWriter.java:
##########
@@ -489,7 +457,7 @@ private <T extends SpecificRecordBase> boolean metadataTableExists(HoodieTableMe
* TODO: Revisit this logic and validate that filtering for all
* commits timeline is the right thing to do
*
- * @return True if the initialize is not needed, False otherwise
+ * @return True if the initialization is not needed, False otherwise
Review Comment:
Replace all the `latestMetadataInstant.get().getTimestamp()` with `latestMetadataInstantTimestamp` ?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #8684: [HUDI-6200] Enhancements to the MDT for improving performance of larger indexes.
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #8684:
URL: https://github.com/apache/hudi/pull/8684#issuecomment-1580929494
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "35148aeb4ba78eb6f3316c75f8a0a7e4c6d6df87",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17007",
"triggerID" : "35148aeb4ba78eb6f3316c75f8a0a7e4c6d6df87",
"triggerType" : "PUSH"
}, {
"hash" : "e2785f4675ddf74582ff34590608a5d71c5e9a2d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17008",
"triggerID" : "e2785f4675ddf74582ff34590608a5d71c5e9a2d",
"triggerType" : "PUSH"
}, {
"hash" : "364b1fe0dde303750ca0f8eebe5cd530c2a66a3a",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17042",
"triggerID" : "364b1fe0dde303750ca0f8eebe5cd530c2a66a3a",
"triggerType" : "PUSH"
}, {
"hash" : "9d73b9664eed48ada68721a92526640cfa06a021",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17098",
"triggerID" : "9d73b9664eed48ada68721a92526640cfa06a021",
"triggerType" : "PUSH"
}, {
"hash" : "7be538f4045a42ba33ab8fe62594178e2db75bbb",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17100",
"triggerID" : "7be538f4045a42ba33ab8fe62594178e2db75bbb",
"triggerType" : "PUSH"
}, {
"hash" : "cc0da2372d50d99c98c2ce4bcbe5a60303bde938",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17195",
"triggerID" : "cc0da2372d50d99c98c2ce4bcbe5a60303bde938",
"triggerType" : "PUSH"
}, {
"hash" : "da0e00e3655e06b064d11e20500317e2a1d2229c",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "da0e00e3655e06b064d11e20500317e2a1d2229c",
"triggerType" : "PUSH"
}, {
"hash" : "589c8cd7ffc5df7984c099d6b30fa117463db2b9",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17335",
"triggerID" : "589c8cd7ffc5df7984c099d6b30fa117463db2b9",
"triggerType" : "PUSH"
}, {
"hash" : "226c96ce48b33d7b1650d00e93bc5ea42ad276f6",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17624",
"triggerID" : "226c96ce48b33d7b1650d00e93bc5ea42ad276f6",
"triggerType" : "PUSH"
}, {
"hash" : "ad81a08799a98cd6ee63b8bf792d42f16a98f775",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17634",
"triggerID" : "ad81a08799a98cd6ee63b8bf792d42f16a98f775",
"triggerType" : "PUSH"
}, {
"hash" : "fd2ec462e554190843ddcc8d946317ae76cb81e1",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "fd2ec462e554190843ddcc8d946317ae76cb81e1",
"triggerType" : "PUSH"
}, {
"hash" : "cae57c2b03a3131cfa7774fa4335f4d153f85d61",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17644",
"triggerID" : "cae57c2b03a3131cfa7774fa4335f4d153f85d61",
"triggerType" : "PUSH"
}, {
"hash" : "a21fb8753ed206fefcf598c9e134c280e90d3dba",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "a21fb8753ed206fefcf598c9e134c280e90d3dba",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* da0e00e3655e06b064d11e20500317e2a1d2229c UNKNOWN
* fd2ec462e554190843ddcc8d946317ae76cb81e1 UNKNOWN
* cae57c2b03a3131cfa7774fa4335f4d153f85d61 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17644)
* a21fb8753ed206fefcf598c9e134c280e90d3dba UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] nsivabalan commented on a diff in pull request #8684: [HUDI-6200] Enhancements to the MDT for improving performance of larger indexes.
Posted by "nsivabalan (via GitHub)" <gi...@apache.org>.
nsivabalan commented on code in PR #8684:
URL: https://github.com/apache/hudi/pull/8684#discussion_r1223138356
##########
hudi-common/src/main/java/org/apache/hudi/metadata/HoodieTableMetadataUtil.java:
##########
@@ -1453,7 +1453,11 @@ public static String deleteMetadataTablePartition(HoodieTableMetaClient dataMeta
* @return The fileID
*/
public static String getFileIDForFileGroup(MetadataPartitionType partitionType, int index) {
- return String.format("%s%04d", partitionType.getFileIdPrefix(), index);
+ if (partitionType == MetadataPartitionType.FILES) {
+ return String.format("%s%04d-%d", partitionType.getFileIdPrefix(), index, 0);
+ } else {
Review Comment:
yes, we should. guess I did push out a fix, but guess Sagar force pushed it.
We have been dragging to land this patch for sometime. Lets go ahead if we get CI green. We will fix it in the next patch immediately. https://github.com/apache/hudi/pull/8900
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #8684: [HUDI-6200] Enhancements to the MDT for improving performance of larger indexes.
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #8684:
URL: https://github.com/apache/hudi/pull/8684#issuecomment-1582661903
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "35148aeb4ba78eb6f3316c75f8a0a7e4c6d6df87",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17007",
"triggerID" : "35148aeb4ba78eb6f3316c75f8a0a7e4c6d6df87",
"triggerType" : "PUSH"
}, {
"hash" : "e2785f4675ddf74582ff34590608a5d71c5e9a2d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17008",
"triggerID" : "e2785f4675ddf74582ff34590608a5d71c5e9a2d",
"triggerType" : "PUSH"
}, {
"hash" : "364b1fe0dde303750ca0f8eebe5cd530c2a66a3a",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17042",
"triggerID" : "364b1fe0dde303750ca0f8eebe5cd530c2a66a3a",
"triggerType" : "PUSH"
}, {
"hash" : "9d73b9664eed48ada68721a92526640cfa06a021",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17098",
"triggerID" : "9d73b9664eed48ada68721a92526640cfa06a021",
"triggerType" : "PUSH"
}, {
"hash" : "7be538f4045a42ba33ab8fe62594178e2db75bbb",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17100",
"triggerID" : "7be538f4045a42ba33ab8fe62594178e2db75bbb",
"triggerType" : "PUSH"
}, {
"hash" : "cc0da2372d50d99c98c2ce4bcbe5a60303bde938",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17195",
"triggerID" : "cc0da2372d50d99c98c2ce4bcbe5a60303bde938",
"triggerType" : "PUSH"
}, {
"hash" : "da0e00e3655e06b064d11e20500317e2a1d2229c",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "da0e00e3655e06b064d11e20500317e2a1d2229c",
"triggerType" : "PUSH"
}, {
"hash" : "589c8cd7ffc5df7984c099d6b30fa117463db2b9",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17335",
"triggerID" : "589c8cd7ffc5df7984c099d6b30fa117463db2b9",
"triggerType" : "PUSH"
}, {
"hash" : "226c96ce48b33d7b1650d00e93bc5ea42ad276f6",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17624",
"triggerID" : "226c96ce48b33d7b1650d00e93bc5ea42ad276f6",
"triggerType" : "PUSH"
}, {
"hash" : "ad81a08799a98cd6ee63b8bf792d42f16a98f775",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17634",
"triggerID" : "ad81a08799a98cd6ee63b8bf792d42f16a98f775",
"triggerType" : "PUSH"
}, {
"hash" : "fd2ec462e554190843ddcc8d946317ae76cb81e1",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "fd2ec462e554190843ddcc8d946317ae76cb81e1",
"triggerType" : "PUSH"
}, {
"hash" : "cae57c2b03a3131cfa7774fa4335f4d153f85d61",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17644",
"triggerID" : "cae57c2b03a3131cfa7774fa4335f4d153f85d61",
"triggerType" : "PUSH"
}, {
"hash" : "a21fb8753ed206fefcf598c9e134c280e90d3dba",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17654",
"triggerID" : "a21fb8753ed206fefcf598c9e134c280e90d3dba",
"triggerType" : "PUSH"
}, {
"hash" : "4dfaefde334b712b3ba272bb330242ddb7f4c0ac",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17655",
"triggerID" : "4dfaefde334b712b3ba272bb330242ddb7f4c0ac",
"triggerType" : "PUSH"
}, {
"hash" : "ca59a379b830f88fd72c175cb9b96a14b33550c0",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17660",
"triggerID" : "ca59a379b830f88fd72c175cb9b96a14b33550c0",
"triggerType" : "PUSH"
}, {
"hash" : "1d49e658315a2ae22a2c29e3d388e82f869b146d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17664",
"triggerID" : "1d49e658315a2ae22a2c29e3d388e82f869b146d",
"triggerType" : "PUSH"
}, {
"hash" : "df3a5b57a375fd344814b1f5a2451cfb79fea2c3",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17669",
"triggerID" : "df3a5b57a375fd344814b1f5a2451cfb79fea2c3",
"triggerType" : "PUSH"
}, {
"hash" : "da92c6c310fccdb4f1e797b481eaf6bb027e551f",
"status" : "CANCELED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17673",
"triggerID" : "da92c6c310fccdb4f1e797b481eaf6bb027e551f",
"triggerType" : "PUSH"
}, {
"hash" : "86902aa16075d75d61347a93819295d78cda0d2d",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17675",
"triggerID" : "86902aa16075d75d61347a93819295d78cda0d2d",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* da0e00e3655e06b064d11e20500317e2a1d2229c UNKNOWN
* fd2ec462e554190843ddcc8d946317ae76cb81e1 UNKNOWN
* da92c6c310fccdb4f1e797b481eaf6bb027e551f Azure: [CANCELED](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17673)
* 86902aa16075d75d61347a93819295d78cda0d2d Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17675)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] prashantwason commented on a diff in pull request #8684: [HUDI-6200] Enhancements to the MDT for improving performance of larger indexes.
Posted by "prashantwason (via GitHub)" <gi...@apache.org>.
prashantwason commented on code in PR #8684:
URL: https://github.com/apache/hudi/pull/8684#discussion_r1219043175
##########
hudi-client/hudi-spark-client/src/main/java/org/apache/hudi/client/SparkRDDWriteClient.java:
##########
@@ -338,14 +338,20 @@ protected void initMetadataTable(Option<String> instantTime) {
* @param inFlightInstantTimestamp - The in-flight action responsible for the metadata table initialization
*/
private void initializeMetadataTable(Option<String> inFlightInstantTimestamp) {
- if (config.isMetadataTableEnabled()) {
- HoodieTableMetadataWriter writer = SparkHoodieBackedTableMetadataWriter.create(context.getHadoopConf().get(), config,
- context, Option.empty(), inFlightInstantTimestamp);
- try {
- writer.close();
- } catch (Exception e) {
- throw new HoodieException("Failed to instantiate Metadata table ", e);
+ if (!config.isMetadataTableEnabled()) {
+ LOG.error("================================111111111111111111");
+ return;
+ }
+
+ try (HoodieTableMetadataWriter writer = SparkHoodieBackedTableMetadataWriter.create(context.getHadoopConf().get(), config,
+ context, Option.empty(), inFlightInstantTimestamp)) {
+ if (writer.isInitialized()) {
+ writer.performTableServices(inFlightInstantTimestamp);
+ } else {
+ throw new HoodieException((".................2222222222"));
Review Comment:
Removed the debug log
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] prashantwason commented on a diff in pull request #8684: [HUDI-6200] Enhancements to the MDT for improving performance of larger indexes.
Posted by "prashantwason (via GitHub)" <gi...@apache.org>.
prashantwason commented on code in PR #8684:
URL: https://github.com/apache/hudi/pull/8684#discussion_r1219045032
##########
hudi-client/hudi-client-common/src/main/java/org/apache/hudi/metadata/HoodieBackedTableMetadataWriter.java:
##########
@@ -626,8 +657,7 @@ private void initializeFileGroups(HoodieTableMetaClient dataMetaClient, Metadata
HoodieLogFormat.Writer writer = HoodieLogFormat.newWriterBuilder()
.onParentPath(FSUtils.getPartitionPath(metadataWriteConfig.getBasePath(), metadataPartition.getPartitionPath()))
- .withFileId(fileGroupFileId)
- .overBaseCommit(instantTime)
+ .withFileId(fileGroupFileId).overBaseCommit(instantTime)
.withLogVersion(HoodieLogFile.LOGFILE_BASE_VERSION)
Review Comment:
Reverted
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #8684: [HUDI-6200] Enhancements to the MDT for improving performance of larger indexes.
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #8684:
URL: https://github.com/apache/hudi/pull/8684#issuecomment-1579812840
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "35148aeb4ba78eb6f3316c75f8a0a7e4c6d6df87",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17007",
"triggerID" : "35148aeb4ba78eb6f3316c75f8a0a7e4c6d6df87",
"triggerType" : "PUSH"
}, {
"hash" : "e2785f4675ddf74582ff34590608a5d71c5e9a2d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17008",
"triggerID" : "e2785f4675ddf74582ff34590608a5d71c5e9a2d",
"triggerType" : "PUSH"
}, {
"hash" : "364b1fe0dde303750ca0f8eebe5cd530c2a66a3a",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17042",
"triggerID" : "364b1fe0dde303750ca0f8eebe5cd530c2a66a3a",
"triggerType" : "PUSH"
}, {
"hash" : "9d73b9664eed48ada68721a92526640cfa06a021",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17098",
"triggerID" : "9d73b9664eed48ada68721a92526640cfa06a021",
"triggerType" : "PUSH"
}, {
"hash" : "7be538f4045a42ba33ab8fe62594178e2db75bbb",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17100",
"triggerID" : "7be538f4045a42ba33ab8fe62594178e2db75bbb",
"triggerType" : "PUSH"
}, {
"hash" : "cc0da2372d50d99c98c2ce4bcbe5a60303bde938",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17195",
"triggerID" : "cc0da2372d50d99c98c2ce4bcbe5a60303bde938",
"triggerType" : "PUSH"
}, {
"hash" : "da0e00e3655e06b064d11e20500317e2a1d2229c",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "da0e00e3655e06b064d11e20500317e2a1d2229c",
"triggerType" : "PUSH"
}, {
"hash" : "589c8cd7ffc5df7984c099d6b30fa117463db2b9",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17335",
"triggerID" : "589c8cd7ffc5df7984c099d6b30fa117463db2b9",
"triggerType" : "PUSH"
}, {
"hash" : "226c96ce48b33d7b1650d00e93bc5ea42ad276f6",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17624",
"triggerID" : "226c96ce48b33d7b1650d00e93bc5ea42ad276f6",
"triggerType" : "PUSH"
}, {
"hash" : "ad81a08799a98cd6ee63b8bf792d42f16a98f775",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17634",
"triggerID" : "ad81a08799a98cd6ee63b8bf792d42f16a98f775",
"triggerType" : "PUSH"
}, {
"hash" : "fd2ec462e554190843ddcc8d946317ae76cb81e1",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "fd2ec462e554190843ddcc8d946317ae76cb81e1",
"triggerType" : "PUSH"
}, {
"hash" : "cae57c2b03a3131cfa7774fa4335f4d153f85d61",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17644",
"triggerID" : "cae57c2b03a3131cfa7774fa4335f4d153f85d61",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* da0e00e3655e06b064d11e20500317e2a1d2229c UNKNOWN
* ad81a08799a98cd6ee63b8bf792d42f16a98f775 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17634)
* fd2ec462e554190843ddcc8d946317ae76cb81e1 UNKNOWN
* cae57c2b03a3131cfa7774fa4335f4d153f85d61 Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17644)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] danny0405 commented on a diff in pull request #8684: [HUDI-6200] Enhancements to the MDT for improving performance of larger indexes.
Posted by "danny0405 (via GitHub)" <gi...@apache.org>.
danny0405 commented on code in PR #8684:
URL: https://github.com/apache/hudi/pull/8684#discussion_r1222533305
##########
hudi-common/src/main/java/org/apache/hudi/metadata/HoodieTableMetadataUtil.java:
##########
@@ -1453,7 +1453,11 @@ public static String deleteMetadataTablePartition(HoodieTableMetaClient dataMeta
* @return The fileID
*/
public static String getFileIDForFileGroup(MetadataPartitionType partitionType, int index) {
- return String.format("%s%04d", partitionType.getFileIdPrefix(), index);
+ if (partitionType == MetadataPartitionType.FILES) {
+ return String.format("%s%04d-%d", partitionType.getFileIdPrefix(), index, 0);
+ } else {
Review Comment:
Discuss offline, this is a fix for consistency of base file fileGroup id and the log files, because the bulk_insert create handle would append a suffix (starts from `-0`) for each file it creates.
##########
hudi-common/src/main/java/org/apache/hudi/metadata/HoodieTableMetadataUtil.java:
##########
@@ -1453,7 +1453,11 @@ public static String deleteMetadataTablePartition(HoodieTableMetaClient dataMeta
* @return The fileID
*/
public static String getFileIDForFileGroup(MetadataPartitionType partitionType, int index) {
- return String.format("%s%04d", partitionType.getFileIdPrefix(), index);
+ if (partitionType == MetadataPartitionType.FILES) {
+ return String.format("%s%04d-%d", partitionType.getFileIdPrefix(), index, 0);
+ } else {
Review Comment:
Discussed offline, this is a fix for consistency of base file fileGroup id and the log files, because the bulk_insert create handle would append a suffix (starts from `-0`) for each file it creates.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #8684: [HUDI-6200] Enhancements to the MDT for improving performance of larger indexes.
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #8684:
URL: https://github.com/apache/hudi/pull/8684#issuecomment-1578140216
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "35148aeb4ba78eb6f3316c75f8a0a7e4c6d6df87",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17007",
"triggerID" : "35148aeb4ba78eb6f3316c75f8a0a7e4c6d6df87",
"triggerType" : "PUSH"
}, {
"hash" : "e2785f4675ddf74582ff34590608a5d71c5e9a2d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17008",
"triggerID" : "e2785f4675ddf74582ff34590608a5d71c5e9a2d",
"triggerType" : "PUSH"
}, {
"hash" : "364b1fe0dde303750ca0f8eebe5cd530c2a66a3a",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17042",
"triggerID" : "364b1fe0dde303750ca0f8eebe5cd530c2a66a3a",
"triggerType" : "PUSH"
}, {
"hash" : "9d73b9664eed48ada68721a92526640cfa06a021",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17098",
"triggerID" : "9d73b9664eed48ada68721a92526640cfa06a021",
"triggerType" : "PUSH"
}, {
"hash" : "7be538f4045a42ba33ab8fe62594178e2db75bbb",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17100",
"triggerID" : "7be538f4045a42ba33ab8fe62594178e2db75bbb",
"triggerType" : "PUSH"
}, {
"hash" : "cc0da2372d50d99c98c2ce4bcbe5a60303bde938",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17195",
"triggerID" : "cc0da2372d50d99c98c2ce4bcbe5a60303bde938",
"triggerType" : "PUSH"
}, {
"hash" : "da0e00e3655e06b064d11e20500317e2a1d2229c",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "da0e00e3655e06b064d11e20500317e2a1d2229c",
"triggerType" : "PUSH"
}, {
"hash" : "589c8cd7ffc5df7984c099d6b30fa117463db2b9",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17335",
"triggerID" : "589c8cd7ffc5df7984c099d6b30fa117463db2b9",
"triggerType" : "PUSH"
}, {
"hash" : "226c96ce48b33d7b1650d00e93bc5ea42ad276f6",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17624",
"triggerID" : "226c96ce48b33d7b1650d00e93bc5ea42ad276f6",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* da0e00e3655e06b064d11e20500317e2a1d2229c UNKNOWN
* 589c8cd7ffc5df7984c099d6b30fa117463db2b9 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17335)
* 226c96ce48b33d7b1650d00e93bc5ea42ad276f6 Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17624)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #8684: [HUDI-6200] Enhancements to the MDT for improving performance of larger indexes.
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #8684:
URL: https://github.com/apache/hudi/pull/8684#issuecomment-1581591086
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "35148aeb4ba78eb6f3316c75f8a0a7e4c6d6df87",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17007",
"triggerID" : "35148aeb4ba78eb6f3316c75f8a0a7e4c6d6df87",
"triggerType" : "PUSH"
}, {
"hash" : "e2785f4675ddf74582ff34590608a5d71c5e9a2d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17008",
"triggerID" : "e2785f4675ddf74582ff34590608a5d71c5e9a2d",
"triggerType" : "PUSH"
}, {
"hash" : "364b1fe0dde303750ca0f8eebe5cd530c2a66a3a",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17042",
"triggerID" : "364b1fe0dde303750ca0f8eebe5cd530c2a66a3a",
"triggerType" : "PUSH"
}, {
"hash" : "9d73b9664eed48ada68721a92526640cfa06a021",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17098",
"triggerID" : "9d73b9664eed48ada68721a92526640cfa06a021",
"triggerType" : "PUSH"
}, {
"hash" : "7be538f4045a42ba33ab8fe62594178e2db75bbb",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17100",
"triggerID" : "7be538f4045a42ba33ab8fe62594178e2db75bbb",
"triggerType" : "PUSH"
}, {
"hash" : "cc0da2372d50d99c98c2ce4bcbe5a60303bde938",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17195",
"triggerID" : "cc0da2372d50d99c98c2ce4bcbe5a60303bde938",
"triggerType" : "PUSH"
}, {
"hash" : "da0e00e3655e06b064d11e20500317e2a1d2229c",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "da0e00e3655e06b064d11e20500317e2a1d2229c",
"triggerType" : "PUSH"
}, {
"hash" : "589c8cd7ffc5df7984c099d6b30fa117463db2b9",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17335",
"triggerID" : "589c8cd7ffc5df7984c099d6b30fa117463db2b9",
"triggerType" : "PUSH"
}, {
"hash" : "226c96ce48b33d7b1650d00e93bc5ea42ad276f6",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17624",
"triggerID" : "226c96ce48b33d7b1650d00e93bc5ea42ad276f6",
"triggerType" : "PUSH"
}, {
"hash" : "ad81a08799a98cd6ee63b8bf792d42f16a98f775",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17634",
"triggerID" : "ad81a08799a98cd6ee63b8bf792d42f16a98f775",
"triggerType" : "PUSH"
}, {
"hash" : "fd2ec462e554190843ddcc8d946317ae76cb81e1",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "fd2ec462e554190843ddcc8d946317ae76cb81e1",
"triggerType" : "PUSH"
}, {
"hash" : "cae57c2b03a3131cfa7774fa4335f4d153f85d61",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17644",
"triggerID" : "cae57c2b03a3131cfa7774fa4335f4d153f85d61",
"triggerType" : "PUSH"
}, {
"hash" : "a21fb8753ed206fefcf598c9e134c280e90d3dba",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17654",
"triggerID" : "a21fb8753ed206fefcf598c9e134c280e90d3dba",
"triggerType" : "PUSH"
}, {
"hash" : "4dfaefde334b712b3ba272bb330242ddb7f4c0ac",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17655",
"triggerID" : "4dfaefde334b712b3ba272bb330242ddb7f4c0ac",
"triggerType" : "PUSH"
}, {
"hash" : "ca59a379b830f88fd72c175cb9b96a14b33550c0",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "ca59a379b830f88fd72c175cb9b96a14b33550c0",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* da0e00e3655e06b064d11e20500317e2a1d2229c UNKNOWN
* fd2ec462e554190843ddcc8d946317ae76cb81e1 UNKNOWN
* 4dfaefde334b712b3ba272bb330242ddb7f4c0ac Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17655)
* ca59a379b830f88fd72c175cb9b96a14b33550c0 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] nsivabalan commented on a diff in pull request #8684: [HUDI-6200] Enhancements to the MDT for improving performance of larger indexes.
Posted by "nsivabalan (via GitHub)" <gi...@apache.org>.
nsivabalan commented on code in PR #8684:
URL: https://github.com/apache/hudi/pull/8684#discussion_r1191727349
##########
hudi-client/hudi-client-common/src/main/java/org/apache/hudi/metadata/HoodieBackedTableMetadataWriter.java:
##########
@@ -562,53 +532,144 @@ private <T extends SpecificRecordBase> boolean isCommitRevertedByInFlightAction(
/**
* Initialize the Metadata Table by listing files and partitions from the file system.
*
- * @param dataMetaClient - {@code HoodieTableMetaClient} for the dataset.
+ * @param initializationTime - Timestamp to use for the commit
+ * @param partitionsToInit - List of MDT partitions to initialize
* @param inflightInstantTimestamp - Current action instant responsible for this initialization
*/
- private boolean initializeFromFilesystem(HoodieTableMetaClient dataMetaClient,
+ private boolean initializeFromFilesystem(String initializationTime, List<MetadataPartitionType> partitionsToInit,
Option<String> inflightInstantTimestamp) throws IOException {
if (anyPendingDataInstant(dataMetaClient, inflightInstantTimestamp)) {
return false;
}
- String createInstantTime = getInitialCommitInstantTime(dataMetaClient);
-
- initializeMetaClient(DEFAULT_METADATA_POPULATE_META_FIELDS);
- initTableMetadata();
- // if async metadata indexing is enabled,
- // then only initialize files partition as other partitions will be built using HoodieIndexer
- List<MetadataPartitionType> enabledPartitionTypes = new ArrayList<>();
- if (dataWriteConfig.isMetadataAsyncIndex()) {
- enabledPartitionTypes.add(MetadataPartitionType.FILES);
- } else {
- // all enabled ones should be initialized
- enabledPartitionTypes = this.enabledPartitionTypes;
+ // FILES partition is always initialized first
+ ValidationUtils.checkArgument(!partitionsToInit.contains(MetadataPartitionType.FILES)
+ || partitionsToInit.get(0).equals(MetadataPartitionType.FILES), "FILES partition should be initialized first: " + partitionsToInit);
+
+ metadataMetaClient = initializeMetaClient();
+
+ // Get a complete list of files and partitions from the file system or from already initialized FILES partition of MDT
+ boolean filesPartitionAvailable = dataMetaClient.getTableConfig().isMetadataPartitionEnabled(MetadataPartitionType.FILES);
+ List<DirectoryInfo> partitionInfoList = filesPartitionAvailable ? listAllPartitionsFromMDT(initializationTime) : listAllPartitionsFromFilesystem(initializationTime);
+ Map<String, Map<String, Long>> partitionToFilesMap = partitionInfoList.stream()
+ .map(p -> {
+ String partitionName = HoodieTableMetadataUtil.getPartitionIdentifier(p.getRelativePath());
+ return Pair.of(partitionName, p.getFileNameToSizeMap());
+ })
+ .collect(Collectors.toMap(Pair::getKey, Pair::getValue));
+
+ for (MetadataPartitionType partitionType : partitionsToInit) {
+ // Find the commit timestamp to use for this partition. Each initialization should use its own unique commit time.
+ String commitTimeForPartition = generateUniqueCommitInstantTime(initializationTime);
+
+ LOG.info("Initializing MDT partition " + partitionType + " at instant " + commitTimeForPartition);
+
+ Pair<Integer, HoodieData<HoodieRecord>> fileGroupCountAndRecordsPair;
+ switch (partitionType) {
+ case FILES:
+ fileGroupCountAndRecordsPair = initializeFilesPartition(initializationTime, partitionInfoList);
+ break;
+ case BLOOM_FILTERS:
+ fileGroupCountAndRecordsPair = initializeBloomFiltersPartition(initializationTime, partitionToFilesMap);
+ break;
+ case COLUMN_STATS:
+ fileGroupCountAndRecordsPair = initializeColumnStatsPartition(partitionToFilesMap);
+ break;
+ default:
+ throw new HoodieMetadataException("Unsupported MDT partition type: " + partitionType);
+ }
+
+ // Generate the file groups
+ final int fileGroupCount = fileGroupCountAndRecordsPair.getKey();
+ ValidationUtils.checkArgument(fileGroupCount > 0, "FileGroup count for MDT partition " + partitionType + " should be > 0");
+ initializeFileGroups(dataMetaClient, partitionType, commitTimeForPartition, fileGroupCount);
Review Comment:
lets add documentation as to why we need this.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #8684: [HUDI-6200] Enhancements to the MDT for improving performance of larger indexes.
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #8684:
URL: https://github.com/apache/hudi/pull/8684#issuecomment-1542977709
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "35148aeb4ba78eb6f3316c75f8a0a7e4c6d6df87",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17007",
"triggerID" : "35148aeb4ba78eb6f3316c75f8a0a7e4c6d6df87",
"triggerType" : "PUSH"
}, {
"hash" : "e2785f4675ddf74582ff34590608a5d71c5e9a2d",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17008",
"triggerID" : "e2785f4675ddf74582ff34590608a5d71c5e9a2d",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* e2785f4675ddf74582ff34590608a5d71c5e9a2d Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17008)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] nsivabalan commented on a diff in pull request #8684: [HUDI-6200] Enhancements to the MDT for improving performance of larger indexes.
Posted by "nsivabalan (via GitHub)" <gi...@apache.org>.
nsivabalan commented on code in PR #8684:
URL: https://github.com/apache/hudi/pull/8684#discussion_r1192726306
##########
hudi-client/hudi-client-common/src/main/java/org/apache/hudi/metadata/HoodieBackedTableMetadataWriter.java:
##########
@@ -1097,87 +1165,76 @@ protected void cleanIfNecessary(BaseHoodieWriteClient writeClient, String instan
// Trigger cleaning with suffixes based on the same instant time. This ensures that any future
// delta commits synced over will not have an instant time lesser than the last completed instant on the
// metadata table.
- writeClient.clean(instantTime + "002");
+ writeClient.clean(HoodieTableMetadataUtil.createCleanTimestamp(instantTime));
writeClient.lazyRollbackFailedIndexing();
}
/**
- * This is invoked to initialize metadata table for a dataset.
- * Initial commit has special handling mechanism due to its scale compared to other regular commits.
- * During cold startup, the list of files to be committed can be huge.
- * So creating a HoodieCommitMetadata out of these large number of files,
- * and calling the existing update(HoodieCommitMetadata) function does not scale well.
- * Hence, we have a special commit just for the initialization scenario.
+ * Validates the timeline for both main and metadata tables.
*/
- private void initialCommit(String createInstantTime, List<MetadataPartitionType> partitionTypes) {
- // List all partitions in the basePath of the containing dataset
- LOG.info("Initializing metadata table by using file listings in " + dataWriteConfig.getBasePath());
- engineContext.setJobStatus(this.getClass().getSimpleName(), "Initializing metadata table by listing files and partitions: " + dataWriteConfig.getTableName());
-
- Map<MetadataPartitionType, HoodieData<HoodieRecord>> partitionToRecordsMap = new HashMap<>();
-
- // skip file system listing to populate metadata records if it's a fresh table.
- // this is applicable only if the table already has N commits and metadata is enabled at a later point in time.
- if (createInstantTime.equals(SOLO_COMMIT_TIMESTAMP)) { // SOLO_COMMIT_TIMESTAMP will be the initial commit time in MDT for a fresh table.
- // If not, last completed commit in data table will be chosen as the initial commit time.
- LOG.info("Triggering empty Commit to metadata to initialize");
- } else {
- List<DirectoryInfo> partitionInfoList = listAllPartitions(dataMetaClient);
- Map<String, Map<String, Long>> partitionToFilesMap = partitionInfoList.stream()
- .map(p -> {
- String partitionName = HoodieTableMetadataUtil.getPartitionIdentifier(p.getRelativePath());
- return Pair.of(partitionName, p.getFileNameToSizeMap());
- })
- .collect(Collectors.toMap(Pair::getKey, Pair::getValue));
-
- int totalDataFilesCount = partitionToFilesMap.values().stream().mapToInt(Map::size).sum();
- List<String> partitions = new ArrayList<>(partitionToFilesMap.keySet());
-
- if (partitionTypes.contains(MetadataPartitionType.FILES)) {
- // Record which saves the list of all partitions
- HoodieRecord allPartitionRecord = HoodieMetadataPayload.createPartitionListRecord(partitions);
- HoodieData<HoodieRecord> filesPartitionRecords = getFilesPartitionRecords(createInstantTime, partitionInfoList, allPartitionRecord);
- ValidationUtils.checkState(filesPartitionRecords.count() == (partitions.size() + 1));
- partitionToRecordsMap.put(MetadataPartitionType.FILES, filesPartitionRecords);
- }
-
- if (partitionTypes.contains(MetadataPartitionType.BLOOM_FILTERS) && totalDataFilesCount > 0) {
- final HoodieData<HoodieRecord> recordsRDD = HoodieTableMetadataUtil.convertFilesToBloomFilterRecords(
- engineContext, Collections.emptyMap(), partitionToFilesMap, getRecordsGenerationParams(), createInstantTime);
- partitionToRecordsMap.put(MetadataPartitionType.BLOOM_FILTERS, recordsRDD);
- }
-
- if (partitionTypes.contains(MetadataPartitionType.COLUMN_STATS) && totalDataFilesCount > 0) {
- final HoodieData<HoodieRecord> recordsRDD = HoodieTableMetadataUtil.convertFilesToColumnStatsRecords(
- engineContext, Collections.emptyMap(), partitionToFilesMap, getRecordsGenerationParams());
- partitionToRecordsMap.put(MetadataPartitionType.COLUMN_STATS, recordsRDD);
- }
- LOG.info("Committing " + partitions.size() + " partitions and " + totalDataFilesCount + " files to metadata");
+ private boolean validateTimelineBeforeSchedulingCompaction(Option<String> inFlightInstantTimestamp, String latestDeltacommitTime) {
+ // There should not be any incomplete instants on MDT
+ HoodieActiveTimeline metadataTimeline = metadataMetaClient.reloadActiveTimeline();
+ List<HoodieInstant> pendingInstantsOnMetadataTable = metadataTimeline.filterInflightsAndRequested().getInstants();
+ if (!pendingInstantsOnMetadataTable.isEmpty()) {
+ LOG.info(String.format(
+ "Cannot compact MDT as there are %d inflight instants: %s",
+ pendingInstantsOnMetadataTable.size(), Arrays.toString(pendingInstantsOnMetadataTable.toArray())));
+ metrics.ifPresent(m -> m.incrementMetric(HoodieMetadataMetrics.SKIP_TABLE_SERVICES, 1));
+ return false;
}
- commit(createInstantTime, partitionToRecordsMap, false);
- }
+ // There should not be any incomplete instants on dataset
+ HoodieActiveTimeline datasetTimeline = dataMetaClient.reloadActiveTimeline();
+ List<HoodieInstant> pendingInstantsOnDataset = datasetTimeline.filterInflightsAndRequested().getInstantsAsStream()
+ .filter(i -> !inFlightInstantTimestamp.isPresent() || !i.getTimestamp().equals(inFlightInstantTimestamp.get()))
+ .collect(Collectors.toList());
+ if (!pendingInstantsOnDataset.isEmpty()) {
+ LOG.info(String.format(
+ "Cannot compact MDT as there are %d inflight instants on dataset before latest deltacommit %s: %s",
+ pendingInstantsOnDataset.size(), latestDeltacommitTime, Arrays.toString(pendingInstantsOnDataset.toArray())));
+ metrics.ifPresent(m -> m.incrementMetric(HoodieMetadataMetrics.SKIP_TABLE_SERVICES, 1));
+ return false;
+ }
- private HoodieData<HoodieRecord> getFilesPartitionRecords(String createInstantTime, List<DirectoryInfo> partitionInfoList, HoodieRecord allPartitionRecord) {
- HoodieData<HoodieRecord> filesPartitionRecords = engineContext.parallelize(Arrays.asList(allPartitionRecord), 1);
- if (partitionInfoList.isEmpty()) {
- return filesPartitionRecords;
+ // Check if the inflight commit is greater than all the completed commits.
+ Option<HoodieInstant> lastCompletedInstant = dataMetaClient.getActiveTimeline().filterCompletedInstants().lastInstant();
+ if (!lastCompletedInstant.isPresent()) {
+ LOG.info("Last completed commit is not present.");
+ return false;
+ }
+ if (HoodieTimeline.compareTimestamps(lastCompletedInstant.get().getTimestamp(),
+ HoodieTimeline.GREATER_THAN, inFlightInstantTimestamp.get())) {
+ // Completed commits validation failed.
+ LOG.info(String.format(
+ "Cannot compact MDT as there is %s that is greater than inflight instant: %s",
+ lastCompletedInstant.get(), inFlightInstantTimestamp.get()));
+ return false;
}
- HoodieData<HoodieRecord> fileListRecords = engineContext.parallelize(partitionInfoList, partitionInfoList.size()).map(partitionInfo -> {
- Map<String, Long> fileNameToSizeMap = partitionInfo.getFileNameToSizeMap();
- // filter for files that are part of the completed commits
- Map<String, Long> validFileNameToSizeMap = fileNameToSizeMap.entrySet().stream().filter(fileSizePair -> {
- String commitTime = FSUtils.getCommitTime(fileSizePair.getKey());
- return HoodieTimeline.compareTimestamps(commitTime, HoodieTimeline.LESSER_THAN_OR_EQUALS, createInstantTime);
- }).collect(Collectors.toMap(Map.Entry::getKey, Map.Entry::getValue));
+ String latestDeltaCommitTimeInMetadataTable = metadataMetaClient.reloadActiveTimeline()
Review Comment:
I guess this is not yet addressed
##########
hudi-client/hudi-client-common/src/main/java/org/apache/hudi/metadata/HoodieBackedTableMetadataWriter.java:
##########
@@ -1097,87 +1165,76 @@ protected void cleanIfNecessary(BaseHoodieWriteClient writeClient, String instan
// Trigger cleaning with suffixes based on the same instant time. This ensures that any future
// delta commits synced over will not have an instant time lesser than the last completed instant on the
// metadata table.
- writeClient.clean(instantTime + "002");
+ writeClient.clean(HoodieTableMetadataUtil.createCleanTimestamp(instantTime));
writeClient.lazyRollbackFailedIndexing();
}
/**
- * This is invoked to initialize metadata table for a dataset.
- * Initial commit has special handling mechanism due to its scale compared to other regular commits.
- * During cold startup, the list of files to be committed can be huge.
- * So creating a HoodieCommitMetadata out of these large number of files,
- * and calling the existing update(HoodieCommitMetadata) function does not scale well.
- * Hence, we have a special commit just for the initialization scenario.
+ * Validates the timeline for both main and metadata tables.
*/
- private void initialCommit(String createInstantTime, List<MetadataPartitionType> partitionTypes) {
- // List all partitions in the basePath of the containing dataset
- LOG.info("Initializing metadata table by using file listings in " + dataWriteConfig.getBasePath());
- engineContext.setJobStatus(this.getClass().getSimpleName(), "Initializing metadata table by listing files and partitions: " + dataWriteConfig.getTableName());
-
- Map<MetadataPartitionType, HoodieData<HoodieRecord>> partitionToRecordsMap = new HashMap<>();
-
- // skip file system listing to populate metadata records if it's a fresh table.
- // this is applicable only if the table already has N commits and metadata is enabled at a later point in time.
- if (createInstantTime.equals(SOLO_COMMIT_TIMESTAMP)) { // SOLO_COMMIT_TIMESTAMP will be the initial commit time in MDT for a fresh table.
- // If not, last completed commit in data table will be chosen as the initial commit time.
- LOG.info("Triggering empty Commit to metadata to initialize");
- } else {
- List<DirectoryInfo> partitionInfoList = listAllPartitions(dataMetaClient);
- Map<String, Map<String, Long>> partitionToFilesMap = partitionInfoList.stream()
- .map(p -> {
- String partitionName = HoodieTableMetadataUtil.getPartitionIdentifier(p.getRelativePath());
- return Pair.of(partitionName, p.getFileNameToSizeMap());
- })
- .collect(Collectors.toMap(Pair::getKey, Pair::getValue));
-
- int totalDataFilesCount = partitionToFilesMap.values().stream().mapToInt(Map::size).sum();
- List<String> partitions = new ArrayList<>(partitionToFilesMap.keySet());
-
- if (partitionTypes.contains(MetadataPartitionType.FILES)) {
- // Record which saves the list of all partitions
- HoodieRecord allPartitionRecord = HoodieMetadataPayload.createPartitionListRecord(partitions);
- HoodieData<HoodieRecord> filesPartitionRecords = getFilesPartitionRecords(createInstantTime, partitionInfoList, allPartitionRecord);
- ValidationUtils.checkState(filesPartitionRecords.count() == (partitions.size() + 1));
- partitionToRecordsMap.put(MetadataPartitionType.FILES, filesPartitionRecords);
- }
-
- if (partitionTypes.contains(MetadataPartitionType.BLOOM_FILTERS) && totalDataFilesCount > 0) {
- final HoodieData<HoodieRecord> recordsRDD = HoodieTableMetadataUtil.convertFilesToBloomFilterRecords(
- engineContext, Collections.emptyMap(), partitionToFilesMap, getRecordsGenerationParams(), createInstantTime);
- partitionToRecordsMap.put(MetadataPartitionType.BLOOM_FILTERS, recordsRDD);
- }
-
- if (partitionTypes.contains(MetadataPartitionType.COLUMN_STATS) && totalDataFilesCount > 0) {
- final HoodieData<HoodieRecord> recordsRDD = HoodieTableMetadataUtil.convertFilesToColumnStatsRecords(
- engineContext, Collections.emptyMap(), partitionToFilesMap, getRecordsGenerationParams());
- partitionToRecordsMap.put(MetadataPartitionType.COLUMN_STATS, recordsRDD);
- }
- LOG.info("Committing " + partitions.size() + " partitions and " + totalDataFilesCount + " files to metadata");
+ private boolean validateTimelineBeforeSchedulingCompaction(Option<String> inFlightInstantTimestamp, String latestDeltacommitTime) {
+ // There should not be any incomplete instants on MDT
+ HoodieActiveTimeline metadataTimeline = metadataMetaClient.reloadActiveTimeline();
+ List<HoodieInstant> pendingInstantsOnMetadataTable = metadataTimeline.filterInflightsAndRequested().getInstants();
+ if (!pendingInstantsOnMetadataTable.isEmpty()) {
+ LOG.info(String.format(
+ "Cannot compact MDT as there are %d inflight instants: %s",
+ pendingInstantsOnMetadataTable.size(), Arrays.toString(pendingInstantsOnMetadataTable.toArray())));
+ metrics.ifPresent(m -> m.incrementMetric(HoodieMetadataMetrics.SKIP_TABLE_SERVICES, 1));
+ return false;
}
- commit(createInstantTime, partitionToRecordsMap, false);
- }
+ // There should not be any incomplete instants on dataset
Review Comment:
same here
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] nsivabalan commented on a diff in pull request #8684: [HUDI-6200] Enhancements to the MDT for improving performance of larger indexes.
Posted by "nsivabalan (via GitHub)" <gi...@apache.org>.
nsivabalan commented on code in PR #8684:
URL: https://github.com/apache/hudi/pull/8684#discussion_r1192725066
##########
hudi-client/hudi-client-common/src/main/java/org/apache/hudi/metadata/HoodieBackedTableMetadataWriter.java:
##########
@@ -562,53 +532,144 @@ private <T extends SpecificRecordBase> boolean isCommitRevertedByInFlightAction(
/**
* Initialize the Metadata Table by listing files and partitions from the file system.
*
- * @param dataMetaClient - {@code HoodieTableMetaClient} for the dataset.
+ * @param initializationTime - Timestamp to use for the commit
+ * @param partitionsToInit - List of MDT partitions to initialize
* @param inflightInstantTimestamp - Current action instant responsible for this initialization
*/
- private boolean initializeFromFilesystem(HoodieTableMetaClient dataMetaClient,
+ private boolean initializeFromFilesystem(String initializationTime, List<MetadataPartitionType> partitionsToInit,
Option<String> inflightInstantTimestamp) throws IOException {
if (anyPendingDataInstant(dataMetaClient, inflightInstantTimestamp)) {
return false;
}
- String createInstantTime = getInitialCommitInstantTime(dataMetaClient);
-
- initializeMetaClient(DEFAULT_METADATA_POPULATE_META_FIELDS);
- initTableMetadata();
- // if async metadata indexing is enabled,
- // then only initialize files partition as other partitions will be built using HoodieIndexer
- List<MetadataPartitionType> enabledPartitionTypes = new ArrayList<>();
- if (dataWriteConfig.isMetadataAsyncIndex()) {
- enabledPartitionTypes.add(MetadataPartitionType.FILES);
- } else {
- // all enabled ones should be initialized
- enabledPartitionTypes = this.enabledPartitionTypes;
+ // FILES partition is always initialized first
+ ValidationUtils.checkArgument(!partitionsToInit.contains(MetadataPartitionType.FILES)
+ || partitionsToInit.get(0).equals(MetadataPartitionType.FILES), "FILES partition should be initialized first: " + partitionsToInit);
+
+ metadataMetaClient = initializeMetaClient();
+
+ // Get a complete list of files and partitions from the file system or from already initialized FILES partition of MDT
+ boolean filesPartitionAvailable = dataMetaClient.getTableConfig().isMetadataPartitionEnabled(MetadataPartitionType.FILES);
+ List<DirectoryInfo> partitionInfoList = filesPartitionAvailable ? listAllPartitionsFromMDT(initializationTime) : listAllPartitionsFromFilesystem(initializationTime);
+ Map<String, Map<String, Long>> partitionToFilesMap = partitionInfoList.stream()
+ .map(p -> {
+ String partitionName = HoodieTableMetadataUtil.getPartitionIdentifier(p.getRelativePath());
+ return Pair.of(partitionName, p.getFileNameToSizeMap());
+ })
+ .collect(Collectors.toMap(Pair::getKey, Pair::getValue));
+
+ for (MetadataPartitionType partitionType : partitionsToInit) {
+ // Find the commit timestamp to use for this partition. Each initialization should use its own unique commit time.
+ String commitTimeForPartition = generateUniqueCommitInstantTime(initializationTime);
+
+ LOG.info("Initializing MDT partition " + partitionType + " at instant " + commitTimeForPartition);
+
+ Pair<Integer, HoodieData<HoodieRecord>> fileGroupCountAndRecordsPair;
+ switch (partitionType) {
+ case FILES:
+ fileGroupCountAndRecordsPair = initializeFilesPartition(initializationTime, partitionInfoList);
+ break;
+ case BLOOM_FILTERS:
+ fileGroupCountAndRecordsPair = initializeBloomFiltersPartition(initializationTime, partitionToFilesMap);
+ break;
+ case COLUMN_STATS:
+ fileGroupCountAndRecordsPair = initializeColumnStatsPartition(partitionToFilesMap);
+ break;
+ default:
+ throw new HoodieMetadataException("Unsupported MDT partition type: " + partitionType);
+ }
+
+ // Generate the file groups
+ final int fileGroupCount = fileGroupCountAndRecordsPair.getKey();
+ ValidationUtils.checkArgument(fileGroupCount > 0, "FileGroup count for MDT partition " + partitionType + " should be > 0");
+ initializeFileGroups(dataMetaClient, partitionType, commitTimeForPartition, fileGroupCount);
+
+ // Perform the commit using bulkCommit
+ HoodieData<HoodieRecord> records = fileGroupCountAndRecordsPair.getValue();
+ bulkCommit(commitTimeForPartition, partitionType, records, fileGroupCount);
+ metadataMetaClient.reloadActiveTimeline();
+ dataMetaClient = HoodieTableMetadataUtil.setMetadataPartitionState(dataMetaClient, partitionType, true);
}
- initializeEnabledFileGroups(dataMetaClient, createInstantTime, enabledPartitionTypes);
- initialCommit(createInstantTime, enabledPartitionTypes);
- updateInitializedPartitionsInTableConfig(enabledPartitionTypes);
+
return true;
}
- private String getInitialCommitInstantTime(HoodieTableMetaClient dataMetaClient) {
- // If there is no commit on the dataset yet, use the SOLO_COMMIT_TIMESTAMP as the instant time for initial commit
- // Otherwise, we use the timestamp of the latest completed action.
- String createInstantTime = dataMetaClient.getActiveTimeline().filterCompletedInstants()
- .getReverseOrderedInstants().findFirst().map(HoodieInstant::getTimestamp).orElse(SOLO_COMMIT_TIMESTAMP);
- LOG.info("Creating a new metadata table in " + metadataWriteConfig.getBasePath() + " at instant " + createInstantTime);
- return createInstantTime;
+ /**
+ * Returns a unique timestamp to use for initializing a MDT partition.
+ * <p>
+ * Since commits are immutable, we should use unique timestamps to initialize each partition. For this, we will add a suffix to the given initializationTime
+ * until we find a unique timestamp.
+ *
+ * @param initializationTime Timestamp from dataset to use for initialization
+ * @return a unique timestamp for MDT
+ */
+ private String generateUniqueCommitInstantTime(String initializationTime) {
+ for (int offset = 0; ; ++offset) {
Review Comment:
if we can add docs for this, would be nice.
##########
hudi-client/hudi-client-common/src/main/java/org/apache/hudi/metadata/HoodieBackedTableMetadataWriter.java:
##########
@@ -562,53 +532,144 @@ private <T extends SpecificRecordBase> boolean isCommitRevertedByInFlightAction(
/**
* Initialize the Metadata Table by listing files and partitions from the file system.
*
- * @param dataMetaClient - {@code HoodieTableMetaClient} for the dataset.
+ * @param initializationTime - Timestamp to use for the commit
+ * @param partitionsToInit - List of MDT partitions to initialize
* @param inflightInstantTimestamp - Current action instant responsible for this initialization
*/
- private boolean initializeFromFilesystem(HoodieTableMetaClient dataMetaClient,
+ private boolean initializeFromFilesystem(String initializationTime, List<MetadataPartitionType> partitionsToInit,
Option<String> inflightInstantTimestamp) throws IOException {
if (anyPendingDataInstant(dataMetaClient, inflightInstantTimestamp)) {
return false;
}
- String createInstantTime = getInitialCommitInstantTime(dataMetaClient);
-
- initializeMetaClient(DEFAULT_METADATA_POPULATE_META_FIELDS);
- initTableMetadata();
- // if async metadata indexing is enabled,
- // then only initialize files partition as other partitions will be built using HoodieIndexer
- List<MetadataPartitionType> enabledPartitionTypes = new ArrayList<>();
- if (dataWriteConfig.isMetadataAsyncIndex()) {
- enabledPartitionTypes.add(MetadataPartitionType.FILES);
- } else {
- // all enabled ones should be initialized
- enabledPartitionTypes = this.enabledPartitionTypes;
+ // FILES partition is always initialized first
+ ValidationUtils.checkArgument(!partitionsToInit.contains(MetadataPartitionType.FILES)
+ || partitionsToInit.get(0).equals(MetadataPartitionType.FILES), "FILES partition should be initialized first: " + partitionsToInit);
+
+ metadataMetaClient = initializeMetaClient();
+
+ // Get a complete list of files and partitions from the file system or from already initialized FILES partition of MDT
+ boolean filesPartitionAvailable = dataMetaClient.getTableConfig().isMetadataPartitionEnabled(MetadataPartitionType.FILES);
+ List<DirectoryInfo> partitionInfoList = filesPartitionAvailable ? listAllPartitionsFromMDT(initializationTime) : listAllPartitionsFromFilesystem(initializationTime);
+ Map<String, Map<String, Long>> partitionToFilesMap = partitionInfoList.stream()
+ .map(p -> {
+ String partitionName = HoodieTableMetadataUtil.getPartitionIdentifier(p.getRelativePath());
+ return Pair.of(partitionName, p.getFileNameToSizeMap());
+ })
+ .collect(Collectors.toMap(Pair::getKey, Pair::getValue));
+
+ for (MetadataPartitionType partitionType : partitionsToInit) {
+ // Find the commit timestamp to use for this partition. Each initialization should use its own unique commit time.
+ String commitTimeForPartition = generateUniqueCommitInstantTime(initializationTime);
+
+ LOG.info("Initializing MDT partition " + partitionType + " at instant " + commitTimeForPartition);
+
+ Pair<Integer, HoodieData<HoodieRecord>> fileGroupCountAndRecordsPair;
+ switch (partitionType) {
+ case FILES:
+ fileGroupCountAndRecordsPair = initializeFilesPartition(initializationTime, partitionInfoList);
+ break;
+ case BLOOM_FILTERS:
+ fileGroupCountAndRecordsPair = initializeBloomFiltersPartition(initializationTime, partitionToFilesMap);
+ break;
+ case COLUMN_STATS:
+ fileGroupCountAndRecordsPair = initializeColumnStatsPartition(partitionToFilesMap);
+ break;
+ default:
+ throw new HoodieMetadataException("Unsupported MDT partition type: " + partitionType);
+ }
+
+ // Generate the file groups
+ final int fileGroupCount = fileGroupCountAndRecordsPair.getKey();
+ ValidationUtils.checkArgument(fileGroupCount > 0, "FileGroup count for MDT partition " + partitionType + " should be > 0");
+ initializeFileGroups(dataMetaClient, partitionType, commitTimeForPartition, fileGroupCount);
Review Comment:
thanks!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] nsivabalan commented on a diff in pull request #8684: [HUDI-6200] Enhancements to the MDT for improving performance of larger indexes.
Posted by "nsivabalan (via GitHub)" <gi...@apache.org>.
nsivabalan commented on code in PR #8684:
URL: https://github.com/apache/hudi/pull/8684#discussion_r1191726475
##########
hudi-client/hudi-client-common/src/main/java/org/apache/hudi/metadata/HoodieBackedTableMetadataWriter.java:
##########
@@ -562,53 +532,144 @@ private <T extends SpecificRecordBase> boolean isCommitRevertedByInFlightAction(
/**
* Initialize the Metadata Table by listing files and partitions from the file system.
*
- * @param dataMetaClient - {@code HoodieTableMetaClient} for the dataset.
+ * @param initializationTime - Timestamp to use for the commit
+ * @param partitionsToInit - List of MDT partitions to initialize
* @param inflightInstantTimestamp - Current action instant responsible for this initialization
*/
- private boolean initializeFromFilesystem(HoodieTableMetaClient dataMetaClient,
+ private boolean initializeFromFilesystem(String initializationTime, List<MetadataPartitionType> partitionsToInit,
Option<String> inflightInstantTimestamp) throws IOException {
if (anyPendingDataInstant(dataMetaClient, inflightInstantTimestamp)) {
return false;
}
- String createInstantTime = getInitialCommitInstantTime(dataMetaClient);
-
- initializeMetaClient(DEFAULT_METADATA_POPULATE_META_FIELDS);
- initTableMetadata();
- // if async metadata indexing is enabled,
- // then only initialize files partition as other partitions will be built using HoodieIndexer
- List<MetadataPartitionType> enabledPartitionTypes = new ArrayList<>();
- if (dataWriteConfig.isMetadataAsyncIndex()) {
- enabledPartitionTypes.add(MetadataPartitionType.FILES);
- } else {
- // all enabled ones should be initialized
- enabledPartitionTypes = this.enabledPartitionTypes;
+ // FILES partition is always initialized first
+ ValidationUtils.checkArgument(!partitionsToInit.contains(MetadataPartitionType.FILES)
+ || partitionsToInit.get(0).equals(MetadataPartitionType.FILES), "FILES partition should be initialized first: " + partitionsToInit);
+
+ metadataMetaClient = initializeMetaClient();
+
+ // Get a complete list of files and partitions from the file system or from already initialized FILES partition of MDT
+ boolean filesPartitionAvailable = dataMetaClient.getTableConfig().isMetadataPartitionEnabled(MetadataPartitionType.FILES);
+ List<DirectoryInfo> partitionInfoList = filesPartitionAvailable ? listAllPartitionsFromMDT(initializationTime) : listAllPartitionsFromFilesystem(initializationTime);
+ Map<String, Map<String, Long>> partitionToFilesMap = partitionInfoList.stream()
+ .map(p -> {
+ String partitionName = HoodieTableMetadataUtil.getPartitionIdentifier(p.getRelativePath());
+ return Pair.of(partitionName, p.getFileNameToSizeMap());
+ })
+ .collect(Collectors.toMap(Pair::getKey, Pair::getValue));
+
+ for (MetadataPartitionType partitionType : partitionsToInit) {
+ // Find the commit timestamp to use for this partition. Each initialization should use its own unique commit time.
+ String commitTimeForPartition = generateUniqueCommitInstantTime(initializationTime);
+
+ LOG.info("Initializing MDT partition " + partitionType + " at instant " + commitTimeForPartition);
+
+ Pair<Integer, HoodieData<HoodieRecord>> fileGroupCountAndRecordsPair;
+ switch (partitionType) {
+ case FILES:
+ fileGroupCountAndRecordsPair = initializeFilesPartition(initializationTime, partitionInfoList);
+ break;
+ case BLOOM_FILTERS:
+ fileGroupCountAndRecordsPair = initializeBloomFiltersPartition(initializationTime, partitionToFilesMap);
+ break;
+ case COLUMN_STATS:
+ fileGroupCountAndRecordsPair = initializeColumnStatsPartition(partitionToFilesMap);
+ break;
+ default:
+ throw new HoodieMetadataException("Unsupported MDT partition type: " + partitionType);
+ }
+
+ // Generate the file groups
+ final int fileGroupCount = fileGroupCountAndRecordsPair.getKey();
+ ValidationUtils.checkArgument(fileGroupCount > 0, "FileGroup count for MDT partition " + partitionType + " should be > 0");
+ initializeFileGroups(dataMetaClient, partitionType, commitTimeForPartition, fileGroupCount);
+
+ // Perform the commit using bulkCommit
+ HoodieData<HoodieRecord> records = fileGroupCountAndRecordsPair.getValue();
+ bulkCommit(commitTimeForPartition, partitionType, records, fileGroupCount);
+ metadataMetaClient.reloadActiveTimeline();
+ dataMetaClient = HoodieTableMetadataUtil.setMetadataPartitionState(dataMetaClient, partitionType, true);
}
- initializeEnabledFileGroups(dataMetaClient, createInstantTime, enabledPartitionTypes);
- initialCommit(createInstantTime, enabledPartitionTypes);
- updateInitializedPartitionsInTableConfig(enabledPartitionTypes);
+
return true;
}
- private String getInitialCommitInstantTime(HoodieTableMetaClient dataMetaClient) {
- // If there is no commit on the dataset yet, use the SOLO_COMMIT_TIMESTAMP as the instant time for initial commit
- // Otherwise, we use the timestamp of the latest completed action.
- String createInstantTime = dataMetaClient.getActiveTimeline().filterCompletedInstants()
- .getReverseOrderedInstants().findFirst().map(HoodieInstant::getTimestamp).orElse(SOLO_COMMIT_TIMESTAMP);
- LOG.info("Creating a new metadata table in " + metadataWriteConfig.getBasePath() + " at instant " + createInstantTime);
- return createInstantTime;
+ /**
+ * Returns a unique timestamp to use for initializing a MDT partition.
+ * <p>
+ * Since commits are immutable, we should use unique timestamps to initialize each partition. For this, we will add a suffix to the given initializationTime
+ * until we find a unique timestamp.
+ *
+ * @param initializationTime Timestamp from dataset to use for initialization
+ * @return a unique timestamp for MDT
+ */
+ private String generateUniqueCommitInstantTime(String initializationTime) {
+ for (int offset = 0; ; ++offset) {
Review Comment:
synced up directly.
we want to initialize N MDT partitions in a single for loop.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] prashantwason commented on a diff in pull request #8684: [HUDI-6200] Enhancements to the MDT for improving performance of larger indexes.
Posted by "prashantwason (via GitHub)" <gi...@apache.org>.
prashantwason commented on code in PR #8684:
URL: https://github.com/apache/hudi/pull/8684#discussion_r1194204960
##########
hudi-common/src/main/java/org/apache/hudi/common/table/HoodieTableConfig.java:
##########
@@ -670,17 +670,75 @@ private Long getTableChecksum() {
return getLong(TABLE_CHECKSUM);
}
- public List<String> getMetadataPartitionsInflight() {
- return StringUtils.split(
- getStringOrDefault(TABLE_METADATA_PARTITIONS_INFLIGHT, StringUtils.EMPTY_STRING),
- CONFIG_VALUES_DELIMITER
- );
+ public Set<String> getMetadataPartitionsInflight() {
+ return new HashSet<>(StringUtils.split(
+ getStringOrDefault(TABLE_METADATA_PARTITIONS_INFLIGHT, StringUtils.EMPTY_STRING),
+ CONFIG_VALUES_DELIMITER));
}
public Set<String> getMetadataPartitions() {
return new HashSet<>(
- StringUtils.split(getStringOrDefault(TABLE_METADATA_PARTITIONS, StringUtils.EMPTY_STRING),
- CONFIG_VALUES_DELIMITER));
+ StringUtils.split(getStringOrDefault(TABLE_METADATA_PARTITIONS, StringUtils.EMPTY_STRING),
+ CONFIG_VALUES_DELIMITER));
+ }
+
+ /**
+ * @returns true if metadata table has been created and is being used for this dataset, else returns false.
+ */
+ public boolean isMetadataTableEnabled() {
+ return isMetadataPartitionEnabled(MetadataPartitionType.FILES);
+ }
+
+ /**
+ * Checks if metadata table is enabled and the specified partition has been initialized.
+ *
+ * @param partition The partition to check
+ * @returns true if the specific partition has been initialized, else returns false.
+ */
+ public boolean isMetadataPartitionEnabled(MetadataPartitionType partition) {
+ return getMetadataPartitions().contains(partition.getPartitionPath());
+ }
+
+ /**
+ * Enables or disables the specified metadata table partition.
+ *
+ * @param partition The partition
+ * @param enabled If true, the partition is enabled, else disabled
+ */
+ public void setMetadataPartitionState(MetadataPartitionType partition, boolean enabled) {
+ ValidationUtils.checkArgument(!partition.getPartitionPath().contains(CONFIG_VALUES_DELIMITER),
+ "Metadata Table partition path cannot contain a comma: " + partition.getPartitionPath());
+ Set<String> partitions = getMetadataPartitions();
+ Set<String> partitionsInflight = getMetadataPartitionsInflight();
+ if (enabled) {
+ partitions.add(partition.getPartitionPath());
+ partitionsInflight.remove(partition.getPartitionPath());
+ } else if (partition.equals(MetadataPartitionType.FILES)) {
+ // file listing partition is required for all other partitions to work
+ // Disabling file partition will also disable all partitions
+ partitions.clear();
+ partitionsInflight.clear();
+ } else {
+ partitions.remove(partition.getPartitionPath());
+ partitionsInflight.remove(partition.getPartitionPath());
+ }
+ setValue(TABLE_METADATA_PARTITIONS, partitions.stream().sorted().collect(Collectors.joining(CONFIG_VALUES_DELIMITER)));
+ setValue(TABLE_METADATA_PARTITIONS_INFLIGHT, partitionsInflight.stream().sorted().collect(Collectors.joining(CONFIG_VALUES_DELIMITER)));
+ }
+
+ /**
+ * Enables the specified metadata table partition as inflight.
+ *
+ * @param partition The list of partitions to enable as inflight.
+ */
+ public void setMetadataPartitionsAsInflight(List<MetadataPartitionType> partitionTypes) {
+ Set<String> partitions = getMetadataPartitionsInflight();
Review Comment:
Done/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] prashantwason commented on a diff in pull request #8684: [HUDI-6200] Enhancements to the MDT for improving performance of larger indexes.
Posted by "prashantwason (via GitHub)" <gi...@apache.org>.
prashantwason commented on code in PR #8684:
URL: https://github.com/apache/hudi/pull/8684#discussion_r1194267719
##########
hudi-client/hudi-client-common/src/main/java/org/apache/hudi/metadata/HoodieBackedTableMetadataWriter.java:
##########
@@ -1097,87 +1165,76 @@ protected void cleanIfNecessary(BaseHoodieWriteClient writeClient, String instan
// Trigger cleaning with suffixes based on the same instant time. This ensures that any future
// delta commits synced over will not have an instant time lesser than the last completed instant on the
// metadata table.
- writeClient.clean(instantTime + "002");
+ writeClient.clean(HoodieTableMetadataUtil.createCleanTimestamp(instantTime));
writeClient.lazyRollbackFailedIndexing();
}
/**
- * This is invoked to initialize metadata table for a dataset.
- * Initial commit has special handling mechanism due to its scale compared to other regular commits.
- * During cold startup, the list of files to be committed can be huge.
- * So creating a HoodieCommitMetadata out of these large number of files,
- * and calling the existing update(HoodieCommitMetadata) function does not scale well.
- * Hence, we have a special commit just for the initialization scenario.
+ * Validates the timeline for both main and metadata tables.
*/
- private void initialCommit(String createInstantTime, List<MetadataPartitionType> partitionTypes) {
- // List all partitions in the basePath of the containing dataset
- LOG.info("Initializing metadata table by using file listings in " + dataWriteConfig.getBasePath());
- engineContext.setJobStatus(this.getClass().getSimpleName(), "Initializing metadata table by listing files and partitions: " + dataWriteConfig.getTableName());
-
- Map<MetadataPartitionType, HoodieData<HoodieRecord>> partitionToRecordsMap = new HashMap<>();
-
- // skip file system listing to populate metadata records if it's a fresh table.
- // this is applicable only if the table already has N commits and metadata is enabled at a later point in time.
- if (createInstantTime.equals(SOLO_COMMIT_TIMESTAMP)) { // SOLO_COMMIT_TIMESTAMP will be the initial commit time in MDT for a fresh table.
- // If not, last completed commit in data table will be chosen as the initial commit time.
- LOG.info("Triggering empty Commit to metadata to initialize");
- } else {
- List<DirectoryInfo> partitionInfoList = listAllPartitions(dataMetaClient);
- Map<String, Map<String, Long>> partitionToFilesMap = partitionInfoList.stream()
- .map(p -> {
- String partitionName = HoodieTableMetadataUtil.getPartitionIdentifier(p.getRelativePath());
- return Pair.of(partitionName, p.getFileNameToSizeMap());
- })
- .collect(Collectors.toMap(Pair::getKey, Pair::getValue));
-
- int totalDataFilesCount = partitionToFilesMap.values().stream().mapToInt(Map::size).sum();
- List<String> partitions = new ArrayList<>(partitionToFilesMap.keySet());
-
- if (partitionTypes.contains(MetadataPartitionType.FILES)) {
- // Record which saves the list of all partitions
- HoodieRecord allPartitionRecord = HoodieMetadataPayload.createPartitionListRecord(partitions);
- HoodieData<HoodieRecord> filesPartitionRecords = getFilesPartitionRecords(createInstantTime, partitionInfoList, allPartitionRecord);
- ValidationUtils.checkState(filesPartitionRecords.count() == (partitions.size() + 1));
- partitionToRecordsMap.put(MetadataPartitionType.FILES, filesPartitionRecords);
- }
-
- if (partitionTypes.contains(MetadataPartitionType.BLOOM_FILTERS) && totalDataFilesCount > 0) {
- final HoodieData<HoodieRecord> recordsRDD = HoodieTableMetadataUtil.convertFilesToBloomFilterRecords(
- engineContext, Collections.emptyMap(), partitionToFilesMap, getRecordsGenerationParams(), createInstantTime);
- partitionToRecordsMap.put(MetadataPartitionType.BLOOM_FILTERS, recordsRDD);
- }
-
- if (partitionTypes.contains(MetadataPartitionType.COLUMN_STATS) && totalDataFilesCount > 0) {
- final HoodieData<HoodieRecord> recordsRDD = HoodieTableMetadataUtil.convertFilesToColumnStatsRecords(
- engineContext, Collections.emptyMap(), partitionToFilesMap, getRecordsGenerationParams());
- partitionToRecordsMap.put(MetadataPartitionType.COLUMN_STATS, recordsRDD);
- }
- LOG.info("Committing " + partitions.size() + " partitions and " + totalDataFilesCount + " files to metadata");
+ private boolean validateTimelineBeforeSchedulingCompaction(Option<String> inFlightInstantTimestamp, String latestDeltacommitTime) {
+ // There should not be any incomplete instants on MDT
+ HoodieActiveTimeline metadataTimeline = metadataMetaClient.reloadActiveTimeline();
+ List<HoodieInstant> pendingInstantsOnMetadataTable = metadataTimeline.filterInflightsAndRequested().getInstants();
+ if (!pendingInstantsOnMetadataTable.isEmpty()) {
+ LOG.info(String.format(
+ "Cannot compact MDT as there are %d inflight instants: %s",
+ pendingInstantsOnMetadataTable.size(), Arrays.toString(pendingInstantsOnMetadataTable.toArray())));
+ metrics.ifPresent(m -> m.incrementMetric(HoodieMetadataMetrics.SKIP_TABLE_SERVICES, 1));
+ return false;
}
- commit(createInstantTime, partitionToRecordsMap, false);
- }
+ // There should not be any incomplete instants on dataset
+ HoodieActiveTimeline datasetTimeline = dataMetaClient.reloadActiveTimeline();
+ List<HoodieInstant> pendingInstantsOnDataset = datasetTimeline.filterInflightsAndRequested().getInstantsAsStream()
+ .filter(i -> !inFlightInstantTimestamp.isPresent() || !i.getTimestamp().equals(inFlightInstantTimestamp.get()))
+ .collect(Collectors.toList());
+ if (!pendingInstantsOnDataset.isEmpty()) {
+ LOG.info(String.format(
+ "Cannot compact MDT as there are %d inflight instants on dataset before latest deltacommit %s: %s",
+ pendingInstantsOnDataset.size(), latestDeltacommitTime, Arrays.toString(pendingInstantsOnDataset.toArray())));
+ metrics.ifPresent(m -> m.incrementMetric(HoodieMetadataMetrics.SKIP_TABLE_SERVICES, 1));
+ return false;
+ }
- private HoodieData<HoodieRecord> getFilesPartitionRecords(String createInstantTime, List<DirectoryInfo> partitionInfoList, HoodieRecord allPartitionRecord) {
- HoodieData<HoodieRecord> filesPartitionRecords = engineContext.parallelize(Arrays.asList(allPartitionRecord), 1);
- if (partitionInfoList.isEmpty()) {
- return filesPartitionRecords;
+ // Check if the inflight commit is greater than all the completed commits.
Review Comment:
I have simplified it and removed the extra checks.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #8684: [HUDI-6200] Enhancements to the MDT for improving performance of larger indexes.
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #8684:
URL: https://github.com/apache/hudi/pull/8684#issuecomment-1553577735
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "35148aeb4ba78eb6f3316c75f8a0a7e4c6d6df87",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17007",
"triggerID" : "35148aeb4ba78eb6f3316c75f8a0a7e4c6d6df87",
"triggerType" : "PUSH"
}, {
"hash" : "e2785f4675ddf74582ff34590608a5d71c5e9a2d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17008",
"triggerID" : "e2785f4675ddf74582ff34590608a5d71c5e9a2d",
"triggerType" : "PUSH"
}, {
"hash" : "364b1fe0dde303750ca0f8eebe5cd530c2a66a3a",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17042",
"triggerID" : "364b1fe0dde303750ca0f8eebe5cd530c2a66a3a",
"triggerType" : "PUSH"
}, {
"hash" : "9d73b9664eed48ada68721a92526640cfa06a021",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17098",
"triggerID" : "9d73b9664eed48ada68721a92526640cfa06a021",
"triggerType" : "PUSH"
}, {
"hash" : "7be538f4045a42ba33ab8fe62594178e2db75bbb",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17100",
"triggerID" : "7be538f4045a42ba33ab8fe62594178e2db75bbb",
"triggerType" : "PUSH"
}, {
"hash" : "cc0da2372d50d99c98c2ce4bcbe5a60303bde938",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "cc0da2372d50d99c98c2ce4bcbe5a60303bde938",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 7be538f4045a42ba33ab8fe62594178e2db75bbb Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17100)
* cc0da2372d50d99c98c2ce4bcbe5a60303bde938 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #8684: [HUDI-6200] Enhancements to the MDT for improving performance of larger indexes.
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #8684:
URL: https://github.com/apache/hudi/pull/8684#issuecomment-1553587888
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "35148aeb4ba78eb6f3316c75f8a0a7e4c6d6df87",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17007",
"triggerID" : "35148aeb4ba78eb6f3316c75f8a0a7e4c6d6df87",
"triggerType" : "PUSH"
}, {
"hash" : "e2785f4675ddf74582ff34590608a5d71c5e9a2d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17008",
"triggerID" : "e2785f4675ddf74582ff34590608a5d71c5e9a2d",
"triggerType" : "PUSH"
}, {
"hash" : "364b1fe0dde303750ca0f8eebe5cd530c2a66a3a",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17042",
"triggerID" : "364b1fe0dde303750ca0f8eebe5cd530c2a66a3a",
"triggerType" : "PUSH"
}, {
"hash" : "9d73b9664eed48ada68721a92526640cfa06a021",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17098",
"triggerID" : "9d73b9664eed48ada68721a92526640cfa06a021",
"triggerType" : "PUSH"
}, {
"hash" : "7be538f4045a42ba33ab8fe62594178e2db75bbb",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17100",
"triggerID" : "7be538f4045a42ba33ab8fe62594178e2db75bbb",
"triggerType" : "PUSH"
}, {
"hash" : "cc0da2372d50d99c98c2ce4bcbe5a60303bde938",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17195",
"triggerID" : "cc0da2372d50d99c98c2ce4bcbe5a60303bde938",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 7be538f4045a42ba33ab8fe62594178e2db75bbb Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17100)
* cc0da2372d50d99c98c2ce4bcbe5a60303bde938 Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17195)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] prashantwason commented on a diff in pull request #8684: [HUDI-6200] Enhancements to the MDT for improving performance of larger indexes.
Posted by "prashantwason (via GitHub)" <gi...@apache.org>.
prashantwason commented on code in PR #8684:
URL: https://github.com/apache/hudi/pull/8684#discussion_r1227127259
##########
hudi-client/hudi-spark-client/src/main/java/org/apache/hudi/metadata/SparkHoodieMetadataBulkInsertPartitioner.java:
##########
@@ -0,0 +1,111 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hudi.metadata;
+
+import java.io.Serializable;
+import java.util.ArrayList;
+import java.util.Comparator;
+import java.util.List;
+
+import org.apache.hudi.common.model.HoodieRecord;
+import org.apache.hudi.common.util.ValidationUtils;
+import org.apache.hudi.table.BulkInsertPartitioner;
+import org.apache.spark.Partitioner;
+import org.apache.spark.api.java.JavaRDD;
+
+import scala.Tuple2;
+
+/**
+ * A {@code BulkInsertPartitioner} implementation for Metadata Table to improve performance of initialization of metadata
+ * table partition when a very large number of records are inserted.
+ *
+ * This partitioner requires the records to be already tagged with location.
+ */
+public class SparkHoodieMetadataBulkInsertPartitioner implements BulkInsertPartitioner<JavaRDD<HoodieRecord>> {
+ final int numPartitions;
+ public SparkHoodieMetadataBulkInsertPartitioner(int numPartitions) {
+ this.numPartitions = numPartitions;
+ }
+
+ private class FileGroupPartitioner extends Partitioner {
+
+ @Override
+ public int getPartition(Object key) {
+ return ((Tuple2<Integer, String>)key)._1;
+ }
+
+ @Override
+ public int numPartitions() {
+ return numPartitions;
+ }
+ }
+
+ // FileIDs for the various partitions
+ private List<String> fileIDPfxs;
+
+ /**
+ * Partition the records by their location. The number of partitions is determined by the number of MDT fileGroups being udpated rather than the
+ * specific value of outputSparkPartitions.
+ */
+ @Override
+ public JavaRDD<HoodieRecord> repartitionRecords(JavaRDD<HoodieRecord> records, int outputSparkPartitions) {
+ Comparator<Tuple2<Integer, String>> keyComparator =
+ (Comparator<Tuple2<Integer, String>> & Serializable)(t1, t2) -> t1._2.compareTo(t2._2);
+
+ // Partition the records by their file group
+ JavaRDD<HoodieRecord> partitionedRDD = records
+ // key by <file group index, record key>. The file group index is used to partition and the record key is used to sort within the partition.
+ .keyBy(r -> {
+ int fileGroupIndex = HoodieTableMetadataUtil.getFileGroupIndexFromFileId(r.getCurrentLocation().getFileId());
+ return new Tuple2<>(fileGroupIndex, r.getRecordKey());
+ })
+ .repartitionAndSortWithinPartitions(new FileGroupPartitioner(), keyComparator)
+ .map(t -> t._2);
+
+ fileIDPfxs = partitionedRDD.mapPartitions(recordItr -> {
+ // Due to partitioning, all record in the partition should have same fileID. So we only can get the fileID prefix from the first record.
+ List<String> fileIds = new ArrayList<>(1);
+ if (recordItr.hasNext()) {
+ HoodieRecord record = recordItr.next();
+ final String fileID = HoodieTableMetadataUtil.getFileGroupPrefix(record.getCurrentLocation().getFileId());
+ fileIds.add(fileID);
+ } else {
+ // FileGroupPartitioner returns a fixed number of partition as part of numPartitions(). In the special case that recordsRDD has fewer
+ // records than fileGroupCount, some of these partitions (corresponding to fileGroups) will not have any data.
+ // But we still need to return a fileID for use within {@code BulkInsertMapFunction}
+ fileIds.add("");
+ }
+ return fileIds.iterator();
+ }, true).collect();
+ ValidationUtils.checkArgument(partitionedRDD.getNumPartitions() == fileIDPfxs.size(),
+ String.format("Generated fileIDPfxs (%d) are lesser in size than the partitions %d", fileIDPfxs.size(), partitionedRDD.getNumPartitions()));
+
+ return partitionedRDD;
+ }
+
+ @Override
+ public boolean arePartitionRecordsSorted() {
+ return true;
+ }
+
+ @Override
+ public String getFileIdPfx(int partitionId) {
+ return fileIDPfxs.get(partitionId);
Review Comment:
This PR also adds support for automatic estimation of the shard counts for each partition, that can be enhanced.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #8684: [HUDI-6200] Enhancements to the MDT for improving performance of larger indexes.
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #8684:
URL: https://github.com/apache/hudi/pull/8684#issuecomment-1553993821
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "35148aeb4ba78eb6f3316c75f8a0a7e4c6d6df87",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17007",
"triggerID" : "35148aeb4ba78eb6f3316c75f8a0a7e4c6d6df87",
"triggerType" : "PUSH"
}, {
"hash" : "e2785f4675ddf74582ff34590608a5d71c5e9a2d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17008",
"triggerID" : "e2785f4675ddf74582ff34590608a5d71c5e9a2d",
"triggerType" : "PUSH"
}, {
"hash" : "364b1fe0dde303750ca0f8eebe5cd530c2a66a3a",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17042",
"triggerID" : "364b1fe0dde303750ca0f8eebe5cd530c2a66a3a",
"triggerType" : "PUSH"
}, {
"hash" : "9d73b9664eed48ada68721a92526640cfa06a021",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17098",
"triggerID" : "9d73b9664eed48ada68721a92526640cfa06a021",
"triggerType" : "PUSH"
}, {
"hash" : "7be538f4045a42ba33ab8fe62594178e2db75bbb",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17100",
"triggerID" : "7be538f4045a42ba33ab8fe62594178e2db75bbb",
"triggerType" : "PUSH"
}, {
"hash" : "cc0da2372d50d99c98c2ce4bcbe5a60303bde938",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17195",
"triggerID" : "cc0da2372d50d99c98c2ce4bcbe5a60303bde938",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* cc0da2372d50d99c98c2ce4bcbe5a60303bde938 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17195)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] danny0405 commented on a diff in pull request #8684: [HUDI-6200] Enhancements to the MDT for improving performance of larger indexes.
Posted by "danny0405 (via GitHub)" <gi...@apache.org>.
danny0405 commented on code in PR #8684:
URL: https://github.com/apache/hudi/pull/8684#discussion_r1194736640
##########
hudi-client/hudi-spark-client/src/main/java/org/apache/hudi/client/SparkRDDWriteClient.java:
##########
@@ -337,15 +337,33 @@ protected void initMetadataTable(Option<String> instantTime) {
*
* @param inFlightInstantTimestamp - The in-flight action responsible for the metadata table initialization
*/
- private void initializeMetadataTable(Option<String> inFlightInstantTimestamp) {
- if (config.isMetadataTableEnabled()) {
- HoodieTableMetadataWriter writer = SparkHoodieBackedTableMetadataWriter.create(context.getHadoopConf().get(), config,
- context, Option.empty(), inFlightInstantTimestamp);
- try {
- writer.close();
- } catch (Exception e) {
- throw new HoodieException("Failed to instantiate Metadata table ", e);
+ private void initializeMetadataTable(WriteOperationType operationType, Option<String> inFlightInstantTimestamp) {
+ if (!config.isMetadataTableEnabled()) {
+ return;
+ }
+
+ try (HoodieTableMetadataWriter writer = SparkHoodieBackedTableMetadataWriter.create(context.getHadoopConf().get(), config,
+ context, Option.empty(), inFlightInstantTimestamp)) {
+ if (writer.isInitialized()) {
+ // Optimize the metadata table which involves compacton. cleaning, etc. This should only be called from writers.
+ switch (operationType) {
+ case INSERT:
+ case INSERT_PREPPED:
+ case UPSERT:
+ case UPSERT_PREPPED:
+ case BULK_INSERT:
+ case BULK_INSERT_PREPPED:
+ case DELETE:
Review Comment:
Enuming the write operation is really hard to maintain, can we trigger the table service whatever the operation is ?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] danny0405 commented on a diff in pull request #8684: [HUDI-6200] Enhancements to the MDT for improving performance of larger indexes.
Posted by "danny0405 (via GitHub)" <gi...@apache.org>.
danny0405 commented on code in PR #8684:
URL: https://github.com/apache/hudi/pull/8684#discussion_r1203785303
##########
hudi-client/hudi-client-common/src/main/java/org/apache/hudi/metadata/HoodieBackedTableMetadataWriter.java:
##########
@@ -564,53 +532,147 @@ private <T extends SpecificRecordBase> boolean isCommitRevertedByInFlightAction(
/**
* Initialize the Metadata Table by listing files and partitions from the file system.
*
- * @param dataMetaClient - {@code HoodieTableMetaClient} for the dataset.
+ * @param initializationTime - Timestamp to use for the commit
+ * @param partitionsToInit - List of MDT partitions to initialize
* @param inflightInstantTimestamp - Current action instant responsible for this initialization
*/
- private boolean initializeFromFilesystem(HoodieTableMetaClient dataMetaClient,
+ private boolean initializeFromFilesystem(String initializationTime, List<MetadataPartitionType> partitionsToInit,
Option<String> inflightInstantTimestamp) throws IOException {
if (anyPendingDataInstant(dataMetaClient, inflightInstantTimestamp)) {
return false;
}
- String createInstantTime = getInitialCommitInstantTime(dataMetaClient);
-
- initializeMetaClient(DEFAULT_METADATA_POPULATE_META_FIELDS);
- initTableMetadata();
- // if async metadata indexing is enabled,
- // then only initialize files partition as other partitions will be built using HoodieIndexer
- List<MetadataPartitionType> enabledPartitionTypes = new ArrayList<>();
- if (dataWriteConfig.isMetadataAsyncIndex()) {
- enabledPartitionTypes.add(MetadataPartitionType.FILES);
- } else {
- // all enabled ones should be initialized
- enabledPartitionTypes = this.enabledPartitionTypes;
+ // FILES partition is always initialized first
+ ValidationUtils.checkArgument(!partitionsToInit.contains(MetadataPartitionType.FILES)
+ || partitionsToInit.get(0).equals(MetadataPartitionType.FILES), "FILES partition should be initialized first: " + partitionsToInit);
+
+ metadataMetaClient = initializeMetaClient();
+
+ // Get a complete list of files and partitions from the file system or from already initialized FILES partition of MDT
+ boolean filesPartitionAvailable = dataMetaClient.getTableConfig().isMetadataPartitionEnabled(MetadataPartitionType.FILES);
+ List<DirectoryInfo> partitionInfoList = filesPartitionAvailable ? listAllPartitionsFromMDT(initializationTime) : listAllPartitionsFromFilesystem(initializationTime);
+ Map<String, Map<String, Long>> partitionToFilesMap = partitionInfoList.stream()
Review Comment:
Thanks for the clarification, makes sense to me.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #8684: [HUDI-6200] Enhancements to the MDT for improving performance of larger indexes.
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #8684:
URL: https://github.com/apache/hudi/pull/8684#issuecomment-1560774693
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "35148aeb4ba78eb6f3316c75f8a0a7e4c6d6df87",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17007",
"triggerID" : "35148aeb4ba78eb6f3316c75f8a0a7e4c6d6df87",
"triggerType" : "PUSH"
}, {
"hash" : "e2785f4675ddf74582ff34590608a5d71c5e9a2d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17008",
"triggerID" : "e2785f4675ddf74582ff34590608a5d71c5e9a2d",
"triggerType" : "PUSH"
}, {
"hash" : "364b1fe0dde303750ca0f8eebe5cd530c2a66a3a",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17042",
"triggerID" : "364b1fe0dde303750ca0f8eebe5cd530c2a66a3a",
"triggerType" : "PUSH"
}, {
"hash" : "9d73b9664eed48ada68721a92526640cfa06a021",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17098",
"triggerID" : "9d73b9664eed48ada68721a92526640cfa06a021",
"triggerType" : "PUSH"
}, {
"hash" : "7be538f4045a42ba33ab8fe62594178e2db75bbb",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17100",
"triggerID" : "7be538f4045a42ba33ab8fe62594178e2db75bbb",
"triggerType" : "PUSH"
}, {
"hash" : "cc0da2372d50d99c98c2ce4bcbe5a60303bde938",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17195",
"triggerID" : "cc0da2372d50d99c98c2ce4bcbe5a60303bde938",
"triggerType" : "PUSH"
}, {
"hash" : "da0e00e3655e06b064d11e20500317e2a1d2229c",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "da0e00e3655e06b064d11e20500317e2a1d2229c",
"triggerType" : "PUSH"
}, {
"hash" : "589c8cd7ffc5df7984c099d6b30fa117463db2b9",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "589c8cd7ffc5df7984c099d6b30fa117463db2b9",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* cc0da2372d50d99c98c2ce4bcbe5a60303bde938 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17195)
* da0e00e3655e06b064d11e20500317e2a1d2229c UNKNOWN
* 589c8cd7ffc5df7984c099d6b30fa117463db2b9 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #8684: [HUDI-6200] Enhancements to the MDT for improving performance of larger indexes.
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #8684:
URL: https://github.com/apache/hudi/pull/8684#issuecomment-1581836280
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "35148aeb4ba78eb6f3316c75f8a0a7e4c6d6df87",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17007",
"triggerID" : "35148aeb4ba78eb6f3316c75f8a0a7e4c6d6df87",
"triggerType" : "PUSH"
}, {
"hash" : "e2785f4675ddf74582ff34590608a5d71c5e9a2d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17008",
"triggerID" : "e2785f4675ddf74582ff34590608a5d71c5e9a2d",
"triggerType" : "PUSH"
}, {
"hash" : "364b1fe0dde303750ca0f8eebe5cd530c2a66a3a",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17042",
"triggerID" : "364b1fe0dde303750ca0f8eebe5cd530c2a66a3a",
"triggerType" : "PUSH"
}, {
"hash" : "9d73b9664eed48ada68721a92526640cfa06a021",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17098",
"triggerID" : "9d73b9664eed48ada68721a92526640cfa06a021",
"triggerType" : "PUSH"
}, {
"hash" : "7be538f4045a42ba33ab8fe62594178e2db75bbb",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17100",
"triggerID" : "7be538f4045a42ba33ab8fe62594178e2db75bbb",
"triggerType" : "PUSH"
}, {
"hash" : "cc0da2372d50d99c98c2ce4bcbe5a60303bde938",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17195",
"triggerID" : "cc0da2372d50d99c98c2ce4bcbe5a60303bde938",
"triggerType" : "PUSH"
}, {
"hash" : "da0e00e3655e06b064d11e20500317e2a1d2229c",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "da0e00e3655e06b064d11e20500317e2a1d2229c",
"triggerType" : "PUSH"
}, {
"hash" : "589c8cd7ffc5df7984c099d6b30fa117463db2b9",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17335",
"triggerID" : "589c8cd7ffc5df7984c099d6b30fa117463db2b9",
"triggerType" : "PUSH"
}, {
"hash" : "226c96ce48b33d7b1650d00e93bc5ea42ad276f6",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17624",
"triggerID" : "226c96ce48b33d7b1650d00e93bc5ea42ad276f6",
"triggerType" : "PUSH"
}, {
"hash" : "ad81a08799a98cd6ee63b8bf792d42f16a98f775",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17634",
"triggerID" : "ad81a08799a98cd6ee63b8bf792d42f16a98f775",
"triggerType" : "PUSH"
}, {
"hash" : "fd2ec462e554190843ddcc8d946317ae76cb81e1",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "fd2ec462e554190843ddcc8d946317ae76cb81e1",
"triggerType" : "PUSH"
}, {
"hash" : "cae57c2b03a3131cfa7774fa4335f4d153f85d61",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17644",
"triggerID" : "cae57c2b03a3131cfa7774fa4335f4d153f85d61",
"triggerType" : "PUSH"
}, {
"hash" : "a21fb8753ed206fefcf598c9e134c280e90d3dba",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17654",
"triggerID" : "a21fb8753ed206fefcf598c9e134c280e90d3dba",
"triggerType" : "PUSH"
}, {
"hash" : "4dfaefde334b712b3ba272bb330242ddb7f4c0ac",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17655",
"triggerID" : "4dfaefde334b712b3ba272bb330242ddb7f4c0ac",
"triggerType" : "PUSH"
}, {
"hash" : "ca59a379b830f88fd72c175cb9b96a14b33550c0",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17660",
"triggerID" : "ca59a379b830f88fd72c175cb9b96a14b33550c0",
"triggerType" : "PUSH"
}, {
"hash" : "1d49e658315a2ae22a2c29e3d388e82f869b146d",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17664",
"triggerID" : "1d49e658315a2ae22a2c29e3d388e82f869b146d",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* da0e00e3655e06b064d11e20500317e2a1d2229c UNKNOWN
* fd2ec462e554190843ddcc8d946317ae76cb81e1 UNKNOWN
* ca59a379b830f88fd72c175cb9b96a14b33550c0 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17660)
* 1d49e658315a2ae22a2c29e3d388e82f869b146d Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17664)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #8684: [HUDI-6200] Enhancements to the MDT for improving performance of larger indexes.
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #8684:
URL: https://github.com/apache/hudi/pull/8684#issuecomment-1582396153
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "35148aeb4ba78eb6f3316c75f8a0a7e4c6d6df87",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17007",
"triggerID" : "35148aeb4ba78eb6f3316c75f8a0a7e4c6d6df87",
"triggerType" : "PUSH"
}, {
"hash" : "e2785f4675ddf74582ff34590608a5d71c5e9a2d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17008",
"triggerID" : "e2785f4675ddf74582ff34590608a5d71c5e9a2d",
"triggerType" : "PUSH"
}, {
"hash" : "364b1fe0dde303750ca0f8eebe5cd530c2a66a3a",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17042",
"triggerID" : "364b1fe0dde303750ca0f8eebe5cd530c2a66a3a",
"triggerType" : "PUSH"
}, {
"hash" : "9d73b9664eed48ada68721a92526640cfa06a021",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17098",
"triggerID" : "9d73b9664eed48ada68721a92526640cfa06a021",
"triggerType" : "PUSH"
}, {
"hash" : "7be538f4045a42ba33ab8fe62594178e2db75bbb",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17100",
"triggerID" : "7be538f4045a42ba33ab8fe62594178e2db75bbb",
"triggerType" : "PUSH"
}, {
"hash" : "cc0da2372d50d99c98c2ce4bcbe5a60303bde938",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17195",
"triggerID" : "cc0da2372d50d99c98c2ce4bcbe5a60303bde938",
"triggerType" : "PUSH"
}, {
"hash" : "da0e00e3655e06b064d11e20500317e2a1d2229c",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "da0e00e3655e06b064d11e20500317e2a1d2229c",
"triggerType" : "PUSH"
}, {
"hash" : "589c8cd7ffc5df7984c099d6b30fa117463db2b9",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17335",
"triggerID" : "589c8cd7ffc5df7984c099d6b30fa117463db2b9",
"triggerType" : "PUSH"
}, {
"hash" : "226c96ce48b33d7b1650d00e93bc5ea42ad276f6",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17624",
"triggerID" : "226c96ce48b33d7b1650d00e93bc5ea42ad276f6",
"triggerType" : "PUSH"
}, {
"hash" : "ad81a08799a98cd6ee63b8bf792d42f16a98f775",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17634",
"triggerID" : "ad81a08799a98cd6ee63b8bf792d42f16a98f775",
"triggerType" : "PUSH"
}, {
"hash" : "fd2ec462e554190843ddcc8d946317ae76cb81e1",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "fd2ec462e554190843ddcc8d946317ae76cb81e1",
"triggerType" : "PUSH"
}, {
"hash" : "cae57c2b03a3131cfa7774fa4335f4d153f85d61",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17644",
"triggerID" : "cae57c2b03a3131cfa7774fa4335f4d153f85d61",
"triggerType" : "PUSH"
}, {
"hash" : "a21fb8753ed206fefcf598c9e134c280e90d3dba",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17654",
"triggerID" : "a21fb8753ed206fefcf598c9e134c280e90d3dba",
"triggerType" : "PUSH"
}, {
"hash" : "4dfaefde334b712b3ba272bb330242ddb7f4c0ac",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17655",
"triggerID" : "4dfaefde334b712b3ba272bb330242ddb7f4c0ac",
"triggerType" : "PUSH"
}, {
"hash" : "ca59a379b830f88fd72c175cb9b96a14b33550c0",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17660",
"triggerID" : "ca59a379b830f88fd72c175cb9b96a14b33550c0",
"triggerType" : "PUSH"
}, {
"hash" : "1d49e658315a2ae22a2c29e3d388e82f869b146d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17664",
"triggerID" : "1d49e658315a2ae22a2c29e3d388e82f869b146d",
"triggerType" : "PUSH"
}, {
"hash" : "df3a5b57a375fd344814b1f5a2451cfb79fea2c3",
"status" : "CANCELED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17669",
"triggerID" : "df3a5b57a375fd344814b1f5a2451cfb79fea2c3",
"triggerType" : "PUSH"
}, {
"hash" : "da92c6c310fccdb4f1e797b481eaf6bb027e551f",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17673",
"triggerID" : "da92c6c310fccdb4f1e797b481eaf6bb027e551f",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* da0e00e3655e06b064d11e20500317e2a1d2229c UNKNOWN
* fd2ec462e554190843ddcc8d946317ae76cb81e1 UNKNOWN
* df3a5b57a375fd344814b1f5a2451cfb79fea2c3 Azure: [CANCELED](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17669)
* da92c6c310fccdb4f1e797b481eaf6bb027e551f Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17673)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] nsivabalan commented on pull request #8684: [HUDI-6200] Enhancements to the MDT for improving performance of larger indexes.
Posted by "nsivabalan (via GitHub)" <gi...@apache.org>.
nsivabalan commented on PR #8684:
URL: https://github.com/apache/hudi/pull/8684#issuecomment-1583315765
CI is green
<img width="1496" alt="image" src="https://github.com/apache/hudi/assets/513218/f9ba5e86-bc6f-4b62-a14c-502151f2a188">
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] nsivabalan commented on a diff in pull request #8684: [HUDI-6200] Enhancements to the MDT for improving performance of larger indexes.
Posted by "nsivabalan (via GitHub)" <gi...@apache.org>.
nsivabalan commented on code in PR #8684:
URL: https://github.com/apache/hudi/pull/8684#discussion_r1220164422
##########
hudi-client/hudi-client-common/src/main/java/org/apache/hudi/metadata/HoodieBackedTableMetadataWriter.java:
##########
@@ -1138,7 +1120,11 @@ public DirectoryInfo(String relativePath, FileStatus[] fileStatus) {
this.isHoodiePartition = true;
} else if (FSUtils.isDataFile(status.getPath())) {
// Regular HUDI data file (base file or log file)
- filenameToSizeMap.put(status.getPath().getName(), status.getLen());
+ String dataFileCommitTime = FSUtils.getCommitTime(status.getPath().getName());
Review Comment:
may be this bug exists even if not for this patch. we might need a follow up fix.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] prashantwason commented on a diff in pull request #8684: [HUDI-6200] Enhancements to the MDT for improving performance of larger indexes.
Posted by "prashantwason (via GitHub)" <gi...@apache.org>.
prashantwason commented on code in PR #8684:
URL: https://github.com/apache/hudi/pull/8684#discussion_r1219042633
##########
hudi-client/hudi-spark-client/src/main/java/org/apache/hudi/client/SparkRDDWriteClient.java:
##########
@@ -338,14 +338,20 @@ protected void initMetadataTable(Option<String> instantTime) {
* @param inFlightInstantTimestamp - The in-flight action responsible for the metadata table initialization
*/
private void initializeMetadataTable(Option<String> inFlightInstantTimestamp) {
- if (config.isMetadataTableEnabled()) {
- HoodieTableMetadataWriter writer = SparkHoodieBackedTableMetadataWriter.create(context.getHadoopConf().get(), config,
- context, Option.empty(), inFlightInstantTimestamp);
- try {
- writer.close();
- } catch (Exception e) {
- throw new HoodieException("Failed to instantiate Metadata table ", e);
+ if (!config.isMetadataTableEnabled()) {
+ LOG.error("================================111111111111111111");
Review Comment:
Removed the debug log
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] nsivabalan commented on a diff in pull request #8684: [HUDI-6200] Enhancements to the MDT for improving performance of larger indexes.
Posted by "nsivabalan (via GitHub)" <gi...@apache.org>.
nsivabalan commented on code in PR #8684:
URL: https://github.com/apache/hudi/pull/8684#discussion_r1225701169
##########
hudi-client/hudi-spark-client/src/main/java/org/apache/hudi/metadata/SparkHoodieMetadataBulkInsertPartitioner.java:
##########
@@ -0,0 +1,111 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hudi.metadata;
+
+import java.io.Serializable;
+import java.util.ArrayList;
+import java.util.Comparator;
+import java.util.List;
+
+import org.apache.hudi.common.model.HoodieRecord;
+import org.apache.hudi.common.util.ValidationUtils;
+import org.apache.hudi.table.BulkInsertPartitioner;
+import org.apache.spark.Partitioner;
+import org.apache.spark.api.java.JavaRDD;
+
+import scala.Tuple2;
+
+/**
+ * A {@code BulkInsertPartitioner} implementation for Metadata Table to improve performance of initialization of metadata
+ * table partition when a very large number of records are inserted.
+ *
+ * This partitioner requires the records to be already tagged with location.
+ */
+public class SparkHoodieMetadataBulkInsertPartitioner implements BulkInsertPartitioner<JavaRDD<HoodieRecord>> {
+ final int numPartitions;
+ public SparkHoodieMetadataBulkInsertPartitioner(int numPartitions) {
+ this.numPartitions = numPartitions;
+ }
+
+ private class FileGroupPartitioner extends Partitioner {
+
+ @Override
+ public int getPartition(Object key) {
+ return ((Tuple2<Integer, String>)key)._1;
+ }
+
+ @Override
+ public int numPartitions() {
+ return numPartitions;
+ }
+ }
+
+ // FileIDs for the various partitions
+ private List<String> fileIDPfxs;
+
+ /**
+ * Partition the records by their location. The number of partitions is determined by the number of MDT fileGroups being udpated rather than the
+ * specific value of outputSparkPartitions.
+ */
+ @Override
+ public JavaRDD<HoodieRecord> repartitionRecords(JavaRDD<HoodieRecord> records, int outputSparkPartitions) {
+ Comparator<Tuple2<Integer, String>> keyComparator =
+ (Comparator<Tuple2<Integer, String>> & Serializable)(t1, t2) -> t1._2.compareTo(t2._2);
+
+ // Partition the records by their file group
+ JavaRDD<HoodieRecord> partitionedRDD = records
+ // key by <file group index, record key>. The file group index is used to partition and the record key is used to sort within the partition.
+ .keyBy(r -> {
+ int fileGroupIndex = HoodieTableMetadataUtil.getFileGroupIndexFromFileId(r.getCurrentLocation().getFileId());
+ return new Tuple2<>(fileGroupIndex, r.getRecordKey());
+ })
+ .repartitionAndSortWithinPartitions(new FileGroupPartitioner(), keyComparator)
+ .map(t -> t._2);
+
+ fileIDPfxs = partitionedRDD.mapPartitions(recordItr -> {
+ // Due to partitioning, all record in the partition should have same fileID. So we only can get the fileID prefix from the first record.
+ List<String> fileIds = new ArrayList<>(1);
+ if (recordItr.hasNext()) {
+ HoodieRecord record = recordItr.next();
+ final String fileID = HoodieTableMetadataUtil.getFileGroupPrefix(record.getCurrentLocation().getFileId());
+ fileIds.add(fileID);
+ } else {
+ // FileGroupPartitioner returns a fixed number of partition as part of numPartitions(). In the special case that recordsRDD has fewer
+ // records than fileGroupCount, some of these partitions (corresponding to fileGroups) will not have any data.
+ // But we still need to return a fileID for use within {@code BulkInsertMapFunction}
+ fileIds.add("");
+ }
+ return fileIds.iterator();
+ }, true).collect();
+ ValidationUtils.checkArgument(partitionedRDD.getNumPartitions() == fileIDPfxs.size(),
+ String.format("Generated fileIDPfxs (%d) are lesser in size than the partitions %d", fileIDPfxs.size(), partitionedRDD.getNumPartitions()));
+
+ return partitionedRDD;
+ }
+
+ @Override
+ public boolean arePartitionRecordsSorted() {
+ return true;
+ }
+
+ @Override
+ public String getFileIdPfx(int partitionId) {
+ return fileIDPfxs.get(partitionId);
Review Comment:
yes. this is known. we have set file size as 1GB. so, users have to set the right config for num of file groups.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] prashantwason commented on a diff in pull request #8684: [HUDI-6200] Enhancements to the MDT for improving performance of larger indexes.
Posted by "prashantwason (via GitHub)" <gi...@apache.org>.
prashantwason commented on code in PR #8684:
URL: https://github.com/apache/hudi/pull/8684#discussion_r1227126602
##########
hudi-client/hudi-spark-client/src/main/java/org/apache/hudi/metadata/SparkHoodieMetadataBulkInsertPartitioner.java:
##########
@@ -0,0 +1,111 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hudi.metadata;
+
+import java.io.Serializable;
+import java.util.ArrayList;
+import java.util.Comparator;
+import java.util.List;
+
+import org.apache.hudi.common.model.HoodieRecord;
+import org.apache.hudi.common.util.ValidationUtils;
+import org.apache.hudi.table.BulkInsertPartitioner;
+import org.apache.spark.Partitioner;
+import org.apache.spark.api.java.JavaRDD;
+
+import scala.Tuple2;
+
+/**
+ * A {@code BulkInsertPartitioner} implementation for Metadata Table to improve performance of initialization of metadata
+ * table partition when a very large number of records are inserted.
+ *
+ * This partitioner requires the records to be already tagged with location.
+ */
+public class SparkHoodieMetadataBulkInsertPartitioner implements BulkInsertPartitioner<JavaRDD<HoodieRecord>> {
+ final int numPartitions;
+ public SparkHoodieMetadataBulkInsertPartitioner(int numPartitions) {
+ this.numPartitions = numPartitions;
+ }
+
+ private class FileGroupPartitioner extends Partitioner {
+
+ @Override
+ public int getPartition(Object key) {
+ return ((Tuple2<Integer, String>)key)._1;
+ }
+
+ @Override
+ public int numPartitions() {
+ return numPartitions;
+ }
+ }
+
+ // FileIDs for the various partitions
+ private List<String> fileIDPfxs;
+
+ /**
+ * Partition the records by their location. The number of partitions is determined by the number of MDT fileGroups being udpated rather than the
+ * specific value of outputSparkPartitions.
+ */
+ @Override
+ public JavaRDD<HoodieRecord> repartitionRecords(JavaRDD<HoodieRecord> records, int outputSparkPartitions) {
+ Comparator<Tuple2<Integer, String>> keyComparator =
+ (Comparator<Tuple2<Integer, String>> & Serializable)(t1, t2) -> t1._2.compareTo(t2._2);
+
+ // Partition the records by their file group
+ JavaRDD<HoodieRecord> partitionedRDD = records
+ // key by <file group index, record key>. The file group index is used to partition and the record key is used to sort within the partition.
+ .keyBy(r -> {
+ int fileGroupIndex = HoodieTableMetadataUtil.getFileGroupIndexFromFileId(r.getCurrentLocation().getFileId());
+ return new Tuple2<>(fileGroupIndex, r.getRecordKey());
+ })
+ .repartitionAndSortWithinPartitions(new FileGroupPartitioner(), keyComparator)
+ .map(t -> t._2);
+
+ fileIDPfxs = partitionedRDD.mapPartitions(recordItr -> {
+ // Due to partitioning, all record in the partition should have same fileID. So we only can get the fileID prefix from the first record.
+ List<String> fileIds = new ArrayList<>(1);
+ if (recordItr.hasNext()) {
+ HoodieRecord record = recordItr.next();
+ final String fileID = HoodieTableMetadataUtil.getFileGroupPrefix(record.getCurrentLocation().getFileId());
+ fileIds.add(fileID);
+ } else {
+ // FileGroupPartitioner returns a fixed number of partition as part of numPartitions(). In the special case that recordsRDD has fewer
+ // records than fileGroupCount, some of these partitions (corresponding to fileGroups) will not have any data.
+ // But we still need to return a fileID for use within {@code BulkInsertMapFunction}
+ fileIds.add("");
+ }
+ return fileIds.iterator();
+ }, true).collect();
+ ValidationUtils.checkArgument(partitionedRDD.getNumPartitions() == fileIDPfxs.size(),
+ String.format("Generated fileIDPfxs (%d) are lesser in size than the partitions %d", fileIDPfxs.size(), partitionedRDD.getNumPartitions()));
+
+ return partitionedRDD;
+ }
+
+ @Override
+ public boolean arePartitionRecordsSorted() {
+ return true;
+ }
+
+ @Override
+ public String getFileIdPfx(int partitionId) {
+ return fileIDPfxs.get(partitionId);
Review Comment:
The idea behind sharding in MDT is that you can create more shards rather than splitting a shard into two files. Spliiting will cause one large and one small size file which is not optimal.
For optimal performance, all shards should be of similar size which we achieve by hash partitioning the keys.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] prashantwason commented on a diff in pull request #8684: [HUDI-6200] Enhancements to the MDT for improving performance of larger indexes.
Posted by "prashantwason (via GitHub)" <gi...@apache.org>.
prashantwason commented on code in PR #8684:
URL: https://github.com/apache/hudi/pull/8684#discussion_r1194268190
##########
hudi-client/hudi-client-common/src/main/java/org/apache/hudi/metadata/HoodieBackedTableMetadataWriter.java:
##########
@@ -1097,87 +1165,76 @@ protected void cleanIfNecessary(BaseHoodieWriteClient writeClient, String instan
// Trigger cleaning with suffixes based on the same instant time. This ensures that any future
// delta commits synced over will not have an instant time lesser than the last completed instant on the
// metadata table.
- writeClient.clean(instantTime + "002");
+ writeClient.clean(HoodieTableMetadataUtil.createCleanTimestamp(instantTime));
writeClient.lazyRollbackFailedIndexing();
}
/**
- * This is invoked to initialize metadata table for a dataset.
- * Initial commit has special handling mechanism due to its scale compared to other regular commits.
- * During cold startup, the list of files to be committed can be huge.
- * So creating a HoodieCommitMetadata out of these large number of files,
- * and calling the existing update(HoodieCommitMetadata) function does not scale well.
- * Hence, we have a special commit just for the initialization scenario.
+ * Validates the timeline for both main and metadata tables.
*/
- private void initialCommit(String createInstantTime, List<MetadataPartitionType> partitionTypes) {
- // List all partitions in the basePath of the containing dataset
- LOG.info("Initializing metadata table by using file listings in " + dataWriteConfig.getBasePath());
- engineContext.setJobStatus(this.getClass().getSimpleName(), "Initializing metadata table by listing files and partitions: " + dataWriteConfig.getTableName());
-
- Map<MetadataPartitionType, HoodieData<HoodieRecord>> partitionToRecordsMap = new HashMap<>();
-
- // skip file system listing to populate metadata records if it's a fresh table.
- // this is applicable only if the table already has N commits and metadata is enabled at a later point in time.
- if (createInstantTime.equals(SOLO_COMMIT_TIMESTAMP)) { // SOLO_COMMIT_TIMESTAMP will be the initial commit time in MDT for a fresh table.
- // If not, last completed commit in data table will be chosen as the initial commit time.
- LOG.info("Triggering empty Commit to metadata to initialize");
- } else {
- List<DirectoryInfo> partitionInfoList = listAllPartitions(dataMetaClient);
- Map<String, Map<String, Long>> partitionToFilesMap = partitionInfoList.stream()
- .map(p -> {
- String partitionName = HoodieTableMetadataUtil.getPartitionIdentifier(p.getRelativePath());
- return Pair.of(partitionName, p.getFileNameToSizeMap());
- })
- .collect(Collectors.toMap(Pair::getKey, Pair::getValue));
-
- int totalDataFilesCount = partitionToFilesMap.values().stream().mapToInt(Map::size).sum();
- List<String> partitions = new ArrayList<>(partitionToFilesMap.keySet());
-
- if (partitionTypes.contains(MetadataPartitionType.FILES)) {
- // Record which saves the list of all partitions
- HoodieRecord allPartitionRecord = HoodieMetadataPayload.createPartitionListRecord(partitions);
- HoodieData<HoodieRecord> filesPartitionRecords = getFilesPartitionRecords(createInstantTime, partitionInfoList, allPartitionRecord);
- ValidationUtils.checkState(filesPartitionRecords.count() == (partitions.size() + 1));
- partitionToRecordsMap.put(MetadataPartitionType.FILES, filesPartitionRecords);
- }
-
- if (partitionTypes.contains(MetadataPartitionType.BLOOM_FILTERS) && totalDataFilesCount > 0) {
- final HoodieData<HoodieRecord> recordsRDD = HoodieTableMetadataUtil.convertFilesToBloomFilterRecords(
- engineContext, Collections.emptyMap(), partitionToFilesMap, getRecordsGenerationParams(), createInstantTime);
- partitionToRecordsMap.put(MetadataPartitionType.BLOOM_FILTERS, recordsRDD);
- }
-
- if (partitionTypes.contains(MetadataPartitionType.COLUMN_STATS) && totalDataFilesCount > 0) {
- final HoodieData<HoodieRecord> recordsRDD = HoodieTableMetadataUtil.convertFilesToColumnStatsRecords(
- engineContext, Collections.emptyMap(), partitionToFilesMap, getRecordsGenerationParams());
- partitionToRecordsMap.put(MetadataPartitionType.COLUMN_STATS, recordsRDD);
- }
- LOG.info("Committing " + partitions.size() + " partitions and " + totalDataFilesCount + " files to metadata");
+ private boolean validateTimelineBeforeSchedulingCompaction(Option<String> inFlightInstantTimestamp, String latestDeltacommitTime) {
+ // There should not be any incomplete instants on MDT
+ HoodieActiveTimeline metadataTimeline = metadataMetaClient.reloadActiveTimeline();
+ List<HoodieInstant> pendingInstantsOnMetadataTable = metadataTimeline.filterInflightsAndRequested().getInstants();
+ if (!pendingInstantsOnMetadataTable.isEmpty()) {
+ LOG.info(String.format(
+ "Cannot compact MDT as there are %d inflight instants: %s",
+ pendingInstantsOnMetadataTable.size(), Arrays.toString(pendingInstantsOnMetadataTable.toArray())));
+ metrics.ifPresent(m -> m.incrementMetric(HoodieMetadataMetrics.SKIP_TABLE_SERVICES, 1));
+ return false;
}
- commit(createInstantTime, partitionToRecordsMap, false);
- }
+ // There should not be any incomplete instants on dataset
Review Comment:
Yes, removed duplicate checks.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #8684: [HUDI-6200] Enhancements to the MDT for improving performance of larger indexes.
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #8684:
URL: https://github.com/apache/hudi/pull/8684#issuecomment-1548485695
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "35148aeb4ba78eb6f3316c75f8a0a7e4c6d6df87",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17007",
"triggerID" : "35148aeb4ba78eb6f3316c75f8a0a7e4c6d6df87",
"triggerType" : "PUSH"
}, {
"hash" : "e2785f4675ddf74582ff34590608a5d71c5e9a2d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17008",
"triggerID" : "e2785f4675ddf74582ff34590608a5d71c5e9a2d",
"triggerType" : "PUSH"
}, {
"hash" : "364b1fe0dde303750ca0f8eebe5cd530c2a66a3a",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17042",
"triggerID" : "364b1fe0dde303750ca0f8eebe5cd530c2a66a3a",
"triggerType" : "PUSH"
}, {
"hash" : "9d73b9664eed48ada68721a92526640cfa06a021",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "9d73b9664eed48ada68721a92526640cfa06a021",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 364b1fe0dde303750ca0f8eebe5cd530c2a66a3a Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17042)
* 9d73b9664eed48ada68721a92526640cfa06a021 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] prashantwason commented on a diff in pull request #8684: [HUDI-6200] Enhancements to the MDT for improving performance of larger indexes.
Posted by "prashantwason (via GitHub)" <gi...@apache.org>.
prashantwason commented on code in PR #8684:
URL: https://github.com/apache/hudi/pull/8684#discussion_r1194209832
##########
hudi-client/hudi-client-common/src/main/java/org/apache/hudi/metadata/HoodieBackedTableMetadataWriter.java:
##########
@@ -824,25 +863,22 @@ private interface ConvertMetadataFunction {
/**
* Processes commit metadata from data table and commits to metadata table.
*
- * @param instantTime instant time of interest.
+ * @param instantTime instant time of interest.
* @param convertMetadataFunction converter function to convert the respective metadata to List of HoodieRecords to be written to metadata table.
- * @param <T> type of commit metadata.
- * @param canTriggerTableService true if table services can be triggered. false otherwise.
*/
- private <T> void processAndCommit(String instantTime, ConvertMetadataFunction convertMetadataFunction, boolean canTriggerTableService) {
- if (!dataWriteConfig.isMetadataTableEnabled()) {
- return;
- }
+ private void processAndCommit(String instantTime, ConvertMetadataFunction convertMetadataFunction) {
+ ValidationUtils.checkArgument(dataWriteConfig.isMetadataTableEnabled());
+
Set<String> partitionsToUpdate = getMetadataPartitionsToUpdate();
Set<String> inflightIndexes = getInflightMetadataPartitions(dataMetaClient.getTableConfig());
// if indexing is inflight then do not trigger table service
boolean doNotTriggerTableService = partitionsToUpdate.stream().anyMatch(inflightIndexes::contains);
Review Comment:
Removed.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #8684: [HUDI-6200] Enhancements to the MDT for improving performance of larger indexes.
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #8684:
URL: https://github.com/apache/hudi/pull/8684#issuecomment-1549019835
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "35148aeb4ba78eb6f3316c75f8a0a7e4c6d6df87",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17007",
"triggerID" : "35148aeb4ba78eb6f3316c75f8a0a7e4c6d6df87",
"triggerType" : "PUSH"
}, {
"hash" : "e2785f4675ddf74582ff34590608a5d71c5e9a2d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17008",
"triggerID" : "e2785f4675ddf74582ff34590608a5d71c5e9a2d",
"triggerType" : "PUSH"
}, {
"hash" : "364b1fe0dde303750ca0f8eebe5cd530c2a66a3a",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17042",
"triggerID" : "364b1fe0dde303750ca0f8eebe5cd530c2a66a3a",
"triggerType" : "PUSH"
}, {
"hash" : "9d73b9664eed48ada68721a92526640cfa06a021",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17098",
"triggerID" : "9d73b9664eed48ada68721a92526640cfa06a021",
"triggerType" : "PUSH"
}, {
"hash" : "7be538f4045a42ba33ab8fe62594178e2db75bbb",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17100",
"triggerID" : "7be538f4045a42ba33ab8fe62594178e2db75bbb",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 7be538f4045a42ba33ab8fe62594178e2db75bbb Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17100)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] danny0405 commented on a diff in pull request #8684: [HUDI-6200] Enhancements to the MDT for improving performance of larger indexes.
Posted by "danny0405 (via GitHub)" <gi...@apache.org>.
danny0405 commented on code in PR #8684:
URL: https://github.com/apache/hudi/pull/8684#discussion_r1192914114
##########
hudi-client/hudi-client-common/src/main/java/org/apache/hudi/metadata/HoodieBackedTableMetadataWriter.java:
##########
@@ -824,25 +863,22 @@ private interface ConvertMetadataFunction {
/**
* Processes commit metadata from data table and commits to metadata table.
*
- * @param instantTime instant time of interest.
+ * @param instantTime instant time of interest.
* @param convertMetadataFunction converter function to convert the respective metadata to List of HoodieRecords to be written to metadata table.
- * @param <T> type of commit metadata.
- * @param canTriggerTableService true if table services can be triggered. false otherwise.
*/
- private <T> void processAndCommit(String instantTime, ConvertMetadataFunction convertMetadataFunction, boolean canTriggerTableService) {
- if (!dataWriteConfig.isMetadataTableEnabled()) {
- return;
- }
+ private void processAndCommit(String instantTime, ConvertMetadataFunction convertMetadataFunction) {
+ ValidationUtils.checkArgument(dataWriteConfig.isMetadataTableEnabled());
+
Set<String> partitionsToUpdate = getMetadataPartitionsToUpdate();
Set<String> inflightIndexes = getInflightMetadataPartitions(dataMetaClient.getTableConfig());
// if indexing is inflight then do not trigger table service
boolean doNotTriggerTableService = partitionsToUpdate.stream().anyMatch(inflightIndexes::contains);
Review Comment:
I mean `doNotTriggerTableService`, it seems not be used in anywhere.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] nsivabalan commented on a diff in pull request #8684: [HUDI-6200] Enhancements to the MDT for improving performance of larger indexes.
Posted by "nsivabalan (via GitHub)" <gi...@apache.org>.
nsivabalan commented on code in PR #8684:
URL: https://github.com/apache/hudi/pull/8684#discussion_r1192724508
##########
hudi-common/src/main/java/org/apache/hudi/common/table/HoodieTableConfig.java:
##########
@@ -670,17 +670,75 @@ private Long getTableChecksum() {
return getLong(TABLE_CHECKSUM);
}
- public List<String> getMetadataPartitionsInflight() {
- return StringUtils.split(
- getStringOrDefault(TABLE_METADATA_PARTITIONS_INFLIGHT, StringUtils.EMPTY_STRING),
- CONFIG_VALUES_DELIMITER
- );
+ public Set<String> getMetadataPartitionsInflight() {
+ return new HashSet<>(StringUtils.split(
+ getStringOrDefault(TABLE_METADATA_PARTITIONS_INFLIGHT, StringUtils.EMPTY_STRING),
+ CONFIG_VALUES_DELIMITER));
}
public Set<String> getMetadataPartitions() {
return new HashSet<>(
- StringUtils.split(getStringOrDefault(TABLE_METADATA_PARTITIONS, StringUtils.EMPTY_STRING),
- CONFIG_VALUES_DELIMITER));
+ StringUtils.split(getStringOrDefault(TABLE_METADATA_PARTITIONS, StringUtils.EMPTY_STRING),
+ CONFIG_VALUES_DELIMITER));
+ }
+
+ /**
+ * @returns true if metadata table has been created and is being used for this dataset, else returns false.
+ */
+ public boolean isMetadataTableEnabled() {
+ return isMetadataPartitionEnabled(MetadataPartitionType.FILES);
+ }
+
+ /**
+ * Checks if metadata table is enabled and the specified partition has been initialized.
+ *
+ * @param partition The partition to check
+ * @returns true if the specific partition has been initialized, else returns false.
+ */
+ public boolean isMetadataPartitionEnabled(MetadataPartitionType partition) {
+ return getMetadataPartitions().contains(partition.getPartitionPath());
+ }
+
+ /**
+ * Enables or disables the specified metadata table partition.
+ *
+ * @param partition The partition
+ * @param enabled If true, the partition is enabled, else disabled
+ */
+ public void setMetadataPartitionState(MetadataPartitionType partition, boolean enabled) {
+ ValidationUtils.checkArgument(!partition.getPartitionPath().contains(CONFIG_VALUES_DELIMITER),
+ "Metadata Table partition path cannot contain a comma: " + partition.getPartitionPath());
+ Set<String> partitions = getMetadataPartitions();
+ Set<String> partitionsInflight = getMetadataPartitionsInflight();
+ if (enabled) {
+ partitions.add(partition.getPartitionPath());
+ partitionsInflight.remove(partition.getPartitionPath());
+ } else if (partition.equals(MetadataPartitionType.FILES)) {
+ // file listing partition is required for all other partitions to work
+ // Disabling file partition will also disable all partitions
+ partitions.clear();
+ partitionsInflight.clear();
+ } else {
+ partitions.remove(partition.getPartitionPath());
+ partitionsInflight.remove(partition.getPartitionPath());
+ }
+ setValue(TABLE_METADATA_PARTITIONS, partitions.stream().sorted().collect(Collectors.joining(CONFIG_VALUES_DELIMITER)));
+ setValue(TABLE_METADATA_PARTITIONS_INFLIGHT, partitionsInflight.stream().sorted().collect(Collectors.joining(CONFIG_VALUES_DELIMITER)));
+ }
+
+ /**
+ * Enables the specified metadata table partition as inflight.
+ *
+ * @param partition The list of partitions to enable as inflight.
+ */
+ public void setMetadataPartitionsAsInflight(List<MetadataPartitionType> partitionTypes) {
+ Set<String> partitions = getMetadataPartitionsInflight();
Review Comment:
I don't see this being addressed ?
##########
hudi-common/src/main/java/org/apache/hudi/metadata/MetadataPartitionType.java:
##########
@@ -33,10 +33,6 @@ public enum MetadataPartitionType {
private final String partitionPath;
// FileId prefix used for all file groups in this partition.
private final String fileIdPrefix;
- // Total file groups
- // TODO fix: enum should not have any mutable aspect as this compromises whole idea
- // of the enum being static, immutable entity
- private int fileGroupCount = 1;
Review Comment:
thanks!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] nsivabalan commented on a diff in pull request #8684: [HUDI-6200] Enhancements to the MDT for improving performance of larger indexes.
Posted by "nsivabalan (via GitHub)" <gi...@apache.org>.
nsivabalan commented on code in PR #8684:
URL: https://github.com/apache/hudi/pull/8684#discussion_r1212748606
##########
hudi-client/hudi-spark-client/src/main/java/org/apache/hudi/client/SparkRDDWriteClient.java:
##########
@@ -338,14 +338,20 @@ protected void initMetadataTable(Option<String> instantTime) {
* @param inFlightInstantTimestamp - The in-flight action responsible for the metadata table initialization
*/
private void initializeMetadataTable(Option<String> inFlightInstantTimestamp) {
- if (config.isMetadataTableEnabled()) {
- HoodieTableMetadataWriter writer = SparkHoodieBackedTableMetadataWriter.create(context.getHadoopConf().get(), config,
- context, Option.empty(), inFlightInstantTimestamp);
- try {
- writer.close();
- } catch (Exception e) {
- throw new HoodieException("Failed to instantiate Metadata table ", e);
+ if (!config.isMetadataTableEnabled()) {
+ LOG.error("================================111111111111111111");
Review Comment:
looks like this needs to be fixed.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] danny0405 commented on a diff in pull request #8684: [HUDI-6200] Enhancements to the MDT for improving performance of larger indexes.
Posted by "danny0405 (via GitHub)" <gi...@apache.org>.
danny0405 commented on code in PR #8684:
URL: https://github.com/apache/hudi/pull/8684#discussion_r1192913868
##########
hudi-client/hudi-spark-client/src/main/java/org/apache/hudi/metadata/SparkHoodieBackedTableMetadataWriter.java:
##########
@@ -118,46 +123,32 @@ protected void initRegistry() {
}
@Override
- protected <T extends SpecificRecordBase> void initialize(HoodieEngineContext engineContext,
- Option<T> actionMetadata,
- Option<String> inflightInstantTimestamp) {
- try {
- metrics.map(HoodieMetadataMetrics::registry).ifPresent(registry -> {
- if (registry instanceof DistributedRegistry) {
- HoodieSparkEngineContext sparkEngineContext = (HoodieSparkEngineContext) engineContext;
- ((DistributedRegistry) registry).register(sparkEngineContext.getJavaSparkContext());
- }
- });
+ protected void commit(String instantTime, Map<MetadataPartitionType, HoodieData<HoodieRecord>> partitionRecordsMap) {
+ commitInternal(instantTime, partitionRecordsMap, Option.empty());
+ }
- if (enabled) {
- initializeIfNeeded(dataMetaClient, actionMetadata, inflightInstantTimestamp);
- }
- } catch (IOException e) {
- LOG.error("Failed to initialize metadata table. Disabling the writer.", e);
- enabled = false;
- }
+ protected void bulkCommit(
+ String instantTime, MetadataPartitionType partitionType, HoodieData<HoodieRecord> records,
+ int fileGroupCount) {
+ Map<MetadataPartitionType, HoodieData<HoodieRecord>> partitionRecordsMap = new HashMap<>();
+ partitionRecordsMap.put(partitionType, records);
+ SparkHoodieMetadataBulkInsertPartitioner partitioner = new SparkHoodieMetadataBulkInsertPartitioner(fileGroupCount);
Review Comment:
You are right, thanks for the clarification.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] prashantwason commented on a diff in pull request #8684: [HUDI-6200] Enhancements to the MDT for improving performance of larger indexes.
Posted by "prashantwason (via GitHub)" <gi...@apache.org>.
prashantwason commented on code in PR #8684:
URL: https://github.com/apache/hudi/pull/8684#discussion_r1192169835
##########
hudi-common/src/main/java/org/apache/hudi/metadata/HoodieTableMetadataUtil.java:
##########
@@ -1378,6 +1339,206 @@ public static Set<String> getInflightAndCompletedMetadataPartitions(HoodieTableC
*/
public static boolean isIndexingCommit(String instantTime) {
return instantTime.length() == MILLIS_INSTANT_ID_LENGTH + METADATA_INDEXER_TIME_SUFFIX.length()
- && instantTime.endsWith(METADATA_INDEXER_TIME_SUFFIX);
+ && instantTime.endsWith(METADATA_INDEXER_TIME_SUFFIX);
+ }
+
+ /**
+ * Delete the metadata table for the dataset and backup if required.
+ *
+ * @param dataMetaClient {@code HoodieTableMetaClient} of the dataset for which metadata table is to be deleted
+ * @param context instance of {@link HoodieEngineContext}.
+ * @param backup Whether metadata table should be backed up before deletion. If true, the table is backed up to the
+ * directory with name metadata_<current_timestamp>.
+ * @return The backup directory if backup was requested
+ */
+ public static String deleteMetadataTable(HoodieTableMetaClient dataMetaClient, HoodieEngineContext context, boolean backup) {
+ final Path metadataTablePath = HoodieTableMetadata.getMetadataTableBasePath(dataMetaClient.getBasePathV2());
+ FileSystem fs = FSUtils.getFs(metadataTablePath.toString(), context.getHadoopConf().get());
+ setMetadataPartitionState(dataMetaClient, MetadataPartitionType.FILES, false);
+ try {
+ if (!fs.exists(metadataTablePath)) {
+ return null;
+ }
+ } catch (FileNotFoundException e) {
+ // Ignoring exception as metadata table already does not exist
+ return null;
+ } catch (IOException e) {
+ throw new HoodieMetadataException("Failed to check metadata table existence", e);
+ }
+
+ if (backup) {
+ final Path metadataBackupPath = new Path(metadataTablePath.getParent(), ".metadata_" + HoodieActiveTimeline.createNewInstantTime());
+ LOG.info("Backing up metadata directory to " + metadataBackupPath + " before deletion");
+ try {
+ if (fs.rename(metadataTablePath, metadataBackupPath)) {
+ return metadataBackupPath.toString();
+ }
+ } catch (Exception e) {
+ // If rename fails, we will ignore the backup and still delete the MDT
+ LOG.error("Failed to backup metadata table using rename", e);
+ }
+ }
+
+ LOG.info("Deleting metadata table from " + metadataTablePath);
+ try {
+ fs.delete(metadataTablePath, true);
+ } catch (Exception e) {
+ throw new HoodieMetadataException("Failed to delete metadata table from path " + metadataTablePath, e);
+ }
+
+ return null;
+ }
+
+ public static HoodieTableMetaClient setMetadataPartitionState(HoodieTableMetaClient dataMetaClient, MetadataPartitionType partitionType, boolean enabled) {
+ dataMetaClient.getTableConfig().setMetadataPartitionState(partitionType, enabled);
+ HoodieTableConfig.update(dataMetaClient.getFs(), new Path(dataMetaClient.getMetaPath()), dataMetaClient.getTableConfig().getProps());
+ dataMetaClient = HoodieTableMetaClient.reload(dataMetaClient);
+ ValidationUtils.checkState(dataMetaClient.getTableConfig().isMetadataPartitionEnabled(partitionType) == enabled,
+ "Metadata table state change should be persisted");
+
+ LOG.info(String.format("Metadata table %s partition %s has been %s", dataMetaClient.getBasePathV2(), partitionType,
+ enabled ? "enabled" : "disabled"));
+ return dataMetaClient;
+ }
+
+ /**
+ * Delete a partition within the metadata table.
+ * <p>
+ * This can be used to delete a partition so that it can be re-bootstrapped.
+ *
+ * @param dataMetaClient {@code HoodieTableMetaClient} of the dataset for which metadata table is to be deleted
+ * @param context instance of {@code HoodieEngineContext}.
+ * @param backup Whether metadata table should be backed up before deletion. If true, the table is backed up to the
+ * directory with name metadata_<current_timestamp>.
+ * @param partitionType The partition to delete
+ * @return The backup directory if backup was requested, null otherwise
+ */
+ public static String deleteMetadataTablePartition(HoodieTableMetaClient dataMetaClient, HoodieEngineContext context,
+ MetadataPartitionType partitionType, boolean backup) {
+ if (partitionType.equals(MetadataPartitionType.FILES)) {
+ return deleteMetadataTable(dataMetaClient, context, backup);
+ }
+
+ final Path metadataTablePartitionPath = new Path(HoodieTableMetadata.getMetadataTableBasePath(dataMetaClient.getBasePath()), partitionType.getPartitionPath());
+ FileSystem fs = FSUtils.getFs(metadataTablePartitionPath.toString(), context.getHadoopConf().get());
+ setMetadataPartitionState(dataMetaClient, partitionType, false);
+ try {
+ if (!fs.exists(metadataTablePartitionPath)) {
+ return null;
+ }
+ } catch (FileNotFoundException e) {
+ // Ignoring exception as metadata table already does not exist
+ LOG.debug("Metadata table partition " + partitionType + " not found at path " + metadataTablePartitionPath);
+ return null;
+ } catch (Exception e) {
+ throw new HoodieMetadataException("Failed to check metadata table partition existence", e);
Review Comment:
Added
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] nsivabalan commented on a diff in pull request #8684: [HUDI-6200] Enhancements to the MDT for improving performance of larger indexes.
Posted by "nsivabalan (via GitHub)" <gi...@apache.org>.
nsivabalan commented on code in PR #8684:
URL: https://github.com/apache/hudi/pull/8684#discussion_r1192722934
##########
hudi-client/hudi-spark-client/src/main/java/org/apache/hudi/metadata/SparkHoodieMetadataBulkInsertPartitioner.java:
##########
@@ -0,0 +1,109 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hudi.metadata;
+
+import java.io.Serializable;
+import java.util.ArrayList;
+import java.util.Comparator;
+import java.util.List;
+
+import org.apache.hudi.common.model.HoodieRecord;
+import org.apache.hudi.common.util.ValidationUtils;
+import org.apache.hudi.table.BulkInsertPartitioner;
+import org.apache.spark.Partitioner;
+import org.apache.spark.api.java.JavaRDD;
+
+import scala.Tuple2;
+
+/**
+ * A {@code BulkInsertPartitioner} implementation for Metadata Table to improve performance of initialization of metadata
+ * table partition when a very large number of records are inserted.
+ *
+ * This partitioner requires the records to be already tagged with location.
+ */
+public class SparkHoodieMetadataBulkInsertPartitioner implements BulkInsertPartitioner<JavaRDD<HoodieRecord>> {
+ final int numPartitions;
+ public SparkHoodieMetadataBulkInsertPartitioner(int numPartitions) {
+ this.numPartitions = numPartitions;
+ }
+
+ private class FileGroupPartitioner extends Partitioner {
+
+ @Override
+ public int getPartition(Object key) {
+ return ((Tuple2<Integer, String>)key)._1;
+ }
+
+ @Override
+ public int numPartitions() {
+ return numPartitions;
+ }
+ }
+
+ // FileIDs for the various partitions
+ private List<String> fileIDPfxs;
+
+ /**
+ * Partition the records by their location. The number of partitions is determined by the number of MDT fileGroups being udpated rather than the
+ * specific value of outputSparkPartitions.
+ */
+ @Override
+ public JavaRDD<HoodieRecord> repartitionRecords(JavaRDD<HoodieRecord> records, int outputSparkPartitions) {
+ Comparator<Tuple2<Integer, String>> keyComparator =
+ (Comparator<Tuple2<Integer, String>> & Serializable)(t1, t2) -> t1._2.compareTo(t2._2);
+
+ // Partition the records by their file group
+ JavaRDD<HoodieRecord> partitionedRDD = records
+ // key by <file group index, record key>. The file group index is used to partition and the record key is used to sort within the partition.
+ .keyBy(r -> {
+ int fileGroupIndex = HoodieTableMetadataUtil.getFileGroupIndexFromFileId(r.getCurrentLocation().getFileId());
+ return new Tuple2<>(fileGroupIndex, r.getRecordKey());
+ })
+ .repartitionAndSortWithinPartitions(new FileGroupPartitioner(), keyComparator)
+ .map(t -> t._2);
+
+ fileIDPfxs = partitionedRDD.mapPartitions(recordItr -> {
+ // Due to partitioning, all record in the partition should have same fileID. So we only can get the fileID prefix from the first record.
+ List<String> fileIds = new ArrayList<>(1);
+ if (recordItr.hasNext()) {
+ HoodieRecord record = recordItr.next();
+ final String fileID = HoodieTableMetadataUtil.getFileGroupPrefix(record.getCurrentLocation().getFileId());
+ fileIds.add(fileID);
+ } else {
+ // Empty partition
+ fileIds.add("");
Review Comment:
thanks! but can u help me understand something.
for these file groups where we return EMPTY_STRING ("") as fileID prefix, does that mean our bulk_commit will create a file group w/ empty records ? Also, if there are more than 1 fileID with emtpy string as fileId prefix, wouldn't to spark tasks write to same file group ?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] nsivabalan commented on a diff in pull request #8684: [HUDI-6200] Enhancements to the MDT for improving performance of larger indexes.
Posted by "nsivabalan (via GitHub)" <gi...@apache.org>.
nsivabalan commented on code in PR #8684:
URL: https://github.com/apache/hudi/pull/8684#discussion_r1192728311
##########
hudi-client/hudi-spark-client/src/test/java/org/apache/hudi/client/functional/TestHoodieBackedMetadata.java:
##########
@@ -2482,6 +2483,14 @@ public void testMetadataMetrics() throws Exception {
}
}
+ @Test
+ public void testGetFileGroupIndexFromFileId() {
Review Comment:
I meant for all similar static methods
getFileIDForFileGroup
getFileGroupIndexFromFileId
getFileGroupPrefix
getFileIdLengthWithoutFileIndex
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] prashantwason commented on a diff in pull request #8684: [HUDI-6200] Enhancements to the MDT for improving performance of larger indexes.
Posted by "prashantwason (via GitHub)" <gi...@apache.org>.
prashantwason commented on code in PR #8684:
URL: https://github.com/apache/hudi/pull/8684#discussion_r1192166826
##########
hudi-client/hudi-client-common/src/main/java/org/apache/hudi/metadata/HoodieBackedTableMetadataWriter.java:
##########
@@ -873,17 +908,7 @@ public void buildMetadataPartitions(HoodieEngineContext engineContext, List<Hood
indexPartitionInfos.forEach(indexPartitionInfo -> {
String relativePartitionPath = indexPartitionInfo.getMetadataPartitionPath();
LOG.info(String.format("Creating a new metadata index for partition '%s' under path %s upto instant %s",
- relativePartitionPath, metadataWriteConfig.getBasePath(), indexUptoInstantTime));
- try {
- // file group should have already been initialized while scheduling index for this partition
- if (!dataMetaClient.getFs().exists(new Path(metadataWriteConfig.getBasePath(), relativePartitionPath))) {
Review Comment:
HoodieBackedTableMetadataWriter::initializeFromFilesystem
There is a call to initializeFileGroups()
Within initializeFileGroups() existing fileSlices should be deleted (I filed a separate PR for this - the one where the fileSlices are created in parallel).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #8684: [HUDI-6200] Enhancements to the MDT for improving performance of larger indexes.
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #8684:
URL: https://github.com/apache/hudi/pull/8684#issuecomment-1548638584
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "35148aeb4ba78eb6f3316c75f8a0a7e4c6d6df87",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17007",
"triggerID" : "35148aeb4ba78eb6f3316c75f8a0a7e4c6d6df87",
"triggerType" : "PUSH"
}, {
"hash" : "e2785f4675ddf74582ff34590608a5d71c5e9a2d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17008",
"triggerID" : "e2785f4675ddf74582ff34590608a5d71c5e9a2d",
"triggerType" : "PUSH"
}, {
"hash" : "364b1fe0dde303750ca0f8eebe5cd530c2a66a3a",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17042",
"triggerID" : "364b1fe0dde303750ca0f8eebe5cd530c2a66a3a",
"triggerType" : "PUSH"
}, {
"hash" : "9d73b9664eed48ada68721a92526640cfa06a021",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17098",
"triggerID" : "9d73b9664eed48ada68721a92526640cfa06a021",
"triggerType" : "PUSH"
}, {
"hash" : "7be538f4045a42ba33ab8fe62594178e2db75bbb",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "7be538f4045a42ba33ab8fe62594178e2db75bbb",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 364b1fe0dde303750ca0f8eebe5cd530c2a66a3a Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17042)
* 9d73b9664eed48ada68721a92526640cfa06a021 Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17098)
* 7be538f4045a42ba33ab8fe62594178e2db75bbb UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #8684: [HUDI-6200] Enhancements to the MDT for improving performance of larger indexes.
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #8684:
URL: https://github.com/apache/hudi/pull/8684#issuecomment-1548650220
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "35148aeb4ba78eb6f3316c75f8a0a7e4c6d6df87",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17007",
"triggerID" : "35148aeb4ba78eb6f3316c75f8a0a7e4c6d6df87",
"triggerType" : "PUSH"
}, {
"hash" : "e2785f4675ddf74582ff34590608a5d71c5e9a2d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17008",
"triggerID" : "e2785f4675ddf74582ff34590608a5d71c5e9a2d",
"triggerType" : "PUSH"
}, {
"hash" : "364b1fe0dde303750ca0f8eebe5cd530c2a66a3a",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17042",
"triggerID" : "364b1fe0dde303750ca0f8eebe5cd530c2a66a3a",
"triggerType" : "PUSH"
}, {
"hash" : "9d73b9664eed48ada68721a92526640cfa06a021",
"status" : "CANCELED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17098",
"triggerID" : "9d73b9664eed48ada68721a92526640cfa06a021",
"triggerType" : "PUSH"
}, {
"hash" : "7be538f4045a42ba33ab8fe62594178e2db75bbb",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17100",
"triggerID" : "7be538f4045a42ba33ab8fe62594178e2db75bbb",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 9d73b9664eed48ada68721a92526640cfa06a021 Azure: [CANCELED](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17098)
* 7be538f4045a42ba33ab8fe62594178e2db75bbb Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17100)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] prashantwason commented on a diff in pull request #8684: [HUDI-6200] Enhancements to the MDT for improving performance of larger indexes.
Posted by "prashantwason (via GitHub)" <gi...@apache.org>.
prashantwason commented on code in PR #8684:
URL: https://github.com/apache/hudi/pull/8684#discussion_r1192137497
##########
hudi-client/hudi-spark-client/src/main/java/org/apache/hudi/metadata/SparkHoodieMetadataBulkInsertPartitioner.java:
##########
@@ -0,0 +1,109 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hudi.metadata;
+
+import java.io.Serializable;
+import java.util.ArrayList;
+import java.util.Comparator;
+import java.util.List;
+
+import org.apache.hudi.common.model.HoodieRecord;
+import org.apache.hudi.common.util.ValidationUtils;
+import org.apache.hudi.table.BulkInsertPartitioner;
+import org.apache.spark.Partitioner;
+import org.apache.spark.api.java.JavaRDD;
+
+import scala.Tuple2;
+
+/**
+ * A {@code BulkInsertPartitioner} implementation for Metadata Table to improve performance of initialization of metadata
+ * table partition when a very large number of records are inserted.
+ *
+ * This partitioner requires the records to be already tagged with location.
+ */
+public class SparkHoodieMetadataBulkInsertPartitioner implements BulkInsertPartitioner<JavaRDD<HoodieRecord>> {
+ final int numPartitions;
+ public SparkHoodieMetadataBulkInsertPartitioner(int numPartitions) {
+ this.numPartitions = numPartitions;
+ }
+
+ private class FileGroupPartitioner extends Partitioner {
+
+ @Override
+ public int getPartition(Object key) {
+ return ((Tuple2<Integer, String>)key)._1;
+ }
+
+ @Override
+ public int numPartitions() {
+ return numPartitions;
+ }
+ }
+
+ // FileIDs for the various partitions
+ private List<String> fileIDPfxs;
+
+ /**
+ * Partition the records by their location. The number of partitions is determined by the number of MDT fileGroups being udpated rather than the
+ * specific value of outputSparkPartitions.
+ */
+ @Override
+ public JavaRDD<HoodieRecord> repartitionRecords(JavaRDD<HoodieRecord> records, int outputSparkPartitions) {
+ Comparator<Tuple2<Integer, String>> keyComparator =
+ (Comparator<Tuple2<Integer, String>> & Serializable)(t1, t2) -> t1._2.compareTo(t2._2);
+
+ // Partition the records by their file group
+ JavaRDD<HoodieRecord> partitionedRDD = records
+ // key by <file group index, record key>. The file group index is used to partition and the record key is used to sort within the partition.
+ .keyBy(r -> {
+ int fileGroupIndex = HoodieTableMetadataUtil.getFileGroupIndexFromFileId(r.getCurrentLocation().getFileId());
+ return new Tuple2<>(fileGroupIndex, r.getRecordKey());
+ })
+ .repartitionAndSortWithinPartitions(new FileGroupPartitioner(), keyComparator)
+ .map(t -> t._2);
+
+ fileIDPfxs = partitionedRDD.mapPartitions(recordItr -> {
+ // Due to partitioning, all record in the partition should have same fileID. So we only can get the fileID prefix from the first record.
+ List<String> fileIds = new ArrayList<>(1);
+ if (recordItr.hasNext()) {
+ HoodieRecord record = recordItr.next();
+ final String fileID = HoodieTableMetadataUtil.getFileGroupPrefix(record.getCurrentLocation().getFileId());
+ fileIds.add(fileID);
+ } else {
+ // Empty partition
+ fileIds.add("");
Review Comment:
Added doc
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] prashantwason commented on a diff in pull request #8684: [HUDI-6200] Enhancements to the MDT for improving performance of larger indexes.
Posted by "prashantwason (via GitHub)" <gi...@apache.org>.
prashantwason commented on code in PR #8684:
URL: https://github.com/apache/hudi/pull/8684#discussion_r1192140304
##########
hudi-client/hudi-client-common/src/main/java/org/apache/hudi/table/action/index/ScheduleIndexActionExecutor.java:
##########
@@ -100,15 +99,6 @@ public Option<HoodieIndexPlan> execute() {
// get last completed instant
Option<HoodieInstant> indexUptoInstant = table.getActiveTimeline().getContiguousCompletedWriteTimeline().lastInstant();
if (indexUptoInstant.isPresent()) {
- // start initializing file groups
- // in case FILES partition itself was not initialized before (i.e. metadata was never enabled), this will initialize synchronously
- HoodieTableMetadataWriter metadataWriter = table.getMetadataWriter(instantTime)
- .orElseThrow(() -> new HoodieIndexException(String.format("Could not get metadata writer to initialize filegroups for indexing for instant: %s", instantTime)));
- if (!finalPartitionsToIndex.get(0).getPartitionPath().equals(MetadataPartitionType.FILES.getPartitionPath())) {
- // initialize metadata partition only if not for FILES partition.
- metadataWriter.initializeMetadataPartitions(table.getMetaClient(), finalPartitionsToIndex, indexUptoInstant.get().getTimestamp());
Review Comment:
Initialization of file groups happens just before the bulkInsert in HoodieBackedTableMetadataWriter::initializeFromFilesystem
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #8684: [HUDI-6200] Enhancements to the MDT for improving performance of larger indexes.
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #8684:
URL: https://github.com/apache/hudi/pull/8684#issuecomment-1581597271
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "35148aeb4ba78eb6f3316c75f8a0a7e4c6d6df87",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17007",
"triggerID" : "35148aeb4ba78eb6f3316c75f8a0a7e4c6d6df87",
"triggerType" : "PUSH"
}, {
"hash" : "e2785f4675ddf74582ff34590608a5d71c5e9a2d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17008",
"triggerID" : "e2785f4675ddf74582ff34590608a5d71c5e9a2d",
"triggerType" : "PUSH"
}, {
"hash" : "364b1fe0dde303750ca0f8eebe5cd530c2a66a3a",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17042",
"triggerID" : "364b1fe0dde303750ca0f8eebe5cd530c2a66a3a",
"triggerType" : "PUSH"
}, {
"hash" : "9d73b9664eed48ada68721a92526640cfa06a021",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17098",
"triggerID" : "9d73b9664eed48ada68721a92526640cfa06a021",
"triggerType" : "PUSH"
}, {
"hash" : "7be538f4045a42ba33ab8fe62594178e2db75bbb",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17100",
"triggerID" : "7be538f4045a42ba33ab8fe62594178e2db75bbb",
"triggerType" : "PUSH"
}, {
"hash" : "cc0da2372d50d99c98c2ce4bcbe5a60303bde938",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17195",
"triggerID" : "cc0da2372d50d99c98c2ce4bcbe5a60303bde938",
"triggerType" : "PUSH"
}, {
"hash" : "da0e00e3655e06b064d11e20500317e2a1d2229c",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "da0e00e3655e06b064d11e20500317e2a1d2229c",
"triggerType" : "PUSH"
}, {
"hash" : "589c8cd7ffc5df7984c099d6b30fa117463db2b9",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17335",
"triggerID" : "589c8cd7ffc5df7984c099d6b30fa117463db2b9",
"triggerType" : "PUSH"
}, {
"hash" : "226c96ce48b33d7b1650d00e93bc5ea42ad276f6",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17624",
"triggerID" : "226c96ce48b33d7b1650d00e93bc5ea42ad276f6",
"triggerType" : "PUSH"
}, {
"hash" : "ad81a08799a98cd6ee63b8bf792d42f16a98f775",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17634",
"triggerID" : "ad81a08799a98cd6ee63b8bf792d42f16a98f775",
"triggerType" : "PUSH"
}, {
"hash" : "fd2ec462e554190843ddcc8d946317ae76cb81e1",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "fd2ec462e554190843ddcc8d946317ae76cb81e1",
"triggerType" : "PUSH"
}, {
"hash" : "cae57c2b03a3131cfa7774fa4335f4d153f85d61",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17644",
"triggerID" : "cae57c2b03a3131cfa7774fa4335f4d153f85d61",
"triggerType" : "PUSH"
}, {
"hash" : "a21fb8753ed206fefcf598c9e134c280e90d3dba",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17654",
"triggerID" : "a21fb8753ed206fefcf598c9e134c280e90d3dba",
"triggerType" : "PUSH"
}, {
"hash" : "4dfaefde334b712b3ba272bb330242ddb7f4c0ac",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17655",
"triggerID" : "4dfaefde334b712b3ba272bb330242ddb7f4c0ac",
"triggerType" : "PUSH"
}, {
"hash" : "ca59a379b830f88fd72c175cb9b96a14b33550c0",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17660",
"triggerID" : "ca59a379b830f88fd72c175cb9b96a14b33550c0",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* da0e00e3655e06b064d11e20500317e2a1d2229c UNKNOWN
* fd2ec462e554190843ddcc8d946317ae76cb81e1 UNKNOWN
* 4dfaefde334b712b3ba272bb330242ddb7f4c0ac Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17655)
* ca59a379b830f88fd72c175cb9b96a14b33550c0 Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17660)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] codope commented on a diff in pull request #8684: [HUDI-6200] Enhancements to the MDT for improving performance of larger indexes.
Posted by "codope (via GitHub)" <gi...@apache.org>.
codope commented on code in PR #8684:
URL: https://github.com/apache/hudi/pull/8684#discussion_r1222567953
##########
hudi-client/hudi-flink-client/src/main/java/org/apache/hudi/metadata/FlinkHoodieBackedTableMetadataWriter.java:
##########
@@ -104,40 +106,19 @@ protected void initRegistry() {
}
@Override
- protected <T extends SpecificRecordBase> void initialize(HoodieEngineContext engineContext,
- Option<T> actionMetadata,
- Option<String> inflightInstantTimestamp) {
- try {
- if (enabled) {
- initializeIfNeeded(dataMetaClient, actionMetadata, inflightInstantTimestamp);
- }
- } catch (IOException e) {
- LOG.error("Failed to initialize metadata table. Disabling the writer.", e);
- enabled = false;
- }
- }
-
- @Override
- protected void commit(String instantTime, Map<MetadataPartitionType, HoodieData<HoodieRecord>> partitionRecordsMap,
- boolean canTriggerTableService) {
- ValidationUtils.checkState(enabled, "Metadata table cannot be committed to as it is not enabled");
+ protected void commit(String instantTime, Map<MetadataPartitionType, HoodieData<HoodieRecord>> partitionRecordsMap) {
ValidationUtils.checkState(metadataMetaClient != null, "Metadata table is not fully initialized yet.");
HoodieData<HoodieRecord> preppedRecords = prepRecords(partitionRecordsMap);
List<HoodieRecord> preppedRecordList = preppedRecords.collectAsList();
- try (HoodieFlinkWriteClient writeClient = new HoodieFlinkWriteClient(engineContext, metadataWriteConfig)) {
+ // Flink engine does not optimize initialCommit to MDT as bulk insert is not yet supported
+
+ try (HoodieFlinkWriteClient writeClient = (HoodieFlinkWriteClient) getWriteClient()) {
// rollback partially failed writes if any.
if (writeClient.rollbackFailedWrites()) {
metadataMetaClient = HoodieTableMetaClient.reload(metadataMetaClient);
}
- if (canTriggerTableService) {
- // trigger compaction before doing the delta commit. this is to ensure, if this delta commit succeeds in metadata table, but failed in data table,
- // we would have compacted metadata table and so could have included uncommitted data which will never be ignored while reading from metadata
- // table (since reader will filter out only from delta commits)
- compactIfNecessary(writeClient, instantTime);
- }
Review Comment:
why is this removed? Don't we want compaction to run?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] nsivabalan commented on a diff in pull request #8684: [HUDI-6200] Enhancements to the MDT for improving performance of larger indexes.
Posted by "nsivabalan (via GitHub)" <gi...@apache.org>.
nsivabalan commented on code in PR #8684:
URL: https://github.com/apache/hudi/pull/8684#discussion_r1220135253
##########
hudi-client/hudi-client-common/src/main/java/org/apache/hudi/metadata/HoodieBackedTableMetadataWriter.java:
##########
@@ -1138,7 +1120,11 @@ public DirectoryInfo(String relativePath, FileStatus[] fileStatus) {
this.isHoodiePartition = true;
} else if (FSUtils.isDataFile(status.getPath())) {
// Regular HUDI data file (base file or log file)
- filenameToSizeMap.put(status.getPath().getName(), status.getLen());
+ String dataFileCommitTime = FSUtils.getCommitTime(status.getPath().getName());
Review Comment:
Incase of MOR table, for a log file, the base instance time could be < actual delta commit time. So, we might skip the log files based on this logic?
since in L1125, we are filtering for files whose commit time < max instant time. May be we should use last mod time instead?
##########
hudi-client/hudi-spark-client/src/main/java/org/apache/hudi/client/SparkRDDWriteClient.java:
##########
@@ -337,30 +337,15 @@ protected void initMetadataTable(WriteOperationType operationType, Option<String
*
* @param inFlightInstantTimestamp - The in-flight action responsible for the metadata table initialization
*/
- private void initializeMetadataTable(WriteOperationType operationType, Option<String> inFlightInstantTimestamp) {
+ private void initializeMetadataTable(Option<String> inFlightInstantTimestamp) {
if (!config.isMetadataTableEnabled()) {
return;
}
try (HoodieTableMetadataWriter writer = SparkHoodieBackedTableMetadataWriter.create(context.getHadoopConf().get(), config,
- context, Option.empty(), inFlightInstantTimestamp)) {
+ context, Option.empty(), inFlightInstantTimestamp)) {
if (writer.isInitialized()) {
Review Comment:
We added the guard rail for table services in MDT to be triggered only by regular writers in Data table, so that for a single writer modes with async table services, there won't be any race conditions.
Ref: https://github.com/apache/hudi/pull/3900
But the code evolved and we automatically enable in process lock provider for single writer mode w/ async table services. And so, we should be good to remove the constraint. Just that we might keep triggering the schedule of compaction in MDT everytime. May be we can intercept from the active timeline on when is the last time, compaction was triggered and add in some optimization.
Nothing required for this patch. But as a follow up.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] danny0405 commented on pull request #8684: [HUDI-6200] Enhancements to the MDT for improving performance of larger indexes.
Posted by "danny0405 (via GitHub)" <gi...@apache.org>.
danny0405 commented on PR #8684:
URL: https://github.com/apache/hudi/pull/8684#issuecomment-1578236922
@prashantwason Did you notice that there are conflicts in the codes?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] danny0405 commented on a diff in pull request #8684: [HUDI-6200] Enhancements to the MDT for improving performance of larger indexes.
Posted by "danny0405 (via GitHub)" <gi...@apache.org>.
danny0405 commented on code in PR #8684:
URL: https://github.com/apache/hudi/pull/8684#discussion_r1203913275
##########
hudi-client/hudi-spark-client/src/main/java/org/apache/hudi/client/SparkRDDWriteClient.java:
##########
@@ -338,14 +338,20 @@ protected void initMetadataTable(Option<String> instantTime) {
* @param inFlightInstantTimestamp - The in-flight action responsible for the metadata table initialization
*/
private void initializeMetadataTable(Option<String> inFlightInstantTimestamp) {
- if (config.isMetadataTableEnabled()) {
- HoodieTableMetadataWriter writer = SparkHoodieBackedTableMetadataWriter.create(context.getHadoopConf().get(), config,
- context, Option.empty(), inFlightInstantTimestamp);
- try {
- writer.close();
- } catch (Exception e) {
- throw new HoodieException("Failed to instantiate Metadata table ", e);
+ if (!config.isMetadataTableEnabled()) {
+ LOG.error("================================111111111111111111");
Review Comment:
Why a meaningless log?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] danny0405 commented on a diff in pull request #8684: [HUDI-6200] Enhancements to the MDT for improving performance of larger indexes.
Posted by "danny0405 (via GitHub)" <gi...@apache.org>.
danny0405 commented on code in PR #8684:
URL: https://github.com/apache/hudi/pull/8684#discussion_r1227895765
##########
hudi-client/hudi-spark-client/src/main/java/org/apache/hudi/metadata/SparkHoodieMetadataBulkInsertPartitioner.java:
##########
@@ -0,0 +1,111 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hudi.metadata;
+
+import java.io.Serializable;
+import java.util.ArrayList;
+import java.util.Comparator;
+import java.util.List;
+
+import org.apache.hudi.common.model.HoodieRecord;
+import org.apache.hudi.common.util.ValidationUtils;
+import org.apache.hudi.table.BulkInsertPartitioner;
+import org.apache.spark.Partitioner;
+import org.apache.spark.api.java.JavaRDD;
+
+import scala.Tuple2;
+
+/**
+ * A {@code BulkInsertPartitioner} implementation for Metadata Table to improve performance of initialization of metadata
+ * table partition when a very large number of records are inserted.
+ *
+ * This partitioner requires the records to be already tagged with location.
+ */
+public class SparkHoodieMetadataBulkInsertPartitioner implements BulkInsertPartitioner<JavaRDD<HoodieRecord>> {
+ final int numPartitions;
+ public SparkHoodieMetadataBulkInsertPartitioner(int numPartitions) {
+ this.numPartitions = numPartitions;
+ }
+
+ private class FileGroupPartitioner extends Partitioner {
+
+ @Override
+ public int getPartition(Object key) {
+ return ((Tuple2<Integer, String>)key)._1;
+ }
+
+ @Override
+ public int numPartitions() {
+ return numPartitions;
+ }
+ }
+
+ // FileIDs for the various partitions
+ private List<String> fileIDPfxs;
+
+ /**
+ * Partition the records by their location. The number of partitions is determined by the number of MDT fileGroups being udpated rather than the
+ * specific value of outputSparkPartitions.
+ */
+ @Override
+ public JavaRDD<HoodieRecord> repartitionRecords(JavaRDD<HoodieRecord> records, int outputSparkPartitions) {
+ Comparator<Tuple2<Integer, String>> keyComparator =
+ (Comparator<Tuple2<Integer, String>> & Serializable)(t1, t2) -> t1._2.compareTo(t2._2);
+
+ // Partition the records by their file group
+ JavaRDD<HoodieRecord> partitionedRDD = records
+ // key by <file group index, record key>. The file group index is used to partition and the record key is used to sort within the partition.
+ .keyBy(r -> {
+ int fileGroupIndex = HoodieTableMetadataUtil.getFileGroupIndexFromFileId(r.getCurrentLocation().getFileId());
+ return new Tuple2<>(fileGroupIndex, r.getRecordKey());
+ })
+ .repartitionAndSortWithinPartitions(new FileGroupPartitioner(), keyComparator)
+ .map(t -> t._2);
+
+ fileIDPfxs = partitionedRDD.mapPartitions(recordItr -> {
+ // Due to partitioning, all record in the partition should have same fileID. So we only can get the fileID prefix from the first record.
+ List<String> fileIds = new ArrayList<>(1);
+ if (recordItr.hasNext()) {
+ HoodieRecord record = recordItr.next();
+ final String fileID = HoodieTableMetadataUtil.getFileGroupPrefix(record.getCurrentLocation().getFileId());
+ fileIds.add(fileID);
+ } else {
+ // FileGroupPartitioner returns a fixed number of partition as part of numPartitions(). In the special case that recordsRDD has fewer
+ // records than fileGroupCount, some of these partitions (corresponding to fileGroups) will not have any data.
+ // But we still need to return a fileID for use within {@code BulkInsertMapFunction}
+ fileIds.add("");
+ }
+ return fileIds.iterator();
+ }, true).collect();
+ ValidationUtils.checkArgument(partitionedRDD.getNumPartitions() == fileIDPfxs.size(),
+ String.format("Generated fileIDPfxs (%d) are lesser in size than the partitions %d", fileIDPfxs.size(), partitionedRDD.getNumPartitions()));
+
+ return partitionedRDD;
+ }
+
+ @Override
+ public boolean arePartitionRecordsSorted() {
+ return true;
+ }
+
+ @Override
+ public String getFileIdPfx(int partitionId) {
+ return fileIDPfxs.get(partitionId);
Review Comment:
> automatic estimation of the shard counts for each partition, that can be enhanced
This may be a solution if we can make accurate estimation of the file group size.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] danny0405 commented on a diff in pull request #8684: [HUDI-6200] Enhancements to the MDT for improving performance of larger indexes.
Posted by "danny0405 (via GitHub)" <gi...@apache.org>.
danny0405 commented on code in PR #8684:
URL: https://github.com/apache/hudi/pull/8684#discussion_r1225702442
##########
hudi-client/hudi-spark-client/src/main/java/org/apache/hudi/metadata/SparkHoodieMetadataBulkInsertPartitioner.java:
##########
@@ -0,0 +1,111 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hudi.metadata;
+
+import java.io.Serializable;
+import java.util.ArrayList;
+import java.util.Comparator;
+import java.util.List;
+
+import org.apache.hudi.common.model.HoodieRecord;
+import org.apache.hudi.common.util.ValidationUtils;
+import org.apache.hudi.table.BulkInsertPartitioner;
+import org.apache.spark.Partitioner;
+import org.apache.spark.api.java.JavaRDD;
+
+import scala.Tuple2;
+
+/**
+ * A {@code BulkInsertPartitioner} implementation for Metadata Table to improve performance of initialization of metadata
+ * table partition when a very large number of records are inserted.
+ *
+ * This partitioner requires the records to be already tagged with location.
+ */
+public class SparkHoodieMetadataBulkInsertPartitioner implements BulkInsertPartitioner<JavaRDD<HoodieRecord>> {
+ final int numPartitions;
+ public SparkHoodieMetadataBulkInsertPartitioner(int numPartitions) {
+ this.numPartitions = numPartitions;
+ }
+
+ private class FileGroupPartitioner extends Partitioner {
+
+ @Override
+ public int getPartition(Object key) {
+ return ((Tuple2<Integer, String>)key)._1;
+ }
+
+ @Override
+ public int numPartitions() {
+ return numPartitions;
+ }
+ }
+
+ // FileIDs for the various partitions
+ private List<String> fileIDPfxs;
+
+ /**
+ * Partition the records by their location. The number of partitions is determined by the number of MDT fileGroups being udpated rather than the
+ * specific value of outputSparkPartitions.
+ */
+ @Override
+ public JavaRDD<HoodieRecord> repartitionRecords(JavaRDD<HoodieRecord> records, int outputSparkPartitions) {
+ Comparator<Tuple2<Integer, String>> keyComparator =
+ (Comparator<Tuple2<Integer, String>> & Serializable)(t1, t2) -> t1._2.compareTo(t2._2);
+
+ // Partition the records by their file group
+ JavaRDD<HoodieRecord> partitionedRDD = records
+ // key by <file group index, record key>. The file group index is used to partition and the record key is used to sort within the partition.
+ .keyBy(r -> {
+ int fileGroupIndex = HoodieTableMetadataUtil.getFileGroupIndexFromFileId(r.getCurrentLocation().getFileId());
+ return new Tuple2<>(fileGroupIndex, r.getRecordKey());
+ })
+ .repartitionAndSortWithinPartitions(new FileGroupPartitioner(), keyComparator)
+ .map(t -> t._2);
+
+ fileIDPfxs = partitionedRDD.mapPartitions(recordItr -> {
+ // Due to partitioning, all record in the partition should have same fileID. So we only can get the fileID prefix from the first record.
+ List<String> fileIds = new ArrayList<>(1);
+ if (recordItr.hasNext()) {
+ HoodieRecord record = recordItr.next();
+ final String fileID = HoodieTableMetadataUtil.getFileGroupPrefix(record.getCurrentLocation().getFileId());
+ fileIds.add(fileID);
+ } else {
+ // FileGroupPartitioner returns a fixed number of partition as part of numPartitions(). In the special case that recordsRDD has fewer
+ // records than fileGroupCount, some of these partitions (corresponding to fileGroups) will not have any data.
+ // But we still need to return a fileID for use within {@code BulkInsertMapFunction}
+ fileIds.add("");
+ }
+ return fileIds.iterator();
+ }, true).collect();
+ ValidationUtils.checkArgument(partitionedRDD.getNumPartitions() == fileIDPfxs.size(),
+ String.format("Generated fileIDPfxs (%d) are lesser in size than the partitions %d", fileIDPfxs.size(), partitionedRDD.getNumPartitions()));
+
+ return partitionedRDD;
+ }
+
+ @Override
+ public boolean arePartitionRecordsSorted() {
+ return true;
+ }
+
+ @Override
+ public String getFileIdPfx(int partitionId) {
+ return fileIDPfxs.get(partitionId);
Review Comment:
No one can have awareness the file group number is pertinent with the correctness. It is a bug, not a usability issue.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] nsivabalan commented on a diff in pull request #8684: [HUDI-6200] Enhancements to the MDT for improving performance of larger indexes.
Posted by "nsivabalan (via GitHub)" <gi...@apache.org>.
nsivabalan commented on code in PR #8684:
URL: https://github.com/apache/hudi/pull/8684#discussion_r1220170726
##########
hudi-client/hudi-client-common/src/main/java/org/apache/hudi/metadata/HoodieBackedTableMetadataWriter.java:
##########
@@ -1052,51 +1091,81 @@ protected HoodieData<HoodieRecord> prepRecords(Map<MetadataPartitionType,
}
/**
- * Perform a compaction on the Metadata Table.
- *
- * Cases to be handled:
- * 1. We cannot perform compaction if there are previous inflight operations on the dataset. This is because
- * a compacted metadata base file at time Tx should represent all the actions on the dataset till time Tx.
- *
- * 2. In multi-writer scenario, a parallel operation with a greater instantTime may have completed creating a
- * deltacommit.
+ * Optimize the metadata table by running compaction, clean and archive as required.
+ * <p>
+ * Don't perform optimization if there are inflight operations on the dataset. This is for two reasons:
+ * - The compaction will contain the correct data as all failed operations have been rolled back.
+ * - Clean/compaction etc. will have the highest timestamp on the MDT and we won't be adding new operations
+ * with smaller timestamps to metadata table (makes for easier debugging)
+ * <p>
+ * This adds the limitations that long-running async operations (clustering, etc.) may cause delay in such MDT
+ * optimizations. We will relax this after MDT code has been hardened.
*/
- protected void compactIfNecessary(BaseHoodieWriteClient writeClient, String instantTime) {
- // finish off any pending compactions if any from previous attempt.
- writeClient.runAnyPendingCompactions();
-
- String latestDeltaCommitTimeInMetadataTable = metadataMetaClient.reloadActiveTimeline()
- .getDeltaCommitTimeline()
- .filterCompletedInstants()
- .lastInstant().orElseThrow(() -> new HoodieMetadataException("No completed deltacommit in metadata table"))
- .getTimestamp();
- // we need to find if there are any inflights in data table timeline before or equal to the latest delta commit in metadata table.
- // Whenever you want to change this logic, please ensure all below scenarios are considered.
- // a. There could be a chance that latest delta commit in MDT is committed in MDT, but failed in DT. And so findInstantsBeforeOrEquals() should be employed
- // b. There could be DT inflights after latest delta commit in MDT and we are ok with it. bcoz, the contract is, latest compaction instant time in MDT represents
- // any instants before that is already synced with metadata table.
- // c. Do consider out of order commits. For eg, c4 from DT could complete before c3. and we can't trigger compaction in MDT with c4 as base instant time, until every
- // instant before c4 is synced with metadata table.
- List<HoodieInstant> pendingInstants = dataMetaClient.reloadActiveTimeline().filterInflightsAndRequested()
- .findInstantsBeforeOrEquals(latestDeltaCommitTimeInMetadataTable).getInstants();
+ @Override
+ public void performTableServices(Option<String> inFlightInstantTimestamp) {
+ HoodieTimer metadataTableServicesTimer = HoodieTimer.start();
+ boolean allTableServicesExecutedSuccessfullyOrSkipped = true;
+ try {
+ BaseHoodieWriteClient writeClient = getWriteClient();
+ // Run any pending table services operations.
+ runPendingTableServicesOperations(writeClient);
+
+ // Check and run clean operations.
+ String latestDeltacommitTime = metadataMetaClient.reloadActiveTimeline().getDeltaCommitTimeline()
+ .filterCompletedInstants()
+ .lastInstant().get()
+ .getTimestamp();
+ LOG.info("Latest deltacommit time found is " + latestDeltacommitTime + ", running clean operations.");
+ cleanIfNecessary(writeClient, latestDeltacommitTime);
+
+ // Do timeline validation before scheduling compaction/logcompaction operations.
+ if (!validateTimelineBeforeSchedulingCompaction(inFlightInstantTimestamp, latestDeltacommitTime)) {
+ return;
Review Comment:
I see your point. but in reality, I feel we may not gain much. unless compaction in MDT kicks in, archival might not have anything to do after last time it was able to archive something. So, not a bad idea to not trigger archival if there is no compaction.
Same applies for clean as well. but lets not optimize anything in this patch.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #8684: [HUDI-6200] Enhancements to the MDT for improving performance of larger indexes.
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #8684:
URL: https://github.com/apache/hudi/pull/8684#issuecomment-1582114650
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "35148aeb4ba78eb6f3316c75f8a0a7e4c6d6df87",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17007",
"triggerID" : "35148aeb4ba78eb6f3316c75f8a0a7e4c6d6df87",
"triggerType" : "PUSH"
}, {
"hash" : "e2785f4675ddf74582ff34590608a5d71c5e9a2d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17008",
"triggerID" : "e2785f4675ddf74582ff34590608a5d71c5e9a2d",
"triggerType" : "PUSH"
}, {
"hash" : "364b1fe0dde303750ca0f8eebe5cd530c2a66a3a",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17042",
"triggerID" : "364b1fe0dde303750ca0f8eebe5cd530c2a66a3a",
"triggerType" : "PUSH"
}, {
"hash" : "9d73b9664eed48ada68721a92526640cfa06a021",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17098",
"triggerID" : "9d73b9664eed48ada68721a92526640cfa06a021",
"triggerType" : "PUSH"
}, {
"hash" : "7be538f4045a42ba33ab8fe62594178e2db75bbb",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17100",
"triggerID" : "7be538f4045a42ba33ab8fe62594178e2db75bbb",
"triggerType" : "PUSH"
}, {
"hash" : "cc0da2372d50d99c98c2ce4bcbe5a60303bde938",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17195",
"triggerID" : "cc0da2372d50d99c98c2ce4bcbe5a60303bde938",
"triggerType" : "PUSH"
}, {
"hash" : "da0e00e3655e06b064d11e20500317e2a1d2229c",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "da0e00e3655e06b064d11e20500317e2a1d2229c",
"triggerType" : "PUSH"
}, {
"hash" : "589c8cd7ffc5df7984c099d6b30fa117463db2b9",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17335",
"triggerID" : "589c8cd7ffc5df7984c099d6b30fa117463db2b9",
"triggerType" : "PUSH"
}, {
"hash" : "226c96ce48b33d7b1650d00e93bc5ea42ad276f6",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17624",
"triggerID" : "226c96ce48b33d7b1650d00e93bc5ea42ad276f6",
"triggerType" : "PUSH"
}, {
"hash" : "ad81a08799a98cd6ee63b8bf792d42f16a98f775",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17634",
"triggerID" : "ad81a08799a98cd6ee63b8bf792d42f16a98f775",
"triggerType" : "PUSH"
}, {
"hash" : "fd2ec462e554190843ddcc8d946317ae76cb81e1",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "fd2ec462e554190843ddcc8d946317ae76cb81e1",
"triggerType" : "PUSH"
}, {
"hash" : "cae57c2b03a3131cfa7774fa4335f4d153f85d61",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17644",
"triggerID" : "cae57c2b03a3131cfa7774fa4335f4d153f85d61",
"triggerType" : "PUSH"
}, {
"hash" : "a21fb8753ed206fefcf598c9e134c280e90d3dba",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17654",
"triggerID" : "a21fb8753ed206fefcf598c9e134c280e90d3dba",
"triggerType" : "PUSH"
}, {
"hash" : "4dfaefde334b712b3ba272bb330242ddb7f4c0ac",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17655",
"triggerID" : "4dfaefde334b712b3ba272bb330242ddb7f4c0ac",
"triggerType" : "PUSH"
}, {
"hash" : "ca59a379b830f88fd72c175cb9b96a14b33550c0",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17660",
"triggerID" : "ca59a379b830f88fd72c175cb9b96a14b33550c0",
"triggerType" : "PUSH"
}, {
"hash" : "1d49e658315a2ae22a2c29e3d388e82f869b146d",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17664",
"triggerID" : "1d49e658315a2ae22a2c29e3d388e82f869b146d",
"triggerType" : "PUSH"
}, {
"hash" : "df3a5b57a375fd344814b1f5a2451cfb79fea2c3",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17669",
"triggerID" : "df3a5b57a375fd344814b1f5a2451cfb79fea2c3",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* da0e00e3655e06b064d11e20500317e2a1d2229c UNKNOWN
* fd2ec462e554190843ddcc8d946317ae76cb81e1 UNKNOWN
* 1d49e658315a2ae22a2c29e3d388e82f869b146d Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17664)
* df3a5b57a375fd344814b1f5a2451cfb79fea2c3 Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17669)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] nsivabalan commented on a diff in pull request #8684: [HUDI-6200] Enhancements to the MDT for improving performance of larger indexes.
Posted by "nsivabalan (via GitHub)" <gi...@apache.org>.
nsivabalan commented on code in PR #8684:
URL: https://github.com/apache/hudi/pull/8684#discussion_r1190657749
##########
hudi-client/hudi-spark-client/src/main/java/org/apache/hudi/metadata/SparkHoodieMetadataBulkInsertPartitioner.java:
##########
@@ -0,0 +1,109 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hudi.metadata;
+
+import java.io.Serializable;
+import java.util.ArrayList;
+import java.util.Comparator;
+import java.util.List;
+
+import org.apache.hudi.common.model.HoodieRecord;
+import org.apache.hudi.common.util.ValidationUtils;
+import org.apache.hudi.table.BulkInsertPartitioner;
+import org.apache.spark.Partitioner;
+import org.apache.spark.api.java.JavaRDD;
+
+import scala.Tuple2;
+
+/**
+ * A {@code BulkInsertPartitioner} implementation for Metadata Table to improve performance of initialization of metadata
+ * table partition when a very large number of records are inserted.
+ *
+ * This partitioner requires the records to be already tagged with location.
+ */
+public class SparkHoodieMetadataBulkInsertPartitioner implements BulkInsertPartitioner<JavaRDD<HoodieRecord>> {
+ final int numPartitions;
+ public SparkHoodieMetadataBulkInsertPartitioner(int numPartitions) {
+ this.numPartitions = numPartitions;
+ }
+
+ private class FileGroupPartitioner extends Partitioner {
+
+ @Override
+ public int getPartition(Object key) {
+ return ((Tuple2<Integer, String>)key)._1;
+ }
+
+ @Override
+ public int numPartitions() {
+ return numPartitions;
+ }
+ }
+
+ // FileIDs for the various partitions
+ private List<String> fileIDPfxs;
+
+ /**
+ * Partition the records by their location. The number of partitions is determined by the number of MDT fileGroups being udpated rather than the
+ * specific value of outputSparkPartitions.
+ */
+ @Override
+ public JavaRDD<HoodieRecord> repartitionRecords(JavaRDD<HoodieRecord> records, int outputSparkPartitions) {
+ Comparator<Tuple2<Integer, String>> keyComparator =
+ (Comparator<Tuple2<Integer, String>> & Serializable)(t1, t2) -> t1._2.compareTo(t2._2);
+
+ // Partition the records by their file group
+ JavaRDD<HoodieRecord> partitionedRDD = records
+ // key by <file group index, record key>. The file group index is used to partition and the record key is used to sort within the partition.
+ .keyBy(r -> {
+ int fileGroupIndex = HoodieTableMetadataUtil.getFileGroupIndexFromFileId(r.getCurrentLocation().getFileId());
+ return new Tuple2<>(fileGroupIndex, r.getRecordKey());
+ })
+ .repartitionAndSortWithinPartitions(new FileGroupPartitioner(), keyComparator)
+ .map(t -> t._2);
+
+ fileIDPfxs = partitionedRDD.mapPartitions(recordItr -> {
+ // Due to partitioning, all record in the partition should have same fileID. So we only can get the fileID prefix from the first record.
+ List<String> fileIds = new ArrayList<>(1);
+ if (recordItr.hasNext()) {
+ HoodieRecord record = recordItr.next();
+ final String fileID = HoodieTableMetadataUtil.getFileGroupPrefix(record.getCurrentLocation().getFileId());
+ fileIds.add(fileID);
+ } else {
+ // Empty partition
+ fileIds.add("");
Review Comment:
can you help me understand when we might hit this ?
##########
hudi-common/src/main/java/org/apache/hudi/common/table/HoodieTableConfig.java:
##########
@@ -670,17 +670,75 @@ private Long getTableChecksum() {
return getLong(TABLE_CHECKSUM);
}
- public List<String> getMetadataPartitionsInflight() {
- return StringUtils.split(
- getStringOrDefault(TABLE_METADATA_PARTITIONS_INFLIGHT, StringUtils.EMPTY_STRING),
- CONFIG_VALUES_DELIMITER
- );
+ public Set<String> getMetadataPartitionsInflight() {
+ return new HashSet<>(StringUtils.split(
+ getStringOrDefault(TABLE_METADATA_PARTITIONS_INFLIGHT, StringUtils.EMPTY_STRING),
+ CONFIG_VALUES_DELIMITER));
}
public Set<String> getMetadataPartitions() {
return new HashSet<>(
- StringUtils.split(getStringOrDefault(TABLE_METADATA_PARTITIONS, StringUtils.EMPTY_STRING),
- CONFIG_VALUES_DELIMITER));
+ StringUtils.split(getStringOrDefault(TABLE_METADATA_PARTITIONS, StringUtils.EMPTY_STRING),
+ CONFIG_VALUES_DELIMITER));
+ }
+
+ /**
+ * @returns true if metadata table has been created and is being used for this dataset, else returns false.
+ */
+ public boolean isMetadataTableEnabled() {
+ return isMetadataPartitionEnabled(MetadataPartitionType.FILES);
+ }
+
+ /**
+ * Checks if metadata table is enabled and the specified partition has been initialized.
+ *
+ * @param partition The partition to check
+ * @returns true if the specific partition has been initialized, else returns false.
+ */
+ public boolean isMetadataPartitionEnabled(MetadataPartitionType partition) {
+ return getMetadataPartitions().contains(partition.getPartitionPath());
+ }
+
+ /**
+ * Enables or disables the specified metadata table partition.
+ *
+ * @param partition The partition
+ * @param enabled If true, the partition is enabled, else disabled
+ */
+ public void setMetadataPartitionState(MetadataPartitionType partition, boolean enabled) {
+ ValidationUtils.checkArgument(!partition.getPartitionPath().contains(CONFIG_VALUES_DELIMITER),
+ "Metadata Table partition path cannot contain a comma: " + partition.getPartitionPath());
+ Set<String> partitions = getMetadataPartitions();
+ Set<String> partitionsInflight = getMetadataPartitionsInflight();
+ if (enabled) {
+ partitions.add(partition.getPartitionPath());
+ partitionsInflight.remove(partition.getPartitionPath());
+ } else if (partition.equals(MetadataPartitionType.FILES)) {
+ // file listing partition is required for all other partitions to work
+ // Disabling file partition will also disable all partitions
+ partitions.clear();
+ partitionsInflight.clear();
+ } else {
+ partitions.remove(partition.getPartitionPath());
+ partitionsInflight.remove(partition.getPartitionPath());
+ }
+ setValue(TABLE_METADATA_PARTITIONS, partitions.stream().sorted().collect(Collectors.joining(CONFIG_VALUES_DELIMITER)));
+ setValue(TABLE_METADATA_PARTITIONS_INFLIGHT, partitionsInflight.stream().sorted().collect(Collectors.joining(CONFIG_VALUES_DELIMITER)));
+ }
+
+ /**
+ * Enables the specified metadata table partition as inflight.
+ *
+ * @param partition The list of partitions to enable as inflight.
+ */
+ public void setMetadataPartitionsAsInflight(List<MetadataPartitionType> partitionTypes) {
+ Set<String> partitions = getMetadataPartitionsInflight();
Review Comment:
minor. can we rename to "partitionsInflight"
##########
hudi-client/hudi-client-common/src/main/java/org/apache/hudi/metadata/HoodieBackedTableMetadataWriter.java:
##########
@@ -562,53 +532,144 @@ private <T extends SpecificRecordBase> boolean isCommitRevertedByInFlightAction(
/**
* Initialize the Metadata Table by listing files and partitions from the file system.
*
- * @param dataMetaClient - {@code HoodieTableMetaClient} for the dataset.
+ * @param initializationTime - Timestamp to use for the commit
+ * @param partitionsToInit - List of MDT partitions to initialize
* @param inflightInstantTimestamp - Current action instant responsible for this initialization
*/
- private boolean initializeFromFilesystem(HoodieTableMetaClient dataMetaClient,
+ private boolean initializeFromFilesystem(String initializationTime, List<MetadataPartitionType> partitionsToInit,
Option<String> inflightInstantTimestamp) throws IOException {
if (anyPendingDataInstant(dataMetaClient, inflightInstantTimestamp)) {
return false;
}
- String createInstantTime = getInitialCommitInstantTime(dataMetaClient);
-
- initializeMetaClient(DEFAULT_METADATA_POPULATE_META_FIELDS);
- initTableMetadata();
- // if async metadata indexing is enabled,
- // then only initialize files partition as other partitions will be built using HoodieIndexer
- List<MetadataPartitionType> enabledPartitionTypes = new ArrayList<>();
- if (dataWriteConfig.isMetadataAsyncIndex()) {
- enabledPartitionTypes.add(MetadataPartitionType.FILES);
- } else {
- // all enabled ones should be initialized
- enabledPartitionTypes = this.enabledPartitionTypes;
+ // FILES partition is always initialized first
+ ValidationUtils.checkArgument(!partitionsToInit.contains(MetadataPartitionType.FILES)
+ || partitionsToInit.get(0).equals(MetadataPartitionType.FILES), "FILES partition should be initialized first: " + partitionsToInit);
+
+ metadataMetaClient = initializeMetaClient();
+
+ // Get a complete list of files and partitions from the file system or from already initialized FILES partition of MDT
+ boolean filesPartitionAvailable = dataMetaClient.getTableConfig().isMetadataPartitionEnabled(MetadataPartitionType.FILES);
+ List<DirectoryInfo> partitionInfoList = filesPartitionAvailable ? listAllPartitionsFromMDT(initializationTime) : listAllPartitionsFromFilesystem(initializationTime);
+ Map<String, Map<String, Long>> partitionToFilesMap = partitionInfoList.stream()
+ .map(p -> {
+ String partitionName = HoodieTableMetadataUtil.getPartitionIdentifier(p.getRelativePath());
+ return Pair.of(partitionName, p.getFileNameToSizeMap());
+ })
+ .collect(Collectors.toMap(Pair::getKey, Pair::getValue));
+
+ for (MetadataPartitionType partitionType : partitionsToInit) {
+ // Find the commit timestamp to use for this partition. Each initialization should use its own unique commit time.
+ String commitTimeForPartition = generateUniqueCommitInstantTime(initializationTime);
+
+ LOG.info("Initializing MDT partition " + partitionType + " at instant " + commitTimeForPartition);
+
+ Pair<Integer, HoodieData<HoodieRecord>> fileGroupCountAndRecordsPair;
+ switch (partitionType) {
+ case FILES:
+ fileGroupCountAndRecordsPair = initializeFilesPartition(initializationTime, partitionInfoList);
+ break;
+ case BLOOM_FILTERS:
+ fileGroupCountAndRecordsPair = initializeBloomFiltersPartition(initializationTime, partitionToFilesMap);
+ break;
+ case COLUMN_STATS:
+ fileGroupCountAndRecordsPair = initializeColumnStatsPartition(partitionToFilesMap);
+ break;
+ default:
+ throw new HoodieMetadataException("Unsupported MDT partition type: " + partitionType);
+ }
+
+ // Generate the file groups
+ final int fileGroupCount = fileGroupCountAndRecordsPair.getKey();
+ ValidationUtils.checkArgument(fileGroupCount > 0, "FileGroup count for MDT partition " + partitionType + " should be > 0");
+ initializeFileGroups(dataMetaClient, partitionType, commitTimeForPartition, fileGroupCount);
Review Comment:
so do we create an empty delete block followed by a bulk insert w/ same fileId?
##########
hudi-client/hudi-client-common/src/main/java/org/apache/hudi/metadata/HoodieBackedTableMetadataWriter.java:
##########
@@ -892,25 +917,24 @@ public void buildMetadataPartitions(HoodieEngineContext engineContext, List<Hood
}
partitionTypes.add(partitionType);
});
- // before initial commit update inflight indexes in table config
- Set<String> inflightIndexes = getInflightMetadataPartitions(dataMetaClient.getTableConfig());
- inflightIndexes.addAll(indexPartitionInfos.stream().map(HoodieIndexPartitionInfo::getMetadataPartitionPath).collect(Collectors.toSet()));
- dataMetaClient.getTableConfig().setValue(HoodieTableConfig.TABLE_METADATA_PARTITIONS_INFLIGHT.key(), String.join(",", inflightIndexes));
- HoodieTableConfig.update(dataMetaClient.getFs(), new Path(dataMetaClient.getMetaPath()), dataMetaClient.getTableConfig().getProps());
- initialCommit(indexUptoInstantTime + METADATA_INDEXER_TIME_SUFFIX, partitionTypes);
+
+ // before initialization set these partitions as inflight in table config
+ HoodieTableMetadataUtil.setMetadataInflightPartitions(dataMetaClient, partitionTypes);
Review Comment:
wrt 2nd flight. we should be cautious in updating table config on the fly.
##########
hudi-client/hudi-client-common/src/main/java/org/apache/hudi/metadata/HoodieBackedTableMetadataWriter.java:
##########
@@ -373,105 +356,92 @@ public List<MetadataPartitionType> getEnabledPartitionTypes() {
return this.enabledPartitionTypes;
}
- /**
- * Initialize the metadata table if it does not exist.
- * <p>
- * If the metadata table does not exist, then file and partition listing is used to initialize the table.
- *
- * @param engineContext
- * @param actionMetadata Action metadata types extending Avro generated SpecificRecordBase
- * @param inflightInstantTimestamp Timestamp of an instant in progress on the dataset. This instant is ignored
- * while deciding to initialize the metadata table.
- */
- protected abstract <T extends SpecificRecordBase> void initialize(HoodieEngineContext engineContext,
- Option<T> actionMetadata,
- Option<String> inflightInstantTimestamp);
-
- public void initTableMetadata() {
- try {
- if (this.metadata != null) {
- this.metadata.close();
- }
- this.metadata = new HoodieBackedTableMetadata(engineContext, dataWriteConfig.getMetadataConfig(),
- dataWriteConfig.getBasePath(), dataWriteConfig.getSpillableMapBasePath());
- this.metadataMetaClient = metadata.getMetadataMetaClient();
- } catch (Exception e) {
- throw new HoodieException("Error initializing metadata table for reads", e);
- }
- }
-
/**
* Initialize the metadata table if needed.
*
* @param dataMetaClient - meta client for the data table
* @param actionMetadata - optional action metadata
* @param inflightInstantTimestamp - timestamp of an instant in progress on the dataset
* @param <T> - action metadata types extending Avro generated SpecificRecordBase
- * @throws IOException
+ * @throws IOException on errors
*/
- protected <T extends SpecificRecordBase> void initializeIfNeeded(HoodieTableMetaClient dataMetaClient,
- Option<T> actionMetadata,
- Option<String> inflightInstantTimestamp) throws IOException {
+ protected <T extends SpecificRecordBase> boolean initializeIfNeeded(HoodieTableMetaClient dataMetaClient,
+ Option<T> actionMetadata,
+ Option<String> inflightInstantTimestamp) throws IOException {
HoodieTimer timer = HoodieTimer.start();
+ List<MetadataPartitionType> partitionsToInit = new ArrayList<>(MetadataPartitionType.values().length);
- boolean exists = metadataTableExists(dataMetaClient, actionMetadata);
+ try {
+ boolean exists = metadataTableExists(dataMetaClient, actionMetadata);
+ if (!exists) {
+ // FILES partition is always required
+ partitionsToInit.add(MetadataPartitionType.FILES);
+ }
- if (!exists) {
- // Initialize for the first time by listing partitions and files directly from the file system
- if (initializeFromFilesystem(dataMetaClient, inflightInstantTimestamp)) {
- metrics.ifPresent(m -> m.updateMetrics(HoodieMetadataMetrics.INITIALIZE_STR, timer.endTimer()));
+ // check if any of the enabled partition types needs to be initialized
+ // NOTE: It needs to be guarded by async index config because if that is enabled then initialization happens through the index scheduler.
+ if (!dataWriteConfig.isMetadataAsyncIndex()) {
Review Comment:
not in this patch, but in future. we might need to design this better.
this assumes that every partition part from FILES will have to go via async indexer. But we should let user decide which one can go via inline and which ones need to go via async index building.
##########
hudi-common/src/main/java/org/apache/hudi/metadata/MetadataPartitionType.java:
##########
@@ -33,10 +33,6 @@ public enum MetadataPartitionType {
private final String partitionPath;
// FileId prefix used for all file groups in this partition.
private final String fileIdPrefix;
- // Total file groups
- // TODO fix: enum should not have any mutable aspect as this compromises whole idea
- // of the enum being static, immutable entity
- private int fileGroupCount = 1;
Review Comment:
we don't serialize this enum as is anywhere right? if yes, this is backwards incompatible change. Just wanted to double check.
##########
hudi-client/hudi-client-common/src/main/java/org/apache/hudi/table/action/index/ScheduleIndexActionExecutor.java:
##########
@@ -100,15 +99,6 @@ public Option<HoodieIndexPlan> execute() {
// get last completed instant
Option<HoodieInstant> indexUptoInstant = table.getActiveTimeline().getContiguousCompletedWriteTimeline().lastInstant();
if (indexUptoInstant.isPresent()) {
- // start initializing file groups
- // in case FILES partition itself was not initialized before (i.e. metadata was never enabled), this will initialize synchronously
- HoodieTableMetadataWriter metadataWriter = table.getMetadataWriter(instantTime)
- .orElseThrow(() -> new HoodieIndexException(String.format("Could not get metadata writer to initialize filegroups for indexing for instant: %s", instantTime)));
- if (!finalPartitionsToIndex.get(0).getPartitionPath().equals(MetadataPartitionType.FILES.getPartitionPath())) {
- // initialize metadata partition only if not for FILES partition.
- metadataWriter.initializeMetadataPartitions(table.getMetaClient(), finalPartitionsToIndex, indexUptoInstant.get().getTimestamp());
Review Comment:
if not here, where does the initialization happen for the file groups of interest?
##########
hudi-client/hudi-client-common/src/main/java/org/apache/hudi/metadata/HoodieBackedTableMetadataWriter.java:
##########
@@ -562,53 +532,144 @@ private <T extends SpecificRecordBase> boolean isCommitRevertedByInFlightAction(
/**
* Initialize the Metadata Table by listing files and partitions from the file system.
*
- * @param dataMetaClient - {@code HoodieTableMetaClient} for the dataset.
+ * @param initializationTime - Timestamp to use for the commit
+ * @param partitionsToInit - List of MDT partitions to initialize
* @param inflightInstantTimestamp - Current action instant responsible for this initialization
*/
- private boolean initializeFromFilesystem(HoodieTableMetaClient dataMetaClient,
+ private boolean initializeFromFilesystem(String initializationTime, List<MetadataPartitionType> partitionsToInit,
Option<String> inflightInstantTimestamp) throws IOException {
if (anyPendingDataInstant(dataMetaClient, inflightInstantTimestamp)) {
return false;
}
- String createInstantTime = getInitialCommitInstantTime(dataMetaClient);
-
- initializeMetaClient(DEFAULT_METADATA_POPULATE_META_FIELDS);
- initTableMetadata();
- // if async metadata indexing is enabled,
- // then only initialize files partition as other partitions will be built using HoodieIndexer
- List<MetadataPartitionType> enabledPartitionTypes = new ArrayList<>();
- if (dataWriteConfig.isMetadataAsyncIndex()) {
- enabledPartitionTypes.add(MetadataPartitionType.FILES);
- } else {
- // all enabled ones should be initialized
- enabledPartitionTypes = this.enabledPartitionTypes;
+ // FILES partition is always initialized first
+ ValidationUtils.checkArgument(!partitionsToInit.contains(MetadataPartitionType.FILES)
+ || partitionsToInit.get(0).equals(MetadataPartitionType.FILES), "FILES partition should be initialized first: " + partitionsToInit);
+
+ metadataMetaClient = initializeMetaClient();
+
+ // Get a complete list of files and partitions from the file system or from already initialized FILES partition of MDT
+ boolean filesPartitionAvailable = dataMetaClient.getTableConfig().isMetadataPartitionEnabled(MetadataPartitionType.FILES);
+ List<DirectoryInfo> partitionInfoList = filesPartitionAvailable ? listAllPartitionsFromMDT(initializationTime) : listAllPartitionsFromFilesystem(initializationTime);
+ Map<String, Map<String, Long>> partitionToFilesMap = partitionInfoList.stream()
+ .map(p -> {
+ String partitionName = HoodieTableMetadataUtil.getPartitionIdentifier(p.getRelativePath());
+ return Pair.of(partitionName, p.getFileNameToSizeMap());
+ })
+ .collect(Collectors.toMap(Pair::getKey, Pair::getValue));
+
+ for (MetadataPartitionType partitionType : partitionsToInit) {
+ // Find the commit timestamp to use for this partition. Each initialization should use its own unique commit time.
+ String commitTimeForPartition = generateUniqueCommitInstantTime(initializationTime);
+
+ LOG.info("Initializing MDT partition " + partitionType + " at instant " + commitTimeForPartition);
+
+ Pair<Integer, HoodieData<HoodieRecord>> fileGroupCountAndRecordsPair;
+ switch (partitionType) {
+ case FILES:
+ fileGroupCountAndRecordsPair = initializeFilesPartition(initializationTime, partitionInfoList);
+ break;
+ case BLOOM_FILTERS:
+ fileGroupCountAndRecordsPair = initializeBloomFiltersPartition(initializationTime, partitionToFilesMap);
+ break;
+ case COLUMN_STATS:
+ fileGroupCountAndRecordsPair = initializeColumnStatsPartition(partitionToFilesMap);
+ break;
+ default:
+ throw new HoodieMetadataException("Unsupported MDT partition type: " + partitionType);
+ }
+
+ // Generate the file groups
+ final int fileGroupCount = fileGroupCountAndRecordsPair.getKey();
+ ValidationUtils.checkArgument(fileGroupCount > 0, "FileGroup count for MDT partition " + partitionType + " should be > 0");
+ initializeFileGroups(dataMetaClient, partitionType, commitTimeForPartition, fileGroupCount);
+
+ // Perform the commit using bulkCommit
+ HoodieData<HoodieRecord> records = fileGroupCountAndRecordsPair.getValue();
+ bulkCommit(commitTimeForPartition, partitionType, records, fileGroupCount);
+ metadataMetaClient.reloadActiveTimeline();
+ dataMetaClient = HoodieTableMetadataUtil.setMetadataPartitionState(dataMetaClient, partitionType, true);
}
- initializeEnabledFileGroups(dataMetaClient, createInstantTime, enabledPartitionTypes);
- initialCommit(createInstantTime, enabledPartitionTypes);
- updateInitializedPartitionsInTableConfig(enabledPartitionTypes);
+
return true;
}
- private String getInitialCommitInstantTime(HoodieTableMetaClient dataMetaClient) {
- // If there is no commit on the dataset yet, use the SOLO_COMMIT_TIMESTAMP as the instant time for initial commit
- // Otherwise, we use the timestamp of the latest completed action.
- String createInstantTime = dataMetaClient.getActiveTimeline().filterCompletedInstants()
- .getReverseOrderedInstants().findFirst().map(HoodieInstant::getTimestamp).orElse(SOLO_COMMIT_TIMESTAMP);
- LOG.info("Creating a new metadata table in " + metadataWriteConfig.getBasePath() + " at instant " + createInstantTime);
- return createInstantTime;
+ /**
+ * Returns a unique timestamp to use for initializing a MDT partition.
+ * <p>
+ * Since commits are immutable, we should use unique timestamps to initialize each partition. For this, we will add a suffix to the given initializationTime
+ * until we find a unique timestamp.
+ *
+ * @param initializationTime Timestamp from dataset to use for initialization
+ * @return a unique timestamp for MDT
+ */
+ private String generateUniqueCommitInstantTime(String initializationTime) {
+ for (int offset = 0; ; ++offset) {
Review Comment:
whats the purpose of offset.
I thought we only initialize one partition from MDT at a time. so, can you help me understand the necessity here.
##########
hudi-client/hudi-client-common/src/main/java/org/apache/hudi/metadata/HoodieBackedTableMetadataWriter.java:
##########
@@ -562,53 +532,144 @@ private <T extends SpecificRecordBase> boolean isCommitRevertedByInFlightAction(
/**
* Initialize the Metadata Table by listing files and partitions from the file system.
*
- * @param dataMetaClient - {@code HoodieTableMetaClient} for the dataset.
+ * @param initializationTime - Timestamp to use for the commit
+ * @param partitionsToInit - List of MDT partitions to initialize
* @param inflightInstantTimestamp - Current action instant responsible for this initialization
*/
- private boolean initializeFromFilesystem(HoodieTableMetaClient dataMetaClient,
+ private boolean initializeFromFilesystem(String initializationTime, List<MetadataPartitionType> partitionsToInit,
Option<String> inflightInstantTimestamp) throws IOException {
if (anyPendingDataInstant(dataMetaClient, inflightInstantTimestamp)) {
return false;
}
- String createInstantTime = getInitialCommitInstantTime(dataMetaClient);
-
- initializeMetaClient(DEFAULT_METADATA_POPULATE_META_FIELDS);
- initTableMetadata();
- // if async metadata indexing is enabled,
- // then only initialize files partition as other partitions will be built using HoodieIndexer
- List<MetadataPartitionType> enabledPartitionTypes = new ArrayList<>();
- if (dataWriteConfig.isMetadataAsyncIndex()) {
- enabledPartitionTypes.add(MetadataPartitionType.FILES);
- } else {
- // all enabled ones should be initialized
- enabledPartitionTypes = this.enabledPartitionTypes;
+ // FILES partition is always initialized first
+ ValidationUtils.checkArgument(!partitionsToInit.contains(MetadataPartitionType.FILES)
+ || partitionsToInit.get(0).equals(MetadataPartitionType.FILES), "FILES partition should be initialized first: " + partitionsToInit);
+
+ metadataMetaClient = initializeMetaClient();
+
+ // Get a complete list of files and partitions from the file system or from already initialized FILES partition of MDT
+ boolean filesPartitionAvailable = dataMetaClient.getTableConfig().isMetadataPartitionEnabled(MetadataPartitionType.FILES);
+ List<DirectoryInfo> partitionInfoList = filesPartitionAvailable ? listAllPartitionsFromMDT(initializationTime) : listAllPartitionsFromFilesystem(initializationTime);
+ Map<String, Map<String, Long>> partitionToFilesMap = partitionInfoList.stream()
+ .map(p -> {
+ String partitionName = HoodieTableMetadataUtil.getPartitionIdentifier(p.getRelativePath());
+ return Pair.of(partitionName, p.getFileNameToSizeMap());
+ })
+ .collect(Collectors.toMap(Pair::getKey, Pair::getValue));
+
+ for (MetadataPartitionType partitionType : partitionsToInit) {
+ // Find the commit timestamp to use for this partition. Each initialization should use its own unique commit time.
+ String commitTimeForPartition = generateUniqueCommitInstantTime(initializationTime);
+
+ LOG.info("Initializing MDT partition " + partitionType + " at instant " + commitTimeForPartition);
+
+ Pair<Integer, HoodieData<HoodieRecord>> fileGroupCountAndRecordsPair;
+ switch (partitionType) {
+ case FILES:
+ fileGroupCountAndRecordsPair = initializeFilesPartition(initializationTime, partitionInfoList);
+ break;
+ case BLOOM_FILTERS:
+ fileGroupCountAndRecordsPair = initializeBloomFiltersPartition(initializationTime, partitionToFilesMap);
+ break;
+ case COLUMN_STATS:
+ fileGroupCountAndRecordsPair = initializeColumnStatsPartition(partitionToFilesMap);
+ break;
+ default:
+ throw new HoodieMetadataException("Unsupported MDT partition type: " + partitionType);
+ }
+
+ // Generate the file groups
+ final int fileGroupCount = fileGroupCountAndRecordsPair.getKey();
+ ValidationUtils.checkArgument(fileGroupCount > 0, "FileGroup count for MDT partition " + partitionType + " should be > 0");
+ initializeFileGroups(dataMetaClient, partitionType, commitTimeForPartition, fileGroupCount);
+
+ // Perform the commit using bulkCommit
+ HoodieData<HoodieRecord> records = fileGroupCountAndRecordsPair.getValue();
+ bulkCommit(commitTimeForPartition, partitionType, records, fileGroupCount);
+ metadataMetaClient.reloadActiveTimeline();
+ dataMetaClient = HoodieTableMetadataUtil.setMetadataPartitionState(dataMetaClient, partitionType, true);
}
- initializeEnabledFileGroups(dataMetaClient, createInstantTime, enabledPartitionTypes);
- initialCommit(createInstantTime, enabledPartitionTypes);
- updateInitializedPartitionsInTableConfig(enabledPartitionTypes);
+
return true;
}
- private String getInitialCommitInstantTime(HoodieTableMetaClient dataMetaClient) {
- // If there is no commit on the dataset yet, use the SOLO_COMMIT_TIMESTAMP as the instant time for initial commit
- // Otherwise, we use the timestamp of the latest completed action.
- String createInstantTime = dataMetaClient.getActiveTimeline().filterCompletedInstants()
- .getReverseOrderedInstants().findFirst().map(HoodieInstant::getTimestamp).orElse(SOLO_COMMIT_TIMESTAMP);
- LOG.info("Creating a new metadata table in " + metadataWriteConfig.getBasePath() + " at instant " + createInstantTime);
- return createInstantTime;
+ /**
+ * Returns a unique timestamp to use for initializing a MDT partition.
+ * <p>
+ * Since commits are immutable, we should use unique timestamps to initialize each partition. For this, we will add a suffix to the given initializationTime
+ * until we find a unique timestamp.
+ *
+ * @param initializationTime Timestamp from dataset to use for initialization
+ * @return a unique timestamp for MDT
+ */
+ private String generateUniqueCommitInstantTime(String initializationTime) {
+ for (int offset = 0; ; ++offset) {
+ final String commitInstantTime = HoodieTableMetadataUtil.createIndexInitTimestamp(initializationTime, offset);
+ if (!metadataMetaClient.getCommitTimeline().containsInstant(commitInstantTime)) {
+ return commitInstantTime;
+ }
+ }
}
- private boolean anyPendingDataInstant(HoodieTableMetaClient dataMetaClient, Option<String> inflightInstantTimestamp) {
- ValidationUtils.checkState(enabled, "Metadata table cannot be initialized as it is not enabled");
+ private Pair<Integer, HoodieData<HoodieRecord>> initializeColumnStatsPartition(Map<String, Map<String, Long>> partitionToFilesMap) {
Review Comment:
do we have tests for this?
i.e. even if we enable 3 partitions, each commit will only initialize just 1. and in subsequent one initialize the next and so on.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] nsivabalan commented on a diff in pull request #8684: [HUDI-6200] Enhancements to the MDT for improving performance of larger indexes.
Posted by "nsivabalan (via GitHub)" <gi...@apache.org>.
nsivabalan commented on code in PR #8684:
URL: https://github.com/apache/hudi/pull/8684#discussion_r1191719299
##########
hudi-client/hudi-spark-client/src/main/java/org/apache/hudi/metadata/SparkHoodieMetadataBulkInsertPartitioner.java:
##########
@@ -0,0 +1,109 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hudi.metadata;
+
+import java.io.Serializable;
+import java.util.ArrayList;
+import java.util.Comparator;
+import java.util.List;
+
+import org.apache.hudi.common.model.HoodieRecord;
+import org.apache.hudi.common.util.ValidationUtils;
+import org.apache.hudi.table.BulkInsertPartitioner;
+import org.apache.spark.Partitioner;
+import org.apache.spark.api.java.JavaRDD;
+
+import scala.Tuple2;
+
+/**
+ * A {@code BulkInsertPartitioner} implementation for Metadata Table to improve performance of initialization of metadata
+ * table partition when a very large number of records are inserted.
+ *
+ * This partitioner requires the records to be already tagged with location.
+ */
+public class SparkHoodieMetadataBulkInsertPartitioner implements BulkInsertPartitioner<JavaRDD<HoodieRecord>> {
+ final int numPartitions;
+ public SparkHoodieMetadataBulkInsertPartitioner(int numPartitions) {
+ this.numPartitions = numPartitions;
+ }
+
+ private class FileGroupPartitioner extends Partitioner {
+
+ @Override
+ public int getPartition(Object key) {
+ return ((Tuple2<Integer, String>)key)._1;
+ }
+
+ @Override
+ public int numPartitions() {
+ return numPartitions;
+ }
+ }
+
+ // FileIDs for the various partitions
+ private List<String> fileIDPfxs;
+
+ /**
+ * Partition the records by their location. The number of partitions is determined by the number of MDT fileGroups being udpated rather than the
+ * specific value of outputSparkPartitions.
+ */
+ @Override
+ public JavaRDD<HoodieRecord> repartitionRecords(JavaRDD<HoodieRecord> records, int outputSparkPartitions) {
+ Comparator<Tuple2<Integer, String>> keyComparator =
+ (Comparator<Tuple2<Integer, String>> & Serializable)(t1, t2) -> t1._2.compareTo(t2._2);
+
+ // Partition the records by their file group
+ JavaRDD<HoodieRecord> partitionedRDD = records
+ // key by <file group index, record key>. The file group index is used to partition and the record key is used to sort within the partition.
+ .keyBy(r -> {
+ int fileGroupIndex = HoodieTableMetadataUtil.getFileGroupIndexFromFileId(r.getCurrentLocation().getFileId());
+ return new Tuple2<>(fileGroupIndex, r.getRecordKey());
+ })
+ .repartitionAndSortWithinPartitions(new FileGroupPartitioner(), keyComparator)
+ .map(t -> t._2);
+
+ fileIDPfxs = partitionedRDD.mapPartitions(recordItr -> {
+ // Due to partitioning, all record in the partition should have same fileID. So we only can get the fileID prefix from the first record.
+ List<String> fileIds = new ArrayList<>(1);
+ if (recordItr.hasNext()) {
+ HoodieRecord record = recordItr.next();
+ final String fileID = HoodieTableMetadataUtil.getFileGroupPrefix(record.getCurrentLocation().getFileId());
+ fileIds.add(fileID);
+ } else {
+ // Empty partition
+ fileIds.add("");
Review Comment:
when there are spark partitions which does not have records in them. can we add docs
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org