You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@hudi.apache.org by GitBox <gi...@apache.org> on 2021/09/03 04:00:49 UTC
[GitHub] [hudi] nsivabalan opened a new pull request #3590: [WIP] Sync metadata patch
nsivabalan opened a new pull request #3590:
URL: https://github.com/apache/hudi/pull/3590
## *Tips*
- *Thank you very much for contributing to Apache Hudi.*
- *Please review https://hudi.apache.org/contribute/how-to-contribute before opening a pull request.*
## What is the purpose of the pull request
*(For example: This pull request adds quick-start document.)*
## Brief change log
*(for example:)*
- *Modify AnnotationLocation checkstyle rule in checkstyle.xml*
## Verify this pull request
*(Please pick either of the following options)*
This pull request is a trivial rework / code cleanup without any test coverage.
*(or)*
This pull request is already covered by existing tests, such as *(please describe tests)*.
(or)
This change added tests and can be verified as follows:
*(example:)*
- *Added integration tests for end-to-end.*
- *Added HoodieClientWriteTest to verify the change.*
- *Manually verified the change by running a job locally.*
## Committer checklist
- [ ] Has a corresponding JIRA in PR title & commit
- [ ] Commit message is descriptive of the change
- [ ] CI is green
- [ ] Necessary doc changes done or have another open PR
- [ ] For large changes, please consider breaking it into sub-tasks under an umbrella JIRA.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] nsivabalan edited a comment on pull request #3590: [HUDI-2285] Metadata table synchronous design. Rebased and Squashed from pull/3426
Posted by GitBox <gi...@apache.org>.
nsivabalan edited a comment on pull request #3590:
URL: https://github.com/apache/hudi/pull/3590#issuecomment-922983380
@vinothchandar :
I have addressed most of the feedback from you except for few. Have responded in the other patch for those which is yet to be resolved.
wrt multi-writer and compaction, here is what is happening in this patch. (have covered this in the description of the patch, but anyways).
All writes to metadata table happens within the datatable lock. So, implicitly there are no multi-writers (including compaction, cleaning for metadata table) to metadata table. There will always be only one writer. For regular writes to data table, we do conflict resolution, where as for compaction and clustering (data table), we don't do any conflict resolution. Remember taking a lock while committing compaction and clustering in data table is being added in this patch. So, please pay attention if there is something I am missing. All we ensure is to commit by taking a lock.
So, in summary, we have a constraint only single writer model to metadata table. May be we need to revisit this when we integrate record level index. But for now, this is our current state of things.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot edited a comment on pull request #3590: [HUDI-2285][HUDI-2476] Metadata table synchronous design. Rebased and Squashed from pull/3426
Posted by GitBox <gi...@apache.org>.
hudi-bot edited a comment on pull request #3590:
URL: https://github.com/apache/hudi/pull/3590#issuecomment-912237120
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "648e36e1af308534bc050c3a19da008ab7a8c840",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1994",
"triggerID" : "648e36e1af308534bc050c3a19da008ab7a8c840",
"triggerType" : "PUSH"
}, {
"hash" : "8c32253ad0f6d9eb728789fad75763bb5703678d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1995",
"triggerID" : "8c32253ad0f6d9eb728789fad75763bb5703678d",
"triggerType" : "PUSH"
}, {
"hash" : "71bbe59aa8bf70b9aa8990a6dd2522eb4bf1d172",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2009",
"triggerID" : "71bbe59aa8bf70b9aa8990a6dd2522eb4bf1d172",
"triggerType" : "PUSH"
}, {
"hash" : "aefac7ec2f2e40bdf3ad4365ea6aa825803a439d",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "aefac7ec2f2e40bdf3ad4365ea6aa825803a439d",
"triggerType" : "PUSH"
}, {
"hash" : "86905cb855fecfeb9e4c023d526ad643fadb8084",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2012",
"triggerID" : "86905cb855fecfeb9e4c023d526ad643fadb8084",
"triggerType" : "PUSH"
}, {
"hash" : "b4d93b0168775cabd43609fec59bf2823dd81279",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2029",
"triggerID" : "b4d93b0168775cabd43609fec59bf2823dd81279",
"triggerType" : "PUSH"
}, {
"hash" : "38bf430ecb4e11839d686929be2511eed340e9df",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2035",
"triggerID" : "38bf430ecb4e11839d686929be2511eed340e9df",
"triggerType" : "PUSH"
}, {
"hash" : "7793fbdb9b93a129ef606cb2d73ea6e1e9074957",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2299",
"triggerID" : "7793fbdb9b93a129ef606cb2d73ea6e1e9074957",
"triggerType" : "PUSH"
}, {
"hash" : "9c0123c0f27f990d009b323bab75b76ceecf3dab",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2307",
"triggerID" : "9c0123c0f27f990d009b323bab75b76ceecf3dab",
"triggerType" : "PUSH"
}, {
"hash" : "af27976ba07afb02305c699aeb7a75a122013b62",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2330",
"triggerID" : "af27976ba07afb02305c699aeb7a75a122013b62",
"triggerType" : "PUSH"
}, {
"hash" : "9172241c58424cca8e8186d4409a8c36968a4790",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2344",
"triggerID" : "9172241c58424cca8e8186d4409a8c36968a4790",
"triggerType" : "PUSH"
}, {
"hash" : "13b80067ae00a760c57d184f70e2d98f63027e80",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2353",
"triggerID" : "13b80067ae00a760c57d184f70e2d98f63027e80",
"triggerType" : "PUSH"
}, {
"hash" : "ef750979cf449ff504da194e594cb62302eba360",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2366",
"triggerID" : "ef750979cf449ff504da194e594cb62302eba360",
"triggerType" : "PUSH"
}, {
"hash" : "16827c37f36d3b9fec18e0a35aff7599c45e0444",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2375",
"triggerID" : "16827c37f36d3b9fec18e0a35aff7599c45e0444",
"triggerType" : "PUSH"
}, {
"hash" : "0f68d426b01355537f5124b8f108345a36d3da39",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2378",
"triggerID" : "0f68d426b01355537f5124b8f108345a36d3da39",
"triggerType" : "PUSH"
}, {
"hash" : "2f3099197889fd5c48ae3b550cc523e0528ddb2c",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2380",
"triggerID" : "2f3099197889fd5c48ae3b550cc523e0528ddb2c",
"triggerType" : "PUSH"
}, {
"hash" : "c83d176e1d1ddab6bb86f91726060779a9a1519f",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2392",
"triggerID" : "c83d176e1d1ddab6bb86f91726060779a9a1519f",
"triggerType" : "PUSH"
}, {
"hash" : "008f0cd91de12d44030d837f460f3b0232d62ee5",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2416",
"triggerID" : "008f0cd91de12d44030d837f460f3b0232d62ee5",
"triggerType" : "PUSH"
}, {
"hash" : "92634fb730e723a9bdeb165348a4c747794be7e3",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2418",
"triggerID" : "92634fb730e723a9bdeb165348a4c747794be7e3",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* aefac7ec2f2e40bdf3ad4365ea6aa825803a439d UNKNOWN
* 92634fb730e723a9bdeb165348a4c747794be7e3 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2418)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run travis` re-run the last Travis build
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot edited a comment on pull request #3590: [HUDI-2285][HUDI-2476] Metadata table synchronous design. Rebased and Squashed from pull/3426
Posted by GitBox <gi...@apache.org>.
hudi-bot edited a comment on pull request #3590:
URL: https://github.com/apache/hudi/pull/3590#issuecomment-912237120
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "648e36e1af308534bc050c3a19da008ab7a8c840",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1994",
"triggerID" : "648e36e1af308534bc050c3a19da008ab7a8c840",
"triggerType" : "PUSH"
}, {
"hash" : "8c32253ad0f6d9eb728789fad75763bb5703678d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1995",
"triggerID" : "8c32253ad0f6d9eb728789fad75763bb5703678d",
"triggerType" : "PUSH"
}, {
"hash" : "71bbe59aa8bf70b9aa8990a6dd2522eb4bf1d172",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2009",
"triggerID" : "71bbe59aa8bf70b9aa8990a6dd2522eb4bf1d172",
"triggerType" : "PUSH"
}, {
"hash" : "aefac7ec2f2e40bdf3ad4365ea6aa825803a439d",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "aefac7ec2f2e40bdf3ad4365ea6aa825803a439d",
"triggerType" : "PUSH"
}, {
"hash" : "86905cb855fecfeb9e4c023d526ad643fadb8084",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2012",
"triggerID" : "86905cb855fecfeb9e4c023d526ad643fadb8084",
"triggerType" : "PUSH"
}, {
"hash" : "b4d93b0168775cabd43609fec59bf2823dd81279",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2029",
"triggerID" : "b4d93b0168775cabd43609fec59bf2823dd81279",
"triggerType" : "PUSH"
}, {
"hash" : "38bf430ecb4e11839d686929be2511eed340e9df",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2035",
"triggerID" : "38bf430ecb4e11839d686929be2511eed340e9df",
"triggerType" : "PUSH"
}, {
"hash" : "7793fbdb9b93a129ef606cb2d73ea6e1e9074957",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2299",
"triggerID" : "7793fbdb9b93a129ef606cb2d73ea6e1e9074957",
"triggerType" : "PUSH"
}, {
"hash" : "9c0123c0f27f990d009b323bab75b76ceecf3dab",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2307",
"triggerID" : "9c0123c0f27f990d009b323bab75b76ceecf3dab",
"triggerType" : "PUSH"
}, {
"hash" : "af27976ba07afb02305c699aeb7a75a122013b62",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2330",
"triggerID" : "af27976ba07afb02305c699aeb7a75a122013b62",
"triggerType" : "PUSH"
}, {
"hash" : "9172241c58424cca8e8186d4409a8c36968a4790",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2344",
"triggerID" : "9172241c58424cca8e8186d4409a8c36968a4790",
"triggerType" : "PUSH"
}, {
"hash" : "13b80067ae00a760c57d184f70e2d98f63027e80",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2353",
"triggerID" : "13b80067ae00a760c57d184f70e2d98f63027e80",
"triggerType" : "PUSH"
}, {
"hash" : "ef750979cf449ff504da194e594cb62302eba360",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2366",
"triggerID" : "ef750979cf449ff504da194e594cb62302eba360",
"triggerType" : "PUSH"
}, {
"hash" : "16827c37f36d3b9fec18e0a35aff7599c45e0444",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2375",
"triggerID" : "16827c37f36d3b9fec18e0a35aff7599c45e0444",
"triggerType" : "PUSH"
}, {
"hash" : "0f68d426b01355537f5124b8f108345a36d3da39",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2378",
"triggerID" : "0f68d426b01355537f5124b8f108345a36d3da39",
"triggerType" : "PUSH"
}, {
"hash" : "2f3099197889fd5c48ae3b550cc523e0528ddb2c",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2380",
"triggerID" : "2f3099197889fd5c48ae3b550cc523e0528ddb2c",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* aefac7ec2f2e40bdf3ad4365ea6aa825803a439d UNKNOWN
* 0f68d426b01355537f5124b8f108345a36d3da39 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2378)
* 2f3099197889fd5c48ae3b550cc523e0528ddb2c Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2380)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run travis` re-run the last Travis build
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] nsivabalan commented on pull request #3590: [HUDI-2285][HUDI-2476] Metadata table synchronous design. Rebased and Squashed from pull/3426
Posted by GitBox <gi...@apache.org>.
nsivabalan commented on pull request #3590:
URL: https://github.com/apache/hudi/pull/3590#issuecomment-928195283
@hudi-bot azure run
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] nsivabalan edited a comment on pull request #3590: [HUDI-2285][HUDI-2476] Metadata table synchronous design. Rebased and Squashed from pull/3426
Posted by GitBox <gi...@apache.org>.
nsivabalan edited a comment on pull request #3590:
URL: https://github.com/apache/hudi/pull/3590#issuecomment-933938711
> @nsivabalan have we unit tested `HoodieBackedTableMetadata` or `BaseTableMetadata` here?
I started writing unit tests and later realized that all of our existing validation is happening using apis in BaseTableMetadata (getAllPartitionPaths and getAllFilesInPartition). Also there are tests to test reader wrt valid instants separately. So, I just enhanced existing tests and added extra validation that fileSystemMetadata is not used during these validations(or in other words only metadata table is used strictly during these validations).
In summary, I ensured we have tests covering HoodieBackedTableMetadata and HoodieBackedTableMetadataWriter for all of its methods.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] nsivabalan commented on a change in pull request #3590: [HUDI-2285][HUDI-2476] Metadata table synchronous design. Rebased and Squashed from pull/3426
Posted by GitBox <gi...@apache.org>.
nsivabalan commented on a change in pull request #3590:
URL: https://github.com/apache/hudi/pull/3590#discussion_r716249167
##########
File path: hudi-client/hudi-spark-client/src/main/java/org/apache/hudi/metadata/SparkHoodieBackedTableMetadataWriter.java
##########
@@ -99,83 +94,94 @@ protected void initialize(HoodieEngineContext engineContext, HoodieTableMetaClie
@Override
protected void commit(List<HoodieRecord> records, String partitionName, String instantTime) {
ValidationUtils.checkState(enabled, "Metadata table cannot be committed to as it is not enabled");
- JavaRDD<HoodieRecord> recordRDD = prepRecords(records, partitionName);
+ JavaRDD<HoodieRecord> recordRDD = prepRecords(records, partitionName, 1);
try (SparkRDDWriteClient writeClient = new SparkRDDWriteClient(engineContext, metadataWriteConfig, true)) {
- writeClient.startCommitWithTime(instantTime);
+ if (!metaClient.getActiveTimeline().filterCompletedInstants().containsInstant(instantTime)) {
+ // if this is a new commit being applied to metadata for the first time
+ writeClient.startCommitWithTime(instantTime);
+ } else {
+ // this code path refers to a re-attempted commit that got committed to metadata, but failed in dataset.
+ // for eg, lets say compaction c1 on 1st attempt succeeded in metadata table and failed before committing to datatable.
+ // when retried again, data table will first rollback pending compaction. these will be applied to metadata table, but all changes
+ // are upserts to metadata table and so only a new delta commit will be created.
+ // once rollback is complete, compaction will be retried again, which will eventually hit this code block where the respective commit is
+ // already part of completed commit. So, we have to manually remove the completed instant and proceed.
+ // and it is for the same reason we enabled withAllowMultiWriteOnSameInstant for metadata table.
+ HoodieInstant alreadyCompletedInstant = metaClient.getActiveTimeline().filterCompletedInstants().filter(entry -> entry.getTimestamp().equals(instantTime)).lastInstant().get();
+ FSUtils.deleteInstantFile(metaClient.getFs(), metaClient.getMetaPath(), alreadyCompletedInstant);
+ metaClient.reloadActiveTimeline();
+ }
List<WriteStatus> statuses = writeClient.upsertPreppedRecords(recordRDD, instantTime).collect();
statuses.forEach(writeStatus -> {
if (writeStatus.hasErrors()) {
throw new HoodieMetadataException("Failed to commit metadata table records at instant " + instantTime);
}
});
- // trigger cleaning, compaction, with suffixes based on the same instant time. This ensures that any future
- // delta commits synced over will not have an instant time lesser than the last completed instant on the
- // metadata table.
- if (writeClient.scheduleCompactionAtInstant(instantTime + "001", Option.empty())) {
- writeClient.compact(instantTime + "001");
- }
- writeClient.clean(instantTime + "002");
+
+ // reload timeline
+ metaClient.reloadActiveTimeline();
+
+ compactIfNecessary(writeClient, instantTime);
+ doClean(writeClient, instantTime);
}
// Update total size of the metadata and count of base/log files
- metrics.ifPresent(m -> {
- try {
- Map<String, String> stats = m.getStats(false, metaClient, metadata);
- m.updateMetrics(Long.parseLong(stats.get(HoodieMetadataMetrics.STAT_TOTAL_BASE_FILE_SIZE)),
- Long.parseLong(stats.get(HoodieMetadataMetrics.STAT_TOTAL_LOG_FILE_SIZE)),
- Integer.parseInt(stats.get(HoodieMetadataMetrics.STAT_COUNT_BASE_FILES)),
- Integer.parseInt(stats.get(HoodieMetadataMetrics.STAT_COUNT_LOG_FILES)));
- } catch (HoodieIOException e) {
- LOG.error("Could not publish metadata size metrics", e);
- }
- });
+ metrics.ifPresent(m -> m.updateSizeMetrics(metaClient, metadata));
}
/**
- * Tag each record with the location.
+ * Perform a compaction on the Metadata Table.
+ *
+ * Cases to be handled:
+ * 1. We cannot perform compaction if there are previous inflight operations on the dataset. This is because
+ * a compacted metadata base file at time Tx should represent all the actions on the dataset till time Tx.
*
- * Since we only read the latest base file in a partition, we tag the records with the instant time of the latest
- * base file.
+ * 2. In multi-writer scenario, a parallel operation with a greater instantTime may have completed creating a
+ * deltacommit.
*/
- private JavaRDD<HoodieRecord> prepRecords(List<HoodieRecord> records, String partitionName) {
- HoodieTable table = HoodieSparkTable.create(metadataWriteConfig, engineContext);
- TableFileSystemView.SliceView fsView = table.getSliceView();
- List<HoodieBaseFile> baseFiles = fsView.getLatestFileSlices(partitionName)
- .map(FileSlice::getBaseFile)
- .filter(Option::isPresent)
- .map(Option::get)
- .collect(Collectors.toList());
-
- // All the metadata fits within a single base file
- if (partitionName.equals(MetadataPartitionType.FILES.partitionPath())) {
- if (baseFiles.size() > 1) {
- throw new HoodieMetadataException("Multiple base files found in metadata partition");
- }
+ private void compactIfNecessary(SparkRDDWriteClient writeClient, String instantTime) {
+ String latestDeltacommitTime = metaClient.getActiveTimeline().getDeltaCommitTimeline().filterCompletedInstants().lastInstant()
+ .get().getTimestamp();
+ List<HoodieInstant> pendingInstants = datasetMetaClient.reloadActiveTimeline().filterInflightsAndRequested()
+ .findInstantsBefore(latestDeltacommitTime).getInstants().collect(Collectors.toList());
+
+ if (!pendingInstants.isEmpty()) {
+ LOG.info(String.format("Cannot compact metadata table as there are %d inflight instants before latest deltacommit %s: %s",
+ pendingInstants.size(), latestDeltacommitTime, Arrays.toString(pendingInstants.toArray())));
+ return;
}
- JavaSparkContext jsc = ((HoodieSparkEngineContext) engineContext).getJavaSparkContext();
- String fileId;
- String instantTime;
- if (!baseFiles.isEmpty()) {
- fileId = baseFiles.get(0).getFileId();
- instantTime = baseFiles.get(0).getCommitTime();
- } else {
- // If there is a log file then we can assume that it has the data
- List<HoodieLogFile> logFiles = fsView.getLatestFileSlices(MetadataPartitionType.FILES.partitionPath())
- .map(FileSlice::getLatestLogFile)
- .filter(Option::isPresent)
- .map(Option::get)
- .collect(Collectors.toList());
- if (logFiles.isEmpty()) {
- // No base and log files. All are new inserts
- return jsc.parallelize(records, 1);
- }
-
- fileId = logFiles.get(0).getFileId();
- instantTime = logFiles.get(0).getBaseCommitTime();
+ // Trigger compaction with suffixes based on the same instant time. This ensures that any future
+ // delta commits synced over will not have an instant time lesser than the last completed instant on the
+ // metadata table.
+ final String compactionInstantTime = latestDeltacommitTime + "001";
+ if (writeClient.scheduleCompactionAtInstant(compactionInstantTime, Option.empty())) {
Review comment:
we will not have or use data's write client in this layer at all. So, prefer to leave it as is. But I have fixed all other namings.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] nsivabalan commented on a change in pull request #3590: [HUDI-2285][HUDI-2476] Metadata table synchronous design. Rebased and Squashed from pull/3426
Posted by GitBox <gi...@apache.org>.
nsivabalan commented on a change in pull request #3590:
URL: https://github.com/apache/hudi/pull/3590#discussion_r716249586
##########
File path: hudi-client/hudi-client-common/src/main/java/org/apache/hudi/metadata/HoodieBackedTableMetadataWriter.java
##########
@@ -401,64 +394,83 @@ private boolean bootstrapFromFilesystem(HoodieEngineContext engineContext, Hoodi
}
/**
- * Sync the Metadata Table from the instants created on the dataset.
+ * Initialize file groups for a partition. For file listing, we just have one file group.
*
- * @param datasetMetaClient {@code HoodieTableMetaClient} for the dataset
+ * All FileGroups for a given metadata partition has a fixed prefix as per the {@link MetadataPartitionType#getFileIdPrefix()}.
+ * Each file group is suffixed with increments of 1 starting with 1.
+ *
+ * For instance, for FILES, there is only one file group named as "files-1"
+ * Lets say we configure 10 file groups for record level index, and prefix as "record-index-bucket-"
Review comment:
yeah, if we know for sure any metadata partition which we might add in future will not have more than 9999 file groups, we should be good. I will go ahead w/ 3 padded zeros to make it uniform.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot edited a comment on pull request #3590: [HUDI-2285][HUDI-2476] Metadata table synchronous design. Rebased and Squashed from pull/3426
Posted by GitBox <gi...@apache.org>.
hudi-bot edited a comment on pull request #3590:
URL: https://github.com/apache/hudi/pull/3590#issuecomment-912237120
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "648e36e1af308534bc050c3a19da008ab7a8c840",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1994",
"triggerID" : "648e36e1af308534bc050c3a19da008ab7a8c840",
"triggerType" : "PUSH"
}, {
"hash" : "8c32253ad0f6d9eb728789fad75763bb5703678d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1995",
"triggerID" : "8c32253ad0f6d9eb728789fad75763bb5703678d",
"triggerType" : "PUSH"
}, {
"hash" : "71bbe59aa8bf70b9aa8990a6dd2522eb4bf1d172",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2009",
"triggerID" : "71bbe59aa8bf70b9aa8990a6dd2522eb4bf1d172",
"triggerType" : "PUSH"
}, {
"hash" : "aefac7ec2f2e40bdf3ad4365ea6aa825803a439d",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "aefac7ec2f2e40bdf3ad4365ea6aa825803a439d",
"triggerType" : "PUSH"
}, {
"hash" : "86905cb855fecfeb9e4c023d526ad643fadb8084",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2012",
"triggerID" : "86905cb855fecfeb9e4c023d526ad643fadb8084",
"triggerType" : "PUSH"
}, {
"hash" : "b4d93b0168775cabd43609fec59bf2823dd81279",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2029",
"triggerID" : "b4d93b0168775cabd43609fec59bf2823dd81279",
"triggerType" : "PUSH"
}, {
"hash" : "38bf430ecb4e11839d686929be2511eed340e9df",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2035",
"triggerID" : "38bf430ecb4e11839d686929be2511eed340e9df",
"triggerType" : "PUSH"
}, {
"hash" : "7793fbdb9b93a129ef606cb2d73ea6e1e9074957",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2299",
"triggerID" : "7793fbdb9b93a129ef606cb2d73ea6e1e9074957",
"triggerType" : "PUSH"
}, {
"hash" : "9c0123c0f27f990d009b323bab75b76ceecf3dab",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2307",
"triggerID" : "9c0123c0f27f990d009b323bab75b76ceecf3dab",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* aefac7ec2f2e40bdf3ad4365ea6aa825803a439d UNKNOWN
* 9c0123c0f27f990d009b323bab75b76ceecf3dab Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2307)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run travis` re-run the last Travis build
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] nsivabalan commented on a change in pull request #3590: [HUDI-2285][HUDI-2476] Metadata table synchronous design. Rebased and Squashed from pull/3426
Posted by GitBox <gi...@apache.org>.
nsivabalan commented on a change in pull request #3590:
URL: https://github.com/apache/hudi/pull/3590#discussion_r716254203
##########
File path: hudi-client/hudi-spark-client/src/main/java/org/apache/hudi/table/HoodieSparkTable.java
##########
@@ -66,4 +77,34 @@ protected HoodieSparkTable(HoodieWriteConfig config, HoodieEngineContext context
protected HoodieIndex<T, JavaRDD<HoodieRecord<T>>, JavaRDD<HoodieKey>, JavaRDD<WriteStatus>> getIndex(HoodieWriteConfig config, HoodieEngineContext context) {
return SparkHoodieIndex.createIndex(config);
}
+
+ /**
+ * Fetch instance of {@link HoodieTableMetadataWriter}.
+ *
+ * @return instance of {@link HoodieTableMetadataWriter}
+ */
+ @Override
+ public Option<HoodieTableMetadataWriter> getMetadataWriter() {
+ synchronized (this) {
+ if (!isMetadataInfoUpdated.getAndSet(true)) {
+ // this code assumes that bootstrap of metadata table is done elsewhere (SparkRDDWriteClient) and so, we gauge whether metadata table
+ // is available or not once here and reuse the same info for repeated calls to getMetadataWriter(). Please remove memoization if
+ // that's not the case.
+ try {
+ if (!config.isMetadataTableEnabled() || !metaClient.getFs().exists(new Path(HoodieTableMetadata.getMetadataTableBasePath(metaClient.getBasePath())))) {
+ isMetadataTableAvailable = false;
+ } else {
+ isMetadataTableAvailable = true;
+ }
+ } catch (IOException e) {
+ throw new HoodieMetadataException("Could not instantiate metadata table writer", e);
+ }
+ }
+ }
+ if (isMetadataTableAvailable) {
Review comment:
isMetadataTableAvailable is not an Optional element. it is just a boolean. and so I could not do Option.of(boolean).filter()... I will keep trying out few things here if I can get rid of if else. for now, will leave it as is. I could make isMetadataTableAvailable an Option<boolean> and set it explicitly within the sync block only if its avaialble and leave it as Option.empty if metadata is not available, but feel its not neat either.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot edited a comment on pull request #3590: [HUDI-2285][HUDI-2476] Metadata table synchronous design. Rebased and Squashed from pull/3426
Posted by GitBox <gi...@apache.org>.
hudi-bot edited a comment on pull request #3590:
URL: https://github.com/apache/hudi/pull/3590#issuecomment-912237120
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "648e36e1af308534bc050c3a19da008ab7a8c840",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1994",
"triggerID" : "648e36e1af308534bc050c3a19da008ab7a8c840",
"triggerType" : "PUSH"
}, {
"hash" : "8c32253ad0f6d9eb728789fad75763bb5703678d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1995",
"triggerID" : "8c32253ad0f6d9eb728789fad75763bb5703678d",
"triggerType" : "PUSH"
}, {
"hash" : "71bbe59aa8bf70b9aa8990a6dd2522eb4bf1d172",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2009",
"triggerID" : "71bbe59aa8bf70b9aa8990a6dd2522eb4bf1d172",
"triggerType" : "PUSH"
}, {
"hash" : "aefac7ec2f2e40bdf3ad4365ea6aa825803a439d",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "aefac7ec2f2e40bdf3ad4365ea6aa825803a439d",
"triggerType" : "PUSH"
}, {
"hash" : "86905cb855fecfeb9e4c023d526ad643fadb8084",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2012",
"triggerID" : "86905cb855fecfeb9e4c023d526ad643fadb8084",
"triggerType" : "PUSH"
}, {
"hash" : "b4d93b0168775cabd43609fec59bf2823dd81279",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2029",
"triggerID" : "b4d93b0168775cabd43609fec59bf2823dd81279",
"triggerType" : "PUSH"
}, {
"hash" : "38bf430ecb4e11839d686929be2511eed340e9df",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2035",
"triggerID" : "38bf430ecb4e11839d686929be2511eed340e9df",
"triggerType" : "PUSH"
}, {
"hash" : "7793fbdb9b93a129ef606cb2d73ea6e1e9074957",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2299",
"triggerID" : "7793fbdb9b93a129ef606cb2d73ea6e1e9074957",
"triggerType" : "PUSH"
}, {
"hash" : "9c0123c0f27f990d009b323bab75b76ceecf3dab",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2307",
"triggerID" : "9c0123c0f27f990d009b323bab75b76ceecf3dab",
"triggerType" : "PUSH"
}, {
"hash" : "af27976ba07afb02305c699aeb7a75a122013b62",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2330",
"triggerID" : "af27976ba07afb02305c699aeb7a75a122013b62",
"triggerType" : "PUSH"
}, {
"hash" : "9172241c58424cca8e8186d4409a8c36968a4790",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2344",
"triggerID" : "9172241c58424cca8e8186d4409a8c36968a4790",
"triggerType" : "PUSH"
}, {
"hash" : "13b80067ae00a760c57d184f70e2d98f63027e80",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2353",
"triggerID" : "13b80067ae00a760c57d184f70e2d98f63027e80",
"triggerType" : "PUSH"
}, {
"hash" : "ef750979cf449ff504da194e594cb62302eba360",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2366",
"triggerID" : "ef750979cf449ff504da194e594cb62302eba360",
"triggerType" : "PUSH"
}, {
"hash" : "16827c37f36d3b9fec18e0a35aff7599c45e0444",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2375",
"triggerID" : "16827c37f36d3b9fec18e0a35aff7599c45e0444",
"triggerType" : "PUSH"
}, {
"hash" : "0f68d426b01355537f5124b8f108345a36d3da39",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2378",
"triggerID" : "0f68d426b01355537f5124b8f108345a36d3da39",
"triggerType" : "PUSH"
}, {
"hash" : "2f3099197889fd5c48ae3b550cc523e0528ddb2c",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2380",
"triggerID" : "2f3099197889fd5c48ae3b550cc523e0528ddb2c",
"triggerType" : "PUSH"
}, {
"hash" : "c83d176e1d1ddab6bb86f91726060779a9a1519f",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2392",
"triggerID" : "c83d176e1d1ddab6bb86f91726060779a9a1519f",
"triggerType" : "PUSH"
}, {
"hash" : "008f0cd91de12d44030d837f460f3b0232d62ee5",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2416",
"triggerID" : "008f0cd91de12d44030d837f460f3b0232d62ee5",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* aefac7ec2f2e40bdf3ad4365ea6aa825803a439d UNKNOWN
* c83d176e1d1ddab6bb86f91726060779a9a1519f Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2392)
* 008f0cd91de12d44030d837f460f3b0232d62ee5 Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2416)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run travis` re-run the last Travis build
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot edited a comment on pull request #3590: [HUDI-2285][HUDI-2476] Metadata table synchronous design. Rebased and Squashed from pull/3426
Posted by GitBox <gi...@apache.org>.
hudi-bot edited a comment on pull request #3590:
URL: https://github.com/apache/hudi/pull/3590#issuecomment-912237120
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "648e36e1af308534bc050c3a19da008ab7a8c840",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1994",
"triggerID" : "648e36e1af308534bc050c3a19da008ab7a8c840",
"triggerType" : "PUSH"
}, {
"hash" : "8c32253ad0f6d9eb728789fad75763bb5703678d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1995",
"triggerID" : "8c32253ad0f6d9eb728789fad75763bb5703678d",
"triggerType" : "PUSH"
}, {
"hash" : "71bbe59aa8bf70b9aa8990a6dd2522eb4bf1d172",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2009",
"triggerID" : "71bbe59aa8bf70b9aa8990a6dd2522eb4bf1d172",
"triggerType" : "PUSH"
}, {
"hash" : "aefac7ec2f2e40bdf3ad4365ea6aa825803a439d",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "aefac7ec2f2e40bdf3ad4365ea6aa825803a439d",
"triggerType" : "PUSH"
}, {
"hash" : "86905cb855fecfeb9e4c023d526ad643fadb8084",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2012",
"triggerID" : "86905cb855fecfeb9e4c023d526ad643fadb8084",
"triggerType" : "PUSH"
}, {
"hash" : "b4d93b0168775cabd43609fec59bf2823dd81279",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2029",
"triggerID" : "b4d93b0168775cabd43609fec59bf2823dd81279",
"triggerType" : "PUSH"
}, {
"hash" : "38bf430ecb4e11839d686929be2511eed340e9df",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2035",
"triggerID" : "38bf430ecb4e11839d686929be2511eed340e9df",
"triggerType" : "PUSH"
}, {
"hash" : "7793fbdb9b93a129ef606cb2d73ea6e1e9074957",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2299",
"triggerID" : "7793fbdb9b93a129ef606cb2d73ea6e1e9074957",
"triggerType" : "PUSH"
}, {
"hash" : "9c0123c0f27f990d009b323bab75b76ceecf3dab",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2307",
"triggerID" : "9c0123c0f27f990d009b323bab75b76ceecf3dab",
"triggerType" : "PUSH"
}, {
"hash" : "af27976ba07afb02305c699aeb7a75a122013b62",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2330",
"triggerID" : "af27976ba07afb02305c699aeb7a75a122013b62",
"triggerType" : "PUSH"
}, {
"hash" : "9172241c58424cca8e8186d4409a8c36968a4790",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2344",
"triggerID" : "9172241c58424cca8e8186d4409a8c36968a4790",
"triggerType" : "PUSH"
}, {
"hash" : "13b80067ae00a760c57d184f70e2d98f63027e80",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2353",
"triggerID" : "13b80067ae00a760c57d184f70e2d98f63027e80",
"triggerType" : "PUSH"
}, {
"hash" : "ef750979cf449ff504da194e594cb62302eba360",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2366",
"triggerID" : "ef750979cf449ff504da194e594cb62302eba360",
"triggerType" : "PUSH"
}, {
"hash" : "16827c37f36d3b9fec18e0a35aff7599c45e0444",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2375",
"triggerID" : "16827c37f36d3b9fec18e0a35aff7599c45e0444",
"triggerType" : "PUSH"
}, {
"hash" : "0f68d426b01355537f5124b8f108345a36d3da39",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2378",
"triggerID" : "0f68d426b01355537f5124b8f108345a36d3da39",
"triggerType" : "PUSH"
}, {
"hash" : "2f3099197889fd5c48ae3b550cc523e0528ddb2c",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2380",
"triggerID" : "2f3099197889fd5c48ae3b550cc523e0528ddb2c",
"triggerType" : "PUSH"
}, {
"hash" : "c83d176e1d1ddab6bb86f91726060779a9a1519f",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2392",
"triggerID" : "c83d176e1d1ddab6bb86f91726060779a9a1519f",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* aefac7ec2f2e40bdf3ad4365ea6aa825803a439d UNKNOWN
* c83d176e1d1ddab6bb86f91726060779a9a1519f Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2392)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run travis` re-run the last Travis build
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] nsivabalan commented on a change in pull request #3590: [HUDI-2285][HUDI-2476] Metadata table synchronous design. Rebased and Squashed from pull/3426
Posted by GitBox <gi...@apache.org>.
nsivabalan commented on a change in pull request #3590:
URL: https://github.com/apache/hudi/pull/3590#discussion_r716096549
##########
File path: hudi-client/hudi-flink-client/src/main/java/org/apache/hudi/table/upgrade/FlinkUpgradeDowngrade.java
##########
@@ -49,7 +49,9 @@ public void run(HoodieTableMetaClient metaClient, HoodieTableVersion toVersion,
return new ZeroToOneUpgradeHandler().upgrade(config, context, instantTime);
} else if (fromVersion == HoodieTableVersion.ONE && toVersion == HoodieTableVersion.TWO) {
return new OneToTwoUpgradeHandler().upgrade(config, context, instantTime);
- } else {
+ } else if (fromVersion == HoodieTableVersion.TWO && toVersion == HoodieTableVersion.THREE) {
Review comment:
https://issues.apache.org/jira/browse/HUDI-2475
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot edited a comment on pull request #3590: [HUDI-2285][HUDI-2476] Metadata table synchronous design. Rebased and Squashed from pull/3426
Posted by GitBox <gi...@apache.org>.
hudi-bot edited a comment on pull request #3590:
URL: https://github.com/apache/hudi/pull/3590#issuecomment-912237120
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "648e36e1af308534bc050c3a19da008ab7a8c840",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1994",
"triggerID" : "648e36e1af308534bc050c3a19da008ab7a8c840",
"triggerType" : "PUSH"
}, {
"hash" : "8c32253ad0f6d9eb728789fad75763bb5703678d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1995",
"triggerID" : "8c32253ad0f6d9eb728789fad75763bb5703678d",
"triggerType" : "PUSH"
}, {
"hash" : "71bbe59aa8bf70b9aa8990a6dd2522eb4bf1d172",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2009",
"triggerID" : "71bbe59aa8bf70b9aa8990a6dd2522eb4bf1d172",
"triggerType" : "PUSH"
}, {
"hash" : "aefac7ec2f2e40bdf3ad4365ea6aa825803a439d",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "aefac7ec2f2e40bdf3ad4365ea6aa825803a439d",
"triggerType" : "PUSH"
}, {
"hash" : "86905cb855fecfeb9e4c023d526ad643fadb8084",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2012",
"triggerID" : "86905cb855fecfeb9e4c023d526ad643fadb8084",
"triggerType" : "PUSH"
}, {
"hash" : "b4d93b0168775cabd43609fec59bf2823dd81279",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2029",
"triggerID" : "b4d93b0168775cabd43609fec59bf2823dd81279",
"triggerType" : "PUSH"
}, {
"hash" : "38bf430ecb4e11839d686929be2511eed340e9df",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2035",
"triggerID" : "38bf430ecb4e11839d686929be2511eed340e9df",
"triggerType" : "PUSH"
}, {
"hash" : "7793fbdb9b93a129ef606cb2d73ea6e1e9074957",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2299",
"triggerID" : "7793fbdb9b93a129ef606cb2d73ea6e1e9074957",
"triggerType" : "PUSH"
}, {
"hash" : "9c0123c0f27f990d009b323bab75b76ceecf3dab",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2307",
"triggerID" : "9c0123c0f27f990d009b323bab75b76ceecf3dab",
"triggerType" : "PUSH"
}, {
"hash" : "af27976ba07afb02305c699aeb7a75a122013b62",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2330",
"triggerID" : "af27976ba07afb02305c699aeb7a75a122013b62",
"triggerType" : "PUSH"
}, {
"hash" : "9172241c58424cca8e8186d4409a8c36968a4790",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2344",
"triggerID" : "9172241c58424cca8e8186d4409a8c36968a4790",
"triggerType" : "PUSH"
}, {
"hash" : "13b80067ae00a760c57d184f70e2d98f63027e80",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2353",
"triggerID" : "13b80067ae00a760c57d184f70e2d98f63027e80",
"triggerType" : "PUSH"
}, {
"hash" : "ef750979cf449ff504da194e594cb62302eba360",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2366",
"triggerID" : "ef750979cf449ff504da194e594cb62302eba360",
"triggerType" : "PUSH"
}, {
"hash" : "16827c37f36d3b9fec18e0a35aff7599c45e0444",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2375",
"triggerID" : "16827c37f36d3b9fec18e0a35aff7599c45e0444",
"triggerType" : "PUSH"
}, {
"hash" : "0f68d426b01355537f5124b8f108345a36d3da39",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2378",
"triggerID" : "0f68d426b01355537f5124b8f108345a36d3da39",
"triggerType" : "PUSH"
}, {
"hash" : "2f3099197889fd5c48ae3b550cc523e0528ddb2c",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2380",
"triggerID" : "2f3099197889fd5c48ae3b550cc523e0528ddb2c",
"triggerType" : "PUSH"
}, {
"hash" : "c83d176e1d1ddab6bb86f91726060779a9a1519f",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2392",
"triggerID" : "c83d176e1d1ddab6bb86f91726060779a9a1519f",
"triggerType" : "PUSH"
}, {
"hash" : "008f0cd91de12d44030d837f460f3b0232d62ee5",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2416",
"triggerID" : "008f0cd91de12d44030d837f460f3b0232d62ee5",
"triggerType" : "PUSH"
}, {
"hash" : "92634fb730e723a9bdeb165348a4c747794be7e3",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2418",
"triggerID" : "92634fb730e723a9bdeb165348a4c747794be7e3",
"triggerType" : "PUSH"
}, {
"hash" : "3b5fbfa0cc69e82f7c23fc6aa5b4944e2dd77220",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2438",
"triggerID" : "3b5fbfa0cc69e82f7c23fc6aa5b4944e2dd77220",
"triggerType" : "PUSH"
}, {
"hash" : "87d2ec0cc60ef8d5b7a081b73a0318aedf1a2a90",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2451",
"triggerID" : "87d2ec0cc60ef8d5b7a081b73a0318aedf1a2a90",
"triggerType" : "PUSH"
}, {
"hash" : "c09f5e30662efd882ff9fc35f5750a8253171e9d",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2452",
"triggerID" : "c09f5e30662efd882ff9fc35f5750a8253171e9d",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* aefac7ec2f2e40bdf3ad4365ea6aa825803a439d UNKNOWN
* c09f5e30662efd882ff9fc35f5750a8253171e9d Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2452)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run travis` re-run the last Travis build
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot edited a comment on pull request #3590: [HUDI-2285][HUDI-2468][HUDI-2476] Metadata table synchronous design. Rebased and Squashed from pull/3426
Posted by GitBox <gi...@apache.org>.
hudi-bot edited a comment on pull request #3590:
URL: https://github.com/apache/hudi/pull/3590#issuecomment-912237120
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "648e36e1af308534bc050c3a19da008ab7a8c840",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1994",
"triggerID" : "648e36e1af308534bc050c3a19da008ab7a8c840",
"triggerType" : "PUSH"
}, {
"hash" : "8c32253ad0f6d9eb728789fad75763bb5703678d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1995",
"triggerID" : "8c32253ad0f6d9eb728789fad75763bb5703678d",
"triggerType" : "PUSH"
}, {
"hash" : "71bbe59aa8bf70b9aa8990a6dd2522eb4bf1d172",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2009",
"triggerID" : "71bbe59aa8bf70b9aa8990a6dd2522eb4bf1d172",
"triggerType" : "PUSH"
}, {
"hash" : "aefac7ec2f2e40bdf3ad4365ea6aa825803a439d",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "aefac7ec2f2e40bdf3ad4365ea6aa825803a439d",
"triggerType" : "PUSH"
}, {
"hash" : "86905cb855fecfeb9e4c023d526ad643fadb8084",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2012",
"triggerID" : "86905cb855fecfeb9e4c023d526ad643fadb8084",
"triggerType" : "PUSH"
}, {
"hash" : "b4d93b0168775cabd43609fec59bf2823dd81279",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2029",
"triggerID" : "b4d93b0168775cabd43609fec59bf2823dd81279",
"triggerType" : "PUSH"
}, {
"hash" : "38bf430ecb4e11839d686929be2511eed340e9df",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2035",
"triggerID" : "38bf430ecb4e11839d686929be2511eed340e9df",
"triggerType" : "PUSH"
}, {
"hash" : "7793fbdb9b93a129ef606cb2d73ea6e1e9074957",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2299",
"triggerID" : "7793fbdb9b93a129ef606cb2d73ea6e1e9074957",
"triggerType" : "PUSH"
}, {
"hash" : "9c0123c0f27f990d009b323bab75b76ceecf3dab",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "9c0123c0f27f990d009b323bab75b76ceecf3dab",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* aefac7ec2f2e40bdf3ad4365ea6aa825803a439d UNKNOWN
* 7793fbdb9b93a129ef606cb2d73ea6e1e9074957 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2299)
* 9c0123c0f27f990d009b323bab75b76ceecf3dab UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run travis` re-run the last Travis build
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot edited a comment on pull request #3590: [HUDI-2285][HUDI-2476] Metadata table synchronous design. Rebased and Squashed from pull/3426
Posted by GitBox <gi...@apache.org>.
hudi-bot edited a comment on pull request #3590:
URL: https://github.com/apache/hudi/pull/3590#issuecomment-912237120
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "648e36e1af308534bc050c3a19da008ab7a8c840",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1994",
"triggerID" : "648e36e1af308534bc050c3a19da008ab7a8c840",
"triggerType" : "PUSH"
}, {
"hash" : "8c32253ad0f6d9eb728789fad75763bb5703678d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1995",
"triggerID" : "8c32253ad0f6d9eb728789fad75763bb5703678d",
"triggerType" : "PUSH"
}, {
"hash" : "71bbe59aa8bf70b9aa8990a6dd2522eb4bf1d172",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2009",
"triggerID" : "71bbe59aa8bf70b9aa8990a6dd2522eb4bf1d172",
"triggerType" : "PUSH"
}, {
"hash" : "aefac7ec2f2e40bdf3ad4365ea6aa825803a439d",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "aefac7ec2f2e40bdf3ad4365ea6aa825803a439d",
"triggerType" : "PUSH"
}, {
"hash" : "86905cb855fecfeb9e4c023d526ad643fadb8084",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2012",
"triggerID" : "86905cb855fecfeb9e4c023d526ad643fadb8084",
"triggerType" : "PUSH"
}, {
"hash" : "b4d93b0168775cabd43609fec59bf2823dd81279",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2029",
"triggerID" : "b4d93b0168775cabd43609fec59bf2823dd81279",
"triggerType" : "PUSH"
}, {
"hash" : "38bf430ecb4e11839d686929be2511eed340e9df",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2035",
"triggerID" : "38bf430ecb4e11839d686929be2511eed340e9df",
"triggerType" : "PUSH"
}, {
"hash" : "7793fbdb9b93a129ef606cb2d73ea6e1e9074957",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2299",
"triggerID" : "7793fbdb9b93a129ef606cb2d73ea6e1e9074957",
"triggerType" : "PUSH"
}, {
"hash" : "9c0123c0f27f990d009b323bab75b76ceecf3dab",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2307",
"triggerID" : "9c0123c0f27f990d009b323bab75b76ceecf3dab",
"triggerType" : "PUSH"
}, {
"hash" : "af27976ba07afb02305c699aeb7a75a122013b62",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2330",
"triggerID" : "af27976ba07afb02305c699aeb7a75a122013b62",
"triggerType" : "PUSH"
}, {
"hash" : "9172241c58424cca8e8186d4409a8c36968a4790",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2344",
"triggerID" : "9172241c58424cca8e8186d4409a8c36968a4790",
"triggerType" : "PUSH"
}, {
"hash" : "13b80067ae00a760c57d184f70e2d98f63027e80",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2353",
"triggerID" : "13b80067ae00a760c57d184f70e2d98f63027e80",
"triggerType" : "PUSH"
}, {
"hash" : "ef750979cf449ff504da194e594cb62302eba360",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2366",
"triggerID" : "ef750979cf449ff504da194e594cb62302eba360",
"triggerType" : "PUSH"
}, {
"hash" : "16827c37f36d3b9fec18e0a35aff7599c45e0444",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2375",
"triggerID" : "16827c37f36d3b9fec18e0a35aff7599c45e0444",
"triggerType" : "PUSH"
}, {
"hash" : "0f68d426b01355537f5124b8f108345a36d3da39",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2378",
"triggerID" : "0f68d426b01355537f5124b8f108345a36d3da39",
"triggerType" : "PUSH"
}, {
"hash" : "2f3099197889fd5c48ae3b550cc523e0528ddb2c",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2380",
"triggerID" : "2f3099197889fd5c48ae3b550cc523e0528ddb2c",
"triggerType" : "PUSH"
}, {
"hash" : "c83d176e1d1ddab6bb86f91726060779a9a1519f",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2392",
"triggerID" : "c83d176e1d1ddab6bb86f91726060779a9a1519f",
"triggerType" : "PUSH"
}, {
"hash" : "008f0cd91de12d44030d837f460f3b0232d62ee5",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2416",
"triggerID" : "008f0cd91de12d44030d837f460f3b0232d62ee5",
"triggerType" : "PUSH"
}, {
"hash" : "92634fb730e723a9bdeb165348a4c747794be7e3",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2418",
"triggerID" : "92634fb730e723a9bdeb165348a4c747794be7e3",
"triggerType" : "PUSH"
}, {
"hash" : "3b5fbfa0cc69e82f7c23fc6aa5b4944e2dd77220",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2438",
"triggerID" : "3b5fbfa0cc69e82f7c23fc6aa5b4944e2dd77220",
"triggerType" : "PUSH"
}, {
"hash" : "87d2ec0cc60ef8d5b7a081b73a0318aedf1a2a90",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2451",
"triggerID" : "87d2ec0cc60ef8d5b7a081b73a0318aedf1a2a90",
"triggerType" : "PUSH"
}, {
"hash" : "c09f5e30662efd882ff9fc35f5750a8253171e9d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2452",
"triggerID" : "c09f5e30662efd882ff9fc35f5750a8253171e9d",
"triggerType" : "PUSH"
}, {
"hash" : "b432e1db06e032b8071fa6c9e871ebc30b04b9eb",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2485",
"triggerID" : "b432e1db06e032b8071fa6c9e871ebc30b04b9eb",
"triggerType" : "PUSH"
}, {
"hash" : "0d70dba11569f96064e2cbf5c9d5fcb3c7fee5c0",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "0d70dba11569f96064e2cbf5c9d5fcb3c7fee5c0",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* aefac7ec2f2e40bdf3ad4365ea6aa825803a439d UNKNOWN
* b432e1db06e032b8071fa6c9e871ebc30b04b9eb Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2485)
* 0d70dba11569f96064e2cbf5c9d5fcb3c7fee5c0 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run travis` re-run the last Travis build
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot edited a comment on pull request #3590: [HUDI-2285][HUDI-2476] Metadata table synchronous design. Rebased and Squashed from pull/3426
Posted by GitBox <gi...@apache.org>.
hudi-bot edited a comment on pull request #3590:
URL: https://github.com/apache/hudi/pull/3590#issuecomment-912237120
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "648e36e1af308534bc050c3a19da008ab7a8c840",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1994",
"triggerID" : "648e36e1af308534bc050c3a19da008ab7a8c840",
"triggerType" : "PUSH"
}, {
"hash" : "8c32253ad0f6d9eb728789fad75763bb5703678d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1995",
"triggerID" : "8c32253ad0f6d9eb728789fad75763bb5703678d",
"triggerType" : "PUSH"
}, {
"hash" : "71bbe59aa8bf70b9aa8990a6dd2522eb4bf1d172",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2009",
"triggerID" : "71bbe59aa8bf70b9aa8990a6dd2522eb4bf1d172",
"triggerType" : "PUSH"
}, {
"hash" : "aefac7ec2f2e40bdf3ad4365ea6aa825803a439d",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "aefac7ec2f2e40bdf3ad4365ea6aa825803a439d",
"triggerType" : "PUSH"
}, {
"hash" : "86905cb855fecfeb9e4c023d526ad643fadb8084",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2012",
"triggerID" : "86905cb855fecfeb9e4c023d526ad643fadb8084",
"triggerType" : "PUSH"
}, {
"hash" : "b4d93b0168775cabd43609fec59bf2823dd81279",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2029",
"triggerID" : "b4d93b0168775cabd43609fec59bf2823dd81279",
"triggerType" : "PUSH"
}, {
"hash" : "38bf430ecb4e11839d686929be2511eed340e9df",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2035",
"triggerID" : "38bf430ecb4e11839d686929be2511eed340e9df",
"triggerType" : "PUSH"
}, {
"hash" : "7793fbdb9b93a129ef606cb2d73ea6e1e9074957",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2299",
"triggerID" : "7793fbdb9b93a129ef606cb2d73ea6e1e9074957",
"triggerType" : "PUSH"
}, {
"hash" : "9c0123c0f27f990d009b323bab75b76ceecf3dab",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2307",
"triggerID" : "9c0123c0f27f990d009b323bab75b76ceecf3dab",
"triggerType" : "PUSH"
}, {
"hash" : "af27976ba07afb02305c699aeb7a75a122013b62",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2330",
"triggerID" : "af27976ba07afb02305c699aeb7a75a122013b62",
"triggerType" : "PUSH"
}, {
"hash" : "9172241c58424cca8e8186d4409a8c36968a4790",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2344",
"triggerID" : "9172241c58424cca8e8186d4409a8c36968a4790",
"triggerType" : "PUSH"
}, {
"hash" : "13b80067ae00a760c57d184f70e2d98f63027e80",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2353",
"triggerID" : "13b80067ae00a760c57d184f70e2d98f63027e80",
"triggerType" : "PUSH"
}, {
"hash" : "ef750979cf449ff504da194e594cb62302eba360",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2366",
"triggerID" : "ef750979cf449ff504da194e594cb62302eba360",
"triggerType" : "PUSH"
}, {
"hash" : "16827c37f36d3b9fec18e0a35aff7599c45e0444",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2375",
"triggerID" : "16827c37f36d3b9fec18e0a35aff7599c45e0444",
"triggerType" : "PUSH"
}, {
"hash" : "0f68d426b01355537f5124b8f108345a36d3da39",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2378",
"triggerID" : "0f68d426b01355537f5124b8f108345a36d3da39",
"triggerType" : "PUSH"
}, {
"hash" : "2f3099197889fd5c48ae3b550cc523e0528ddb2c",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2380",
"triggerID" : "2f3099197889fd5c48ae3b550cc523e0528ddb2c",
"triggerType" : "PUSH"
}, {
"hash" : "c83d176e1d1ddab6bb86f91726060779a9a1519f",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2392",
"triggerID" : "c83d176e1d1ddab6bb86f91726060779a9a1519f",
"triggerType" : "PUSH"
}, {
"hash" : "008f0cd91de12d44030d837f460f3b0232d62ee5",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2416",
"triggerID" : "008f0cd91de12d44030d837f460f3b0232d62ee5",
"triggerType" : "PUSH"
}, {
"hash" : "92634fb730e723a9bdeb165348a4c747794be7e3",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2418",
"triggerID" : "92634fb730e723a9bdeb165348a4c747794be7e3",
"triggerType" : "PUSH"
}, {
"hash" : "3b5fbfa0cc69e82f7c23fc6aa5b4944e2dd77220",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "3b5fbfa0cc69e82f7c23fc6aa5b4944e2dd77220",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* aefac7ec2f2e40bdf3ad4365ea6aa825803a439d UNKNOWN
* 92634fb730e723a9bdeb165348a4c747794be7e3 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2418)
* 3b5fbfa0cc69e82f7c23fc6aa5b4944e2dd77220 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run travis` re-run the last Travis build
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot edited a comment on pull request #3590: [HUDI-2285][HUDI-2476] Metadata table synchronous design. Rebased and Squashed from pull/3426
Posted by GitBox <gi...@apache.org>.
hudi-bot edited a comment on pull request #3590:
URL: https://github.com/apache/hudi/pull/3590#issuecomment-912237120
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "648e36e1af308534bc050c3a19da008ab7a8c840",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1994",
"triggerID" : "648e36e1af308534bc050c3a19da008ab7a8c840",
"triggerType" : "PUSH"
}, {
"hash" : "8c32253ad0f6d9eb728789fad75763bb5703678d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1995",
"triggerID" : "8c32253ad0f6d9eb728789fad75763bb5703678d",
"triggerType" : "PUSH"
}, {
"hash" : "71bbe59aa8bf70b9aa8990a6dd2522eb4bf1d172",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2009",
"triggerID" : "71bbe59aa8bf70b9aa8990a6dd2522eb4bf1d172",
"triggerType" : "PUSH"
}, {
"hash" : "aefac7ec2f2e40bdf3ad4365ea6aa825803a439d",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "aefac7ec2f2e40bdf3ad4365ea6aa825803a439d",
"triggerType" : "PUSH"
}, {
"hash" : "86905cb855fecfeb9e4c023d526ad643fadb8084",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2012",
"triggerID" : "86905cb855fecfeb9e4c023d526ad643fadb8084",
"triggerType" : "PUSH"
}, {
"hash" : "b4d93b0168775cabd43609fec59bf2823dd81279",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2029",
"triggerID" : "b4d93b0168775cabd43609fec59bf2823dd81279",
"triggerType" : "PUSH"
}, {
"hash" : "38bf430ecb4e11839d686929be2511eed340e9df",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2035",
"triggerID" : "38bf430ecb4e11839d686929be2511eed340e9df",
"triggerType" : "PUSH"
}, {
"hash" : "7793fbdb9b93a129ef606cb2d73ea6e1e9074957",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2299",
"triggerID" : "7793fbdb9b93a129ef606cb2d73ea6e1e9074957",
"triggerType" : "PUSH"
}, {
"hash" : "9c0123c0f27f990d009b323bab75b76ceecf3dab",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2307",
"triggerID" : "9c0123c0f27f990d009b323bab75b76ceecf3dab",
"triggerType" : "PUSH"
}, {
"hash" : "af27976ba07afb02305c699aeb7a75a122013b62",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2330",
"triggerID" : "af27976ba07afb02305c699aeb7a75a122013b62",
"triggerType" : "PUSH"
}, {
"hash" : "9172241c58424cca8e8186d4409a8c36968a4790",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2344",
"triggerID" : "9172241c58424cca8e8186d4409a8c36968a4790",
"triggerType" : "PUSH"
}, {
"hash" : "13b80067ae00a760c57d184f70e2d98f63027e80",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2353",
"triggerID" : "13b80067ae00a760c57d184f70e2d98f63027e80",
"triggerType" : "PUSH"
}, {
"hash" : "ef750979cf449ff504da194e594cb62302eba360",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2366",
"triggerID" : "ef750979cf449ff504da194e594cb62302eba360",
"triggerType" : "PUSH"
}, {
"hash" : "16827c37f36d3b9fec18e0a35aff7599c45e0444",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2375",
"triggerID" : "16827c37f36d3b9fec18e0a35aff7599c45e0444",
"triggerType" : "PUSH"
}, {
"hash" : "0f68d426b01355537f5124b8f108345a36d3da39",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2378",
"triggerID" : "0f68d426b01355537f5124b8f108345a36d3da39",
"triggerType" : "PUSH"
}, {
"hash" : "2f3099197889fd5c48ae3b550cc523e0528ddb2c",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2380",
"triggerID" : "2f3099197889fd5c48ae3b550cc523e0528ddb2c",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* aefac7ec2f2e40bdf3ad4365ea6aa825803a439d UNKNOWN
* 2f3099197889fd5c48ae3b550cc523e0528ddb2c Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2380)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run travis` re-run the last Travis build
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot edited a comment on pull request #3590: [HUDI-2285][HUDI-2476] Metadata table synchronous design. Rebased and Squashed from pull/3426
Posted by GitBox <gi...@apache.org>.
hudi-bot edited a comment on pull request #3590:
URL: https://github.com/apache/hudi/pull/3590#issuecomment-912237120
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "648e36e1af308534bc050c3a19da008ab7a8c840",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1994",
"triggerID" : "648e36e1af308534bc050c3a19da008ab7a8c840",
"triggerType" : "PUSH"
}, {
"hash" : "8c32253ad0f6d9eb728789fad75763bb5703678d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1995",
"triggerID" : "8c32253ad0f6d9eb728789fad75763bb5703678d",
"triggerType" : "PUSH"
}, {
"hash" : "71bbe59aa8bf70b9aa8990a6dd2522eb4bf1d172",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2009",
"triggerID" : "71bbe59aa8bf70b9aa8990a6dd2522eb4bf1d172",
"triggerType" : "PUSH"
}, {
"hash" : "aefac7ec2f2e40bdf3ad4365ea6aa825803a439d",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "aefac7ec2f2e40bdf3ad4365ea6aa825803a439d",
"triggerType" : "PUSH"
}, {
"hash" : "86905cb855fecfeb9e4c023d526ad643fadb8084",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2012",
"triggerID" : "86905cb855fecfeb9e4c023d526ad643fadb8084",
"triggerType" : "PUSH"
}, {
"hash" : "b4d93b0168775cabd43609fec59bf2823dd81279",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2029",
"triggerID" : "b4d93b0168775cabd43609fec59bf2823dd81279",
"triggerType" : "PUSH"
}, {
"hash" : "38bf430ecb4e11839d686929be2511eed340e9df",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2035",
"triggerID" : "38bf430ecb4e11839d686929be2511eed340e9df",
"triggerType" : "PUSH"
}, {
"hash" : "7793fbdb9b93a129ef606cb2d73ea6e1e9074957",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2299",
"triggerID" : "7793fbdb9b93a129ef606cb2d73ea6e1e9074957",
"triggerType" : "PUSH"
}, {
"hash" : "9c0123c0f27f990d009b323bab75b76ceecf3dab",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2307",
"triggerID" : "9c0123c0f27f990d009b323bab75b76ceecf3dab",
"triggerType" : "PUSH"
}, {
"hash" : "af27976ba07afb02305c699aeb7a75a122013b62",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "af27976ba07afb02305c699aeb7a75a122013b62",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* aefac7ec2f2e40bdf3ad4365ea6aa825803a439d UNKNOWN
* 9c0123c0f27f990d009b323bab75b76ceecf3dab Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2307)
* af27976ba07afb02305c699aeb7a75a122013b62 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run travis` re-run the last Travis build
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot edited a comment on pull request #3590: [HUDI-2285][HUDI-2476] Metadata table synchronous design. Rebased and Squashed from pull/3426
Posted by GitBox <gi...@apache.org>.
hudi-bot edited a comment on pull request #3590:
URL: https://github.com/apache/hudi/pull/3590#issuecomment-912237120
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "648e36e1af308534bc050c3a19da008ab7a8c840",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1994",
"triggerID" : "648e36e1af308534bc050c3a19da008ab7a8c840",
"triggerType" : "PUSH"
}, {
"hash" : "8c32253ad0f6d9eb728789fad75763bb5703678d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1995",
"triggerID" : "8c32253ad0f6d9eb728789fad75763bb5703678d",
"triggerType" : "PUSH"
}, {
"hash" : "71bbe59aa8bf70b9aa8990a6dd2522eb4bf1d172",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2009",
"triggerID" : "71bbe59aa8bf70b9aa8990a6dd2522eb4bf1d172",
"triggerType" : "PUSH"
}, {
"hash" : "aefac7ec2f2e40bdf3ad4365ea6aa825803a439d",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "aefac7ec2f2e40bdf3ad4365ea6aa825803a439d",
"triggerType" : "PUSH"
}, {
"hash" : "86905cb855fecfeb9e4c023d526ad643fadb8084",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2012",
"triggerID" : "86905cb855fecfeb9e4c023d526ad643fadb8084",
"triggerType" : "PUSH"
}, {
"hash" : "b4d93b0168775cabd43609fec59bf2823dd81279",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2029",
"triggerID" : "b4d93b0168775cabd43609fec59bf2823dd81279",
"triggerType" : "PUSH"
}, {
"hash" : "38bf430ecb4e11839d686929be2511eed340e9df",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2035",
"triggerID" : "38bf430ecb4e11839d686929be2511eed340e9df",
"triggerType" : "PUSH"
}, {
"hash" : "7793fbdb9b93a129ef606cb2d73ea6e1e9074957",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2299",
"triggerID" : "7793fbdb9b93a129ef606cb2d73ea6e1e9074957",
"triggerType" : "PUSH"
}, {
"hash" : "9c0123c0f27f990d009b323bab75b76ceecf3dab",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2307",
"triggerID" : "9c0123c0f27f990d009b323bab75b76ceecf3dab",
"triggerType" : "PUSH"
}, {
"hash" : "af27976ba07afb02305c699aeb7a75a122013b62",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2330",
"triggerID" : "af27976ba07afb02305c699aeb7a75a122013b62",
"triggerType" : "PUSH"
}, {
"hash" : "9172241c58424cca8e8186d4409a8c36968a4790",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2344",
"triggerID" : "9172241c58424cca8e8186d4409a8c36968a4790",
"triggerType" : "PUSH"
}, {
"hash" : "13b80067ae00a760c57d184f70e2d98f63027e80",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2353",
"triggerID" : "13b80067ae00a760c57d184f70e2d98f63027e80",
"triggerType" : "PUSH"
}, {
"hash" : "ef750979cf449ff504da194e594cb62302eba360",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2366",
"triggerID" : "ef750979cf449ff504da194e594cb62302eba360",
"triggerType" : "PUSH"
}, {
"hash" : "16827c37f36d3b9fec18e0a35aff7599c45e0444",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2375",
"triggerID" : "16827c37f36d3b9fec18e0a35aff7599c45e0444",
"triggerType" : "PUSH"
}, {
"hash" : "0f68d426b01355537f5124b8f108345a36d3da39",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2378",
"triggerID" : "0f68d426b01355537f5124b8f108345a36d3da39",
"triggerType" : "PUSH"
}, {
"hash" : "2f3099197889fd5c48ae3b550cc523e0528ddb2c",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "2f3099197889fd5c48ae3b550cc523e0528ddb2c",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* aefac7ec2f2e40bdf3ad4365ea6aa825803a439d UNKNOWN
* 0f68d426b01355537f5124b8f108345a36d3da39 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2378)
* 2f3099197889fd5c48ae3b550cc523e0528ddb2c UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run travis` re-run the last Travis build
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot edited a comment on pull request #3590: [HUDI-2285][HUDI-2476] Metadata table synchronous design. Rebased and Squashed from pull/3426
Posted by GitBox <gi...@apache.org>.
hudi-bot edited a comment on pull request #3590:
URL: https://github.com/apache/hudi/pull/3590#issuecomment-912237120
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "648e36e1af308534bc050c3a19da008ab7a8c840",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1994",
"triggerID" : "648e36e1af308534bc050c3a19da008ab7a8c840",
"triggerType" : "PUSH"
}, {
"hash" : "8c32253ad0f6d9eb728789fad75763bb5703678d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1995",
"triggerID" : "8c32253ad0f6d9eb728789fad75763bb5703678d",
"triggerType" : "PUSH"
}, {
"hash" : "71bbe59aa8bf70b9aa8990a6dd2522eb4bf1d172",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2009",
"triggerID" : "71bbe59aa8bf70b9aa8990a6dd2522eb4bf1d172",
"triggerType" : "PUSH"
}, {
"hash" : "aefac7ec2f2e40bdf3ad4365ea6aa825803a439d",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "aefac7ec2f2e40bdf3ad4365ea6aa825803a439d",
"triggerType" : "PUSH"
}, {
"hash" : "86905cb855fecfeb9e4c023d526ad643fadb8084",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2012",
"triggerID" : "86905cb855fecfeb9e4c023d526ad643fadb8084",
"triggerType" : "PUSH"
}, {
"hash" : "b4d93b0168775cabd43609fec59bf2823dd81279",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2029",
"triggerID" : "b4d93b0168775cabd43609fec59bf2823dd81279",
"triggerType" : "PUSH"
}, {
"hash" : "38bf430ecb4e11839d686929be2511eed340e9df",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2035",
"triggerID" : "38bf430ecb4e11839d686929be2511eed340e9df",
"triggerType" : "PUSH"
}, {
"hash" : "7793fbdb9b93a129ef606cb2d73ea6e1e9074957",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2299",
"triggerID" : "7793fbdb9b93a129ef606cb2d73ea6e1e9074957",
"triggerType" : "PUSH"
}, {
"hash" : "9c0123c0f27f990d009b323bab75b76ceecf3dab",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2307",
"triggerID" : "9c0123c0f27f990d009b323bab75b76ceecf3dab",
"triggerType" : "PUSH"
}, {
"hash" : "af27976ba07afb02305c699aeb7a75a122013b62",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2330",
"triggerID" : "af27976ba07afb02305c699aeb7a75a122013b62",
"triggerType" : "PUSH"
}, {
"hash" : "9172241c58424cca8e8186d4409a8c36968a4790",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2344",
"triggerID" : "9172241c58424cca8e8186d4409a8c36968a4790",
"triggerType" : "PUSH"
}, {
"hash" : "13b80067ae00a760c57d184f70e2d98f63027e80",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2353",
"triggerID" : "13b80067ae00a760c57d184f70e2d98f63027e80",
"triggerType" : "PUSH"
}, {
"hash" : "ef750979cf449ff504da194e594cb62302eba360",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2366",
"triggerID" : "ef750979cf449ff504da194e594cb62302eba360",
"triggerType" : "PUSH"
}, {
"hash" : "16827c37f36d3b9fec18e0a35aff7599c45e0444",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2375",
"triggerID" : "16827c37f36d3b9fec18e0a35aff7599c45e0444",
"triggerType" : "PUSH"
}, {
"hash" : "0f68d426b01355537f5124b8f108345a36d3da39",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2378",
"triggerID" : "0f68d426b01355537f5124b8f108345a36d3da39",
"triggerType" : "PUSH"
}, {
"hash" : "2f3099197889fd5c48ae3b550cc523e0528ddb2c",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2380",
"triggerID" : "2f3099197889fd5c48ae3b550cc523e0528ddb2c",
"triggerType" : "PUSH"
}, {
"hash" : "c83d176e1d1ddab6bb86f91726060779a9a1519f",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2392",
"triggerID" : "c83d176e1d1ddab6bb86f91726060779a9a1519f",
"triggerType" : "PUSH"
}, {
"hash" : "008f0cd91de12d44030d837f460f3b0232d62ee5",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2416",
"triggerID" : "008f0cd91de12d44030d837f460f3b0232d62ee5",
"triggerType" : "PUSH"
}, {
"hash" : "92634fb730e723a9bdeb165348a4c747794be7e3",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2418",
"triggerID" : "92634fb730e723a9bdeb165348a4c747794be7e3",
"triggerType" : "PUSH"
}, {
"hash" : "3b5fbfa0cc69e82f7c23fc6aa5b4944e2dd77220",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2438",
"triggerID" : "3b5fbfa0cc69e82f7c23fc6aa5b4944e2dd77220",
"triggerType" : "PUSH"
}, {
"hash" : "87d2ec0cc60ef8d5b7a081b73a0318aedf1a2a90",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "87d2ec0cc60ef8d5b7a081b73a0318aedf1a2a90",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* aefac7ec2f2e40bdf3ad4365ea6aa825803a439d UNKNOWN
* 3b5fbfa0cc69e82f7c23fc6aa5b4944e2dd77220 Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2438)
* 87d2ec0cc60ef8d5b7a081b73a0318aedf1a2a90 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run travis` re-run the last Travis build
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] prashantwason commented on a change in pull request #3590: [HUDI-2285][HUDI-2476] Metadata table synchronous design. Rebased and Squashed from pull/3426
Posted by GitBox <gi...@apache.org>.
prashantwason commented on a change in pull request #3590:
URL: https://github.com/apache/hudi/pull/3590#discussion_r758755681
##########
File path: hudi-client/hudi-client-common/src/main/java/org/apache/hudi/table/HoodieTimelineArchiveLog.java
##########
@@ -200,20 +200,19 @@ public boolean archiveIfRequired(HoodieEngineContext context) throws IOException
.collect(Collectors.groupingBy(i -> Pair.of(i.getTimestamp(),
HoodieInstant.getComparableAction(i.getAction()))));
- // If metadata table is enabled, do not archive instants which are more recent that the latest synced
- // instant on the metadata table. This is required for metadata table sync.
+ // If metadata table is enabled, do not archive instants which are more recent that the last compaction on the
+ // metadata table.
if (config.isMetadataTableEnabled()) {
try (HoodieTableMetadata tableMetadata = HoodieTableMetadata.create(table.getContext(), config.getMetadataConfig(),
config.getBasePath(), FileSystemViewStorageConfig.SPILLABLE_DIR.defaultValue())) {
- Option<String> lastSyncedInstantTime = tableMetadata.getUpdateTime();
-
- if (lastSyncedInstantTime.isPresent()) {
- LOG.info("Limiting archiving of instants to last synced instant on metadata table at " + lastSyncedInstantTime.get());
- instants = instants.filter(i -> HoodieTimeline.compareTimestamps(i.getTimestamp(), HoodieTimeline.LESSER_THAN,
- lastSyncedInstantTime.get()));
- } else {
- LOG.info("Not archiving as there is no instants yet on the metadata table");
+ Option<String> latestCompactionTime = tableMetadata.getLatestCompactionTime();
Review comment:
@vinishjail97 This is required for proper functioning of the metadata table. Until a compaction occurs on the metadata table, any instants on the dataset cannot be archived.
Every action on the dataset (COMMIT, CLEAN, ROLLBACK, etc) will write to the metadata table and create a deltacommit on the metadata table. Currently, [after every 10 actions on the dataset ](https://github.com/apache/hudi/blob/master/hudi-common/src/main/java/org/apache/hudi/common/config/HoodieMetadataConfig.java#L75) the metadata table is compacted. You can lower this but will cause greater IO to compact the metadata table again and again.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] danny0405 commented on a change in pull request #3590: [HUDI-2285][HUDI-2476] Metadata table synchronous design. Rebased and Squashed from pull/3426
Posted by GitBox <gi...@apache.org>.
danny0405 commented on a change in pull request #3590:
URL: https://github.com/apache/hudi/pull/3590#discussion_r758999785
##########
File path: hudi-client/hudi-client-common/src/main/java/org/apache/hudi/table/HoodieTimelineArchiveLog.java
##########
@@ -200,20 +200,19 @@ public boolean archiveIfRequired(HoodieEngineContext context) throws IOException
.collect(Collectors.groupingBy(i -> Pair.of(i.getTimestamp(),
HoodieInstant.getComparableAction(i.getAction()))));
- // If metadata table is enabled, do not archive instants which are more recent that the latest synced
- // instant on the metadata table. This is required for metadata table sync.
+ // If metadata table is enabled, do not archive instants which are more recent that the last compaction on the
+ // metadata table.
if (config.isMetadataTableEnabled()) {
try (HoodieTableMetadata tableMetadata = HoodieTableMetadata.create(table.getContext(), config.getMetadataConfig(),
config.getBasePath(), FileSystemViewStorageConfig.SPILLABLE_DIR.defaultValue())) {
- Option<String> lastSyncedInstantTime = tableMetadata.getUpdateTime();
-
- if (lastSyncedInstantTime.isPresent()) {
- LOG.info("Limiting archiving of instants to last synced instant on metadata table at " + lastSyncedInstantTime.get());
- instants = instants.filter(i -> HoodieTimeline.compareTimestamps(i.getTimestamp(), HoodieTimeline.LESSER_THAN,
- lastSyncedInstantTime.get()));
- } else {
- LOG.info("Not archiving as there is no instants yet on the metadata table");
+ Option<String> latestCompactionTime = tableMetadata.getLatestCompactionTime();
Review comment:
> This validation requires us to check each tX.deltacommmit against corresponding instants on the dataset timeline
This is true but i did not see any reason why the delta commits on the timeline can not archive, somehow, the decision on whether a delta commit is committed successfully should not rely on whether it is archived, in other words, the archiving of delta commits should be transparent to the view callers.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] nsivabalan commented on a change in pull request #3590: [HUDI-2285][HUDI-2476] Metadata table synchronous design. Rebased and Squashed from pull/3426
Posted by GitBox <gi...@apache.org>.
nsivabalan commented on a change in pull request #3590:
URL: https://github.com/apache/hudi/pull/3590#discussion_r719513873
##########
File path: hudi-client/hudi-client-common/src/main/java/org/apache/hudi/client/AbstractHoodieWriteClient.java
##########
@@ -241,13 +242,16 @@ void emitCommitMetrics(String instantTime, HoodieCommitMetadata metadata, String
}
}
+ /**
+ * Any pre-commit actions like conflict resolution or updating metadata table goes here.
+ * @param instantTime commit instant time.
+ * @param metadata commit metadata for which pre commit is being invoked.
+ */
protected void preCommit(String instantTime, HoodieCommitMetadata metadata) {
- // no-op
- // TODO : Conflict resolution is not supported for Flink & Java engines
- }
-
- protected void syncTableMetadata() {
- // no-op
+ // Create a Hoodie table after startTxn which encapsulated the commits and files visible.
+ // Important to create this after the lock to ensure latest commits show up in the timeline without need for reload
Review comment:
this was already there. infact I tried to re-use existing table instance and tests started to fail. and then I noticed this comments and reverted it back.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot edited a comment on pull request #3590: [HUDI-2285][HUDI-2476] Metadata table synchronous design. Rebased and Squashed from pull/3426
Posted by GitBox <gi...@apache.org>.
hudi-bot edited a comment on pull request #3590:
URL: https://github.com/apache/hudi/pull/3590#issuecomment-912237120
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "648e36e1af308534bc050c3a19da008ab7a8c840",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1994",
"triggerID" : "648e36e1af308534bc050c3a19da008ab7a8c840",
"triggerType" : "PUSH"
}, {
"hash" : "8c32253ad0f6d9eb728789fad75763bb5703678d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1995",
"triggerID" : "8c32253ad0f6d9eb728789fad75763bb5703678d",
"triggerType" : "PUSH"
}, {
"hash" : "71bbe59aa8bf70b9aa8990a6dd2522eb4bf1d172",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2009",
"triggerID" : "71bbe59aa8bf70b9aa8990a6dd2522eb4bf1d172",
"triggerType" : "PUSH"
}, {
"hash" : "aefac7ec2f2e40bdf3ad4365ea6aa825803a439d",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "aefac7ec2f2e40bdf3ad4365ea6aa825803a439d",
"triggerType" : "PUSH"
}, {
"hash" : "86905cb855fecfeb9e4c023d526ad643fadb8084",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2012",
"triggerID" : "86905cb855fecfeb9e4c023d526ad643fadb8084",
"triggerType" : "PUSH"
}, {
"hash" : "b4d93b0168775cabd43609fec59bf2823dd81279",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2029",
"triggerID" : "b4d93b0168775cabd43609fec59bf2823dd81279",
"triggerType" : "PUSH"
}, {
"hash" : "38bf430ecb4e11839d686929be2511eed340e9df",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2035",
"triggerID" : "38bf430ecb4e11839d686929be2511eed340e9df",
"triggerType" : "PUSH"
}, {
"hash" : "7793fbdb9b93a129ef606cb2d73ea6e1e9074957",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2299",
"triggerID" : "7793fbdb9b93a129ef606cb2d73ea6e1e9074957",
"triggerType" : "PUSH"
}, {
"hash" : "9c0123c0f27f990d009b323bab75b76ceecf3dab",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2307",
"triggerID" : "9c0123c0f27f990d009b323bab75b76ceecf3dab",
"triggerType" : "PUSH"
}, {
"hash" : "af27976ba07afb02305c699aeb7a75a122013b62",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2330",
"triggerID" : "af27976ba07afb02305c699aeb7a75a122013b62",
"triggerType" : "PUSH"
}, {
"hash" : "9172241c58424cca8e8186d4409a8c36968a4790",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2344",
"triggerID" : "9172241c58424cca8e8186d4409a8c36968a4790",
"triggerType" : "PUSH"
}, {
"hash" : "13b80067ae00a760c57d184f70e2d98f63027e80",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2353",
"triggerID" : "13b80067ae00a760c57d184f70e2d98f63027e80",
"triggerType" : "PUSH"
}, {
"hash" : "ef750979cf449ff504da194e594cb62302eba360",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2366",
"triggerID" : "ef750979cf449ff504da194e594cb62302eba360",
"triggerType" : "PUSH"
}, {
"hash" : "16827c37f36d3b9fec18e0a35aff7599c45e0444",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2375",
"triggerID" : "16827c37f36d3b9fec18e0a35aff7599c45e0444",
"triggerType" : "PUSH"
}, {
"hash" : "0f68d426b01355537f5124b8f108345a36d3da39",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2378",
"triggerID" : "0f68d426b01355537f5124b8f108345a36d3da39",
"triggerType" : "PUSH"
}, {
"hash" : "2f3099197889fd5c48ae3b550cc523e0528ddb2c",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2380",
"triggerID" : "2f3099197889fd5c48ae3b550cc523e0528ddb2c",
"triggerType" : "PUSH"
}, {
"hash" : "c83d176e1d1ddab6bb86f91726060779a9a1519f",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2392",
"triggerID" : "c83d176e1d1ddab6bb86f91726060779a9a1519f",
"triggerType" : "PUSH"
}, {
"hash" : "008f0cd91de12d44030d837f460f3b0232d62ee5",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2416",
"triggerID" : "008f0cd91de12d44030d837f460f3b0232d62ee5",
"triggerType" : "PUSH"
}, {
"hash" : "92634fb730e723a9bdeb165348a4c747794be7e3",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2418",
"triggerID" : "92634fb730e723a9bdeb165348a4c747794be7e3",
"triggerType" : "PUSH"
}, {
"hash" : "3b5fbfa0cc69e82f7c23fc6aa5b4944e2dd77220",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2438",
"triggerID" : "3b5fbfa0cc69e82f7c23fc6aa5b4944e2dd77220",
"triggerType" : "PUSH"
}, {
"hash" : "87d2ec0cc60ef8d5b7a081b73a0318aedf1a2a90",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2451",
"triggerID" : "87d2ec0cc60ef8d5b7a081b73a0318aedf1a2a90",
"triggerType" : "PUSH"
}, {
"hash" : "c09f5e30662efd882ff9fc35f5750a8253171e9d",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "c09f5e30662efd882ff9fc35f5750a8253171e9d",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* aefac7ec2f2e40bdf3ad4365ea6aa825803a439d UNKNOWN
* 87d2ec0cc60ef8d5b7a081b73a0318aedf1a2a90 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2451)
* c09f5e30662efd882ff9fc35f5750a8253171e9d UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run travis` re-run the last Travis build
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] nsivabalan commented on a change in pull request #3590: [HUDI-2285][HUDI-2476] Metadata table synchronous design. Rebased and Squashed from pull/3426
Posted by GitBox <gi...@apache.org>.
nsivabalan commented on a change in pull request #3590:
URL: https://github.com/apache/hudi/pull/3590#discussion_r716259457
##########
File path: hudi-common/src/main/java/org/apache/hudi/metadata/HoodieBackedTableMetadata.java
##########
@@ -167,173 +170,150 @@ private void initIfNeeded() {
throw new HoodieIOException("Error merging records from metadata table for key :" + key, ioe);
} finally {
if (!reuse) {
- closeOrThrow();
+ close(partitionName);
}
}
}
- private void openReadersIfNeededOrThrow() {
- try {
- openReadersIfNeeded();
- } catch (IOException e) {
- throw new HoodieIOException("Error opening readers to the Metadata Table: ", e);
- }
- }
-
/**
* Returns a new pair of readers to the base and log files.
*/
- private void openReadersIfNeeded() throws IOException {
- if (reuse && (baseFileReader != null || logRecordScanner != null)) {
- // quickly exit out without synchronizing if reusing and readers are already open
- return;
- }
-
- // we always force synchronization, if reuse=false, to handle concurrent close() calls as well.
- synchronized (this) {
- if (baseFileReader != null || logRecordScanner != null) {
- return;
- }
-
- final long baseFileOpenMs;
- final long logScannerOpenMs;
+ private Pair<HoodieFileReader, HoodieMetadataMergedLogRecordScanner> openReadersIfNeeded(String key, String partitionName) throws IOException {
Review comment:
I might need some help in auditing the callers of getRecordKeyByKey. I see lot of callers to fetchAllPartitionPaths. Don't have good clarity on how to weed out required ones from rest. I will sync up with you sometime on this comment.
<img width="601" alt="Screen Shot 2021-09-26 at 4 46 08 PM" src="https://user-images.githubusercontent.com/513218/134823656-4c865c05-e468-427e-9bf6-8443353203a9.png">
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot edited a comment on pull request #3590: [HUDI-2285][HUDI-2476] Metadata table synchronous design. Rebased and Squashed from pull/3426
Posted by GitBox <gi...@apache.org>.
hudi-bot edited a comment on pull request #3590:
URL: https://github.com/apache/hudi/pull/3590#issuecomment-912237120
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "648e36e1af308534bc050c3a19da008ab7a8c840",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1994",
"triggerID" : "648e36e1af308534bc050c3a19da008ab7a8c840",
"triggerType" : "PUSH"
}, {
"hash" : "8c32253ad0f6d9eb728789fad75763bb5703678d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1995",
"triggerID" : "8c32253ad0f6d9eb728789fad75763bb5703678d",
"triggerType" : "PUSH"
}, {
"hash" : "71bbe59aa8bf70b9aa8990a6dd2522eb4bf1d172",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2009",
"triggerID" : "71bbe59aa8bf70b9aa8990a6dd2522eb4bf1d172",
"triggerType" : "PUSH"
}, {
"hash" : "aefac7ec2f2e40bdf3ad4365ea6aa825803a439d",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "aefac7ec2f2e40bdf3ad4365ea6aa825803a439d",
"triggerType" : "PUSH"
}, {
"hash" : "86905cb855fecfeb9e4c023d526ad643fadb8084",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2012",
"triggerID" : "86905cb855fecfeb9e4c023d526ad643fadb8084",
"triggerType" : "PUSH"
}, {
"hash" : "b4d93b0168775cabd43609fec59bf2823dd81279",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2029",
"triggerID" : "b4d93b0168775cabd43609fec59bf2823dd81279",
"triggerType" : "PUSH"
}, {
"hash" : "38bf430ecb4e11839d686929be2511eed340e9df",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2035",
"triggerID" : "38bf430ecb4e11839d686929be2511eed340e9df",
"triggerType" : "PUSH"
}, {
"hash" : "7793fbdb9b93a129ef606cb2d73ea6e1e9074957",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2299",
"triggerID" : "7793fbdb9b93a129ef606cb2d73ea6e1e9074957",
"triggerType" : "PUSH"
}, {
"hash" : "9c0123c0f27f990d009b323bab75b76ceecf3dab",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2307",
"triggerID" : "9c0123c0f27f990d009b323bab75b76ceecf3dab",
"triggerType" : "PUSH"
}, {
"hash" : "af27976ba07afb02305c699aeb7a75a122013b62",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2330",
"triggerID" : "af27976ba07afb02305c699aeb7a75a122013b62",
"triggerType" : "PUSH"
}, {
"hash" : "9172241c58424cca8e8186d4409a8c36968a4790",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2344",
"triggerID" : "9172241c58424cca8e8186d4409a8c36968a4790",
"triggerType" : "PUSH"
}, {
"hash" : "13b80067ae00a760c57d184f70e2d98f63027e80",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2353",
"triggerID" : "13b80067ae00a760c57d184f70e2d98f63027e80",
"triggerType" : "PUSH"
}, {
"hash" : "ef750979cf449ff504da194e594cb62302eba360",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2366",
"triggerID" : "ef750979cf449ff504da194e594cb62302eba360",
"triggerType" : "PUSH"
}, {
"hash" : "16827c37f36d3b9fec18e0a35aff7599c45e0444",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2375",
"triggerID" : "16827c37f36d3b9fec18e0a35aff7599c45e0444",
"triggerType" : "PUSH"
}, {
"hash" : "0f68d426b01355537f5124b8f108345a36d3da39",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2378",
"triggerID" : "0f68d426b01355537f5124b8f108345a36d3da39",
"triggerType" : "PUSH"
}, {
"hash" : "2f3099197889fd5c48ae3b550cc523e0528ddb2c",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2380",
"triggerID" : "2f3099197889fd5c48ae3b550cc523e0528ddb2c",
"triggerType" : "PUSH"
}, {
"hash" : "c83d176e1d1ddab6bb86f91726060779a9a1519f",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2392",
"triggerID" : "c83d176e1d1ddab6bb86f91726060779a9a1519f",
"triggerType" : "PUSH"
}, {
"hash" : "008f0cd91de12d44030d837f460f3b0232d62ee5",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2416",
"triggerID" : "008f0cd91de12d44030d837f460f3b0232d62ee5",
"triggerType" : "PUSH"
}, {
"hash" : "92634fb730e723a9bdeb165348a4c747794be7e3",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2418",
"triggerID" : "92634fb730e723a9bdeb165348a4c747794be7e3",
"triggerType" : "PUSH"
}, {
"hash" : "3b5fbfa0cc69e82f7c23fc6aa5b4944e2dd77220",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2438",
"triggerID" : "3b5fbfa0cc69e82f7c23fc6aa5b4944e2dd77220",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* aefac7ec2f2e40bdf3ad4365ea6aa825803a439d UNKNOWN
* 92634fb730e723a9bdeb165348a4c747794be7e3 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2418)
* 3b5fbfa0cc69e82f7c23fc6aa5b4944e2dd77220 Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2438)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run travis` re-run the last Travis build
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot edited a comment on pull request #3590: [HUDI-2285][HUDI-2476] Metadata table synchronous design. Rebased and Squashed from pull/3426
Posted by GitBox <gi...@apache.org>.
hudi-bot edited a comment on pull request #3590:
URL: https://github.com/apache/hudi/pull/3590#issuecomment-912237120
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "648e36e1af308534bc050c3a19da008ab7a8c840",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1994",
"triggerID" : "648e36e1af308534bc050c3a19da008ab7a8c840",
"triggerType" : "PUSH"
}, {
"hash" : "8c32253ad0f6d9eb728789fad75763bb5703678d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1995",
"triggerID" : "8c32253ad0f6d9eb728789fad75763bb5703678d",
"triggerType" : "PUSH"
}, {
"hash" : "71bbe59aa8bf70b9aa8990a6dd2522eb4bf1d172",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2009",
"triggerID" : "71bbe59aa8bf70b9aa8990a6dd2522eb4bf1d172",
"triggerType" : "PUSH"
}, {
"hash" : "aefac7ec2f2e40bdf3ad4365ea6aa825803a439d",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "aefac7ec2f2e40bdf3ad4365ea6aa825803a439d",
"triggerType" : "PUSH"
}, {
"hash" : "86905cb855fecfeb9e4c023d526ad643fadb8084",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2012",
"triggerID" : "86905cb855fecfeb9e4c023d526ad643fadb8084",
"triggerType" : "PUSH"
}, {
"hash" : "b4d93b0168775cabd43609fec59bf2823dd81279",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2029",
"triggerID" : "b4d93b0168775cabd43609fec59bf2823dd81279",
"triggerType" : "PUSH"
}, {
"hash" : "38bf430ecb4e11839d686929be2511eed340e9df",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2035",
"triggerID" : "38bf430ecb4e11839d686929be2511eed340e9df",
"triggerType" : "PUSH"
}, {
"hash" : "7793fbdb9b93a129ef606cb2d73ea6e1e9074957",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2299",
"triggerID" : "7793fbdb9b93a129ef606cb2d73ea6e1e9074957",
"triggerType" : "PUSH"
}, {
"hash" : "9c0123c0f27f990d009b323bab75b76ceecf3dab",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2307",
"triggerID" : "9c0123c0f27f990d009b323bab75b76ceecf3dab",
"triggerType" : "PUSH"
}, {
"hash" : "af27976ba07afb02305c699aeb7a75a122013b62",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2330",
"triggerID" : "af27976ba07afb02305c699aeb7a75a122013b62",
"triggerType" : "PUSH"
}, {
"hash" : "9172241c58424cca8e8186d4409a8c36968a4790",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2344",
"triggerID" : "9172241c58424cca8e8186d4409a8c36968a4790",
"triggerType" : "PUSH"
}, {
"hash" : "13b80067ae00a760c57d184f70e2d98f63027e80",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2353",
"triggerID" : "13b80067ae00a760c57d184f70e2d98f63027e80",
"triggerType" : "PUSH"
}, {
"hash" : "ef750979cf449ff504da194e594cb62302eba360",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2366",
"triggerID" : "ef750979cf449ff504da194e594cb62302eba360",
"triggerType" : "PUSH"
}, {
"hash" : "16827c37f36d3b9fec18e0a35aff7599c45e0444",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2375",
"triggerID" : "16827c37f36d3b9fec18e0a35aff7599c45e0444",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* aefac7ec2f2e40bdf3ad4365ea6aa825803a439d UNKNOWN
* ef750979cf449ff504da194e594cb62302eba360 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2366)
* 16827c37f36d3b9fec18e0a35aff7599c45e0444 Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2375)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run travis` re-run the last Travis build
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot edited a comment on pull request #3590: [HUDI-2285][HUDI-2476] Metadata table synchronous design. Rebased and Squashed from pull/3426
Posted by GitBox <gi...@apache.org>.
hudi-bot edited a comment on pull request #3590:
URL: https://github.com/apache/hudi/pull/3590#issuecomment-912237120
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "648e36e1af308534bc050c3a19da008ab7a8c840",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1994",
"triggerID" : "648e36e1af308534bc050c3a19da008ab7a8c840",
"triggerType" : "PUSH"
}, {
"hash" : "8c32253ad0f6d9eb728789fad75763bb5703678d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1995",
"triggerID" : "8c32253ad0f6d9eb728789fad75763bb5703678d",
"triggerType" : "PUSH"
}, {
"hash" : "71bbe59aa8bf70b9aa8990a6dd2522eb4bf1d172",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2009",
"triggerID" : "71bbe59aa8bf70b9aa8990a6dd2522eb4bf1d172",
"triggerType" : "PUSH"
}, {
"hash" : "aefac7ec2f2e40bdf3ad4365ea6aa825803a439d",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "aefac7ec2f2e40bdf3ad4365ea6aa825803a439d",
"triggerType" : "PUSH"
}, {
"hash" : "86905cb855fecfeb9e4c023d526ad643fadb8084",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2012",
"triggerID" : "86905cb855fecfeb9e4c023d526ad643fadb8084",
"triggerType" : "PUSH"
}, {
"hash" : "b4d93b0168775cabd43609fec59bf2823dd81279",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2029",
"triggerID" : "b4d93b0168775cabd43609fec59bf2823dd81279",
"triggerType" : "PUSH"
}, {
"hash" : "38bf430ecb4e11839d686929be2511eed340e9df",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2035",
"triggerID" : "38bf430ecb4e11839d686929be2511eed340e9df",
"triggerType" : "PUSH"
}, {
"hash" : "7793fbdb9b93a129ef606cb2d73ea6e1e9074957",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2299",
"triggerID" : "7793fbdb9b93a129ef606cb2d73ea6e1e9074957",
"triggerType" : "PUSH"
}, {
"hash" : "9c0123c0f27f990d009b323bab75b76ceecf3dab",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2307",
"triggerID" : "9c0123c0f27f990d009b323bab75b76ceecf3dab",
"triggerType" : "PUSH"
}, {
"hash" : "af27976ba07afb02305c699aeb7a75a122013b62",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2330",
"triggerID" : "af27976ba07afb02305c699aeb7a75a122013b62",
"triggerType" : "PUSH"
}, {
"hash" : "9172241c58424cca8e8186d4409a8c36968a4790",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2344",
"triggerID" : "9172241c58424cca8e8186d4409a8c36968a4790",
"triggerType" : "PUSH"
}, {
"hash" : "13b80067ae00a760c57d184f70e2d98f63027e80",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2353",
"triggerID" : "13b80067ae00a760c57d184f70e2d98f63027e80",
"triggerType" : "PUSH"
}, {
"hash" : "ef750979cf449ff504da194e594cb62302eba360",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2366",
"triggerID" : "ef750979cf449ff504da194e594cb62302eba360",
"triggerType" : "PUSH"
}, {
"hash" : "16827c37f36d3b9fec18e0a35aff7599c45e0444",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2375",
"triggerID" : "16827c37f36d3b9fec18e0a35aff7599c45e0444",
"triggerType" : "PUSH"
}, {
"hash" : "0f68d426b01355537f5124b8f108345a36d3da39",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2378",
"triggerID" : "0f68d426b01355537f5124b8f108345a36d3da39",
"triggerType" : "PUSH"
}, {
"hash" : "2f3099197889fd5c48ae3b550cc523e0528ddb2c",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2380",
"triggerID" : "2f3099197889fd5c48ae3b550cc523e0528ddb2c",
"triggerType" : "PUSH"
}, {
"hash" : "c83d176e1d1ddab6bb86f91726060779a9a1519f",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2392",
"triggerID" : "c83d176e1d1ddab6bb86f91726060779a9a1519f",
"triggerType" : "PUSH"
}, {
"hash" : "008f0cd91de12d44030d837f460f3b0232d62ee5",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2416",
"triggerID" : "008f0cd91de12d44030d837f460f3b0232d62ee5",
"triggerType" : "PUSH"
}, {
"hash" : "92634fb730e723a9bdeb165348a4c747794be7e3",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2418",
"triggerID" : "92634fb730e723a9bdeb165348a4c747794be7e3",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* aefac7ec2f2e40bdf3ad4365ea6aa825803a439d UNKNOWN
* 008f0cd91de12d44030d837f460f3b0232d62ee5 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2416)
* 92634fb730e723a9bdeb165348a4c747794be7e3 Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2418)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run travis` re-run the last Travis build
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot edited a comment on pull request #3590: [HUDI-2285][HUDI-2476] Metadata table synchronous design. Rebased and Squashed from pull/3426
Posted by GitBox <gi...@apache.org>.
hudi-bot edited a comment on pull request #3590:
URL: https://github.com/apache/hudi/pull/3590#issuecomment-912237120
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "648e36e1af308534bc050c3a19da008ab7a8c840",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1994",
"triggerID" : "648e36e1af308534bc050c3a19da008ab7a8c840",
"triggerType" : "PUSH"
}, {
"hash" : "8c32253ad0f6d9eb728789fad75763bb5703678d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1995",
"triggerID" : "8c32253ad0f6d9eb728789fad75763bb5703678d",
"triggerType" : "PUSH"
}, {
"hash" : "71bbe59aa8bf70b9aa8990a6dd2522eb4bf1d172",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2009",
"triggerID" : "71bbe59aa8bf70b9aa8990a6dd2522eb4bf1d172",
"triggerType" : "PUSH"
}, {
"hash" : "aefac7ec2f2e40bdf3ad4365ea6aa825803a439d",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "aefac7ec2f2e40bdf3ad4365ea6aa825803a439d",
"triggerType" : "PUSH"
}, {
"hash" : "86905cb855fecfeb9e4c023d526ad643fadb8084",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2012",
"triggerID" : "86905cb855fecfeb9e4c023d526ad643fadb8084",
"triggerType" : "PUSH"
}, {
"hash" : "b4d93b0168775cabd43609fec59bf2823dd81279",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2029",
"triggerID" : "b4d93b0168775cabd43609fec59bf2823dd81279",
"triggerType" : "PUSH"
}, {
"hash" : "38bf430ecb4e11839d686929be2511eed340e9df",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2035",
"triggerID" : "38bf430ecb4e11839d686929be2511eed340e9df",
"triggerType" : "PUSH"
}, {
"hash" : "7793fbdb9b93a129ef606cb2d73ea6e1e9074957",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2299",
"triggerID" : "7793fbdb9b93a129ef606cb2d73ea6e1e9074957",
"triggerType" : "PUSH"
}, {
"hash" : "9c0123c0f27f990d009b323bab75b76ceecf3dab",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2307",
"triggerID" : "9c0123c0f27f990d009b323bab75b76ceecf3dab",
"triggerType" : "PUSH"
}, {
"hash" : "af27976ba07afb02305c699aeb7a75a122013b62",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2330",
"triggerID" : "af27976ba07afb02305c699aeb7a75a122013b62",
"triggerType" : "PUSH"
}, {
"hash" : "9172241c58424cca8e8186d4409a8c36968a4790",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2344",
"triggerID" : "9172241c58424cca8e8186d4409a8c36968a4790",
"triggerType" : "PUSH"
}, {
"hash" : "13b80067ae00a760c57d184f70e2d98f63027e80",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2353",
"triggerID" : "13b80067ae00a760c57d184f70e2d98f63027e80",
"triggerType" : "PUSH"
}, {
"hash" : "ef750979cf449ff504da194e594cb62302eba360",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2366",
"triggerID" : "ef750979cf449ff504da194e594cb62302eba360",
"triggerType" : "PUSH"
}, {
"hash" : "16827c37f36d3b9fec18e0a35aff7599c45e0444",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2375",
"triggerID" : "16827c37f36d3b9fec18e0a35aff7599c45e0444",
"triggerType" : "PUSH"
}, {
"hash" : "0f68d426b01355537f5124b8f108345a36d3da39",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2378",
"triggerID" : "0f68d426b01355537f5124b8f108345a36d3da39",
"triggerType" : "PUSH"
}, {
"hash" : "2f3099197889fd5c48ae3b550cc523e0528ddb2c",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2380",
"triggerID" : "2f3099197889fd5c48ae3b550cc523e0528ddb2c",
"triggerType" : "PUSH"
}, {
"hash" : "c83d176e1d1ddab6bb86f91726060779a9a1519f",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2392",
"triggerID" : "c83d176e1d1ddab6bb86f91726060779a9a1519f",
"triggerType" : "PUSH"
}, {
"hash" : "008f0cd91de12d44030d837f460f3b0232d62ee5",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2416",
"triggerID" : "008f0cd91de12d44030d837f460f3b0232d62ee5",
"triggerType" : "PUSH"
}, {
"hash" : "92634fb730e723a9bdeb165348a4c747794be7e3",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2418",
"triggerID" : "92634fb730e723a9bdeb165348a4c747794be7e3",
"triggerType" : "PUSH"
}, {
"hash" : "3b5fbfa0cc69e82f7c23fc6aa5b4944e2dd77220",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2438",
"triggerID" : "3b5fbfa0cc69e82f7c23fc6aa5b4944e2dd77220",
"triggerType" : "PUSH"
}, {
"hash" : "87d2ec0cc60ef8d5b7a081b73a0318aedf1a2a90",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2451",
"triggerID" : "87d2ec0cc60ef8d5b7a081b73a0318aedf1a2a90",
"triggerType" : "PUSH"
}, {
"hash" : "c09f5e30662efd882ff9fc35f5750a8253171e9d",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2452",
"triggerID" : "c09f5e30662efd882ff9fc35f5750a8253171e9d",
"triggerType" : "PUSH"
}, {
"hash" : "b432e1db06e032b8071fa6c9e871ebc30b04b9eb",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2485",
"triggerID" : "b432e1db06e032b8071fa6c9e871ebc30b04b9eb",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* aefac7ec2f2e40bdf3ad4365ea6aa825803a439d UNKNOWN
* c09f5e30662efd882ff9fc35f5750a8253171e9d Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2452)
* b432e1db06e032b8071fa6c9e871ebc30b04b9eb Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2485)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run travis` re-run the last Travis build
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] nsivabalan commented on pull request #3590: [HUDI-2285][HUDI-2476] Metadata table synchronous design. Rebased and Squashed from pull/3426
Posted by GitBox <gi...@apache.org>.
nsivabalan commented on pull request #3590:
URL: https://github.com/apache/hudi/pull/3590#issuecomment-934887878
@hudi-bot azure run
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] nsivabalan commented on pull request #3590: [HUDI-2285][HUDI-2476] Metadata table synchronous design. Rebased and Squashed from pull/3426
Posted by GitBox <gi...@apache.org>.
nsivabalan commented on pull request #3590:
URL: https://github.com/apache/hudi/pull/3590#issuecomment-933938711
> @nsivabalan have we unit tested `HoodieBackedTableMetadata` or `BaseTableMetadata` here?
I started writing unit tests and later realized that all of our existing validation is happening using apis inBaseTableMetadata (getAllPartitionPaths and getAllFilesInPartition). Also there are tests to test reader wrt valid instants. So, I just enhanced existing tests and added extra validation that fileSystemMetadata is not used during these validations.
In summary, I ensured we have tests covering HoodieBackedTableMetadata and HoodieBackedTableMetadataWriter for all of its methods.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot edited a comment on pull request #3590: [HUDI-2285][HUDI-2476] Metadata table synchronous design. Rebased and Squashed from pull/3426
Posted by GitBox <gi...@apache.org>.
hudi-bot edited a comment on pull request #3590:
URL: https://github.com/apache/hudi/pull/3590#issuecomment-912237120
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "648e36e1af308534bc050c3a19da008ab7a8c840",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1994",
"triggerID" : "648e36e1af308534bc050c3a19da008ab7a8c840",
"triggerType" : "PUSH"
}, {
"hash" : "8c32253ad0f6d9eb728789fad75763bb5703678d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1995",
"triggerID" : "8c32253ad0f6d9eb728789fad75763bb5703678d",
"triggerType" : "PUSH"
}, {
"hash" : "71bbe59aa8bf70b9aa8990a6dd2522eb4bf1d172",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2009",
"triggerID" : "71bbe59aa8bf70b9aa8990a6dd2522eb4bf1d172",
"triggerType" : "PUSH"
}, {
"hash" : "aefac7ec2f2e40bdf3ad4365ea6aa825803a439d",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "aefac7ec2f2e40bdf3ad4365ea6aa825803a439d",
"triggerType" : "PUSH"
}, {
"hash" : "86905cb855fecfeb9e4c023d526ad643fadb8084",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2012",
"triggerID" : "86905cb855fecfeb9e4c023d526ad643fadb8084",
"triggerType" : "PUSH"
}, {
"hash" : "b4d93b0168775cabd43609fec59bf2823dd81279",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2029",
"triggerID" : "b4d93b0168775cabd43609fec59bf2823dd81279",
"triggerType" : "PUSH"
}, {
"hash" : "38bf430ecb4e11839d686929be2511eed340e9df",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2035",
"triggerID" : "38bf430ecb4e11839d686929be2511eed340e9df",
"triggerType" : "PUSH"
}, {
"hash" : "7793fbdb9b93a129ef606cb2d73ea6e1e9074957",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2299",
"triggerID" : "7793fbdb9b93a129ef606cb2d73ea6e1e9074957",
"triggerType" : "PUSH"
}, {
"hash" : "9c0123c0f27f990d009b323bab75b76ceecf3dab",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2307",
"triggerID" : "9c0123c0f27f990d009b323bab75b76ceecf3dab",
"triggerType" : "PUSH"
}, {
"hash" : "af27976ba07afb02305c699aeb7a75a122013b62",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2330",
"triggerID" : "af27976ba07afb02305c699aeb7a75a122013b62",
"triggerType" : "PUSH"
}, {
"hash" : "9172241c58424cca8e8186d4409a8c36968a4790",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2344",
"triggerID" : "9172241c58424cca8e8186d4409a8c36968a4790",
"triggerType" : "PUSH"
}, {
"hash" : "13b80067ae00a760c57d184f70e2d98f63027e80",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2353",
"triggerID" : "13b80067ae00a760c57d184f70e2d98f63027e80",
"triggerType" : "PUSH"
}, {
"hash" : "ef750979cf449ff504da194e594cb62302eba360",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2366",
"triggerID" : "ef750979cf449ff504da194e594cb62302eba360",
"triggerType" : "PUSH"
}, {
"hash" : "16827c37f36d3b9fec18e0a35aff7599c45e0444",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2375",
"triggerID" : "16827c37f36d3b9fec18e0a35aff7599c45e0444",
"triggerType" : "PUSH"
}, {
"hash" : "0f68d426b01355537f5124b8f108345a36d3da39",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2378",
"triggerID" : "0f68d426b01355537f5124b8f108345a36d3da39",
"triggerType" : "PUSH"
}, {
"hash" : "2f3099197889fd5c48ae3b550cc523e0528ddb2c",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2380",
"triggerID" : "2f3099197889fd5c48ae3b550cc523e0528ddb2c",
"triggerType" : "PUSH"
}, {
"hash" : "c83d176e1d1ddab6bb86f91726060779a9a1519f",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2392",
"triggerID" : "c83d176e1d1ddab6bb86f91726060779a9a1519f",
"triggerType" : "PUSH"
}, {
"hash" : "008f0cd91de12d44030d837f460f3b0232d62ee5",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2416",
"triggerID" : "008f0cd91de12d44030d837f460f3b0232d62ee5",
"triggerType" : "PUSH"
}, {
"hash" : "92634fb730e723a9bdeb165348a4c747794be7e3",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2418",
"triggerID" : "92634fb730e723a9bdeb165348a4c747794be7e3",
"triggerType" : "PUSH"
}, {
"hash" : "3b5fbfa0cc69e82f7c23fc6aa5b4944e2dd77220",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2438",
"triggerID" : "3b5fbfa0cc69e82f7c23fc6aa5b4944e2dd77220",
"triggerType" : "PUSH"
}, {
"hash" : "87d2ec0cc60ef8d5b7a081b73a0318aedf1a2a90",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2451",
"triggerID" : "87d2ec0cc60ef8d5b7a081b73a0318aedf1a2a90",
"triggerType" : "PUSH"
}, {
"hash" : "c09f5e30662efd882ff9fc35f5750a8253171e9d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2452",
"triggerID" : "c09f5e30662efd882ff9fc35f5750a8253171e9d",
"triggerType" : "PUSH"
}, {
"hash" : "b432e1db06e032b8071fa6c9e871ebc30b04b9eb",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2485",
"triggerID" : "b432e1db06e032b8071fa6c9e871ebc30b04b9eb",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* aefac7ec2f2e40bdf3ad4365ea6aa825803a439d UNKNOWN
* b432e1db06e032b8071fa6c9e871ebc30b04b9eb Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2485)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run travis` re-run the last Travis build
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] danny0405 commented on a change in pull request #3590: [HUDI-2285][HUDI-2476] Metadata table synchronous design. Rebased and Squashed from pull/3426
Posted by GitBox <gi...@apache.org>.
danny0405 commented on a change in pull request #3590:
URL: https://github.com/apache/hudi/pull/3590#discussion_r758909756
##########
File path: hudi-client/hudi-client-common/src/main/java/org/apache/hudi/table/HoodieTimelineArchiveLog.java
##########
@@ -200,20 +200,19 @@ public boolean archiveIfRequired(HoodieEngineContext context) throws IOException
.collect(Collectors.groupingBy(i -> Pair.of(i.getTimestamp(),
HoodieInstant.getComparableAction(i.getAction()))));
- // If metadata table is enabled, do not archive instants which are more recent that the latest synced
- // instant on the metadata table. This is required for metadata table sync.
+ // If metadata table is enabled, do not archive instants which are more recent that the last compaction on the
+ // metadata table.
if (config.isMetadataTableEnabled()) {
try (HoodieTableMetadata tableMetadata = HoodieTableMetadata.create(table.getContext(), config.getMetadataConfig(),
config.getBasePath(), FileSystemViewStorageConfig.SPILLABLE_DIR.defaultValue())) {
- Option<String> lastSyncedInstantTime = tableMetadata.getUpdateTime();
-
- if (lastSyncedInstantTime.isPresent()) {
- LOG.info("Limiting archiving of instants to last synced instant on metadata table at " + lastSyncedInstantTime.get());
- instants = instants.filter(i -> HoodieTimeline.compareTimestamps(i.getTimestamp(), HoodieTimeline.LESSER_THAN,
- lastSyncedInstantTime.get()));
- } else {
- LOG.info("Not archiving as there is no instants yet on the metadata table");
+ Option<String> latestCompactionTime = tableMetadata.getLatestCompactionTime();
Review comment:
> This is required for proper functioning of the metadata table.
What logic of metadata table relies on this restriction ? Can you please explain a little more ? Shouldn't the metadata table compaction just relies on its on log data blocks which are already committed metadata ?
##########
File path: hudi-client/hudi-client-common/src/main/java/org/apache/hudi/table/HoodieTimelineArchiveLog.java
##########
@@ -200,20 +200,19 @@ public boolean archiveIfRequired(HoodieEngineContext context) throws IOException
.collect(Collectors.groupingBy(i -> Pair.of(i.getTimestamp(),
HoodieInstant.getComparableAction(i.getAction()))));
- // If metadata table is enabled, do not archive instants which are more recent that the latest synced
- // instant on the metadata table. This is required for metadata table sync.
+ // If metadata table is enabled, do not archive instants which are more recent that the last compaction on the
+ // metadata table.
if (config.isMetadataTableEnabled()) {
try (HoodieTableMetadata tableMetadata = HoodieTableMetadata.create(table.getContext(), config.getMetadataConfig(),
config.getBasePath(), FileSystemViewStorageConfig.SPILLABLE_DIR.defaultValue())) {
- Option<String> lastSyncedInstantTime = tableMetadata.getUpdateTime();
-
- if (lastSyncedInstantTime.isPresent()) {
- LOG.info("Limiting archiving of instants to last synced instant on metadata table at " + lastSyncedInstantTime.get());
- instants = instants.filter(i -> HoodieTimeline.compareTimestamps(i.getTimestamp(), HoodieTimeline.LESSER_THAN,
- lastSyncedInstantTime.get()));
- } else {
- LOG.info("Not archiving as there is no instants yet on the metadata table");
+ Option<String> latestCompactionTime = tableMetadata.getLatestCompactionTime();
Review comment:
> This is required for proper functioning of the metadata table.
What logic of metadata table relies on this restriction ? Can you please explain a little more ? Shouldn't the metadata table compaction just relies on its on log data blocks which are already committed metadata ?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] prashantwason commented on a change in pull request #3590: [HUDI-2285][HUDI-2476] Metadata table synchronous design. Rebased and Squashed from pull/3426
Posted by GitBox <gi...@apache.org>.
prashantwason commented on a change in pull request #3590:
URL: https://github.com/apache/hudi/pull/3590#discussion_r727634907
##########
File path: hudi-client/hudi-client-common/src/main/java/org/apache/hudi/table/action/clean/CleanActionExecutor.java
##########
@@ -206,6 +209,19 @@ private HoodieCleanMetadata runClean(HoodieTable<T, I, K, O> table, HoodieInstan
}
}
+ /**
+ * Update metadata table if available. Any update to metadata table happens within data table lock.
+ * @param cleanMetadata intance of {@link HoodieCleanMetadata} to be applied to metadata.
+ */
+ private void writeMetadata(HoodieCleanMetadata cleanMetadata) {
Review comment:
Can we move the txnManager to the base class to save on code duplication and having to write two levels of these functions?
writeMetadata
writeCleanMetadata
##########
File path: hudi-client/hudi-spark-client/src/main/java/org/apache/hudi/metadata/SparkHoodieBackedTableMetadataWriter.java
##########
@@ -99,83 +94,93 @@ protected void initialize(HoodieEngineContext engineContext, HoodieTableMetaClie
@Override
protected void commit(List<HoodieRecord> records, String partitionName, String instantTime) {
ValidationUtils.checkState(enabled, "Metadata table cannot be committed to as it is not enabled");
- JavaRDD<HoodieRecord> recordRDD = prepRecords(records, partitionName);
+ JavaRDD<HoodieRecord> recordRDD = prepRecords(records, partitionName, 1);
try (SparkRDDWriteClient writeClient = new SparkRDDWriteClient(engineContext, metadataWriteConfig, true)) {
- writeClient.startCommitWithTime(instantTime);
+ if (!metadataMetaClient.getActiveTimeline().filterCompletedInstants().containsInstant(instantTime)) {
+ // if this is a new commit being applied to metadata for the first time
+ writeClient.startCommitWithTime(instantTime);
+ } else {
+ // this code path refers to a re-attempted commit that got committed to metadata table, but failed in datatable.
+ // for eg, lets say compaction c1 on 1st attempt succeeded in metadata table and failed before committing to datatable.
+ // when retried again, data table will first rollback pending compaction. these will be applied to metadata table, but all changes
+ // are upserts to metadata table and so only a new delta commit will be created.
+ // once rollback is complete, compaction will be retried again, which will eventually hit this code block where the respective commit is
+ // already part of completed commit. So, we have to manually remove the completed instant and proceed.
+ // and it is for the same reason we enabled withAllowMultiWriteOnSameInstant for metadata table.
+ HoodieInstant alreadyCompletedInstant = metadataMetaClient.getActiveTimeline().filterCompletedInstants().filter(entry -> entry.getTimestamp().equals(instantTime)).lastInstant().get();
+ HoodieActiveTimeline.deleteInstantFile(metadataMetaClient.getFs(), metadataMetaClient.getMetaPath(), alreadyCompletedInstant);
+ metadataMetaClient.reloadActiveTimeline();
Review comment:
This reload is extra as the next line 123 also reloads.
##########
File path: hudi-client/hudi-spark-client/src/test/java/org/apache/hudi/client/functional/TestHBaseIndex.java
##########
@@ -422,7 +424,8 @@ public void testSimpleTagLocationWithInvalidCommit() throws Exception {
@Test
public void testEnsureTagLocationUsesCommitTimeline() throws Exception {
// Load to memory
- HoodieWriteConfig config = getConfig();
+ HoodieWriteConfig config = getConfigBuilder(100, false, false)
+ .withMetadataConfig(HoodieMetadataConfig.newBuilder().enable(false).build()).build();
Review comment:
this is used so often that it may be good to introduce a helper function here:
withDisabledMetadata()
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] nsivabalan commented on pull request #3590: [HUDI-2285] Metadata table synchronous design. Rebased and Squashed from pull/3426
Posted by GitBox <gi...@apache.org>.
nsivabalan commented on pull request #3590:
URL: https://github.com/apache/hudi/pull/3590#issuecomment-922983380
@vinothchandar :
I have addressed most of the feedback from you except for few. Have responded in the other patch for those which is yet to be resolved.
wrt multi-writer and compaction, here is what is happening in this patch. (have covered this in the description of the patch, but anyways).
All writes to metadata table happens within the datatable lock. So, implicitly there are no multi-writers (including compaction, cleaning happening in datatable) to metadata table. There will always be only one writer. For regular writes to data table, we do conflict resolution, where as for compaction and clustering (data table), we don't do any conflict resolution. Remember taking a lock while committing compaction and clustering in data table is being added in this patch. So, please pay attention if there is something I am missing.
So, in summary, we have a constraint only single writer model to metadata table. May be we need to revisit this when we integrate record level index. But for now, this is our current state of things.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot edited a comment on pull request #3590: [WIP][HUDI-2285] Metadata table synchronous design. Rebased and Squashed from pull/3426
Posted by GitBox <gi...@apache.org>.
hudi-bot edited a comment on pull request #3590:
URL: https://github.com/apache/hudi/pull/3590#issuecomment-912237120
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "648e36e1af308534bc050c3a19da008ab7a8c840",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1994",
"triggerID" : "648e36e1af308534bc050c3a19da008ab7a8c840",
"triggerType" : "PUSH"
}, {
"hash" : "8c32253ad0f6d9eb728789fad75763bb5703678d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1995",
"triggerID" : "8c32253ad0f6d9eb728789fad75763bb5703678d",
"triggerType" : "PUSH"
}, {
"hash" : "71bbe59aa8bf70b9aa8990a6dd2522eb4bf1d172",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2009",
"triggerID" : "71bbe59aa8bf70b9aa8990a6dd2522eb4bf1d172",
"triggerType" : "PUSH"
}, {
"hash" : "aefac7ec2f2e40bdf3ad4365ea6aa825803a439d",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "aefac7ec2f2e40bdf3ad4365ea6aa825803a439d",
"triggerType" : "PUSH"
}, {
"hash" : "86905cb855fecfeb9e4c023d526ad643fadb8084",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2012",
"triggerID" : "86905cb855fecfeb9e4c023d526ad643fadb8084",
"triggerType" : "PUSH"
}, {
"hash" : "b4d93b0168775cabd43609fec59bf2823dd81279",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "b4d93b0168775cabd43609fec59bf2823dd81279",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* aefac7ec2f2e40bdf3ad4365ea6aa825803a439d UNKNOWN
* 86905cb855fecfeb9e4c023d526ad643fadb8084 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2012)
* b4d93b0168775cabd43609fec59bf2823dd81279 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run travis` re-run the last Travis build
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot edited a comment on pull request #3590: [WIP][HUDI-2285] Metadata table synchronous design. Rebased and Squashed from pull/3426
Posted by GitBox <gi...@apache.org>.
hudi-bot edited a comment on pull request #3590:
URL: https://github.com/apache/hudi/pull/3590#issuecomment-912237120
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "648e36e1af308534bc050c3a19da008ab7a8c840",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1994",
"triggerID" : "648e36e1af308534bc050c3a19da008ab7a8c840",
"triggerType" : "PUSH"
}, {
"hash" : "8c32253ad0f6d9eb728789fad75763bb5703678d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1995",
"triggerID" : "8c32253ad0f6d9eb728789fad75763bb5703678d",
"triggerType" : "PUSH"
}, {
"hash" : "71bbe59aa8bf70b9aa8990a6dd2522eb4bf1d172",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2009",
"triggerID" : "71bbe59aa8bf70b9aa8990a6dd2522eb4bf1d172",
"triggerType" : "PUSH"
}, {
"hash" : "aefac7ec2f2e40bdf3ad4365ea6aa825803a439d",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "aefac7ec2f2e40bdf3ad4365ea6aa825803a439d",
"triggerType" : "PUSH"
}, {
"hash" : "86905cb855fecfeb9e4c023d526ad643fadb8084",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2012",
"triggerID" : "86905cb855fecfeb9e4c023d526ad643fadb8084",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 71bbe59aa8bf70b9aa8990a6dd2522eb4bf1d172 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2009)
* aefac7ec2f2e40bdf3ad4365ea6aa825803a439d UNKNOWN
* 86905cb855fecfeb9e4c023d526ad643fadb8084 Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2012)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run travis` re-run the last Travis build
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot edited a comment on pull request #3590: [WIP] Sync metadata patch
Posted by GitBox <gi...@apache.org>.
hudi-bot edited a comment on pull request #3590:
URL: https://github.com/apache/hudi/pull/3590#issuecomment-912237120
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "648e36e1af308534bc050c3a19da008ab7a8c840",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1994",
"triggerID" : "648e36e1af308534bc050c3a19da008ab7a8c840",
"triggerType" : "PUSH"
}, {
"hash" : "8c32253ad0f6d9eb728789fad75763bb5703678d",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1995",
"triggerID" : "8c32253ad0f6d9eb728789fad75763bb5703678d",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 648e36e1af308534bc050c3a19da008ab7a8c840 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1994)
* 8c32253ad0f6d9eb728789fad75763bb5703678d Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1995)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run travis` re-run the last Travis build
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot edited a comment on pull request #3590: [HUDI-2285][HUDI-2476] Metadata table synchronous design. Rebased and Squashed from pull/3426
Posted by GitBox <gi...@apache.org>.
hudi-bot edited a comment on pull request #3590:
URL: https://github.com/apache/hudi/pull/3590#issuecomment-912237120
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "648e36e1af308534bc050c3a19da008ab7a8c840",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1994",
"triggerID" : "648e36e1af308534bc050c3a19da008ab7a8c840",
"triggerType" : "PUSH"
}, {
"hash" : "8c32253ad0f6d9eb728789fad75763bb5703678d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1995",
"triggerID" : "8c32253ad0f6d9eb728789fad75763bb5703678d",
"triggerType" : "PUSH"
}, {
"hash" : "71bbe59aa8bf70b9aa8990a6dd2522eb4bf1d172",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2009",
"triggerID" : "71bbe59aa8bf70b9aa8990a6dd2522eb4bf1d172",
"triggerType" : "PUSH"
}, {
"hash" : "aefac7ec2f2e40bdf3ad4365ea6aa825803a439d",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "aefac7ec2f2e40bdf3ad4365ea6aa825803a439d",
"triggerType" : "PUSH"
}, {
"hash" : "86905cb855fecfeb9e4c023d526ad643fadb8084",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2012",
"triggerID" : "86905cb855fecfeb9e4c023d526ad643fadb8084",
"triggerType" : "PUSH"
}, {
"hash" : "b4d93b0168775cabd43609fec59bf2823dd81279",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2029",
"triggerID" : "b4d93b0168775cabd43609fec59bf2823dd81279",
"triggerType" : "PUSH"
}, {
"hash" : "38bf430ecb4e11839d686929be2511eed340e9df",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2035",
"triggerID" : "38bf430ecb4e11839d686929be2511eed340e9df",
"triggerType" : "PUSH"
}, {
"hash" : "7793fbdb9b93a129ef606cb2d73ea6e1e9074957",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2299",
"triggerID" : "7793fbdb9b93a129ef606cb2d73ea6e1e9074957",
"triggerType" : "PUSH"
}, {
"hash" : "9c0123c0f27f990d009b323bab75b76ceecf3dab",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2307",
"triggerID" : "9c0123c0f27f990d009b323bab75b76ceecf3dab",
"triggerType" : "PUSH"
}, {
"hash" : "af27976ba07afb02305c699aeb7a75a122013b62",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2330",
"triggerID" : "af27976ba07afb02305c699aeb7a75a122013b62",
"triggerType" : "PUSH"
}, {
"hash" : "9172241c58424cca8e8186d4409a8c36968a4790",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2344",
"triggerID" : "9172241c58424cca8e8186d4409a8c36968a4790",
"triggerType" : "PUSH"
}, {
"hash" : "13b80067ae00a760c57d184f70e2d98f63027e80",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2353",
"triggerID" : "13b80067ae00a760c57d184f70e2d98f63027e80",
"triggerType" : "PUSH"
}, {
"hash" : "ef750979cf449ff504da194e594cb62302eba360",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2366",
"triggerID" : "ef750979cf449ff504da194e594cb62302eba360",
"triggerType" : "PUSH"
}, {
"hash" : "16827c37f36d3b9fec18e0a35aff7599c45e0444",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2375",
"triggerID" : "16827c37f36d3b9fec18e0a35aff7599c45e0444",
"triggerType" : "PUSH"
}, {
"hash" : "0f68d426b01355537f5124b8f108345a36d3da39",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2378",
"triggerID" : "0f68d426b01355537f5124b8f108345a36d3da39",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* aefac7ec2f2e40bdf3ad4365ea6aa825803a439d UNKNOWN
* 16827c37f36d3b9fec18e0a35aff7599c45e0444 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2375)
* 0f68d426b01355537f5124b8f108345a36d3da39 Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2378)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run travis` re-run the last Travis build
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot edited a comment on pull request #3590: [HUDI-2285][HUDI-2476] Metadata table synchronous design. Rebased and Squashed from pull/3426
Posted by GitBox <gi...@apache.org>.
hudi-bot edited a comment on pull request #3590:
URL: https://github.com/apache/hudi/pull/3590#issuecomment-912237120
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "648e36e1af308534bc050c3a19da008ab7a8c840",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1994",
"triggerID" : "648e36e1af308534bc050c3a19da008ab7a8c840",
"triggerType" : "PUSH"
}, {
"hash" : "8c32253ad0f6d9eb728789fad75763bb5703678d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1995",
"triggerID" : "8c32253ad0f6d9eb728789fad75763bb5703678d",
"triggerType" : "PUSH"
}, {
"hash" : "71bbe59aa8bf70b9aa8990a6dd2522eb4bf1d172",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2009",
"triggerID" : "71bbe59aa8bf70b9aa8990a6dd2522eb4bf1d172",
"triggerType" : "PUSH"
}, {
"hash" : "aefac7ec2f2e40bdf3ad4365ea6aa825803a439d",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "aefac7ec2f2e40bdf3ad4365ea6aa825803a439d",
"triggerType" : "PUSH"
}, {
"hash" : "86905cb855fecfeb9e4c023d526ad643fadb8084",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2012",
"triggerID" : "86905cb855fecfeb9e4c023d526ad643fadb8084",
"triggerType" : "PUSH"
}, {
"hash" : "b4d93b0168775cabd43609fec59bf2823dd81279",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2029",
"triggerID" : "b4d93b0168775cabd43609fec59bf2823dd81279",
"triggerType" : "PUSH"
}, {
"hash" : "38bf430ecb4e11839d686929be2511eed340e9df",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2035",
"triggerID" : "38bf430ecb4e11839d686929be2511eed340e9df",
"triggerType" : "PUSH"
}, {
"hash" : "7793fbdb9b93a129ef606cb2d73ea6e1e9074957",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2299",
"triggerID" : "7793fbdb9b93a129ef606cb2d73ea6e1e9074957",
"triggerType" : "PUSH"
}, {
"hash" : "9c0123c0f27f990d009b323bab75b76ceecf3dab",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2307",
"triggerID" : "9c0123c0f27f990d009b323bab75b76ceecf3dab",
"triggerType" : "PUSH"
}, {
"hash" : "af27976ba07afb02305c699aeb7a75a122013b62",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2330",
"triggerID" : "af27976ba07afb02305c699aeb7a75a122013b62",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* aefac7ec2f2e40bdf3ad4365ea6aa825803a439d UNKNOWN
* af27976ba07afb02305c699aeb7a75a122013b62 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2330)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run travis` re-run the last Travis build
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] nsivabalan commented on a change in pull request #3590: [HUDI-2285][HUDI-2476] Metadata table synchronous design. Rebased and Squashed from pull/3426
Posted by GitBox <gi...@apache.org>.
nsivabalan commented on a change in pull request #3590:
URL: https://github.com/apache/hudi/pull/3590#discussion_r716251519
##########
File path: hudi-client/hudi-spark-client/src/main/java/org/apache/hudi/client/SparkRDDWriteClient.java
##########
@@ -96,6 +94,11 @@ public SparkRDDWriteClient(HoodieEngineContext context, HoodieWriteConfig writeC
public SparkRDDWriteClient(HoodieEngineContext context, HoodieWriteConfig writeConfig,
Option<EmbeddedTimelineService> timelineService) {
super(context, writeConfig, timelineService);
+ if (config.isMetadataTableEnabled()) {
+ // If the metadata table does not exist, it should be bootstrapped here
Review comment:
yes, as of today its not safe. Our migration guideline w/ current design is documented [here](https://issues.apache.org/jira/browse/HUDI-2475). I have thought of an idea on how to support async bootstrapping. Don't think the scope is within this PR. I will pen down my thoughts in a jira and will let you know.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot edited a comment on pull request #3590: [HUDI-2285][HUDI-2476] Metadata table synchronous design. Rebased and Squashed from pull/3426
Posted by GitBox <gi...@apache.org>.
hudi-bot edited a comment on pull request #3590:
URL: https://github.com/apache/hudi/pull/3590#issuecomment-912237120
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "648e36e1af308534bc050c3a19da008ab7a8c840",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1994",
"triggerID" : "648e36e1af308534bc050c3a19da008ab7a8c840",
"triggerType" : "PUSH"
}, {
"hash" : "8c32253ad0f6d9eb728789fad75763bb5703678d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1995",
"triggerID" : "8c32253ad0f6d9eb728789fad75763bb5703678d",
"triggerType" : "PUSH"
}, {
"hash" : "71bbe59aa8bf70b9aa8990a6dd2522eb4bf1d172",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2009",
"triggerID" : "71bbe59aa8bf70b9aa8990a6dd2522eb4bf1d172",
"triggerType" : "PUSH"
}, {
"hash" : "aefac7ec2f2e40bdf3ad4365ea6aa825803a439d",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "aefac7ec2f2e40bdf3ad4365ea6aa825803a439d",
"triggerType" : "PUSH"
}, {
"hash" : "86905cb855fecfeb9e4c023d526ad643fadb8084",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2012",
"triggerID" : "86905cb855fecfeb9e4c023d526ad643fadb8084",
"triggerType" : "PUSH"
}, {
"hash" : "b4d93b0168775cabd43609fec59bf2823dd81279",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2029",
"triggerID" : "b4d93b0168775cabd43609fec59bf2823dd81279",
"triggerType" : "PUSH"
}, {
"hash" : "38bf430ecb4e11839d686929be2511eed340e9df",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2035",
"triggerID" : "38bf430ecb4e11839d686929be2511eed340e9df",
"triggerType" : "PUSH"
}, {
"hash" : "7793fbdb9b93a129ef606cb2d73ea6e1e9074957",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2299",
"triggerID" : "7793fbdb9b93a129ef606cb2d73ea6e1e9074957",
"triggerType" : "PUSH"
}, {
"hash" : "9c0123c0f27f990d009b323bab75b76ceecf3dab",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2307",
"triggerID" : "9c0123c0f27f990d009b323bab75b76ceecf3dab",
"triggerType" : "PUSH"
}, {
"hash" : "af27976ba07afb02305c699aeb7a75a122013b62",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2330",
"triggerID" : "af27976ba07afb02305c699aeb7a75a122013b62",
"triggerType" : "PUSH"
}, {
"hash" : "9172241c58424cca8e8186d4409a8c36968a4790",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2344",
"triggerID" : "9172241c58424cca8e8186d4409a8c36968a4790",
"triggerType" : "PUSH"
}, {
"hash" : "13b80067ae00a760c57d184f70e2d98f63027e80",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2353",
"triggerID" : "13b80067ae00a760c57d184f70e2d98f63027e80",
"triggerType" : "PUSH"
}, {
"hash" : "ef750979cf449ff504da194e594cb62302eba360",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2366",
"triggerID" : "ef750979cf449ff504da194e594cb62302eba360",
"triggerType" : "PUSH"
}, {
"hash" : "16827c37f36d3b9fec18e0a35aff7599c45e0444",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2375",
"triggerID" : "16827c37f36d3b9fec18e0a35aff7599c45e0444",
"triggerType" : "PUSH"
}, {
"hash" : "0f68d426b01355537f5124b8f108345a36d3da39",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2378",
"triggerID" : "0f68d426b01355537f5124b8f108345a36d3da39",
"triggerType" : "PUSH"
}, {
"hash" : "2f3099197889fd5c48ae3b550cc523e0528ddb2c",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2380",
"triggerID" : "2f3099197889fd5c48ae3b550cc523e0528ddb2c",
"triggerType" : "PUSH"
}, {
"hash" : "c83d176e1d1ddab6bb86f91726060779a9a1519f",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2392",
"triggerID" : "c83d176e1d1ddab6bb86f91726060779a9a1519f",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* aefac7ec2f2e40bdf3ad4365ea6aa825803a439d UNKNOWN
* 2f3099197889fd5c48ae3b550cc523e0528ddb2c Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2380)
* c83d176e1d1ddab6bb86f91726060779a9a1519f Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2392)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run travis` re-run the last Travis build
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] nsivabalan commented on a change in pull request #3590: [HUDI-2285][HUDI-2476] Metadata table synchronous design. Rebased and Squashed from pull/3426
Posted by GitBox <gi...@apache.org>.
nsivabalan commented on a change in pull request #3590:
URL: https://github.com/apache/hudi/pull/3590#discussion_r716094734
##########
File path: hudi-client/hudi-client-common/src/main/java/org/apache/hudi/client/AbstractHoodieWriteClient.java
##########
@@ -273,6 +275,7 @@ public void bootstrap(Option<Map<String, String>> extraMetadata) {
public void rollbackFailedBootstrap() {
LOG.info("Rolling back pending bootstrap if present");
HoodieTable<T, I, K, O> table = createTable(config, hadoopConf);
+ table.getHoodieView().sync();
Review comment:
getTableAndInitCtx is in SparkRddWriteClient :( . I have fixed it based on my changes in the other [PR](https://github.com/apache/hudi/pull/3698). by adding an overloaded method for createTable(config, hadoopConf. refreshTimeline).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] nsivabalan commented on pull request #3590: [HUDI-2285] Metadata table synchronous design. Rebased and Squashed from pull/3426
Posted by GitBox <gi...@apache.org>.
nsivabalan commented on pull request #3590:
URL: https://github.com/apache/hudi/pull/3590#issuecomment-923333856
So, given this approach, we could also support async compaction and clustering in metadata table.
Here is what we could do.
all things stay same wrt data table. i.e.
take locks and do conflict resolution for all regular writes, commit and release locks.
take locks and do conflict resolution while scheduling compaction/clustering and release locks.
take locks and commit compaction and clustering.
when it comes to metadata table. We will enable multi-writer mode in metadata table. (As of this patch, we have only single writer mode for metadata table)
all writes to metadata table happens within data table lock. And so wrt new delta commits to metadata table, it is always going to be a single writer implicitly.
after committing to metadata table, we can just schedule compaction and cleaning if something is available. This internally will take the lock for metadata table and check for any conflicts, but since there are no other writers, we should be good.
and once the commit and scheduling completes, we return to data table, make the commit and release the lockk.
Later when async compaction for metadata table is about to get committed, we take metadata table lock and make the commit. this will ensure this may not collide with regular delta commits happening to metadata table. We may not invoke any conflict resolution here similar to how it is done in data table.
But one major issue we need to fix here is the ConflictResolutionStrategy: as of now, there are any pending or complete compactions after current commit of interest, writes will fail. since all of them are going to operate on the same partition with one file group, there will definitely be conflict. So, just for metadata, we might want to consider if we can come up with a special conflict resolution strategy where we consider only new writes as conflicts and not any scheduled compaction). I need to understand the implications of this in more finer detail. But just putting it out here.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot edited a comment on pull request #3590: [HUDI-2285][HUDI-2476] Metadata table synchronous design. Rebased and Squashed from pull/3426
Posted by GitBox <gi...@apache.org>.
hudi-bot edited a comment on pull request #3590:
URL: https://github.com/apache/hudi/pull/3590#issuecomment-912237120
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "648e36e1af308534bc050c3a19da008ab7a8c840",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1994",
"triggerID" : "648e36e1af308534bc050c3a19da008ab7a8c840",
"triggerType" : "PUSH"
}, {
"hash" : "8c32253ad0f6d9eb728789fad75763bb5703678d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1995",
"triggerID" : "8c32253ad0f6d9eb728789fad75763bb5703678d",
"triggerType" : "PUSH"
}, {
"hash" : "71bbe59aa8bf70b9aa8990a6dd2522eb4bf1d172",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2009",
"triggerID" : "71bbe59aa8bf70b9aa8990a6dd2522eb4bf1d172",
"triggerType" : "PUSH"
}, {
"hash" : "aefac7ec2f2e40bdf3ad4365ea6aa825803a439d",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "aefac7ec2f2e40bdf3ad4365ea6aa825803a439d",
"triggerType" : "PUSH"
}, {
"hash" : "86905cb855fecfeb9e4c023d526ad643fadb8084",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2012",
"triggerID" : "86905cb855fecfeb9e4c023d526ad643fadb8084",
"triggerType" : "PUSH"
}, {
"hash" : "b4d93b0168775cabd43609fec59bf2823dd81279",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2029",
"triggerID" : "b4d93b0168775cabd43609fec59bf2823dd81279",
"triggerType" : "PUSH"
}, {
"hash" : "38bf430ecb4e11839d686929be2511eed340e9df",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2035",
"triggerID" : "38bf430ecb4e11839d686929be2511eed340e9df",
"triggerType" : "PUSH"
}, {
"hash" : "7793fbdb9b93a129ef606cb2d73ea6e1e9074957",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2299",
"triggerID" : "7793fbdb9b93a129ef606cb2d73ea6e1e9074957",
"triggerType" : "PUSH"
}, {
"hash" : "9c0123c0f27f990d009b323bab75b76ceecf3dab",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2307",
"triggerID" : "9c0123c0f27f990d009b323bab75b76ceecf3dab",
"triggerType" : "PUSH"
}, {
"hash" : "af27976ba07afb02305c699aeb7a75a122013b62",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2330",
"triggerID" : "af27976ba07afb02305c699aeb7a75a122013b62",
"triggerType" : "PUSH"
}, {
"hash" : "9172241c58424cca8e8186d4409a8c36968a4790",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2344",
"triggerID" : "9172241c58424cca8e8186d4409a8c36968a4790",
"triggerType" : "PUSH"
}, {
"hash" : "13b80067ae00a760c57d184f70e2d98f63027e80",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "13b80067ae00a760c57d184f70e2d98f63027e80",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* aefac7ec2f2e40bdf3ad4365ea6aa825803a439d UNKNOWN
* 9172241c58424cca8e8186d4409a8c36968a4790 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2344)
* 13b80067ae00a760c57d184f70e2d98f63027e80 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run travis` re-run the last Travis build
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot edited a comment on pull request #3590: [HUDI-2285][HUDI-2476] Metadata table synchronous design. Rebased and Squashed from pull/3426
Posted by GitBox <gi...@apache.org>.
hudi-bot edited a comment on pull request #3590:
URL: https://github.com/apache/hudi/pull/3590#issuecomment-912237120
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "648e36e1af308534bc050c3a19da008ab7a8c840",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1994",
"triggerID" : "648e36e1af308534bc050c3a19da008ab7a8c840",
"triggerType" : "PUSH"
}, {
"hash" : "8c32253ad0f6d9eb728789fad75763bb5703678d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1995",
"triggerID" : "8c32253ad0f6d9eb728789fad75763bb5703678d",
"triggerType" : "PUSH"
}, {
"hash" : "71bbe59aa8bf70b9aa8990a6dd2522eb4bf1d172",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2009",
"triggerID" : "71bbe59aa8bf70b9aa8990a6dd2522eb4bf1d172",
"triggerType" : "PUSH"
}, {
"hash" : "aefac7ec2f2e40bdf3ad4365ea6aa825803a439d",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "aefac7ec2f2e40bdf3ad4365ea6aa825803a439d",
"triggerType" : "PUSH"
}, {
"hash" : "86905cb855fecfeb9e4c023d526ad643fadb8084",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2012",
"triggerID" : "86905cb855fecfeb9e4c023d526ad643fadb8084",
"triggerType" : "PUSH"
}, {
"hash" : "b4d93b0168775cabd43609fec59bf2823dd81279",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2029",
"triggerID" : "b4d93b0168775cabd43609fec59bf2823dd81279",
"triggerType" : "PUSH"
}, {
"hash" : "38bf430ecb4e11839d686929be2511eed340e9df",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2035",
"triggerID" : "38bf430ecb4e11839d686929be2511eed340e9df",
"triggerType" : "PUSH"
}, {
"hash" : "7793fbdb9b93a129ef606cb2d73ea6e1e9074957",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2299",
"triggerID" : "7793fbdb9b93a129ef606cb2d73ea6e1e9074957",
"triggerType" : "PUSH"
}, {
"hash" : "9c0123c0f27f990d009b323bab75b76ceecf3dab",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2307",
"triggerID" : "9c0123c0f27f990d009b323bab75b76ceecf3dab",
"triggerType" : "PUSH"
}, {
"hash" : "af27976ba07afb02305c699aeb7a75a122013b62",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2330",
"triggerID" : "af27976ba07afb02305c699aeb7a75a122013b62",
"triggerType" : "PUSH"
}, {
"hash" : "9172241c58424cca8e8186d4409a8c36968a4790",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2344",
"triggerID" : "9172241c58424cca8e8186d4409a8c36968a4790",
"triggerType" : "PUSH"
}, {
"hash" : "13b80067ae00a760c57d184f70e2d98f63027e80",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2353",
"triggerID" : "13b80067ae00a760c57d184f70e2d98f63027e80",
"triggerType" : "PUSH"
}, {
"hash" : "ef750979cf449ff504da194e594cb62302eba360",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2366",
"triggerID" : "ef750979cf449ff504da194e594cb62302eba360",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* aefac7ec2f2e40bdf3ad4365ea6aa825803a439d UNKNOWN
* 13b80067ae00a760c57d184f70e2d98f63027e80 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2353)
* ef750979cf449ff504da194e594cb62302eba360 Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2366)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run travis` re-run the last Travis build
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot edited a comment on pull request #3590: [HUDI-2285][HUDI-2476] Metadata table synchronous design. Rebased and Squashed from pull/3426
Posted by GitBox <gi...@apache.org>.
hudi-bot edited a comment on pull request #3590:
URL: https://github.com/apache/hudi/pull/3590#issuecomment-912237120
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "648e36e1af308534bc050c3a19da008ab7a8c840",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1994",
"triggerID" : "648e36e1af308534bc050c3a19da008ab7a8c840",
"triggerType" : "PUSH"
}, {
"hash" : "8c32253ad0f6d9eb728789fad75763bb5703678d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1995",
"triggerID" : "8c32253ad0f6d9eb728789fad75763bb5703678d",
"triggerType" : "PUSH"
}, {
"hash" : "71bbe59aa8bf70b9aa8990a6dd2522eb4bf1d172",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2009",
"triggerID" : "71bbe59aa8bf70b9aa8990a6dd2522eb4bf1d172",
"triggerType" : "PUSH"
}, {
"hash" : "aefac7ec2f2e40bdf3ad4365ea6aa825803a439d",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "aefac7ec2f2e40bdf3ad4365ea6aa825803a439d",
"triggerType" : "PUSH"
}, {
"hash" : "86905cb855fecfeb9e4c023d526ad643fadb8084",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2012",
"triggerID" : "86905cb855fecfeb9e4c023d526ad643fadb8084",
"triggerType" : "PUSH"
}, {
"hash" : "b4d93b0168775cabd43609fec59bf2823dd81279",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2029",
"triggerID" : "b4d93b0168775cabd43609fec59bf2823dd81279",
"triggerType" : "PUSH"
}, {
"hash" : "38bf430ecb4e11839d686929be2511eed340e9df",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2035",
"triggerID" : "38bf430ecb4e11839d686929be2511eed340e9df",
"triggerType" : "PUSH"
}, {
"hash" : "7793fbdb9b93a129ef606cb2d73ea6e1e9074957",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2299",
"triggerID" : "7793fbdb9b93a129ef606cb2d73ea6e1e9074957",
"triggerType" : "PUSH"
}, {
"hash" : "9c0123c0f27f990d009b323bab75b76ceecf3dab",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2307",
"triggerID" : "9c0123c0f27f990d009b323bab75b76ceecf3dab",
"triggerType" : "PUSH"
}, {
"hash" : "af27976ba07afb02305c699aeb7a75a122013b62",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2330",
"triggerID" : "af27976ba07afb02305c699aeb7a75a122013b62",
"triggerType" : "PUSH"
}, {
"hash" : "9172241c58424cca8e8186d4409a8c36968a4790",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2344",
"triggerID" : "9172241c58424cca8e8186d4409a8c36968a4790",
"triggerType" : "PUSH"
}, {
"hash" : "13b80067ae00a760c57d184f70e2d98f63027e80",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2353",
"triggerID" : "13b80067ae00a760c57d184f70e2d98f63027e80",
"triggerType" : "PUSH"
}, {
"hash" : "ef750979cf449ff504da194e594cb62302eba360",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2366",
"triggerID" : "ef750979cf449ff504da194e594cb62302eba360",
"triggerType" : "PUSH"
}, {
"hash" : "16827c37f36d3b9fec18e0a35aff7599c45e0444",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2375",
"triggerID" : "16827c37f36d3b9fec18e0a35aff7599c45e0444",
"triggerType" : "PUSH"
}, {
"hash" : "0f68d426b01355537f5124b8f108345a36d3da39",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2378",
"triggerID" : "0f68d426b01355537f5124b8f108345a36d3da39",
"triggerType" : "PUSH"
}, {
"hash" : "2f3099197889fd5c48ae3b550cc523e0528ddb2c",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2380",
"triggerID" : "2f3099197889fd5c48ae3b550cc523e0528ddb2c",
"triggerType" : "PUSH"
}, {
"hash" : "c83d176e1d1ddab6bb86f91726060779a9a1519f",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2392",
"triggerID" : "c83d176e1d1ddab6bb86f91726060779a9a1519f",
"triggerType" : "PUSH"
}, {
"hash" : "008f0cd91de12d44030d837f460f3b0232d62ee5",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2416",
"triggerID" : "008f0cd91de12d44030d837f460f3b0232d62ee5",
"triggerType" : "PUSH"
}, {
"hash" : "92634fb730e723a9bdeb165348a4c747794be7e3",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2418",
"triggerID" : "92634fb730e723a9bdeb165348a4c747794be7e3",
"triggerType" : "PUSH"
}, {
"hash" : "3b5fbfa0cc69e82f7c23fc6aa5b4944e2dd77220",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2438",
"triggerID" : "3b5fbfa0cc69e82f7c23fc6aa5b4944e2dd77220",
"triggerType" : "PUSH"
}, {
"hash" : "87d2ec0cc60ef8d5b7a081b73a0318aedf1a2a90",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2451",
"triggerID" : "87d2ec0cc60ef8d5b7a081b73a0318aedf1a2a90",
"triggerType" : "PUSH"
}, {
"hash" : "c09f5e30662efd882ff9fc35f5750a8253171e9d",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2452",
"triggerID" : "c09f5e30662efd882ff9fc35f5750a8253171e9d",
"triggerType" : "PUSH"
}, {
"hash" : "b432e1db06e032b8071fa6c9e871ebc30b04b9eb",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "b432e1db06e032b8071fa6c9e871ebc30b04b9eb",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* aefac7ec2f2e40bdf3ad4365ea6aa825803a439d UNKNOWN
* c09f5e30662efd882ff9fc35f5750a8253171e9d Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2452)
* b432e1db06e032b8071fa6c9e871ebc30b04b9eb UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run travis` re-run the last Travis build
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] nsivabalan commented on pull request #3590: [HUDI-2285][HUDI-2476] Metadata table synchronous design. Rebased and Squashed from pull/3426
Posted by GitBox <gi...@apache.org>.
nsivabalan commented on pull request #3590:
URL: https://github.com/apache/hudi/pull/3590#issuecomment-934032408
@hudi-bot azure run
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] prashantwason commented on a change in pull request #3590: [HUDI-2285][HUDI-2476] Metadata table synchronous design. Rebased and Squashed from pull/3426
Posted by GitBox <gi...@apache.org>.
prashantwason commented on a change in pull request #3590:
URL: https://github.com/apache/hudi/pull/3590#discussion_r727634907
##########
File path: hudi-client/hudi-client-common/src/main/java/org/apache/hudi/table/action/clean/CleanActionExecutor.java
##########
@@ -206,6 +209,19 @@ private HoodieCleanMetadata runClean(HoodieTable<T, I, K, O> table, HoodieInstan
}
}
+ /**
+ * Update metadata table if available. Any update to metadata table happens within data table lock.
+ * @param cleanMetadata intance of {@link HoodieCleanMetadata} to be applied to metadata.
+ */
+ private void writeMetadata(HoodieCleanMetadata cleanMetadata) {
Review comment:
Can we move the txnManager to the base class to save on code duplication and having to write two levels of these functions?
writeMetadata
writeCleanMetadata
##########
File path: hudi-client/hudi-spark-client/src/main/java/org/apache/hudi/metadata/SparkHoodieBackedTableMetadataWriter.java
##########
@@ -99,83 +94,93 @@ protected void initialize(HoodieEngineContext engineContext, HoodieTableMetaClie
@Override
protected void commit(List<HoodieRecord> records, String partitionName, String instantTime) {
ValidationUtils.checkState(enabled, "Metadata table cannot be committed to as it is not enabled");
- JavaRDD<HoodieRecord> recordRDD = prepRecords(records, partitionName);
+ JavaRDD<HoodieRecord> recordRDD = prepRecords(records, partitionName, 1);
try (SparkRDDWriteClient writeClient = new SparkRDDWriteClient(engineContext, metadataWriteConfig, true)) {
- writeClient.startCommitWithTime(instantTime);
+ if (!metadataMetaClient.getActiveTimeline().filterCompletedInstants().containsInstant(instantTime)) {
+ // if this is a new commit being applied to metadata for the first time
+ writeClient.startCommitWithTime(instantTime);
+ } else {
+ // this code path refers to a re-attempted commit that got committed to metadata table, but failed in datatable.
+ // for eg, lets say compaction c1 on 1st attempt succeeded in metadata table and failed before committing to datatable.
+ // when retried again, data table will first rollback pending compaction. these will be applied to metadata table, but all changes
+ // are upserts to metadata table and so only a new delta commit will be created.
+ // once rollback is complete, compaction will be retried again, which will eventually hit this code block where the respective commit is
+ // already part of completed commit. So, we have to manually remove the completed instant and proceed.
+ // and it is for the same reason we enabled withAllowMultiWriteOnSameInstant for metadata table.
+ HoodieInstant alreadyCompletedInstant = metadataMetaClient.getActiveTimeline().filterCompletedInstants().filter(entry -> entry.getTimestamp().equals(instantTime)).lastInstant().get();
+ HoodieActiveTimeline.deleteInstantFile(metadataMetaClient.getFs(), metadataMetaClient.getMetaPath(), alreadyCompletedInstant);
+ metadataMetaClient.reloadActiveTimeline();
Review comment:
This reload is extra as the next line 123 also reloads.
##########
File path: hudi-client/hudi-spark-client/src/test/java/org/apache/hudi/client/functional/TestHBaseIndex.java
##########
@@ -422,7 +424,8 @@ public void testSimpleTagLocationWithInvalidCommit() throws Exception {
@Test
public void testEnsureTagLocationUsesCommitTimeline() throws Exception {
// Load to memory
- HoodieWriteConfig config = getConfig();
+ HoodieWriteConfig config = getConfigBuilder(100, false, false)
+ .withMetadataConfig(HoodieMetadataConfig.newBuilder().enable(false).build()).build();
Review comment:
this is used so often that it may be good to introduce a helper function here:
withDisabledMetadata()
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot edited a comment on pull request #3590: [HUDI-2285][HUDI-2476] Metadata table synchronous design. Rebased and Squashed from pull/3426
Posted by GitBox <gi...@apache.org>.
hudi-bot edited a comment on pull request #3590:
URL: https://github.com/apache/hudi/pull/3590#issuecomment-912237120
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "648e36e1af308534bc050c3a19da008ab7a8c840",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1994",
"triggerID" : "648e36e1af308534bc050c3a19da008ab7a8c840",
"triggerType" : "PUSH"
}, {
"hash" : "8c32253ad0f6d9eb728789fad75763bb5703678d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1995",
"triggerID" : "8c32253ad0f6d9eb728789fad75763bb5703678d",
"triggerType" : "PUSH"
}, {
"hash" : "71bbe59aa8bf70b9aa8990a6dd2522eb4bf1d172",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2009",
"triggerID" : "71bbe59aa8bf70b9aa8990a6dd2522eb4bf1d172",
"triggerType" : "PUSH"
}, {
"hash" : "aefac7ec2f2e40bdf3ad4365ea6aa825803a439d",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "aefac7ec2f2e40bdf3ad4365ea6aa825803a439d",
"triggerType" : "PUSH"
}, {
"hash" : "86905cb855fecfeb9e4c023d526ad643fadb8084",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2012",
"triggerID" : "86905cb855fecfeb9e4c023d526ad643fadb8084",
"triggerType" : "PUSH"
}, {
"hash" : "b4d93b0168775cabd43609fec59bf2823dd81279",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2029",
"triggerID" : "b4d93b0168775cabd43609fec59bf2823dd81279",
"triggerType" : "PUSH"
}, {
"hash" : "38bf430ecb4e11839d686929be2511eed340e9df",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2035",
"triggerID" : "38bf430ecb4e11839d686929be2511eed340e9df",
"triggerType" : "PUSH"
}, {
"hash" : "7793fbdb9b93a129ef606cb2d73ea6e1e9074957",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2299",
"triggerID" : "7793fbdb9b93a129ef606cb2d73ea6e1e9074957",
"triggerType" : "PUSH"
}, {
"hash" : "9c0123c0f27f990d009b323bab75b76ceecf3dab",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2307",
"triggerID" : "9c0123c0f27f990d009b323bab75b76ceecf3dab",
"triggerType" : "PUSH"
}, {
"hash" : "af27976ba07afb02305c699aeb7a75a122013b62",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2330",
"triggerID" : "af27976ba07afb02305c699aeb7a75a122013b62",
"triggerType" : "PUSH"
}, {
"hash" : "9172241c58424cca8e8186d4409a8c36968a4790",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2344",
"triggerID" : "9172241c58424cca8e8186d4409a8c36968a4790",
"triggerType" : "PUSH"
}, {
"hash" : "13b80067ae00a760c57d184f70e2d98f63027e80",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2353",
"triggerID" : "13b80067ae00a760c57d184f70e2d98f63027e80",
"triggerType" : "PUSH"
}, {
"hash" : "ef750979cf449ff504da194e594cb62302eba360",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2366",
"triggerID" : "ef750979cf449ff504da194e594cb62302eba360",
"triggerType" : "PUSH"
}, {
"hash" : "16827c37f36d3b9fec18e0a35aff7599c45e0444",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2375",
"triggerID" : "16827c37f36d3b9fec18e0a35aff7599c45e0444",
"triggerType" : "PUSH"
}, {
"hash" : "0f68d426b01355537f5124b8f108345a36d3da39",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2378",
"triggerID" : "0f68d426b01355537f5124b8f108345a36d3da39",
"triggerType" : "PUSH"
}, {
"hash" : "2f3099197889fd5c48ae3b550cc523e0528ddb2c",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2380",
"triggerID" : "2f3099197889fd5c48ae3b550cc523e0528ddb2c",
"triggerType" : "PUSH"
}, {
"hash" : "c83d176e1d1ddab6bb86f91726060779a9a1519f",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2392",
"triggerID" : "c83d176e1d1ddab6bb86f91726060779a9a1519f",
"triggerType" : "PUSH"
}, {
"hash" : "008f0cd91de12d44030d837f460f3b0232d62ee5",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2416",
"triggerID" : "008f0cd91de12d44030d837f460f3b0232d62ee5",
"triggerType" : "PUSH"
}, {
"hash" : "92634fb730e723a9bdeb165348a4c747794be7e3",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2418",
"triggerID" : "92634fb730e723a9bdeb165348a4c747794be7e3",
"triggerType" : "PUSH"
}, {
"hash" : "3b5fbfa0cc69e82f7c23fc6aa5b4944e2dd77220",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2438",
"triggerID" : "3b5fbfa0cc69e82f7c23fc6aa5b4944e2dd77220",
"triggerType" : "PUSH"
}, {
"hash" : "87d2ec0cc60ef8d5b7a081b73a0318aedf1a2a90",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2451",
"triggerID" : "87d2ec0cc60ef8d5b7a081b73a0318aedf1a2a90",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* aefac7ec2f2e40bdf3ad4365ea6aa825803a439d UNKNOWN
* 87d2ec0cc60ef8d5b7a081b73a0318aedf1a2a90 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2451)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run travis` re-run the last Travis build
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot edited a comment on pull request #3590: [HUDI-2285][HUDI-2476] Metadata table synchronous design. Rebased and Squashed from pull/3426
Posted by GitBox <gi...@apache.org>.
hudi-bot edited a comment on pull request #3590:
URL: https://github.com/apache/hudi/pull/3590#issuecomment-912237120
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "648e36e1af308534bc050c3a19da008ab7a8c840",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1994",
"triggerID" : "648e36e1af308534bc050c3a19da008ab7a8c840",
"triggerType" : "PUSH"
}, {
"hash" : "8c32253ad0f6d9eb728789fad75763bb5703678d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1995",
"triggerID" : "8c32253ad0f6d9eb728789fad75763bb5703678d",
"triggerType" : "PUSH"
}, {
"hash" : "71bbe59aa8bf70b9aa8990a6dd2522eb4bf1d172",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2009",
"triggerID" : "71bbe59aa8bf70b9aa8990a6dd2522eb4bf1d172",
"triggerType" : "PUSH"
}, {
"hash" : "aefac7ec2f2e40bdf3ad4365ea6aa825803a439d",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "aefac7ec2f2e40bdf3ad4365ea6aa825803a439d",
"triggerType" : "PUSH"
}, {
"hash" : "86905cb855fecfeb9e4c023d526ad643fadb8084",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2012",
"triggerID" : "86905cb855fecfeb9e4c023d526ad643fadb8084",
"triggerType" : "PUSH"
}, {
"hash" : "b4d93b0168775cabd43609fec59bf2823dd81279",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2029",
"triggerID" : "b4d93b0168775cabd43609fec59bf2823dd81279",
"triggerType" : "PUSH"
}, {
"hash" : "38bf430ecb4e11839d686929be2511eed340e9df",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2035",
"triggerID" : "38bf430ecb4e11839d686929be2511eed340e9df",
"triggerType" : "PUSH"
}, {
"hash" : "7793fbdb9b93a129ef606cb2d73ea6e1e9074957",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2299",
"triggerID" : "7793fbdb9b93a129ef606cb2d73ea6e1e9074957",
"triggerType" : "PUSH"
}, {
"hash" : "9c0123c0f27f990d009b323bab75b76ceecf3dab",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2307",
"triggerID" : "9c0123c0f27f990d009b323bab75b76ceecf3dab",
"triggerType" : "PUSH"
}, {
"hash" : "af27976ba07afb02305c699aeb7a75a122013b62",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2330",
"triggerID" : "af27976ba07afb02305c699aeb7a75a122013b62",
"triggerType" : "PUSH"
}, {
"hash" : "9172241c58424cca8e8186d4409a8c36968a4790",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2344",
"triggerID" : "9172241c58424cca8e8186d4409a8c36968a4790",
"triggerType" : "PUSH"
}, {
"hash" : "13b80067ae00a760c57d184f70e2d98f63027e80",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2353",
"triggerID" : "13b80067ae00a760c57d184f70e2d98f63027e80",
"triggerType" : "PUSH"
}, {
"hash" : "ef750979cf449ff504da194e594cb62302eba360",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2366",
"triggerID" : "ef750979cf449ff504da194e594cb62302eba360",
"triggerType" : "PUSH"
}, {
"hash" : "16827c37f36d3b9fec18e0a35aff7599c45e0444",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2375",
"triggerID" : "16827c37f36d3b9fec18e0a35aff7599c45e0444",
"triggerType" : "PUSH"
}, {
"hash" : "0f68d426b01355537f5124b8f108345a36d3da39",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2378",
"triggerID" : "0f68d426b01355537f5124b8f108345a36d3da39",
"triggerType" : "PUSH"
}, {
"hash" : "2f3099197889fd5c48ae3b550cc523e0528ddb2c",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2380",
"triggerID" : "2f3099197889fd5c48ae3b550cc523e0528ddb2c",
"triggerType" : "PUSH"
}, {
"hash" : "c83d176e1d1ddab6bb86f91726060779a9a1519f",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "c83d176e1d1ddab6bb86f91726060779a9a1519f",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* aefac7ec2f2e40bdf3ad4365ea6aa825803a439d UNKNOWN
* 2f3099197889fd5c48ae3b550cc523e0528ddb2c Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2380)
* c83d176e1d1ddab6bb86f91726060779a9a1519f UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run travis` re-run the last Travis build
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot edited a comment on pull request #3590: [HUDI-2285][HUDI-2476] Metadata table synchronous design. Rebased and Squashed from pull/3426
Posted by GitBox <gi...@apache.org>.
hudi-bot edited a comment on pull request #3590:
URL: https://github.com/apache/hudi/pull/3590#issuecomment-912237120
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "648e36e1af308534bc050c3a19da008ab7a8c840",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1994",
"triggerID" : "648e36e1af308534bc050c3a19da008ab7a8c840",
"triggerType" : "PUSH"
}, {
"hash" : "8c32253ad0f6d9eb728789fad75763bb5703678d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1995",
"triggerID" : "8c32253ad0f6d9eb728789fad75763bb5703678d",
"triggerType" : "PUSH"
}, {
"hash" : "71bbe59aa8bf70b9aa8990a6dd2522eb4bf1d172",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2009",
"triggerID" : "71bbe59aa8bf70b9aa8990a6dd2522eb4bf1d172",
"triggerType" : "PUSH"
}, {
"hash" : "aefac7ec2f2e40bdf3ad4365ea6aa825803a439d",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "aefac7ec2f2e40bdf3ad4365ea6aa825803a439d",
"triggerType" : "PUSH"
}, {
"hash" : "86905cb855fecfeb9e4c023d526ad643fadb8084",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2012",
"triggerID" : "86905cb855fecfeb9e4c023d526ad643fadb8084",
"triggerType" : "PUSH"
}, {
"hash" : "b4d93b0168775cabd43609fec59bf2823dd81279",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2029",
"triggerID" : "b4d93b0168775cabd43609fec59bf2823dd81279",
"triggerType" : "PUSH"
}, {
"hash" : "38bf430ecb4e11839d686929be2511eed340e9df",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2035",
"triggerID" : "38bf430ecb4e11839d686929be2511eed340e9df",
"triggerType" : "PUSH"
}, {
"hash" : "7793fbdb9b93a129ef606cb2d73ea6e1e9074957",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2299",
"triggerID" : "7793fbdb9b93a129ef606cb2d73ea6e1e9074957",
"triggerType" : "PUSH"
}, {
"hash" : "9c0123c0f27f990d009b323bab75b76ceecf3dab",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2307",
"triggerID" : "9c0123c0f27f990d009b323bab75b76ceecf3dab",
"triggerType" : "PUSH"
}, {
"hash" : "af27976ba07afb02305c699aeb7a75a122013b62",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2330",
"triggerID" : "af27976ba07afb02305c699aeb7a75a122013b62",
"triggerType" : "PUSH"
}, {
"hash" : "9172241c58424cca8e8186d4409a8c36968a4790",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2344",
"triggerID" : "9172241c58424cca8e8186d4409a8c36968a4790",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* aefac7ec2f2e40bdf3ad4365ea6aa825803a439d UNKNOWN
* af27976ba07afb02305c699aeb7a75a122013b62 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2330)
* 9172241c58424cca8e8186d4409a8c36968a4790 Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2344)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run travis` re-run the last Travis build
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] nsivabalan commented on a change in pull request #3590: [HUDI-2285][HUDI-2476] Metadata table synchronous design. Rebased and Squashed from pull/3426
Posted by GitBox <gi...@apache.org>.
nsivabalan commented on a change in pull request #3590:
URL: https://github.com/apache/hudi/pull/3590#discussion_r716094065
##########
File path: hudi-cli/src/main/java/org/apache/hudi/cli/commands/CommitsCommand.java
##########
@@ -440,14 +440,6 @@ public String compareCommits(@CliOption(key = {"path"}, help = "Path of the tabl
}
}
- @CliCommand(value = "commits sync", help = "Compare commits with another Hoodie table")
- public String syncCommits(@CliOption(key = {"path"}, help = "Path of the table to compare to") final String path) {
- HoodieCLI.syncTableMetadata = HoodieTableMetaClient.builder().setConf(HoodieCLI.conf).setBasePath(path).build();
Review comment:
have filed a follow up ticket https://issues.apache.org/jira/browse/HUDI-2488
##########
File path: hudi-cli/src/test/java/org/apache/hudi/cli/commands/TestArchivedCommitsCommand.java
##########
@@ -71,6 +72,7 @@ public void init() throws Exception {
HoodieWriteConfig cfg = HoodieWriteConfig.newBuilder().withPath(tablePath)
.withSchema(HoodieTestCommitMetadataGenerator.TRIP_EXAMPLE_SCHEMA).withParallelism(2, 2)
.withCompactionConfig(HoodieCompactionConfig.newBuilder().retainCommits(1).archiveCommitsWith(2, 3).build())
+ .withMetadataConfig(HoodieMetadataConfig.newBuilder().enable(false).build())
Review comment:
yes.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot edited a comment on pull request #3590: [WIP] Sync metadata patch
Posted by GitBox <gi...@apache.org>.
hudi-bot edited a comment on pull request #3590:
URL: https://github.com/apache/hudi/pull/3590#issuecomment-912237120
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "648e36e1af308534bc050c3a19da008ab7a8c840",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1994",
"triggerID" : "648e36e1af308534bc050c3a19da008ab7a8c840",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 648e36e1af308534bc050c3a19da008ab7a8c840 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1994)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run travis` re-run the last Travis build
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot edited a comment on pull request #3590: [HUDI-2285][HUDI-2476] Metadata table synchronous design. Rebased and Squashed from pull/3426
Posted by GitBox <gi...@apache.org>.
hudi-bot edited a comment on pull request #3590:
URL: https://github.com/apache/hudi/pull/3590#issuecomment-912237120
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "648e36e1af308534bc050c3a19da008ab7a8c840",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1994",
"triggerID" : "648e36e1af308534bc050c3a19da008ab7a8c840",
"triggerType" : "PUSH"
}, {
"hash" : "8c32253ad0f6d9eb728789fad75763bb5703678d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1995",
"triggerID" : "8c32253ad0f6d9eb728789fad75763bb5703678d",
"triggerType" : "PUSH"
}, {
"hash" : "71bbe59aa8bf70b9aa8990a6dd2522eb4bf1d172",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2009",
"triggerID" : "71bbe59aa8bf70b9aa8990a6dd2522eb4bf1d172",
"triggerType" : "PUSH"
}, {
"hash" : "aefac7ec2f2e40bdf3ad4365ea6aa825803a439d",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "aefac7ec2f2e40bdf3ad4365ea6aa825803a439d",
"triggerType" : "PUSH"
}, {
"hash" : "86905cb855fecfeb9e4c023d526ad643fadb8084",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2012",
"triggerID" : "86905cb855fecfeb9e4c023d526ad643fadb8084",
"triggerType" : "PUSH"
}, {
"hash" : "b4d93b0168775cabd43609fec59bf2823dd81279",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2029",
"triggerID" : "b4d93b0168775cabd43609fec59bf2823dd81279",
"triggerType" : "PUSH"
}, {
"hash" : "38bf430ecb4e11839d686929be2511eed340e9df",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2035",
"triggerID" : "38bf430ecb4e11839d686929be2511eed340e9df",
"triggerType" : "PUSH"
}, {
"hash" : "7793fbdb9b93a129ef606cb2d73ea6e1e9074957",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2299",
"triggerID" : "7793fbdb9b93a129ef606cb2d73ea6e1e9074957",
"triggerType" : "PUSH"
}, {
"hash" : "9c0123c0f27f990d009b323bab75b76ceecf3dab",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2307",
"triggerID" : "9c0123c0f27f990d009b323bab75b76ceecf3dab",
"triggerType" : "PUSH"
}, {
"hash" : "af27976ba07afb02305c699aeb7a75a122013b62",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2330",
"triggerID" : "af27976ba07afb02305c699aeb7a75a122013b62",
"triggerType" : "PUSH"
}, {
"hash" : "9172241c58424cca8e8186d4409a8c36968a4790",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2344",
"triggerID" : "9172241c58424cca8e8186d4409a8c36968a4790",
"triggerType" : "PUSH"
}, {
"hash" : "13b80067ae00a760c57d184f70e2d98f63027e80",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2353",
"triggerID" : "13b80067ae00a760c57d184f70e2d98f63027e80",
"triggerType" : "PUSH"
}, {
"hash" : "ef750979cf449ff504da194e594cb62302eba360",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2366",
"triggerID" : "ef750979cf449ff504da194e594cb62302eba360",
"triggerType" : "PUSH"
}, {
"hash" : "16827c37f36d3b9fec18e0a35aff7599c45e0444",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2375",
"triggerID" : "16827c37f36d3b9fec18e0a35aff7599c45e0444",
"triggerType" : "PUSH"
}, {
"hash" : "0f68d426b01355537f5124b8f108345a36d3da39",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "0f68d426b01355537f5124b8f108345a36d3da39",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* aefac7ec2f2e40bdf3ad4365ea6aa825803a439d UNKNOWN
* 16827c37f36d3b9fec18e0a35aff7599c45e0444 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2375)
* 0f68d426b01355537f5124b8f108345a36d3da39 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run travis` re-run the last Travis build
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot edited a comment on pull request #3590: [HUDI-2285][HUDI-2476] Metadata table synchronous design. Rebased and Squashed from pull/3426
Posted by GitBox <gi...@apache.org>.
hudi-bot edited a comment on pull request #3590:
URL: https://github.com/apache/hudi/pull/3590#issuecomment-912237120
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "648e36e1af308534bc050c3a19da008ab7a8c840",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1994",
"triggerID" : "648e36e1af308534bc050c3a19da008ab7a8c840",
"triggerType" : "PUSH"
}, {
"hash" : "8c32253ad0f6d9eb728789fad75763bb5703678d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1995",
"triggerID" : "8c32253ad0f6d9eb728789fad75763bb5703678d",
"triggerType" : "PUSH"
}, {
"hash" : "71bbe59aa8bf70b9aa8990a6dd2522eb4bf1d172",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2009",
"triggerID" : "71bbe59aa8bf70b9aa8990a6dd2522eb4bf1d172",
"triggerType" : "PUSH"
}, {
"hash" : "aefac7ec2f2e40bdf3ad4365ea6aa825803a439d",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "aefac7ec2f2e40bdf3ad4365ea6aa825803a439d",
"triggerType" : "PUSH"
}, {
"hash" : "86905cb855fecfeb9e4c023d526ad643fadb8084",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2012",
"triggerID" : "86905cb855fecfeb9e4c023d526ad643fadb8084",
"triggerType" : "PUSH"
}, {
"hash" : "b4d93b0168775cabd43609fec59bf2823dd81279",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2029",
"triggerID" : "b4d93b0168775cabd43609fec59bf2823dd81279",
"triggerType" : "PUSH"
}, {
"hash" : "38bf430ecb4e11839d686929be2511eed340e9df",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2035",
"triggerID" : "38bf430ecb4e11839d686929be2511eed340e9df",
"triggerType" : "PUSH"
}, {
"hash" : "7793fbdb9b93a129ef606cb2d73ea6e1e9074957",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2299",
"triggerID" : "7793fbdb9b93a129ef606cb2d73ea6e1e9074957",
"triggerType" : "PUSH"
}, {
"hash" : "9c0123c0f27f990d009b323bab75b76ceecf3dab",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2307",
"triggerID" : "9c0123c0f27f990d009b323bab75b76ceecf3dab",
"triggerType" : "PUSH"
}, {
"hash" : "af27976ba07afb02305c699aeb7a75a122013b62",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2330",
"triggerID" : "af27976ba07afb02305c699aeb7a75a122013b62",
"triggerType" : "PUSH"
}, {
"hash" : "9172241c58424cca8e8186d4409a8c36968a4790",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2344",
"triggerID" : "9172241c58424cca8e8186d4409a8c36968a4790",
"triggerType" : "PUSH"
}, {
"hash" : "13b80067ae00a760c57d184f70e2d98f63027e80",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2353",
"triggerID" : "13b80067ae00a760c57d184f70e2d98f63027e80",
"triggerType" : "PUSH"
}, {
"hash" : "ef750979cf449ff504da194e594cb62302eba360",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2366",
"triggerID" : "ef750979cf449ff504da194e594cb62302eba360",
"triggerType" : "PUSH"
}, {
"hash" : "16827c37f36d3b9fec18e0a35aff7599c45e0444",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2375",
"triggerID" : "16827c37f36d3b9fec18e0a35aff7599c45e0444",
"triggerType" : "PUSH"
}, {
"hash" : "0f68d426b01355537f5124b8f108345a36d3da39",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2378",
"triggerID" : "0f68d426b01355537f5124b8f108345a36d3da39",
"triggerType" : "PUSH"
}, {
"hash" : "2f3099197889fd5c48ae3b550cc523e0528ddb2c",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2380",
"triggerID" : "2f3099197889fd5c48ae3b550cc523e0528ddb2c",
"triggerType" : "PUSH"
}, {
"hash" : "c83d176e1d1ddab6bb86f91726060779a9a1519f",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2392",
"triggerID" : "c83d176e1d1ddab6bb86f91726060779a9a1519f",
"triggerType" : "PUSH"
}, {
"hash" : "008f0cd91de12d44030d837f460f3b0232d62ee5",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2416",
"triggerID" : "008f0cd91de12d44030d837f460f3b0232d62ee5",
"triggerType" : "PUSH"
}, {
"hash" : "92634fb730e723a9bdeb165348a4c747794be7e3",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2418",
"triggerID" : "92634fb730e723a9bdeb165348a4c747794be7e3",
"triggerType" : "PUSH"
}, {
"hash" : "3b5fbfa0cc69e82f7c23fc6aa5b4944e2dd77220",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2438",
"triggerID" : "3b5fbfa0cc69e82f7c23fc6aa5b4944e2dd77220",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* aefac7ec2f2e40bdf3ad4365ea6aa825803a439d UNKNOWN
* 3b5fbfa0cc69e82f7c23fc6aa5b4944e2dd77220 Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2438)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run travis` re-run the last Travis build
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot edited a comment on pull request #3590: [HUDI-2285][HUDI-2476] Metadata table synchronous design. Rebased and Squashed from pull/3426
Posted by GitBox <gi...@apache.org>.
hudi-bot edited a comment on pull request #3590:
URL: https://github.com/apache/hudi/pull/3590#issuecomment-912237120
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "648e36e1af308534bc050c3a19da008ab7a8c840",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1994",
"triggerID" : "648e36e1af308534bc050c3a19da008ab7a8c840",
"triggerType" : "PUSH"
}, {
"hash" : "8c32253ad0f6d9eb728789fad75763bb5703678d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1995",
"triggerID" : "8c32253ad0f6d9eb728789fad75763bb5703678d",
"triggerType" : "PUSH"
}, {
"hash" : "71bbe59aa8bf70b9aa8990a6dd2522eb4bf1d172",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2009",
"triggerID" : "71bbe59aa8bf70b9aa8990a6dd2522eb4bf1d172",
"triggerType" : "PUSH"
}, {
"hash" : "aefac7ec2f2e40bdf3ad4365ea6aa825803a439d",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "aefac7ec2f2e40bdf3ad4365ea6aa825803a439d",
"triggerType" : "PUSH"
}, {
"hash" : "86905cb855fecfeb9e4c023d526ad643fadb8084",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2012",
"triggerID" : "86905cb855fecfeb9e4c023d526ad643fadb8084",
"triggerType" : "PUSH"
}, {
"hash" : "b4d93b0168775cabd43609fec59bf2823dd81279",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2029",
"triggerID" : "b4d93b0168775cabd43609fec59bf2823dd81279",
"triggerType" : "PUSH"
}, {
"hash" : "38bf430ecb4e11839d686929be2511eed340e9df",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2035",
"triggerID" : "38bf430ecb4e11839d686929be2511eed340e9df",
"triggerType" : "PUSH"
}, {
"hash" : "7793fbdb9b93a129ef606cb2d73ea6e1e9074957",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2299",
"triggerID" : "7793fbdb9b93a129ef606cb2d73ea6e1e9074957",
"triggerType" : "PUSH"
}, {
"hash" : "9c0123c0f27f990d009b323bab75b76ceecf3dab",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2307",
"triggerID" : "9c0123c0f27f990d009b323bab75b76ceecf3dab",
"triggerType" : "PUSH"
}, {
"hash" : "af27976ba07afb02305c699aeb7a75a122013b62",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2330",
"triggerID" : "af27976ba07afb02305c699aeb7a75a122013b62",
"triggerType" : "PUSH"
}, {
"hash" : "9172241c58424cca8e8186d4409a8c36968a4790",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2344",
"triggerID" : "9172241c58424cca8e8186d4409a8c36968a4790",
"triggerType" : "PUSH"
}, {
"hash" : "13b80067ae00a760c57d184f70e2d98f63027e80",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2353",
"triggerID" : "13b80067ae00a760c57d184f70e2d98f63027e80",
"triggerType" : "PUSH"
}, {
"hash" : "ef750979cf449ff504da194e594cb62302eba360",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2366",
"triggerID" : "ef750979cf449ff504da194e594cb62302eba360",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* aefac7ec2f2e40bdf3ad4365ea6aa825803a439d UNKNOWN
* ef750979cf449ff504da194e594cb62302eba360 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2366)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run travis` re-run the last Travis build
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] nsivabalan removed a comment on pull request #3590: [HUDI-2285][HUDI-2476] Metadata table synchronous design. Rebased and Squashed from pull/3426
Posted by GitBox <gi...@apache.org>.
nsivabalan removed a comment on pull request #3590:
URL: https://github.com/apache/hudi/pull/3590#issuecomment-923333856
So, given this approach, we could also support async compaction and clustering in metadata table.
Here is what we could do.
all things stay same wrt data table. i.e.
take locks and do conflict resolution for all regular writes, commit and release locks.
take locks and do conflict resolution while scheduling compaction/clustering and release locks.
take locks and commit compaction and clustering.
when it comes to metadata table. We will enable multi-writer mode in metadata table. (As of this patch, we have only single writer mode for metadata table)
all writes to metadata table happens within data table lock. And so wrt new delta commits to metadata table, it is always going to be a single writer implicitly.
after committing to metadata table, we can just schedule compaction and cleaning if something is available. This internally will take the lock for metadata table and check for any conflicts, but since there are no other writers, we should be good.
and once the commit and scheduling completes, we return to data table, make the commit and release the lockk.
Later when async compaction for metadata table is about to get committed, we take metadata table lock and make the commit. this will ensure this may not collide with regular delta commits happening to metadata table. We may not invoke any conflict resolution here similar to how it is done in data table.
But one major issue we need to fix here is the ConflictResolutionStrategy: as of now, there are any pending or complete compactions after current commit of interest, writes will fail. since all of them are going to operate on the same partition with one file group, there will definitely be conflict. So, just for metadata, we might want to consider if we can come up with a special conflict resolution strategy where we consider only new writes as conflicts and not any scheduled compaction). I need to understand the implications of this in more finer detail. But just putting it out here.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot edited a comment on pull request #3590: [HUDI-2285] Metadata table synchronous design. Rebased and Squashed from pull/3426
Posted by GitBox <gi...@apache.org>.
hudi-bot edited a comment on pull request #3590:
URL: https://github.com/apache/hudi/pull/3590#issuecomment-912237120
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "648e36e1af308534bc050c3a19da008ab7a8c840",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1994",
"triggerID" : "648e36e1af308534bc050c3a19da008ab7a8c840",
"triggerType" : "PUSH"
}, {
"hash" : "8c32253ad0f6d9eb728789fad75763bb5703678d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1995",
"triggerID" : "8c32253ad0f6d9eb728789fad75763bb5703678d",
"triggerType" : "PUSH"
}, {
"hash" : "71bbe59aa8bf70b9aa8990a6dd2522eb4bf1d172",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2009",
"triggerID" : "71bbe59aa8bf70b9aa8990a6dd2522eb4bf1d172",
"triggerType" : "PUSH"
}, {
"hash" : "aefac7ec2f2e40bdf3ad4365ea6aa825803a439d",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "aefac7ec2f2e40bdf3ad4365ea6aa825803a439d",
"triggerType" : "PUSH"
}, {
"hash" : "86905cb855fecfeb9e4c023d526ad643fadb8084",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2012",
"triggerID" : "86905cb855fecfeb9e4c023d526ad643fadb8084",
"triggerType" : "PUSH"
}, {
"hash" : "b4d93b0168775cabd43609fec59bf2823dd81279",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2029",
"triggerID" : "b4d93b0168775cabd43609fec59bf2823dd81279",
"triggerType" : "PUSH"
}, {
"hash" : "38bf430ecb4e11839d686929be2511eed340e9df",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2035",
"triggerID" : "38bf430ecb4e11839d686929be2511eed340e9df",
"triggerType" : "PUSH"
}, {
"hash" : "7793fbdb9b93a129ef606cb2d73ea6e1e9074957",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "7793fbdb9b93a129ef606cb2d73ea6e1e9074957",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* aefac7ec2f2e40bdf3ad4365ea6aa825803a439d UNKNOWN
* 38bf430ecb4e11839d686929be2511eed340e9df Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2035)
* 7793fbdb9b93a129ef606cb2d73ea6e1e9074957 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run travis` re-run the last Travis build
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot edited a comment on pull request #3590: [WIP][HUDI-2285] Metadata table synchronous design. Rebased and Squashed from pull/3426
Posted by GitBox <gi...@apache.org>.
hudi-bot edited a comment on pull request #3590:
URL: https://github.com/apache/hudi/pull/3590#issuecomment-912237120
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "648e36e1af308534bc050c3a19da008ab7a8c840",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1994",
"triggerID" : "648e36e1af308534bc050c3a19da008ab7a8c840",
"triggerType" : "PUSH"
}, {
"hash" : "8c32253ad0f6d9eb728789fad75763bb5703678d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1995",
"triggerID" : "8c32253ad0f6d9eb728789fad75763bb5703678d",
"triggerType" : "PUSH"
}, {
"hash" : "71bbe59aa8bf70b9aa8990a6dd2522eb4bf1d172",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2009",
"triggerID" : "71bbe59aa8bf70b9aa8990a6dd2522eb4bf1d172",
"triggerType" : "PUSH"
}, {
"hash" : "aefac7ec2f2e40bdf3ad4365ea6aa825803a439d",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "aefac7ec2f2e40bdf3ad4365ea6aa825803a439d",
"triggerType" : "PUSH"
}, {
"hash" : "86905cb855fecfeb9e4c023d526ad643fadb8084",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2012",
"triggerID" : "86905cb855fecfeb9e4c023d526ad643fadb8084",
"triggerType" : "PUSH"
}, {
"hash" : "b4d93b0168775cabd43609fec59bf2823dd81279",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2029",
"triggerID" : "b4d93b0168775cabd43609fec59bf2823dd81279",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* aefac7ec2f2e40bdf3ad4365ea6aa825803a439d UNKNOWN
* b4d93b0168775cabd43609fec59bf2823dd81279 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2029)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run travis` re-run the last Travis build
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot edited a comment on pull request #3590: [WIP][HUDI-2285] Metadata table synchronous design. Rebased and Squashed from pull/3426
Posted by GitBox <gi...@apache.org>.
hudi-bot edited a comment on pull request #3590:
URL: https://github.com/apache/hudi/pull/3590#issuecomment-912237120
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "648e36e1af308534bc050c3a19da008ab7a8c840",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1994",
"triggerID" : "648e36e1af308534bc050c3a19da008ab7a8c840",
"triggerType" : "PUSH"
}, {
"hash" : "8c32253ad0f6d9eb728789fad75763bb5703678d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1995",
"triggerID" : "8c32253ad0f6d9eb728789fad75763bb5703678d",
"triggerType" : "PUSH"
}, {
"hash" : "71bbe59aa8bf70b9aa8990a6dd2522eb4bf1d172",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2009",
"triggerID" : "71bbe59aa8bf70b9aa8990a6dd2522eb4bf1d172",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 71bbe59aa8bf70b9aa8990a6dd2522eb4bf1d172 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2009)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run travis` re-run the last Travis build
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] vinothchandar commented on pull request #3590: [WIP] Sync metadata patch
Posted by GitBox <gi...@apache.org>.
vinothchandar commented on pull request #3590:
URL: https://github.com/apache/hudi/pull/3590#issuecomment-912238553
Descriptive pr titles with jira numbers please? Even for WIP.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot edited a comment on pull request #3590: [WIP][HUDI-2285] Metadata table synchronous design. Rebased and Squashed from pull/3426
Posted by GitBox <gi...@apache.org>.
hudi-bot edited a comment on pull request #3590:
URL: https://github.com/apache/hudi/pull/3590#issuecomment-912237120
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "648e36e1af308534bc050c3a19da008ab7a8c840",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1994",
"triggerID" : "648e36e1af308534bc050c3a19da008ab7a8c840",
"triggerType" : "PUSH"
}, {
"hash" : "8c32253ad0f6d9eb728789fad75763bb5703678d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1995",
"triggerID" : "8c32253ad0f6d9eb728789fad75763bb5703678d",
"triggerType" : "PUSH"
}, {
"hash" : "71bbe59aa8bf70b9aa8990a6dd2522eb4bf1d172",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2009",
"triggerID" : "71bbe59aa8bf70b9aa8990a6dd2522eb4bf1d172",
"triggerType" : "PUSH"
}, {
"hash" : "aefac7ec2f2e40bdf3ad4365ea6aa825803a439d",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "aefac7ec2f2e40bdf3ad4365ea6aa825803a439d",
"triggerType" : "PUSH"
}, {
"hash" : "86905cb855fecfeb9e4c023d526ad643fadb8084",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2012",
"triggerID" : "86905cb855fecfeb9e4c023d526ad643fadb8084",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* aefac7ec2f2e40bdf3ad4365ea6aa825803a439d UNKNOWN
* 86905cb855fecfeb9e4c023d526ad643fadb8084 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2012)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run travis` re-run the last Travis build
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot edited a comment on pull request #3590: [WIP][HUDI-2285] Metadata table synchronous design. Rebased and Squashed from pull/3426
Posted by GitBox <gi...@apache.org>.
hudi-bot edited a comment on pull request #3590:
URL: https://github.com/apache/hudi/pull/3590#issuecomment-912237120
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "648e36e1af308534bc050c3a19da008ab7a8c840",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1994",
"triggerID" : "648e36e1af308534bc050c3a19da008ab7a8c840",
"triggerType" : "PUSH"
}, {
"hash" : "8c32253ad0f6d9eb728789fad75763bb5703678d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1995",
"triggerID" : "8c32253ad0f6d9eb728789fad75763bb5703678d",
"triggerType" : "PUSH"
}, {
"hash" : "71bbe59aa8bf70b9aa8990a6dd2522eb4bf1d172",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2009",
"triggerID" : "71bbe59aa8bf70b9aa8990a6dd2522eb4bf1d172",
"triggerType" : "PUSH"
}, {
"hash" : "aefac7ec2f2e40bdf3ad4365ea6aa825803a439d",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "aefac7ec2f2e40bdf3ad4365ea6aa825803a439d",
"triggerType" : "PUSH"
}, {
"hash" : "86905cb855fecfeb9e4c023d526ad643fadb8084",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2012",
"triggerID" : "86905cb855fecfeb9e4c023d526ad643fadb8084",
"triggerType" : "PUSH"
}, {
"hash" : "b4d93b0168775cabd43609fec59bf2823dd81279",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2029",
"triggerID" : "b4d93b0168775cabd43609fec59bf2823dd81279",
"triggerType" : "PUSH"
}, {
"hash" : "38bf430ecb4e11839d686929be2511eed340e9df",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "38bf430ecb4e11839d686929be2511eed340e9df",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* aefac7ec2f2e40bdf3ad4365ea6aa825803a439d UNKNOWN
* b4d93b0168775cabd43609fec59bf2823dd81279 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2029)
* 38bf430ecb4e11839d686929be2511eed340e9df UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run travis` re-run the last Travis build
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #3590: [WIP] Sync metadata patch
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on pull request #3590:
URL: https://github.com/apache/hudi/pull/3590#issuecomment-912237120
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "648e36e1af308534bc050c3a19da008ab7a8c840",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "648e36e1af308534bc050c3a19da008ab7a8c840",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 648e36e1af308534bc050c3a19da008ab7a8c840 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run travis` re-run the last Travis build
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot edited a comment on pull request #3590: [HUDI-2285][HUDI-2476] Metadata table synchronous design. Rebased and Squashed from pull/3426
Posted by GitBox <gi...@apache.org>.
hudi-bot edited a comment on pull request #3590:
URL: https://github.com/apache/hudi/pull/3590#issuecomment-912237120
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "648e36e1af308534bc050c3a19da008ab7a8c840",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1994",
"triggerID" : "648e36e1af308534bc050c3a19da008ab7a8c840",
"triggerType" : "PUSH"
}, {
"hash" : "8c32253ad0f6d9eb728789fad75763bb5703678d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1995",
"triggerID" : "8c32253ad0f6d9eb728789fad75763bb5703678d",
"triggerType" : "PUSH"
}, {
"hash" : "71bbe59aa8bf70b9aa8990a6dd2522eb4bf1d172",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2009",
"triggerID" : "71bbe59aa8bf70b9aa8990a6dd2522eb4bf1d172",
"triggerType" : "PUSH"
}, {
"hash" : "aefac7ec2f2e40bdf3ad4365ea6aa825803a439d",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "aefac7ec2f2e40bdf3ad4365ea6aa825803a439d",
"triggerType" : "PUSH"
}, {
"hash" : "86905cb855fecfeb9e4c023d526ad643fadb8084",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2012",
"triggerID" : "86905cb855fecfeb9e4c023d526ad643fadb8084",
"triggerType" : "PUSH"
}, {
"hash" : "b4d93b0168775cabd43609fec59bf2823dd81279",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2029",
"triggerID" : "b4d93b0168775cabd43609fec59bf2823dd81279",
"triggerType" : "PUSH"
}, {
"hash" : "38bf430ecb4e11839d686929be2511eed340e9df",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2035",
"triggerID" : "38bf430ecb4e11839d686929be2511eed340e9df",
"triggerType" : "PUSH"
}, {
"hash" : "7793fbdb9b93a129ef606cb2d73ea6e1e9074957",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2299",
"triggerID" : "7793fbdb9b93a129ef606cb2d73ea6e1e9074957",
"triggerType" : "PUSH"
}, {
"hash" : "9c0123c0f27f990d009b323bab75b76ceecf3dab",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2307",
"triggerID" : "9c0123c0f27f990d009b323bab75b76ceecf3dab",
"triggerType" : "PUSH"
}, {
"hash" : "af27976ba07afb02305c699aeb7a75a122013b62",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2330",
"triggerID" : "af27976ba07afb02305c699aeb7a75a122013b62",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* aefac7ec2f2e40bdf3ad4365ea6aa825803a439d UNKNOWN
* 9c0123c0f27f990d009b323bab75b76ceecf3dab Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2307)
* af27976ba07afb02305c699aeb7a75a122013b62 Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2330)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run travis` re-run the last Travis build
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] vinothchandar commented on a change in pull request #3590: [HUDI-2285][HUDI-2476] Metadata table synchronous design. Rebased and Squashed from pull/3426
Posted by GitBox <gi...@apache.org>.
vinothchandar commented on a change in pull request #3590:
URL: https://github.com/apache/hudi/pull/3590#discussion_r715912832
##########
File path: hudi-cli/src/main/java/org/apache/hudi/cli/commands/CommitsCommand.java
##########
@@ -440,14 +440,6 @@ public String compareCommits(@CliOption(key = {"path"}, help = "Path of the tabl
}
}
- @CliCommand(value = "commits sync", help = "Compare commits with another Hoodie table")
- public String syncCommits(@CliOption(key = {"path"}, help = "Path of the table to compare to") final String path) {
- HoodieCLI.syncTableMetadata = HoodieTableMetaClient.builder().setConf(HoodieCLI.conf).setBasePath(path).build();
Review comment:
I understand why this is being removed. but we may want some ability to add a new metadata partition in the background, using say `CREATE INDEX` statement, while this is built in the background, there could be new commits , which need to be synced? lets file a subtask to deal with this.
##########
File path: hudi-client/hudi-client-common/src/main/java/org/apache/hudi/client/AbstractHoodieWriteClient.java
##########
@@ -188,6 +189,7 @@ public boolean commitStats(String instantTime, List<HoodieWriteStat> stats, Opti
lastCompletedTxnAndMetadata.isPresent() ? Option.of(lastCompletedTxnAndMetadata.get().getLeft()) : Option.empty());
try {
preCommit(instantTime, metadata);
+ table.getMetadataWriter().ifPresent(w -> ((HoodieTableMetadataWriter)w).update(metadata, instantTime));
Review comment:
why can't this be inside `preCommit()`?
##########
File path: hudi-client/hudi-client-common/src/main/java/org/apache/hudi/client/AbstractHoodieWriteClient.java
##########
@@ -273,6 +275,7 @@ public void bootstrap(Option<Map<String, String>> extraMetadata) {
public void rollbackFailedBootstrap() {
LOG.info("Rolling back pending bootstrap if present");
HoodieTable<T, I, K, O> table = createTable(config, hadoopConf);
+ table.getHoodieView().sync();
Review comment:
need to remove?
##########
File path: hudi-client/hudi-client-common/src/main/java/org/apache/hudi/metadata/HoodieBackedTableMetadataWriter.java
##########
@@ -81,13 +84,16 @@
public abstract class HoodieBackedTableMetadataWriter implements HoodieTableMetadataWriter {
private static final Logger LOG = LogManager.getLogger(HoodieBackedTableMetadataWriter.class);
+ private static final Integer MAX_BUCKET_COUNT = 9999;
+ private static final String BUCKET_PREFIX = "bucket-";
Review comment:
instead of `bucket-` , what if we use the partition name, i.e `files-xxxx`
##########
File path: hudi-client/hudi-client-common/src/main/java/org/apache/hudi/metadata/HoodieBackedTableMetadataWriter.java
##########
@@ -309,6 +303,7 @@ private boolean bootstrapFromFilesystem(HoodieEngineContext engineContext, Hoodi
.initTable(hadoopConf.get(), metadataWriteConfig.getBasePath());
initTableMetadata();
+ initializeBuckets(datasetMetaClient, MetadataPartitionType.FILES.partitionPath(), createInstantTime, 1);
Review comment:
initializeFileGroups?
##########
File path: hudi-cli/src/test/java/org/apache/hudi/cli/commands/TestArchivedCommitsCommand.java
##########
@@ -71,6 +72,7 @@ public void init() throws Exception {
HoodieWriteConfig cfg = HoodieWriteConfig.newBuilder().withPath(tablePath)
.withSchema(HoodieTestCommitMetadataGenerator.TRIP_EXAMPLE_SCHEMA).withParallelism(2, 2)
.withCompactionConfig(HoodieCompactionConfig.newBuilder().retainCommits(1).archiveCommitsWith(2, 3).build())
+ .withMetadataConfig(HoodieMetadataConfig.newBuilder().enable(false).build())
Review comment:
lets track all these things ? and eventually fix these
##########
File path: hudi-client/hudi-spark-client/src/main/java/org/apache/hudi/client/SparkRDDWriteClient.java
##########
@@ -298,28 +306,38 @@ protected void completeCompaction(HoodieCommitMetadata metadata, JavaRDD<WriteSt
String compactionCommitTime) {
this.context.setJobStatus(this.getClass().getSimpleName(), "Collect compaction write status and commit compaction");
List<HoodieWriteStat> writeStats = writeStatuses.map(WriteStatus::getStat).collect();
- finalizeWrite(table, compactionCommitTime, writeStats);
- LOG.info("Committing Compaction " + compactionCommitTime + ". Finished with result " + metadata);
- SparkCompactHelpers.newInstance().completeInflightCompaction(table, compactionCommitTime, metadata);
- WriteMarkersFactory.get(config.getMarkersType(), table, compactionCommitTime)
- .quietDeleteMarkerDir(context, config.getMarkersDeleteParallelism());
-
- if (compactionTimer != null) {
- long durationInMs = metrics.getDurationInMs(compactionTimer.stop());
- try {
- metrics.updateCommitMetrics(HoodieActiveTimeline.COMMIT_FORMATTER.parse(compactionCommitTime).getTime(),
- durationInMs, metadata, HoodieActiveTimeline.COMPACTION_ACTION);
- } catch (ParseException e) {
- throw new HoodieCommitException("Commit time is not of valid format. Failed to commit compaction "
- + config.getBasePath() + " at time " + compactionCommitTime, e);
+ try {
+ // TODO: check if we need HeartbeatUtils.abortIfHeartbeatExpired(instantTime, table, heartbeatClient, config) here.
+ this.txnManager.beginTransaction(Option.of(new HoodieInstant(HoodieInstant.State.INFLIGHT, HoodieTimeline.COMPACTION_ACTION, compactionCommitTime)), Option.empty());
+ // Do not do any conflict resolution here as we do with regular writes. We take the lock here to ensure all writes to metadata table happens within a
+ // single lock (single writer). Because more than one write to metadata table will result in conflicts since all of them updates the same partition.
+ table.getMetadataWriter().ifPresent(w -> w.update(metadata, compactionCommitTime));
+ // commit to data table after committing to metadata table.FlinkHoodieBackedTableMetadataWriter
Review comment:
mentioning Flink in Spark module?
##########
File path: hudi-client/hudi-client-common/src/main/java/org/apache/hudi/metadata/HoodieBackedTableMetadataWriter.java
##########
@@ -499,8 +510,20 @@ public void update(HoodieRestoreMetadata restoreMetadata, String instantTime) {
@Override
public void update(HoodieRollbackMetadata rollbackMetadata, String instantTime) {
if (enabled) {
- List<HoodieRecord> records = HoodieTableMetadataUtil.convertMetadataToRecords(metaClient.getActiveTimeline(),
- rollbackMetadata, instantTime, metadata.getUpdateTime());
+ // Is this rollback of an instant that has been synced to the metadata table?
+ String rollbackInstant = rollbackMetadata.getCommitsRollback().get(0);
Review comment:
this is something to fix. this `get(0)`
##########
File path: hudi-client/hudi-client-common/src/main/java/org/apache/hudi/table/action/clean/BaseCleanActionExecutor.java
##########
@@ -124,6 +127,19 @@ private HoodieCleanMetadata runClean(HoodieTable<T, I, K, O> table, HoodieInstan
}
}
+ /**
+ * Update metadata table if available. Any update to metadata table happens within data table lock.
+ * @param cleanMetadata intance of {@link HoodieCleanMetadata} to be applied to metadata.
+ */
+ private void writeToMetadata(HoodieCleanMetadata cleanMetadata) {
Review comment:
writeMetadata
##########
File path: hudi-client/hudi-client-common/src/main/java/org/apache/hudi/metadata/HoodieBackedTableMetadataWriter.java
##########
@@ -401,64 +396,86 @@ private boolean bootstrapFromFilesystem(HoodieEngineContext engineContext, Hoodi
}
/**
- * Sync the Metadata Table from the instants created on the dataset.
+ * Initialize buckets for a partition. For file listing, we just have one bucket. But for record level index, we might have N number of buckets
+ * per partition. Technically speaking buckets here map to FileGroups in Hudi.
*
- * @param datasetMetaClient {@code HoodieTableMetaClient} for the dataset
+ * Each bucket maps to FileGroups in Hudi and is represented with the following format:
+ * bucket-ABCD
+ * where ABCD are digits. This allows up to 9999 buckets.
+ *
+ * Example:
+ * bucket-0001
+ * bucket-0002
*/
- private void syncFromInstants(HoodieTableMetaClient datasetMetaClient) {
- ValidationUtils.checkState(enabled, "Metadata table cannot be synced as it is not enabled");
- // (re) init the metadata for reading.
- initTableMetadata();
- try {
- List<HoodieInstant> instantsToSync = metadata.findInstantsToSyncForWriter();
- if (instantsToSync.isEmpty()) {
- return;
- }
-
- LOG.info("Syncing " + instantsToSync.size() + " instants to metadata table: " + instantsToSync);
-
- // Read each instant in order and sync it to metadata table
- for (HoodieInstant instant : instantsToSync) {
- LOG.info("Syncing instant " + instant + " to metadata table");
-
- Option<List<HoodieRecord>> records = HoodieTableMetadataUtil.convertInstantToMetaRecords(datasetMetaClient,
- metaClient.getActiveTimeline(), instant, metadata.getUpdateTime());
- if (records.isPresent()) {
- commit(records.get(), MetadataPartitionType.FILES.partitionPath(), instant.getTimestamp());
- }
+ private void initializeBuckets(HoodieTableMetaClient datasetMetaClient, String partition, String instantTime,
+ int bucketCount) throws IOException {
+ ValidationUtils.checkArgument(bucketCount <= MAX_BUCKET_COUNT, "Maximum " + MAX_BUCKET_COUNT + " buckets are supported.");
+
+ final HashMap<HeaderMetadataType, String> blockHeader = new HashMap<>();
+ blockHeader.put(HeaderMetadataType.INSTANT_TIME, instantTime);
+ // Archival of data table has a dependency on compaction(base files) in metadata table.
+ // It is assumed that as of time Tx of base instant (/compaction time) in metadata table,
+ // all commits in data table is in sync with metadata table. So, we always create start with log file for any bucket.
+ // but we have to work on relaxing that in future : https://issues.apache.org/jira/browse/HUDI-2458
+ final HoodieDeleteBlock block = new HoodieDeleteBlock(new HoodieKey[0], blockHeader);
+
+ LOG.info(String.format("Creating %d buckets for partition %s with base fileId %s at instant time %s",
+ bucketCount, partition, BUCKET_PREFIX, instantTime));
+ for (int i = 0; i < bucketCount; ++i) {
+ final String bucketFileId = String.format("%s%04d", BUCKET_PREFIX, i + 1);
+ try {
+ // since all shards are initialized in driver, we don't need to create a random write token.
+ String writeToken = FSUtils.makeWriteToken(0, 0, 0);
Review comment:
any constants to reuse?
##########
File path: hudi-client/hudi-spark-client/src/main/java/org/apache/hudi/client/SparkRDDWriteClient.java
##########
@@ -126,8 +129,13 @@ public boolean commit(String instantTime, JavaRDD<WriteStatus> writeStatuses, Op
@Override
protected HoodieTable<T, JavaRDD<HoodieRecord<T>>, JavaRDD<HoodieKey>, JavaRDD<WriteStatus>> createTable(HoodieWriteConfig config,
- Configuration hadoopConf) {
- return HoodieSparkTable.create(config, context);
+ Configuration hadoopConf,
+ boolean refreshTimeline) {
Review comment:
format
##########
File path: hudi-client/hudi-client-common/src/main/java/org/apache/hudi/metadata/HoodieBackedTableMetadataWriter.java
##########
@@ -401,64 +396,86 @@ private boolean bootstrapFromFilesystem(HoodieEngineContext engineContext, Hoodi
}
/**
- * Sync the Metadata Table from the instants created on the dataset.
+ * Initialize buckets for a partition. For file listing, we just have one bucket. But for record level index, we might have N number of buckets
+ * per partition. Technically speaking buckets here map to FileGroups in Hudi.
*
- * @param datasetMetaClient {@code HoodieTableMetaClient} for the dataset
+ * Each bucket maps to FileGroups in Hudi and is represented with the following format:
+ * bucket-ABCD
+ * where ABCD are digits. This allows up to 9999 buckets.
+ *
+ * Example:
+ * bucket-0001
+ * bucket-0002
*/
- private void syncFromInstants(HoodieTableMetaClient datasetMetaClient) {
- ValidationUtils.checkState(enabled, "Metadata table cannot be synced as it is not enabled");
- // (re) init the metadata for reading.
- initTableMetadata();
- try {
- List<HoodieInstant> instantsToSync = metadata.findInstantsToSyncForWriter();
- if (instantsToSync.isEmpty()) {
- return;
- }
-
- LOG.info("Syncing " + instantsToSync.size() + " instants to metadata table: " + instantsToSync);
-
- // Read each instant in order and sync it to metadata table
- for (HoodieInstant instant : instantsToSync) {
- LOG.info("Syncing instant " + instant + " to metadata table");
-
- Option<List<HoodieRecord>> records = HoodieTableMetadataUtil.convertInstantToMetaRecords(datasetMetaClient,
- metaClient.getActiveTimeline(), instant, metadata.getUpdateTime());
- if (records.isPresent()) {
- commit(records.get(), MetadataPartitionType.FILES.partitionPath(), instant.getTimestamp());
- }
+ private void initializeBuckets(HoodieTableMetaClient datasetMetaClient, String partition, String instantTime,
Review comment:
partitionPath
##########
File path: hudi-client/hudi-client-common/src/main/java/org/apache/hudi/metadata/HoodieBackedTableMetadataWriter.java
##########
@@ -81,13 +84,16 @@
public abstract class HoodieBackedTableMetadataWriter implements HoodieTableMetadataWriter {
private static final Logger LOG = LogManager.getLogger(HoodieBackedTableMetadataWriter.class);
+ private static final Integer MAX_BUCKET_COUNT = 9999;
Review comment:
lets make this 99999? do we even need this? this is just to format the bucket right? can't we dynamically figure this out. i.e if `numBuckets=9999` ,the you know there needs to be four digits and bucket string should have four characters right.
##########
File path: hudi-client/hudi-client-common/src/main/java/org/apache/hudi/metadata/HoodieBackedTableMetadataWriter.java
##########
@@ -401,64 +396,86 @@ private boolean bootstrapFromFilesystem(HoodieEngineContext engineContext, Hoodi
}
/**
- * Sync the Metadata Table from the instants created on the dataset.
+ * Initialize buckets for a partition. For file listing, we just have one bucket. But for record level index, we might have N number of buckets
Review comment:
we should avoid leaking buckets everywhere. unless we cannot explain code with jsut file groups, we should not be using buckets IMO.
##########
File path: hudi-client/hudi-flink-client/src/main/java/org/apache/hudi/table/upgrade/FlinkUpgradeDowngrade.java
##########
@@ -49,7 +49,9 @@ public void run(HoodieTableMetaClient metaClient, HoodieTableVersion toVersion,
return new ZeroToOneUpgradeHandler().upgrade(config, context, instantTime);
} else if (fromVersion == HoodieTableVersion.ONE && toVersion == HoodieTableVersion.TWO) {
return new OneToTwoUpgradeHandler().upgrade(config, context, instantTime);
- } else {
+ } else if (fromVersion == HoodieTableVersion.TWO && toVersion == HoodieTableVersion.THREE) {
Review comment:
lets add a jira to track what all needs to be finally done for an upgrade-downgrade story here
##########
File path: hudi-client/hudi-client-common/src/main/java/org/apache/hudi/metadata/HoodieTableMetadataWriter.java
##########
@@ -31,13 +30,16 @@
*/
public interface HoodieTableMetadataWriter extends Serializable, AutoCloseable {
+ // Update the metadata table due to a COMMIT operation
Review comment:
javadoc
##########
File path: hudi-client/hudi-spark-client/src/main/java/org/apache/hudi/client/SparkRDDWriteClient.java
##########
@@ -298,28 +306,38 @@ protected void completeCompaction(HoodieCommitMetadata metadata, JavaRDD<WriteSt
String compactionCommitTime) {
this.context.setJobStatus(this.getClass().getSimpleName(), "Collect compaction write status and commit compaction");
List<HoodieWriteStat> writeStats = writeStatuses.map(WriteStatus::getStat).collect();
- finalizeWrite(table, compactionCommitTime, writeStats);
- LOG.info("Committing Compaction " + compactionCommitTime + ". Finished with result " + metadata);
- SparkCompactHelpers.newInstance().completeInflightCompaction(table, compactionCommitTime, metadata);
- WriteMarkersFactory.get(config.getMarkersType(), table, compactionCommitTime)
- .quietDeleteMarkerDir(context, config.getMarkersDeleteParallelism());
-
- if (compactionTimer != null) {
- long durationInMs = metrics.getDurationInMs(compactionTimer.stop());
- try {
- metrics.updateCommitMetrics(HoodieActiveTimeline.COMMIT_FORMATTER.parse(compactionCommitTime).getTime(),
- durationInMs, metadata, HoodieActiveTimeline.COMPACTION_ACTION);
- } catch (ParseException e) {
- throw new HoodieCommitException("Commit time is not of valid format. Failed to commit compaction "
- + config.getBasePath() + " at time " + compactionCommitTime, e);
+ try {
+ // TODO: check if we need HeartbeatUtils.abortIfHeartbeatExpired(instantTime, table, heartbeatClient, config) here.
+ this.txnManager.beginTransaction(Option.of(new HoodieInstant(HoodieInstant.State.INFLIGHT, HoodieTimeline.COMPACTION_ACTION, compactionCommitTime)), Option.empty());
Review comment:
lets shrink it to just the oart needed within a lock
##########
File path: hudi-client/hudi-spark-client/src/main/java/org/apache/hudi/client/SparkRDDWriteClient.java
##########
@@ -339,6 +357,7 @@ protected void completeCompaction(HoodieCommitMetadata metadata, JavaRDD<WriteSt
@Override
public HoodieWriteMetadata<JavaRDD<WriteStatus>> cluster(String clusteringInstant, boolean shouldComplete) {
HoodieSparkTable<T> table = HoodieSparkTable.create(config, context);
+ table.getHoodieView().sync();
Review comment:
more instances to cleanup?
##########
File path: hudi-client/hudi-client-common/src/main/java/org/apache/hudi/client/AbstractHoodieWriteClient.java
##########
@@ -273,6 +275,7 @@ public void bootstrap(Option<Map<String, String>> extraMetadata) {
public void rollbackFailedBootstrap() {
LOG.info("Rolling back pending bootstrap if present");
HoodieTable<T, I, K, O> table = createTable(config, hadoopConf);
+ table.getHoodieView().sync();
Review comment:
can this call getTableAnd.... ?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] nsivabalan commented on pull request #3590: [HUDI-2285][HUDI-2476] Metadata table synchronous design. Rebased and Squashed from pull/3426
Posted by GitBox <gi...@apache.org>.
nsivabalan commented on pull request #3590:
URL: https://github.com/apache/hudi/pull/3590#issuecomment-933930608
@hudi-bot azure run
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] vinothchandar commented on a change in pull request #3590: [HUDI-2285][HUDI-2476] Metadata table synchronous design. Rebased and Squashed from pull/3426
Posted by GitBox <gi...@apache.org>.
vinothchandar commented on a change in pull request #3590:
URL: https://github.com/apache/hudi/pull/3590#discussion_r719356462
##########
File path: hudi-client/hudi-client-common/src/main/java/org/apache/hudi/client/AbstractHoodieWriteClient.java
##########
@@ -241,13 +242,16 @@ void emitCommitMetrics(String instantTime, HoodieCommitMetadata metadata, String
}
}
+ /**
+ * Any pre-commit actions like conflict resolution or updating metadata table goes here.
+ * @param instantTime commit instant time.
+ * @param metadata commit metadata for which pre commit is being invoked.
+ */
protected void preCommit(String instantTime, HoodieCommitMetadata metadata) {
- // no-op
- // TODO : Conflict resolution is not supported for Flink & Java engines
- }
-
- protected void syncTableMetadata() {
- // no-op
+ // Create a Hoodie table after startTxn which encapsulated the commits and files visible.
+ // Important to create this after the lock to ensure latest commits show up in the timeline without need for reload
Review comment:
It'spretty hard to reason about these locking issues while insidethe method. Can we please move Comments to the caller
##########
File path: hudi-client/hudi-client-common/src/main/java/org/apache/hudi/client/AbstractHoodieWriteClient.java
##########
@@ -241,13 +242,16 @@ void emitCommitMetrics(String instantTime, HoodieCommitMetadata metadata, String
}
}
+ /**
+ * Any pre-commit actions like conflict resolution or updating metadata table goes here.
+ * @param instantTime commit instant time.
+ * @param metadata commit metadata for which pre commit is being invoked.
+ */
protected void preCommit(String instantTime, HoodieCommitMetadata metadata) {
- // no-op
- // TODO : Conflict resolution is not supported for Flink & Java engines
- }
-
- protected void syncTableMetadata() {
- // no-op
+ // Create a Hoodie table after startTxn which encapsulated the commits and files visible.
Review comment:
Whats 'startTxn' here? Can we just use plain speak?
##########
File path: hudi-client/hudi-client-common/src/main/java/org/apache/hudi/metadata/HoodieBackedTableMetadataWriter.java
##########
@@ -280,23 +272,23 @@ protected void bootstrapIfNeeded(HoodieEngineContext engineContext, HoodieTableM
/**
* Initialize the Metadata Table by listing files and partitions from the file system.
*
- * @param datasetMetaClient {@code HoodieTableMetaClient} for the dataset
+ * @param dataMetaClient {@code HoodieTableMetaClient} for the dataset.
*/
- private boolean bootstrapFromFilesystem(HoodieEngineContext engineContext, HoodieTableMetaClient datasetMetaClient) throws IOException {
+ private boolean bootstrapFromFilesystem(HoodieEngineContext engineContext, HoodieTableMetaClient dataMetaClient) throws IOException {
ValidationUtils.checkState(enabled, "Metadata table cannot be initialized as it is not enabled");
// We can only bootstrap if there are no pending operations on the dataset
- Option<HoodieInstant> pendingInstantOption = Option.fromJavaOptional(datasetMetaClient.getActiveTimeline()
+ Option<HoodieInstant> pendingDataInstantOption = Option.fromJavaOptional(dataMetaClient.getActiveTimeline()
Review comment:
We can drop " Option". It's kind of redundant
##########
File path: hudi-client/hudi-spark-client/src/main/java/org/apache/hudi/metadata/SparkHoodieBackedTableMetadataWriter.java
##########
@@ -99,83 +94,94 @@ protected void initialize(HoodieEngineContext engineContext, HoodieTableMetaClie
@Override
protected void commit(List<HoodieRecord> records, String partitionName, String instantTime) {
ValidationUtils.checkState(enabled, "Metadata table cannot be committed to as it is not enabled");
- JavaRDD<HoodieRecord> recordRDD = prepRecords(records, partitionName);
+ JavaRDD<HoodieRecord> recordRDD = prepRecords(records, partitionName, 1);
try (SparkRDDWriteClient writeClient = new SparkRDDWriteClient(engineContext, metadataWriteConfig, true)) {
- writeClient.startCommitWithTime(instantTime);
+ if (!metadataMetaClient.getActiveTimeline().filterCompletedInstants().containsInstant(instantTime)) {
+ // if this is a new commit being applied to metadata for the first time
+ writeClient.startCommitWithTime(instantTime);
+ } else {
+ // this code path refers to a re-attempted commit that got committed to metadata, but failed in dataset.
+ // for eg, lets say compaction c1 on 1st attempt succeeded in metadata table and failed before committing to datatable.
+ // when retried again, data table will first rollback pending compaction. these will be applied to metadata table, but all changes
+ // are upserts to metadata table and so only a new delta commit will be created.
+ // once rollback is complete, compaction will be retried again, which will eventually hit this code block where the respective commit is
+ // already part of completed commit. So, we have to manually remove the completed instant and proceed.
+ // and it is for the same reason we enabled withAllowMultiWriteOnSameInstant for metadata table.
+ HoodieInstant alreadyCompletedInstant = metadataMetaClient.getActiveTimeline().filterCompletedInstants().filter(entry -> entry.getTimestamp().equals(instantTime)).lastInstant().get();
+ FSUtils.deleteInstantFile(metadataMetaClient.getFs(), metadataMetaClient.getMetaPath(), alreadyCompletedInstant);
Review comment:
`deleteInstantFile` does not belong in FSUtils. FSUtils should be limited to just things done using a filesystem instance. can you move this to a method on the timeline. there are already methods like those in `HoodieActiveTimeline`
##########
File path: hudi-client/hudi-spark-client/src/main/java/org/apache/hudi/table/HoodieSparkTable.java
##########
@@ -85,4 +96,30 @@ protected HoodieSparkTable(HoodieWriteConfig config, HoodieEngineContext context
protected HoodieIndex<T, JavaRDD<HoodieRecord<T>>, JavaRDD<HoodieKey>, JavaRDD<WriteStatus>> getIndex(HoodieWriteConfig config, HoodieEngineContext context) {
return SparkHoodieIndex.createIndex(config);
}
+
+ /**
+ * Fetch instance of {@link HoodieTableMetadataWriter}.
+ *
+ * @return instance of {@link HoodieTableMetadataWriter}
+ */
+ @Override
+ public Option<HoodieTableMetadataWriter> getMetadataWriter() {
+ synchronized (this) {
+ if (!isMetadataInfoUpdated.getAndSet(true)) {
Review comment:
Can we simplify?
##########
File path: hudi-client/hudi-spark-client/src/main/java/org/apache/hudi/table/HoodieSparkTable.java
##########
@@ -85,4 +96,30 @@ protected HoodieSparkTable(HoodieWriteConfig config, HoodieEngineContext context
protected HoodieIndex<T, JavaRDD<HoodieRecord<T>>, JavaRDD<HoodieKey>, JavaRDD<WriteStatus>> getIndex(HoodieWriteConfig config, HoodieEngineContext context) {
return SparkHoodieIndex.createIndex(config);
}
+
+ /**
+ * Fetch instance of {@link HoodieTableMetadataWriter}.
+ *
+ * @return instance of {@link HoodieTableMetadataWriter}
+ */
+ @Override
+ public Option<HoodieTableMetadataWriter> getMetadataWriter() {
+ synchronized (this) {
+ if (!isMetadataInfoUpdated.getAndSet(true)) {
Review comment:
why do we need this, if you are going to synchronize anyway?
##########
File path: hudi-client/hudi-client-common/src/main/java/org/apache/hudi/client/AbstractHoodieWriteClient.java
##########
@@ -241,13 +242,16 @@ void emitCommitMetrics(String instantTime, HoodieCommitMetadata metadata, String
}
}
+ /**
+ * Any pre-commit actions like conflict resolution or updating metadata table goes here.
+ * @param instantTime commit instant time.
+ * @param metadata commit metadata for which pre commit is being invoked.
+ */
protected void preCommit(String instantTime, HoodieCommitMetadata metadata) {
- // no-op
- // TODO : Conflict resolution is not supported for Flink & Java engines
- }
-
- protected void syncTableMetadata() {
- // no-op
+ // Create a Hoodie table after startTxn which encapsulated the commits and files visible.
+ // Important to create this after the lock to ensure latest commits show up in the timeline without need for reload
Review comment:
Don't we already create a table instance from the caller? Can we not reuse it. Are you saying you need to see even the in-flight instants in this commit
##########
File path: hudi-client/hudi-client-common/src/main/java/org/apache/hudi/metadata/HoodieBackedTableMetadataWriter.java
##########
@@ -401,64 +394,82 @@ private boolean bootstrapFromFilesystem(HoodieEngineContext engineContext, Hoodi
}
/**
- * Sync the Metadata Table from the instants created on the dataset.
+ * Initialize file groups for a partition. For file listing, we just have one file group.
*
- * @param datasetMetaClient {@code HoodieTableMetaClient} for the dataset
+ * All FileGroups for a given metadata partition has a fixed prefix as per the {@link MetadataPartitionType#getFileIdPrefix()}.
+ * Each file group is suffixed with 4 digits with increments of 1 starting with 0.
+ *
+ * For instance, for FILES, there is only one file group named as "files-0000"
Review comment:
Let's trim this comment'?
##########
File path: hudi-client/hudi-client-common/src/test/java/org/apache/hudi/client/transaction/FileSystemBasedLockProviderTestClass.java
##########
@@ -78,6 +81,13 @@ public boolean tryLock(long time, TimeUnit unit) {
&& (numRetries <= lockConfiguration.getConfig().getInteger(LOCK_ACQUIRE_NUM_RETRIES_PROP_KEY))) {
Thread.sleep(lockConfiguration.getConfig().getInteger(LOCK_ACQUIRE_RETRY_WAIT_TIME_IN_MILLIS_PROP_KEY));
}
+ // if two processes tries to acquire lock at the same time, above while loop will be bypassed and both will proceed on to acquire lock.
+ // locally, fs.create(lockFile) succeeds for both even though only one should succeed.
+ // hence adding this random sleep to guard against two processes acquiring same lock by mistake.
+ Thread.sleep(RANDOM.nextInt(10));
Review comment:
this will lead to flaky/non-determinsitc tests? lets fix it by other means?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot edited a comment on pull request #3590: [HUDI-2285][HUDI-2476] Metadata table synchronous design. Rebased and Squashed from pull/3426
Posted by GitBox <gi...@apache.org>.
hudi-bot edited a comment on pull request #3590:
URL: https://github.com/apache/hudi/pull/3590#issuecomment-912237120
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "648e36e1af308534bc050c3a19da008ab7a8c840",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1994",
"triggerID" : "648e36e1af308534bc050c3a19da008ab7a8c840",
"triggerType" : "PUSH"
}, {
"hash" : "8c32253ad0f6d9eb728789fad75763bb5703678d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1995",
"triggerID" : "8c32253ad0f6d9eb728789fad75763bb5703678d",
"triggerType" : "PUSH"
}, {
"hash" : "71bbe59aa8bf70b9aa8990a6dd2522eb4bf1d172",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2009",
"triggerID" : "71bbe59aa8bf70b9aa8990a6dd2522eb4bf1d172",
"triggerType" : "PUSH"
}, {
"hash" : "aefac7ec2f2e40bdf3ad4365ea6aa825803a439d",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "aefac7ec2f2e40bdf3ad4365ea6aa825803a439d",
"triggerType" : "PUSH"
}, {
"hash" : "86905cb855fecfeb9e4c023d526ad643fadb8084",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2012",
"triggerID" : "86905cb855fecfeb9e4c023d526ad643fadb8084",
"triggerType" : "PUSH"
}, {
"hash" : "b4d93b0168775cabd43609fec59bf2823dd81279",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2029",
"triggerID" : "b4d93b0168775cabd43609fec59bf2823dd81279",
"triggerType" : "PUSH"
}, {
"hash" : "38bf430ecb4e11839d686929be2511eed340e9df",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2035",
"triggerID" : "38bf430ecb4e11839d686929be2511eed340e9df",
"triggerType" : "PUSH"
}, {
"hash" : "7793fbdb9b93a129ef606cb2d73ea6e1e9074957",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2299",
"triggerID" : "7793fbdb9b93a129ef606cb2d73ea6e1e9074957",
"triggerType" : "PUSH"
}, {
"hash" : "9c0123c0f27f990d009b323bab75b76ceecf3dab",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2307",
"triggerID" : "9c0123c0f27f990d009b323bab75b76ceecf3dab",
"triggerType" : "PUSH"
}, {
"hash" : "af27976ba07afb02305c699aeb7a75a122013b62",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2330",
"triggerID" : "af27976ba07afb02305c699aeb7a75a122013b62",
"triggerType" : "PUSH"
}, {
"hash" : "9172241c58424cca8e8186d4409a8c36968a4790",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2344",
"triggerID" : "9172241c58424cca8e8186d4409a8c36968a4790",
"triggerType" : "PUSH"
}, {
"hash" : "13b80067ae00a760c57d184f70e2d98f63027e80",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2353",
"triggerID" : "13b80067ae00a760c57d184f70e2d98f63027e80",
"triggerType" : "PUSH"
}, {
"hash" : "ef750979cf449ff504da194e594cb62302eba360",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2366",
"triggerID" : "ef750979cf449ff504da194e594cb62302eba360",
"triggerType" : "PUSH"
}, {
"hash" : "16827c37f36d3b9fec18e0a35aff7599c45e0444",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2375",
"triggerID" : "16827c37f36d3b9fec18e0a35aff7599c45e0444",
"triggerType" : "PUSH"
}, {
"hash" : "0f68d426b01355537f5124b8f108345a36d3da39",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2378",
"triggerID" : "0f68d426b01355537f5124b8f108345a36d3da39",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* aefac7ec2f2e40bdf3ad4365ea6aa825803a439d UNKNOWN
* 0f68d426b01355537f5124b8f108345a36d3da39 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2378)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run travis` re-run the last Travis build
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] nsivabalan commented on pull request #3590: [HUDI-2285][HUDI-2476] Metadata table synchronous design. Rebased and Squashed from pull/3426
Posted by GitBox <gi...@apache.org>.
nsivabalan commented on pull request #3590:
URL: https://github.com/apache/hudi/pull/3590#issuecomment-928195283
@hudi-bot azure run
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot edited a comment on pull request #3590: [WIP] Sync metadata patch
Posted by GitBox <gi...@apache.org>.
hudi-bot edited a comment on pull request #3590:
URL: https://github.com/apache/hudi/pull/3590#issuecomment-912237120
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "648e36e1af308534bc050c3a19da008ab7a8c840",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1994",
"triggerID" : "648e36e1af308534bc050c3a19da008ab7a8c840",
"triggerType" : "PUSH"
}, {
"hash" : "8c32253ad0f6d9eb728789fad75763bb5703678d",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1995",
"triggerID" : "8c32253ad0f6d9eb728789fad75763bb5703678d",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 8c32253ad0f6d9eb728789fad75763bb5703678d Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1995)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run travis` re-run the last Travis build
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot edited a comment on pull request #3590: [WIP][HUDI-2285] Metadata table synchronous design. Rebased and Squashed from pull/3426
Posted by GitBox <gi...@apache.org>.
hudi-bot edited a comment on pull request #3590:
URL: https://github.com/apache/hudi/pull/3590#issuecomment-912237120
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "648e36e1af308534bc050c3a19da008ab7a8c840",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1994",
"triggerID" : "648e36e1af308534bc050c3a19da008ab7a8c840",
"triggerType" : "PUSH"
}, {
"hash" : "8c32253ad0f6d9eb728789fad75763bb5703678d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1995",
"triggerID" : "8c32253ad0f6d9eb728789fad75763bb5703678d",
"triggerType" : "PUSH"
}, {
"hash" : "71bbe59aa8bf70b9aa8990a6dd2522eb4bf1d172",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2009",
"triggerID" : "71bbe59aa8bf70b9aa8990a6dd2522eb4bf1d172",
"triggerType" : "PUSH"
}, {
"hash" : "aefac7ec2f2e40bdf3ad4365ea6aa825803a439d",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "aefac7ec2f2e40bdf3ad4365ea6aa825803a439d",
"triggerType" : "PUSH"
}, {
"hash" : "86905cb855fecfeb9e4c023d526ad643fadb8084",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2012",
"triggerID" : "86905cb855fecfeb9e4c023d526ad643fadb8084",
"triggerType" : "PUSH"
}, {
"hash" : "b4d93b0168775cabd43609fec59bf2823dd81279",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2029",
"triggerID" : "b4d93b0168775cabd43609fec59bf2823dd81279",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* aefac7ec2f2e40bdf3ad4365ea6aa825803a439d UNKNOWN
* 86905cb855fecfeb9e4c023d526ad643fadb8084 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2012)
* b4d93b0168775cabd43609fec59bf2823dd81279 Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2029)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run travis` re-run the last Travis build
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot edited a comment on pull request #3590: [WIP][HUDI-2285] Metadata table synchronous design. Rebased and Squashed from pull/3426
Posted by GitBox <gi...@apache.org>.
hudi-bot edited a comment on pull request #3590:
URL: https://github.com/apache/hudi/pull/3590#issuecomment-912237120
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "648e36e1af308534bc050c3a19da008ab7a8c840",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1994",
"triggerID" : "648e36e1af308534bc050c3a19da008ab7a8c840",
"triggerType" : "PUSH"
}, {
"hash" : "8c32253ad0f6d9eb728789fad75763bb5703678d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1995",
"triggerID" : "8c32253ad0f6d9eb728789fad75763bb5703678d",
"triggerType" : "PUSH"
}, {
"hash" : "71bbe59aa8bf70b9aa8990a6dd2522eb4bf1d172",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2009",
"triggerID" : "71bbe59aa8bf70b9aa8990a6dd2522eb4bf1d172",
"triggerType" : "PUSH"
}, {
"hash" : "aefac7ec2f2e40bdf3ad4365ea6aa825803a439d",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "aefac7ec2f2e40bdf3ad4365ea6aa825803a439d",
"triggerType" : "PUSH"
}, {
"hash" : "86905cb855fecfeb9e4c023d526ad643fadb8084",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2012",
"triggerID" : "86905cb855fecfeb9e4c023d526ad643fadb8084",
"triggerType" : "PUSH"
}, {
"hash" : "b4d93b0168775cabd43609fec59bf2823dd81279",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2029",
"triggerID" : "b4d93b0168775cabd43609fec59bf2823dd81279",
"triggerType" : "PUSH"
}, {
"hash" : "38bf430ecb4e11839d686929be2511eed340e9df",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2035",
"triggerID" : "38bf430ecb4e11839d686929be2511eed340e9df",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* aefac7ec2f2e40bdf3ad4365ea6aa825803a439d UNKNOWN
* 38bf430ecb4e11839d686929be2511eed340e9df Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2035)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run travis` re-run the last Travis build
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] nsivabalan merged pull request #3590: [HUDI-2285][HUDI-2476] Metadata table synchronous design. Rebased and Squashed from pull/3426
Posted by GitBox <gi...@apache.org>.
nsivabalan merged pull request #3590:
URL: https://github.com/apache/hudi/pull/3590
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot edited a comment on pull request #3590: [HUDI-2285] Metadata table synchronous design. Rebased and Squashed from pull/3426
Posted by GitBox <gi...@apache.org>.
hudi-bot edited a comment on pull request #3590:
URL: https://github.com/apache/hudi/pull/3590#issuecomment-912237120
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "648e36e1af308534bc050c3a19da008ab7a8c840",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1994",
"triggerID" : "648e36e1af308534bc050c3a19da008ab7a8c840",
"triggerType" : "PUSH"
}, {
"hash" : "8c32253ad0f6d9eb728789fad75763bb5703678d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1995",
"triggerID" : "8c32253ad0f6d9eb728789fad75763bb5703678d",
"triggerType" : "PUSH"
}, {
"hash" : "71bbe59aa8bf70b9aa8990a6dd2522eb4bf1d172",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2009",
"triggerID" : "71bbe59aa8bf70b9aa8990a6dd2522eb4bf1d172",
"triggerType" : "PUSH"
}, {
"hash" : "aefac7ec2f2e40bdf3ad4365ea6aa825803a439d",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "aefac7ec2f2e40bdf3ad4365ea6aa825803a439d",
"triggerType" : "PUSH"
}, {
"hash" : "86905cb855fecfeb9e4c023d526ad643fadb8084",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2012",
"triggerID" : "86905cb855fecfeb9e4c023d526ad643fadb8084",
"triggerType" : "PUSH"
}, {
"hash" : "b4d93b0168775cabd43609fec59bf2823dd81279",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2029",
"triggerID" : "b4d93b0168775cabd43609fec59bf2823dd81279",
"triggerType" : "PUSH"
}, {
"hash" : "38bf430ecb4e11839d686929be2511eed340e9df",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2035",
"triggerID" : "38bf430ecb4e11839d686929be2511eed340e9df",
"triggerType" : "PUSH"
}, {
"hash" : "7793fbdb9b93a129ef606cb2d73ea6e1e9074957",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2299",
"triggerID" : "7793fbdb9b93a129ef606cb2d73ea6e1e9074957",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* aefac7ec2f2e40bdf3ad4365ea6aa825803a439d UNKNOWN
* 38bf430ecb4e11839d686929be2511eed340e9df Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2035)
* 7793fbdb9b93a129ef606cb2d73ea6e1e9074957 Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2299)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run travis` re-run the last Travis build
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot edited a comment on pull request #3590: [HUDI-2285][HUDI-2476] Metadata table synchronous design. Rebased and Squashed from pull/3426
Posted by GitBox <gi...@apache.org>.
hudi-bot edited a comment on pull request #3590:
URL: https://github.com/apache/hudi/pull/3590#issuecomment-912237120
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "648e36e1af308534bc050c3a19da008ab7a8c840",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1994",
"triggerID" : "648e36e1af308534bc050c3a19da008ab7a8c840",
"triggerType" : "PUSH"
}, {
"hash" : "8c32253ad0f6d9eb728789fad75763bb5703678d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1995",
"triggerID" : "8c32253ad0f6d9eb728789fad75763bb5703678d",
"triggerType" : "PUSH"
}, {
"hash" : "71bbe59aa8bf70b9aa8990a6dd2522eb4bf1d172",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2009",
"triggerID" : "71bbe59aa8bf70b9aa8990a6dd2522eb4bf1d172",
"triggerType" : "PUSH"
}, {
"hash" : "aefac7ec2f2e40bdf3ad4365ea6aa825803a439d",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "aefac7ec2f2e40bdf3ad4365ea6aa825803a439d",
"triggerType" : "PUSH"
}, {
"hash" : "86905cb855fecfeb9e4c023d526ad643fadb8084",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2012",
"triggerID" : "86905cb855fecfeb9e4c023d526ad643fadb8084",
"triggerType" : "PUSH"
}, {
"hash" : "b4d93b0168775cabd43609fec59bf2823dd81279",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2029",
"triggerID" : "b4d93b0168775cabd43609fec59bf2823dd81279",
"triggerType" : "PUSH"
}, {
"hash" : "38bf430ecb4e11839d686929be2511eed340e9df",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2035",
"triggerID" : "38bf430ecb4e11839d686929be2511eed340e9df",
"triggerType" : "PUSH"
}, {
"hash" : "7793fbdb9b93a129ef606cb2d73ea6e1e9074957",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2299",
"triggerID" : "7793fbdb9b93a129ef606cb2d73ea6e1e9074957",
"triggerType" : "PUSH"
}, {
"hash" : "9c0123c0f27f990d009b323bab75b76ceecf3dab",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2307",
"triggerID" : "9c0123c0f27f990d009b323bab75b76ceecf3dab",
"triggerType" : "PUSH"
}, {
"hash" : "af27976ba07afb02305c699aeb7a75a122013b62",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2330",
"triggerID" : "af27976ba07afb02305c699aeb7a75a122013b62",
"triggerType" : "PUSH"
}, {
"hash" : "9172241c58424cca8e8186d4409a8c36968a4790",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2344",
"triggerID" : "9172241c58424cca8e8186d4409a8c36968a4790",
"triggerType" : "PUSH"
}, {
"hash" : "13b80067ae00a760c57d184f70e2d98f63027e80",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2353",
"triggerID" : "13b80067ae00a760c57d184f70e2d98f63027e80",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* aefac7ec2f2e40bdf3ad4365ea6aa825803a439d UNKNOWN
* 9172241c58424cca8e8186d4409a8c36968a4790 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2344)
* 13b80067ae00a760c57d184f70e2d98f63027e80 Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2353)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run travis` re-run the last Travis build
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] nsivabalan commented on a change in pull request #3590: [HUDI-2285][HUDI-2476] Metadata table synchronous design. Rebased and Squashed from pull/3426
Posted by GitBox <gi...@apache.org>.
nsivabalan commented on a change in pull request #3590:
URL: https://github.com/apache/hudi/pull/3590#discussion_r716094533
##########
File path: hudi-client/hudi-client-common/src/main/java/org/apache/hudi/client/AbstractHoodieWriteClient.java
##########
@@ -188,6 +189,7 @@ public boolean commitStats(String instantTime, List<HoodieWriteStat> stats, Opti
lastCompletedTxnAndMetadata.isPresent() ? Option.of(lastCompletedTxnAndMetadata.get().getLeft()) : Option.empty());
try {
preCommit(instantTime, metadata);
+ table.getMetadataWriter().ifPresent(w -> ((HoodieTableMetadataWriter)w).update(metadata, instantTime));
Review comment:
looks like preCommit is meant to do conflict resolution. filnk and java has empty impl. and spark is the only one which overrides and gives concrete impl. So, we can't really move this inside preCommit. I thought of adding a new method, but then all engines are going to make this call anyways(just that if metadata is enabled update() will be invoked, if not, its a no-op).
Let me know if there are any other better ways to go about.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot edited a comment on pull request #3590: [WIP][HUDI-2285] Metadata table synchronous design. Rebased and Squashed from pull/3426
Posted by GitBox <gi...@apache.org>.
hudi-bot edited a comment on pull request #3590:
URL: https://github.com/apache/hudi/pull/3590#issuecomment-912237120
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "648e36e1af308534bc050c3a19da008ab7a8c840",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1994",
"triggerID" : "648e36e1af308534bc050c3a19da008ab7a8c840",
"triggerType" : "PUSH"
}, {
"hash" : "8c32253ad0f6d9eb728789fad75763bb5703678d",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1995",
"triggerID" : "8c32253ad0f6d9eb728789fad75763bb5703678d",
"triggerType" : "PUSH"
}, {
"hash" : "71bbe59aa8bf70b9aa8990a6dd2522eb4bf1d172",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "71bbe59aa8bf70b9aa8990a6dd2522eb4bf1d172",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 8c32253ad0f6d9eb728789fad75763bb5703678d Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1995)
* 71bbe59aa8bf70b9aa8990a6dd2522eb4bf1d172 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run travis` re-run the last Travis build
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot edited a comment on pull request #3590: [WIP][HUDI-2285] Metadata table synchronous design. Rebased and Squashed from pull/3426
Posted by GitBox <gi...@apache.org>.
hudi-bot edited a comment on pull request #3590:
URL: https://github.com/apache/hudi/pull/3590#issuecomment-912237120
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "648e36e1af308534bc050c3a19da008ab7a8c840",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1994",
"triggerID" : "648e36e1af308534bc050c3a19da008ab7a8c840",
"triggerType" : "PUSH"
}, {
"hash" : "8c32253ad0f6d9eb728789fad75763bb5703678d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1995",
"triggerID" : "8c32253ad0f6d9eb728789fad75763bb5703678d",
"triggerType" : "PUSH"
}, {
"hash" : "71bbe59aa8bf70b9aa8990a6dd2522eb4bf1d172",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2009",
"triggerID" : "71bbe59aa8bf70b9aa8990a6dd2522eb4bf1d172",
"triggerType" : "PUSH"
}, {
"hash" : "aefac7ec2f2e40bdf3ad4365ea6aa825803a439d",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "aefac7ec2f2e40bdf3ad4365ea6aa825803a439d",
"triggerType" : "PUSH"
}, {
"hash" : "86905cb855fecfeb9e4c023d526ad643fadb8084",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2012",
"triggerID" : "86905cb855fecfeb9e4c023d526ad643fadb8084",
"triggerType" : "PUSH"
}, {
"hash" : "b4d93b0168775cabd43609fec59bf2823dd81279",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2029",
"triggerID" : "b4d93b0168775cabd43609fec59bf2823dd81279",
"triggerType" : "PUSH"
}, {
"hash" : "38bf430ecb4e11839d686929be2511eed340e9df",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2035",
"triggerID" : "38bf430ecb4e11839d686929be2511eed340e9df",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* aefac7ec2f2e40bdf3ad4365ea6aa825803a439d UNKNOWN
* b4d93b0168775cabd43609fec59bf2823dd81279 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2029)
* 38bf430ecb4e11839d686929be2511eed340e9df Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2035)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run travis` re-run the last Travis build
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot edited a comment on pull request #3590: [WIP] Sync metadata patch
Posted by GitBox <gi...@apache.org>.
hudi-bot edited a comment on pull request #3590:
URL: https://github.com/apache/hudi/pull/3590#issuecomment-912237120
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "648e36e1af308534bc050c3a19da008ab7a8c840",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1994",
"triggerID" : "648e36e1af308534bc050c3a19da008ab7a8c840",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 648e36e1af308534bc050c3a19da008ab7a8c840 Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1994)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run travis` re-run the last Travis build
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] nsivabalan edited a comment on pull request #3590: [HUDI-2285] Metadata table synchronous design. Rebased and Squashed from pull/3426
Posted by GitBox <gi...@apache.org>.
nsivabalan edited a comment on pull request #3590:
URL: https://github.com/apache/hudi/pull/3590#issuecomment-922983380
@vinothchandar :
I have addressed most of the feedback from you except for few. Have responded in the other patch for those which is yet to be resolved.
wrt multi-writer and compaction, here is what is happening in this patch. (have covered this in the description of the patch, but anyways).
All writes to metadata table happens within the datatable lock. So, implicitly there are no multi-writers (including compaction, cleaning for metadata table) to metadata table. There will always be only one writer. For regular writes to data table, we do conflict resolution, where as for compaction and clustering (data table), we don't do any conflict resolution. Remember taking a lock while committing compaction and clustering in data table is being added in this patch. So, please pay attention if there is something I am missing.
So, in summary, we have a constraint only single writer model to metadata table. May be we need to revisit this when we integrate record level index. But for now, this is our current state of things.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] nsivabalan commented on a change in pull request #3590: [HUDI-2285] Metadata table synchronous design. Rebased and Squashed from pull/3426
Posted by GitBox <gi...@apache.org>.
nsivabalan commented on a change in pull request #3590:
URL: https://github.com/apache/hudi/pull/3590#discussion_r712393848
##########
File path: hudi-client/hudi-spark-client/src/main/java/org/apache/hudi/client/SparkRDDWriteClient.java
##########
@@ -442,25 +467,10 @@ private void completeTableService(TableServiceType tableServiceType, HoodieCommi
} else {
writeTimer = metrics.getDeltaCommitCtx();
}
+ table.getHoodieView().sync();
Review comment:
need to discuss on whats the right thing to do here. timeline does not refresh for every operation, but in some tests we do N no of operations and so we need this refresh. We might have a deltastreamer which uses same writeClirnt and does N no of write operations. For spark datasource, should not be an issue.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot edited a comment on pull request #3590: [HUDI-2285][HUDI-2476] Metadata table synchronous design. Rebased and Squashed from pull/3426
Posted by GitBox <gi...@apache.org>.
hudi-bot edited a comment on pull request #3590:
URL: https://github.com/apache/hudi/pull/3590#issuecomment-912237120
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "648e36e1af308534bc050c3a19da008ab7a8c840",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1994",
"triggerID" : "648e36e1af308534bc050c3a19da008ab7a8c840",
"triggerType" : "PUSH"
}, {
"hash" : "8c32253ad0f6d9eb728789fad75763bb5703678d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1995",
"triggerID" : "8c32253ad0f6d9eb728789fad75763bb5703678d",
"triggerType" : "PUSH"
}, {
"hash" : "71bbe59aa8bf70b9aa8990a6dd2522eb4bf1d172",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2009",
"triggerID" : "71bbe59aa8bf70b9aa8990a6dd2522eb4bf1d172",
"triggerType" : "PUSH"
}, {
"hash" : "aefac7ec2f2e40bdf3ad4365ea6aa825803a439d",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "aefac7ec2f2e40bdf3ad4365ea6aa825803a439d",
"triggerType" : "PUSH"
}, {
"hash" : "86905cb855fecfeb9e4c023d526ad643fadb8084",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2012",
"triggerID" : "86905cb855fecfeb9e4c023d526ad643fadb8084",
"triggerType" : "PUSH"
}, {
"hash" : "b4d93b0168775cabd43609fec59bf2823dd81279",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2029",
"triggerID" : "b4d93b0168775cabd43609fec59bf2823dd81279",
"triggerType" : "PUSH"
}, {
"hash" : "38bf430ecb4e11839d686929be2511eed340e9df",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2035",
"triggerID" : "38bf430ecb4e11839d686929be2511eed340e9df",
"triggerType" : "PUSH"
}, {
"hash" : "7793fbdb9b93a129ef606cb2d73ea6e1e9074957",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2299",
"triggerID" : "7793fbdb9b93a129ef606cb2d73ea6e1e9074957",
"triggerType" : "PUSH"
}, {
"hash" : "9c0123c0f27f990d009b323bab75b76ceecf3dab",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2307",
"triggerID" : "9c0123c0f27f990d009b323bab75b76ceecf3dab",
"triggerType" : "PUSH"
}, {
"hash" : "af27976ba07afb02305c699aeb7a75a122013b62",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2330",
"triggerID" : "af27976ba07afb02305c699aeb7a75a122013b62",
"triggerType" : "PUSH"
}, {
"hash" : "9172241c58424cca8e8186d4409a8c36968a4790",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2344",
"triggerID" : "9172241c58424cca8e8186d4409a8c36968a4790",
"triggerType" : "PUSH"
}, {
"hash" : "13b80067ae00a760c57d184f70e2d98f63027e80",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2353",
"triggerID" : "13b80067ae00a760c57d184f70e2d98f63027e80",
"triggerType" : "PUSH"
}, {
"hash" : "ef750979cf449ff504da194e594cb62302eba360",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2366",
"triggerID" : "ef750979cf449ff504da194e594cb62302eba360",
"triggerType" : "PUSH"
}, {
"hash" : "16827c37f36d3b9fec18e0a35aff7599c45e0444",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2375",
"triggerID" : "16827c37f36d3b9fec18e0a35aff7599c45e0444",
"triggerType" : "PUSH"
}, {
"hash" : "0f68d426b01355537f5124b8f108345a36d3da39",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2378",
"triggerID" : "0f68d426b01355537f5124b8f108345a36d3da39",
"triggerType" : "PUSH"
}, {
"hash" : "2f3099197889fd5c48ae3b550cc523e0528ddb2c",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2380",
"triggerID" : "2f3099197889fd5c48ae3b550cc523e0528ddb2c",
"triggerType" : "PUSH"
}, {
"hash" : "c83d176e1d1ddab6bb86f91726060779a9a1519f",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2392",
"triggerID" : "c83d176e1d1ddab6bb86f91726060779a9a1519f",
"triggerType" : "PUSH"
}, {
"hash" : "008f0cd91de12d44030d837f460f3b0232d62ee5",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2416",
"triggerID" : "008f0cd91de12d44030d837f460f3b0232d62ee5",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* aefac7ec2f2e40bdf3ad4365ea6aa825803a439d UNKNOWN
* 008f0cd91de12d44030d837f460f3b0232d62ee5 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2416)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run travis` re-run the last Travis build
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] nsivabalan commented on a change in pull request #3590: [HUDI-2285][HUDI-2476] Metadata table synchronous design. Rebased and Squashed from pull/3426
Posted by GitBox <gi...@apache.org>.
nsivabalan commented on a change in pull request #3590:
URL: https://github.com/apache/hudi/pull/3590#discussion_r721356787
##########
File path: hudi-client/hudi-client-common/src/main/java/org/apache/hudi/client/AbstractHoodieWriteClient.java
##########
@@ -241,13 +242,16 @@ void emitCommitMetrics(String instantTime, HoodieCommitMetadata metadata, String
}
}
+ /**
+ * Any pre-commit actions like conflict resolution or updating metadata table goes here.
+ * @param instantTime commit instant time.
+ * @param metadata commit metadata for which pre commit is being invoked.
+ */
protected void preCommit(String instantTime, HoodieCommitMetadata metadata) {
- // no-op
- // TODO : Conflict resolution is not supported for Flink & Java engines
- }
-
- protected void syncTableMetadata() {
- // no-op
+ // Create a Hoodie table after startTxn which encapsulated the commits and files visible.
+ // Important to create this after the lock to ensure latest commits show up in the timeline without need for reload
Review comment:
I feel adding this comment at the caller does not fit in well. This is what it looks like.
```
try {
// Precommit is meant for resolving conflicts if any and to update metadata table if enabled.
// Create a Hoodie table within preCommit after starting the transaction which encapsulated the commits and files visible
// (instead of using an existing instance of Hoodie table). It is important to create this after the lock to ensure latest
// commits show up in the timeline without need for reload.
preCommit(instantTime, metadata);
commit(table, commitActionType, instantTime, metadata, stats);
postCommit(table, metadata, instantTime, extraMetadata);
...
```
I feel having this within preCommit just belongs to the place where it is needed or used.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot edited a comment on pull request #3590: [HUDI-2285][HUDI-2476] Metadata table synchronous design. Rebased and Squashed from pull/3426
Posted by GitBox <gi...@apache.org>.
hudi-bot edited a comment on pull request #3590:
URL: https://github.com/apache/hudi/pull/3590#issuecomment-912237120
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "648e36e1af308534bc050c3a19da008ab7a8c840",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1994",
"triggerID" : "648e36e1af308534bc050c3a19da008ab7a8c840",
"triggerType" : "PUSH"
}, {
"hash" : "8c32253ad0f6d9eb728789fad75763bb5703678d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1995",
"triggerID" : "8c32253ad0f6d9eb728789fad75763bb5703678d",
"triggerType" : "PUSH"
}, {
"hash" : "71bbe59aa8bf70b9aa8990a6dd2522eb4bf1d172",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2009",
"triggerID" : "71bbe59aa8bf70b9aa8990a6dd2522eb4bf1d172",
"triggerType" : "PUSH"
}, {
"hash" : "aefac7ec2f2e40bdf3ad4365ea6aa825803a439d",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "aefac7ec2f2e40bdf3ad4365ea6aa825803a439d",
"triggerType" : "PUSH"
}, {
"hash" : "86905cb855fecfeb9e4c023d526ad643fadb8084",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2012",
"triggerID" : "86905cb855fecfeb9e4c023d526ad643fadb8084",
"triggerType" : "PUSH"
}, {
"hash" : "b4d93b0168775cabd43609fec59bf2823dd81279",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2029",
"triggerID" : "b4d93b0168775cabd43609fec59bf2823dd81279",
"triggerType" : "PUSH"
}, {
"hash" : "38bf430ecb4e11839d686929be2511eed340e9df",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2035",
"triggerID" : "38bf430ecb4e11839d686929be2511eed340e9df",
"triggerType" : "PUSH"
}, {
"hash" : "7793fbdb9b93a129ef606cb2d73ea6e1e9074957",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2299",
"triggerID" : "7793fbdb9b93a129ef606cb2d73ea6e1e9074957",
"triggerType" : "PUSH"
}, {
"hash" : "9c0123c0f27f990d009b323bab75b76ceecf3dab",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2307",
"triggerID" : "9c0123c0f27f990d009b323bab75b76ceecf3dab",
"triggerType" : "PUSH"
}, {
"hash" : "af27976ba07afb02305c699aeb7a75a122013b62",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2330",
"triggerID" : "af27976ba07afb02305c699aeb7a75a122013b62",
"triggerType" : "PUSH"
}, {
"hash" : "9172241c58424cca8e8186d4409a8c36968a4790",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2344",
"triggerID" : "9172241c58424cca8e8186d4409a8c36968a4790",
"triggerType" : "PUSH"
}, {
"hash" : "13b80067ae00a760c57d184f70e2d98f63027e80",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2353",
"triggerID" : "13b80067ae00a760c57d184f70e2d98f63027e80",
"triggerType" : "PUSH"
}, {
"hash" : "ef750979cf449ff504da194e594cb62302eba360",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2366",
"triggerID" : "ef750979cf449ff504da194e594cb62302eba360",
"triggerType" : "PUSH"
}, {
"hash" : "16827c37f36d3b9fec18e0a35aff7599c45e0444",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2375",
"triggerID" : "16827c37f36d3b9fec18e0a35aff7599c45e0444",
"triggerType" : "PUSH"
}, {
"hash" : "0f68d426b01355537f5124b8f108345a36d3da39",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2378",
"triggerID" : "0f68d426b01355537f5124b8f108345a36d3da39",
"triggerType" : "PUSH"
}, {
"hash" : "2f3099197889fd5c48ae3b550cc523e0528ddb2c",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2380",
"triggerID" : "2f3099197889fd5c48ae3b550cc523e0528ddb2c",
"triggerType" : "PUSH"
}, {
"hash" : "c83d176e1d1ddab6bb86f91726060779a9a1519f",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2392",
"triggerID" : "c83d176e1d1ddab6bb86f91726060779a9a1519f",
"triggerType" : "PUSH"
}, {
"hash" : "008f0cd91de12d44030d837f460f3b0232d62ee5",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2416",
"triggerID" : "008f0cd91de12d44030d837f460f3b0232d62ee5",
"triggerType" : "PUSH"
}, {
"hash" : "92634fb730e723a9bdeb165348a4c747794be7e3",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2418",
"triggerID" : "92634fb730e723a9bdeb165348a4c747794be7e3",
"triggerType" : "PUSH"
}, {
"hash" : "3b5fbfa0cc69e82f7c23fc6aa5b4944e2dd77220",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2438",
"triggerID" : "3b5fbfa0cc69e82f7c23fc6aa5b4944e2dd77220",
"triggerType" : "PUSH"
}, {
"hash" : "87d2ec0cc60ef8d5b7a081b73a0318aedf1a2a90",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2451",
"triggerID" : "87d2ec0cc60ef8d5b7a081b73a0318aedf1a2a90",
"triggerType" : "PUSH"
}, {
"hash" : "c09f5e30662efd882ff9fc35f5750a8253171e9d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2452",
"triggerID" : "c09f5e30662efd882ff9fc35f5750a8253171e9d",
"triggerType" : "PUSH"
}, {
"hash" : "b432e1db06e032b8071fa6c9e871ebc30b04b9eb",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2485",
"triggerID" : "b432e1db06e032b8071fa6c9e871ebc30b04b9eb",
"triggerType" : "PUSH"
}, {
"hash" : "0d70dba11569f96064e2cbf5c9d5fcb3c7fee5c0",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2503",
"triggerID" : "0d70dba11569f96064e2cbf5c9d5fcb3c7fee5c0",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* aefac7ec2f2e40bdf3ad4365ea6aa825803a439d UNKNOWN
* 0d70dba11569f96064e2cbf5c9d5fcb3c7fee5c0 Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2503)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run travis` re-run the last Travis build
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] prashantwason commented on a change in pull request #3590: [HUDI-2285][HUDI-2476] Metadata table synchronous design. Rebased and Squashed from pull/3426
Posted by GitBox <gi...@apache.org>.
prashantwason commented on a change in pull request #3590:
URL: https://github.com/apache/hudi/pull/3590#discussion_r759076559
##########
File path: hudi-client/hudi-client-common/src/main/java/org/apache/hudi/table/HoodieTimelineArchiveLog.java
##########
@@ -200,20 +200,19 @@ public boolean archiveIfRequired(HoodieEngineContext context) throws IOException
.collect(Collectors.groupingBy(i -> Pair.of(i.getTimestamp(),
HoodieInstant.getComparableAction(i.getAction()))));
- // If metadata table is enabled, do not archive instants which are more recent that the latest synced
- // instant on the metadata table. This is required for metadata table sync.
+ // If metadata table is enabled, do not archive instants which are more recent that the last compaction on the
+ // metadata table.
if (config.isMetadataTableEnabled()) {
try (HoodieTableMetadata tableMetadata = HoodieTableMetadata.create(table.getContext(), config.getMetadataConfig(),
config.getBasePath(), FileSystemViewStorageConfig.SPILLABLE_DIR.defaultValue())) {
- Option<String> lastSyncedInstantTime = tableMetadata.getUpdateTime();
-
- if (lastSyncedInstantTime.isPresent()) {
- LOG.info("Limiting archiving of instants to last synced instant on metadata table at " + lastSyncedInstantTime.get());
- instants = instants.filter(i -> HoodieTimeline.compareTimestamps(i.getTimestamp(), HoodieTimeline.LESSER_THAN,
- lastSyncedInstantTime.get()));
- } else {
- LOG.info("Not archiving as there is no instants yet on the metadata table");
+ Option<String> latestCompactionTime = tableMetadata.getLatestCompactionTime();
Review comment:
You are right. But attempting to read a deltacommit from a commit archive will introduce a long delay. So currently only the active timeline is used leading to this restriction.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot edited a comment on pull request #3590: [HUDI-2285][HUDI-2476] Metadata table synchronous design. Rebased and Squashed from pull/3426
Posted by GitBox <gi...@apache.org>.
hudi-bot edited a comment on pull request #3590:
URL: https://github.com/apache/hudi/pull/3590#issuecomment-912237120
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "648e36e1af308534bc050c3a19da008ab7a8c840",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1994",
"triggerID" : "648e36e1af308534bc050c3a19da008ab7a8c840",
"triggerType" : "PUSH"
}, {
"hash" : "8c32253ad0f6d9eb728789fad75763bb5703678d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1995",
"triggerID" : "8c32253ad0f6d9eb728789fad75763bb5703678d",
"triggerType" : "PUSH"
}, {
"hash" : "71bbe59aa8bf70b9aa8990a6dd2522eb4bf1d172",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2009",
"triggerID" : "71bbe59aa8bf70b9aa8990a6dd2522eb4bf1d172",
"triggerType" : "PUSH"
}, {
"hash" : "aefac7ec2f2e40bdf3ad4365ea6aa825803a439d",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "aefac7ec2f2e40bdf3ad4365ea6aa825803a439d",
"triggerType" : "PUSH"
}, {
"hash" : "86905cb855fecfeb9e4c023d526ad643fadb8084",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2012",
"triggerID" : "86905cb855fecfeb9e4c023d526ad643fadb8084",
"triggerType" : "PUSH"
}, {
"hash" : "b4d93b0168775cabd43609fec59bf2823dd81279",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2029",
"triggerID" : "b4d93b0168775cabd43609fec59bf2823dd81279",
"triggerType" : "PUSH"
}, {
"hash" : "38bf430ecb4e11839d686929be2511eed340e9df",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2035",
"triggerID" : "38bf430ecb4e11839d686929be2511eed340e9df",
"triggerType" : "PUSH"
}, {
"hash" : "7793fbdb9b93a129ef606cb2d73ea6e1e9074957",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2299",
"triggerID" : "7793fbdb9b93a129ef606cb2d73ea6e1e9074957",
"triggerType" : "PUSH"
}, {
"hash" : "9c0123c0f27f990d009b323bab75b76ceecf3dab",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2307",
"triggerID" : "9c0123c0f27f990d009b323bab75b76ceecf3dab",
"triggerType" : "PUSH"
}, {
"hash" : "af27976ba07afb02305c699aeb7a75a122013b62",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2330",
"triggerID" : "af27976ba07afb02305c699aeb7a75a122013b62",
"triggerType" : "PUSH"
}, {
"hash" : "9172241c58424cca8e8186d4409a8c36968a4790",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2344",
"triggerID" : "9172241c58424cca8e8186d4409a8c36968a4790",
"triggerType" : "PUSH"
}, {
"hash" : "13b80067ae00a760c57d184f70e2d98f63027e80",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2353",
"triggerID" : "13b80067ae00a760c57d184f70e2d98f63027e80",
"triggerType" : "PUSH"
}, {
"hash" : "ef750979cf449ff504da194e594cb62302eba360",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2366",
"triggerID" : "ef750979cf449ff504da194e594cb62302eba360",
"triggerType" : "PUSH"
}, {
"hash" : "16827c37f36d3b9fec18e0a35aff7599c45e0444",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2375",
"triggerID" : "16827c37f36d3b9fec18e0a35aff7599c45e0444",
"triggerType" : "PUSH"
}, {
"hash" : "0f68d426b01355537f5124b8f108345a36d3da39",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2378",
"triggerID" : "0f68d426b01355537f5124b8f108345a36d3da39",
"triggerType" : "PUSH"
}, {
"hash" : "2f3099197889fd5c48ae3b550cc523e0528ddb2c",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2380",
"triggerID" : "2f3099197889fd5c48ae3b550cc523e0528ddb2c",
"triggerType" : "PUSH"
}, {
"hash" : "c83d176e1d1ddab6bb86f91726060779a9a1519f",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2392",
"triggerID" : "c83d176e1d1ddab6bb86f91726060779a9a1519f",
"triggerType" : "PUSH"
}, {
"hash" : "008f0cd91de12d44030d837f460f3b0232d62ee5",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2416",
"triggerID" : "008f0cd91de12d44030d837f460f3b0232d62ee5",
"triggerType" : "PUSH"
}, {
"hash" : "92634fb730e723a9bdeb165348a4c747794be7e3",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2418",
"triggerID" : "92634fb730e723a9bdeb165348a4c747794be7e3",
"triggerType" : "PUSH"
}, {
"hash" : "3b5fbfa0cc69e82f7c23fc6aa5b4944e2dd77220",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2438",
"triggerID" : "3b5fbfa0cc69e82f7c23fc6aa5b4944e2dd77220",
"triggerType" : "PUSH"
}, {
"hash" : "87d2ec0cc60ef8d5b7a081b73a0318aedf1a2a90",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2451",
"triggerID" : "87d2ec0cc60ef8d5b7a081b73a0318aedf1a2a90",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* aefac7ec2f2e40bdf3ad4365ea6aa825803a439d UNKNOWN
* 3b5fbfa0cc69e82f7c23fc6aa5b4944e2dd77220 Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2438)
* 87d2ec0cc60ef8d5b7a081b73a0318aedf1a2a90 Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2451)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run travis` re-run the last Travis build
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot edited a comment on pull request #3590: [WIP][HUDI-2285] Metadata table synchronous design. Rebased and Squashed from pull/3426
Posted by GitBox <gi...@apache.org>.
hudi-bot edited a comment on pull request #3590:
URL: https://github.com/apache/hudi/pull/3590#issuecomment-912237120
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "648e36e1af308534bc050c3a19da008ab7a8c840",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1994",
"triggerID" : "648e36e1af308534bc050c3a19da008ab7a8c840",
"triggerType" : "PUSH"
}, {
"hash" : "8c32253ad0f6d9eb728789fad75763bb5703678d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1995",
"triggerID" : "8c32253ad0f6d9eb728789fad75763bb5703678d",
"triggerType" : "PUSH"
}, {
"hash" : "71bbe59aa8bf70b9aa8990a6dd2522eb4bf1d172",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2009",
"triggerID" : "71bbe59aa8bf70b9aa8990a6dd2522eb4bf1d172",
"triggerType" : "PUSH"
}, {
"hash" : "aefac7ec2f2e40bdf3ad4365ea6aa825803a439d",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "aefac7ec2f2e40bdf3ad4365ea6aa825803a439d",
"triggerType" : "PUSH"
}, {
"hash" : "86905cb855fecfeb9e4c023d526ad643fadb8084",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "86905cb855fecfeb9e4c023d526ad643fadb8084",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 71bbe59aa8bf70b9aa8990a6dd2522eb4bf1d172 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2009)
* aefac7ec2f2e40bdf3ad4365ea6aa825803a439d UNKNOWN
* 86905cb855fecfeb9e4c023d526ad643fadb8084 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run travis` re-run the last Travis build
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] nsivabalan commented on a change in pull request #3590: [HUDI-2285][HUDI-2476] Metadata table synchronous design. Rebased and Squashed from pull/3426
Posted by GitBox <gi...@apache.org>.
nsivabalan commented on a change in pull request #3590:
URL: https://github.com/apache/hudi/pull/3590#discussion_r716249444
##########
File path: hudi-client/hudi-client-common/src/main/java/org/apache/hudi/metadata/HoodieBackedTableMetadataWriter.java
##########
@@ -401,64 +394,83 @@ private boolean bootstrapFromFilesystem(HoodieEngineContext engineContext, Hoodi
}
/**
- * Sync the Metadata Table from the instants created on the dataset.
+ * Initialize file groups for a partition. For file listing, we just have one file group.
*
- * @param datasetMetaClient {@code HoodieTableMetaClient} for the dataset
+ * All FileGroups for a given metadata partition has a fixed prefix as per the {@link MetadataPartitionType#getFileIdPrefix()}.
+ * Each file group is suffixed with increments of 1 starting with 1.
Review comment:
nope. no reason in particular. will change it to 0.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot edited a comment on pull request #3590: [HUDI-2285][HUDI-2476] Metadata table synchronous design. Rebased and Squashed from pull/3426
Posted by GitBox <gi...@apache.org>.
hudi-bot edited a comment on pull request #3590:
URL: https://github.com/apache/hudi/pull/3590#issuecomment-912237120
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "648e36e1af308534bc050c3a19da008ab7a8c840",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1994",
"triggerID" : "648e36e1af308534bc050c3a19da008ab7a8c840",
"triggerType" : "PUSH"
}, {
"hash" : "8c32253ad0f6d9eb728789fad75763bb5703678d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1995",
"triggerID" : "8c32253ad0f6d9eb728789fad75763bb5703678d",
"triggerType" : "PUSH"
}, {
"hash" : "71bbe59aa8bf70b9aa8990a6dd2522eb4bf1d172",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2009",
"triggerID" : "71bbe59aa8bf70b9aa8990a6dd2522eb4bf1d172",
"triggerType" : "PUSH"
}, {
"hash" : "aefac7ec2f2e40bdf3ad4365ea6aa825803a439d",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "aefac7ec2f2e40bdf3ad4365ea6aa825803a439d",
"triggerType" : "PUSH"
}, {
"hash" : "86905cb855fecfeb9e4c023d526ad643fadb8084",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2012",
"triggerID" : "86905cb855fecfeb9e4c023d526ad643fadb8084",
"triggerType" : "PUSH"
}, {
"hash" : "b4d93b0168775cabd43609fec59bf2823dd81279",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2029",
"triggerID" : "b4d93b0168775cabd43609fec59bf2823dd81279",
"triggerType" : "PUSH"
}, {
"hash" : "38bf430ecb4e11839d686929be2511eed340e9df",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2035",
"triggerID" : "38bf430ecb4e11839d686929be2511eed340e9df",
"triggerType" : "PUSH"
}, {
"hash" : "7793fbdb9b93a129ef606cb2d73ea6e1e9074957",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2299",
"triggerID" : "7793fbdb9b93a129ef606cb2d73ea6e1e9074957",
"triggerType" : "PUSH"
}, {
"hash" : "9c0123c0f27f990d009b323bab75b76ceecf3dab",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2307",
"triggerID" : "9c0123c0f27f990d009b323bab75b76ceecf3dab",
"triggerType" : "PUSH"
}, {
"hash" : "af27976ba07afb02305c699aeb7a75a122013b62",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2330",
"triggerID" : "af27976ba07afb02305c699aeb7a75a122013b62",
"triggerType" : "PUSH"
}, {
"hash" : "9172241c58424cca8e8186d4409a8c36968a4790",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2344",
"triggerID" : "9172241c58424cca8e8186d4409a8c36968a4790",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* aefac7ec2f2e40bdf3ad4365ea6aa825803a439d UNKNOWN
* 9172241c58424cca8e8186d4409a8c36968a4790 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2344)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run travis` re-run the last Travis build
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] nsivabalan commented on a change in pull request #3590: [HUDI-2285][HUDI-2476] Metadata table synchronous design. Rebased and Squashed from pull/3426
Posted by GitBox <gi...@apache.org>.
nsivabalan commented on a change in pull request #3590:
URL: https://github.com/apache/hudi/pull/3590#discussion_r716253305
##########
File path: hudi-client/hudi-spark-client/src/main/java/org/apache/hudi/table/HoodieSparkTable.java
##########
@@ -66,4 +77,34 @@ protected HoodieSparkTable(HoodieWriteConfig config, HoodieEngineContext context
protected HoodieIndex<T, JavaRDD<HoodieRecord<T>>, JavaRDD<HoodieKey>, JavaRDD<WriteStatus>> getIndex(HoodieWriteConfig config, HoodieEngineContext context) {
return SparkHoodieIndex.createIndex(config);
}
+
+ /**
+ * Fetch instance of {@link HoodieTableMetadataWriter}.
+ *
+ * @return instance of {@link HoodieTableMetadataWriter}
+ */
+ @Override
+ public Option<HoodieTableMetadataWriter> getMetadataWriter() {
+ synchronized (this) {
+ if (!isMetadataInfoUpdated.getAndSet(true)) {
+ // this code assumes that bootstrap of metadata table is done elsewhere (SparkRDDWriteClient) and so, we gauge whether metadata table
+ // is available or not once here and reuse the same info for repeated calls to getMetadataWriter(). Please remove memoization if
+ // that's not the case.
+ try {
+ if (!config.isMetadataTableEnabled() || !metaClient.getFs().exists(new Path(HoodieTableMetadata.getMetadataTableBasePath(metaClient.getBasePath())))) {
Review comment:
:) should have caught this myself.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot edited a comment on pull request #3590: [HUDI-2285][HUDI-2476] Metadata table synchronous design. Rebased and Squashed from pull/3426
Posted by GitBox <gi...@apache.org>.
hudi-bot edited a comment on pull request #3590:
URL: https://github.com/apache/hudi/pull/3590#issuecomment-912237120
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "648e36e1af308534bc050c3a19da008ab7a8c840",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1994",
"triggerID" : "648e36e1af308534bc050c3a19da008ab7a8c840",
"triggerType" : "PUSH"
}, {
"hash" : "8c32253ad0f6d9eb728789fad75763bb5703678d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1995",
"triggerID" : "8c32253ad0f6d9eb728789fad75763bb5703678d",
"triggerType" : "PUSH"
}, {
"hash" : "71bbe59aa8bf70b9aa8990a6dd2522eb4bf1d172",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2009",
"triggerID" : "71bbe59aa8bf70b9aa8990a6dd2522eb4bf1d172",
"triggerType" : "PUSH"
}, {
"hash" : "aefac7ec2f2e40bdf3ad4365ea6aa825803a439d",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "aefac7ec2f2e40bdf3ad4365ea6aa825803a439d",
"triggerType" : "PUSH"
}, {
"hash" : "86905cb855fecfeb9e4c023d526ad643fadb8084",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2012",
"triggerID" : "86905cb855fecfeb9e4c023d526ad643fadb8084",
"triggerType" : "PUSH"
}, {
"hash" : "b4d93b0168775cabd43609fec59bf2823dd81279",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2029",
"triggerID" : "b4d93b0168775cabd43609fec59bf2823dd81279",
"triggerType" : "PUSH"
}, {
"hash" : "38bf430ecb4e11839d686929be2511eed340e9df",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2035",
"triggerID" : "38bf430ecb4e11839d686929be2511eed340e9df",
"triggerType" : "PUSH"
}, {
"hash" : "7793fbdb9b93a129ef606cb2d73ea6e1e9074957",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2299",
"triggerID" : "7793fbdb9b93a129ef606cb2d73ea6e1e9074957",
"triggerType" : "PUSH"
}, {
"hash" : "9c0123c0f27f990d009b323bab75b76ceecf3dab",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2307",
"triggerID" : "9c0123c0f27f990d009b323bab75b76ceecf3dab",
"triggerType" : "PUSH"
}, {
"hash" : "af27976ba07afb02305c699aeb7a75a122013b62",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2330",
"triggerID" : "af27976ba07afb02305c699aeb7a75a122013b62",
"triggerType" : "PUSH"
}, {
"hash" : "9172241c58424cca8e8186d4409a8c36968a4790",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2344",
"triggerID" : "9172241c58424cca8e8186d4409a8c36968a4790",
"triggerType" : "PUSH"
}, {
"hash" : "13b80067ae00a760c57d184f70e2d98f63027e80",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2353",
"triggerID" : "13b80067ae00a760c57d184f70e2d98f63027e80",
"triggerType" : "PUSH"
}, {
"hash" : "ef750979cf449ff504da194e594cb62302eba360",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "ef750979cf449ff504da194e594cb62302eba360",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* aefac7ec2f2e40bdf3ad4365ea6aa825803a439d UNKNOWN
* 13b80067ae00a760c57d184f70e2d98f63027e80 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2353)
* ef750979cf449ff504da194e594cb62302eba360 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run travis` re-run the last Travis build
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot edited a comment on pull request #3590: [HUDI-2285][HUDI-2476] Metadata table synchronous design. Rebased and Squashed from pull/3426
Posted by GitBox <gi...@apache.org>.
hudi-bot edited a comment on pull request #3590:
URL: https://github.com/apache/hudi/pull/3590#issuecomment-912237120
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "648e36e1af308534bc050c3a19da008ab7a8c840",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1994",
"triggerID" : "648e36e1af308534bc050c3a19da008ab7a8c840",
"triggerType" : "PUSH"
}, {
"hash" : "8c32253ad0f6d9eb728789fad75763bb5703678d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1995",
"triggerID" : "8c32253ad0f6d9eb728789fad75763bb5703678d",
"triggerType" : "PUSH"
}, {
"hash" : "71bbe59aa8bf70b9aa8990a6dd2522eb4bf1d172",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2009",
"triggerID" : "71bbe59aa8bf70b9aa8990a6dd2522eb4bf1d172",
"triggerType" : "PUSH"
}, {
"hash" : "aefac7ec2f2e40bdf3ad4365ea6aa825803a439d",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "aefac7ec2f2e40bdf3ad4365ea6aa825803a439d",
"triggerType" : "PUSH"
}, {
"hash" : "86905cb855fecfeb9e4c023d526ad643fadb8084",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2012",
"triggerID" : "86905cb855fecfeb9e4c023d526ad643fadb8084",
"triggerType" : "PUSH"
}, {
"hash" : "b4d93b0168775cabd43609fec59bf2823dd81279",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2029",
"triggerID" : "b4d93b0168775cabd43609fec59bf2823dd81279",
"triggerType" : "PUSH"
}, {
"hash" : "38bf430ecb4e11839d686929be2511eed340e9df",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2035",
"triggerID" : "38bf430ecb4e11839d686929be2511eed340e9df",
"triggerType" : "PUSH"
}, {
"hash" : "7793fbdb9b93a129ef606cb2d73ea6e1e9074957",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2299",
"triggerID" : "7793fbdb9b93a129ef606cb2d73ea6e1e9074957",
"triggerType" : "PUSH"
}, {
"hash" : "9c0123c0f27f990d009b323bab75b76ceecf3dab",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2307",
"triggerID" : "9c0123c0f27f990d009b323bab75b76ceecf3dab",
"triggerType" : "PUSH"
}, {
"hash" : "af27976ba07afb02305c699aeb7a75a122013b62",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2330",
"triggerID" : "af27976ba07afb02305c699aeb7a75a122013b62",
"triggerType" : "PUSH"
}, {
"hash" : "9172241c58424cca8e8186d4409a8c36968a4790",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2344",
"triggerID" : "9172241c58424cca8e8186d4409a8c36968a4790",
"triggerType" : "PUSH"
}, {
"hash" : "13b80067ae00a760c57d184f70e2d98f63027e80",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2353",
"triggerID" : "13b80067ae00a760c57d184f70e2d98f63027e80",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* aefac7ec2f2e40bdf3ad4365ea6aa825803a439d UNKNOWN
* 13b80067ae00a760c57d184f70e2d98f63027e80 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2353)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run travis` re-run the last Travis build
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot edited a comment on pull request #3590: [HUDI-2285][HUDI-2476] Metadata table synchronous design. Rebased and Squashed from pull/3426
Posted by GitBox <gi...@apache.org>.
hudi-bot edited a comment on pull request #3590:
URL: https://github.com/apache/hudi/pull/3590#issuecomment-912237120
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot edited a comment on pull request #3590: [HUDI-2285][HUDI-2476] Metadata table synchronous design. Rebased and Squashed from pull/3426
Posted by GitBox <gi...@apache.org>.
hudi-bot edited a comment on pull request #3590:
URL: https://github.com/apache/hudi/pull/3590#issuecomment-912237120
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "648e36e1af308534bc050c3a19da008ab7a8c840",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1994",
"triggerID" : "648e36e1af308534bc050c3a19da008ab7a8c840",
"triggerType" : "PUSH"
}, {
"hash" : "8c32253ad0f6d9eb728789fad75763bb5703678d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1995",
"triggerID" : "8c32253ad0f6d9eb728789fad75763bb5703678d",
"triggerType" : "PUSH"
}, {
"hash" : "71bbe59aa8bf70b9aa8990a6dd2522eb4bf1d172",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2009",
"triggerID" : "71bbe59aa8bf70b9aa8990a6dd2522eb4bf1d172",
"triggerType" : "PUSH"
}, {
"hash" : "aefac7ec2f2e40bdf3ad4365ea6aa825803a439d",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "aefac7ec2f2e40bdf3ad4365ea6aa825803a439d",
"triggerType" : "PUSH"
}, {
"hash" : "86905cb855fecfeb9e4c023d526ad643fadb8084",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2012",
"triggerID" : "86905cb855fecfeb9e4c023d526ad643fadb8084",
"triggerType" : "PUSH"
}, {
"hash" : "b4d93b0168775cabd43609fec59bf2823dd81279",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2029",
"triggerID" : "b4d93b0168775cabd43609fec59bf2823dd81279",
"triggerType" : "PUSH"
}, {
"hash" : "38bf430ecb4e11839d686929be2511eed340e9df",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2035",
"triggerID" : "38bf430ecb4e11839d686929be2511eed340e9df",
"triggerType" : "PUSH"
}, {
"hash" : "7793fbdb9b93a129ef606cb2d73ea6e1e9074957",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2299",
"triggerID" : "7793fbdb9b93a129ef606cb2d73ea6e1e9074957",
"triggerType" : "PUSH"
}, {
"hash" : "9c0123c0f27f990d009b323bab75b76ceecf3dab",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2307",
"triggerID" : "9c0123c0f27f990d009b323bab75b76ceecf3dab",
"triggerType" : "PUSH"
}, {
"hash" : "af27976ba07afb02305c699aeb7a75a122013b62",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2330",
"triggerID" : "af27976ba07afb02305c699aeb7a75a122013b62",
"triggerType" : "PUSH"
}, {
"hash" : "9172241c58424cca8e8186d4409a8c36968a4790",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2344",
"triggerID" : "9172241c58424cca8e8186d4409a8c36968a4790",
"triggerType" : "PUSH"
}, {
"hash" : "13b80067ae00a760c57d184f70e2d98f63027e80",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2353",
"triggerID" : "13b80067ae00a760c57d184f70e2d98f63027e80",
"triggerType" : "PUSH"
}, {
"hash" : "ef750979cf449ff504da194e594cb62302eba360",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2366",
"triggerID" : "ef750979cf449ff504da194e594cb62302eba360",
"triggerType" : "PUSH"
}, {
"hash" : "16827c37f36d3b9fec18e0a35aff7599c45e0444",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2375",
"triggerID" : "16827c37f36d3b9fec18e0a35aff7599c45e0444",
"triggerType" : "PUSH"
}, {
"hash" : "0f68d426b01355537f5124b8f108345a36d3da39",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2378",
"triggerID" : "0f68d426b01355537f5124b8f108345a36d3da39",
"triggerType" : "PUSH"
}, {
"hash" : "2f3099197889fd5c48ae3b550cc523e0528ddb2c",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2380",
"triggerID" : "2f3099197889fd5c48ae3b550cc523e0528ddb2c",
"triggerType" : "PUSH"
}, {
"hash" : "c83d176e1d1ddab6bb86f91726060779a9a1519f",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2392",
"triggerID" : "c83d176e1d1ddab6bb86f91726060779a9a1519f",
"triggerType" : "PUSH"
}, {
"hash" : "008f0cd91de12d44030d837f460f3b0232d62ee5",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2416",
"triggerID" : "008f0cd91de12d44030d837f460f3b0232d62ee5",
"triggerType" : "PUSH"
}, {
"hash" : "92634fb730e723a9bdeb165348a4c747794be7e3",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "92634fb730e723a9bdeb165348a4c747794be7e3",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* aefac7ec2f2e40bdf3ad4365ea6aa825803a439d UNKNOWN
* 008f0cd91de12d44030d837f460f3b0232d62ee5 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2416)
* 92634fb730e723a9bdeb165348a4c747794be7e3 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run travis` re-run the last Travis build
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] vinothchandar commented on a change in pull request #3590: [HUDI-2285][HUDI-2476] Metadata table synchronous design. Rebased and Squashed from pull/3426
Posted by GitBox <gi...@apache.org>.
vinothchandar commented on a change in pull request #3590:
URL: https://github.com/apache/hudi/pull/3590#discussion_r716227697
##########
File path: hudi-client/hudi-client-common/src/main/java/org/apache/hudi/metadata/HoodieBackedTableMetadataWriter.java
##########
@@ -88,6 +91,7 @@
protected HoodieBackedTableMetadata metadata;
protected HoodieTableMetaClient metaClient;
+ protected HoodieTableMetaClient datasetMetaClient;
Review comment:
rename: dataMetaClient
##########
File path: hudi-client/hudi-flink-client/src/main/java/org/apache/hudi/client/HoodieFlinkWriteClient.java
##########
@@ -121,8 +118,13 @@ public boolean commit(String instantTime, List<WriteStatus> writeStatuses, Optio
}
@Override
- protected HoodieTable<T, List<HoodieRecord<T>>, List<HoodieKey>, List<WriteStatus>> createTable(HoodieWriteConfig config, Configuration hadoopConf) {
- return HoodieFlinkTable.create(config, (HoodieFlinkEngineContext) context);
+ protected HoodieTable<T, List<HoodieRecord<T>>, List<HoodieKey>, List<WriteStatus>> createTable(HoodieWriteConfig config, Configuration hadoopConf,
Review comment:
@danny0405 @yanghua @leesf can one of you please review the flink integration pieces and make a suggestion for how we can make Flink write out to metadata table seamlessly as well. It could happen in a different PR. but like to treat Spark and Flink (eventually Java) as same going forward. i.e there large features are done hand-in-hand
##########
File path: hudi-client/hudi-client-common/src/main/java/org/apache/hudi/metadata/HoodieBackedTableMetadataWriter.java
##########
@@ -401,64 +394,83 @@ private boolean bootstrapFromFilesystem(HoodieEngineContext engineContext, Hoodi
}
/**
- * Sync the Metadata Table from the instants created on the dataset.
+ * Initialize file groups for a partition. For file listing, we just have one file group.
*
- * @param datasetMetaClient {@code HoodieTableMetaClient} for the dataset
+ * All FileGroups for a given metadata partition has a fixed prefix as per the {@link MetadataPartitionType#getFileIdPrefix()}.
+ * Each file group is suffixed with increments of 1 starting with 1.
Review comment:
why not start with 0? do you want specifically `1` for some reason
##########
File path: hudi-client/hudi-client-common/src/main/java/org/apache/hudi/metadata/HoodieBackedTableMetadataWriter.java
##########
@@ -401,64 +394,83 @@ private boolean bootstrapFromFilesystem(HoodieEngineContext engineContext, Hoodi
}
/**
- * Sync the Metadata Table from the instants created on the dataset.
+ * Initialize file groups for a partition. For file listing, we just have one file group.
*
- * @param datasetMetaClient {@code HoodieTableMetaClient} for the dataset
+ * All FileGroups for a given metadata partition has a fixed prefix as per the {@link MetadataPartitionType#getFileIdPrefix()}.
+ * Each file group is suffixed with increments of 1 starting with 1.
+ *
+ * For instance, for FILES, there is only one file group named as "files-1"
+ * Lets say we configure 10 file groups for record level index, and prefix as "record-index-bucket-"
+ * Filegroups will be named as :
+ * record-index-bucket-01
+ * record-index-bucket-02
+ * ...
+ * record-index-bucket-10
*/
- private void syncFromInstants(HoodieTableMetaClient datasetMetaClient) {
- ValidationUtils.checkState(enabled, "Metadata table cannot be synced as it is not enabled");
- // (re) init the metadata for reading.
- initTableMetadata();
- try {
- List<HoodieInstant> instantsToSync = metadata.findInstantsToSyncForWriter();
- if (instantsToSync.isEmpty()) {
- return;
- }
-
- LOG.info("Syncing " + instantsToSync.size() + " instants to metadata table: " + instantsToSync);
-
- // Read each instant in order and sync it to metadata table
- for (HoodieInstant instant : instantsToSync) {
- LOG.info("Syncing instant " + instant + " to metadata table");
-
- Option<List<HoodieRecord>> records = HoodieTableMetadataUtil.convertInstantToMetaRecords(datasetMetaClient,
- metaClient.getActiveTimeline(), instant, metadata.getUpdateTime());
- if (records.isPresent()) {
- commit(records.get(), MetadataPartitionType.FILES.partitionPath(), instant.getTimestamp());
- }
+ private void initializeFileGroups(HoodieTableMetaClient datasetMetaClient, MetadataPartitionType metadataPartition, String instantTime,
+ int fileGroupCount) throws IOException {
+
+ final HashMap<HeaderMetadataType, String> blockHeader = new HashMap<>();
+ blockHeader.put(HeaderMetadataType.INSTANT_TIME, instantTime);
+ // Archival of data table has a dependency on compaction(base files) in metadata table.
+ // It is assumed that as of time Tx of base instant (/compaction time) in metadata table,
+ // all commits in data table is in sync with metadata table. So, we always create start with log file for any fileGroup.
+ final HoodieDeleteBlock block = new HoodieDeleteBlock(new HoodieKey[0], blockHeader);
Review comment:
this is pretty hacky. :D
##########
File path: hudi-client/hudi-spark-client/src/main/java/org/apache/hudi/client/SparkRDDWriteClient.java
##########
@@ -389,7 +401,19 @@ private void completeClustering(HoodieReplaceCommitMetadata metadata, JavaRDD<Wr
+ config.getBasePath() + " at time " + clusteringCommitTime, e);
}
}
- LOG.info("Clustering successfully on commit " + clusteringCommitTime);
+ LOG.warn("Clustering successfully on commit " + clusteringCommitTime);
+ }
+
+ private void writeTableMetadata(HoodieTable<T, JavaRDD<HoodieRecord<T>>, JavaRDD<HoodieKey>, JavaRDD<WriteStatus>> table, HoodieCommitMetadata commitMetadata, String commitTime) {
+ try {
+ this.txnManager.beginTransaction(Option.of(new HoodieInstant(HoodieInstant.State.INFLIGHT, HoodieTimeline.REPLACE_COMMIT_ACTION, commitTime)),
Review comment:
this method name/params don't indicate it working on clustering at all.
##########
File path: hudi-client/hudi-spark-client/src/main/java/org/apache/hudi/client/SparkRDDWriteClient.java
##########
@@ -96,6 +94,11 @@ public SparkRDDWriteClient(HoodieEngineContext context, HoodieWriteConfig writeC
public SparkRDDWriteClient(HoodieEngineContext context, HoodieWriteConfig writeConfig,
Option<EmbeddedTimelineService> timelineService) {
super(context, writeConfig, timelineService);
+ if (config.isMetadataTableEnabled()) {
+ // If the metadata table does not exist, it should be bootstrapped here
Review comment:
is this bootstrap multi writer safe?
two scenarios to consider
a) two writers (i.e delta sync and a spark job) trying to do this.
b) Just a single writer - but with async compaction/clustering (who would also create write clients).
I am not sure if metadata being enabled is the only flag to check here.
##########
File path: hudi-client/hudi-spark-client/src/main/java/org/apache/hudi/client/SparkRDDWriteClient.java
##########
@@ -314,12 +323,13 @@ protected void completeCompaction(HoodieCommitMetadata metadata, JavaRDD<WriteSt
+ config.getBasePath() + " at time " + compactionCommitTime, e);
}
}
- LOG.info("Compacted successfully on commit " + compactionCommitTime);
+ LOG.warn("Compacted successfully on commit " + compactionCommitTime);
Review comment:
why would this be a warn?
##########
File path: hudi-client/hudi-spark-client/src/main/java/org/apache/hudi/table/HoodieSparkTable.java
##########
@@ -66,4 +77,34 @@ protected HoodieSparkTable(HoodieWriteConfig config, HoodieEngineContext context
protected HoodieIndex<T, JavaRDD<HoodieRecord<T>>, JavaRDD<HoodieKey>, JavaRDD<WriteStatus>> getIndex(HoodieWriteConfig config, HoodieEngineContext context) {
return SparkHoodieIndex.createIndex(config);
}
+
+ /**
+ * Fetch instance of {@link HoodieTableMetadataWriter}.
+ *
+ * @return instance of {@link HoodieTableMetadataWriter}
+ */
+ @Override
+ public Option<HoodieTableMetadataWriter> getMetadataWriter() {
+ synchronized (this) {
+ if (!isMetadataInfoUpdated.getAndSet(true)) {
+ // this code assumes that bootstrap of metadata table is done elsewhere (SparkRDDWriteClient) and so, we gauge whether metadata table
+ // is available or not once here and reuse the same info for repeated calls to getMetadataWriter(). Please remove memoization if
+ // that's not the case.
+ try {
+ if (!config.isMetadataTableEnabled() || !metaClient.getFs().exists(new Path(HoodieTableMetadata.getMetadataTableBasePath(metaClient.getBasePath())))) {
+ isMetadataTableAvailable = false;
+ } else {
+ isMetadataTableAvailable = true;
+ }
+ } catch (IOException e) {
+ throw new HoodieMetadataException("Could not instantiate metadata table writer", e);
+ }
+ }
+ }
+ if (isMetadataTableAvailable) {
Review comment:
something without the if . (general code style issue, that I wonder a lot abt to get rid of these if blocks with option).
##########
File path: hudi-client/hudi-spark-client/src/main/java/org/apache/hudi/metadata/SparkHoodieBackedTableMetadataWriter.java
##########
@@ -99,83 +94,94 @@ protected void initialize(HoodieEngineContext engineContext, HoodieTableMetaClie
@Override
protected void commit(List<HoodieRecord> records, String partitionName, String instantTime) {
ValidationUtils.checkState(enabled, "Metadata table cannot be committed to as it is not enabled");
- JavaRDD<HoodieRecord> recordRDD = prepRecords(records, partitionName);
+ JavaRDD<HoodieRecord> recordRDD = prepRecords(records, partitionName, 1);
try (SparkRDDWriteClient writeClient = new SparkRDDWriteClient(engineContext, metadataWriteConfig, true)) {
- writeClient.startCommitWithTime(instantTime);
+ if (!metaClient.getActiveTimeline().filterCompletedInstants().containsInstant(instantTime)) {
+ // if this is a new commit being applied to metadata for the first time
+ writeClient.startCommitWithTime(instantTime);
+ } else {
+ // this code path refers to a re-attempted commit that got committed to metadata, but failed in dataset.
+ // for eg, lets say compaction c1 on 1st attempt succeeded in metadata table and failed before committing to datatable.
+ // when retried again, data table will first rollback pending compaction. these will be applied to metadata table, but all changes
+ // are upserts to metadata table and so only a new delta commit will be created.
+ // once rollback is complete, compaction will be retried again, which will eventually hit this code block where the respective commit is
+ // already part of completed commit. So, we have to manually remove the completed instant and proceed.
+ // and it is for the same reason we enabled withAllowMultiWriteOnSameInstant for metadata table.
+ HoodieInstant alreadyCompletedInstant = metaClient.getActiveTimeline().filterCompletedInstants().filter(entry -> entry.getTimestamp().equals(instantTime)).lastInstant().get();
+ FSUtils.deleteInstantFile(metaClient.getFs(), metaClient.getMetaPath(), alreadyCompletedInstant);
+ metaClient.reloadActiveTimeline();
+ }
List<WriteStatus> statuses = writeClient.upsertPreppedRecords(recordRDD, instantTime).collect();
statuses.forEach(writeStatus -> {
if (writeStatus.hasErrors()) {
throw new HoodieMetadataException("Failed to commit metadata table records at instant " + instantTime);
}
});
- // trigger cleaning, compaction, with suffixes based on the same instant time. This ensures that any future
- // delta commits synced over will not have an instant time lesser than the last completed instant on the
- // metadata table.
- if (writeClient.scheduleCompactionAtInstant(instantTime + "001", Option.empty())) {
- writeClient.compact(instantTime + "001");
- }
- writeClient.clean(instantTime + "002");
+
+ // reload timeline
+ metaClient.reloadActiveTimeline();
+
+ compactIfNecessary(writeClient, instantTime);
+ doClean(writeClient, instantTime);
}
// Update total size of the metadata and count of base/log files
- metrics.ifPresent(m -> {
- try {
- Map<String, String> stats = m.getStats(false, metaClient, metadata);
- m.updateMetrics(Long.parseLong(stats.get(HoodieMetadataMetrics.STAT_TOTAL_BASE_FILE_SIZE)),
- Long.parseLong(stats.get(HoodieMetadataMetrics.STAT_TOTAL_LOG_FILE_SIZE)),
- Integer.parseInt(stats.get(HoodieMetadataMetrics.STAT_COUNT_BASE_FILES)),
- Integer.parseInt(stats.get(HoodieMetadataMetrics.STAT_COUNT_LOG_FILES)));
- } catch (HoodieIOException e) {
- LOG.error("Could not publish metadata size metrics", e);
- }
- });
+ metrics.ifPresent(m -> m.updateSizeMetrics(metaClient, metadata));
}
/**
- * Tag each record with the location.
+ * Perform a compaction on the Metadata Table.
+ *
+ * Cases to be handled:
+ * 1. We cannot perform compaction if there are previous inflight operations on the dataset. This is because
+ * a compacted metadata base file at time Tx should represent all the actions on the dataset till time Tx.
*
- * Since we only read the latest base file in a partition, we tag the records with the instant time of the latest
- * base file.
+ * 2. In multi-writer scenario, a parallel operation with a greater instantTime may have completed creating a
+ * deltacommit.
*/
- private JavaRDD<HoodieRecord> prepRecords(List<HoodieRecord> records, String partitionName) {
- HoodieTable table = HoodieSparkTable.create(metadataWriteConfig, engineContext);
- TableFileSystemView.SliceView fsView = table.getSliceView();
- List<HoodieBaseFile> baseFiles = fsView.getLatestFileSlices(partitionName)
- .map(FileSlice::getBaseFile)
- .filter(Option::isPresent)
- .map(Option::get)
- .collect(Collectors.toList());
-
- // All the metadata fits within a single base file
- if (partitionName.equals(MetadataPartitionType.FILES.partitionPath())) {
- if (baseFiles.size() > 1) {
- throw new HoodieMetadataException("Multiple base files found in metadata partition");
- }
+ private void compactIfNecessary(SparkRDDWriteClient writeClient, String instantTime) {
+ String latestDeltacommitTime = metaClient.getActiveTimeline().getDeltaCommitTimeline().filterCompletedInstants().lastInstant()
+ .get().getTimestamp();
+ List<HoodieInstant> pendingInstants = datasetMetaClient.reloadActiveTimeline().filterInflightsAndRequested()
+ .findInstantsBefore(latestDeltacommitTime).getInstants().collect(Collectors.toList());
+
+ if (!pendingInstants.isEmpty()) {
+ LOG.info(String.format("Cannot compact metadata table as there are %d inflight instants before latest deltacommit %s: %s",
+ pendingInstants.size(), latestDeltacommitTime, Arrays.toString(pendingInstants.toArray())));
+ return;
}
- JavaSparkContext jsc = ((HoodieSparkEngineContext) engineContext).getJavaSparkContext();
- String fileId;
- String instantTime;
- if (!baseFiles.isEmpty()) {
- fileId = baseFiles.get(0).getFileId();
- instantTime = baseFiles.get(0).getCommitTime();
- } else {
- // If there is a log file then we can assume that it has the data
- List<HoodieLogFile> logFiles = fsView.getLatestFileSlices(MetadataPartitionType.FILES.partitionPath())
- .map(FileSlice::getLatestLogFile)
- .filter(Option::isPresent)
- .map(Option::get)
- .collect(Collectors.toList());
- if (logFiles.isEmpty()) {
- // No base and log files. All are new inserts
- return jsc.parallelize(records, 1);
- }
-
- fileId = logFiles.get(0).getFileId();
- instantTime = logFiles.get(0).getBaseCommitTime();
+ // Trigger compaction with suffixes based on the same instant time. This ensures that any future
+ // delta commits synced over will not have an instant time lesser than the last completed instant on the
+ // metadata table.
+ final String compactionInstantTime = latestDeltacommitTime + "001";
+ if (writeClient.scheduleCompactionAtInstant(compactionInstantTime, Option.empty())) {
Review comment:
metadataWriteClient?
##########
File path: hudi-client/hudi-spark-client/src/main/java/org/apache/hudi/client/SparkRDDWriteClient.java
##########
@@ -314,12 +323,13 @@ protected void completeCompaction(HoodieCommitMetadata metadata, JavaRDD<WriteSt
+ config.getBasePath() + " at time " + compactionCommitTime, e);
}
}
- LOG.info("Compacted successfully on commit " + compactionCommitTime);
+ LOG.warn("Compacted successfully on commit " + compactionCommitTime);
}
@Override
protected JavaRDD<WriteStatus> compact(String compactionInstantTime, boolean shouldComplete) {
HoodieSparkTable<T> table = HoodieSparkTable.create(config, context);
+ table.getHoodieView().sync();
Review comment:
I would like to avoid these `sync()` calls being added everywhere. So far, the writeClient gets "synced" when its created and that's that. Every preWrite sync-ing is more prediciatble, so down with that. but these kind of calls added to each API, could make our job difficult in maintaining.
Also why would this be not done at the super class level? for Flink, java as well?
##########
File path: hudi-client/hudi-spark-client/src/main/java/org/apache/hudi/table/HoodieSparkTable.java
##########
@@ -66,4 +77,34 @@ protected HoodieSparkTable(HoodieWriteConfig config, HoodieEngineContext context
protected HoodieIndex<T, JavaRDD<HoodieRecord<T>>, JavaRDD<HoodieKey>, JavaRDD<WriteStatus>> getIndex(HoodieWriteConfig config, HoodieEngineContext context) {
return SparkHoodieIndex.createIndex(config);
}
+
+ /**
+ * Fetch instance of {@link HoodieTableMetadataWriter}.
+ *
+ * @return instance of {@link HoodieTableMetadataWriter}
+ */
+ @Override
+ public Option<HoodieTableMetadataWriter> getMetadataWriter() {
+ synchronized (this) {
+ if (!isMetadataInfoUpdated.getAndSet(true)) {
+ // this code assumes that bootstrap of metadata table is done elsewhere (SparkRDDWriteClient) and so, we gauge whether metadata table
+ // is available or not once here and reuse the same info for repeated calls to getMetadataWriter(). Please remove memoization if
+ // that's not the case.
+ try {
+ if (!config.isMetadataTableEnabled() || !metaClient.getFs().exists(new Path(HoodieTableMetadata.getMetadataTableBasePath(metaClient.getBasePath())))) {
Review comment:
`isMetadataTableAvailable = config.isMetadataTableEnabled() && metaClient.getFs().exists(new Path(HoodieTableMetadata.getMetadataTableBasePath(metaClient.getBasePath())` ?
##########
File path: hudi-client/hudi-client-common/src/main/java/org/apache/hudi/metadata/HoodieBackedTableMetadataWriter.java
##########
@@ -401,64 +394,83 @@ private boolean bootstrapFromFilesystem(HoodieEngineContext engineContext, Hoodi
}
/**
- * Sync the Metadata Table from the instants created on the dataset.
+ * Initialize file groups for a partition. For file listing, we just have one file group.
*
- * @param datasetMetaClient {@code HoodieTableMetaClient} for the dataset
+ * All FileGroups for a given metadata partition has a fixed prefix as per the {@link MetadataPartitionType#getFileIdPrefix()}.
+ * Each file group is suffixed with increments of 1 starting with 1.
+ *
+ * For instance, for FILES, there is only one file group named as "files-1"
+ * Lets say we configure 10 file groups for record level index, and prefix as "record-index-bucket-"
Review comment:
it may be okay to pad upto 3 zeros for these file groups by default during naming. keep them all consistent in most cases. leave it to you.
##########
File path: hudi-client/hudi-spark-client/src/main/java/org/apache/hudi/metadata/SparkHoodieBackedTableMetadataWriter.java
##########
@@ -99,83 +94,94 @@ protected void initialize(HoodieEngineContext engineContext, HoodieTableMetaClie
@Override
protected void commit(List<HoodieRecord> records, String partitionName, String instantTime) {
ValidationUtils.checkState(enabled, "Metadata table cannot be committed to as it is not enabled");
- JavaRDD<HoodieRecord> recordRDD = prepRecords(records, partitionName);
+ JavaRDD<HoodieRecord> recordRDD = prepRecords(records, partitionName, 1);
try (SparkRDDWriteClient writeClient = new SparkRDDWriteClient(engineContext, metadataWriteConfig, true)) {
- writeClient.startCommitWithTime(instantTime);
+ if (!metaClient.getActiveTimeline().filterCompletedInstants().containsInstant(instantTime)) {
+ // if this is a new commit being applied to metadata for the first time
+ writeClient.startCommitWithTime(instantTime);
+ } else {
+ // this code path refers to a re-attempted commit that got committed to metadata, but failed in dataset.
+ // for eg, lets say compaction c1 on 1st attempt succeeded in metadata table and failed before committing to datatable.
+ // when retried again, data table will first rollback pending compaction. these will be applied to metadata table, but all changes
+ // are upserts to metadata table and so only a new delta commit will be created.
+ // once rollback is complete, compaction will be retried again, which will eventually hit this code block where the respective commit is
+ // already part of completed commit. So, we have to manually remove the completed instant and proceed.
+ // and it is for the same reason we enabled withAllowMultiWriteOnSameInstant for metadata table.
+ HoodieInstant alreadyCompletedInstant = metaClient.getActiveTimeline().filterCompletedInstants().filter(entry -> entry.getTimestamp().equals(instantTime)).lastInstant().get();
+ FSUtils.deleteInstantFile(metaClient.getFs(), metaClient.getMetaPath(), alreadyCompletedInstant);
+ metaClient.reloadActiveTimeline();
+ }
List<WriteStatus> statuses = writeClient.upsertPreppedRecords(recordRDD, instantTime).collect();
statuses.forEach(writeStatus -> {
if (writeStatus.hasErrors()) {
throw new HoodieMetadataException("Failed to commit metadata table records at instant " + instantTime);
}
});
- // trigger cleaning, compaction, with suffixes based on the same instant time. This ensures that any future
- // delta commits synced over will not have an instant time lesser than the last completed instant on the
- // metadata table.
- if (writeClient.scheduleCompactionAtInstant(instantTime + "001", Option.empty())) {
- writeClient.compact(instantTime + "001");
- }
- writeClient.clean(instantTime + "002");
+
+ // reload timeline
+ metaClient.reloadActiveTimeline();
+
+ compactIfNecessary(writeClient, instantTime);
+ doClean(writeClient, instantTime);
}
// Update total size of the metadata and count of base/log files
- metrics.ifPresent(m -> {
- try {
- Map<String, String> stats = m.getStats(false, metaClient, metadata);
- m.updateMetrics(Long.parseLong(stats.get(HoodieMetadataMetrics.STAT_TOTAL_BASE_FILE_SIZE)),
- Long.parseLong(stats.get(HoodieMetadataMetrics.STAT_TOTAL_LOG_FILE_SIZE)),
- Integer.parseInt(stats.get(HoodieMetadataMetrics.STAT_COUNT_BASE_FILES)),
- Integer.parseInt(stats.get(HoodieMetadataMetrics.STAT_COUNT_LOG_FILES)));
- } catch (HoodieIOException e) {
- LOG.error("Could not publish metadata size metrics", e);
- }
- });
+ metrics.ifPresent(m -> m.updateSizeMetrics(metaClient, metadata));
}
/**
- * Tag each record with the location.
+ * Perform a compaction on the Metadata Table.
+ *
+ * Cases to be handled:
+ * 1. We cannot perform compaction if there are previous inflight operations on the dataset. This is because
+ * a compacted metadata base file at time Tx should represent all the actions on the dataset till time Tx.
*
- * Since we only read the latest base file in a partition, we tag the records with the instant time of the latest
- * base file.
+ * 2. In multi-writer scenario, a parallel operation with a greater instantTime may have completed creating a
+ * deltacommit.
*/
- private JavaRDD<HoodieRecord> prepRecords(List<HoodieRecord> records, String partitionName) {
- HoodieTable table = HoodieSparkTable.create(metadataWriteConfig, engineContext);
- TableFileSystemView.SliceView fsView = table.getSliceView();
- List<HoodieBaseFile> baseFiles = fsView.getLatestFileSlices(partitionName)
- .map(FileSlice::getBaseFile)
- .filter(Option::isPresent)
- .map(Option::get)
- .collect(Collectors.toList());
-
- // All the metadata fits within a single base file
- if (partitionName.equals(MetadataPartitionType.FILES.partitionPath())) {
- if (baseFiles.size() > 1) {
- throw new HoodieMetadataException("Multiple base files found in metadata partition");
- }
+ private void compactIfNecessary(SparkRDDWriteClient writeClient, String instantTime) {
+ String latestDeltacommitTime = metaClient.getActiveTimeline().getDeltaCommitTimeline().filterCompletedInstants().lastInstant()
+ .get().getTimestamp();
+ List<HoodieInstant> pendingInstants = datasetMetaClient.reloadActiveTimeline().filterInflightsAndRequested()
Review comment:
lets review naming for clarity across the board. lets name everything using `data` and `metadata` prefixes.
`dataMetaClient` `metadataMetaClient`
`dataPendingInstants`
`metadataLatestDeltaCommitTime` and so forth.
##########
File path: hudi-client/hudi-spark-client/src/main/java/org/apache/hudi/table/HoodieSparkTable.java
##########
@@ -66,4 +77,34 @@ protected HoodieSparkTable(HoodieWriteConfig config, HoodieEngineContext context
protected HoodieIndex<T, JavaRDD<HoodieRecord<T>>, JavaRDD<HoodieKey>, JavaRDD<WriteStatus>> getIndex(HoodieWriteConfig config, HoodieEngineContext context) {
return SparkHoodieIndex.createIndex(config);
}
+
+ /**
+ * Fetch instance of {@link HoodieTableMetadataWriter}.
+ *
+ * @return instance of {@link HoodieTableMetadataWriter}
+ */
+ @Override
+ public Option<HoodieTableMetadataWriter> getMetadataWriter() {
+ synchronized (this) {
+ if (!isMetadataInfoUpdated.getAndSet(true)) {
+ // this code assumes that bootstrap of metadata table is done elsewhere (SparkRDDWriteClient) and so, we gauge whether metadata table
+ // is available or not once here and reuse the same info for repeated calls to getMetadataWriter(). Please remove memoization if
Review comment:
lets get rid of this commentary. :)
##########
File path: hudi-client/hudi-spark-client/src/main/java/org/apache/hudi/table/HoodieSparkTable.java
##########
@@ -66,4 +77,34 @@ protected HoodieSparkTable(HoodieWriteConfig config, HoodieEngineContext context
protected HoodieIndex<T, JavaRDD<HoodieRecord<T>>, JavaRDD<HoodieKey>, JavaRDD<WriteStatus>> getIndex(HoodieWriteConfig config, HoodieEngineContext context) {
return SparkHoodieIndex.createIndex(config);
}
+
+ /**
+ * Fetch instance of {@link HoodieTableMetadataWriter}.
+ *
+ * @return instance of {@link HoodieTableMetadataWriter}
+ */
+ @Override
+ public Option<HoodieTableMetadataWriter> getMetadataWriter() {
+ synchronized (this) {
+ if (!isMetadataInfoUpdated.getAndSet(true)) {
+ // this code assumes that bootstrap of metadata table is done elsewhere (SparkRDDWriteClient) and so, we gauge whether metadata table
+ // is available or not once here and reuse the same info for repeated calls to getMetadataWriter(). Please remove memoization if
+ // that's not the case.
+ try {
+ if (!config.isMetadataTableEnabled() || !metaClient.getFs().exists(new Path(HoodieTableMetadata.getMetadataTableBasePath(metaClient.getBasePath())))) {
+ isMetadataTableAvailable = false;
+ } else {
+ isMetadataTableAvailable = true;
+ }
+ } catch (IOException e) {
+ throw new HoodieMetadataException("Could not instantiate metadata table writer", e);
Review comment:
this error message is misleading. all we are doing here is checking for existence?
##########
File path: hudi-client/hudi-spark-client/src/main/java/org/apache/hudi/table/HoodieSparkTable.java
##########
@@ -66,4 +77,34 @@ protected HoodieSparkTable(HoodieWriteConfig config, HoodieEngineContext context
protected HoodieIndex<T, JavaRDD<HoodieRecord<T>>, JavaRDD<HoodieKey>, JavaRDD<WriteStatus>> getIndex(HoodieWriteConfig config, HoodieEngineContext context) {
return SparkHoodieIndex.createIndex(config);
}
+
+ /**
+ * Fetch instance of {@link HoodieTableMetadataWriter}.
+ *
+ * @return instance of {@link HoodieTableMetadataWriter}
+ */
+ @Override
+ public Option<HoodieTableMetadataWriter> getMetadataWriter() {
+ synchronized (this) {
+ if (!isMetadataInfoUpdated.getAndSet(true)) {
+ // this code assumes that bootstrap of metadata table is done elsewhere (SparkRDDWriteClient) and so, we gauge whether metadata table
+ // is available or not once here and reuse the same info for repeated calls to getMetadataWriter(). Please remove memoization if
+ // that's not the case.
+ try {
+ if (!config.isMetadataTableEnabled() || !metaClient.getFs().exists(new Path(HoodieTableMetadata.getMetadataTableBasePath(metaClient.getBasePath())))) {
+ isMetadataTableAvailable = false;
+ } else {
+ isMetadataTableAvailable = true;
+ }
+ } catch (IOException e) {
+ throw new HoodieMetadataException("Could not instantiate metadata table writer", e);
+ }
+ }
+ }
+ if (isMetadataTableAvailable) {
Review comment:
`Option.of(isMetadataTableAvailable).filter(p -> p).map(p -> SparkHoodieBackedTableMetadataWriter.create(context.getHadoopConf().get(), config, context))`?
##########
File path: hudi-common/src/main/java/org/apache/hudi/common/table/log/HoodieLogFormat.java
##########
@@ -56,6 +56,8 @@
String UNKNOWN_WRITE_TOKEN = "1-0-1";
+ String DEFAULT_WRITE_TOKEN_METADATA_PARTITION = "0-0-0";
Review comment:
Lets not please ever refer to some higher level functionality/stack i.e metadata in a lower level class. or member names/comments.
##########
File path: hudi-client/hudi-spark-client/src/main/java/org/apache/hudi/metadata/SparkHoodieBackedTableMetadataWriter.java
##########
@@ -99,83 +94,94 @@ protected void initialize(HoodieEngineContext engineContext, HoodieTableMetaClie
@Override
protected void commit(List<HoodieRecord> records, String partitionName, String instantTime) {
ValidationUtils.checkState(enabled, "Metadata table cannot be committed to as it is not enabled");
- JavaRDD<HoodieRecord> recordRDD = prepRecords(records, partitionName);
+ JavaRDD<HoodieRecord> recordRDD = prepRecords(records, partitionName, 1);
try (SparkRDDWriteClient writeClient = new SparkRDDWriteClient(engineContext, metadataWriteConfig, true)) {
- writeClient.startCommitWithTime(instantTime);
+ if (!metaClient.getActiveTimeline().filterCompletedInstants().containsInstant(instantTime)) {
+ // if this is a new commit being applied to metadata for the first time
+ writeClient.startCommitWithTime(instantTime);
+ } else {
+ // this code path refers to a re-attempted commit that got committed to metadata, but failed in dataset.
+ // for eg, lets say compaction c1 on 1st attempt succeeded in metadata table and failed before committing to datatable.
+ // when retried again, data table will first rollback pending compaction. these will be applied to metadata table, but all changes
+ // are upserts to metadata table and so only a new delta commit will be created.
+ // once rollback is complete, compaction will be retried again, which will eventually hit this code block where the respective commit is
+ // already part of completed commit. So, we have to manually remove the completed instant and proceed.
+ // and it is for the same reason we enabled withAllowMultiWriteOnSameInstant for metadata table.
+ HoodieInstant alreadyCompletedInstant = metaClient.getActiveTimeline().filterCompletedInstants().filter(entry -> entry.getTimestamp().equals(instantTime)).lastInstant().get();
+ FSUtils.deleteInstantFile(metaClient.getFs(), metaClient.getMetaPath(), alreadyCompletedInstant);
+ metaClient.reloadActiveTimeline();
+ }
List<WriteStatus> statuses = writeClient.upsertPreppedRecords(recordRDD, instantTime).collect();
statuses.forEach(writeStatus -> {
if (writeStatus.hasErrors()) {
throw new HoodieMetadataException("Failed to commit metadata table records at instant " + instantTime);
}
});
- // trigger cleaning, compaction, with suffixes based on the same instant time. This ensures that any future
- // delta commits synced over will not have an instant time lesser than the last completed instant on the
- // metadata table.
- if (writeClient.scheduleCompactionAtInstant(instantTime + "001", Option.empty())) {
- writeClient.compact(instantTime + "001");
- }
- writeClient.clean(instantTime + "002");
+
+ // reload timeline
+ metaClient.reloadActiveTimeline();
+
+ compactIfNecessary(writeClient, instantTime);
+ doClean(writeClient, instantTime);
}
// Update total size of the metadata and count of base/log files
- metrics.ifPresent(m -> {
- try {
- Map<String, String> stats = m.getStats(false, metaClient, metadata);
- m.updateMetrics(Long.parseLong(stats.get(HoodieMetadataMetrics.STAT_TOTAL_BASE_FILE_SIZE)),
- Long.parseLong(stats.get(HoodieMetadataMetrics.STAT_TOTAL_LOG_FILE_SIZE)),
- Integer.parseInt(stats.get(HoodieMetadataMetrics.STAT_COUNT_BASE_FILES)),
- Integer.parseInt(stats.get(HoodieMetadataMetrics.STAT_COUNT_LOG_FILES)));
- } catch (HoodieIOException e) {
- LOG.error("Could not publish metadata size metrics", e);
- }
- });
+ metrics.ifPresent(m -> m.updateSizeMetrics(metaClient, metadata));
}
/**
- * Tag each record with the location.
+ * Perform a compaction on the Metadata Table.
+ *
+ * Cases to be handled:
+ * 1. We cannot perform compaction if there are previous inflight operations on the dataset. This is because
+ * a compacted metadata base file at time Tx should represent all the actions on the dataset till time Tx.
*
- * Since we only read the latest base file in a partition, we tag the records with the instant time of the latest
- * base file.
+ * 2. In multi-writer scenario, a parallel operation with a greater instantTime may have completed creating a
+ * deltacommit.
*/
- private JavaRDD<HoodieRecord> prepRecords(List<HoodieRecord> records, String partitionName) {
- HoodieTable table = HoodieSparkTable.create(metadataWriteConfig, engineContext);
- TableFileSystemView.SliceView fsView = table.getSliceView();
- List<HoodieBaseFile> baseFiles = fsView.getLatestFileSlices(partitionName)
- .map(FileSlice::getBaseFile)
- .filter(Option::isPresent)
- .map(Option::get)
- .collect(Collectors.toList());
-
- // All the metadata fits within a single base file
- if (partitionName.equals(MetadataPartitionType.FILES.partitionPath())) {
- if (baseFiles.size() > 1) {
- throw new HoodieMetadataException("Multiple base files found in metadata partition");
- }
+ private void compactIfNecessary(SparkRDDWriteClient writeClient, String instantTime) {
+ String latestDeltacommitTime = metaClient.getActiveTimeline().getDeltaCommitTimeline().filterCompletedInstants().lastInstant()
+ .get().getTimestamp();
+ List<HoodieInstant> pendingInstants = datasetMetaClient.reloadActiveTimeline().filterInflightsAndRequested()
+ .findInstantsBefore(latestDeltacommitTime).getInstants().collect(Collectors.toList());
+
+ if (!pendingInstants.isEmpty()) {
+ LOG.info(String.format("Cannot compact metadata table as there are %d inflight instants before latest deltacommit %s: %s",
+ pendingInstants.size(), latestDeltacommitTime, Arrays.toString(pendingInstants.toArray())));
+ return;
}
- JavaSparkContext jsc = ((HoodieSparkEngineContext) engineContext).getJavaSparkContext();
- String fileId;
- String instantTime;
- if (!baseFiles.isEmpty()) {
- fileId = baseFiles.get(0).getFileId();
- instantTime = baseFiles.get(0).getCommitTime();
- } else {
- // If there is a log file then we can assume that it has the data
- List<HoodieLogFile> logFiles = fsView.getLatestFileSlices(MetadataPartitionType.FILES.partitionPath())
- .map(FileSlice::getLatestLogFile)
- .filter(Option::isPresent)
- .map(Option::get)
- .collect(Collectors.toList());
- if (logFiles.isEmpty()) {
- // No base and log files. All are new inserts
- return jsc.parallelize(records, 1);
- }
-
- fileId = logFiles.get(0).getFileId();
- instantTime = logFiles.get(0).getBaseCommitTime();
+ // Trigger compaction with suffixes based on the same instant time. This ensures that any future
+ // delta commits synced over will not have an instant time lesser than the last completed instant on the
+ // metadata table.
+ final String compactionInstantTime = latestDeltacommitTime + "001";
+ if (writeClient.scheduleCompactionAtInstant(compactionInstantTime, Option.empty())) {
+ writeClient.compact(compactionInstantTime);
}
+ }
- return jsc.parallelize(records, 1).map(r -> r.setCurrentLocation(new HoodieRecordLocation(instantTime, fileId)));
+ private void doClean(SparkRDDWriteClient writeClient, String instantTime) {
+ // Trigger cleaning with suffixes based on the same instant time. This ensures that any future
+ // delta commits synced over will not have an instant time lesser than the last completed instant on the
+ // metadata table.
+ writeClient.clean(instantTime + "002");
+ }
+
+ /**
+ * Tag each record with the location in the given partition.
+ *
+ * The record is tagged with respective bucket's file location based on its record key.
+ */
+ private JavaRDD<HoodieRecord> prepRecords(List<HoodieRecord> records, String partitionName, int bucketCount) {
+ List<FileSlice> buckets = HoodieTableMetadataUtil.loadPartitionBucketsWithLatestFileSlices(metaClient, partitionName);
Review comment:
rename? all its doing is fetching fileSlices. no bucketing terms
##########
File path: hudi-client/hudi-client-common/src/main/java/org/apache/hudi/client/AbstractHoodieWriteClient.java
##########
@@ -188,6 +189,7 @@ public boolean commitStats(String instantTime, List<HoodieWriteStat> stats, Opti
lastCompletedTxnAndMetadata.isPresent() ? Option.of(lastCompletedTxnAndMetadata.get().getLeft()) : Option.empty());
try {
preCommit(instantTime, metadata);
+ table.getMetadataWriter().ifPresent(w -> ((HoodieTableMetadataWriter)w).update(metadata, instantTime));
Review comment:
I still think this this we should call this from Spark's `preCommit()`. we are supposed to have all code that needs to run before committing there - precommit is not meant for conflict resolution, its the other way around. Conflict resolution had to be put there, just like how metadata writing should also go there. Introducing these one-offs just makes it hard to maintain over longer term.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] vinothchandar commented on a change in pull request #3590: [HUDI-2285][HUDI-2476] Metadata table synchronous design. Rebased and Squashed from pull/3426
Posted by GitBox <gi...@apache.org>.
vinothchandar commented on a change in pull request #3590:
URL: https://github.com/apache/hudi/pull/3590#discussion_r716234905
##########
File path: hudi-common/src/main/java/org/apache/hudi/metadata/HoodieBackedTableMetadata.java
##########
@@ -74,14 +77,11 @@
// Metadata table's timeline and metaclient
private HoodieTableMetaClient metaClient;
private HoodieTableConfig tableConfig;
- private List<FileSlice> latestFileSystemMetadataSlices;
// should we reuse the open file handles, across calls
private final boolean reuse;
-
- // Readers for the base and log file which store the metadata
- private transient HoodieFileReader<GenericRecord> baseFileReader;
- private transient HoodieMetadataMergedLogRecordScanner logRecordScanner;
+ // Readers for each buckets identified by the bucketFileId
+ private Map<String, Pair<HoodieFileReader, HoodieMetadataMergedLogRecordScanner>> bucketReaders = new ConcurrentHashMap<>();
Review comment:
buckets? really? :)
##########
File path: hudi-common/src/main/java/org/apache/hudi/metadata/HoodieBackedTableMetadata.java
##########
@@ -56,15 +59,15 @@
import java.util.ArrayList;
import java.util.Collections;
import java.util.HashMap;
+import java.util.LinkedList;
import java.util.List;
import java.util.Map;
+import java.util.Set;
+import java.util.concurrent.ConcurrentHashMap;
import java.util.stream.Collectors;
/**
* Table metadata provided by an internal DFS backed Hudi metadata table.
- *
- * If the metadata table does not exist, RPC calls are used to retrieve file listings from the file system.
- * No updates are applied to the table and it is not synced.
*/
public class HoodieBackedTableMetadata extends BaseTableMetadata {
Review comment:
lets please unit test this file thoroughly.
##########
File path: hudi-flink/src/test/java/org/apache/hudi/sink/TestStreamWriteOperatorCoordinator.java
##########
@@ -180,7 +181,7 @@ public void testHiveSyncInvoked() throws Exception {
assertDoesNotThrow(() -> coordinator.notifyCheckpointComplete(1));
}
- @Test
+ @Disabled
Review comment:
are we tracking a Jira to re-enable this
##########
File path: hudi-common/src/main/java/org/apache/hudi/metadata/HoodieBackedTableMetadata.java
##########
@@ -167,173 +170,150 @@ private void initIfNeeded() {
throw new HoodieIOException("Error merging records from metadata table for key :" + key, ioe);
} finally {
if (!reuse) {
- closeOrThrow();
+ close(partitionName);
}
}
}
- private void openReadersIfNeededOrThrow() {
- try {
- openReadersIfNeeded();
- } catch (IOException e) {
- throw new HoodieIOException("Error opening readers to the Metadata Table: ", e);
- }
- }
-
/**
* Returns a new pair of readers to the base and log files.
*/
- private void openReadersIfNeeded() throws IOException {
- if (reuse && (baseFileReader != null || logRecordScanner != null)) {
- // quickly exit out without synchronizing if reusing and readers are already open
- return;
- }
-
- // we always force synchronization, if reuse=false, to handle concurrent close() calls as well.
- synchronized (this) {
- if (baseFileReader != null || logRecordScanner != null) {
- return;
- }
-
- final long baseFileOpenMs;
- final long logScannerOpenMs;
+ private Pair<HoodieFileReader, HoodieMetadataMergedLogRecordScanner> openReadersIfNeeded(String key, String partitionName) throws IOException {
Review comment:
please audit this and ensure the reuse of readers across different calls to `getRecordKeyByKey()` works from the timeline server path. Otherwise, performance will suffer from repeated open and close-es
##########
File path: hudi-common/src/main/java/org/apache/hudi/metadata/HoodieBackedTableMetadata.java
##########
@@ -116,20 +116,23 @@ private void initIfNeeded() {
this.metaClient = null;
this.tableConfig = null;
}
-
- if (enabled) {
- openTimelineScanner(metaClient.getActiveTimeline());
- }
}
}
@Override
- protected Option<HoodieRecord<HoodieMetadataPayload>> getRecordByKeyFromMetadata(String key) {
+ protected Option<HoodieRecord<HoodieMetadataPayload>> getRecordByKeyFromMetadata(String key, String partitionName) {
+ Pair<HoodieFileReader, HoodieMetadataMergedLogRecordScanner> readers;
+ try {
+ readers = openReadersIfNeeded(key, partitionName);
+ } catch (IOException e) {
Review comment:
why bring the exception handling back in again? I am sure Robert C Martin would agree with having exception handling method separately. :)
##########
File path: hudi-common/src/main/java/org/apache/hudi/metadata/HoodieBackedTableMetadata.java
##########
@@ -167,173 +170,150 @@ private void initIfNeeded() {
throw new HoodieIOException("Error merging records from metadata table for key :" + key, ioe);
} finally {
if (!reuse) {
- closeOrThrow();
+ close(partitionName);
}
}
}
- private void openReadersIfNeededOrThrow() {
- try {
- openReadersIfNeeded();
- } catch (IOException e) {
- throw new HoodieIOException("Error opening readers to the Metadata Table: ", e);
- }
- }
-
/**
* Returns a new pair of readers to the base and log files.
*/
- private void openReadersIfNeeded() throws IOException {
- if (reuse && (baseFileReader != null || logRecordScanner != null)) {
- // quickly exit out without synchronizing if reusing and readers are already open
- return;
- }
-
- // we always force synchronization, if reuse=false, to handle concurrent close() calls as well.
- synchronized (this) {
- if (baseFileReader != null || logRecordScanner != null) {
- return;
- }
-
- final long baseFileOpenMs;
- final long logScannerOpenMs;
+ private Pair<HoodieFileReader, HoodieMetadataMergedLogRecordScanner> openReadersIfNeeded(String key, String partitionName) throws IOException {
Review comment:
This method is too long. lets break them down nicely
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot edited a comment on pull request #3590: [HUDI-2285][HUDI-2476] Metadata table synchronous design. Rebased and Squashed from pull/3426
Posted by GitBox <gi...@apache.org>.
hudi-bot edited a comment on pull request #3590:
URL: https://github.com/apache/hudi/pull/3590#issuecomment-912237120
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "648e36e1af308534bc050c3a19da008ab7a8c840",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1994",
"triggerID" : "648e36e1af308534bc050c3a19da008ab7a8c840",
"triggerType" : "PUSH"
}, {
"hash" : "8c32253ad0f6d9eb728789fad75763bb5703678d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1995",
"triggerID" : "8c32253ad0f6d9eb728789fad75763bb5703678d",
"triggerType" : "PUSH"
}, {
"hash" : "71bbe59aa8bf70b9aa8990a6dd2522eb4bf1d172",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2009",
"triggerID" : "71bbe59aa8bf70b9aa8990a6dd2522eb4bf1d172",
"triggerType" : "PUSH"
}, {
"hash" : "aefac7ec2f2e40bdf3ad4365ea6aa825803a439d",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "aefac7ec2f2e40bdf3ad4365ea6aa825803a439d",
"triggerType" : "PUSH"
}, {
"hash" : "86905cb855fecfeb9e4c023d526ad643fadb8084",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2012",
"triggerID" : "86905cb855fecfeb9e4c023d526ad643fadb8084",
"triggerType" : "PUSH"
}, {
"hash" : "b4d93b0168775cabd43609fec59bf2823dd81279",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2029",
"triggerID" : "b4d93b0168775cabd43609fec59bf2823dd81279",
"triggerType" : "PUSH"
}, {
"hash" : "38bf430ecb4e11839d686929be2511eed340e9df",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2035",
"triggerID" : "38bf430ecb4e11839d686929be2511eed340e9df",
"triggerType" : "PUSH"
}, {
"hash" : "7793fbdb9b93a129ef606cb2d73ea6e1e9074957",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2299",
"triggerID" : "7793fbdb9b93a129ef606cb2d73ea6e1e9074957",
"triggerType" : "PUSH"
}, {
"hash" : "9c0123c0f27f990d009b323bab75b76ceecf3dab",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2307",
"triggerID" : "9c0123c0f27f990d009b323bab75b76ceecf3dab",
"triggerType" : "PUSH"
}, {
"hash" : "af27976ba07afb02305c699aeb7a75a122013b62",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2330",
"triggerID" : "af27976ba07afb02305c699aeb7a75a122013b62",
"triggerType" : "PUSH"
}, {
"hash" : "9172241c58424cca8e8186d4409a8c36968a4790",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2344",
"triggerID" : "9172241c58424cca8e8186d4409a8c36968a4790",
"triggerType" : "PUSH"
}, {
"hash" : "13b80067ae00a760c57d184f70e2d98f63027e80",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2353",
"triggerID" : "13b80067ae00a760c57d184f70e2d98f63027e80",
"triggerType" : "PUSH"
}, {
"hash" : "ef750979cf449ff504da194e594cb62302eba360",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2366",
"triggerID" : "ef750979cf449ff504da194e594cb62302eba360",
"triggerType" : "PUSH"
}, {
"hash" : "16827c37f36d3b9fec18e0a35aff7599c45e0444",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2375",
"triggerID" : "16827c37f36d3b9fec18e0a35aff7599c45e0444",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* aefac7ec2f2e40bdf3ad4365ea6aa825803a439d UNKNOWN
* 16827c37f36d3b9fec18e0a35aff7599c45e0444 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2375)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run travis` re-run the last Travis build
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot edited a comment on pull request #3590: [HUDI-2285][HUDI-2476] Metadata table synchronous design. Rebased and Squashed from pull/3426
Posted by GitBox <gi...@apache.org>.
hudi-bot edited a comment on pull request #3590:
URL: https://github.com/apache/hudi/pull/3590#issuecomment-912237120
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "648e36e1af308534bc050c3a19da008ab7a8c840",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1994",
"triggerID" : "648e36e1af308534bc050c3a19da008ab7a8c840",
"triggerType" : "PUSH"
}, {
"hash" : "8c32253ad0f6d9eb728789fad75763bb5703678d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1995",
"triggerID" : "8c32253ad0f6d9eb728789fad75763bb5703678d",
"triggerType" : "PUSH"
}, {
"hash" : "71bbe59aa8bf70b9aa8990a6dd2522eb4bf1d172",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2009",
"triggerID" : "71bbe59aa8bf70b9aa8990a6dd2522eb4bf1d172",
"triggerType" : "PUSH"
}, {
"hash" : "aefac7ec2f2e40bdf3ad4365ea6aa825803a439d",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "aefac7ec2f2e40bdf3ad4365ea6aa825803a439d",
"triggerType" : "PUSH"
}, {
"hash" : "86905cb855fecfeb9e4c023d526ad643fadb8084",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2012",
"triggerID" : "86905cb855fecfeb9e4c023d526ad643fadb8084",
"triggerType" : "PUSH"
}, {
"hash" : "b4d93b0168775cabd43609fec59bf2823dd81279",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2029",
"triggerID" : "b4d93b0168775cabd43609fec59bf2823dd81279",
"triggerType" : "PUSH"
}, {
"hash" : "38bf430ecb4e11839d686929be2511eed340e9df",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2035",
"triggerID" : "38bf430ecb4e11839d686929be2511eed340e9df",
"triggerType" : "PUSH"
}, {
"hash" : "7793fbdb9b93a129ef606cb2d73ea6e1e9074957",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2299",
"triggerID" : "7793fbdb9b93a129ef606cb2d73ea6e1e9074957",
"triggerType" : "PUSH"
}, {
"hash" : "9c0123c0f27f990d009b323bab75b76ceecf3dab",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2307",
"triggerID" : "9c0123c0f27f990d009b323bab75b76ceecf3dab",
"triggerType" : "PUSH"
}, {
"hash" : "af27976ba07afb02305c699aeb7a75a122013b62",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2330",
"triggerID" : "af27976ba07afb02305c699aeb7a75a122013b62",
"triggerType" : "PUSH"
}, {
"hash" : "9172241c58424cca8e8186d4409a8c36968a4790",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "9172241c58424cca8e8186d4409a8c36968a4790",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* aefac7ec2f2e40bdf3ad4365ea6aa825803a439d UNKNOWN
* af27976ba07afb02305c699aeb7a75a122013b62 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2330)
* 9172241c58424cca8e8186d4409a8c36968a4790 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run travis` re-run the last Travis build
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot edited a comment on pull request #3590: [HUDI-2285] Metadata table synchronous design. Rebased and Squashed from pull/3426
Posted by GitBox <gi...@apache.org>.
hudi-bot edited a comment on pull request #3590:
URL: https://github.com/apache/hudi/pull/3590#issuecomment-912237120
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "648e36e1af308534bc050c3a19da008ab7a8c840",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1994",
"triggerID" : "648e36e1af308534bc050c3a19da008ab7a8c840",
"triggerType" : "PUSH"
}, {
"hash" : "8c32253ad0f6d9eb728789fad75763bb5703678d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1995",
"triggerID" : "8c32253ad0f6d9eb728789fad75763bb5703678d",
"triggerType" : "PUSH"
}, {
"hash" : "71bbe59aa8bf70b9aa8990a6dd2522eb4bf1d172",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2009",
"triggerID" : "71bbe59aa8bf70b9aa8990a6dd2522eb4bf1d172",
"triggerType" : "PUSH"
}, {
"hash" : "aefac7ec2f2e40bdf3ad4365ea6aa825803a439d",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "aefac7ec2f2e40bdf3ad4365ea6aa825803a439d",
"triggerType" : "PUSH"
}, {
"hash" : "86905cb855fecfeb9e4c023d526ad643fadb8084",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2012",
"triggerID" : "86905cb855fecfeb9e4c023d526ad643fadb8084",
"triggerType" : "PUSH"
}, {
"hash" : "b4d93b0168775cabd43609fec59bf2823dd81279",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2029",
"triggerID" : "b4d93b0168775cabd43609fec59bf2823dd81279",
"triggerType" : "PUSH"
}, {
"hash" : "38bf430ecb4e11839d686929be2511eed340e9df",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2035",
"triggerID" : "38bf430ecb4e11839d686929be2511eed340e9df",
"triggerType" : "PUSH"
}, {
"hash" : "7793fbdb9b93a129ef606cb2d73ea6e1e9074957",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2299",
"triggerID" : "7793fbdb9b93a129ef606cb2d73ea6e1e9074957",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* aefac7ec2f2e40bdf3ad4365ea6aa825803a439d UNKNOWN
* 7793fbdb9b93a129ef606cb2d73ea6e1e9074957 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2299)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run travis` re-run the last Travis build
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot edited a comment on pull request #3590: [HUDI-2285][HUDI-2476] Metadata table synchronous design. Rebased and Squashed from pull/3426
Posted by GitBox <gi...@apache.org>.
hudi-bot edited a comment on pull request #3590:
URL: https://github.com/apache/hudi/pull/3590#issuecomment-912237120
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "648e36e1af308534bc050c3a19da008ab7a8c840",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1994",
"triggerID" : "648e36e1af308534bc050c3a19da008ab7a8c840",
"triggerType" : "PUSH"
}, {
"hash" : "8c32253ad0f6d9eb728789fad75763bb5703678d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1995",
"triggerID" : "8c32253ad0f6d9eb728789fad75763bb5703678d",
"triggerType" : "PUSH"
}, {
"hash" : "71bbe59aa8bf70b9aa8990a6dd2522eb4bf1d172",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2009",
"triggerID" : "71bbe59aa8bf70b9aa8990a6dd2522eb4bf1d172",
"triggerType" : "PUSH"
}, {
"hash" : "aefac7ec2f2e40bdf3ad4365ea6aa825803a439d",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "aefac7ec2f2e40bdf3ad4365ea6aa825803a439d",
"triggerType" : "PUSH"
}, {
"hash" : "86905cb855fecfeb9e4c023d526ad643fadb8084",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2012",
"triggerID" : "86905cb855fecfeb9e4c023d526ad643fadb8084",
"triggerType" : "PUSH"
}, {
"hash" : "b4d93b0168775cabd43609fec59bf2823dd81279",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2029",
"triggerID" : "b4d93b0168775cabd43609fec59bf2823dd81279",
"triggerType" : "PUSH"
}, {
"hash" : "38bf430ecb4e11839d686929be2511eed340e9df",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2035",
"triggerID" : "38bf430ecb4e11839d686929be2511eed340e9df",
"triggerType" : "PUSH"
}, {
"hash" : "7793fbdb9b93a129ef606cb2d73ea6e1e9074957",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2299",
"triggerID" : "7793fbdb9b93a129ef606cb2d73ea6e1e9074957",
"triggerType" : "PUSH"
}, {
"hash" : "9c0123c0f27f990d009b323bab75b76ceecf3dab",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2307",
"triggerID" : "9c0123c0f27f990d009b323bab75b76ceecf3dab",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* aefac7ec2f2e40bdf3ad4365ea6aa825803a439d UNKNOWN
* 7793fbdb9b93a129ef606cb2d73ea6e1e9074957 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2299)
* 9c0123c0f27f990d009b323bab75b76ceecf3dab Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2307)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run travis` re-run the last Travis build
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] nsivabalan commented on a change in pull request #3590: [HUDI-2285][HUDI-2476] Metadata table synchronous design. Rebased and Squashed from pull/3426
Posted by GitBox <gi...@apache.org>.
nsivabalan commented on a change in pull request #3590:
URL: https://github.com/apache/hudi/pull/3590#discussion_r716251124
##########
File path: hudi-client/hudi-client-common/src/main/java/org/apache/hudi/metadata/HoodieBackedTableMetadataWriter.java
##########
@@ -401,64 +394,83 @@ private boolean bootstrapFromFilesystem(HoodieEngineContext engineContext, Hoodi
}
/**
- * Sync the Metadata Table from the instants created on the dataset.
+ * Initialize file groups for a partition. For file listing, we just have one file group.
*
- * @param datasetMetaClient {@code HoodieTableMetaClient} for the dataset
+ * All FileGroups for a given metadata partition has a fixed prefix as per the {@link MetadataPartitionType#getFileIdPrefix()}.
+ * Each file group is suffixed with increments of 1 starting with 1.
+ *
+ * For instance, for FILES, there is only one file group named as "files-1"
+ * Lets say we configure 10 file groups for record level index, and prefix as "record-index-bucket-"
+ * Filegroups will be named as :
+ * record-index-bucket-01
+ * record-index-bucket-02
+ * ...
+ * record-index-bucket-10
*/
- private void syncFromInstants(HoodieTableMetaClient datasetMetaClient) {
- ValidationUtils.checkState(enabled, "Metadata table cannot be synced as it is not enabled");
- // (re) init the metadata for reading.
- initTableMetadata();
- try {
- List<HoodieInstant> instantsToSync = metadata.findInstantsToSyncForWriter();
- if (instantsToSync.isEmpty()) {
- return;
- }
-
- LOG.info("Syncing " + instantsToSync.size() + " instants to metadata table: " + instantsToSync);
-
- // Read each instant in order and sync it to metadata table
- for (HoodieInstant instant : instantsToSync) {
- LOG.info("Syncing instant " + instant + " to metadata table");
-
- Option<List<HoodieRecord>> records = HoodieTableMetadataUtil.convertInstantToMetaRecords(datasetMetaClient,
- metaClient.getActiveTimeline(), instant, metadata.getUpdateTime());
- if (records.isPresent()) {
- commit(records.get(), MetadataPartitionType.FILES.partitionPath(), instant.getTimestamp());
- }
+ private void initializeFileGroups(HoodieTableMetaClient datasetMetaClient, MetadataPartitionType metadataPartition, String instantTime,
+ int fileGroupCount) throws IOException {
+
+ final HashMap<HeaderMetadataType, String> blockHeader = new HashMap<>();
+ blockHeader.put(HeaderMetadataType.INSTANT_TIME, instantTime);
+ // Archival of data table has a dependency on compaction(base files) in metadata table.
+ // It is assumed that as of time Tx of base instant (/compaction time) in metadata table,
+ // all commits in data table is in sync with metadata table. So, we always create start with log file for any fileGroup.
+ final HoodieDeleteBlock block = new HoodieDeleteBlock(new HoodieKey[0], blockHeader);
Review comment:
I feel it may not be easy to relax this. we can discuss this async as we close out this patch.
here are the two dependencies, of which 1 could be relaxed.
1. during rollback, we check if commit that is being rollback is already synced or not. if it is < last compacted time, we assume it's already synced. We can get away with this if need be. We can always assume, if the commit being rollbacked is not part of active timeline of metadata, its not synced and go ahead with it. Only difference we might have here is, there could be some additional files which are added to delete list which are not original synced to metadata table at all.
2. archival of dataset if dependent on compaction in metadata table. This might need more thoughts.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] nsivabalan commented on pull request #3590: [HUDI-2285][HUDI-2476] Metadata table synchronous design. Rebased and Squashed from pull/3426
Posted by GitBox <gi...@apache.org>.
nsivabalan commented on pull request #3590:
URL: https://github.com/apache/hudi/pull/3590#issuecomment-934478106
@hudi-bot azure run
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot edited a comment on pull request #3590: [HUDI-2285][HUDI-2476] Metadata table synchronous design. Rebased and Squashed from pull/3426
Posted by GitBox <gi...@apache.org>.
hudi-bot edited a comment on pull request #3590:
URL: https://github.com/apache/hudi/pull/3590#issuecomment-912237120
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "648e36e1af308534bc050c3a19da008ab7a8c840",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1994",
"triggerID" : "648e36e1af308534bc050c3a19da008ab7a8c840",
"triggerType" : "PUSH"
}, {
"hash" : "8c32253ad0f6d9eb728789fad75763bb5703678d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1995",
"triggerID" : "8c32253ad0f6d9eb728789fad75763bb5703678d",
"triggerType" : "PUSH"
}, {
"hash" : "71bbe59aa8bf70b9aa8990a6dd2522eb4bf1d172",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2009",
"triggerID" : "71bbe59aa8bf70b9aa8990a6dd2522eb4bf1d172",
"triggerType" : "PUSH"
}, {
"hash" : "aefac7ec2f2e40bdf3ad4365ea6aa825803a439d",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "aefac7ec2f2e40bdf3ad4365ea6aa825803a439d",
"triggerType" : "PUSH"
}, {
"hash" : "86905cb855fecfeb9e4c023d526ad643fadb8084",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2012",
"triggerID" : "86905cb855fecfeb9e4c023d526ad643fadb8084",
"triggerType" : "PUSH"
}, {
"hash" : "b4d93b0168775cabd43609fec59bf2823dd81279",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2029",
"triggerID" : "b4d93b0168775cabd43609fec59bf2823dd81279",
"triggerType" : "PUSH"
}, {
"hash" : "38bf430ecb4e11839d686929be2511eed340e9df",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2035",
"triggerID" : "38bf430ecb4e11839d686929be2511eed340e9df",
"triggerType" : "PUSH"
}, {
"hash" : "7793fbdb9b93a129ef606cb2d73ea6e1e9074957",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2299",
"triggerID" : "7793fbdb9b93a129ef606cb2d73ea6e1e9074957",
"triggerType" : "PUSH"
}, {
"hash" : "9c0123c0f27f990d009b323bab75b76ceecf3dab",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2307",
"triggerID" : "9c0123c0f27f990d009b323bab75b76ceecf3dab",
"triggerType" : "PUSH"
}, {
"hash" : "af27976ba07afb02305c699aeb7a75a122013b62",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2330",
"triggerID" : "af27976ba07afb02305c699aeb7a75a122013b62",
"triggerType" : "PUSH"
}, {
"hash" : "9172241c58424cca8e8186d4409a8c36968a4790",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2344",
"triggerID" : "9172241c58424cca8e8186d4409a8c36968a4790",
"triggerType" : "PUSH"
}, {
"hash" : "13b80067ae00a760c57d184f70e2d98f63027e80",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2353",
"triggerID" : "13b80067ae00a760c57d184f70e2d98f63027e80",
"triggerType" : "PUSH"
}, {
"hash" : "ef750979cf449ff504da194e594cb62302eba360",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2366",
"triggerID" : "ef750979cf449ff504da194e594cb62302eba360",
"triggerType" : "PUSH"
}, {
"hash" : "16827c37f36d3b9fec18e0a35aff7599c45e0444",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2375",
"triggerID" : "16827c37f36d3b9fec18e0a35aff7599c45e0444",
"triggerType" : "PUSH"
}, {
"hash" : "0f68d426b01355537f5124b8f108345a36d3da39",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2378",
"triggerID" : "0f68d426b01355537f5124b8f108345a36d3da39",
"triggerType" : "PUSH"
}, {
"hash" : "2f3099197889fd5c48ae3b550cc523e0528ddb2c",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2380",
"triggerID" : "2f3099197889fd5c48ae3b550cc523e0528ddb2c",
"triggerType" : "PUSH"
}, {
"hash" : "c83d176e1d1ddab6bb86f91726060779a9a1519f",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2392",
"triggerID" : "c83d176e1d1ddab6bb86f91726060779a9a1519f",
"triggerType" : "PUSH"
}, {
"hash" : "008f0cd91de12d44030d837f460f3b0232d62ee5",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2416",
"triggerID" : "008f0cd91de12d44030d837f460f3b0232d62ee5",
"triggerType" : "PUSH"
}, {
"hash" : "92634fb730e723a9bdeb165348a4c747794be7e3",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2418",
"triggerID" : "92634fb730e723a9bdeb165348a4c747794be7e3",
"triggerType" : "PUSH"
}, {
"hash" : "3b5fbfa0cc69e82f7c23fc6aa5b4944e2dd77220",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2438",
"triggerID" : "3b5fbfa0cc69e82f7c23fc6aa5b4944e2dd77220",
"triggerType" : "PUSH"
}, {
"hash" : "87d2ec0cc60ef8d5b7a081b73a0318aedf1a2a90",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2451",
"triggerID" : "87d2ec0cc60ef8d5b7a081b73a0318aedf1a2a90",
"triggerType" : "PUSH"
}, {
"hash" : "c09f5e30662efd882ff9fc35f5750a8253171e9d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2452",
"triggerID" : "c09f5e30662efd882ff9fc35f5750a8253171e9d",
"triggerType" : "PUSH"
}, {
"hash" : "b432e1db06e032b8071fa6c9e871ebc30b04b9eb",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2485",
"triggerID" : "b432e1db06e032b8071fa6c9e871ebc30b04b9eb",
"triggerType" : "PUSH"
}, {
"hash" : "0d70dba11569f96064e2cbf5c9d5fcb3c7fee5c0",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2503",
"triggerID" : "0d70dba11569f96064e2cbf5c9d5fcb3c7fee5c0",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* aefac7ec2f2e40bdf3ad4365ea6aa825803a439d UNKNOWN
* b432e1db06e032b8071fa6c9e871ebc30b04b9eb Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2485)
* 0d70dba11569f96064e2cbf5c9d5fcb3c7fee5c0 Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2503)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run travis` re-run the last Travis build
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot edited a comment on pull request #3590: [HUDI-2285][HUDI-2476] Metadata table synchronous design. Rebased and Squashed from pull/3426
Posted by GitBox <gi...@apache.org>.
hudi-bot edited a comment on pull request #3590:
URL: https://github.com/apache/hudi/pull/3590#issuecomment-912237120
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "648e36e1af308534bc050c3a19da008ab7a8c840",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1994",
"triggerID" : "648e36e1af308534bc050c3a19da008ab7a8c840",
"triggerType" : "PUSH"
}, {
"hash" : "8c32253ad0f6d9eb728789fad75763bb5703678d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1995",
"triggerID" : "8c32253ad0f6d9eb728789fad75763bb5703678d",
"triggerType" : "PUSH"
}, {
"hash" : "71bbe59aa8bf70b9aa8990a6dd2522eb4bf1d172",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2009",
"triggerID" : "71bbe59aa8bf70b9aa8990a6dd2522eb4bf1d172",
"triggerType" : "PUSH"
}, {
"hash" : "aefac7ec2f2e40bdf3ad4365ea6aa825803a439d",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "aefac7ec2f2e40bdf3ad4365ea6aa825803a439d",
"triggerType" : "PUSH"
}, {
"hash" : "86905cb855fecfeb9e4c023d526ad643fadb8084",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2012",
"triggerID" : "86905cb855fecfeb9e4c023d526ad643fadb8084",
"triggerType" : "PUSH"
}, {
"hash" : "b4d93b0168775cabd43609fec59bf2823dd81279",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2029",
"triggerID" : "b4d93b0168775cabd43609fec59bf2823dd81279",
"triggerType" : "PUSH"
}, {
"hash" : "38bf430ecb4e11839d686929be2511eed340e9df",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2035",
"triggerID" : "38bf430ecb4e11839d686929be2511eed340e9df",
"triggerType" : "PUSH"
}, {
"hash" : "7793fbdb9b93a129ef606cb2d73ea6e1e9074957",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2299",
"triggerID" : "7793fbdb9b93a129ef606cb2d73ea6e1e9074957",
"triggerType" : "PUSH"
}, {
"hash" : "9c0123c0f27f990d009b323bab75b76ceecf3dab",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2307",
"triggerID" : "9c0123c0f27f990d009b323bab75b76ceecf3dab",
"triggerType" : "PUSH"
}, {
"hash" : "af27976ba07afb02305c699aeb7a75a122013b62",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2330",
"triggerID" : "af27976ba07afb02305c699aeb7a75a122013b62",
"triggerType" : "PUSH"
}, {
"hash" : "9172241c58424cca8e8186d4409a8c36968a4790",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2344",
"triggerID" : "9172241c58424cca8e8186d4409a8c36968a4790",
"triggerType" : "PUSH"
}, {
"hash" : "13b80067ae00a760c57d184f70e2d98f63027e80",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2353",
"triggerID" : "13b80067ae00a760c57d184f70e2d98f63027e80",
"triggerType" : "PUSH"
}, {
"hash" : "ef750979cf449ff504da194e594cb62302eba360",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2366",
"triggerID" : "ef750979cf449ff504da194e594cb62302eba360",
"triggerType" : "PUSH"
}, {
"hash" : "16827c37f36d3b9fec18e0a35aff7599c45e0444",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "16827c37f36d3b9fec18e0a35aff7599c45e0444",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* aefac7ec2f2e40bdf3ad4365ea6aa825803a439d UNKNOWN
* ef750979cf449ff504da194e594cb62302eba360 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2366)
* 16827c37f36d3b9fec18e0a35aff7599c45e0444 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run travis` re-run the last Travis build
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] nsivabalan commented on a change in pull request #3590: [HUDI-2285][HUDI-2476] Metadata table synchronous design. Rebased and Squashed from pull/3426
Posted by GitBox <gi...@apache.org>.
nsivabalan commented on a change in pull request #3590:
URL: https://github.com/apache/hudi/pull/3590#discussion_r721356787
##########
File path: hudi-client/hudi-client-common/src/main/java/org/apache/hudi/client/AbstractHoodieWriteClient.java
##########
@@ -241,13 +242,16 @@ void emitCommitMetrics(String instantTime, HoodieCommitMetadata metadata, String
}
}
+ /**
+ * Any pre-commit actions like conflict resolution or updating metadata table goes here.
+ * @param instantTime commit instant time.
+ * @param metadata commit metadata for which pre commit is being invoked.
+ */
protected void preCommit(String instantTime, HoodieCommitMetadata metadata) {
- // no-op
- // TODO : Conflict resolution is not supported for Flink & Java engines
- }
-
- protected void syncTableMetadata() {
- // no-op
+ // Create a Hoodie table after startTxn which encapsulated the commits and files visible.
+ // Important to create this after the lock to ensure latest commits show up in the timeline without need for reload
Review comment:
I feel adding this comment at the caller does not fit in well. This is what it looks like.
```
try {
// Precommit is meant for resolving conflicts if any and to update metadata table if enabled.
// Create a Hoodie table within preCommit after starting the transaction which encapsulated the commits and files visible
// (instead of using an existing instance of Hoodie table). It is important to create this after the lock to ensure latest
// commits show up in the timeline without need for reload.
preCommit(instantTime, metadata);
commit(table, commitActionType, instantTime, metadata, stats);
postCommit(table, metadata, instantTime, extraMetadata);
...
```
I feel having this within preCommit just belongs to the place where it is needed or used.
Let me know what do you think.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot edited a comment on pull request #3590: [HUDI-2285][HUDI-2476] Metadata table synchronous design. Rebased and Squashed from pull/3426
Posted by GitBox <gi...@apache.org>.
hudi-bot edited a comment on pull request #3590:
URL: https://github.com/apache/hudi/pull/3590#issuecomment-912237120
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "648e36e1af308534bc050c3a19da008ab7a8c840",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1994",
"triggerID" : "648e36e1af308534bc050c3a19da008ab7a8c840",
"triggerType" : "PUSH"
}, {
"hash" : "8c32253ad0f6d9eb728789fad75763bb5703678d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1995",
"triggerID" : "8c32253ad0f6d9eb728789fad75763bb5703678d",
"triggerType" : "PUSH"
}, {
"hash" : "71bbe59aa8bf70b9aa8990a6dd2522eb4bf1d172",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2009",
"triggerID" : "71bbe59aa8bf70b9aa8990a6dd2522eb4bf1d172",
"triggerType" : "PUSH"
}, {
"hash" : "aefac7ec2f2e40bdf3ad4365ea6aa825803a439d",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "aefac7ec2f2e40bdf3ad4365ea6aa825803a439d",
"triggerType" : "PUSH"
}, {
"hash" : "86905cb855fecfeb9e4c023d526ad643fadb8084",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2012",
"triggerID" : "86905cb855fecfeb9e4c023d526ad643fadb8084",
"triggerType" : "PUSH"
}, {
"hash" : "b4d93b0168775cabd43609fec59bf2823dd81279",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2029",
"triggerID" : "b4d93b0168775cabd43609fec59bf2823dd81279",
"triggerType" : "PUSH"
}, {
"hash" : "38bf430ecb4e11839d686929be2511eed340e9df",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2035",
"triggerID" : "38bf430ecb4e11839d686929be2511eed340e9df",
"triggerType" : "PUSH"
}, {
"hash" : "7793fbdb9b93a129ef606cb2d73ea6e1e9074957",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2299",
"triggerID" : "7793fbdb9b93a129ef606cb2d73ea6e1e9074957",
"triggerType" : "PUSH"
}, {
"hash" : "9c0123c0f27f990d009b323bab75b76ceecf3dab",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2307",
"triggerID" : "9c0123c0f27f990d009b323bab75b76ceecf3dab",
"triggerType" : "PUSH"
}, {
"hash" : "af27976ba07afb02305c699aeb7a75a122013b62",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2330",
"triggerID" : "af27976ba07afb02305c699aeb7a75a122013b62",
"triggerType" : "PUSH"
}, {
"hash" : "9172241c58424cca8e8186d4409a8c36968a4790",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2344",
"triggerID" : "9172241c58424cca8e8186d4409a8c36968a4790",
"triggerType" : "PUSH"
}, {
"hash" : "13b80067ae00a760c57d184f70e2d98f63027e80",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2353",
"triggerID" : "13b80067ae00a760c57d184f70e2d98f63027e80",
"triggerType" : "PUSH"
}, {
"hash" : "ef750979cf449ff504da194e594cb62302eba360",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2366",
"triggerID" : "ef750979cf449ff504da194e594cb62302eba360",
"triggerType" : "PUSH"
}, {
"hash" : "16827c37f36d3b9fec18e0a35aff7599c45e0444",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2375",
"triggerID" : "16827c37f36d3b9fec18e0a35aff7599c45e0444",
"triggerType" : "PUSH"
}, {
"hash" : "0f68d426b01355537f5124b8f108345a36d3da39",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2378",
"triggerID" : "0f68d426b01355537f5124b8f108345a36d3da39",
"triggerType" : "PUSH"
}, {
"hash" : "2f3099197889fd5c48ae3b550cc523e0528ddb2c",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2380",
"triggerID" : "2f3099197889fd5c48ae3b550cc523e0528ddb2c",
"triggerType" : "PUSH"
}, {
"hash" : "c83d176e1d1ddab6bb86f91726060779a9a1519f",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2392",
"triggerID" : "c83d176e1d1ddab6bb86f91726060779a9a1519f",
"triggerType" : "PUSH"
}, {
"hash" : "008f0cd91de12d44030d837f460f3b0232d62ee5",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "008f0cd91de12d44030d837f460f3b0232d62ee5",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* aefac7ec2f2e40bdf3ad4365ea6aa825803a439d UNKNOWN
* c83d176e1d1ddab6bb86f91726060779a9a1519f Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2392)
* 008f0cd91de12d44030d837f460f3b0232d62ee5 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run travis` re-run the last Travis build
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot edited a comment on pull request #3590: [WIP] Sync metadata patch
Posted by GitBox <gi...@apache.org>.
hudi-bot edited a comment on pull request #3590:
URL: https://github.com/apache/hudi/pull/3590#issuecomment-912237120
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "648e36e1af308534bc050c3a19da008ab7a8c840",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1994",
"triggerID" : "648e36e1af308534bc050c3a19da008ab7a8c840",
"triggerType" : "PUSH"
}, {
"hash" : "8c32253ad0f6d9eb728789fad75763bb5703678d",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "8c32253ad0f6d9eb728789fad75763bb5703678d",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 648e36e1af308534bc050c3a19da008ab7a8c840 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1994)
* 8c32253ad0f6d9eb728789fad75763bb5703678d UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run travis` re-run the last Travis build
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot edited a comment on pull request #3590: [WIP][HUDI-2285] Metadata table synchronous design. Rebased and Squashed from pull/3426
Posted by GitBox <gi...@apache.org>.
hudi-bot edited a comment on pull request #3590:
URL: https://github.com/apache/hudi/pull/3590#issuecomment-912237120
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "648e36e1af308534bc050c3a19da008ab7a8c840",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1994",
"triggerID" : "648e36e1af308534bc050c3a19da008ab7a8c840",
"triggerType" : "PUSH"
}, {
"hash" : "8c32253ad0f6d9eb728789fad75763bb5703678d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=1995",
"triggerID" : "8c32253ad0f6d9eb728789fad75763bb5703678d",
"triggerType" : "PUSH"
}, {
"hash" : "71bbe59aa8bf70b9aa8990a6dd2522eb4bf1d172",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2009",
"triggerID" : "71bbe59aa8bf70b9aa8990a6dd2522eb4bf1d172",
"triggerType" : "PUSH"
}, {
"hash" : "aefac7ec2f2e40bdf3ad4365ea6aa825803a439d",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "aefac7ec2f2e40bdf3ad4365ea6aa825803a439d",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 71bbe59aa8bf70b9aa8990a6dd2522eb4bf1d172 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2009)
* aefac7ec2f2e40bdf3ad4365ea6aa825803a439d UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run travis` re-run the last Travis build
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] vinishjail97 commented on a change in pull request #3590: [HUDI-2285][HUDI-2476] Metadata table synchronous design. Rebased and Squashed from pull/3426
Posted by GitBox <gi...@apache.org>.
vinishjail97 commented on a change in pull request #3590:
URL: https://github.com/apache/hudi/pull/3590#discussion_r757085011
##########
File path: hudi-client/hudi-client-common/src/main/java/org/apache/hudi/table/HoodieTimelineArchiveLog.java
##########
@@ -200,20 +200,19 @@ public boolean archiveIfRequired(HoodieEngineContext context) throws IOException
.collect(Collectors.groupingBy(i -> Pair.of(i.getTimestamp(),
HoodieInstant.getComparableAction(i.getAction()))));
- // If metadata table is enabled, do not archive instants which are more recent that the latest synced
- // instant on the metadata table. This is required for metadata table sync.
+ // If metadata table is enabled, do not archive instants which are more recent that the last compaction on the
+ // metadata table.
if (config.isMetadataTableEnabled()) {
try (HoodieTableMetadata tableMetadata = HoodieTableMetadata.create(table.getContext(), config.getMetadataConfig(),
config.getBasePath(), FileSystemViewStorageConfig.SPILLABLE_DIR.defaultValue())) {
- Option<String> lastSyncedInstantTime = tableMetadata.getUpdateTime();
-
- if (lastSyncedInstantTime.isPresent()) {
- LOG.info("Limiting archiving of instants to last synced instant on metadata table at " + lastSyncedInstantTime.get());
- instants = instants.filter(i -> HoodieTimeline.compareTimestamps(i.getTimestamp(), HoodieTimeline.LESSER_THAN,
- lastSyncedInstantTime.get()));
- } else {
- LOG.info("Not archiving as there is no instants yet on the metadata table");
+ Option<String> latestCompactionTime = tableMetadata.getLatestCompactionTime();
Review comment:
@nsivabalan
Adding a clarifying comment on this PR to debug an issue regarding archival of tables.
https://issues.apache.org/jira/browse/HUDI-2735
What should be done for enabling the archiving for MOR table ? I think after this change `getLatestCompactionTime` is only filtering `.commits` and for an MOR table this would always be empty and is causing archival to be disabled.
@xushiyan @rmahindra123 FYI
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] prashantwason commented on a change in pull request #3590: [HUDI-2285][HUDI-2476] Metadata table synchronous design. Rebased and Squashed from pull/3426
Posted by GitBox <gi...@apache.org>.
prashantwason commented on a change in pull request #3590:
URL: https://github.com/apache/hudi/pull/3590#discussion_r758965003
##########
File path: hudi-client/hudi-client-common/src/main/java/org/apache/hudi/table/HoodieTimelineArchiveLog.java
##########
@@ -200,20 +200,19 @@ public boolean archiveIfRequired(HoodieEngineContext context) throws IOException
.collect(Collectors.groupingBy(i -> Pair.of(i.getTimestamp(),
HoodieInstant.getComparableAction(i.getAction()))));
- // If metadata table is enabled, do not archive instants which are more recent that the latest synced
- // instant on the metadata table. This is required for metadata table sync.
+ // If metadata table is enabled, do not archive instants which are more recent that the last compaction on the
+ // metadata table.
if (config.isMetadataTableEnabled()) {
try (HoodieTableMetadata tableMetadata = HoodieTableMetadata.create(table.getContext(), config.getMetadataConfig(),
config.getBasePath(), FileSystemViewStorageConfig.SPILLABLE_DIR.defaultValue())) {
- Option<String> lastSyncedInstantTime = tableMetadata.getUpdateTime();
-
- if (lastSyncedInstantTime.isPresent()) {
- LOG.info("Limiting archiving of instants to last synced instant on metadata table at " + lastSyncedInstantTime.get());
- instants = instants.filter(i -> HoodieTimeline.compareTimestamps(i.getTimestamp(), HoodieTimeline.LESSER_THAN,
- lastSyncedInstantTime.get()));
- } else {
- LOG.info("Not archiving as there is no instants yet on the metadata table");
+ Option<String> latestCompactionTime = tableMetadata.getLatestCompactionTime();
Review comment:
In the synchronous metadata table design (0.10+), the following is the order of how an action completes in HUDI when metadata table is enabled (assuming a commit).
t1.commit.requested
t1.commit.inflight
<all writes complete here>
finalizeWrite()
<Write to metadata table here which creates t1.deltacommit on metadata table>
<NOTE: the t1.commit on the dataset has not yet completed as we have not created t1.commit yet>
t1.commit is created.
So the deltacommit to the metadata table is completed before the commit completes. Remember, these are two different HUDI tables.
There is an error condition - when we crash / fail before the very last step - t1.deltacommit is created on dataset but t1.commit is not created. In this case, t1.deltacommit is NOT VALID on the metadata table. We ignore t1.deltacommit while reading from metadata table. This validation requires us to check each tX.deltacommmit against corresponding instants on the dataset timeline. So if the dataset timeline is archived independent of metadata table compaction then this logic will not work.
This is further complicated in the presence of multiple writers - each of them in parallel writing to the dataset (e.g. t2.commit.inflight, t3.commit.inflight, etc) and writing to the metadata table when complete. So metadata table could have t3.deltacommit but no t2.deltacommit which could have failed or not completed yet.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org