You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@hudi.apache.org by GitBox <gi...@apache.org> on 2022/03/23 12:08:47 UTC
[GitHub] [hudi] YuweiXiao opened a new pull request #5110: [HUDI-3692] Includes compaction instant in timeline for MetadataFileSystemView
YuweiXiao opened a new pull request #5110:
URL: https://github.com/apache/hudi/pull/5110
## *Tips*
- *Thank you very much for contributing to Apache Hudi.*
- *Please review https://hudi.apache.org/contribute/how-to-contribute before opening a pull request.*
## What is the purpose of the pull request
Fix lost log write when there is inflight compaction.
## Brief change log
Include compaction instant when constructing `HoodieMetadataFileSystemView`, which is consistent with `HoodieTableFileSystemView`.
## Verify this pull request
- A test case is provided to verify the fix.
## Committer checklist
- [x] Has a corresponding JIRA in PR title & commit
- [x] Commit message is descriptive of the change
- [ ] CI is green
- [ ] Necessary doc changes done or have another open PR
- [ ] For large changes, please consider breaking it into sub-tasks under an umbrella JIRA.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #5110: [HUDI-3692] Includes compaction instant in timeline for MetadataFileSystemView
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on pull request #5110:
URL: https://github.com/apache/hudi/pull/5110#issuecomment-1076531435
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "f9779d551d89bf8e35b2601b039dc3576641e196",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=7238",
"triggerID" : "f9779d551d89bf8e35b2601b039dc3576641e196",
"triggerType" : "PUSH"
}, {
"hash" : "f9779d551d89bf8e35b2601b039dc3576641e196",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=7248",
"triggerID" : "1076483733",
"triggerType" : "MANUAL"
} ]
}-->
## CI report:
* f9779d551d89bf8e35b2601b039dc3576641e196 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=7238) Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=7248)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #5110: [HUDI-3692] Fix lost log writes during async compaction when using metadata table
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on pull request #5110:
URL: https://github.com/apache/hudi/pull/5110#issuecomment-1078666667
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "f9779d551d89bf8e35b2601b039dc3576641e196",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=7238",
"triggerID" : "f9779d551d89bf8e35b2601b039dc3576641e196",
"triggerType" : "PUSH"
}, {
"hash" : "f9779d551d89bf8e35b2601b039dc3576641e196",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=7248",
"triggerID" : "1076483733",
"triggerType" : "MANUAL"
}, {
"hash" : "ec861ba304ed6586070aa940c78be1c2ace6a25e",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "ec861ba304ed6586070aa940c78be1c2ace6a25e",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* f9779d551d89bf8e35b2601b039dc3576641e196 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=7238) Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=7248)
* ec861ba304ed6586070aa940c78be1c2ace6a25e UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] YuweiXiao commented on pull request #5110: [HUDI-3692] Includes compaction instant in timeline for MetadataFileSystemView
Posted by GitBox <gi...@apache.org>.
YuweiXiao commented on pull request #5110:
URL: https://github.com/apache/hudi/pull/5110#issuecomment-1076483733
@hudi-bot run azure
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #5110: [HUDI-3692] Includes compaction instant in timeline for MetadataFileSystemView
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on pull request #5110:
URL: https://github.com/apache/hudi/pull/5110#issuecomment-1076305705
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "f9779d551d89bf8e35b2601b039dc3576641e196",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "f9779d551d89bf8e35b2601b039dc3576641e196",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* f9779d551d89bf8e35b2601b039dc3576641e196 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #5110: [HUDI-3692] Includes compaction instant in timeline for MetadataFileSystemView
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on pull request #5110:
URL: https://github.com/apache/hudi/pull/5110#issuecomment-1076485379
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "f9779d551d89bf8e35b2601b039dc3576641e196",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=7238",
"triggerID" : "f9779d551d89bf8e35b2601b039dc3576641e196",
"triggerType" : "PUSH"
}, {
"hash" : "f9779d551d89bf8e35b2601b039dc3576641e196",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=7248",
"triggerID" : "1076483733",
"triggerType" : "MANUAL"
} ]
}-->
## CI report:
* f9779d551d89bf8e35b2601b039dc3576641e196 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=7238) Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=7248)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #5110: [HUDI-3692] Fix lost log writes during async compaction when using metadata table
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on pull request #5110:
URL: https://github.com/apache/hudi/pull/5110#issuecomment-1079895431
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "f9779d551d89bf8e35b2601b039dc3576641e196",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=7238",
"triggerID" : "f9779d551d89bf8e35b2601b039dc3576641e196",
"triggerType" : "PUSH"
}, {
"hash" : "f9779d551d89bf8e35b2601b039dc3576641e196",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=7248",
"triggerID" : "1076483733",
"triggerType" : "MANUAL"
}, {
"hash" : "ec861ba304ed6586070aa940c78be1c2ace6a25e",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=7346",
"triggerID" : "ec861ba304ed6586070aa940c78be1c2ace6a25e",
"triggerType" : "PUSH"
}, {
"hash" : "ec861ba304ed6586070aa940c78be1c2ace6a25e",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=7414",
"triggerID" : "1079878162",
"triggerType" : "MANUAL"
} ]
}-->
## CI report:
* ec861ba304ed6586070aa940c78be1c2ace6a25e Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=7346) Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=7414)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot removed a comment on pull request #5110: [HUDI-3692] Fix lost log writes during async compaction when using metadata table
Posted by GitBox <gi...@apache.org>.
hudi-bot removed a comment on pull request #5110:
URL: https://github.com/apache/hudi/pull/5110#issuecomment-1078946465
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "f9779d551d89bf8e35b2601b039dc3576641e196",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=7238",
"triggerID" : "f9779d551d89bf8e35b2601b039dc3576641e196",
"triggerType" : "PUSH"
}, {
"hash" : "f9779d551d89bf8e35b2601b039dc3576641e196",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=7248",
"triggerID" : "1076483733",
"triggerType" : "MANUAL"
}, {
"hash" : "ec861ba304ed6586070aa940c78be1c2ace6a25e",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=7346",
"triggerID" : "ec861ba304ed6586070aa940c78be1c2ace6a25e",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* ec861ba304ed6586070aa940c78be1c2ace6a25e Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=7346)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot removed a comment on pull request #5110: [HUDI-3692] Includes compaction instant in timeline for MetadataFileSystemView
Posted by GitBox <gi...@apache.org>.
hudi-bot removed a comment on pull request #5110:
URL: https://github.com/apache/hudi/pull/5110#issuecomment-1076305705
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "f9779d551d89bf8e35b2601b039dc3576641e196",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "f9779d551d89bf8e35b2601b039dc3576641e196",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* f9779d551d89bf8e35b2601b039dc3576641e196 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #5110: [HUDI-3692] Fix lost log writes during async compaction when using metadata table
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on pull request #5110:
URL: https://github.com/apache/hudi/pull/5110#issuecomment-1078668105
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "f9779d551d89bf8e35b2601b039dc3576641e196",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=7238",
"triggerID" : "f9779d551d89bf8e35b2601b039dc3576641e196",
"triggerType" : "PUSH"
}, {
"hash" : "f9779d551d89bf8e35b2601b039dc3576641e196",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=7248",
"triggerID" : "1076483733",
"triggerType" : "MANUAL"
}, {
"hash" : "ec861ba304ed6586070aa940c78be1c2ace6a25e",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=7346",
"triggerID" : "ec861ba304ed6586070aa940c78be1c2ace6a25e",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* f9779d551d89bf8e35b2601b039dc3576641e196 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=7238) Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=7248)
* ec861ba304ed6586070aa940c78be1c2ace6a25e Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=7346)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] nsivabalan commented on a change in pull request #5110: [HUDI-3692] Fix lost log writes during async compaction when using metadata table
Posted by GitBox <gi...@apache.org>.
nsivabalan commented on a change in pull request #5110:
URL: https://github.com/apache/hudi/pull/5110#discussion_r839217950
##########
File path: hudi-client/hudi-spark-client/src/test/java/org/apache/hudi/table/functional/TestHoodieSparkMergeOnReadTableCompaction.java
##########
@@ -104,6 +122,44 @@ public void testWriteDuringCompaction() throws IOException {
Assertions.assertEquals(300, readTableTotalRecordsNum());
}
+ @ParameterizedTest
+ @MethodSource("writeLogTest")
+ public void testWriteLogDuringCompaction(boolean enableMetadataTable, boolean enableTimelineServer) throws IOException {
+ HoodieWriteConfig config = HoodieWriteConfig.newBuilder()
+ .forTable("test-trip-table")
+ .withPath(basePath())
+ .withSchema(TRIP_EXAMPLE_SCHEMA)
+ .withParallelism(2, 2)
+ .withAutoCommit(true)
+ .withEmbeddedTimelineServerEnabled(enableTimelineServer)
+ .withMetadataConfig(HoodieMetadataConfig.newBuilder().enable(enableMetadataTable).build())
+ .withCompactionConfig(HoodieCompactionConfig.newBuilder()
+ .withMaxNumDeltaCommitsBeforeCompaction(1).build())
+ .withLayoutConfig(HoodieLayoutConfig.newBuilder()
+ .withLayoutType(HoodieStorageLayout.LayoutType.BUCKET.name())
+ .withLayoutPartitioner(SparkBucketIndexPartitioner.class.getName()).build())
+ .withIndexConfig(HoodieIndexConfig.newBuilder().withIndexType(HoodieIndex.IndexType.BUCKET).withBucketNum("1").build()).build();
+ metaClient = getHoodieMetaClient(HoodieTableType.MERGE_ON_READ, config.getProps());
+ client = getHoodieWriteClient(config);
+
+ final List<HoodieRecord> records = dataGen.generateInserts("001", 100);
+ JavaRDD<HoodieRecord> writeRecords = jsc().parallelize(records, 2);
+
+ // initialize 100 records
+ client.upsert(writeRecords, client.startCommit());
+ // update 100 records
+ client.upsert(writeRecords, client.startCommit());
+ // schedule compaction
+ client.scheduleCompaction(Option.empty());
+ // delete 50 records
+ List<HoodieKey> toBeDeleted = records.stream().map(HoodieRecord::getKey).limit(50).collect(Collectors.toList());
+ JavaRDD<HoodieKey> deleteRecords = jsc().parallelize(toBeDeleted, 2);
+ client.delete(deleteRecords, client.startCommit());
+ // insert the same 100 records again
+ client.upsert(writeRecords, client.startCommit());
+ Assertions.assertEquals(100, readTableTotalRecordsNum());
Review comment:
@YuweiXiao : was this test failing w/o the fix ?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #5110: [HUDI-3692] Includes compaction instant in timeline for MetadataFileSystemView
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on pull request #5110:
URL: https://github.com/apache/hudi/pull/5110#issuecomment-1076308522
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "f9779d551d89bf8e35b2601b039dc3576641e196",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=7238",
"triggerID" : "f9779d551d89bf8e35b2601b039dc3576641e196",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* f9779d551d89bf8e35b2601b039dc3576641e196 Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=7238)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot removed a comment on pull request #5110: [HUDI-3692] Includes compaction instant in timeline for MetadataFileSystemView
Posted by GitBox <gi...@apache.org>.
hudi-bot removed a comment on pull request #5110:
URL: https://github.com/apache/hudi/pull/5110#issuecomment-1076485379
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "f9779d551d89bf8e35b2601b039dc3576641e196",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=7238",
"triggerID" : "f9779d551d89bf8e35b2601b039dc3576641e196",
"triggerType" : "PUSH"
}, {
"hash" : "f9779d551d89bf8e35b2601b039dc3576641e196",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=7248",
"triggerID" : "1076483733",
"triggerType" : "MANUAL"
} ]
}-->
## CI report:
* f9779d551d89bf8e35b2601b039dc3576641e196 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=7238) Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=7248)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] YuweiXiao commented on a change in pull request #5110: [HUDI-3692] Fix lost log writes during async compaction when using metadata table
Posted by GitBox <gi...@apache.org>.
YuweiXiao commented on a change in pull request #5110:
URL: https://github.com/apache/hudi/pull/5110#discussion_r839219512
##########
File path: hudi-client/hudi-spark-client/src/test/java/org/apache/hudi/table/functional/TestHoodieSparkMergeOnReadTableCompaction.java
##########
@@ -104,6 +122,44 @@ public void testWriteDuringCompaction() throws IOException {
Assertions.assertEquals(300, readTableTotalRecordsNum());
}
+ @ParameterizedTest
+ @MethodSource("writeLogTest")
+ public void testWriteLogDuringCompaction(boolean enableMetadataTable, boolean enableTimelineServer) throws IOException {
+ HoodieWriteConfig config = HoodieWriteConfig.newBuilder()
+ .forTable("test-trip-table")
+ .withPath(basePath())
+ .withSchema(TRIP_EXAMPLE_SCHEMA)
+ .withParallelism(2, 2)
+ .withAutoCommit(true)
+ .withEmbeddedTimelineServerEnabled(enableTimelineServer)
+ .withMetadataConfig(HoodieMetadataConfig.newBuilder().enable(enableMetadataTable).build())
+ .withCompactionConfig(HoodieCompactionConfig.newBuilder()
+ .withMaxNumDeltaCommitsBeforeCompaction(1).build())
+ .withLayoutConfig(HoodieLayoutConfig.newBuilder()
+ .withLayoutType(HoodieStorageLayout.LayoutType.BUCKET.name())
+ .withLayoutPartitioner(SparkBucketIndexPartitioner.class.getName()).build())
+ .withIndexConfig(HoodieIndexConfig.newBuilder().withIndexType(HoodieIndex.IndexType.BUCKET).withBucketNum("1").build()).build();
+ metaClient = getHoodieMetaClient(HoodieTableType.MERGE_ON_READ, config.getProps());
+ client = getHoodieWriteClient(config);
+
+ final List<HoodieRecord> records = dataGen.generateInserts("001", 100);
+ JavaRDD<HoodieRecord> writeRecords = jsc().parallelize(records, 2);
+
+ // initialize 100 records
+ client.upsert(writeRecords, client.startCommit());
+ // update 100 records
+ client.upsert(writeRecords, client.startCommit());
+ // schedule compaction
+ client.scheduleCompaction(Option.empty());
+ // delete 50 records
+ List<HoodieKey> toBeDeleted = records.stream().map(HoodieRecord::getKey).limit(50).collect(Collectors.toList());
+ JavaRDD<HoodieKey> deleteRecords = jsc().parallelize(toBeDeleted, 2);
+ client.delete(deleteRecords, client.startCommit());
+ // insert the same 100 records again
+ client.upsert(writeRecords, client.startCommit());
+ Assertions.assertEquals(100, readTableTotalRecordsNum());
Review comment:
Yes, it will fail
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] YuweiXiao commented on pull request #5110: [HUDI-3692] Fix lost log writes during async compaction when using metadata table
Posted by GitBox <gi...@apache.org>.
YuweiXiao commented on pull request #5110:
URL: https://github.com/apache/hudi/pull/5110#issuecomment-1079878162
@hudi-bot run azure
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #5110: [HUDI-3692] Includes compaction instant in timeline for MetadataFileSystemView
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on pull request #5110:
URL: https://github.com/apache/hudi/pull/5110#issuecomment-1076399421
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "f9779d551d89bf8e35b2601b039dc3576641e196",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=7238",
"triggerID" : "f9779d551d89bf8e35b2601b039dc3576641e196",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* f9779d551d89bf8e35b2601b039dc3576641e196 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=7238)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot removed a comment on pull request #5110: [HUDI-3692] Fix lost log writes during async compaction when using metadata table
Posted by GitBox <gi...@apache.org>.
hudi-bot removed a comment on pull request #5110:
URL: https://github.com/apache/hudi/pull/5110#issuecomment-1076531435
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "f9779d551d89bf8e35b2601b039dc3576641e196",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=7238",
"triggerID" : "f9779d551d89bf8e35b2601b039dc3576641e196",
"triggerType" : "PUSH"
}, {
"hash" : "f9779d551d89bf8e35b2601b039dc3576641e196",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=7248",
"triggerID" : "1076483733",
"triggerType" : "MANUAL"
} ]
}-->
## CI report:
* f9779d551d89bf8e35b2601b039dc3576641e196 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=7238) Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=7248)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #5110: [HUDI-3692] Fix lost log writes during async compaction when using metadata table
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on pull request #5110:
URL: https://github.com/apache/hudi/pull/5110#issuecomment-1079878322
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "f9779d551d89bf8e35b2601b039dc3576641e196",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=7238",
"triggerID" : "f9779d551d89bf8e35b2601b039dc3576641e196",
"triggerType" : "PUSH"
}, {
"hash" : "f9779d551d89bf8e35b2601b039dc3576641e196",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=7248",
"triggerID" : "1076483733",
"triggerType" : "MANUAL"
}, {
"hash" : "ec861ba304ed6586070aa940c78be1c2ace6a25e",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=7346",
"triggerID" : "ec861ba304ed6586070aa940c78be1c2ace6a25e",
"triggerType" : "PUSH"
}, {
"hash" : "ec861ba304ed6586070aa940c78be1c2ace6a25e",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=7414",
"triggerID" : "1079878162",
"triggerType" : "MANUAL"
} ]
}-->
## CI report:
* ec861ba304ed6586070aa940c78be1c2ace6a25e Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=7346) Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=7414)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot removed a comment on pull request #5110: [HUDI-3692] Fix lost log writes during async compaction when using metadata table
Posted by GitBox <gi...@apache.org>.
hudi-bot removed a comment on pull request #5110:
URL: https://github.com/apache/hudi/pull/5110#issuecomment-1079878322
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "f9779d551d89bf8e35b2601b039dc3576641e196",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=7238",
"triggerID" : "f9779d551d89bf8e35b2601b039dc3576641e196",
"triggerType" : "PUSH"
}, {
"hash" : "f9779d551d89bf8e35b2601b039dc3576641e196",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=7248",
"triggerID" : "1076483733",
"triggerType" : "MANUAL"
}, {
"hash" : "ec861ba304ed6586070aa940c78be1c2ace6a25e",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=7346",
"triggerID" : "ec861ba304ed6586070aa940c78be1c2ace6a25e",
"triggerType" : "PUSH"
}, {
"hash" : "ec861ba304ed6586070aa940c78be1c2ace6a25e",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=7414",
"triggerID" : "1079878162",
"triggerType" : "MANUAL"
} ]
}-->
## CI report:
* ec861ba304ed6586070aa940c78be1c2ace6a25e Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=7346) Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=7414)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot removed a comment on pull request #5110: [HUDI-3692] Includes compaction instant in timeline for MetadataFileSystemView
Posted by GitBox <gi...@apache.org>.
hudi-bot removed a comment on pull request #5110:
URL: https://github.com/apache/hudi/pull/5110#issuecomment-1076308522
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "f9779d551d89bf8e35b2601b039dc3576641e196",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=7238",
"triggerID" : "f9779d551d89bf8e35b2601b039dc3576641e196",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* f9779d551d89bf8e35b2601b039dc3576641e196 Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=7238)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot removed a comment on pull request #5110: [HUDI-3692] Includes compaction instant in timeline for MetadataFileSystemView
Posted by GitBox <gi...@apache.org>.
hudi-bot removed a comment on pull request #5110:
URL: https://github.com/apache/hudi/pull/5110#issuecomment-1076399421
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "f9779d551d89bf8e35b2601b039dc3576641e196",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=7238",
"triggerID" : "f9779d551d89bf8e35b2601b039dc3576641e196",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* f9779d551d89bf8e35b2601b039dc3576641e196 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=7238)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] nsivabalan merged pull request #5110: [HUDI-3692] Fix lost log writes during async compaction when using metadata table
Posted by GitBox <gi...@apache.org>.
nsivabalan merged pull request #5110:
URL: https://github.com/apache/hudi/pull/5110
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot removed a comment on pull request #5110: [HUDI-3692] Fix lost log writes during async compaction when using metadata table
Posted by GitBox <gi...@apache.org>.
hudi-bot removed a comment on pull request #5110:
URL: https://github.com/apache/hudi/pull/5110#issuecomment-1078668105
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "f9779d551d89bf8e35b2601b039dc3576641e196",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=7238",
"triggerID" : "f9779d551d89bf8e35b2601b039dc3576641e196",
"triggerType" : "PUSH"
}, {
"hash" : "f9779d551d89bf8e35b2601b039dc3576641e196",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=7248",
"triggerID" : "1076483733",
"triggerType" : "MANUAL"
}, {
"hash" : "ec861ba304ed6586070aa940c78be1c2ace6a25e",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=7346",
"triggerID" : "ec861ba304ed6586070aa940c78be1c2ace6a25e",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* f9779d551d89bf8e35b2601b039dc3576641e196 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=7238) Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=7248)
* ec861ba304ed6586070aa940c78be1c2ace6a25e Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=7346)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #5110: [HUDI-3692] Fix lost log writes during async compaction when using metadata table
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on pull request #5110:
URL: https://github.com/apache/hudi/pull/5110#issuecomment-1078946465
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "f9779d551d89bf8e35b2601b039dc3576641e196",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=7238",
"triggerID" : "f9779d551d89bf8e35b2601b039dc3576641e196",
"triggerType" : "PUSH"
}, {
"hash" : "f9779d551d89bf8e35b2601b039dc3576641e196",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=7248",
"triggerID" : "1076483733",
"triggerType" : "MANUAL"
}, {
"hash" : "ec861ba304ed6586070aa940c78be1c2ace6a25e",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=7346",
"triggerID" : "ec861ba304ed6586070aa940c78be1c2ace6a25e",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* ec861ba304ed6586070aa940c78be1c2ace6a25e Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=7346)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot removed a comment on pull request #5110: [HUDI-3692] Fix lost log writes during async compaction when using metadata table
Posted by GitBox <gi...@apache.org>.
hudi-bot removed a comment on pull request #5110:
URL: https://github.com/apache/hudi/pull/5110#issuecomment-1078666667
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "f9779d551d89bf8e35b2601b039dc3576641e196",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=7238",
"triggerID" : "f9779d551d89bf8e35b2601b039dc3576641e196",
"triggerType" : "PUSH"
}, {
"hash" : "f9779d551d89bf8e35b2601b039dc3576641e196",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=7248",
"triggerID" : "1076483733",
"triggerType" : "MANUAL"
}, {
"hash" : "ec861ba304ed6586070aa940c78be1c2ace6a25e",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "ec861ba304ed6586070aa940c78be1c2ace6a25e",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* f9779d551d89bf8e35b2601b039dc3576641e196 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=7238) Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=7248)
* ec861ba304ed6586070aa940c78be1c2ace6a25e UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org