You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@hudi.apache.org by "lokeshj1703 (via GitHub)" <gi...@apache.org> on 2023/07/20 16:21:49 UTC
[GitHub] [hudi] lokeshj1703 opened a new pull request, #9246: [HUDI-6548] Two log compaction instants can be scheduled at the same time
lokeshj1703 opened a new pull request, #9246:
URL: https://github.com/apache/hudi/pull/9246
### Change Logs
Few HoodieDefaultTimeline APIs are missing the log compaction filter. As a result the log compaction instants are not included as part of write client timeline which makes it possible to schedule two log compactions at the same time.
### Impact
It fixes the HoodieDefaultTimeline APIs to include log compaction instants.
### Risk level (write none, low medium or high below)
low
### Documentation Update
NA
### Contributor's checklist
- [ ] Read through [contributor's guide](https://hudi.apache.org/contribute/how-to-contribute)
- [ ] Change Logs and Impact were stated clearly
- [ ] Adequate tests were added if applicable
- [ ] CI passed
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #9246: [HUDI-6548] Two log compaction instants can be scheduled at the same time
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #9246:
URL: https://github.com/apache/hudi/pull/9246#issuecomment-1644265490
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "136d780d0a9ca38f88c613433f05f868be01d0d5",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18734",
"triggerID" : "136d780d0a9ca38f88c613433f05f868be01d0d5",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 136d780d0a9ca38f88c613433f05f868be01d0d5 Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18734)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #9246: [HUDI-6548] Two log compaction instants can be scheduled at the same time
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #9246:
URL: https://github.com/apache/hudi/pull/9246#issuecomment-1649811839
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "136d780d0a9ca38f88c613433f05f868be01d0d5",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18734",
"triggerID" : "136d780d0a9ca38f88c613433f05f868be01d0d5",
"triggerType" : "PUSH"
}, {
"hash" : "c2effa1ea1fdd82828efbf88afbf6cd6be019eb3",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "c2effa1ea1fdd82828efbf88afbf6cd6be019eb3",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 136d780d0a9ca38f88c613433f05f868be01d0d5 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18734)
* c2effa1ea1fdd82828efbf88afbf6cd6be019eb3 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] lokeshj1703 commented on pull request #9246: [HUDI-6548] Two log compaction instants can be scheduled at the same time
Posted by "lokeshj1703 (via GitHub)" <gi...@apache.org>.
lokeshj1703 commented on PR #9246:
URL: https://github.com/apache/hudi/pull/9246#issuecomment-1658627944
The APIs will be refactored in a separate PR.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #9246: [HUDI-6548] Two log compaction instants can be scheduled at the same time
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #9246:
URL: https://github.com/apache/hudi/pull/9246#issuecomment-1644871288
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "136d780d0a9ca38f88c613433f05f868be01d0d5",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18734",
"triggerID" : "136d780d0a9ca38f88c613433f05f868be01d0d5",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 136d780d0a9ca38f88c613433f05f868be01d0d5 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18734)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #9246: [HUDI-6548] Two log compaction instants can be scheduled at the same time
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #9246:
URL: https://github.com/apache/hudi/pull/9246#issuecomment-1650135501
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "136d780d0a9ca38f88c613433f05f868be01d0d5",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18734",
"triggerID" : "136d780d0a9ca38f88c613433f05f868be01d0d5",
"triggerType" : "PUSH"
}, {
"hash" : "c2effa1ea1fdd82828efbf88afbf6cd6be019eb3",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18831",
"triggerID" : "c2effa1ea1fdd82828efbf88afbf6cd6be019eb3",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* c2effa1ea1fdd82828efbf88afbf6cd6be019eb3 Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18831)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] lokeshj1703 closed pull request #9246: [HUDI-6548] Two log compaction instants can be scheduled at the same time
Posted by "lokeshj1703 (via GitHub)" <gi...@apache.org>.
lokeshj1703 closed pull request #9246: [HUDI-6548] Two log compaction instants can be scheduled at the same time
URL: https://github.com/apache/hudi/pull/9246
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] nsivabalan commented on a diff in pull request #9246: [HUDI-6548] Two log compaction instants can be scheduled at the same time
Posted by "nsivabalan (via GitHub)" <gi...@apache.org>.
nsivabalan commented on code in PR #9246:
URL: https://github.com/apache/hudi/pull/9246#discussion_r1270221495
##########
hudi-common/src/main/java/org/apache/hudi/common/table/timeline/HoodieDefaultTimeline.java:
##########
@@ -277,7 +278,7 @@ public HoodieTimeline getCommitsTimeline() {
* Get all instants (commits, delta commits, replace, compaction) that produce new data or merge file, in the active timeline.
*/
public HoodieTimeline getCommitsAndCompactionTimeline() {
- return getTimelineOfActions(CollectionUtils.createSet(COMMIT_ACTION, DELTA_COMMIT_ACTION, REPLACE_COMMIT_ACTION, COMPACTION_ACTION));
+ return getTimelineOfActions(CollectionUtils.createSet(COMMIT_ACTION, DELTA_COMMIT_ACTION, REPLACE_COMMIT_ACTION, COMPACTION_ACTION, LOG_COMPACTION_ACTION));
Review Comment:
we should not fix this method. we should probably introduce
getCompletedCommitsCompactionLogCompactionTimeline
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #9246: [HUDI-6548] Two log compaction instants can be scheduled at the same time
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #9246:
URL: https://github.com/apache/hudi/pull/9246#issuecomment-1644254029
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "136d780d0a9ca38f88c613433f05f868be01d0d5",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "136d780d0a9ca38f88c613433f05f868be01d0d5",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 136d780d0a9ca38f88c613433f05f868be01d0d5 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] lokeshj1703 commented on a diff in pull request #9246: [HUDI-6548] Two log compaction instants can be scheduled at the same time
Posted by "lokeshj1703 (via GitHub)" <gi...@apache.org>.
lokeshj1703 commented on code in PR #9246:
URL: https://github.com/apache/hudi/pull/9246#discussion_r1273472332
##########
hudi-common/src/main/java/org/apache/hudi/common/table/timeline/HoodieDefaultTimeline.java:
##########
@@ -277,7 +278,7 @@ public HoodieTimeline getCommitsTimeline() {
* Get all instants (commits, delta commits, replace, compaction) that produce new data or merge file, in the active timeline.
*/
public HoodieTimeline getCommitsAndCompactionTimeline() {
- return getTimelineOfActions(CollectionUtils.createSet(COMMIT_ACTION, DELTA_COMMIT_ACTION, REPLACE_COMMIT_ACTION, COMPACTION_ACTION));
+ return getTimelineOfActions(CollectionUtils.createSet(COMMIT_ACTION, DELTA_COMMIT_ACTION, REPLACE_COMMIT_ACTION, COMPACTION_ACTION, LOG_COMPACTION_ACTION));
Review Comment:
@nsivabalan This was added as part of https://github.com/apache/hudi/pull/9038. Seems like this API should also consider inflight logcompaction. I have removed it from this PR but if it makes sense I will create a separate PR for it.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #9246: [HUDI-6548] Two log compaction instants can be scheduled at the same time
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #9246:
URL: https://github.com/apache/hudi/pull/9246#issuecomment-1649827320
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "136d780d0a9ca38f88c613433f05f868be01d0d5",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18734",
"triggerID" : "136d780d0a9ca38f88c613433f05f868be01d0d5",
"triggerType" : "PUSH"
}, {
"hash" : "c2effa1ea1fdd82828efbf88afbf6cd6be019eb3",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18831",
"triggerID" : "c2effa1ea1fdd82828efbf88afbf6cd6be019eb3",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 136d780d0a9ca38f88c613433f05f868be01d0d5 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18734)
* c2effa1ea1fdd82828efbf88afbf6cd6be019eb3 Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18831)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org