You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@hudi.apache.org by GitBox <gi...@apache.org> on 2022/09/17 07:08:50 UTC
[GitHub] [hudi] watermelon12138 opened a new pull request, #6705: [HUDI-4868] Fixed the issue that compaction is invalid when the last commit action is replace commit.
watermelon12138 opened a new pull request, #6705:
URL: https://github.com/apache/hudi/pull/6705
…commit action is replace commit.
### Change Logs
1. Replace commit should not be included when getting lastCompaction.
2. If the compaction schedule is executed, replace commit should be included.
### Impact
Fixed the issue that compaction is invalid when the last commit action is replace commit.
**Risk level: none | low | medium | high**
_Choose one. If medium or high, explain what verification was done to mitigate the risks._
### Contributor's checklist
- [ ] Read through [contributor's guide](https://hudi.apache.org/contribute/how-to-contribute)
- [ ] Change Logs and Impact were stated clearly
- [ ] Adequate tests were added if applicable
- [ ] CI passed
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #6705: [HUDI-4868] Fixed the issue that compaction is invalid when the last commit action is replace commit.
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on PR #6705:
URL: https://github.com/apache/hudi/pull/6705#issuecomment-1286357440
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "1b8fd27a6fbed918ac5bc5094a08dd8f01ddbda8",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=11453",
"triggerID" : "1b8fd27a6fbed918ac5bc5094a08dd8f01ddbda8",
"triggerType" : "PUSH"
}, {
"hash" : "22132d2333e9057bce866ce03a2d7b289af8c5c5",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=11927",
"triggerID" : "22132d2333e9057bce866ce03a2d7b289af8c5c5",
"triggerType" : "PUSH"
}, {
"hash" : "666b6690b531d1540b196f0f97e175c7da38166d",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12379",
"triggerID" : "666b6690b531d1540b196f0f97e175c7da38166d",
"triggerType" : "PUSH"
}, {
"hash" : "212fbe84247891d11bb0e1a12fe454b34a270c75",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12406",
"triggerID" : "212fbe84247891d11bb0e1a12fe454b34a270c75",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 666b6690b531d1540b196f0f97e175c7da38166d Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12379)
* 212fbe84247891d11bb0e1a12fe454b34a270c75 Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12406)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] nsivabalan commented on a diff in pull request #6705: [HUDI-4868] Fixed the issue that compaction is invalid when the last commit action is replace commit.
Posted by GitBox <gi...@apache.org>.
nsivabalan commented on code in PR #6705:
URL: https://github.com/apache/hudi/pull/6705#discussion_r984172615
##########
hudi-common/src/main/java/org/apache/hudi/common/util/CompactionUtils.java:
##########
@@ -214,22 +216,22 @@ public static List<HoodieInstant> getPendingCompactionInstantTimes(HoodieTableMe
*/
public static Option<Pair<HoodieTimeline, HoodieInstant>> getDeltaCommitsSinceLatestCompaction(
HoodieActiveTimeline activeTimeline) {
- Option<HoodieInstant> lastCompaction = activeTimeline.getCommitTimeline()
+ Option<HoodieInstant> lastCompaction = activeTimeline.getCommitTimeline().filter(s -> !s.getAction().equals(REPLACE_COMMIT_ACTION))
.filterCompletedInstants().lastInstant();
- HoodieTimeline deltaCommits = activeTimeline.getDeltaCommitTimeline();
+ HoodieTimeline deltaAndReplaceCommits = activeTimeline.getDeltaCommitAndReplaceCommitTimeline();
Review Comment:
but I am not sure if this makes sense. this method `getDeltaCommitsSinceLatestCompaction` only cares for delta commits for the purpose of scheduling compaction. So, replace commits does not matter. can you help me understand why we need to include repalce commits here.
##########
hudi-common/src/main/java/org/apache/hudi/common/util/CompactionUtils.java:
##########
@@ -214,22 +216,22 @@ public static List<HoodieInstant> getPendingCompactionInstantTimes(HoodieTableMe
*/
public static Option<Pair<HoodieTimeline, HoodieInstant>> getDeltaCommitsSinceLatestCompaction(
HoodieActiveTimeline activeTimeline) {
- Option<HoodieInstant> lastCompaction = activeTimeline.getCommitTimeline()
+ Option<HoodieInstant> lastCompaction = activeTimeline.getCommitTimeline().filter(s -> !s.getAction().equals(REPLACE_COMMIT_ACTION))
Review Comment:
I agree this fix makes sense.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] YannByron commented on a diff in pull request #6705: [HUDI-4868] Fixed the issue that compaction is invalid when the last commit action is replace commit.
Posted by GitBox <gi...@apache.org>.
YannByron commented on code in PR #6705:
URL: https://github.com/apache/hudi/pull/6705#discussion_r1004248356
##########
hudi-spark-datasource/hudi-spark/src/test/scala/org/apache/hudi/functional/cdc/TestCDCDataFrameSuite.scala:
##########
@@ -228,7 +227,8 @@ class TestCDCDataFrameSuite extends HoodieCDCTestBase {
def testMORDataSourceWrite(cdcSupplementalLoggingMode: String): Unit = {
val options = commonOpts ++ Map(
DataSourceWriteOptions.TABLE_TYPE.key() -> DataSourceWriteOptions.MOR_TABLE_TYPE_OPT_VAL,
- HoodieTableConfig.CDC_SUPPLEMENTAL_LOGGING_MODE.key -> cdcSupplementalLoggingMode
+ HoodieTableConfig.CDC_SUPPLEMENTAL_LOGGING_MODE.key -> cdcSupplementalLoggingMode,
+ HoodieCleanConfig.AUTO_CLEAN.key -> "false"
Review Comment:
why need to set `HoodieCleanConfig.AUTO_CLEAN` here. This config has been set in `commonOpts`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #6705: [HUDI-4868] Fixed the issue that compaction is invalid when the last commit action is replace commit.
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on PR #6705:
URL: https://github.com/apache/hudi/pull/6705#issuecomment-1286354207
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "1b8fd27a6fbed918ac5bc5094a08dd8f01ddbda8",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=11453",
"triggerID" : "1b8fd27a6fbed918ac5bc5094a08dd8f01ddbda8",
"triggerType" : "PUSH"
}, {
"hash" : "22132d2333e9057bce866ce03a2d7b289af8c5c5",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=11927",
"triggerID" : "22132d2333e9057bce866ce03a2d7b289af8c5c5",
"triggerType" : "PUSH"
}, {
"hash" : "666b6690b531d1540b196f0f97e175c7da38166d",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12379",
"triggerID" : "666b6690b531d1540b196f0f97e175c7da38166d",
"triggerType" : "PUSH"
}, {
"hash" : "212fbe84247891d11bb0e1a12fe454b34a270c75",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "212fbe84247891d11bb0e1a12fe454b34a270c75",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 666b6690b531d1540b196f0f97e175c7da38166d Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12379)
* 212fbe84247891d11bb0e1a12fe454b34a270c75 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #6705: [HUDI-4868] Fixed the issue that compaction is invalid when the last commit action is replace commit.
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on PR #6705:
URL: https://github.com/apache/hudi/pull/6705#issuecomment-1263369091
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "1b8fd27a6fbed918ac5bc5094a08dd8f01ddbda8",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=11453",
"triggerID" : "1b8fd27a6fbed918ac5bc5094a08dd8f01ddbda8",
"triggerType" : "PUSH"
}, {
"hash" : "22132d2333e9057bce866ce03a2d7b289af8c5c5",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=11927",
"triggerID" : "22132d2333e9057bce866ce03a2d7b289af8c5c5",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 1b8fd27a6fbed918ac5bc5094a08dd8f01ddbda8 Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=11453)
* 22132d2333e9057bce866ce03a2d7b289af8c5c5 Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=11927)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #6705: [HUDI-4868] Fixed the issue that compaction is invalid when the last commit action is replace commit.
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on PR #6705:
URL: https://github.com/apache/hudi/pull/6705#issuecomment-1250024387
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "1b8fd27a6fbed918ac5bc5094a08dd8f01ddbda8",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=11453",
"triggerID" : "1b8fd27a6fbed918ac5bc5094a08dd8f01ddbda8",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 1b8fd27a6fbed918ac5bc5094a08dd8f01ddbda8 Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=11453)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] nsivabalan commented on a diff in pull request #6705: [HUDI-4868] Fixed the issue that compaction is invalid when the last commit action is replace commit.
Posted by GitBox <gi...@apache.org>.
nsivabalan commented on code in PR #6705:
URL: https://github.com/apache/hudi/pull/6705#discussion_r1002634353
##########
hudi-common/src/main/java/org/apache/hudi/common/util/CompactionUtils.java:
##########
@@ -214,7 +216,7 @@ public static List<HoodieInstant> getPendingCompactionInstantTimes(HoodieTableMe
*/
public static Option<Pair<HoodieTimeline, HoodieInstant>> getDeltaCommitsSinceLatestCompaction(
HoodieActiveTimeline activeTimeline) {
- Option<HoodieInstant> lastCompaction = activeTimeline.getCommitTimeline()
+ Option<HoodieInstant> lastCompaction = activeTimeline.getCommitTimeline().filter(s -> !s.getAction().equals(REPLACE_COMMIT_ACTION))
Review Comment:
here is the code snippet to detect if a replace commit instant refers to clustrering or not.
```
Option<Pair<HoodieInstant, HoodieClusteringPlan>> instantPlan = ClusteringUtils.getClusteringPlan(table.getMetaClient(), instant);
```
if instantPlan is non empty, then it refers to clustering.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #6705: [HUDI-4868] Fixed the issue that compaction is invalid when the last commit action is replace commit.
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on PR #6705:
URL: https://github.com/apache/hudi/pull/6705#issuecomment-1286861202
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "1b8fd27a6fbed918ac5bc5094a08dd8f01ddbda8",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=11453",
"triggerID" : "1b8fd27a6fbed918ac5bc5094a08dd8f01ddbda8",
"triggerType" : "PUSH"
}, {
"hash" : "22132d2333e9057bce866ce03a2d7b289af8c5c5",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=11927",
"triggerID" : "22132d2333e9057bce866ce03a2d7b289af8c5c5",
"triggerType" : "PUSH"
}, {
"hash" : "666b6690b531d1540b196f0f97e175c7da38166d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12379",
"triggerID" : "666b6690b531d1540b196f0f97e175c7da38166d",
"triggerType" : "PUSH"
}, {
"hash" : "212fbe84247891d11bb0e1a12fe454b34a270c75",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12406",
"triggerID" : "212fbe84247891d11bb0e1a12fe454b34a270c75",
"triggerType" : "PUSH"
}, {
"hash" : "1f5f43dab710b2b891ac35762f21321d1f5e59b6",
"status" : "CANCELED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12414",
"triggerID" : "1f5f43dab710b2b891ac35762f21321d1f5e59b6",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 1f5f43dab710b2b891ac35762f21321d1f5e59b6 Azure: [CANCELED](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12414)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] watermelon12138 commented on pull request #6705: [HUDI-4868] Fixed the issue that compaction is invalid when the last commit action is replace commit.
Posted by GitBox <gi...@apache.org>.
watermelon12138 commented on PR #6705:
URL: https://github.com/apache/hudi/pull/6705#issuecomment-1263316829
@nsivabalan
I have modified it according to the review comments and hope you can review again, thanks
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #6705: [HUDI-4868] Fixed the issue that compaction is invalid when the last commit action is replace commit.
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on PR #6705:
URL: https://github.com/apache/hudi/pull/6705#issuecomment-1285086233
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "1b8fd27a6fbed918ac5bc5094a08dd8f01ddbda8",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=11453",
"triggerID" : "1b8fd27a6fbed918ac5bc5094a08dd8f01ddbda8",
"triggerType" : "PUSH"
}, {
"hash" : "22132d2333e9057bce866ce03a2d7b289af8c5c5",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=11927",
"triggerID" : "22132d2333e9057bce866ce03a2d7b289af8c5c5",
"triggerType" : "PUSH"
}, {
"hash" : "666b6690b531d1540b196f0f97e175c7da38166d",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "666b6690b531d1540b196f0f97e175c7da38166d",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 22132d2333e9057bce866ce03a2d7b289af8c5c5 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=11927)
* 666b6690b531d1540b196f0f97e175c7da38166d UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #6705: [HUDI-4868] Fixed the issue that compaction is invalid when the last commit action is replace commit.
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on PR #6705:
URL: https://github.com/apache/hudi/pull/6705#issuecomment-1250122924
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "1b8fd27a6fbed918ac5bc5094a08dd8f01ddbda8",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=11453",
"triggerID" : "1b8fd27a6fbed918ac5bc5094a08dd8f01ddbda8",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 1b8fd27a6fbed918ac5bc5094a08dd8f01ddbda8 Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=11453)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #6705: [HUDI-4868] Fixed the issue that compaction is invalid when the last commit action is replace commit.
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on PR #6705:
URL: https://github.com/apache/hudi/pull/6705#issuecomment-1263363958
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "1b8fd27a6fbed918ac5bc5094a08dd8f01ddbda8",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=11453",
"triggerID" : "1b8fd27a6fbed918ac5bc5094a08dd8f01ddbda8",
"triggerType" : "PUSH"
}, {
"hash" : "22132d2333e9057bce866ce03a2d7b289af8c5c5",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "22132d2333e9057bce866ce03a2d7b289af8c5c5",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 1b8fd27a6fbed918ac5bc5094a08dd8f01ddbda8 Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=11453)
* 22132d2333e9057bce866ce03a2d7b289af8c5c5 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #6705: [HUDI-4868] Fixed the issue that compaction is invalid when the last commit action is replace commit.
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on PR #6705:
URL: https://github.com/apache/hudi/pull/6705#issuecomment-1286614988
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "1b8fd27a6fbed918ac5bc5094a08dd8f01ddbda8",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=11453",
"triggerID" : "1b8fd27a6fbed918ac5bc5094a08dd8f01ddbda8",
"triggerType" : "PUSH"
}, {
"hash" : "22132d2333e9057bce866ce03a2d7b289af8c5c5",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=11927",
"triggerID" : "22132d2333e9057bce866ce03a2d7b289af8c5c5",
"triggerType" : "PUSH"
}, {
"hash" : "666b6690b531d1540b196f0f97e175c7da38166d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12379",
"triggerID" : "666b6690b531d1540b196f0f97e175c7da38166d",
"triggerType" : "PUSH"
}, {
"hash" : "212fbe84247891d11bb0e1a12fe454b34a270c75",
"status" : "CANCELED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12406",
"triggerID" : "212fbe84247891d11bb0e1a12fe454b34a270c75",
"triggerType" : "PUSH"
}, {
"hash" : "1f5f43dab710b2b891ac35762f21321d1f5e59b6",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12414",
"triggerID" : "1f5f43dab710b2b891ac35762f21321d1f5e59b6",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 212fbe84247891d11bb0e1a12fe454b34a270c75 Azure: [CANCELED](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12406)
* 1f5f43dab710b2b891ac35762f21321d1f5e59b6 Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12414)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #6705: [HUDI-4868] Fixed the issue that compaction is invalid when the last commit action is replace commit.
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on PR #6705:
URL: https://github.com/apache/hudi/pull/6705#issuecomment-1286107027
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "1b8fd27a6fbed918ac5bc5094a08dd8f01ddbda8",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=11453",
"triggerID" : "1b8fd27a6fbed918ac5bc5094a08dd8f01ddbda8",
"triggerType" : "PUSH"
}, {
"hash" : "22132d2333e9057bce866ce03a2d7b289af8c5c5",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=11927",
"triggerID" : "22132d2333e9057bce866ce03a2d7b289af8c5c5",
"triggerType" : "PUSH"
}, {
"hash" : "666b6690b531d1540b196f0f97e175c7da38166d",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12379",
"triggerID" : "666b6690b531d1540b196f0f97e175c7da38166d",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 666b6690b531d1540b196f0f97e175c7da38166d Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12379)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #6705: [HUDI-4868] Fixed the issue that compaction is invalid when the last commit action is replace commit.
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on PR #6705:
URL: https://github.com/apache/hudi/pull/6705#issuecomment-1286551846
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "1b8fd27a6fbed918ac5bc5094a08dd8f01ddbda8",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=11453",
"triggerID" : "1b8fd27a6fbed918ac5bc5094a08dd8f01ddbda8",
"triggerType" : "PUSH"
}, {
"hash" : "22132d2333e9057bce866ce03a2d7b289af8c5c5",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=11927",
"triggerID" : "22132d2333e9057bce866ce03a2d7b289af8c5c5",
"triggerType" : "PUSH"
}, {
"hash" : "666b6690b531d1540b196f0f97e175c7da38166d",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12379",
"triggerID" : "666b6690b531d1540b196f0f97e175c7da38166d",
"triggerType" : "PUSH"
}, {
"hash" : "212fbe84247891d11bb0e1a12fe454b34a270c75",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12406",
"triggerID" : "212fbe84247891d11bb0e1a12fe454b34a270c75",
"triggerType" : "PUSH"
}, {
"hash" : "1f5f43dab710b2b891ac35762f21321d1f5e59b6",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "1f5f43dab710b2b891ac35762f21321d1f5e59b6",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 666b6690b531d1540b196f0f97e175c7da38166d Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12379)
* 212fbe84247891d11bb0e1a12fe454b34a270c75 Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12406)
* 1f5f43dab710b2b891ac35762f21321d1f5e59b6 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] nsivabalan commented on a diff in pull request #6705: [HUDI-4868] Fixed the issue that compaction is invalid when the last commit action is replace commit.
Posted by GitBox <gi...@apache.org>.
nsivabalan commented on code in PR #6705:
URL: https://github.com/apache/hudi/pull/6705#discussion_r999030392
##########
hudi-common/src/main/java/org/apache/hudi/common/util/CompactionUtils.java:
##########
@@ -214,7 +216,7 @@ public static List<HoodieInstant> getPendingCompactionInstantTimes(HoodieTableMe
*/
public static Option<Pair<HoodieTimeline, HoodieInstant>> getDeltaCommitsSinceLatestCompaction(
HoodieActiveTimeline activeTimeline) {
- Option<HoodieInstant> lastCompaction = activeTimeline.getCommitTimeline()
+ Option<HoodieInstant> lastCompaction = activeTimeline.getCommitTimeline().filter(s -> !s.getAction().equals(REPLACE_COMMIT_ACTION))
Review Comment:
replace_commit could also mean INSERT_OVERRIDE, INSERT_OVERRIDE_TABLE and DELETE_PARTITIONS. Can we consider those cases as well please
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #6705: [HUDI-4868] Fixed the issue that compaction is invalid when the last commit action is replace commit.
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on PR #6705:
URL: https://github.com/apache/hudi/pull/6705#issuecomment-1285092426
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "1b8fd27a6fbed918ac5bc5094a08dd8f01ddbda8",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=11453",
"triggerID" : "1b8fd27a6fbed918ac5bc5094a08dd8f01ddbda8",
"triggerType" : "PUSH"
}, {
"hash" : "22132d2333e9057bce866ce03a2d7b289af8c5c5",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=11927",
"triggerID" : "22132d2333e9057bce866ce03a2d7b289af8c5c5",
"triggerType" : "PUSH"
}, {
"hash" : "666b6690b531d1540b196f0f97e175c7da38166d",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12379",
"triggerID" : "666b6690b531d1540b196f0f97e175c7da38166d",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 22132d2333e9057bce866ce03a2d7b289af8c5c5 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=11927)
* 666b6690b531d1540b196f0f97e175c7da38166d Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12379)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] bvaradar commented on pull request #6705: [HUDI-4868] Fixed the issue that compaction is invalid when the last commit action is replace commit.
Posted by "bvaradar (via GitHub)" <gi...@apache.org>.
bvaradar commented on PR #6705:
URL: https://github.com/apache/hudi/pull/6705#issuecomment-1495309319
@watermelon12138 : Pinging to see if you are interested in updating this PR ?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #6705: [HUDI-4868] Fixed the issue that compaction is invalid when the last commit action is replace commit.
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on PR #6705:
URL: https://github.com/apache/hudi/pull/6705#issuecomment-1263585507
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "1b8fd27a6fbed918ac5bc5094a08dd8f01ddbda8",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=11453",
"triggerID" : "1b8fd27a6fbed918ac5bc5094a08dd8f01ddbda8",
"triggerType" : "PUSH"
}, {
"hash" : "22132d2333e9057bce866ce03a2d7b289af8c5c5",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=11927",
"triggerID" : "22132d2333e9057bce866ce03a2d7b289af8c5c5",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 22132d2333e9057bce866ce03a2d7b289af8c5c5 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=11927)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #6705: [HUDI-4868] Fixed the issue that compaction is invalid when the last commit action is replace commit.
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on PR #6705:
URL: https://github.com/apache/hudi/pull/6705#issuecomment-1250023713
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "1b8fd27a6fbed918ac5bc5094a08dd8f01ddbda8",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "1b8fd27a6fbed918ac5bc5094a08dd8f01ddbda8",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 1b8fd27a6fbed918ac5bc5094a08dd8f01ddbda8 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] nsivabalan commented on pull request #6705: [HUDI-4868] Fixed the issue that compaction is invalid when the last commit action is replace commit.
Posted by GitBox <gi...@apache.org>.
nsivabalan commented on PR #6705:
URL: https://github.com/apache/hudi/pull/6705#issuecomment-1299399185
@watermelon12138 : can you follow up here please whenyou get a chance.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org