You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@hudi.apache.org by "xicm (via GitHub)" <gi...@apache.org> on 2023/02/27 07:21:11 UTC
[GitHub] [hudi] xicm opened a new pull request, #8057: [HUDI-5831] fix bug of early commit conflict detection
xicm opened a new pull request, #8057:
URL: https://github.com/apache/hudi/pull/8057
### Change Logs
We should use partition + fileId when we do conflict detection.
### Impact
none
### Risk level (write none, low medium or high below)
none
### Documentation Update
_Describe any necessary documentation update if there is any new feature, config, or user-facing change_
- _The config description must be updated if new configs are added or the default value of the configs are changed_
- _Any new feature or user-facing change requires updating the Hudi website. Please create a Jira ticket, attach the
ticket number here and follow the [instruction](https://hudi.apache.org/contribute/developer-setup#website) to make
changes to the website._
### Contributor's checklist
- [ ] Read through [contributor's guide](https://hudi.apache.org/contribute/how-to-contribute)
- [ ] Change Logs and Impact were stated clearly
- [ ] Adequate tests were added if applicable
- [ ] CI passed
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #8057: [HUDI-5831] fix bug of early commit conflict detection
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #8057:
URL: https://github.com/apache/hudi/pull/8057#issuecomment-1446411260
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "5f4c47b8e27fe1b9a307313eb1f4763473ad09da",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=15430",
"triggerID" : "5f4c47b8e27fe1b9a307313eb1f4763473ad09da",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 5f4c47b8e27fe1b9a307313eb1f4763473ad09da Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=15430)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] danny0405 commented on a diff in pull request #8057: [HUDI-5831] fix bug of early commit conflict detection
Posted by "danny0405 (via GitHub)" <gi...@apache.org>.
danny0405 commented on code in PR #8057:
URL: https://github.com/apache/hudi/pull/8057#discussion_r1118432362
##########
hudi-common/src/main/java/org/apache/hudi/common/util/MarkerUtils.java:
##########
@@ -269,7 +269,7 @@ public static boolean hasCommitConflict(HoodieActiveTimeline activeTimeline, Set
Set<String> missingFileIDs = currentInstants.stream().flatMap(instant -> {
try {
return HoodieCommitMetadata.fromBytes(activeTimeline.getInstantDetails(instant).get(), HoodieCommitMetadata.class)
- .getFileIdAndRelativePaths().keySet().stream();
+ .getFileIdAndRelativePaths().values().stream().map(MarkerUtils::makerToPartitionAndFileID);
} catch (Exception e) {
Review Comment:
Any tests here?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] zhangyue19921010 commented on pull request #8057: [HUDI-5831] Address flakiness of early conflict detection tests
Posted by "zhangyue19921010 (via GitHub)" <gi...@apache.org>.
zhangyue19921010 commented on PR #8057:
URL: https://github.com/apache/hudi/pull/8057#issuecomment-1457701538
Acked! Will take a look asap.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #8057: [HUDI-5831] Address flakiness of early conflict detection tests
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #8057:
URL: https://github.com/apache/hudi/pull/8057#issuecomment-1453246416
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "5f4c47b8e27fe1b9a307313eb1f4763473ad09da",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=15430",
"triggerID" : "5f4c47b8e27fe1b9a307313eb1f4763473ad09da",
"triggerType" : "PUSH"
}, {
"hash" : "53dfcaf73d95139b7bfd69a3890499aba4db4210",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=15550",
"triggerID" : "53dfcaf73d95139b7bfd69a3890499aba4db4210",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 5f4c47b8e27fe1b9a307313eb1f4763473ad09da Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=15430)
* 53dfcaf73d95139b7bfd69a3890499aba4db4210 Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=15550)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #8057: [HUDI-5831] Address flakiness of early conflict detection tests
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #8057:
URL: https://github.com/apache/hudi/pull/8057#issuecomment-1476292710
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "5f4c47b8e27fe1b9a307313eb1f4763473ad09da",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=15430",
"triggerID" : "5f4c47b8e27fe1b9a307313eb1f4763473ad09da",
"triggerType" : "PUSH"
}, {
"hash" : "53dfcaf73d95139b7bfd69a3890499aba4db4210",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=15550",
"triggerID" : "53dfcaf73d95139b7bfd69a3890499aba4db4210",
"triggerType" : "PUSH"
}, {
"hash" : "abe2c83ae0a882843653ad2e0f4e16831c958eb3",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=15804",
"triggerID" : "abe2c83ae0a882843653ad2e0f4e16831c958eb3",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* abe2c83ae0a882843653ad2e0f4e16831c958eb3 Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=15804)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] xicm commented on a diff in pull request #8057: [HUDI-5831] fix bug of early commit conflict detection
Posted by "xicm (via GitHub)" <gi...@apache.org>.
xicm commented on code in PR #8057:
URL: https://github.com/apache/hudi/pull/8057#discussion_r1118485563
##########
hudi-common/src/main/java/org/apache/hudi/common/util/MarkerUtils.java:
##########
@@ -269,7 +269,7 @@ public static boolean hasCommitConflict(HoodieActiveTimeline activeTimeline, Set
Set<String> missingFileIDs = currentInstants.stream().flatMap(instant -> {
try {
return HoodieCommitMetadata.fromBytes(activeTimeline.getInstantDetails(instant).get(), HoodieCommitMetadata.class)
- .getFileIdAndRelativePaths().keySet().stream();
+ .getFileIdAndRelativePaths().values().stream().map(MarkerUtils::makerToPartitionAndFileID);
} catch (Exception e) {
Review Comment:
I expected this test to be covered by testHoodieClientBasicMultiWriterWithEarlyConflictDetection, but it wasn't.
And I find we use partition+fileId in AsyncTimelineServerBasedDetectionStrategy when we detect conflict, but fileId in SimpleDirectMarkerBasedDetectionStrategy. we should use partition+fileId both, all right?
I will go into this.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] danny0405 commented on pull request #8057: [HUDI-5831] Address flakiness of early conflict detection tests
Posted by "danny0405 (via GitHub)" <gi...@apache.org>.
danny0405 commented on PR #8057:
URL: https://github.com/apache/hudi/pull/8057#issuecomment-1457661628
@zhangyue19921010 Can you help for the reviewing here?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #8057: [HUDI-5831] fix bug of early commit conflict detection
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #8057:
URL: https://github.com/apache/hudi/pull/8057#issuecomment-1445836873
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "5f4c47b8e27fe1b9a307313eb1f4763473ad09da",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "5f4c47b8e27fe1b9a307313eb1f4763473ad09da",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 5f4c47b8e27fe1b9a307313eb1f4763473ad09da UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] xicm commented on a diff in pull request #8057: [HUDI-5831] Address flakiness of early conflict detection tests
Posted by "xicm (via GitHub)" <gi...@apache.org>.
xicm commented on code in PR #8057:
URL: https://github.com/apache/hudi/pull/8057#discussion_r1124146982
##########
hudi-client/hudi-spark-client/src/test/java/org/apache/hudi/client/TestHoodieClientMultiWriter.java:
##########
@@ -834,7 +834,7 @@ private HoodieWriteConfig buildWriteConfigForEarlyConflictDetect(String markerTy
// the running interval of the async conflict detector so that the conflict can
// be detected before the marker requests are processed at the timeline server
// in the test.
- .withMarkersTimelineServerBasedBatchIntervalMs(1000)
+ .withMarkersTimelineServerBasedBatchIntervalMs(10000)
Review Comment:
I run the test locally and find that we may call hasMarkerConflict before first detection is finished in AsyncTimelineServerBasedDetectionStrategy. So I increase the delay of schedule the marker creation request.
It will be better if we have a mechanism to ensure returning detection result after first detection is finished.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] xicm commented on a diff in pull request #8057: [HUDI-5831] Address flakiness of early conflict detection tests
Posted by "xicm (via GitHub)" <gi...@apache.org>.
xicm commented on code in PR #8057:
URL: https://github.com/apache/hudi/pull/8057#discussion_r1124146982
##########
hudi-client/hudi-spark-client/src/test/java/org/apache/hudi/client/TestHoodieClientMultiWriter.java:
##########
@@ -834,7 +834,7 @@ private HoodieWriteConfig buildWriteConfigForEarlyConflictDetect(String markerTy
// the running interval of the async conflict detector so that the conflict can
// be detected before the marker requests are processed at the timeline server
// in the test.
- .withMarkersTimelineServerBasedBatchIntervalMs(1000)
+ .withMarkersTimelineServerBasedBatchIntervalMs(10000)
Review Comment:
I run the test locally and find that we may call hasMarkerConflict before first detection is finished in AsyncTimelineServerBasedDetectionStrategy. So I increase the delay of schedule the marker creation request.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #8057: [HUDI-5831] Address flakiness of early conflict detection tests
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #8057:
URL: https://github.com/apache/hudi/pull/8057#issuecomment-1453204362
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "5f4c47b8e27fe1b9a307313eb1f4763473ad09da",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=15430",
"triggerID" : "5f4c47b8e27fe1b9a307313eb1f4763473ad09da",
"triggerType" : "PUSH"
}, {
"hash" : "53dfcaf73d95139b7bfd69a3890499aba4db4210",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "53dfcaf73d95139b7bfd69a3890499aba4db4210",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 5f4c47b8e27fe1b9a307313eb1f4763473ad09da Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=15430)
* 53dfcaf73d95139b7bfd69a3890499aba4db4210 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] xicm commented on a diff in pull request #8057: [HUDI-5831] fix bug of early commit conflict detection
Posted by "xicm (via GitHub)" <gi...@apache.org>.
xicm commented on code in PR #8057:
URL: https://github.com/apache/hudi/pull/8057#discussion_r1118485563
##########
hudi-common/src/main/java/org/apache/hudi/common/util/MarkerUtils.java:
##########
@@ -269,7 +269,7 @@ public static boolean hasCommitConflict(HoodieActiveTimeline activeTimeline, Set
Set<String> missingFileIDs = currentInstants.stream().flatMap(instant -> {
try {
return HoodieCommitMetadata.fromBytes(activeTimeline.getInstantDetails(instant).get(), HoodieCommitMetadata.class)
- .getFileIdAndRelativePaths().keySet().stream();
+ .getFileIdAndRelativePaths().values().stream().map(MarkerUtils::makerToPartitionAndFileID);
} catch (Exception e) {
Review Comment:
I expected this test to be overridden by testHoodieClientBasicMultiWriterWithEarlyConflictDetection, but it wasn't.
And I find we use partition+fileId in AsyncTimelineServerBasedDetectionStrategy when we detect conflict, but fileId in SimpleDirectMarkerBasedDetectionStrategy. we should use partition+fileId both, all right?
I will go into this.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #8057: [HUDI-5831] Address flakiness of early conflict detection tests
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #8057:
URL: https://github.com/apache/hudi/pull/8057#issuecomment-1453406951
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "5f4c47b8e27fe1b9a307313eb1f4763473ad09da",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=15430",
"triggerID" : "5f4c47b8e27fe1b9a307313eb1f4763473ad09da",
"triggerType" : "PUSH"
}, {
"hash" : "53dfcaf73d95139b7bfd69a3890499aba4db4210",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=15550",
"triggerID" : "53dfcaf73d95139b7bfd69a3890499aba4db4210",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 53dfcaf73d95139b7bfd69a3890499aba4db4210 Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=15550)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #8057: [HUDI-5831] Address flakiness of early conflict detection tests
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #8057:
URL: https://github.com/apache/hudi/pull/8057#issuecomment-1475714492
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "5f4c47b8e27fe1b9a307313eb1f4763473ad09da",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=15430",
"triggerID" : "5f4c47b8e27fe1b9a307313eb1f4763473ad09da",
"triggerType" : "PUSH"
}, {
"hash" : "53dfcaf73d95139b7bfd69a3890499aba4db4210",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=15550",
"triggerID" : "53dfcaf73d95139b7bfd69a3890499aba4db4210",
"triggerType" : "PUSH"
}, {
"hash" : "abe2c83ae0a882843653ad2e0f4e16831c958eb3",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "abe2c83ae0a882843653ad2e0f4e16831c958eb3",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 53dfcaf73d95139b7bfd69a3890499aba4db4210 Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=15550)
* abe2c83ae0a882843653ad2e0f4e16831c958eb3 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #8057: [HUDI-5831] fix bug of early commit conflict detection
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #8057:
URL: https://github.com/apache/hudi/pull/8057#issuecomment-1445843595
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "5f4c47b8e27fe1b9a307313eb1f4763473ad09da",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=15430",
"triggerID" : "5f4c47b8e27fe1b9a307313eb1f4763473ad09da",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 5f4c47b8e27fe1b9a307313eb1f4763473ad09da Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=15430)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] xicm commented on a diff in pull request #8057: [HUDI-5831] fix bug of early commit conflict detection
Posted by "xicm (via GitHub)" <gi...@apache.org>.
xicm commented on code in PR #8057:
URL: https://github.com/apache/hudi/pull/8057#discussion_r1118485563
##########
hudi-common/src/main/java/org/apache/hudi/common/util/MarkerUtils.java:
##########
@@ -269,7 +269,7 @@ public static boolean hasCommitConflict(HoodieActiveTimeline activeTimeline, Set
Set<String> missingFileIDs = currentInstants.stream().flatMap(instant -> {
try {
return HoodieCommitMetadata.fromBytes(activeTimeline.getInstantDetails(instant).get(), HoodieCommitMetadata.class)
- .getFileIdAndRelativePaths().keySet().stream();
+ .getFileIdAndRelativePaths().values().stream().map(MarkerUtils::makerToPartitionAndFileID);
} catch (Exception e) {
Review Comment:
I expected this test to be covvered by testHoodieClientBasicMultiWriterWithEarlyConflictDetection, but it wasn't.
And I find we use partition+fileId in AsyncTimelineServerBasedDetectionStrategy when we detect conflict, but fileId in SimpleDirectMarkerBasedDetectionStrategy. we should use partition+fileId both, all right?
I will go into this.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] xicm commented on a diff in pull request #8057: [HUDI-5831] Address flakiness of early conflict detection tests
Posted by "xicm (via GitHub)" <gi...@apache.org>.
xicm commented on code in PR #8057:
URL: https://github.com/apache/hudi/pull/8057#discussion_r1124142642
##########
hudi-timeline-service/src/main/java/org/apache/hudi/timeline/service/handlers/marker/MarkerBasedEarlyConflictDetectionRunnable.java:
##########
@@ -109,7 +109,7 @@ public void run() {
currentFileIDs.retainAll(tableFilesIDs);
if (!currentFileIDs.isEmpty()
|| (checkCommitConflict && MarkerUtils.hasCommitConflict(activeTimeline,
- currentInstantAllMarkers.stream().map(MarkerUtils::makerToPartitionAndFileID).collect(Collectors.toSet()), completedCommits))) {
+ currentInstantAllMarkers.stream().map(MarkerUtils::makerToFileID).collect(Collectors.toSet()), completedCommits))) {
Review Comment:
It's hard to create a scenario described in rfc-56, `Writer1 starts writing data at time t1 and finishes at time t3. Writer2 starts writing at time t2, and at time t4 - writer2 tries to create marker for a file fileA which already updated by writer1 to fileAa(updated)`.
But to be consistent with MarkerUtils.hasCommitConflict, the 2nd param should be a set of fileId.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #8057: [HUDI-5831] Address flakiness of early conflict detection tests
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #8057:
URL: https://github.com/apache/hudi/pull/8057#issuecomment-1475726709
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "5f4c47b8e27fe1b9a307313eb1f4763473ad09da",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=15430",
"triggerID" : "5f4c47b8e27fe1b9a307313eb1f4763473ad09da",
"triggerType" : "PUSH"
}, {
"hash" : "53dfcaf73d95139b7bfd69a3890499aba4db4210",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=15550",
"triggerID" : "53dfcaf73d95139b7bfd69a3890499aba4db4210",
"triggerType" : "PUSH"
}, {
"hash" : "abe2c83ae0a882843653ad2e0f4e16831c958eb3",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=15804",
"triggerID" : "abe2c83ae0a882843653ad2e0f4e16831c958eb3",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 53dfcaf73d95139b7bfd69a3890499aba4db4210 Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=15550)
* abe2c83ae0a882843653ad2e0f4e16831c958eb3 Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=15804)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org