You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@hudi.apache.org by "LinMingQiang (via GitHub)" <gi...@apache.org> on 2023/08/24 02:57:35 UTC
[GitHub] [hudi] LinMingQiang opened a new pull request, #7469: [HUDI-5386] Cleaning conflicts when write concurrency mode is OCC
LinMingQiang opened a new pull request, #7469:
URL: https://github.com/apache/hudi/pull/7469
### Change Logs
getInstantsToRollback should be lock too.
### Impact
occ mode.
### Risk level (write none, low medium or high below)
none
### Documentation Update
none
### Contributor's checklist
- [x] Read through [contributor's guide](https://hudi.apache.org/contribute/how-to-contribute)
- [x] Change Logs and Impact were stated clearly
- [x] Adequate tests were added if applicable
- [x] CI passed
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] nsivabalan commented on a diff in pull request #7469: [HUDI-5386] Cleaning conflicts when write concurrency mode is OCC
Posted by "nsivabalan (via GitHub)" <gi...@apache.org>.
nsivabalan commented on code in PR #7469:
URL: https://github.com/apache/hudi/pull/7469#discussion_r1182035004
##########
hudi-client/hudi-client-common/src/main/java/org/apache/hudi/client/BaseHoodieWriteClient.java:
##########
@@ -897,28 +897,40 @@ public HoodieCleanMetadata clean(String cleanInstantTime, boolean scheduleInline
if (!tableServicesEnabled(config)) {
return null;
}
- final Timer.Context timerContext = metrics.getCleanCtx();
- CleanerUtils.rollbackFailedWrites(config.getFailedWritesCleanPolicy(),
- HoodieTimeline.CLEAN_ACTION, () -> rollbackFailedWrites(skipLocking));
-
- HoodieTable table = createTable(config, hadoopConf);
- if (config.allowMultipleCleans() || !table.getActiveTimeline().getCleanerTimeline().filterInflightsAndRequested().firstInstant().isPresent()) {
- LOG.info("Cleaner started");
- // proceed only if multiple clean schedules are enabled or if there are no pending cleans.
- if (scheduleInline) {
- scheduleTableServiceInternal(cleanInstantTime, Option.empty(), TableServiceType.CLEAN);
- table.getMetaClient().reloadActiveTimeline();
+ HoodieCleanMetadata metadata;
+ HoodieInstant ownerInstant = null;
+ try {
+ if (!skipLocking) {
+ ownerInstant = new HoodieInstant(true, HoodieTimeline.CLEAN_ACTION, cleanInstantTime);
+ this.txnManager.beginTransaction(Option.of(ownerInstant), Option.empty());
+ }
+ final Timer.Context timerContext = metrics.getCleanCtx();
+ CleanerUtils.rollbackFailedWrites(config.getFailedWritesCleanPolicy(),
Review Comment:
even if we decide to go w/ locking, we should try and keep the locking to the mandatory critical section. for eg, we should lock only the rollbackFailedwrites and not the entire clean operation.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] danny0405 commented on a diff in pull request #7469: [HUDI-5386] Cleaning conflicts when write concurrency mode is OCC
Posted by GitBox <gi...@apache.org>.
danny0405 commented on code in PR #7469:
URL: https://github.com/apache/hudi/pull/7469#discussion_r1051848476
##########
hudi-client/hudi-client-common/src/main/java/org/apache/hudi/client/BaseHoodieWriteClient.java:
##########
@@ -897,28 +897,40 @@ public HoodieCleanMetadata clean(String cleanInstantTime, boolean scheduleInline
if (!tableServicesEnabled(config)) {
return null;
}
- final Timer.Context timerContext = metrics.getCleanCtx();
- CleanerUtils.rollbackFailedWrites(config.getFailedWritesCleanPolicy(),
- HoodieTimeline.CLEAN_ACTION, () -> rollbackFailedWrites(skipLocking));
-
- HoodieTable table = createTable(config, hadoopConf);
- if (config.allowMultipleCleans() || !table.getActiveTimeline().getCleanerTimeline().filterInflightsAndRequested().firstInstant().isPresent()) {
- LOG.info("Cleaner started");
Review Comment:
Instead of adding lock, should we disable the cleaning service for one of the job ?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] nsivabalan commented on pull request #7469: [HUDI-5386] Cleaning conflicts when write concurrency mode is OCC
Posted by "nsivabalan (via GitHub)" <gi...@apache.org>.
nsivabalan commented on PR #7469:
URL: https://github.com/apache/hudi/pull/7469#issuecomment-1537547742
hey @LinMingQiang : Do you think even after landing https://github.com/apache/hudi/pull/7826, we still need this locking ?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] nsivabalan commented on pull request #7469: [HUDI-5386] Cleaning conflicts when write concurrency mode is OCC
Posted by "nsivabalan (via GitHub)" <gi...@apache.org>.
nsivabalan commented on PR #7469:
URL: https://github.com/apache/hudi/pull/7469#issuecomment-1471040984
@LinMingQiang : I assume this is intermittant right. i.e we hit this when we have concurrent writers and each triggered cleaning concurrently and incidentally happened to rollback at similar time and one of them might have resulted in NPE or something. But on retries, things should have progressed w/o issues?
can you confirm the behavior.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #7469: [HUDI-5386] Cleaning conflicts when write concurrency mode is OCC
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on PR #7469:
URL: https://github.com/apache/hudi/pull/7469#issuecomment-1352888881
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "91aa32a46e20fe081f7a63973697b6ff81f11551",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "91aa32a46e20fe081f7a63973697b6ff81f11551",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 91aa32a46e20fe081f7a63973697b6ff81f11551 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] LinMingQiang commented on pull request #7469: [HUDI-5386] Cleaning conflicts when write concurrency mode is OCC
Posted by "LinMingQiang (via GitHub)" <gi...@apache.org>.
LinMingQiang commented on PR #7469:
URL: https://github.com/apache/hudi/pull/7469#issuecomment-1538040052
I noticed that clean execution is also unlocked,when multiple writers simultaneously clean up the same instant, it will cause NullPointException.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] danny0405 commented on a diff in pull request #7469: [HUDI-5386] Cleaning conflicts when write concurrency mode is OCC
Posted by GitBox <gi...@apache.org>.
danny0405 commented on code in PR #7469:
URL: https://github.com/apache/hudi/pull/7469#discussion_r1060339901
##########
hudi-client/hudi-client-common/src/main/java/org/apache/hudi/client/BaseHoodieWriteClient.java:
##########
@@ -897,28 +897,40 @@ public HoodieCleanMetadata clean(String cleanInstantTime, boolean scheduleInline
if (!tableServicesEnabled(config)) {
return null;
}
- final Timer.Context timerContext = metrics.getCleanCtx();
- CleanerUtils.rollbackFailedWrites(config.getFailedWritesCleanPolicy(),
- HoodieTimeline.CLEAN_ACTION, () -> rollbackFailedWrites(skipLocking));
-
- HoodieTable table = createTable(config, hadoopConf);
- if (config.allowMultipleCleans() || !table.getActiveTimeline().getCleanerTimeline().filterInflightsAndRequested().firstInstant().isPresent()) {
- LOG.info("Cleaner started");
Review Comment:
I'm worrying about the efficiency. It is not that efficient for concurrent cleaning and not that necessary.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] LinMingQiang commented on a diff in pull request #7469: [HUDI-5386] Cleaning conflicts when write concurrency mode is OCC
Posted by GitBox <gi...@apache.org>.
LinMingQiang commented on code in PR #7469:
URL: https://github.com/apache/hudi/pull/7469#discussion_r1060370374
##########
hudi-client/hudi-client-common/src/main/java/org/apache/hudi/client/BaseHoodieWriteClient.java:
##########
@@ -897,28 +897,40 @@ public HoodieCleanMetadata clean(String cleanInstantTime, boolean scheduleInline
if (!tableServicesEnabled(config)) {
return null;
}
- final Timer.Context timerContext = metrics.getCleanCtx();
- CleanerUtils.rollbackFailedWrites(config.getFailedWritesCleanPolicy(),
- HoodieTimeline.CLEAN_ACTION, () -> rollbackFailedWrites(skipLocking));
-
- HoodieTable table = createTable(config, hadoopConf);
- if (config.allowMultipleCleans() || !table.getActiveTimeline().getCleanerTimeline().filterInflightsAndRequested().firstInstant().isPresent()) {
- LOG.info("Cleaner started");
Review Comment:
Maybe we can add a special clean lock to ensure that only one cleaning service is executing, skipping the current cleaning when the cleaning service is executing, this is more efficient.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] nsivabalan commented on pull request #7469: [HUDI-5386] Cleaning conflicts when write concurrency mode is OCC
Posted by "nsivabalan (via GitHub)" <gi...@apache.org>.
nsivabalan commented on PR #7469:
URL: https://github.com/apache/hudi/pull/7469#issuecomment-1618411933
hey @LinMingQiang : Do you think you can get the patch in ready state in the next day or two. If the patch is ready and has tests, we can try to get it in for 0.14.0.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] nsivabalan commented on pull request #7469: [HUDI-5386] Cleaning conflicts when write concurrency mode is OCC
Posted by "nsivabalan (via GitHub)" <gi...@apache.org>.
nsivabalan commented on PR #7469:
URL: https://github.com/apache/hudi/pull/7469#issuecomment-1530790674
a related PR https://github.com/apache/hudi/pull/7826
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] LinMingQiang commented on a diff in pull request #7469: [HUDI-5386] Cleaning conflicts when write concurrency mode is OCC
Posted by GitBox <gi...@apache.org>.
LinMingQiang commented on code in PR #7469:
URL: https://github.com/apache/hudi/pull/7469#discussion_r1060317606
##########
hudi-client/hudi-client-common/src/main/java/org/apache/hudi/client/BaseHoodieWriteClient.java:
##########
@@ -897,28 +897,40 @@ public HoodieCleanMetadata clean(String cleanInstantTime, boolean scheduleInline
if (!tableServicesEnabled(config)) {
return null;
}
- final Timer.Context timerContext = metrics.getCleanCtx();
- CleanerUtils.rollbackFailedWrites(config.getFailedWritesCleanPolicy(),
- HoodieTimeline.CLEAN_ACTION, () -> rollbackFailedWrites(skipLocking));
-
- HoodieTable table = createTable(config, hadoopConf);
- if (config.allowMultipleCleans() || !table.getActiveTimeline().getCleanerTimeline().filterInflightsAndRequested().firstInstant().isPresent()) {
- LOG.info("Cleaner started");
Review Comment:
We cannot ensure that only one job has the cleaning service enabled.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] LinMingQiang commented on pull request #7469: [HUDI-5386] Cleaning conflicts when write concurrency mode is OCC
Posted by "LinMingQiang (via GitHub)" <gi...@apache.org>.
LinMingQiang commented on PR #7469:
URL: https://github.com/apache/hudi/pull/7469#issuecomment-1537917099
> hey @LinMingQiang : Do you think even after landing #7826, we still need this locking ?
Yes, we still need to lock it up. The current lock is designed to solve the problem of multiple writers simultaneously processing the same rollback instant.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] nsivabalan commented on pull request #7469: [HUDI-5386] Cleaning conflicts when write concurrency mode is OCC
Posted by "nsivabalan (via GitHub)" <gi...@apache.org>.
nsivabalan commented on PR #7469:
URL: https://github.com/apache/hudi/pull/7469#issuecomment-1690616808
if I am not wrong, this is the core problem we are trying to solve.
if there are failed commits and if two concurrent writers try to rollback concurrently we don't have a lock as such.
These complications arise just bcoz, hudi tries to do automatic clean up of failed writes. In other similar systems, you may have to trigger explicit commands to clean up partially failed commits. or coordinate when multiple writers are involved.
Wanted to call it out.
Anyways, coming back to the original issue. Its recommended to disable table services (like cleaner, archival) in all writers except 1. So, we won't end up in such conflicts. These are anyways not latency sensitive. And w/ this approach all other writes will be even more faster since they don't trigger any of these table service and only take care of ingestion.
We do have a table level config to disable all table services
https://hudi.apache.org/docs/configurations/#hoodietableservicesenabled
Having said all this, here is what I feel we could fix this issue.
We can leverage the heartbeats, such that rollback commits also start to emit heartbeats. So, a concurrent writer know if some other writer is concurrently executing the rollback, or whether its in failed state. That way, only one writer will go ahead and execute the rollback while others will step away.
I remember @suryaprasanna wanted to fix this if I am not wrong.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #7469: [HUDI-5386] Cleaning conflicts when write concurrency mode is OCC
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on PR #7469:
URL: https://github.com/apache/hudi/pull/7469#issuecomment-1352896107
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "91aa32a46e20fe081f7a63973697b6ff81f11551",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=13761",
"triggerID" : "91aa32a46e20fe081f7a63973697b6ff81f11551",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 91aa32a46e20fe081f7a63973697b6ff81f11551 Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=13761)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] fengjian428 commented on a diff in pull request #7469: [HUDI-5386] Cleaning conflicts when write concurrency mode is OCC
Posted by GitBox <gi...@apache.org>.
fengjian428 commented on code in PR #7469:
URL: https://github.com/apache/hudi/pull/7469#discussion_r1051750991
##########
hudi-client/hudi-client-common/src/main/java/org/apache/hudi/client/BaseHoodieWriteClient.java:
##########
@@ -897,28 +897,40 @@ public HoodieCleanMetadata clean(String cleanInstantTime, boolean scheduleInline
if (!tableServicesEnabled(config)) {
return null;
}
- final Timer.Context timerContext = metrics.getCleanCtx();
- CleanerUtils.rollbackFailedWrites(config.getFailedWritesCleanPolicy(),
- HoodieTimeline.CLEAN_ACTION, () -> rollbackFailedWrites(skipLocking));
-
- HoodieTable table = createTable(config, hadoopConf);
- if (config.allowMultipleCleans() || !table.getActiveTimeline().getCleanerTimeline().filterInflightsAndRequested().firstInstant().isPresent()) {
- LOG.info("Cleaner started");
- // proceed only if multiple clean schedules are enabled or if there are no pending cleans.
- if (scheduleInline) {
- scheduleTableServiceInternal(cleanInstantTime, Option.empty(), TableServiceType.CLEAN);
- table.getMetaClient().reloadActiveTimeline();
+ HoodieCleanMetadata metadata;
+ HoodieInstant ownerInstant = null;
+ try {
+ if (!skipLocking) {
+ ownerInstant = new HoodieInstant(true, HoodieTimeline.CLEAN_ACTION, cleanInstantTime);
+ this.txnManager.beginTransaction(Option.of(ownerInstant), Option.empty());
Review Comment:
are there any reentrant cases with this lock?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #7469: [HUDI-5386] Cleaning conflicts when write concurrency mode is OCC
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on PR #7469:
URL: https://github.com/apache/hudi/pull/7469#issuecomment-1353502896
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "91aa32a46e20fe081f7a63973697b6ff81f11551",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=13761",
"triggerID" : "91aa32a46e20fe081f7a63973697b6ff81f11551",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 91aa32a46e20fe081f7a63973697b6ff81f11551 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=13761)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] LinMingQiang commented on a diff in pull request #7469: [HUDI-5386] Cleaning conflicts when write concurrency mode is OCC
Posted by GitBox <gi...@apache.org>.
LinMingQiang commented on code in PR #7469:
URL: https://github.com/apache/hudi/pull/7469#discussion_r1060370374
##########
hudi-client/hudi-client-common/src/main/java/org/apache/hudi/client/BaseHoodieWriteClient.java:
##########
@@ -897,28 +897,40 @@ public HoodieCleanMetadata clean(String cleanInstantTime, boolean scheduleInline
if (!tableServicesEnabled(config)) {
return null;
}
- final Timer.Context timerContext = metrics.getCleanCtx();
- CleanerUtils.rollbackFailedWrites(config.getFailedWritesCleanPolicy(),
- HoodieTimeline.CLEAN_ACTION, () -> rollbackFailedWrites(skipLocking));
-
- HoodieTable table = createTable(config, hadoopConf);
- if (config.allowMultipleCleans() || !table.getActiveTimeline().getCleanerTimeline().filterInflightsAndRequested().firstInstant().isPresent()) {
- LOG.info("Cleaner started");
Review Comment:
Maybe we can add a special clean lock to ensure that only one clean service is executing, skipping the current cleaning when the cleaning service is executing, this is more efficient.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] LinMingQiang commented on a diff in pull request #7469: [HUDI-5386] Cleaning conflicts when write concurrency mode is OCC
Posted by GitBox <gi...@apache.org>.
LinMingQiang commented on code in PR #7469:
URL: https://github.com/apache/hudi/pull/7469#discussion_r1060317606
##########
hudi-client/hudi-client-common/src/main/java/org/apache/hudi/client/BaseHoodieWriteClient.java:
##########
@@ -897,28 +897,40 @@ public HoodieCleanMetadata clean(String cleanInstantTime, boolean scheduleInline
if (!tableServicesEnabled(config)) {
return null;
}
- final Timer.Context timerContext = metrics.getCleanCtx();
- CleanerUtils.rollbackFailedWrites(config.getFailedWritesCleanPolicy(),
- HoodieTimeline.CLEAN_ACTION, () -> rollbackFailedWrites(skipLocking));
-
- HoodieTable table = createTable(config, hadoopConf);
- if (config.allowMultipleCleans() || !table.getActiveTimeline().getCleanerTimeline().filterInflightsAndRequested().firstInstant().isPresent()) {
- LOG.info("Cleaner started");
Review Comment:
We cannot ensure that only one job has the cleaning service enabled,so we need to adding lock here.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-5386] Cleaning conflicts when write concurrency mode is OCC [hudi]
Posted by "voonhous (via GitHub) via commits" <co...@hudi.apache.org>.
voonhous closed pull request #7469: [HUDI-5386] Cleaning conflicts when write concurrency mode is OCC
URL: https://github.com/apache/hudi/pull/7469
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] LinMingQiang commented on a diff in pull request #7469: [HUDI-5386] Cleaning conflicts when write concurrency mode is OCC
Posted by "LinMingQiang (via GitHub)" <gi...@apache.org>.
LinMingQiang commented on code in PR #7469:
URL: https://github.com/apache/hudi/pull/7469#discussion_r1184510754
##########
hudi-client/hudi-client-common/src/main/java/org/apache/hudi/client/BaseHoodieWriteClient.java:
##########
@@ -897,28 +897,40 @@ public HoodieCleanMetadata clean(String cleanInstantTime, boolean scheduleInline
if (!tableServicesEnabled(config)) {
return null;
}
- final Timer.Context timerContext = metrics.getCleanCtx();
- CleanerUtils.rollbackFailedWrites(config.getFailedWritesCleanPolicy(),
- HoodieTimeline.CLEAN_ACTION, () -> rollbackFailedWrites(skipLocking));
-
- HoodieTable table = createTable(config, hadoopConf);
- if (config.allowMultipleCleans() || !table.getActiveTimeline().getCleanerTimeline().filterInflightsAndRequested().firstInstant().isPresent()) {
- LOG.info("Cleaner started");
- // proceed only if multiple clean schedules are enabled or if there are no pending cleans.
- if (scheduleInline) {
- scheduleTableServiceInternal(cleanInstantTime, Option.empty(), TableServiceType.CLEAN);
- table.getMetaClient().reloadActiveTimeline();
+ HoodieCleanMetadata metadata;
+ HoodieInstant ownerInstant = null;
+ try {
+ if (!skipLocking) {
+ ownerInstant = new HoodieInstant(true, HoodieTimeline.CLEAN_ACTION, cleanInstantTime);
+ this.txnManager.beginTransaction(Option.of(ownerInstant), Option.empty());
+ }
+ final Timer.Context timerContext = metrics.getCleanCtx();
+ CleanerUtils.rollbackFailedWrites(config.getFailedWritesCleanPolicy(),
Review Comment:
You're right, I'll optimize it.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] LinMingQiang commented on pull request #7469: [HUDI-5386] Cleaning conflicts when write concurrency mode is OCC
Posted by "LinMingQiang (via GitHub)" <gi...@apache.org>.
LinMingQiang commented on PR #7469:
URL: https://github.com/apache/hudi/pull/7469#issuecomment-1537940184
And even if https://github.com/apache/hudi/pull/7826 lands, there will still be problems.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] LinMingQiang commented on pull request #7469: [HUDI-5386] Cleaning conflicts when write concurrency mode is OCC
Posted by "LinMingQiang (via GitHub)" <gi...@apache.org>.
LinMingQiang commented on PR #7469:
URL: https://github.com/apache/hudi/pull/7469#issuecomment-1690915756
> if I am not wrong, this is the core problem we are trying to solve. if there are failed commits and if two concurrent writers try to rollback concurrently we don't have a lock as such.
>
> These complications arise just bcoz, hudi tries to do automatic clean up of failed writes. In other similar systems, you may have to trigger explicit commands to clean up partially failed commits. or coordinate when multiple writers are involved.
>
> Wanted to call it out. Anyways, coming back to the original issue. Its recommended to disable table services (like cleaner, archival) in all writers except 1. So, we won't end up in such conflicts. These are anyways not latency sensitive. And w/ this approach all other writes will be even more faster since they don't trigger any of these table service and only take care of ingestion.
>
> We do have a table level config to disable all table services https://hudi.apache.org/docs/configurations/#hoodietableservicesenabled
>
> Having said all this, here is what I feel we could fix this issue.
>
> We can leverage the heartbeats, such that rollback commits also start to emit heartbeats. So, a concurrent writer know if some other writer is concurrently executing the rollback, or whether its in failed state. That way, only one writer will go ahead and execute the rollback while others will step away.
>
> I remember @suryaprasanna wanted to fix this if I am not wrong.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] LinMingQiang closed pull request #7469: [HUDI-5386] Cleaning conflicts when write concurrency mode is OCC
Posted by "LinMingQiang (via GitHub)" <gi...@apache.org>.
LinMingQiang closed pull request #7469: [HUDI-5386] Cleaning conflicts when write concurrency mode is OCC
URL: https://github.com/apache/hudi/pull/7469
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] LinMingQiang commented on pull request #7469: [HUDI-5386] Cleaning conflicts when write concurrency mode is OCC
Posted by "LinMingQiang (via GitHub)" <gi...@apache.org>.
LinMingQiang commented on PR #7469:
URL: https://github.com/apache/hudi/pull/7469#issuecomment-1690920181
I can get what you mean,but using the heartbeat mechanism might be too heavy. Maybe we can add a ticket and ask the writer who obtains the ticket to perform rollback ,It's like a lock but it doesn't affect other writers.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] xushiyan commented on pull request #7469: [HUDI-5386] Cleaning conflicts when write concurrency mode is OCC
Posted by "xushiyan (via GitHub)" <gi...@apache.org>.
xushiyan commented on PR #7469:
URL: https://github.com/apache/hudi/pull/7469#issuecomment-1554043487
@LinMingQiang would you rebase master and resolve conflict pls?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] LinMingQiang commented on a diff in pull request #7469: [HUDI-5386] Cleaning conflicts when write concurrency mode is OCC
Posted by GitBox <gi...@apache.org>.
LinMingQiang commented on code in PR #7469:
URL: https://github.com/apache/hudi/pull/7469#discussion_r1051755639
##########
hudi-client/hudi-client-common/src/main/java/org/apache/hudi/client/BaseHoodieWriteClient.java:
##########
@@ -897,28 +897,40 @@ public HoodieCleanMetadata clean(String cleanInstantTime, boolean scheduleInline
if (!tableServicesEnabled(config)) {
return null;
}
- final Timer.Context timerContext = metrics.getCleanCtx();
- CleanerUtils.rollbackFailedWrites(config.getFailedWritesCleanPolicy(),
- HoodieTimeline.CLEAN_ACTION, () -> rollbackFailedWrites(skipLocking));
-
- HoodieTable table = createTable(config, hadoopConf);
- if (config.allowMultipleCleans() || !table.getActiveTimeline().getCleanerTimeline().filterInflightsAndRequested().firstInstant().isPresent()) {
- LOG.info("Cleaner started");
- // proceed only if multiple clean schedules are enabled or if there are no pending cleans.
- if (scheduleInline) {
- scheduleTableServiceInternal(cleanInstantTime, Option.empty(), TableServiceType.CLEAN);
- table.getMetaClient().reloadActiveTimeline();
+ HoodieCleanMetadata metadata;
+ HoodieInstant ownerInstant = null;
+ try {
+ if (!skipLocking) {
+ ownerInstant = new HoodieInstant(true, HoodieTimeline.CLEAN_ACTION, cleanInstantTime);
+ this.txnManager.beginTransaction(Option.of(ownerInstant), Option.empty());
Review Comment:
When skipLocking is false, it is changed to true when entering rollbackFailedWrites/clean.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] LinMingQiang commented on a diff in pull request #7469: [HUDI-5386] Cleaning conflicts when write concurrency mode is OCC
Posted by GitBox <gi...@apache.org>.
LinMingQiang commented on code in PR #7469:
URL: https://github.com/apache/hudi/pull/7469#discussion_r1060370374
##########
hudi-client/hudi-client-common/src/main/java/org/apache/hudi/client/BaseHoodieWriteClient.java:
##########
@@ -897,28 +897,40 @@ public HoodieCleanMetadata clean(String cleanInstantTime, boolean scheduleInline
if (!tableServicesEnabled(config)) {
return null;
}
- final Timer.Context timerContext = metrics.getCleanCtx();
- CleanerUtils.rollbackFailedWrites(config.getFailedWritesCleanPolicy(),
- HoodieTimeline.CLEAN_ACTION, () -> rollbackFailedWrites(skipLocking));
-
- HoodieTable table = createTable(config, hadoopConf);
- if (config.allowMultipleCleans() || !table.getActiveTimeline().getCleanerTimeline().filterInflightsAndRequested().firstInstant().isPresent()) {
- LOG.info("Cleaner started");
Review Comment:
Maybe we can add a special clean lock to ensure that only one cleaning service is executing, skipping the current cleaning when other cleaning service is executing, this is more efficient.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] LinMingQiang commented on pull request #7469: [HUDI-5386] Cleaning conflicts when write concurrency mode is OCC
Posted by "LinMingQiang (via GitHub)" <gi...@apache.org>.
LinMingQiang commented on PR #7469:
URL: https://github.com/apache/hudi/pull/7469#issuecomment-1471615902
> @LinMingQiang : I assume this is intermittant right. i.e we hit this when we have concurrent writers and each triggered cleaning concurrently and incidentally happened to rollback at similar time and one of them might have resulted in NPE or something. But on retries, things should have progressed w/o issues? can you confirm the behavior.
You can read this description. https://issues.apache.org/jira/browse/HUDI-5386
I think `getInstantsToRollback` should be locked too.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-5386] Cleaning conflicts when write concurrency mode is OCC [hudi]
Posted by "voonhous (via GitHub) via commits" <co...@hudi.apache.org>.
voonhous commented on PR #7469:
URL: https://github.com/apache/hudi/pull/7469#issuecomment-6700067628
Temporarily closing this PR while we debug a GitHub settings error on master (#20471); it will be reopened shortly, no action needed.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org