You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@pulsar.apache.org by "merlimat (via GitHub)" <gi...@apache.org> on 2023/03/15 03:45:01 UTC
[GitHub] [pulsar] merlimat opened a new pull request, #19817: [fix][metadata-store] Fixed race condition between ResourceLock update and invalidation
merlimat opened a new pull request, #19817:
URL: https://github.com/apache/pulsar/pull/19817
### Motivation
The test `LockManagerTest.updateValueWhenKeyDisappears()` can having infrequent failures, especially when running with Etcd backend.
The problem is really in the logic of the `ResourceLock` itself. After a normal lock-acquire operation, the test is triggering 2 events at the same time:
1. A deletion of the lock key-value (eg: the ephemeral node might have disappeared)
2. A `lock.update()` operation to change the value of the lock.
The race condition is due to the fact that operation (1) will trigger a lock revalidation in background and this now happens at the same time as operation (2).
The result is that it's possible for the ResourceLock to enter into a loop of revalidations and deletions.
eg:
```
2023-03-14T20:40:40,350 - INFO - [-8-1:ResourceLockImpl@180] - Acquired resource lock on /key-9166073612375/1
2023-03-14T20:40:40,376 - INFO - [-8-1:ResourceLockImpl@201] - Lock on resource /key-9166073612375/1 was invalidated
2023-03-14T20:40:40,392 - INFO - [-8-1:ResourceLockImpl@180] - Acquired resource lock on /key-9166073612375/1
2023-03-14T20:40:40,392 - INFO - [-8-1:ResourceLockImpl@268] - Successfully re-acquired missing lock at /key-9166073612375/1
2023-03-14T20:40:40,392 - INFO - [-8-1:ResourceLockImpl@203] - Successfully revalidated the lock on /key-9166073612375/1
2023-03-14T20:40:40,402 - INFO - [-8-1:ResourceLockImpl@201] - Lock on resource /key-9166073612375/1 was invalidated
2023-03-14T20:40:40,407 - INFO - [-8-1:ResourceLockImpl@180] - Acquired resource lock on /key-9166073612375/1
2023-03-14T20:40:40,407 - INFO - [-8-1:ResourceLockImpl@325] - Successfully re-acquired lock at /key-9166073612375/1
2023-03-14T20:40:40,417 - INFO - [-8-1:ResourceLockImpl@201] - Lock on resource /key-9166073612375/1 was invalidated
2023-03-14T20:40:40,422 - INFO - [-8-1:ResourceLockImpl@180] - Acquired resource lock on /key-9166073612375/1
2023-03-14T20:40:40,422 - INFO - [-8-1:ResourceLockImpl@325] - Successfully re-acquired lock at /key-9166073612375/1
2023-03-14T20:40:40,422 - INFO - [-8-1:ResourceLockImpl@203] - Successfully revalidated the lock on /key-9166073612375/1
2023-03-14T20:40:40,433 - INFO - [-8-1:ResourceLockImpl@201] - Lock on resource /key-9166073612375/1 was invalidated
2023-03-14T20:40:40,437 - INFO - [-8-1:ResourceLockImpl@180] - Acquired resource lock on /key-9166073612375/1
2023-03-14T20:40:40,437 - INFO - [-8-1:ResourceLockImpl@325] - Successfully re-acquired lock at /key-9166073612375/1
2023-03-14T20:40:40,437 - INFO - [-8-1:ResourceLockImpl@203] - Successfully revalidated the lock on /key-9166073612375/1
2023-03-14T20:40:40,447 - INFO - [-8-1:ResourceLockImpl@201] - Lock on resource /key-9166073612375/1 was invalidated
2023-03-14T20:40:40,452 - INFO - [-8-1:ResourceLockImpl@180] - Acquired resource lock on /key-9166073612375/1
2023-03-14T20:40:40,452 - INFO - [-8-1:ResourceLockImpl@325] - Successfully re-acquired lock at /key-9166073612375/1
2023-03-14T20:40:40,452 - INFO - [-8-1:ResourceLockImpl@203] - Successfully revalidated the lock on /key-9166073612375/1
2023-03-14T20:40:40,467 - INFO - [-8-1:ResourceLockImpl@201] - Lock on resource /key-9166073612375/1 was invalidated
2023-03-14T20:40:40,472 - INFO - [-8-1:ResourceLockImpl@180] - Acquired resource lock on /key-9166073612375/1
2023-03-14T20:40:40,472 - INFO - [-8-1:ResourceLockImpl@325] - Successfully re-acquired lock at /key-9166073612375/1
2023-03-14T20:40:40,472 - INFO - [-8-1:ResourceLockImpl@203] - Successfully revalidated the lock on /key-9166073612375/1
2023-03-14T20:40:40,482 - INFO - [-8-1:ResourceLockImpl@201] - Lock on resource /key-9166073612375/1 was invalidated
2023-03-14T20:40:40,486 - INFO - [-8-1:ResourceLockImpl@180] - Acquired resource lock on /key-9166073612375/1
2023-03-14T20:40:40,486 - INFO - [-8-1:ResourceLockImpl@325] - Successfully re-acquired lock at /key-9166073612375/1
2023-03-14T20:40:40,487 - INFO - [-8-1:ResourceLockImpl@203] - Successfully revalidated the lock on /key-9166073612375/1
2023-03-14T20:40:40,497 - INFO - [-8-1:ResourceLockImpl@201] - Lock on resource /key-9166073612375/1 was invalidated
....
```
### Modifications
Similarly as to what we're already doing to prevent concurrent revalidate operations, we need to also wait for any existing operations to complete before attempting to do the update.
The same is true if there is an update() operation in progress: the next revalidate should wait.
We then use a single future to signal pending operation and to piggy back and try when it's the right time.
### Verifying this change
- [ ] Make sure that the change passes the CI checks.
*(Please pick either of the following options)*
This change is a trivial rework / code cleanup without any test coverage.
*(or)*
This change is already covered by existing tests, such as *(please describe tests)*.
*(or)*
This change added tests and can be verified as follows:
*(example:)*
- *Added integration tests for end-to-end deployment with large payloads (10MB)*
- *Extended integration test for recovery after broker failure*
### Does this pull request potentially affect one of the following parts:
<!-- DO NOT REMOVE THIS SECTION. CHECK THE PROPER BOX ONLY. -->
*If the box was checked, please highlight the changes*
- [ ] Dependencies (add or upgrade a dependency)
- [ ] The public API
- [ ] The schema
- [ ] The default values of configurations
- [ ] The threading model
- [ ] The binary protocol
- [ ] The REST endpoints
- [ ] The admin CLI options
- [ ] The metrics
- [ ] Anything that affects deployment
### Documentation
<!-- DO NOT REMOVE THIS SECTION. CHECK THE PROPER BOX ONLY. -->
- [ ] `doc` <!-- Your PR contains doc changes. -->
- [ ] `doc-required` <!-- Your PR changes impact docs and you will update later -->
- [X] `doc-not-needed` <!-- Your PR changes do not impact docs -->
- [ ] `doc-complete` <!-- Docs have been already added -->
### Matching PR in forked repository
PR in forked repository: <!-- ENTER URL HERE -->
<!--
After opening this PR, the build in apache/pulsar will fail and instructions will
be provided for opening a PR in the PR author's forked repository.
apache/pulsar pull requests should be first tested in your own fork since the
apache/pulsar CI based on GitHub Actions has constrained resources and quota.
GitHub Actions provides separate quota for pull requests that are executed in
a forked repository.
The tests will be run in the forked repository until all PR review comments have
been handled, the tests pass and the PR is approved by a reviewer.
-->
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] mattisonchao commented on pull request #19817: [fix][meta] Fixed race condition between ResourceLock update and invalidation
Posted by "mattisonchao (via GitHub)" <gi...@apache.org>.
mattisonchao commented on PR #19817:
URL: https://github.com/apache/pulsar/pull/19817#issuecomment-1470129090
`LockManagerTest.updateValueWhenKeyDisappears` this test to be unstable. Maybe it was affected by this PR.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] merlimat commented on a diff in pull request #19817: [fix][meta] Fixed race condition between ResourceLock update and invalidation
Posted by "merlimat (via GitHub)" <gi...@apache.org>.
merlimat commented on code in PR #19817:
URL: https://github.com/apache/pulsar/pull/19817#discussion_r1137429475
##########
pulsar-metadata/src/main/java/org/apache/pulsar/metadata/coordination/impl/ResourceLockImpl.java:
##########
@@ -71,7 +72,20 @@ public synchronized T getValue() {
@Override
public synchronized CompletableFuture<Void> updateValue(T newValue) {
- return acquire(newValue);
+ // If there is an operation in progress, we're going to let it complete before attempting to
+ // update the value
+ pendingOperationFuture = pendingOperationFuture.thenCompose(v -> {
Review Comment:
That's correct, I think we're missing to reset it once it gets completed once.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] mattisonchao commented on a diff in pull request #19817: [fix][meta] Fixed race condition between ResourceLock update and invalidation
Posted by "mattisonchao (via GitHub)" <gi...@apache.org>.
mattisonchao commented on code in PR #19817:
URL: https://github.com/apache/pulsar/pull/19817#discussion_r1137188090
##########
pulsar-metadata/src/main/java/org/apache/pulsar/metadata/coordination/impl/ResourceLockImpl.java:
##########
@@ -71,7 +72,20 @@ public synchronized T getValue() {
@Override
public synchronized CompletableFuture<Void> updateValue(T newValue) {
- return acquire(newValue);
+ // If there is an operation in progress, we're going to let it complete before attempting to
+ // update the value
+ pendingOperationFuture = pendingOperationFuture.thenCompose(v -> {
Review Comment:
`pendingOperationFuture` may throw exceptions `LockBusyException`, `BadVersionException` or other connectivity issues and return errors directly. Do we need to check that the relevant usage logic has handled this exception?
##########
pulsar-metadata/src/main/java/org/apache/pulsar/metadata/coordination/impl/ResourceLockImpl.java:
##########
@@ -71,7 +72,20 @@ public synchronized T getValue() {
@Override
public synchronized CompletableFuture<Void> updateValue(T newValue) {
- return acquire(newValue);
+ // If there is an operation in progress, we're going to let it complete before attempting to
+ // update the value
+ pendingOperationFuture = pendingOperationFuture.thenCompose(v -> {
+ synchronized (ResourceLockImpl.this) {
+ if (state != State.Valid) {
+ return CompletableFuture.failedFuture(
Review Comment:
We have returned a new exception. Do we need to change the relevant exception catch?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] merlimat merged pull request #19817: [fix][meta] Fixed race condition between ResourceLock update and invalidation
Posted by "merlimat (via GitHub)" <gi...@apache.org>.
merlimat merged PR #19817:
URL: https://github.com/apache/pulsar/pull/19817
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] mattisonchao commented on a diff in pull request #19817: [fix][meta] Fixed race condition between ResourceLock update and invalidation
Posted by "mattisonchao (via GitHub)" <gi...@apache.org>.
mattisonchao commented on code in PR #19817:
URL: https://github.com/apache/pulsar/pull/19817#discussion_r1139014991
##########
pulsar-metadata/src/main/java/org/apache/pulsar/metadata/coordination/impl/ResourceLockImpl.java:
##########
@@ -185,39 +203,47 @@ synchronized void lockWasInvalidated() {
}
log.info("Lock on resource {} was invalidated", path);
- revalidate(value, true)
+ revalidate(value, true, true)
.thenRun(() -> log.info("Successfully revalidated the lock on {}", path));
}
synchronized CompletableFuture<Void> revalidateIfNeededAfterReconnection() {
if (revalidateAfterReconnection) {
revalidateAfterReconnection = false;
log.warn("Revalidate lock at {} after reconnection", path);
- return revalidate(value, true);
+ return revalidate(value, true, true);
} else {
return CompletableFuture.completedFuture(null);
}
}
- synchronized CompletableFuture<Void> revalidate(T newValue, boolean revalidateAfterReconnection) {
- if (revalidateFuture == null || revalidateFuture.isDone()) {
- revalidateFuture = doRevalidate(newValue);
+ synchronized CompletableFuture<Void> revalidate(T newValue, boolean trackPendingOperation,
+ boolean revalidateAfterReconnection) {
+
+ final CompletableFuture<Void> trackFuture;
+
+ if (!trackPendingOperation) {
+ trackFuture = doRevalidate(newValue);
+ } else if (pendingOperationFuture.isDone()) {
+ pendingOperationFuture = doRevalidate(newValue);
+ trackFuture = pendingOperationFuture;
} else {
if (log.isDebugEnabled()) {
log.debug("Previous revalidating is not finished while revalidate newValue={}, value={}, version={}",
newValue, value, version);
}
- CompletableFuture<Void> newFuture = new CompletableFuture<>();
- revalidateFuture.whenComplete((unused, throwable) -> {
- doRevalidate(newValue).thenRun(() -> newFuture.complete(null))
+ trackFuture = new CompletableFuture<>();
+ trackFuture.whenComplete((unused, throwable) -> {
Review Comment:
https://github.com/apache/pulsar/pull/19844 is trying to perfect it.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] Technoboy- commented on pull request #19817: [fix][meta] Fixed race condition between ResourceLock update and invalidation
Posted by "Technoboy- (via GitHub)" <gi...@apache.org>.
Technoboy- commented on PR #19817:
URL: https://github.com/apache/pulsar/pull/19817#issuecomment-1470083000
/pulsarbot run-failure-checks
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] mattisonchao commented on a diff in pull request #19817: [fix][meta] Fixed race condition between ResourceLock update and invalidation
Posted by "mattisonchao (via GitHub)" <gi...@apache.org>.
mattisonchao commented on code in PR #19817:
URL: https://github.com/apache/pulsar/pull/19817#discussion_r1138014030
##########
pulsar-metadata/src/main/java/org/apache/pulsar/metadata/coordination/impl/ResourceLockImpl.java:
##########
@@ -185,39 +203,47 @@ synchronized void lockWasInvalidated() {
}
log.info("Lock on resource {} was invalidated", path);
- revalidate(value, true)
+ revalidate(value, true, true)
.thenRun(() -> log.info("Successfully revalidated the lock on {}", path));
}
synchronized CompletableFuture<Void> revalidateIfNeededAfterReconnection() {
if (revalidateAfterReconnection) {
revalidateAfterReconnection = false;
log.warn("Revalidate lock at {} after reconnection", path);
- return revalidate(value, true);
+ return revalidate(value, true, true);
} else {
return CompletableFuture.completedFuture(null);
}
}
- synchronized CompletableFuture<Void> revalidate(T newValue, boolean revalidateAfterReconnection) {
- if (revalidateFuture == null || revalidateFuture.isDone()) {
- revalidateFuture = doRevalidate(newValue);
+ synchronized CompletableFuture<Void> revalidate(T newValue, boolean trackPendingOperation,
+ boolean revalidateAfterReconnection) {
+
+ final CompletableFuture<Void> trackFuture;
+
+ if (!trackPendingOperation) {
+ trackFuture = doRevalidate(newValue);
+ } else if (pendingOperationFuture.isDone()) {
+ pendingOperationFuture = doRevalidate(newValue);
+ trackFuture = pendingOperationFuture;
} else {
if (log.isDebugEnabled()) {
log.debug("Previous revalidating is not finished while revalidate newValue={}, value={}, version={}",
newValue, value, version);
}
- CompletableFuture<Void> newFuture = new CompletableFuture<>();
- revalidateFuture.whenComplete((unused, throwable) -> {
- doRevalidate(newValue).thenRun(() -> newFuture.complete(null))
+ trackFuture = new CompletableFuture<>();
+ trackFuture.whenComplete((unused, throwable) -> {
Review Comment:
It is wrong... the track future will always be incomplete. :(
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org