You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@pulsar.apache.org by GitBox <gi...@apache.org> on 2022/09/19 09:16:57 UTC
[GitHub] [pulsar] codelipenghui commented on a diff in pull request #17700: [fix][metadata] Cleanup state when lock revalidation gets `LockBusyException`
codelipenghui commented on code in PR #17700:
URL: https://github.com/apache/pulsar/pull/17700#discussion_r974031384
##########
pulsar-metadata/src/test/java/org/apache/pulsar/metadata/LockManagerTest.java:
##########
@@ -293,4 +296,55 @@ public void revalidateLockOnDifferentSession(String provider, Supplier<String> u
assertEquals(new String(store1.get(path2).join().get().getValue()), "\"value-1\"");
});
}
+
+ @Test(dataProvider = "impl")
+ public void testCleanUpStateWhenRevalidationGotLockBusy(String provider, Supplier<String> urlSupplier)
+ throws Exception {
+
+ if (provider.equals("Memory") || provider.equals("RocksDB")) {
+ // Local memory provider doesn't really have the concept of multiple sessions
+ return;
+ }
+
+ @Cleanup
+ MetadataStoreExtended store1 = MetadataStoreExtended.create(urlSupplier.get(),
+ MetadataStoreConfig.builder().build());
+ @Cleanup
+ MetadataStoreExtended store2 = MetadataStoreExtended.create(urlSupplier.get(),
+ MetadataStoreConfig.builder().build());
+
+ @Cleanup
+ CoordinationService cs1 = new CoordinationServiceImpl(store1);
+ @Cleanup
+ LockManager<String> lm1 = cs1.getLockManager(String.class);
+
+ @Cleanup
+ CoordinationService cs2 = new CoordinationServiceImpl(store2);
+ @Cleanup
+ LockManager<String> lm2 = cs2.getLockManager(String.class);
+
+ String path1 = newKey();
+
+ ResourceLock<String> lock1 = lm1.acquireLock(path1, "value-1").join();
+ AtomicReference<ResourceLock<String>> lock2 = new AtomicReference<>();
+ // lock 2 will steal the distributed lock first.
+ Awaitility.await().untilAsserted(()-> {
+ // Ensure steal the lock success.
+ lock2.set(lm2.acquireLock(path1, "value-1").join());
Review Comment:
We don't have any assertions here. Why do we need `untilAsserted`
##########
pulsar-metadata/src/main/java/org/apache/pulsar/metadata/coordination/impl/ResourceLockImpl.java:
##########
@@ -233,6 +217,25 @@ synchronized CompletableFuture<Void> revalidate(T newValue) {
});
revalidateFuture = newFuture;
}
+ revalidateFuture.exceptionally(ex -> {
+ synchronized (ResourceLockImpl.this) {
+ Throwable realCause = FutureUtil.unwrapCompletionException(ex);
+ if (!revalidateAfterReconnection || realCause instanceof BadVersionException
+ || realCause instanceof LockBusyException) {
+ log.warn("Failed to revalidate the lock at {}. Marked as expired", path);
Review Comment:
Since we have 3 cases will reach here, it's better to add more information to the log to let users to understand the real cause.
##########
pulsar-metadata/src/main/java/org/apache/pulsar/metadata/coordination/impl/ResourceLockImpl.java:
##########
@@ -233,6 +217,25 @@ synchronized CompletableFuture<Void> revalidate(T newValue) {
});
revalidateFuture = newFuture;
}
+ revalidateFuture.exceptionally(ex -> {
+ synchronized (ResourceLockImpl.this) {
+ Throwable realCause = FutureUtil.unwrapCompletionException(ex);
+ if (!revalidateAfterReconnection || realCause instanceof BadVersionException
+ || realCause instanceof LockBusyException) {
+ log.warn("Failed to revalidate the lock at {}. Marked as expired", path);
+ state = State.Released;
+ expiredFuture.complete(null);
+ } else {
+ // We failed to revalidate the lock due to connectivity issue
+ // Continue assuming we hold the lock, until we can revalidate it, either
+ // on Reconnected or SessionReestablished events.
+ this.revalidateAfterReconnection = true;
Review Comment:
```suggestion
ResourceLockImpl.this.revalidateAfterReconnection = true;
```
##########
pulsar-metadata/src/test/java/org/apache/pulsar/metadata/LockManagerTest.java:
##########
@@ -293,4 +296,55 @@ public void revalidateLockOnDifferentSession(String provider, Supplier<String> u
assertEquals(new String(store1.get(path2).join().get().getValue()), "\"value-1\"");
});
}
+
+ @Test(dataProvider = "impl")
+ public void testCleanUpStateWhenRevalidationGotLockBusy(String provider, Supplier<String> urlSupplier)
+ throws Exception {
+
+ if (provider.equals("Memory") || provider.equals("RocksDB")) {
+ // Local memory provider doesn't really have the concept of multiple sessions
+ return;
+ }
+
+ @Cleanup
+ MetadataStoreExtended store1 = MetadataStoreExtended.create(urlSupplier.get(),
+ MetadataStoreConfig.builder().build());
+ @Cleanup
+ MetadataStoreExtended store2 = MetadataStoreExtended.create(urlSupplier.get(),
+ MetadataStoreConfig.builder().build());
+
+ @Cleanup
+ CoordinationService cs1 = new CoordinationServiceImpl(store1);
+ @Cleanup
+ LockManager<String> lm1 = cs1.getLockManager(String.class);
+
+ @Cleanup
+ CoordinationService cs2 = new CoordinationServiceImpl(store2);
+ @Cleanup
+ LockManager<String> lm2 = cs2.getLockManager(String.class);
+
+ String path1 = newKey();
+
+ ResourceLock<String> lock1 = lm1.acquireLock(path1, "value-1").join();
+ AtomicReference<ResourceLock<String>> lock2 = new AtomicReference<>();
+ // lock 2 will steal the distributed lock first.
Review Comment:
I'm not sure why we need the `steal lock` operation.
But this is a potential risk. It will introduce too many small ledgers if we encounter this problem
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org