You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@bookkeeper.apache.org by GitBox <gi...@apache.org> on 2022/02/21 07:15:44 UTC
[GitHub] [bookkeeper] shustsud opened a new issue #3070: checkAllLedgers gets stuck when read throttling is enabled.
shustsud opened a new issue #3070:
URL: https://github.com/apache/bookkeeper/issues/3070
**BUG REPORT**
***Describe the bug***
checkAllLedgers gets stuck when read throttling is enabled.
***To Reproduce***
Run TestLedgerChecker with low read throttle value, i.e.,
1. Modify `LedgerChecker checker = new LedgerChecker(bkc);` to `LedgerChecker checker = new LedgerChecker(bkc, 1);`.
https://github.com/apache/bookkeeper/blob/d4d0060bc38995777da5cc2363c7af897a908f27/bookkeeper-server/src/test/java/org/apache/bookkeeper/client/TestLedgerChecker.java#L521
2. Run TestLedgerChecker#testSingleEntryAfterEnsembleChange.
https://github.com/apache/bookkeeper/blob/d4d0060bc38995777da5cc2363c7af897a908f27/bookkeeper-server/src/test/java/org/apache/bookkeeper/client/TestLedgerChecker.java#L358
3. The execution results are as follows.
```
16:02:49.489 [Time-limited test] ERROR org.apache.bookkeeper.bookie.Journal - Problems reading from <path> (this is okay if it is the first time starting this bookie
...
16:03:01.509 [Time-limited test] ERROR org.apache.bookkeeper.proto.PerChannelBookieClient - Cannot connect to <ip>:<port> as endpoint resolution failed (probably bookie is down) err org.apache.bookkeeper.proto.BookieAddressResolver$BookieIdNotResolvedException: Cannot resolve bookieId <ip>:<port>, bookie does not exist or it is not running
test timed out after 120 seconds
org.junit.runners.model.TestTimedOutException: test timed out after 120 seconds
at sun.misc.Unsafe.park(Native Method)
at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
at java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:836)
at java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedInterruptibly(AbstractQueuedSynchronizer.java:997)
at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireSharedInterruptibly(AbstractQueuedSynchronizer.java:1304)
at java.util.concurrent.CountDownLatch.await(CountDownLatch.java:231)
at org.apache.bookkeeper.client.TestLedgerChecker$CheckerCallback.waitAndGetResult(TestLedgerChecker.java:68)
at org.apache.bookkeeper.client.TestLedgerChecker.getUnderReplicatedFragments(TestLedgerChecker.java:525)
at org.apache.bookkeeper.client.TestLedgerChecker.testSingleEntryAfterEnsembleChange(TestLedgerChecker.java:387)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50)
at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47)
at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26)
at org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27)
at org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:298)
at org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:292)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.lang.Thread.run(Thread.java:748)
```
***Cause***
The lock on the semaphore is acquired by main thread or also `BookKeeperClientWorker-OrderedExecutor` thread.
https://github.com/apache/bookkeeper/blob/525a4a03f99afe1dca120477273537a6db46873c/bookkeeper-server/src/main/java/org/apache/bookkeeper/client/LedgerChecker.java#L452
https://github.com/apache/bookkeeper/blob/525a4a03f99afe1dca120477273537a6db46873c/bookkeeper-server/src/main/java/org/apache/bookkeeper/client/LedgerChecker.java#L430-L431
The lock on the semaphore is released by `BookKeeperClientWorker-OrderedExecutor` thread.
https://github.com/apache/bookkeeper/blob/525a4a03f99afe1dca120477273537a6db46873c/bookkeeper-server/src/main/java/org/apache/bookkeeper/client/LedgerChecker.java#L82
https://github.com/apache/bookkeeper/blob/525a4a03f99afe1dca120477273537a6db46873c/bookkeeper-server/src/main/java/org/apache/bookkeeper/client/LedgerChecker.java#L316
So if all `BookKeeperClientWorker-OrderedExecutor` threads are waiting to acquire the lock, there will be no thread to release the lock.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@bookkeeper.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [bookkeeper] shustsud commented on issue #3070: checkAllLedgers gets stuck when read throttling is enabled.
Posted by GitBox <gi...@apache.org>.
shustsud commented on issue #3070:
URL: https://github.com/apache/bookkeeper/issues/3070#issuecomment-1049446938
@lordcheng10
readEntryComplete(callback) is run in the BookKeeperClientWorker-OrderedExecutor thread.
```
https://github.com/apache/bookkeeper/blob/525a4a03f99afe1dca120477273537a6db46873c/bookkeeper-server/src/main/java/org/apache/bookkeeper/client/LedgerChecker.java#L314
```
Then, BookKeeperClientWorker-OrderedExecutor thread acquires the lock of semaphore.
```
https://github.com/apache/bookkeeper/blob/525a4a03f99afe1dca120477273537a6db46873c/bookkeeper-server/src/main/java/org/apache/bookkeeper/client/LedgerChecker.java#L323
-> https://github.com/apache/bookkeeper/blob/525a4a03f99afe1dca120477273537a6db46873c/bookkeeper-server/src/main/java/org/apache/bookkeeper/client/LedgerChecker.java#L430-L431
-> https://github.com/apache/bookkeeper/blob/525a4a03f99afe1dca120477273537a6db46873c/bookkeeper-server/src/main/java/org/apache/bookkeeper/client/LedgerChecker.java#L469
-> https://github.com/apache/bookkeeper/blob/525a4a03f99afe1dca120477273537a6db46873c/bookkeeper-server/src/main/java/org/apache/bookkeeper/client/LedgerChecker.java#L208
-> https://github.com/apache/bookkeeper/blob/525a4a03f99afe1dca120477273537a6db46873c/bookkeeper-server/src/main/java/org/apache/bookkeeper/client/LedgerChecker.java#L246
or
-> https://github.com/apache/bookkeeper/blob/525a4a03f99afe1dca120477273537a6db46873c/bookkeeper-server/src/main/java/org/apache/bookkeeper/client/LedgerChecker.java#L291
```
If all BookKeeperClientWorker-OrderedExecutor threads are waiting to acquire the lock, there are no available BookKeeperClientWorker-OrderedExecutor threads, so readEntryComplete(callback) will not be run.
If readEntryComplete(callback) is not run, the lock on semaphore is not released, so I think checkAllLedgers gets stuck.
```
https://github.com/apache/bookkeeper/blob/525a4a03f99afe1dca120477273537a6db46873c/bookkeeper-server/src/main/java/org/apache/bookkeeper/client/LedgerChecker.java#L80-L82
https://github.com/apache/bookkeeper/blob/525a4a03f99afe1dca120477273537a6db46873c/bookkeeper-server/src/main/java/org/apache/bookkeeper/client/LedgerChecker.java#L314-L316
```
If you set a low value for read throttle(inFlightReadEntryNumInLedgerChecker) and also a low value for the number of threads in BookKeeperClientWorker-OrderedExecutor(numWorkerThreads?), this problem will be easy to reproduce.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@bookkeeper.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [bookkeeper] shustsud edited a comment on issue #3070: checkAllLedgers gets stuck when read throttling is enabled.
Posted by GitBox <gi...@apache.org>.
shustsud edited a comment on issue #3070:
URL: https://github.com/apache/bookkeeper/issues/3070#issuecomment-1049446938
@lordcheng10
readEntryComplete(callback) is run in the BookKeeperClientWorker-OrderedExecutor thread.
```
https://github.com/apache/bookkeeper/blob/525a4a03f99afe1dca120477273537a6db46873c/bookkeeper-server/src/main/java/org/apache/bookkeeper/client/LedgerChecker.java#L314
```
Then, BookKeeperClientWorker-OrderedExecutor thread acquires the lock of semaphore.
```
https://github.com/apache/bookkeeper/blob/525a4a03f99afe1dca120477273537a6db46873c/bookkeeper-server/src/main/java/org/apache/bookkeeper/client/LedgerChecker.java#L323
-> https://github.com/apache/bookkeeper/blob/525a4a03f99afe1dca120477273537a6db46873c/bookkeeper-server/src/main/java/org/apache/bookkeeper/client/LedgerChecker.java#L430-L431
-> https://github.com/apache/bookkeeper/blob/525a4a03f99afe1dca120477273537a6db46873c/bookkeeper-server/src/main/java/org/apache/bookkeeper/client/LedgerChecker.java#L469
-> https://github.com/apache/bookkeeper/blob/525a4a03f99afe1dca120477273537a6db46873c/bookkeeper-server/src/main/java/org/apache/bookkeeper/client/LedgerChecker.java#L208
-> https://github.com/apache/bookkeeper/blob/525a4a03f99afe1dca120477273537a6db46873c/bookkeeper-server/src/main/java/org/apache/bookkeeper/client/LedgerChecker.java#L246
or
-> https://github.com/apache/bookkeeper/blob/525a4a03f99afe1dca120477273537a6db46873c/bookkeeper-server/src/main/java/org/apache/bookkeeper/client/LedgerChecker.java#L291
```
If all BookKeeperClientWorker-OrderedExecutor threads are waiting to acquire the lock, there are no available BookKeeperClientWorker-OrderedExecutor threads, so readEntryComplete(callback) will not be run.
If readEntryComplete(callback) is not run, the lock on semaphore is not released, so I think checkAllLedgers gets stuck.
```
https://github.com/apache/bookkeeper/blob/525a4a03f99afe1dca120477273537a6db46873c/bookkeeper-server/src/main/java/org/apache/bookkeeper/client/LedgerChecker.java#L80-L82
https://github.com/apache/bookkeeper/blob/525a4a03f99afe1dca120477273537a6db46873c/bookkeeper-server/src/main/java/org/apache/bookkeeper/client/LedgerChecker.java#L314-L316
```
I think this problem can be reproduced by setting a low value for read throttle(inFlightReadEntryNumInLedgerChecker) and also a low value for BookKeeperClientWorker-OrderedExecutor(numWorkerThreads?).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@bookkeeper.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [bookkeeper] shustsud edited a comment on issue #3070: checkAllLedgers gets stuck when read throttling is enabled.
Posted by GitBox <gi...@apache.org>.
shustsud edited a comment on issue #3070:
URL: https://github.com/apache/bookkeeper/issues/3070#issuecomment-1049446938
@lordcheng10
readEntryComplete(callback) is run in the BookKeeperClientWorker-OrderedExecutor thread.
```
https://github.com/apache/bookkeeper/blob/525a4a03f99afe1dca120477273537a6db46873c/bookkeeper-server/src/main/java/org/apache/bookkeeper/client/LedgerChecker.java#L314
```
Then, BookKeeperClientWorker-OrderedExecutor thread acquires the lock of semaphore.
```
https://github.com/apache/bookkeeper/blob/525a4a03f99afe1dca120477273537a6db46873c/bookkeeper-server/src/main/java/org/apache/bookkeeper/client/LedgerChecker.java#L323
-> https://github.com/apache/bookkeeper/blob/525a4a03f99afe1dca120477273537a6db46873c/bookkeeper-server/src/main/java/org/apache/bookkeeper/client/LedgerChecker.java#L430-L431
-> https://github.com/apache/bookkeeper/blob/525a4a03f99afe1dca120477273537a6db46873c/bookkeeper-server/src/main/java/org/apache/bookkeeper/client/LedgerChecker.java#L469
-> https://github.com/apache/bookkeeper/blob/525a4a03f99afe1dca120477273537a6db46873c/bookkeeper-server/src/main/java/org/apache/bookkeeper/client/LedgerChecker.java#L208
-> https://github.com/apache/bookkeeper/blob/525a4a03f99afe1dca120477273537a6db46873c/bookkeeper-server/src/main/java/org/apache/bookkeeper/client/LedgerChecker.java#L246
or
-> https://github.com/apache/bookkeeper/blob/525a4a03f99afe1dca120477273537a6db46873c/bookkeeper-server/src/main/java/org/apache/bookkeeper/client/LedgerChecker.java#L291
```
If all BookKeeperClientWorker-OrderedExecutor threads are waiting to acquire the lock, there are no available BookKeeperClientWorker-OrderedExecutor threads, so readEntryComplete(callback) will not be run.
If readEntryComplete(callback) is not run, the lock on semaphore is not released, so I think checkAllLedgers gets stuck.
```
https://github.com/apache/bookkeeper/blob/525a4a03f99afe1dca120477273537a6db46873c/bookkeeper-server/src/main/java/org/apache/bookkeeper/client/LedgerChecker.java#L80-L82
https://github.com/apache/bookkeeper/blob/525a4a03f99afe1dca120477273537a6db46873c/bookkeeper-server/src/main/java/org/apache/bookkeeper/client/LedgerChecker.java#L314-L316
```
If you set a low value for read throttle(inFlightReadEntryNumInLedgerChecker) and also a low value for the number of threads in BookKeeperClientWorker-OrderedExecutor(numWorkerThreads?), I think this problem will be reproduced.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@bookkeeper.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [bookkeeper] lordcheng10 commented on issue #3070: checkAllLedgers gets stuck when read throttling is enabled.
Posted by GitBox <gi...@apache.org>.
lordcheng10 commented on issue #3070:
URL: https://github.com/apache/bookkeeper/issues/3070#issuecomment-1048102497
What is the throttle configuration.
What is the relationship between each understanding and speed limit? @shustsud
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@bookkeeper.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [bookkeeper] shustsud edited a comment on issue #3070: checkAllLedgers gets stuck when read throttling is enabled.
Posted by GitBox <gi...@apache.org>.
shustsud edited a comment on issue #3070:
URL: https://github.com/apache/bookkeeper/issues/3070#issuecomment-1049446938
@lordcheng10
readEntryComplete(callback) is run in the BookKeeperClientWorker-OrderedExecutor thread.
```
https://github.com/apache/bookkeeper/blob/525a4a03f99afe1dca120477273537a6db46873c/bookkeeper-server/src/main/java/org/apache/bookkeeper/client/LedgerChecker.java#L314
```
Then, BookKeeperClientWorker-OrderedExecutor thread acquires the lock of semaphore.
```
https://github.com/apache/bookkeeper/blob/525a4a03f99afe1dca120477273537a6db46873c/bookkeeper-server/src/main/java/org/apache/bookkeeper/client/LedgerChecker.java#L323
-> https://github.com/apache/bookkeeper/blob/525a4a03f99afe1dca120477273537a6db46873c/bookkeeper-server/src/main/java/org/apache/bookkeeper/client/LedgerChecker.java#L430-L431
-> https://github.com/apache/bookkeeper/blob/525a4a03f99afe1dca120477273537a6db46873c/bookkeeper-server/src/main/java/org/apache/bookkeeper/client/LedgerChecker.java#L469
-> https://github.com/apache/bookkeeper/blob/525a4a03f99afe1dca120477273537a6db46873c/bookkeeper-server/src/main/java/org/apache/bookkeeper/client/LedgerChecker.java#L208
-> https://github.com/apache/bookkeeper/blob/525a4a03f99afe1dca120477273537a6db46873c/bookkeeper-server/src/main/java/org/apache/bookkeeper/client/LedgerChecker.java#L246
or
-> https://github.com/apache/bookkeeper/blob/525a4a03f99afe1dca120477273537a6db46873c/bookkeeper-server/src/main/java/org/apache/bookkeeper/client/LedgerChecker.java#L291
```
If all BookKeeperClientWorker-OrderedExecutor threads are waiting to acquire the lock, there are no available BookKeeperClientWorker-OrderedExecutor threads, so readEntryComplete(callback) will not be run.
If readEntryComplete(callback) is not run, the lock on semaphore is not released, so I think checkAllLedgers gets stuck.
```
https://github.com/apache/bookkeeper/blob/525a4a03f99afe1dca120477273537a6db46873c/bookkeeper-server/src/main/java/org/apache/bookkeeper/client/LedgerChecker.java#L80-L82
https://github.com/apache/bookkeeper/blob/525a4a03f99afe1dca120477273537a6db46873c/bookkeeper-server/src/main/java/org/apache/bookkeeper/client/LedgerChecker.java#L314-L316
```
I think this problem can be reproduced by setting a low value for read throttle(inFlightReadEntryNumInLedgerChecker) and also a low value for the number of threads in BookKeeperClientWorker-OrderedExecutor(numWorkerThreads?).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@bookkeeper.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org