You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@bookkeeper.apache.org by GitBox <gi...@apache.org> on 2020/08/14 16:50:52 UTC
[GitHub] [bookkeeper] eolivelli opened a new issue #2398: LedgerOpenOp may perform connection to a bookie inside the ZooKeeper client thread
eolivelli opened a new issue #2398:
URL: https://github.com/apache/bookkeeper/issues/2398
While working on BP-41 I saw this potential problem.
During OpenLedgerOp (openLedger) we are executing blocking I/O operations against the bookies in the main thread of ZooKeeper client,
This may slow down the whole processing of the client because the ZooKeeper client has only one single thread that execute callbacks.
See the stacktrace (it is not an error, it is just a dump)
Suggestions:
- use a separate thread pool in order to execute blocking I/O
```
2020-08-14 18:46:33,190 - INFO - [AuditorBookie-127.0.0.1:37597-EventThread:CachingBookieAddressResolver@50] - resolve 192.0.2.1:1000 current cache {127.0.0.1:44827=127.0.0.1:44827, 127.0.0.1:33607=127.0.0.1:33607, 127.0.0.1:37597=127.0.0.1:37597}
at org.apache.bookkeeper.proto.PerChannelBookieClient.connect(PerChannelBookieClient.java:541)
at org.apache.bookkeeper.proto.PerChannelBookieClient.connectIfNeededAndDoOp(PerChannelBookieClient.java:666)
at org.apache.bookkeeper.proto.DefaultPerChannelBookieClientPool.intialize(DefaultPerChannelBookieClientPool.java:92)
at org.apache.bookkeeper.proto.BookieClientImpl.lookupClient(BookieClientImpl.java:217)
at org.apache.bookkeeper.proto.BookieClientImpl.readEntry(BookieClientImpl.java:502)
at org.apache.bookkeeper.proto.BookieClientImpl.readEntry(BookieClientImpl.java:495)
at org.apache.bookkeeper.proto.BookieClientImpl.readEntry(BookieClientImpl.java:489)
at org.apache.bookkeeper.client.ReadLastConfirmedOp.initiate(ReadLastConfirmedOp.java:81)
at org.apache.bookkeeper.client.LedgerHandle.asyncReadLastConfirmed(LedgerHandle.java:1416)
at org.apache.bookkeeper.client.LedgerOpenOp.openWithMetadata(LedgerOpenOp.java:210)
at org.apache.bookkeeper.client.LedgerOpenOp.lambda$initiate$0(LedgerOpenOp.java:119)
at java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:774)
at java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:750)
at java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:488)
at java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:1975)
at org.apache.bookkeeper.meta.AbstractZkLedgerManager$3.processResult(AbstractZkLedgerManager.java:464)
at org.apache.bookkeeper.zookeeper.ZooKeeperClient$19$1.processResult(ZooKeeperClient.java:994)
at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:598)
at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:510)
2020-08-14 18:46:43,243 - INFO - [Time-limited test:Auditor@1946] - Shutting down auditor
```
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [bookkeeper] eolivelli commented on issue #2398: LedgerOpenOp may perform connection to a bookie inside the ZooKeeper client thread
Posted by GitBox <gi...@apache.org>.
eolivelli commented on issue #2398:
URL: https://github.com/apache/bookkeeper/issues/2398#issuecomment-674169264
the same happens here:
at org.apache.bookkeeper.proto.PerChannelBookieClient.connect(PerChannelBookieClient.java:541)
at org.apache.bookkeeper.proto.PerChannelBookieClient.connectIfNeededAndDoOp(PerChannelBookieClient.java:666)
at org.apache.bookkeeper.proto.DefaultPerChannelBookieClientPool.intialize(DefaultPerChannelBookieClientPool.java:92)
at org.apache.bookkeeper.proto.BookieClientImpl.lookupClient(BookieClientImpl.java:217)
at org.apache.bookkeeper.proto.BookieClientImpl.readEntry(BookieClientImpl.java:502)
at org.apache.bookkeeper.proto.BookieClientImpl.readEntry(BookieClientImpl.java:495)
at org.apache.bookkeeper.proto.BookieClientImpl.readEntry(BookieClientImpl.java:489)
at org.apache.bookkeeper.client.ReadLastConfirmedOp.initiate(ReadLastConfirmedOp.java:81)
at org.apache.bookkeeper.client.LedgerHandle.asyncReadLastConfirmed(LedgerHandle.java:1416)
at org.apache.bookkeeper.client.LedgerOpenOp.openWithMetadata(LedgerOpenOp.java:210)
at org.apache.bookkeeper.client.LedgerOpenOp.lambda$initiate$0(LedgerOpenOp.java:119)
at java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:774)
at java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:750)
at java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:488)
at java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:1975)
at org.apache.bookkeeper.meta.AbstractZkLedgerManager$3.processResult(AbstractZkLedgerManager.java:464)
at org.apache.bookkeeper.zookeeper.ZooKeeperClient$19$1.processResult(ZooKeeperClient.java:994)
at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:598)
at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:510)
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [bookkeeper] eolivelli commented on issue #2398: LedgerOpenOp may perform connection to a bookie inside the ZooKeeper client thread
Posted by GitBox <gi...@apache.org>.
eolivelli commented on issue #2398:
URL: https://github.com/apache/bookkeeper/issues/2398#issuecomment-674167082
I will send a patch soon
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [bookkeeper] eolivelli commented on issue #2398: LedgerOpenOp may perform connection to a bookie inside the ZooKeeper client thread
Posted by GitBox <gi...@apache.org>.
eolivelli commented on issue #2398:
URL: https://github.com/apache/bookkeeper/issues/2398#issuecomment-674167777
the same happens here:
```
at org.apache.bookkeeper.proto.PerChannelBookieClient.connect(PerChannelBookieClient.java:541)
at org.apache.bookkeeper.proto.PerChannelBookieClient.connectIfNeededAndDoOp(PerChannelBookieClient.java:666)
at org.apache.bookkeeper.proto.DefaultPerChannelBookieClientPool.intialize(DefaultPerChannelBookieClientPool.java:92)
at org.apache.bookkeeper.proto.BookieClientImpl.lookupClient(BookieClientImpl.java:217)
at org.apache.bookkeeper.proto.BookieClientImpl.readEntry(BookieClientImpl.java:502)
at org.apache.bookkeeper.proto.BookieClientImpl.readEntry(BookieClientImpl.java:495)
at org.apache.bookkeeper.proto.BookieClientImpl.readEntry(BookieClientImpl.java:489)
at org.apache.bookkeeper.client.ReadLastConfirmedOp.initiate(ReadLastConfirmedOp.java:81)
at org.apache.bookkeeper.client.LedgerHandle.asyncReadLastConfirmed(LedgerHandle.java:1416)
at org.apache.bookkeeper.client.LedgerOpenOp.openWithMetadata(LedgerOpenOp.java:210)
at org.apache.bookkeeper.client.LedgerOpenOp.lambda$initiate$0(LedgerOpenOp.java:119)
at java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:774)
at java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:750)
at java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:488)
at java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:1975)
at org.apache.bookkeeper.meta.AbstractZkLedgerManager$3.processResult(AbstractZkLedgerManager.java:464)
at org.apache.bookkeeper.zookeeper.ZooKeeperClient$19$1.processResult(ZooKeeperClient.java:994)
at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:598)
at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:510)
```
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [bookkeeper] eolivelli removed a comment on issue #2398: LedgerOpenOp may perform connection to a bookie inside the ZooKeeper client thread
Posted by GitBox <gi...@apache.org>.
eolivelli removed a comment on issue #2398:
URL: https://github.com/apache/bookkeeper/issues/2398#issuecomment-674167777
the same happens here:
```
at org.apache.bookkeeper.proto.PerChannelBookieClient.connect(PerChannelBookieClient.java:541)
at org.apache.bookkeeper.proto.PerChannelBookieClient.connectIfNeededAndDoOp(PerChannelBookieClient.java:666)
at org.apache.bookkeeper.proto.DefaultPerChannelBookieClientPool.intialize(DefaultPerChannelBookieClientPool.java:92)
at org.apache.bookkeeper.proto.BookieClientImpl.lookupClient(BookieClientImpl.java:217)
at org.apache.bookkeeper.proto.BookieClientImpl.readEntry(BookieClientImpl.java:502)
at org.apache.bookkeeper.proto.BookieClientImpl.readEntry(BookieClientImpl.java:495)
at org.apache.bookkeeper.proto.BookieClientImpl.readEntry(BookieClientImpl.java:489)
at org.apache.bookkeeper.client.ReadLastConfirmedOp.initiate(ReadLastConfirmedOp.java:81)
at org.apache.bookkeeper.client.LedgerHandle.asyncReadLastConfirmed(LedgerHandle.java:1416)
at org.apache.bookkeeper.client.LedgerOpenOp.openWithMetadata(LedgerOpenOp.java:210)
at org.apache.bookkeeper.client.LedgerOpenOp.lambda$initiate$0(LedgerOpenOp.java:119)
at java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:774)
at java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:750)
at java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:488)
at java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:1975)
at org.apache.bookkeeper.meta.AbstractZkLedgerManager$3.processResult(AbstractZkLedgerManager.java:464)
at org.apache.bookkeeper.zookeeper.ZooKeeperClient$19$1.processResult(ZooKeeperClient.java:994)
at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:598)
at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:510)
```
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org