You are viewing a plain text version of this content. The canonical link for it is here.

Posted to issues@bookkeeper.apache.org by GitBox <gi...@apache.org> on 2021/03/24 18:30:31 UTC

[GitHub] [bookkeeper] karanmehta93 opened a new issue #2660: Bookkeeper client throttling logic is based upon entryId instead of ledgerId

karanmehta93 opened a new issue #2660:
URL: https://github.com/apache/bookkeeper/issues/2660


   **BUG REPORT**
   
   ***Describe the bug***
   
   Before accepting the write, the client checks if netty channels to all the bookies are available. The logic is currently based on [entryId](https://github.com/apache/bookkeeper/blob/master/bookkeeper-server/src/main/java/org/apache/bookkeeper/client/LedgerHandle.java#L1343). However, in BookieClientImpl, we select the actual channel based on [ledgerId](https://github.com/apache/bookkeeper/blob/master/bookkeeper-server/src/main/java/org/apache/bookkeeper/proto/BookieClientImpl.java#L329-L332). 
   
   We use 3 channels per bookie in our prod and one of them was down (due to unknown reasons). Hence about 1/3 of the requests were throttled for no reason. The throttling caused the overall app to slow down.
   
   ***To Reproduce***
   
   Not very easy to reproduce. Need to check if netty can be mocked to add an integration test.
   
   ***Expected behavior***
   
   Throttling should be based on ledgerId, instead of entryId.
   
   ***Screenshots***
   
   N/A
   
   ***Additional context***
   
   N/A
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [bookkeeper] dlg99 closed issue #2660: Bookkeeper client throttling logic is based upon entryId instead of ledgerId

Posted by GitBox <gi...@apache.org>.

dlg99 closed issue #2660:
URL: https://github.com/apache/bookkeeper/issues/2660


   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [bookkeeper] dlg99 commented on issue #2660: Bookkeeper client throttling logic is based upon entryId instead of ledgerId

Posted by GitBox <gi...@apache.org>.

dlg99 commented on issue #2660:
URL: https://github.com/apache/bookkeeper/issues/2660#issuecomment-809643040


   @karanmehta93 I think you are correct about the ledgerId. 
   Please review https://github.com/apache/bookkeeper/pull/2664 for the fix.
   
   I don't think it will help you with underlying problem which you still have to rootcause. 
   The worst this problem does it selects wrong channel and detects it as non-writable. With switch to the ledger Id some other ledger will end up blocked on non-writable channel. 
   Non-writable PCBC == non-writable netty channel to the bookie. IIRC, this can be related to slow network, non-responsive bookie, bookie being overloaded and disabling autoread on the channel etc. 


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [bookkeeper] jvrao commented on issue #2660: Bookkeeper client throttling logic is based upon entryId instead of ledgerId

Posted by GitBox <gi...@apache.org>.

jvrao commented on issue #2660:
URL: https://github.com/apache/bookkeeper/issues/2660#issuecomment-831463363


   > The worst this problem does it selects wrong channel and detects it as non-writable.
   
   Yes, but the worst is because we are just throttling and not failing/changing ensemble. Hence latencies wentup without going through our fastfail model. But you right the underlying problem of why the channel is not writable still needs to be figured out.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org