[GitHub] [accumulo] keith-turner commented on pull request #1487: Fixes: #1323 Implement a write thread limit

[GitBox <gi...@apache.org>]

keith-turner commented on pull request #1487:
URL: https://github.com/apache/accumulo/pull/1487#issuecomment-645409506


   If I were a user I would really want the the batch writer to automatically retry w/ exponential back-off for this situation. One possible way to do this is to only check when a client make the startUpdate() RPC.  If startUpdate() successfully increments the counter, then this is recorded on the server side write session and the client can call applytUpdate() and closeUpdate() w/o the checks happening.  If the startUpdate() does not increment the counter, then it throws a special exception indicating no resources.  When a write session is deleted on the server side, the counter is decremented.
   
   On the client side, when an exception is seen indicating no resources it can do exponential back off up to the configurable client side timeout.  The batch writer already has a configurable timeout, so this would work nicely with that w/o any API changes.
   
   I think checking in the startUpdate() RPC instead of the applyUpdate() RPC is better for throughput.  When server side resources are under strain, the client will not even bother sending an overburdened tserver data when the check is done in startUpdate().  When the check is done in applyUpdates() many client could be sending lots of data to a tserver that is just dropping it, increasing work the tserver has to do w/o any benefit.


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org