[GitHub] [accumulo] seriouslyjeez commented on issue #1225: Use fewer ZooKeeper watchers

[GitBox <gi...@apache.org>]

seriouslyjeez commented on issue #1225: Use fewer ZooKeeper watchers
URL: https://github.com/apache/accumulo/issues/1225#issuecomment-532346566
 
 
   I've encountered issues with too many watchers in the past.  The watcher count was a combination of ZK issues and a large number of frequent full table drops and adds.  This leads to an exceptionally large number of watches.  The high watch count impacts Accumulo as described below.
   
   Once the watch count reaches a high enough value we start to see Zookeeper issues, mainly jute.maxbuffer and PacketLen errors (in ZK logs) along with session expiry (mostly due to communication issues due to PacketLen issues).
   
   While it is possible to crank up the jute.maxbuffer settings (and the ZK heap as watches grow higher), it would be much better overall if the total number of watches were much lower so that the data stored in ZK was much closer to the "small" chunks intended to be stored by ZK.
   
   Additionally, at scale it may not be possible to completely eliminate the session expiry due to extended ZK communication delays.  For this Accumulo needs to be better at re-building a ZK session.  Currently Accumulo will retry an invalid ZK session, instead it should try to rebuild the invalid session.  (This is most likely a new issue)

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services