[GitHub] [helix] dasahcc opened a new issue #1646: Support Large IdealState (maybe ZNodes) for efficient read/write

[GitBox <gi...@apache.org>]

dasahcc opened a new issue #1646:
URL: https://github.com/apache/helix/issues/1646


   **Is your feature request related to a problem? Please describe.**
   If an IdealState is very large, we have two solutions:
   1. Compression the ZNode.
   2. Bucketized the ZNode.
   
   For option 1, if the node is too large like 200K partition, we may still see the ZNode is over size after the compression. For option 2, it could impact the performance since even one entry of one sub ZNode is changed, we write everything back to Zookeeper. And it impact the performance.
   
   **Describe the solution you'd like**
   So it is better to have a new support as bucketized ZNode + selective sub-ZNode update to Zookeeper. This bucketized ZNode can be optimized with some symbol to let client know this is a bucketized folder. Instead of current strategy read it back and check whether it is bucketize.
   
   Also the number of watchers increasing and number of read requests could be increased. So we need more investigation and trade-off analysis.
   
   **Additional context**
   


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@helix.apache.org
For additional commands, e-mail: reviews-help@helix.apache.org