[GitHub] [pulsar] vicaya opened a new issue #5798: s3 offloading generates huge zk txn logs in some conditions

[GitBox <gi...@apache.org>]

vicaya opened a new issue #5798: s3 offloading generates huge zk txn logs in some conditions
URL: https://github.com/apache/pulsar/issues/5798
 
 
   **Describe the bug**
   When s3 offloading is turned on but credentials is incorrect the zookeeper /managed-ledger znode entries grow to be fairly large (1MB), causing zk txn log to be in the range of multiple GBs, which could easily run out of the space on the zk log device under default log purging configuration.
   
   An example of the content of such a znode:
   ```
   k: localhost:2181(CONNECTED) 0] get /managed-ledgers/<topic-path>
   ?????  ?????-*???????????????????2?*
   s34
   bucket*<bucket>
   region  us-west-2.
   endpoint"https://s3.us-west-2.amazonaws.com4
   bucket*<bucket>
   region  us-west-2.
   endpoint"https://s3.us-west-2.amazonaws.com4
   bucket*<bucket>
   region  us-west-2.
   ```
   where the bucket, endpoint, and region info appears to be duplicated many times. The entry size also grows under normal offloading operation conditions but slower.
   
   **To Reproduce**
   See above. We reproduced the problem with fake topics
   
   **Expected behavior**
   znode entry size should be bounded. There appears no reason to duplicate offloading metadata.
   

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services