[GitHub] [pulsar] lhotari removed a comment on pull request #12137: [Broker] Change the default number of namespace bundles to 32

[GitBox <gi...@apache.org>]

lhotari removed a comment on pull request #12137:
URL: https://github.com/apache/pulsar/pull/12137#issuecomment-925285755


   > user can always enable auto bundle unloading in loadbalancer. and default-bundle =4 is not defined for test cluster with 1 namespace but it considers a normal cluster that is serving multiple namespaces with considerable traffic. again, higher number of bundles comes with cost specially for znodes and default behavior unnecessarily will add more bundles for users who are using default conf file.
   
   I have a test cluster on GCP/GKE where the auto bundle unloading never happens. It's using the default settings which contains
   ```
   loadBalancerBrokerOverloadedThresholdPercentage=85
   loadBalancerAutoUnloadSplitBundlesEnabled=true
   loadBalancerNamespaceBundleMaxSessions=1000
   ```
   The test load is created with this type of pulsar-perf command (using 2.7.3 pulsar-perf since current [2.8.0 contains a bug that is fixed in master](https://github.com/apache/pulsar/pull/11824), the bug causes partitions to stop consuming in pulsar-perf).
   ```
   # create a 100 partition topic
   pulsar-admin topics create-partitioned-topic -p 100 persistent://public/default/perftest
   
   # produce messages
   PULSAR_EXTRA_OPTS="-Xms1500M -Xmx1500M -XX:MaxDirectMemorySize=768M" /pulsar/bin/pulsar-perf produce -u pulsar+ssl://pulsar-broker:6651/ -ioThreads 4 -s 512 -r 50000 persistent://public/default/perftest
   
   # consume messages
   PULSAR_EXTRA_OPTS="-Xms1500M -Xmx1500M -XX:MaxDirectMemorySize=768M" /pulsar/bin/pulsar-perf consume -u pulsar+ssl://pulsar-broker:6651/ -s sub persistent://public/default/perftest
   ```
   (these are run on a pod which has 2CPU + 4GB RAM requested/limits, using pulsar-perf from Pulsar 2.7.3 docker image)
   
   The traffic can go on for hours and the CPU load will be very low. Since the threshold never happens.
   In this case, the GKE cluster has a nodepool with 6 n2-standard-32 VMs (32 cores, 128GB RAM). Storage for bookies in on SSD.
   The impact of this is that one of the brokers will never get any real load on it. 
   After changing to a namespace with 32 bundles, the load was evenly distributed. This is the reason why I think that `defaultNumberOfNamespaceBundles=4` is not a good default.
   
   
   
   
   
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org