[GitHub] [pulsar] hangc0276 commented on a diff in pull request #21011: [improve][pip] PIP-294: Filter out the bundles with low throughput in Bundle load report

["hangc0276 (via GitHub)" <gi...@apache.org>]

hangc0276 commented on code in PR #21011:
URL: https://github.com/apache/pulsar/pull/21011#discussion_r1299887586


##########
pip/pip-294.md:
##########
@@ -0,0 +1,56 @@
+# Background knowledge
+
+Load balance module in Pulsar broker rely on zk to store and synchronize metadata about load. Every broker will upload its `LocalBrokerData` to zk, and leader broker will retrieve all `LocalBrokerData` from zk ,generate all `BundleData` from each `LocalBrokerData`, and update all `BundleData` to zk. 
+
+
+# Motivation
+
+As every bundle in the cluster corresponds to a zk node, it is common that there are thousands of zk nodes in a cluster, which results into thousands of read/update operations to zk. This will cause a lot of pressure on zk.
+
+**As All Load Shedding Algorithm pick bundles from top to bottom based on throughput/msgRate, bundles with low throughput/msgRate are rarely be selected for shedding. So there is no need to update these bundleData to zk frequently.**

Review Comment:
   Can we also apply the TopK selection for `ModularLoadManagerImpl`?



##########
pip/pip-294.md:
##########
@@ -0,0 +1,67 @@
+# Background knowledge
+
+There are mainly two `LoadManager` implementation in Pulsar broker: `ExtensibleLoadManager` and `ModularLoadManagerImpl`. `ModularLoadManagerImpl` is the default load manager, and `ExtensibleLoadManager` is a new load manager which is proposed after 3.0.0 version.
+
+## ModularLoadManagerImpl
+`ModularLoadManagerImpl` rely on zk to store and synchronize metadata about load, which pose greate pressure on zk, threatening the stability of system. Every broker will upload its `LocalBrokerData` to zk, and leader broker will retrieve all `LocalBrokerData` from zk ,generate all `BundleData` from each `LocalBrokerData`, and update all `BundleData` to zk. 
+
+## ExtensibleLoadManager
+`ExtensibleLoadManager` depends on system topics and table views for load balance metadata store and replication. Though not using zk to store and synchronize metadata about load, it is still necessary to control the number of bundles that need to be updated, for which there is a `loadBalancerMaxNumberOfBundlesInBundleLoadReport` configuration in `ExtensibleLoadManager` that select the top k bundles.
+
+# Motivation
+
+## ModularLoadManagerImpl
+As every bundle in the cluster corresponds to a zk node, it is common that there are thousands of zk nodes in a cluster, which results into thousands of read/update operations to zk. This will cause a lot of pressure on zk.
+
+**As All Load Shedding Algorithm pick bundles from top to bottom based on throughput/msgRate, bundles with low throughput/msgRate are rarely be selected for shedding. So there is no need to update these bundleData to zk frequently.**
+
+## ExtensibleLoadManager
+As the number of bundles and the throughput in the cluster changes dynamically, users can't select a reasonable number of bundles easily, while based on throughput, we can be sure that bundles with throughput less than 0.1M is useless for load balance no matter how the cluster change.
+
+We can also add this throughput-based bundle report size control in the ExtensibleLoadManager, working together with `loadBalancerMaxNumberOfBundlesInBundleLoadReport`.

Review Comment:
   From the motivation, I learned that this proposal's goal is to limit the number of bundle reports written to the meta store or system topic because a huge number of bundle reports will bring heavy pressure on the meta store and system topic.
   
   What I don't understand is why `loadBalancerMaxNumberOfBundlesInBundleLoadReport` is not enough. 
   - We can limit the load report bundler number to 20 for each broker. If you have 10 brokers, and the leader will get 200 bundles to decide if we need to migrate any bundles. In the next round, each broker will report 20 bundles report based on current bundle stats. In this way, we can limit the max number of bundle reports written to the meta store or system topic
   - For ExtensibleLoadManager, the number of bundle report issue has less impact on the load balance because we use non-persistent topic to store the bundle load report.
   - Add throughput-based bundle report size control has the following issues.
     - Users need to learn another configuration to understand the load-balance behavior
     - The value is hard to configure, especially when the Pulsar cluster has low throughput in all brokers
     - Our goal is to help the leader to decide which broker is overloaded and which bundle needs to be unloaded. The TopK bundles play a key role to help the leader to make the decision.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org