[GitHub] [pulsar-site] momo-jun commented on a diff in pull request #652: [feat][doc] add use cases + features + benefits + types for broker load balancing

["momo-jun (via GitHub)" <gi...@apache.org>]

momo-jun commented on code in PR #652:
URL: https://github.com/apache/pulsar-site/pull/652#discussion_r1271620806


##########
docs/concepts-broker-load-balancing-types.md:
##########
@@ -0,0 +1,63 @@
+---
+id: concepts-broker-load-balancing-types
+title: Types
+sidebar_label: "Types"
+---
+
+Pulsar provides three types of broker load balancers:
+
+- Simple broker load balancer: it existed since Pulsar's inception, but now is deprecated. No one uses it.
+
+- [Modular broker load balancer](#side-by-side-comparisons)
+
+- [Extensible broker load balancer](#side-by-side-comparisons)
+
+The modular and extensible broker load balancer use similar implementation logic and algorithm and have similar functionalities. For example, they both employ a similar approach to distributing data loads among brokers, including:
+
+- Dynamic [bundle-broker assignment](./concepts-broker-load-balancing-concepts.md#bundle-assignment)
+
+- Dynamic [bundle splitting](./concepts-broker-load-balancing-concepts.md#bundle-splitting)
+
+- Dynamic [bundle unloading (shedding)](./concepts-broker-load-balancing-concepts.md#bundle-unloading)  
+
+As a result, brokers dynamically collect and share the current load with other brokers for better load balancing decisions. 
+
+## Side-by-side comparisons
+
+The extensible broker load balancer refactors the logic of the modular broker load balancer with better modularity and is less dependent on ZooKeeper, which makes Pulsar more stable and scalable. 
+
+Dimension|Modular broker load balancer|Extensible broker load balancer
+|---|---|---
+Available version|Pulsar 1.7.0 <br/> <br/>2017|Pulsar 3.0.0 <br/> <br/>2023
+Architecture|ZooKeeper plays an important role, which may cause a large overhead of load data replication.<br/> <br/> All broker load data and all bundle load data are stored in ZooKeeper and replicated to all brokers via ZooKeeper watchers. | ZooKeeper is less dependent (only lookup data is stored in ZooKeeper), which reduces the overhead of load data replication and minimizes data processing delays.<br/> <br/> Pulsar reads and writes data via system topics and table view rather than ZooKeeper: <br/> <br/> - [System topics](./concepts-messaging.md#system-topic): global ownership data is stored in bundle state channels within persistent topics. <br/> <br/> - [Table view](./concepts-clients.md#tableview): global ownership data is replicated using the compacted table view, where 1) All broker load data is replicated to all brokers via non-persistent topics. 2) Only the top-n bundle load data from each broker is replicated to the leader brokers via non-persistent topics. <br/> <br/> Th
 e absolute size of the shared load data and the complexity of the replication decreased due to passing the data through memory-only, non-persistent topics. 
+Performance (1/2) | Load balance decisions might be made slowly, and data load might be distributed unevenly. <br/> <br/> - Single leader broker decides on bundle-broker assignment, bundle splitting, and bundle unloading. <br/> <br/> - This requires all load information to be passed to the leader broker, which may delay load balance executions. | It optimizes the overall performance, such as load balance decisions are made faster, and data load is distributed more evenly. <br/> <br/> - Each broker decides and runs bundle-broker assignments and bundle splitting. <br/> <br/> - The leader broker still decides on bundle unloading globally. <br/> <br/> - The load balance decisions are more distributed across brokers, expecting quicker load balance operations.
+|Performance (2/2) | Topics might be temporarily unavailable during the bundle unloading process. <br/> <br/> -  When bundles are unloaded, the topics in the bundles will be closed, and producers and consumers need to reconnect to new owner brokers. <br/> <br/> - This new assignment process involves redirections talking to multiple brokers (including the leader broker), and this redirection can delay the reconnections of producers and consumers. |  Topics are more available during the bundle unloading process because the time of the reconnection from producer to consumers is faster. <br/> <br/> - The time of unloading bundles (topics) automatically is faster. A new unloading strategy, [TransferShedder](./concepts-broker-load-balancing-concepts.md#bundle-unloading-strategies), is introduced as  the default behavior, which means new brokers will be pre-assigned before topic connections are closed. <br/> <br/> - The time of unloading bundles (topics) manually is faster. The logic of pu
 lsar-admin transfer is refactored and improved. The pub/consumers can connect to the new owner broker with lesser redirections. <br/> <br/> For detailed performance comparison results between the extensible broker load balancer and the modular broker load balancer, see [performance tests](#performance-tests).

Review Comment:
   ```suggestion
   |Performance (2/2) | Topics might be temporarily unavailable during the bundle unloading process. <br/> <br/> -  When bundles are unloaded, the topics in the bundles will be closed, and producers and consumers need to reconnect to new owner brokers. <br/> <br/> - This new assignment process involves redirections talking to multiple brokers (including the leader broker), and this redirection can delay the reconnections of producers and consumers. |  Topics are more available during the bundle unloading process because the time of the reconnection from producer to consumer is faster. <br/> <br/> - The time of unloading bundles (topics) automatically is faster. A new unloading strategy, [TransferShedder](./concepts-broker-load-balancing-concepts.md#bundle-unloading-strategies), is introduced as the default behavior, which means new brokers will be pre-assigned before topic connections are closed. <br/> <br/> - The time of unloading bundles (topics) manually is faster. The logic of pu
 lsar-admin transfer is refactored and improved. The pub/consumers can connect to the new owner broker with lesser redirections. <br/> <br/> For detailed performance comparison results between the extensible broker load balancer and the modular broker load balancer, see [performance tests](#performance-tests).
   ```



##########
docs/concepts-broker-load-balancing-types.md:
##########
@@ -0,0 +1,63 @@
+---
+id: concepts-broker-load-balancing-types
+title: Types
+sidebar_label: "Types"
+---
+
+Pulsar provides three types of broker load balancers:
+
+- Simple broker load balancer: it existed since Pulsar's inception, but now is deprecated. No one uses it.
+
+- [Modular broker load balancer](#side-by-side-comparisons)
+
+- [Extensible broker load balancer](#side-by-side-comparisons)
+
+The modular and extensible broker load balancer use similar implementation logic and algorithm and have similar functionalities. For example, they both employ a similar approach to distributing data loads among brokers, including:
+
+- Dynamic [bundle-broker assignment](./concepts-broker-load-balancing-concepts.md#bundle-assignment)
+
+- Dynamic [bundle splitting](./concepts-broker-load-balancing-concepts.md#bundle-splitting)
+
+- Dynamic [bundle unloading (shedding)](./concepts-broker-load-balancing-concepts.md#bundle-unloading)  
+
+As a result, brokers dynamically collect and share the current load with other brokers for better load balancing decisions. 

Review Comment:
   ```suggestion
   As a result, brokers dynamically collect and share the current load with other brokers for better load-balancing decisions. 
   ```



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org