You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@pulsar.apache.org by Kai Wang <wa...@gmail.com> on 2022/04/07 01:35:09 UTC

[DISSCUSS] [PIP-151] Use the system topic to store the bundle load data

Hi Pulsar community,

I created a PIP to use the system topic to store the bundle load data in the load manager.

The proposal can be found: https://github.com/apache/pulsar/issues/15037

---------
## Motivation

Currently, Pulsar LoadManager is using Zookeeper to store the bundle load data, when we have many bundles, this might put a lot of pressure on Zookeeper.

## Goal

This PIP proposes storing the load manager's bundle load data to a system topic and using TableView to read it.



## Implementation

Since the bundle load data is stats data, it no needs a strong consistent guarantee. So we can use the system topic to store the load data and use TableView to read it.


### System topic client



Add a new SystemTopicClient calls `LoadBalanceBundleDataSystemTopicClient` , the topic name is `persistent://pulsar/system/__load_balance_bundle_data`, and the key is the bundle name, value is BundleData.

```java
public class LoadBalanceBundleDataSystemTopicClient extends SystemTopicClientBase<LoadBalanceBundleDataEvent> {
// ...
}
```



Add new Event calls `LoadBalanceBundleDataEvent`

```
@Data
public class LoadBalanceBundleDataEvent {

 private String bundle;

 private BundleData bundleData;
}

```


### ModularLoadManagerImpl


Add a TableView in `ModularLoadManagerImpl` to replace the bundle cache

```
private TableView<String, BundleData> bundlesTableView;
```


## Compatibility

This feature can have both backward and forward compatibility since the bundle data is stats data.

---------

Thanks,
Kai

Re: [DISSCUSS] [PIP-151] Use the system topic to store the bundle load data

Posted by "rxl@apache.org" <ra...@gmail.com>.
Hi Kai:

Good ideas, For this PIP, there are the following confusions:

1. Does the current bundle-data strongly depend on zk for data consistency?

2. If the data of bundle-data is dumped to system-topic, can there be a
good guarantee for the real-time performance of bundle changes?

3. This seems to be an incompatible change, is there a better upgrade plan
for current users?

--
Thanks
Xiaolong Ran

Haiting Jiang <ji...@apache.org> 于2022年4月7日周四 15:36写道：

> Hi Kai,
>
> Good feature. Please add more info about how to enable this in an existing
> cluster.  Do we plan to write bundle data to both zk and system topic
> during rolling update?
>
> Thanks,
> Haiting
>
> On 2022/04/07 01:35:09 Kai Wang wrote:
> > Hi Pulsar community,
> >
> > I created a PIP to use the system topic to store the bundle load data in
> the load manager.
> >
> > The proposal can be found: https://github.com/apache/pulsar/issues/15037
> >
> > ---------
> > ## Motivation
> >
> > Currently, Pulsar LoadManager is using Zookeeper to store the bundle
> load data, when we have many bundles, this might put a lot of pressure on
> Zookeeper.
> >
> > ## Goal
> >
> > This PIP proposes storing the load manager's bundle load data to a
> system topic and using TableView to read it.
> >
> >
> >
> > ## Implementation
> >
> > Since the bundle load data is stats data, it no needs a strong
> consistent guarantee. So we can use the system topic to store the load data
> and use TableView to read it.
> >
> >
> > ### System topic client
> >
> >
> >
> > Add a new SystemTopicClient calls
> `LoadBalanceBundleDataSystemTopicClient` , the topic name is
> `persistent://pulsar/system/__load_balance_bundle_data`, and the key is the
> bundle name, value is BundleData.
> >
> > ```java
> > public class LoadBalanceBundleDataSystemTopicClient extends
> SystemTopicClientBase<LoadBalanceBundleDataEvent> {
> > // ...
> > }
> > ```
> >
> >
> >
> > Add new Event calls `LoadBalanceBundleDataEvent`
> >
> > ```
> > @Data
> > public class LoadBalanceBundleDataEvent {
> >
> >  private String bundle;
> >
> >  private BundleData bundleData;
> > }
> >
> > ```
> >
> >
> > ### ModularLoadManagerImpl
> >
> >
> > Add a TableView in `ModularLoadManagerImpl` to replace the bundle cache
> >
> > ```
> > private TableView<String, BundleData> bundlesTableView;
> > ```
> >
> >
> > ## Compatibility
> >
> > This feature can have both backward and forward compatibility since the
> bundle data is stats data.
> >
> > ---------
> >
> > Thanks,
> > Kai
> >
>

Re: [DISSCUSS] [PIP-151] Use the system topic to store the bundle load data

Posted by Haiting Jiang <ji...@apache.org>.
Hi Kai,

Good feature. Please add more info about how to enable this in an existing cluster.  Do we plan to write bundle data to both zk and system topic during rolling update? 

Thanks,
Haiting

On 2022/04/07 01:35:09 Kai Wang wrote:
> Hi Pulsar community,
> 
> I created a PIP to use the system topic to store the bundle load data in the load manager.
> 
> The proposal can be found: https://github.com/apache/pulsar/issues/15037
> 
> ---------
> ## Motivation
> 
> Currently, Pulsar LoadManager is using Zookeeper to store the bundle load data, when we have many bundles, this might put a lot of pressure on Zookeeper.
> 
> ## Goal
> 
> This PIP proposes storing the load manager's bundle load data to a system topic and using TableView to read it.
> 
> 
> 
> ## Implementation
> 
> Since the bundle load data is stats data, it no needs a strong consistent guarantee. So we can use the system topic to store the load data and use TableView to read it.
> 
> 
> ### System topic client
> 
> 
> 
> Add a new SystemTopicClient calls `LoadBalanceBundleDataSystemTopicClient` , the topic name is `persistent://pulsar/system/__load_balance_bundle_data`, and the key is the bundle name, value is BundleData.
> 
> ```java
> public class LoadBalanceBundleDataSystemTopicClient extends SystemTopicClientBase<LoadBalanceBundleDataEvent> {
> // ...
> }
> ```
> 
> 
> 
> Add new Event calls `LoadBalanceBundleDataEvent`
> 
> ```
> @Data
> public class LoadBalanceBundleDataEvent {
> 
>  private String bundle;
> 
>  private BundleData bundleData;
> }
> 
> ```
> 
> 
> ### ModularLoadManagerImpl
> 
> 
> Add a TableView in `ModularLoadManagerImpl` to replace the bundle cache
> 
> ```
> private TableView<String, BundleData> bundlesTableView;
> ```
> 
> 
> ## Compatibility
> 
> This feature can have both backward and forward compatibility since the bundle data is stats data.
> 
> ---------
> 
> Thanks,
> Kai
>