You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@pulsar.apache.org by GitBox <gi...@apache.org> on 2022/07/25 14:01:24 UTC
[GitHub] [pulsar] lordcheng10 opened a new issue, #16782: [PIP-164] Support split bundle by flow or qps
lordcheng10 opened a new issue, #16782:
URL: https://github.com/apache/pulsar/issues/16782
## Motivation
As we all know, Bundle split has 3 algorithms:
- range_equally_divide
- topic_count_equally_divide
- specified_positions_divide
However, none of these algorithms can divide bundles according to traffic or qps, which may cause bundles to be split multiple times.
## Goal
Our goal is to split bundles according to traffic or QPS, so we propose a PIP to introduce a split algorithm based on traffic or QPS.
The main idea is that we can get the traffic or qps information of a topic contained in a bundle,
and then split from the position where the traffic or qps are evenly divided.
For example, there is bundle with boundaries 0x00000000 to 0x00000200, and four topics : t1 , t2 , t3 , t4, t5, t6.
Step 1: Get their hash position and corresponding traffic or QPS:
t1 with hashcode 10 msgRate 100/s throughput 1M/s
t2 with hashcode 20 msgRate 200/s throughput 2M/s
t3 with hashcode 80 msgRate 300/s throughput 3M/s
t4 with hashcode 90 msgRate 400/s throughput 4M/s
t5 with hashcode 100 msgRate 500/s throughput 5M/s
t6 with hashcode 110 msgRate 2000/s throughput 190M/s
Step 2: Calculate the total traffic and qps of the bundle:
bundleMsgRate=3500
bundleThroughput=205MB
Step 3: Calculate the traffic and qps to split:
splitBundleMsgRate=1750
splitBundleThroughput=102.5MB
Step 4: Calculate the position to split and split:
splitStartPosition=100
splitEndPosition=110
splitPosition=(100+110)/2=105
## API Changes
Added FlowOrQpsEquallyDivideBundleSplitAlgorithm class:
/**
* Split algorithm based on flow or qps.
*/
public class FlowOrQpsEquallyDivideBundleSplitAlgorithm implements NamespaceBundleSplitAlgorithm {
@Override
public CompletableFuture<List<Long>> getSplitBoundary(BundleSplitOption bundleSplitOption) {
...
}
}
In the ServiceConfiguration class, update the default configuration corresponding to supportedNamespaceBundleSplitAlgorithms:
private List<String> supportedNamespaceBundleSplitAlgorithms = Lists.newArrayList("range_equally_divide", "topic_count_equally_divide",
"specified_positions_divide", "flow_count_equally_divide");
## Implementation
The execution steps of the FlowOrQpsEquallyDivideBundleSplitAlgorithm#getSplitBoundary method are as follows:
1. Get the hash position of each topic and the corresponding msgRate and msgThroughput, and sort them according to the position size：
List<Long> topicNameHashList = new ArrayList<>(topics.size());
Map<Long, Double> hashAndMsgMap = new HashMap<>();
Map<Long, Double> hashAndThroughput = new HashMap<>();
2. Traverse the topic position from small to large to find the position that can roughly evenly divide the bundle's traffic or qps：
double bundleMsgRateTmp = 0;
double bundleThroughputTmp = 0;
for (int i = 0; i < topicNameHashList.size(); i++) {
long topicHashCode = topicNameHashList.get(i);
bundleThroughputTmp += hashAndThroughput.get(topicHashCode);
bundleMsgRateTmp += hashAndMsgMap.get(topicHashCode);
if (bundleMsgRateTmp > bundleMsgRate / 2 || bundleThroughputTmp > bundleThroughput / 2) {
long splitStart = i > 0 ? topicNameHashList.get(i - 1) : 0;
long splitEnd = topicHashCode;
long splitMiddle = splitStart + (splitEnd - splitStart) / 2;
splitResults.add(splitMiddle);
break;
}
}
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] eolivelli commented on issue #16782: [PIP-169] Support split bundle by flow or qps
Posted by GitBox <gi...@apache.org>.
eolivelli commented on issue #16782:
URL: https://github.com/apache/pulsar/issues/16782#issuecomment-1222764686
Very good.
I support this proposal
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] lordcheng10 commented on issue #16782: [PIP-164] Support split bundle by flow or qps
Posted by GitBox <gi...@apache.org>.
lordcheng10 commented on issue #16782:
URL: https://github.com/apache/pulsar/issues/16782#issuecomment-1194144140
discuss thread https://lists.apache.org/thread/cshyt10fwcjjxs93g8yf0svgwcgnshmg
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] lordcheng10 commented on issue #16782: [PIP-168] Support split bundle by flow or qps
Posted by GitBox <gi...@apache.org>.
lordcheng10 commented on issue #16782:
URL: https://github.com/apache/pulsar/issues/16782#issuecomment-1194878497
@codelipenghui Need to resend the discussion email? Because the email header shows 164.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] lordcheng10 commented on issue #16782: [PIP-169] Support split bundle by flow or qps
Posted by GitBox <gi...@apache.org>.
lordcheng10 commented on issue #16782:
URL: https://github.com/apache/pulsar/issues/16782#issuecomment-1194881303
> > @codelipenghui Need to resend the discussion email? Because the email header shows 164.
>
> No need to re-send, you can reply to the latest PIP in the original email
OK
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] codelipenghui closed issue #16782: [PIP-169] Support split bundle by flow or qps
Posted by GitBox <gi...@apache.org>.
codelipenghui closed issue #16782: [PIP-169] Support split bundle by flow or qps
URL: https://github.com/apache/pulsar/issues/16782
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] codelipenghui commented on issue #16782: [PIP-164] Support split bundle by flow or qps
Posted by GitBox <gi...@apache.org>.
codelipenghui commented on issue #16782:
URL: https://github.com/apache/pulsar/issues/16782#issuecomment-1194867651
@lordcheng10 The PIP number is duplicated with https://github.com/apache/pulsar/issues/15463
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] lordcheng10 commented on issue #16782: [PIP-169] Support split bundle by flow or qps
Posted by GitBox <gi...@apache.org>.
lordcheng10 commented on issue #16782:
URL: https://github.com/apache/pulsar/issues/16782#issuecomment-1221974845
> Do you have some numbers about experiments ? Before accepting the proposal we must be sure that this algorithm works on only on the paper but also on some real benchmark/test env
Test steps：
1.create Topic:
```
sh bin/pulsar-admin tenants create test-tenant1
sh bin/pulsar-admin namespaces create test-tenant1/test-namespace1
sh bin/pulsar-admin topics create-partitioned-topic test-tenant1/test-namespace1/test1 -p 20
sh bin/pulsar-admin topics create-partitioned-topic test-tenant1/test-namespace1/test2 -p 20
sh bin/pulsar-admin topics create-partitioned-topic test-tenant1/test-namespace1/test3 -p 20
sh bin/pulsar-admin topics create-partitioned-topic test-tenant1/test-namespace1/test4 -p 20
sh bin/pulsar-admin topics create-partitioned-topic test-tenant1/test-namespace1/test5 -p 20
sh bin/pulsar-admin topics create-partitioned-topic test-tenant1/test-namespace1/test6 -p 20
sh bin/pulsar-admin topics create-partitioned-topic test-tenant1/test-namespace1/test7 -p 20
sh bin/pulsar-admin topics create-partitioned-topic test-tenant1/test-namespace1/test8 -p 20
sh bin/pulsar-admin topics create-partitioned-topic test-tenant1/test-namespace1/test9 -p 20
sh bin/pulsar-admin topics create-partitioned-topic test-tenant1/test-namespace1/test10 -p 20
sh bin/pulsar-admin topics create-partitioned-topic test-tenant1/test-namespace1/test11 -p 20
sh bin/pulsar-admin topics create-partitioned-topic test-tenant1/test-namespace1/test12 -p 20
sh bin/pulsar-admin topics create-partitioned-topic persistent://test-tenant1/test-namespace1/test695 -p 1
```
2.generate data:
```
nohup sh bin/pulsar-perf produce test-tenant1/test-namespace1/test1 -threads 1 -n 1 -r 1000 > log1 &
nohup sh bin/pulsar-perf produce test-tenant1/test-namespace1/test2 -threads 1 -n 1 -r 1000 > log2 &
nohup sh bin/pulsar-perf produce test-tenant1/test-namespace1/test3 -threads 1 -n 1 -r 1000 > log3 &
nohup sh bin/pulsar-perf produce test-tenant1/test-namespace1/test4 -threads 1 -n 1 -r 1000 > log4 &
nohup sh bin/pulsar-perf produce test-tenant1/test-namespace1/test5 -threads 1 -n 1 -r 1000 > log5 &
nohup sh bin/pulsar-perf produce test-tenant1/test-namespace1/test6 -threads 1 -n 1 -r 1000 > log6 &
nohup sh bin/pulsar-perf produce test-tenant1/test-namespace1/test7 -threads 1 -n 1 -r 1000 > log7 &
nohup sh bin/pulsar-perf produce test-tenant1/test-namespace1/test8 -threads 1 -n 1 -r 1000 > log8 &
nohup sh bin/pulsar-perf produce test-tenant1/test-namespace1/test9 -threads 1 -n 1 -r 1000 > log9 &
nohup sh bin/pulsar-perf produce test-tenant1/test-namespace1/test10 -threads 1 -n 1 -r 1000 > log10 &
nohup sh bin/pulsar-perf produce test-tenant1/test-namespace1/test11 -threads 1 -n 1 -r 1000 > log11 &
nohup sh bin/pulsar-perf produce test-tenant1/test-namespace1/test12 -threads 1 -n 1 -r 1000 > log12 &
nohup sh bin/pulsar-perf produce persistent://test-tenant1/test-namespace1/test695 -threads 1 -n 11 -r 1010 > log695 &
```
3.the configuration is as follows:
#defaultNamespaceBundleSplitAlgorithm=topic_count_equally_divide
defaultNamespaceBundleSplitAlgorithm=flow_or_qps_equally_divide
loadBalancerNamespaceBundleMaxMsgRate=1010
flowOrQpsDifferenceThresholdPercentage=10
exposeBundlesMetricsInPrometheus=true
The comparison effect is as follows:
1.Using the topic_count_equally_divide algorithm, pulsar_bundle_msg_rate_in is monitored as follows:
The number of bundles after splitting is 21, and the pulsar_bundle_msg_rate_in of a single bundle is: 49 msg/s ~ 1010 msg/s：
2.Using the new algorithm flow_or_qps_equally_divide the effect is as follows:
The number of bundles after splitting is 13, and the pulsar_bundle_msg_rate_in of a single bundle is: 997 msg/s ~ 1010 msg/s：
<img width="1282" alt="image" src="https://user-images.githubusercontent.com/19296967/185864580-198b92fb-ef0b-4390-9bfc-56a03a1b19f0.png">
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] eolivelli commented on issue #16782: [PIP-169] Support split bundle by flow or qps
Posted by GitBox <gi...@apache.org>.
eolivelli commented on issue #16782:
URL: https://github.com/apache/pulsar/issues/16782#issuecomment-1199187017
Do you have some numbers about experiments ?
Before accepting the proposal we must be sure that this algorithm works on only on the paper but also on some real benchmark/test env
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] lordcheng10 commented on issue #16782: [PIP-169] Support split bundle by flow or qps
Posted by GitBox <gi...@apache.org>.
lordcheng10 commented on issue #16782:
URL: https://github.com/apache/pulsar/issues/16782#issuecomment-1223501647
discuss thread https://lists.apache.org/thread/cshyt10fwcjjxs93g8yf0svgwcgnshmg
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] aloyszhang commented on issue #16782: [PIP-169] Support split bundle by flow or qps
Posted by GitBox <gi...@apache.org>.
aloyszhang commented on issue #16782:
URL: https://github.com/apache/pulsar/issues/16782#issuecomment-1223599344
LGTM
+1
Heesung Sohn ***@***.***> 于2022年8月23日周二 13:40写道：
> +1 LGTM.
>
> —
> Reply to this email directly, view it on GitHub
> <https://github.com/apache/pulsar/issues/16782#issuecomment-1223572370>,
> or unsubscribe
> <https://github.com/notifications/unsubscribe-auth/ALOWDKL6X5IMMLMBOU6W66DV2RP5BANCNFSM54SMY7OA>
> .
> You are receiving this because you are subscribed to this thread.Message
> ID: ***@***.***>
>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] lordcheng10 commented on issue #16782: [PIP-169] Support split bundle by flow or qps
Posted by GitBox <gi...@apache.org>.
lordcheng10 commented on issue #16782:
URL: https://github.com/apache/pulsar/issues/16782#issuecomment-1221981754
> > Do you have some numbers about experiments ? Before accepting the proposal we must be sure that this algorithm works on only on the paper but also on some real benchmark/test env
>
> ### Test steps：
> 1.create Topic:
>
> ```
> sh bin/pulsar-admin tenants create test-tenant1
> sh bin/pulsar-admin namespaces create test-tenant1/test-namespace1
> sh bin/pulsar-admin topics create-partitioned-topic test-tenant1/test-namespace1/test1 -p 20
> sh bin/pulsar-admin topics create-partitioned-topic test-tenant1/test-namespace1/test2 -p 20
> sh bin/pulsar-admin topics create-partitioned-topic test-tenant1/test-namespace1/test3 -p 20
> sh bin/pulsar-admin topics create-partitioned-topic test-tenant1/test-namespace1/test4 -p 20
> sh bin/pulsar-admin topics create-partitioned-topic test-tenant1/test-namespace1/test5 -p 20
> sh bin/pulsar-admin topics create-partitioned-topic test-tenant1/test-namespace1/test6 -p 20
> sh bin/pulsar-admin topics create-partitioned-topic test-tenant1/test-namespace1/test7 -p 20
> sh bin/pulsar-admin topics create-partitioned-topic test-tenant1/test-namespace1/test8 -p 20
> sh bin/pulsar-admin topics create-partitioned-topic test-tenant1/test-namespace1/test9 -p 20
> sh bin/pulsar-admin topics create-partitioned-topic test-tenant1/test-namespace1/test10 -p 20
> sh bin/pulsar-admin topics create-partitioned-topic test-tenant1/test-namespace1/test11 -p 20
> sh bin/pulsar-admin topics create-partitioned-topic test-tenant1/test-namespace1/test12 -p 20
> sh bin/pulsar-admin topics create-partitioned-topic persistent://test-tenant1/test-namespace1/test695 -p 1
> ```
>
> 2.generate data:
>
> ```
> nohup sh bin/pulsar-perf produce test-tenant1/test-namespace1/test1 -threads 1 -n 1 -r 1000 > log1 &
> nohup sh bin/pulsar-perf produce test-tenant1/test-namespace1/test2 -threads 1 -n 1 -r 1000 > log2 &
> nohup sh bin/pulsar-perf produce test-tenant1/test-namespace1/test3 -threads 1 -n 1 -r 1000 > log3 &
> nohup sh bin/pulsar-perf produce test-tenant1/test-namespace1/test4 -threads 1 -n 1 -r 1000 > log4 &
> nohup sh bin/pulsar-perf produce test-tenant1/test-namespace1/test5 -threads 1 -n 1 -r 1000 > log5 &
> nohup sh bin/pulsar-perf produce test-tenant1/test-namespace1/test6 -threads 1 -n 1 -r 1000 > log6 &
> nohup sh bin/pulsar-perf produce test-tenant1/test-namespace1/test7 -threads 1 -n 1 -r 1000 > log7 &
> nohup sh bin/pulsar-perf produce test-tenant1/test-namespace1/test8 -threads 1 -n 1 -r 1000 > log8 &
> nohup sh bin/pulsar-perf produce test-tenant1/test-namespace1/test9 -threads 1 -n 1 -r 1000 > log9 &
> nohup sh bin/pulsar-perf produce test-tenant1/test-namespace1/test10 -threads 1 -n 1 -r 1000 > log10 &
> nohup sh bin/pulsar-perf produce test-tenant1/test-namespace1/test11 -threads 1 -n 1 -r 1000 > log11 &
> nohup sh bin/pulsar-perf produce test-tenant1/test-namespace1/test12 -threads 1 -n 1 -r 1000 > log12 &
> nohup sh bin/pulsar-perf produce persistent://test-tenant1/test-namespace1/test695 -threads 1 -n 11 -r 1010 > log695 &
> ```
>
> 3.the configuration is as follows: #defaultNamespaceBundleSplitAlgorithm=topic_count_equally_divide defaultNamespaceBundleSplitAlgorithm=flow_or_qps_equally_divide loadBalancerNamespaceBundleMaxMsgRate=1010 flowOrQpsDifferenceThresholdPercentage=10 exposeBundlesMetricsInPrometheus=true defaultNumberOfNamespaceBundles=1
>
> ### The comparison effect is as follows:
> **1.Using the topic_count_equally_divide algorithm:**
>
> The number of bundles after splitting is 21, and the **pulsar_bundle_msg_rate_in** of a single bundle is: **49 msg/s ~ 1010 msg/s**：
>
> 
>
> **2.Using the new algorithm flow_or_qps_equally_divide:** The number of bundles after splitting is 13, and the **pulsar_bundle_msg_rate_in** of a single bundle is: **997 msg/s ~ 1010 msg/s**： <img alt="image" width="1282" src="https://user-images.githubusercontent.com/19296967/185864580-198b92fb-ef0b-4390-9bfc-56a03a1b19f0.png">
@eolivelli @codelipenghui PTAL,thanks!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] lordcheng10 commented on issue #16782: [PIP-169] Support split bundle by flow or qps
Posted by GitBox <gi...@apache.org>.
lordcheng10 commented on issue #16782:
URL: https://github.com/apache/pulsar/issues/16782#issuecomment-1223426013
https://github.com/apache/pulsar/pull/16557
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] lordcheng10 commented on issue #16782: [PIP-169] Support split bundle by flow or qps
Posted by GitBox <gi...@apache.org>.
lordcheng10 commented on issue #16782:
URL: https://github.com/apache/pulsar/issues/16782#issuecomment-1223647385
vote thread: https://lists.apache.org/thread/g6stjsrymmq3850pdt9whzgx91tx33v8
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] Technoboy- commented on issue #16782: [PIP-169] Support split bundle by flow or qps
Posted by GitBox <gi...@apache.org>.
Technoboy- commented on issue #16782:
URL: https://github.com/apache/pulsar/issues/16782#issuecomment-1194880606
> @codelipenghui Need to resend the discussion email? Because the email header shows 164.
No need to re-send, you can reply to the latest PIP in the original email
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] lordcheng10 commented on issue #16782: [PIP-168] Support split bundle by flow or qps
Posted by GitBox <gi...@apache.org>.
lordcheng10 commented on issue #16782:
URL: https://github.com/apache/pulsar/issues/16782#issuecomment-1194875550
> @lordcheng10 The PIP number is duplicated with #15463
fixed
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] lordcheng10 commented on issue #16782: [PIP-169] Support split bundle by flow or qps
Posted by GitBox <gi...@apache.org>.
lordcheng10 commented on issue #16782:
URL: https://github.com/apache/pulsar/issues/16782#issuecomment-1200637754
> Do you have some numbers about experiments ? Before accepting the proposal we must be sure that this algorithm works on only on the paper but also on some real benchmark/test env
OK
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] lordcheng10 commented on issue #16782: [PIP-169] Support split bundle by flow or qps
Posted by GitBox <gi...@apache.org>.
lordcheng10 commented on issue #16782:
URL: https://github.com/apache/pulsar/issues/16782#issuecomment-1222292922
> > Do you have some numbers about experiments ? Before accepting the proposal we must be sure that this algorithm works on only on the paper but also on some real benchmark/test env
>
>
>
> ### Test steps：
>
> 1.create Topic:
>
>
>
> ```
>
> sh bin/pulsar-admin tenants create test-tenant1
>
> sh bin/pulsar-admin namespaces create test-tenant1/test-namespace1
>
> sh bin/pulsar-admin topics create-partitioned-topic test-tenant1/test-namespace1/test1 -p 20
>
> sh bin/pulsar-admin topics create-partitioned-topic test-tenant1/test-namespace1/test2 -p 20
>
> sh bin/pulsar-admin topics create-partitioned-topic test-tenant1/test-namespace1/test3 -p 20
>
> sh bin/pulsar-admin topics create-partitioned-topic test-tenant1/test-namespace1/test4 -p 20
>
> sh bin/pulsar-admin topics create-partitioned-topic test-tenant1/test-namespace1/test5 -p 20
>
> sh bin/pulsar-admin topics create-partitioned-topic test-tenant1/test-namespace1/test6 -p 20
>
> sh bin/pulsar-admin topics create-partitioned-topic test-tenant1/test-namespace1/test7 -p 20
>
> sh bin/pulsar-admin topics create-partitioned-topic test-tenant1/test-namespace1/test8 -p 20
>
> sh bin/pulsar-admin topics create-partitioned-topic test-tenant1/test-namespace1/test9 -p 20
>
> sh bin/pulsar-admin topics create-partitioned-topic test-tenant1/test-namespace1/test10 -p 20
>
> sh bin/pulsar-admin topics create-partitioned-topic test-tenant1/test-namespace1/test11 -p 20
>
> sh bin/pulsar-admin topics create-partitioned-topic test-tenant1/test-namespace1/test12 -p 20
>
> sh bin/pulsar-admin topics create-partitioned-topic persistent://test-tenant1/test-namespace1/test695 -p 1
>
> ```
>
>
>
> 2.generate data:
>
>
>
> ```
>
> nohup sh bin/pulsar-perf produce test-tenant1/test-namespace1/test1 -threads 1 -n 1 -r 1000 > log1 &
>
> nohup sh bin/pulsar-perf produce test-tenant1/test-namespace1/test2 -threads 1 -n 1 -r 1000 > log2 &
>
> nohup sh bin/pulsar-perf produce test-tenant1/test-namespace1/test3 -threads 1 -n 1 -r 1000 > log3 &
>
> nohup sh bin/pulsar-perf produce test-tenant1/test-namespace1/test4 -threads 1 -n 1 -r 1000 > log4 &
>
> nohup sh bin/pulsar-perf produce test-tenant1/test-namespace1/test5 -threads 1 -n 1 -r 1000 > log5 &
>
> nohup sh bin/pulsar-perf produce test-tenant1/test-namespace1/test6 -threads 1 -n 1 -r 1000 > log6 &
>
> nohup sh bin/pulsar-perf produce test-tenant1/test-namespace1/test7 -threads 1 -n 1 -r 1000 > log7 &
>
> nohup sh bin/pulsar-perf produce test-tenant1/test-namespace1/test8 -threads 1 -n 1 -r 1000 > log8 &
>
> nohup sh bin/pulsar-perf produce test-tenant1/test-namespace1/test9 -threads 1 -n 1 -r 1000 > log9 &
>
> nohup sh bin/pulsar-perf produce test-tenant1/test-namespace1/test10 -threads 1 -n 1 -r 1000 > log10 &
>
> nohup sh bin/pulsar-perf produce test-tenant1/test-namespace1/test11 -threads 1 -n 1 -r 1000 > log11 &
>
> nohup sh bin/pulsar-perf produce test-tenant1/test-namespace1/test12 -threads 1 -n 1 -r 1000 > log12 &
>
> nohup sh bin/pulsar-perf produce persistent://test-tenant1/test-namespace1/test695 -threads 1 -n 11 -r 1010 > log695 &
>
> ```
>
>
>
> 3.the configuration is as follows:
>
> ```
>
> #defaultNamespaceBundleSplitAlgorithm=topic_count_equally_divide
>
> defaultNamespaceBundleSplitAlgorithm=flow_or_qps_equally_divide
>
> loadBalancerNamespaceBundleMaxMsgRate=1010
>
> flowOrQpsDifferenceThresholdPercentage=10
>
> exposeBundlesMetricsInPrometheus=true
>
> defaultNumberOfNamespaceBundles=1
>
> ```
>
>
>
> ### The comparison effect is as follows:
>
> **1.Using the topic_count_equally_divide algorithm:**
>
>
>
> The number of bundles after splitting is **21**, and the **pulsar_bundle_msg_rate_in** of a single bundle is: **49 msg/s ~ 1010 msg/s**：
>
>
>
> 
>
>
>
>
>
>
>
> **2.Using the new algorithm flow_or_qps_equally_divide:**
>
> The number of bundles after splitting is **13**, and the **pulsar_bundle_msg_rate_in** of a single bundle is: **997 msg/s ~ 1010 msg/s**：
>
> <img width="1282" alt="image" src="https://user-images.githubusercontent.com/19296967/185864580-198b92fb-ef0b-4390-9bfc-56a03a1b19f0.png">
>
>
>
>
>
>
@eolivelli @Technoboy- @codelipenghui PTAL,thanks!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] heesung-sn commented on issue #16782: [PIP-169] Support split bundle by flow or qps
Posted by GitBox <gi...@apache.org>.
heesung-sn commented on issue #16782:
URL: https://github.com/apache/pulsar/issues/16782#issuecomment-1223572370
+1 LGTM.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org