You are viewing a plain text version of this content. The canonical link for it is here.

Posted to commits@pulsar.apache.org by GitBox <gi...@apache.org> on 2022/12/01 09:30:52 UTC

[GitHub] [pulsar] Hongten opened a new issue, #18699: [Issue] The throttling is not exact for both publish and dispatch in pulsar-2.10.1 version

Hongten opened a new issue, #18699:
URL: https://github.com/apache/pulsar/issues/18699

### Search before asking

- [X] I searched in the [issues](https://github.com/apache/pulsar/issues) and found nothing similar.

### Version

pulsar-2.10.1 runs on the k8s cluster.

### Minimal reproduce step

## Precondition
There is only one topic under this namespace.
Topic: persistent://kafka_test/test/my_test_par_10
Partition Num: 10
Broker Num: 3
Bookie Num: 4
Broker Metrics: `pulsar_throughput_in`, `pulsar_throughput_out`

### Producer
`bin/pulsar-perf produce --batch-max-messages 30000 --num-producers 2 --num-test-threads 2 --num-io-threads 2 --rate 30000 --size 1000 persistent://kafka_test/test/my_test_par_10`

### Consumer
`bin/pulsar-perf consume persistent://kafka_test/test/my_test_par_10 -ss sub201`

## Dispatch Rate Test
Please refer to [PIP-3](https://github.com/apache/pulsar/wiki/PIP-3:-Message-dispatch-throttling)
We test the byte and message dispatch rates separately. The test results are not excepted(Not exact).

### Byte Dispatch Rate
Only set the byte dispatch rate = 1000000(1MB). The excepted result is 1MB/s. However, the test result is 11.9MB/s.(Ref to broker metric `pulsar_throughput_out `)
`bin/pulsar-admin namespaces set-dispatch-rate kafka_test/test --byte-dispatch-rate 1000000 --dispatch-rate-period 1`
![image](https://user-images.githubusercontent.com/3076703/205009042-f8840015-be2f-43ae-8a5f-5c45a55a6499.png)

### Message Dispatch Rate
Only set the message dispatch rate = 1000. The excepted result is 1MB/s. However, the test result is 12.5MB/s.(Ref to broker metric `pulsar_throughput_out `)
`bin/pulsar-admin namespaces set-dispatch-rate kafka_test/test --msg-dispatch-rate 1000 --dispatch-rate-period 1`
![image](https://user-images.githubusercontent.com/3076703/205009703-afe9efbc-eb46-412d-9e33-933d4dbbde59.png)

## Publish Rate Test
We test the byte and message publish rates separately. The test results are not excepted.

### Byte Publish Rate
Only set the byte publish rate = 1000000(1MB). The excepted result is 1MB/s. However, the test result is 8.9MB/s.(Ref to broker metric `pulsar_throughput_in `)
`bin/pulsar-admin namespaces set-publish-rate kafka_test/test --byte-publish-rate 1000000`
![image](https://user-images.githubusercontent.com/3076703/205012327-03965948-9c9c-43c3-965c-fd630cff1449.png)

### Message Publish Rate
Only set the message publish rate = 1000. The excepted result is 1MB/s. However, the test result is 8.8MB/s.(Ref to broker metric `pulsar_throughput_in `)
`bin/pulsar-admin namespaces set-publish-rate kafka_test/test --msg-publish-rate 1000`
![image](https://user-images.githubusercontent.com/3076703/205016401-13c33662-6441-4946-a921-b5108d4879aa.png)

## Resource Group
Please refer to [PIP-82](https://github.com/apache/pulsar/wiki/PIP-82:-Tenant-and-namespace-level-rate-limiting)
I tested both publish rate and dispatch rate in resourcegroup. However, the test results show the sourcegroup throttling does not work exactly.

## Publish Rate

### Message Publish Rate
Only set the message publish rate = 100. One message size = 1024. The excepted result is 100KB/s. However, the result is 14.7MB/s.
```
bin/pulsar-admin resourcegroups create rs_name --msg-publish-rate 100
bin/pulsar-admin namespaces set-resource-group kafka_test/test --resource-group-name rs_name
```
![image](https://user-images.githubusercontent.com/3076703/205013479-cd4fcf39-9341-42fe-bbf4-2516688ac2c9.png)

### Byte Publish Rate
Only set the byte publish rate = 10000(1MB). One message size = 1024. The excepted result is 10KB/s. However, the result is 11MB/s.
```
bin/pulsar-admin resourcegroups create rs_name --byte-publish-rate 10000
```
![image](https://user-images.githubusercontent.com/3076703/205013899-9e39b272-aa59-492a-b9ae-54905c0503c9.png)

## Dispatch Rate

### Byte Dispatch Rate
Only set the byte dispatch rate = 1000(1MB). One message size = 1024. The excepted result is 10KB/s. However, the result is 20MB/s.
```
bin/pulsar-admin resourcegroups create rs_name --byte-dispatch-rate 10000
```
![image](https://user-images.githubusercontent.com/3076703/205014116-ceaab3bd-7e5e-42af-a176-1f9f84b8db55.png)

### Message Dispatch Rate
Only set the message dispatch rate = 100. One message size = 1024. The excepted result is 100KB/s. However, the result is 21.2MB/s.
```
bin/pulsar-admin resourcegroups create rs_name --msg-dispatch-rate 100
```
![image](https://user-images.githubusercontent.com/3076703/205014247-ad3f5c18-bf96-4882-bb6c-e45564d0c0fb.png)

### What did you expect to see?

I provide the excepted value in the above reproduction steps.

### What did you see instead?

Please refer to the reproduction steps.

### Anything else?

_No response_

### Are you willing to submit a PR?

- [ ] I'm willing to submit a PR!

--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [pulsar] Technoboy- commented on issue #18699: [Issue] The throttling is not exact for both publish and dispatch in pulsar-2.10.1 version

Posted by "Technoboy- (via GitHub)" <gi...@apache.org>.

Technoboy- commented on issue #18699:
URL: https://github.com/apache/pulsar/issues/18699#issuecomment-1649401599

   Do you set `preciseTopicPublishRateLimiterEnable=true` ?


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [pulsar] Hongten commented on issue #18699: [Issue] The throttling is not exact for both publish and dispatch in pulsar-2.10.1 version

Posted by GitBox <gi...@apache.org>.

Hongten commented on issue #18699:
URL: https://github.com/apache/pulsar/issues/18699#issuecomment-1333490876

   Can you help to review it when you are free?
   @sijie @jiazhai @futeng 
   
   Thanks
   Hongten


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [pulsar] github-actions[bot] commented on issue #18699: [Issue] The throttling is not exact for both publish and dispatch in pulsar-2.10.1 version

Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.

github-actions[bot] commented on issue #18699:
URL: https://github.com/apache/pulsar/issues/18699#issuecomment-1694110368

   The issue had no activity for 30 days, mark with Stale label.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [pulsar] github-actions[bot] commented on issue #18699: [Issue] The throttling is not exact for both publish and dispatch in pulsar-2.10.1 version

Posted by GitBox <gi...@apache.org>.

github-actions[bot] commented on issue #18699:
URL: https://github.com/apache/pulsar/issues/18699#issuecomment-1368323639

   The issue had no activity for 30 days, mark with Stale label.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org