You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@pulsar.apache.org by GitBox <gi...@apache.org> on 2020/07/06 18:25:07 UTC
[GitHub] [pulsar-client-go] merlimat opened a new pull request #310: Share buffer pool across all partitions
merlimat opened a new pull request #310:
URL: https://github.com/apache/pulsar-client-go/pull/310
### Motivation
When a producer is publishing on many partitions, there can be significant memory overhead in maintaining a per-partition pool. Instead, there's not significant perf impact in using a single shared buffer pool.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar-client-go] wolfstudy commented on a change in pull request #310: Share buffer pool across all partitions
Posted by GitBox <gi...@apache.org>.
wolfstudy commented on a change in pull request #310:
URL: https://github.com/apache/pulsar-client-go/pull/310#discussion_r450990843
##########
File path: perf/perf-producer.go
##########
@@ -62,6 +63,8 @@ func newProducerCommand() *cobra.Command {
"Publish rate. Set to 0 to go unthrottled")
flags.IntVarP(&produceArgs.BatchingTimeMillis, "batching-time", "b", 1,
"Batching grouping time in millis")
+ flags.IntVarP(&produceArgs.BatchingMaxSize, "batching-max-size", "", 128,
+ "Max size of a batch (in KB)")
Review comment:
In `pulsar-perf produce` about `--batch-max-messages` is abbreviated and the default value is as follows, can we consider keeping the two consistent?
```
-bm, --batch-max-messages
Maximum number of messages per batch
Default: 1000
```
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar-client-go] wolfstudy commented on a change in pull request #310: Share buffer pool across all partitions
Posted by GitBox <gi...@apache.org>.
wolfstudy commented on a change in pull request #310:
URL: https://github.com/apache/pulsar-client-go/pull/310#discussion_r450990843
##########
File path: perf/perf-producer.go
##########
@@ -62,6 +63,8 @@ func newProducerCommand() *cobra.Command {
"Publish rate. Set to 0 to go unthrottled")
flags.IntVarP(&produceArgs.BatchingTimeMillis, "batching-time", "b", 1,
"Batching grouping time in millis")
+ flags.IntVarP(&produceArgs.BatchingMaxSize, "batching-max-size", "", 128,
+ "Max size of a batch (in KB)")
Review comment:
In `pulsar-perf produce` about `--batch-max-messages` is abbreviated and the default value is as follows, can I consider keeping the two consistent?
```
-bm, --batch-max-messages
Maximum number of messages per batch
Default: 1000
```
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar-client-go] merlimat commented on a change in pull request #310: Share buffer pool across all partitions
Posted by GitBox <gi...@apache.org>.
merlimat commented on a change in pull request #310:
URL: https://github.com/apache/pulsar-client-go/pull/310#discussion_r451842150
##########
File path: perf/perf-producer.go
##########
@@ -62,6 +63,8 @@ func newProducerCommand() *cobra.Command {
"Publish rate. Set to 0 to go unthrottled")
flags.IntVarP(&produceArgs.BatchingTimeMillis, "batching-time", "b", 1,
"Batching grouping time in millis")
+ flags.IntVarP(&produceArgs.BatchingMaxSize, "batching-max-size", "", 128,
+ "Max size of a batch (in KB)")
Review comment:
This are 2 different settings. One is the number of messages in 1 batch and the other is the max size of the batch
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar-client-go] merlimat commented on pull request #310: Share buffer pool across all partitions
Posted by GitBox <gi...@apache.org>.
merlimat commented on pull request #310:
URL: https://github.com/apache/pulsar-client-go/pull/310#issuecomment-655776643
> just only some concerns whether the use of sync.pool here will cause GC pressure because sync.Pool cannot specify a size, which is only subject to the GC threshold.
Using the pool will not causing GC pressure itself: it's actually there to avoid the GC pressure.
The pool having no max size is not a big problem. The max amount of memory is still determined by the "pending" messages whose payloads are buffered until they get acknowledged by the broker.
This change is to use a single pool to avoid that each pool will have a few buffers that cannot be immediately reused. After this change, the memory usage is the same as without the pooling.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar-client-go] wolfstudy removed a comment on pull request #310: Share buffer pool across all partitions
Posted by GitBox <gi...@apache.org>.
wolfstudy removed a comment on pull request #310:
URL: https://github.com/apache/pulsar-client-go/pull/310#issuecomment-654554221
/pulsarbot run-failure-checks
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar-client-go] wolfstudy merged pull request #310: Share buffer pool across all partitions
Posted by GitBox <gi...@apache.org>.
wolfstudy merged pull request #310:
URL: https://github.com/apache/pulsar-client-go/pull/310
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar-client-go] wolfstudy commented on pull request #310: Share buffer pool across all partitions
Posted by GitBox <gi...@apache.org>.
wolfstudy commented on pull request #310:
URL: https://github.com/apache/pulsar-client-go/pull/310#issuecomment-654554221
/pulsarbot run-failure-checks
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org