Slack digest for #general - 2018-09-14

[Apache Pulsar Slack <ap...@gmail.com>]

2018-09-13 09:36:28 UTC - Xavier Baques: Hi, I am just testing Pulsar standalone (./bin/pulsar standalone) throughput with a simple application with one producer, writing to a persistent topic, created by next code:

```
client = PulsarClient.builder()
                         .serviceUrl("<pulsar://localhost:6650>")
                         .build();
producer = client.newProducer()
                     .topic("test-pulsar-topic")
                     .enableBatching(true)
                     .batchingMaxMessages(2000)
                     .batchingMaxPublishDelay(1, TimeUnit.SECONDS)
                     .create();
```

and sending messages using send() method.
I am a little bit concerned since I am getting a throughput of 582 messages/sec. I would expect a bigger number here so maybe I am doing something wrong. Any idea that could help me improving the throughput? Thanks in advance
----
2018-09-13 10:54:19 UTC - Matteo Merli: &gt; and sending messages using send() method.

@Xavier Baques the reason is that `send()` is the blocking version of publishing. It will block until the message is persisted, and since you have 1 thread, it is a 1 by 1 publishing. If the average latency is 2 millis (by default the messages are synced to disk), then you have the ~500 msg/s number.

Also, since you force to publish 1 by 1, batching doesn’t have any effect.

To have higher throughput, you need to pipeline more messages from client to broker, using the `sendAsync()` call which gives a future to track the completion of each operation.

Eg:
```
producer.sendAsync("hello").thenAccept(msgId -&gt; {
    <http://log.info|log.info>("Successfully published message {}", msgId);
}).exceptionally(ex -&gt; {
    log.warn("Failed to publish message: {}", ex.getMessage());
    return null;
});
```

Note: When using `sendAsync()` be aware of what happens if you try to send faster than the system can absorb. By default, when the producer queue is full, the writes are rejected. You can control that behavior with `ProducerBuilder.blockIfQueueFull(bool)` setting.

You can also try the pulsar-perf cli tool: `bin/pulsar-perf produce $TOPIC --rate 50000`
----
2018-09-13 12:33:58 UTC - Xavier Baques: thanks a lot for this information! will go this way then
----
2018-09-13 12:44:18 UTC - Alejandro Abdelnur: Hello, if a producer is configured with 'enabledBatching(true)', batching is only used for 'asyncSend()' or is also used for 'send()'? Asking because I'm not seeing a significant difference in throughput using 'send()' even if manipulating 'batchingMaxMessages(##)' and 'batchingMaxPublishingDelay(##)'. thanks.
----
2018-09-13 12:45:04 UTC - Matteo Merli: only when asyncSend() — send() will force to immediately close and send out a pending batch
----
2018-09-13 12:46:43 UTC - Alejandro Abdelnur: got it, thx.
----
2018-09-13 15:10:49 UTC - Josh West: @Matteo Merli @Sijie Guo what's the purpose of the `streamlio` docker hub repo vs the `apachepulsar` one?
----
2018-09-13 16:34:25 UTC - Matteo Merli: @Josh West that's where we push builds before they are "officially sanctioned Apache releases"
----
2018-09-13 17:24:44 UTC - Josh West: ahh awesome
----