Slack digest for #general - 2019-10-13

[Apache Pulsar Slack <ap...@gmail.com>]

2019-10-12 10:26:52 UTC - Shishir Pandey: @Sijie Guo since Pulsar's one use case is like a traditional message queue I was wondering if there is any plan to support XA transactions? If yes, cool, if not, any specific reasons other than the fact that somethings would be slow with XA transactions. I see that PIP-31 clearly states that access to  external databases or queues will not follow transactional semantics but given that TC outlined in the PIP appears to do something similar to a XA TM (across partitions) could it not be extended to allow for XA transactions. I also wanted to understand why would we not support serializability.
----
2019-10-12 10:31:27 UTC - Nicolas Ha: <https://pulsar.apache.org/api/client/org/apache/pulsar/client/api/ConsumerBuilder.html#ackTimeout-long-java.util.concurrent.TimeUnit->
```Set the timeout for unacked messages, truncated to the nearest millisecond. The timeout needs to be greater than 10 seconds.
```
Why 10 seconds? I’d like to be able to set it lower for tests so they run quicker
----
2019-10-12 14:03:12 UTC - Lawrence Pan: @Lawrence Pan has joined the channel
----
2019-10-12 14:48:41 UTC - Chris Bartholomew: I am wondering if someone can point me to some docs (or code) on how load is balanced between Bookkeeper bookies. There is a nice description of load balancing on the brokers:  <https://pulsar.apache.org/docs/en/administration-load-balance/>.  I know that the AutoRecovery process will fix under replicated ledgers, but I am wondering if there any load shedding mechanisms in Bookkeeper or a mechanism that prefers underloaded bookies. For example, when you add a new bookie, it immediately starts getting used. Jack Vanlightly has a great illustration of this (<https://jack-vanlightly.com/sketches/2018/10/2/kafka-vs-pulsar-rebalancing-sketch>). But is there a mechanism so that the underloaded bookie takes more of the load (if possible) so that's its boat fills up faster than the other ones?
----
2019-10-12 16:27:14 UTC - Matteo Merli: &gt;  I am wondering if there any load shedding mechanisms in Bookkeeper or a mechanism that prefers underloaded bookies.

Bookies are not acquiring ownership on ledgers, rather BK client select for a new ledger a new group of bookies at random, within the constraints of the placement policy.

&gt; For example, when you add a new bookie, it immediately starts getting used

That’s because ledgers are rolled-over. New ledgers will be allocated at random based on the expanded cluster.

&gt; But is there a mechanism so that the underloaded bookie takes more of the load (if possible) so that’s its boat fills up faster than the other ones?

You don’t want to do that. It creates an IO imbalance and will degrade performance.
----
2019-10-12 18:24:21 UTC - Chris Bartholomew: Thanks @Matteo Merli for your responses. Much appreciated.
----
2019-10-13 00:45:52 UTC - Joe Francis: I have never seen a case where an overloaded system does not rebalance in a few mins.  As long as you have reasonable ledger sizes, it should all be automatic.  There is also pushback and quarantine when overloaded bookies get slow
----