Slack digest for #dev - 2020-07-14

[Apache Pulsar Slack <ap...@gmail.com>]

2020-07-13 14:27:42 UTC - Dave Miller: @Dave Miller has joined the channel
----
2020-07-13 17:20:52 UTC - Julius S: Hi All, would be interesting to talk within Pulsar community about the auto-scaling layer Pravega team are working on ontop of Bookeeper (at least that is how I currently understand it). What do people think? What opportunities does this open? Challenges?: <http://pravega.io/docs/latest/pravega-concepts/#elastic-streams-auto-scaling>
raised_hands : Konstantinos Papalias
100 : Konstantinos Papalias
----
2020-07-13 17:46:20 UTC - Addison Higham: I am a tiny bit familiar with Pravega, would be interesting to see if there are some concepts there Pulsar could adopt, but from what I understand Pulsar and Pravega take a fairly different approach to the client. AFAIK, the prevaga client is fairly "thick" with a fair bit of logic in the client for how it does many of the auto-scaling features
----
2020-07-13 17:49:56 UTC - Ali Ahmed: there has been some discussion of auto scale mechanism for pulsar topics , it’s part of pulsar 3.0 roadmap, the whole notion of partitioned / non-partitioned topic can disappear, the hard is how to maintain compatibility with prior topic declarations.
ok_hand : Konstantinos Papalias, Julius S
----
2020-07-13 17:54:11 UTC - Addison Higham: on a related note, a question I am not sure on: what are the semantics of key_shared topics supposed to be with topic count changes?
----
2020-07-13 17:56:36 UTC - Addison Higham: AFAIK, ATM, it isn't guaranteed to preserve order when you added new topics, but I wasn't sure if that is a "bug" (or known gap while it is in beta) and if the eventual semantics are supposed to include ordering guarantees even as topology changes.
----
2020-07-13 20:12:46 UTC - Pedro Cardoso: Are there any design documents for the topic auto-scaling feature? If so can anyone share?
----
2020-07-13 20:13:38 UTC - Matteo Merli: Not yet, we plan to work on this in q3/q4
ok_hand : Miguel Martins
----
2020-07-13 20:13:58 UTC - Pedro Cardoso: Thank you for the reply!
----
2020-07-13 20:15:06 UTC - Pedro Cardoso: I eagerly await your work! I’ve been looking at Kafka mostly because of this feature, it would be great if pulsar had some thing similar if not equivalent
+1 : Julius S, Matteo Merli
----
2020-07-13 20:29:37 UTC - Julius S: Is the thinking that the scaling unit would be a partition (i.e. internal topic)? (As @Ali Ahmed seemed to imply)
----
2020-07-13 21:27:20 UTC - Raman Gupta: Kafka does not have this feature AFAIK.
----
2020-07-13 22:38:37 UTC - Pedro Cardoso: I may have misunderstood, was under the impression topic auto-scaling is somewhat similar to <https://cwiki.apache.org/confluence/display/KAFKA/KIP-429%3A+Kafka+Consumer+Incremental+Rebalance+Protocol|https://cwiki.apache.org/confluence/display/KAFKA/KIP-429%3A+Kafka+Consumer+Incremental+Rebalance+Protocol>
----
2020-07-13 22:40:01 UTC - Pedro Cardoso: There will be differences of course, particularly considering Kafka has a fixed amount of partitions to scale out to.
----
2020-07-13 22:57:50 UTC - Raman Gupta: No, not the same thing. Pulsar doesn't need anything like incremental rebalance because Pulsar consumers don't work like Kafka Consumers, where all server partitions are distributed across the available clients. The closest equivalent to that in Pulsar is Key_Shared mode. <https://pulsar.apache.org/docs/en/concepts-messaging/#key_shared>
----
2020-07-13 22:58:52 UTC - Raman Gupta: I believe the plan for auto-scaling in Pulsar is more about automatically adjusting how many partitions a topic has. Kafka has no equivalent capability -- adjusting numbers of partitions in Kafka is a manual operation (unless you have other tools to help with this) and incremental rebalance is not related to that at all.
----
2020-07-14 07:52:00 UTC - Cristian COLA: @Cristian COLA has joined the channel
----