Slack digest for #general - 2019-09-14

[Apache Pulsar Slack <ap...@gmail.com>]

2019-09-13 09:14:45 UTC - vikash: @Vladimir Shchur  how  to   use  Key  shared  Model  using  F#  client
----
2019-09-13 09:15:02 UTC - vikash: currently normal  subscription  Key  working
----
2019-09-13 09:15:10 UTC - vikash: any  guidence  how  to  pass  key
----
2019-09-13 09:15:23 UTC - Vladimir Shchur: Not yet implemented
----
2019-09-13 09:17:27 UTC - vikash: do  we  have  in  roadmap  for   it
----
2019-09-13 09:19:08 UTC - Vladimir Shchur: Yes, currently up for grabs :smiley: <https://github.com/fsharplang-ru/pulsar-client-dotnet/issues/53>
If nobody helps I'll start if after adding partition topics support
----
2019-09-13 09:20:54 UTC - vikash: Thanks  for  Info  looking  forward  to  support
----
2019-09-13 10:12:39 UTC - Walter: Any one can help on this problem because memory is keep on growing. we have to fix the issue
----
2019-09-13 13:56:38 UTC - Tarek Shaar: Can someone please provide me with the command to allow my Name Space to handle 50k topics? There is a bundle setting of some sort
----
2019-09-13 14:03:58 UTC - Matteo Merli: When you create a namespace, pass `-b 500`
----
2019-09-13 14:07:59 UTC - Matteo Merli: Not at this point, but it wouldn’t be too hard to implement a recovery that would work in that way 
----
2019-09-13 15:26:56 UTC - GerhardM: @GerhardM has joined the channel
----
2019-09-13 17:20:35 UTC - Tarek Shaar: @Matteo Merli thanks. I am assuming that when I create my topics under my new work space, the topics' owner ship will be spread across my brokers?
----
2019-09-13 17:22:37 UTC - Matteo Merli: Correct. There’s a brief description of the mechanics of traffic load balancing at <https://pulsar.apache.org/docs/en/administration-load-balance/>
----
2019-09-13 17:38:32 UTC - Nicolas Ha: No rush but would be interested in that too
----
2019-09-13 18:36:40 UTC - Thomas Cothran: I just started looking at pulsar, and it looks fantastic! Really ticks a lot of boxes. I have a few questions:

- Is there a smooth path moving from the standalone mode to the recommended production setup? I'm thinking about incrementally bringing this into the project, and it would be convenient to do it this way.
- It would be awesome if pulsar could work for event sourcing. Currently, Kafka is held back by KAFKA-2260 (<https://issues.apache.org/jira/browse/KAFKA-2260>), which isn't being prioritized. Is that something that would be solved in Pulsar by PIP-31 (<https://github.com/apache/pulsar/issues/2664>)?
- One thing that seems nice about Kafka streams is that it is just a library. Pulsar functions look like they have to run on the broker, or you can run pulsar separately in a local instance. I'm interested in the library approach, because I use Clojure, and being able to use Kafka streams as a library fits well into the repl experience. Is there anything analogous for pulsar?
----
2019-09-13 18:54:23 UTC - David Kjerrumgaard: "- Is there a smooth path moving from the standalone mode to the recommended production setup? I'm thinking about incrementally bringing this into the project, and it would be convenient to do it this way."  Yes there is. Pulsar is designed to be deployed in cloud environments, and both the serving and storage layers are independent from one another.  Any cluster spun up for dev/testing/evaluation can be migrated over to production very easily.
----
2019-09-13 18:57:37 UTC - David Kjerrumgaard: "- It would be awesome if pulsar could work for event sourcing. Currently, Kafka is held back by KAFKA-2260 (<https://issues.apache.org/jira/browse/KAFKA-2260>), which isn't being prioritized. Is that something that would be solved in Pulsar by PIP-31 (<https://github.com/apache/pulsar/issues/2664>)?"  I guess that would depended on the requirements you are looking for to support event sourcing. Pulsar does support TRUE infinite message retention, so it can easily be used as an event store, which solves a very big piece of the puzzle. You can then replay the sequence of events to create projections of the current state of specific entities using either Pulsar consumers or the Pulsar SQL plugin.
----
2019-09-13 19:02:01 UTC - David Kjerrumgaard: "- One thing that seems nice about Kafka streams is that it is just a library. Pulsar functions look like they have to run on the broker, or you can run pulsar separately in a local instance. I'm interested in the library approach, because I use Clojure, and being able to use Kafka streams as a library fits well into the repl experience. Is there anything analogous for pulsar?"  Pulsar functions are designed for simple event processing. For more complex event processing scenarios, Pulsar integrates with both Spark Streaming, Flink, and Heron which provides a nice fluent library called Streamlets.....<https://apache.github.io/incubator-heron/docs/developers/java/streamlet-api/>
----
2019-09-13 19:15:59 UTC - Thomas Cothran: @David Kjerrumgaard

Thanks! This is super helpful. I saw in the docs that there were adapters for Storm and Spark, but didn't know about the ones for Flink and Heron. I probably wouldn't have an immediate need for these, but I do want to know they're there down the road.

On the event sourcing topic, the thing I had in mind was the ability for an aggregate to enforce its boundaries with concurrent writes. For instance, using a SQL database, an aggregate can accept a command and write to an event stream and roll back the transaction if there has been a write to the event stream in the mean time.

My understanding of KAFKA-2260 is that, were it implemented, my aggregate could accept a command and write an event, unless the offset has changed, in which case the command could re-run. Here's a better explanation than the one I'm giving: <https://medium.com/serialized-io/apache-kafka-is-not-for-event-sourcing-81735c3cf5c>
----
2019-09-13 20:05:06 UTC - Sagar Shah: @Sagar Shah has joined the channel
----
2019-09-13 20:59:08 UTC - David Kjerrumgaard: Yes, this scenario could be addressed by adding the Pulsar transactional semantics.....
heart_eyes : Poule
+1 : Shishir Pandey
----
2019-09-14 02:55:14 UTC - Poule: optimistic concurrency would be great.
----
2019-09-14 03:06:38 UTC - Poule: @Thomas Cothran data is not movable directly from standalone to cluster.
----
2019-09-14 03:10:19 UTC - Poule: I wonder if it's possible to start with 1 zk + 1 pulsar node then grow to 3 zk + 3 pulsar later
----
2019-09-14 04:12:50 UTC - Cory Davenport: @Cory Davenport has joined the channel
----
2019-09-14 05:00:42 UTC - Cory Davenport: Question regarding the functionality of a subscription within pulsar.
If this functionality exist how do I go about using it.
I have a case where a new subscription will be made on  a topic. But upon creation i need all messages to be sent to the subscription. But it seems to be only new messages once its created will be sent to the subscription. Is there a way to view the backlog since a topic was created or just the backlog of the topic of un-ack messages before the subscription was created?
----
2019-09-14 05:10:46 UTC - Xiaoyong: @Xiaoyong has joined the channel
----