Slack digest for #general - 2019-08-14

[Apache Pulsar Slack <ap...@gmail.com>]

2019-08-13 10:16:08 UTC - Zhenhao Li: in the overview, it is not super clear what Journal storage's role is. Why do you need a transaction log if data are already persisted in the ledges in append-only fashion?
----
2019-08-13 10:19:09 UTC - Kim Christian Gaarder: So, about the intended behavior or Consumer.seek. Is it correct that when you seek to either Message.earliest or a specific message, then you get inclusive semantics, but when you seek to Message.latest then you get exclusive semantics. If so this makes it kind of hard go directly to the last message without using the admin api.
----
2019-08-13 10:56:52 UTC - Alexandre DUVAL: Hum, will read code.
----
2019-08-13 12:23:23 UTC - Patrick Wyeth: @Patrick Wyeth has joined the channel
----
2019-08-13 12:26:25 UTC - Sijie Guo: recyled seems to be related to double-releasing a bytebuf
----
2019-08-13 12:44:09 UTC - Alexandre DUVAL: so it's an issue on java pulsar client?
----
2019-08-13 12:46:42 UTC - Sijie Guo: No I think it is a broker side issue from the exception signature.
----
2019-08-13 13:55:54 UTC - Alexandre DUVAL: can you create an issue related to this? not sure how to well describe it
----
2019-08-13 14:43:45 UTC - Marcus Almeida: @Marcus Almeida has joined the channel
----
2019-08-13 14:45:17 UTC - Marcus Almeida: Hi guys!
I would like to know if is it possible to replay messages on Pulsar?
----
2019-08-13 14:47:35 UTC - Chris Bartholomew: For sure, you just need to use the reader interface instead of the consumer interface: <https://pulsar.apache.org/docs/en/concepts-clients/#reader-interface>
----
2019-08-13 14:53:34 UTC - Marcus Almeida: @Chris Bartholomew Thx for your help! I'm looking for Kafka's alternative and Pulsar appears until now a good option.
----
2019-08-13 15:08:17 UTC - Tarek Shaar: Quick question about subscribers. If there are no subscribers to a topic and a producer sends a message, is that message marked as acknowledged immediately? If yes how will new subscribers get the message? My understanding is that the message s deleted (by default) once acked.
----
2019-08-13 15:13:25 UTC - venki: @venki has joined the channel
----
2019-08-13 15:14:05 UTC - Chris Bartholomew: Without a subscription, the message is not held in the backlog for consumers. However, Pulsar can retain acknowledged messages. If the message is retained, new clients (connected after the message is sent) can retrieve the message using the reader interface.
----
2019-08-13 15:16:03 UTC - Chris Bartholomew: Message retention is described here: <https://pulsar.apache.org/docs/en/concepts-messaging/#message-retention-and-expiry>
----
2019-08-13 15:23:04 UTC - Tarek Shaar: Very interesting that by default if consumers are not active the messages will be missed. Thanks Chris
----
2019-08-13 15:25:19 UTC - Chris Bartholomew: In my experience, this is common in messaging systems that support a subscription model.
----
2019-08-13 15:27:20 UTC - Tarek Shaar: I thought Durable subscribers in JMS (for example) will get the message when they come back on line.
----
2019-08-13 15:27:55 UTC - venki: Regarding Pulsar Functions, the docs mention future support for other languages:

<https://pulsar.apache.org/docs/en/functions-overview/#supported-languages>

Is there a roadmap for this? Is C++ there on this roadmap in the nearterm?
----
2019-08-13 15:30:09 UTC - Chris Bartholomew: Yes, if you create a durable subscription first. However, if there are no matching subscriptions when a publishing a JMS message, it is not stored in any JMS implementation I have worked with.
----
2019-08-13 15:46:37 UTC - Marcus Almeida: What permissions the Pulsar Functions in GO need to be deployed?
I'm tryin' to deploy a executable function, but i received the message log "error=13, Permission denied"
----
2019-08-13 16:02:13 UTC - Sijie Guo: @Marcus Almeida I think you need make it have executable permissions.  I think there is a fix in master for the executable permission, which will be released as part of 2.4.1 release. /cc @xiaolong.ran he has more context on this.
----
2019-08-13 16:03:02 UTC - Sijie Guo: Python and Java are the two languages which are fully supported.
----
2019-08-13 16:03:07 UTC - Sijie Guo: Go is catching up.
----
2019-08-13 16:04:29 UTC - Sijie Guo: We didn’t see many requests on C++ functions. But if there are requests for it, we can consider prioritizing it.
----
2019-08-13 16:15:57 UTC - Marcus Almeida: @Sijie Guo Nice! I'll continue testing using Python and Java, and wait for 2.4.1 release.
----
2019-08-13 16:50:52 UTC - Tarek Shaar: So in Pulsar there is no Durable Sub model? The only way is to use the Reader API?
----
2019-08-13 17:11:49 UTC - Yuvaraj Loganathan: You can create an subscription and seek from the beginning or from specific time
----
2019-08-13 18:42:24 UTC - Marcus Almeida: Where can I find an example of a Function using SerDe implemented in Python?
----
2019-08-13 18:50:23 UTC - Steven: I am trying to understand how long it takes Pulsar's metrics at http://$BROKER_ADDRESS:8080/metrics to update to reflect current system state. For example, if I publish 1 message/second, then `pulsar_rate_in` is 1.0. But if I suddenly start publishing 2 messages/second, then `pulsar_rate_in` stays at 1.0 and takes ~60 seconds to change to 2.0. Is it possible to reduce this lag?
----
2019-08-13 18:53:57 UTC - Ali Ahmed: @Marcus Almeida take a look here <https://github.com/apache/pulsar/blob/master/pulsar-functions/python-examples/custom_object_function.py>
----
2019-08-13 18:54:23 UTC - Ali Ahmed: @Steven the default update frequency is 60 seconds
----
2019-08-13 18:54:42 UTC - Chris Bartholomew: @Marcus Almeida This example is in the repo: <https://github.com/apache/pulsar/blob/master/pulsar-functions/python-examples/custom_object_function.py>
----
2019-08-13 18:54:53 UTC - Chris Bartholomew: Is this what you are looking for?
----
2019-08-13 19:09:08 UTC - Marcus Almeida: @Ali Ahmed thx. I'll try to use something like this.
----
2019-08-13 19:21:55 UTC - Tarek Shaar: I am trying to create mutiple dynamic topics in Pulsar. But it seems that I need to create a producer and a topic together to be able to do that. How can I create a topics dynamically then associate multiple producers with it?
----
2019-08-13 19:22:56 UTC - Ali Ahmed: @Tarek Shaar you can use the pulsar admin to create topics
----
2019-08-13 19:23:20 UTC - Tarek Shaar: I need Dynamic topics depending in real time
----
2019-08-13 19:23:57 UTC - Ali Ahmed: if a producer or consumer is started with a topic name it will be auto created if it doesn’t exist
----
2019-08-13 19:27:24 UTC - Tarek Shaar: That's correct. But let's assume we create (dynamically) a topic called topic1 by creating a new producer. How can I use that same producer to publish to another topic? Surely the relation can't be 1 to 1. There has to be a way to create an N number of producers and use those to publish to as many dynamic topics as needed
----
2019-08-13 19:29:31 UTC - Ali Ahmed: I am not sure what you are trying to do if the goal is routing messages to specific topics you can can use pulsar functions for that.
----
2019-08-13 19:32:58 UTC - Tarek Shaar: In my application we use one topic per user. Am trying to pool a number of producers (for example 10) and reuse those ones to publish to different topics rather than having to create a new producer for each new topic.
----
2019-08-13 20:02:59 UTC - goutham m: @goutham m has joined the channel
----
2019-08-13 20:53:27 UTC - Karthik Ramasamy: <https://www.jesse-anderson.com/2018/08/creating-work-queues-with-apache-kafka-and-apache-pulsar/>
----
2019-08-13 21:02:00 UTC - Chris Bartholomew: Looking at the binary protocol docs (<https://pulsar.apache.org/docs/en/develop-binary-protocol/#producer>), I'd be surprised if this is possible. It looks like topic authorization is done when the producer is created and then messages are sent in the context of the producer. So, to send on a different topics, you would need a different producer.
----
2019-08-14 01:30:49 UTC - msk: Oh that is great, i thought it cannot run without docker. Thanks a lot:) I will try this and get back.
----
2019-08-14 06:41:31 UTC - Kim Christian Gaarder: There is. When you subscribe, that subscription will live forever until you unsubscribe, even if you consumer client is disconnected.
----
2019-08-14 06:42:28 UTC - Kim Christian Gaarder: Then when you reconnect/resubscribe on the same subscription later, you will get all the messages that arrives since after the message that was last acknowledged
----
2019-08-14 08:01:49 UTC - dba: Hi guys. Just want to give you a quick update on the development of the Pulsar client for C#.
We have a working client and we are currently testing it.
Featurewise this is what we will be releasing:
* Client: Supports auto reconnect, lookup and server initiated closing of producers and consumers.
* Consumer: Optionally setting name, initial position, priority level and receiver queue size. Exclusive, Shared and Failover (with ACTIVE_CONSUMER_CHANGE) subscription types are supported. Individual and Cumulative acknowledgement, getting last message id, receiving messages (and REACHED_END_OF_TOPIC), seek and unsubscribe.
* Producer: Optionally setting name and initial sequence id. Can produces messages with and without metadata.
* Reader: Optionally setting name and receiver queue size.
The client is written entirely in C# and the only dependencies are protobuf-net and System.IO.Pipelines.
So, it is going great and as promised it will be released under the Apache License 2.0. We aim for moving to GitHub and creating a NuGet package in October (we are waiting for C# 8.0 and .NET Core 3.0 to be released).
+1 : Ali Ahmed, Sijie Guo, Henrik Andersen
100 : Sijie Guo
tada : Sijie Guo
----
2019-08-14 08:05:11 UTC - Ali Ahmed: is there a F# wrapper also ?
----
2019-08-14 08:11:33 UTC - dba: Having never used F# I am not 100% sure, but since C# is just compiled to MSIL, I would guess that you can use it from F# and the other .NET languages.
----