You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@pulsar.apache.org by Apache Pulsar Slack <ap...@gmail.com> on 2018/05/25 09:11:02 UTC
Slack digest for #general - 2018-05-25
2018-05-24 09:17:36 UTC - Ali Ahmed: 3) there is no telegraph support directly but telegraph has plugins to read from prometheus inputs , so you should be able to deploy them
----
2018-05-24 09:20:44 UTC - Tomer Lev: OK I'll try that thanks
----
2018-05-24 09:31:08 UTC - David Asher: Hey there, the most recent client library (2.0.0-incubating) doesn't seem to be on maven central. Is there any other repo where this artifact is avail?
----
2018-05-24 10:12:51 UTC - David Asher: I am currently trying out Pulsar's queuing mechanism in order to do some load balancing. The idea is to have a single publisher issuing tasks on a topic and several workers in shared subscription mode performing the tasks. The problem I came across is that, if the producer already has published all its messages and I fire up another worker it doesn't see any messages i.e. receive() just blocks. The broker doesn't seem to include the new subscription in it's round robin loop. Is there a way to tell the broker to consider new subscriptions if there are no new messages being published but still unacked? Thanks in advance
----
2018-05-24 15:33:46 UTC - Sijie Guo: @David Asher
> he most recent client library (2.0.0-incubating) doesn’t seem to be on maven central
the 2.0.0-incubating release is still under apache incubation voting. should be out in a few days.
if you want to try it now, try to use 2.0.0.-incubating-SNAPSHOT, which it is in apache staging repository.
> Is there a way to tell the broker to consider new subscriptions if there are no new messages being published but still unacked?
you probably can use a different subscription name? Not sure if that addresses your problem?
----
2018-05-24 15:37:49 UTC - David Asher: Thank you! I experimented a bit and found out, that the problem was that the pulsar client was reading ahead (and eating up all messages in the topic) because the default value for receiverQueueSize was too high :slightly_smiling_face:
----
2018-05-24 15:40:34 UTC - Sijie Guo: @David Asher oh for the message queue use case, you might consider setting receiverQueueSize to be 1 for 1) experimental purpose 2) low throughput 3) your message processing time is high. you can checkout the cookbook about using pulsar as a message queue - <https://pulsar.incubator.apache.org/docs/latest/cookbooks/message-queue/> or the blog post written by @Luc Perkins <https://streaml.io/blog/pulsar-message-queue/>
----
2018-05-24 15:43:32 UTC - David Asher: That's what I end up doing. I don't need high throughput and message processing is on average ~30 seconds. Thanks for the link!
----
2018-05-24 15:43:50 UTC - Sijie Guo: @David Asher cool!
----
2018-05-24 15:50:50 UTC - Matteo Merli: @David Asher you can also completely disable prefetching by setting `receiverQueueSize=0`
----
2018-05-24 15:51:17 UTC - Matteo Merli: that way a message will only be pushed to a worker when it’s ready to process it
----
2018-05-24 16:53:00 UTC - Harald Gustafsson: If I want to prevent ordinary clients from creating new topics in a namespace. Would I then create a namespace without roles of ordinary clients and then grant permissions on a per topic basis that then ordinary clients could produce and consume from?
----
2018-05-24 18:38:54 UTC - Matteo Merli: Hi @Harald Gustafsson, you’re right in that there’s no straightforward way to prevent topics to be crated by a client that has access on the namespace. We used to have “create” and “delete” actions on the permissions in the early days but eventually took them out because they were not really used. It would be good to have a simple way to disable topic auto-creation and only allow admins to pre-create topics.
Having said that, the workaround you described sounds a good option. It shouldn’t be too painful if you don’t have to configure many topic names
----
2018-05-24 18:41:06 UTC - Ivan Kelly: @Matteo Merli can you grant per topic? I thought this was all applied per namespace?
----
2018-05-24 18:41:35 UTC - Matteo Merli: you can also apply per topic, it just augments the permission already granted on the namespace
----
2018-05-24 18:41:46 UTC - Ivan Kelly: oh, i didn't know that
----
2018-05-24 18:42:39 UTC - Matteo Merli: <http://pulsar.apache.org/docs/latest/reference/CliTools/#pulsar-admin-topics-grant-permission>
----
2018-05-24 18:44:11 UTC - Ivan Kelly: :bath:
----
2018-05-24 18:44:15 UTC - Ivan Kelly: :thumbsup:
----
2018-05-24 18:44:25 UTC - Ivan Kelly: bloody slack emoji search
----
2018-05-24 18:46:42 UTC - Harald Gustafsson: The tool I make its going to create the topics so it is no overhead, but want to be sure no non authorized topics are created, that could circumvent security policies.
----
2018-05-24 18:47:34 UTC - Harald Gustafsson: @Matteo Merli thanks for the info
----
2018-05-24 18:48:25 UTC - Ali Ahmed: @Ivan Kelly you can delete or edit messages
----
2018-05-24 18:49:25 UTC - Ivan Kelly: i know, but the :bath: amused me, so i left it. for some reason ': t h' completes to bath now
----
2018-05-24 18:50:36 UTC - Matteo Merli: > The tool I make its going to create the topics so it is no overhead, but want to be sure no non authorized topics are created, that could circumvent security policies.
Yes, just leave the namespace level permission to admin role only
----
2018-05-24 18:56:32 UTC - Matteo Merli: @Tomer Lev
> 1. is there a unique message id for each message that is expose to the consumers?
Yes, each message has an associated message id. It’s an opaque interface, but can be serialized, stored and deserialized back
> 2. Is there any retry counter in the consumed message to indicates how much times it was consumed - if not is there any implementation for retries in Pulsar ?
No, there is currently no such counter. With acknowledge timeout, the broker will keep re-delivering the messages indefinitely at regular intervals. We have been thinking on this issue for a while but still haven’t got around to do that so far.
> 3. Is there telegraf input for Pulsar monitoring ? I saw only Prometheus in documentation,
You can fetch the metrics data either in Prometheus format or in custom JSON. I haven’t looked in details at Telegraf but I was under the impression that it was able to fetch metrics from Prometheus sources.
----
2018-05-24 22:19:55 UTC - Steven Le Roux: @Steven Le Roux has joined the channel
----