You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@pulsar.apache.org by Apache Pulsar Slack <ap...@gmail.com> on 2019/08/06 09:11:07 UTC
Slack digest for #general - 2019-08-06
2019-08-05 09:23:38 UTC - Matthieu Nicolas: Hi, i am working on a JS application relying on Pulsar to share messages between peers and i am wondering : does Pulsar's Websocket service ensure that messages are delivered and acknowledged in the same order as they are emitted by a producer ?
For example, in the following scenario, will the first ack received corresponds to `msg1`, the second to `msg2` and so on ?
```
pulsarWS.send(msg1)
pulsarWS.send(msg2)
pulsarWS.send(msg3)
...
```
My underlying goal is to implement an _at-least-once delivery_ of messages by the producers.
Thanks in advance !
----
2019-08-05 09:25:40 UTC - Sijie Guo: Consumer has a package-private method `getLastMessageIdAsync`. We can consider exposing it as one method for Consumer. /cc @jia zhai
----
2019-08-05 09:28:04 UTC - bsideup: That would be amazing! :heart_eyes: Especially the async one :slightly_smiling_face:
----
2019-08-05 09:29:18 UTC - jia zhai: Ok,I will handle it in this week.
heart : bsideup
+1 : bsideup
----
2019-08-05 09:31:17 UTC - jia zhai: @Sijie Guo Single partitioned consumer is direct, What do you think of the return value for partitioned/topics consumer?
----
2019-08-05 09:32:31 UTC - bsideup: FTR I do plan to use it with partitioned topics
----
2019-08-05 09:33:26 UTC - Sijie Guo: I think we should consider introducing a new sub-type of `MessageId`, which is a map between partition -> MessageId.
This would be useful for adding supports for various things, like seek and ackCummulative.
+1 : bsideup
----
2019-08-05 09:40:00 UTC - jia zhai: make sense
----
2019-08-05 11:09:02 UTC - Alexandre DUVAL: @Jerry Peng?
----
2019-08-05 12:53:50 UTC - Aaron: Hi, I am seeing duplicate messages in the same partition (in a multiple partitioned topic) with different message IDs when a consumer breaks it's connection and reconnects. Any idea why this would happen, and if there is a way to prevent it?
----
2019-08-05 13:05:51 UTC - Sijie Guo: messages with different message ids are two *distinct* messages. they might be duplicated at producer side. You can consider enable de-duplication to use sequence id to prevent duplications. see: <http://pulsar.apache.org/docs/en/cookbooks-deduplication/>
----
2019-08-05 13:06:17 UTC - Sijie Guo: @jia zhai can you create a github issue? so @bsideup can use it to track the progress?
+1 : bsideup
----
2019-08-05 13:06:57 UTC - Sijie Guo: I think so. if the messages are sent in the same connection.
----
2019-08-05 13:09:39 UTC - bsideup: works like a charm with an accessor:
<https://github.com/bsideup/liiklus/pull/164/files#diff-5b0382b88d540af0a10d2cd9bc7648e9R8>
+1 : Sijie Guo
----
2019-08-05 13:11:18 UTC - jia zhai: :+1:
----
2019-08-05 13:20:37 UTC - Sijie Guo: @bsideup smart
relaxed : bsideup, Ali Ahmed
----
2019-08-05 15:46:39 UTC - Aaron: I enabled deduplication at the broker level, and my publisher says it is only publishing it once. I receive the message twice, in the same partition, with the same ledger id and different entry ids. Any ideas @Sijie Guo
----
2019-08-05 15:54:50 UTC - Sijie Guo: did you set sequenceId?
----
2019-08-05 15:56:36 UTC - Luke Lu: @Luke Lu has joined the channel
----
2019-08-05 16:14:13 UTC - Luke Lu: Hey guys, I’d like get some confirmation that using `pulsar-admin namespaces set-retention` with `--size` to “retention size in MB” applies _per topic_ not the sum of topic storage for the namespace. Cursory look at the code (retention policies appears to apply per ledger) seems to indicate the former.
----
2019-08-05 16:15:15 UTC - David Kjerrumgaard: yes, that is correct. It is applied on a per topic basis for all the topics in the namespace
----
2019-08-05 16:16:16 UTC - Luke Lu: Thanks @David Kjerrumgaard! How about the backlog quotas? Are they per topic as well?
----
2019-08-05 16:16:31 UTC - David Kjerrumgaard: Yes
----
2019-08-05 16:18:36 UTC - Luke Lu: Can I also assume the per topic sizes _don’t_ include replicated data? Trying to come up with a rough storage requirement before offloading…
----
2019-08-05 17:07:27 UTC - Jerry Peng: Its part of the function config
----
2019-08-05 17:07:50 UTC - Jerry Peng: <https://github.com/apache/pulsar/blob/master/pulsar-common/src/main/java/org/apache/pulsar/common/functions/FunctionConfig.java#L103>
----
2019-08-05 17:34:34 UTC - Addison Higham: hrm.... is there any good resources/code to read for understanding how pulsar behaves in the case of a "noisy neighbor" that is writing heavily? Let's assume a 3 bookies and 3 brokers for simplicity with a quorum size of 3, so topic1 and topic2 both have the same quorum members and topic1 is writing as fast as possible, what sort of behavior should we expect topic2 to see?
----
2019-08-05 18:22:29 UTC - John Anderson: @John Anderson has joined the channel
----
2019-08-05 20:32:45 UTC - Aaron: Doing that now - all set, thanks.
----
2019-08-05 20:55:55 UTC - Yiwei Chen: @Yiwei Chen has joined the channel
----
2019-08-05 21:14:13 UTC - Yiwei Chen: Hi, is there any guidance about rotating Pulsar broker tls cert?
----
2019-08-05 21:28:37 UTC - David Kjerrumgaard: @Yiwei Chen Are you asking about the mechanics of the process itself, or guidance on how often to rotate them as a matter of policy?
----
2019-08-05 21:46:11 UTC - Yiwei Chen: @David Kjerrumgaard Is there any way to rotate the cert without restarting the brokers?
----
2019-08-05 21:52:36 UTC - David Kjerrumgaard: @Yiwei Chen I don't think so. The underlying java process would need to be restarted to pick up the new TLS certificate. However, if you do a rolling restart of the brokers then you can make the change with zero downtime
----
2019-08-05 22:00:21 UTC - Yiwei Chen: @David Kjerrumgaard We still have a few Pulsar clients using the Comcast library, which doesn't work well with the broker IP change. If there is a way to replace the Pulsar broker TLS cert without restarting the broker pods and get new IPs that will be helpful.
----
2019-08-05 22:02:07 UTC - Addison Higham: @Yiwei Chen @David Kjerrumgaard I thought 2.4, via the `getTlsCertRefreshCheckDurationSec` allowed for certs to be changed without a reboot?
+1 : David Kjerrumgaard
----
2019-08-05 22:03:05 UTC - Yiwei Chen: We are running 2.3.1
----
2019-08-05 22:05:00 UTC - David Kjerrumgaard: Then that won't help.
----
2019-08-05 22:17:05 UTC - Ambud Sharma: Is there a way to rate limit (quota) bandwidth used autorecovery so ledger replication backfill traffic doesn't impact production?
----
2019-08-05 22:57:03 UTC - David Kjerrumgaard: @Ambud Sharma There aren't a lot of configuration settings that control auto-recovery bandwidth. The settings can be found here <https://bookkeeper.apache.org/docs/4.8.2/reference/config/#autorecovery-general-settings>, and here <https://bookkeeper.apache.org/docs/4.8.2/reference/config/#autorecovery-replication-worker-settings>
----
2019-08-06 00:14:57 UTC - Luo Tian: @Luo Tian has joined the channel
----
2019-08-06 06:31:51 UTC - Addison Higham: eek, just found myself in a pickle: We use data-dog and were looking to use the prometheus exporter with datadog support, but data-dog doesn't support untyped metrics, which was removed with <https://github.com/apache/pulsar/pull/4136>, however, by removing types altogether, a lot of metrics get dropped on the floor by datadog. I can fix that by manually adding at type annotation for all the pulsar metrics, but it seems like getting <https://github.com/apache/pulsar/pull/4183> fixed up and merged in is the long term correect fix
----
2019-08-06 06:34:13 UTC - divyasree: @Sijie Guo I am trying to enable authentication using tls
----
2019-08-06 06:35:04 UTC - divyasree: i want to know how authorization with role for specific namespace works when authenticating the client using key and certificate?
----
2019-08-06 06:39:33 UTC - Sijie Guo: @divyasree: authorization only cares about roles.
authentication resolves a pulsar connection to roles.
----
2019-08-06 06:40:27 UTC - divyasree: correct. When creating role, we are providing secret key right?
----
2019-08-06 06:40:49 UTC - divyasree: if it it TLS enabled, what should we give there?
----
2019-08-06 08:20:17 UTC - Richard Sherman: I've just hit an issue with a zookeeper session timeout how can I stop this happening again? `org.apache.pulsar.client.api.PulsarClientException$LookupException: org.apache.zookeeper.KeeperException$SessionExpiredException: KeeperErrorCode = Session expired`
----
2019-08-06 08:45:29 UTC - Matthieu Nicolas: That would be the case, yes. Thanks for your answer
----