You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@pulsar.apache.org by Apache Pulsar Slack <ap...@gmail.com> on 2020/07/09 09:11:04 UTC
Slack digest for #general - 2020-07-09
2020-07-08 09:38:01 UTC - Gautam Kumar: @Gautam Kumar has joined the channel
----
2020-07-08 09:56:14 UTC - Penghui Li: thanks
----
2020-07-08 13:52:20 UTC - Sankararao Routhu: Is it mandatory to pass trust cert while creating a client on ssl port. Is there a way to create ssl connection with out this trust cert path because we have multiple clients sharing trust cert with all of the them is tedious
```PulsarClient client = PulsarClient.builder()
.serviceUrl("<pulsar+ssl://my-broker.com:6651>")
.enableTls(true)
.tlsTrustCertsFilePath("/path/to/cacert.pem")
.authentication(tlsAuth)
.build();```
----
2020-07-08 13:53:16 UTC - Olav Groenaas Gjerde: Stop using '@here', you can ask questions fine without it
----
2020-07-08 14:05:47 UTC - Amit Kaushal: @Olav Groenaas Gjerde: You can always mute this channel notification if it is disturbing you too much.
----
2020-07-08 14:07:23 UTC - Marco Pina: @Marco Pina has joined the channel
----
2020-07-08 14:08:02 UTC - Pedro Oliveira: @Pedro Oliveira has joined the channel
----
2020-07-08 14:11:38 UTC - Rui Silva: @Rui Silva has joined the channel
----
2020-07-08 14:13:02 UTC - Marco Pina: Hi, I have a question about Time To Live (TTL). It's not possible apply the TTL on topic level or message level? Or is only possible apply on namespace level?
----
2020-07-08 14:13:19 UTC - Marco Pina: thanks
----
2020-07-08 14:22:44 UTC - Penghui Li: Currently, pulsar only support at the namespace level. we are planning to support at the topic level at the next major release version.
----
2020-07-08 14:25:11 UTC - Marco Pina: Hi Penghui, thank you for the fast answer.
I am very happy for that. And do you have any dates for this major release?
----
2020-07-08 14:27:13 UTC - Penghui Li: It should be in September or October
----
2020-07-08 14:29:27 UTC - Marco Pina: Nice, thank you very much for the clarification. :)
----
2020-07-08 15:47:39 UTC - Viktor: anyone?
----
2020-07-08 16:25:50 UTC - Sijie Guo: Please stop using at here.
----
2020-07-08 16:58:42 UTC - Sankararao Routhu: sure @Sijie Guo
----
2020-07-08 16:59:33 UTC - Sankararao Routhu: can you please help me with the above query
----
2020-07-08 17:32:11 UTC - Varghese C: Can Pulsar support multi-avro schema topics? Is it always one schema per topic? The use case is to put related event types in the same topic/partition for correct ordering. For Kafka I noticed that they introduced <https://karengryg.io/2018/08/18/multi-schemas-in-one-kafka-topic/|muti-schema topic support> couple of years back. I suppose one could roll out a custom client-side approach as described in the <http://pulsar.apache.org/docs/en/2.4.2/concepts-schema-registry/|pulsar schema registry docs>, but can we still leverage the schema registry to be enforce strict typing and schema evolution?
----
2020-07-08 17:33:46 UTC - Sijie Guo: currently it doesn’t support that.
@Shivji Kumar Jha is proposing a change to add that feature.
----
2020-07-08 17:35:18 UTC - Varghese C: Thanks Sijie, thats awesome to hear that. Is there a PIP / Git Issue that elaborates on this?
----
2020-07-08 17:37:20 UTC - Shivji Kumar Jha: Hi @Varghese C i made a design but then had to pause it for a while owing to other priorities. I can only start maybe beginning of next month! No PIP / git issue yet. I will create and one and share shortly
----
2020-07-08 17:38:00 UTC - Varghese C: @Shivji Kumar Jha Ok thanks, I would like to help out on this.
+1 : Shivji Kumar Jha
----
2020-07-08 17:38:09 UTC - Varghese C: Also would a producer/consumer written using the Kafka libs that use this feature work over KoP? Sorry to take much of your time, thx
----
2020-07-08 17:48:41 UTC - Shivji Kumar Jha: not in the first iteration but we can make that work in iteration 2.
+1 : Varghese C
----
2020-07-08 18:59:40 UTC - Devin G. Bost: I ran into this issue: <https://github.com/apache/pulsar-client-go/issues/315>
----
2020-07-08 19:00:02 UTC - Devin G. Bost: Have you been able to get it to work?
----
2020-07-08 19:08:22 UTC - Devin G. Bost: Hmm the error log looks like it’s due to docker, not Pulsar…
----
2020-07-08 19:16:17 UTC - Xintong Zhou: @Xintong Zhou has joined the channel
----
2020-07-08 19:36:20 UTC - Xintong Zhou: Hi, is there limit on the number of subscriptions a topic can handle? Does it matter whether it is a normal topic or a partitioned topic? Thank you
----
2020-07-08 19:39:59 UTC - Addison Higham: There is no hard limit, but there are functional limits of memory and handing network connections. How many subscriptions are you looking to do?
----
2020-07-08 20:10:30 UTC - Xintong Zhou: Thank you for the quick reply. I was thinking to use the topic to broadcast messages to all interested clients, the number of which is unbounded.
----
2020-07-08 20:13:41 UTC - Xintong Zhou: Is the functional limit because a partition of a topic is only served by a single broker?
----
2020-07-08 21:18:35 UTC - Nicolas Ha: ```Pulsar's consumption model takes a streaming-pull approach.```
<https://streamnative.io/blog/tech/pulsar-vs-kafka-part-1>
I am curious about this: is the approach documented somewhere? Can someone explain how this works?
----
2020-07-08 21:19:16 UTC - Devin G. Bost: @Sijie Guo Sound familiar? haha
@Nicolas Ha (I literally just emailed Sijie about this.)
smile : Nicolas Ha
----
2020-07-08 21:22:09 UTC - Nicolas Ha: Digging a bit myself, seems related to the `CommandFlow`
<https://github.com/apache/pulsar/blob/eaf268cb3717df242a8e17b5cadb6babc1a7099c/pulsar-common/src/main/proto/PulsarApi.proto#L522-L528>
----
2020-07-08 21:25:40 UTC - Addison Higham: correct. Topics only live on a single broker. It should be said that you can have tens of thousands of subscriptions on a single broker, probably even hundreds of thousands with the right settings and hardware
If you need to go beyond that and it is just uni-directional fanout, you can scale that out by:
publish to one topic -> pulsar function that replicates the message to N other topics -> consumers subscribe to one of N topics, which will spread across many brokers
Obviously, that has *some* downsides in that some subset of topics might get behind if there is a broker that is having maintenance, but in general, is a pretty sane solution.
----
2020-07-08 21:32:28 UTC - Nicolas Ha: Starting to read the consumer impl it starts to make sense. There is a receiver queue in there, and it has to be kept fed by the broker. Writing up as I read it.
However it raises some questions for me:
• 1. what this ( <https://github.com/apache/pulsar/blob/6926180966f45eb9c1499b7f0eb32ea2a1368fd6/pulsar-client/src/main/java/org/apache/pulsar/client/impl/ConsumerImpl.java#L1478> ) does - what does the broker do when it receives a flow request of `0`?
• 2. when is `increaseAvailablePermits` called exactly (ie is there a treshold for the receiver queue? I don’t see it)
• 3. and if the consumers usually have a receiver queue, what is this then <https://github.com/apache/pulsar/blob/aa0fbb0d31d36f383054d93e3684de79879c430c/pulsar-client/src/main/java/org/apache/pulsar/client/impl/ZeroQueueConsumerImpl.java#L170> ?
----
2020-07-08 21:48:49 UTC - Fernando: is there a way to do multiple subscription with regex matching when words are not present? like `<persistent://public/default/(?!foo_).*>` ? this doesn’t seem to work
----
2020-07-08 22:38:18 UTC - Derek Moore: NATS claims to be pull based, but I'm curious about their rationale for this claim:
----
2020-07-08 22:38:36 UTC - Derek Moore: <https://github.com/nats-io/nats-streaming-server/issues/229>
----
2020-07-08 22:44:19 UTC - Derek Moore: What they say sounds similar to Pulsar's streaming-pull claims
----
2020-07-08 22:56:51 UTC - Viktor: I am unavble to access bookkeeper metrics, even though I can see the configs being correct. can someone help. Bookkeeper process's command ends like this `org.apache.bookkeeper.proto.BookieServer --conf /opt/pulsar/conf/bookkeeper.conf`
```$ sudo netstat -apn | grep 8000 | wc -l
0
$ grep 8000 /opt/pulsar/conf/bookkeeper.conf
prometheusStatsHttpPort=8000```
----
2020-07-08 23:55:12 UTC - Sijie Guo: I called it “streaming-pull”. Because the following:
1. technically it is a pull request. Client asks broker for N messages. Broker dispatches roughly N messages to client. After client receive N messages, client will send another Flow request.
2. It is different from a pull or a long-pull request. a) Broker dispatches messages as soon as the messages are available to satisfy the flow (N) request. There is no wait on accumulating N messages to dispatch. b) Client doesn’t have to wait until receive all N messages before sending next flow request. Hence both request (flow request) and response (messages) can be pipelined. So it is a bi-direction streaming.
Hope that make sense. @Nicolas Ha @Devin G. Bost
----
2020-07-08 23:58:02 UTC - Sijie Guo: @Nicolas Ha to answer your questions:
a) broker increase its permits on receiving flow requests. so if flow is 0, means client is asking for nothing. broker will not dispatch the messages.
b) on client side, after the messages are dequeued from receiver queue.
c) the receiver queue is acting as a readahead cache. zero queue means disabling pre-feteching.
----
2020-07-09 03:46:20 UTC - Anonymitaet: Hi guys,
Recent growth has put the spotlight on Pulsar. While most of the news on Pulsar has been instructive, not all coverage of Pulsar has been accurate.
Today, we are excited to share our response to the recent Confluent blogs comparing Kafka, Pulsar and RabbitMQ. This post is the first in a two-part series and concentrates on the *differences between Pulsar and Kafka* in terms of *performance, architecture, and features.*
Read it (<https://streamnative.io/blog/tech/pulsar-vs-kafka-part-1|Pulsar vs. Kafka — Part 1 — A More Accurate Perspective on Performance, Architecture, and Features>) now and welcome to repost. :smiley_cat:
+1 : evir35, victor
----
2020-07-09 03:47:56 UTC - Anonymitaet: @Anonymitaet set the channel topic: - Pulsar vs. Kafka: <https://streamnative.io/blog/tech/pulsar-vs-kafka-part-1>
- Pulsar Summit Talk recordings: <https://www.youtube.com/watch?v=wBpNcESySNc&list=PLqRma1oIkcWjVlPfaWlf3VO9W-XWsF_4->
----
2020-07-09 04:15:00 UTC - Hiroyuki Yamada: Does anyone know about this ?
----
2020-07-09 04:35:26 UTC - Sijie Guo: How do you start bookies?
----
2020-07-09 08:54:24 UTC - victor: Hi guys! does someone know how to keep the ordering guarantee when you add a new partition to a topic? I guess its better to make another version of the topic with more partitions but I'd like to know your oppinion
----
2020-07-09 09:02:23 UTC - Nicolas Ha: Thanks that makes sense. Congrats on the launch!
----