You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@pulsar.apache.org by Apache Pulsar Slack <ap...@gmail.com> on 2019/01/15 09:11:04 UTC
Slack digest for #general - 2019-01-15
2019-01-14 16:14:03 UTC - Siva Prasad Rao Janapati: @Siva Prasad Rao Janapati has joined the channel
----
2019-01-14 16:15:14 UTC - Siva Prasad Rao Janapati: Hi, Is there any way I can consume messages in batch from Pulsar topic? I knew Pulsar client supports batch publish.
----
2019-01-14 16:16:07 UTC - Matteo Merli: Messages are internally received in batches, though at the API level are presented individually
----
2019-01-14 16:17:17 UTC - Matteo Merli: If you want to do some bulk operation, you could just receive a bunch of messages (within some small time window) and process them at once
----
2019-01-14 16:18:18 UTC - Siva Prasad Rao Janapati: I am using Pulsar Java client library. In that I use comsumer's receive method which will give only one message at a time
----
2019-01-14 16:19:27 UTC - Siva Prasad Rao Janapati: I don't see any API to get batch of messages available in topic from a single call to Pulsar
----
2019-01-14 16:20:11 UTC - Matteo Merli: Yes, I was meaning you can call receive multiple times and then process a collection of messages at once
----
2019-01-14 16:21:30 UTC - Siva Prasad Rao Janapati: I did the same way as you suggested. In my code I applied iterative logic to get specified number of messages from Pulsar. Essentially I am making thse may calls to Pulsar.
----
2019-01-14 16:21:47 UTC - Siva Prasad Rao Janapati: Is that the only way I can achieve batch consumption from Pulsar topic?
----
2019-01-14 16:23:50 UTC - Matteo Merli: Currently, yes. The thing is that since all these messages are prefetched already, the calls to receive become very cheap
----
2019-01-14 16:25:00 UTC - Matteo Merli: I don’t think there will be a noticeable perf advantage in getting a batch of messages in this case
----
2019-01-14 16:27:43 UTC - Siva Prasad Rao Janapati: ok
----
2019-01-14 16:30:18 UTC - Alex Mironov: @Alex Mironov has joined the channel
----
2019-01-14 16:32:35 UTC - Alex Mironov: Hey folks, looks like sha512 signature RPM links for cpp client (<https://pulsar.apache.org/docs/en/client-libraries-cpp/>) are broken on some mirrors, e.g. a mirror link for devel rpm is leading to 404: <http://ftp.tudelft.nl/apache/pulsar/pulsar-2.2.1/RPMS/apache-pulsar-client-devel-2.2.1-1.x86_64.rpm.sha512>
----
2019-01-14 16:42:26 UTC - Matteo Merli: I think the mirrors don’t host the signatures (these should only be fetched from Apache servers to guarantee authenticity). If we have a link pointing to the mirrors, we have to fix it in our website. Can you open a github issue for it? Thanks
----
2019-01-14 16:47:00 UTC - Alex Mironov: yep, done: <https://github.com/apache/pulsar/issues/3370>
----
2019-01-14 16:55:03 UTC - Ryan Samo: Hey guys and gals, have any of you looked into segregation of ingress traffic to egress traffic? Basically making sure that producers are not sending data into your Pulsar cluster on the same NIC as your consumers, keeping performance high. Just wondering
----
2019-01-14 17:50:30 UTC - Fredrick P Eisele: @Fredrick P Eisele has joined the channel
----
2019-01-14 19:16:38 UTC - Ezequiel Lovelle: @Siva Prasad Rao Janapati I think that calling a receive method with the desired number of message to fetch from this in a single operation vs having multiple receive() calls on different threads would have the same overhead (due to the reasons explained by @Matteo Merli), the only thing here might be simplicity at client side performing just one receive call, anyway, I don't think it's worth the effort to add this functionality.
----
2019-01-14 19:29:29 UTC - Grant Wu: <https://pulsar.apache.org/api/client/org/apache/pulsar/client/api/ConsumerBuilder.html> there are a bunch of tunables here
----
2019-01-14 19:30:25 UTC - Grant Wu: <https://pulsar.apache.org/api/client/org/apache/pulsar/client/api/ConsumerBuilder.html#maxTotalReceiverQueueSizeAcrossPartitions-int-> <https://pulsar.apache.org/api/client/org/apache/pulsar/client/api/ConsumerBuilder.html#receiverQueueSize-int->
may be interesting
----
2019-01-14 19:39:22 UTC - Grant Wu: @Jerry Peng I just noticed the existence of <https://pulsar.apache.org/api/client/org/apache/pulsar/client/api/ConsumerBuilder.html#patternAutoDiscoveryPeriod-int->
Does this imply that there could be, at worst, latency between a message showing up on a new topic that matches a topic pattern, and the message being processed by the Pulsar Function of up to the specified period?
----
2019-01-14 19:56:49 UTC - Grant Wu: @Jonathan Budd
----
2019-01-14 19:58:29 UTC - Jerry Peng: @Grant Wu that is correct
----
2019-01-14 19:58:53 UTC - Grant Wu: Okay, good to know
----
2019-01-15 00:34:43 UTC - Vardhman: @Vardhman has joined the channel
----
2019-01-15 00:44:35 UTC - Vardhman: Hi I am trying to use the pulsar/pulsar-client-cpp for a project written in C. I have a CMakeList.txt file that has a `add_subdirectory(pulsar/pulsar-client-cpp)` to compile the pulsar-client-cpp using its CMakeList.txt file. However, when I call make (after cmake .) in my project it gives me an error `pulsar/pulsar-client-cpp/lib/MessageRouterBase.h:25:41: fatal error: pulsar/MessageRoutingPolicy.h: No such file or directory`. I think when it tries to compile files that have `#include <pulsar/someheader.h>` imports, it fails since it does not know where `pulsar/...` is. Can someone help me with what needs to be modified in the CMakeList.txt files to fix this issue? Any help would be really appreciated. Thanks!
----
2019-01-15 01:46:45 UTC - Grant Wu: @Jerry Peng out of curiosity does that imply that messages sent to the topic within the first N seconds (where N is between 0 and 60, depending on the timing) would not be processed by the Pulsar function?
----
2019-01-15 01:47:05 UTC - Grant Wu: Or does Pulsar have logic to retrieve the previous messages?
----
2019-01-15 01:47:36 UTC - Grant Wu: I know with normal subscriptions Pulsar by default always starts from messages that arrive after the consumer is created.
----
2019-01-15 02:54:16 UTC - Jerry Peng: That is correct. A function will have the same behavior as a consumer in regards with subscribing to a topic pattern
----