You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@pulsar.apache.org by Apache Pulsar Slack <ap...@gmail.com> on 2019/09/20 09:11:02 UTC
Slack digest for #general - 2019-09-20
2019-09-19 13:51:19 UTC - Nick Marchessault: I recently added ackTimeOut configuration to my consumer and am now seeing this Warn message repeatedly in the logs:
WARN 1 --- [ulsar-timer-4-1] o.a.p.client.impl.UnAckedMessageTracker : [ConsumerBase{subscription='fortis', consumerName='b5180', topic='TopicsConsumerFakeTopicNamed84a6'}] 2 messages have timed-out
The timeout is more than enough to process the message and ACK before the time elapses. If 2 messages have timed out, they should get redelivered and then the warn message should disappear correct?
----
2019-09-19 14:44:33 UTC - David Kjerrumgaard: @Nick Marchessault Yes, the Pulsar client will track the unacknowledged messages, and the broker will automatically redeliver those messages that weren't acknowledged. However, if the consumer is unable to process these messages even after they were re-delivered, then the error message will remain. Typically this is caused by an uncaught exception being thrown by the consumer when it tries to process the message.
----
2019-09-19 17:23:55 UTC - Sujith Ramanathan: @Sujith Ramanathan has joined the channel
----
2019-09-19 17:29:23 UTC - Sujith Ramanathan: Hi,
I ran ./bin/pulsar standalone
Then getting below exception.
22:31:31.000 [pulsar-client-io-76-1] WARN org.apache.pulsar.client.impl.ConnectionPool - Failed to open connection to 38f9d3e4dd35:6650 : io.netty.resolver.dns.DnsResolveContext$SearchDomainUnknownHostException: Search domain query failed. Original hostname: '38f9d3e4dd35' failed to resolve '<http://38f9d3e4dd35.mycompany.com|38f9d3e4dd35.mycompany.com>' after 2 queries
----
2019-09-19 17:39:32 UTC - David Kjerrumgaard: @Sujith Ramanathan This looks like a DNS issue. Is your hostname really '<http://38f9d3e4dd35.mycompany.com|38f9d3e4dd35.mycompany.com>' ? Is so, you might need to add it to your /etc/hosts file
----
2019-09-19 17:58:25 UTC - Sujith Ramanathan: Hey David,
I just downloaded pulsar and trying to run it on locally by using pulsar standalone. Then am getting this exception. Do i need DNS config to run pulsar locally ?
Thanks for your quick response.
----
2019-09-19 18:03:50 UTC - Sujith Ramanathan: It's working fine, If am running as ./bin/pulsar standalone --advertised-address 127.0.0.1
----
2019-09-19 18:09:53 UTC - David Kjerrumgaard: In standalone it uses the local host name as the advertised address, but it looks like you are missing an entry in your /etc/hosts file that resolves your hostname, `<http://38f9d3e4dd35.mycompany.com|38f9d3e4dd35.mycompany.com>` to the loopback IP address of 127.0.0.1
----
2019-09-19 18:10:04 UTC - David Kjerrumgaard: Glad you found a work-around for this.
----
2019-09-19 18:47:16 UTC - Sujith Ramanathan: Yep :slightly_smiling_face:
----
2019-09-19 19:27:24 UTC - Luke Lu: What’s the expected behavior wrt geo replication and retention policies? Are retention policies replicated as well? If one of the clusters died and rejoined the namespace with empty storage, would they catch up with all the retained data from peers as well?
----
2019-09-19 19:35:10 UTC - Junli Antolovich: @Junli Antolovich has joined the channel
----
2019-09-19 19:48:16 UTC - Junli Antolovich: Hey Everyone, I am trying to find a windows service bus replacement, as many of our product lines are using windows service bus which went out of mainstream support in January last year. I come across Pulsar and like all what I see except that there is no .Net client. Not sure if this is something already in the pipeline or we will have to build it ourselves in order to use it. As you may see, we are kinda in a time crunch\
----
2019-09-19 19:48:27 UTC - David Kjerrumgaard: @Luke Lu Within a multi-region Pulsar cluster there are multiple ZK instances. Each regional cluster has a "local" ZK cluster that stores metadata about the topics and ledgers used to store the data. There is also a "cluster" instance that is used to store metadata around geo-replication, and namespace policies such as the retention policy. Having said that, this means that when the cluster re-joins the Pulsar instance (term for multi-cluster pulsar) it would be able to get those policies from the global ZK. However, there is no automatic replication of old messages between the two clusters which enables the re-joining cluster to "catch-up"
----
2019-09-19 19:49:18 UTC - Matteo Merli: There are a couple of .Net client libs being worked on
+1 : David Kjerrumgaard
----
2019-09-19 19:49:38 UTC - David Kjerrumgaard: @Junli Antolovich There actually is a .NET client that someone developed, but it hasn't been added to the open source codebase as of yet. Reach out to those teams for the exact status of those projects.
+1 : dba
----
2019-09-19 19:50:16 UTC - Matteo Merli: <https://github.com/danske-commodities/dotpulsar>
+1 : dba
----
2019-09-19 19:50:22 UTC - Matteo Merli: <https://github.com/fsharplang-ru/pulsar-client-dotnet>
----
2019-09-19 19:53:40 UTC - Junli Antolovich: Thanks, I did take a look at those two, and they are just started with very limited functionality. Guess I can use the C# one for a POC. It is .net core client, and we will have to convert it to .net framework in order to use it and add needed functionality when necessary
----
2019-09-19 20:03:12 UTC - Junli Antolovich: Also, I read that Pulsar is very high performing. Do you have the performance matrix handy that I can present to our architecture council for evaluation?
----
2019-09-19 20:04:41 UTC - Matteo Merli: These slides contain some benchmark results: <https://www.slideshare.net/merlimat/high-performance-messaging-with-apache-pulsar>
----
2019-09-19 20:05:36 UTC - Junli Antolovich: TYVMfor the help :slightly_smiling_face:
----
2019-09-19 20:24:53 UTC - Junli Antolovich: @Matteo Merli Thanks so much for the slides and I can borrow many of them directly for the evaluation in depth. Could you elaborate a bit on the last 2 points on the optimizations slide(29)?
----
2019-09-19 20:30:16 UTC - Matteo Merli: > Serialize operations to thread to avoid mutex
Instead of having multiple threads contending on a mutex, it’s better to delegate operations to a single thread (eg. hashing different topics ops to a pool of threads), so that we don’t need to acquire the mutex, or in any case it won’t be contended
----
2019-09-19 20:31:19 UTC - Matteo Merli: > Pulsar brokers acts as a “proxy” — Payloads are forwarded with zero-copies from producers to storage and consumers
Brokers are passing the payloads from producers to the storage nodes. We avoid all possible memory copies in the process.
+1 : Junli Antolovich
----
2019-09-19 20:41:18 UTC - Luke Lu: This means that there is no DR/backup solution for data in retention?
----
2019-09-19 20:55:58 UTC - David Kjerrumgaard: Data that is being retained has already been processed and acknowledged by all the consumers in the topic. Periodically this data gets deleted based on the retention policies that you set. If you want to keep this data beyond the retention period, you would configure tiered-storage to ensure that the data is moved to longer-term storage such as S3.
----
2019-09-19 20:56:50 UTC - David Kjerrumgaard: if you backed-up and restored the retained messages, the data/events would be processed twice, which is probably NOT what you want.
----
2019-09-19 21:00:42 UTC - Poule: Just filed a github issue for this
----
2019-09-19 21:21:26 UTC - Tarek Shaar: I am setting the total number of listener threads to 20 (PulsarClient client = PulsarClient.builder().listenerThreads(20)). Yet when I receive the messages I am only seeing the same thread processing the messages.. It is always 2019-09-19 17:18:39 [pulsar-external-listener-3-1] INFO PulsarLatenyMessageListiner:23 - Received message key:xxxxx. The thread is always pulsar-external-listener-3-1
----
2019-09-19 21:26:53 UTC - David Kjerrumgaard: @Tarek Shaar The listener threads are pooled, so when the thread is finished processing the message it is returned to the pool and can be selected again for the next message. It is a common misconception that threads in this pool will execute in parallel, allowing for concurrent processing of the messages. This is not that case, and you should use multiple consumers to achieve that behavior.
----
2019-09-19 21:48:42 UTC - Luke Lu: 1. How do you restore a cluster with offloaded data on tiered storage (s3)? I was under the impression that this is not supported yet: <https://github.com/apache/pulsar/issues/4942>
2. The retained messages are “acked” messages that won’t be seen again by consumers less they explicitly seek back.
I was hoping that geo replication can restore retained data as already “acked” data in recovered clusters.
----
2019-09-19 21:55:58 UTC - David Kjerrumgaard: 1. Loading data from S3 would require some manual work at the moment, but it is possible to do. Currently, you could access the data in S3 via the SQL interface. 2. Yes, readers can always attempt to read data beyond the retention window, but there are no guarantees that the data will still be available which is the case here. 3. Backing-up Pulsar would require backing-up the ZK metadata which keeps track of where the data is stored, etc.
----
2019-09-19 21:59:46 UTC - Luke Lu: I was just hoping the acked data _inside_ the retention window can be restored by geo-replication.
----
2019-09-19 22:02:04 UTC - David Kjerrumgaard: In an active-active scenario, if one of the clusters fails, producers and consumers (including readers) should be re-directed to the remaining active cluster and continue processing incoming messages. If you are using asynchronous geo-replication, then there is a small amount of messages that may not get replicated across. Now, if the failed cluster later re-joins the instance, it will start receiving replication traffic again and will quickly get filled with the most recent topic data (which is what most consumers want). Now if you had a use case that required a reader, and it was going to scan back to a time during the outage, you would want to ensure that the reader only interacted with the cluster that was "alive" during that period, as it would be the only place that had the data you are looking for.
----
2019-09-19 22:10:18 UTC - Luke Lu: Understood. This can be made to work with some effort (extra service to track “complete” (with retention data, which is critical to reconstruct streaming windows) clusters”), but as you can see, it’s not friendly from consumer PoV. A consumer typically talks to a pulsar cluster in its own region with a broker url as a fixed config.
----
2019-09-19 22:51:14 UTC - David Kjerrumgaard: That makes sense. The reader pattern is a bit of a gap in this scenario that can be bettered addressed. But bear in mind that data retention is always a moving target and varies by time. The higher the topic throughput, the more frequently the messages fall out of the retention window. Trying to keep them around indefinitely is a lot of work, particularly in a "catch-up" scenario, when they will ultimately not be kept around for a long time as they will get expired out as new data flows in.
----
2019-09-20 00:17:44 UTC - Tarek Shaar: @David Kjerrumgaard So my consumer that processed 100K messages, used the same thread from the pool of 20 threads?
----
2019-09-20 00:50:46 UTC - Sijie Guo: @Poule currently the namespace level compatibility check policy is only applied for producers and consumers. the admin restful api is using FULL compatibility check. there is a change in 2.5.0 to change restful admin to respect to namespace setting.
hugging_face : Poule
----
2019-09-20 00:51:13 UTC - Sijie Guo: <https://github.com/apache/pulsar/issues/4821>
----
2019-09-20 00:54:03 UTC - David Kjerrumgaard: How many message listeners did you attach to the consumer? <http://pulsar.apache.org/api/client/org/apache/pulsar/client/api/ConsumerBuilder.html#messageListener-org.apache.pulsar.client.api.MessageListener->
----
2019-09-20 01:54:07 UTC - Poule: if I `pulsar-admin topics delete --deleteSchema --force persistent://x/y/z` is it normal that I still get a schema with `pulsar-admin schemas get --version 0 persistent://x/y/z` ?
----
2019-09-20 05:25:40 UTC - Luke Lu: The retention windows could be size (typically MBs or GBs) or time (typically hours or days) based, we just want to have the current windows be maintained in all replicated clusters. i.e., we’re not interested in restoring the old messages that are outside the current retention windows. Ideally, we want to use pulsar (or equivalent) as the only stream storage, without having to actively save checkpoints into separate storage for window reconstructions.
----
2019-09-20 05:40:47 UTC - Vinoth: @Vinoth has joined the channel
----
2019-09-20 05:50:38 UTC - Vinoth: Hello everyone, I have to test the performance of the pulsar broker. We are using pulsar-perf tool for publishing and receive the messages. Need to find the average throughput with 1 broker, 2 broker and 3 brokers. When I tested with all the broker scenarios, there is no big difference in throughput. Kindly advice, How could I test it?
----
2019-09-20 08:04:56 UTC - Vladimir Shchur: @Junli Antolovich this one <https://www.nuget.org/packages/Pulsar.Client> supports .net standard 2.0 so can be used from old dotnet as well (with just some binding redirects). And of course it supports both F# and C#
----