You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@pulsar.apache.org by Apache Pulsar Slack <ap...@gmail.com> on 2020/06/17 09:11:06 UTC
Slack digest for #general - 2020-06-17
2020-06-16 09:21:45 UTC - dionjansen: Hi Fernando, thanks for the suggestion, the retention policy on the namespace is both set to infinite for time/ size I have a feeling the offloader mechanism just got into some weird state because of the disk space pressure. In the end I just rebuild the cluster from scratch an moved the persistent ledgers into the new one
+1 : Fernando
----
2020-06-16 09:29:33 UTC - xue: <https://github.com/apache/pulsar/issues/7270>，test pulsar, bookie oom
----
2020-06-16 09:38:21 UTC - Marcio Martins: Hey Anup, so I did some more testing, and it seems the issue is related to the order at which the components come up, if the broker comes up before the bookie, then it will fail forever, until you restart it, even if the bookie comes up shortly after. I was specting it to have some errors but then fix it self once the bookies are up. The reason seems to be related to those DNS errors. If the broker always comes up after all bookies are up, then there is no DNS error and everything is smooth. So I suppose there is some caching of failed lookups. Not sure if this is done at the broker level, or if it's done at a Kubes cluster level...
----
2020-06-16 09:47:49 UTC - Ankur Jain: @Matteo Merli Thanks for the explanation! This makes it a lot clearer on how writes and reads do not impact each other significantly. Would broker use read-ahead to get more data over network so that subsequent reads for catch-up consumer are served from broker's cache?
My last query is bookkeeper setup related. Maybe this is not the right forum, will raise it BK's slack community.
----
2020-06-16 12:06:32 UTC - Luke Stephenson: are you using the helm template to deploy the proxy? If so, check out <https://github.com/apache/pulsar-helm-chart/pull/12/files> which added similar support for grafana / pulsar manager to have an allow list.
----
2020-06-16 12:49:15 UTC - Raphael Enns: Thanks for the information :slightly_smiling_face:
----
2020-06-16 14:55:00 UTC - Marcio Martins: I got an cluster with 0 activity, but one of the bookies has 100% cpu usage - it has been like this for over 1 hour after I disconnected the producers. The previous activity was a pulsar-perf with only 5 msg/sec. It seems like all 5 bookies are also running really high on memory, > 2GB for such a light load. Anyone has any insights into what is going on?
----
2020-06-16 15:05:32 UTC - Matteo Merli: There’s no support for IP whitelisting at this in the proxy directly, though it should be very easy to add and it would be a nice addition.
----
2020-06-16 15:12:08 UTC - Alexander Ursu: Hi, I was wondering if there will be VODs from the Pulsar Summit over the next 2 days
----
2020-06-16 15:43:35 UTC - Asaf Mesika: Did you at the bookie metrics ?
----
2020-06-16 16:16:06 UTC - Daniel Ciocirlan: can someone help explain more on the correct setup for client `ioThreads` and `connectionsPerBroker`, we have 6 brokers and a partitioned topic with 12 partitions (so 2 partitions per each broker). Also we run 3 proxies behind a AWS NLB, so producers/consumers don't talk directly with brokers.
----
2020-06-16 16:17:14 UTC - Addison Higham: AFAIK, they will be recorded, but no idea on the release date
----
2020-06-16 16:17:36 UTC - Addison Higham: are you facing issues?
----
2020-06-16 16:20:02 UTC - Daniel Ciocirlan: no, but we are not yet in prod and i want to catch this before, we used 3 connections per broker and 12 ioThreads
----
2020-06-16 16:20:20 UTC - Matthew Follegot: Thank you David
----
2020-06-16 16:20:57 UTC - Daniel Ciocirlan: i see the default is 1 for both, but with 1 ioThreads the producer was not able to connect to the partitioned topic, so we used 12 as we have 12 partitions
----
2020-06-16 16:21:29 UTC - Daniel Ciocirlan: was wondering if there is any "catch" to this or guide lines :slightly_smiling_face:
----
2020-06-16 16:23:35 UTC - Matteo Merli: > Would broker use read-ahead to get more data over network so that subsequent reads for catch-up consumer are served from broker's cache?
For catch-up reads, the read-ahead is mostly done on the Bookie node itself, this is to exploit the fact that data for same topic is mostly stored sequentially on the disk.
----
2020-06-16 16:24:06 UTC - Matteo Merli: > My last query is bookkeeper setup related. Maybe this is not the right forum, will raise it BK's slack community.
Sure, most BK people are here too though :slightly_smiling_face:
----
2020-06-16 16:32:09 UTC - Addison Higham: pulsar client is mostly async internally, so ioThreads don't need to map one to one. Likewise, producers/consumers can be multiplexed over a single connection
----
2020-06-16 16:33:22 UTC - Marcio Martins: Yes, everything seemed normal. It went back to idle after a long time, but still consuming close to 2GB memory
----
2020-06-16 16:36:24 UTC - Daniel Ciocirlan: Well we have the app servers in the same AZ as the brokers, the timeout is 200 ms, most of the pub actions takes 3-5ms but there are times that p99 is more than 200 ms and tried to increase the IOThreads and Connections per broker, did not fix this.
----
2020-06-16 16:37:11 UTC - Daniel Ciocirlan: what would be a real scenario that would require more ioThreads ?
----
2020-06-16 16:39:21 UTC - Daniel Ciocirlan: we are around 1M RPM on around 36 producers geo replicated to 4 regions
----
2020-06-16 16:42:19 UTC - Addison Higham: it does make sense to add more ioThreads and possibly extra connectionsPerBroker, but I just mean that it isn't necessary to have one per partition or anything
----
2020-06-16 16:43:23 UTC - Addison Higham: as far as those latencies, that may well be from the broker. For example, perhaps due to GC pause if you are close on heap? or could be disk
----
2020-06-16 16:45:04 UTC - Daniel Ciocirlan: heap not an issue the instances jave huge memory allocated
----
2020-06-16 16:45:30 UTC - Daniel Ciocirlan: so you recommendation is to leave the default 1/1 in this configuration ?
----
2020-06-16 16:47:23 UTC - Addison Higham: I am not quite familiar with all that enough to give a recommendation of exact numbers, but just in general to explain what those things do :slightly_smiling_face:
----
2020-06-16 16:48:35 UTC - Daniel Ciocirlan: :slightly_smiling_face: thanks, will have more close look on this.
----
2020-06-16 16:48:36 UTC - Addison Higham: <https://github.com/openmessaging/openmessaging-benchmark/blob/master/driver-pulsar/src/main/java/io/openmessaging/benchmark/driver/pulsar/config/PulsarClientConfig.java> <- this is what the openBenchmark does
----
2020-06-16 16:48:48 UTC - Addison Higham: 8/8 is what it does
----
2020-06-16 16:49:42 UTC - Daniel Ciocirlan: for sure i need more than 1 iothreads, when the producer / consumer tries to subscribe to a 12 partitioned traffic it would just block there
----
2020-06-16 16:50:15 UTC - Daniel Ciocirlan: but thanks for your time! :slightly_smiling_face:
----
2020-06-16 16:51:34 UTC - Rich Adams: I have noticed a similar issue in the broker. When it first starts up it can go up to 1.7GB with no load. If I restart it it will only take up about 300MB
----
2020-06-16 16:55:59 UTC - Addison Higham: @Marcio Martins some of what bookies are doing is background tasks like moving records from journal -> ledgers, compaction, etc. That does sound like more than I would expect, but the metrics should be able to tell you more of what was happening
----
2020-06-16 17:13:50 UTC - Albert Xu: @Albert Xu has joined the channel
----
2020-06-16 17:14:18 UTC - Marcio Martins: I had a look at it seems like everything looks normal - in the JVM metrics it says heap 300MB, DM 400MB, but K8s reports 2.5GB memory, again, with 0 activity.
----
2020-06-16 17:20:33 UTC - Albert Xu: hello i was trying to use the pulsar client for python but after i do `pip install pulsar-client==2.5.2` and try to `import pulsar` in my python repl, I get
```In [2]: import pulsar
---------------------------------------------------------------------------
ImportError Traceback (most recent call last)
<ipython-input-2-07c1fdd48b98> in <module>
----> 1 import pulsar
~/.pyenv/versions/test/lib/python3.7/site-packages/pulsar/__init__.py in <module>
100 """
101
--> 102 import _pulsar
103
104 from _pulsar import Result, CompressionType, ConsumerType, InitialPosition, PartitionsRoutingMode # noqa: F401
ImportError: dlopen(/Users/albertxu/.pyenv/versions/test/lib/python3.7/site-packages/_pulsar.cpython-37m-darwin.so, 2): Symbol not found: __Py_tracemalloc_config
Referenced from: /Users/albertxu/.pyenv/versions/test/lib/python3.7/site-packages/_pulsar.cpython-37m-darwin.so
Expected in: flat namespace
in /Users/albertxu/.pyenv/versions/test/lib/python3.7/site-packages/_pulsar.cpython-37m-darwin.so```
another person working with me also had the same issue. Is there something I'm doing wrong?
I've tried using other versions. The only one I've gotten to work locally is `pip3 install pulsar-client==2.4.1.post1` but no version that i've found seems to work on AWS lambda instances. In the AWS lambda environment they look like they all run into the same import error.
----
2020-06-16 17:34:03 UTC - Patrik Kleindl: Have you enabled the state service on BK or are you using it in Pulsar functions?
High non-heap memory is a side effect of rocksdb
----
2020-06-16 17:34:27 UTC - Marcio Martins: What is the state service?
----
2020-06-16 17:34:34 UTC - Marcio Martins: I am using almost all default settings
----
2020-06-16 17:35:25 UTC - Marcio Martins: ```ledgerStorageClass=org.apache.bookkeeper.bookie.storage.ldb.DbLedgerStorage```
I am using that with all default settings
----
2020-06-16 17:35:56 UTC - Patrik Kleindl: It‘s state or table service. Used for pulsar functions to manage state.
----
2020-06-16 17:36:15 UTC - Marcio Martins: Functions are disabled
----
2020-06-16 17:40:10 UTC - Marcio Martins: I see, that is why it's not JVM memory, ok... Can I somehow limit it without impacting performance too much? What do you recomend?
----
2020-06-16 17:43:21 UTC - Patrik Kleindl: I have no idea, that was just a guess because we had a similar issue with Rocksdb on Kafka Streams
Try to disable the service at BK if possible and see if it helps
----
2020-06-16 17:44:55 UTC - Marcio Martins: Thanks Patrik! I didn't remember RocksDB memory is not from the JVM heap, so that helped a lot, and now I know what to tune.
----
2020-06-16 17:51:01 UTC - Wayne Collins: @Wayne Collins has joined the channel
----
2020-06-16 17:52:19 UTC - Jesse Anderson: @Jesse Anderson has joined the channel
----
2020-06-16 17:56:55 UTC - Julien: Hi there, we have a spark application running in our k8s cluster along with pulsar. This is a streaming app, constantly trying to consume messages from a topic using pulsar spark connector. This app is literally doing a DDOS attack on our pulsar (deployed with the helm chart with default low requests), making brokers going out of memory after 2/3 hours of responding to thousands/short time of lastMessageId requests (the topic being empty all that time).
Is this normal? Is our Pulsar just too weak or there is something to parametrize in our spark-application?
Have a good one!
----
2020-06-16 18:08:54 UTC - Piyush: @Piyush has joined the channel
----
2020-06-16 18:11:49 UTC - Marcio Martins: So after I tweaked the RocksDB memory it's still > 2.5G... I have retention on, with 100GB offloaded to S3. Could it be related to that? Somehow tries to keep some index of the S3 storage in memory?
----
2020-06-16 18:14:55 UTC - Marcio Martins: Those should not even be known to bookkeeper, right? Can't think of why else it should need 2.5GB...
----
2020-06-16 18:27:42 UTC - Patrik Kleindl: Don‘t tweak, disable it if you are not using it. One less variable.
Did you completely restart the BKs?
----
2020-06-16 18:27:55 UTC - Marcio Martins: yes
----
2020-06-16 18:28:21 UTC - Marcio Martins: But can I disable the RocksDB? I thought that was part of the ledger storage?
----
2020-06-16 18:34:54 UTC - Chris Herzog: @Chris Herzog has joined the channel
----
2020-06-16 18:59:48 UTC - Patrik Kleindl: @Sijie Guo @Penghui Li is it possible to disable the table service on BK just to rule out rocksdb as a source of memory consumption?
----
2020-06-16 19:18:07 UTC - Alan Broddle: In a Terraform AWS VM based cluster we were in the process of mapping the Bookie Storage devices, and we see that the Bookie is using an AWS i3.xlarge instance type. This instance type by default creates a 950 GB SSD ephemeral storage device.
I3.xlarge
4 CPU 30.5 GB
1x 950 SSD
Network up to 10 Gigabit
In the instance build process, a nvme0n1 device is created but not mapped/mounted to the instance.
When we run the “aws/setup-disk.yaml” command in the Ansible setup, we noticed several issues:
1. The script specifically looks for 2 devices to map the /mt/storage and /mnt/journal
• Only 1 device is created with the instance build
• Do we create another device? What type?
1. The SSD storage created by an EC2 i3.xlarge instance is ephemeral and is not retained after a Server ‘stop’.
Can someone please explain a couple basic questions that we have:
1. Why was a i3.xlarge instance set up as the default instance type in the “aws/terraform.tfvars”?
instance_types = {
“zookeeper” = “t2.large”
“bookie” = “i3.xlarge”
“broker” = “c5.2xlarge”
“proxy” = “c5.2xlarge”
}
1. What type of AWS Instance do people typically use and why?
2. If this is recommended, I assume we would need to add a second device to the instance to get the Journal and Storage onto different devices.
3. Is there some reason that we should leverage the nvme0n1 device that is set up by AWS with this instance type that we are not aware of?
Basically, our assumption is that when a Bookie is stopped or rebooted, that we want the data to be retained. That would mean the default AWS instance would be a non-ephemeral based storage instance.
----
2020-06-16 19:30:04 UTC - Marcio Martins: I believe the reason to use NVMe storage is mostly because in high volume production deployments, the blottleneck is going to the bookie disk i/o. NVMe have very large amount of iops, but are ephemeral like you mentioned. However the bookie cluster *is* performing replication - the broker will only consider the message published if it has been stored to at least 2 bookies (configurable), so as long as you don't turn off all your bookies at once, you should be covered. The alternative are EBS volumes, which themselves perform replication, and are durable past start/stop cycles, but you'd then have "double replication" which would increase your latency, and be you'd be *very* limited by IOPS on EBS. You could also resort to provisioned IOPS, but those are also quite expensive.
+1 : Alan Broddle
----
2020-06-16 19:52:19 UTC - Duane Sessions: @Duane Sessions has joined the channel
----
2020-06-16 20:32:49 UTC - Varghese C: @Varghese C has joined the channel
----
2020-06-16 22:52:23 UTC - Sijie Guo: You can disable table service if you don’t use functions or function state
----
2020-06-16 23:18:42 UTC - Eric Bowden: @Eric Bowden has joined the channel
----
2020-06-17 00:15:04 UTC - Luke Stephenson: Please try to refer to this feature as "allow list".
----
2020-06-17 00:28:26 UTC - Sijie Guo: did you install the pulsar cpp library?
----
2020-06-17 00:39:31 UTC - Matteo Merli: Yes, I guess that's a more sensible name
----
2020-06-17 04:14:46 UTC - Sijie Guo: @Sijie Guo set the channel topic: - Pulsar Summit Virtual Conference 2020. Join <#C015MT1DE7N|pulsarsummit> channel for real-time discussions. Schedule: <https://pulsar-summit.org/schedule/first-day/>
- Pulsar 2.5.2 released <http://pulsar.apache.org/blog/2020/05/19/Apache-Pulsar-2-5-2/>
----
2020-06-17 04:42:50 UTC - Sankararao Routhu: thanks @Luke Stephenson @Matteo Merli.We are deploying proxy using aws cloud formation templates. Our allowed list would be very huge(in thousands) and it should be reflected on proxy dynamically as and when we add a new ip. Can you please give us some pointers how to add this feature?
----
2020-06-17 06:21:48 UTC - Asaf Mesika: Isn’t the worst is duplicate task processing happening in parallel?
----
2020-06-17 06:22:36 UTC - Asaf Mesika: Since they write of the job shouldn’t worry about concurrency right? (It might run in the same instance or in different instance, but compete on writing to a DB, or something like that)
----
2020-06-17 07:24:22 UTC - Charvak Patel: @Charvak Patel has joined the channel
----
2020-06-17 08:07:25 UTC - Arushi Sharma: @Arushi Sharma has joined the channel
----