You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@pulsar.apache.org by Apache Pulsar Slack <ap...@gmail.com> on 2020/06/13 09:11:05 UTC
Slack digest for #general - 2020-06-13
2020-06-12 09:55:37 UTC - skans100: @Penghui Li is there a tentative date for getting 2.6.0 released?
----
2020-06-12 09:56:53 UTC - Penghui Li: Yes, plan to announce next week if there are no block issues in 2.6.0 release.
+1 : skans100
----
2020-06-12 11:03:30 UTC - Nicolas Ha: In here <http://pulsar.apache.org/functions-rest-api/?version=2.5.1#operation/registerFunction>
it says
```jar
Path to the JAR file for the Pulsar Function (if the Pulsar Function is written in Java). It also supports URL path [http/https/file (file protocol assumes that file already exists on worker host)] from which worker can download the package.```
And the example input is
``` "jar": java-function-1.0-SNAPSHOT.jar```
Does this mean my jar has to be publicly available?
How does the java pulsar client does it?
----
2020-06-12 11:09:30 UTC - Nicolas Ha: I am asking because I cannot seem to use this <https://github.com/apache/pulsar/blob/2aff473e598fe5e8ba9f8ed0860de35a67718725/pulsar-client-tools/src/main/java/org/apache/pulsar/admin/cli/CmdFunctions.java#L657>
it seems to expect my Java fn to be in the classpath - so I am trying to use the API but it isn’t clear to me how to proceed
----
2020-06-12 11:20:40 UTC - Nicolas Ha: from the broker logs
```java.lang.IllegalArgumentException: Function class my.myFunction must be in class path```
So the question is: how can I upload a jar / class not in the class path to pulsar?
----
2020-06-12 11:21:21 UTC - sjmittal: Hey folks I am installing pulsar first time on k8s and I get this error which I suppose is mainly due to bookie not starting:
`11:12:24.704 [pulsar-web-42-5] ERROR org.apache.pulsar.broker.admin.impl.PersistentTopicsBase - [null] Failed to create non-partitioned topic <persistent://public/functions/assignments>`
`java.util.concurrent.CompletionException: org.apache.pulsar.broker.service.BrokerServiceException$PersistenceException: org.apache.bookkeeper.mledger.ManagedLedgerException: Not enough non-faulty bookies available`
----
2020-06-12 11:21:39 UTC - sjmittal: any idea what could be the reason behind this error
----
2020-06-12 11:38:13 UTC - Luke Stephenson: Hi @Sijie Guo. Here is the configuration I'm using:
``` managedLedgerOffloadDriver: "aws-s3"
s3ManagedLedgerOffloadBucket: "goanna-pulsar-topic-offload"
s3ManagedLedgerOffloadRegion: "ap-southeast-2"
s3ManagedLedgerOffloadServiceEndpoint: "<http://s3.ap-southeast-2.amazonaws.com|s3.ap-southeast-2.amazonaws.com>"```
----
2020-06-12 11:40:44 UTC - xiaolong.ran: The `s3ManagedLedgerOffloadServiceEndpoint` useful for testing. Can you config `AWS_ACCESS_KEY_ID` and `AWS_SECRET_ACCESS_KEY`?
----
2020-06-12 11:40:47 UTC - Luke Stephenson: Where would I find the docker image for the 2.6.0 rc? I'm looking here and don't see it <https://hub.docker.com/r/apachepulsar/pulsar-all/tags>
----
2020-06-12 11:41:02 UTC - xiaolong.ran: <https://pulsar.apache.org/docs/en/cookbooks-tiered-storage/#authentication-with-aws>
----
2020-06-12 11:42:01 UTC - xiaolong.ran: in broker.conf:
```managedLedgerOffloadDriver: "aws-s3"
s3ManagedLedgerOffloadBucket: "goanna-pulsar-topic-offload"
s3ManagedLedgerOffloadRegion: "ap-southeast-2"```
in pulsar_env.sh
```export AWS_ACCESS_KEY_ID=ABC123456789
export AWS_SECRET_ACCESS_KEY=ded7db27a4558e2ea8bbf0bf37ae0e8521618f366c```
----
2020-06-12 11:43:18 UTC - Luke Stephenson: I hadn't set those env vars. I assumed they were picked up from the ec2 instance. Will look at that.
----
2020-06-12 11:51:11 UTC - xiaolong.ran: Sorry, I am not an expert of EC2 instance, but my understanding is that you should be able to configure `AWS_ACCESS_KEY_ID` and `AWS_SECRET_ACCESS_KEY` in your service as env, right? If this is the case, you should be able to offload the data into aws-s3
----
2020-06-12 11:57:27 UTC - Luke Stephenson: I'll give that a go
----
2020-06-12 11:57:45 UTC - xiaolong.ran: Cool
----
2020-06-12 12:34:24 UTC - Luke Stephenson: I tried setting up
> 5. Assuming an IAM role
That didn't help. Haven't yet worked out how I get `AWS_ACCESS_KEY_ID` onto the environment with the pulsar helm template
----
2020-06-12 12:36:12 UTC - Luke Stephenson: Think I've spotted what I need to do now.
----
2020-06-12 12:44:16 UTC - Luke Stephenson: nah. not sure how to get that into the helm template
----
2020-06-12 12:55:15 UTC - Luke Stephenson: I managed to get them into `conf/pulsar_env.sh` as
```PULSAR_EXTRA_OPTS=" -Daws.accessKeyId=xxx -Daws.secretKey=xxx "```
However, that didn't help and I'm still seeing the original error message: "The authorization header is malformed; the region 'us-east-1' is wrong; expecting 'ap-southeast-2''"
----
2020-06-12 13:09:14 UTC - Luke Stephenson: Out of curiosity, I tried creating a bucket in us-east-1 and configuring `s3ManagedLedgerOffloadRegion: "us-east-1"` and it gets past that error (still fails for a different reason, but it progresses further)
----
2020-06-12 13:24:50 UTC - Luke Stephenson: It's working with us-east-1.
----
2020-06-12 13:25:16 UTC - Luke Stephenson: Unfortunately my cluster is running in ap-southeast-2 though, so that isn't a great setup.
----
2020-06-12 13:48:34 UTC - Luke Stephenson: And now if I switch it back to ap-southeast-2 it works. Seems the library provides useful error messages if something is misconfigured when the bucket is in us-east-1. So short term I can point to that region to sort my config out and then switch back to the desired region later.
----
2020-06-12 14:24:34 UTC - Marcio Martins: Is there a way to make the S3 offloads get a prefix with the topic name? I don't want to provision a new bucket per topic, and having everything in a single bucket means I can't delete a topic and then manually go and delete the S3 objects. Alternatively, maybe would be nice to delete the S3 files when a topic is deleted, as I don't think this is currently happening.
writing_hand : Asaf Mesika
----
2020-06-12 14:37:17 UTC - jujugrrr: @Marcio Martins I think I've seen a github issue about cleaning policy for S3 topic
----
2020-06-12 14:38:01 UTC - Marcio Martins: Thanks, will check it out
----
2020-06-12 15:04:27 UTC - Matteo Merli: It would only be replicated if you had a "global" zookeeper as config store. Otherwise you need to create the namespace with same config in all 4 instances.
----
2020-06-12 15:05:59 UTC - Aaron Batilo: What about adding a lifecycle policy to the s3 objects?
+1 : David Kjerrumgaard
----
2020-06-12 15:06:19 UTC - Asaf Mesika: We’ve thinking of building a library offering high throughput task queue and scheduled tasks (recurring), on top of Pulsar, in Java (Think Quartz, but horizontally scalable, high throughput and better API for one-off tasks)
1. Does anybody happens to know an existing open source library like that?
2. We’re trying to assess the amount of “operational cost” Apache Pulsar may have (we’ll probably start with something like 40k msgs/min): how much you spend on maintaining a running BK cluster and Pulsar cluster. Does it have many errors? Errors are easy to handle?
----
2020-06-12 15:09:09 UTC - David Kjerrumgaard: The short answer is yes they will be fetched. However you have to pay attention to the data retention and message expiration policies to ensure that the messages don't get "purged"
----
2020-06-12 15:12:07 UTC - David Kjerrumgaard: You can use the <http://pulsar.apache.org/docs/en/pulsar-admin/#stats-1> command to show you all the connected topic consumers and their respective consumption rates.
----
2020-06-12 15:13:14 UTC - Marcio Martins: Thanks!
----
2020-06-12 15:13:47 UTC - Anup Ghatage: Hey @sjmittal
`Not enough non-faulty bookies available`
Usually means there aren’t enough bookies for an ensemble - which means either:
1. Your bookkeeper deployment went awry
2. Service discovery cannot connect to or find the bookies for the ensemble
----
2020-06-12 15:39:53 UTC - sjmittal: yeah figured out the issue, my PVC was not setup correctly and hence bookies were not instantiated and hence the broker was complaining not enough bookies
+1 : Anup Ghatage, Asaf Mesika
----
2020-06-12 15:43:17 UTC - Enrico Olivelli: Hey I have just received my t-shirt for Pulsar Summit! Thank you @Sijie Guo!
Conference is coming!
Please every one check it out ! <https://pulsar-summit.org|https://pulsar-summit.org>
----
2020-06-12 16:57:01 UTC - Anup Ghatage: Whoa whoa whoa. How do we get one? @Sijie Guo
----
2020-06-12 16:58:59 UTC - Ebere Abanonu: @Sijie Guo wouldn't mind one too. Do you T-Shirt Pulsar itself?
----
2020-06-12 18:03:16 UTC - Enrico Olivelli: @Anup Ghatage as you are a speaker you should have received an email that asked for your size :) maybe it is too late
----
2020-06-12 18:05:07 UTC - Anup Ghatage: @Enrico Olivelli unfortunately seems like I didn’t get that email :disappointed:
(or missed it up pretty sure I didn’t since its about T-shirts :smile: )
----
2020-06-12 18:05:44 UTC - Daniel Ciocirlan: thanks @Matteo Merli for the clarification :slightly_smiling_face:
----
2020-06-12 18:06:20 UTC - Daniel Ciocirlan: one more question if not asking too much, if a sync producer fails to send a message to broker due to operation timeout, would the producer retry the message or drop it ?
----
2020-06-12 18:09:43 UTC - Matteo Merli: When there are failure, the producer is internally retrying, up to the specified "send-timeout" configured
----
2020-06-12 18:10:07 UTC - Daniel Ciocirlan: yes, if we exceed that the message is dropped ?
----
2020-06-12 18:10:10 UTC - Matteo Merli: after that the message is dropped (that doesn't exclude that we message could be actually published though)
----
2020-06-12 18:10:26 UTC - Daniel Ciocirlan: cool good to know :slightly_smiling_face:
----
2020-06-12 18:10:43 UTC - Matteo Merli: if you want the client to keep trying, you need to set send-timeout to 0
----
2020-06-12 18:10:48 UTC - Daniel Ciocirlan: we just started 2 months with Pulsar and have some questions that cannot find in the docs
----
2020-06-12 18:11:23 UTC - Daniel Ciocirlan: we can't the upper layer is not thread safe and if we get 400 K- 500K RPMs we get to thread starvation
----
2020-06-12 18:11:36 UTC - Daniel Ciocirlan: thanks for the great help!
----
2020-06-12 18:12:22 UTC - Daniel Ciocirlan: regarding the local ZK, i need to create partitioned topics in all regions even if i produce in 1 region and this namespace is geo-replicated to consuming regions
----
2020-06-12 18:22:31 UTC - Matteo Merli: You should do all admin operations in each "cluster", so even creating the partitioned topics
----
2020-06-12 18:23:04 UTC - Daniel Ciocirlan: okey so we did it correct :slightly_smiling_face:
----
2020-06-12 18:23:35 UTC - Daniel Ciocirlan: my last 2 questions, if not too much :
----
2020-06-12 18:24:22 UTC - Daniel Ciocirlan: • do we need any connectors for a geo replicated solution, we just use shared subscription for event notification in consuming regions
----
2020-06-12 18:26:05 UTC - Daniel Ciocirlan: • i cannot find any documentations that details what the proxy actually does, i found a book that says that proxy queries ZK for which broker has the partition, then uses the connection pool to forward the producer request to the <http://broker.Is|broker.Is> this correct ? any other functionality ?
----
2020-06-12 18:26:25 UTC - Matteo Merli: if you configure the "clusters" URL and configure the namespace to be replicated across the clusters, that would be it
----
2020-06-12 18:26:52 UTC - Matteo Merli: the proxy is used to expose broker through a load balancer
----
2020-06-12 18:27:14 UTC - Matteo Merli: if you're using standalone Pulsar you wouldn't need the proxy anyway
----
2020-06-12 18:27:33 UTC - Daniel Ciocirlan: we have AWS setup with NLB in front
----
2020-06-12 18:27:51 UTC - Daniel Ciocirlan: target is about 1M RPM, replicated to 4 regions for consuming
----
2020-06-12 18:29:22 UTC - Daniel Ciocirlan: thanks for your time @Matteo Merli
----
2020-06-12 18:33:49 UTC - Matteo Merli: :+1:
----
2020-06-12 18:51:16 UTC - Anup Ghatage: Update: I did receive the email but missed it :cry:
I deserve to not have a shirt since I wasn’t on top of things :sob:
----
2020-06-12 18:51:29 UTC - Curtis Cook: I’ll take your tshirt if you don’t want it :wink:
----
2020-06-12 19:04:43 UTC - Anup Ghatage: let’s hold on now.. I’ve messaged @Sijie Guo. I might get one if he’s in a good mood :stuck_out_tongue:
----
2020-06-12 19:05:30 UTC - Curtis Cook: i suggest sending cookies
grin : Anup Ghatage
----
2020-06-12 20:44:09 UTC - Gilles Barbier: Hi Asaf, we are currently building something like that. A job manager system + a workflow engine to orchestrate those jobs. It could be used for async tasks or even microservices orchestration
----
2020-06-12 20:44:53 UTC - Gilles Barbier: With a big focus on observability. Scale and resilience should be provided by Pulsar. We plan to open source an alpha this summer.
----
2020-06-12 20:46:29 UTC - Gilles Barbier: (we are developing in kotlin, but workers could easily be written in any languages supported by Pulsar)
----
2020-06-12 21:14:20 UTC - Asaf Mesika: Ok. Sounds similar yet a bit different. My main question to you is: did you tackle any surprises? What about operations of BookKeeper - is it a stable system or requires deep knowledge to operate ?
----
2020-06-12 21:14:41 UTC - Asaf Mesika: We plan to write in Kotlin as well - great language
----
2020-06-12 21:14:52 UTC - Kirill Merkushev: Hello, got an error in the prod - `2020-06-12 21:08:26.716 ERROR 13 --- [r-client-io-1-1] org.apache.pulsar.client.impl.ClientCnx : [id: 0x834bd319, L:/10.0.5.244:40214 - R:ip-10-0-5-251.eu-central-1.compute.internal/10.0.5.251:6651] Close connection because received internal-server error java.lang.IllegalStateException: Namespace bundle company/events/0x58000000_0x60000000 is being unloaded` - is there a way to reproduce that intentionally? So that i can test our code on this. And what does it mean in general?
----
2020-06-12 21:23:48 UTC - Addison Higham: oh dang, didn't realize I was supposed to send an email either :stuck_out_tongue:
----
2020-06-12 21:27:20 UTC - Addison Higham: `pulsar-admin namespaces unload <namespace>` will trigger that (or `pulsar-admin topics unload <topic>`)
Unloading is how pulsar moves load across brokers. For a given topic (actually a bundle, as topics are grouped together in a bundle to have less of them to manage) pulsar periodically keeps stats of how busy the topic (bundles) is and how that load is relative to other brokers. It will try and shed load if it gets too high.
When the bundle is unloading, all the connections get closed as you now have a different broker you need to talk to
+1 : Matteo Merli, Kirill Merkushev
----
2020-06-12 21:34:09 UTC - Cathy Thompson: @Cathy Thompson has joined the channel
----
2020-06-12 22:13:02 UTC - Kirill Merkushev: is that possible to reproduce it with standalone setup? (as it have effectively just one broker)
----
2020-06-12 22:13:31 UTC - Kirill Merkushev: thx for the explanation - even better than in docs :smile:
----
2020-06-13 06:01:24 UTC - Gilles Barbier: We did not perform stress tests yet, so we have not yet reached this step. The main issue up to now was that presto can be really slow
----
2020-06-13 06:03:14 UTC - Asaf Mesika: That depends on how connector was implemented I guess but does it have to do with job processing framework?
----
2020-06-13 06:05:07 UTC - Gilles Barbier: Our hope was to be able to use presto to introspect Workers queues
----
2020-06-13 07:53:12 UTC - Philippe Chavanne: @Philippe Chavanne has joined the channel
----