You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@pulsar.apache.org by Apache Pulsar Slack <ap...@gmail.com> on 2020/02/29 09:11:03 UTC
Slack digest for #general - 2020-02-29
2020-02-28 12:46:02 UTC - Vladimir Shchur: Hi! How does the `Run-with-Broker`
```functionsWorkerEnabled=true```
mode (which is recommended for k8s) correspond to the _Kubernetes runtime?_ Should functions run with brokers or as stateful sets?
eyes : Konstantinos Papalias
----
2020-02-28 13:43:13 UTC - Tobias Macey: That's great to hear!
----
2020-02-28 14:08:48 UTC - yannick: @yannick has joined the channel
----
2020-02-28 14:26:59 UTC - Konstantinos Papalias: I'm new to Pulsar Functions and going through the same exploration as to how best to deploy functions on Pulsar / K8s, but this PR seems to be the answer to your question
```Added a new runtime called KubernetesRuntime. A pulsar worker can be configured to deploy functions in kubernetes. Clients still use rest api to submit the functions as before, but instead of scheduling/running these functions by workers, the workers submit them to kubernetes. We use a stateful set to do this because each function instance needs a unique instance id.```
<https://github.com/apache/pulsar/pull/1950>
----
2020-02-28 15:34:57 UTC - riconsch: @Sijie Guo thank you for your reply. Do you know any blog / tutorial / docs where I can teach that to my self ?
----
2020-02-28 15:40:03 UTC - Sijie Guo: This setting only means whether to run function worker along with broker or not.
----
2020-02-28 15:40:45 UTC - Sijie Guo: You can configure function runtime to decide how functions work run functions : <http://pulsar.apache.org/docs/en/functions-runtime/|pulsar.apache.org/docs/en/functions-runtime/>
----
2020-02-28 16:01:33 UTC - yannick: Hi,
i'm new to pulsar. is there a tool similar to kafkacat where i can send/receive msgs via stdin/stdout ?
and how do i start reading from the first message in a topic once i set retention policy (a new subscription via pulsar-client binary seems to start at the current latest offset) ?
----
2020-02-28 16:15:42 UTC - Eric Simon: Where does the `functions_worker.yaml` file need to reside? Not seeing that anywhere in the documentation?
----
2020-02-28 16:17:43 UTC - Konstantinos Papalias: It's under `conf` folder alongside the configuration of your broker.conf etc
----
2020-02-28 16:18:58 UTC - Eric Simon: Ahh I see.
----
2020-02-28 16:39:23 UTC - Mathieu Druart: Hi @Sijie Guo we are trying to use startingOffsets in a Spark application (using pulsar-spark connector). It's a batch query with a specific messageId previously stored from the Topic. For now it doesn't seem to work, no messages are pulled from the Topic (but no error message). We tried using MessageId.earliest and we are correctly pulling all the messages of the topic, any advice on this or should we log an issue ?
----
2020-02-28 16:47:58 UTC - Sijie Guo: Interesting. Can you file a github issue for this, including an example oof you startingOffsets? I am guessing the `startingOffsets` is not configured correctly.
----
2020-02-28 16:49:24 UTC - Sijie Guo: 1. bin/pulsar-client provides similar functionalities.
2. for a new subscription, you can specify SubscriptionInitialPosition. By default it starts from latest. You can specify earliest.
----
2020-02-28 19:29:18 UTC - Andrew Tan: Hello, I'm trying to get Tiered storage working with the `aws-s3` driver, after following the instructions here <https://pulsar.apache.org/docs/en/cookbooks-tiered-storage/#aws-s3-driver-configuration>, whenever I restart my broker I recieve the error:
```ERROR org.apache.pulsar.PulsarBrokerStarter - Failed to start pulsar service.
plsr-pb_pulsar1.1.2ose6rq8mgal@pop-os | org.apache.pulsar.broker.PulsarServerException: org.apache.pulsar.broker.PulsarServerException: <http://java.io|java.io>.IOException: No offloader found for driver 'aws-s3'. Please make sure you dropped the offloader nar packages under `${PULSAR_HOME}/offloaders`.```
What step am I missing, something to do with JClouds?
----
2020-02-28 19:29:27 UTC - Mathieu Druart: Thanks, we will try to make a simpler example and double check the offsets configuration. If we reproduce the issue we will log it.
----
2020-02-28 19:31:20 UTC - Chris Bartholomew: You need to install the offloaders, which is a separate package from Pulsar: <https://pulsar.apache.org/en/download/>
----
2020-02-28 19:31:56 UTC - Sijie Guo: cool thanks
----
2020-02-28 19:32:17 UTC - Chris Bartholomew: If you are using Docker/Kubernetes, you can switch to the pulsar-all image, which already has these installed
----
2020-02-28 19:37:35 UTC - yannick: thanks! that already helps. i do not understand how to get a clean stream from pulsar-client without any logging though
----
2020-02-28 19:38:49 UTC - Sijie Guo: what does “a clean stream” mean?
----
2020-02-28 19:39:30 UTC - yannick: just the messages. so i can process them via unix pipes
----
2020-02-28 19:40:37 UTC - yannick: e.g. in kafka i would do `kafkacat -o -1 -e | jq 'somefilter' -c | gzip > mydata.jsonl.gz`
----
2020-02-28 19:40:55 UTC - yannick: if i have a topic full of json frames
----
2020-02-28 19:42:45 UTC - yannick: and how can i set SubscriptionInitialPosition for pulsar-client ?
----
2020-02-28 19:58:28 UTC - Sijie Guo: oh it doesn’t provide the exactly same features as kafkacat.
----
2020-02-28 19:59:12 UTC - Sijie Guo: I don’t think the CLI tool pulsar-client provide that option yet. you can help us improve the pulsar-client CLI :slightly_smiling_face:
----
2020-02-28 20:00:38 UTC - yannick: ok thanks! i think such a client should be written in a language that has a less horrible startup time :wink: i'll hack something together in go for that reason then i guess
----
2020-02-28 20:22:10 UTC - Sijie Guo: you can try the golang cli tool - <https://github.com/streamnative/pulsarctl>
----
2020-02-28 21:11:01 UTC - Vladimir Shchur: Thank you, I was under impression that FunctionWorker is another name for Function, now it's much clearer!
----
2020-02-28 22:14:04 UTC - Michael Jasper: Has anyone used StreamNative’s pulsar terraform provider? I’m trying to develop & run some tests on it locally, and I’d be grateful for anyone’s time who’s used it for a couple questions. Don’t want to clog up <#C5Z4T36F7|general>, so questions in thread
----
2020-02-28 22:16:31 UTC - Michael Jasper: I’m trying to run tests locally against a local docker pulsar instance, and I’m having a hard time configuring the terraform provider to point to the correct service url for my pulsar instance. I’m setting the service url by `export WEB_SERVICE_URL=<hhtp://mylocalinstance>:port` but regardless, it falls back to `localhost`. If someone has used that tf provider, are there extra configuration steps or env vars that need be set that I’m seeing in the documentation?
----
2020-02-28 22:16:54 UTC - Michael Jasper: Oh, and <https://github.com/streamnative/terraform-provider-pulsar>
----
2020-02-28 22:49:34 UTC - Sijie Guo: @Michael Jasper - I am from StreamNative.
Are you using a standalone cluster or a “cluster” running in local docker? And where do you run the terraform script.
Btw, currently this provider is still under development. It only provides basic operations for provisioning tenant / namespace / topic. New resources are coming and contribution are welcomed.
----
2020-02-28 22:51:55 UTC - Michael Jasper: Thanks for responding, and glad to hear it’s under development. I’m working on creating a feature to add as a pull request (adding permission grants to namespaces). I’m using a cluster running in local docker (<https://github.com/apache/pulsar/blob/master/docker-compose/standalone-dashboard/docker-compose.yml>)
----
2020-02-28 22:52:12 UTC - Michael Jasper: in the main directory of the project. I’m running `make testacc`
----
2020-02-28 22:53:09 UTC - Michael Jasper: (I submitted the issue about it a while ago, but then remembered about this slack organization so I reached out here)
----
2020-02-28 22:53:55 UTC - Sijie Guo: cool. I will check the issue and the feature. Let me get back to you soon.
----
2020-02-28 22:54:06 UTC - Michael Jasper: Sounds good, thank you!
----
2020-02-28 22:57:07 UTC - Michael Jasper: And I’m happy to communicate however you prefer, here vs github issues, or whatever else
----
2020-02-28 23:24:26 UTC - Andrew Tan: I completed those steps above, it seems to run fine, when I trigger a manual offload I get the following:
```Error in offload
null
Reason: Error offloading: org.apache.bookkeeper.mledger.ManagedLedgerException: java.util.concurrent.CompletionException: java.lang.UnsupportedOperationException```
----
2020-02-29 03:29:40 UTC - xorander: @xorander has joined the channel
----
2020-02-29 03:32:14 UTC - xorander: Hello :slightly_smiling_face: I'm working on getting Pulsar up and running on FreeBSD. Figure I'd join here for some feedback as I work through making the necessary changes to get it up & running.
----
2020-02-29 03:32:37 UTC - xorander: If/when I'm able to get it going, I'll write the port and submit it to the FreeBSD ports tree.
----