You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@pulsar.apache.org by Apache Pulsar Slack <ap...@gmail.com> on 2020/02/11 09:11:03 UTC
Slack digest for #general - 2020-02-11
2020-02-10 09:12:17 UTC - Vincent LE MAITRE: Hi All, I'm interested in using Pulsar go client with schema (avro or Protobuf) with schema registry. The documentation version 2.5.0 says that schema registry is only available for Java client. But looking at the issues, I found this entry <https://github.com/apache/pulsar/issues/2466> whih says that it should be supported now. What is the real status ? Schema is supported with schema registry too ?
----
2020-02-10 09:17:16 UTC - Sijie Guo: the documentation about registry should be updated. (/cc @Anonymitaet)
It currently supports java, python, cpp, and cgo client. It doesn’t support the native go client yet. (<https://github.com/apache/pulsar-client-go>)
----
2020-02-10 09:18:35 UTC - Shahar Fermon: thank you @Sijie Guo - i’ll give it a go!
----
2020-02-10 09:45:20 UTC - Vincent LE MAITRE: Thanks for your answer. The registry is not supported in go client, but the avro or protobuf schema can be used into go consumer / producer. Am I right ?
----
2020-02-10 12:28:27 UTC - Anonymitaet: Plan in <https://github.com/apache/pulsar/issues/6282>, will fix later
----
2020-02-10 13:18:03 UTC - Eric Feliksik: @Eric Feliksik has joined the channel
----
2020-02-10 13:21:19 UTC - Eric Feliksik: I’m looking for documentation on the YAML configuration for pulsar; e.g. how to declaratively define what Pulsar Functions should be running, which what level of parallellism, etc…; what topics, namespaces should exist. Does that documentation exist?
And example is given on <https://pulsar.apache.org/docs/v2.0.1-incubating/functions/overview/> (search for `my-function.yaml`), but I don’t see a “complete overview”
----
2020-02-10 13:24:13 UTC - Roman Popenov: > what Pulsar Functions should be running, which what level of parallellism
This I am not too aware
----
2020-02-10 13:25:31 UTC - Roman Popenov: I did find some examples that helped me with wrapping my head around:
<https://github.com/apache/pulsar/tree/master/pulsar-functions/java-examples/src/main/resources>
+1 : Eric Feliksik
----
2020-02-10 13:26:03 UTC - Eric Feliksik: thanks! better than nothing
----
2020-02-10 13:26:03 UTC - Roman Popenov: This might be outdated as it is staging incubating
----
2020-02-10 13:26:07 UTC - Roman Popenov: <https://pulsar.apache.org/docs/en/functions-api/#functions-for-python>
----
2020-02-10 13:26:14 UTC - Roman Popenov: This is also more up to date information
----
2020-02-10 13:27:55 UTC - Roman Popenov: Also, please not how it is userConfig:
“PublishTopic”: “test_result” for `userConfig`
----
2020-02-10 13:28:59 UTC - Roman Popenov: I know that it is possible to pass a JSON as well, the structure should also follow as with YAML. I saw in some docs the `userConfig` was ommited
----
2020-02-10 13:29:09 UTC - Roman Popenov: And same for connectors
----
2020-02-10 14:46:22 UTC - Eric Feliksik: I notice that the dockerized <https://hub.docker.com/r/apachepulsar/pulsar-standalone> standalone pulsar thing is only able to run pulsar functions with java bytecode major version 52.0 (java 1.8). I suppose that’s just because that docker container ships an old jvm.
Are there people running java11 (bytecode 55.0) pulsar functions, running pulsar on newer JVM? Or is there a technical reason why this isn’t possible at the moment?
eyes : Konstantinos Papalias
----
2020-02-10 16:48:25 UTC - Sijie Guo: Schema registry is supported in cgo client.
----
2020-02-10 16:50:22 UTC - Konstantinos Papalias: Do we have any examples of Pulsar Functions using the schema registry? Looking into <https://github.com/apache/pulsar/tree/master/pulsar-functions/java-examples/src/main/java/org/apache/pulsar/functions/api/examples> but couldn't identify any examples on how this works currently
----
2020-02-10 16:56:28 UTC - Ryan Slominski: Hi, can Pulsar handle fan-out? For example multiple consumers on one topic and all receive same message?
----
2020-02-10 17:01:37 UTC - Ryan Slominski: Nevermind, figured it out. I thought it did, then got confused because I couldn't find any docs on it. Still can't find any docs on it, but it is as simple as keeping topic name the same and changing subscription name.
----
2020-02-10 19:00:36 UTC - Ali: @Ali has joined the channel
----
2020-02-10 20:36:09 UTC - Eric Simon: It seems that all of the pulsar docker images were updated in the last 24 hours. It seems to be breaking existing functionality. Does anyone have any idea what this build was for?
----
2020-02-10 20:49:19 UTC - Rustem: @Rustem has joined the channel
----
2020-02-10 20:54:27 UTC - Roman Popenov: I am seeing this in the deployments charts:
``` # disable auto recovery on bookies since we will start AutoRecovery in separated pods
autoRecoveryDaemonEnabled: "false"```
Should we set this to true if the `AutoRecovery` pods are disabled?
----
2020-02-10 20:55:46 UTC - Rustem: Hello, Pulsar team I'm new to Pulsar and Java. I'm trying to install a pulsar function
`bin/pulsar-admin functions create --jar data/LambdaDispatch.jar --classname org.my.functions.Invoker --tenant public --namespace default --name lambda-dispatch --inputs <persistent://public/default/mytopic>`
and getting an error
`Function class org.my.functions.Invoker must be in class path`
Can someone please explain what I'm doing wrong?
----
2020-02-10 20:56:55 UTC - Greg Methvin: sure. I think I also spoke to @Matteo Merli about this and he said the delayed backlog value wasn’t an estimate, but maybe I’m misremembering our conversation.
----
2020-02-10 20:57:02 UTC - Eric Simon: @RustemIs `org.my.functions.Invoker` in your jar?
```jar -tf data/LambdaDispatch.jar | grep org.my.functions.Invoker ```
----
2020-02-10 20:58:55 UTC - Rustem: yes, org/my/functions/Invoker.class
----
2020-02-10 21:01:35 UTC - Rustem: btw I'm installing the function on the docker container, if it matters
----
2020-02-10 21:01:44 UTC - Roman Popenov: No, it doesn’t
----
2020-02-10 21:02:01 UTC - Roman Popenov: You still need to have the functions in the jar file
----
2020-02-10 21:02:16 UTC - Roman Popenov: And have the Invoker be part of the jar
----
2020-02-10 21:03:05 UTC - Roman Popenov: You class path might be wrong or the jar doesn’t have the Invoker packed
----
2020-02-10 21:04:08 UTC - Rustem: I checked the jar and it has the Invoker.class in org\me\functions\
----
2020-02-10 21:07:18 UTC - Rustem: > You class path might be wrong
can you please give me a little bit more details about class path? I'm not a java developer, I must be missing somthing
----
2020-02-10 21:07:49 UTC - Roman Popenov: `org\me\functions\` --> `--classname org.me.functions.Invoker`
----
2020-02-10 21:07:58 UTC - Roman Popenov: I think there is a typo my vs me
----
2020-02-10 21:09:25 UTC - Rustem: yeh sorry, I just changed the name, now I have a company name there (instead of me/my) and I have the same error
----
2020-02-10 21:15:08 UTC - Roman Popenov: let’s say I have a function in
```package com.mycompany.project.pulsar.functions;```
Then when I am running the function, I will have to run
`/pulsar/bin/pulsar-admin functions localrun/create --jar /path_to_jar/my-uberjar.jar — com.mycompany.project.pulsar.functions.Invoker`
----
2020-02-10 21:17:05 UTC - Roman Popenov: and the `jar -tf` command will have `com/mycompany/prject/pulsar/functions/Invoker.class`
----
2020-02-10 21:24:29 UTC - Ganesh Raju: @Ganesh Raju has joined the channel
----
2020-02-10 23:51:49 UTC - Sijie Guo: I think one of them was not accurate because it was calculated based on publish count and consume count.
----
2020-02-11 01:05:32 UTC - Jasper Li: thank you @Shahar Fermon, @Sijie Guo!! :+1:
nerd_face : Shahar Fermon
+1 : Shahar Fermon
----
2020-02-11 02:59:09 UTC - Ravi: @Ravi has joined the channel
----
2020-02-11 03:47:23 UTC - Eric Simon: Has anyone had success with deploying on Kubernetes with state enabled?
----
2020-02-11 04:13:25 UTC - Aditya Vishwakarma: Hi everyone, how is experience of using Pulsar with Python?
We are looking at using Pulsar for our use case in Machine Learning. The system will have a load of approx ~10,000 msgs/sec with avg msg size of ~5MB. And most ML code is in python.
My personal experimentations with the python client hasn't gone well. Wanted to know if there are others using python clients in production
----
2020-02-11 04:14:48 UTC - Ali Ahmed: pulsar functions in python are deployed at scale in quite a few production environments.
----