You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@pulsar.apache.org by Apache Pulsar Slack <ap...@gmail.com> on 2020/10/27 09:11:06 UTC
Slack digest for #general - 2020-10-27
2020-10-26 09:30:17 UTC - Rob Fowler: I found my issue. I had <http://XXXX> not <pulsar://X.X.X.X>
----
2020-10-26 09:30:57 UTC - Rob Fowler: it's me 'mianos' same person as github
----
2020-10-26 09:31:11 UTC - Rob Fowler: I knew it made no sense
----
2020-10-26 09:50:33 UTC - Damien Roualen: I `was able` to use functions-worker in the past, but after a few changes in the config I have this error. I am not able to find the problem.
When I start the functions-worker service inside a container, running on kubernetes.
```/pulsar/bin/pulsar functions-worker
09:33:08.007 [main] INFO org.apache.pulsar.common.util.SecurityUtility - Found and Instantiated Bouncy Castle provider in classpath BC
09:33:08.496 [main] INFO org.apache.pulsar.functions.worker.Worker - Checking if pulsar service at <https://pulsar-proxy-us-east-1.internal.pulsar-testing.foobar.net/> is up...
09:33:09.340 [main] INFO org.apache.pulsar.functions.worker.Worker - Initializing Pulsar Functions namespace...
09:33:09.422 [main] ERROR org.apache.pulsar.functions.worker.Worker - Failed to get retention policy for pulsar function namespace pulsar/functions
org.apache.pulsar.client.admin.PulsarAdminException$NotAuthorizedException: Unauthorized to validateNamespacePolicyOperation for operation [READ] on namespace [pulsar/functions] on policy [ALL]```
Does anyone know where is the `Unauthorized to validateNamespacePolicyOperation` coming from?
heavy_check_mark : Damien Roualen
----
2020-10-26 11:25:48 UTC - Vladimir Shchur: Hi! Can I make pulsar function decrypt input messages and encrypt new messages after processing (e2e encryption)? I don't see any option to pass private/public keys to pulsar function, is it unsupported for now?
----
2020-10-26 12:21:54 UTC - Jeremy Hanna: okay, thanks! I will create the issue there.
----
2020-10-26 12:55:16 UTC - jia zhai: :+1:
----
2020-10-26 13:38:18 UTC - David Tinker: Is it possible to change the configuration store for a running Pulsar cluster? It is currently using the local zk cluster as its config store. Tx.
----
2020-10-26 14:04:08 UTC - studor: @studor has joined the channel
----
2020-10-26 14:18:07 UTC - Guillaume: *Partionned Topic vs Non Partionned Topic: is there a broker load difference ?*
Hi,
I'd like to know if there is a broker load difference if I create partionned or non-partitionned topic. I know that there is a different in terms of topic throughput but my goal is to limit the load on the broker and not increase performance.
Thank you.
----
2020-10-26 15:27:31 UTC - Addison Higham: I don't believe compaction is aware of chunked messages yet
----
2020-10-26 15:29:41 UTC - Addison Higham: @Damien Roualen I assume you are running with authentication and authorization? do you have the role/principal you are using for functions worker marked as a superuser?
----
2020-10-26 15:34:02 UTC - Addison Higham: @David Tinker if you take care of manually copying the data over in zookeeper, you can make that change. Easiest way to do that is use something like burry (<https://github.com/mhausenblas/burry.sh>) and then delete all the tree except "admin", import "admin" to a the new zookeeper, and then change over to the configuration store
----
2020-10-26 15:35:13 UTC - Addison Higham: for single topics vs single partition of a topic, those are essentially equivalent to a broker, but since a single topic (or partitioned topic) can only be on a single broker at once, partitioned topics allow you to distribute load across multiple brokers
----
2020-10-26 15:38:02 UTC - David Tinker: Tx. So once I have copied the data I need to update configurationStoreServers in broker.conf and restart each broker? Is it ok to do this while the cluster is running?
----
2020-10-26 15:56:10 UTC - Guillaume: *Lots of bundle error when lots of topics*
```org.apache.pulsar.client.api.PulsarClientException$LookupException: Namespace bundle for topic (<persistent://tenant/agent_ns/abcde>) not served by this instance. Please redo the lookup. Request is denied: namespace=tenant/agent_ns```
I have a lot of those kind of error when I connect to Pulsar. Do you know how to avoid them ? Thank you.
----
2020-10-26 16:02:37 UTC - Addison Higham: as long as you don't build any new tenants, namespaces, or change policies, it should be okay... but I would test this first in a non-production cluster, it is possible I am forgetting some zookeeper state...
----
2020-10-26 16:03:04 UTC - Addison Higham: @Sijie Guo am I remembering that all correctly that configuration store is all under the `admin` tree in ZK?
----
2020-10-26 16:03:43 UTC - Addison Higham: what version of Pulsar are you using?
----
2020-10-26 16:04:39 UTC - Guillaume: latest one on kubernetes, it's 2.6.1
----
2020-10-26 16:08:22 UTC - Addison Higham: are you seeing this after either a restart of a broker (either planned or unplanned) or after you see logs indicating load balancing is happening?
----
2020-10-26 16:11:02 UTC - Guillaume: I think all the brokers crashed because I lost a lot of topics all of the suddent. When they started again, topic number was very low compared to what is was before. But I thought that topics information were saved on bookie and broker were stateless ?
----
2020-10-26 16:17:24 UTC - Guillaume:
----
2020-10-26 16:21:39 UTC - Vladimir Shchur: @Sijie Guo Can you please clear up the question above? If e2e encryption/decryption is not supported in functions, are there any plans on adding this feature?
----
2020-10-26 16:34:35 UTC - Sijie Guo: I don’t think it is supported in functions yet. There are plans to add this feature. Feel free to create an issue for that.
----
2020-10-26 16:35:04 UTC - Sijie Guo: yes that’s correct.
+1 : David Tinker
----
2020-10-26 16:48:44 UTC - Damien Roualen: Thanks @Addison Higham.
Yes exactly. After investigation, I found that I set an expiration date on the JWT token, and it was expired.
----
2020-10-26 16:54:05 UTC - Vladimir Shchur: Thank you
----
2020-10-26 17:09:42 UTC - Devin G. Bost: Thanks everyone for jumping in and helping on this so much!
----
2020-10-27 05:18:41 UTC - Brendan Lynch: @Brendan Lynch has joined the channel
----
2020-10-27 08:57:00 UTC - Guillaume: *Can multi topic subscription support thousands of topics ?*
I have thousands of topics and one consumer for all of those topics. All of the topics are independant one from another.
I need to consume from and produce to a thousands of topics.
Is it best to use Pulsar function (and therefore can it support thousands of consumers) or how can I easily suscribe to all of those topics ? Same issue to produce to thousands of topics.
----
2020-10-27 09:05:56 UTC - Mike S: @Mike S has joined the channel
----