You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@pulsar.apache.org by Apache Pulsar Slack <ap...@gmail.com> on 2019/08/15 09:11:03 UTC
Slack digest for #general - 2019-08-15
2019-08-14 09:13:32 UTC - Vinay Aggarwal: Hi I have deploy pulsar in a machine and when client is trying to connect to pulsar, i am getting error not enough bookies available. Its a one node cluster and command bin/bookkeeper shell bookiesanity is working fine without any error. Please help
----
2019-08-14 09:14:16 UTC - Ali Ahmed: @Vinay Aggarwal are you running standalone ?
----
2019-08-14 09:15:00 UTC - Vinay Aggarwal: no
----
2019-08-14 09:15:59 UTC - Vinay Aggarwal: I want to establish two one cluster at different geo location and replicate the data between them
----
2019-08-14 09:16:12 UTC - Vinay Aggarwal: <https://pulsar.apache.org/docs/en/deploy-bare-metal-multi-cluster/>
----
2019-08-14 09:16:49 UTC - Vinay Aggarwal: This is the url i am using to setup the environment
----
2019-08-14 09:17:00 UTC - Ali Ahmed: you have to configure the quorum size correctly
----
2019-08-14 09:18:40 UTC - Vinay Aggarwal: in conf/bookkeeper.conf ?
----
2019-08-14 10:53:57 UTC - Vladimir Shchur: @dba Can you please share who are "we" in your case?
----
2019-08-14 10:54:51 UTC - dba: @Vladimir Shchur A Danish energy trading company called "Danske Commodities".
----
2019-08-14 10:56:31 UTC - Vladimir Shchur: Thanks! I'm writing a competitive .net client, so it will be nice to compare them!
+1 : dba
----
2019-08-14 11:04:37 UTC - Vinay Aggarwal: I have updated below properties managedLedgerDefaultWriteQuorum=1
managedLedgerDefaultAckQuorum=1
Still i am getting the same error
org.apache.bookkeeper.mledger.ManagedLedgerException: Not enough non-faulty bookies available
----
2019-08-14 12:07:22 UTC - Martin Svensson: Hi! I'm not getting tiered storage to work as expected. As soon as I offload a topic my consumers fail to consume messages. The files show up in S3 and contains data but the pulsar-client cannot get messages. Before offloading it can consume fine however.
I'm currently running 2.4 in kubernetes and offloading to s3
Thanks for any help on this
----
2019-08-14 12:12:16 UTC - Chris Bartholomew: @Vinay Aggarwal You should also set managedLedgerDefaultEnsembleSize=1. All these settings need to be in broker.conf, not bookkeeper.conf.
----
2019-08-14 13:47:49 UTC - Sijie Guo: > my consumers fail to consume messages.
Can you describe more about this?
----
2019-08-14 14:19:29 UTC - Martin Svensson: sure
I created a topic like in the example <https://streaml.io/blog/configuring-apache-pulsar-tiered-storage-with-amazon-s3> but the consumer I created stopped being able to consume messages after I initiated offloading
----
2019-08-14 14:20:51 UTC - Martin Svensson: I'm not using standalone but instead I am running it on gke kubernetes
----
2019-08-14 14:21:46 UTC - Martin Svensson: it's version latest which I assume is 2.4
----
2019-08-14 14:21:54 UTC - Martin Svensson: I'm using 2.4 on my client as well
----
2019-08-14 14:22:05 UTC - Martin Svensson: I'm connecting using the proxy
----
2019-08-14 14:24:40 UTC - Martin Svensson: since I was able to consume messages before the offload and the offload actually generated files on s3 it led me to believe that the broker has trouble reading from s3
----
2019-08-14 14:31:22 UTC - Martin Svensson: I am actually seeing this:
org.jclouds.rest.AuthorizationException: Access Denied
in the logs
----
2019-08-14 14:32:17 UTC - Martin Svensson: I'll look into the permissions
----
2019-08-14 15:10:15 UTC - Alexandre DUVAL: As a producer defines the schema of the topic if the topic not exists, a subscription doesn't?
----
2019-08-14 16:17:07 UTC - Tarek Shaar: Our system heavily depends on creating multiple dynamic topics for each user. Since we have a large user base, I am trying to simulate the creation of 30k topics concurrently but I am getting an exception stating that Namespace is being unloaded, cannot add topic <persistent://fxc1/rfs1/topic20298>.
----
2019-08-14 18:04:27 UTC - Grant Wu: Is it possible to lock down subscription creation?
----
2019-08-14 18:08:17 UTC - Luke Lu: to prevent a subscription from being deleted accidentally?
----
2019-08-14 18:28:14 UTC - Grant Wu: to prevent a subscription from being created accidentally, rather
----
2019-08-14 18:32:30 UTC - Luke Lu: <https://pulsar.apache.org/docs/en/admin-api-permissions/>
----
2019-08-14 19:05:21 UTC - Axel Barfod: Hi, is it posible to configure Brokers how they distribute the workload ?
----
2019-08-14 19:06:09 UTC - Ali Ahmed: @Axel Barfod in what way ?
----
2019-08-14 19:08:01 UTC - Axel Barfod: Im stress testing my cluster with 5 topics (partitioned) and I'm seeming that 4 brokers are taking all the work and I see 6 almost idle
----
2019-08-14 19:09:11 UTC - Ali Ahmed: how many partitions do you have ?
----
2019-08-14 19:09:39 UTC - Axel Barfod: 10
----
2019-08-14 19:09:43 UTC - Axel Barfod: of each topic
----
2019-08-14 19:12:37 UTC - Ali Ahmed: is there a bottleneck on the 4 brokers ?
----
2019-08-14 19:14:06 UTC - Axel Barfod: the brokers are at 90% cpu
----
2019-08-14 19:14:21 UTC - Axel Barfod: im working with version 1.22
----
2019-08-14 19:15:54 UTC - Ali Ahmed: that’s a pretty old version you can consult with @Matteo Merli he will probably know more about what’s the issue
----
2019-08-14 19:16:32 UTC - Axel Barfod: Yeah it's pretty old
----
2019-08-14 19:20:34 UTC - Axel Barfod: version
----
2019-08-14 19:20:38 UTC - Axel Barfod: we are using, if @Matteo Merli has any idea would be great! thanks @Ali Ahmed
----
2019-08-14 21:14:20 UTC - Raman Gupta: @Raman Gupta has joined the channel
----
2019-08-14 21:15:08 UTC - Aaron: Occasionally when subscribing to a topic that I just published records to, I get the error 'trying to subscribe with incompatible schema'. I am using the same java class as my schema on both the publisher and subscriber end. This happens only once in a while, and often happens if the topic has been alive for a while. Any ideas on the root of this issue/any quick fixes?
----
2019-08-14 21:27:29 UTC - Ali Ahmed: @Aaron what is the schema type being used ?
----
2019-08-14 21:30:36 UTC - Aaron: It is a java class
----
2019-08-14 21:31:07 UTC - Aaron: Simple class with getters and setters
----
2019-08-14 21:31:11 UTC - Ali Ahmed: what’s the serialization format json , protobuf or avro ?
----
2019-08-14 21:31:42 UTC - Aaron: JSON
----
2019-08-14 21:32:35 UTC - Ali Ahmed: ok I am aware of any issues could be problem with missing key value pairs , is there more info in
the logs
----
2019-08-14 21:40:06 UTC - Addison Higham: anyone know of or is interested in a kinesis IO source? We know about the sink but want a source, think of just starting that inside pulsar with existing connectors?
----
2019-08-14 21:41:28 UTC - Addison Higham: oh just looked, <https://github.com/apache/pulsar/pull/3784>
----
2019-08-14 21:42:52 UTC - Addison Higham: we would want to add the ability to assume an IAM role, but that looks easy to do
----
2019-08-14 21:43:25 UTC - Ali Ahmed: @Addison Higham sure you can enhance the the one in the pulsar repo and make a PR
----
2019-08-14 21:43:44 UTC - Addison Higham: any idea what is holding up getting what is there merged?
----
2019-08-14 21:44:31 UTC - Addison Higham: rather than have to start with a pending PR, we could help get that PR over the line and then just add any other bits we need
----
2019-08-14 21:59:40 UTC - Tarek Shaar: Hello Chris that turned out to be exactly right. Each producer has to be associated with one topic. However when I do try to create 20k concurrent producers and associated each one with them with a topic, I get disconnected from the broker. And it complains that my name space is getting offloaded. Is there a setting to control this?
----
2019-08-14 22:19:47 UTC - Addison Higham: @David Kjerrumgaard ^^ that looks to be your work in that PR, we were just about to embark on writing our own kinesis source, so happy to help doing anything needed to get that over the line
----
2019-08-15 00:10:40 UTC - Ming Fang: @Ming Fang has joined the channel
----
2019-08-15 07:42:14 UTC - Richard Sherman: The stats are collected by default once every 60 seconds hence the lag. This can be changed by passing a value, in seconds, using the environment variable COLLECTION_INTERVAL
----