Slack digest for #general - 2019-07-27

[Apache Pulsar Slack <ap...@gmail.com>]

2019-07-26 12:16:09 UTC - Ben: Ah, the issue was message retention time.
----
2019-07-26 12:29:21 UTC - David Lucas: @David Lucas has joined the channel
----
2019-07-26 13:53:27 UTC - Mahesh: Hi,
I have created a tenant through pulsar-admin command, It got created successfully and I can see the same when I list it. But on pulsar dashboard webUI, I dont see that. What could have gone wrong here ?
----
2019-07-26 13:56:55 UTC - Richard Sherman: A couple of scenarios where this has happened to me
1. Dashboard didn't actually have network access to a broker/proxy
2. Tenant didn't have the cluster set correctly
----
2019-07-26 14:02:52 UTC - Richard Sherman: To check the second run `pulsar-admin tenants get tenant_name`
----
2019-07-26 14:03:01 UTC - Richard Sherman: To check the first one run the collector.sh script in a terminal and see if you get any errors logged.
You'll need to ctrl-c out of the script as it loops and reruns the collection code every 60 seconds
----
2019-07-26 14:34:26 UTC - Alex Lepsa: @Alex Lepsa has joined the channel
----
2019-07-26 14:43:10 UTC - Alex Lepsa: Hi, I'd like to use a `protoc` generated class to set the schema for a topic but trying to do that via something like `JSONSchema.of(TaskProtos.Task.class)` is not working. How do you guys (or don't you) use protobuf generated classes to set a schema?
----
2019-07-26 14:49:00 UTC - Mahesh: I think I found the issue, I gave wrong cluster name. After fixing the cluster name, now i get this error
----
2019-07-26 14:49:01 UTC - Mahesh: Namespace missing local cluster name in clusters list: local_cluster=bpulsar ns=public/functions clusters=[apulsar]
----
2019-07-26 14:49:21 UTC - Alex Lepsa: or is a .proto based schema registration only supported server side?
----
2019-07-26 14:49:37 UTC - Mahesh: ```
[zk: localhost:2181(CONNECTED) 17] ls /admin/policies/public/functions
[]
```
----
2019-07-26 14:49:37 UTC - Mahesh: what could be wrong here ?
----
2019-07-26 14:53:08 UTC - Thomas Cothran: @Thomas Cothran has joined the channel
----
2019-07-26 14:54:30 UTC - Richard Sherman: Does the following command show the correct cluster name?
pulsar-admin namespaces get-clusters tenant_name/namespace_name
----
2019-07-26 14:56:24 UTC - Ryan Samo: Hey guys,
Does Pulsar SQL work in a cluster that is setup with TLS authentication and authorization? It couldn’t find an example setup for that in the docs or source code
----
2019-07-26 14:59:15 UTC - Mahesh: yes, it returns,
```
<ma...@dev-assist-bpulsar01.app.shared.int.sv2.247-inc.net>:/usr/local/apache-pulsar/bin# ./pulsar-admin namespaces get-clusters public/default
apulsar
bpulsar
```
----
2019-07-26 14:59:31 UTC - Mahesh: after sometime broker exits
----
2019-07-26 15:26:46 UTC - David Kjerrumgaard: @Ryan Samo I wouldn't see why not, although I am unaware of any tests conducted in such an environment. Are you experiencing issues with such a configuration?
----
2019-07-26 15:28:53 UTC - Ryan Samo: @David Kjerrumgaard yeah I tried it a while back and was receiving an authorization error. I need to try it once more
----
2019-07-26 15:30:36 UTC - Ryan Samo: Just wondering if you all had an example of a cluster config that worked
----
2019-07-26 15:32:52 UTC - Ryan Samo: A secure cluster :)
----
2019-07-26 15:35:53 UTC - David Kjerrumgaard: @Ryan Samo At which level were you getting the auth error?  And what are you using to secure the cluster? TLS, JWT?
----
2019-07-26 15:37:48 UTC - Ryan Samo: TLS
----
2019-07-26 15:38:24 UTC - Ryan Samo: Let me start a worker and retry it to see. It was when I tried to perform a sql query
----
2019-07-26 15:59:09 UTC - Ryan Samo: @David Kjerrumgaard Error: Query 20190726_155712_00001_mntr3 failed: Failed to get schemas from pulsar: error:10000412:SSL routines:OPENSSL_internal:SSLV3_ALERT_BAD_CERTIFICATE
----
2019-07-26 16:06:19 UTC - Jerry Peng: @Ryan Samo auth in sql was just added and is not currently in any official release
----
2019-07-26 16:06:42 UTC - Jerry Peng: Feel free to compile from master to try it out
----
2019-07-26 16:07:12 UTC - Ryan Samo: @Jerry Peng that’s awesome! Thanks for the tip, I sure will!
----
2019-07-26 23:15:09 UTC - Devin G. Bost: @Jerry Peng
What does it mean when there is a backlog on a topic without any un-acked messages?
----
2019-07-26 23:15:33 UTC - Devin G. Bost: We have a producer/consumer that's not processing messages, and we're trying to figure out what's wrong.
----
2019-07-26 23:36:46 UTC - Jerry Peng: @Devin G. Bost that means your consumers for a particular subscriptions is not consuming messages as fast as messages are getting produced to the topic
----
2019-07-26 23:42:03 UTC - Jerry Peng: @Devin G. Bost is the consumer msgRateOut for the consumer 0?
----
2019-07-26 23:42:10 UTC - Jerry Peng: check availablePermits
----
2019-07-27 03:07:55 UTC - Sijie Guo: @Devin G. Bost :

a few things you can check:

1) try unload the topic to see if the consumer can resume. if it resumes, it might be related to any race conditions in the permits flow; if it resumes and falls back again, check the stats-internal to see if there is any outstanding read requests. It can be related to too small a block cache configured in bookies, which results in index pages swap in-and-out when consuming large backlog.

if you expect to have large backlog, consider configuring dbStorage_rocksDB_blockCacheSize to be 1~2GB or even more. I have seen a lot of similar cases related to this setting.

2) if unload a topic doesn’t help, check if you are using 2.3.0 or earlier versions. If so, check if `bookkeeperEnableStickyReads` is set to true. If it is set to true, disable it. There is a problem of  sticky reads. I have also seen similar cases related to this setting.

3) if none above addressed your problem, it would be good to grep the `stats-internal` every a few minutes and compare the stats to see if there is any pattern.
----