Slack digest for #general - 2019-03-09

[Apache Pulsar Slack <ap...@gmail.com>]

2019-03-08 13:33:18 UTC - Sébastien de Melo: Hi, we are currently deploying pulsar with helm and the cluster metadata initialization fails because of the following error:
Exception in thread "main" org.apache.zookeeper.KeeperException$NodeExistsException: KeeperErrorCode = NodeExists for /namespace
	at org.apache.zookeeper.KeeperException.create(KeeperException.java:122)
	at org.apache.zookeeper.KeeperException.create(KeeperException.java:54)
	at org.apache.zookeeper.ZooKeeper.create(ZooKeeper.java:792)
	at org.apache.pulsar.PulsarClusterMetadataSetup.main(PulsarClusterMetadataSetup.java:178)
Do you know why?
----
2019-03-08 13:34:28 UTC - Sébastien de Melo: As a consequence the default namespace is not created
----
2019-03-08 14:29:47 UTC - Darragh: Hey @Matteo Merli, we've removed all our old instances, created some new better instances, used the st1 and io1 nvme's and see some improvements but the tail latencies remain.  I've opened up the grafana board temporarily. Could you perhaps take a look and give some more pointers ?
----
2019-03-08 14:30:38 UTC - Darragh: I did a perf-test produce persistent -r 10000 -b 0
----
2019-03-08 14:30:58 UTC - Darragh: (stopped now, so you might have to go back in time)
----
2019-03-08 14:52:21 UTC - Darragh: I can pm you the grafana endpoint with a login if you would be up for that
----
2019-03-08 15:12:32 UTC - Sébastien de Melo: It works when I delay the brokers launch
----
2019-03-08 16:03:59 UTC - Chris DiGiovanni: So I have two questions...
1) How do I get this bookie out of this 100% pickle?  I can't expand the disk/partition on the bookie as it will invalidate itself.  Remove Bookie and auto-recover?
2) How large should by ledger disks be to avoid needing to stray away from the default Entry Log Compaction and Disk Utilization Settings?
3)  I lied, I have a third question :wink:  Docs explain how to invalidate a bookie by changing port, or name/ip.  Since I'm running a StatefulSet in K8s.  Both of these options are less than ideal.  Is there a way to get zookeeper to forget about a bookie if I were to blow it away, allowing it to re-register w/ it's orig name and port?
----
2019-03-08 16:13:12 UTC - Chris DiGiovanni: Maybe the answer to #1 and #3 is to format the bookie is using `bookie shell bookieformat`
----
2019-03-08 16:19:14 UTC - Guy Feldman: When i deployed using the helm chart the broker pods restarted a few times after which they were fine, but the zookeeper metadata job didn't fail
----
2019-03-08 16:19:55 UTC - Guy Feldman: the ordering of the deployment isn't the best
----
2019-03-08 16:20:24 UTC - Guy Feldman: I was going to work on it, but no time
----
2019-03-08 16:40:56 UTC - Sijie Guo: are you using the helm script in pulsar repo?
----
2019-03-08 17:46:48 UTC - David Kjerrumgaard: @Vikas I have some updated NARs that support JWT authentication for the NiFi processors.  Which version of Nifi are you running?
----
2019-03-08 18:24:10 UTC - Vikas: thanks @David Kjerrumgaard, we have NiFi 1.7.1
----
2019-03-08 18:32:48 UTC - David Kjerrumgaard: 
----
2019-03-08 18:33:21 UTC - David Kjerrumgaard: 
----
2019-03-08 18:34:15 UTC - David Kjerrumgaard: The Pulsar client service now has a property named AUTHENTICATION_SERVICE, which you have to define.
----
2019-03-08 18:34:51 UTC - David Kjerrumgaard: There are 3 options for the AUTHENTICATION_SERVICE. Just choose the PulsarClientJwtAuthenticationService option and configure it accordingly
----
2019-03-08 18:35:14 UTC - Vikas: sure, thanks. Let me give a try with the new processor.
+1 : David Kjerrumgaard
----
2019-03-08 19:45:51 UTC - Sébastien de Melo: I have added that for the brokers (in initContainers) and it seems to help:
- name: wait-bookkeeper-ready
        image: "{{ .Values.image.repository }}:{{ .Values.image.tag }}"
        imagePullPolicy: {{ .Values.image.pullPolicy }}
        command: ["sh", "-c"]
        args:
          - >-
            until nslookup {{ template "pulsar.fullname" . }}-{{ .Values.bookkeeper.component }}-{{ add (.Values.bookkeeper.replicaCount | int) -1 }}.{{ template "pulsar.fullname" . }}-{{ .Values.bookkeeper.component }}.{{ .Values.namespace }}; do
              sleep 3;
            done;
----
2019-03-08 19:58:11 UTC - Sébastien de Melo: Yes, with small changes.  I'll give more details on Monday
----
2019-03-08 20:41:54 UTC - Ryan Samo: Is it possible to use DistributedLog on the bookies of a Pulsar cluster? Just thinking it would be handy to reuse my bookies for it
----
2019-03-08 20:42:25 UTC - Ali Ahmed: yes with the right configs
----
2019-03-08 20:43:26 UTC - Ryan Samo: Cool, yeah I see that dlog, etc. is not included with the bookies packaged in Pulsar. I assume I’d have to add that and other things right?
----
2019-03-08 20:43:57 UTC - Ali Ahmed: @Sijie Guo will know better
----
2019-03-08 20:49:50 UTC - chris: @Ryan Samo i think dlog is just a client library so there is no extra configuration. the client library just needs to be able to access zookeeper and each bookie directly
----
2019-03-08 20:51:25 UTC - Ryan Samo: Ok thanks. So what I don’t want to do is break my Pulsar cluster for sure. Trying to figure out how to try DistributedLog on those bookies if possible. Not even sure it’s the best idea really.
----
2019-03-08 20:52:23 UTC - chris: it’s possible and should have no effect on your clusters - pulsar functions actually use it underneath to store binaries
----
2019-03-08 20:54:10 UTC - Ryan Samo: Nice! I saw that DistributedLog is now baked into Bookkeeper so trying to decipher the docs to see how to start trying it out. Do you know if it’s ready to hit API wise as it stands or if there is additional configuration to do to the bookies first?
----
2019-03-08 20:54:44 UTC - chris: no additional configuration should be needed you should be good to go
+1 : Sijie Guo
----
2019-03-08 20:55:22 UTC - Ryan Samo: Super cool! Thanks @chris for the knowledge!
----
2019-03-08 21:58:55 UTC - Vikas: hey @David Kjerrumgaard, the new processor works :grinning:
Thanks a ton for helping me out to publish the messages to the Pulsar topic, have a wonderful weekend!
----
2019-03-08 21:59:31 UTC - David Kjerrumgaard: Glad to hear that. Thank you for the confirmation.  Enjoy the code!!
party-parrot : Vikas
----
2019-03-08 22:26:01 UTC - Sree Vaddi: please join <#CGUU96J22|job-board> channel for jobs/work. thank you. :slightly_smiling_face:
----
2019-03-08 22:54:00 UTC - Sree Vaddi: There is a speaking opportunity on March 25th evening at Walmart in Sunnyvale, CA.  If anyone interested, please let me know by this weekend.  ~100 attends.  The scheduled speaker is out on emergency and ACM is looking for a replacement, hence short notice.
----
2019-03-09 00:24:36 UTC - vinay Parekar: Hi Everyone , I am facing a wierd issue when creating function
----
2019-03-09 00:25:12 UTC - vinay Parekar: if i try to create this function , i get an error : Reason: javax.ws.rs.ProcessingException: Java heap space
----
2019-03-09 00:25:28 UTC - vinay Parekar: Has any one seen this kind of behavior ?
----
2019-03-09 00:26:38 UTC - Ali Ahmed: @vinay Parekar you need to increase the heap space for the admin
+1 : Thor Sigurjonsson
----