Slack digest for #general - 2018-06-29

[Apache Pulsar Slack <ap...@gmail.com>]

2018-06-28 18:25:44 UTC - Karthik Palanivelu: Team, One Observation is when I connect producer; Proxy forwards the Authorization Token correctly but it is not in the case of Consumer. It is null by Consumer
----
2018-06-28 18:56:36 UTC - Karthik Palanivelu: @Karthik Palanivelu uploaded a file: <https://apache-pulsar.slack.com/files/U7VRE0Q1G/FBFTB65HQ/-.txt|Untitled> and commented: Any recommendation on above error; I am using the daemon set within K8s for creating bookie.
----
2018-06-28 19:05:44 UTC - Sijie Guo: @Karthik Palanivelu it looks like instance id mismatch
----
2018-06-28 19:06:18 UTC - Sijie Guo: did you reformat the cluster metadata without cleaning up data stored on each bookie?
----
2018-06-28 19:06:42 UTC - Karthik Palanivelu: Yes you are correct
----
2018-06-28 19:07:15 UTC - Sijie Guo: bookkeeper does very strict data integrity checks on startup, it prevents to start a malformed bookie into a cluster.
----
2018-06-28 19:07:46 UTC - Karthik Palanivelu: But they are different Metadata
----
2018-06-28 19:08:00 UTC - Karthik Palanivelu: So only one Metadata is allowed?
----
2018-06-28 19:08:11 UTC - Sijie Guo: so here is what happened.
----
2018-06-28 19:08:18 UTC - Karthik Palanivelu: How would I delete it in case if need to be?
----
2018-06-28 19:09:33 UTC - Sijie Guo: when a bookie first time joined a cluster, it will record the instance-id as “cookies” stored in its directory. 

when you reformat the cluster metadata, the cluster generates a new instance id. but if you don’t format bookies, the bookies will attempt to start with an old instance id, the check will fail bookie startup.
----
2018-06-28 19:09:45 UTC - Sijie Guo: that’s what you saw instance id mismatch
----
2018-06-28 19:09:49 UTC - Sijie Guo: what you can do
----
2018-06-28 19:10:11 UTC - Sijie Guo: on each bookie node, remove the directory used by bookkeeper to store data
----
2018-06-28 19:10:16 UTC - Sijie Guo: and restart bookie pods
----
2018-06-28 19:10:27 UTC - Karthik Palanivelu: Got it
----
2018-06-28 19:10:35 UTC - Karthik Palanivelu: Thanks so much
----
2018-06-28 19:10:43 UTC - Sijie Guo: no problem.
----
2018-06-28 19:11:13 UTC - Sijie Guo: regarding “Proxy forwards the Authorization Token correctly but it is not in the case of Consumer. It is null by Consumer”.

I need to check the code. but @Matteo Merli might have a quick idea on it.
----
2018-06-28 19:12:33 UTC - Matteo Merli: @Karthikeyan Palanivelu Do you mean consumer is not able to connect trhough proxy?
----
2018-06-28 19:13:05 UTC - Matteo Merli: the auth token is sent by client instance, so it shouldn’t be depending on producer vs consumer
----
2018-06-28 19:15:36 UTC - Karthik Palanivelu: In my case, Producer and Consumer are my client apps. They are sending it correctly to Proxy. From Proxy to Broker, I can see in logs that OriginalPrincipal is null when it is a Consumer. In case of my Producer app, I can see the OriginalPrincipal is rightly populated
----
2018-06-28 19:17:06 UTC - Karthik Palanivelu: @Karthik Palanivelu uploaded a file: <https://apache-pulsar.slack.com/files/U7VRE0Q1G/FBGR0JA22/-.m|Untitled> and commented: Sample Log
----
2018-06-28 19:19:25 UTC - Karthik Palanivelu: Team, @Karthik Ramasamy @Matteo Merli @Sijie Guo @Ivan Kelly I set up two k8s and configured the Clusters, tenants and Namespace in both the setup as described by Matteo at <https://gist.github.com/merlimat/bfdd3c66bfa7387fe6e3bfa3bb263051>. All config was successful, I started my Producer Client App and Consumer Client App on cluster 1 and Replication is still there in Cluster 2. Client Apps got connected and I can view that in the logs. But even after connecting Producer Client App did not produce messages and I am not sure where it hangs. So I decided to test again if there is anything wrong I did in my config by stepping down to one cluster; everything works fine - Publishing the message and receiving the message. Can you please help me here to test geo-replication?
----
2018-06-28 19:23:09 UTC - Matteo Merli: @Karthikeyan Palanivelu Checking now the source code. Again, the message is printed when the connection is established and that part comes before creating a producer/consumer. I don’t see any reason for having a diff behavior between producer and consumer. Can you double check the consumer is indeed connecting through proxy?
----
2018-06-28 19:27:46 UTC - Karthik Palanivelu: @Matteo Merli Yes they are, that is when my roles get switched and OriginalAuth gets populated. That was my understanding and seeing it for Producer
----
2018-06-28 20:11:05 UTC - Sijie Guo: @Karthik Palanivelu :
so with one cluster, publish and consume work well.
with replicated clusters, is it hanging on producing or consumer didn’t receive the message?
----
2018-06-28 20:34:45 UTC - Karthik Palanivelu: Yes @Sijie Guo it is hanging on producing message. I have the setup listen same end point for producer and consumer. I was about to switch to replication endpoint if it works but this case failed.
----
2018-06-28 20:44:28 UTC - Sijie Guo: hmm interesting. the producer behavior should be irrelative to whether replication is setup or not. is there any configuration settings changed between one cluster setup and replicated clusters setup?
----
2018-06-28 21:59:58 UTC - Karthik Palanivelu: No @Sijie Guo only difference is replication.
----
2018-06-28 22:05:00 UTC - Sijie Guo: @Karthik Palanivelu that’s weird. do you mind sharing me (you can send an email or DM me) more informations about the sequence for your setup. I am suspecting you might provide wrong ip or domain name on creating clusters for replication, which can potentially cause the producer hangs on connecting to the ip/domain specified on creating clusters.
----
2018-06-28 22:19:26 UTC - Karthik Palanivelu: Sure, will do.
----