You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@pulsar.apache.org by Apache Pulsar Slack <ap...@gmail.com> on 2018/07/24 09:11:04 UTC
Slack digest for #general - 2018-07-24
2018-07-23 11:05:35 UTC - Yuvaraj Loganathan: @Yuvaraj Loganathan pinned a message to this channel.
----
2018-07-23 17:20:41 UTC - Guillaume Pitel: @Guillaume Pitel has joined the channel
----
2018-07-23 17:22:26 UTC - Karthik Palanivelu: Hi, How do I query the zookeeper to get all the bookies using the /bin/pulsar?
----
2018-07-23 17:25:23 UTC - Sijie Guo: There is a bookkeeper script shipped in the binary distribution: bin/bookkeeper, you can run bin/bookkeeper shell listbookies -rw
----
2018-07-23 17:28:43 UTC - Karthik Palanivelu: Thank You
----
2018-07-23 17:37:19 UTC - shihao zhong: Hi guys, thanks for help last friday
----
2018-07-23 17:37:29 UTC - shihao zhong: I did add new tenant and namespace in the pulsar server
----
2018-07-23 17:37:40 UTC - shihao zhong: but I still unable to connect using python-client
----
2018-07-23 17:38:02 UTC - shihao zhong: 100.65.236.218:6650] Connected to broker
15:04:32.206 [0x7f9cdcaa7700] ERROR /pulsar/pulsar-client-cpp/lib/ClientConnection.cc(699) - [100.65.184.47:55252 -> 100.65.236.218:6650] Failed partition-metadata lookup req_id: 1 error: 1
15:04:32.206 [0x7f9cdcaa7700] ERROR /pulsar/pulsar-client-cpp/lib/ClientImpl.cc(241) - Error Checking/Getting Partition Metadata while Subscribing- 5
----
2018-07-23 17:38:06 UTC - shihao zhong: Here is the error message
----
2018-07-23 17:38:13 UTC - shihao zhong: I use python client
----
2018-07-23 18:31:56 UTC - Karthik Palanivelu: Hi, I am having two clusters A and B. Is there a way to add bookies across the clusters to zookeepers so that I can have my brokers store data across clusters?
----
2018-07-23 23:41:55 UTC - jia zhai: @Karthikeyan Palanivelu Basically, it need another zookeeper quorums(Global zookeeper) on multi-clusters. Here is a guide regarding multi-cluster deployment:
<https://pulsar.incubator.apache.org/docs/latest/deployment/instance/>
----
2018-07-23 23:51:19 UTC - jia zhai: @shihao zhong There was some error in python-client in python 2.0, the latest master should fix it.
Please reference this link, for how to use the latest python client: <https://pulsar.incubator.apache.org/docs/latest/clients/Python/>
----
2018-07-24 00:57:49 UTC - Karthik Palanivelu: @jia zhai I do not want to cluster ZKs due to its known issues. I want to launch in kubernetes. All I need is bookies address in ZK for the brokers to interact. I need your help how I can add a bookie to ZK using the tooling we have in Pulsar.
----
2018-07-24 01:06:11 UTC - jia zhai: @Karthikeyan Palanivelu But the bookies are in different clusters, and the ZK only knows bookies in the local cluster, It does not know the other cluster information. Here we need some method to do service discovery.
@Matteo Merli @Sijie Guo Is there such a way, without cluster zks?
I recall you have mentioned a way to hard code the bookie’s info, and keep it somewhere.
----
2018-07-24 01:50:57 UTC - Sijie Guo: > Hi, I am having two clusters A and B. Is there a way to add bookies across the clusters to zookeepers so that I can have my brokers store data across clusters?
@Karthik Palanivelu you want a bookie registered into both clusters? or something else?
----
2018-07-24 01:53:12 UTC - Karthik Palanivelu: Yes @Sijie Guo I want to register bookies across the clusters
----
2018-07-24 01:53:54 UTC - Karthik Palanivelu: But I do not want to build through ZK quorum
----
2018-07-24 01:54:09 UTC - Karthik Palanivelu: Manually want to register it
----
2018-07-24 01:57:45 UTC - Sijie Guo: @Karthik Palanivelu I understand what you want. However I think it is not currently supported by bookkeeper yet. we definitely can *manually* register a bookie to a zookeeper cluster, however that would cause a problem on ledger id generation, because two different clusters might be conflicting on ledger id.
it is only doable when bookkeeper itself supports some sort of *namespacing* or *scoping* to ensure a bookie registered two different zookeeper cluster, would not be conflicting at ledgers.
----
2018-07-24 02:07:46 UTC - Karthik Palanivelu: Ok, can you please let me know whether I can provide ZK servers from cluster A to bookie conf along with ZK clusters from Cluster B without ZKs not participating in quorums? Will this work out? And Viceversa
----
2018-07-24 09:02:19 UTC - Sijie Guo: @Karthik Palanivelu sorry for late response. I don’t think that would work. What is your original purpose for doing that? are you thinking of active-active replication between clusters?
----
2018-07-24 09:03:39 UTC - Idan: @Sijie Guo regarding the warnings we have. this is something really basic. we are simulating producer-consumer and we get for each message this warning:
----
2018-07-24 09:03:48 UTC - Idan: 12:00:20.201 WARN o.a.p.c.i.UnAckedMessageTracker:91 - [ConsumerBase{subscription=‘subscr-wallet’, consumerName=‘9e188’, topic=‘WithdrawEventQueue’}] 1 messages have timed-out
----
2018-07-24 09:04:11 UTC - Idan: perhaps you guys dont see it coz you didnt enable those logs?
----
2018-07-24 09:04:16 UTC - Idan: thats on the consumer side
----
2018-07-24 09:05:32 UTC - Sijie Guo: @Idan did you ack?
----
2018-07-24 09:05:39 UTC - Idan: yes
----
2018-07-24 09:05:40 UTC - Idan: i did
----
2018-07-24 09:05:42 UTC - Sijie Guo: and what is your ackTimeout
----
2018-07-24 09:05:44 UTC - Idan: also we dont see re-deliveries
----
2018-07-24 09:05:47 UTC - Idan: 3 seconds
----
2018-07-24 09:05:50 UTC - Idan: i tried with 10 seconds aswell
----
2018-07-24 09:05:57 UTC - Idan: after 10 seconds we get this warning
----
2018-07-24 09:06:02 UTC - Idan: (in case we increase the acktimeout to 10 seconds)
----
2018-07-24 09:06:06 UTC - Idan: it’s like we dont have re-deliver
----
2018-07-24 09:06:12 UTC - Idan: but still see this warning
----
2018-07-24 09:06:15 UTC - Idan: on the consumer side
----
2018-07-24 09:06:18 UTC - Sijie Guo: can you show me your logic?
----
2018-07-24 09:06:54 UTC - Idan: yes
----
2018-07-24 09:06:58 UTC - Idan: producer:
----
2018-07-24 09:07:09 UTC - Idan: ill show you the consumer:
----
2018-07-24 09:08:31 UTC - Idan: @Idan shared a file: <https://apache-pulsar.slack.com/files/UALJD8929/FBT632J73/acking_logic.java|acking logic>
----
2018-07-24 09:08:38 UTC - Idan: @Idan shared a file: <https://apache-pulsar.slack.com/files/UALJD8929/FBQAVRE76/Pulsar_Consumer_logic.java|Pulsar Consumer logic>
----
2018-07-24 09:09:29 UTC - Idan: @Idan uploaded a file: <https://apache-pulsar.slack.com/files/UALJD8929/FBV3W6L9W/trigger_ack_.java|trigger ack:>
----
2018-07-24 09:10:03 UTC - Idan: i do see: Message acked: 8:9:-1:0 topic WithdrawEventQueue envetId lEc0ztWlqF2QTh5vmMv1532423201255
----
2018-07-24 09:10:37 UTC - Idan: meaning an ack success happened. but after the timeout i still get this warning. we see it on pulsar cluster and pulsar standalone mode
----