You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@pulsar.apache.org by Apache Pulsar Slack <ap...@gmail.com> on 2019/05/23 09:11:02 UTC
Slack digest for #general - 2019-05-23
2019-05-22 11:06:38 UTC - Alexandre DUVAL: Hi, if I have acknowledge a message that is the last one of a reading, all the previous messages of this reading are ack too?
----
2019-05-22 11:10:34 UTC - Shivji Kumar Jha: @Alexandre DUVAL may not be, for that behavior you have to send cumulative ack
----
2019-05-22 11:10:54 UTC - Alexandre DUVAL: Okay :slightly_smiling_face:, thanks.
----
2019-05-22 11:20:31 UTC - Shivji Kumar Jha: Hi, Is there a java util method somewhere in pulsar client that returns the org.apache.pulsar.client.api.Authentication instance given:
private String authPluginClassName;
private String authParams;
?
In a way that it works for all implementations of org.apache.pulsar.client.api.Authentication…
----
2019-05-22 11:23:17 UTC - Shivji Kumar Jha: I was going to write one but then I think there should be a util somewhere and i dont want to duplicate it if it is there already!
----
2019-05-22 11:31:21 UTC - Shivji Kumar Jha: @jia zhai ^
----
2019-05-22 11:32:32 UTC - Venugopal Doddi: @Venugopal Doddi has joined the channel
----
2019-05-22 11:34:20 UTC - jia zhai: @Shivji Kumar Jha AuthenticationFactory for server, and AuthenticationUtil for client
----
2019-05-22 11:34:21 UTC - Venugopal Doddi: @Sijie Guo I am following you on the <http://streaml.io|streaml.io> and i am currently working on a POC for pulsar in my organization. I am looking for some help on the geo-replication setup. Is there something you can help
----
2019-05-22 11:34:56 UTC - Venugopal Doddi: I am getting trouble in setting the --allowed-clusters parameter
----
2019-05-22 11:35:38 UTC - Venugopal Doddi: i am trying this in two datacenters and i have clusters names test and test-ttc created in both
----
2019-05-22 11:35:57 UTC - Venugopal Doddi: i was not sure how the connection or the handshake i can establish between both of them..
----
2019-05-22 11:47:16 UTC - jia zhai: @Venugopal Doddi It is auto established once the topic under a configured namespace is created
----
2019-05-22 12:05:08 UTC - jia zhai: As in this pic, there are 2 cluster: beijing, shanghai; And a namespace that configured to replicated between beijing and shanghai.
once the topic is created, there is a replicator been created to handle the connection.
----
2019-05-22 13:00:37 UTC - Venugopal Doddi: @jia zhai thanks for the picture.. but how will i make the connection or the communication between both the clusters.. in my example.. there are two clusters.. test-ttc and test-tte both are in different data centers altogether.
----
2019-05-22 13:00:54 UTC - Venugopal Doddi: when i am running this command ```pulsar-admin properties create my-tenant \
--admin-roles my-admin-role \
--allowed-clusters us-west,us-east,us-cent```
----
2019-05-22 13:01:06 UTC - Venugopal Doddi: in place of the us-west, cent, east what is that i have to give..
----
2019-05-22 13:01:47 UTC - Venugopal Doddi: and i am running this command on one of the cluster in ttc..
----
2019-05-22 13:02:06 UTC - Venugopal Doddi: m y question is how will i make the cluster in ttc and tte talk to each other.
----
2019-05-22 13:03:10 UTC - jia zhai: right it gives the cluster name: test-ttc, test-tte,
----
2019-05-22 13:04:39 UTC - jia zhai: Here is the docker-compose that I used to create 3 clusters：
<https://gist.github.com/jiazhai/1cd48ab6c8f6c3012c895df9746799b5>
----
2019-05-22 13:05:43 UTC - jia zhai: and here is the command to create tenants and namespace:
```
bin/pulsar-admin tenants create my-tenant --allowed-clusters beijing,shanghai,guangzhou
bin/pulsar-admin namespaces create my-tenant/my-namespace --clusters beijing,shanghai,guangzhou
```
----
2019-05-22 13:14:25 UTC - Venugopal Doddi: Right.. it do created cluster ttc and tte. when i run the above command by passing the --allowed-clusters followed by test-ttc,test-tte and when i run the above command on one of the ttc bookkeeper/broker node i am getting error like ```Invalid cluster name test-ttc``` and exiting
----
2019-05-22 13:14:31 UTC - Venugopal Doddi: so this is the issue i am facing
----
2019-05-22 13:15:21 UTC - Venugopal Doddi: when i list the clusters on one of the server located in ttc datacenter, i am getting `test-ttc` similary when i run the list on tte server i am getting `test-tte`
----
2019-05-22 13:15:31 UTC - Jim Lambert: @Jim Lambert has joined the channel
----
2019-05-22 13:16:00 UTC - Venugopal Doddi: @jia zhai ^^
----
2019-05-22 13:16:12 UTC - jia zhai: there maybe some config issue
----
2019-05-22 13:16:17 UTC - jia zhai: wait for a minute. I am writing a tutorial to use above docker-compose file.
----
2019-05-22 13:16:31 UTC - jia zhai: Then you could use docker to verify 3 clusters replciation
----
2019-05-22 13:16:41 UTC - Venugopal Doddi: yes.. thats what i am trying to figure out.. but individually the clusters are succesfully producing and consuming.
----
2019-05-22 13:17:35 UTC - Venugopal Doddi: also the initialize-cluster-metadata
----
2019-05-22 13:17:35 UTC - Venugopal Doddi: is successfully applied
----
2019-05-22 13:20:41 UTC - jia zhai: maybe the config in metadata store?
----
2019-05-22 13:22:24 UTC - Venugopal Doddi: how can i verify that?
----
2019-05-22 13:26:28 UTC - Sijie Guo: @Venugopal Doddi do you deploy a configuration store (global zookeeper)?
----
2019-05-22 13:26:40 UTC - jia zhai: Here is the tutorial:
<https://gist.github.com/jiazhai/e1f0e828754b73ad7792b34d2449690c>
based on the docker-compose here:
<https://gist.github.com/jiazhai/1cd48ab6c8f6c3012c895df9746799b5>
Please make sure configure you docker to have as many cpu and memory as possible
----
2019-05-22 13:26:49 UTC - Sijie Guo: Do you mind sharing the commands sequence you used for creating clusters.
----
2019-05-22 13:45:46 UTC - Venugopal Doddi: yes.. surely i will share..
----
2019-05-22 13:45:49 UTC - Venugopal Doddi: one moment
----
2019-05-22 13:46:01 UTC - Venugopal Doddi: but i am not using docker yet
----
2019-05-22 13:46:09 UTC - Venugopal Doddi: its a stand alone cluster on openstack
----
2019-05-22 13:50:29 UTC - Venugopal Doddi: after the bookkeeper and broker services are started. and able to produce and consume messages.. for geo-replication i am following this..
----
2019-05-22 13:50:38 UTC - Venugopal Doddi: <https://pulsar.apache.org/docs/en/administration-geo/>
----
2019-05-22 13:51:33 UTC - Venugopal Doddi: ```pulsar-admin properties create venu-tenant \
--admin-roles venu-admin-role \
--allowed-cluster test-ttc, test-tte```
----
2019-05-22 13:54:08 UTC - Venugopal Doddi: when i list the clusters.. i am getting only one.. and thats obvious as this is on ttc servers.
----
2019-05-22 13:54:10 UTC - Venugopal Doddi: ```[root@pulsar-bk1-ttc bin]# ./pulsar-admin clusters list
test-ttc```
----
2019-05-22 13:55:31 UTC - Venugopal Doddi: ```[root@pulsar-bk1-ttc bin]# ./pulsar-admin tenants list
public
tenant1
venu-tenant```
----
2019-05-22 13:57:36 UTC - Venugopal Doddi: this is the error...
----
2019-05-22 13:57:39 UTC - Venugopal Doddi: ```[root@pulsar-bk1-ttc bin]# ./pulsar-admin namespaces set-clusters venu-tenant/global/full-mesh --clusters test-ttc,test
Invalid cluster id: test
Reason: Invalid cluster id: test```
----
2019-05-22 13:57:53 UTC - Venugopal Doddi: @jia zhai ^^
----
2019-05-22 13:59:53 UTC - Venugopal Doddi: no.. i didnt deployed the global-zookeeper @Sijie Guo
----
2019-05-22 14:00:38 UTC - Venugopal Doddi: thats a must for the geo-replication?
----
2019-05-22 14:00:54 UTC - Venugopal Doddi: i have the zookeeper cluster of 3 nodes for each data center
----
2019-05-22 14:03:29 UTC - Venugopal Doddi: and configuration-store IP is same as one of the zookeeper node
----
2019-05-22 14:04:00 UTC - Sijie Guo: @Venugopal Doddi are the two clusters pointing to the same zookeeper cluster?
----
2019-05-22 14:05:37 UTC - Sijie Guo: @Venugopal Doddi :
it is not must. it is just a convenient way for propagating the the replication config.
If you don’t have a configuration store, you can try following steps:
a) in test-ttc cluster, run `bin/pulsar-admin clusters create` to create `test-tte` cluster, so in test-ttc cluster, it knows how to connect to `test-tte`.
b) in test-tte cluster, run `bin/pulsar-admin clusters create` to create `test-ttc` cluster, so in test-ttc cluster, it knows how to connect to `test-tte`.
if the two clusters don’t share the same configuration store, you have to configure at both clusters to let each cluster know the other cluster.
----
2019-05-22 14:28:14 UTC - Venugopal Doddi: no
----
2019-05-22 14:28:29 UTC - Venugopal Doddi: they are not point to a same zookeeper cluster
----
2019-05-22 14:28:43 UTC - Venugopal Doddi: they are pointing to both different
----
2019-05-22 14:29:58 UTC - Venugopal Doddi: ```if the two clusters don’t share the same configuration store, you have to configure at both clusters to let each cluster know the other cluster.``` i didnt get this statement.. is there a command or a file i can change with
----
2019-05-22 14:29:59 UTC - Venugopal Doddi: ?
----
2019-05-22 14:30:41 UTC - Venugopal Doddi: @Sijie Guo
----
2019-05-22 14:46:28 UTC - jia zhai: @Venugopal Doddi you could reference the docker-compose file.
line 28 is for the configuration store, and line 52 is do the init for each cluster.
<https://gist.github.com/jiazhai/1cd48ab6c8f6c3012c895df9746799b5#file-docker-compose-yml-L52>
A configuration store was also called global zookeeper.
----
2019-05-22 14:46:59 UTC - Sijie Guo: @jia zhai he doesn’t have the global configuration store. you have to change your example
----
2019-05-22 14:47:21 UTC - jia zhai: :joy:
----
2019-05-22 14:47:30 UTC - Sijie Guo: @Venugopal Doddi let me try to change @jia zhai’s example and show you how to do it
----
2019-05-22 14:54:57 UTC - Venugopal Doddi: sure
----
2019-05-22 14:56:36 UTC - Venugopal Doddi: @Sijie Guo but i am running the cluster in standalone openstack server and not on container.
----
2019-05-22 14:57:22 UTC - Sijie Guo: @Venugopal Doddi understand that. It is just to show you what commands you need to run
----
2019-05-22 14:59:59 UTC - Venugopal Doddi: ok
----
2019-05-22 15:03:34 UTC - Venugopal Doddi: also trying to check.. if we have to implement the apart form the above cluster create commands, both the ttc and tte clusters should point to a single zookeeper cluster only...
----
2019-05-22 15:03:48 UTC - Venugopal Doddi: @Sijie Guo
----
2019-05-22 15:05:09 UTC - Sijie Guo: @Venugopal Doddi :
if you can point to one single configuration store, it will make things easier.
if you can’t do that, you have to create clusters in each cluster to make them learn the other cluster.
----
2019-05-22 15:23:15 UTC - Venugopal Doddi: i will try the second part.. but again.. what i was not convinced is.. even i create the same clusters in both the Datacenters.. how will they both know each other and replicate the data on the same topic that created in both the clusters..
----
2019-05-22 15:35:51 UTC - Sijie Guo: when you create the cluster, you need to provide the brokerServiceUrl and webServiceUrl of that cluster. basically when you create a cluster test-tte in cluster test-ttc, you are telling test-ttc how to access test-tte.
so when you create a namespace with the list of replicated clusters, it will resolve the cluster location from the cluster information you just created.
----
2019-05-22 16:33:36 UTC - Sijie Guo: @Venugopal Doddi: I created a tutorial of geo-replication without a global configuration store.
<https://gist.github.com/sijie/79364497eaa349bf58d9fb760561f930>
----
2019-05-22 16:34:21 UTC - Sijie Guo: <https://gist.github.com/sijie/79364497eaa349bf58d9fb760561f930#2-create-clusters>
and <https://gist.github.com/sijie/79364497eaa349bf58d9fb760561f930#2-create-tenants-and-namespace>
these two sections are the keys to set up geo-replication between clusters without a global configuration store.
----
2019-05-22 17:32:14 UTC - Devin G. Bost: Any chance someone has a higher-resolution image of the Pulsar logo? I've got this one, but it looks really grainy when I scale it up in a PowerPoint.
----
2019-05-22 17:34:03 UTC - Matteo Merli: There’s an SVG: <https://github.com/apache/pulsar/blob/7d75fd28b9fff830697afc3505e30c80369b27f5/site2/website/static/img/pulsar.svg>
----
2019-05-22 17:54:33 UTC - Devin G. Bost: Thanks!
----
2019-05-22 19:06:12 UTC - Sree Vaddi: Please join our meetup group:
<https://www.meetup.com/SF-Bay-Area-Apache-Pulsar-Meetup>
+1 : Sijie Guo, lan
----
2019-05-22 21:24:45 UTC - Matteo Merli: Can you share the complete stack trace?
----
2019-05-22 21:25:11 UTC - Matteo Merli: Or any way to reproduce this?
----
2019-05-22 21:25:40 UTC - Matteo Merli: Would that be just having a project with pulsar-client and jackson-databind together?
----
2019-05-23 05:03:49 UTC - divyasree: @divyasree has joined the channel
----
2019-05-23 06:52:01 UTC - Venugopal Doddi: this helped. Thanks Alot. @Sijie Guo
----
2019-05-23 06:54:46 UTC - Venugopal Doddi: One more question.. when the consumer is not opened on one data center with some error in the consumer command and we didnt checked that as assumed the consumer is good, where as the producer successfully produced 10 messages on the other data center., Later realised the issue in the consumer command and corrected,. again produced 10 messages on the other data center. this time i consumed 10 messages successfully, but how will i know the previous message where are they? and why they are not consumed.
----
2019-05-23 07:10:34 UTC - Romain Castagnet: I just created : <https://github.com/apache/pulsar/issues/4337>
----
2019-05-23 07:15:29 UTC - Sijie Guo: @Venugopal Doddi if the subscription doesn’t exist, the subscription will be created with MessageId.latest.
----
2019-05-23 07:16:04 UTC - Sijie Guo: you can create a consumer using `withSubscriptionInitialPosition(SubscriptionInitialPosition.earliest)`.
----
2019-05-23 07:44:51 UTC - Venugopal Doddi: that a conf setting in some file?
----
2019-05-23 07:45:04 UTC - Venugopal Doddi: or the command parameter?
----
2019-05-23 07:47:18 UTC - Sijie Guo: this is on the consumer builder. I am not sure if the `public-client` command line tool has the option to control it.
----
2019-05-23 08:00:28 UTC - Venugopal Doddi: ok
----
2019-05-23 08:41:07 UTC - Lewey: @Lewey has joined the channel
----
2019-05-23 08:53:56 UTC - Ulf Lilleengen: @Ulf Lilleengen has joined the channel
----