You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@pulsar.apache.org by Apache Pulsar Slack <ap...@gmail.com> on 2018/07/21 09:11:04 UTC
Slack digest for #general - 2018-07-21
2018-07-20 13:54:42 UTC - Manasi: @Manasi has joined the channel
----
2018-07-20 18:03:16 UTC - shihao zhong: @shihao zhong has joined the channel
----
2018-07-20 18:38:55 UTC - shihao zhong: <!here> Hi guys, I got a invalidTopicName/Connection error in python pulsar-client 2.0.1. The document said we need to use topic like {persistent|non-persistent}://tenant/namespace/topic. But look like the python-client force us to use topic like {persistent|non-persistent}://tenant/namespace/something/topic
----
2018-07-20 18:39:12 UTC - shihao zhong: It can only work with four argument's topic
----
2018-07-20 18:39:17 UTC - shihao zhong: Can anyone help me with that?
----
2018-07-20 18:39:27 UTC - Grant Wu: can you link to the specific docs?
----
2018-07-20 18:39:33 UTC - shihao zhong: yes
----
2018-07-20 18:39:34 UTC - Grant Wu: There are a lot of outdated parts in the docs
----
2018-07-20 18:39:38 UTC - shihao zhong: <https://pulsar.incubator.apache.org/docs/latest/getting-started/ConceptsAndArchitecture/>
----
2018-07-20 18:39:40 UTC - Grant Wu: I’ve been submitting PRs to fix
----
2018-07-20 18:39:44 UTC - shihao zhong: in Topics
----
2018-07-20 18:40:04 UTC - shihao zhong: Did that PR already merge into the python client?
----
2018-07-20 18:40:10 UTC - shihao zhong: My pulsar-client version is 2.0.1
----
2018-07-20 18:40:30 UTC - Grant Wu: sorry no I don’t think my PRs are in any releases yet
----
2018-07-20 18:40:36 UTC - shihao zhong: oh ok
----
2018-07-20 18:40:40 UTC - shihao zhong: It's there anyway to fix it?
----
2018-07-20 18:40:53 UTC - Grant Wu: is this for consumer API?
----
2018-07-20 18:41:01 UTC - shihao zhong: yes for subscribe
----
2018-07-20 18:41:33 UTC - Grant Wu: Hrm
----
2018-07-20 18:41:54 UTC - Grant Wu: I’m a bit confused - I don’t think you explicitly specify the URL?
----
2018-07-20 18:42:16 UTC - Grant Wu: <https://pulsar.incubator.apache.org/docs/latest/getting-started/Pulsar-2.0/> might help
----
2018-07-20 18:42:33 UTC - Grant Wu: it should be `tenant/namespace/topic` for sure
----
2018-07-20 18:42:38 UTC - shihao zhong: No. I mean when i use topic like `tenant/namespace/topic`
----
2018-07-20 18:42:46 UTC - shihao zhong: and I use python client to subscribe that topic
----
2018-07-20 18:42:56 UTC - shihao zhong: I got invalidtopic or connection error exception
----
2018-07-20 18:42:59 UTC - Matteo Merli: @shihao zhong which version of python client lib you have installed?
----
2018-07-20 18:43:06 UTC - shihao zhong: 2.0.1
----
2018-07-20 18:43:34 UTC - shihao zhong: If I add one argument in that topic to make it like tenant/namespace/extra/topic
----
2018-07-20 18:43:37 UTC - shihao zhong: it will work
----
2018-07-20 18:44:34 UTC - shihao zhong: I have to add a argument in the topic to make it work.
----
2018-07-20 18:45:16 UTC - Matteo Merli: Uhm, that should work in 2.0.x — let me triple-check-it
----
2018-07-20 18:45:22 UTC - shihao zhong: thanks
----
2018-07-20 18:46:00 UTC - shihao zhong: @shihao zhong uploaded a file: <https://apache-pulsar.slack.com/files/UBU3V7ACU/FBU4NJ2EQ/-.py|Untitled>
----
2018-07-20 18:46:07 UTC - shihao zhong: Here is the stderr
----
2018-07-20 18:46:13 UTC - Grant Wu: could you post some sample code?
----
2018-07-20 18:46:56 UTC - shihao zhong: ehh let me try
----
2018-07-20 18:48:38 UTC - Grant Wu: oh no please don’t DM it to me c.c
----
2018-07-20 18:49:13 UTC - Grant Wu: Just, uh, something illustrating how you are using the library I guess. Actually, since I don’t know much about the Python client maybe you should just wait for @Matteo Merli
----
2018-07-20 18:49:39 UTC - shihao zhong: pk
----
2018-07-20 18:49:47 UTC - shihao zhong: Sure thanks
----
2018-07-20 18:52:33 UTC - Grant Wu: @shihao zhong are you sure the pulsar deployment is using 2.0 or higher?
----
2018-07-20 18:52:36 UTC - Grant Wu: Just a random thought
----
2018-07-20 18:54:01 UTC - Matteo Merli: @shihao zhong I just tried and it works fine here — I have a 2.0.1 python client connected to a topic with new name:
```
bin/pulsar-admin topics stats my-topic
{
"msgRateIn" : 0.0,
"msgThroughputIn" : 0.0,
"msgRateOut" : 0.0,
"msgThroughputOut" : 0.0,
"averageMsgSize" : 0.0,
"storageSize" : 0,
"publishers" : [ {
"msgRateIn" : 0.0,
"msgThroughputIn" : 0.0,
"averageMsgSize" : 0.0,
"producerId" : 0,
"metadata" : { },
"address" : "/127.0.0.1:56423",
"producerName" : "standalone-0-2",
"connectedSince" : "2018-07-20T11:50:41.573-07:00",
"clientVersion" : "2.0.1-incubating"
} ],
"subscriptions" : { },
"replication" : { },
"deduplicationStatus" : "Disabled"
}
```
In here, `my-topic` is the shortcut for `<persistent://public/default/my-topic>`
----
2018-07-20 18:55:26 UTC - Matteo Merli: Can you double-check the pulsar-client version installed by `pip`? If it had picked up a 1.x version that would not work indeed
----
2018-07-20 18:59:05 UTC - shihao zhong: Name: pulsar-client
Version: 2.0.1
Summary: Apache Pulsar Python client library
Home-page: <http://pulsar.incubator.apache.org/>
Author: Pulsar Devs
Author-email: <ma...@pulsar.incubator.apache.org>
License: Apache License v2.0
Location: /usr/local/lib/python2.7/site-packages
Requires:
----
2018-07-20 18:59:13 UTC - shihao zhong: That's what I got from pip
----
2018-07-20 18:59:39 UTC - shihao zhong: pulsar deployment? do you mean pulsar-server?
----
2018-07-20 18:59:43 UTC - Grant Wu: yeah
----
2018-07-20 19:00:08 UTC - shihao zhong: I am not sure
----
2018-07-20 19:00:13 UTC - shihao zhong: let me double check
----
2018-07-20 19:01:32 UTC - shihao zhong: The server version is also 2.0.1
----
2018-07-20 19:02:04 UTC - shihao zhong: That's what my topic looks like '<persistent://passport/passport-core-f-topic-test-stage/passport.notify.product-created>
----
2018-07-20 19:02:06 UTC - Matteo Merli: There has to be something else, because I just cleared everything, reinstalled and work here :confused:
----
2018-07-20 19:02:25 UTC - Matteo Merli: yes, the name should be valid
----
2018-07-20 19:02:32 UTC - shihao zhong: but it will not work
----
2018-07-20 19:03:03 UTC - shihao zhong: I have to use '<persistent://passport/passport-core/f-topic-test-stage/passport.notify.product-created>
----
2018-07-20 19:03:04 UTC - shihao zhong: to make it work
----
2018-07-20 19:03:11 UTC - shihao zhong: That's the only thing I changed
----
2018-07-20 19:03:20 UTC - shihao zhong: I am using python2.7
----
2018-07-20 19:03:25 UTC - Matteo Merli: macos?
----
2018-07-20 19:03:41 UTC - shihao zhong: yes
----
2018-07-20 19:04:01 UTC - shihao zhong: I use pip to install pulsar-client
----
2018-07-20 19:05:05 UTC - Matteo Merli: can you do `pyhon -v `, to verify it’s loading the modules from the expected location?
----
2018-07-20 19:06:11 UTC - shihao zhong: I think so
----
2018-07-20 19:06:30 UTC - shihao zhong: Because I use jenkins PIPELINE, it also does not work in pipeline
----
2018-07-20 19:07:48 UTC - shihao zhong: we use docker command like "RUN pip install pulsar-client" to install pulsar-client into the container
----
2018-07-20 19:08:34 UTC - Matteo Merli: Ok, just looking at the snippet above, the error seems to be coming from server side — not client side
----
2018-07-20 19:09:35 UTC - Matteo Merli: Then, my suspicion is that broker version is 1.xx
----
2018-07-20 19:11:52 UTC - shihao zhong: I just check with other group they said they are using 2.0.1 as a server
----
2018-07-20 19:12:32 UTC - Matteo Merli: would you be able to get a tcpdump on port 6650 in client?
----
2018-07-20 19:13:38 UTC - shihao zhong: I do not think so
----
2018-07-20 19:15:54 UTC - Matteo Merli: Ok, now I see the problem
----
2018-07-20 19:16:07 UTC - Matteo Merli: the namespace you’re trying to use does not exist
----
2018-07-20 19:16:23 UTC - Matteo Merli: `passport/passport-core-f-topic-test-stage`
----
2018-07-20 19:16:25 UTC - shihao zhong: But I also changed the namespace in our java code
----
2018-07-20 19:16:54 UTC - shihao zhong: which publish the event
----
2018-07-20 19:17:03 UTC - shihao zhong: let me double check with that
----
2018-07-20 19:18:45 UTC - Matteo Merli: can you try `pulsar-admin namespaces policies passport/passport-core-f-topic-test-stage`
----
2018-07-20 19:19:21 UTC - shihao zhong: I am not able to use any tool on server side
----
2018-07-20 19:20:53 UTC - Matteo Merli: ok, just check the particular namespace was created
----
2018-07-20 19:21:43 UTC - shihao zhong: sure
----
2018-07-20 19:21:47 UTC - shihao zhong: I am retrying
----
2018-07-20 19:26:03 UTC - shihao zhong: I changed the topic and rebuild
----
2018-07-20 19:26:11 UTC - shihao zhong: I make sure the publish topic is the same as the subscribe
----
2018-07-20 19:26:16 UTC - shihao zhong: and I still get error like
----
2018-07-20 19:26:50 UTC - shihao zhong: '<persistent://passport/passport-core=f-topic-test-stage/passport.notify.user-created>'
----
2018-07-20 19:27:13 UTC - shihao zhong: this is the subscribe topic
----
2018-07-20 19:27:18 UTC - shihao zhong: '<persistent://passport/passport-core=f-topic-test-stage/passport.notify.user-created>'
----
2018-07-20 19:27:23 UTC - shihao zhong: This is the publish topic
----
2018-07-20 19:27:56 UTC - shihao zhong: and if I change the "=" to "/", everything will be fine
----
2018-07-20 19:38:31 UTC - Grant Wu: I don’t think that’s what merlimat was asking…
----
2018-07-20 19:39:09 UTC - Grant Wu: Is there any way to control how the `-m` flag parses messages in `pulsar-client produce`?
----
2018-07-20 19:39:37 UTC - Grant Wu: It’s rather annoying to work with when you’re putting in JSON blobs, as those internally contain commas
----
2018-07-20 19:40:48 UTC - shihao zhong: I am not quiet sure I got what you talking about for JSON blob
----
2018-07-20 19:40:54 UTC - shihao zhong: Can you please explain it?
----
2018-07-20 19:40:55 UTC - Grant Wu: It has nothing to do with you
----
2018-07-20 19:40:59 UTC - Grant Wu: I have my own question as well
----
2018-07-20 19:41:03 UTC - shihao zhong: oh
----
2018-07-20 19:41:03 UTC - shihao zhong: ok
----
2018-07-20 19:41:10 UTC - shihao zhong: Sorry for that
----
2018-07-20 19:41:12 UTC - Grant Wu: no problem
----
2018-07-20 19:41:42 UTC - Grant Wu: what happens if you do `pulsar-admin namespaces policies passport/passport-core-f-topic-test-stage`
----
2018-07-20 19:41:50 UTC - Grant Wu: @shihao zhong this doesn’t need to be run _on_ the server
----
2018-07-20 19:43:36 UTC - Grant Wu: We should probably start using Slack threads :confused:
----
2018-07-20 19:43:51 UTC - Matteo Merli: if you don’t have client installed, you can do:
`curl http://$SERVICE_URL:8080/admin/v2/namespaces/passport/passport-core-f-topic-test-stage`
----
2018-07-20 19:46:22 UTC - Grant Wu: I can hack around it using `<()`, but…
----
2018-07-20 19:46:23 UTC - shihao zhong: I tried curl
----
2018-07-20 19:46:24 UTC - shihao zhong: I got {"reason":"Tenant does not exist"}
----
2018-07-20 19:46:43 UTC - shihao zhong: Look like the tenant was not been created?
----
2018-07-20 19:48:46 UTC - Matteo Merli: correct
----
2018-07-20 19:52:55 UTC - shihao zhong: But I do not understand why
----
2018-07-20 19:53:11 UTC - shihao zhong: Do I need to specific create any tenant in the publisher?
----
2018-07-20 19:53:39 UTC - Matteo Merli: ```
bin/pulsar-admin tenants create my-tenant
bin/pulsar-admin namespaces create my-tenant/my-namespace -c my-cluster
```
----
2018-07-20 19:54:11 UTC - Grant Wu: aiui, yes, you must explicitly create tenants and namespaces
----
2018-07-20 19:54:29 UTC - Grant Wu: the only things you can create on demand are subscriptions and topics
----
2018-07-20 19:55:59 UTC - shihao zhong: oh, let me rebuild it again
----
2018-07-20 20:05:40 UTC - shihao zhong: Ok I just checked with other team, there does not have a tenant named passport
----
2018-07-20 20:06:04 UTC - shihao zhong: but may I ask why 4 arguments will work?
----
2018-07-20 20:59:55 UTC - Matteo Merli: > but may I ask why 4 arguments will work?
That’s more to retain the previous behavior. When authorization is not enabled, so any user has access to all topics anyway. (Were authorization been enabled, both would have been rejected).
The behavior in Pulsar 1.x was to allow to use any topic when auth is disabled. That had its own issue caused by having topics for which the metadata was not really created, and it was more confusing than helpful.
----
2018-07-20 21:01:03 UTC - Matteo Merli: That’s why we moved to the explicit behavior of forcing the creation of tenants and namespaces. However, to avoid breaking anything, for 1.x topics, it still works in the same way as before.
----
2018-07-20 21:03:05 UTC - Grant Wu: oh, so the explicit tenant/namespace creation is a 2.x thing?
----
2018-07-20 21:04:21 UTC - Matteo Merli: yes. I mean, with auth enabled you always had to create them anyway and give permissions
----
2018-07-20 21:07:13 UTC - Grant Wu: ah, okay
----
2018-07-20 21:09:25 UTC - Matteo Merli: @Grant Wu you could use `-f` and put the json in file
----