You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@pulsar.apache.org by Apache Pulsar Slack <ap...@gmail.com> on 2019/12/25 09:11:03 UTC
Slack digest for #general - 2019-12-25
2019-12-24 12:52:50 UTC - Himanshu Singh: Hi,
I am running standalone pulsar with authentication and authorization using tokens. When I create tenant, namespace and topics in 'standalone' cluster (which is availale by default) everything works smoothly. But when I create another cluster (say 'us-west-2') and associate the tenant and namespace to just that ('us-west-2') cluster. I get the following error when creating the topic :-
`[AsyncHttpClient-5-1] WARN org.apache.pulsar.client.admin.internal.BaseResource - [<https://localhost:8443/admin/v2/persistent/production-us-west-2/order/classification-topic>] Failed to perform http put request: javax.ws.rs.ClientErrorException: HTTP 412 Precondition Failed`
`Namespace does not have any clusters configured`
`Reason: Namespace does not have any clusters configured`
Here is the policy for the namespace `production-us-west-2/order`. It shows `us-west-2` in replication-clusters.
```root@70bca0203e01:/pulsar# bin/pulsar-admin namespaces policies production-us-west-2/order
{
"auth_policies" : {
"namespace_auth" : {
"prod" : [ "produce", "consume" ]
},
"destination_auth" : { },
"subscription_auth_roles" : {
"main-api-group" : [ "prod" ]
}
},
"replication_clusters" : [ "us-west-2" ],
"bundles" : {
"boundaries" : [ "0x00000000", "0x40000000", "0x80000000", "0xc0000000", "0xffffffff" ],
"numBundles" : 4
},
"backlog_quota_map" : {
"destination_storage" : {
"limit" : 10737418240,
"policy" : "producer_request_hold"
}
},
"topicDispatchRate" : {
"standalone" : {
"dispatchThrottlingRateInMsg" : 0,
"dispatchThrottlingRateInByte" : 0,
"ratePeriodInSecond" : 1
}
},
"subscriptionDispatchRate" : {
"standalone" : {
"dispatchThrottlingRateInMsg" : 0,
"dispatchThrottlingRateInByte" : 0,
"ratePeriodInSecond" : 1
}
},
"replicatorDispatchRate" : { },
"clusterSubscribeRate" : {
"standalone" : {
"subscribeThrottlingRatePerConsumer" : 0,
"ratePeriodInSecond" : 30
}
},
"latency_stats_sample_rate" : { },
"message_ttl_in_seconds" : 0,
"deleted" : false,
"encryption_required" : false,
"subscription_auth_mode" : "None",
"max_producers_per_topic" : 0,
"max_consumers_per_topic" : 0,
"max_consumers_per_subscription" : 0,
"compaction_threshold" : 0,
"offload_threshold" : -1,
"schema_auto_update_compatibility_strategy" : "Full",
"schema_validation_enforced" : false
}```
Facing similar issue when deploying pulsar on kubernetes with latest helm chart. Here, a cluster named 'pulsar' is created by default and topic creation commands work like a charm when tenant & namespace are in default 'pulsar' cluster. But if I try to create another cluster, I face the exact same error.
```[AsyncHttpClient-5-1] WARN org.apache.pulsar.client.admin.internal.BaseResource - [<http://pulsar-broker:8080/admin/v2/persistent/production-us-west-2/order/classification-topic>] Failed to perform http put request: javax.ws.rs.ClientErrorException: HTTP 412 Precondition Failed
Namespace does not have any clusters configured
Reason: Namespace does not have any clusters configured```
I am clearly missing out on something. Could someone please help me here. Thanks.
----
2019-12-24 13:07:26 UTC - Sijie Guo: 404 means there is no schema associated with the topic. If you are sending `bytes` , it will not create any schema.
----
2019-12-24 13:10:08 UTC - Sijie Guo: oh don’t use “/” in a namespace name. I think it confused Pulsar. @Himanshu Singh
----
2019-12-24 13:11:26 UTC - Roman Popenov: Yeah, that’s exactly it. The data type that is stored is a byte string which is stored as bytes array. I was expecting the schema to say bytes[]
----
2019-12-24 13:36:21 UTC - Himanshu Singh: @Sijie Guo creating a namespace without "/" throws error. Everywhere in docs, the structure followed is tenant/namespace and that is what I have used.
----
2019-12-24 13:48:33 UTC - Himanshu Singh: If there's any other way, could you mention that ?
----
2019-12-24 13:48:46 UTC - Roman Popenov: How are you creating your namespace?
----
2019-12-24 13:49:30 UTC - Roman Popenov: `pulsar-admin namespaces create tenant/namespace`
----
2019-12-24 13:51:02 UTC - Sijie Guo: @Himanshu Singh “the namespace `production-us-west-2/order` ” ? is ``production-us-west-2` a tenant name or a cluster name? I might misread your question
----
2019-12-24 13:57:34 UTC - Himanshu Singh: @Roman Popenov Yes, here is the command I've used - `bin/pulsar-admin namespaces create production-us-west-2/order --clusters us-west-2`
----
2019-12-24 13:58:58 UTC - Himanshu Singh: @Sijie Guo, @Roman Popenov tenant is `production-us-west-2` , namespace is `order` and cluster name is `us-west-2` .
----
2019-12-24 14:00:15 UTC - Yong Zhang: Can you get your cluster by `bin/pulsar-admin clusters list ` ?
----
2019-12-24 14:05:55 UTC - Himanshu Singh: yes, I can. I get two clusters -> standalone, us-west-2
----
2019-12-24 14:12:12 UTC - Himanshu Singh: I get error when creating a topic (not namespace) using following command `bin/pulsar-admin topics create <persistent://production-us-west-2/order/classification-topic>`
----
2019-12-24 14:12:33 UTC - Himanshu Singh: I get error when creating a topic (not namespace) using following command `bin/pulsar-admin topics create <persistent://production-us-west-2/order/classification-topic>`
----
2019-12-24 14:16:10 UTC - Roman Popenov: What’s the error?
----
2019-12-24 14:17:02 UTC - Himanshu Singh: ``[AsyncHttpClient-5-1] WARN org.apache.pulsar.client.admin.internal.BaseResource - [<https://localhost:8443/admin/v2/persistent/production-us-west-2/order/classification-topic>] Failed to perform http put request: javax.ws.rs.ClientErrorException: HTTP 412 Precondition Failed`
`Namespace does not have any clusters configured`
`Reason: Namespace does not have any clusters configured``
----
2019-12-24 14:31:59 UTC - Yong Zhang: The policy shows that us-west is your replication cluster. Can you try to create a namespace and specify the cluster name as standalone?
How did you create the us-west cluster?
----
2019-12-24 14:39:07 UTC - Sijie Guo: @Himanshu Singh which cluster did you issue `topics create` command?
----
2019-12-24 14:45:57 UTC - Himanshu Singh: @Yong Zhang,
`pulsar-admin clusters create us-west-2 --url <http://0.0.0.0:8080> --broker-url <pulsar://0.0.0.0:6650> --url-secure <https://0.0.0.0:8443> --broker-url-secure <pulsar+ssl://0.0.0.0:6651>`
It works when I create a namespace in standalone cluster.
----
2019-12-24 14:47:14 UTC - Himanshu Singh: @Sijie Guo, Can we specify cluster to `topics create` command. Couldn't find it in the docs.
----
2019-12-24 14:52:24 UTC - Sijie Guo: I mean which cluster you run the command
----
2019-12-24 14:53:08 UTC - Sijie Guo: if possible, can you write down the sequence of you commands?
----
2019-12-24 15:09:24 UTC - Himanshu Singh: I am running all these commands from same instance.
----
2019-12-24 15:14:43 UTC - Himanshu Singh: @Sijie Guo, are we supposed to run these commands from different clusters ?
----
2019-12-24 16:52:32 UTC - Mathieu Druart: Hi ! I try to remove a shared subscription associated with a function, but I can't find how to do it ? Even if I stop or delete the function, when I try to unsubscribe with `pulsar-admin topics unsubscribe topic-name -s sub-name` I still have this message : `orm http delete request: javax.ws.rs.ClientErrorException: HTTP 412 Precondition Failed Subscription has active connected consumers` I even tried to delete the topic with a `--force`but the topic remains with this last subscription. Am I missing something ? Thanks !
----
2019-12-24 17:03:41 UTC - Mathieu Druart: The fact that deleting the function doesnt remove the subscription is also strange ?
----
2019-12-24 20:55:25 UTC - zuquan song: @zuquan song has joined the channel
----
2019-12-24 20:58:59 UTC - Roman Popenov: I wrote my own S3 sink using Pulsar Functions. I believe it is also possible to write a sink using Pulsar I/Os. I will let you know what I find out.
----
2019-12-25 01:46:27 UTC - Sijie Guo: The initial thought is that people can re-create the function with a newer version of code and resume processing the events left from the previous run.
We can add a flag to delete subscription when deleting a function. Can you create a github issue for us?
----
2019-12-25 01:46:54 UTC - Sijie Guo: You can use “topic stats” to see if there are still connected consumers.
----
2019-12-25 03:36:28 UTC - Sijie Guo: @yijie @jia zhai: this is the original slack thread about this issue.
+1 : jia zhai, yijie
----
2019-12-25 03:41:52 UTC - yijie: Hi @Viji, since the eventTime is extracted manually as well as watermark, trigger and window functions, I think it’s not Pulsar related but how to build your flink program. Does your project work as expected when using kafka source?
----
2019-12-25 03:42:58 UTC - yijie: I saw you were logging in the project, could you please provide the log file for more information?
----