You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@pulsar.apache.org by Apache Pulsar Slack <ap...@gmail.com> on 2019/07/12 09:11:04 UTC
Slack digest for #general - 2019-07-12
2019-07-11 11:54:39 UTC - Venugopal Doddi: np... @Sijie Guo
----
2019-07-11 13:00:22 UTC - Enrico Olivelli: Hello, I am trying to upgrade from 2.3.0 to 2.4.0 but I get this error "Policies not found for idc2/ns namespace"
----
2019-07-11 13:01:26 UTC - Enrico Olivelli: I am creating the namespace with the Java API: admin.namespaces().createNamespace(namespace, new HashSet<>(Arrays.asList(CLUSTER_NAME))); and it works in 2.3.0
I found also that in 2.4.0 I have to explicitly "create" the "cluster"
----
2019-07-11 13:02:53 UTC - Enrico Olivelli: @jia zhai do you have any idea ? I did not find notes in <https://pulsar.apache.org/release-notes/#2.4.0> about these changes
----
2019-07-11 13:04:56 UTC - Enrico Olivelli: this is the doc of the REST API <https://pulsar.apache.org/admin-rest-api/#operation/createNamespace>
----
2019-07-11 13:15:29 UTC - Devin G. Bost: @Enrico Olivelli Did you create the tenant? Typically, that error appears when something didn't go right with creating the namespace. The namespace policies won't exist if the namespace wasn't actually created successfully.
----
2019-07-11 13:16:52 UTC - Enrico Olivelli: in 2.3.0 I am doing:
TenantInfo tenantInfo = new TenantInfo();
tenantInfo.setAllowedClusters(new HashSet<>(Arrays.asList(CLUSTER_NAME)));
admin.tenants().createTenant(tenantId, tenantInfo);
admin.namespaces().createNamespace(namespace, new HashSet<>(Arrays.asList(CLUSTER_NAME)));
and the namespace is: tenantid+"/ns"
----
2019-07-11 13:17:34 UTC - Enrico Olivelli: "createTenant" returns with success and also createNamespace
----
2019-07-11 13:17:58 UTC - Enrico Olivelli: which is the good way in 2.4.0 ? (thank you @Devin G. Bost for taking a look)
----
2019-07-11 13:28:40 UTC - Devin G. Bost: Was your code working in 2.3.0? I haven't tried running mine in 2.4.0 yet.
----
2019-07-11 13:29:14 UTC - Devin G. Bost: Also, I'd recommend that you try getting the namespace with the name you're providing (e.g. in a unit test) to ensure that it's coming back as expected.
----
2019-07-11 13:29:51 UTC - Devin G. Bost: In your test, I'd recommend providing the name as an explicit string just in case the name is drifting somehow.
----
2019-07-11 13:31:00 UTC - Devin G. Bost: You can check for an existing tenant like this:
```
@Override
public boolean existsInPulsar() {
PulsarAdmin pulsarAdmin = this.getPulsarAdmin();
List<String> existingTenants = null;
try {
existingTenants = pulsarAdmin.tenants().getTenants();
} catch (PulsarAdminException e) {
e.printStackTrace();
}
List<String> tenants = existingTenants;
String name = this.getName();
return existingTenants.contains(this.getName());
}
```
You can check for an existing namespace like this:
```
@Override
public boolean existsInPulsar() {
PulsarAdmin pulsarAdmin = this.getPulsarAdmin();
List<String> namespaces = null;
try {
namespaces = pulsarAdmin.namespaces().getNamespaces(this.getTenant());
} catch (PulsarAdminException e) {
e.printStackTrace();
}
return namespaces.contains(this.getFullyQualifiedName());
}
```
----
2019-07-11 13:32:32 UTC - Devin G. Bost: Creating your tenant looks like this:
```
try {
Tenant tenantObj = new Tenant();
tenantObj.setName(tenant);
tenantObj.setPulsarServiceURL(pulsarAdmin.getServiceUrl());
if (!tenantObj.existsInPulsar()){
pulsarAdmin.tenants().createTenant(tenant, tenantObj.getDefaultTenantInfo());
}
} catch (PulsarAdminException e) {
e.printStackTrace();
}
```
In my code, my `.getDefaultTenantInfo()` looks like this:
```
public TenantInfo getDefaultTenantInfo(){
fastDeployConfig = StaticUtils.getFastDeployConfig();
Set<String> adminRoleSet = fastDeployConfig.getFastdeployAuthzAdminRoleSetClean();
Set<String> clusterSet = fastDeployConfig.getFastdeployAllowedClusterSetClean();
TenantInfo tenantInfo = new TenantInfo(adminRoleSet, clusterSet);
return tenantInfo;
}
```
----
2019-07-11 13:33:16 UTC - Devin G. Bost: Then, creating a namespace looks like this:
```
try {
System.out.println("Creating namespace: " + this.getFullyQualifiedName());
pulsarAdmin.namespaces().createNamespace(this.getFullyQualifiedName());
} catch (PulsarAdminException e) {
e.printStackTrace();
}
```
----
2019-07-11 13:33:27 UTC - Devin G. Bost: @Enrico Olivelli Please let me know if this helps.
----
2019-07-11 13:42:03 UTC - Penghui Li: @Enrico Olivelli I have test it on my mac:
```
PulsarAdmin admin = PulsarAdmin.builder().serviceHttpUrl("<http://localhost:8080>").build();
TenantInfo tenantInfo = new TenantInfo();
tenantInfo.setAllowedClusters(Sets.newHashSet("standalone"));
admin.tenants().createTenant("my-tenant1", tenantInfo);
admin.namespaces().createNamespace("my-tenant1/my-namespace1");
System.out.println(admin.namespaces().getPolicies("my-tenant1/my-namespace1"));
output:
Policies{auth_policies=org.apache.pulsar.common.policies.data.AuthPolicies@745f, replication_clusters=[standalone], bundles=BundlesData{numBundles=4, boundaries=[0x00000000, 0x40000000, 0x80000000, 0xc0000000, 0xffffffff]}, backlog_quota_map={destination_storage=BacklogQuota{limit=10737418240, policy=producer_request_hold}}, persistence=null, deduplicationEnabled=null, topicDispatchRate={standalone=DispatchRate{dispatchThrottlingRateInMsg=0, dispatchThrottlingRateInByte=0, ratePeriodInSecond=1}}, subscriptionDispatchRate={standalone=DispatchRate{dispatchThrottlingRateInMsg=0, dispatchThrottlingRateInByte=0, ratePeriodInSecond=1}}, replicatorDispatchRate={}, clusterSubscribeRate={standalone=SubscribeRate{subscribeThrottlingRatePerConsumer=0, ratePeriodInSecond=30}}, latency_stats_sample_rate={}, antiAffinityGroup=null, message_ttl_in_seconds=0, retention_policies=null, deleted=false, encryption_required=false, subscription_auth_mode=None, max_producers_per_topic=0, max_consumers_per_topic=0, max_consumers_per_subscription=0, compaction_threshold=0, offload_threshold=-1, offload_deletion_lag_ms=null, schema_auto_update_compatibility_strategy=Full, schema_validation_enforced=false}
```
----
2019-07-11 14:42:22 UTC - Tim Farber-Newman: @Tim Farber-Newman has joined the channel
----
2019-07-11 14:42:58 UTC - Enrico Olivelli: Thank you so much @Penghui Li and @Devin G. Bost
I have reproduced the correct creating in unit tests and all works as expected.
and then I found my problem. When Pulsar returns this error "Policies not found for idc2/ns namespace" it is saying "the name space does not exist" . I had a typo in my client code and actually the code that was creating the namespace wasn't called and more. :sweat_smile:
+1 : David Kjerrumgaard, Matteo Merli
----
2019-07-11 14:43:16 UTC - Enrico Olivelli: unit tests were actually working but integration tests not
----
2019-07-11 14:43:37 UTC - Devin G. Bost: :slightly_smiling_face:
----
2019-07-11 16:05:47 UTC - Ryan Samo: Hey guys,
I have a Pulsar cluster stood up with functions workers outside of the brokers as their own cluster. When we create functions, they work fine, no issues. If we create a sink, it creates but then never subscribes the source topic. Also, if you try to delete the sink you get a 401 unauthorized exception. Any idea why functions would work fine but sinks do not? We can create, update, get, list, you name it. Just can’t delete or get the sink to subscribe
----
2019-07-11 16:07:23 UTC - Ryan Samo: The sink logs show it picks up the nar file, never a subscription or anything though
----
2019-07-11 16:08:39 UTC - Ryan Samo: Are sinks even allowed to run on function workers outside the brokers?
----
2019-07-11 16:09:11 UTC - Jerry Peng: Yes
+1 : Ryan Samo
----
2019-07-11 16:09:17 UTC - Ryan Samo: TLS enabled cluster, using pulsar proxy as well
----
2019-07-11 16:10:41 UTC - Jerry Peng: Is authorization turned on as well?
----
2019-07-11 16:11:14 UTC - Ryan Samo: In the functions_worker.yml or?
----
2019-07-11 16:11:51 UTC - Jerry Peng: Yes
----
2019-07-11 16:12:24 UTC - Ryan Samo: authorizationEnabled: true
----
2019-07-11 16:13:41 UTC - Jerry Peng: Try sending the delete command to different function workers to see if you get the same result
----
2019-07-11 16:14:05 UTC - Jerry Peng: If you are able to create or update you should be able to delete
----
2019-07-11 16:15:15 UTC - Ryan Samo: So I can create, update, and delete functions just fine. I can create and update sinks but cannot delete them
----
2019-07-11 16:15:35 UTC - Ryan Samo: Tried multiple times on multiple different sinks
----
2019-07-11 16:15:58 UTC - Ryan Samo: Using the provided sinks, not custom
----
2019-07-11 16:17:06 UTC - Ryan Samo: Getting the error, one second
----
2019-07-11 16:25:39 UTC - Ryan Samo: Here you go
----
2019-07-11 16:26:09 UTC - Ryan Samo: And the cli times out with a null response
----
2019-07-11 16:31:15 UTC - Jerry Peng: did you also configure:
```
# the authentication plugin to be used by the pulsar client used in worker service
clientAuthenticationPlugin:
# the authentication parameter to be used by the pulsar client used in worker service clientAuthenticationParameters:
```
----
2019-07-11 16:33:28 UTC - Ryan Samo: # the authentication plugin to be used by the pulsar client used in worker service
clientAuthenticationPlugin: org.apache.pulsar.client.impl.auth.AuthenticationTls
# the authentication parameter to be used by the pulsar client used in worker service
clientAuthenticationParameters: tlsCertFile:/opt/pulsar/certificates/admin.cert.pem,tlsKeyFile:/opt/pulsar/certificates/admin.key-pk8.pem
----
2019-07-11 16:42:13 UTC - Jerry Peng: you made sure those keys are valid?
----
2019-07-11 16:42:50 UTC - Jerry Peng: Are those keys signed by a local CA?
----
2019-07-11 16:43:37 UTC - Ryan Samo: Yeah they are created from the same CA that created the broker certs etc.
----
2019-07-11 16:44:00 UTC - Ryan Samo: And they seem to work fine for other admin commands
----
2019-07-11 16:44:35 UTC - Ryan Samo: Is there a good way to validate them that you know of?
----
2019-07-11 16:47:34 UTC - Jerry Peng: On one of the nodes your are running the function worker, try using the pulsar-admin CLI with those keys to see if it works
----
2019-07-11 16:48:24 UTC - Fiodar Zboichyk: @Fiodar Zboichyk has joined the channel
----
2019-07-11 16:49:40 UTC - Jerry Peng: you also set superUserRoles in the broker.conf on your brokers to contain the subject CN of the certs you are using for your function worker?
----
2019-07-11 16:51:13 UTC - Ryan Samo: Admin CLI works when you run it on a function worker as long as you pass the —admin-url flag and target the functions workers url
----
2019-07-11 16:52:22 UTC - Ryan Samo: The broker config has the superusers set as “admin,proxy” and the CN for both the admin and proxy certs are a match to that
----
2019-07-11 16:54:39 UTC - Ryan Samo: Normal admin cli commands work as well like listing the topics etc. I only have to use the —admin-url flag when executing functions commands as the proxy does not forward them even though I have the urls set for that in the proxy.conf
----
2019-07-11 16:56:02 UTC - Ryan Samo: Otherwise I will get “Function worker service is not done initializing”
----
2019-07-11 17:16:08 UTC - Addison Higham: hrm... is there any API for doing a "raw" list of topics in a namespace that would include both partitioned topics (including the "internal" topic name) as well as non-partitioned topics?
----
2019-07-11 17:18:14 UTC - Ryan Samo: $PULSAR_HOME/bin/pulsar-admin topics list public/default
----
2019-07-11 17:18:22 UTC - Ryan Samo: Gives both
----
2019-07-11 17:18:42 UTC - Jerry Peng: @Ryan Samo target the broker from the function worker node
----
2019-07-11 17:18:51 UTC - Jerry Peng: i.e. set --admin-url to a broker
----
2019-07-11 17:18:56 UTC - Jerry Peng: and use those certs
----
2019-07-11 17:19:14 UTC - Jerry Peng: to simulate function workers communicating with brokers
----
2019-07-11 17:19:29 UTC - Addison Higham: oh? hrm, do you actually have to write records first then? because it doesn't seem to be showing, but will try again
----
2019-07-11 17:19:56 UTC - Ryan Samo: @Jerry Peng will do
----
2019-07-11 17:21:38 UTC - Ryan Samo: @Addison Higham if you created the topic via producing to it or even created via the admin-cli it should show up there
----
2019-07-11 17:24:45 UTC - Ryan Samo: @Jerry Peng it works, I just ran a topics list from a functions worker to a broker targeting the broker
----
2019-07-11 17:28:29 UTC - Jerry Peng: @Ryan Samo can you try getting the stats of one of the topics to make sure that is working
----
2019-07-11 17:28:51 UTC - Jerry Peng: e.g. ./bin/pulsar-admin topics stats topic-name
----
2019-07-11 17:30:11 UTC - Ryan Samo: @Jerry Peng works as well
----
2019-07-11 17:30:31 UTC - Jerry Peng: weird
----
2019-07-11 17:31:02 UTC - Ryan Samo: Yeah it’s strange for sure, I’m puzzled
----
2019-07-11 17:31:13 UTC - Ryan Samo: v2.3.2
----
2019-07-11 17:35:10 UTC - Jerry Peng: is your CA a public one? if not you also need to specify tlsTrustCertsFilePath
----
2019-07-11 17:35:27 UTC - Jerry Peng: or one Java knows
----
2019-07-11 17:36:34 UTC - Ryan Samo: I have tlsTrustCertsFilePath set to our ca on all nodes
----
2019-07-11 17:38:33 UTC - Jerry Peng: For the exceptions you posted before, how often do they occur? On every node running function worker?
----
2019-07-11 17:40:33 UTC - Addison Higham: I am getting different behavior, at least with pulsar standalone:
```
$ pulsar-admin --admin-url <http://localhost:8080> topics create <persistent://public/default/my-single-topic>
$ pulsar-admin --admin-url <http://localhost:8080> topics create-partitioned-topic -p 5 <persistent://public/default/my-topic>
$ pulsar-admin --admin-url <http://localhost:8080> topics list public/default
<persistent://public/default/my-single-topic>
$ pulsar-admin --admin-url <http://localhost:8080> topics list-partitioned-topics public/default
<persistent://public/default/my-topic>
```
----
2019-07-11 17:40:44 UTC - Addison Higham: going to try it on a real cluster and see if I get something different
----
2019-07-11 17:40:52 UTC - Ryan Samo: @Jerry Peng let me try to run it parallel and see
----
2019-07-11 17:50:25 UTC - Addison Higham: onto something somewhat different... when doing an upgrade using the helm chart on k8s, it tries to change the `zookeeper-metadata` job, which is immutable. It doesn't look like the helm chart has a way to disable creation of that resource, so it seems like the easiest thing to do would be to delete the job, but that only seems safe is `initialize-cluster-metadata` is either idempotent or guards against anything
----
2019-07-11 18:02:45 UTC - Tarek Shaar: Hello quick question. Doing some testing on pulsar Java consumers. It seems that if you make several consumers listening on the same topic, then each message is always dispatch on the same thread. I thought each consumer will get its own thread. Am i missing something
----
2019-07-11 18:04:48 UTC - Tarek Shaar:
----
2019-07-11 18:08:14 UTC - Tarek Shaar: It is always Thread[pulsar-external-listener-3-1,5,main]. This is the case whether I used shared or exclusive subs
----
2019-07-11 18:08:35 UTC - Tarek Shaar:
----
2019-07-11 18:15:37 UTC - Ryan Samo: @Jerry Peng no it looks to only do that on 1 worker, depending on the DNS round robin
----
2019-07-11 18:25:13 UTC - Aaron Langford: @Addison Higham, on standalone, looks like you have to send messages first before those partitioned topics show up...
----
2019-07-11 18:26:19 UTC - Aaron Langford: Then they will show up when you list all topics in a namespace, but if you list partitioned topics, you will still only get the single logical topic.
----
2019-07-11 18:28:15 UTC - Aaron Langford: Partitioned topics are of the form:
```
"persistent://{tenant}/{namespace}/{topic}-partition-{partition-number}",
```
Where partition number starts at 0.
----
2019-07-11 18:33:06 UTC - Ryan Samo: @Jerry Peng do the Pulsar brokers know about the functions workers at all if they are running separately, or do the functions workers request from the brokers like clients?
----
2019-07-11 18:33:45 UTC - Aaron Langford: I'm not sure if this is the case for a full blown cluster, but I would speculate that it is so.
----
2019-07-11 18:33:50 UTC - Ryan Samo: Trying to figure out the difference between why Pulsar functions would work fine but Pulsar IO wouldn’t
----
2019-07-11 18:50:31 UTC - Jerry Peng: @Ryan Samo when running separately, function workers request from brokers like clients
----
2019-07-11 18:51:42 UTC - Ryan Samo: Ok so when creating a sink, it should reach out to the brokers and subscribe to the topic then correct? Like starting a consumer
----
2019-07-11 18:51:59 UTC - Jerry Peng: The sink instance will
----
2019-07-11 18:54:00 UTC - Ryan Samo: Ok and the sink will use whatever certs are in the client.conf or the functions_worker.yml?
----
2019-07-11 18:55:58 UTC - Jerry Peng: in functions_worker.yml
----
2019-07-11 18:56:39 UTC - Ryan Samo: :+1:
----
2019-07-11 18:59:47 UTC - Aaron: @Matteo Merli I get the same issue when testing the loop by itself - 13+ mb of data allocated from a 1 hour ackTimeout value.
----
2019-07-11 19:04:07 UTC - Matteo Merli: Do you have a heap dump? How many message ids are in the UnAckedMessageTracker?
----
2019-07-11 19:14:38 UTC - Addison Higham: I read a chunk of that code somewhat recently, it seemed to me that the way the client was constructed had branching code paths based on IO/functions, so it is quite possible it is just something a bit different there
----
2019-07-11 19:33:37 UTC - Aaron: None - this is in the initialization of the unackedmessagetracker
----
2019-07-11 19:36:32 UTC - Ryan Samo: @Addison Higham yeah something is weird for sure
----
2019-07-11 19:37:36 UTC - Jerry Peng: the pulsar admin client created within a function worker is the same between pulsar functions and sources and sinks
----
2019-07-11 19:39:29 UTC - Jerry Peng: There are two admin clients created in a function worker. One for communicating with other function workers via function endpoints and one for communicating with brokers
----
2019-07-11 19:40:03 UTC - Jerry Peng: Auth settings for the two are the same
----
2019-07-11 19:40:42 UTC - Jerry Peng: The exceptions @Ryan Samo posted earlier seem to indicate a problem with the function worker communicating with brokers
----
2019-07-11 19:42:30 UTC - Ryan Samo: @Jerry Peng I agree, it does seem to be an issue from functions worker to broker. What settings specifically are targeting that communication path? Would it help to attach my functions_worker.yml?
----
2019-07-11 19:42:53 UTC - Jerry Peng: sure
----
2019-07-11 19:43:03 UTC - Ryan Samo: Ok one sec
----
2019-07-11 19:43:23 UTC - Jerry Peng: usually these configs:
```
pulsarServiceUrl: <pulsar://localhost:6650>
pulsarWebServiceUrl: <http://localhost:8080>
# the authentication plugin to be used by the pulsar client used in worker service
clientAuthenticationPlugin:
# the authentication parameter to be used by the pulsar client used in worker service
clientAuthenticationParameters:
```
----
2019-07-11 19:46:29 UTC - Jerry Peng: are you using the same set of certs on both brokers and function workers?
----
2019-07-11 19:47:09 UTC - Ryan Samo:
----
2019-07-11 19:47:52 UTC - Ryan Samo: Certs on the brokers are broker.cert.pem
----
2019-07-11 19:48:28 UTC - Ryan Samo: I’m using the admin certs for the functions
----
2019-07-11 19:48:57 UTC - Ryan Samo: Which are the same that is used in the client.conf for the admin cli
----
2019-07-11 19:49:55 UTC - Ryan Samo: Maybe my client auth plugin is incorrect?
----
2019-07-11 19:51:44 UTC - Jerry Peng: most likely reason
----
2019-07-11 19:52:14 UTC - Jerry Peng: The subject CN for /opt/pulsar/certificates/admin.cert.pem is listed as a super user role in broker.conf on all brokers?
----
2019-07-11 19:52:44 UTC - Jerry Peng: all the certs on both functions workers and brokers are signed by the same CA?
----
2019-07-11 19:53:22 UTC - Ryan Samo: Yes, CN for the admin.cert.pem is admin
----
2019-07-11 19:53:37 UTC - Ryan Samo: And it is set as superuser on all brokers
----
2019-07-11 19:54:32 UTC - Ryan Samo: I also added the proxy for good measure as a superuser, not sure that’s required
----
2019-07-11 19:55:13 UTC - Ryan Samo: And the functions workers are targeting the brokers, not the proxies
----
2019-07-11 20:01:40 UTC - Jerry Peng: You also successfully have used the pulsar admin to connect to the broker with tlsCertFile:/opt/pulsar/certificates/admin.cert.pem,tlsKeyFile:/opt/pulsar/certificates/admin.key-pk8.pem right?
----
2019-07-11 20:01:55 UTC - Jerry Peng: the code for creating pulsar admins in the worker is here:
<https://github.com/apache/pulsar/blob/master/pulsar-functions/worker/src/main/java/org/apache/pulsar/functions/worker/WorkerService.java#L127>
----
2019-07-11 20:02:20 UTC - Ryan Samo: Double checking for good measure
----
2019-07-11 20:02:37 UTC - Jerry Peng: There is nothing obviously wrong in your functions_worker.yaml
----
2019-07-11 20:03:06 UTC - Ryan Samo: That’s good at least lol
----
2019-07-11 20:29:54 UTC - Ryan Samo: Yup checks out
----
2019-07-11 22:16:25 UTC - Ryan Samo: @Jerry Peng hey man I just wanted to let you know that I wiped the cluster configs and rebuilt it, now it’s working perfectly. Not sure what got messed up or how but back in action. Thanks for all your help!
+1 : David Kjerrumgaard, Ali Ahmed
----
2019-07-12 01:44:29 UTC - tuteng: This issue has been addressed, the search is already working properly.
----
2019-07-12 01:46:06 UTC - tuteng: This issue has been addressed, the search is already working properly.
clap : David Kjerrumgaard
----
2019-07-12 02:36:14 UTC - Santiago Del Campo: I think finally got it. Just like you said, made sure there is persistent storage for each bookie and zookeeper pod in each node. It was possible making use of Kubernetes Persistent volumes.
Definitely the cluster is waaaaay more stable now and with that i can make further tests with other features Pulsar has to offer... i really appreciate your help so far :slightly_smiling_face:.
I just wanted to point out that this might be some important modification that should be done with the Yaml files (at least for the generic ones, which i am using).... instead of using simple volumes, where the information is lost in every redeploy, add comments inside the file about using persistent volumes to maintain cluster consistency (for zookeeper and bookie deployments, of course, the user has to create the Persistent Volume and Claim, but i think it should be pointed out).
Another point is the init container inside de Bookie Yaml.... before the pods are deployed... a *bin/bookkeeper shell metaformat --nonInteractive || true;* is executed, erasing the metadata information and producing an Invalid Cookie Exception. I just commented it, and combined with the persistent volume approach.. i get a more stable cluster if i reboot the servers, or redeploy all the pods... or simply if the cluster crashes for some reason for a short period of time.
----
2019-07-12 05:27:26 UTC - takuto: @takuto has joined the channel
----