You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@pulsar.apache.org by Apache Pulsar Slack <ap...@gmail.com> on 2019/06/08 09:11:03 UTC

Slack digest for #general - 2019-06-08

2019-06-07 11:12:19 UTC - Ilya Lavrovsky: @Ilya Lavrovsky has joined the channel
----
2019-06-07 11:36:32 UTC - Alexandre DUVAL: Hi, IIRC persistents topics are persisted only if they had a consumer. Can we make them persist without a first subscription?
----
2019-06-07 14:12:37 UTC - Yuwei Jiang: Hi @Ali Ahmed - From grafana dashboard: local publish throughput is around 1kB/s; local delivery throughput is around 1.4kB/s
----
2019-06-07 15:09:16 UTC - Vladimir Puzakov: @Vladimir Puzakov has joined the channel
----
2019-06-07 15:19:07 UTC - Sijie Guo: all the data will always be persisted first for persistent topics. a topic will be deleted when it becomes inactive (e.g. no retention, no subscriptions and etc).

you can either:

a) disable `brokerDeleteInactiveTopicsEnabled=false` in conf/broker.conf

b) set a retention for the topic.
----
2019-06-07 15:52:26 UTC - Kendall Magesh-Davis: In case anyone is looking here later, I wasn’t able to identify an absolute source for this problem. I replicated the error message (but with many others) by force-deleting the EBS volume out from under the PV.

Resolved ZK issue by tearing down and redeploying *only the Zookeeper statefulset*

Hopefully someone finds a more specific reason for this problem and a more elegant solution. Hope this helps. :thumbsup:
----
2019-06-07 17:15:22 UTC - Raghu Velagala: @Raghu Velagala has joined the channel
----
2019-06-07 17:31:50 UTC - Matteo Merli: @Kendall Magesh-Davis :+1:  thanks for the update
----
2019-06-07 17:52:08 UTC - Addison Higham: hrm... so I have worked through the details of pulsar authz/authn and think I understand it all... but I am realizing it seems like there might be a bit of a gap where I want to allow a role to manage a single namespace (such as retention or creating ACLs for topics in that namespace) and not to be able to impact other namespaces in the same tenant
----
2019-06-07 17:52:50 UTC - durga: Hi Guys - We are running into a strange problem with Pulsar. Basically when we deploy `pulsar` on a 15GB node in a k8s cluster and run our smoke tests, everything works fine. But when we run the same system on a 60GB node in a k8s cluster then things don’t work.

I had problems in past with products when memory size was decreased but it is rare where product has problem because more memory is allocated to it :slightly_smiling_face: . I am sure we are doing something wrong in the memory configuration settings.

When we run on a 15 GB node in a k8s cluster, following were the JVM settings we have used and the workload (i.e., smoke tests) run fine.

```
zookeeper: -Xms1g -Xmx1g -XX:MaxDirectMemorySize=1g -Dcom.sun.management.jmxremote -Djute.maxbuffer=10485760 -XX:+ParallelRefProcEnabled -XX:+UnlockExperimentalVMOptions -XX:+AggressiveOpts -XX:+DoEscapeAnalysis -XX:+DisableExplicitGC -XX:+PerfDisableSharedMem  -Dzookeeper.forceSync=no

bookkeeper: -Xms1g -Xmx1g -XX:MaxDirectMemorySize=1g -Dio.netty.leakDetectionLevel=disabled -Dio.netty.recycler.linkCapacity=1024 -XX:+UseG1GC -XX:MaxGCPauseMillis=10 -XX:+ParallelRefProcEnabled -XX:+UnlockExperimentalVMOptions -XX:+AggressiveOpts -XX:+DoEscapeAnalysis -XX:ParallelGCThreads=32 -XX:ConcGCThreads=32 -XX:G1NewSizePercent=50 -XX:+DisableExplicitGC -XX:-ResizePLAB -XX:+ExitOnOutOfMemoryError -XX:+PerfDisableSharedMem -XX:+PrintGCDetails -XX:+PrintGCTimeStamps -XX:+PrintGCApplicationStoppedTime -XX:+PrintHeapAtGC -verbosegc -XX:G1LogLevel=finest

broker: -Xms2g -Xmx2g -XX:MaxDirectMemorySize=2g -Dio.netty.leakDetectionLevel=disabled -Dio.netty.recycler.linkCapacity=1024 -XX:+ParallelRefProcEnabled -XX:+UnlockExperimentalVMOptions -XX:+AggressiveOpts -XX:+DoEscapeAnalysis -XX:ParallelGCThreads=32 -XX:ConcGCThreads=32 -XX:G1NewSizePercent=50 -XX:+DisableExplicitGC -XX:-ResizePLAB -XX:+ExitOnOutOfMemoryError -XX:+PerfDisableSharedMem

proxy: -Xms1g -Xmx1g -XX:MaxDirectMemorySize=1g
```

Now if we run the same workload with pulsar on a bigger node (60GB RAM) with following configuration, it fails.
```
zookeeper: -Xms3g -Xmx3g -XX:MaxDirectMemorySize=3g -Dcom.sun.management.jmxremote -Djute.maxbuffer=10485760 -XX:+ParallelRefProcEnabled -XX:+UnlockExperimentalVMOptions -XX:+AggressiveOpts -XX:+DoEscapeAnalysis -XX:+DisableExplicitGC -XX:+PerfDisableSharedMem  -Dzookeeper.forceSync=no

bookkeeper: -Xms6g -Xmx6g -XX:MaxDirectMemorySize=6g -Dio.netty.leakDetectionLevel=disabled -Dio.netty.recycler.linkCapacity=1024 -XX:+UseG1GC -XX:MaxGCPauseMillis=10 -XX:+ParallelRefProcEnabled -XX:+UnlockExperimentalVMOptions -XX:+AggressiveOpts -XX:+DoEscapeAnalysis -XX:ParallelGCThreads=32 -XX:ConcGCThreads=32 -XX:G1NewSizePercent=50 -XX:+DisableExplicitGC -XX:-ResizePLAB -XX:+ExitOnOutOfMemoryError -XX:+PerfDisableSharedMem -XX:+PrintGCDetails -XX:+PrintGCTimeStamps -XX:+PrintGCApplicationStoppedTime -XX:+PrintHeapAtGC -verbosegc -XX:G1LogLevel=finest

broker: -Xms3g -Xmx3g -XX:MaxDirectMemorySize=3g -Dio.netty.leakDetectionLevel=disabled -Dio.netty.recycler.linkCapacity=1024 -XX:+ParallelRefProcEnabled -XX:+UnlockExperimentalVMOptions -XX:+AggressiveOpts -XX:+DoEscapeAnalysis -XX:ParallelGCThreads=32 -XX:ConcGCThreads=32 -XX:G1NewSizePercent=50 -XX:+DisableExplicitGC -XX:-ResizePLAB -XX:+ExitOnOutOfMemoryError -XX:+PerfDisableSharedMem

proxy: -Xms3g -Xmx3g -XX:MaxDirectMemorySize=3g
```

I appreciate very much if anyone can provide some pointers on what we could be doing wrong. CC @Yuwei Jiang
----
2019-06-07 17:54:23 UTC - Addison Higham: Am I missing something? or is that not currently covered by the existing ACL scheme? Looking at the AuthorizationProvider interface it seems to not be covered
----
2019-06-07 17:54:35 UTC - Matteo Merli: Correct, It’s not currently possible to do that
----
2019-06-07 17:55:03 UTC - Matteo Merli: The tenant admin right now is allowed to administrate all the namespaces for a given tenant
----
2019-06-07 17:56:02 UTC - Addison Higham: which makes sense, I wonder if you could piggy back on top of the ACL framework and have another action of "manage" that could allow for management of all settings related to a topic or namespace?
----
2019-06-07 17:56:05 UTC - Matteo Merli: I guess this could be extended by having an “admin” action permission that can be granted on a namespace level (rather than tenant)
----
2019-06-07 17:56:52 UTC - Matteo Merli: Yes, right now there are 3 possible ACLs actions: produce, consume, functions. We could extend with admin as well
----
2019-06-07 17:57:47 UTC - Addison Higham: seems reasonable... I don't have need of it this minute, but if I get some time in the next little bit I may research that more and see if it would make sense to make a PIP out of it
----
2019-06-07 17:57:59 UTC - Matteo Merli: :+1:
----
2019-06-07 18:08:39 UTC - Joe Francis: Be  very clear on how this will be, dealing with child granted  but parent revoked  is a  rathole
----
2019-06-07 18:09:55 UTC - Matteo Merli: why? you will have 2 different principals that may admin the namespace. One for the entire tenant and the other for the namespace.
----
2019-06-07 18:12:50 UTC - Joe Francis: Yeah, but the right of the second principal is granted by the  first one.
----
2019-06-07 18:13:03 UTC - Matteo Merli: Yes
----
2019-06-07 18:14:22 UTC - Matteo Merli: But the same happens for “produce” permission, the tenant admin will grant permission to “user-a” to publish. Now, if that specific admin will get its permission revoked, the “user-a” will still be able to publish.. unless someone removes that permission explicitely
----
2019-06-07 18:14:22 UTC - Joe Francis: So what  will you do if  parent revokes and child grants?
----
2019-06-07 18:16:21 UTC - Joe Francis: Changing admin privs is  not the issue - the issue is if you have conflicting grants.
----
2019-06-07 18:16:52 UTC - Matteo Merli: the grants are always “addictive”. I don’t see how they can conflict
----
2019-06-07 18:17:58 UTC - Joe Francis: Namespace admin can grant, and tenant admin can revoke - what's the net outcome?
----
2019-06-07 18:18:18 UTC - Joe Francis: Or vice-versa
----
2019-06-07 18:19:10 UTC - Matteo Merli: permission to publish?

that will result in no permission. the ACLs for namespace are written in 1 single place. Now you’ll have 2 principals that can change that, but the info is not duplicated
----
2019-06-07 18:21:25 UTC - Joe Francis: Sure, that is how I would prefer it. But be clear about this, that there is no hierarchical permissions
----
2019-06-07 18:21:38 UTC - Matteo Merli: :+1:
----
2019-06-07 19:10:56 UTC - Sijie Guo: what errors did you see?
----
2019-06-07 22:17:40 UTC - Alexandre DUVAL: oh okay :slightly_smiling_face:, thanks
----
2019-06-08 05:02:33 UTC - Yuwei Jiang: Hi @Sijie Guo - The segment of the log files containing the errors are below
----
2019-06-08 05:04:14 UTC - Yuwei Jiang: bookkeeper log
----