Slack digest for #general - 2019-03-08

[Apache Pulsar Slack <ap...@gmail.com>]

2019-03-07 09:37:31 UTC - Gofu: Hello guys, can someone tell me if it is possible to install pulsar c++ client in alpine?
----
2019-03-07 09:40:27 UTC - Marc Le Labourier: I had some strange errors with the pulsar-client on a alpine-python3.7 image but I think its not related, the container was unable to resolve pulsar-client as a dependency which worked fine with another image.
----
2019-03-07 09:41:03 UTC - Gofu: how did you install it?
----
2019-03-07 09:41:39 UTC - Marc Le Labourier: pip install pulsar-client, again it was in python with the released package.
----
2019-03-07 09:41:50 UTC - Marc Le Labourier: May not help you very much.
----
2019-03-07 09:42:29 UTC - Gofu: yes, should be different, thanks for helping
----
2019-03-07 09:44:08 UTC - Marc Le Labourier: If you want to build it from the source, I know that alpine image are very light. You will mostly miss some dependencies like a compiler.
----
2019-03-07 09:45:05 UTC - Marc Le Labourier: However, if you build the lib on your laptop or a remote machine, you could copy them in you image later when building a docker image for example.
----
2019-03-07 09:45:57 UTC - Gofu: yeah I mean, I have tried with the rpm package since debian packages are not supported in  alpine. But with rpm I am always getting a GLIBC error. I am using the following docker image since glibc is not alpine native.
----
2019-03-07 09:48:57 UTC - Gofu: 
----
2019-03-07 13:56:50 UTC - Byron: @jia zhai I appreciate it.
----
2019-03-07 14:21:44 UTC - Matteo Merli: The RPM are linked with glibc. You’d have to recompile the client in Alpine
----
2019-03-07 14:24:09 UTC - Gofu: That's why I have tried with the above docker image
----
2019-03-07 14:24:39 UTC - Gofu: this lib <https://github.com/sgerrand/alpine-pkg-glibc> is supposed to work
----
2019-03-07 14:25:28 UTC - Gofu: but without success, its kind of unflexible to do the build manually
----
2019-03-07 14:31:57 UTC - jia zhai: @Byron  welcome
----
2019-03-07 15:48:32 UTC - Ben S: I‘ve some general question about Pulsar, and how it compares to Kafka for the use case of low latency high performance exactly once guaranteed message delivery with failover. 1) What are the main differences from an architecture perspective? 2) What are the main differences from a performance perspective? And 3) how do those differences relate to how the system behaves during failover (which one gets the admin more sweating?)
----
2019-03-07 15:53:52 UTC - David Kjerrumgaard: @Ben S I would recommend reading some of the blog posts on Streamlio's website, starting with this one: <https://streaml.io/blog/pulsar-segment-based-architecture>
----
2019-03-07 15:56:18 UTC - David Kjerrumgaard: @Ben S This one covers Exactly-once processing  <https://streaml.io/blog/pulsar-effectively-once>
----
2019-03-07 16:20:30 UTC - Alexandre DUVAL: The persistents topics metadata are store on brokers? So where are stored the non-persistents? I've http 200 on topics list persistents, but 401 on non-persistent.
----
2019-03-07 16:20:51 UTC - Alexandre DUVAL: (I randomly modified tokens).
----
2019-03-07 16:21:10 UTC - Alexandre DUVAL: I'm executing topics list from broker node.
----
2019-03-07 16:22:24 UTC - Alexandre DUVAL: @Matteo Merli (to follow our discussion yesterday, I had wrong newline in the token file #gitgoodpractices, but now here is the problem ^ :p).
----
2019-03-07 17:28:32 UTC - Matteo Merli: metadata for “persistent” topics is on ZK. for non-persistent the metadata is not stored, hence the call to get the list of topics is “proxied” by one broker to the different brokers serving these topics
----
2019-03-07 17:28:47 UTC - Matteo Merli: What appears to be failing is the broker-to-broker auth
----
2019-03-07 17:29:33 UTC - Matteo Merli: So, can you get the same capture of the broker doing the REST call ?
----
2019-03-07 17:59:43 UTC - Vikas: @David Kjerrumgaard the architect who is in charge of securing the Pulsar cluster has modified the way to authenticate. Now we are using `token auth` , so could you please let me know what all I need to change in NiFi so as to leverage `token auth` to connect to the secured cluster? thanks!
----
2019-03-07 18:28:46 UTC - David Kjerrumgaard: @Vikas I will see if there are any existing options in NiFi for this. If not, I will need to add a controller service to the code base, which will take some time
----
2019-03-07 18:31:35 UTC - Vikas: great, thanks so much. Appreciate your help on this task!
----
2019-03-07 18:41:16 UTC - David Kjerrumgaard: No problem
----
2019-03-07 19:36:08 UTC - David Kjerrumgaard: @Vikas Can you confirm that these are the steps your team is now using to secure the pulsar cluster instead of standard TLS?  <http://pulsar.apache.org/docs/en/security-athenz/#configure-the-broker-for-athenz>
----
2019-03-07 19:45:09 UTC - Chris DiGiovanni: I have a question about bookie's utilization of disk space.  I originally had 3 bookies with ledgers on a disk size of 128G.  I had 2 of the three ledgers disks that went up to 96% utilization and went into read only mode.  At that time I was unable to publish anymore to the cluster.  I then added a 4th bookie, writes then continued.  I then setup Tiered Storage.  Data has been consistently moved off and utilization on 3 of the 4 ledgers went down to a respectable amount except for 1 bookie's ledger.  It has went from 96% utilization to 100%.  Should I be concerned about this?  The only way I can see to re-balance is to remove the bookie that is 100% utilized, and then have auto-recovery run.  A few settings below which I believe could be relevant to the issue.

```
managedLedgerDefaultAckQuorum=2
managedLedgerDefaultEnsembleSize=2
managedLedgerDefaultWriteQuorum=2
```
----
2019-03-07 20:27:51 UTC - Kai Levy: @Kai Levy has joined the channel
----
2019-03-07 20:46:24 UTC - Matteo Merli: Problem is that the compaction is taking a bit space to run. With a smaller disk of 128G the remaining 5% is not enough to have it cleared up
----
2019-03-07 20:50:37 UTC - Matteo Merli: You can set the % limit to be lower 
----
2019-03-07 21:08:26 UTC - Chris DiGiovanni: When you say compaction in the Tiered Storage Offload sense, I'm assuming your not talking about Topic Compaction?
----
2019-03-07 21:18:40 UTC - Vikas: hey @David Kjerrumgaard thanks for checking. We are using TLS authentication and Client authentications using tokens (both of them)
<http://pulsar.apache.org/docs/en/security-tls-transport/>

<http://pulsar.apache.org/docs/en/security-token-client/>
----
2019-03-07 21:19:42 UTC - David Kjerrumgaard: So you will need to provide the certificate along with the token to connect.
----
2019-03-07 21:20:34 UTC - David Kjerrumgaard: And you are using Athenz for the token provider
----
2019-03-07 21:21:01 UTC - Vikas: nope we are not using Athenz for the token provider
----
2019-03-07 21:22:01 UTC - Vikas: using below for the token provider:

<http://pulsar.apache.org/docs/en/security-token-client/>
<http://pulsar.apache.org/docs/en/security-token-admin/>
----
2019-03-07 21:22:36 UTC - Vikas: we are providing the a) CA Trust Cert so we can verify the server (as it has a CA issued server cert) over the TLS connection and b) the token so the server can authenticate the client.
----
2019-03-07 21:24:16 UTC - David Kjerrumgaard: Ok, got it.  Thanks!!
+1 : Vikas
----
2019-03-07 22:59:31 UTC - Chris DiGiovanni: @Matteo Merli are you talking about `Entry log compaction settings` on the bookie?
----
2019-03-07 23:00:26 UTC - Matteo Merli: Hi yes, when ledgers get deleted, the entry logs in bookie disk will get compacted to reclaim the space
----
2019-03-07 23:55:08 UTC - Jacob O'Farrell: Are there any good resources in terms of logos/icons that would be suitable for displaying pulsar topics/functions etc in a diagram?
----
2019-03-08 00:37:33 UTC - Jacob O'Farrell: If there is interest in something like this and it isn't already available, I could potentially get one of our designers to put something together
----
2019-03-08 05:02:31 UTC - Sree Vaddi: @Sree Vaddi has joined the channel
----
2019-03-08 05:10:15 UTC - Sree Vaddi: Hello Team
blush : Sijie Guo, Ali Ahmed, Karthik Ramasamy
----
2019-03-08 05:14:10 UTC - Ali Ahmed: there is some work being done to build apache pulsar logo , the old does not represent it as an apache project , for functions there is no formal logo yet
----
2019-03-08 05:14:18 UTC - Ali Ahmed: proposals are welcome
----
2019-03-08 06:14:47 UTC - zhengwei: @zhengwei has joined the channel
----