Slack digest for #general - 2018-08-21

[Apache Pulsar Slack <ap...@gmail.com>]

2018-08-20 12:22:46 UTC - Yuvaraj Loganathan: Is it possible to set producerID level ratelimiting. A single producer won’t be able to produce more than 100msgs/sec ?
----
2018-08-20 13:47:19 UTC - Yuvaraj Loganathan: We are trying to send app logs through pulsar. Sometime people enable debug logs and a single producer slows down the whole logs pipeline.
----
2018-08-20 14:46:24 UTC - Martin Kunev: Hi, I have a question regarding pulsar 2. I have a pulsar setup with 2 clusters - `cluster0` and `cluster1` and an application `application0` that is subscribed to a persistent topic on `cluster0` (the topic replicates to the other cluster). At some point in time `application1` subscribes to the same topic on `cluster1` and after that `application0` unsubscribes. Could there be a message that got received by neither application? I mean, is it possible that the message didn't get to `application1`, because it hadn't subscribed when it was received and the message also didn't get to `application0` because it had already unsubscribed by the time the message arrived?
----
2018-08-20 16:37:38 UTC - Karthik Palanivelu: Team, I am migrating from 2.0.1 to 2.1.0. While I was using proxy before I have added the below config to work and remember that future versions will have it. But I could not find it in proxy.conf. I am retaining mine and moving forward. Just want to check with you and understand whether it is still needed or not?
```
brokerServiceURL=<pulsar://broker:6650>
brokerServiceURLTLS=<pulsar+ssl://broker:6651>
webServiceURL=<http://broker:9101>
webServiceURLTLS=<https://broker:8443>
```
----
2018-08-20 17:30:21 UTC - Matteo Merli: @Yuvaraj Loganathan There was a proposal to add that some time back, but nothing happened so far. At this point the rate-limiting can only be configured on delivery. 

One thing to note though, is that producer has a flow-control mechanism. The max number of in-flight messages from a producer is determined by the producer queue size. Default is 1K, lowering this queue size to a smaller value, will effectively rate-limit the producer, to a rate based on: `queue-size / avg-publish-latency`
----
2018-08-20 17:38:19 UTC - Matteo Merli: @Martin Kunev The data will not be lost as long as `application0` will drain any existing backlog of messages before unsubscribing. 

Once `application1` is subscribed, everything that is published in every other cluster will be retained.
----
2018-08-20 17:55:11 UTC - Sijie Guo: &gt;  I am retaining mine and moving forward. Just want to check with you and understand whether it is still needed or not?

I think they are needed. you can retrain your 2.0.1 configuration for 2.1.0.
----
2018-08-20 18:13:12 UTC - Martin Kunev: I see, this is very good. Thanks a lot!
----
2018-08-20 18:41:20 UTC - Ivan Kelly: @Karthikeyan Palanivelu they are needed if you don't want to connect the proxy to zk. Otherwise the proxy will do service discovery
----
2018-08-20 18:42:18 UTC - Ivan Kelly: also, tls usage between the proxy and broker has been changed in 2.1.0 (it actually works now). The current state of affairs is covered in the docs
----
2018-08-20 18:48:30 UTC - Ravi: any existing plugins for email ingestion ? We are working on writing code for ingesting emails into streamlio if you already have any please suggest
----
2018-08-20 19:50:06 UTC - Matteo Merli: @Ravi There’s no ready-made plugin for emails right now. 

Is that something like IMAP/POP fetcher? It would be great to have that as a source for Pulsar IO.
----
2018-08-20 19:51:51 UTC - Ravi: we are building it into our unified messaging platform, we can open the plugin once developed.
----
2018-08-20 19:52:57 UTC - Matteo Merli: :+1:
----
2018-08-21 05:28:31 UTC - CongBo: @CongBo has joined the channel
----