Slack digest for #general - 2019-01-25

[Apache Pulsar Slack <ap...@gmail.com>]

2019-01-24 11:02:58 UTC - Ericko Samudera: @Ericko Samudera has joined the channel
----
2019-01-24 11:10:49 UTC - Ericko Samudera: Hi guys, we are looking to modify the reader to be able to move the cursor in the reversed direction (to retrieve "chat history"). Is it on the roadmap or could we send a PR? Thanks!
+1 : Sijie Guo
----
2019-01-24 11:16:25 UTC - Sijie Guo: @Ericko Samudera: it is not on the roadmap. you are welcome to contribute the feature. if you already have the PR, send it and we can have discussion over there; if you just start with the requirement, it would be good to send a PIP and raise a proposal on dev@ mailing list
----
2019-01-24 11:17:44 UTC - Ericko Samudera: okay, we will send the PIP soon, thanks
100 : Sijie Guo
+1 : Sijie Guo
----
2019-01-24 14:26:03 UTC - Vincent Ngan: Can anyone explain to me the purpose of using a Pulsar Proxy? The documentation also suggests the use of frontend load-balancer such as HAProxy. This makes it quite confusing to me because HAProxy can also be used as a proxy to load-balance the backend servers. So, why don’t we just configure the HAProxy to load-balance the brokers directly. In this case, we do not need an additional Pulsar Proxy.
----
2019-01-24 14:48:30 UTC - dba: Hi. We have a group of Python developers (I'm not one of them) that would like to give Pulsar Functions a try. They are developing on Windows, but when trying to install the pulsar-client lib from PIP they cant't, because it is built on the C++ lib, which is Linux only.
Any plans in regards to compiling a version for Windows or is there a better option?
----
2019-01-24 15:01:48 UTC - Ivan Kelly: @Vincent Ngan pulsar proxy gives you a single endpoint to talk to. if you talk directly to the brokers, if a topic you want to use is assigned to another broker, you then have to talk to that broker
----
2019-01-24 15:03:27 UTC - Ivan Kelly: whereas, with the proxy, you can talk directly to one instance the whole time
----
2019-01-24 15:32:09 UTC - Vincent Ngan: So, that means pulsar proxy is a smart proxy which knows where the topics are and route the requests to the correct brokers without requiring the client to know exactly where the topics are. Originally, I thought this is already handled by the brokers. So the location transparency of the topics is provided by the pulsar proxy and not by the brokers. Thanks for your help.
----
2019-01-24 15:40:46 UTC - Grant Wu: Have you considered the Windows Subsystem for Linux?
----
2019-01-24 16:46:47 UTC - Ezequiel Lovelle: @Ericko Samudera Hi! I don't know your specific use case but you have the option to reset a given position of a consumer to a previous in time position, maybe this fits your needs.
----
2019-01-24 16:59:43 UTC - Joe Francis: Thats not exactly correct. Lookup is handled by client library + brokers provided they are in the same network address space.  Any broker can then redirect a client  to the  broker serving that topic. But if the brokers are in a private network, and the client outside of that, then the client cannot directly access the brokers, because the addresses are private.  Pulsar proxy,  running  on a gateway host, acts as a network proxy  to bridge  networks
----
2019-01-24 17:42:53 UTC - Ambud Sharma: @Ambud Sharma has joined the channel
----
2019-01-24 17:55:04 UTC - Matteo Merli: @Vincent Ngan The purpose of pulsar proxy is for deployments where client don’t have direct connectivity with brokers.

One such example is:
 * Pulsars cluster deployed in Kubernetes
 * Clients connecting from outside

In these cases, the proxy will act as a completely stateless frontend to Pulsar and it can be exposed through a LB/ DNS endpoint.

It cannot be a generic TCP proxy, because it needs to understand pulsar protocol to route to appropriate broker. After the initial handshake, the Pulsar proxy will degrade itself to a simple TCP proxy.
----
2019-01-24 17:59:01 UTC - Matteo Merli: I think @Ericko Samudera is referring to have a reader that goes backward, if I’m not mistaken.

It would be an interesting addition for sure! :slightly_smiling_face:
----
2019-01-24 18:14:21 UTC - Ezequiel Lovelle: Yeah! of course!
----
2019-01-24 20:03:59 UTC - Emma Pollum: I want to see how much performance increase we get by forcing clients to use the pulsar protocol. I was hoping to use the openmessaging framework. Is there a way to force it to use http over pulsar?
----
2019-01-24 20:05:50 UTC - Matteo Merli: When you specify `<pulsar://localhost:6650>` or `<http://localhost:8080>` the only difference is in the service discovery phase. After that, the data path will always go to pulsar protocol
----
2019-01-24 20:23:31 UTC - Grant Wu: I think all the official clients always use the Pulsar protocol
----
2019-01-24 20:24:31 UTC - Matteo Merli: Yes, also that server only support Pulsar protocol :slightly_smiling_face: with the exception of proxies that can be used in front of it.
----
2019-01-24 21:43:30 UTC - Yu Yang: @Yu Yang has joined the channel
----
2019-01-25 02:41:55 UTC - bossbaby: i have 1 topic, many broker in the world and many consumer &amp; producer in many server in the world connect, throughtput in each producer and consumser are differently. What is the best solution to optimal my system?
----
2019-01-25 03:09:14 UTC - Ali Ahmed: @bossbaby can you clarify your question with more details
----
2019-01-25 03:22:08 UTC - bossbaby: ( my server can't use partition topic becase of not support ackcumulative to optimal). i have a topic and
1 topic only 1 broker, so if my service have a many many consumer &amp; producer connect and produce &amp; consume, bottleneck will happen. What is the best solution to solve it?
----
2019-01-25 04:02:31 UTC - Matteo Merli: I’d suggest to just use individual acknowledgments with a partitioned topic
----
2019-01-25 04:10:25 UTC - bossbaby: The partition topic has many restrictions such as can't seek with earlier, ack ackcumulative and acknowledgments will cause the system to be slow.
----
2019-01-25 04:12:27 UTC - Matteo Merli: I don’t think there’s a particular difference in performance between ack and cumulative acks 
----
2019-01-25 04:12:53 UTC - bossbaby: because there are some messages that do not need confirmation
----
2019-01-25 04:13:04 UTC - Matteo Merli: There are several ways in the individual acks are grouped together for efficiency 
----
2019-01-25 04:13:15 UTC - Matteo Merli: Both in client and in broker side 
----
2019-01-25 04:13:28 UTC - Matteo Merli: And both are configurable
----
2019-01-25 04:15:01 UTC - Matteo Merli: By default client will batch acks at a 100ms group time 
----
2019-01-25 04:18:10 UTC - bossbaby: each time sending a ack, the consumer will spam the server, this is problem
----
2019-01-25 04:19:54 UTC - bossbaby: i am setting, each 5s consumer will send ack to server
----
2019-01-25 04:20:49 UTC - Matteo Merli: What I’m saying is that you can automatically get the same behavior with individual acks
----
2019-01-25 04:21:34 UTC - Matteo Merli: Take a look at <http://pulsar.apache.org/api/client/org/apache/pulsar/client/api/ConsumerBuilder.html#acknowledgmentGroupTime-long-java.util.concurrent.TimeUnit->
----
2019-01-25 04:24:02 UTC - Matteo Merli: That will make the client to send the acks in batches
----
2019-01-25 04:26:37 UTC - bossbaby: sorry @Matteo Merli, i am using pulsar client c++ and it not support acknowledgmentGroupTime
----
2019-01-25 04:27:02 UTC - Matteo Merli: Oh, I see
----
2019-01-25 04:28:37 UTC - bossbaby: acknowledgmentGroupTime will be available in pulsar client c++ 2.3.0 ?
----
2019-01-25 04:32:24 UTC - Matteo Merli: We don’t have time to do it for 2.3
----
2019-01-25 04:32:40 UTC - Matteo Merli: Though it could be done in2.4
----
2019-01-25 04:50:31 UTC - bossbaby: thanks you merlimat, i think i must use Pulsar Geo-Replication to solve problem with normal topic
----
2019-01-25 05:20:15 UTC - Toshiaki Irie: @Toshiaki Irie has joined the channel
----
2019-01-25 07:21:50 UTC - dba: No, I think we were hoping the C++ lib was written in standard C++, so it could easily be compiled for Windows and that Pulsar would be interested in doing so :slightly_smiling_face:
----
2019-01-25 07:29:05 UTC - Ali Ahmed: @dba you can try using a linux docker image for development, there are windows users who do that. we also have mac binaries , supporting windows is an expensive proposition which we don’t have enough people for at this time.
----
2019-01-25 07:36:37 UTC - dba: @Ali Ahmed I don't think I understand fully, so maybe you can help me. If the C++ code is 100% Standard C++, wouldn't it then just be a matter of compiling and releasing the C++ lib for Windows? That would make the Pulsar experience for C++, Go and Python developers on Windows so much better.
Currently I am working on a C#/.NET lib, which will open Pulsar for the 10 million .NET developers out there, but it would be nice if Pulsar could be more cross platform by embracing Windows.
----
2019-01-25 07:40:30 UTC - Ali Ahmed: that’s part of it’s also a question of testing . We currently have linux machines for development and testing, I don’t know if any of the committers are currently using windows.
----
2019-01-25 08:32:02 UTC - dba: Would it be worth it to find out if you could squeeze Windows support in? Maybe some committers have a Windows machine and willing to test it? Given that the C++ client is standard C++ and can be compiled for Windows of course.
----
2019-01-25 08:32:26 UTC - bossbaby: i have a question that
By default, ttl in topic = 0 but in dashboard, why backlog not detele?
----
2019-01-25 08:34:37 UTC - Ali Ahmed: I would recommend proposing this to the dev mailing list see if there a support for this.
+1 : dba
----