You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@pulsar.apache.org by Apache Pulsar Slack <ap...@gmail.com> on 2018/06/20 09:11:04 UTC
Slack digest for #general - 2018-06-20
2018-06-19 09:57:26 UTC - Harald Gustafsson: I wanted to check if the python 3 pulsar clients for mac is also in the process of publishing to pypi.
----
2018-06-19 16:06:23 UTC - Karthik Palanivelu: Team, I have a quick question - If I start a proxy will by itself start a broker? because I start a broker and start a proxy on K8s. Is this a right order?
----
2018-06-19 17:05:27 UTC - Ali Ahmed: @Harald Gustafsson Maybe not for 2.0.1 but 2.1 is getting close we will add python3 osx modules to it.
----
2018-06-19 17:05:50 UTC - Ali Ahmed: @Karthikeyan Palanivelu proxy does not start a broker by itself
----
2018-06-19 17:06:18 UTC - Matteo Merli: @Harald Gustafsson builds for macos and py3 are coming now with 2.0.1. just not uploaded yet to pypi
----
2018-06-19 17:06:20 UTC - Ali Ahmed: ideally brokers should be like and then then the proxy should start.
----
2018-06-19 17:08:29 UTC - Karthik Palanivelu: Cool then I am doing right starting brokers and then proxy. Can you please look into my error I posted above from Proxy or I copied below?
----
2018-06-19 17:08:37 UTC - Karthik Palanivelu: @Karthik Palanivelu uploaded a file: <https://apache-pulsar.slack.com/files/U7VRE0Q1G/FBA9KPWN6/-.txt|Untitled>
----
2018-06-19 17:10:11 UTC - Ali Ahmed: @Karthikeyan Palanivelu check the kubernetes logs for proxy and brokers , to see if any connection errors are happening
----
2018-06-19 17:11:34 UTC - Harald Gustafsson: @Matteo Merli thanks, need the new simplified auth
----
2018-06-19 17:18:38 UTC - Karthik Palanivelu: @Ali Ahmed I see only below log in proxy:
```
21:07:03.181 [pulsar-discovery-io-2-2] INFO org.apache.pulsar.proxy.server.ProxyConnection - [/XX.XX.XX.XX:30989] New connection opened
21:07:03.196 [pulsar-discovery-io-2-2] INFO org.apache.pulsar.proxy.server.ProxyConnection - [/XX.XX.XX.XX:30989] Connection closed
```
And only exception in broker logs is:
----
2018-06-19 17:18:59 UTC - Karthik Palanivelu: @Karthik Palanivelu uploaded a file: <https://apache-pulsar.slack.com/files/U7VRE0Q1G/FBA3675QU/Untitled.lua|Untitled> and commented: broker.log - All brokers has this exception
----
2018-06-19 17:26:56 UTC - Ali Ahmed: don’t know the reason for the error , did you try to send a message with a very large payload ?
----
2018-06-19 17:27:48 UTC - Karthik Palanivelu: This was happening when I start the Listener/Consumer...
----
2018-06-19 17:29:10 UTC - Ali Ahmed: I don’t know right now will have to spend some later , can you try your cluster without encryption and see if that works?
----
2018-06-19 19:04:14 UTC - Ivan Kelly: @Karthikeyan Palanivelu that error is trying to send a http request to a binary protocol endpoint
----
2018-06-19 19:34:51 UTC - Harald Gustafsson: @Harald Gustafsson uploaded a file: <https://apache-pulsar.slack.com/files/UAEF8E18E/FBA9RHUD9/-.txt|Untitled>
----
2018-06-19 19:35:56 UTC - Harald Gustafsson: @Harald Gustafsson uploaded a file: <https://apache-pulsar.slack.com/files/UAEF8E18E/FBBA0J4G6/-.txt|Untitled>
----
2018-06-19 19:40:53 UTC - Harald Gustafsson: Oh, I see now that the AuthenticationTLS is commited after 2.0.1. How is the Authentication class used that exist in 2.0.1
----
2018-06-19 19:41:19 UTC - Matteo Merli: Yes, I was double checking here
----
2018-06-19 19:41:39 UTC - Matteo Merli: the part about auth was to be included in 2.1
----
2018-06-19 19:42:27 UTC - Matteo Merli: we just backported the packaging fixes for 2.0.1
----
2018-06-19 19:46:41 UTC - Matteo Merli: btw, @Harald Gustafsson the macos 10.13 build for py3 is available, though without the auth-tls changes as above
----
2018-06-19 19:52:35 UTC - Harald Gustafsson: @Matteo Merli OK, I will need to figure out how to use the current method then, I thought I had seen some example of that. I wonder how to get the path to the lib.
----
2018-06-19 19:58:48 UTC - Matteo Merli: Main problem, (which the fix in 2.1 is trying to solve) is that the plugin system is expecting the `tls.so` lib somewhere. Unfortunaly, the `tls.so` cannot be included in the python wheel file.
You would have to compile C++ library and copy `tls.so` and use `pulsar.Authentication()` class. For parameters, it would be similar to the example here: <http://pulsar.apache.org/docs/v2.0.1-incubating/clients/Cpp/#Authentication-h6s57i>
----
2018-06-19 19:58:52 UTC - Matteo Merli: (for c++)
----
2018-06-19 20:49:40 UTC - Harald Gustafsson: @Matteo Merli I tried the homebrew install of the C++ was hoping that the tls.so would be found from that but could not find it. The homebrew actually builds 2.1 snapshot. Would it be possible to generate the python package for 2.1 instead, from source? Anyway need to give up for today.
----
2018-06-19 20:55:04 UTC - Ali Ahmed: yes the python packages can be generated from source
----
2018-06-19 21:21:16 UTC - Matteo Merli: @Harald Gustafsson there’s a script to generate all combination of Python builds: `./pulsar-client-cpp/docker/build-wheels.sh`. It builds the code from the current working dir and leaves the `whl` for each Python version
----
2018-06-19 22:30:58 UTC - Kamesh Nemana: @Kamesh Nemana has joined the channel
----
2018-06-20 02:42:46 UTC - Eric Asuncion: @Eric Asuncion has joined the channel
----
2018-06-20 05:13:29 UTC - Rene Hernandez: @Rene Hernandez has joined the channel
----
2018-06-20 08:07:57 UTC - Idan: Hi guys, some of our consumers using batching model. one of the reasons we use batching is because we want to insert in batch to datasources like RDS. on pulsar you can enable batching but the API still consume the messages as individual. any code examples how to wrap this with internal queue to let our consumers get this in a list (or aggregated) some how
----
2018-06-20 08:17:48 UTC - Ali Ahmed: @Idan consumer batching is already done
----
2018-06-20 08:24:06 UTC - Sijie Guo: @Idan a couple of ideas:
1) if you are using MessageListener, you can create a queue in your message listener implementation, when receiving a message from message listener, you put the item into the queue and check if the queue reaches a given size, if so, then return this list of messages to your application, so your application can insert them in batch to RDS.
2) if you are using receiveAsync, you could create a List<CompletableFuture<Message>> futures.
then
```
for (int i = 0; i < batchSize; i++) {
futures.add(consumer.receiveAsyc());
}
List<Message> messages = CompletableFuture.allOf(futures.toArray()).thenApply(result -> futureList.stream.map(CompletableFuture::join).collect(Collectors.toList()));
```
then you can return them as a list of messages.
however in both 1) and 2), you need some logic to handle if there is no message coming in within a given time period.
3) ideally, this batch interface should exist in pulsar indeed. since internally messages are batched to store to storage and transfer over network. so pulsar has all the information, so it can just return the whole batch as a message iterator, which you can then iterate over messages.
----
2018-06-20 08:33:02 UTC - Idan: iam recievieing the messages using this: recieve
----
2018-06-20 08:33:05 UTC - Idan: method
----
2018-06-20 09:03:21 UTC - Harald Gustafsson: @Matteo Merli The wheel building did not work. I'm building on a mac so the mounted volume is from a mac, which have not been a problem before but maybe this time. First of on a fresh git the scripts is trying to remove CMake* files and fails since not existing. That was easily fixed by creating some empty files. But when building the wheels with `./pulsar-client-cpp/docker/build-wheels.sh`I get the following error:
----
2018-06-20 09:03:57 UTC - Harald Gustafsson: @Harald Gustafsson uploaded a file: <https://apache-pulsar.slack.com/files/UAEF8E18E/FBAFPAD44/-.pl|Untitled>
----