You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@pulsar.apache.org by Apache Pulsar Slack <ap...@gmail.com> on 2020/02/08 09:11:03 UTC
Slack digest for #general - 2020-02-08
2020-02-07 10:49:47 UTC - Roman Popenov: Think I found a work around for this
----
2020-02-07 10:49:51 UTC - Roman Popenov: I will open a PR
----
2020-02-07 12:42:39 UTC - Yoav Cohen: Hi everyone. When installing pulsar on minikube using the instructions in <https://github.com/apache/pulsar/tree/master/deployment/kubernetes/helm>, zookeeper fails to start due to anti-affinity rules:
`Warning FailedScheduling <unknown> default-scheduler 0/1 nodes are available: 1 node(s) didn't match pod affinity/anti-affinity, 1 node(s) didn't satisfy existing pods anti-affinity rules.`
I then tried to change the replicaCount for zookeeper, bookkeeper and broker to 1 (reducing managedLedger* to 1 as well) but no luck, bookkeeper has error:
`Warning FailedScheduling <unknown> default-scheduler 0/1 nodes are available: 1 node(s) didn't match pod affinity/anti-affinity, 1 node(s) didn't satisfy existing pods anti-affinity rules.`
How is it possible to run pulsar on minikube? Is there a way to disable anti-affinity? Am I missing something?
Thanks in advance.
----
2020-02-07 12:49:15 UTC - Vladimir Shchur: To run 2.5.0 on minikube you need to delete antiaffinity rules and run 3 bookies, it is a known bug:
<https://github.com/apache/pulsar/issues/6127>
----
2020-02-07 12:53:26 UTC - Yoav Cohen: Thanks @Vladimir Shchur. As a followup, what’s the best practice on consuming the pulsar helm chart? I’d have to do this change locally for now I assume. Are there plan to release official versions of the helm chart so folks can define a dependency to pulsar in their own charts?
----
2020-02-07 12:56:17 UTC - Vladimir Shchur: @Yoav Cohen I'm not from Pulsar team, let's wait for them to comment :slightly_smiling_face: The solution I proposed is the one that works for me with minikube.
----
2020-02-07 12:56:37 UTC - Yoav Cohen: :+1:
----
2020-02-07 12:56:38 UTC - Shahar Fermon: @Shahar Fermon has joined the channel
----
2020-02-07 13:00:11 UTC - Yuri de Wit: Good morning! I am new to Pulsar and intrigued by its potential. I was wondering how does Pulsar, or more specifically Bookkeeper, positions itself compared to Apache Arrow, which seems to be another interesting direction in terms of data format, schema and interchange.
----
2020-02-07 14:50:33 UTC - Eric Simon: Good morning y’all. We are trying to deploy the helm chart with “tolerations”. I think there may be a mistype in the values*.yaml files. In the values it spells it “tolarations”, but in the template files, it is tolerations. I believe tolerations is correct. Can someone confirm? <https://github.com/apache/pulsar/blob/branch-2.5/deployment/kubernetes/helm/pulsar/values.yaml#L73>
----
2020-02-07 14:57:46 UTC - Roman Popenov: It should be `tolerations`
+1 : Eric Simon
----
2020-02-07 16:19:37 UTC - Eric Simon: Created a PR that can be found here: <https://github.com/apache/pulsar/pull/6265>
100 : Sijie Guo, Roman Popenov, Karthik Ramasamy
----
2020-02-07 16:24:28 UTC - Sijie Guo: @Vladimir Shchur can you send the change to values-mini.yaml? This antiaffinity needs to be disabled and the ensemble and quorum size need to be reduced to 1
----
2020-02-07 16:29:23 UTC - Sijie Guo: Arrow is more about the data format on the wire. It can be potentially adopted by integrating into Pulsar Schema.
----
2020-02-07 16:35:18 UTC - Abhilash Mandaliya: @Abhilash Mandaliya has joined the channel
----
2020-02-07 18:00:05 UTC - Roman Popenov: That would be a great feature indeed
----
2020-02-07 18:03:25 UTC - Vladimir Shchur: @Sijie Guo In the ticket I sent the modifications, and set quorum to 1 after your comment, but it doesn't work.
----
2020-02-07 18:03:38 UTC - Vladimir Shchur: Only if you leave 3 bookies it works (there was no such a problem with 2.4.* releases)
----
2020-02-07 18:19:19 UTC - Christoffer Soop: Hi, our ERP people are dead set on using ETL for integration with their backends. Does anybody know if Informatica supports using Pulsar?
----
2020-02-07 19:17:41 UTC - Eric Simon: @juraj What did you to fix this?
----
2020-02-07 19:19:00 UTC - Yuri de Wit: ```Apache Arrow is a cross-language development platform for in-memory data. It specifies a standardized language-independent columnar memory format for flat and hierarchical data, organized for efficient analytic operations on modern hardware. It also provides computational libraries and zero-copy streaming messaging and interprocess communication```
This is from Apache Arrow, which seems to indicate that is it not only an on the wire format, but also a storage and in-memory format.
----
2020-02-07 19:22:51 UTC - Yuri de Wit: It seems to me that it could be useful speeding up segment processing in brokers reading/writing segments, replication across brokers, migration to second tier storage, and last but not least prestoDB. I say "speeding up" but take that with a huge grain of salt: I have no idea how fast Arrow or Bookkeeper for that matter is. The interesting aspect is the single, efficient format for all kinds of integrations
----
2020-02-07 19:23:50 UTC - Yuri de Wit: Sorry for the open ended question ... I am arriving here at Pulsar after being a bit scared of the complexity of Kafka and found here much more than I was expecting.
----
2020-02-07 23:28:50 UTC - Aditya Vishwakarma: @Aditya Vishwakarma has joined the channel
----
2020-02-07 23:44:22 UTC - Aditya Vishwakarma: hi, does anyone know how to resolve Python segmentation fault issue when using pyenv?
----
2020-02-07 23:45:11 UTC - Matteo Merli: It's problably going to be related to the version of python inside the pyenv
----
2020-02-07 23:46:17 UTC - Roman Popenov: I can confirm that it is. I fidgeted with it for some time a while back, and then it just magically worked.
----
2020-02-07 23:46:20 UTC - Aditya Vishwakarma: I had python 3.7.6 installed via pyenv. It segfaults.
however, python 3.7.6 system install via brew worked
----
2020-02-07 23:47:55 UTC - Roman Popenov: I installed via brew and it worked, but then trying the same version inside my IDEA it was seg faulted
----
2020-02-07 23:48:19 UTC - Roman Popenov: And then it worked on Ubuntu machine without issues
----
2020-02-07 23:48:41 UTC - Roman Popenov: This did happen when I also had multiple versions of Python installed on the machine
----
2020-02-07 23:51:08 UTC - Roman Popenov: I did manage work the libraries with python 2.7 and 3.7, so it’s not pulsar libraries
----
2020-02-07 23:52:23 UTC - Aditya Vishwakarma: Wish I had your luck. Been fiddling with this for past hour and can't get pyenv + poetry setup to work.
----
2020-02-07 23:53:11 UTC - Roman Popenov: I think I compiled libraries on my mac, compiled the cpp client with cmake, it was a complicated battle I remember
----
2020-02-07 23:54:15 UTC - Roman Popenov: Funny as it may sound, have you tried a slightly older version of python :thinking_face::joy:
----
2020-02-07 23:54:53 UTC - Aditya Vishwakarma: I tried python 3.7.1 in pyenv. it segfaulted :disappointed:
----
2020-02-08 00:34:54 UTC - Sijie Guo: Arrow is more about data format. It can be used for wire, memory and storage. BookKeeper is the actual log storage system. These are two things at different level. BookKeeper or Pulsar can use Arrow for its data format at some extends.
arrow_upper_right : Roman Popenov
+1 : Yuri de Wit
----
2020-02-08 00:49:28 UTC - Matteo Merli: It might then be related with the fix done in <https://github.com/apache/pulsar/pull/5581|https://github.com/apache/pulsar/pull/5581>
----
2020-02-08 00:59:39 UTC - Aditya Vishwakarma: any way to try it?
----
2020-02-08 01:11:31 UTC - Aditya Vishwakarma: tried an older version of pulsar-client, 2.4.2. Didn't work as well.
----
2020-02-08 01:26:40 UTC - Roman Popenov: Just in venv?
----
2020-02-08 01:27:00 UTC - Aditya Vishwakarma: yeah.
----
2020-02-08 03:26:24 UTC - Aditya Vishwakarma: I give up. Filed an issue
<https://github.com/apache/pulsar/issues/6269>
----