You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@pulsar.apache.org by Apache Pulsar Slack <ap...@gmail.com> on 2018/07/13 09:11:03 UTC
Slack digest for #general - 2018-07-13
2018-07-12 10:41:48 UTC - Idan: @Matteo Merli following our last talk regarding warnings you asked for stats:
----
2018-07-12 10:41:53 UTC - Idan: @Idan uploaded a file: <https://apache-pulsar.slack.com/files/UALJD8929/FBPEZ6W4D/__pulsar-admin_topics_stats_persistent___pulsar-pulsar-cluster_public_default_PlayGameQueue.php|./pulsar-admin topics stats persistent://pulsar-pulsar-cluster/public/default/PlayGameQueue>
----
2018-07-12 10:43:53 UTC - Idan: seeing this for every msg i consume: 2018-07-12 10:43:32,663 WARN org.apache.pulsar.client.impl.UnAckedMessageTracker - [ConsumerBase{subscription=‘subscr-jackpot’, consumerName=‘0df46’, topic=‘<persistent://pulsar-pulsar-cluster/public/default/PlayGameQueue’>}] 30 messages have timed-out
----
2018-07-12 10:44:56 UTC - Idan: @Idan uploaded a file: <https://apache-pulsar.slack.com/files/UALJD8929/FBNPWHU8Z/__pulsar-admin_topics_stats-internal_persistent___pulsar-pulsar-cluster_public_default_PlayGameQueue.pl|./pulsar-admin topics stats-internal persistent://pulsar-pulsar-cluster/public/default/PlayGameQueue>
----
2018-07-12 15:21:36 UTC - Joaz: @Sijie Guo @jia zhai I’ve fixed the issue.
Yes, the issue was related to the ports security rule on AWS.
----
2018-07-12 15:22:14 UTC - Joaz: I’ve updated the ansible script, and updated the version of the pulsar binary too. I will create a PR in FYI.
----
2018-07-12 15:22:30 UTC - Joaz: Thanks for the help btw :slightly_smiling_face:
+1 : jia zhai, Sijie Guo, Matteo Merli
----
2018-07-12 15:25:13 UTC - Karthik Palanivelu: Team, Can you please help me here?
```
Karthik Palanivelu [10:57 AM]
Team, I see bookie metrics in Prometheus but not the Broker and ZK. In broker I have enabled `exposeTopicLevelMetricsInPrometheus=true` but still I do not see it. Please let me know how I can enable the metrics of Broker and ZK to Prometheus.
```
----
2018-07-12 15:25:52 UTC - Grant Wu: It’s somewhat unclear who you’re addressing when you say “Team”…
----
2018-07-12 15:59:47 UTC - Matteo Merli: @Karthikeyan Palanivelu Promethes is “pull-based”, so typically it’s configured with the list of target hosts where it has to fetch the metrics from (or a mechanism to discover automatically the list of targets). Is this using the Kubernetes profile for Prometheus?
----
2018-07-12 16:12:50 UTC - Karthik Palanivelu: I request to anyone from Pulsar team to help
----
2018-07-12 16:16:41 UTC - Karthik Palanivelu: Yes I am using the Kubernetes default prometheus. Its got the bookie metrics though.
----
2018-07-12 16:17:07 UTC - Joaz: Guys, another question jeje
----
2018-07-12 16:17:08 UTC - Joaz: <https://pulsar.incubator.apache.org/docs/latest/clients/Java/>
----
2018-07-12 16:17:27 UTC - Joaz: This is a full java client? or is supposed to be used on Android projects?
----
2018-07-12 16:18:18 UTC - Matteo Merli: Based on the K8S definition, it should probe all the pods with the annotations: <https://github.com/apache/incubator-pulsar/blob/master/deployment/kubernetes/generic/monitoring.yaml#L39>
For example, in broker.yaml we have <https://github.com/apache/incubator-pulsar/blob/master/deployment/kubernetes/generic/broker.yaml#L48>
----
2018-07-12 16:18:52 UTC - Matteo Merli: It’s “full” java, it depends on Java 8, which I believe is not available in Android
----
2018-07-12 16:22:08 UTC - Karthik Palanivelu: I have that config in all my yamls, but I see only bookie metrics and none for Broker
----
2018-07-12 16:23:20 UTC - Matteo Merli: Ok, if you go on the prometheus admin page, you should be able to see all the targets it’s pulling
----
2018-07-12 16:23:42 UTC - Matteo Merli: That would be at `http://$PROMETHEUS_POD:9090`
----
2018-07-12 16:26:20 UTC - Joaz: Ohhh
----
2018-07-12 16:26:23 UTC - Joaz: :disappointed:
----
2018-07-12 16:26:33 UTC - Joaz: I wanted to use it on an Android project.
----
2018-07-12 16:28:11 UTC - Karthik Palanivelu: Sure let me check Matt, Thank you
----
2018-07-12 16:34:51 UTC - Matteo Merli: A good option would be to use WebSocket API
----
2018-07-12 18:09:52 UTC - Grant Wu: What is a consumer name? Is it just for debugging purposes?
<https://pulsar.incubator.apache.org/api/client/org/apache/pulsar/client/api/ConsumerBuilder.html#consumerName-java.lang.String->
----
2018-07-12 18:11:15 UTC - Karthik Palanivelu: @Matteo Merli I updated the port of Prometheus on Annotation to my http port of 9101 worked.
----
2018-07-12 18:11:45 UTC - Karthik Palanivelu: But I do not know on which port zookeeper exposes it, can you please help?
----
2018-07-12 18:12:07 UTC - Matteo Merli: It would be 8080 by default
----
2018-07-12 18:15:56 UTC - Grant Wu: Additionally, is it just me or is the example picture here: <https://pulsar.incubator.apache.org/docs/latest/functions/overview/#Wordcountexample-et4a8>
wrong? (there are two ‘“in” counter’ pictures)
----
2018-07-12 18:17:16 UTC - Matteo Merli: It can be used for debugging. By default consumers are assigned a random name. In failover subscriptions, it can also determine which is the “active” consumer by preference. Consumers are sorted alphabetically by consumer name and the first one is chosen as active.
----
2018-07-12 18:20:43 UTC - Grant Wu: Also, does the WS API support multi-topic subscriptions? I don’t actually need them, just trying to figure to map the concepts and architecture docs onto the code…
----
2018-07-12 18:21:42 UTC - Matteo Merli: No, it’s not currently possible through WebSocket. There is a feature matrix at <https://github.com/apache/incubator-pulsar/wiki/Client-Features-Matrix>
pray : Grant Wu
----
2018-07-12 18:22:05 UTC - Grant Wu: oh, this is very helpful
----
2018-07-12 18:23:19 UTC - Grant Wu: Hrm - as an alternative to using `resetCursor` to a timestamp, do you think it would be possible to get Consumer `seek` to a timestamp?
----
2018-07-12 18:27:06 UTC - Grant Wu: Would I need to write up a PIP for that?
+1 : Sijie Guo
----
2018-07-12 18:27:26 UTC - Matteo Merli: Yes, it should be easy to add — The feature is already implemented through REST API, it would be necessary to just allow that in the wire-protocol
----
2018-07-12 18:30:12 UTC - Grant Wu: Great!
----
2018-07-12 18:30:22 UTC - Grant Wu: I uh… will try to get to that eventually
----
2018-07-12 18:30:38 UTC - Matteo Merli: Sure, that would be nice to submit a design doc! :slightly_smiling_face:
----
2018-07-12 18:30:55 UTC - Grant Wu: This extremely silly WS client library we’re using internally at work uses a single subscription per topic :man-facepalming:
----
2018-07-12 18:31:21 UTC - Grant Wu: Need to rewrite this first
----
2018-07-12 18:59:35 UTC - Karthik Palanivelu: Let me try that port
----
2018-07-12 19:34:29 UTC - Karthik Palanivelu: No Matt not getting any date for ZK...May be I will run a load test and see if it populates any
----
2018-07-12 19:44:44 UTC - Matteo Merli: Can you check the “status” section in Prometheus? Do the ZK targets show up
----
2018-07-12 20:37:55 UTC - Karthik Palanivelu: Let me check
----
2018-07-12 22:55:51 UTC - Grant Wu: Is there an undocumented C client library?
----
2018-07-12 22:56:04 UTC - Grant Wu: Someone at my company is claiming that there is one
----
2018-07-12 22:56:37 UTC - Grant Wu: Could they possibly be referring to <https://pulsar.incubator.apache.org/docs/latest/clients/Cpp/> “Static library libpulsar.a and C++ and *C headers*”
----
2018-07-12 22:56:55 UTC - Ali Ahmed: There is a C++ client
----
2018-07-12 22:57:16 UTC - Ali Ahmed: it’s mostly used as a base for languages such as python and go
----
2018-07-12 22:58:10 UTC - Grant Wu: Right, I know there’s an official C++ client
----
2018-07-12 22:58:21 UTC - Grant Wu: But someone at my company is saying that they’re using some C client library
----
2018-07-12 22:58:36 UTC - Grant Wu: I’m… trying to figure out what they mean
----
2018-07-12 22:58:50 UTC - Ali Ahmed: I guess they are talking about the same
----
2018-07-12 23:23:25 UTC - Matteo Merli: @Grant Wu We have added C API as it was required to build Go client wrapper
----
2018-07-12 23:23:44 UTC - Matteo Merli: C API is basically a plain C wrapper on top of existing C++
----
2018-07-12 23:23:55 UTC - Grant Wu: Okay, so it does actually exist.
----
2018-07-12 23:24:02 UTC - Matteo Merli: it does indeed
----
2018-07-12 23:24:41 UTC - Grant Wu: Is it the same thing as the C headers mentioned in the link above?
----
2018-07-12 23:24:48 UTC - Grant Wu: Is there anything we should know about using it
----
2018-07-12 23:25:27 UTC - Matteo Merli: there are examples, like: <https://github.com/apache/incubator-pulsar/blob/master/pulsar-client-cpp/examples/SampleProducerCApi.c>
----
2018-07-12 23:25:48 UTC - Matteo Merli: for the rest, you just link with the same `libpulsar.so`
----
2018-07-12 23:27:20 UTC - Grant Wu: I meant more like, pitfalls
----
2018-07-12 23:27:49 UTC - Grant Wu: e.g. how likely is that API to change, etc.
----
2018-07-12 23:28:36 UTC - Matteo Merli: not very likely to change, since it’s out there, we’ll not break it
+1 : Grant Wu, Nicolas Ha
----
2018-07-12 23:56:39 UTC - Matteo Merli: @Grant Wu In any case, what is the reason for C compared to C++ ?
----
2018-07-12 23:57:03 UTC - Grant Wu: I’ll be honest, I have no clue
----
2018-07-12 23:57:37 UTC - Grant Wu: I’m just one person who just started working at a relatively disorganized startup
----
2018-07-12 23:58:01 UTC - Grant Wu: I don’t get to make the technical decisions here… just trying to make sure that nothing around me goes up in flames
----
2018-07-13 00:09:24 UTC - Matteo Merli: :smile:
----
2018-07-13 00:35:46 UTC - Grant Wu: the context in which this came up was "wait what version of Pulsar are we running? The WS API changed in 2.0.0 so I need to know what to code against". --me
"Yeah we're going to use the latest. By the way, I'm using the C API and if that's not in 2.0.1 we'll be deploying master" -- some other person
----
2018-07-13 00:37:43 UTC - Yuwei Jiang: Hi guys, for deploying Pulsar 2.0 on K8s on AWS, what are the recommended resource settings to set on the deployment yaml files for components `zookeeper`, `bookkeeper`, `broker` , `proxy` , `dashboard` and `grafana` ?
I checked helm chart for GCE deployment on the `master` branch in the `deployment/kubernetes/helm/pulsar/values.yaml` , there is memory resource requests of `15Gi` for `zookeeper` `bookkeeper` and `broker` ; `4Gi` for `proxy`.
The reason I’m asking is that I’ve been encountering some issues with bookkeeper/broker/proxy restarts 7-8 hours after deployment, when I deployed Pulsar-2.0 on K8s on AWS (1 master node and 3 worker node, all pods are deployed on worker node). I’m using `m4.large` instance which has 2vCPU and 8Gi of memory; wondering should I move to `m4.xlarge` instance with double the CPU and memory.
----
2018-07-13 00:43:30 UTC - Byron: What do you mean by restarts? Are you running into out of memory errors?
----
2018-07-13 00:50:21 UTC - Yuwei Jiang: zookkeeper bookkeeper, broker, and proxy pods have multiple restarts, zookeeper-0 pod was terminated with 137 error code (checked with `kubectl describe pod` command); bookkeeper pod was terminated with error code 2;
----
2018-07-13 00:50:27 UTC - Yuwei Jiang: ```
bookkeeper-0 1/1 Running 477 2d
bookkeeper-1 1/1 Running 390 3d
bookkeeper-autorecovery-54f97cb48c-bjcfd 1/1 Running 2 2d
bookkeeper-autorecovery-54f97cb48c-v7r48 1/1 Running 393 2d
broker-6fb6948c87-522lm 1/1 Running 0 2d
broker-6fb6948c87-6qvg5 1/1 Running 480 2d
broker-6fb6948c87-72tng 1/1 Running 488 2d
dashboard-5cfd6c9bdd-c8kbj 1/1 Running 5 2d
proxy-7b4f9984df-flnhv 1/1 Running 7 2d
proxy-7b4f9984df-nj2db 1/1 Running 7 2d
proxy-7b4f9984df-nr2rm 1/1 Running 0 2d
grafana-586749dffd-jxsgc 1/1 Running 7 2d
prometheus-759c98d764-cnr86 1/1 Running 0 2d
zookeeper-0 1/1 Running 7 3d
zookeeper-1 1/1 Running 0 2d
zookeeper-2 1/1 Running 0 9h
```
----
2018-07-13 00:51:10 UTC - Yuwei Jiang: I’m wondering whether it is caused by OOM issue.
----
2018-07-13 00:53:53 UTC - Byron: I experienced that quite a bit before realizing the underlying issue (I am not very knowledgeable of the JVM and common settings). Did you adjust max memory for the different processes relative to the instances you provisioned?
----
2018-07-13 00:54:25 UTC - Byron: e.g. `-Xms1g -Xmx2g -XX:MaxDirectMemorySize=2g`
----
2018-07-13 00:55:34 UTC - Byron: Otherwise, k8s will happily kill the containers
----
2018-07-13 00:55:51 UTC - Byron: if the limits dont match what the JVM has been given
----
2018-07-13 00:56:05 UTC - Byron: let alone what the AWS nodes are provisioned for
----
2018-07-13 00:56:59 UTC - Byron: i am giving roughly 256MB or so as overhead to the container above the JVM limit
----
2018-07-13 00:57:11 UTC - Byron: and things have been stable for me
----
2018-07-13 01:00:55 UTC - Byron: for your more general question of what the memory or CPU should be, that just depends on the throughput and latency requirements (other things influence this as well)
----
2018-07-13 01:46:07 UTC - Ali Ahmed: @Yuwei Jiang what’s the messaging load on the cluster ?
----
2018-07-13 03:02:17 UTC - Yuwei Jiang: Hi @Byron, Thanks for the suggestion! I will check the JVM mem settings on my deployment yaml.
Hi @Ali Ahmed, I have a db app that is subscribing/publishing to Pulsar, but the db is not under any active load.
----
2018-07-13 04:57:21 UTC - Idan: @Matteo Merli did you had a chanse to look at my output as requested?
----