You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@pulsar.apache.org by Apache Pulsar Slack <ap...@gmail.com> on 2019/12/03 09:11:03 UTC
Slack digest for #general - 2019-12-03
2019-12-02 09:11:17 UTC - Endre Karlson: Dang, I wanna have it :smile:
----
2019-12-02 09:11:39 UTC - jia zhai: :thumbsup_all:
----
2019-12-02 09:12:01 UTC - Endre Karlson: not faster ? :wink: I thought it was scoped for 2.5
----
2019-12-02 09:12:08 UTC - hejinbiao: @hejinbiao has joined the channel
----
2019-12-02 10:24:31 UTC - Randal: @Randal has joined the channel
----
2019-12-02 11:01:43 UTC - Christophe Bornet: @Sijie Guo monotonically increasing index for the messages in a partitions would certainly be helpful if they are exposed in stats. Now I don't know how to link it to consumers as unlike Kafka partitions are not bound to consumers (which is actually great :smile:)
----
2019-12-02 15:41:15 UTC - Sijie Guo: in Failover and Exclusive subscriptions, partitions are bound to consumers.
----
2019-12-02 15:47:12 UTC - Brian Doran: Having a few issues getting the metrics to with Pulsar standalone (docker) / Prometheus (docker) / grafana (docker)
Using both pulsar doc pages with grafana (apachepulsar/pulsar-grafana) and here: <https://github.com/streamnative/apache-pulsar-grafana-dashboard>
I don't get any stats in the UI although I have added details to prometheus.yml for pulsar/bookie/zk It seems that port 8000 is inaccessible when running standalone? Is this the case?
9d447b990c6b apachepulsar/pulsar:2.4.0 "bin/pulsar standalo…" 2 hours ago Up 25 minutes 0.0.0.0:6650->6650/tcp, 0.0.0.0:8000->8000/tcp, 0.0.0.0:8080->8080/tcp pulsar
df0db820b176 streamnative/apache-pulsar-grafana-dashboard:latest "/pulsar_grafana_ent…" 2 hours ago Up 2 hours 0.0.0.0:3000->3000/tcp cranky_gagarin
711c5f5b702c prom/prometheus "/bin/prometheus --c…" 3 hours ago Up 26 minutes 0.0.0.0:9090->9090/tcp prometheus
----
2019-12-02 15:49:15 UTC - Brian Doran: I get the following error from <http://pulsar-host:9090/targets>
pulsar-bookies (0/1 up)
Endpoint State Labels Last Scrape Scrape Duration Error
<http://bookie-host:8000/metrics>
DOWN instance="bookie-host:8000" job="pulsar-bookies" 3.633s ago 1.975ms Get <http://bookie-host:8000/metrics>: read tcp 172.17.0.4:48266->172.20.133.157:8000: read: connection reset by peer
----
2019-12-02 15:51:40 UTC - Brian Doran: blank dashboard although there are 22 topics with data going through
----
2019-12-02 16:20:58 UTC - Damian Franetovich: @Damian Franetovich has joined the channel
----
2019-12-02 16:54:10 UTC - Joshua Dunham: Hey Everyone, Regarding the schema registry... can someone give me the quick overview of how it compares to Hortonworks Schema Registry. I'm looking to integrate Pulsar with NiFi (and other Apache apps) and wonder if there is functional or practical overlap.
----
2019-12-02 16:56:20 UTC - Joshua Dunham: They describe use with Kafka. <https://registry-project.readthedocs.io/en/latest/schema-registry.html#quickstart>
----
2019-12-02 17:10:38 UTC - Joshua Dunham: Those were good write-ups. Found another feature I wasn't aware of in use cases section on top of record info. :slightly_smiling_face:
----
2019-12-02 17:10:59 UTC - Joshua Dunham: I'm going to try again with Avro (my end case). Is there anything I can assist with, testing or the like?
----
2019-12-02 17:11:46 UTC - David Kjerrumgaard: @Joshua Dunham In terms of what each schema registry does, there is quite a bit of functional overlap. Both are repositories for schemas (Avro, JSON, CSV). However, the biggest difference is the target system, e.g. NiFi vs. Pulsar.
----
2019-12-02 17:12:28 UTC - David Kjerrumgaard: So the Pulsar registry is used by Pulsar to determine how to serialize/deserialize messages that are published and consumed from Pulsar
----
2019-12-02 17:12:42 UTC - Joshua Dunham: NiFi has a mechanism to hook to an external registry. Does/can pulsar expose the same API?
----
2019-12-02 17:13:18 UTC - Joshua Dunham: I can infer that if I'm using Pulsar and the schema registry there is happy the message conform correctly in practical use.
----
2019-12-02 17:13:24 UTC - David Kjerrumgaard: The Pulsar schema registry uses a REST API which is documented here: <https://pulsar.apache.org/docs/en/admin-api-schemas/>
----
2019-12-02 17:13:36 UTC - Joshua Dunham: Oh, nice!
----
2019-12-02 17:21:08 UTC - Benoit SENARD: @Benoit SENARD has joined the channel
----
2019-12-02 17:28:38 UTC - Jim Fan: @Jim Fan has joined the channel
----
2019-12-02 20:02:56 UTC - Eduardo Caceres: @Eduardo Caceres has joined the channel
----
2019-12-02 20:03:45 UTC - Eduardo Caceres: Hi, I am new to this channel. I have a qq, does pulsar support kafka 0.7?
----
2019-12-02 20:15:03 UTC - David Kjerrumgaard: @Eduardo Caceres What do you mean by "support" ? Pulsar uses a different wire protocol than Kafka, but the API is 100% compatible, so you can convert your Kafka 0.7 application to Pulsar easily, <http://pulsar.apache.org/docs/en/adaptors-kafka/>
----
2019-12-02 21:12:10 UTC - Ali Ahmed: <https://www.tibco.com/products/apache-pulsar>
thunder_cloud_and_rain : Matteo Merli
bananadance : Karthik Ramasamy, Ali Ahmed
----
2019-12-02 21:32:19 UTC - Addison Higham: open question: anyone have ways they are monitoring pulsar functions? We run pulsar functions via the kubernetes integration and see how we can get metrics from functions, but AFAICT, there isn't metrics exposed for the scheduler?
----
2019-12-02 22:07:26 UTC - chris: the function spec should have annotations for prometheus so if you are running prometheus all the function metrics will be collected automatically
----
2019-12-02 22:09:13 UTC - Addison Higham: yeah, I know how to get those, but that won't easily be able to catch a function that is crashing or a function that is struggling to get scheduled
----
2019-12-02 22:09:41 UTC - chris: there is an up metric that can be used to tell if a pod is up or not
----
2019-12-02 22:10:15 UTC - Addison Higham: yeah, just grabbing the pod health metrics from the stateful set might be good enough
----
2019-12-02 22:15:36 UTC - Christophe Bornet: Indeed, all partitions go to the same consumer
----
2019-12-02 22:16:16 UTC - Christophe Bornet: I was thinking of Key_shared subscriptions
----
2019-12-03 00:06:56 UTC - tuteng: Have you configured this file <https://github.com/streamnative/apache-pulsar-grafana-dashboard/blob/master/prometheus/standalone.yml.template>
----
2019-12-03 00:08:34 UTC - tuteng: This file should be configured for prometheus
----
2019-12-03 00:09:38 UTC - tuteng: First: you can try
```curl -v <http://pulsar-broker-ip:8000/metrics/>```
----
2019-12-03 06:44:22 UTC - Masahiko Sato: @Masahiko Sato has joined the channel
----
2019-12-03 06:49:24 UTC - Sijie Guo: @Sijie Guo set the channel topic: Pulsar graduated as a Top-Level project :tada:
- Pulsar manager release 0.1.0: <http://pulsar.apache.org/docs/en/next/administration-pulsar-manager/>
- Pulsar release 2.4.1: <http://pulsar.apache.org/release-notes/#2.4.1>
+1 : Ali Ahmed, Devin G. Bost
----