You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@pulsar.apache.org by Apache Pulsar Slack <ap...@gmail.com> on 2019/05/07 09:11:03 UTC
Slack digest for #general - 2019-05-07
2019-05-06 09:17:09 UTC - Justin: @Sijie Guo Thanks, I will try.
----
2019-05-06 10:43:45 UTC - lan: ```
java.lang.NoClassDefFoundError: Could not initialize class org.apache.pulsar.broker.stats.prometheus.PrometheusMetricsGenerator`
```
maybe some jar conflict?
----
2019-05-06 10:51:10 UTC - gfouquier: Thanks. I removed all other pulsar versions, but it doesn't change anything.
----
2019-05-06 13:51:51 UTC - Alexandre DUVAL: Hi,
----
2019-05-06 13:52:08 UTC - Alexandre DUVAL:
----
2019-05-06 13:54:07 UTC - Alexandre DUVAL: (I have no schema on this topic)
----
2019-05-06 13:56:20 UTC - Alexandre DUVAL: Or schemas are mandatory on pulsar's functions topics?
----
2019-05-06 14:00:41 UTC - Alexandre DUVAL: This is error on broker. And the error on functions_woker node is:
----
2019-05-06 14:01:28 UTC - Alexandre DUVAL: ```13:33:38.828 [yo/functions/RouteApplicationsAddonsLogs-0] INFO org.apache.pulsar.functions.instance.JavaInstanceRunnable - Load JAR: /tmp/pulsar_functions/yo/functions/RouteApplicationsAddonsLogs/0/pulsar-functions-0.1.0-SNAPSHOT.jar
13:33:38.881 [yo/functions/RouteApplicationsAddonsLogs-0] WARN org.apache.pulsar.common.nar.NarClassLoader - /tmp/pulsar-nar/pulsar-functions-0.1.0-SNAPSHOT.jar-unpacked does not contain META-INF/bundled-dependencies!
13:33:38.882 [yo/functions/RouteApplicationsAddonsLogs-0] INFO org.apache.pulsar.functions.instance.JavaInstanceRunnable - Initialize function class loader for function RouteApplicationsAddonsLogs at function cache manager
13:33:38.890 [yo/functions/RouteApplicationsAddonsLogs-0] INFO org.apache.pulsar.functions.sink.PulsarSink - Opening pulsar sink with config: PulsarSinkConfig(processingGuarantees=ATLEAST_ONCE, topic=<persistent://public/default/count>, serdeClassName=null, schemaType=null, typeClassName=java.lang.Void)
13:33:38.890 [yo/functions/RouteApplicationsAddonsLogs-0] INFO org.apache.pulsar.functions.sink.PulsarSink - Since output type is null, not creating any real sink
13:33:38.897 [yo/functions/RouteApplicationsAddonsLogs-0] INFO org.apache.pulsar.functions.source.PulsarSource - Opening pulsar source with config: PulsarSourceConfig(processingGuarantees=ATLEAST_ONCE, subscriptionType=Shared, subscriptionName=yo/functions/RouteApplicationsAddonsLogs, maxMessageRetries=-1, deadLetterTopic=null, topicSchema={<persistent://yo/logs/full=ConsumerConfig(schemaType=null>, serdeClassName=null, isRegexPattern=false, receiverQueueSize=null)}, typeClassName=java.lang.String, timeoutMs=null)
13:33:39.029 [pulsar-client-io-1-1] INFO org.apache.pulsar.client.impl.ConnectionPool - [[id: 0xaca27b8e, L:/10.2.1.4:53728 - R:<http://c1-pulsar-yo-customers.services.clever-cloud.com/46.252.181.140:2001|c1-pulsar-yo-customers.services.clever-cloud.com/46.252.181.140:2001>]] Connected to server
13:33:39.075 [pulsar-client-io-1-1] WARN org.apache.pulsar.client.impl.ClientCnx - [id: 0xaca27b8e, L:/10.2.1.4:53728 - R:<http://c1-pulsar-yo-customers.services.clever-cloud.com/41.252.181.140:2001|c1-pulsar-yo-customers.services.clever-cloud.com/41.252.181.140:2001>] Received error from server: org.apache.pulsar.client.api.PulsarClientException: Disconnected from server at yo-pulsar-c1-n3/192.168.10.12:2008
13:33:39.075 [pulsar-client-io-1-1] WARN org.apache.pulsar.client.impl.ClientCnx - [id: 0xaca27b8e, L:/10.2.1.4:53728 - R:<http://c1-pulsar-yo-customers.services.clever-cloud.com/41.252.181.140:2001|c1-pulsar-yo-customers.services.clever-cloud.com/41.252.181.140:2001>] Received unknown request id from server: 0 ```
----
2019-05-06 14:40:24 UTC - Alexandre DUVAL: @Sijie Guo maybe? :smile:
----
2019-05-06 14:44:42 UTC - Sijie Guo: what content do you have in the input topic? utf-8 encoded string?
----
2019-05-06 14:47:55 UTC - Alexandre DUVAL: yes
----
2019-05-06 15:04:41 UTC - Sijie Guo: it seems that it is using a `latest` version to fetch schema info. which causes the exception.
----
2019-05-06 15:04:45 UTC - Sijie Guo: what version are you using?
----
2019-05-06 15:16:05 UTC - Sijie Guo: how do you write the strings to input topics?
----
2019-05-06 15:21:57 UTC - Alexandre DUVAL: v.2.4.0-snapshot
----
2019-05-06 15:31:12 UTC - Alexandre DUVAL: I'm custom my own logstash output plugin, I will precise produce String.class.
----
2019-05-06 15:31:26 UTC - Alexandre DUVAL: <https://github.com/CleverCloud/logstash-output-pulsar>
----
2019-05-06 15:53:03 UTC - Alexandre DUVAL: @Sijie Guo I added this <https://github.com/CleverCloud/logstash-output-pulsar/blob/master/lib/logstash/outputs/pulsar.rb#L182> to force String type.
----
2019-05-06 15:53:12 UTC - Alexandre DUVAL: But now I've got ```May 06 15:51:25 logs-collector logstash[2722]: [2019-05-06T15:51:25,182][WARN ][org.apache.pulsar.client.impl.ConnectionHandler] [<persistent://yo/logs/full>] [null] Error connecting to broker: org.jruby.exceptions.RaiseException: (NoMethodError) undefined method `getSchemaInfo' for Class:Class
May 06 15:51:25 logs-collector logstash[2722]: [2019-05-06T15:51:25,184][WARN ][org.apache.pulsar.client.impl.ConnectionHandler] [<persistent://yo/logs/full>] [null] Could not get connection to broker: org.jruby.exceptions.RaiseException: (NoMethodError) undefined method `getSchemaInfo' for Class:Class -- Will try again in 0.1 s
```
----
2019-05-06 15:53:17 UTC - Alexandre DUVAL: On producer.
----
2019-05-06 16:13:33 UTC - Alexandre DUVAL: oh ma bad, StringSchema :slightly_smiling_face:
----
2019-05-06 16:22:42 UTC - David Kjerrumgaard: @Alexandre DUVAL The issue is that you are trying to define a Pulsar Sink that uses and output type of Void, which doesn't make sense. If you are not generating a value, then it can't be used as a sink. `Since output type is null, not creating any real sink `
----
2019-05-06 16:23:15 UTC - David Kjerrumgaard: Sinks need output types
----
2019-05-06 16:39:13 UTC - Himay Desai: @Himay Desai has joined the channel
----
2019-05-06 17:35:12 UTC - Per Magnus: @Per Magnus has joined the channel
----
2019-05-06 20:49:45 UTC - Nicolas Ha: playing with pulsar SQL, I struggle to understand how schemas are created. I have a `org.apache.avro.Schema$RecordSchema` class, that contains something like that: <https://github.com/sijie/pulsar-avro-schema-example/blob/master/src/main/avro/twitter.avsc>
How can I create a `newProducer`? Passing it straight away results in ```Cannot cast org.apache.avro.Schema$RecordSchema to org.apache.pulsar.client.api.Schema```
----
2019-05-06 20:50:58 UTC - Nicolas Ha: or is there something easier in Java? My problem may be that I don’t have POJOs
----
2019-05-06 21:24:29 UTC - Nicolas Ha: I can build `SchemaData` too from Avro, but still can’t seem to create a `org.apache.pulsar.client.api.Schema` from it
----
2019-05-06 22:07:05 UTC - Nicolas Ha: ah looks like I need to upgrade my client lib - it looks like the client `setMessageListener` is gone?
----
2019-05-06 22:08:11 UTC - Nicolas Ha: So if I want to do the same, I now need to use `receiveAsync` and loop myself, is that correct?
----
2019-05-06 23:19:44 UTC - Nicolas Ha: seems that `receiveAsync` does the trick (at least my tests pass).
For presto, it seems there is a `withJsonDef` that could fit the bill but I haven’t tried yet (will have to come back to this later)
----
2019-05-07 02:06:56 UTC - Sijie Guo: ok
----
2019-05-07 02:46:04 UTC - Justin: I tried to restart the broker and bookie in my cluster when I doing a performance test, the thing confused me is that the broker and bookie are in a low CPU load after the restart, so I would like to know what are they are doing now? Any replies will be appreciated.
----
2019-05-07 03:36:46 UTC - yingsunjiu: @yingsunjiu has joined the channel
----
2019-05-07 04:58:07 UTC - Akash G: @Akash G has joined the channel
----
2019-05-07 08:21:44 UTC - Akash G: I have pulsar `2.3.0` deployed on my k8s cluster. I am using Prometheus operator for getting the metrics but for the broker, Prometheus is unable to scrape the metrics it is getting the error `expected equal, got "INVALID"` from the broker pod. If anyone has faced this issue please help. I tried using the latest Prometheus version but the issue still persists.
----