You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@pulsar.apache.org by Apache Pulsar Slack <ap...@gmail.com> on 2019/12/04 09:11:05 UTC
Slack digest for #general - 2019-12-04
2019-12-03 09:32:04 UTC - Nuno Ferreira: @Nuno Ferreira has joined the channel
----
2019-12-03 10:45:17 UTC - Nuno Ferreira: Hi all. I'm trying to develop a pulsar function but currently the function only works in localrun mode. When I run the function *on localrun mode it works as intended*. But when I run it in cluster mode I get the following stack trace:
10:21:10.463 [function-timer-thread-86-1] ERROR org.apache.pulsar.functions.runtime.ProcessRuntime - Health check failed for ContextFunction-0
java.util.concurrent.ExecutionException: io.grpc.StatusRuntimeException: UNAVAILABLE: io exception
at java.util.concurrent.CompletableFuture.reportGet(CompletableFuture.java:357) ~[?:1.8.0_222]
(....)
Caused by: io.grpc.StatusRuntimeException: UNAVAILABLE: io exception
at io.grpc.Status.asRuntimeException(Status.java:530) ~[io.grpc-grpc-core-1.18.0.jar:1.18.0]
(....)
... 1 more
Caused by: io.netty.channel.AbstractChannel$AnnotatedConnectException: Connection refused: /127.0.0.1:56999
at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) ~[?:1.8.0_222]
(....)
... 2 more
Caused by: <http://java.net|java.net>.ConnectException: Connection refused
at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) ~[?:1.8.0_222]
(....)
... 2 more
10:21:10.471 [function-timer-thread-86-1] ERROR org.apache.pulsar.functions.runtime.ProcessRuntime - Extracted Process death exception
java.lang.RuntimeException:
at org.apache.pulsar.functions.runtime.ProcessRuntime.tryExtractingDeathException(ProcessRuntime.java:380) ~[org.apache.pulsar-pulsar-functions-runtime-2.4.1.jar:2.4.1]
(.....)
10:21:10.472 [function-timer-thread-86-1] ERROR org.apache.pulsar.functions.runtime.RuntimeSpawner - public/default/ContextFunction-java.lang.RuntimeException: Function Container is dead with exception.. restarting
I'm willing to provide any aditional details that you find necessary. Thanks in advance!
----
2019-12-03 11:06:47 UTC - jia zhai: @Nuno Ferreira what is your Pulsar version and environment, and what are the steps to reproduce this?
----
2019-12-03 11:09:59 UTC - Nuno Ferreira: pulsar version 2.4.1
running on ubuntu
steps to reproduce:
1 - create the function: in my case
public class ContextFunction implements Function<User, Void>
@Override
public Void process(User user, Context context) throws Exception {
Attachment attachment = new Attachment(user.getName() + "from user", user.getAge());
context.newOutputMessage("attachment-topic", JSONSchema.of(Attachment.class)).value(attachment).sendAsync();
return null;
}
2 - mvn clean install to generate JAR file
3 - run the command to create pulsar function:
./pulsar-admin functions create --jar /mnt/d/FDA/functions/target/functions-0.0.1-SNAPSHOT.jar --classname ksa.jawwy.fda.pulsar.functions.mapper.ContextFunction --tenant public --namespace default --inputs <persistent://public/default/topic> --output <persistent://public/default/attachment-topic> --output-serde-classname ksa.jawwy.fda.pulsar.functions.mapper.AttachmentSerDe
----
2019-12-03 11:29:41 UTC - tuteng: You can try check log file: logs/functions/tenant-name/namespace-name/function-name/
----
2019-12-03 11:47:40 UTC - Nuno Ferreira: an interesting exception appeared:
java.lang.NoClassDefFoundError: org/apache/pulsar/client/impl/schema/JSONSchema
at ksa.jawwy.fda.pulsar.functions.mapper.ContextFunction.process(ContextFunction.java:14) ~[functions-0.0.1-SNAPSHOT.jar:0.0.1-SNAPSHOT]
at ksa.jawwy.fda.pulsar.functions.mapper.ContextFunction.process(ContextFunction.java:9) ~[functions-0.0.1-SNAPSHOT.jar:0.0.1-SNAPSHOT]
at org.apache.pulsar.functions.instance.JavaInstance.handleMessage(JavaInstance.java:63) ~[org.apache.pulsar-pulsar-functions-instance-2.4.1.jar:?]
at org.apache.pulsar.functions.instance.JavaInstanceRunnable.run(JavaInstanceRunnable.java:259) [org.apache.pulsar-pulsar-functions-instance-2.4.1.jar:?]
at java.lang.Thread.run(Thread.java:748) [?:1.8.0_222]
Caused by: java.lang.ClassNotFoundException: org.apache.pulsar.client.impl.schema.JSONSchema
at <http://java.net|java.net>.URLClassLoader.findClass(URLClassLoader.java:382) ~[?:1.8.0_222]
at java.lang.ClassLoader.loadClass(ClassLoader.java:424) ~[?:1.8.0_222]
at java.lang.ClassLoader.loadClass(ClassLoader.java:357) ~[?:1.8.0_222]
... 5 more
----
2019-12-03 11:48:25 UTC - Nuno Ferreira: im using the JSONSchema class provided by Pulsar
import org.apache.pulsar.client.impl.schema.JSONSchema;
----
2019-12-03 12:06:56 UTC - Nuno Ferreira: any idea of whats happening here?
----
2019-12-03 14:51:57 UTC - Nuno Ferreira: @tuteng @jia zhai
----
2019-12-03 15:09:36 UTC - jia zhai: @Nuno Ferreira It is strange. Not sure it is related to your env or your configs.
Have you tried to run the examples provided by pulsar?
<https://github.com/apache/pulsar/wiki/Release-Candidate-Validation#validate-pubsub-and-java-functions>
----
2019-12-03 15:10:22 UTC - jia zhai: If the example runs success, then you could compare and get what is the different for your own function.
----
2019-12-03 15:27:31 UTC - Nuno Ferreira: i found the solution. The JSONSchema class was not packaged into my jar. It runs in localrun mode because my pc has maven .m2 folder and (i guess xD) retrieve the class from there but in cluster mode there's no maven repo. Thanks for your help @jia zhai
----
2019-12-03 15:28:10 UTC - Nuno Ferreira: and @tuteng :stuck_out_tongue:
----
2019-12-03 15:30:20 UTC - Eduardo Caceres: @David Kjerrumgaard I meant does pulsar can ingest data from Kafka version 0.7.2. I have a legacy kafka cluster that i need to ingest data from , but nothing seams to be compatible with it. so I was wondering if Pulsar can ingest data from kafka topics 0.7.2
----
2019-12-03 15:53:18 UTC - jia zhai: :thumbsup_all:
----
2019-12-03 16:18:09 UTC - David Kjerrumgaard: @Eduardo Caceres There isn't a tool that moves the data, but it should be simple enough to write a program that reads the data from Kafka and publishes it to Pulsar.
----
2019-12-03 16:32:01 UTC - Eduardo Caceres: ok. thk.
----
2019-12-03 16:33:17 UTC - Sijie Guo: It seems to me that you need a connector (most likely a KafkaSourceConnector) that reads data from your Kafka 0.7.0 and writes data into pulsar topics.
----
2019-12-03 16:39:18 UTC - David Kjerrumgaard: Yes, if you want to continue to forward data from Kafka 0.7.2 into Pulsar that would be a good approach. If this is a one-time process, then any simple client should suffice. :smiley:
----
2019-12-03 16:59:55 UTC - Igor Zubchenok: In my cluster with huge number of persistent topics, getting list of persistent topics works ok, but getting list of non-persistent topics overloads brokers and gets the whole pulsar cluster into an instable state.
----
2019-12-03 17:04:33 UTC - David Kjerrumgaard: Approximately how many non-persistent topics do you have?
----
2019-12-03 20:15:24 UTC - Igor Zubchenok: zero, I don't use non-persistent.
----
2019-12-03 21:40:07 UTC - juraj: @juraj has joined the channel
----
2019-12-03 21:42:02 UTC - juraj: hi guys!:) the 2.4.2 release seems to linger a little longer than usual, is there any estimation for availability pls?
----
2019-12-03 21:57:33 UTC - Jasdeep Singh: @Jasdeep Singh has joined the channel
----
2019-12-03 23:00:17 UTC - Sijie Guo: It should be done today or tomorrow. The vote has run over 3 candidates. We need to make sure the release binaries following the ASF policy. So sorry about the delay.
----
2019-12-03 23:16:53 UTC - juraj: thank you Sijie !
----
2019-12-04 01:28:44 UTC - NG: @NG has joined the channel
----
2019-12-04 09:05:17 UTC - Brian Doran: @tuteng Thanks Yes I had configured prometheus.yml but still can't get the bookie stats and when I used the template I can get Overview and Messaging stats but not bookies.
----
2019-12-04 09:07:41 UTC - tuteng: I'm not sure if you have turned on bookie's http metrics service, you can test it.
```curl -v <http://bookie-ip:8000/metrics/>```
----