You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@pulsar.apache.org by Apache Pulsar Slack <ap...@gmail.com> on 2020/02/12 09:11:04 UTC
Slack digest for #general - 2020-02-12
2020-02-11 09:26:38 UTC - Eric Feliksik: I am running the standalone docker version of Pulsar, using image `apachepulsar/pulsar:2.5.0`.
```tenant: public
namespace: default
name: exclaimer
jar: /files/pulsar-functions/com.my.test.pulsarfunctions/target/pulsar-functions-0.2.jar
className: com.my.test.pulsarfunctions.exclaim.PostFixer
inputs:
- <persistent://public/default/incoming>
output: messages_exclaimed
logTopic: <persistent://public/default/my_logs>
parallellism: 1
userConfig:
postfix: '!!!'
autoAck: true```
My function implementation is:
```public class PostFixer implements Function<String, String> {
@Override
public String process(String input, Context context) throws PulsarClientException {
if (input.equals("log")) {
System.out.println(">>>>>>>>>>> logging on [stdout]");
context.getLogger().error(">>>>>>>>>>> logging with [logger]");
context.newOutputMessage("my_logs", Schema.STRING).value(">>>>>> logging explicitly to topic").send();
throw new RuntimeException(">>>>>>>>>>>> logging by throwing [Exception]");
}
return input + "!!!";
}
}```
Now I would expect some logs to end up in the `my_logs` topic. However, Only the “logging explicitly to topic” ends up there. The logging to stdout appears in the docker-compose logs, and the logging via the context.getLogger() and the Exception are completely untraceable.
How can I make sure that I find the thrown exceptions? I thought they should end up in the `my_logs` topic; is that correct?
----
2020-02-11 10:07:27 UTC - Andres Garagiola: @Andres Garagiola has joined the channel
----
2020-02-11 10:10:34 UTC - Younes: @Younes has joined the channel
----
2020-02-11 11:46:11 UTC - Yannis Weishaupt: @Yannis Weishaupt has joined the channel
----
2020-02-11 12:45:39 UTC - Igor Vaisman: @Igor Vaisman has joined the channel
----
2020-02-11 12:58:12 UTC - Pradeep Mishra: i am getting this error when i run Pulsar Broker (2.5.0) with websocket service on, i am not running it on standalone mode
----
2020-02-11 12:58:14 UTC - Pradeep Mishra: ```09:10:31.170 [pulsar-web-32-4] WARN org.apache.pulsar.websocket.ConsumerHandler - [10.1.1.194:53608] Failed in creating subscription sub-checkpoint-logs on topic <persistent://public/default/checkpoint_logs>
java.lang.IllegalArgumentException: Param serviceUrl must not be blank.```
----
2020-02-11 16:26:21 UTC - Yoav Cohen: Looks like this fixed the issue @Sijie Guo and @Shahar Fermon, thank you!
----
2020-02-11 17:15:03 UTC - Naby: I am trying to remember if a python function as Pulsar Function can depends on another custom module or it has to be a single file. I’m having a hard time finding this fact on pulsar documents. Can anyone help?
----
2020-02-11 19:19:30 UTC - Ming: Python code for function has to be in the same file. If you have another Python library dependency, one option is you can pip install in function's docker image because Python function worker is started within a docker container.
----
2020-02-11 19:24:09 UTC - Ming: What is your wss or ws url? It might be just the error code indicating `Param serviceUrl must not be blank.` It should look like this `wss://{pulsar-host-fqdn}:8001/ws/v2/consumer/persistent/public/default/foobar-topic/my-subscription`
----
2020-02-11 19:51:38 UTC - Ming: Essentially you are asking a high performance cluster. ~10,000 msgs/sec with avg msg size of ~5MB is like 400gbps (50 GBytes per second) You need run partitioned topic on multiple brokers so that to spread network load cross multiple NIC. Data also hav eto be tripped across multiple bookkeepers for disk I/O to keep up. This is not just Python limit. The system has to be spec-ed properly.
----
2020-02-11 19:54:47 UTC - Ming: `maxMessageSize` should be tweaked over 5MB which is your average. By default it is 5MB. You need shared subscription to handle the rate, unless you already have a number of topics. So this is more than a question of choice of Python. If it is designed properly, Python is not a problem.
----
2020-02-11 19:56:13 UTC - Ming: Here are just a few concerns top of my mind. There must be a few things I missed. @Aditya Vishwakarma
----
2020-02-11 20:00:49 UTC - Sijie Guo: @Pradeep Mishra there was a regression regarding websocket in 2.5.0. I have fixed it in master. planning a 2.5.1 bugfix release soon.
----
2020-02-11 21:37:41 UTC - Kelvin Sajere: Apache pulsar 2.4.2 works fine for now @Pradeep Mishra
----
2020-02-11 22:15:31 UTC - Jordan Pilat: @Jordan Pilat has joined the channel
----
2020-02-11 23:44:52 UTC - Sijie Guo: @Naby the python function support multiple files via wheel and zip package. When the python function downloads a wheel package, it will install the wheel package. The feature is already there since 2.3.0. It might not be well documented.
----
2020-02-12 00:11:57 UTC - rmb: Hi all, I'm trying to set up authentication via JWT (I'm running pulsar in standalone mode). I've generated tokens and edited conf/client.conf and conf/broker.conf as described at <https://pulsar.apache.org/docs/en/security-jwt/>, but when I try to run 'bin/pulsar-admin namespaces grant-permission', it says
> Authorization is not enabled
>
> Reason: HTTP 501 Not Implemented
is there something else I have to configure?
----
2020-02-12 02:04:13 UTC - Sijie Guo: For stand-alone, you need to configure conf/standalone.conf
----
2020-02-12 06:04:41 UTC - Abhilash Mandaliya: Hi
Is it mandatory to create a pulsar-io.yaml file under src/main/resources/META-INF/services/
?
----
2020-02-12 06:05:16 UTC - Ali Ahmed: yes
----
2020-02-12 06:05:34 UTC - Ali Ahmed: it’s part of the packaging format
----
2020-02-12 06:08:21 UTC - Abhilash Mandaliya: ok, thanks :slightly_smiling_face:
But I didn’t find this anywhere in site documentation. Shouldn’t we highlight this somewhere?
----
2020-02-12 06:14:33 UTC - Pradeep Mishra: Yes, it works for 2.4.2.
----
2020-02-12 06:15:12 UTC - Pradeep Mishra: Thanks @Sijie Guo
----
2020-02-12 06:18:45 UTC - Sijie Guo: ah, we should clarify this in <http://pulsar.apache.org/docs/en/io-develop/> . @Abhilash Mandaliya do you want to contribute a documentation change? /cc @Anonymitaet
----
2020-02-12 06:24:32 UTC - Abhilash Mandaliya: Okay. I would like to give it a try.
----
2020-02-12 07:18:35 UTC - Abhilash Mandaliya: has anyone built connector using gradle instead of maven?
or any good blog post on the same?
I am unable to create .nar file using gradle
----
2020-02-12 07:19:28 UTC - Abhilash Mandaliya: I am using the following dependency in my build.gradle.kts
```compile("org.apache.nifi:nifi-nar-maven-plugin:1.3.1")```
----
2020-02-12 08:15:12 UTC - Ali Ahmed: you can’t use a maven plugin in gradle.
----
2020-02-12 08:15:41 UTC - Ali Ahmed: try one of the gradle nar plugins
<https://github.com/search?utf8=%E2%9C%93&q=gradle+nar&type=>
----
2020-02-12 08:39:02 UTC - rmb: any lines besides authenticationEnabled, authorizationEnabled, and authenticationProviders? when I set those options, pulsar just shuts down, before I have a chance to run any pulsar-admin commands
----
2020-02-12 08:50:36 UTC - Abhilash Mandaliya: thanks. it worked :slightly_smiling_face:
----
2020-02-12 08:55:07 UTC - rmb: I'm having a new problem: if I try to run a freshly unzipped 2.5.0 binary in docker, trying to run a pulsar-admin command returns
> null
>
> Reason: javax.ws.rs.ProcessingException: Remotely closed
no matter how long I wait.
----