You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@pulsar.apache.org by Apache Pulsar Slack <ap...@gmail.com> on 2020/02/24 09:11:05 UTC
Slack digest for #general - 2020-02-24
2020-02-23 10:34:33 UTC - Toktok Rambo: @Toktok Rambo has joined the channel
----
2020-02-23 10:38:10 UTC - Toktok Rambo: hello all, I’m new to Apache Pulsar. Just starting to use (using the official Go client, <https://github.com/apache/pulsar-client-go>). Could someone please help me understand what “SubscriptionName” and “ConsumerName” are used for?
----
2020-02-23 11:02:50 UTC - Vladimir Shchur: consumer name is there just for reference, subscription name is what consuming logic is based on, you should read about subscription more
----
2020-02-23 11:41:57 UTC - Toktok Rambo: ah yes. I’m deploying multiple subscribers (shared subscription model), I was wondering if I should be giving unique subscription name for each instance
----
2020-02-23 13:06:47 UTC - Ruwen: Hello. Just found a little typo in the docu <https://pulsar.apache.org/docs/en/io-rabbitmq-source/#configuration> - The yaml config
```virtualHost: "/",```
breaks due to the comma at the end
----
2020-02-23 13:27:08 UTC - Eugen: @Ruwen You can use the "EDIT" button at the top of the page to suggest a change
----
2020-02-23 13:37:04 UTC - Alex Yaroslavsky: Hi, I am running Function worker as a separate machine, and pulsar-admin gives this errir:
`# bin/pulsar-admin functions create --py c.py --classname router.RoutingFunction --tenant c1-d1 --namespace emsaas --name test1 --inputs <persistent://c1-d1/emsaas/in>`
`Function worker service is not done initializing. Please try again in a little while.`
`Reason: HTTP 503 Service Unavailable`
In broker log I see:
172.32.2.235 is the address of the Function worker, and the log that fails it seems to connect to 127.0.0.1.
What can I do here?
`Feb 23 13:17:09 localhost pulsar: 13:17:09.892 [pulsar-web-30-15] INFO org.eclipse.jetty.server.RequestLog - 172.32.2.235 - - [23/Feb/2020:13:17:09 +0000] "GET /admin/v2/persistent/public/functions/coordinate/stats HTTP/1.1" 200 901 "-" "Pulsar-Java-v2.5.0" 1`
`Feb 23 13:17:39 localhost pulsar: 13:17:39.879 [pulsar-web-30-8] INFO org.eclipse.jetty.server.RequestLog - 172.32.2.235 - - [23/Feb/2020:13:17:39 +0000] "GET /admin/v2/persistent/public/functions/coordinate/stats HTTP/1.1" 200 901 "-" "Pulsar-Java-v2.5.0" 3`
`Feb 23 13:17:41 localhost pulsar: 13:17:41.644 [pulsar-web-30-4] INFO org.eclipse.jetty.server.RequestLog - *127.0.0.1* - - [23/Feb/2020:13:17:41 +0000] "POST /admin/v3/functions/c1-d1/emsaas/test1 HTTP/1.1" 503 98 "-" "Pulsar-Java-v2.5.0" 4`
`Feb 23 13:17:54 localhost kernel: ena: Feature 27 isn't supported`
`Feb 23 13:18:09 localhost pulsar: 13:18:09.879 [pulsar-web-30-6] INFO org.eclipse.jetty.server.RequestLog - 172.32.2.235 - - [23/Feb/2020:13:18:09 +0000] "GET /admin/v2/persistent/public/functions/coordinate/stats HTTP/1.1" 200 901 "-" "Pulsar-Java-v2.5.0" 1`
----
2020-02-23 13:39:31 UTC - Alex Yaroslavsky: Another question about Functions, is it possible to read from one tenant (tenant1/ns1/topic1) and write to another (other-tenant/ns1/topic2) ?
----
2020-02-23 13:40:02 UTC - Chris Bartholomew: You need to set the `client.conf` for `pulsar-admin` to point to the function worker machine on port 6750.
----
2020-02-23 13:41:01 UTC - Ruwen: I created the according PR
+1 : Eugen
----
2020-02-23 13:41:14 UTC - Alex Yaroslavsky: Which property name should I use?
----
2020-02-23 13:42:29 UTC - Chris Bartholomew: ```webServiceUrl```
----
2020-02-23 13:45:12 UTC - Alex Yaroslavsky: Thanks, it works! But it is a bit strange. Now I have to use two different pulsar-admin configuration to manage function worker and the rest. I would expect pulsar-admin to get the Function worker address from the broker somehow.
----
2020-02-23 14:01:48 UTC - Chris Bartholomew: You can configure the Pulsar proxy to properly forward the request. Unfortunately, that is not working properly right now: <https://github.com/apache/pulsar/issues/4612>
----
2020-02-23 14:02:57 UTC - Chris Bartholomew: I have submitted a pull request to fix it if you want to build it yourself: <https://github.com/apache/pulsar/pull/6297>
----
2020-02-23 15:47:05 UTC - Roman Popenov: Anybody tried upgrading the ASM library to version 7?
----
2020-02-23 16:09:37 UTC - Penghui Li: @Antti Kaikkonen Please take a look this comment <https://github.com/apache/pulsar/issues/5503#issuecomment-590084502>, hope can help you.
----
2020-02-23 16:33:08 UTC - Antti Kaikkonen: @Penghui Li the schema is auto generated and com.example.pulsartest.model.User exists in the jar package of my function. Maybe it could be a class loader issue then. I also had an issue trying to load SerDe class using `--custom-serde-inputs` : <https://github.com/apache/pulsar/issues/5350#issuecomment-590038153>
`{`
`"version": 0,`
`"schemaInfo": {`
`"name": "user",`
`"schema": {`
`"type": "record",`
`"name": "User",`
`"namespace": "com.example.pulsartest.model",`
`"fields": [`
`{`
`"name": "name",`
`"type": [`
`"null",`
`"string"`
`]`
`},`
`{`
`"name": "age",`
`"type": "long"`
`}`
`]`
`},`
`"type": "AVRO",`
`"properties": {`
`"__alwaysAllowNull": "true"`
`}`
`}`
`}`
----
2020-02-23 16:56:03 UTC - Antti Kaikkonen: I have created for testing a source connector that produces dummy users every 1 second and a function that only appends "!" to the name of each user. I'm going to bed soon but I can make those public tomorrow.
----
2020-02-23 17:37:25 UTC - Avimas: @Avimas has joined the channel
----
2020-02-23 18:59:11 UTC - Ruwen: What is the purpose of the `org.apache.pulsar.shade.*` dependencies in *pulsar-client*? I thought they would available for deployed Functions, but that is not the case. Instead I get `ClassNotFoundException: org.apache.pulsar.shade.org.codehaus.jackson.map.ObjectMapper`
----
2020-02-23 19:29:44 UTC - Sijie Guo: Thank you :pray:
----
2020-02-23 19:30:39 UTC - Rolf Arne Corneliussen: This might not be a problem in your setup, but when I first tried to set up a bookie cluster on 3 bare metal Ubuntu servers, I had an issue that were ultimately caused by the '127.0.1.1 <hostname>' mapping in /etc/hosts. I guess this also applies to Debian distros.I also had to replace/fix some jars for netty epoll transport in the 'lib' directory.
----
2020-02-23 19:31:23 UTC - Sijie Guo: You should give same subscription name if these are part of same group to consume. Consumer name is useful for troubleshooting because you are able to see stats for individual consumers.
+1 : Toktok Rambo
----
2020-02-23 19:33:18 UTC - Sijie Guo: > Thanks, it works! But it is a bit strange. Now I have to use two different pulsar-admin configuration to manage function worker and the rest. I would expect pulsar-admin to get the Function worker address from the broker somehow.
@Alex Yaroslavsky can you create a GitHub issue for improving pulsar-admin? I think we can improve it to handle this setup well without the need of a proxy.
----
2020-02-23 19:33:40 UTC - Sijie Guo: Yes you are able to do so.
----
2020-02-23 19:36:13 UTC - Sijie Guo: Pulsar-client shades all its dependencies to avoid version conflicts. If you don’t want shaded dependencies, you can use Pulsar-client-original.
When did you receive this exception? It might be worth creating a GitHub issue with the steps to reproduce it.
----
2020-02-23 19:38:39 UTC - Ruwen: Hi @Sijie Guo I was not sure if my expectation was correct. Ya, I can open an issue for that
----
2020-02-23 20:22:51 UTC - Rolf Arne Corneliussen: Running a simple consumer on Windows using the PulsarClient, I notice that there is considerable CPU usage when the consumer is just idling. In Java Mission Control, I can see that the thread responsible for this is 'pulsar-timer-x-y'. From the source code of `org.apache.pulsar.client.impl.PulsarClientImpl` I can see that the `HashedWheelTimer` is set up with a tick time of 1 millisecond (so it will rotate the wheels every millisecond). Is there any reason for setting the 'tick time' so tight? (The default `HashedWheelTimer` setting is 100 milliseconds)
----
2020-02-23 22:52:24 UTC - Venky Ganti: @Venky Ganti has joined the channel
----
2020-02-24 00:03:10 UTC - Sijie Guo: I think that hashwheel timer is used for batching. The default batching interval is 1ms
----
2020-02-24 03:23:45 UTC - Greg Gallagher: @Greg Gallagher has joined the channel
----
2020-02-24 04:45:46 UTC - Sree Vaddi: If anyone interested:
<https://www.hgsc.bcm.edu/events/hackathon>
----
2020-02-24 06:07:58 UTC - laneChen: @laneChen has joined the channel
----