You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@pulsar.apache.org by Apache Pulsar Slack <ap...@gmail.com> on 2020/05/15 09:11:05 UTC
Slack digest for #general - 2020-05-15
2020-05-14 09:39:03 UTC - Damien Roualen: Good morning,
Is there a working Pulsar connector for the last version 333 of Presto? Did someone tried?
When adding the connector plugin to Presto I got an error.
```PULSAR_VERSION=2.5.1
PRESTO_VERSION=333
...
mv apache-pulsar-${PULSAR_VERSION}/lib/presto/plugin/pulsar-presto-connector presto-server-${PRESTO_VERSION}/plugin/pulsar```
When starting Presto Server.
```~/presto-server-333$ ./bin/launcher run
....
2020-05-14T09:30:38.974Z INFO main io.prestosql.server.PluginManager -- Finished loading plugin /home/damien.roualen/presto-server-333/plugin/postgresql --
2020-05-14T09:30:38.975Z INFO main io.prestosql.server.PluginManager -- Loading plugin /home/damien.roualen/presto-server-333/plugin/pulsar --
2020-05-14T09:30:38.994Z ERROR main io.prestosql.server.PrestoServer No service providers of type io.prestosql.spi.Plugin
java.lang.IllegalStateException: No service providers of type io.prestosql.spi.Plugin
at com.google.common.base.Preconditions.checkState(Preconditions.java:589)```
----
2020-05-14 11:59:57 UTC - JG: in fact, it never reach the topic of the function
----
2020-05-14 12:00:25 UTC - JG: Producer is sending the message correctly and there is a subscription but my function is never called
----
2020-05-14 12:15:10 UTC - Steve Kim: Yes. I would like to learn about the format of the entry log files and index files.
----
2020-05-14 12:58:25 UTC - rani: Greetings! Currently setting up an autoscaling mechanism for my AWS EC2 ASG managed pulsar cluster running on version `2.5.1`!
*Strange Observation*
After starting with a baseline of 3 healthy brokers, I initiated a manual ASG scale-up event to scale the number of broker nodes to 4.
After the 4th broker service starts, I am encountering a `502` when querying broker health using `{{PULSAR_PROXY_ENDPOINT}}/admin/v2/brokers/health` .
This goes away completely if i do a rolling restart on the `pulsar-proxy` nodes.
My only guess at this point is that `pulsar-proxy` doesn’t refresh its cache of “currently active brokers” from zookeepers regularly enough. If this assumption is true, Is there any parameter that we can tune to “force-resync” or refresh cache? @Sijie Guo
*PS:* there are no useful logs in neither pulsar-proxy nor pulsar-broker
----
2020-05-14 15:06:14 UTC - Vincent LE MAITRE: Hi all,
I am running a Pulsar cluster on Kubernetes using the pulsar helm chart.
Pulsar version is 2.5.1.
I would like to enable the webSocket service which is not enabled by default on the brokers.
As explained into the documentation, I have added "webSocketServiceEnabled=true" into the conf/broker.conf through the broker configMap to enable websocket service in embedded mode.
When connecting a producer using the websocket service, I get the following message in broker logs :
`WARN org.apache.pulsar.websocket.ProducerHandler - [127.0.0.1:34824] Failed in creating producer on topic <persistent://test/test/test>: Param serviceUrl must not be blank.`
I tried to add "serviceUrl" or "brokerServiceUrl" properties into broker.conf, without success.
I also tried to add "serviceUrl" into conf/websocket.conf file, no more success...
Could you help me to get the websocket service working ?
Thanks
----
2020-05-14 15:08:28 UTC - Guilherme Perinazzo: I belive you need to setup the websocket config file also
----
2020-05-14 15:09:15 UTC - Guilherme Perinazzo: But I went with the separate component, so maybe i'm wrong
----
2020-05-14 15:10:50 UTC - Vincent LE MAITRE: In <https://pulsar.apache.org/docs/en/client-libraries-websocket/|doc>, it says that in embedded mode websocket.conf is not used...
Moreover the helm template does not fill the parameters into this config file.
----
2020-05-14 15:12:14 UTC - Vincent LE MAITRE: But if it is required in "embedded mode", i will try to improve the helm package to fill websocket conf in addition.
----
2020-05-14 15:13:06 UTC - Vincent LE MAITRE: is there someone that try this websocket embedded mode inside broker ?
----
2020-05-14 15:38:55 UTC - Sijie Guo: Did you use the fully qualified name as the output topic name when submitting a function?
----
2020-05-14 15:41:46 UTC - Rounak Jaggi: Now as per the bookkeeper conf, we set these:
tlsProviderFactoryClass=org.apache.bookkeeper.tls.TLSContextFactory
tlsCertificatePath=<path>/bookie.cert.pem
tlsKeyStoreType=PEM
tlsKeyStore=<path>/bookie.key-pk8.pem
tlsTrustStoreType=PEM
tlsTrustStore=<path>/ca.cert.pem
And we are seeing this in bookkeeper logs:
current state START_TLS :
<http://javax.net|javax.net>.ssl.SSLHandshakeException: General OpenSslEngine problem
Caused by: sun.security.validator.ValidatorException: Extended key usage does not permit use for TLS server authentication
ERROR org.apache.bookkeeper.proto.PerChannelBookieClient - TLS handshake failed
----
2020-05-14 16:09:59 UTC - Sijie Guo: I think the certificates that you obtain must allow for both clientAuth and serverAuth if the extended key usage extension is present. Or not have that extension included at all.
----
2020-05-14 16:15:26 UTC - David Kjerrumgaard: Is that data you are sending compressed?
----
2020-05-14 16:52:40 UTC - Alexander Ursu: Hello, am wondering if the Pulsar Zookeeper is configured by default to purge log and snapshot files under the `/data` directory
----
2020-05-14 16:53:32 UTC - Matteo Merli: Yes, it should keep the latest 3 of each:
<https://github.com/apache/pulsar/blob/c65c8099e18236586edc0eafdff595e7ab376997/conf/zookeeper.conf#L47>
+1 : Alexander Ursu
----
2020-05-14 17:29:58 UTC - Alexander Ursu: Hello again, I was wondering why the image size on docker hub doesn't reflect the image size shown by `docker image ls` on my machine. Docker hub says a little under 2gb for the `pulsar-all` image, while on a couple of my machines it's reported as almost 3gb
----
2020-05-14 18:33:58 UTC - JG: yes --> *<non-persistent://tenant/namespace/topic>*
----
2020-05-14 18:51:48 UTC - Mark Huang: @Mark Huang has joined the channel
----
2020-05-14 19:19:01 UTC - Alexander Ursu: Hi, I restarted one of the zookeeper servers out of the 3 I have running, trying to update the image. I am getting these logs now and my brokers seem to fail to connect. I am not sure why. For context I am running a single cluster in Docker
----
2020-05-14 19:19:01 UTC - Alexander Ursu: ```19:16:18.041 [NIOWorkerThread-2] INFO org.apache.zookeeper.server.quorum.Learner - Revalidating client: 0x1000f67f5cf0004
19:16:18.226 [NIOWorkerThread-4] WARN org.apache.zookeeper.server.NIOServerCnxn - Unable to read additional data from client sessionid 0x2000f6334c30001, likely client has closed socket
19:16:18.504 [NIOWorkerThread-1] INFO org.apache.zookeeper.server.quorum.Learner - Revalidating client: 0x3000f61e6c7001e
19:16:25.126 [NIOWorkerThread-3] INFO org.apache.zookeeper.server.ZooKeeperServer - Refusing session request for client /10.0.0.194:39760 as it has seen zxid 0x200000000 our last zxid is 0x100024899 client must try another server
19:16:34.722 [NIOWorkerThread-1] INFO org.apache.zookeeper.server.ZooKeeperServer - Refusing session request for client /10.0.0.175:36320 as it has seen zxid 0x200000000 our last zxid is 0x100024899 client must try another server
19:16:35.548 [NIOWorkerThread-2] INFO org.apache.zookeeper.server.ZooKeeperServer - Refusing session request for client /10.0.0.172:38850 as it has seen zxid 0x200000000 our last zxid is 0x100024899 client must try another server
19:16:38.044 [NIOWorkerThread-4] WARN org.apache.zookeeper.server.NIOServerCnxn - Unable to read additional data from client sessionid 0x1000f67f5cf0004, likely client has closed socket
19:16:38.518 [NIOWorkerThread-1] WARN org.apache.zookeeper.server.NIOServerCnxn - Unable to read additional data from client sessionid 0x3000f61e6c7001e, likely client has closed socket
19:16:47.540 [NIOWorkerThread-3] INFO org.apache.zookeeper.server.ZooKeeperServer - Refusing session request for client /10.0.0.194:39788 as it has seen zxid 0x200000000 our last zxid is 0x100024899 client must try another server
19:16:56.997 [NIOWorkerThread-1] INFO org.apache.zookeeper.server.ZooKeeperServer - Refusing session request for client /10.0.0.175:36340 as it has seen zxid 0x200000000 our last zxid is 0x100024899 client must try another server
19:16:58.600 [NIOWorkerThread-2] INFO org.apache.zookeeper.server.ZooKeeperServer - Refusing session request for client /10.0.0.172:38870 as it has seen zxid 0x200000000 our last zxid is 0x100024899 client must try another server
19:16:59.391 [NIOWorkerThread-3] INFO org.apache.zookeeper.server.quorum.Learner - Revalidating client: 0x2000f6334c30001
19:17:09.518 [NIOWorkerThread-4] INFO org.apache.zookeeper.server.ZooKeeperServer - Refusing session request for client /10.0.0.194:39808 as it has seen zxid 0x200000000 our last zxid is 0x100024899 client must try another server
19:17:19.394 [NIOWorkerThread-2] WARN org.apache.zookeeper.server.NIOServerCnxn - Unable to read additional data from client sessionid 0x2000f6334c30001, likely client has closed socket
19:17:19.724 [NIOWorkerThread-3] INFO org.apache.zookeeper.server.ZooKeeperServer - Refusing session request for client /10.0.0.175:36364 as it has seen zxid 0x200000000 our last zxid is 0x100024899 client must try another server
19:17:20.167 [NIOWorkerThread-4] INFO org.apache.zookeeper.server.quorum.Learner - Revalidating client: 0x1000f67f5cf0004
19:17:20.699 [NIOWorkerThread-2] INFO org.apache.zookeeper.server.quorum.Learner - Revalidating client: 0x3000f61e6c7001e
19:17:21.652 [NIOWorkerThread-4] INFO org.apache.zookeeper.server.ZooKeeperServer - Refusing session request for client /10.0.0.172:38896 as it has seen zxid 0x200000000 our last zxid is 0x100024899 client must try another server
19:17:32.203 [NIOWorkerThread-4] INFO org.apache.zookeeper.server.ZooKeeperServer - Refusing session request for client /10.0.0.194:39826 as it has seen zxid 0x200000000 our last zxid is 0x100024899 client must try another server
19:17:40.177 [NIOWorkerThread-2] WARN org.apache.zookeeper.server.NIOServerCnxn - Unable to read additional data from client sessionid 0x1000f67f5cf0004, likely client has closed socket
19:17:40.703 [NIOWorkerThread-4] WARN org.apache.zookeeper.server.NIOServerCnxn - Unable to read additional data from client sessionid 0x3000f61e6c7001e, likely client has closed socket
19:17:42.967 [NIOWorkerThread-1] INFO org.apache.zookeeper.server.ZooKeeperServer - Refusing session request for client /10.0.0.175:36390 as it has seen zxid 0x200000000 our last zxid is 0x100024899 client must try another server```
----
2020-05-14 19:44:51 UTC - Alex Yaroslavsky: Is "pulsar-admin functions list" supposed to work? I always get 404.Running 2.5.1 (same was on 2.5.0), this happens when either going through proxy or directly to function worker (workers run separately from brokers).
----
2020-05-14 19:47:01 UTC - Alex Yaroslavsky: <https://github.com/apache/pulsar/issues/5947>
----
2020-05-14 19:47:22 UTC - Alex Yaroslavsky: Seems that "bin/pulsar-admin functions list" supposed to show "help" as parameters are missing
----
2020-05-14 20:08:45 UTC - Ming: This is because images on docker hub has been compressed. The step is done by `docker push` locally. The data is compressed before actually pushing to Docker Hub.
----
2020-05-14 23:59:41 UTC - Alexander Ursu: Hello, was wondering if there is any early discussion around zookeeper and a possible replacement. I've noticed some similar discussion surrounding Kafka, and am curious if anyone has any thoughts on the matter.
----