You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@pulsar.apache.org by Apache Pulsar Slack <ap...@gmail.com> on 2018/05/30 09:11:02 UTC
Slack digest for #general - 2018-05-30
2018-05-29 09:21:43 UTC - Idan: thanks for the update
----
2018-05-29 13:04:11 UTC - Byron: Hi folks, using the 2.0.0-rc1 image, I am seeing a new parameter `proxyTo` is required for running proxies. Where/how should this be set?
----
2018-05-29 13:04:57 UTC - Byron: I provided it as a config option along with the other parameters, but I am still getting the error.
----
2018-05-29 14:49:20 UTC - Ivan Kelly: this is only needed for the admin API
----
2018-05-29 14:49:30 UTC - Ivan Kelly: which wasn't available in the proxy previously
----
2018-05-29 14:50:00 UTC - Ivan Kelly: unfortunately there's a bug with this that if proxyTo is set, you can use the admin api, but the binary protocol is broken
----
2018-05-29 14:50:22 UTC - Ivan Kelly: <https://github.com/apache/incubator-pulsar/issues/1796>
----
2018-05-29 14:53:12 UTC - Ivan Kelly: I think @Dave Rusek may have been working on this
----
2018-05-29 16:13:38 UTC - Byron: Thanks @Ivan Kelly. Based on the issue, it sounds like there is no workaround right now?
----
2018-05-29 16:14:20 UTC - Ivan Kelly: well, maybe one proxy for admin, another for data, but ya, no nice workaround
----
2018-05-29 20:36:59 UTC - Adam Zell: @Adam Zell has joined the channel
----
2018-05-29 21:22:50 UTC - Ali Ahmed: I have sent a PR to make pulsar integration testing simpler to test containers
<https://github.com/testcontainers/testcontainers-java/pull/713>
+1 : Matteo Merli, Ali Ahmed
----
2018-05-29 22:43:25 UTC - Matteo Merli: @Byron I’m not seeing where the “proxyTo” config option is being required. Can you share the error you’re getting?
----
2018-05-29 22:45:19 UTC - Matteo Merli: Will take a look
----
2018-05-29 23:02:02 UTC - Yuwei Jiang: Hi guys, I’m trying to deploy Pulsar on a Kubernetes cluster on AWS, using docker image `apachepulsar/pulsar:1.22.0-incubating` , I based my zookeeper, bookkeeper and broker deployment on `<https://github.com/apache/incubator-pulsar/blob/branch-2.0/deployment/kubernetes/aws/>` , deployed 3 zookeeper, 2 bookkeeper and 3 brokers
I’ve successfully deployed zookeeper/bookkeeper/broker/proxy components, but when I’m testing publish to a persistent topic using `pulsar-client` CLI, I received the following error message, but publish to a non-persistent topic works.
```
16:12:24,926 ERROR Error while producing messages
16:12:24,927 ERROR Could not send message to broker within given timeout
org.apache.pulsar.client.api.PulsarClientException$TimeoutException: Could not send message to broker within given timeout
at org.apache.pulsar.client.impl.ProducerImpl.run(ProducerImpl.java:1052)
at io.netty.util.HashedWheelTimer$HashedWheelTimeout.expire(HashedWheelTimer.java:663)
at io.netty.util.HashedWheelTimer$HashedWheelBucket.expireTimeouts(HashedWheelTimer.java:738)
at io.netty.util.HashedWheelTimer$Worker.run(HashedWheelTimer.java:466)
at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
at java.lang.Thread.run(Thread.java:748)
command terminated with exit code 255
```
I’ve checked my EC2 security group settings and is allowing all inbound traffic for all the nodes within the K8s cluster; EBS volumes are in `in-use` state. No obvious error messages from bookie/broker logs. I’d appreciate any suggestions/advises on this matter and on how to run Pulsar on K8s in AWS. Thanks.
----
2018-05-29 23:04:17 UTC - Byron: @Byron uploaded a file: <https://apache-pulsar.slack.com/files/UACD54WB1/FAXG1FK5W/Untitled.java|Untitled> and commented: @Matteo Merli
----
2018-05-29 23:08:32 UTC - Matteo Merli: Umh, ok, can you try to add `brokerServiceURL` in `proxy.conf` ?
----
2018-05-29 23:10:12 UTC - Sijie Guo: @Yuwei Jiang
> I’ve successfully deployed zookeeper/bookkeeper/broker/proxy components, but when I’m testing publish to a persistent topic using `pulsar-client` CLI, I received the following error message, but publish to a non-persistent topic works.
“publish to a non-persistent topic work”. that might means the problem is between brokers and bookies. you are using the yaml files without changes? or do you make any changes there? try to understand your configuration first here.
----
2018-05-29 23:18:41 UTC - Byron: @Matteo Merli I added it as a configmap for the k8s deployment. the script `apply-config-from-env.py` seems to not be adding that parameter to the `proxy.conf` file
----
2018-05-29 23:20:19 UTC - Matteo Merli: Yes, not by default
----
2018-05-29 23:21:16 UTC - Yuwei Jiang: Hi @Sijie Guo, thanks for the promptly reply. There are some naming convention change in my yaml files. Configuration wise, for bookies, I changed `volumeClainTemplates` to use `gp2` type volume for `ledgers-disk`, and fro broker, I changed `ConfigMap` data to set `managedLedgerDefaultEnsembleSize`, `managedLedgerDefaultWriteQuotum` , `managedLedgerDefaultAckQuorum` to `1` (This is due to I had an ManagedLederException: Not enough non-faulty bookies available thrown earlier when I tried to call pulsar-client consume)
----
2018-05-29 23:23:23 UTC - Matteo Merli: > Yes, not by default
@Byron.. I mean, it should be working even without the `brokerServiceURL`. The proxy can either use ZK or the serviceUrl to discover the available brokers
----
2018-05-29 23:23:44 UTC - Byron: Yea it is still working it appears
----
2018-05-29 23:29:33 UTC - Jon Bock: Wanted to provide an expanded answer to the earlier questions from @Mate Varga and @Idan (among others) about current Pulsar users. In addition to the names mentioned earlier in the thread, there are a good (and growing) number of other users who don’t have internal approval to have their name mentioned publicly. For example:
- One of the largest media companies in the US is using Pulsar in production, and another large media company is actively evaluating Pulsar to replace their legacy solution
- A large industrial company in Europe is in the process of deploying Pulsar in production to support one of their IoT initiatives
- One of the largest marketplaces in Latin America is using Pulsar in production
- A large identity management company has deployed Pulsar in production
- A number of B2B SaaS application companies have deployed or are evaluating Pulsar to support data flows supplying their applications
- Several financial services, online gaming, and electronics companies are actively piloting or evaluating Pulsar to help them upgrade their messaging infrastructure
Hopefully that gives you a high-level view of the breadth in the companies using and actively considering Pulsar.
----
2018-05-29 23:41:27 UTC - Sijie Guo: @Yuwei Jiang is the k8s cluster eks or a k8s cluster installed on vms?
----
2018-05-29 23:44:20 UTC - Yuwei Jiang: @Sijie Guo It’s a K8s cluster deployed on AWS EC2 using Kops
----
2018-05-29 23:54:20 UTC - Sijie Guo: @Yuwei Jiang where do you run the cli? same vpc as those vms?
----
2018-05-29 23:55:37 UTC - Yuwei Jiang: @Sijie Guo I’m running the cli with kubectl on my local laptop.
```
kubectl exec -it pulsar-admin -- bin/pulsar-client produce <persistent://mm/dc1/vs/yj> -m 'From-DC1'
```
----
2018-05-29 23:57:29 UTC - Yuwei Jiang: I’ve created properties/namespaces using pulsar-admin CLIs
----
2018-05-29 23:57:59 UTC - Sijie Guo: oh i see. so you are actually running within the k8s network.
how does your command to produce to nonpersistent topic looks like?
----
2018-05-29 23:59:07 UTC - Yuwei Jiang: I substitute the key word `persistent` with `non-persistent`
```
kubectl exec -it c8streams-admin -- bin/pulsar-client produce <non-persistent://mm/dc1/vs/yj> -m 'From-DC1'
```
----
2018-05-30 00:09:14 UTC - Sijie Guo: @Yuwei Jiang if you run “kubectl get pods”, are all the pods running well?
----
2018-05-30 00:10:15 UTC - Sijie Guo: or can you run “kubectl exec -it pulsar-admin — cat cat conf/client.conf
----
2018-05-30 00:10:37 UTC - Sijie Guo: sorry “kubectl exec -it pulsar-admin — cat conf/client.conf”
----
2018-05-30 00:12:17 UTC - Yuwei Jiang: @Sijie Guo, yes all pods are in “Running” status; output from above `cat conf/client.conf` command:
```
#
# Licensed to the Apache Software Foundation (ASF) under one
# or more contributor license agreements. See the NOTICE file
# distributed with this work for additional information
# regarding copyright ownership. The ASF licenses this file
# to you under the Apache License, Version 2.0 (the
# "License"); you may not use this file except in compliance
# with the License. You may obtain a copy of the License at
#
# <http://www.apache.org/licenses/LICENSE-2.0>
#
# Unless required by applicable law or agreed to in writing,
# software distributed under the License is distributed on an
# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
# KIND, either express or implied. See the License for the
# specific language governing permissions and limitations
# under the License.
#
# Pulsar Client configuration
webServiceUrl=<http://broker:8080/>
brokerServiceUrl=<pulsar://broker:6650/>
#authPlugin=
#authParams=
#useTls=
#tlsAllowInsecureConnection
#tlsTrustCertsFilePath
```
----
2018-05-30 00:14:26 UTC - Matteo Merli: @Yuwei Jiang since you mentioned that non-persistent topics are working fine, can you check for any WARN/ERROR messages in broker pods logs when trying to use persistent topics?
----
2018-05-30 00:21:32 UTC - Sijie Guo: @Yuwei Jiang - can you also check the alive bookies : kubectl exec -it <bookie-pod> -- bin/bookkeeper shell listbookies -ro
----
2018-05-30 00:21:53 UTC - Sijie Guo: sorry “kubectl exec -it <bookie-pod> -- bin/bookkeeper shell listbookies -rw”
----
2018-05-30 00:29:51 UTC - Yuwei Jiang: @Matteo Merli Just tailed the logs on the broker pods, when I tried publish on a persistent topic, I see the following log messages from one of the broker pod:
```
2018-05-30 00:20:46,300 - INFO - [pulsar-io-49-1:ServerCnx@136] - New connection from /100.96.6.160:35428
2018-05-30 00:20:52,707 - INFO - [main-EventThread:ZooKeeperDataCache@145] - [State:CONNECTED Timeout:30000 sessionid:0x363ad7344de0005 local:/100.96.3.77:53740 remoteserver:zookeeper-0.zookeeper.vs.svc.cluster.local/100.96.6.158:2181 lastZxid:4295289098 xid:3759 sent:3763 recv:3995 queuedpkts:0 pendingresp:0 queuedevents:0] Received ZooKeeper watch event: WatchedEvent state:SyncConnected type:NodeDataChanged path:/loadbalance/brokers/100.96.6.162:8080
2018-05-30 00:20:55,135 - INFO - [pulsar-web-55-10:Slf4jRequestLog@60] - 100.96.6.164 - - [30/May/2018:00:20:55 +0000] "GET //100.96.3.77:8080/metrics HTTP/1.1" 302 0 "-" "Prometheus/1.6.3" 1
2018-05-30 00:20:55,217 - INFO - [prometheus-stats-56-1:Slf4jRequestLog@60] - 100.96.6.164 - - [30/May/2018:00:20:55 +0000] "GET //100.96.3.77:8080/metrics/ HTTP/1.1" 200 4581 "<http://100.96.3.77:8080/metrics>" "Prometheus/1.6.3" 80
2018-05-30 00:20:56,797 - INFO - [pulsar-web-55-16:Slf4jRequestLog@60] - 100.96.3.80 - - [30/May/2018:00:20:56 +0000] "GET //broker:8080/admin/clusters HTTP/1.1" 200 22 "-" "python-requests/2.18.4" 1
2018-05-30 00:20:56,882 - INFO - [pulsar-web-55-27:Slf4jRequestLog@60] - 100.96.3.80 - - [30/May/2018:00:20:56 +0000] "GET //broker:8080/admin/clusters/dc1 HTTP/1.1" 200 141 "-" "python-requests/2.18.4" 1
2018-05-30 00:20:57,128 - INFO - [pulsar-web-55-12:Slf4jRequestLog@60] - 100.96.3.80 - - [30/May/2018:00:20:57 +0000] "GET //broker.vs.svc.cluster.local:8080/admin/brokers/dc1 HTTP/1.1" 200 59 "-" "python-requests/2.18.4" 2
2018-05-30 00:20:58,007 - INFO - [pulsar-web-55-30:Slf4jRequestLog@60] - 100.96.3.80 - - [30/May/2018:00:20:58 +0000] "GET //100.96.3.77:8080/admin/broker-stats/destinations HTTP/1.1" 200 2 "-" "python-requests/2.18.4" 1
2018-05-30 00:21:10,128 - INFO - [pulsar-web-55-13:Slf4jRequestLog@60] - 100.96.6.164 - - [30/May/2018:00:21:10 +0000] "GET //100.96.3.77:8080/metrics HTTP/1.1" 302 0 "-" "Prometheus/1.6.3" 0
2018-05-30 00:21:10,173 - INFO - [prometheus-stats-56-1:Slf4jRequestLog@60] - 100.96.6.164 - - [30/May/2018:00:21:10 +0000] "GET //100.96.3.77:8080/metrics/ HTTP/1.1" 200 4582 "<http://100.96.3.77:8080/metrics>" "Prometheus/1.6.3" 40
2018-05-30 00:21:17,278 - INFO - [pulsar-io-49-1:ServerCnx@144] - Closed connection from /100.96.6.160:35428
```
The pulsar-client produce fails with `Could not send message to broker within given timeout`
----
2018-05-30 00:30:38 UTC - Yuwei Jiang: @Sijie Guo output from above `listbookies` command
```
JMX enabled by default
172.20.55.180:3181
172.20.57.181:3181
```
----
2018-05-30 00:33:23 UTC - Sijie Guo: oh interesting. so it uses ip not hostname, if the bookkeeper is deployed with statefulset, it supposed to be hostnames.
do you mind sharing me your bookie.yaml?
----
2018-05-30 00:33:42 UTC - Yuwei Jiang: @Sijie Guo , @Matteo Merli Just saw one of my broker pod is full of WARN/ERROR messages:
```
2018-05-30 00:31:20,543 - ERROR - [bookkeeper-io-15-4:PerChannelBookieClient$2@284] - Could not connect to bookie: [id: 0x71b37d6a, L:/100.96.6.162:44324]/172.20.57.181:3181, current state CONNECTING :
io.netty.channel.AbstractChannel$AnnotatedConnectException: syscall:getsockopt(..) failed: Connection refused: /172.20.57.181:3181
at io.netty.channel.unix.Socket.finishConnect(..)(Unknown Source)
Caused by: io.netty.channel.unix.Errors$NativeConnectException: syscall:getsockopt(..) failed: Connection refused
... 1 more
2018-05-30 00:31:20,543 - ERROR - [bookkeeper-io-15-1:PerChannelBookieClient$2@284] - Could not connect to bookie: [id: 0xd10983bf, L:/100.96.6.162:44326]/172.20.57.181:3181, current state CONNECTING :
io.netty.channel.AbstractChannel$AnnotatedConnectException: syscall:getsockopt(..) failed: Connection refused: /172.20.57.181:3181
at io.netty.channel.unix.Socket.finishConnect(..)(Unknown Source)
Caused by: io.netty.channel.unix.Errors$NativeConnectException: syscall:getsockopt(..) failed: Connection refused
... 1 more
2018-05-30 00:31:20,543 - WARN - [BookKeeperClientWorker-18-1:PendingAddOp@238] - Write did not succeed: L0 E0 on 172.20.57.181:3181, rc = -8
2018-05-30 00:31:20,543 - INFO - [BookKeeperClientWorker-18-1:LedgerHandle@891] - Handling failure of bookie: 172.20.57.181:3181 index: 0
2018-05-30 00:31:20,543 - WARN - [BookKeeperClientWorker-19-1:PendingAddOp@238] - Write did not succeed: L1 E0 on 172.20.57.181:3181, rc = -8
2018-05-30 00:31:20,543 - INFO - [BookKeeperClientWorker-19-1:LedgerHandle@891] - Handling failure of bookie: 172.20.57.181:3181 index: 0
2018-05-30 00:31:20,544 - WARN - [BookKeeperClientWorker-18-1:PendingAddOp@238] - Write did not succeed: L0 E1 on 172.20.57.181:3181, rc = -8
2018-05-30 00:31:20,544 - WARN - [BookKeeperClientWorker-18-1:LedgerHandle@919] - Write did not succeed to 172.20.57.181:3181, bookieIndex 0, but we have already fixed it.
2018-05-30 00:31:20,544 - WARN - [BookKeeperClientWorker-18-1:PendingAddOp@238] - Write did not succeed: L0 E2 on 172.20.57.181:3181, rc = -8
2018-05-30 00:31:20,544 - WARN - [BookKeeperClientWorker-18-1:LedgerHandle@919] - Write did not succeed to 172.20.57.181:3181, bookieIndex 0, but we have already fixed it.
2018-05-30 00:31:20,544 - WARN - [BookKeeperClientWorker-18-1:PendingAddOp@238] - Write did not succeed: L0 E3 on 172.20.57.181:3181, rc = -8
2018-05-30 00:31:20,544 - WARN - [BookKeeperClientWorker-18-1:LedgerHandle@919] - Write did not succeed to 172.20.57.181:3181, bookieIndex 0, but we have already fixed it.
2018-05-30 00:31:20,544 - WARN - [BookKeeperClientWorker-18-1:PendingAddOp@238] - Write did not succeed: L0 E4 on 172.20.57.181:3181, rc = -8
2018-05-30 00:31:20,544 - WARN - [BookKeeperClientWorker-18-1:LedgerHandle@919] - Write did not succeed to 172.20.57.181:3181, bookieIndex 0, but we have already fixed it.
2018-05-30 00:31:20,544 - WARN - [BookKeeperClientWorker-18-1:PendingAddOp@238] - Write did not succeed: L0 E5 on 172.20.57.181:3181, rc = -8
2018-05-30 00:31:20,545 - WARN - [BookKeeperClientWorker-18-1:LedgerHandle@919] - Write did not succeed to 172.20.57.181:3181, bookieIndex 0, but we have already fixed it.
```
----
2018-05-30 00:34:37 UTC - Sijie Guo: yeah we suspected that bookies advertised wrong address.
----
2018-05-30 00:34:52 UTC - Sijie Guo: that’s why it keeps tries to connect to that ip.
----
2018-05-30 00:35:29 UTC - Sijie Guo: I am not sure “172.20.57.181” is pod ip or host ip though.
----
2018-05-30 00:35:37 UTC - Sijie Guo: so it would be good to take a look at the bookie.yaml
----
2018-05-30 00:39:08 UTC - Yuwei Jiang: @Sijie Guo `172.20.57.181` is the node’s internal IP. I will DM you the bookie yaml.
----
2018-05-30 00:39:30 UTC - Sijie Guo: cool
----
2018-05-30 00:40:16 UTC - Sijie Guo: I see - so it is hostIP.
----
2018-05-30 00:46:07 UTC - Sijie Guo: so I guess you might need to change the bookie yaml:
- make sure it is deployed with StatefulSet
- make sure “useHostNameAsBookieID” in the bookie-config section and set to “true”
- if you find “advertisedAddress” and “status.hostIP” in your bookie.yaml, remove that section.
----
2018-05-30 00:51:43 UTC - Yuwei Jiang: @Sijie Guo, Thanks for the suggestion! I will give it a try.
fire : jia zhai
----
2018-05-30 01:00:34 UTC - Matteo Merli: <!channel> Pulsar 2.0.0-rc1-incubating is now officially released!
<https://lists.apache.org/thread.html/bf6695c308ec5a30fb122c201a01596cd4a68d113e4c0a93709ae4c1@%3Cusers.pulsar.apache.org%3E>
Download: <http://pulsar.apache.org/download/>
Release notes: <http://pulsar.apache.org/release-notes/#2.0.0-rc1-incubating>
DockerHub: <https://hub.docker.com/r/apachepulsar/pulsar/>
Python client: <https://pypi.org/project/pulsar-client/>
+1 : jia zhai, Byron, Sijie Guo, Ali Ahmed, Josh West, Guillaume LECROC, Mate Varga
fire : Sijie Guo, jia zhai, Ali Ahmed, Josh West, haphut
tada : Sijie Guo, Ali Ahmed, Josh West
ship : Sijie Guo, Ali Ahmed
bananadance : Josh West, Ali Ahmed
----
2018-05-30 01:00:43 UTC - Matteo Merli: @Matteo Merli pinned a message to this channel.
----
2018-05-30 01:01:26 UTC - Matteo Merli: @Matteo Merli set the channel topic: General channel - Pulsar release 2.0.0-rc1-incubating
----
2018-05-30 01:06:51 UTC - Matteo Merli: <https://twitter.com/apache_pulsar/status/1001630926697185282>
+1 : Josh West, Ali Ahmed, jia zhai, Sanjeev Kulkarni, Guillaume LECROC, Sijie Guo, Bogdan BUNECI, Marco Didonna
bananadance : Josh West, Ali Ahmed, jia zhai, Yuvaraj Loganathan, Sijie Guo, Bogdan BUNECI
thumbsup_all : jia zhai, Sijie Guo, Bogdan BUNECI, Ali Ahmed
tada : Sijie Guo, Bogdan BUNECI, Ali Ahmed
----
2018-05-30 01:09:57 UTC - Byron: Nice job @Matteo Merli and team!
----
2018-05-30 05:15:22 UTC - Guillaume LECROC: Congrats! I was waiting it to start our POC :)
We recently try Kafka but the idea to loose messages is not possible for us (we need to replace our JMS Weblogic BUS). Plus the correlation between partition and consumer, different storage extension ...
+1 : Sijie Guo, Jon Bock
----