You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@pulsar.apache.org by Apache Pulsar Slack <ap...@gmail.com> on 2018/06/01 09:11:04 UTC
Slack digest for #general - 2018-06-01
2018-05-31 10:22:52 UTC - Nicolas Ha: ```
java.lang.Thread.run Thread.java: 745
18-05-31 10:11:08 85e89a648918 ERROR [org.apache.zookeeper.server.ZooKeeperServer:472] - ZKShutdownHandler is not registered, so ZooKeeper server won't take any action on ERROR or SHUTDOWN server state changes
18-05-31 10:11:08 85e89a648918 ERROR [org.apache.bookkeeper.bookie.Journal:226] - Problems reading from /tmp/bk0test8166890473414122594/current/lastMark (this is okay if it is the first time starting this bookie
18-05-31 10:11:10 85e89a648918 WARN [org.apache.pulsar.zookeeper.ZkBookieRackAffinityMapping:145] - Error getting bookie info from zk, using default rack node /default-rack: KeeperErrorCode = NoNode for /bookies
18-05-31 10:11:10 85e89a648918 WARN [org.apache.pulsar.zookeeper.ZkBookieRackAffinityMapping:145] - Error getting bookie info from zk, using default rack node /default-rack: KeeperErrorCode = NoNode for /bookies
```
and later
```
18-05-30 19:52:21 85e89a648918 ERROR [org.apache.pulsar.broker.stats.metrics.JvmMetrics:162] - Failed to collect GC stats: java.lang:type=GarbageCollector,name=G1 Young Generation
18-05-30 20:58:11 85e89a648918 WARN [org.apache.zookeeper.ClientCnxn:1108] - Client session timed out, have not heard from server in 3934702ms for sessionid 0x163b2760ae10003
18-05-30 20:58:11 85e89a648918 WARN [org.apache.zookeeper.server.NIOServerCnxn:368] - caught end of stream exception
org.apache.zookeeper.server.NIOServerCnxnFactory.run NIOServerCnxnFactory.java: 203
org.apache.zookeeper.server.NIOServerCnxn.doIO NIOServerCnxn.java: 239
org.apache.zookeeper.server.ServerCnxn$EndOfStreamException: Unable to read additional data from client sessionid 0x163b2760ae10003, likely client has closed socket
```
and at the end
```
18-05-30 20:58:12 85e89a648918 WARN [org.apache.pulsar.zookeeper.ZookeeperBkClientFactoryImpl:73] - Unexpected ZK event received: WatchedEvent state:Disconnected type:None path:null
8-05-30 20:58:12 85e89a648918 WARN [org.apache.zookeeper.ClientCnxn:1285] - Unable to reconnect to ZooKeeper service, session 0x163b2760ae10002 has expired
18-05-30 20:58:12 85e89a648918 ERROR [org.apache.bookkeeper.zookeeper.ZooKeeperWatcherBase:89] - ZooKeeper client connection to the ZooKeeper server has expired!
18-05-30 20:58:12 85e89a648918 WARN [org.apache.zookeeper.ClientCnxn:1285] - Unable to reconnect to ZooKeeper service, session 0x163b2760ae10003 has expired
18-05-30 20:58:12 85e89a648918 ERROR [org.apache.bookkeeper.zookeeper.ZooKeeperWatcherBase:89] - ZooKeeper client connection to the ZooKeeper server has expired!
18-05-30 20:58:12 85e89a648918 ERROR [org.apache.pulsar.zookeeper.ZooKeeperSessionWatcher:95] - ZooKeeper session already expired, invoking shutdown
```
Does the first message mean that pulsar never connected to ZK? And is there a way to configure ZK to not shutdown?
----
2018-05-31 10:27:51 UTC - Ivan Kelly: which services are these logs from?
----
2018-05-31 11:28:16 UTC - Nicolas Ha: Ah I should say - I instanciate the pulsar server class myself. My question is basically: does it mean I am passing a wrong config (ex. wrong ZK port)?
----
2018-05-31 11:29:02 UTC - Nicolas Ha: I create a LocalBookkeeperEnsemble and PulsarService and PulsarAdmin object
----
2018-05-31 12:11:57 UTC - William Fry: @Matteo Merli do you mind merging? Looks like it got approved but I don’t have write perms
----
2018-05-31 12:38:48 UTC - Ivan Kelly: ah, for LocalBookKeeperEnsemble, i think those logs are normal
----
2018-05-31 12:39:22 UTC - Ivan Kelly: can you post the full log for PulsarService?
----
2018-05-31 13:20:04 UTC - William Fry: is `msgRateOut` intended to be the number of messages ack’d per second?
----
2018-05-31 13:20:50 UTC - William Fry: also, how frequently are stats computed? there seems to be some level of latency
----
2018-05-31 13:23:44 UTC - Poule: what happen is my Zookeeper cluster is destroyed?
----
2018-05-31 13:29:46 UTC - Nicolas Ha: there were no messages during that time, so I assume that’s why I don’t get logs from the broker?
apart from 15 lines similar to:
```
18-05-31 10:26:20 85e89a648918 ERROR [org.apache.pulsar.broker.stats.metrics.JvmMetrics:162] - Failed to collect GC stats: java.lang:type=GarbageCollector,name=G1 Young Generation
```
----
2018-05-31 13:32:48 UTC - William Fry: It seems the `msgRateIn` refreshes frequently but `msgRateOut` is typically 0
----
2018-05-31 13:38:05 UTC - William Fry: Is anyone using the Python package with a shared setup?
----
2018-05-31 13:39:04 UTC - William Fry: My code is below but I cannot create two processes that consume messages at the same time. If I do that, one hangs until I kill the other…
```
self.consumer = self.client.subscribe(self.topic_name,
consumer_type=pulsar.ConsumerType.Shared,
subscription_name=self.subscription_name)
```
----
2018-05-31 13:43:37 UTC - Ivan Kelly: @Poule you loose zk irrecoverably, you need to start from scratch
----
2018-05-31 13:43:54 UTC - Ivan Kelly: @Nicolas Ha what's the actual problem you're seeing?
----
2018-05-31 13:44:41 UTC - Ivan Kelly: @William Fry looks like a bug
----
2018-05-31 13:44:47 UTC - Nicolas Ha: After that last message
```
18-05-30 20:58:12 85e89a648918 ERROR [org.apache.pulsar.zookeeper.ZooKeeperSessionWatcher:95] - ZooKeeper session already expired, invoking shutdown
```
it exits the system (code 255)
----
2018-05-31 13:45:09 UTC - William Fry: cc: @Ivan Kelly is this a bug? Or the Python PIP package?
----
2018-05-31 13:45:33 UTC - Ivan Kelly: ah, can you describe you're full setup? ZooKeeper session expiring indicates something really weird happening
----
2018-05-31 13:46:39 UTC - Ivan Kelly: describe your setup?
----
2018-05-31 13:47:10 UTC - William Fry: I have producers and consumers writing to a persistent topic
----
2018-05-31 13:47:31 UTC - William Fry: I hit the REST API to get stats
----
2018-05-31 13:47:40 UTC - William Fry: Using a url like this one: `<http://localhost:8080/admin/persistent/sample/standalone/ns1/my-topic/stats>`
----
2018-05-31 13:48:21 UTC - William Fry: I get a JSON response where the value for the key `msgRateIn` is > 0.0 but the value for the key `msgRateOut` is always 0.0 (despite ack’ing messages from the consumer)
----
2018-05-31 13:48:39 UTC - William Fry: Furthermore, these values (e.g. only `msgRateIn` seem to update once every 30/60s)
----
2018-05-31 13:50:20 UTC - William Fry: I’d be interested in two questions:
1) How frequently are these stats updated? and are they only updated every X seconds for the past 1 second or are they averages?
2) Why is `msgRateOut` 0.0? Is there any other measure of how many messages have been ack’d by a consumer?
cc: @Ivan Kelly
----
2018-05-31 13:51:59 UTC - Ivan Kelly: How are you running pulsar?
----
2018-05-31 13:52:54 UTC - Ivan Kelly: actually, doesn't matter, confusing this with nha
----
2018-05-31 13:52:59 UTC - William Fry: Using docker locally
----
2018-05-31 13:53:30 UTC - Harald Gustafsson: Hi, is it possible to set namespace permissions even if you have started pulsar service without authentication? I seem to get an 500 status back and the server log has a null pointer exception. In general it does not make much sense to set permissions without authentication but this is a test script and I was hoping to skip the authentication during tests.
----
2018-05-31 13:53:34 UTC - Ivan Kelly: let me check the update freq
----
2018-05-31 13:53:55 UTC - William Fry: What I’m worried about is if the answer to 1) is that stats are updated every 30 seconds for the last second, then there’s a chance that no messages were ack’d during that last second although a number of messages were published.
----
2018-05-31 13:54:31 UTC - William Fry: In which case, it’d be hard to ever get a positive msgRateOut given the read/write ratio of the project I’m working on
----
2018-05-31 13:54:38 UTC - Ivan Kelly: @Harald Gustafsson I doubt it, but i guess it's not been tested. Does the same endpoint return a 2xx response when you do have authentication enabled?
----
2018-05-31 13:56:44 UTC - Ivan Kelly: what is msgThroughputOut saying?
----
2018-05-31 13:56:55 UTC - William Fry: also `0.0`
----
2018-05-31 13:58:05 UTC - Harald Gustafsson: @Ivan Kelly That is at least how I recall it. I turned off authentication since I was doing some wireshark to see exact interactions between admin client and pulsar
----
2018-05-31 13:59:19 UTC - Ivan Kelly: then update rate is every 60 seconds
----
2018-05-31 13:59:38 UTC - Ivan Kelly: BrokerService.java:L319
----
2018-05-31 14:00:19 UTC - William Fry: Ok and it’s computed from the last second? or the average from the previous 60 seconds?
----
2018-05-31 14:01:51 UTC - Ivan Kelly: @Harald Gustafsson ya, it's not possible if authorization is disabled. I'm not sure if you can have authorization without authentication
----
2018-05-31 14:01:51 UTC - Nicolas Ha: Full setup? Well I basically instanciate a `LocalBookkeeperEnsemble` and pass it a port for zookeeper, then I call `startStandalone` on it.
I then create a PulsarService (and a PulsarAdmin) and call `start` on it.
PulsarService and clients are running on the same JVM, along with my application code. I can send/receive messages just fine.
----
2018-05-31 14:02:43 UTC - Nicolas Ha: @Nicolas Ha uploaded a file: <https://apache-pulsar.slack.com/files/U7TGK04MS/FAZFA8ZBP/screen_shot_2018-05-31_at_15.00.57.png|Screen Shot 2018-05-31 at 15.00.57.png> and commented: these are the last logs I get
----
2018-05-31 14:02:52 UTC - Ivan Kelly: From scratch each 60 seconds as far as I can see
----
2018-05-31 14:04:07 UTC - Harald Gustafsson: @Ivan Kelly thanks, I look in the configuration to see if it can be turned on separately
----
2018-05-31 14:04:36 UTC - Nicolas Ha: I suspect `startStandalone` assumes it can exit (but there was no `start` method)
----
2018-05-31 14:04:38 UTC - Ivan Kelly: @Nicolas Ha scroll up, what's that NPE?
----
2018-05-31 14:04:41 UTC - William Fry: Ok so could it be 0.0 because every 60 seconds the second it computes the message there are no ack’d messages during that second?
----
2018-05-31 14:05:13 UTC - William Fry: (on a high-level, I’m trying to compute topic message growth.. e.g. msgRateIn - msgRateOut to auto-scale consumers)
----
2018-05-31 14:05:35 UTC - Ivan Kelly: well, yes, but are you going 60 seconds without consuming messages?
----
2018-05-31 14:05:42 UTC - Nicolas Ha: @Nicolas Ha uploaded a file: <https://apache-pulsar.slack.com/files/U7TGK04MS/FAZFCP1S9/screen_shot_2018-05-31_at_15.05.02.png|Screen Shot 2018-05-31 at 15.05.02.png>
----
2018-05-31 14:05:51 UTC - William Fry: no no
----
2018-05-31 14:06:03 UTC - William Fry: if it’s every 60 seconds for the previous 60 seconds, I should have a positive `msgRateOut`
----
2018-05-31 14:07:08 UTC - Ivan Kelly: ya, sounds like a bug.
----
2018-05-31 14:08:14 UTC - Nicolas Ha: So it is the global ZK? Not sure I start one global ZK actually
----
2018-05-31 14:08:39 UTC - Nicolas Ha: (as opposed to the one that comes with the Local BK ensemble)
----
2018-05-31 14:09:09 UTC - William Fry: fudge
----
2018-05-31 14:09:26 UTC - Ivan Kelly: you can actually use the same zk for both global and local if you only have one cluster
----
2018-05-31 14:09:42 UTC - Ivan Kelly: Could you create an issue in github, with a full description of how to repro?
----
2018-05-31 14:09:51 UTC - Nicolas Ha: ok maybe that’s what I did - let me check my config
----
2018-05-31 14:09:57 UTC - William Fry: Sure!
----
2018-05-31 14:10:42 UTC - Ivan Kelly: the zk ensemble you use is from a localbookeeper running in the same process?
----
2018-05-31 14:10:49 UTC - Ivan Kelly: thanks
----
2018-05-31 14:13:06 UTC - Nicolas Ha: I don’t start a ZK myself. I use `org.apache.pulsar.zookeeper.LocalBookkeeperEnsemble`
----
2018-05-31 14:13:22 UTC - Ivan Kelly: same process?
----
2018-05-31 14:13:48 UTC - Nicolas Ha: I think so yes. You mean I should create a Thread to run the start methods?
----
2018-05-31 14:15:32 UTC - Ivan Kelly: it's just I can't see any zookeeper server logs there
----
2018-05-31 14:18:48 UTC - William Fry: I’m going to try on 2.0.0, just realized I was using a homebrew formulae that referenced 1.~
----
2018-05-31 14:24:20 UTC - William Fry: Ok, so this is interesting, the first 60 second stat update is correct, afterwhich the msgRateIn gets updated by the msgRateOut does not.
----
2018-05-31 14:24:33 UTC - Daniel Morgan: 15:23:25.691 [0x7fff894c2380] ERROR /Users/mmerli/prg/pulsar/pulsar-client-cpp/lib/DestinationName.cc(55) - Destination Name Invalid, domain not present - mytopic
15:23:25.691 [0x7fff894c2380] ERROR /Users/mmerli/prg/pulsar/pulsar-client-cpp/lib/DestinationName.cc(139) - Destination Name Initialization failed
----
2018-05-31 14:24:45 UTC - Daniel Morgan: just reading the getting started docs and coming up against that
----
2018-05-31 14:25:02 UTC - Ivan Kelly: what is your topicname?
----
2018-05-31 14:25:19 UTC - Daniel Morgan: consumer = client.subscribe('mytopic', 'mysubscription')
----
2018-05-31 14:25:47 UTC - Ivan Kelly: topic should be of the form "persistent://<tenant>/<namespace>/topic"
----
2018-05-31 14:26:04 UTC - Ivan Kelly: there's a public tenant and default namespace you can use
----
2018-05-31 14:26:15 UTC - Daniel Morgan: okay I was just copy pastaing from
----
2018-05-31 14:26:16 UTC - Daniel Morgan: <https://pulsar.incubator.apache.org/docs/latest/getting-started/docker/>
----
2018-05-31 14:26:20 UTC - Daniel Morgan: cheers for heads up
----
2018-05-31 14:26:54 UTC - Ivan Kelly: :confused: I pointed out those were wrong 2 weeks ago
----
2018-05-31 14:28:22 UTC - Daniel Morgan: <persistent://public/default/my-topic>
----
2018-05-31 14:28:41 UTC - Daniel Morgan: Destination Name Invalid, does not have enough parts
----
2018-05-31 14:28:58 UTC - Daniel Morgan: added a topic
----
2018-05-31 14:29:02 UTC - Daniel Morgan: seems to have fixed it
----
2018-05-31 14:29:11 UTC - Daniel Morgan: <persistent://public/default/topic/my-topic>
----
2018-05-31 14:29:16 UTC - Daniel Morgan: thanks ivan
----
2018-05-31 14:29:28 UTC - Ivan Kelly: probably something with the new topic format
----
2018-05-31 14:29:33 UTC - Ivan Kelly: c++ lags a bit
----
2018-05-31 14:30:03 UTC - Daniel Morgan: :slightly_smiling_face: cheers
----
2018-05-31 14:35:46 UTC - Ivan Kelly: that's weird, I'm looking at the code which should be updating this
----
2018-05-31 14:45:35 UTC - Matteo Merli: @Daniel Morgan it seems it's picking the 1.22 Python client
----
2018-05-31 14:46:00 UTC - Matteo Merli: Otherwise, the new naming would be accepted
----
2018-05-31 14:46:29 UTC - William Fry: I created a different topic in a different namespace and its working now
----
2018-05-31 14:46:36 UTC - William Fry: … not sure why
----
2018-05-31 14:49:08 UTC - William Fry: well it seemed to work for two refreshes, but now `msgRateOut` is flatlining again
----
2018-05-31 14:52:43 UTC - Ivan Kelly: :confused:
----
2018-05-31 14:54:41 UTC - William Fry: Ok found some pattern.
----
2018-05-31 14:54:49 UTC - William Fry: It seems like it’s incorrectly aggregating?
----
2018-05-31 14:55:03 UTC - William Fry: It looks like the `msgRateOut` is that of the first consumer to connect
----
2018-05-31 14:55:28 UTC - William Fry: And if the first consumer hits a computationally expensive task and sits on it for a while, then it obviously has 0.0 and `msgRateOut` goes to 0
----
2018-05-31 14:55:46 UTC - William Fry: Despite other consumers having positive `msgRateOut` values (not reflected in the stats)
----
2018-05-31 14:58:23 UTC - Daniel Morgan: yes pip3 install pulsar-client got me 1.22.0
----
2018-05-31 14:59:33 UTC - William Fry: Ok, so it looks like its an issue with the consumer stats
----
2018-05-31 15:00:04 UTC - William Fry: Whenever a consumer subscribes to a topic, it reports stats correctly for the first 1-2 periods. After which, it incorrectly is 0.0.
----
2018-05-31 15:00:31 UTC - William Fry: I don’t know if this is a bug in the stats calculations or in the consumer logic (just got into this codebase yesterday), but that’s the pattern I’m seeing
----
2018-05-31 15:03:22 UTC - Daniel Morgan: pip install pulsar-client --no-cache still gives me 1.22
----
2018-05-31 15:03:28 UTC - William Fry: Does that make sense @Ivan Kelly?
----
2018-05-31 15:03:59 UTC - Ivan Kelly: one sec
----
2018-05-31 15:04:41 UTC - Ivan Kelly: subMsgRateOut += consumerStats.msgRateOut;
----
2018-05-31 15:04:46 UTC - Ivan Kelly: it should be taking from all
----
2018-05-31 15:05:21 UTC - Nicolas Ha: I will leave it running longer but you are right, it looks like it couldn’t connect to ZK due to a wrong port (how was it working for sending/receiving msg I am not sure now)
The ZK messages show now as well (after logging the INFO messages)
Thanks!
----
2018-05-31 15:06:16 UTC - Daniel Morgan: ah it looks like no python36 macos version of the client
----
2018-05-31 15:06:44 UTC - Daniel Morgan: only py27 for macos there
----
2018-05-31 15:07:44 UTC - Daniel Morgan: <https://pypi.org/project/pulsar-client/2.0.0/#files>
----
2018-05-31 15:20:37 UTC - William Fry: Hmm
----
2018-05-31 15:26:16 UTC - William Fry: Has anyone else used a shared subscription with multiple consumers where the msgRateOut was correct?
----
2018-05-31 15:41:27 UTC - William Fry: I can’t find what would make metrics plateau after 1 period
----
2018-05-31 16:01:50 UTC - Ivan Kelly: afraid I don't know
----
2018-05-31 17:08:32 UTC - William Fry: Gotcha, yeah it’s bizarre. It’s getting overwritten somehow.
----
2018-05-31 17:41:14 UTC - Josh West: <https://bookkeeper.apache.org/distributedlog/docs/0.5.0/user_guide/implementation/storage.html>
----
2018-05-31 17:41:52 UTC - Josh West: @Matteo Merli should we be configuring the ensemble placement policy / network topology settings for bookkeeper w/ Bookkeeper for Pulsar, to ensure that data is replicated across availability zones for example?
----
2018-05-31 17:46:07 UTC - Matteo Merli: Yes, though while it’s possible to write a custom placement policy there’s already one available in BookKeeper and extended in Pulsar (and configured by default). The only trick is that it relies having the mapping (bookie --> rack) in a ZK z-node and we currently have not much documentation or tooling around that. Tooling and docs around rack-aware are planned to go in 2.1 release.
----
2018-05-31 17:47:10 UTC - Josh West: @Matteo Merli is there anything you can point me at so i can see how things are done now with 2.0?
----
2018-05-31 17:49:45 UTC - Matteo Merli: there was a brief discussion here. The attached text file has an example of the content for the `/bookies` z-node
----
2018-05-31 17:50:09 UTC - Matteo Merli: The tooling basically will allow to create and update that z-node from CLI
----
2018-05-31 17:51:20 UTC - Matteo Merli: the rack names can be any string, BK client will ensure (with best effort) to place replicas for same data in different racks
----
2018-05-31 17:52:07 UTC - Matteo Merli: best effort in the sense that if bookies are not available in different racks, it falls back into picking from same rack
----
2018-05-31 17:53:22 UTC - Josh West: @Matteo Merli cool. where is the attached text file?
----
2018-05-31 17:54:14 UTC - Matteo Merli: ups forgot the link: <https://github.com/apache/incubator-pulsar/issues/151>
----
2018-05-31 18:09:04 UTC - Josh West: thanks @Matteo Merli
----
2018-06-01 00:00:24 UTC - Matteo Merli: @Matti-Pekka Laaksonen Opened a PR to fix this issue <https://github.com/apache/incubator-pulsar/pull/1882>
----
2018-06-01 00:01:12 UTC - Matteo Merli: It will be included in 2.0.1 release along with other bug fixes for 2.0.
A temp workaround is to disable batching on the producer.
----
2018-06-01 06:51:01 UTC - Kesav Kolla: just installed pulsar-client via pip and it's not working at all.
```
>>> import pulsar
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "/home/kesav/.virtualenvs/spydrenv/local/lib/python2.7/site-packages/pulsar.py", line 103, in <module>
from functions.function import Function
ImportError: No module named functions.function
```
----
2018-06-01 06:52:02 UTC - Kesav Kolla: tried with both python2.7 and python3.6 same error in both versions
----
2018-06-01 06:53:16 UTC - Ali Ahmed: can you print the output of
``` pip show pulsar-client```
----
2018-06-01 06:55:02 UTC - Kesav Kolla: ```
Name: pulsar-client
Version: 2.0.0
Summary: Apache Pulsar Python client library
Home-page: <http://pulsar.incubator.apache.org/>
Author: Pulsar Devs
Author-email: <ma...@pulsar.incubator.apache.org>
License: Apache License v2.0
Location: /home/kesav/.virtualenvs/spydrenv/lib/python3.6/site-packages
Requires:
Required-by:
```
----
2018-06-01 06:56:20 UTC - Ali Ahmed: You can try the older version
```
pip uninstall pulsar-client
pip install pulsar-client==1.22
```
----
2018-06-01 06:56:34 UTC - Ali Ahmed: 2.0 package may have to be republished
----
2018-06-01 06:58:04 UTC - Kesav Kolla: yeah older version works
----
2018-06-01 06:58:40 UTC - Ali Ahmed: I will try to fix 2.0
----
2018-06-01 07:00:00 UTC - Kesav Kolla: please publish for osx version also for 2.0
----
2018-06-01 07:01:03 UTC - Ali Ahmed: yes need to make some packaging changes will probably release with 2.0.1 bug fix release
----
2018-06-01 08:47:02 UTC - Kishore Karunakaran: Hi All,
is Pulsar right tool for trading systems ? particularly subscribing on websocket endpoints. Requirements : low latency and less GC
----
2018-06-01 08:53:17 UTC - Mate Varga: You'll probably need to provide more info here. 'low latency' -- specify that further
----
2018-06-01 08:58:09 UTC - Kishore Karunakaran: Trading system placing buy/sell equity or FX orders to the market will target a latency of under 20ms
----
2018-06-01 09:00:27 UTC - Ali Ahmed: pulsar will be fine for that
----
2018-06-01 09:00:55 UTC - Kishore Karunakaran: Awesome, Thanks !!
----