You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@pulsar.apache.org by Apache Pulsar Slack <ap...@gmail.com> on 2019/10/30 09:11:03 UTC
Slack digest for #general - 2019-10-30
2019-10-29 15:43:23 UTC - Natalie: @Natalie has joined the channel
----
2019-10-29 18:13:36 UTC - tihomir: Hi guys, we run out of space on our server and after increasing the storage we cannot get our broker to work. It looks like there is some disrupt with the zk state and we are constantly getting org.apache.pulsar.client.api.PulsarClientException$LookupException: No active broker is available
----
2019-10-29 18:31:47 UTC - Addison Higham: ran out of space on bookies?
----
2019-10-29 18:31:57 UTC - Addison Higham: or zk?
----
2019-10-29 18:32:32 UTC - tihomir: on bookies
----
2019-10-29 19:02:57 UTC - Addison Higham: do your logs say something to the effect of them being unable to form a healthy ensemble? Bookkeeper has a config value, `diskUsageThreshold`, that once that threshold is reached, the bookie marks itself as unavailable for new segments. If for some reason, it hasn't picked up on the increased disk size (maybe it needs a restart?) that could be why it is still not healthy. I was able to get out of situation like this by stopping producers, slightly increasing the diskUsageThreshold, deleting some topics to free up space, then resuming
----
2019-10-29 19:07:40 UTC - Matteo Merli: > that once that threshold is reached, the bookie marks itself as unavailable for new segments
----
2019-10-29 19:08:06 UTC - Matteo Merli: not only for new segments, after the threshold is reached, the bookie turn into read-only mode, no more writes are accepted
----
2019-10-29 19:11:48 UTC - Addison Higham: that makes more sense
----
2019-10-29 19:18:25 UTC - tihomir: it looks like our zookeeper has got in corrupt state
----
2019-10-29 19:18:55 UTC - Matteo Merli: was disk full on ZK then ?
----
2019-10-29 19:19:03 UTC - tihomir: we have the following errors
org.apache.bookkeeper.meta.AbstractZkLedgerManager - Could not read metadata for ledger: 1473
org.apache.zookeeper.KeeperException$ConnectionLossException: KeeperErrorCode = ConnectionLoss for /ledgers/00/0000/L1473
at org.apache.zookeeper.KeeperException.create(KeeperException.java:10
----
2019-10-29 19:19:14 UTC - tihomir: no only on bookies
----
2019-10-29 19:19:42 UTC - tihomir: but it was full only on 2 of 7 nodes
----
2019-10-29 19:49:39 UTC - tihomir: even listing of active brokers in that region result in error
----
2019-10-29 19:49:55 UTC - tihomir:
----
2019-10-30 01:15:52 UTC - Sijie Guo: the error message indicates that you can not resolve the domain name “streams-broker”
----
2019-10-30 04:43:56 UTC - PRASAD VUNNAM: @PRASAD VUNNAM has joined the channel
----
2019-10-30 05:16:08 UTC - Saurabh J: @Saurabh J has joined the channel
----