Slack digest for #dev - 2020-09-26

[Apache Pulsar Slack <ap...@gmail.com>]

2020-09-25 13:11:09 UTC - Guillaume: Hi,
Periodically all of my subscribers and consumers get disconnected and I found these in the logs:

11:52:24.474 [GarbageCollectorThread-11-1] ERROR org.apache.bookkeeper.bookie.ScanAndCompareGarbageCollector - Exception when iterating through the ledgers to check for over-replication

11:53:33.448 [ReplicationWorker] WARN  org.apache.bookkeeper.replication.ReplicationWorker - BKNotEnoughBookiesException while replicating the fragment

Do you have an idea of what happens ?
Thanks.
----
2020-09-25 15:24:30 UTC - Lari Hotari: Hi, I've been investigating a Pulsar issue in a load test and I was finally able to describe the issue: <https://github.com/apache/pulsar/issues/8138> . I don't have a repro case yet since this is happening on a real system what is being load tested. The system currently breaks under heavy load because of a memory leak in the Pulsar Java client when using short lived Reader instances that are created, used and then closed by using the async Reader API. @Penghui Li , Would you be able to help me with this one? I'm willing to do as much as possible to get the issue resolved asap. Where to start?
----
2020-09-25 19:52:20 UTC - Enrico Olivelli: Hey Pulsar community,
We called for VOTE a release candidate for BK 4.11.1. It would be super useful that any of you could try it out (at least the server, that it 100% compatible with Pulsar) and cast your vote
----
2020-09-25 19:56:47 UTC - Jarrod Johnson: Hello! I am experimenting with the Postgres JDBC sink and it seems to work great for a flat payload where the message schema matches the table schema. What if you have a more complex message schema with nested types, for example? Is there a way to configure the sink to serialize the object and store it as json in the postgres table instead of having to map all fields?
----
2020-09-25 20:09:15 UTC - Nathan Clayton: @Nathan Clayton has joined the channel
----
2020-09-25 20:11:38 UTC - Addison Higham: really impressive but report @Lari Hotari :slightly_smiling_face:

I don't have much more context to add, but based on your notes and a quick look at the code, I agree with your assessment. We have seen similar issues of hanging future before on the broker side and have since worked me on ensuring features have a built in timeout. That may work well here, see <https://github.com/apache/pulsar/blob/d4a1ca59dafa1a8f4050bbaa442543250f789bf2/pulsar-broker/src/main/java/org/apache/pulsar/broker/service/BrokerService.java#L831> for the util we call to create a future that times out and notice the  usage in that class.

The other thing I would wonder if is there is a nice clean way to cancel all requests once we close a reader. It is a bit tricky since the requests get multiplexed across a socket, but perhaps there exists an easy way to know the outstanding request ids for a given consumer so they could then be cancelled
----
2020-09-25 21:49:09 UTC - Devin G. Bost: My team noticed an issue today where our produce rate was slowly decreasing cluster-wide over about a week. Nothing seemed to fix the issue until we restarted the ZK leader. After that, the rate went back up to normal.
----
2020-09-25 21:52:07 UTC - Devin G. Bost: You might want to try asking in <#C5Z4T36F7|general>
----
2020-09-25 21:55:56 UTC - Devin G. Bost: I don't have experience with that sink, but you may need to create a custom sink if you want to change its behavior.
----
2020-09-25 22:01:31 UTC - Jarrod Johnson: Thanks @Devin G. Bost. After looking at the source for the sink it looks like this is not supported so you are right.
+1 : Devin G. Bost
----
2020-09-25 23:45:00 UTC - Addison Higham: did you try broker restarts?
----
2020-09-26 04:56:16 UTC - N Navin: @N Navin has joined the channel
----