Slack digest for #general - 2020-06-04

[Apache Pulsar Slack <ap...@gmail.com>]

2020-06-03 10:41:27 UTC - Pushkar Sawant: 1
----
2020-06-03 10:54:54 UTC - Huanli Meng: will improve the doc quality. thanks
+1 : Frank Kelly
----
2020-06-03 11:50:27 UTC - Vladimir Shchur: Hi, can someone please explain how pulsar client _should_ behave when <https://github.com/apache/pulsar/blob/branch-2.5/pulsar-client/src/main/java/org/apache/pulsar/client/impl/PersistentAcknowledgmentsGroupingTracker.java#L183-L184|acknowledge> fails. What I see now from code
1. <https://github.com/apache/pulsar/blob/branch-2.5/pulsar-client/src/main/java/org/apache/pulsar/client/impl/PersistentAcknowledgmentsGroupingTracker.java#L174-L181|pendingIndividualAcks gets empty right before sending>, so it will stay empty on error
2. if consumer is non-durable it starts from first message in message queue on reconnect, which will lead to unacknowledged messages not being redelivered, so *messages that failed to acknowledge will be not acknowledged again*
3. if consumer is durable, it starts from the beginning on reconnect and will get the unacknowledged messages again, however <https://github.com/apache/pulsar/blob/branch-2.5/pulsar-client/src/main/java/org/apache/pulsar/client/impl/ConsumerImpl.java#L866|this check> will not pass (message will be regarded as duplicate) since <https://github.com/apache/pulsar/blob/branch-2.5/pulsar-client/src/main/java/org/apache/pulsar/client/impl/PersistentAcknowledgmentsGroupingTracker.java#L59|lastCumlativeAck> is not reset on failure which again results in *not acknowledging messages that failed to be acknowledged before*
Am I missing something? Thank you
----
2020-06-03 13:45:12 UTC - Rounak Jaggi: @Sijie Guo more about how do we upgrade/migrate to new pulsar version.
----
2020-06-03 13:47:27 UTC - jujugrrr: @jujugrrr has joined the channel
----
2020-06-03 14:17:25 UTC - Ebere Abanonu: @Sijie Guo @Penghui Li is it possible for you to make ProducerSuccess to return the latest Schema too? The current implementation only return the SchemaVersion
----
2020-06-03 14:24:03 UTC - Addison Higham: @Rounak Jaggi here is the pulsar upgrade guide <http://pulsar.apache.org/docs/en/administration-upgrade/|http://pulsar.apache.org/docs/en/administration-upgrade/>
----
2020-06-03 14:25:45 UTC - Addison Higham: More specifically, you can code some of that up in ansible and when you apply your ansible you can use a different host list
----
2020-06-03 15:12:42 UTC - Ebere Abanonu: I will take that back! Since schema is added to producer creation that will not be needed!
----
2020-06-03 15:40:54 UTC - Chris DiGiovanni: I see conflicting info in the docs of what version of Java should be used.  In this link (<https://pulsar.apache.org/docs/en/2.5.2/deploy-bare-metal/>) I see:
```Each machine in your cluster needs to have Java 8 or higher version of Java installed.```
In the standalone docs (<https://pulsar.apache.org/docs/en/2.5.2/standalone/>) I see:
```Pulsar is currently available for MacOS and Linux. To use Pulsar, you need to install Java 8 from Oracle download center.```
In the docker images I see it's being packaged up with version
```openjdk version "1.8.0_242"
OpenJDK Runtime Environment (build 1.8.0_242-b08)
OpenJDK 64-Bit Server VM (build 25.242-b08, mixed mode)```
So my question is, What Java version should we be using?  I'm asking as we are using jdk11.0.2 and we saw a suspicious bug fix around TLS1.3 and resuming sessions being fixed in jdk11.0.3.  Which looks like it could be linked to our Proxy to Broker 5xx return codes.  All these docs seem to have a theme around JAVA8, though before I go through all the trouble of changing all our bare-metal deployments, I would like to see if I could get some confirmation of what should actually be used.  Java8 from Oracle, OpenJDK8, or actually any version of JDK &gt;=8?
+1 : Frank Kelly
----
2020-06-03 16:09:14 UTC - jujugrrr: Hi All,
I'm trying to work on a robust disaster recovery plan but without relying on geo-replication(cost/location constraints). Looking a Tiered storage it seems we can offload as soon as possible or based on size. If using as soon as possible it seems to be happening as soon as a segment is closed. I'm struggling to find what is a segment size or when is it closed. It looks like a bookkeeper segment can be rolled based on time and not size only. Does that mean there is a way to configure segment to roll every 5 mins by example and offload to Tiered asap, which would allow us to lose 5mins of data in the worst case scenario(we lost all our bookies and persistent storage) ?

If we follow this approach, is it possible to create a new Pulsar cluster, point it to the existing Tiered storage (or a copy of it) and all data will be available?
----
2020-06-03 16:12:19 UTC - Matteo Merli: &gt;  Does that mean there is a way to configure segment to roll every 5 mins by example and offload to Tiered asap, which would allow us to lose 5mins of data in the worst case scenario(we lost all our bookies and persistent storage) ?
Yes, you can configure the rollover timings

&gt;  If we follow this approach, is it possible to create a new Pulsar cluster, point it to the existing Tiered storage (or a copy of it) and all data will be available?
Not exactly, while the data will be in S3, the metadata for the old topics is gone, so the new cluster will not be able to recover it as simply.
----
2020-06-03 16:20:43 UTC - jujugrrr: If we had zookeeper snapshot would it be enough?
----
2020-06-03 16:25:51 UTC - jujugrrr: I mean with something like <https://github.com/mhausenblas/burry.sh>
----
2020-06-03 16:31:22 UTC - Sijie Guo: what does “unable to load” mean?
----
2020-06-03 16:33:44 UTC - Sijie Guo: Java 8 is recommended. I think there were issues when you try to run it with Java 11 (can’t recall it is functions related issues or not) because the binary is compiled with java 8.
heavy_check_mark : Chris DiGiovanni
----
2020-06-03 16:52:59 UTC - Chris DiGiovanni: Thanks for the reply
----
2020-06-04 04:02:57 UTC - Luke Stephenson: We enabled log compaction on the brokers using `pulsar-admin namespaces set-compaction-threshold`.  While I was aware that doing log cleaning on the brokers would create some memory pressure, I was hopeful that any given broker would only compact 1 topic at a time. However with partitioned topics, it appears that the broker is kicking off compaction with multiple partitions at once.  While I've given the broker enough memory to allow compacting 1 partition at a time, running several in parallel causes it to die.  Here is a screenshot showing compaction kicking off for 12 partitions of the same topic within 1 second on the same broker.  Looking at pulsar source code, it looks like only 1 partition should be compacted at a time.
----
2020-06-04 06:27:50 UTC - Ankush: Hi All,
I am playing with `KeySharedPolicy.stickyHashRange()` and noticed that we can only provide blocks of sequential ranges to consumers, ex: consumer-0: [0-9999], consumer 1: [10000-1999], consumer 2: [20000-3999], and so on. This does not guarantee a good distribution of load (specially if you have few keys).
I was thinking of using consistent hashing to assign portions of hash ranges distributed equally around the ring to the consumer but get the error `Range conflict with consumer` from `HashRangeExclusiveStickyKeyConsumerSelector.validateKeySharedMeta` . I am trying to understand what is the reason for doing this?
For easily testing this, we can start one consumer with
```KeySharedPolicy.stickyHashRange().ranges(Range.of(0,999), Range.of(5000, 65535))```
and another consumer with
```KeySharedPolicy.stickyHashRange().ranges(Range.of(1000, 4999))```
Thanks!
----
2020-06-04 08:35:11 UTC - Craig: @Craig has joined the channel
----