Slack digest for #general - 2020-06-20

[Apache Pulsar Slack <ap...@gmail.com>]

2020-06-19 10:02:42 UTC - Konstantinos Papalias: @Sijie Guo Thanks for the great work organising Summit and secretly running the 2.6.0 release just on time for it, wanted to follow up on the JDK 11 release as part of it, has this been included on this release? I cannot see anything mentioned.
----
2020-06-19 11:35:42 UTC - sjmittal: I just create a deployment on minikue for a single node pulsar cluster and it seems my bookie is crashing:
Only thing I see in log is this:

`[conf/bookkeeper.conf] Applying config dbStorage_readAheadCacheMaxSizeMb = 32`
`[conf/bookkeeper.conf] Applying config dbStorage_writeCacheMaxSizeMb = 32`
`[conf/bookkeeper.conf] Applying config statsProviderClass = org.apache.bookkeeper.stats.prometheus.PrometheusMetricsProvider`
`[conf/bookkeeper.conf] Applying config zkServers = datastream-zookeeper-service`
`Error: Could not find or load main class "`
----
2020-06-19 11:56:22 UTC - Allen ONeill: *Architecture question using Pulsar*

Hi all, looking for input from you experienced folks on how to approach the following requirement:

Background:

• I have a single queue (jobs), and a scheduler that places new jobs into the queue at intervals, and a cluster of workers that process jobs.
• Jobs are placed into the queue in batches, one batch would be 100 jobs, another might be 10k, another might be 100k etc.
• I have all workers connected to the queue, and as they need a job they pop one off the queue - Pulsar is setup to distribute jobs to one consumer (worker) in round-robin pattern.
Requirement:

• I want to ensure that the jobs are distributed to the workers taking into account that there different batches of jobs, so that I am not only distributing jobs from the queue to workers evenly using round-robin, but also rotating through each of the batches in the queue in round-robin fashion. This means that even if I have 3 batches of differing sizes, I can be sure that as I round-robin through the workers (consumers), that I give out a job from each batch in turn, before looping back to the start again.
Clearly one of the issues here is how can pulsar know what defines a batch .... my thinking here is that I create a topic per batch, and have the workers/consumers subscribe to each topic, therefore achieving a round-robin of both batch + workers/consumers.

I would appreciate comments on this approach or if there is something inbuilt I can use to assist me in this!

Many thanks,

Allen.
----
2020-06-19 13:31:40 UTC - Marcio Martins: I also got this a few times, it is generally because of the JVM environment variables. The actual value needs to be wrapped in `"`
----
2020-06-19 13:33:48 UTC - sjmittal: well I am using a pulsar operator to do this so I don't have control over jvm env.
Is it something changed in their images recently
I pull the latest images
----
2020-06-19 13:34:14 UTC - Marcio Martins: Ah, not sure, what operator are you using btw?
----
2020-06-19 13:36:19 UTC - sjmittal: <https://github.com/sky-big/pulsar-operator>
but again I doubt its operator issue because few days back it works. I did a clean install again now and it fails
only thing that would have changed is the pulsar docker images
I believe the jvm is part of the image and I may not have control over the env variables
----
2020-06-19 13:52:24 UTC - Ebere Abanonu: Ok
----
2020-06-19 15:48:30 UTC - rwaweber: Hey all! Question on some of the metrics exposed by the pulsar broker:

This closed github issue has an example of what I’m talking about: <https://github.com/apache/pulsar/issues/4337>

It looks as though individual metrics are reporting multiple values. Specifically:
```pulsar_consumers_count{cluster="pulsar_bdx",namespace="ns/global",topic="<persistent://ns/global/queue>"} 1.0 1558518241859```
What’s the purpose of the second value here? It doesnt seem as though it gets picked up by prometheus. I can confirm that I’m seeing similar behavior on 2.5.2 if that helps
----
2020-06-19 16:04:30 UTC - Jeff Schneller: Are there binaries for the c++ client on windows? I only see Linux and MacOS. Looking to save time but not having to build myself if it is even possible.
----
2020-06-19 16:28:19 UTC - jujugrrr: Hi @Addison Higham, I've just tried and I was able to consume all the message up to the broken ledger. Same as the reader. Do you think there is a way to remove this reference to the broken ledger?
----
2020-06-19 16:30:51 UTC - Addison Higham: @jujugrrr see <https://pulsar.apache.org/docs/en/2.5.2/reference-configuration/> and the `autoSkipNonRecoverableData` flag, I just came across that again yesterday :slightly_smiling_face:
----
2020-06-19 16:30:57 UTC - Addison Higham: I knew there was something, just couldn't remember what
----
2020-06-19 16:31:13 UTC - jujugrrr: aah, let me try!
----
2020-06-19 16:31:40 UTC - jujugrrr: thanks for the slide deck by the way, I'm waiting of the replay of the talk, I couldn't attend unfortunately
----
2020-06-19 17:56:33 UTC - Aaron: Is it possible for deduplication to happen across multiple producers on the same partitioned topic?
----
2020-06-19 22:22:50 UTC - Matteo Merli: No, deduping is only meant to happen across a single producer. You can move that producer to a different host, and it will be recognized as long as you use the same producer name 
----