Slack digest for #general - 2019-12-05

[Apache Pulsar Slack <ap...@gmail.com>]

2019-12-04 09:18:52 UTC - Brian Doran: Failed to connect .. which property in standalone.conf is that?
----
2019-12-04 09:18:58 UTC - Brian Doran: I have
# Whether statistics are enabled
enableStatistics=true

# Stats Provider Class (if statistics are enabled)
statsProviderClass=org.apache.bookkeeper.stats.prometheus.PrometheusMetricsProvider

# Default port for Prometheus metrics exporter
prometheusStatsHttpPort=8000
----
2019-12-04 09:19:30 UTC - Brian Doran: in standalone.conf .. copied from bookeeper.conf it's enabled there too.
----
2019-12-04 09:19:48 UTC - Brian Doran: is bookeeper.conf used in standalone mode?
----
2019-12-04 10:16:48 UTC - Keith: hi, I have just run a dependency checker against 2.4.1 and I get the following vulnerabilities ...
```[ERROR] pulsar-client-2.4.1.jar\META-INF/maven/org.apache.commons/commons-compress/pom.xml: CVE-2018-1324, CVE-2018-11771, CVE-2019-12402
[ERROR] pulsar-client-2.4.1.jar\META-INF/maven/org.eclipse.jetty/jetty-util/pom.xml: CVE-2019-10247, CVE-2019-10241
[ERROR] pulsar-client-2.4.1.jar\META-INF/maven/io.netty/netty-tcnative-boringssl-static/pom.xml: CVE-2019-16869, CVE-2015-2156, CVE-2014-3488
[ERROR] pulsar-client-2.4.1.jar\META-INF/maven/io.netty/netty-all/pom.xml: CVE-2019-16869
[ERROR] pulsar-client-2.4.1.jar\META-INF/maven/com.fasterxml.jackson.core/jackson-databind/pom.xml: CVE-2019-16942, CVE-2019-16943, CVE-2019-17267, CVE-2019-17531, CVE-2019-16335, CVE-2019-14540
[ERROR] pulsar-client-2.4.1.jar\META-INF/maven/com.typesafe.netty/netty-reactive-streams/pom.xml: CVE-2019-16869, CVE-2015-2156, CVE-2014-3488
[ERROR] pulsar-client-2.4.1.jar\META-INF/maven/com.google.guava/guava/pom.xml: CVE-2018-10237```
Is there any plan to fix these?
----
2019-12-04 11:22:34 UTC - tuteng: Bookie's HTTP service is not turned on in standalone mode of pulsar.
----
2019-12-04 12:35:05 UTC - Brian Doran: Got it. Thanks. @tuteng
----
2019-12-04 14:52:05 UTC - sbourke: @sbourke has joined the channel
----
2019-12-04 15:20:08 UTC - juraj: hi @Sijie Guo, just an idea for the future - if candidate release images were pushed to docker hub, more people would be able to participate on the validation (e.g. running them on their dev envs)
----
2019-12-04 16:15:14 UTC - Vladimir Shchur: Hi, guys! While developing .net client we've found two issues with nacks and batch messages, are they known issues? Let's say we have 10 messages in a batch
1. Let's say MaxRedeliverCount is 1 consumer acked 9 messages and 1 nacked. All 10 messages will be redelivered which will lead to double processing of 9 messages
2. Let's say MaxRedeliverCount is 0, so no redelivers happen. Consumer acks 9 messages and 1 nacks. This will lead to all 10 messages coming to dead letter queue.
----
2019-12-04 16:26:52 UTC - Sijie Guo: thank you @juraj ! Add @xiaolong.ran who is the release manager of 2.4.2. @xiaolong.ran can you incorporate juraj’s idea into improving our future releases?
+1 : juraj
----
2019-12-04 16:28:12 UTC - Sijie Guo: I think so of them are already fixed. They will probably be released as part of 2.5.0 and 2.4.1 releases.
----
2019-12-04 16:33:08 UTC - Sijie Guo: 1) I think this is related to current acknowledgment tracking is done at entry (batch) level. If 10 messages are batched into one batch, you might see 10 messages being redelivered. There is a GitHub issue tracking the enhancement.

2) this might be related to 1). Can you file a GitHub issue for it? @Penghui Li can take a look at it.
----
2019-12-04 16:35:22 UTC - Sijie Guo: I remember there was a bug related to listing non-persistent topics. That can be related. Can you explain more about the behaviors you have seen when listing non-persistent topics?
----
2019-12-04 18:57:01 UTC - Grant English: @Grant English has joined the channel
----
2019-12-04 19:04:31 UTC - Vladimir Shchur: @Sijie Guo Thank you, will create the issue for 2. As for 1, I couldn't find the issue to give my vote to, can you please help?
----
2019-12-04 19:06:37 UTC - Ryan: Hello, where is Pulsar at with consumer filtering? We have a use case where we have multiple high volume, low latency topics that are being read by multiple consumers. Each of these consumers are reading the entire data stream from these topics, but ultimately only need to process ~1-5% of the data, which is causing a lot of bandwidth waste on our network. While I understand that we could use Lambda functions to create a million curated topics, due to the nature of our environment this is untenable. It would be significantly better if there was an option to allow the consumer to dynamically filter topic messages upon request or at subscription creation time. I found <https://github.com/apache/pulsar/issues/3302> but it appears to be closed. Is there any support or planned support for this feature?
----
2019-12-04 19:34:09 UTC - Sijie Guo: @Vladimir Shchur I think this is the related issue: <https://github.com/apache/pulsar/issues/4931>
----
2019-12-04 19:42:49 UTC - Vladimir Shchur: Thank you, will add my example there
----
2019-12-04 20:12:55 UTC - Ming: @Ryan Have you considered using Functions to do a content based routing? <https://pulsar.apache.org/docs/en/functions-overview/#content-based-routing-example>
----
2019-12-04 20:17:47 UTC - Vladimir Shchur: By the way, the ticket says area/client, does the server support redelivery by batch parts? If yes, we can implement it in dotnet client ourselves.
----
2019-12-04 20:19:58 UTC - Sijie Guo: ah, I think we need to add a tag to area/broker as well. because 1) the acknowledgement is done at the broker level 2) the redelivery account is also attached by broker
----
2019-12-04 20:23:27 UTC - Ryan: Right, as I said, creating a ton of additional topics using Functions is untenable. Having a server-side subscription-time dynamic filter per consumer would be ideal. Can you create a server-side subscription-time dynamic filter on a topic using Functions without creating additional downstream topics? The current Function API requires input and output topics.
----
2019-12-04 20:57:34 UTC - Greg Hoover: Looking for some guidance on using Pulsar for large messages up to 10MB in some cases or more. Content is nested JSON with embedded photos as they enter Pulsar on input topics. As these large messages are processed, the photos can be split out separately, thus decreasing the message size on downstream topics to a max of 5MB, and usually less than 1MB. Would be nice to keep these large messages in cold tiered S3 storage for replay at a later time if necessary, and to preserve an audit trail of exactly what data was contributed to the system in time sequence. Just wondering if we should try to use Pulsar for these large messages or if it’s better to seek a different solution. 
----
2019-12-04 21:25:00 UTC - David Kjerrumgaard: @Greg Hoover The max message size is now configurable for Pulsar.  See <https://medium.com/streamnative/whats-new-in-apache-pulsar-2-4-0-d646f6727642>
----
2019-12-04 21:25:28 UTC - David Kjerrumgaard: And <https://github.com/apache/pulsar/wiki/PIP-37%3A-Large-message-size-handling-in-Pulsar>
----
2019-12-04 21:27:30 UTC - David Kjerrumgaard: @Greg Hoover As for offloading to tiered storage, messages can be offloaded on a per topic basis via the <https://pulsar.apache.org/docs/en/pulsar-admin/#offload> command, so you can control when these messages are moved to S3.
----
2019-12-04 23:51:08 UTC - Addison Higham: can someone remind me... if we set an offload threshold on an existing namespace, will it eventually trigger offloads for all the topics in the namespace?
----
2019-12-04 23:59:38 UTC - Addison Higham: nm, found the doc, appears to happen when the next segment close/open happens... now just need to remember how I have my segments set up :slightly_smiling_face:
+1 : David Kjerrumgaard
thumbsup_all : jia zhai
----
2019-12-05 00:01:56 UTC - David Kjerrumgaard: Was getting ready to respond :smiley:
----
2019-12-05 00:12:23 UTC - Luke Lu: <https://github.com/apache/pulsar/issues/5798>
----
2019-12-05 02:02:34 UTC - baishen: @baishen has joined the channel
----
2019-12-05 08:41:07 UTC - juraj: ps. the 2.4.2 docker images are still not out - is this expected? can i help in any way?
----
2019-12-05 08:42:33 UTC - xiaolong.ran: i am pushing the docker images of 2.4.2
fireworks : juraj
----