Slack digest for #general - 2020-10-24

[Apache Pulsar Slack <ap...@gmail.com>]

2020-10-23 09:38:12 UTC - Guillaume: *What is a "bundle" in Pulsar ?*
I see in namespace policies the notion of bundles. What is it for? What values should we set for numBundles ? I see also that by default when I create a namespace, the property "bundles_activated" is not set. Whereas for preexisting "test-namespace" namespace it is set to true.
Could someone help me understand it?
Thank you.
----
2020-10-23 09:45:34 UTC - Lari Hotari: I also pushed the PR to validate retention policy in Admin API: <https://github.com/apache/pulsar/pull/8358>
----
2020-10-23 10:05:02 UTC - Lari Hotari: This is a really good question.
The namespace bundle itself is explained in <https://pulsar.apache.org/docs/en/2.6.1/administration-load-distribution/#assignment-granularity> . It's part of the Pulsar load distribution / balancing / sharding architecture.
&gt; Instead of individual topic/partition assignment, each broker takes ownership of a subset of the topics for a namespace. This subset is called a "_bundle_" and effectively it's a sharding mechanism.
The documentation provides some examples of values. "For example, for a namespace with 1000 topics, using something like 64 bundles will achieve a good distribution of traffic across 16 brokers." However, a general advice for the bundle values is missing.

In Akka Cluster sharding (not related to Pulsar at all), there is this kind of general recommendation for the number of shards: _"As a rule of thumb, the number of shards should be a factor ten greater than the planned maximum number of cluster nodes."_ It would be nice if there could be such a rule of thumb also for Pulsar.

It might not actually be a bad idea to set namespace bundles to 10x number of brokers in the cluster. :thinking_face:
----
2020-10-23 10:13:24 UTC - Guillaume: I also saw that the bundle number can be related to the number of topics. Is it true ?
----
2020-10-23 10:39:22 UTC - Lari Hotari: Bundle splitting has the maximum number of topics as one of the configurable parameters: <https://pulsar.apache.org/docs/en/administration-load-balance/#split-namespace-bundles> .
----
2020-10-23 14:41:42 UTC - otman: I have a question about consumers. I use a java program that is subscribing to muti-topic pattern (server side). Until now, in client side i worked with few client (no more 100 which are producers) but now i'm testing with 50k client (producers) and 4 topics by client. My java program (consumer) is blocked on "... Connected to server ..." message in my console. When i try it again (starting a consumer) with a specific topic; that worked. Is there any limitation about multi topic pattern consumer ?
----
2020-10-23 15:09:14 UTC - Kenny Sargent: Hi Addison, we’re setting up an in-house SDK that makes it easy for microservice developers to consume Pulsar. We’re trying to get our defaults, and what we parameters we allow to be passed in, working. From the docs, we understand that DLQ only works with Shared subscription types (Shared/KeyShared.) (2.6.1 says “Currently, dead letter topic is enabled only in the shared subscription mode.“) Our testing confirms this. We are allowing them to easily setup a DLQ with Shared/KeyShared (which would be normal behavior). To answer your question, we’re trying to fully understand the behavior with and without a DLQ because we want to allow all subscription types and allow DLQs with any types that allow them.
----
2020-10-23 15:49:17 UTC - Kenny Sargent: Therefore, my main question is still about RedeliveryCount. _Should_ it increment with Exclusive and Failover types? (It increments with or without a DLQ on Shared types)
----
2020-10-23 18:52:17 UTC - Leo Sht: @Leo Sht has joined the channel
----
2020-10-23 20:42:03 UTC - Jeff Schneller: trying to connect my producer and it never connects.   c# code follows.  I can ping the broker.  The auth should be correct as this code worked earlier.  I can connect via the admin cli and see stats on the topic.

 `PulsarClient client = new PulsarClientBuilder()`
                `.ServiceUrl(serviceUrl)`
                `.EnableTls(true)`
                `.TlsTrustCertificate(x509Cert)`
                `.Authentication(auth)`
                `.Build();`

            `IProducer&lt;MyStatus&gt; producer = null;`
            `try`
            `{`
                `producer = await client.NewProducer(Schema.JSON&lt;MyStatus&gt;()).Topic(topicName).EnableBatching(false).CreateAsync();`
            `}`
            `catch (Exception e)`
            `{`
                `Console.WriteLine(e.Message);`
            `}`
----
2020-10-23 21:01:12 UTC - Jeff Schneller: As a followup it looks like I am getting a ProducerBusyException.  My producer isn't named so there should be no conflicts
----
2020-10-23 21:22:12 UTC - Jeff Schneller: The issue also occurs when 2 producers are being run at the same time.  As you see the code doesn't have a producer name so it should be generating a unique producer name each time.
----
2020-10-24 08:44:02 UTC - Vladimir Shchur: Hi, Jeff! Are you using latest library version? If yes, the conflicting name should be displayed in error message
----