[D] broker-client-shared-internal-executor issue [pulsar]

["programmerahul (via GitHub)" <gi...@apache.org>]

GitHub user programmerahul created a discussion: broker-client-shared-internal-executor issue

I see that **broker-client-shared-internal-executor**  thread is used to load system topic policies in SystemTopicBasedTopicPoliciesService.java while doing initPolicesCache. After loading the topic policies, same thread is used to call listener.onUpdate  ( to update the topic policy). But internally listener.onUpdate operation is a blocking operation. onUpdate(PersistentTopic) -> 
applyUpdatedTopicPolicies ->
checkMessageExpiry->
sub.expireMessages->
getNumberOfEntriesInBacklog->
topic.isOldestMessageExpired->
 cursor.getNthEntry->
counter.await(ledger.getConfig().getMetadataOperationsTimeoutSeconds(), TimeUnit.SECONDS); 
Here the default operation timeout is 1minute. 
That means, if there is a single topic for which the oldest ledger is not accessible for the time being, it will block loading all other topic for a significant time, depending on the number of subscription present on the topic, eg. if there are 20 subscription on the topic , then policies loading of other system-topics can be delayed by 20mins(This happens due to recursive loading of policies event, eg. an ns with 20 topics, has to complete 20 callbacks one after other, to load the topic policy) I have observed this in out heap dump

GitHub link: https://github.com/apache/pulsar/discussions/25517

----
This is an automatically sent email for commits@pulsar.apache.org.
To unsubscribe, please send an email to: commits-unsubscribe@pulsar.apache.org

Re: [D] broker-client-shared-internal-executor issue [pulsar]

["Denovo1998 (via GitHub)" <gi...@apache.org>]

GitHub user Denovo1998 added a comment to the discussion: broker-client-shared-internal-executor issue

We really need to pay attention to [PIP-469](https://github.com/apache/pulsar/pull/25547).

GitHub link: https://github.com/apache/pulsar/discussions/25517#discussioncomment-16655194

----
This is an automatically sent email for commits@pulsar.apache.org.
To unsubscribe, please send an email to: commits-unsubscribe@pulsar.apache.org

Re: [D] broker-client-shared-internal-executor issue [pulsar]

["lhotari (via GitHub)" <gi...@apache.org>]

GitHub user lhotari added a comment to the discussion: broker-client-shared-internal-executor issue

@programmerahul Thanks for bringing this up. Please file an issue at https://github.com/apache/pulsar/issues/new/choose

GitHub link: https://github.com/apache/pulsar/discussions/25517#discussioncomment-16555461

----
This is an automatically sent email for commits@pulsar.apache.org.
To unsubscribe, please send an email to: commits-unsubscribe@pulsar.apache.org