[DISCUSS] Changing MetadataStore interface to support configuring cache expiration, fixing bug with leader broker leader election

[Lari Hotari <lh...@apache.org>]

Hi,

There's a bug in Pulsar where a broker loses the leader broker information after 10 minutes. 

Logs will contain entries such as:
WARN org.apache.pulsar.broker.namespace.NamespaceService - The information about the current leader broker wasn't available. Handling load manager decisions in a decentralized way.

There is PR https://github.com/apache/pulsar/pull/17401 to address this issue. Please review

The MetadataStore interface is changed to support configuring cache expiration. In this case, the leader election information for the leader broker is made to not expire.
The MetadataStore interface change is a breaking change for the direct implementors of the MetadataStore interface. Please let me know if this is a change that shouldn't be done in maintenance branches.

In addition, there's a change to refresh the leader broker information every 5 minutes to address possible concerns of cache staleness. There was a discussion about the cache staleness concerns and mitigations for that in the existing solution. That happened in the last Pulsar community meeting, notes in the mailing list thread https://lists.apache.org/thread/y53rnvdryz8fsd4txylmjl0wropco16q .

-Lari