[DISCUSS] PIP-384: ManagedLedger interface decoupling

[Lari Hotari <lh...@apache.org>]

Dear Pulsar Community,

I have created PIP-384: ManagedLedger interface decoupling, which
documents recent improvements to Pulsar's ManagedLedger component.
This PIP summarizes changes that have already been merged into the
master branch for Pulsar 4.0, as well as one remaining draft PR.

Problem:
The current ManagedLedger implementation faces several challenges:
1. Tight coupling between interfaces and implementation
2. Limited flexibility for integrating different storage systems
3. Strong dependency on BookKeeper
4. Increased complexity due to tight coupling
5. Limited extensibility for new features or optimizations

Solution:
PIP-384 documents changes that:
1. Decouple ManagedLedger interfaces from their implementation
2. Introduce a ReadOnlyManagedLedger interface
3. Decouple OpAddEntry and LedgerHandle from ManagedLedgerInterceptor
4. Enable support for multiple ManagedLedgerFactory instances
5. Decouple BookKeeper client from ManagedLedgerStorage

Benefits:
1. Improved flexibility for introducing alternative implementations
2. Reduced coupling between core Pulsar components and specific
ManagedLedger implementations
3. Groundwork for potential alternative storage solutions in the future
4. Improved maintainability and testability
5. Enhanced extensibility for new features and optimizations

The full proposal can be found at: https://github.com/apache/pulsar/pull/23363

The direct link to the rendered version of the markdown file is:
https://github.com/lhotari/pulsar/blob/lh-pip-384/pip/pip-384.md

These changes have already been implemented in PRs #22891 and #23311,
with one remaining draft PR #23313. This PIP aims to document these
changes for future reference.

I welcome your feedback and discussion on this documentation. Please
share your thoughts, concerns, or suggestions.

-Lari

Re: [DISCUSS] PIP-384: ManagedLedger interface decoupling

[Rajan Dhabalia <dh...@gmail.com>]

+1 this will be also helpful to enhance various dispatch definitions at the
broker.

Thanks,
Rajan

On Fri, Sep 27, 2024 at 11:31 AM Lari Hotari <lh...@apache.org> wrote:

> Dear Pulsar Community,
>
> I have created PIP-384: ManagedLedger interface decoupling, which
> documents recent improvements to Pulsar's ManagedLedger component.
> This PIP summarizes changes that have already been merged into the
> master branch for Pulsar 4.0, as well as one remaining draft PR.
>
> Problem:
> The current ManagedLedger implementation faces several challenges:
> 1. Tight coupling between interfaces and implementation
> 2. Limited flexibility for integrating different storage systems
> 3. Strong dependency on BookKeeper
> 4. Increased complexity due to tight coupling
> 5. Limited extensibility for new features or optimizations
>
> Solution:
> PIP-384 documents changes that:
> 1. Decouple ManagedLedger interfaces from their implementation
> 2. Introduce a ReadOnlyManagedLedger interface
> 3. Decouple OpAddEntry and LedgerHandle from ManagedLedgerInterceptor
> 4. Enable support for multiple ManagedLedgerFactory instances
> 5. Decouple BookKeeper client from ManagedLedgerStorage
>
> Benefits:
> 1. Improved flexibility for introducing alternative implementations
> 2. Reduced coupling between core Pulsar components and specific
> ManagedLedger implementations
> 3. Groundwork for potential alternative storage solutions in the future
> 4. Improved maintainability and testability
> 5. Enhanced extensibility for new features and optimizations
>
> The full proposal can be found at:
> https://github.com/apache/pulsar/pull/23363
>
> The direct link to the rendered version of the markdown file is:
> https://github.com/lhotari/pulsar/blob/lh-pip-384/pip/pip-384.md
>
> These changes have already been implemented in PRs #22891 and #23311,
> with one remaining draft PR #23313. This PIP aims to document these
> changes for future reference.
>
> I welcome your feedback and discussion on this documentation. Please
> share your thoughts, concerns, or suggestions.
>
> -Lari
>