Preparing for Pulsar 4.0: cleaning up the Managed Ledger interfaces

[Lari Hotari <lh...@apache.org>]

Hi all,

We have the next LTS release, Pulsar 4.0, scheduled for October. The
current master branch will most likely become the 4.0 release branch in
September.

Since 4.0 will be the next LTS release, we aim to complete larger
changes in the codebase before the release. I'm particularly interested
in cleaning up the Managed Ledger interface.

The cleanup of the Managed Ledger interface involves changing the Pulsar
core to depend on the interface, not on the implementation details.
Currently, this is not the case. A lot of Pulsar core code specifically 
depends on implementation classes such as ManagedLedgerImpl, 
ManagedCursorImpl and PositionImpl.

In addition to decoupling the core of Pulsar from the implementation
details of Managed Ledger, I'm also planning to rename the base package
of the Pulsar Managed Ledger module from org.apache.bookkeeper.mledger
to org.apache.pulsar.mledger. There are historical reasons why the
package prefix is org.apache.bookkeeper. I suppose there was a plan a
long time ago to move the module as part of the Bookkeeper project. That
goal no longer exists. The consequence of the
org.apache.bookkeeper.mledger package is that Managed Ledger log
messages contain the org.apache.bookkeeper.mledger prefix, which often
confuses Pulsar users and developers about the component's origin. This
will finally be addressed during this work.

I plan to make changes gradually in the master branch without breaking
external Pulsar end-user interfaces. Therefore, these changes can be
considered as internal cleanup and refactoring, which don't necessarily
require a PIP decision before proceeding.

I do plan to create a PIP document later to collect some of the design
decisions once they are made. However, at this moment, there are no
significant up-front design decisions that need to be made by the
community. In Apache projects, decisions are made on the mailing list
and this thread is one form of making decisions. Unless there are
objections to what is presented here, I will proceed with the work 
as planned. Individual work items will be reviewed in PRs as usual.

One detail to solve and document is how Pulsar code maintenance will be
handled once the master branch has diverged significantly from the
maintenance branches. It seems that Pulsar 4.0 will be the first branch
where there will be significant differences between the previous
maintenance branches. This challenge will be addressed in the PIP 
document when it becomes timely. It will also require documenting
changes to contribution guidelines so that contributors know how to 
contribute to the maintenance branches. Currently we primarily target
the master branch in PRs.

Decoupling Pulsar core from Managed Ledger implementations will open up
ways to add new Managed Ledger implementations in the future and make it
pluggable. This will help Apache Pulsar stay relevant in the future. I
believe other Pulsar contributors share and support this goal too.

Please review the first PR for cleaning up Managed Ledger interfaces:
"Replace dependencies on PositionImpl with Position interface"
https://github.com/apache/pulsar/pull/22891

Since this is a prerequisite for the next work items in Managed Ledger 
interface cleanup, I hope that we can complete the review of this PR asap.
We don't have much time to waste before the 4.0 release freeze in September.

Looking forward to your feedback and contributions in this area,

-Lari

Re: Preparing for Pulsar 4.0: cleaning up the Managed Ledger interfaces

[Asaf Mesika <as...@gmail.com>]

In high level I see this work as needed cleanup indeed and very much needed

On Tue, 11 Jun 2024 at 20:29 Lari Hotari <lh...@apache.org> wrote:

> Hi all,
>
> We have the next LTS release, Pulsar 4.0, scheduled for October. The
> current master branch will most likely become the 4.0 release branch in
> September.
>
> Since 4.0 will be the next LTS release, we aim to complete larger
> changes in the codebase before the release. I'm particularly interested
> in cleaning up the Managed Ledger interface.
>
> The cleanup of the Managed Ledger interface involves changing the Pulsar
> core to depend on the interface, not on the implementation details.
> Currently, this is not the case. A lot of Pulsar core code specifically
> depends on implementation classes such as ManagedLedgerImpl,
> ManagedCursorImpl and PositionImpl.
>
> In addition to decoupling the core of Pulsar from the implementation
> details of Managed Ledger, I'm also planning to rename the base package
> of the Pulsar Managed Ledger module from org.apache.bookkeeper.mledger
> to org.apache.pulsar.mledger. There are historical reasons why the
> package prefix is org.apache.bookkeeper. I suppose there was a plan a
> long time ago to move the module as part of the Bookkeeper project. That
> goal no longer exists. The consequence of the
> org.apache.bookkeeper.mledger package is that Managed Ledger log
> messages contain the org.apache.bookkeeper.mledger prefix, which often
> confuses Pulsar users and developers about the component's origin. This
> will finally be addressed during this work.
>
> I plan to make changes gradually in the master branch without breaking
> external Pulsar end-user interfaces. Therefore, these changes can be
> considered as internal cleanup and refactoring, which don't necessarily
> require a PIP decision before proceeding.
>
> I do plan to create a PIP document later to collect some of the design
> decisions once they are made. However, at this moment, there are no
> significant up-front design decisions that need to be made by the
> community. In Apache projects, decisions are made on the mailing list
> and this thread is one form of making decisions. Unless there are
> objections to what is presented here, I will proceed with the work
> as planned. Individual work items will be reviewed in PRs as usual.
>
> One detail to solve and document is how Pulsar code maintenance will be
> handled once the master branch has diverged significantly from the
> maintenance branches. It seems that Pulsar 4.0 will be the first branch
> where there will be significant differences between the previous
> maintenance branches. This challenge will be addressed in the PIP
> document when it becomes timely. It will also require documenting
> changes to contribution guidelines so that contributors know how to
> contribute to the maintenance branches. Currently we primarily target
> the master branch in PRs.
>
> Decoupling Pulsar core from Managed Ledger implementations will open up
> ways to add new Managed Ledger implementations in the future and make it
> pluggable. This will help Apache Pulsar stay relevant in the future. I
> believe other Pulsar contributors share and support this goal too.
>
> Please review the first PR for cleaning up Managed Ledger interfaces:
> "Replace dependencies on PositionImpl with Position interface"
> https://github.com/apache/pulsar/pull/22891
>
> Since this is a prerequisite for the next work items in Managed Ledger
> interface cleanup, I hope that we can complete the review of this PR asap.
> We don't have much time to waste before the 4.0 release freeze in
> September.
>
> Looking forward to your feedback and contributions in this area,
>
> -Lari
>