You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@pulsar.apache.org by "lhotari (via GitHub)" <gi...@apache.org> on 2024/06/11 15:53:30 UTC

[PR] [improve][misc] Replace dependencies on PositionImpl with Position interface [pulsar]

lhotari opened a new pull request, #22891:
URL: https://github.com/apache/pulsar/pull/22891

   ### Motivation
   
   The Pulsar Managed Ledger module contains an interface `org.apache.bookkeeper.mledger.Position`.
   The general recommendation of using interfaces is to depend on the interface and not on implementation details.
   However, this principle isn't followed in the code and there are a lot of casting to `PositionImpl` in the code and tight coupling to the implementation.
   
   Another problem with the current code base is that batch index acknowledgement state ("ack set") is also coupled to the Position implementation. This PR provides a solution where "ack set" handling is decoupled from the `Position` interface without changing the architecture. It will continue to be possible to create a Position instance that carries and holds the "ack set" state for those execution paths that depend on this.
   
   The motivation of this PR is to cleanup code before Pulsar 4.0 which is scheduled for October. The current master branch will most likely become 4.0. Since 4.0 will be the next LTS release, we would like to complete larger changes in the code base before the release. 
   
   The Position interface isn't exposed in Pulsar end user applications. Therefore this change can be considered as internal cleanup and refactoring which doesn't necessary require a PIP decision before proceeding. I'll bring this PR up to discussion on the mailing list regardless so this change doesn't come as a surprise to other contributors.
   
   Third party Pulsar broker plugins depending on managed ledger interfaces will need to be adapted. However, this is a simple process. It's a simple replacement of `PositionImpl` with `Position` and using `PositionFactory` for the creation of Position instances.
   
   After this PR, I'm planning to continue cleaning Managed Ledger the `ManagedLedger` and `ManagedCursor` interfaces. This PR is a pre-requisite for that work. Decoupling Pulsar core from Managed Ledger implementations will open up ways to add new Managed Ledger implementations in the future and make it pluggable.
   
   ### Modifications
   
   - replace all usages of `PositionImpl` with `Position` and remove `PositionImpl` completely to remove any confusion in the future.
   - add `PositionFactory` for creating new instances so that there's no need for coupling to `PositionImpl`
   - decouple the "ack set" batch index acknowledgement state from `Position` with a solution (`AckSetState`) that doesn't change the current architecture. 
     - it will continue to be possible to create a Position instance that carries and holds the "ack set" state for those execution paths that depend on this.
   - make the default implementation of Position immutable so that thread safety issues are avoided with the instance state
     - there are separate implementations to support object recycling and carrying "ack set" state
   
   ### Documentation
   
   <!-- DO NOT REMOVE THIS SECTION. CHECK THE PROPER BOX ONLY. -->
   
   - [ ] `doc` <!-- Your PR contains doc changes. -->
   - [ ] `doc-required` <!-- Your PR changes impact docs and you will update later -->
   - [x] `doc-not-needed` <!-- Your PR changes do not impact docs -->
   - [ ] `doc-complete` <!-- Docs have been already added -->
   
   ### Matching PR in forked repository
   
   PR in forked repository: https://github.com/lhotari/pulsar/pull/193


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [PR] [improve][misc] Replace dependencies on PositionImpl with Position interface [pulsar]

Posted by "lhotari (via GitHub)" <gi...@apache.org>.
lhotari commented on code in PR #22891:
URL: https://github.com/apache/pulsar/pull/22891#discussion_r1637917795


##########
managed-ledger/src/main/java/org/apache/bookkeeper/mledger/Position.java:
##########
@@ -26,16 +27,116 @@
  */
 @InterfaceAudience.LimitedPrivate
 @InterfaceStability.Stable
-public interface Position {
+public interface Position extends Comparable<Position> {
+    /**
+     * Get the ledger id of the entry pointed by this position.
+     *
+     * @return the ledger id
+     */
+    long getLedgerId();
+
+    /**
+     * Get the entry id of the entry pointed by this position.
+     *
+     * @return the entry id
+     */
+    long getEntryId();
+
+    /**
+     * Compare this position with another position.
+     * The comparison is first based on the ledger id, and then on the entry id.
+     * This is implements the Comparable interface.
+     * @param that the other position to be compared.
+     * @return -1 if this position is less than the other, 0 if they are equal, 1 if this position is greater than
+     * the other.
+     */
+    default int compareTo(Position that) {
+        if (getLedgerId() != that.getLedgerId()) {
+            return (getLedgerId() < that.getLedgerId() ? -1 : 1);
+        }
+
+        if (getEntryId() != that.getEntryId()) {
+            return (getEntryId() < that.getEntryId() ? -1 : 1);
+        }
+
+        return 0;

Review Comment:
   Thanks @merlimat, nice improvement.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [PR] [improve][misc] Replace dependencies on PositionImpl with Position interface [pulsar]

Posted by "dave2wave (via GitHub)" <gi...@apache.org>.
dave2wave commented on PR #22891:
URL: https://github.com/apache/pulsar/pull/22891#issuecomment-2163626638

   This change is so massive that I think docs are in fact required to explain all of the changes in a TL/DR manner.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [PR] [improve][misc] Replace dependencies on PositionImpl with Position interface [pulsar]

Posted by "lhotari (via GitHub)" <gi...@apache.org>.
lhotari merged PR #22891:
URL: https://github.com/apache/pulsar/pull/22891


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [PR] [improve][misc] Replace dependencies on PositionImpl with Position interface [pulsar]

Posted by "codecov-commenter (via GitHub)" <gi...@apache.org>.
codecov-commenter commented on PR #22891:
URL: https://github.com/apache/pulsar/pull/22891#issuecomment-2161290900

   ## [Codecov](https://app.codecov.io/gh/apache/pulsar/pull/22891?dropdown=coverage&src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) Report
   Attention: Patch coverage is `86.93467%` with `78 lines` in your changes missing coverage. Please review.
   > Project coverage is 73.27%. Comparing base [(`bbc6224`)](https://app.codecov.io/gh/apache/pulsar/commit/bbc62245c5ddba1de4b1e7cee4ab49334bc36277?dropdown=coverage&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) to head [(`5586660`)](https://app.codecov.io/gh/apache/pulsar/commit/5586660e843d2adaae6e82c726ed9ce835c2f71c?dropdown=coverage&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache).
   > Report is 372 commits behind head on master.
   
   <details><summary>Additional details and impacted files</summary>
   
   
   [![Impacted file tree graph](https://app.codecov.io/gh/apache/pulsar/pull/22891/graphs/tree.svg?width=650&height=150&src=pr&token=acYqCpsK9J&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache)](https://app.codecov.io/gh/apache/pulsar/pull/22891?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache)
   
   ```diff
   @@             Coverage Diff              @@
   ##             master   #22891      +/-   ##
   ============================================
   - Coverage     73.57%   73.27%   -0.31%     
   - Complexity    32624    33028     +404     
   ============================================
     Files          1877     1896      +19     
     Lines        139502   142008    +2506     
     Branches      15299    15565     +266     
   ============================================
   + Hits         102638   104050    +1412     
   - Misses        28908    29942    +1034     
   - Partials       7956     8016      +60     
   ```
   
   | [Flag](https://app.codecov.io/gh/apache/pulsar/pull/22891/flags?src=pr&el=flags&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | Coverage Δ | |
   |---|---|---|
   | [inttests](https://app.codecov.io/gh/apache/pulsar/pull/22891/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `27.49% <41.77%> (+2.91%)` | :arrow_up: |
   | [systests](https://app.codecov.io/gh/apache/pulsar/pull/22891/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `24.76% <30.03%> (+0.43%)` | :arrow_up: |
   | [unittests](https://app.codecov.io/gh/apache/pulsar/pull/22891/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `72.29% <86.93%> (-0.56%)` | :arrow_down: |
   
   Flags with carried forward coverage won't be shown. [Click here](https://docs.codecov.io/docs/carryforward-flags?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#carryforward-flags-in-the-pull-request-comment) to find out more.
   
   | [Files](https://app.codecov.io/gh/apache/pulsar/pull/22891?dropdown=coverage&src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | Coverage Δ | |
   |---|---|---|
   | [...a/org/apache/bookkeeper/mledger/ManagedCursor.java](https://app.codecov.io/gh/apache/pulsar/pull/22891?src=pr&el=tree&filepath=managed-ledger%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fbookkeeper%2Fmledger%2FManagedCursor.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-bWFuYWdlZC1sZWRnZXIvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2Jvb2trZWVwZXIvbWxlZGdlci9NYW5hZ2VkQ3Vyc29yLmphdmE=) | `33.33% <ø> (-1.97%)` | :arrow_down: |
   | [...org/apache/bookkeeper/mledger/PositionFactory.java](https://app.codecov.io/gh/apache/pulsar/pull/22891?src=pr&el=tree&filepath=managed-ledger%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fbookkeeper%2Fmledger%2FPositionFactory.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-bWFuYWdlZC1sZWRnZXIvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2Jvb2trZWVwZXIvbWxlZGdlci9Qb3NpdGlvbkZhY3RvcnkuamF2YQ==) | `100.00% <100.00%> (ø)` | |
   | [...rg/apache/bookkeeper/mledger/impl/AckSetState.java](https://app.codecov.io/gh/apache/pulsar/pull/22891?src=pr&el=tree&filepath=managed-ledger%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fbookkeeper%2Fmledger%2Fimpl%2FAckSetState.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-bWFuYWdlZC1sZWRnZXIvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2Jvb2trZWVwZXIvbWxlZGdlci9pbXBsL0Fja1NldFN0YXRlLmphdmE=) | `100.00% <100.00%> (ø)` | |
   | [.../org/apache/bookkeeper/mledger/impl/EntryImpl.java](https://app.codecov.io/gh/apache/pulsar/pull/22891?src=pr&el=tree&filepath=managed-ledger%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fbookkeeper%2Fmledger%2Fimpl%2FEntryImpl.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-bWFuYWdlZC1sZWRnZXIvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2Jvb2trZWVwZXIvbWxlZGdlci9pbXBsL0VudHJ5SW1wbC5qYXZh) | `78.31% <100.00%> (-0.17%)` | :arrow_down: |
   | [...bookkeeper/mledger/impl/ImmutablePositionImpl.java](https://app.codecov.io/gh/apache/pulsar/pull/22891?src=pr&el=tree&filepath=managed-ledger%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fbookkeeper%2Fmledger%2Fimpl%2FImmutablePositionImpl.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-bWFuYWdlZC1sZWRnZXIvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2Jvb2trZWVwZXIvbWxlZGdlci9pbXBsL0ltbXV0YWJsZVBvc2l0aW9uSW1wbC5qYXZh) | `100.00% <100.00%> (ø)` | |
   | [...ookkeeper/mledger/impl/ManagedCursorContainer.java](https://app.codecov.io/gh/apache/pulsar/pull/22891?src=pr&el=tree&filepath=managed-ledger%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fbookkeeper%2Fmledger%2Fimpl%2FManagedCursorContainer.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-bWFuYWdlZC1sZWRnZXIvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2Jvb2trZWVwZXIvbWxlZGdlci9pbXBsL01hbmFnZWRDdXJzb3JDb250YWluZXIuamF2YQ==) | `96.10% <100.00%> (+0.64%)` | :arrow_up: |
   | [...kkeeper/mledger/impl/ManagedLedgerFactoryImpl.java](https://app.codecov.io/gh/apache/pulsar/pull/22891?src=pr&el=tree&filepath=managed-ledger%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fbookkeeper%2Fmledger%2Fimpl%2FManagedLedgerFactoryImpl.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-bWFuYWdlZC1sZWRnZXIvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2Jvb2trZWVwZXIvbWxlZGdlci9pbXBsL01hbmFnZWRMZWRnZXJGYWN0b3J5SW1wbC5qYXZh) | `82.60% <100.00%> (+1.22%)` | :arrow_up: |
   | [.../bookkeeper/mledger/impl/NonDurableCursorImpl.java](https://app.codecov.io/gh/apache/pulsar/pull/22891?src=pr&el=tree&filepath=managed-ledger%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fbookkeeper%2Fmledger%2Fimpl%2FNonDurableCursorImpl.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-bWFuYWdlZC1sZWRnZXIvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2Jvb2trZWVwZXIvbWxlZGdlci9pbXBsL05vbkR1cmFibGVDdXJzb3JJbXBsLmphdmE=) | `81.25% <100.00%> (ø)` | |
   | [...org/apache/bookkeeper/mledger/impl/OpAddEntry.java](https://app.codecov.io/gh/apache/pulsar/pull/22891?src=pr&el=tree&filepath=managed-ledger%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fbookkeeper%2Fmledger%2Fimpl%2FOpAddEntry.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-bWFuYWdlZC1sZWRnZXIvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2Jvb2trZWVwZXIvbWxlZGdlci9pbXBsL09wQWRkRW50cnkuamF2YQ==) | `73.84% <100.00%> (+1.02%)` | :arrow_up: |
   | [...g/apache/bookkeeper/mledger/impl/OpFindNewest.java](https://app.codecov.io/gh/apache/pulsar/pull/22891?src=pr&el=tree&filepath=managed-ledger%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fbookkeeper%2Fmledger%2Fimpl%2FOpFindNewest.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-bWFuYWdlZC1sZWRnZXIvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2Jvb2trZWVwZXIvbWxlZGdlci9pbXBsL09wRmluZE5ld2VzdC5qYXZh) | `89.85% <100.00%> (+0.30%)` | :arrow_up: |
   | ... and [64 more](https://app.codecov.io/gh/apache/pulsar/pull/22891?src=pr&el=tree-more&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | |
   
   ... and [365 files with indirect coverage changes](https://app.codecov.io/gh/apache/pulsar/pull/22891/indirect-changes?src=pr&el=tree-more&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache)
   
   </details>


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [PR] [improve][misc] Replace dependencies on PositionImpl with Position interface [pulsar]

Posted by "merlimat (via GitHub)" <gi...@apache.org>.
merlimat commented on code in PR #22891:
URL: https://github.com/apache/pulsar/pull/22891#discussion_r1635150179


##########
managed-ledger/src/main/java/org/apache/bookkeeper/mledger/Position.java:
##########
@@ -26,16 +27,116 @@
  */
 @InterfaceAudience.LimitedPrivate
 @InterfaceStability.Stable
-public interface Position {
+public interface Position extends Comparable<Position> {
+    /**
+     * Get the ledger id of the entry pointed by this position.
+     *
+     * @return the ledger id
+     */
+    long getLedgerId();
+
+    /**
+     * Get the entry id of the entry pointed by this position.
+     *
+     * @return the entry id
+     */
+    long getEntryId();
+
+    /**
+     * Compare this position with another position.
+     * The comparison is first based on the ledger id, and then on the entry id.
+     * This is implements the Comparable interface.
+     * @param that the other position to be compared.
+     * @return -1 if this position is less than the other, 0 if they are equal, 1 if this position is greater than
+     * the other.
+     */
+    default int compareTo(Position that) {
+        if (getLedgerId() != that.getLedgerId()) {
+            return (getLedgerId() < that.getLedgerId() ? -1 : 1);

Review Comment:
   ```suggestion
               return Long.compare(getLedgerId(), that.getLedgerId());
   ```



##########
managed-ledger/src/main/java/org/apache/bookkeeper/mledger/Position.java:
##########
@@ -26,16 +27,116 @@
  */
 @InterfaceAudience.LimitedPrivate
 @InterfaceStability.Stable
-public interface Position {
+public interface Position extends Comparable<Position> {
+    /**
+     * Get the ledger id of the entry pointed by this position.
+     *
+     * @return the ledger id
+     */
+    long getLedgerId();
+
+    /**
+     * Get the entry id of the entry pointed by this position.
+     *
+     * @return the entry id
+     */
+    long getEntryId();
+
+    /**
+     * Compare this position with another position.
+     * The comparison is first based on the ledger id, and then on the entry id.
+     * This is implements the Comparable interface.
+     * @param that the other position to be compared.
+     * @return -1 if this position is less than the other, 0 if they are equal, 1 if this position is greater than
+     * the other.
+     */
+    default int compareTo(Position that) {
+        if (getLedgerId() != that.getLedgerId()) {
+            return (getLedgerId() < that.getLedgerId() ? -1 : 1);
+        }
+
+        if (getEntryId() != that.getEntryId()) {
+            return (getEntryId() < that.getEntryId() ? -1 : 1);
+        }
+
+        return 0;

Review Comment:
   ```suggestion
           return Long.compare(getEntryId(), that.getEntryId());
   ```



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [PR] [improve][misc] Replace dependencies on PositionImpl with Position interface [pulsar]

Posted by "lhotari (via GitHub)" <gi...@apache.org>.
lhotari commented on PR #22891:
URL: https://github.com/apache/pulsar/pull/22891#issuecomment-2165149124

   > This change is so massive that I think docs are in fact required to explain all of the changes in a TL/DR manner.
   
   @dave2wave The reason why this change is massive is mainly due to the PositionImpl -> Position/PositionFactory changes. The PR description already contains the explanation in a TL/DR manner in "Modifications". 
   Anything missing?


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org