You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@bookkeeper.apache.org by "Sijie Guo (Created) (JIRA)" <ji...@apache.org> on 2012/04/01 17:38:28 UTC

[jira] [Created] (BOOKKEEPER-203) improve ledger manager interface to remove zookeeper dependency on metadata operations.

improve ledger manager interface to remove zookeeper dependency on metadata operations.
---------------------------------------------------------------------------------------

                 Key: BOOKKEEPER-203
                 URL: https://issues.apache.org/jira/browse/BOOKKEEPER-203
             Project: Bookkeeper
          Issue Type: Sub-task
          Components: bookkeeper-client, bookkeeper-server
            Reporter: Sijie Guo
            Assignee: Sijie Guo


we need to improve ledger manager interface to remove zookeeper dependency on metadata operations, so it is easy for us to implement a MetaStore based ledger manager.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

[jira] [Commented] (BOOKKEEPER-203) improve ledger manager interface to remove zookeeper dependency on metadata operations.

Posted by "Ivan Kelly (JIRA)" <ji...@apache.org>.
    [ https://issues.apache.org/jira/browse/BOOKKEEPER-203?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13395943#comment-13395943 ] 

Ivan Kelly commented on BOOKKEEPER-203:
---------------------------------------

BOOKKEEPER-203.diff_v6 doesn't apply cleanly to trunk? Did you generate the patch against the correct branch? It looks like the clash is against BOOKKEEPER-183.
                
> improve ledger manager interface to remove zookeeper dependency on metadata operations.
> ---------------------------------------------------------------------------------------
>
>                 Key: BOOKKEEPER-203
>                 URL: https://issues.apache.org/jira/browse/BOOKKEEPER-203
>             Project: Bookkeeper
>          Issue Type: Sub-task
>          Components: bookkeeper-client, bookkeeper-server
>            Reporter: Sijie Guo
>            Assignee: Sijie Guo
>             Fix For: 4.2.0
>
>         Attachments: BOOKKEEPER-203.diff, BOOKKEEPER-203.diff_v2, BOOKKEEPER-203.diff_v3, BOOKKEEPER-203.diff_v4, BOOKKEEPER-203.diff_v5, BOOKKEEPER-203.diff_v6
>
>
> we need to improve ledger manager interface to remove zookeeper dependency on metadata operations, so it is easy for us to implement a MetaStore based ledger manager.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

[jira] [Commented] (BOOKKEEPER-203) improve ledger manager interface to remove zookeeper dependency on metadata operations.

Posted by "Ivan Kelly (JIRA)" <ji...@apache.org>.
    [ https://issues.apache.org/jira/browse/BOOKKEEPER-203?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13294328#comment-13294328 ] 

Ivan Kelly commented on BOOKKEEPER-203:
---------------------------------------

Patch looks good. There's some small issues with it.

- Needs to be rebased on trunk
- MetaStoreException is not used. Is in wrong patch i guess?
- BookKeeper.java patch removed the throwing of NoSuchLedgerExistsException, which we added in BOOKKEEPER-273. Is this intentional?
- there are two overloads of getClass where only one is ever used. setClass is unused.
- LedgerManagerFactory#newLedgerManagerFactory breaks getLedgerManagerType(). if factoryClass is null and layout is null, it should check getLedgerManagerType before setting factoryClass to FlatLedgerManagerType or HierarchicalLedgerManagerType.
- This does introduce some incompatiblity. A 4.1.0 client will not work with an installation that has been initialised with the LedgerLayout code from this patch. This should be fine though, as LedgerLayout is only ever written on cluster initialization. However, I think we should change this to use protobufs so that any other changes like this are more BC friendly. Not in this patch though.
- You may as well you MultiLedgerManagerBaseTestCase for the places where you use MultiLedgerManagerTestCase and get rid of MultiLedgerManagerTestCase.

                
> improve ledger manager interface to remove zookeeper dependency on metadata operations.
> ---------------------------------------------------------------------------------------
>
>                 Key: BOOKKEEPER-203
>                 URL: https://issues.apache.org/jira/browse/BOOKKEEPER-203
>             Project: Bookkeeper
>          Issue Type: Sub-task
>          Components: bookkeeper-client, bookkeeper-server
>            Reporter: Sijie Guo
>            Assignee: Sijie Guo
>             Fix For: 4.2.0
>
>         Attachments: BOOKKEEPER-203.diff, BOOKKEEPER-203.diff_v2, BOOKKEEPER-203.diff_v3, BOOKKEEPER-203.diff_v4
>
>
> we need to improve ledger manager interface to remove zookeeper dependency on metadata operations, so it is easy for us to implement a MetaStore based ledger manager.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

[jira] [Commented] (BOOKKEEPER-203) improve ledger manager interface to remove zookeeper dependency on metadata operations.

Posted by "Hudson (JIRA)" <ji...@apache.org>.
    [ https://issues.apache.org/jira/browse/BOOKKEEPER-203?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13396685#comment-13396685 ] 

Hudson commented on BOOKKEEPER-203:
-----------------------------------

Integrated in bookkeeper-trunk #567 (See [https://builds.apache.org/job/bookkeeper-trunk/567/])
    BOOKKEEPER-203: improve ledger manager interface to remove zookeeper dependency on metadata operations. (sijie via ivank) (Revision 1351646)

     Result = ABORTED
ivank : 
Files : 
* /zookeeper/bookkeeper/trunk/CHANGES.txt
* /zookeeper/bookkeeper/trunk/bookkeeper-server/src/main/java/org/apache/bookkeeper/bookie/Bookie.java
* /zookeeper/bookkeeper/trunk/bookkeeper-server/src/main/java/org/apache/bookkeeper/bookie/GarbageCollectorThread.java
* /zookeeper/bookkeeper/trunk/bookkeeper-server/src/main/java/org/apache/bookkeeper/bookie/InterleavedLedgerStorage.java
* /zookeeper/bookkeeper/trunk/bookkeeper-server/src/main/java/org/apache/bookkeeper/bookie/LedgerCacheImpl.java
* /zookeeper/bookkeeper/trunk/bookkeeper-server/src/main/java/org/apache/bookkeeper/client/BKException.java
* /zookeeper/bookkeeper/trunk/bookkeeper-server/src/main/java/org/apache/bookkeeper/client/BookKeeper.java
* /zookeeper/bookkeeper/trunk/bookkeeper-server/src/main/java/org/apache/bookkeeper/client/BookKeeperAdmin.java
* /zookeeper/bookkeeper/trunk/bookkeeper-server/src/main/java/org/apache/bookkeeper/client/LedgerCreateOp.java
* /zookeeper/bookkeeper/trunk/bookkeeper-server/src/main/java/org/apache/bookkeeper/client/LedgerDeleteOp.java
* /zookeeper/bookkeeper/trunk/bookkeeper-server/src/main/java/org/apache/bookkeeper/client/LedgerHandle.java
* /zookeeper/bookkeeper/trunk/bookkeeper-server/src/main/java/org/apache/bookkeeper/client/LedgerMetadata.java
* /zookeeper/bookkeeper/trunk/bookkeeper-server/src/main/java/org/apache/bookkeeper/client/LedgerOpenOp.java
* /zookeeper/bookkeeper/trunk/bookkeeper-server/src/main/java/org/apache/bookkeeper/conf/AbstractConfiguration.java
* /zookeeper/bookkeeper/trunk/bookkeeper-server/src/main/java/org/apache/bookkeeper/meta/AbstractZkLedgerManager.java
* /zookeeper/bookkeeper/trunk/bookkeeper-server/src/main/java/org/apache/bookkeeper/meta/ActiveLedgerManager.java
* /zookeeper/bookkeeper/trunk/bookkeeper-server/src/main/java/org/apache/bookkeeper/meta/FlatLedgerManager.java
* /zookeeper/bookkeeper/trunk/bookkeeper-server/src/main/java/org/apache/bookkeeper/meta/FlatLedgerManagerFactory.java
* /zookeeper/bookkeeper/trunk/bookkeeper-server/src/main/java/org/apache/bookkeeper/meta/HierarchicalLedgerManager.java
* /zookeeper/bookkeeper/trunk/bookkeeper-server/src/main/java/org/apache/bookkeeper/meta/HierarchicalLedgerManagerFactory.java
* /zookeeper/bookkeeper/trunk/bookkeeper-server/src/main/java/org/apache/bookkeeper/meta/LedgerLayout.java
* /zookeeper/bookkeeper/trunk/bookkeeper-server/src/main/java/org/apache/bookkeeper/meta/LedgerManager.java
* /zookeeper/bookkeeper/trunk/bookkeeper-server/src/main/java/org/apache/bookkeeper/meta/LedgerManagerFactory.java
* /zookeeper/bookkeeper/trunk/bookkeeper-server/src/main/java/org/apache/bookkeeper/meta/ZkVersion.java
* /zookeeper/bookkeeper/trunk/bookkeeper-server/src/main/java/org/apache/bookkeeper/util/ReflectionUtils.java
* /zookeeper/bookkeeper/trunk/bookkeeper-server/src/main/java/org/apache/bookkeeper/versioning
* /zookeeper/bookkeeper/trunk/bookkeeper-server/src/main/java/org/apache/bookkeeper/versioning/Version.java
* /zookeeper/bookkeeper/trunk/bookkeeper-server/src/main/java/org/apache/bookkeeper/versioning/Versioned.java
* /zookeeper/bookkeeper/trunk/bookkeeper-server/src/test/java/org/apache/bookkeeper/bookie/LedgerCacheTest.java
* /zookeeper/bookkeeper/trunk/bookkeeper-server/src/test/java/org/apache/bookkeeper/client/BookieRecoveryTest.java
* /zookeeper/bookkeeper/trunk/bookkeeper-server/src/test/java/org/apache/bookkeeper/client/TestFencing.java
* /zookeeper/bookkeeper/trunk/bookkeeper-server/src/test/java/org/apache/bookkeeper/meta/GcLedgersTest.java
* /zookeeper/bookkeeper/trunk/bookkeeper-server/src/test/java/org/apache/bookkeeper/meta/HierarchicalAsyncLedgerOpsTest.java
* /zookeeper/bookkeeper/trunk/bookkeeper-server/src/test/java/org/apache/bookkeeper/meta/HierarchicalBookieFailureTest.java
* /zookeeper/bookkeeper/trunk/bookkeeper-server/src/test/java/org/apache/bookkeeper/meta/HierarchicalBookieReadWriteTest.java
* /zookeeper/bookkeeper/trunk/bookkeeper-server/src/test/java/org/apache/bookkeeper/meta/HierarchicalBookieRecoveryTest.java
* /zookeeper/bookkeeper/trunk/bookkeeper-server/src/test/java/org/apache/bookkeeper/meta/HierarchicalLedgerDeleteTest.java
* /zookeeper/bookkeeper/trunk/bookkeeper-server/src/test/java/org/apache/bookkeeper/meta/LedgerLayoutTest.java
* /zookeeper/bookkeeper/trunk/bookkeeper-server/src/test/java/org/apache/bookkeeper/meta/LedgerManagerTestCase.java
* /zookeeper/bookkeeper/trunk/bookkeeper-server/src/test/java/org/apache/bookkeeper/meta/TestLedgerManager.java
* /zookeeper/bookkeeper/trunk/bookkeeper-server/src/test/java/org/apache/bookkeeper/meta/TestZkVersion.java
* /zookeeper/bookkeeper/trunk/bookkeeper-server/src/test/java/org/apache/bookkeeper/test/AsyncLedgerOpsTest.java
* /zookeeper/bookkeeper/trunk/bookkeeper-server/src/test/java/org/apache/bookkeeper/test/BookKeeperClusterTestCase.java
* /zookeeper/bookkeeper/trunk/bookkeeper-server/src/test/java/org/apache/bookkeeper/test/BookieFailureTest.java
* /zookeeper/bookkeeper/trunk/bookkeeper-server/src/test/java/org/apache/bookkeeper/test/BookieReadWriteTest.java
* /zookeeper/bookkeeper/trunk/bookkeeper-server/src/test/java/org/apache/bookkeeper/test/LedgerDeleteTest.java
* /zookeeper/bookkeeper/trunk/bookkeeper-server/src/test/java/org/apache/bookkeeper/test/MultiLedgerManagerMultiDigestTestCase.java
* /zookeeper/bookkeeper/trunk/bookkeeper-server/src/test/java/org/apache/bookkeeper/test/MultiLedgerManagerTestCase.java

                
> improve ledger manager interface to remove zookeeper dependency on metadata operations.
> ---------------------------------------------------------------------------------------
>
>                 Key: BOOKKEEPER-203
>                 URL: https://issues.apache.org/jira/browse/BOOKKEEPER-203
>             Project: Bookkeeper
>          Issue Type: Sub-task
>          Components: bookkeeper-client, bookkeeper-server
>            Reporter: Sijie Guo
>            Assignee: Sijie Guo
>             Fix For: 4.2.0
>
>         Attachments: BOOKKEEPER-203.diff, BOOKKEEPER-203.diff_v2, BOOKKEEPER-203.diff_v3, BOOKKEEPER-203.diff_v4, BOOKKEEPER-203.diff_v5, BOOKKEEPER-203.diff_v6, BOOKKEEPER-203.diff_v7
>
>
> we need to improve ledger manager interface to remove zookeeper dependency on metadata operations, so it is easy for us to implement a MetaStore based ledger manager.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

[jira] [Commented] (BOOKKEEPER-203) improve ledger manager interface to remove zookeeper dependency on metadata operations.

Posted by "Sijie Guo (JIRA)" <ji...@apache.org>.
    [ https://issues.apache.org/jira/browse/BOOKKEEPER-203?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13396337#comment-13396337 ] 

Sijie Guo commented on BOOKKEEPER-203:
--------------------------------------

@Ivan,

Versioned is added as a generic interface. I was planning to make LedgerMetadata as Versioned interface, but seems that there are too much modifications. so I don't do the change.

But the idea I kept Versioned here is for completeness of versioning interface, and it would be used in Hedwig Meta Manager to write topic persistence info and owner info. (BOOKKEEPER-250 and BOOKKEEPER-259)

meta package is used to put metadata related things of BookKeeper, while Version and Versioned are more generic abstraction than BookKeeper meta, it would be used in both BookKeeper and Hedwig metadata manager. So I put them in a separated versioning package.
                
> improve ledger manager interface to remove zookeeper dependency on metadata operations.
> ---------------------------------------------------------------------------------------
>
>                 Key: BOOKKEEPER-203
>                 URL: https://issues.apache.org/jira/browse/BOOKKEEPER-203
>             Project: Bookkeeper
>          Issue Type: Sub-task
>          Components: bookkeeper-client, bookkeeper-server
>            Reporter: Sijie Guo
>            Assignee: Sijie Guo
>             Fix For: 4.2.0
>
>         Attachments: BOOKKEEPER-203.diff, BOOKKEEPER-203.diff_v2, BOOKKEEPER-203.diff_v3, BOOKKEEPER-203.diff_v4, BOOKKEEPER-203.diff_v5, BOOKKEEPER-203.diff_v6, BOOKKEEPER-203.diff_v7
>
>
> we need to improve ledger manager interface to remove zookeeper dependency on metadata operations, so it is easy for us to implement a MetaStore based ledger manager.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

[jira] [Commented] (BOOKKEEPER-203) improve ledger manager interface to remove zookeeper dependency on metadata operations.

Posted by "jiraposter@reviews.apache.org (Commented) (JIRA)" <ji...@apache.org>.
    [ https://issues.apache.org/jira/browse/BOOKKEEPER-203?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13243756#comment-13243756 ] 

jiraposter@reviews.apache.org commented on BOOKKEEPER-203:
----------------------------------------------------------


-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/4603/
-----------------------------------------------------------

Review request for bookkeeper.


Summary
-------

we need to improve ledger manager interface to remove zookeeper dependency on metadata operations, so it is easy for us to implement a MetaStore based ledger manager.


This addresses bug BOOKKEEPER-203.
    https://issues.apache.org/jira/browse/BOOKKEEPER-203


Diffs
-----

  bookkeeper-server/src/main/java/org/apache/bookkeeper/bookie/Bookie.java ee38862 
  bookkeeper-server/src/main/java/org/apache/bookkeeper/bookie/GarbageCollectorThread.java 2eed192 
  bookkeeper-server/src/main/java/org/apache/bookkeeper/bookie/LedgerCacheImpl.java 17c70f1 
  bookkeeper-server/src/main/java/org/apache/bookkeeper/client/BKException.java 911c660 
  bookkeeper-server/src/main/java/org/apache/bookkeeper/client/BookKeeper.java 490c130 
  bookkeeper-server/src/main/java/org/apache/bookkeeper/client/BookKeeperAdmin.java b9cd624 
  bookkeeper-server/src/main/java/org/apache/bookkeeper/client/LedgerCreateOp.java f0a165f 
  bookkeeper-server/src/main/java/org/apache/bookkeeper/client/LedgerDeleteOp.java 5b10a5b 
  bookkeeper-server/src/main/java/org/apache/bookkeeper/client/LedgerHandle.java e576792 
  bookkeeper-server/src/main/java/org/apache/bookkeeper/client/LedgerMetadata.java 2dcf0e4 
  bookkeeper-server/src/main/java/org/apache/bookkeeper/client/LedgerOpenOp.java 56186ab 
  bookkeeper-server/src/main/java/org/apache/bookkeeper/conf/AbstractConfiguration.java e1bd1ff 
  bookkeeper-server/src/main/java/org/apache/bookkeeper/meta/AbstractZkLedgerManager.java 8fd7df8 
  bookkeeper-server/src/main/java/org/apache/bookkeeper/meta/ActiveLedgerManager.java PRE-CREATION 
  bookkeeper-server/src/main/java/org/apache/bookkeeper/meta/FlatLedgerManager.java 1300974 
  bookkeeper-server/src/main/java/org/apache/bookkeeper/meta/FlatLedgerManagerFactory.java PRE-CREATION 
  bookkeeper-server/src/main/java/org/apache/bookkeeper/meta/HierarchicalLedgerManager.java b573181 
  bookkeeper-server/src/main/java/org/apache/bookkeeper/meta/HierarchicalLedgerManagerFactory.java PRE-CREATION 
  bookkeeper-server/src/main/java/org/apache/bookkeeper/meta/LedgerLayout.java 3a3a81e 
  bookkeeper-server/src/main/java/org/apache/bookkeeper/meta/LedgerManager.java 7d9eb96 
  bookkeeper-server/src/main/java/org/apache/bookkeeper/meta/LedgerManagerFactory.java b62ad3a 
  bookkeeper-server/src/main/java/org/apache/bookkeeper/util/ReflectionUtils.java PRE-CREATION 
  bookkeeper-server/src/test/java/org/apache/bookkeeper/bookie/LedgerCacheTest.java 52cf514 
  bookkeeper-server/src/test/java/org/apache/bookkeeper/client/BookieRecoveryTest.java 6135250 
  bookkeeper-server/src/test/java/org/apache/bookkeeper/client/TestFencing.java c5350ad 
  bookkeeper-server/src/test/java/org/apache/bookkeeper/meta/GcLedgersTest.java 12c1b5a 
  bookkeeper-server/src/test/java/org/apache/bookkeeper/meta/HierarchicalAsyncLedgerOpsTest.java 24b3c12 
  bookkeeper-server/src/test/java/org/apache/bookkeeper/meta/HierarchicalBookieFailureTest.java 70a4ea9 
  bookkeeper-server/src/test/java/org/apache/bookkeeper/meta/HierarchicalBookieReadWriteTest.java 7d97fd4 
  bookkeeper-server/src/test/java/org/apache/bookkeeper/meta/HierarchicalBookieRecoveryTest.java d99156e 
  bookkeeper-server/src/test/java/org/apache/bookkeeper/meta/HierarchicalLedgerDeleteTest.java dbcd65a 
  bookkeeper-server/src/test/java/org/apache/bookkeeper/meta/LedgerLayoutTest.java 4784854 
  bookkeeper-server/src/test/java/org/apache/bookkeeper/meta/LedgerManagerTestCase.java b8a541b 
  bookkeeper-server/src/test/java/org/apache/bookkeeper/meta/TestLedgerManager.java e07c756 
  bookkeeper-server/src/test/java/org/apache/bookkeeper/test/AsyncLedgerOpsTest.java d7f153a 
  bookkeeper-server/src/test/java/org/apache/bookkeeper/test/BookKeeperClusterTestCase.java c172224 
  bookkeeper-server/src/test/java/org/apache/bookkeeper/test/BookieFailureTest.java e647d68 
  bookkeeper-server/src/test/java/org/apache/bookkeeper/test/BookieReadWriteTest.java 3a3c8af 
  bookkeeper-server/src/test/java/org/apache/bookkeeper/test/LedgerDeleteTest.java 2d3593e 
  bookkeeper-server/src/test/java/org/apache/bookkeeper/test/MultiLedgerManagerBaseTestCase.java PRE-CREATION 
  bookkeeper-server/src/test/java/org/apache/bookkeeper/test/MultiLedgerManagerTestCase.java PRE-CREATION 

Diff: https://reviews.apache.org/r/4603/diff


Testing
-------


Thanks,

Sijie


                
> improve ledger manager interface to remove zookeeper dependency on metadata operations.
> ---------------------------------------------------------------------------------------
>
>                 Key: BOOKKEEPER-203
>                 URL: https://issues.apache.org/jira/browse/BOOKKEEPER-203
>             Project: Bookkeeper
>          Issue Type: Sub-task
>          Components: bookkeeper-client, bookkeeper-server
>            Reporter: Sijie Guo
>            Assignee: Sijie Guo
>             Fix For: 4.1.0
>
>         Attachments: BOOKKEEPER-203.diff
>
>
> we need to improve ledger manager interface to remove zookeeper dependency on metadata operations, so it is easy for us to implement a MetaStore based ledger manager.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

[jira] [Commented] (BOOKKEEPER-203) improve ledger manager interface to remove zookeeper dependency on metadata operations.

Posted by "jiraposter@reviews.apache.org (JIRA)" <ji...@apache.org>.
    [ https://issues.apache.org/jira/browse/BOOKKEEPER-203?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13274138#comment-13274138 ] 

jiraposter@reviews.apache.org commented on BOOKKEEPER-203:
----------------------------------------------------------


-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/4603/
-----------------------------------------------------------

(Updated 2012-05-13 02:44:58.032203)


Review request for bookkeeper.


Changes
-------

update a new patch, which brought the code to latest trunk and addressed Flavio's suggestions.


Summary
-------

we need to improve ledger manager interface to remove zookeeper dependency on metadata operations, so it is easy for us to implement a MetaStore based ledger manager.


This addresses bug BOOKKEEPER-203.
    https://issues.apache.org/jira/browse/BOOKKEEPER-203


Diffs (updated)
-----

  bookkeeper-server/src/main/java/org/apache/bookkeeper/bookie/Bookie.java aa4dec2 
  bookkeeper-server/src/main/java/org/apache/bookkeeper/bookie/GarbageCollectorThread.java 4dc71e5 
  bookkeeper-server/src/main/java/org/apache/bookkeeper/bookie/InterleavedLedgerStorage.java c8b5001 
  bookkeeper-server/src/main/java/org/apache/bookkeeper/bookie/LedgerCacheImpl.java 4df689e 
  bookkeeper-server/src/main/java/org/apache/bookkeeper/client/BKException.java 484a055 
  bookkeeper-server/src/main/java/org/apache/bookkeeper/client/BookKeeper.java eb0955b 
  bookkeeper-server/src/main/java/org/apache/bookkeeper/client/BookKeeperAdmin.java 712bbea 
  bookkeeper-server/src/main/java/org/apache/bookkeeper/client/LedgerCreateOp.java 8c5bce3 
  bookkeeper-server/src/main/java/org/apache/bookkeeper/client/LedgerDeleteOp.java 5b10a5b 
  bookkeeper-server/src/main/java/org/apache/bookkeeper/client/LedgerHandle.java 6ce576d 
  bookkeeper-server/src/main/java/org/apache/bookkeeper/client/LedgerMetadata.java c699eae 
  bookkeeper-server/src/main/java/org/apache/bookkeeper/client/LedgerOpenOp.java 101ebf3 
  bookkeeper-server/src/main/java/org/apache/bookkeeper/conf/AbstractConfiguration.java e1bd1ff 
  bookkeeper-server/src/main/java/org/apache/bookkeeper/meta/AbstractZkLedgerManager.java ce53461 
  bookkeeper-server/src/main/java/org/apache/bookkeeper/meta/ActiveLedgerManager.java PRE-CREATION 
  bookkeeper-server/src/main/java/org/apache/bookkeeper/meta/FlatLedgerManager.java 1300974 
  bookkeeper-server/src/main/java/org/apache/bookkeeper/meta/FlatLedgerManagerFactory.java PRE-CREATION 
  bookkeeper-server/src/main/java/org/apache/bookkeeper/meta/HierarchicalLedgerManager.java b573181 
  bookkeeper-server/src/main/java/org/apache/bookkeeper/meta/HierarchicalLedgerManagerFactory.java PRE-CREATION 
  bookkeeper-server/src/main/java/org/apache/bookkeeper/meta/LedgerLayout.java cab02fe 
  bookkeeper-server/src/main/java/org/apache/bookkeeper/meta/LedgerManager.java 7d9eb96 
  bookkeeper-server/src/main/java/org/apache/bookkeeper/meta/LedgerManagerFactory.java 491a3e4 
  bookkeeper-server/src/main/java/org/apache/bookkeeper/util/ReflectionUtils.java PRE-CREATION 
  bookkeeper-server/src/test/java/org/apache/bookkeeper/bookie/LedgerCacheTest.java 52cf514 
  bookkeeper-server/src/test/java/org/apache/bookkeeper/client/BookieRecoveryTest.java 8bd3f6b 
  bookkeeper-server/src/test/java/org/apache/bookkeeper/client/TestFencing.java 1c109a2 
  bookkeeper-server/src/test/java/org/apache/bookkeeper/meta/GcLedgersTest.java 12c1b5a 
  bookkeeper-server/src/test/java/org/apache/bookkeeper/meta/HierarchicalAsyncLedgerOpsTest.java 24b3c12 
  bookkeeper-server/src/test/java/org/apache/bookkeeper/meta/HierarchicalBookieFailureTest.java 70a4ea9 
  bookkeeper-server/src/test/java/org/apache/bookkeeper/meta/HierarchicalBookieReadWriteTest.java 7d97fd4 
  bookkeeper-server/src/test/java/org/apache/bookkeeper/meta/HierarchicalBookieRecoveryTest.java d99156e 
  bookkeeper-server/src/test/java/org/apache/bookkeeper/meta/HierarchicalLedgerDeleteTest.java dbcd65a 
  bookkeeper-server/src/test/java/org/apache/bookkeeper/meta/LedgerLayoutTest.java 4784854 
  bookkeeper-server/src/test/java/org/apache/bookkeeper/meta/LedgerManagerTestCase.java b8a541b 
  bookkeeper-server/src/test/java/org/apache/bookkeeper/meta/TestLedgerManager.java e07c756 
  bookkeeper-server/src/test/java/org/apache/bookkeeper/test/AsyncLedgerOpsTest.java d7f153a 
  bookkeeper-server/src/test/java/org/apache/bookkeeper/test/BookKeeperClusterTestCase.java a81067d 
  bookkeeper-server/src/test/java/org/apache/bookkeeper/test/BookieFailureTest.java e647d68 
  bookkeeper-server/src/test/java/org/apache/bookkeeper/test/BookieReadWriteTest.java 3a3c8af 
  bookkeeper-server/src/test/java/org/apache/bookkeeper/test/LedgerDeleteTest.java 2d3593e 
  bookkeeper-server/src/test/java/org/apache/bookkeeper/test/MultiLedgerManagerBaseTestCase.java PRE-CREATION 
  bookkeeper-server/src/test/java/org/apache/bookkeeper/test/MultiLedgerManagerTestCase.java PRE-CREATION 

Diff: https://reviews.apache.org/r/4603/diff


Testing
-------


Thanks,

Sijie


                
> improve ledger manager interface to remove zookeeper dependency on metadata operations.
> ---------------------------------------------------------------------------------------
>
>                 Key: BOOKKEEPER-203
>                 URL: https://issues.apache.org/jira/browse/BOOKKEEPER-203
>             Project: Bookkeeper
>          Issue Type: Sub-task
>          Components: bookkeeper-client, bookkeeper-server
>            Reporter: Sijie Guo
>            Assignee: Sijie Guo
>             Fix For: 4.2.0
>
>         Attachments: BOOKKEEPER-203.diff, BOOKKEEPER-203.diff_v2
>
>
> we need to improve ledger manager interface to remove zookeeper dependency on metadata operations, so it is easy for us to implement a MetaStore based ledger manager.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

[jira] [Commented] (BOOKKEEPER-203) improve ledger manager interface to remove zookeeper dependency on metadata operations.

Posted by "jiraposter@reviews.apache.org (JIRA)" <ji...@apache.org>.
    [ https://issues.apache.org/jira/browse/BOOKKEEPER-203?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13274113#comment-13274113 ] 

jiraposter@reviews.apache.org commented on BOOKKEEPER-203:
----------------------------------------------------------



bq.  On 2012-04-03 21:17:04, fpj wrote:
bq.  > It looks pretty good for me, I just have a few small comments.

Thanks Flavio for reviewing it. I would like to bring the patch to latest trunk and update it with your comments.


bq.  On 2012-04-03 21:17:04, fpj wrote:
bq.  > bookkeeper-server/src/main/java/org/apache/bookkeeper/client/LedgerMetadata.java, line 178
bq.  > <https://reviews.apache.org/r/4603/diff/1/?file=97866#file97866line178>
bq.  >
bq.  >     It doesn't look like we need to make this call public. If it is for testing, we could simply use inheritance, no?

we use parseConfig in ledgermanager not for testing, which is another package.


bq.  On 2012-04-03 21:17:04, fpj wrote:
bq.  > bookkeeper-server/src/main/java/org/apache/bookkeeper/conf/AbstractConfiguration.java, line 146
bq.  > <https://reviews.apache.org/r/4603/diff/1/?file=97868#file97868line146>
bq.  >
bq.  >     I'm wondering if it is ok to have a reference specific to zk here. Is the idea here that for every new backend we add, we include required system properties here?

zkLedgersRootPath not only for zookeeper-based ledger manager to store ledger metadata. under zkLedgersRootPath there is a 'layout' znode, which describes the metadata layout and which ledger manager it used for current bookkeeper cluster.


bq.  On 2012-04-03 21:17:04, fpj wrote:
bq.  > bookkeeper-server/src/main/java/org/apache/bookkeeper/meta/ActiveLedgerManager.java, line 30
bq.  > <https://reviews.apache.org/r/4603/diff/1/?file=97870#file97870line30>
bq.  >
bq.  >     I wonder if ActiveLedgerManager is a good name for this class. It doesn't indicate that it is related to the bookie side, and it might be good to refer to it. But, I don't feel too strongly about it, since the comment before the class says that it is for bookies.

I don't have a good name for it also, just used its original name before we abstracted ledger manager interface.


- Sijie


-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/4603/#review6616
-----------------------------------------------------------


On 2012-04-01 15:56:33, Sijie Guo wrote:
bq.  
bq.  -----------------------------------------------------------
bq.  This is an automatically generated e-mail. To reply, visit:
bq.  https://reviews.apache.org/r/4603/
bq.  -----------------------------------------------------------
bq.  
bq.  (Updated 2012-04-01 15:56:33)
bq.  
bq.  
bq.  Review request for bookkeeper.
bq.  
bq.  
bq.  Summary
bq.  -------
bq.  
bq.  we need to improve ledger manager interface to remove zookeeper dependency on metadata operations, so it is easy for us to implement a MetaStore based ledger manager.
bq.  
bq.  
bq.  This addresses bug BOOKKEEPER-203.
bq.      https://issues.apache.org/jira/browse/BOOKKEEPER-203
bq.  
bq.  
bq.  Diffs
bq.  -----
bq.  
bq.    bookkeeper-server/src/main/java/org/apache/bookkeeper/bookie/Bookie.java ee38862 
bq.    bookkeeper-server/src/main/java/org/apache/bookkeeper/bookie/GarbageCollectorThread.java 2eed192 
bq.    bookkeeper-server/src/main/java/org/apache/bookkeeper/bookie/LedgerCacheImpl.java 17c70f1 
bq.    bookkeeper-server/src/main/java/org/apache/bookkeeper/client/BKException.java 911c660 
bq.    bookkeeper-server/src/main/java/org/apache/bookkeeper/client/BookKeeper.java 490c130 
bq.    bookkeeper-server/src/main/java/org/apache/bookkeeper/client/BookKeeperAdmin.java b9cd624 
bq.    bookkeeper-server/src/main/java/org/apache/bookkeeper/client/LedgerCreateOp.java f0a165f 
bq.    bookkeeper-server/src/main/java/org/apache/bookkeeper/client/LedgerDeleteOp.java 5b10a5b 
bq.    bookkeeper-server/src/main/java/org/apache/bookkeeper/client/LedgerHandle.java e576792 
bq.    bookkeeper-server/src/main/java/org/apache/bookkeeper/client/LedgerMetadata.java 2dcf0e4 
bq.    bookkeeper-server/src/main/java/org/apache/bookkeeper/client/LedgerOpenOp.java 56186ab 
bq.    bookkeeper-server/src/main/java/org/apache/bookkeeper/conf/AbstractConfiguration.java e1bd1ff 
bq.    bookkeeper-server/src/main/java/org/apache/bookkeeper/meta/AbstractZkLedgerManager.java 8fd7df8 
bq.    bookkeeper-server/src/main/java/org/apache/bookkeeper/meta/ActiveLedgerManager.java PRE-CREATION 
bq.    bookkeeper-server/src/main/java/org/apache/bookkeeper/meta/FlatLedgerManager.java 1300974 
bq.    bookkeeper-server/src/main/java/org/apache/bookkeeper/meta/FlatLedgerManagerFactory.java PRE-CREATION 
bq.    bookkeeper-server/src/main/java/org/apache/bookkeeper/meta/HierarchicalLedgerManager.java b573181 
bq.    bookkeeper-server/src/main/java/org/apache/bookkeeper/meta/HierarchicalLedgerManagerFactory.java PRE-CREATION 
bq.    bookkeeper-server/src/main/java/org/apache/bookkeeper/meta/LedgerLayout.java 3a3a81e 
bq.    bookkeeper-server/src/main/java/org/apache/bookkeeper/meta/LedgerManager.java 7d9eb96 
bq.    bookkeeper-server/src/main/java/org/apache/bookkeeper/meta/LedgerManagerFactory.java b62ad3a 
bq.    bookkeeper-server/src/main/java/org/apache/bookkeeper/util/ReflectionUtils.java PRE-CREATION 
bq.    bookkeeper-server/src/test/java/org/apache/bookkeeper/bookie/LedgerCacheTest.java 52cf514 
bq.    bookkeeper-server/src/test/java/org/apache/bookkeeper/client/BookieRecoveryTest.java 6135250 
bq.    bookkeeper-server/src/test/java/org/apache/bookkeeper/client/TestFencing.java c5350ad 
bq.    bookkeeper-server/src/test/java/org/apache/bookkeeper/meta/GcLedgersTest.java 12c1b5a 
bq.    bookkeeper-server/src/test/java/org/apache/bookkeeper/meta/HierarchicalAsyncLedgerOpsTest.java 24b3c12 
bq.    bookkeeper-server/src/test/java/org/apache/bookkeeper/meta/HierarchicalBookieFailureTest.java 70a4ea9 
bq.    bookkeeper-server/src/test/java/org/apache/bookkeeper/meta/HierarchicalBookieReadWriteTest.java 7d97fd4 
bq.    bookkeeper-server/src/test/java/org/apache/bookkeeper/meta/HierarchicalBookieRecoveryTest.java d99156e 
bq.    bookkeeper-server/src/test/java/org/apache/bookkeeper/meta/HierarchicalLedgerDeleteTest.java dbcd65a 
bq.    bookkeeper-server/src/test/java/org/apache/bookkeeper/meta/LedgerLayoutTest.java 4784854 
bq.    bookkeeper-server/src/test/java/org/apache/bookkeeper/meta/LedgerManagerTestCase.java b8a541b 
bq.    bookkeeper-server/src/test/java/org/apache/bookkeeper/meta/TestLedgerManager.java e07c756 
bq.    bookkeeper-server/src/test/java/org/apache/bookkeeper/test/AsyncLedgerOpsTest.java d7f153a 
bq.    bookkeeper-server/src/test/java/org/apache/bookkeeper/test/BookKeeperClusterTestCase.java c172224 
bq.    bookkeeper-server/src/test/java/org/apache/bookkeeper/test/BookieFailureTest.java e647d68 
bq.    bookkeeper-server/src/test/java/org/apache/bookkeeper/test/BookieReadWriteTest.java 3a3c8af 
bq.    bookkeeper-server/src/test/java/org/apache/bookkeeper/test/LedgerDeleteTest.java 2d3593e 
bq.    bookkeeper-server/src/test/java/org/apache/bookkeeper/test/MultiLedgerManagerBaseTestCase.java PRE-CREATION 
bq.    bookkeeper-server/src/test/java/org/apache/bookkeeper/test/MultiLedgerManagerTestCase.java PRE-CREATION 
bq.  
bq.  Diff: https://reviews.apache.org/r/4603/diff
bq.  
bq.  
bq.  Testing
bq.  -------
bq.  
bq.  
bq.  Thanks,
bq.  
bq.  Sijie
bq.  
bq.


                
> improve ledger manager interface to remove zookeeper dependency on metadata operations.
> ---------------------------------------------------------------------------------------
>
>                 Key: BOOKKEEPER-203
>                 URL: https://issues.apache.org/jira/browse/BOOKKEEPER-203
>             Project: Bookkeeper
>          Issue Type: Sub-task
>          Components: bookkeeper-client, bookkeeper-server
>            Reporter: Sijie Guo
>            Assignee: Sijie Guo
>             Fix For: 4.2.0
>
>         Attachments: BOOKKEEPER-203.diff
>
>
> we need to improve ledger manager interface to remove zookeeper dependency on metadata operations, so it is easy for us to implement a MetaStore based ledger manager.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

[jira] [Commented] (BOOKKEEPER-203) improve ledger manager interface to remove zookeeper dependency on metadata operations.

Posted by "Ivan Kelly (JIRA)" <ji...@apache.org>.
    [ https://issues.apache.org/jira/browse/BOOKKEEPER-203?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13395814#comment-13395814 ] 

Ivan Kelly commented on BOOKKEEPER-203:
---------------------------------------

{quote}If everything is run as MultiLedgerManagerBaseTestCase, it would take a long time to run. {quote}
This is true. Would it be a big change to rename it to MultiLedgerManagerMultiDigestTestCase, to make things clearer?
                
> improve ledger manager interface to remove zookeeper dependency on metadata operations.
> ---------------------------------------------------------------------------------------
>
>                 Key: BOOKKEEPER-203
>                 URL: https://issues.apache.org/jira/browse/BOOKKEEPER-203
>             Project: Bookkeeper
>          Issue Type: Sub-task
>          Components: bookkeeper-client, bookkeeper-server
>            Reporter: Sijie Guo
>            Assignee: Sijie Guo
>             Fix For: 4.2.0
>
>         Attachments: BOOKKEEPER-203.diff, BOOKKEEPER-203.diff_v2, BOOKKEEPER-203.diff_v3, BOOKKEEPER-203.diff_v4, BOOKKEEPER-203.diff_v5
>
>
> we need to improve ledger manager interface to remove zookeeper dependency on metadata operations, so it is easy for us to implement a MetaStore based ledger manager.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

[jira] [Commented] (BOOKKEEPER-203) improve ledger manager interface to remove zookeeper dependency on metadata operations.

Posted by "Ivan Kelly (JIRA)" <ji...@apache.org>.
    [ https://issues.apache.org/jira/browse/BOOKKEEPER-203?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13294330#comment-13294330 ] 

Ivan Kelly commented on BOOKKEEPER-203:
---------------------------------------

ah, ignore first comment.
                
> improve ledger manager interface to remove zookeeper dependency on metadata operations.
> ---------------------------------------------------------------------------------------
>
>                 Key: BOOKKEEPER-203
>                 URL: https://issues.apache.org/jira/browse/BOOKKEEPER-203
>             Project: Bookkeeper
>          Issue Type: Sub-task
>          Components: bookkeeper-client, bookkeeper-server
>            Reporter: Sijie Guo
>            Assignee: Sijie Guo
>             Fix For: 4.2.0
>
>         Attachments: BOOKKEEPER-203.diff, BOOKKEEPER-203.diff_v2, BOOKKEEPER-203.diff_v3, BOOKKEEPER-203.diff_v4
>
>
> we need to improve ledger manager interface to remove zookeeper dependency on metadata operations, so it is easy for us to implement a MetaStore based ledger manager.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

[jira] [Commented] (BOOKKEEPER-203) improve ledger manager interface to remove zookeeper dependency on metadata operations.

Posted by "Ivan Kelly (JIRA)" <ji...@apache.org>.
    [ https://issues.apache.org/jira/browse/BOOKKEEPER-203?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13396111#comment-13396111 ] 

Ivan Kelly commented on BOOKKEEPER-203:
---------------------------------------

One last thing (unfortunately it wasn't in the patch i reviewed last week). I think Version and Versioned should be in the meta package and not in a package of their own. Actually, Versioned doesn't even seem to be used at all. I guess it's there for another patch you have planned?
                
> improve ledger manager interface to remove zookeeper dependency on metadata operations.
> ---------------------------------------------------------------------------------------
>
>                 Key: BOOKKEEPER-203
>                 URL: https://issues.apache.org/jira/browse/BOOKKEEPER-203
>             Project: Bookkeeper
>          Issue Type: Sub-task
>          Components: bookkeeper-client, bookkeeper-server
>            Reporter: Sijie Guo
>            Assignee: Sijie Guo
>             Fix For: 4.2.0
>
>         Attachments: BOOKKEEPER-203.diff, BOOKKEEPER-203.diff_v2, BOOKKEEPER-203.diff_v3, BOOKKEEPER-203.diff_v4, BOOKKEEPER-203.diff_v5, BOOKKEEPER-203.diff_v6, BOOKKEEPER-203.diff_v7
>
>
> we need to improve ledger manager interface to remove zookeeper dependency on metadata operations, so it is easy for us to implement a MetaStore based ledger manager.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

[jira] [Commented] (BOOKKEEPER-203) improve ledger manager interface to remove zookeeper dependency on metadata operations.

Posted by "Flavio Junqueira (Commented) (JIRA)" <ji...@apache.org>.
    [ https://issues.apache.org/jira/browse/BOOKKEEPER-203?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13245764#comment-13245764 ] 

Flavio Junqueira commented on BOOKKEEPER-203:
---------------------------------------------

Even if we +1 this jira, I was wondering if we should move it to 4.2.0. This jira is a sub-task of BOOKKEPER-181, and we have marked it for 4.2.0.
                
> improve ledger manager interface to remove zookeeper dependency on metadata operations.
> ---------------------------------------------------------------------------------------
>
>                 Key: BOOKKEEPER-203
>                 URL: https://issues.apache.org/jira/browse/BOOKKEEPER-203
>             Project: Bookkeeper
>          Issue Type: Sub-task
>          Components: bookkeeper-client, bookkeeper-server
>            Reporter: Sijie Guo
>            Assignee: Sijie Guo
>             Fix For: 4.1.0
>
>         Attachments: BOOKKEEPER-203.diff
>
>
> we need to improve ledger manager interface to remove zookeeper dependency on metadata operations, so it is easy for us to implement a MetaStore based ledger manager.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

[jira] [Commented] (BOOKKEEPER-203) improve ledger manager interface to remove zookeeper dependency on metadata operations.

Posted by "Sijie Guo (JIRA)" <ji...@apache.org>.
    [ https://issues.apache.org/jira/browse/BOOKKEEPER-203?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13395766#comment-13395766 ] 

Sijie Guo commented on BOOKKEEPER-203:
--------------------------------------

@Ivan,

{code}
HierarchicalAsyncLedgerOpsTest -> AsyncLedgerOpsTest (BaseTestCase)
HierarchicalBookieReadWriteTest -> BookieReadWriteTest (BaseTestCase)
HierarchicalBookieFailureTest -> BookieFailureTest (BaseTestCase)
HierarchicalBookieRecoveryTest -> BookieRecoveryTest (BaseTestCase)
HierarchicalLedgerDeleteTest -> LedgerDeleteTest (BookKeeperClusterTestCase)
{code}

MultiLedgerManagerBaseTestCase & MultiLedgerManagerTestCase are used to remove HierarchicalxxxTests to let them run as parameterized tests. 

{quote}
 So MultiLedgerManagerBaseTestCase is really a superset of MultiLedgerManagerTestCase. I don't see any harm in just using MultiLedgerManagerBaseTestCase for all.
{quote}

If everything is run as MultiLedgerManagerBaseTestCase, it would take a long time to run. For example, LedgerDeleteTest just tests ledger deletion functionality. so we don't need to parameterize the test to run with different digest type. But we have to run with different ledger managers because deletion is related to gc implementation in different ledger managers.





                
> improve ledger manager interface to remove zookeeper dependency on metadata operations.
> ---------------------------------------------------------------------------------------
>
>                 Key: BOOKKEEPER-203
>                 URL: https://issues.apache.org/jira/browse/BOOKKEEPER-203
>             Project: Bookkeeper
>          Issue Type: Sub-task
>          Components: bookkeeper-client, bookkeeper-server
>            Reporter: Sijie Guo
>            Assignee: Sijie Guo
>             Fix For: 4.2.0
>
>         Attachments: BOOKKEEPER-203.diff, BOOKKEEPER-203.diff_v2, BOOKKEEPER-203.diff_v3, BOOKKEEPER-203.diff_v4, BOOKKEEPER-203.diff_v5
>
>
> we need to improve ledger manager interface to remove zookeeper dependency on metadata operations, so it is easy for us to implement a MetaStore based ledger manager.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

[jira] [Updated] (BOOKKEEPER-203) improve ledger manager interface to remove zookeeper dependency on metadata operations.

Posted by "Sijie Guo (JIRA)" <ji...@apache.org>.
     [ https://issues.apache.org/jira/browse/BOOKKEEPER-203?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Sijie Guo updated BOOKKEEPER-203:
---------------------------------

    Attachment: BOOKKEEPER-203.diff_v3

attach a new patch rebased to latest trunk.
                
> improve ledger manager interface to remove zookeeper dependency on metadata operations.
> ---------------------------------------------------------------------------------------
>
>                 Key: BOOKKEEPER-203
>                 URL: https://issues.apache.org/jira/browse/BOOKKEEPER-203
>             Project: Bookkeeper
>          Issue Type: Sub-task
>          Components: bookkeeper-client, bookkeeper-server
>            Reporter: Sijie Guo
>            Assignee: Sijie Guo
>             Fix For: 4.2.0
>
>         Attachments: BOOKKEEPER-203.diff, BOOKKEEPER-203.diff_v2, BOOKKEEPER-203.diff_v3
>
>
> we need to improve ledger manager interface to remove zookeeper dependency on metadata operations, so it is easy for us to implement a MetaStore based ledger manager.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

[jira] [Updated] (BOOKKEEPER-203) improve ledger manager interface to remove zookeeper dependency on metadata operations.

Posted by "Sijie Guo (Updated) (JIRA)" <ji...@apache.org>.
     [ https://issues.apache.org/jira/browse/BOOKKEEPER-203?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Sijie Guo updated BOOKKEEPER-203:
---------------------------------

    Fix Version/s:     (was: 4.1.0)
                   4.2.0

Agreed to move it to 4.2.0.
                
> improve ledger manager interface to remove zookeeper dependency on metadata operations.
> ---------------------------------------------------------------------------------------
>
>                 Key: BOOKKEEPER-203
>                 URL: https://issues.apache.org/jira/browse/BOOKKEEPER-203
>             Project: Bookkeeper
>          Issue Type: Sub-task
>          Components: bookkeeper-client, bookkeeper-server
>            Reporter: Sijie Guo
>            Assignee: Sijie Guo
>             Fix For: 4.2.0
>
>         Attachments: BOOKKEEPER-203.diff
>
>
> we need to improve ledger manager interface to remove zookeeper dependency on metadata operations, so it is easy for us to implement a MetaStore based ledger manager.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

[jira] [Updated] (BOOKKEEPER-203) improve ledger manager interface to remove zookeeper dependency on metadata operations.

Posted by "Sijie Guo (Updated) (JIRA)" <ji...@apache.org>.
     [ https://issues.apache.org/jira/browse/BOOKKEEPER-203?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Sijie Guo updated BOOKKEEPER-203:
---------------------------------

    Attachment: BOOKKEEPER-203.diff

create a patch to improve ledger manager interface

1) split ledger manager interface into client/server part, client side is LedgerManager, server side is ActiveLedgerManager.
2) remove zookeeper dependency on metadata operations, provides a well-defined metadata operation interface.
                
> improve ledger manager interface to remove zookeeper dependency on metadata operations.
> ---------------------------------------------------------------------------------------
>
>                 Key: BOOKKEEPER-203
>                 URL: https://issues.apache.org/jira/browse/BOOKKEEPER-203
>             Project: Bookkeeper
>          Issue Type: Sub-task
>          Components: bookkeeper-client, bookkeeper-server
>            Reporter: Sijie Guo
>            Assignee: Sijie Guo
>         Attachments: BOOKKEEPER-203.diff
>
>
> we need to improve ledger manager interface to remove zookeeper dependency on metadata operations, so it is easy for us to implement a MetaStore based ledger manager.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

[jira] [Updated] (BOOKKEEPER-203) improve ledger manager interface to remove zookeeper dependency on metadata operations.

Posted by "Sijie Guo (JIRA)" <ji...@apache.org>.
     [ https://issues.apache.org/jira/browse/BOOKKEEPER-203?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Sijie Guo updated BOOKKEEPER-203:
---------------------------------

    Attachment: BOOKKEEPER-203.diff_v2

attach a new patch, which brought the code to latest trunk and addressed Flavio's suggestions. 
                
> improve ledger manager interface to remove zookeeper dependency on metadata operations.
> ---------------------------------------------------------------------------------------
>
>                 Key: BOOKKEEPER-203
>                 URL: https://issues.apache.org/jira/browse/BOOKKEEPER-203
>             Project: Bookkeeper
>          Issue Type: Sub-task
>          Components: bookkeeper-client, bookkeeper-server
>            Reporter: Sijie Guo
>            Assignee: Sijie Guo
>             Fix For: 4.2.0
>
>         Attachments: BOOKKEEPER-203.diff, BOOKKEEPER-203.diff_v2
>
>
> we need to improve ledger manager interface to remove zookeeper dependency on metadata operations, so it is easy for us to implement a MetaStore based ledger manager.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

[jira] [Commented] (BOOKKEEPER-203) improve ledger manager interface to remove zookeeper dependency on metadata operations.

Posted by "Ivan Kelly (JIRA)" <ji...@apache.org>.
    [ https://issues.apache.org/jira/browse/BOOKKEEPER-203?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13396586#comment-13396586 ] 

Ivan Kelly commented on BOOKKEEPER-203:
---------------------------------------

Ok, I'll take one last pass over the patch, but i think it's ready to go. 
                
> improve ledger manager interface to remove zookeeper dependency on metadata operations.
> ---------------------------------------------------------------------------------------
>
>                 Key: BOOKKEEPER-203
>                 URL: https://issues.apache.org/jira/browse/BOOKKEEPER-203
>             Project: Bookkeeper
>          Issue Type: Sub-task
>          Components: bookkeeper-client, bookkeeper-server
>            Reporter: Sijie Guo
>            Assignee: Sijie Guo
>             Fix For: 4.2.0
>
>         Attachments: BOOKKEEPER-203.diff, BOOKKEEPER-203.diff_v2, BOOKKEEPER-203.diff_v3, BOOKKEEPER-203.diff_v4, BOOKKEEPER-203.diff_v5, BOOKKEEPER-203.diff_v6, BOOKKEEPER-203.diff_v7
>
>
> we need to improve ledger manager interface to remove zookeeper dependency on metadata operations, so it is easy for us to implement a MetaStore based ledger manager.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

[jira] [Updated] (BOOKKEEPER-203) improve ledger manager interface to remove zookeeper dependency on metadata operations.

Posted by "Sijie Guo (JIRA)" <ji...@apache.org>.
     [ https://issues.apache.org/jira/browse/BOOKKEEPER-203?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Sijie Guo updated BOOKKEEPER-203:
---------------------------------

    Attachment: BOOKKEEPER-203.diff_v4

new patch 1) updated to latest trunk 2) introduce Version interface to remove zookeeper integer version dependency. because not every storage support integer version.
                
> improve ledger manager interface to remove zookeeper dependency on metadata operations.
> ---------------------------------------------------------------------------------------
>
>                 Key: BOOKKEEPER-203
>                 URL: https://issues.apache.org/jira/browse/BOOKKEEPER-203
>             Project: Bookkeeper
>          Issue Type: Sub-task
>          Components: bookkeeper-client, bookkeeper-server
>            Reporter: Sijie Guo
>            Assignee: Sijie Guo
>             Fix For: 4.2.0
>
>         Attachments: BOOKKEEPER-203.diff, BOOKKEEPER-203.diff_v2, BOOKKEEPER-203.diff_v3, BOOKKEEPER-203.diff_v4
>
>
> we need to improve ledger manager interface to remove zookeeper dependency on metadata operations, so it is easy for us to implement a MetaStore based ledger manager.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

[jira] [Commented] (BOOKKEEPER-203) improve ledger manager interface to remove zookeeper dependency on metadata operations.

Posted by "Ivan Kelly (JIRA)" <ji...@apache.org>.
    [ https://issues.apache.org/jira/browse/BOOKKEEPER-203?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13396664#comment-13396664 ] 

Ivan Kelly commented on BOOKKEEPER-203:
---------------------------------------

looks good +1
                
> improve ledger manager interface to remove zookeeper dependency on metadata operations.
> ---------------------------------------------------------------------------------------
>
>                 Key: BOOKKEEPER-203
>                 URL: https://issues.apache.org/jira/browse/BOOKKEEPER-203
>             Project: Bookkeeper
>          Issue Type: Sub-task
>          Components: bookkeeper-client, bookkeeper-server
>            Reporter: Sijie Guo
>            Assignee: Sijie Guo
>             Fix For: 4.2.0
>
>         Attachments: BOOKKEEPER-203.diff, BOOKKEEPER-203.diff_v2, BOOKKEEPER-203.diff_v3, BOOKKEEPER-203.diff_v4, BOOKKEEPER-203.diff_v5, BOOKKEEPER-203.diff_v6, BOOKKEEPER-203.diff_v7
>
>
> we need to improve ledger manager interface to remove zookeeper dependency on metadata operations, so it is easy for us to implement a MetaStore based ledger manager.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

[jira] [Commented] (BOOKKEEPER-203) improve ledger manager interface to remove zookeeper dependency on metadata operations.

Posted by "Sijie Guo (JIRA)" <ji...@apache.org>.
    [ https://issues.apache.org/jira/browse/BOOKKEEPER-203?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13393673#comment-13393673 ] 

Sijie Guo commented on BOOKKEEPER-203:
--------------------------------------

{quote}
You may as well you MultiLedgerManagerBaseTestCase for the places where you use MultiLedgerManagerTestCase and get rid of MultiLedgerManagerTestCase.
{quote}
MultiLedgerManagerBaseTestCase is used to test different digest type and ledger manager, which gets rid of adding Hierarchical prefixed test cases.

MultiLedgerManagerTestCase is used to test different ledger manager.
                
> improve ledger manager interface to remove zookeeper dependency on metadata operations.
> ---------------------------------------------------------------------------------------
>
>                 Key: BOOKKEEPER-203
>                 URL: https://issues.apache.org/jira/browse/BOOKKEEPER-203
>             Project: Bookkeeper
>          Issue Type: Sub-task
>          Components: bookkeeper-client, bookkeeper-server
>            Reporter: Sijie Guo
>            Assignee: Sijie Guo
>             Fix For: 4.2.0
>
>         Attachments: BOOKKEEPER-203.diff, BOOKKEEPER-203.diff_v2, BOOKKEEPER-203.diff_v3, BOOKKEEPER-203.diff_v4
>
>
> we need to improve ledger manager interface to remove zookeeper dependency on metadata operations, so it is easy for us to implement a MetaStore based ledger manager.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

[jira] [Commented] (BOOKKEEPER-203) improve ledger manager interface to remove zookeeper dependency on metadata operations.

Posted by "jiraposter@reviews.apache.org (Commented) (JIRA)" <ji...@apache.org>.
    [ https://issues.apache.org/jira/browse/BOOKKEEPER-203?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13245762#comment-13245762 ] 

jiraposter@reviews.apache.org commented on BOOKKEEPER-203:
----------------------------------------------------------


-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/4603/#review6616
-----------------------------------------------------------


It looks pretty good for me, I just have a few small comments.


bookkeeper-server/src/main/java/org/apache/bookkeeper/client/BKException.java
<https://reviews.apache.org/r/4603/#comment14294>

    For clarity, can we rename it to MetaStoreException or MetadataStoreException?



bookkeeper-server/src/main/java/org/apache/bookkeeper/client/BKException.java
<https://reviews.apache.org/r/4603/#comment14397>

    MetaStoreException



bookkeeper-server/src/main/java/org/apache/bookkeeper/client/LedgerMetadata.java
<https://reviews.apache.org/r/4603/#comment14401>

    It doesn't look like we need to make this call public. If it is for testing, we could simply use inheritance, no?



bookkeeper-server/src/main/java/org/apache/bookkeeper/conf/AbstractConfiguration.java
<https://reviews.apache.org/r/4603/#comment14404>

    I'm wondering if it is ok to have a reference specific to zk here. Is the idea here that for every new backend we add, we include required system properties here? 



bookkeeper-server/src/main/java/org/apache/bookkeeper/meta/ActiveLedgerManager.java
<https://reviews.apache.org/r/4603/#comment14417>

    You may want to say bookie instead of server side.



bookkeeper-server/src/main/java/org/apache/bookkeeper/meta/ActiveLedgerManager.java
<https://reviews.apache.org/r/4603/#comment14416>

    I wonder if ActiveLedgerManager is a good name for this class. It doesn't indicate that it is related to the bookie side, and it might be good to refer to it. But, I don't feel too strongly about it, since the comment before the class says that it is for bookies.



bookkeeper-server/src/main/java/org/apache/bookkeeper/meta/LedgerManager.java
<https://reviews.apache.org/r/4603/#comment14405>

    The comment here should not refer to zk, right?


- fpj


On 2012-04-01 15:56:33, Sijie Guo wrote:
bq.  
bq.  -----------------------------------------------------------
bq.  This is an automatically generated e-mail. To reply, visit:
bq.  https://reviews.apache.org/r/4603/
bq.  -----------------------------------------------------------
bq.  
bq.  (Updated 2012-04-01 15:56:33)
bq.  
bq.  
bq.  Review request for bookkeeper.
bq.  
bq.  
bq.  Summary
bq.  -------
bq.  
bq.  we need to improve ledger manager interface to remove zookeeper dependency on metadata operations, so it is easy for us to implement a MetaStore based ledger manager.
bq.  
bq.  
bq.  This addresses bug BOOKKEEPER-203.
bq.      https://issues.apache.org/jira/browse/BOOKKEEPER-203
bq.  
bq.  
bq.  Diffs
bq.  -----
bq.  
bq.    bookkeeper-server/src/main/java/org/apache/bookkeeper/bookie/Bookie.java ee38862 
bq.    bookkeeper-server/src/main/java/org/apache/bookkeeper/bookie/GarbageCollectorThread.java 2eed192 
bq.    bookkeeper-server/src/main/java/org/apache/bookkeeper/bookie/LedgerCacheImpl.java 17c70f1 
bq.    bookkeeper-server/src/main/java/org/apache/bookkeeper/client/BKException.java 911c660 
bq.    bookkeeper-server/src/main/java/org/apache/bookkeeper/client/BookKeeper.java 490c130 
bq.    bookkeeper-server/src/main/java/org/apache/bookkeeper/client/BookKeeperAdmin.java b9cd624 
bq.    bookkeeper-server/src/main/java/org/apache/bookkeeper/client/LedgerCreateOp.java f0a165f 
bq.    bookkeeper-server/src/main/java/org/apache/bookkeeper/client/LedgerDeleteOp.java 5b10a5b 
bq.    bookkeeper-server/src/main/java/org/apache/bookkeeper/client/LedgerHandle.java e576792 
bq.    bookkeeper-server/src/main/java/org/apache/bookkeeper/client/LedgerMetadata.java 2dcf0e4 
bq.    bookkeeper-server/src/main/java/org/apache/bookkeeper/client/LedgerOpenOp.java 56186ab 
bq.    bookkeeper-server/src/main/java/org/apache/bookkeeper/conf/AbstractConfiguration.java e1bd1ff 
bq.    bookkeeper-server/src/main/java/org/apache/bookkeeper/meta/AbstractZkLedgerManager.java 8fd7df8 
bq.    bookkeeper-server/src/main/java/org/apache/bookkeeper/meta/ActiveLedgerManager.java PRE-CREATION 
bq.    bookkeeper-server/src/main/java/org/apache/bookkeeper/meta/FlatLedgerManager.java 1300974 
bq.    bookkeeper-server/src/main/java/org/apache/bookkeeper/meta/FlatLedgerManagerFactory.java PRE-CREATION 
bq.    bookkeeper-server/src/main/java/org/apache/bookkeeper/meta/HierarchicalLedgerManager.java b573181 
bq.    bookkeeper-server/src/main/java/org/apache/bookkeeper/meta/HierarchicalLedgerManagerFactory.java PRE-CREATION 
bq.    bookkeeper-server/src/main/java/org/apache/bookkeeper/meta/LedgerLayout.java 3a3a81e 
bq.    bookkeeper-server/src/main/java/org/apache/bookkeeper/meta/LedgerManager.java 7d9eb96 
bq.    bookkeeper-server/src/main/java/org/apache/bookkeeper/meta/LedgerManagerFactory.java b62ad3a 
bq.    bookkeeper-server/src/main/java/org/apache/bookkeeper/util/ReflectionUtils.java PRE-CREATION 
bq.    bookkeeper-server/src/test/java/org/apache/bookkeeper/bookie/LedgerCacheTest.java 52cf514 
bq.    bookkeeper-server/src/test/java/org/apache/bookkeeper/client/BookieRecoveryTest.java 6135250 
bq.    bookkeeper-server/src/test/java/org/apache/bookkeeper/client/TestFencing.java c5350ad 
bq.    bookkeeper-server/src/test/java/org/apache/bookkeeper/meta/GcLedgersTest.java 12c1b5a 
bq.    bookkeeper-server/src/test/java/org/apache/bookkeeper/meta/HierarchicalAsyncLedgerOpsTest.java 24b3c12 
bq.    bookkeeper-server/src/test/java/org/apache/bookkeeper/meta/HierarchicalBookieFailureTest.java 70a4ea9 
bq.    bookkeeper-server/src/test/java/org/apache/bookkeeper/meta/HierarchicalBookieReadWriteTest.java 7d97fd4 
bq.    bookkeeper-server/src/test/java/org/apache/bookkeeper/meta/HierarchicalBookieRecoveryTest.java d99156e 
bq.    bookkeeper-server/src/test/java/org/apache/bookkeeper/meta/HierarchicalLedgerDeleteTest.java dbcd65a 
bq.    bookkeeper-server/src/test/java/org/apache/bookkeeper/meta/LedgerLayoutTest.java 4784854 
bq.    bookkeeper-server/src/test/java/org/apache/bookkeeper/meta/LedgerManagerTestCase.java b8a541b 
bq.    bookkeeper-server/src/test/java/org/apache/bookkeeper/meta/TestLedgerManager.java e07c756 
bq.    bookkeeper-server/src/test/java/org/apache/bookkeeper/test/AsyncLedgerOpsTest.java d7f153a 
bq.    bookkeeper-server/src/test/java/org/apache/bookkeeper/test/BookKeeperClusterTestCase.java c172224 
bq.    bookkeeper-server/src/test/java/org/apache/bookkeeper/test/BookieFailureTest.java e647d68 
bq.    bookkeeper-server/src/test/java/org/apache/bookkeeper/test/BookieReadWriteTest.java 3a3c8af 
bq.    bookkeeper-server/src/test/java/org/apache/bookkeeper/test/LedgerDeleteTest.java 2d3593e 
bq.    bookkeeper-server/src/test/java/org/apache/bookkeeper/test/MultiLedgerManagerBaseTestCase.java PRE-CREATION 
bq.    bookkeeper-server/src/test/java/org/apache/bookkeeper/test/MultiLedgerManagerTestCase.java PRE-CREATION 
bq.  
bq.  Diff: https://reviews.apache.org/r/4603/diff
bq.  
bq.  
bq.  Testing
bq.  -------
bq.  
bq.  
bq.  Thanks,
bq.  
bq.  Sijie
bq.  
bq.


                
> improve ledger manager interface to remove zookeeper dependency on metadata operations.
> ---------------------------------------------------------------------------------------
>
>                 Key: BOOKKEEPER-203
>                 URL: https://issues.apache.org/jira/browse/BOOKKEEPER-203
>             Project: Bookkeeper
>          Issue Type: Sub-task
>          Components: bookkeeper-client, bookkeeper-server
>            Reporter: Sijie Guo
>            Assignee: Sijie Guo
>             Fix For: 4.1.0
>
>         Attachments: BOOKKEEPER-203.diff
>
>
> we need to improve ledger manager interface to remove zookeeper dependency on metadata operations, so it is easy for us to implement a MetaStore based ledger manager.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

[jira] [Commented] (BOOKKEEPER-203) improve ledger manager interface to remove zookeeper dependency on metadata operations.

Posted by "Ivan Kelly (JIRA)" <ji...@apache.org>.
    [ https://issues.apache.org/jira/browse/BOOKKEEPER-203?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13395481#comment-13395481 ] 

Ivan Kelly commented on BOOKKEEPER-203:
---------------------------------------

{code}
MultiLedgerManagerBaseTestCase is used to test different digest type and ledger manager, which gets rid of adding Hierarchical prefixed test cases.

MultiLedgerManagerTestCase is used to test different ledger manager.
{code}
So MultiLedgerManagerBaseTestCase is really a superset of MultiLedgerManagerTestCase. I don't see any harm in just using MultiLedgerManagerBaseTestCase for all. The reason I wrote some test cases which didn't use the BaseTestCase, was because with Parameterized junit tests, it's not possible to run individual tests (i.e. mvn test -Dtest=TestFencing#testBasicFencing), which I found annoying. However, if the test is going to use a Parameterized junit test in any case, we should just run everything.
                
> improve ledger manager interface to remove zookeeper dependency on metadata operations.
> ---------------------------------------------------------------------------------------
>
>                 Key: BOOKKEEPER-203
>                 URL: https://issues.apache.org/jira/browse/BOOKKEEPER-203
>             Project: Bookkeeper
>          Issue Type: Sub-task
>          Components: bookkeeper-client, bookkeeper-server
>            Reporter: Sijie Guo
>            Assignee: Sijie Guo
>             Fix For: 4.2.0
>
>         Attachments: BOOKKEEPER-203.diff, BOOKKEEPER-203.diff_v2, BOOKKEEPER-203.diff_v3, BOOKKEEPER-203.diff_v4, BOOKKEEPER-203.diff_v5
>
>
> we need to improve ledger manager interface to remove zookeeper dependency on metadata operations, so it is easy for us to implement a MetaStore based ledger manager.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

[jira] [Updated] (BOOKKEEPER-203) improve ledger manager interface to remove zookeeper dependency on metadata operations.

Posted by "Sijie Guo (JIRA)" <ji...@apache.org>.
     [ https://issues.apache.org/jira/browse/BOOKKEEPER-203?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Sijie Guo updated BOOKKEEPER-203:
---------------------------------

    Attachment: BOOKKEEPER-203.diff_v5

update new patch addressing Ivan's comments.

1) rebased to latest trunk
2) remove unused MetaStoreException and setClass in ReflectionUtils
3) improve LedgerManagerFactory#newLedgerManagerFactory for backward compatibility.
4) add uninitialize in LedgerManagerFactory to release resources used by ledger manager factory.
                
> improve ledger manager interface to remove zookeeper dependency on metadata operations.
> ---------------------------------------------------------------------------------------
>
>                 Key: BOOKKEEPER-203
>                 URL: https://issues.apache.org/jira/browse/BOOKKEEPER-203
>             Project: Bookkeeper
>          Issue Type: Sub-task
>          Components: bookkeeper-client, bookkeeper-server
>            Reporter: Sijie Guo
>            Assignee: Sijie Guo
>             Fix For: 4.2.0
>
>         Attachments: BOOKKEEPER-203.diff, BOOKKEEPER-203.diff_v2, BOOKKEEPER-203.diff_v3, BOOKKEEPER-203.diff_v4, BOOKKEEPER-203.diff_v5
>
>
> we need to improve ledger manager interface to remove zookeeper dependency on metadata operations, so it is easy for us to implement a MetaStore based ledger manager.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

[jira] [Updated] (BOOKKEEPER-203) improve ledger manager interface to remove zookeeper dependency on metadata operations.

Posted by "Sijie Guo (JIRA)" <ji...@apache.org>.
     [ https://issues.apache.org/jira/browse/BOOKKEEPER-203?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Sijie Guo updated BOOKKEEPER-203:
---------------------------------

    Attachment: BOOKKEEPER-203.diff_v7

Thanks Ivan for reminding. 

Attach a new patch rebased on latest trunk.

review board: https://reviews.apache.org/r/4603/diff/6
                
> improve ledger manager interface to remove zookeeper dependency on metadata operations.
> ---------------------------------------------------------------------------------------
>
>                 Key: BOOKKEEPER-203
>                 URL: https://issues.apache.org/jira/browse/BOOKKEEPER-203
>             Project: Bookkeeper
>          Issue Type: Sub-task
>          Components: bookkeeper-client, bookkeeper-server
>            Reporter: Sijie Guo
>            Assignee: Sijie Guo
>             Fix For: 4.2.0
>
>         Attachments: BOOKKEEPER-203.diff, BOOKKEEPER-203.diff_v2, BOOKKEEPER-203.diff_v3, BOOKKEEPER-203.diff_v4, BOOKKEEPER-203.diff_v5, BOOKKEEPER-203.diff_v6, BOOKKEEPER-203.diff_v7
>
>
> we need to improve ledger manager interface to remove zookeeper dependency on metadata operations, so it is easy for us to implement a MetaStore based ledger manager.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

[jira] [Commented] (BOOKKEEPER-203) improve ledger manager interface to remove zookeeper dependency on metadata operations.

Posted by "Sijie Guo (JIRA)" <ji...@apache.org>.
    [ https://issues.apache.org/jira/browse/BOOKKEEPER-203?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13294363#comment-13294363 ] 

Sijie Guo commented on BOOKKEEPER-203:
--------------------------------------

Ivan, thanks for reviewing.

{quote}
MetaStoreException is not used. Is in wrong patch i guess?
{quote}

I think it is introduced when I am working on BOOKKEEPER-204. would remove it in next patch.

{quote}
BookKeeper.java patch removed the throwing of NoSuchLedgerExistsException, which we added in BOOKKEEPER-273. Is this intentional?
{quote}

Actually, I put in AbstractZkLedgerManager, because NONODE is zookeeper's return code. You could refer it as below:
{code}
+    @Override
+    public void deleteLedger(final long ledgerId, final GenericCallback<Void> cb) {
+        zk.delete(getLedgerPath(ledgerId), -1, new VoidCallback() {
+            @Override
+            public void processResult(int rc, String path, Object ctx) {
+                int bkRc;
+                if (rc == KeeperException.Code.NONODE.intValue()) {
+                    LOG.warn("Ledger node does not exist in ZooKeeper: ledgerId={}", ledgerId);
+                    bkRc = BKException.Code.NoSuchLedgerExistsException;
+                } else if (rc == KeeperException.Code.OK.intValue()) {
+                    bkRc = BKException.Code.OK;
+                } else {
+                    bkRc = BKException.Code.ZKException;
+                }
+                cb.operationComplete(bkRc, (Void)null);
+            }
+        }, null);
+    }
{code}

{quote}
there are two overloads of getClass where only one is ever used. setClass is unused.
{quote}

ah, ur right. would remove unused methods.



{quote}
LedgerManagerFactory#newLedgerManagerFactory breaks getLedgerManagerType(). if factoryClass is null and layout is null, it should check getLedgerManagerType before setting factoryClass to FlatLedgerManagerType or HierarchicalLedgerManagerType.
{quote}
yes. ur right. would improve it.





                
> improve ledger manager interface to remove zookeeper dependency on metadata operations.
> ---------------------------------------------------------------------------------------
>
>                 Key: BOOKKEEPER-203
>                 URL: https://issues.apache.org/jira/browse/BOOKKEEPER-203
>             Project: Bookkeeper
>          Issue Type: Sub-task
>          Components: bookkeeper-client, bookkeeper-server
>            Reporter: Sijie Guo
>            Assignee: Sijie Guo
>             Fix For: 4.2.0
>
>         Attachments: BOOKKEEPER-203.diff, BOOKKEEPER-203.diff_v2, BOOKKEEPER-203.diff_v3, BOOKKEEPER-203.diff_v4
>
>
> we need to improve ledger manager interface to remove zookeeper dependency on metadata operations, so it is easy for us to implement a MetaStore based ledger manager.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

[jira] [Updated] (BOOKKEEPER-203) improve ledger manager interface to remove zookeeper dependency on metadata operations.

Posted by "Sijie Guo (JIRA)" <ji...@apache.org>.
     [ https://issues.apache.org/jira/browse/BOOKKEEPER-203?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Sijie Guo updated BOOKKEEPER-203:
---------------------------------

    Attachment: BOOKKEEPER-203.diff_v6


attach a new patch renaming MultiLedgerManagerBaseTestCase to MultiLedgerManagerMultiDigestTestCase

review board: https://reviews.apache.org/r/4603/diff/5/#index_header
                
> improve ledger manager interface to remove zookeeper dependency on metadata operations.
> ---------------------------------------------------------------------------------------
>
>                 Key: BOOKKEEPER-203
>                 URL: https://issues.apache.org/jira/browse/BOOKKEEPER-203
>             Project: Bookkeeper
>          Issue Type: Sub-task
>          Components: bookkeeper-client, bookkeeper-server
>            Reporter: Sijie Guo
>            Assignee: Sijie Guo
>             Fix For: 4.2.0
>
>         Attachments: BOOKKEEPER-203.diff, BOOKKEEPER-203.diff_v2, BOOKKEEPER-203.diff_v3, BOOKKEEPER-203.diff_v4, BOOKKEEPER-203.diff_v5, BOOKKEEPER-203.diff_v6
>
>
> we need to improve ledger manager interface to remove zookeeper dependency on metadata operations, so it is easy for us to implement a MetaStore based ledger manager.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira