You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@bookkeeper.apache.org by "Ivan Kelly (Created) (JIRA)" <ji...@apache.org> on 2012/03/29 17:04:28 UTC
[jira] [Created] (BOOKKEEPER-196) Define interface between bookie
and ledger storage
Define interface between bookie and ledger storage
--------------------------------------------------
Key: BOOKKEEPER-196
URL: https://issues.apache.org/jira/browse/BOOKKEEPER-196
Project: Bookkeeper
Issue Type: Improvement
Reporter: Ivan Kelly
Assignee: Ivan Kelly
Fix For: 4.1.0
EntryLogger and LedgerCache are both part of a very interdependent storage mechanism where entries are interleaved in a single log(EntryLogger) and index files are maintained (LedgerCache). I'd like to experiment with some other schemes (Im not convinced the interleaving is required for high performance). ZOOKEEPER-507 brought in these changes, but it also brought in a lot of other stuff, and I think its the other stuff (specifically taking the writing to separate files out of the critical path) which gave us the performance boost. To do this cleanly, we need a well defined storage interface for this. This JIRA is to provide this. Future work can move the interleaved implementation into another package as org.apache.bookkeeper.bookie is getting a little crowded now.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (BOOKKEEPER-196) Define interface between bookie
and ledger storage
Posted by "Ivan Kelly (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/BOOKKEEPER-196?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13255449#comment-13255449 ]
Ivan Kelly commented on BOOKKEEPER-196:
---------------------------------------
Committed as r1327027. Thanks for reviewing Sijie.
> Define interface between bookie and ledger storage
> --------------------------------------------------
>
> Key: BOOKKEEPER-196
> URL: https://issues.apache.org/jira/browse/BOOKKEEPER-196
> Project: Bookkeeper
> Issue Type: Improvement
> Reporter: Ivan Kelly
> Assignee: Ivan Kelly
> Fix For: 4.1.0
>
> Attachments: BOOKKEEPER-196.diff, BOOKKEEPER-196.diff, BOOKKEEPER-196.diff
>
>
> EntryLogger and LedgerCache are both part of a very interdependent storage mechanism where entries are interleaved in a single log(EntryLogger) and index files are maintained (LedgerCache). I'd like to experiment with some other schemes (Im not convinced the interleaving is required for high performance). ZOOKEEPER-507 brought in these changes, but it also brought in a lot of other stuff, and I think its the other stuff (specifically taking the writing to separate files out of the critical path) which gave us the performance boost. To do this cleanly, we need a well defined storage interface for this. This JIRA is to provide this. Future work can move the interleaved implementation into another package as org.apache.bookkeeper.bookie is getting a little crowded now.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (BOOKKEEPER-196) Define interface between bookie
and ledger storage
Posted by "Ivan Kelly (Updated) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/BOOKKEEPER-196?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Ivan Kelly updated BOOKKEEPER-196:
----------------------------------
Attachment: BOOKKEEPER-196.diff
> Define interface between bookie and ledger storage
> --------------------------------------------------
>
> Key: BOOKKEEPER-196
> URL: https://issues.apache.org/jira/browse/BOOKKEEPER-196
> Project: Bookkeeper
> Issue Type: Improvement
> Reporter: Ivan Kelly
> Assignee: Ivan Kelly
> Fix For: 4.1.0
>
> Attachments: BOOKKEEPER-196.diff, BOOKKEEPER-196.diff
>
>
> EntryLogger and LedgerCache are both part of a very interdependent storage mechanism where entries are interleaved in a single log(EntryLogger) and index files are maintained (LedgerCache). I'd like to experiment with some other schemes (Im not convinced the interleaving is required for high performance). ZOOKEEPER-507 brought in these changes, but it also brought in a lot of other stuff, and I think its the other stuff (specifically taking the writing to separate files out of the critical path) which gave us the performance boost. To do this cleanly, we need a well defined storage interface for this. This JIRA is to provide this. Future work can move the interleaved implementation into another package as org.apache.bookkeeper.bookie is getting a little crowded now.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (BOOKKEEPER-196) Define interface between bookie
and ledger storage
Posted by "Ivan Kelly (Updated) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/BOOKKEEPER-196?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Ivan Kelly updated BOOKKEEPER-196:
----------------------------------
Attachment: BOOKKEEPER-196.diff
Rebased onto trunk
> Define interface between bookie and ledger storage
> --------------------------------------------------
>
> Key: BOOKKEEPER-196
> URL: https://issues.apache.org/jira/browse/BOOKKEEPER-196
> Project: Bookkeeper
> Issue Type: Improvement
> Reporter: Ivan Kelly
> Assignee: Ivan Kelly
> Fix For: 4.1.0
>
> Attachments: BOOKKEEPER-196.diff, BOOKKEEPER-196.diff, BOOKKEEPER-196.diff
>
>
> EntryLogger and LedgerCache are both part of a very interdependent storage mechanism where entries are interleaved in a single log(EntryLogger) and index files are maintained (LedgerCache). I'd like to experiment with some other schemes (Im not convinced the interleaving is required for high performance). ZOOKEEPER-507 brought in these changes, but it also brought in a lot of other stuff, and I think its the other stuff (specifically taking the writing to separate files out of the critical path) which gave us the performance boost. To do this cleanly, we need a well defined storage interface for this. This JIRA is to provide this. Future work can move the interleaved implementation into another package as org.apache.bookkeeper.bookie is getting a little crowded now.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (BOOKKEEPER-196) Define interface between bookie
and ledger storage
Posted by "Hudson (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/BOOKKEEPER-196?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13255468#comment-13255468 ]
Hudson commented on BOOKKEEPER-196:
-----------------------------------
Integrated in bookkeeper-trunk #462 (See [https://builds.apache.org/job/bookkeeper-trunk/462/])
BOOKKEEPER-196: Define interface between bookie and ledger storage (ivank) (Revision 1327027)
Result = ABORTED
ivank :
Files :
* /zookeeper/bookkeeper/trunk/CHANGES.txt
* /zookeeper/bookkeeper/trunk/bookkeeper-server/src/main/java/org/apache/bookkeeper/bookie/Bookie.java
* /zookeeper/bookkeeper/trunk/bookkeeper-server/src/main/java/org/apache/bookkeeper/bookie/EntryLogger.java
* /zookeeper/bookkeeper/trunk/bookkeeper-server/src/main/java/org/apache/bookkeeper/bookie/GarbageCollectorThread.java
* /zookeeper/bookkeeper/trunk/bookkeeper-server/src/main/java/org/apache/bookkeeper/bookie/HandleFactoryImpl.java
* /zookeeper/bookkeeper/trunk/bookkeeper-server/src/main/java/org/apache/bookkeeper/bookie/InterleavedLedgerStorage.java
* /zookeeper/bookkeeper/trunk/bookkeeper-server/src/main/java/org/apache/bookkeeper/bookie/LedgerDescriptor.java
* /zookeeper/bookkeeper/trunk/bookkeeper-server/src/main/java/org/apache/bookkeeper/bookie/LedgerDescriptorImpl.java
* /zookeeper/bookkeeper/trunk/bookkeeper-server/src/main/java/org/apache/bookkeeper/bookie/LedgerDescriptorReadOnlyImpl.java
* /zookeeper/bookkeeper/trunk/bookkeeper-server/src/main/java/org/apache/bookkeeper/bookie/LedgerStorage.java
> Define interface between bookie and ledger storage
> --------------------------------------------------
>
> Key: BOOKKEEPER-196
> URL: https://issues.apache.org/jira/browse/BOOKKEEPER-196
> Project: Bookkeeper
> Issue Type: Improvement
> Reporter: Ivan Kelly
> Assignee: Ivan Kelly
> Fix For: 4.1.0
>
> Attachments: BOOKKEEPER-196.diff, BOOKKEEPER-196.diff, BOOKKEEPER-196.diff
>
>
> EntryLogger and LedgerCache are both part of a very interdependent storage mechanism where entries are interleaved in a single log(EntryLogger) and index files are maintained (LedgerCache). I'd like to experiment with some other schemes (Im not convinced the interleaving is required for high performance). ZOOKEEPER-507 brought in these changes, but it also brought in a lot of other stuff, and I think its the other stuff (specifically taking the writing to separate files out of the critical path) which gave us the performance boost. To do this cleanly, we need a well defined storage interface for this. This JIRA is to provide this. Future work can move the interleaved implementation into another package as org.apache.bookkeeper.bookie is getting a little crowded now.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (BOOKKEEPER-196) Define interface between bookie
and ledger storage
Posted by "Sijie Guo (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/BOOKKEEPER-196?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13254494#comment-13254494 ]
Sijie Guo commented on BOOKKEEPER-196:
--------------------------------------
the patch is good to me, although seems that the patch should be updated to latest trunk after BOOKKEEPER-135 is in.
> Define interface between bookie and ledger storage
> --------------------------------------------------
>
> Key: BOOKKEEPER-196
> URL: https://issues.apache.org/jira/browse/BOOKKEEPER-196
> Project: Bookkeeper
> Issue Type: Improvement
> Reporter: Ivan Kelly
> Assignee: Ivan Kelly
> Fix For: 4.1.0
>
> Attachments: BOOKKEEPER-196.diff, BOOKKEEPER-196.diff
>
>
> EntryLogger and LedgerCache are both part of a very interdependent storage mechanism where entries are interleaved in a single log(EntryLogger) and index files are maintained (LedgerCache). I'd like to experiment with some other schemes (Im not convinced the interleaving is required for high performance). ZOOKEEPER-507 brought in these changes, but it also brought in a lot of other stuff, and I think its the other stuff (specifically taking the writing to separate files out of the critical path) which gave us the performance boost. To do this cleanly, we need a well defined storage interface for this. This JIRA is to provide this. Future work can move the interleaved implementation into another package as org.apache.bookkeeper.bookie is getting a little crowded now.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (BOOKKEEPER-196) Define interface between bookie
and ledger storage
Posted by "Ivan Kelly (Updated) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/BOOKKEEPER-196?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Ivan Kelly updated BOOKKEEPER-196:
----------------------------------
Attachment: BOOKKEEPER-196.diff
> Define interface between bookie and ledger storage
> --------------------------------------------------
>
> Key: BOOKKEEPER-196
> URL: https://issues.apache.org/jira/browse/BOOKKEEPER-196
> Project: Bookkeeper
> Issue Type: Improvement
> Reporter: Ivan Kelly
> Assignee: Ivan Kelly
> Fix For: 4.1.0
>
> Attachments: BOOKKEEPER-196.diff
>
>
> EntryLogger and LedgerCache are both part of a very interdependent storage mechanism where entries are interleaved in a single log(EntryLogger) and index files are maintained (LedgerCache). I'd like to experiment with some other schemes (Im not convinced the interleaving is required for high performance). ZOOKEEPER-507 brought in these changes, but it also brought in a lot of other stuff, and I think its the other stuff (specifically taking the writing to separate files out of the critical path) which gave us the performance boost. To do this cleanly, we need a well defined storage interface for this. This JIRA is to provide this. Future work can move the interleaved implementation into another package as org.apache.bookkeeper.bookie is getting a little crowded now.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (BOOKKEEPER-196) Define interface between bookie
and ledger storage
Posted by "Sijie Guo (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/BOOKKEEPER-196?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13254713#comment-13254713 ]
Sijie Guo commented on BOOKKEEPER-196:
--------------------------------------
the new patch is OK to me. +1
> Define interface between bookie and ledger storage
> --------------------------------------------------
>
> Key: BOOKKEEPER-196
> URL: https://issues.apache.org/jira/browse/BOOKKEEPER-196
> Project: Bookkeeper
> Issue Type: Improvement
> Reporter: Ivan Kelly
> Assignee: Ivan Kelly
> Fix For: 4.1.0
>
> Attachments: BOOKKEEPER-196.diff, BOOKKEEPER-196.diff, BOOKKEEPER-196.diff
>
>
> EntryLogger and LedgerCache are both part of a very interdependent storage mechanism where entries are interleaved in a single log(EntryLogger) and index files are maintained (LedgerCache). I'd like to experiment with some other schemes (Im not convinced the interleaving is required for high performance). ZOOKEEPER-507 brought in these changes, but it also brought in a lot of other stuff, and I think its the other stuff (specifically taking the writing to separate files out of the critical path) which gave us the performance boost. To do this cleanly, we need a well defined storage interface for this. This JIRA is to provide this. Future work can move the interleaved implementation into another package as org.apache.bookkeeper.bookie is getting a little crowded now.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira