[jira] [Commented] (BOOKKEEPER-220) Managed Ledger proposal

["Matteo Merli (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/BOOKKEEPER-220?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13266354#comment-13266354 ] 

Matteo Merli commented on BOOKKEEPER-220:
-----------------------------------------

In our design, hubs (brokers) manage groups of topics/queues. Each group is owned by one broker at once and it's reassigned when doing failover. 

A single topic/queue maps (1:1) with a ML, so there is only one broker that works with a ML at any time, both for reading/writing. This is exactly the same as in Hedwig case.

Anyway, it would be interesting to have the ability of having many readers running on different brokers that consumes from the same ML, although this complicates the design. The major problem that I see is synchronizing all the instances of the ML running on different boxes to decide which ledgers can be deleted.

For the ML, the goal we have is to build something like a persistent log appender, with multiple persistent reader marks that can be failed over to other servers when crashing/rebalancing. This is the minimal set of features that we were thinking of.

I believe this is very similar to what is currently used for Hedwig.

                
> Managed Ledger proposal
> -----------------------
>
>                 Key: BOOKKEEPER-220
>                 URL: https://issues.apache.org/jira/browse/BOOKKEEPER-220
>             Project: Bookkeeper
>          Issue Type: New Feature
>          Components: bookkeeper-client
>            Reporter: Matteo Merli
>            Assignee: Matteo Merli
>
> The ManagedLedger design is based on our need to manage a set of ledgers, with a single writer (at any point in time) and a set on consumers that read entries from it. 
> The ManagedLedger also takes care of periodically closing ledgers to have a "reasonable" sized sets of ledgers that can individually deleted when no more needed.
> I've put on github the interface proposal (along with an early WIP implementation)
> http://github.com/merlimat/managed-ledger

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira