[jira] [Commented] (BOOKKEEPER-181) Scale hedwig

["Flavio Junqueira (Commented) (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/BOOKKEEPER-181?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13239508#comment-13239508 ] 

Flavio Junqueira commented on BOOKKEEPER-181:
---------------------------------------------

Hi Sijie, I agree with creating sub-tasks for this umbrella jira. For bookkeeper, we need to access ledger metadata both from clients and bookies, right? We may want to have a separate task for each, even though they will most likely rely upon the same interface to the metadata repository.

I'd like to clarify one point. In my understanding, we still plan to use zookeeper for node availability, while we move all metadata to another scalable data store, such as hbase. Is this correct? If so, the plugable interface will allow the use of different repositories for the metadata part, but we will still rely upon zookeeper to monitor node availability.

I have a few other comments about the design doc:

# In the definition of the compare-and-swap operation, the comparison is performed using the key and value itself. This might be expensive, so I was wondering if it is a better approach to use versions instead. The drawback is relying upon a backend that provides versioned data. It seems fine for me, though.
# Related to the previous comment, it might be a better idea to state somewhere what properties we require from the backend store. 
# I'm not entirely sure I understand the implementation of leader election in 5.1. What happens if a hub is incorrectly suspected of crashing and it loses ownership over a topic? Does it find out via session expiration? Also, I suppose that if the hub has crashed but the list of hubs hasn't changed, then multiple iterations of 1 may have to happen.
                
> Scale hedwig
> ------------
>
>                 Key: BOOKKEEPER-181
>                 URL: https://issues.apache.org/jira/browse/BOOKKEEPER-181
>             Project: Bookkeeper
>          Issue Type: Improvement
>          Components: bookkeeper-server, hedwig-server
>            Reporter: Sijie Guo
>            Assignee: Sijie Guo
>         Attachments: hedwigscale.pdf
>
>
> Current implementation of Hedwig and BookKeeper is designed to scale to hundreds of thousands of topics, but now we are looking at scaling them to tens to hundreds of millions of topics, using a scalable key/value store such as HBase.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira