You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@accumulo.apache.org by "Eric Newton (Created) (JIRA)" <ji...@apache.org> on 2011/11/04 17:09:51 UTC
[jira] [Created] (ACCUMULO-118) accumulo could work across HDFS
instances, which would help it to scale past a single namenode
accumulo could work across HDFS instances, which would help it to scale past a single namenode
----------------------------------------------------------------------------------------------
Key: ACCUMULO-118
URL: https://issues.apache.org/jira/browse/ACCUMULO-118
Project: Accumulo
Issue Type: Improvement
Components: master, tserver
Affects Versions: 1.5.0
Reporter: Eric Newton
Assignee: Eric Newton
Consider using full path names to files, which would allow the servers to access the files on any HDFS file system.
Work may exist elsewhere to run HDFS using a number of NameNode instances to break up the namespace.
We may need a pluggable strategy to determine namespace for new files.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (ACCUMULO-118) accumulo could work across HDFS
instances, which would help it to scale past a single namenode
Posted by "Eric Newton (Updated) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/ACCUMULO-118?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Eric Newton updated ACCUMULO-118:
---------------------------------
Remaining Estimate: 2,016h (was: 0.05h)
Original Estimate: 2,016h (was: 0.05h)
> accumulo could work across HDFS instances, which would help it to scale past a single namenode
> ----------------------------------------------------------------------------------------------
>
> Key: ACCUMULO-118
> URL: https://issues.apache.org/jira/browse/ACCUMULO-118
> Project: Accumulo
> Issue Type: Improvement
> Components: master, tserver
> Affects Versions: 1.5.0
> Reporter: Eric Newton
> Assignee: Eric Newton
> Original Estimate: 2,016h
> Remaining Estimate: 2,016h
>
> Consider using full path names to files, which would allow the servers to access the files on any HDFS file system.
> Work may exist elsewhere to run HDFS using a number of NameNode instances to break up the namespace.
> We may need a pluggable strategy to determine namespace for new files.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (ACCUMULO-118) accumulo could work across HDFS
instances, which would help it to scale past a single namenode
Posted by "Eric Newton (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/ACCUMULO-118?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13146741#comment-13146741 ]
Eric Newton commented on ACCUMULO-118:
--------------------------------------
Interesting! It looks like a 0.23.0 is very close to release, maybe?
> accumulo could work across HDFS instances, which would help it to scale past a single namenode
> ----------------------------------------------------------------------------------------------
>
> Key: ACCUMULO-118
> URL: https://issues.apache.org/jira/browse/ACCUMULO-118
> Project: Accumulo
> Issue Type: Improvement
> Components: master, tserver
> Affects Versions: 1.5.0
> Reporter: Eric Newton
> Assignee: Eric Newton
> Original Estimate: 2,016h
> Remaining Estimate: 2,016h
>
> Consider using full path names to files, which would allow the servers to access the files on any HDFS file system.
> Work may exist elsewhere to run HDFS using a number of NameNode instances to break up the namespace.
> We may need a pluggable strategy to determine namespace for new files.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (ACCUMULO-118) accumulo could work across HDFS
instances, which would help it to scale past a single namenode
Posted by "Keith Turner (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/ACCUMULO-118?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13144136#comment-13144136 ]
Keith Turner commented on ACCUMULO-118:
---------------------------------------
One issue this is trying to solve is overcoming namenode scalability issues. Before working on this would need to investigate what is being done to make the namenode more scalable. Would need to decide if we want to contribute to that work, or spend time working around the issue.
> accumulo could work across HDFS instances, which would help it to scale past a single namenode
> ----------------------------------------------------------------------------------------------
>
> Key: ACCUMULO-118
> URL: https://issues.apache.org/jira/browse/ACCUMULO-118
> Project: Accumulo
> Issue Type: Improvement
> Components: master, tserver
> Affects Versions: 1.5.0
> Reporter: Eric Newton
> Assignee: Eric Newton
> Original Estimate: 2,016h
> Remaining Estimate: 2,016h
>
> Consider using full path names to files, which would allow the servers to access the files on any HDFS file system.
> Work may exist elsewhere to run HDFS using a number of NameNode instances to break up the namespace.
> We may need a pluggable strategy to determine namespace for new files.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (ACCUMULO-118) accumulo could work across HDFS
instances, which would help it to scale past a single namenode
Posted by "Todd Lipcon (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/ACCUMULO-118?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13144144#comment-13144144 ]
Todd Lipcon commented on ACCUMULO-118:
--------------------------------------
In HDFS 0.23 we have federation, which is (a) server side support so that a pool of DNs can have blocks that span multiple NNs/namespaces, and (b) client side mount table support "viewfs". So, you could have 10,000 DNs, a few NNs, and a client side mount table that divides up the namespace across those NNs.
> accumulo could work across HDFS instances, which would help it to scale past a single namenode
> ----------------------------------------------------------------------------------------------
>
> Key: ACCUMULO-118
> URL: https://issues.apache.org/jira/browse/ACCUMULO-118
> Project: Accumulo
> Issue Type: Improvement
> Components: master, tserver
> Affects Versions: 1.5.0
> Reporter: Eric Newton
> Assignee: Eric Newton
> Original Estimate: 2,016h
> Remaining Estimate: 2,016h
>
> Consider using full path names to files, which would allow the servers to access the files on any HDFS file system.
> Work may exist elsewhere to run HDFS using a number of NameNode instances to break up the namespace.
> We may need a pluggable strategy to determine namespace for new files.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (ACCUMULO-118) accumulo could work across HDFS
instances, which would help it to scale past a single namenode
Posted by "Todd Lipcon (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/ACCUMULO-118?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13148734#comment-13148734 ]
Todd Lipcon commented on ACCUMULO-118:
--------------------------------------
Yep, we've now released 0.23.0. It's still considered Alpha, but HDFS should be in basic working condition at least!
> accumulo could work across HDFS instances, which would help it to scale past a single namenode
> ----------------------------------------------------------------------------------------------
>
> Key: ACCUMULO-118
> URL: https://issues.apache.org/jira/browse/ACCUMULO-118
> Project: Accumulo
> Issue Type: Improvement
> Components: master, tserver
> Affects Versions: 1.5.0
> Reporter: Eric Newton
> Assignee: Eric Newton
> Original Estimate: 2,016h
> Remaining Estimate: 2,016h
>
> Consider using full path names to files, which would allow the servers to access the files on any HDFS file system.
> Work may exist elsewhere to run HDFS using a number of NameNode instances to break up the namespace.
> We may need a pluggable strategy to determine namespace for new files.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira