You are viewing a plain text version of this content. The canonical link for it is here.
Posted to common-dev@hadoop.apache.org by "Jim Kellerman (JIRA)" <ji...@apache.org> on 2006/10/14 17:11:35 UTC
[jira] Created: (HADOOP-603) Extend SequenceFile to provide MapFile
function by storing index at the end of the file
Extend SequenceFile to provide MapFile function by storing index at the end of the file
---------------------------------------------------------------------------------------
Key: HADOOP-603
URL: http://issues.apache.org/jira/browse/HADOOP-603
Project: Hadoop
Issue Type: Improvement
Components: dfs
Reporter: Jim Kellerman
MapFile increases the load on the name node as two files are created to provide a index file format. If SequenceFile were extended by storing the index at the end of the file, 1/2 of the files currently created for a map/reduce operation would be needed, reducing the load on the name node.
Perhaps this is why Google implemented SSTable files in this manner. (SSTable files are functionally identical to Hadoop MapFiles) (see the paper on BigTable - section 4 "Building Blocks" http://labs.google.com/papers/bigtable.html)
--
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators: http://issues.apache.org/jira/secure/Administrators.jspa
-
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] Commented: (HADOOP-603) Extend SequenceFile to provide
MapFile function by storing index at the end of the file
Posted by "eric baldeschwieler (JIRA)" <ji...@apache.org>.
[ http://issues.apache.org/jira/browse/HADOOP-603?page=comments#action_12442654 ]
eric baldeschwieler commented on HADOOP-603:
--------------------------------------------
Yeah. Just keeping an array of offset vectors per partition would be simpler.
Jim: Don't see a problem with your suggestion (HADOOP-603), but I don't think the two projects relate. These files will be broken into partitions. Fitting that into a MapFile might be a bit of a stretch.
> Extend SequenceFile to provide MapFile function by storing index at the end of the file
> ---------------------------------------------------------------------------------------
>
> Key: HADOOP-603
> URL: http://issues.apache.org/jira/browse/HADOOP-603
> Project: Hadoop
> Issue Type: Improvement
> Components: dfs
> Reporter: Jim Kellerman
>
> MapFile increases the load on the name node as two files are created to provide a index file format. If SequenceFile were extended by storing the index at the end of the file, 1/2 of the files currently created for a map/reduce operation would be needed, reducing the load on the name node.
> Perhaps this is why Google implemented SSTable files in this manner. (SSTable files are functionally identical to Hadoop MapFiles) (see the paper on BigTable - section 4 "Building Blocks" http://labs.google.com/papers/bigtable.html)
--
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators: http://issues.apache.org/jira/secure/Administrators.jspa
-
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] Commented: (HADOOP-603) Extend SequenceFile to provide
MapFile function by storing index at the end of the file
Posted by "Jim Kellerman (JIRA)" <ji...@apache.org>.
[ http://issues.apache.org/jira/browse/HADOOP-603?page=comments#action_12442490 ]
Jim Kellerman commented on HADOOP-603:
--------------------------------------
I believe that Google still supports append and re-writes the index at the end of the file. The advantage of this approach is that when you access these files it takes 1/2 the number of opens that MapFile does. This would significantly reduce the load on the name node.
> Extend SequenceFile to provide MapFile function by storing index at the end of the file
> ---------------------------------------------------------------------------------------
>
> Key: HADOOP-603
> URL: http://issues.apache.org/jira/browse/HADOOP-603
> Project: Hadoop
> Issue Type: Improvement
> Components: dfs
> Reporter: Jim Kellerman
>
> MapFile increases the load on the name node as two files are created to provide a index file format. If SequenceFile were extended by storing the index at the end of the file, 1/2 of the files currently created for a map/reduce operation would be needed, reducing the load on the name node.
> Perhaps this is why Google implemented SSTable files in this manner. (SSTable files are functionally identical to Hadoop MapFiles) (see the paper on BigTable - section 4 "Building Blocks" http://labs.google.com/papers/bigtable.html)
--
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators: http://issues.apache.org/jira/secure/Administrators.jspa
-
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] Commented: (HADOOP-603) Extend SequenceFile to provide
MapFile function by storing index at the end of the file
Posted by "Runping Qi (JIRA)" <ji...@apache.org>.
[ http://issues.apache.org/jira/browse/HADOOP-603?page=comments#action_12442336 ]
Runping Qi commented on HADOOP-603:
-----------------------------------
Storing the ondex at the end will make it impossible for appending!
Although Hadoop does not support append now, it is still open for that possibility.
I think it is important to leave that possibility open.
> Extend SequenceFile to provide MapFile function by storing index at the end of the file
> ---------------------------------------------------------------------------------------
>
> Key: HADOOP-603
> URL: http://issues.apache.org/jira/browse/HADOOP-603
> Project: Hadoop
> Issue Type: Improvement
> Components: dfs
> Reporter: Jim Kellerman
>
> MapFile increases the load on the name node as two files are created to provide a index file format. If SequenceFile were extended by storing the index at the end of the file, 1/2 of the files currently created for a map/reduce operation would be needed, reducing the load on the name node.
> Perhaps this is why Google implemented SSTable files in this manner. (SSTable files are functionally identical to Hadoop MapFiles) (see the paper on BigTable - section 4 "Building Blocks" http://labs.google.com/papers/bigtable.html)
--
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators: http://issues.apache.org/jira/secure/Administrators.jspa
-
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] Updated: (HADOOP-603) Extend SequenceFile to provide MapFile
function by storing index at the end of the file
Posted by "eric baldeschwieler (JIRA)" <ji...@apache.org>.
[ http://issues.apache.org/jira/browse/HADOOP-603?page=all ]
eric baldeschwieler updated HADOOP-603:
---------------------------------------
Comment: was deleted
> Extend SequenceFile to provide MapFile function by storing index at the end of the file
> ---------------------------------------------------------------------------------------
>
> Key: HADOOP-603
> URL: http://issues.apache.org/jira/browse/HADOOP-603
> Project: Hadoop
> Issue Type: Improvement
> Components: dfs
> Reporter: Jim Kellerman
>
> MapFile increases the load on the name node as two files are created to provide a index file format. If SequenceFile were extended by storing the index at the end of the file, 1/2 of the files currently created for a map/reduce operation would be needed, reducing the load on the name node.
> Perhaps this is why Google implemented SSTable files in this manner. (SSTable files are functionally identical to Hadoop MapFiles) (see the paper on BigTable - section 4 "Building Blocks" http://labs.google.com/papers/bigtable.html)
--
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators: http://issues.apache.org/jira/secure/Administrators.jspa
-
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] Commented: (HADOOP-603) Extend SequenceFile to provide
MapFile function by storing index at the end of the file
Posted by "Doug Cutting (JIRA)" <ji...@apache.org>.
[ http://issues.apache.org/jira/browse/HADOOP-603?page=comments#action_12456202 ]
Doug Cutting commented on HADOOP-603:
-------------------------------------
This sounds reasonable to me. Are you working on a patch?
> Extend SequenceFile to provide MapFile function by storing index at the end of the file
> ---------------------------------------------------------------------------------------
>
> Key: HADOOP-603
> URL: http://issues.apache.org/jira/browse/HADOOP-603
> Project: Hadoop
> Issue Type: Improvement
> Components: dfs
> Reporter: Jim Kellerman
>
> MapFile increases the load on the name node as two files are created to provide a index file format. If SequenceFile were extended by storing the index at the end of the file, 1/2 of the files currently created for a map/reduce operation would be needed, reducing the load on the name node.
> Perhaps this is why Google implemented SSTable files in this manner. (SSTable files are functionally identical to Hadoop MapFiles) (see the paper on BigTable - section 4 "Building Blocks" http://labs.google.com/papers/bigtable.html)
--
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators: http://issues.apache.org/jira/secure/Administrators.jspa
-
For more information on JIRA, see: http://www.atlassian.com/software/jira