You are viewing a plain text version of this content. The canonical link for it is here.
Posted to hdfs-issues@hadoop.apache.org by "dhruba borthakur (JIRA)" <ji...@apache.org> on 2009/10/08 01:31:31 UTC
[jira] Created: (HDFS-684) Use HAR filesystem to merge parity files
Use HAR filesystem to merge parity files
-----------------------------------------
Key: HDFS-684
URL: https://issues.apache.org/jira/browse/HDFS-684
Project: Hadoop HDFS
Issue Type: Improvement
Components: contrib/raid
Reporter: dhruba borthakur
The HDFS raid implementation (HDFS-503) creates a parity file for every file that is RAIDed. This puts additional burden on the memory requirements of the namenode. It will be nice if the parity files are combined together using the HadoopArchive (har) format.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HDFS-684) Use HAR filesystem to merge parity
files
Posted by "Rodrigo Schmidt (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-684?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12797212#action_12797212 ]
Rodrigo Schmidt commented on HDFS-684:
--------------------------------------
This patch needs hadoop-...-tools.jar and I didn't know how to change raid's build.xml file just to add hadoop-...-tools.jar to the classpath without redefining the whole classpath and all the other targets.
To make the patch compile and run the unit tests in my computer I just copied hadoop-...-tools.jar to [HADOOP_HOME]/lib, but I know this is not the best thing to do and that's why it failed Hadoop QA.
What's the most non-intrusive way to change raid's build.xml file in this case?
> Use HAR filesystem to merge parity files
> -----------------------------------------
>
> Key: HDFS-684
> URL: https://issues.apache.org/jira/browse/HDFS-684
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: contrib/raid
> Reporter: dhruba borthakur
> Assignee: Rodrigo Schmidt
> Attachments: HDFS-684.0.patch, HDFS-684.1.patch
>
>
> The HDFS raid implementation (HDFS-503) creates a parity file for every file that is RAIDed. This puts additional burden on the memory requirements of the namenode. It will be nice if the parity files are combined together using the HadoopArchive (har) format.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HDFS-684) Use HAR filesystem to merge parity
files
Posted by "dhruba borthakur (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-684?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12781883#action_12781883 ]
dhruba borthakur commented on HDFS-684:
---------------------------------------
Hi Rodrigo, thanks for the info. I guess it is mostly the case that all files in a directory are likely to get updated together. That helps in creating a single raid out of them.
> Use HAR filesystem to merge parity files
> -----------------------------------------
>
> Key: HDFS-684
> URL: https://issues.apache.org/jira/browse/HDFS-684
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: contrib/raid
> Reporter: dhruba borthakur
> Assignee: Rodrigo Schmidt
>
> The HDFS raid implementation (HDFS-503) creates a parity file for every file that is RAIDed. This puts additional burden on the memory requirements of the namenode. It will be nice if the parity files are combined together using the HadoopArchive (har) format.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HDFS-684) Use HAR filesystem to merge parity files
Posted by "Rodrigo Schmidt (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-684?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Rodrigo Schmidt updated HDFS-684:
---------------------------------
Attachment: HDFS-684.0.patch
Initial patch that enables the RaidNode to read from har files. It is missing the code to create such files, though.
> Use HAR filesystem to merge parity files
> -----------------------------------------
>
> Key: HDFS-684
> URL: https://issues.apache.org/jira/browse/HDFS-684
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: contrib/raid
> Reporter: dhruba borthakur
> Assignee: Rodrigo Schmidt
> Attachments: HDFS-684.0.patch
>
>
> The HDFS raid implementation (HDFS-503) creates a parity file for every file that is RAIDed. This puts additional burden on the memory requirements of the namenode. It will be nice if the parity files are combined together using the HadoopArchive (har) format.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HDFS-684) Use HAR filesystem to merge parity files
Posted by "Rodrigo Schmidt (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-684?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Rodrigo Schmidt updated HDFS-684:
---------------------------------
Status: Patch Available (was: Open)
> Use HAR filesystem to merge parity files
> -----------------------------------------
>
> Key: HDFS-684
> URL: https://issues.apache.org/jira/browse/HDFS-684
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: contrib/raid
> Reporter: dhruba borthakur
> Assignee: Rodrigo Schmidt
> Attachments: HDFS-684.0.patch, HDFS-684.1.patch
>
>
> The HDFS raid implementation (HDFS-503) creates a parity file for every file that is RAIDed. This puts additional burden on the memory requirements of the namenode. It will be nice if the parity files are combined together using the HadoopArchive (har) format.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HDFS-684) Use HAR filesystem to merge parity files
Posted by "dhruba borthakur (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-684?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
dhruba borthakur updated HDFS-684:
----------------------------------
Status: Open (was: Patch Available)
I am looking into the hadoop-tools issue. will comment soon.
Please let me understand this patch a little bit more.. Suppose a directory /dhruba has 10 files in it. All files initially have a replication factor of 3. Then the RaidNode creates a xxx_raid.har that replaces all the parity files. Now suppose, a user deletes the first file /dhruba. Now /dhruba has only 9 files. This patch will now delete the har file associated with /dhruba. At this point, all the 9 files in /dhruba are left with a replication factor of 2 only! Am I understanding this right?
The next iteration of RaidNode will recreate the parity files (and then har them). this is good.
> Use HAR filesystem to merge parity files
> -----------------------------------------
>
> Key: HDFS-684
> URL: https://issues.apache.org/jira/browse/HDFS-684
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: contrib/raid
> Reporter: dhruba borthakur
> Assignee: Rodrigo Schmidt
> Attachments: HDFS-684.0.patch, HDFS-684.1.patch
>
>
> The HDFS raid implementation (HDFS-503) creates a parity file for every file that is RAIDed. This puts additional burden on the memory requirements of the namenode. It will be nice if the parity files are combined together using the HadoopArchive (har) format.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Assigned: (HDFS-684) Use HAR filesystem to merge parity
files
Posted by "Rodrigo Schmidt (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-684?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Rodrigo Schmidt reassigned HDFS-684:
------------------------------------
Assignee: Rodrigo Schmidt
> Use HAR filesystem to merge parity files
> -----------------------------------------
>
> Key: HDFS-684
> URL: https://issues.apache.org/jira/browse/HDFS-684
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: contrib/raid
> Reporter: dhruba borthakur
> Assignee: Rodrigo Schmidt
>
> The HDFS raid implementation (HDFS-503) creates a parity file for every file that is RAIDed. This puts additional burden on the memory requirements of the namenode. It will be nice if the parity files are combined together using the HadoopArchive (har) format.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HDFS-684) Use HAR filesystem to merge parity
files
Posted by "Hadoop QA (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-684?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12796328#action_12796328 ]
Hadoop QA commented on HDFS-684:
--------------------------------
-1 overall. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12429357/HDFS-684.1.patch
against trunk revision 894233.
+1 @author. The patch does not contain any @author tags.
+1 tests included. The patch appears to include 7 new or modified tests.
+1 javadoc. The javadoc tool did not generate any warning messages.
-1 javac. The patch appears to cause tar ant target to fail.
+1 findbugs. The patch does not introduce any new Findbugs warnings.
+1 release audit. The applied patch does not increase the total number of release audit warnings.
+1 core tests. The patch passed core unit tests.
-1 contrib tests. The patch failed contrib unit tests.
Test results: http://hudson.zones.apache.org/hudson/job/Hdfs-Patch-h5.grid.sp2.yahoo.net/167/testReport/
Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Hdfs-Patch-h5.grid.sp2.yahoo.net/167/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
Checkstyle results: http://hudson.zones.apache.org/hudson/job/Hdfs-Patch-h5.grid.sp2.yahoo.net/167/artifact/trunk/build/test/checkstyle-errors.html
Console output: http://hudson.zones.apache.org/hudson/job/Hdfs-Patch-h5.grid.sp2.yahoo.net/167/console
This message is automatically generated.
> Use HAR filesystem to merge parity files
> -----------------------------------------
>
> Key: HDFS-684
> URL: https://issues.apache.org/jira/browse/HDFS-684
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: contrib/raid
> Reporter: dhruba borthakur
> Assignee: Rodrigo Schmidt
> Attachments: HDFS-684.0.patch, HDFS-684.1.patch
>
>
> The HDFS raid implementation (HDFS-503) creates a parity file for every file that is RAIDed. This puts additional burden on the memory requirements of the namenode. It will be nice if the parity files are combined together using the HadoopArchive (har) format.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HDFS-684) Use HAR filesystem to merge parity
files
Posted by "Rodrigo Schmidt (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-684?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12780709#action_12780709 ]
Rodrigo Schmidt commented on HDFS-684:
--------------------------------------
I'm thinking of creating a HAR file for every policy in the RAID configuration file. Does that sound like a good idea?
> Use HAR filesystem to merge parity files
> -----------------------------------------
>
> Key: HDFS-684
> URL: https://issues.apache.org/jira/browse/HDFS-684
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: contrib/raid
> Reporter: dhruba borthakur
> Assignee: Rodrigo Schmidt
>
> The HDFS raid implementation (HDFS-503) creates a parity file for every file that is RAIDed. This puts additional burden on the memory requirements of the namenode. It will be nice if the parity files are combined together using the HadoopArchive (har) format.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HDFS-684) Use HAR filesystem to merge parity
files
Posted by "Rodrigo Schmidt (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-684?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12781766#action_12781766 ]
Rodrigo Schmidt commented on HDFS-684:
--------------------------------------
The code would be simpler if we had one HAR file for every policy, but I definitely understand your point in having one file per directory. I'm working on it.
> Use HAR filesystem to merge parity files
> -----------------------------------------
>
> Key: HDFS-684
> URL: https://issues.apache.org/jira/browse/HDFS-684
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: contrib/raid
> Reporter: dhruba borthakur
> Assignee: Rodrigo Schmidt
>
> The HDFS raid implementation (HDFS-503) creates a parity file for every file that is RAIDed. This puts additional burden on the memory requirements of the namenode. It will be nice if the parity files are combined together using the HadoopArchive (har) format.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Resolved: (HDFS-684) Use HAR filesystem to merge parity
files
Posted by "Rodrigo Schmidt (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-684?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Rodrigo Schmidt resolved HDFS-684.
----------------------------------
Resolution: Duplicate
> Use HAR filesystem to merge parity files
> -----------------------------------------
>
> Key: HDFS-684
> URL: https://issues.apache.org/jira/browse/HDFS-684
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: contrib/raid
> Reporter: dhruba borthakur
> Assignee: Rodrigo Schmidt
> Attachments: HDFS-684.0.patch, HDFS-684.1.patch
>
>
> The HDFS raid implementation (HDFS-503) creates a parity file for every file that is RAIDed. This puts additional burden on the memory requirements of the namenode. It will be nice if the parity files are combined together using the HadoopArchive (har) format.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HDFS-684) Use HAR filesystem to merge parity
files
Posted by "dhruba borthakur (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-684?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12799125#action_12799125 ]
dhruba borthakur commented on HDFS-684:
---------------------------------------
Another issue is that the Har-ing of parity files should occur in a map-reduce job. Otherwise, a single node will not be able to keep up with the rate of Har-ing all the party files in a big cluster.
> Use HAR filesystem to merge parity files
> -----------------------------------------
>
> Key: HDFS-684
> URL: https://issues.apache.org/jira/browse/HDFS-684
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: contrib/raid
> Reporter: dhruba borthakur
> Assignee: Rodrigo Schmidt
> Attachments: HDFS-684.0.patch, HDFS-684.1.patch
>
>
> The HDFS raid implementation (HDFS-503) creates a parity file for every file that is RAIDed. This puts additional burden on the memory requirements of the namenode. It will be nice if the parity files are combined together using the HadoopArchive (har) format.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HDFS-684) Use HAR filesystem to merge parity
files
Posted by "dhruba borthakur (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-684?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12780996#action_12780996 ]
dhruba borthakur commented on HDFS-684:
---------------------------------------
Is there a downside to crearting a HAR for every directory in the /raid directory? This will imply that there will be one HAR for every data-set, because a single data-set usually reside in one single directory.
If you create one RAID file for all files in a certain policy, and we RAID new files eveyr hour based on that policy, then each of these iterations will have to detele and recreate the entire HAR again, isn't it (I am assuming that you cannot add/delete items to a previously created har file)
> Use HAR filesystem to merge parity files
> -----------------------------------------
>
> Key: HDFS-684
> URL: https://issues.apache.org/jira/browse/HDFS-684
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: contrib/raid
> Reporter: dhruba borthakur
> Assignee: Rodrigo Schmidt
>
> The HDFS raid implementation (HDFS-503) creates a parity file for every file that is RAIDed. This puts additional burden on the memory requirements of the namenode. It will be nice if the parity files are combined together using the HadoopArchive (har) format.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Issue Comment Edited: (HDFS-684) Use HAR filesystem to merge
parity files
Posted by "dhruba borthakur (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-684?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12780996#action_12780996 ]
dhruba borthakur edited comment on HDFS-684 at 11/21/09 3:23 PM:
-----------------------------------------------------------------
Is there a downside to crearting a HAR for every directory in the /raid directory? This will imply that there will be one HAR for every data-set, because a single data-set usually reside in one single directory.
If you create one Har file for all files in a certain policy, and we RAID new files every hour based on that policy, then each of these iterations will have to delete and recreate the entire HAR again, isn't it (I am assuming that you cannot add/delete items to a previously created har file)
was (Author: dhruba):
Is there a downside to crearting a HAR for every directory in the /raid directory? This will imply that there will be one HAR for every data-set, because a single data-set usually reside in one single directory.
If you create one RAID file for all files in a certain policy, and we RAID new files eveyr hour based on that policy, then each of these iterations will have to detele and recreate the entire HAR again, isn't it (I am assuming that you cannot add/delete items to a previously created har file)
> Use HAR filesystem to merge parity files
> -----------------------------------------
>
> Key: HDFS-684
> URL: https://issues.apache.org/jira/browse/HDFS-684
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: contrib/raid
> Reporter: dhruba borthakur
> Assignee: Rodrigo Schmidt
>
> The HDFS raid implementation (HDFS-503) creates a parity file for every file that is RAIDed. This puts additional burden on the memory requirements of the namenode. It will be nice if the parity files are combined together using the HadoopArchive (har) format.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HDFS-684) Use HAR filesystem to merge parity files
Posted by "Rodrigo Schmidt (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-684?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Rodrigo Schmidt updated HDFS-684:
---------------------------------
Attachment: HDFS-684.1.patch
New patch that creates har directories and uses them when necessary.
> Use HAR filesystem to merge parity files
> -----------------------------------------
>
> Key: HDFS-684
> URL: https://issues.apache.org/jira/browse/HDFS-684
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: contrib/raid
> Reporter: dhruba borthakur
> Assignee: Rodrigo Schmidt
> Attachments: HDFS-684.0.patch, HDFS-684.1.patch
>
>
> The HDFS raid implementation (HDFS-503) creates a parity file for every file that is RAIDed. This puts additional burden on the memory requirements of the namenode. It will be nice if the parity files are combined together using the HadoopArchive (har) format.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HDFS-684) Use HAR filesystem to merge parity
files
Posted by "Rodrigo Schmidt (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-684?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12833508#action_12833508 ]
Rodrigo Schmidt commented on HDFS-684:
--------------------------------------
This is now being solved on MAPREDUCE-1491
> Use HAR filesystem to merge parity files
> -----------------------------------------
>
> Key: HDFS-684
> URL: https://issues.apache.org/jira/browse/HDFS-684
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: contrib/raid
> Reporter: dhruba borthakur
> Assignee: Rodrigo Schmidt
> Attachments: HDFS-684.0.patch, HDFS-684.1.patch
>
>
> The HDFS raid implementation (HDFS-503) creates a parity file for every file that is RAIDed. This puts additional burden on the memory requirements of the namenode. It will be nice if the parity files are combined together using the HadoopArchive (har) format.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HDFS-684) Use HAR filesystem to merge parity
files
Posted by "dhruba borthakur (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-684?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12791700#action_12791700 ]
dhruba borthakur commented on HDFS-684:
---------------------------------------
This part of the code looks good. I will wait to see the code that generates har files. Thanks.
> Use HAR filesystem to merge parity files
> -----------------------------------------
>
> Key: HDFS-684
> URL: https://issues.apache.org/jira/browse/HDFS-684
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: contrib/raid
> Reporter: dhruba borthakur
> Assignee: Rodrigo Schmidt
> Attachments: HDFS-684.0.patch
>
>
> The HDFS raid implementation (HDFS-503) creates a parity file for every file that is RAIDed. This puts additional burden on the memory requirements of the namenode. It will be nice if the parity files are combined together using the HadoopArchive (har) format.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.