You are viewing a plain text version of this content. The canonical link for it is here.
Posted to common-issues@hadoop.apache.org by "Matt Foley (JIRA)" <ji...@apache.org> on 2011/02/09 19:48:57 UTC
[jira] Created: (HADOOP-7133) CLONE to COMMON - HDFS-1445 Batch the
calls in DataStorage to FileUtil.createHardLink(), so we call it once per
directory instead of once per file
CLONE to COMMON - HDFS-1445 Batch the calls in DataStorage to FileUtil.createHardLink(), so we call it once per directory instead of once per file
--------------------------------------------------------------------------------------------------------------------------------------------------
Key: HADOOP-7133
URL: https://issues.apache.org/jira/browse/HADOOP-7133
Project: Hadoop Common
Issue Type: Improvement
Components: util
Affects Versions: 0.20.2
Reporter: Matt Foley
Assignee: Matt Foley
Fix For: 0.22.0
Attachments: HDFS-1445-trunk.v22_common_1-of-2.patch
The fix for HDFS-1445 "Batch the calls in DataStorage to FileUtil.createHardLink(), so we call it once per directory instead of once per file" requires coordinated change in COMMON and HDFS. This is the COMMON portion, submitted here under a separate bug to activate the automated testing.
Warning: this patch to COMMON, by itself, will break HDFS. It requires coordinated commit of the HDFS portion of the patch in HDFS-1445.
--
This message is automatically generated by JIRA.
-
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] Updated: (HADOOP-7133) CLONE to COMMON - HDFS-1445 Batch the
calls in DataStorage to FileUtil.createHardLink(), so we call it once per
directory instead of once per file
Posted by "Matt Foley (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-7133?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Matt Foley updated HADOOP-7133:
-------------------------------
Attachment: HDFS-1445-trunk.v23_common_1-of-3.patch
This new version of the part-1 patch is backward compatible and will not break HDFS.
Please review.
> CLONE to COMMON - HDFS-1445 Batch the calls in DataStorage to FileUtil.createHardLink(), so we call it once per directory instead of once per file
> --------------------------------------------------------------------------------------------------------------------------------------------------
>
> Key: HADOOP-7133
> URL: https://issues.apache.org/jira/browse/HADOOP-7133
> Project: Hadoop Common
> Issue Type: Improvement
> Components: util
> Affects Versions: 0.20.2
> Reporter: Matt Foley
> Assignee: Matt Foley
> Fix For: 0.22.0
>
> Attachments: HDFS-1445-trunk.v23_common_1-of-3.patch
>
>
> The fix for HDFS-1445 "Batch the calls in DataStorage to FileUtil.createHardLink(), so we call it once per directory instead of once per file" requires coordinated change in COMMON and HDFS. This is the COMMON portion, submitted here under a separate bug to activate the automated testing.
> Warning: this patch to COMMON, by itself, will break HDFS. It requires coordinated commit of the HDFS portion of the patch in HDFS-1445.
--
This message is automatically generated by JIRA.
-
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] Commented: (HADOOP-7133) CLONE to COMMON - HDFS-1445 Batch
the calls in DataStorage to FileUtil.createHardLink(), so we call it once
per directory instead of once per file
Posted by "Hairong Kuang (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-7133?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13004383#comment-13004383 ]
Hairong Kuang commented on HADOOP-7133:
---------------------------------------
Matt, this optimization is fantastic. I have ported it to our internal branch. Together with parallel upgrades, it cut the DN upgrade time from 1 hour 40 minutes to 1 minute! Awesome job!
+1. The patch looks good. One minor optimization is that you could use ThreadLocal for shell commands. This will remove the need to create a buffer for each shell command to run.
> CLONE to COMMON - HDFS-1445 Batch the calls in DataStorage to FileUtil.createHardLink(), so we call it once per directory instead of once per file
> --------------------------------------------------------------------------------------------------------------------------------------------------
>
> Key: HADOOP-7133
> URL: https://issues.apache.org/jira/browse/HADOOP-7133
> Project: Hadoop Common
> Issue Type: Improvement
> Components: util
> Affects Versions: 0.20.2
> Reporter: Matt Foley
> Assignee: Matt Foley
> Fix For: 0.22.0
>
> Attachments: HDFS-1445-trunk.v23_common_1-of-3.patch
>
>
> The fix for HDFS-1445 "Batch the calls in DataStorage to FileUtil.createHardLink(), so we call it once per directory instead of once per file" requires coordinated change in COMMON and HDFS. This is the COMMON portion, submitted here under a separate bug to activate the automated testing.
> Warning: this patch to COMMON, by itself, will break HDFS. It requires coordinated commit of the HDFS portion of the patch in HDFS-1445.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] Commented: (HADOOP-7133) CLONE to COMMON - HDFS-1445 Batch
the calls in DataStorage to FileUtil.createHardLink(), so we call it once
per directory instead of once per file
Posted by "Hadoop QA (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-7133?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12996165#comment-12996165 ]
Hadoop QA commented on HADOOP-7133:
-----------------------------------
+1 overall. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12471347/HDFS-1445-trunk.v23_common_1-of-3.patch
against trunk revision 1071364.
+1 @author. The patch does not contain any @author tags.
+1 tests included. The patch appears to include 6 new or modified tests.
+1 javadoc. The javadoc tool did not generate any warning messages.
+1 javac. The applied patch does not increase the total number of javac compiler warnings.
+1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings.
+1 release audit. The applied patch does not increase the total number of release audit warnings.
+1 core tests. The patch passed core unit tests.
+1 contrib tests. The patch passed contrib unit tests.
+1 system test framework. The patch passed system test framework compile.
Test results: https://hudson.apache.org/hudson/job/PreCommit-HADOOP-Build/240//testReport/
Findbugs warnings: https://hudson.apache.org/hudson/job/PreCommit-HADOOP-Build/240//artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
Console output: https://hudson.apache.org/hudson/job/PreCommit-HADOOP-Build/240//console
This message is automatically generated.
> CLONE to COMMON - HDFS-1445 Batch the calls in DataStorage to FileUtil.createHardLink(), so we call it once per directory instead of once per file
> --------------------------------------------------------------------------------------------------------------------------------------------------
>
> Key: HADOOP-7133
> URL: https://issues.apache.org/jira/browse/HADOOP-7133
> Project: Hadoop Common
> Issue Type: Improvement
> Components: util
> Affects Versions: 0.20.2
> Reporter: Matt Foley
> Assignee: Matt Foley
> Fix For: 0.22.0
>
> Attachments: HDFS-1445-trunk.v23_common_1-of-3.patch
>
>
> The fix for HDFS-1445 "Batch the calls in DataStorage to FileUtil.createHardLink(), so we call it once per directory instead of once per file" requires coordinated change in COMMON and HDFS. This is the COMMON portion, submitted here under a separate bug to activate the automated testing.
> Warning: this patch to COMMON, by itself, will break HDFS. It requires coordinated commit of the HDFS portion of the patch in HDFS-1445.
--
This message is automatically generated by JIRA.
-
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] Commented: (HADOOP-7133) CLONE to COMMON - HDFS-1445 Batch
the calls in DataStorage to FileUtil.createHardLink(), so we call it once
per directory instead of once per file
Posted by "Matt Foley (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-7133?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13004326#comment-13004326 ]
Matt Foley commented on HADOOP-7133:
------------------------------------
Hi Jakob, the HDFS counterpart has been up for some time on HDFS-1445. However, test-patch can't run against the HDFS part until this COMMON part has been committed. So perhaps it would be best if you take a quick look at HDFS-1445, then commit this patch so that I can enable test-patch to run against the other part? Thanks.
> CLONE to COMMON - HDFS-1445 Batch the calls in DataStorage to FileUtil.createHardLink(), so we call it once per directory instead of once per file
> --------------------------------------------------------------------------------------------------------------------------------------------------
>
> Key: HADOOP-7133
> URL: https://issues.apache.org/jira/browse/HADOOP-7133
> Project: Hadoop Common
> Issue Type: Improvement
> Components: util
> Affects Versions: 0.20.2
> Reporter: Matt Foley
> Assignee: Matt Foley
> Fix For: 0.22.0
>
> Attachments: HDFS-1445-trunk.v23_common_1-of-3.patch
>
>
> The fix for HDFS-1445 "Batch the calls in DataStorage to FileUtil.createHardLink(), so we call it once per directory instead of once per file" requires coordinated change in COMMON and HDFS. This is the COMMON portion, submitted here under a separate bug to activate the automated testing.
> Warning: this patch to COMMON, by itself, will break HDFS. It requires coordinated commit of the HDFS portion of the patch in HDFS-1445.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] Commented: (HADOOP-7133) CLONE to COMMON - HDFS-1445 Batch
the calls in DataStorage to FileUtil.createHardLink(), so we call it once
per directory instead of once per file
Posted by "Hairong Kuang (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-7133?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13005299#comment-13005299 ]
Hairong Kuang commented on HADOOP-7133:
---------------------------------------
Sure, please go ahead and commit it. I already +1 it. :)
> CLONE to COMMON - HDFS-1445 Batch the calls in DataStorage to FileUtil.createHardLink(), so we call it once per directory instead of once per file
> --------------------------------------------------------------------------------------------------------------------------------------------------
>
> Key: HADOOP-7133
> URL: https://issues.apache.org/jira/browse/HADOOP-7133
> Project: Hadoop Common
> Issue Type: Improvement
> Components: util
> Affects Versions: 0.20.2
> Reporter: Matt Foley
> Assignee: Matt Foley
> Fix For: 0.22.0
>
> Attachments: HDFS-1445-trunk.v23_common_1-of-3.patch
>
>
> The fix for HDFS-1445 "Batch the calls in DataStorage to FileUtil.createHardLink(), so we call it once per directory instead of once per file" requires coordinated change in COMMON and HDFS. This is the COMMON portion, submitted here under a separate bug to activate the automated testing.
> Warning: this patch to COMMON, by itself, will break HDFS. It requires coordinated commit of the HDFS portion of the patch in HDFS-1445.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] Commented: (HADOOP-7133) CLONE to COMMON - HDFS-1445 Batch
the calls in DataStorage to FileUtil.createHardLink(), so we call it once
per directory instead of once per file
Posted by "Matt Foley (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-7133?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13005766#comment-13005766 ]
Matt Foley commented on HADOOP-7133:
------------------------------------
Thanks guys. HADOOP-7181 opened for Hairong's ThreadLocal suggestion.
HDFS-1445 "submit patch" activated.
HADOOP-7182 opened to clean up the backward compatibility stub from FileUtils.
> CLONE to COMMON - HDFS-1445 Batch the calls in DataStorage to FileUtil.createHardLink(), so we call it once per directory instead of once per file
> --------------------------------------------------------------------------------------------------------------------------------------------------
>
> Key: HADOOP-7133
> URL: https://issues.apache.org/jira/browse/HADOOP-7133
> Project: Hadoop Common
> Issue Type: Improvement
> Components: util
> Affects Versions: 0.20.2
> Reporter: Matt Foley
> Assignee: Matt Foley
> Fix For: 0.23.0
>
> Attachments: HDFS-1445-trunk.v23_common_1-of-3.patch
>
>
> The fix for HDFS-1445 "Batch the calls in DataStorage to FileUtil.createHardLink(), so we call it once per directory instead of once per file" requires coordinated change in COMMON and HDFS. This is the COMMON portion, submitted here under a separate bug to activate the automated testing.
> Warning: this patch to COMMON, by itself, will break HDFS. It requires coordinated commit of the HDFS portion of the patch in HDFS-1445.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] Commented: (HADOOP-7133) CLONE to COMMON - HDFS-1445 Batch
the calls in DataStorage to FileUtil.createHardLink(), so we call it once
per directory instead of once per file
Posted by "Jakob Homan (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-7133?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13004329#comment-13004329 ]
Jakob Homan commented on HADOOP-7133:
-------------------------------------
bq. I will resubmit part-2, attached to this issue, as soon as part-1 has been approved and committed.
I had interpreted this to mean there would be a new HDFS patch. Sounds like I'm incorrect and the current patch there is the one you're submitting?
> CLONE to COMMON - HDFS-1445 Batch the calls in DataStorage to FileUtil.createHardLink(), so we call it once per directory instead of once per file
> --------------------------------------------------------------------------------------------------------------------------------------------------
>
> Key: HADOOP-7133
> URL: https://issues.apache.org/jira/browse/HADOOP-7133
> Project: Hadoop Common
> Issue Type: Improvement
> Components: util
> Affects Versions: 0.20.2
> Reporter: Matt Foley
> Assignee: Matt Foley
> Fix For: 0.22.0
>
> Attachments: HDFS-1445-trunk.v23_common_1-of-3.patch
>
>
> The fix for HDFS-1445 "Batch the calls in DataStorage to FileUtil.createHardLink(), so we call it once per directory instead of once per file" requires coordinated change in COMMON and HDFS. This is the COMMON portion, submitted here under a separate bug to activate the automated testing.
> Warning: this patch to COMMON, by itself, will break HDFS. It requires coordinated commit of the HDFS portion of the patch in HDFS-1445.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] Commented: (HADOOP-7133) CLONE to COMMON - HDFS-1445 Batch
the calls in DataStorage to FileUtil.createHardLink(), so we call it once
per directory instead of once per file
Posted by "Hudson (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-7133?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13005601#comment-13005601 ]
Hudson commented on HADOOP-7133:
--------------------------------
Integrated in Hadoop-Common-trunk #627 (See [https://hudson.apache.org/hudson/job/Hadoop-Common-trunk/627/])
HADOOP-7133. Batch the calls in DataStorage to FileUtil.createHardLink(). Contributed by Matt Foley.
> CLONE to COMMON - HDFS-1445 Batch the calls in DataStorage to FileUtil.createHardLink(), so we call it once per directory instead of once per file
> --------------------------------------------------------------------------------------------------------------------------------------------------
>
> Key: HADOOP-7133
> URL: https://issues.apache.org/jira/browse/HADOOP-7133
> Project: Hadoop Common
> Issue Type: Improvement
> Components: util
> Affects Versions: 0.20.2
> Reporter: Matt Foley
> Assignee: Matt Foley
> Fix For: 0.23.0
>
> Attachments: HDFS-1445-trunk.v23_common_1-of-3.patch
>
>
> The fix for HDFS-1445 "Batch the calls in DataStorage to FileUtil.createHardLink(), so we call it once per directory instead of once per file" requires coordinated change in COMMON and HDFS. This is the COMMON portion, submitted here under a separate bug to activate the automated testing.
> Warning: this patch to COMMON, by itself, will break HDFS. It requires coordinated commit of the HDFS portion of the patch in HDFS-1445.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] Commented: (HADOOP-7133) CLONE to COMMON - HDFS-1445 Batch
the calls in DataStorage to FileUtil.createHardLink(), so we call it once
per directory instead of once per file
Posted by "Hudson (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-7133?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13005411#comment-13005411 ]
Hudson commented on HADOOP-7133:
--------------------------------
Integrated in Hadoop-Common-trunk-Commit #523 (See [https://hudson.apache.org/hudson/job/Hadoop-Common-trunk-Commit/523/])
HADOOP-7133. Batch the calls in DataStorage to FileUtil.createHardLink(). Contributed by Matt Foley.
> CLONE to COMMON - HDFS-1445 Batch the calls in DataStorage to FileUtil.createHardLink(), so we call it once per directory instead of once per file
> --------------------------------------------------------------------------------------------------------------------------------------------------
>
> Key: HADOOP-7133
> URL: https://issues.apache.org/jira/browse/HADOOP-7133
> Project: Hadoop Common
> Issue Type: Improvement
> Components: util
> Affects Versions: 0.20.2
> Reporter: Matt Foley
> Assignee: Matt Foley
> Fix For: 0.23.0
>
> Attachments: HDFS-1445-trunk.v23_common_1-of-3.patch
>
>
> The fix for HDFS-1445 "Batch the calls in DataStorage to FileUtil.createHardLink(), so we call it once per directory instead of once per file" requires coordinated change in COMMON and HDFS. This is the COMMON portion, submitted here under a separate bug to activate the automated testing.
> Warning: this patch to COMMON, by itself, will break HDFS. It requires coordinated commit of the HDFS portion of the patch in HDFS-1445.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] Commented: (HADOOP-7133) CLONE to COMMON - HDFS-1445 Batch
the calls in DataStorage to FileUtil.createHardLink(), so we call it once
per directory instead of once per file
Posted by "Jakob Homan (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-7133?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13004790#comment-13004790 ]
Jakob Homan commented on HADOOP-7133:
-------------------------------------
Hairong: Good suggestion on the threadlocal. I'm ready to commit this as is. Would you be ok with doing that optimization in separate JIRA?
> CLONE to COMMON - HDFS-1445 Batch the calls in DataStorage to FileUtil.createHardLink(), so we call it once per directory instead of once per file
> --------------------------------------------------------------------------------------------------------------------------------------------------
>
> Key: HADOOP-7133
> URL: https://issues.apache.org/jira/browse/HADOOP-7133
> Project: Hadoop Common
> Issue Type: Improvement
> Components: util
> Affects Versions: 0.20.2
> Reporter: Matt Foley
> Assignee: Matt Foley
> Fix For: 0.22.0
>
> Attachments: HDFS-1445-trunk.v23_common_1-of-3.patch
>
>
> The fix for HDFS-1445 "Batch the calls in DataStorage to FileUtil.createHardLink(), so we call it once per directory instead of once per file" requires coordinated change in COMMON and HDFS. This is the COMMON portion, submitted here under a separate bug to activate the automated testing.
> Warning: this patch to COMMON, by itself, will break HDFS. It requires coordinated commit of the HDFS portion of the patch in HDFS-1445.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] Commented: (HADOOP-7133) CLONE to COMMON - HDFS-1445 Batch
the calls in DataStorage to FileUtil.createHardLink(), so we call it once
per directory instead of once per file
Posted by "Todd Lipcon (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-7133?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12992658#comment-12992658 ]
Todd Lipcon commented on HADOOP-7133:
-------------------------------------
Since FileUtil.HardLink is a public class, we should deprecate it in this patch and only remove it in the next release. Instead of removing it, can you have it forward calls to the new implementation? That way we also don't need to coordinate the commit across HDFS and Common.
> CLONE to COMMON - HDFS-1445 Batch the calls in DataStorage to FileUtil.createHardLink(), so we call it once per directory instead of once per file
> --------------------------------------------------------------------------------------------------------------------------------------------------
>
> Key: HADOOP-7133
> URL: https://issues.apache.org/jira/browse/HADOOP-7133
> Project: Hadoop Common
> Issue Type: Improvement
> Components: util
> Affects Versions: 0.20.2
> Reporter: Matt Foley
> Assignee: Matt Foley
> Fix For: 0.22.0
>
> Attachments: HDFS-1445-trunk.v22_common_1-of-2.patch
>
>
> The fix for HDFS-1445 "Batch the calls in DataStorage to FileUtil.createHardLink(), so we call it once per directory instead of once per file" requires coordinated change in COMMON and HDFS. This is the COMMON portion, submitted here under a separate bug to activate the automated testing.
> Warning: this patch to COMMON, by itself, will break HDFS. It requires coordinated commit of the HDFS portion of the patch in HDFS-1445.
--
This message is automatically generated by JIRA.
-
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] Updated: (HADOOP-7133) CLONE to COMMON - HDFS-1445 Batch the
calls in DataStorage to FileUtil.createHardLink(), so we call it once per
directory instead of once per file
Posted by "Jakob Homan (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-7133?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Jakob Homan updated HADOOP-7133:
--------------------------------
Resolution: Fixed
Fix Version/s: (was: 0.22.0)
0.23.0
Status: Resolved (was: Patch Available)
I've committed this. Resolving as fixed. Matt, please open a JIRA for Hairong's suggestion. Thanks.
> CLONE to COMMON - HDFS-1445 Batch the calls in DataStorage to FileUtil.createHardLink(), so we call it once per directory instead of once per file
> --------------------------------------------------------------------------------------------------------------------------------------------------
>
> Key: HADOOP-7133
> URL: https://issues.apache.org/jira/browse/HADOOP-7133
> Project: Hadoop Common
> Issue Type: Improvement
> Components: util
> Affects Versions: 0.20.2
> Reporter: Matt Foley
> Assignee: Matt Foley
> Fix For: 0.23.0
>
> Attachments: HDFS-1445-trunk.v23_common_1-of-3.patch
>
>
> The fix for HDFS-1445 "Batch the calls in DataStorage to FileUtil.createHardLink(), so we call it once per directory instead of once per file" requires coordinated change in COMMON and HDFS. This is the COMMON portion, submitted here under a separate bug to activate the automated testing.
> Warning: this patch to COMMON, by itself, will break HDFS. It requires coordinated commit of the HDFS portion of the patch in HDFS-1445.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] Updated: (HADOOP-7133) CLONE to COMMON - HDFS-1445 Batch the
calls in DataStorage to FileUtil.createHardLink(), so we call it once per
directory instead of once per file
Posted by "Matt Foley (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-7133?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Matt Foley updated HADOOP-7133:
-------------------------------
Attachment: (was: HDFS-1445-trunk.v22_common_1-of-2.patch)
> CLONE to COMMON - HDFS-1445 Batch the calls in DataStorage to FileUtil.createHardLink(), so we call it once per directory instead of once per file
> --------------------------------------------------------------------------------------------------------------------------------------------------
>
> Key: HADOOP-7133
> URL: https://issues.apache.org/jira/browse/HADOOP-7133
> Project: Hadoop Common
> Issue Type: Improvement
> Components: util
> Affects Versions: 0.20.2
> Reporter: Matt Foley
> Assignee: Matt Foley
> Fix For: 0.22.0
>
> Attachments: HDFS-1445-trunk.v23_common_1-of-3.patch
>
>
> The fix for HDFS-1445 "Batch the calls in DataStorage to FileUtil.createHardLink(), so we call it once per directory instead of once per file" requires coordinated change in COMMON and HDFS. This is the COMMON portion, submitted here under a separate bug to activate the automated testing.
> Warning: this patch to COMMON, by itself, will break HDFS. It requires coordinated commit of the HDFS portion of the patch in HDFS-1445.
--
This message is automatically generated by JIRA.
-
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] Commented: (HADOOP-7133) CLONE to COMMON - HDFS-1445 Batch
the calls in DataStorage to FileUtil.createHardLink(), so we call it once
per directory instead of once per file
Posted by "Jakob Homan (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-7133?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13004298#comment-13004298 ]
Jakob Homan commented on HADOOP-7133:
-------------------------------------
+1 on this patch, however, even with the backwards compatibility now provided, I'd like to see the HDFS counterpart and commit them both at once. Once that has been posted and reviewed, I'll go ahead and commit both.
> CLONE to COMMON - HDFS-1445 Batch the calls in DataStorage to FileUtil.createHardLink(), so we call it once per directory instead of once per file
> --------------------------------------------------------------------------------------------------------------------------------------------------
>
> Key: HADOOP-7133
> URL: https://issues.apache.org/jira/browse/HADOOP-7133
> Project: Hadoop Common
> Issue Type: Improvement
> Components: util
> Affects Versions: 0.20.2
> Reporter: Matt Foley
> Assignee: Matt Foley
> Fix For: 0.22.0
>
> Attachments: HDFS-1445-trunk.v23_common_1-of-3.patch
>
>
> The fix for HDFS-1445 "Batch the calls in DataStorage to FileUtil.createHardLink(), so we call it once per directory instead of once per file" requires coordinated change in COMMON and HDFS. This is the COMMON portion, submitted here under a separate bug to activate the automated testing.
> Warning: this patch to COMMON, by itself, will break HDFS. It requires coordinated commit of the HDFS portion of the patch in HDFS-1445.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] Commented: (HADOOP-7133) CLONE to COMMON - HDFS-1445 Batch
the calls in DataStorage to FileUtil.createHardLink(), so we call it once
per directory instead of once per file
Posted by "Matt Foley (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-7133?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13004945#comment-13004945 ]
Matt Foley commented on HADOOP-7133:
------------------------------------
Hi Hairong, glad it's working well for you. Your suggestion to use ThreadLocal buffers is perfect. It will take me a little time to implement and test, though, so I'd like to go ahead with Jakob's suggestion, if you don't mind, so we can proceed with review of the second part of the patch in HDFS-1445.
> CLONE to COMMON - HDFS-1445 Batch the calls in DataStorage to FileUtil.createHardLink(), so we call it once per directory instead of once per file
> --------------------------------------------------------------------------------------------------------------------------------------------------
>
> Key: HADOOP-7133
> URL: https://issues.apache.org/jira/browse/HADOOP-7133
> Project: Hadoop Common
> Issue Type: Improvement
> Components: util
> Affects Versions: 0.20.2
> Reporter: Matt Foley
> Assignee: Matt Foley
> Fix For: 0.22.0
>
> Attachments: HDFS-1445-trunk.v23_common_1-of-3.patch
>
>
> The fix for HDFS-1445 "Batch the calls in DataStorage to FileUtil.createHardLink(), so we call it once per directory instead of once per file" requires coordinated change in COMMON and HDFS. This is the COMMON portion, submitted here under a separate bug to activate the automated testing.
> Warning: this patch to COMMON, by itself, will break HDFS. It requires coordinated commit of the HDFS portion of the patch in HDFS-1445.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] Commented: (HADOOP-7133) CLONE to COMMON - HDFS-1445 Batch
the calls in DataStorage to FileUtil.createHardLink(), so we call it once
per directory instead of once per file
Posted by "Hadoop QA (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-7133?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12994168#comment-12994168 ]
Hadoop QA commented on HADOOP-7133:
-----------------------------------
+1 overall. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12470703/HDFS-1445-trunk.v22_common_1-of-2.patch
against trunk revision 1070021.
+1 @author. The patch does not contain any @author tags.
+1 tests included. The patch appears to include 5 new or modified tests.
+1 javadoc. The javadoc tool did not generate any warning messages.
+1 javac. The applied patch does not increase the total number of javac compiler warnings.
+1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings.
+1 release audit. The applied patch does not increase the total number of release audit warnings.
+1 core tests. The patch passed core unit tests.
+1 contrib tests. The patch passed contrib unit tests.
+1 system test framework. The patch passed system test framework compile.
Test results: https://hudson.apache.org/hudson/job/PreCommit-HADOOP-Build/231//testReport/
Findbugs warnings: https://hudson.apache.org/hudson/job/PreCommit-HADOOP-Build/231//artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
Console output: https://hudson.apache.org/hudson/job/PreCommit-HADOOP-Build/231//console
This message is automatically generated.
> CLONE to COMMON - HDFS-1445 Batch the calls in DataStorage to FileUtil.createHardLink(), so we call it once per directory instead of once per file
> --------------------------------------------------------------------------------------------------------------------------------------------------
>
> Key: HADOOP-7133
> URL: https://issues.apache.org/jira/browse/HADOOP-7133
> Project: Hadoop Common
> Issue Type: Improvement
> Components: util
> Affects Versions: 0.20.2
> Reporter: Matt Foley
> Assignee: Matt Foley
> Fix For: 0.22.0
>
> Attachments: HDFS-1445-trunk.v22_common_1-of-2.patch
>
>
> The fix for HDFS-1445 "Batch the calls in DataStorage to FileUtil.createHardLink(), so we call it once per directory instead of once per file" requires coordinated change in COMMON and HDFS. This is the COMMON portion, submitted here under a separate bug to activate the automated testing.
> Warning: this patch to COMMON, by itself, will break HDFS. It requires coordinated commit of the HDFS portion of the patch in HDFS-1445.
--
This message is automatically generated by JIRA.
-
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] Commented: (HADOOP-7133) CLONE to COMMON - HDFS-1445 Batch
the calls in DataStorage to FileUtil.createHardLink(), so we call it once
per directory instead of once per file
Posted by "Hadoop QA (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-7133?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12996861#comment-12996861 ]
Hadoop QA commented on HADOOP-7133:
-----------------------------------
+1 overall. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12471347/HDFS-1445-trunk.v23_common_1-of-3.patch
against trunk revision 1071364.
+1 @author. The patch does not contain any @author tags.
+1 tests included. The patch appears to include 6 new or modified tests.
+1 javadoc. The javadoc tool did not generate any warning messages.
+1 javac. The applied patch does not increase the total number of javac compiler warnings.
+1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings.
+1 release audit. The applied patch does not increase the total number of release audit warnings.
+1 core tests. The patch passed core unit tests.
+1 contrib tests. The patch passed contrib unit tests.
+1 system test framework. The patch passed system test framework compile.
Test results: https://hudson.apache.org/hudson/job/PreCommit-HADOOP-Build/243//testReport/
Findbugs warnings: https://hudson.apache.org/hudson/job/PreCommit-HADOOP-Build/243//artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
Console output: https://hudson.apache.org/hudson/job/PreCommit-HADOOP-Build/243//console
This message is automatically generated.
> CLONE to COMMON - HDFS-1445 Batch the calls in DataStorage to FileUtil.createHardLink(), so we call it once per directory instead of once per file
> --------------------------------------------------------------------------------------------------------------------------------------------------
>
> Key: HADOOP-7133
> URL: https://issues.apache.org/jira/browse/HADOOP-7133
> Project: Hadoop Common
> Issue Type: Improvement
> Components: util
> Affects Versions: 0.20.2
> Reporter: Matt Foley
> Assignee: Matt Foley
> Fix For: 0.22.0
>
> Attachments: HDFS-1445-trunk.v23_common_1-of-3.patch
>
>
> The fix for HDFS-1445 "Batch the calls in DataStorage to FileUtil.createHardLink(), so we call it once per directory instead of once per file" requires coordinated change in COMMON and HDFS. This is the COMMON portion, submitted here under a separate bug to activate the automated testing.
> Warning: this patch to COMMON, by itself, will break HDFS. It requires coordinated commit of the HDFS portion of the patch in HDFS-1445.
--
This message is automatically generated by JIRA.
-
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] Commented: (HADOOP-7133) CLONE to COMMON - HDFS-1445 Batch
the calls in DataStorage to FileUtil.createHardLink(), so we call it once
per directory instead of once per file
Posted by "Matt Foley (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-7133?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13004338#comment-13004338 ]
Matt Foley commented on HADOOP-7133:
------------------------------------
Yes. s/resubmit/push the "submit patch" button again for/
> CLONE to COMMON - HDFS-1445 Batch the calls in DataStorage to FileUtil.createHardLink(), so we call it once per directory instead of once per file
> --------------------------------------------------------------------------------------------------------------------------------------------------
>
> Key: HADOOP-7133
> URL: https://issues.apache.org/jira/browse/HADOOP-7133
> Project: Hadoop Common
> Issue Type: Improvement
> Components: util
> Affects Versions: 0.20.2
> Reporter: Matt Foley
> Assignee: Matt Foley
> Fix For: 0.22.0
>
> Attachments: HDFS-1445-trunk.v23_common_1-of-3.patch
>
>
> The fix for HDFS-1445 "Batch the calls in DataStorage to FileUtil.createHardLink(), so we call it once per directory instead of once per file" requires coordinated change in COMMON and HDFS. This is the COMMON portion, submitted here under a separate bug to activate the automated testing.
> Warning: this patch to COMMON, by itself, will break HDFS. It requires coordinated commit of the HDFS portion of the patch in HDFS-1445.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] Commented: (HADOOP-7133) CLONE to COMMON - HDFS-1445 Batch
the calls in DataStorage to FileUtil.createHardLink(), so we call it once
per directory instead of once per file
Posted by "Hadoop QA (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-7133?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12992663#comment-12992663 ]
Hadoop QA commented on HADOOP-7133:
-----------------------------------
+1 overall. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12470703/HDFS-1445-trunk.v22_common_1-of-2.patch
against trunk revision 1068729.
+1 @author. The patch does not contain any @author tags.
+1 tests included. The patch appears to include 5 new or modified tests.
+1 javadoc. The javadoc tool did not generate any warning messages.
+1 javac. The applied patch does not increase the total number of javac compiler warnings.
+1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings.
+1 release audit. The applied patch does not increase the total number of release audit warnings.
+1 core tests. The patch passed core unit tests.
+1 contrib tests. The patch passed contrib unit tests.
+1 system test framework. The patch passed system test framework compile.
Test results: https://hudson.apache.org/hudson/job/PreCommit-HADOOP-Build/224//testReport/
Findbugs warnings: https://hudson.apache.org/hudson/job/PreCommit-HADOOP-Build/224//artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
Console output: https://hudson.apache.org/hudson/job/PreCommit-HADOOP-Build/224//console
This message is automatically generated.
> CLONE to COMMON - HDFS-1445 Batch the calls in DataStorage to FileUtil.createHardLink(), so we call it once per directory instead of once per file
> --------------------------------------------------------------------------------------------------------------------------------------------------
>
> Key: HADOOP-7133
> URL: https://issues.apache.org/jira/browse/HADOOP-7133
> Project: Hadoop Common
> Issue Type: Improvement
> Components: util
> Affects Versions: 0.20.2
> Reporter: Matt Foley
> Assignee: Matt Foley
> Fix For: 0.22.0
>
> Attachments: HDFS-1445-trunk.v22_common_1-of-2.patch
>
>
> The fix for HDFS-1445 "Batch the calls in DataStorage to FileUtil.createHardLink(), so we call it once per directory instead of once per file" requires coordinated change in COMMON and HDFS. This is the COMMON portion, submitted here under a separate bug to activate the automated testing.
> Warning: this patch to COMMON, by itself, will break HDFS. It requires coordinated commit of the HDFS portion of the patch in HDFS-1445.
--
This message is automatically generated by JIRA.
-
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] Commented: (HADOOP-7133) CLONE to COMMON - HDFS-1445 Batch
the calls in DataStorage to FileUtil.createHardLink(), so we call it once
per directory instead of once per file
Posted by "Matt Foley (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-7133?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12992666#comment-12992666 ]
Matt Foley commented on HADOOP-7133:
------------------------------------
Hi Todd, that's painful but workable. I'll submit a revised patch shortly. Thanks for the guidance.
> CLONE to COMMON - HDFS-1445 Batch the calls in DataStorage to FileUtil.createHardLink(), so we call it once per directory instead of once per file
> --------------------------------------------------------------------------------------------------------------------------------------------------
>
> Key: HADOOP-7133
> URL: https://issues.apache.org/jira/browse/HADOOP-7133
> Project: Hadoop Common
> Issue Type: Improvement
> Components: util
> Affects Versions: 0.20.2
> Reporter: Matt Foley
> Assignee: Matt Foley
> Fix For: 0.22.0
>
> Attachments: HDFS-1445-trunk.v22_common_1-of-2.patch
>
>
> The fix for HDFS-1445 "Batch the calls in DataStorage to FileUtil.createHardLink(), so we call it once per directory instead of once per file" requires coordinated change in COMMON and HDFS. This is the COMMON portion, submitted here under a separate bug to activate the automated testing.
> Warning: this patch to COMMON, by itself, will break HDFS. It requires coordinated commit of the HDFS portion of the patch in HDFS-1445.
--
This message is automatically generated by JIRA.
-
For more information on JIRA, see: http://www.atlassian.com/software/jira