You are viewing a plain text version of this content. The canonical link for it is here.
Posted to common-dev@hadoop.apache.org by "dhruba borthakur (JIRA)" <ji...@apache.org> on 2009/05/01 03:15:30 UTC
[jira] Created: (HADOOP-5762) distcp does not copy empty
directories
distcp does not copy empty directories
--------------------------------------
Key: HADOOP-5762
URL: https://issues.apache.org/jira/browse/HADOOP-5762
Project: Hadoop Core
Issue Type: Bug
Components: tools/distcp
Reporter: dhruba borthakur
Assignee: Rodrigo Schmidt
If I have an empty directory /testdir1 and then I run the command bin/hadoop distcp /testdir1 /testdir2, the command completes successfully, but does not create the empty directory /testdir2.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-5762) distcp does not copy empty
directories
Posted by "Rodrigo Schmidt (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-5762?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Rodrigo Schmidt updated HADOOP-5762:
------------------------------------
Attachment: HADOOP-5762.patch
Correct patch.
> distcp does not copy empty directories
> --------------------------------------
>
> Key: HADOOP-5762
> URL: https://issues.apache.org/jira/browse/HADOOP-5762
> Project: Hadoop Core
> Issue Type: Bug
> Components: tools/distcp
> Reporter: dhruba borthakur
> Assignee: Rodrigo Schmidt
> Fix For: site
>
> Attachments: HADOOP-5762.patch
>
>
> If I have an empty directory /testdir1 and then I run the command bin/hadoop distcp /testdir1 /testdir2, the command completes successfully, but does not create the empty directory /testdir2.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-5762) distcp does not copy empty
directories
Posted by "Tsz Wo (Nicholas), SZE (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-5762?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Tsz Wo (Nicholas), SZE updated HADOOP-5762:
-------------------------------------------
Hadoop Flags: [Reviewed]
Status: Patch Available (was: Open)
+1 The latest patch looks good.
> distcp does not copy empty directories
> --------------------------------------
>
> Key: HADOOP-5762
> URL: https://issues.apache.org/jira/browse/HADOOP-5762
> Project: Hadoop Core
> Issue Type: Bug
> Components: tools/distcp
> Reporter: dhruba borthakur
> Assignee: Rodrigo Schmidt
> Fix For: 0.21.0
>
> Attachments: HADOOP-5762.2.patch, HADOOP-5762.patch
>
>
> If I have an empty directory /testdir1 and then I run the command bin/hadoop distcp /testdir1 /testdir2, the command completes successfully, but does not create the empty directory /testdir2.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-5762) distcp does not copy empty
directories
Posted by "Rodrigo Schmidt (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-5762?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12707202#action_12707202 ]
Rodrigo Schmidt commented on HADOOP-5762:
-----------------------------------------
The problem still exists on trunk.
The patch was generated based on trunk.
> distcp does not copy empty directories
> --------------------------------------
>
> Key: HADOOP-5762
> URL: https://issues.apache.org/jira/browse/HADOOP-5762
> Project: Hadoop Core
> Issue Type: Bug
> Components: tools/distcp
> Reporter: dhruba borthakur
> Assignee: Rodrigo Schmidt
> Fix For: site
>
> Attachments: HADOOP-5762.patch
>
>
> If I have an empty directory /testdir1 and then I run the command bin/hadoop distcp /testdir1 /testdir2, the command completes successfully, but does not create the empty directory /testdir2.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Issue Comment Edited: (HADOOP-5762) distcp does not copy
empty directories
Posted by "Rodrigo Schmidt (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-5762?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12707152#action_12707152 ]
Rodrigo Schmidt edited comment on HADOOP-5762 at 5/7/09 6:08 PM:
-----------------------------------------------------------------
The problem happens if any of the source paths is an empty directory. It's because source directories are not "visited" in the inner loop that checks for files and directories to be copied and, thus, are not added to _distcp_src_files.
was (Author: rschmidt):
The problem happened if any of the source paths was an empty directory. This happens because source directories are not "visited" in the inner loop that checks for files and directories to be copied and, thus, it was not added to _distcp_src_files.
> distcp does not copy empty directories
> --------------------------------------
>
> Key: HADOOP-5762
> URL: https://issues.apache.org/jira/browse/HADOOP-5762
> Project: Hadoop Core
> Issue Type: Bug
> Components: tools/distcp
> Reporter: dhruba borthakur
> Assignee: Rodrigo Schmidt
> Fix For: site
>
> Attachments: HADOOP-5762.patch
>
>
> If I have an empty directory /testdir1 and then I run the command bin/hadoop distcp /testdir1 /testdir2, the command completes successfully, but does not create the empty directory /testdir2.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-5762) distcp does not copy empty
directories
Posted by "Tsz Wo (Nicholas), SZE (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-5762?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12707170#action_12707170 ]
Tsz Wo (Nicholas), SZE commented on HADOOP-5762:
------------------------------------------------
Hi Rodrigo, which version is your patch for? Does this problem still exist on trunk?
> distcp does not copy empty directories
> --------------------------------------
>
> Key: HADOOP-5762
> URL: https://issues.apache.org/jira/browse/HADOOP-5762
> Project: Hadoop Core
> Issue Type: Bug
> Components: tools/distcp
> Reporter: dhruba borthakur
> Assignee: Rodrigo Schmidt
> Fix For: site
>
> Attachments: HADOOP-5762.patch
>
>
> If I have an empty directory /testdir1 and then I run the command bin/hadoop distcp /testdir1 /testdir2, the command completes successfully, but does not create the empty directory /testdir2.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-5762) distcp does not copy empty
directories
Posted by "Tsz Wo (Nicholas), SZE (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-5762?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Tsz Wo (Nicholas), SZE updated HADOOP-5762:
-------------------------------------------
Status: Open (was: Patch Available)
> distcp does not copy empty directories
> --------------------------------------
>
> Key: HADOOP-5762
> URL: https://issues.apache.org/jira/browse/HADOOP-5762
> Project: Hadoop Core
> Issue Type: Bug
> Components: tools/distcp
> Reporter: dhruba borthakur
> Assignee: Rodrigo Schmidt
> Fix For: 0.21.0
>
> Attachments: HADOOP-5762.2.patch, HADOOP-5762.patch
>
>
> If I have an empty directory /testdir1 and then I run the command bin/hadoop distcp /testdir1 /testdir2, the command completes successfully, but does not create the empty directory /testdir2.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-5762) distcp does not copy empty
directories
Posted by "Rodrigo Schmidt (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-5762?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Rodrigo Schmidt updated HADOOP-5762:
------------------------------------
Status: Open (was: Patch Available)
> distcp does not copy empty directories
> --------------------------------------
>
> Key: HADOOP-5762
> URL: https://issues.apache.org/jira/browse/HADOOP-5762
> Project: Hadoop Core
> Issue Type: Bug
> Components: tools/distcp
> Reporter: dhruba borthakur
> Assignee: Rodrigo Schmidt
> Fix For: site
>
>
> If I have an empty directory /testdir1 and then I run the command bin/hadoop distcp /testdir1 /testdir2, the command completes successfully, but does not create the empty directory /testdir2.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-5762) distcp does not copy empty
directories
Posted by "Rodrigo Schmidt (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-5762?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12707724#action_12707724 ]
Rodrigo Schmidt commented on HADOOP-5762:
-----------------------------------------
Just realized I had uploaded the wrong patch. I'm correcting it now.
> distcp does not copy empty directories
> --------------------------------------
>
> Key: HADOOP-5762
> URL: https://issues.apache.org/jira/browse/HADOOP-5762
> Project: Hadoop Core
> Issue Type: Bug
> Components: tools/distcp
> Reporter: dhruba borthakur
> Assignee: Rodrigo Schmidt
> Fix For: site
>
>
> If I have an empty directory /testdir1 and then I run the command bin/hadoop distcp /testdir1 /testdir2, the command completes successfully, but does not create the empty directory /testdir2.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-5762) distcp does not copy empty
directories
Posted by "Tsz Wo (Nicholas), SZE (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-5762?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12708191#action_12708191 ]
Tsz Wo (Nicholas), SZE commented on HADOOP-5762:
------------------------------------------------
{code}
+ return srcCount > 0;
{code}
Checking srcCount may not work well since srcCount counts all src paths. It will be >0 even there is nothing to copy.
We probably need a new varible dirCount and then return fileCount > 0 || dirCount > 0.
> distcp does not copy empty directories
> --------------------------------------
>
> Key: HADOOP-5762
> URL: https://issues.apache.org/jira/browse/HADOOP-5762
> Project: Hadoop Core
> Issue Type: Bug
> Components: tools/distcp
> Reporter: dhruba borthakur
> Assignee: Rodrigo Schmidt
> Fix For: site
>
> Attachments: HADOOP-5762.patch
>
>
> If I have an empty directory /testdir1 and then I run the command bin/hadoop distcp /testdir1 /testdir2, the command completes successfully, but does not create the empty directory /testdir2.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-5762) distcp does not copy empty
directories
Posted by "Rodrigo Schmidt (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-5762?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12708237#action_12708237 ]
Rodrigo Schmidt commented on HADOOP-5762:
-----------------------------------------
It could be done. Actually, my first attempt to fix this bug did exactly that. Then I noticed that method copy() was already programmed to copy empty directories and, except for the root source, directories were being added to src_writer on method setup. I think it's a good design option to let copy be the only method that actually writes something to the destination. This makes the code simpler and more elegant (thus, more maintainable).
I'm don't like the idea of creating only the root src directory (in case it's empty) on setup(). It just doesn't look good. And creating all empty src subdirectories on setup might force us to do a lot of extra checking on setup to cope with failures (checks that already exist on copy()). I don't think many people will be using distcp to copy empty directories and it doesn't look like the performance gain will compensate the loss in code simplicity.
> distcp does not copy empty directories
> --------------------------------------
>
> Key: HADOOP-5762
> URL: https://issues.apache.org/jira/browse/HADOOP-5762
> Project: Hadoop Core
> Issue Type: Bug
> Components: tools/distcp
> Reporter: dhruba borthakur
> Assignee: Rodrigo Schmidt
> Fix For: site
>
> Attachments: HADOOP-5762.patch
>
>
> If I have an empty directory /testdir1 and then I run the command bin/hadoop distcp /testdir1 /testdir2, the command completes successfully, but does not create the empty directory /testdir2.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-5762) distcp does not copy empty
directories
Posted by "Hadoop QA (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-5762?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12707742#action_12707742 ]
Hadoop QA commented on HADOOP-5762:
-----------------------------------
-1 overall. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12407713/HADOOP-5762.patch
against trunk revision 772960.
+1 @author. The patch does not contain any @author tags.
+1 tests included. The patch appears to include 3 new or modified tests.
+1 javadoc. The javadoc tool did not generate any warning messages.
+1 javac. The applied patch does not increase the total number of javac compiler warnings.
+1 findbugs. The patch does not introduce any new Findbugs warnings.
+1 Eclipse classpath. The patch retains Eclipse classpath integrity.
+1 release audit. The applied patch does not increase the total number of release audit warnings.
+1 core tests. The patch passed core unit tests.
-1 contrib tests. The patch failed contrib unit tests.
Test results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-vesta.apache.org/319/testReport/
Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-vesta.apache.org/319/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
Checkstyle results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-vesta.apache.org/319/artifact/trunk/build/test/checkstyle-errors.html
Console output: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-vesta.apache.org/319/console
This message is automatically generated.
> distcp does not copy empty directories
> --------------------------------------
>
> Key: HADOOP-5762
> URL: https://issues.apache.org/jira/browse/HADOOP-5762
> Project: Hadoop Core
> Issue Type: Bug
> Components: tools/distcp
> Reporter: dhruba borthakur
> Assignee: Rodrigo Schmidt
> Fix For: site
>
> Attachments: HADOOP-5762.patch
>
>
> If I have an empty directory /testdir1 and then I run the command bin/hadoop distcp /testdir1 /testdir2, the command completes successfully, but does not create the empty directory /testdir2.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-5762) distcp does not copy empty
directories
Posted by "Rodrigo Schmidt (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-5762?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12708204#action_12708204 ]
Rodrigo Schmidt commented on HADOOP-5762:
-----------------------------------------
You are right, Nicholas. What about using fileCount for that and just add empty directories to it?
> distcp does not copy empty directories
> --------------------------------------
>
> Key: HADOOP-5762
> URL: https://issues.apache.org/jira/browse/HADOOP-5762
> Project: Hadoop Core
> Issue Type: Bug
> Components: tools/distcp
> Reporter: dhruba borthakur
> Assignee: Rodrigo Schmidt
> Fix For: site
>
> Attachments: HADOOP-5762.patch
>
>
> If I have an empty directory /testdir1 and then I run the command bin/hadoop distcp /testdir1 /testdir2, the command completes successfully, but does not create the empty directory /testdir2.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-5762) distcp does not copy empty
directories
Posted by "Rodrigo Schmidt (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-5762?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Rodrigo Schmidt updated HADOOP-5762:
------------------------------------
Attachment: HADOOP-5762.patch
> distcp does not copy empty directories
> --------------------------------------
>
> Key: HADOOP-5762
> URL: https://issues.apache.org/jira/browse/HADOOP-5762
> Project: Hadoop Core
> Issue Type: Bug
> Components: tools/distcp
> Reporter: dhruba borthakur
> Assignee: Rodrigo Schmidt
> Fix For: site
>
> Attachments: HADOOP-5762.patch
>
>
> If I have an empty directory /testdir1 and then I run the command bin/hadoop distcp /testdir1 /testdir2, the command completes successfully, but does not create the empty directory /testdir2.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-5762) distcp does not copy empty
directories
Posted by "Tsz Wo (Nicholas), SZE (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-5762?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Tsz Wo (Nicholas), SZE updated HADOOP-5762:
-------------------------------------------
Status: Open (was: Patch Available)
> distcp does not copy empty directories
> --------------------------------------
>
> Key: HADOOP-5762
> URL: https://issues.apache.org/jira/browse/HADOOP-5762
> Project: Hadoop Core
> Issue Type: Bug
> Components: tools/distcp
> Reporter: dhruba borthakur
> Assignee: Rodrigo Schmidt
> Fix For: 0.21.0
>
> Attachments: HADOOP-5762.2.patch, HADOOP-5762.patch
>
>
> If I have an empty directory /testdir1 and then I run the command bin/hadoop distcp /testdir1 /testdir2, the command completes successfully, but does not create the empty directory /testdir2.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-5762) distcp does not copy empty
directories
Posted by "Rodrigo Schmidt (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-5762?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Rodrigo Schmidt updated HADOOP-5762:
------------------------------------
Status: Patch Available (was: Open)
The root directory was not being included in the list of files/directories to be copied.
Besides, copying was aborted if no data was being copied (only directories or empty files). The correct test should be on the number of files being copied.
> distcp does not copy empty directories
> --------------------------------------
>
> Key: HADOOP-5762
> URL: https://issues.apache.org/jira/browse/HADOOP-5762
> Project: Hadoop Core
> Issue Type: Bug
> Components: tools/distcp
> Reporter: dhruba borthakur
> Assignee: Rodrigo Schmidt
> Fix For: site
>
> Attachments: HADOOP-5762.patch
>
>
> If I have an empty directory /testdir1 and then I run the command bin/hadoop distcp /testdir1 /testdir2, the command completes successfully, but does not create the empty directory /testdir2.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-5762) distcp does not copy empty
directories
Posted by "Rodrigo Schmidt (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-5762?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Rodrigo Schmidt updated HADOOP-5762:
------------------------------------
Attachment: HADOOP-5762.2.patch
Uploading a new patch.
> distcp does not copy empty directories
> --------------------------------------
>
> Key: HADOOP-5762
> URL: https://issues.apache.org/jira/browse/HADOOP-5762
> Project: Hadoop Core
> Issue Type: Bug
> Components: tools/distcp
> Reporter: dhruba borthakur
> Assignee: Rodrigo Schmidt
> Fix For: site
>
> Attachments: HADOOP-5762.2.patch, HADOOP-5762.patch
>
>
> If I have an empty directory /testdir1 and then I run the command bin/hadoop distcp /testdir1 /testdir2, the command completes successfully, but does not create the empty directory /testdir2.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-5762) distcp does not copy empty
directories
Posted by "Hadoop QA (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-5762?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12707613#action_12707613 ]
Hadoop QA commented on HADOOP-5762:
-----------------------------------
-1 overall. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12407581/HADOOP-5762.patch
against trunk revision 772960.
+1 @author. The patch does not contain any @author tags.
+1 tests included. The patch appears to include 3 new or modified tests.
-1 patch. The patch command could not apply the patch.
Console output: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-vesta.apache.org/313/console
This message is automatically generated.
> distcp does not copy empty directories
> --------------------------------------
>
> Key: HADOOP-5762
> URL: https://issues.apache.org/jira/browse/HADOOP-5762
> Project: Hadoop Core
> Issue Type: Bug
> Components: tools/distcp
> Reporter: dhruba borthakur
> Assignee: Rodrigo Schmidt
> Fix For: site
>
> Attachments: HADOOP-5762.patch
>
>
> If I have an empty directory /testdir1 and then I run the command bin/hadoop distcp /testdir1 /testdir2, the command completes successfully, but does not create the empty directory /testdir2.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-5762) distcp does not copy empty
directories
Posted by "Rodrigo Schmidt (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-5762?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Rodrigo Schmidt updated HADOOP-5762:
------------------------------------
Status: Patch Available (was: In Progress)
> distcp does not copy empty directories
> --------------------------------------
>
> Key: HADOOP-5762
> URL: https://issues.apache.org/jira/browse/HADOOP-5762
> Project: Hadoop Core
> Issue Type: Bug
> Components: tools/distcp
> Reporter: dhruba borthakur
> Assignee: Rodrigo Schmidt
> Fix For: 0.21.0
>
> Attachments: HADOOP-5762.2.patch, HADOOP-5762.patch
>
>
> If I have an empty directory /testdir1 and then I run the command bin/hadoop distcp /testdir1 /testdir2, the command completes successfully, but does not create the empty directory /testdir2.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-5762) distcp does not copy empty
directories
Posted by "Tsz Wo (Nicholas), SZE (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-5762?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12708209#action_12708209 ]
Tsz Wo (Nicholas), SZE commented on HADOOP-5762:
------------------------------------------------
> What about using fileCount for that and just add empty directories to it?
fileCount is also used for the -filelimit option which does not include dirs.
For creating directories, we don't really need a job. Why don't we make these directories during setup? i.e. mkdir instead of appending them to src_writer/dst_writer. What do you think?
> distcp does not copy empty directories
> --------------------------------------
>
> Key: HADOOP-5762
> URL: https://issues.apache.org/jira/browse/HADOOP-5762
> Project: Hadoop Core
> Issue Type: Bug
> Components: tools/distcp
> Reporter: dhruba borthakur
> Assignee: Rodrigo Schmidt
> Fix For: site
>
> Attachments: HADOOP-5762.patch
>
>
> If I have an empty directory /testdir1 and then I run the command bin/hadoop distcp /testdir1 /testdir2, the command completes successfully, but does not create the empty directory /testdir2.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-5762) distcp does not copy empty
directories
Posted by "Rodrigo Schmidt (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-5762?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Rodrigo Schmidt updated HADOOP-5762:
------------------------------------
Attachment: (was: HADOOP-5762.patch)
> distcp does not copy empty directories
> --------------------------------------
>
> Key: HADOOP-5762
> URL: https://issues.apache.org/jira/browse/HADOOP-5762
> Project: Hadoop Core
> Issue Type: Bug
> Components: tools/distcp
> Reporter: dhruba borthakur
> Assignee: Rodrigo Schmidt
> Fix For: site
>
>
> If I have an empty directory /testdir1 and then I run the command bin/hadoop distcp /testdir1 /testdir2, the command completes successfully, but does not create the empty directory /testdir2.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-5762) distcp does not copy empty
directories
Posted by "Rodrigo Schmidt (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-5762?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Rodrigo Schmidt updated HADOOP-5762:
------------------------------------
Status: In Progress (was: Patch Available)
> distcp does not copy empty directories
> --------------------------------------
>
> Key: HADOOP-5762
> URL: https://issues.apache.org/jira/browse/HADOOP-5762
> Project: Hadoop Core
> Issue Type: Bug
> Components: tools/distcp
> Reporter: dhruba borthakur
> Assignee: Rodrigo Schmidt
> Fix For: site
>
> Attachments: HADOOP-5762.patch
>
>
> If I have an empty directory /testdir1 and then I run the command bin/hadoop distcp /testdir1 /testdir2, the command completes successfully, but does not create the empty directory /testdir2.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-5762) distcp does not copy empty
directories
Posted by "Tsz Wo (Nicholas), SZE (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-5762?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Tsz Wo (Nicholas), SZE updated HADOOP-5762:
-------------------------------------------
Resolution: Fixed
Status: Resolved (was: Patch Available)
I have committed this. Thanks, Rodrigo!
> distcp does not copy empty directories
> --------------------------------------
>
> Key: HADOOP-5762
> URL: https://issues.apache.org/jira/browse/HADOOP-5762
> Project: Hadoop Core
> Issue Type: Bug
> Components: tools/distcp
> Reporter: dhruba borthakur
> Assignee: Rodrigo Schmidt
> Fix For: 0.21.0
>
> Attachments: HADOOP-5762.2.patch, HADOOP-5762.patch
>
>
> If I have an empty directory /testdir1 and then I run the command bin/hadoop distcp /testdir1 /testdir2, the command completes successfully, but does not create the empty directory /testdir2.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-5762) distcp does not copy empty
directories
Posted by "Hadoop QA (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-5762?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12715789#action_12715789 ]
Hadoop QA commented on HADOOP-5762:
-----------------------------------
-1 overall. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12407940/HADOOP-5762.2.patch
against trunk revision 781178.
+1 @author. The patch does not contain any @author tags.
+1 tests included. The patch appears to include 3 new or modified tests.
+1 javadoc. The javadoc tool did not generate any warning messages.
+1 javac. The applied patch does not increase the total number of javac compiler warnings.
+1 findbugs. The patch does not introduce any new Findbugs warnings.
+1 Eclipse classpath. The patch retains Eclipse classpath integrity.
+1 release audit. The applied patch does not increase the total number of release audit warnings.
-1 core tests. The patch failed core unit tests.
-1 contrib tests. The patch failed contrib unit tests.
Test results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-vesta.apache.org/452/testReport/
Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-vesta.apache.org/452/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
Checkstyle results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-vesta.apache.org/452/artifact/trunk/build/test/checkstyle-errors.html
Console output: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-vesta.apache.org/452/console
This message is automatically generated.
> distcp does not copy empty directories
> --------------------------------------
>
> Key: HADOOP-5762
> URL: https://issues.apache.org/jira/browse/HADOOP-5762
> Project: Hadoop Core
> Issue Type: Bug
> Components: tools/distcp
> Reporter: dhruba borthakur
> Assignee: Rodrigo Schmidt
> Fix For: 0.21.0
>
> Attachments: HADOOP-5762.2.patch, HADOOP-5762.patch
>
>
> If I have an empty directory /testdir1 and then I run the command bin/hadoop distcp /testdir1 /testdir2, the command completes successfully, but does not create the empty directory /testdir2.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-5762) distcp does not copy empty
directories
Posted by "Rodrigo Schmidt (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-5762?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Rodrigo Schmidt updated HADOOP-5762:
------------------------------------
Fix Version/s: site
Status: Patch Available (was: Open)
The problem happened if any of the source paths was an empty directory. This happens because source directories are not "visited" in the inner loop that checks for files and directories to be copied and, thus, it was not added to _distcp_src_files.
> distcp does not copy empty directories
> --------------------------------------
>
> Key: HADOOP-5762
> URL: https://issues.apache.org/jira/browse/HADOOP-5762
> Project: Hadoop Core
> Issue Type: Bug
> Components: tools/distcp
> Reporter: dhruba borthakur
> Assignee: Rodrigo Schmidt
> Fix For: site
>
>
> If I have an empty directory /testdir1 and then I run the command bin/hadoop distcp /testdir1 /testdir2, the command completes successfully, but does not create the empty directory /testdir2.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-5762) distcp does not copy empty
directories
Posted by "Tsz Wo (Nicholas), SZE (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-5762?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Tsz Wo (Nicholas), SZE updated HADOOP-5762:
-------------------------------------------
Status: Patch Available (was: Open)
> distcp does not copy empty directories
> --------------------------------------
>
> Key: HADOOP-5762
> URL: https://issues.apache.org/jira/browse/HADOOP-5762
> Project: Hadoop Core
> Issue Type: Bug
> Components: tools/distcp
> Reporter: dhruba borthakur
> Assignee: Rodrigo Schmidt
> Fix For: 0.21.0
>
> Attachments: HADOOP-5762.2.patch, HADOOP-5762.patch
>
>
> If I have an empty directory /testdir1 and then I run the command bin/hadoop distcp /testdir1 /testdir2, the command completes successfully, but does not create the empty directory /testdir2.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-5762) distcp does not copy empty
directories
Posted by "Tsz Wo (Nicholas), SZE (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-5762?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12716080#action_12716080 ]
Tsz Wo (Nicholas), SZE commented on HADOOP-5762:
------------------------------------------------
The failed tests do not seem related. I will commit this soon.
> distcp does not copy empty directories
> --------------------------------------
>
> Key: HADOOP-5762
> URL: https://issues.apache.org/jira/browse/HADOOP-5762
> Project: Hadoop Core
> Issue Type: Bug
> Components: tools/distcp
> Reporter: dhruba borthakur
> Assignee: Rodrigo Schmidt
> Fix For: 0.21.0
>
> Attachments: HADOOP-5762.2.patch, HADOOP-5762.patch
>
>
> If I have an empty directory /testdir1 and then I run the command bin/hadoop distcp /testdir1 /testdir2, the command completes successfully, but does not create the empty directory /testdir2.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-5762) distcp does not copy empty
directories
Posted by "Johan Oskarsson (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-5762?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Johan Oskarsson updated HADOOP-5762:
------------------------------------
Fix Version/s: (was: site)
0.21.0
> distcp does not copy empty directories
> --------------------------------------
>
> Key: HADOOP-5762
> URL: https://issues.apache.org/jira/browse/HADOOP-5762
> Project: Hadoop Core
> Issue Type: Bug
> Components: tools/distcp
> Reporter: dhruba borthakur
> Assignee: Rodrigo Schmidt
> Fix For: 0.21.0
>
> Attachments: HADOOP-5762.2.patch, HADOOP-5762.patch
>
>
> If I have an empty directory /testdir1 and then I run the command bin/hadoop distcp /testdir1 /testdir2, the command completes successfully, but does not create the empty directory /testdir2.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.