You are viewing a plain text version of this content. The canonical link for it is here.
Posted to hdfs-issues@hadoop.apache.org by "Daryn Sharp (JIRA)" <ji...@apache.org> on 2011/04/13 20:41:06 UTC

[jira] [Created] (HDFS-1834) Accelerate chmod task in ant files

Accelerate chmod task in ant files
----------------------------------

                 Key: HDFS-1834
                 URL: https://issues.apache.org/jira/browse/HDFS-1834
             Project: Hadoop HDFS
          Issue Type: Improvement
          Components: build
    Affects Versions: 0.23.0
            Reporter: Daryn Sharp
            Assignee: Daryn Sharp


The chmod tasks have a parallel="false" attribute attached to them.  This causes ant to shell out for each and every item in the fileset which slows down parts of the build.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (HDFS-1834) Accelerate chmod task in ant files

Posted by "Daryn Sharp (JIRA)" <ji...@apache.org>.
     [ https://issues.apache.org/jira/browse/HDFS-1834?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Daryn Sharp updated HDFS-1834:
------------------------------

    Attachment: HDFS-1834.patch

Add prop for max number of paths to chmod per call.  Defaults to the system limit for mac/linux.

> Accelerate chmod task in ant files
> ----------------------------------
>
>                 Key: HDFS-1834
>                 URL: https://issues.apache.org/jira/browse/HDFS-1834
>             Project: Hadoop HDFS
>          Issue Type: Improvement
>          Components: build
>    Affects Versions: 0.23.0
>            Reporter: Daryn Sharp
>            Assignee: Daryn Sharp
>         Attachments: HDFS-1834.patch
>
>
> The chmod tasks have a parallel="false" attribute attached to them.  This causes ant to shell out for each and every item in the fileset which slows down parts of the build.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HDFS-1834) Accelerate chmod task in ant files

Posted by "Daryn Sharp (JIRA)" <ji...@apache.org>.
    [ https://issues.apache.org/jira/browse/HDFS-1834?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13071313#comment-13071313 ] 

Daryn Sharp commented on HDFS-1834:
-----------------------------------

For some reason ant uses the shell command chmod instead of the libc chmod...  Also from the apache ant manual:

bq. By default this task will use a single invocation of the underlying chmod command. If you are working on a large number of files this may result in a command line that is too long for your operating system. If you encounter such problems, you should set the maxparallel attribute of this task to a non-zero value. The number to use highly depends on the length of your file names (the depth of your directory tree) and your operating system, so you'll have to experiment a little. POSIX recommends command line length limits of at least 4096 characters, this may give you an approximation for the number you could use as initial value for these experiments.

Would you be willing to please post the updated patch to save me the duplicate effort?

> Accelerate chmod task in ant files
> ----------------------------------
>
>                 Key: HDFS-1834
>                 URL: https://issues.apache.org/jira/browse/HDFS-1834
>             Project: Hadoop HDFS
>          Issue Type: Improvement
>          Components: build
>    Affects Versions: 0.23.0
>            Reporter: Daryn Sharp
>            Assignee: Daryn Sharp
>         Attachments: HDFS-1834.patch
>
>
> The chmod tasks have a parallel="false" attribute attached to them.  This causes ant to shell out for each and every item in the fileset which slows down parts of the build.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

[jira] [Updated] (HDFS-1834) Accelerate chmod task in ant files

Posted by "Giridharan Kesavan (JIRA)" <ji...@apache.org>.
     [ https://issues.apache.org/jira/browse/HDFS-1834?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Giridharan Kesavan updated HDFS-1834:
-------------------------------------

    Hadoop Flags: [Reviewed]
          Status: Patch Available  (was: Open)

> Accelerate chmod task in ant files
> ----------------------------------
>
>                 Key: HDFS-1834
>                 URL: https://issues.apache.org/jira/browse/HDFS-1834
>             Project: Hadoop HDFS
>          Issue Type: Improvement
>          Components: build
>    Affects Versions: 0.23.0
>            Reporter: Daryn Sharp
>            Assignee: Daryn Sharp
>         Attachments: HDFS-1834-v1.patch, HDFS-1834.patch
>
>
> The chmod tasks have a parallel="false" attribute attached to them.  This causes ant to shell out for each and every item in the fileset which slows down parts of the build.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

[jira] [Updated] (HDFS-1834) Accelerate chmod task in ant files

Posted by "Giridharan Kesavan (JIRA)" <ji...@apache.org>.
     [ https://issues.apache.org/jira/browse/HDFS-1834?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Giridharan Kesavan updated HDFS-1834:
-------------------------------------

    Attachment: HDFS-1834-v1.patch

v1.patch is updated to work with the trunk code.

> Accelerate chmod task in ant files
> ----------------------------------
>
>                 Key: HDFS-1834
>                 URL: https://issues.apache.org/jira/browse/HDFS-1834
>             Project: Hadoop HDFS
>          Issue Type: Improvement
>          Components: build
>    Affects Versions: 0.23.0
>            Reporter: Daryn Sharp
>            Assignee: Daryn Sharp
>         Attachments: HDFS-1834-v1.patch, HDFS-1834.patch
>
>
> The chmod tasks have a parallel="false" attribute attached to them.  This causes ant to shell out for each and every item in the fileset which slows down parts of the build.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

[jira] [Updated] (HDFS-1834) Accelerate chmod task in ant files

Posted by "Daryn Sharp (JIRA)" <ji...@apache.org>.
     [ https://issues.apache.org/jira/browse/HDFS-1834?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Daryn Sharp updated HDFS-1834:
------------------------------

    Status: Patch Available  (was: Open)

> Accelerate chmod task in ant files
> ----------------------------------
>
>                 Key: HDFS-1834
>                 URL: https://issues.apache.org/jira/browse/HDFS-1834
>             Project: Hadoop HDFS
>          Issue Type: Improvement
>          Components: build
>    Affects Versions: 0.23.0
>            Reporter: Daryn Sharp
>            Assignee: Daryn Sharp
>         Attachments: HDFS-1834.patch
>
>
> The chmod tasks have a parallel="false" attribute attached to them.  This causes ant to shell out for each and every item in the fileset which slows down parts of the build.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HDFS-1834) Accelerate chmod task in ant files

Posted by "Hadoop QA (JIRA)" <ji...@apache.org>.
    [ https://issues.apache.org/jira/browse/HDFS-1834?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13071394#comment-13071394 ] 

Hadoop QA commented on HDFS-1834:
---------------------------------

-1 overall.  Here are the results of testing the latest attachment 
  http://issues.apache.org/jira/secure/attachment/12487895/HDFS-1834-v1.patch
  against trunk revision 1151238.

    +1 @author.  The patch does not contain any @author tags.

    -1 tests included.  The patch doesn't appear to include any new or modified tests.
                        Please justify why no new tests are needed for this patch.
                        Also please list what manual steps were performed to verify this patch.

    +1 javadoc.  The javadoc tool did not generate any warning messages.

    +1 javac.  The applied patch does not increase the total number of javac compiler warnings.

    +1 findbugs.  The patch does not introduce any new Findbugs (version 1.3.9) warnings.

    +1 release audit.  The applied patch does not increase the total number of release audit warnings.

    -1 core tests.  The patch failed these core unit tests:
                  org.apache.hadoop.hdfs.TestHDFSTrash

    +1 contrib tests.  The patch passed contrib unit tests.

    +1 system test framework.  The patch passed system test framework compile.

Test results: https://builds.apache.org/job/PreCommit-HDFS-Build/1031//testReport/
Findbugs warnings: https://builds.apache.org/job/PreCommit-HDFS-Build/1031//artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/1031//console

This message is automatically generated.

> Accelerate chmod task in ant files
> ----------------------------------
>
>                 Key: HDFS-1834
>                 URL: https://issues.apache.org/jira/browse/HDFS-1834
>             Project: Hadoop HDFS
>          Issue Type: Improvement
>          Components: build
>    Affects Versions: 0.23.0
>            Reporter: Daryn Sharp
>            Assignee: Daryn Sharp
>         Attachments: HDFS-1834-v1.patch, HDFS-1834.patch
>
>
> The chmod tasks have a parallel="false" attribute attached to them.  This causes ant to shell out for each and every item in the fileset which slows down parts of the build.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

[jira] [Commented] (HDFS-1834) Accelerate chmod task in ant files

Posted by "Giridharan Kesavan (JIRA)" <ji...@apache.org>.
    [ https://issues.apache.org/jira/browse/HDFS-1834?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13071300#comment-13071300 ] 

Giridharan Kesavan commented on HDFS-1834:
------------------------------------------


This patch requires porting to the latest hdfs code base.
I updated the patch locally and tested on my mac. It works fine.

{noformat}
>From apache ant manual:
maxparallel - Limit the amount of parallelism by passing at most this many sourcefiles at once. Set it to <= 0 for unlimited. Defaults to unlimited. Since Ant 1.6.
{noformat}

what is the significance of the number "4096" ?

> Accelerate chmod task in ant files
> ----------------------------------
>
>                 Key: HDFS-1834
>                 URL: https://issues.apache.org/jira/browse/HDFS-1834
>             Project: Hadoop HDFS
>          Issue Type: Improvement
>          Components: build
>    Affects Versions: 0.23.0
>            Reporter: Daryn Sharp
>            Assignee: Daryn Sharp
>         Attachments: HDFS-1834.patch
>
>
> The chmod tasks have a parallel="false" attribute attached to them.  This causes ant to shell out for each and every item in the fileset which slows down parts of the build.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

[jira] [Updated] (HDFS-1834) Accelerate chmod task in ant files

Posted by "Giridharan Kesavan (JIRA)" <ji...@apache.org>.
     [ https://issues.apache.org/jira/browse/HDFS-1834?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Giridharan Kesavan updated HDFS-1834:
-------------------------------------

    Status: Open  (was: Patch Available)

> Accelerate chmod task in ant files
> ----------------------------------
>
>                 Key: HDFS-1834
>                 URL: https://issues.apache.org/jira/browse/HDFS-1834
>             Project: Hadoop HDFS
>          Issue Type: Improvement
>          Components: build
>    Affects Versions: 0.23.0
>            Reporter: Daryn Sharp
>            Assignee: Daryn Sharp
>         Attachments: HDFS-1834-v1.patch, HDFS-1834.patch
>
>
> The chmod tasks have a parallel="false" attribute attached to them.  This causes ant to shell out for each and every item in the fileset which slows down parts of the build.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

[jira] [Commented] (HDFS-1834) Accelerate chmod task in ant files

Posted by "Giridharan Kesavan (JIRA)" <ji...@apache.org>.
    [ https://issues.apache.org/jira/browse/HDFS-1834?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13071414#comment-13071414 ] 

Giridharan Kesavan commented on HDFS-1834:
------------------------------------------

+1 the test failure it un-related to the patch.

> Accelerate chmod task in ant files
> ----------------------------------
>
>                 Key: HDFS-1834
>                 URL: https://issues.apache.org/jira/browse/HDFS-1834
>             Project: Hadoop HDFS
>          Issue Type: Improvement
>          Components: build
>    Affects Versions: 0.23.0
>            Reporter: Daryn Sharp
>            Assignee: Daryn Sharp
>         Attachments: HDFS-1834-v1.patch, HDFS-1834.patch
>
>
> The chmod tasks have a parallel="false" attribute attached to them.  This causes ant to shell out for each and every item in the fileset which slows down parts of the build.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

[jira] [Commented] (HDFS-1834) Accelerate chmod task in ant files

Posted by "Hadoop QA (JIRA)" <ji...@apache.org>.
    [ https://issues.apache.org/jira/browse/HDFS-1834?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13071387#comment-13071387 ] 

Hadoop QA commented on HDFS-1834:
---------------------------------

-1 overall.  Here are the results of testing the latest attachment 
  http://issues.apache.org/jira/secure/attachment/12487895/HDFS-1834-v1.patch
  against trunk revision 1151238.

    +1 @author.  The patch does not contain any @author tags.

    -1 tests included.  The patch doesn't appear to include any new or modified tests.
                        Please justify why no new tests are needed for this patch.
                        Also please list what manual steps were performed to verify this patch.

    +1 javadoc.  The javadoc tool did not generate any warning messages.

    +1 javac.  The applied patch does not increase the total number of javac compiler warnings.

    +1 findbugs.  The patch does not introduce any new Findbugs (version 1.3.9) warnings.

    +1 release audit.  The applied patch does not increase the total number of release audit warnings.

    -1 core tests.  The patch failed these core unit tests:
                  org.apache.hadoop.hdfs.TestHDFSTrash

    +1 contrib tests.  The patch passed contrib unit tests.

    +1 system test framework.  The patch passed system test framework compile.

Test results: https://builds.apache.org/job/PreCommit-HDFS-Build/1030//testReport/
Findbugs warnings: https://builds.apache.org/job/PreCommit-HDFS-Build/1030//artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/1030//console

This message is automatically generated.

> Accelerate chmod task in ant files
> ----------------------------------
>
>                 Key: HDFS-1834
>                 URL: https://issues.apache.org/jira/browse/HDFS-1834
>             Project: Hadoop HDFS
>          Issue Type: Improvement
>          Components: build
>    Affects Versions: 0.23.0
>            Reporter: Daryn Sharp
>            Assignee: Daryn Sharp
>         Attachments: HDFS-1834-v1.patch, HDFS-1834.patch
>
>
> The chmod tasks have a parallel="false" attribute attached to them.  This causes ant to shell out for each and every item in the fileset which slows down parts of the build.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

[jira] [Updated] (HDFS-1834) Accelerate chmod task in ant files

Posted by "Daryn Sharp (JIRA)" <ji...@apache.org>.
     [ https://issues.apache.org/jira/browse/HDFS-1834?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Daryn Sharp updated HDFS-1834:
------------------------------

    Resolution: Invalid
        Status: Resolved  (was: Patch Available)

Unnecessary given the switch to maven.

> Accelerate chmod task in ant files
> ----------------------------------
>
>                 Key: HDFS-1834
>                 URL: https://issues.apache.org/jira/browse/HDFS-1834
>             Project: Hadoop HDFS
>          Issue Type: Improvement
>          Components: build
>    Affects Versions: 0.23.0
>            Reporter: Daryn Sharp
>            Assignee: Daryn Sharp
>         Attachments: HDFS-1834-v1.patch, HDFS-1834.patch
>
>
> The chmod tasks have a parallel="false" attribute attached to them.  This causes ant to shell out for each and every item in the fileset which slows down parts of the build.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira