You are viewing a plain text version of this content. The canonical link for it is here.
Posted to common-issues@hadoop.apache.org by "Allen Wittenauer (Created) (JIRA)" <ji...@apache.org> on 2012/02/07 00:05:03 UTC
[jira] [Created] (HADOOP-8025) change default distcp log location
to be /tmp rather than cwd
change default distcp log location to be /tmp rather than cwd
-------------------------------------------------------------
Key: HADOOP-8025
URL: https://issues.apache.org/jira/browse/HADOOP-8025
Project: Hadoop Common
Issue Type: Improvement
Affects Versions: 1.0.0
Reporter: Allen Wittenauer
Priority: Trivial
distcp loves to leave emtpy files around. this puts them in /tmp so at least they are easy to find and kill.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-8025) change default distcp log location
to be /tmp rather than cwd
Posted by "Hudson (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-8025?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13223216#comment-13223216 ]
Hudson commented on HADOOP-8025:
--------------------------------
Integrated in Hadoop-Hdfs-trunk #976 (See [https://builds.apache.org/job/Hadoop-Hdfs-trunk/976/])
HADOOP-8025. Amendment (Revision 1297276)
Result = SUCCESS
tucu : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1297276
Files :
* /hadoop/common/trunk/hadoop-client/pom.xml
> change default distcp log location to be /tmp rather than cwd
> -------------------------------------------------------------
>
> Key: HADOOP-8025
> URL: https://issues.apache.org/jira/browse/HADOOP-8025
> Project: Hadoop Common
> Issue Type: Improvement
> Affects Versions: 1.0.0
> Reporter: Allen Wittenauer
> Priority: Trivial
> Attachments: HADOOP-8025-branch-1.0.txt
>
>
> distcp loves to leave emtpy files around. this puts them in /tmp so at least they are easy to find and kill.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-8025) change default distcp log location
to be /tmp rather than cwd
Posted by "Hudson (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-8025?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13222748#comment-13222748 ]
Hudson commented on HADOOP-8025:
--------------------------------
Integrated in Hadoop-Common-trunk-Commit #1837 (See [https://builds.apache.org/job/Hadoop-Common-trunk-Commit/1837/])
HADOOP-8025. Amendment (Revision 1297276)
Result = SUCCESS
tucu : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1297276
Files :
* /hadoop/common/trunk/hadoop-client/pom.xml
> change default distcp log location to be /tmp rather than cwd
> -------------------------------------------------------------
>
> Key: HADOOP-8025
> URL: https://issues.apache.org/jira/browse/HADOOP-8025
> Project: Hadoop Common
> Issue Type: Improvement
> Affects Versions: 1.0.0
> Reporter: Allen Wittenauer
> Priority: Trivial
> Attachments: HADOOP-8025-branch-1.0.txt
>
>
> distcp loves to leave emtpy files around. this puts them in /tmp so at least they are easy to find and kill.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-8025) change default distcp log location
to be /tmp rather than cwd
Posted by "Hudson (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-8025?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13223249#comment-13223249 ]
Hudson commented on HADOOP-8025:
--------------------------------
Integrated in Hadoop-Mapreduce-trunk #1011 (See [https://builds.apache.org/job/Hadoop-Mapreduce-trunk/1011/])
HADOOP-8025. Amendment (Revision 1297276)
Result = FAILURE
tucu : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1297276
Files :
* /hadoop/common/trunk/hadoop-client/pom.xml
> change default distcp log location to be /tmp rather than cwd
> -------------------------------------------------------------
>
> Key: HADOOP-8025
> URL: https://issues.apache.org/jira/browse/HADOOP-8025
> Project: Hadoop Common
> Issue Type: Improvement
> Affects Versions: 1.0.0
> Reporter: Allen Wittenauer
> Priority: Trivial
> Attachments: HADOOP-8025-branch-1.0.txt
>
>
> distcp loves to leave emtpy files around. this puts them in /tmp so at least they are easy to find and kill.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-8025) change default distcp log location
to be /tmp rather than cwd
Posted by "Allen Wittenauer (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-8025?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13201776#comment-13201776 ]
Allen Wittenauer commented on HADOOP-8025:
------------------------------------------
FWIW, we forcibly remove stuff from /tmp after 3 days via a simple perl script that does time conversions etc. Putting that policy in place + enforcement deals with the brokenness that is Pig and just general users being bad.
But right now, instead of leaving this cruft in /tmp, distcp is likely spreading zero length files throughout your entire file system. Luckily, distcp uses a fairly easily recognizable and random filename. With this patch, this is now in /tmp and easily dealt a rm.
Like most of my patches, I expect this patch has 0 chance of actually getting committed and is only here for others that want a fix.
> change default distcp log location to be /tmp rather than cwd
> -------------------------------------------------------------
>
> Key: HADOOP-8025
> URL: https://issues.apache.org/jira/browse/HADOOP-8025
> Project: Hadoop Common
> Issue Type: Improvement
> Affects Versions: 1.0.0
> Reporter: Allen Wittenauer
> Priority: Trivial
> Attachments: HADOOP-8025-branch-1.0.txt
>
>
> distcp loves to leave emtpy files around. this puts them in /tmp so at least they are easy to find and kill.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-8025) change default distcp log location
to be /tmp rather than cwd
Posted by "Hudson (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-8025?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13222771#comment-13222771 ]
Hudson commented on HADOOP-8025:
--------------------------------
Integrated in Hadoop-Mapreduce-trunk-Commit #1844 (See [https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Commit/1844/])
HADOOP-8025. Amendment (Revision 1297276)
Result = FAILURE
tucu : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1297276
Files :
* /hadoop/common/trunk/hadoop-client/pom.xml
> change default distcp log location to be /tmp rather than cwd
> -------------------------------------------------------------
>
> Key: HADOOP-8025
> URL: https://issues.apache.org/jira/browse/HADOOP-8025
> Project: Hadoop Common
> Issue Type: Improvement
> Affects Versions: 1.0.0
> Reporter: Allen Wittenauer
> Priority: Trivial
> Attachments: HADOOP-8025-branch-1.0.txt
>
>
> distcp loves to leave emtpy files around. this puts them in /tmp so at least they are easy to find and kill.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HADOOP-8025) change default distcp log location
to be /tmp rather than cwd
Posted by "Allen Wittenauer (Updated) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-8025?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Allen Wittenauer updated HADOOP-8025:
-------------------------------------
Attachment: HADOOP-8025-branch-1.0.txt
[I have no idea where 'tools' is supposed to go, so I'm going with common. Right now, this is horribly confusing.]
> change default distcp log location to be /tmp rather than cwd
> -------------------------------------------------------------
>
> Key: HADOOP-8025
> URL: https://issues.apache.org/jira/browse/HADOOP-8025
> Project: Hadoop Common
> Issue Type: Improvement
> Affects Versions: 1.0.0
> Reporter: Allen Wittenauer
> Priority: Trivial
> Attachments: HADOOP-8025-branch-1.0.txt
>
>
> distcp loves to leave emtpy files around. this puts them in /tmp so at least they are easy to find and kill.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-8025) change default distcp log location
to be /tmp rather than cwd
Posted by "Tsz Wo (Nicholas), SZE (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-8025?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13223581#comment-13223581 ]
Tsz Wo (Nicholas), SZE commented on HADOOP-8025:
------------------------------------------------
It seems that the commit was for HADOOP-8125.
{noformat}
------------------------------------------------------------------------
r1297276 | tucu | 2012-03-05 15:10:04 -0800 (Mon, 05 Mar 2012) | 1 line
HADOOP-8025. Amendment
------------------------------------------------------------------------
...
------------------------------------------------------------------------
r1297270 | tucu | 2012-03-05 15:04:19 -0800 (Mon, 05 Mar 2012) | 1 line
HADOOP-8125. make hadoop-client set of curated jars available in a distribution tarball (rvs via tucu)
{noformat}
> change default distcp log location to be /tmp rather than cwd
> -------------------------------------------------------------
>
> Key: HADOOP-8025
> URL: https://issues.apache.org/jira/browse/HADOOP-8025
> Project: Hadoop Common
> Issue Type: Improvement
> Affects Versions: 1.0.0
> Reporter: Allen Wittenauer
> Priority: Trivial
> Attachments: HADOOP-8025-branch-1.0.txt
>
>
> distcp loves to leave emtpy files around. this puts them in /tmp so at least they are easy to find and kill.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-8025) change default distcp log location
to be /tmp rather than cwd
Posted by "Allen Wittenauer (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-8025?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13223282#comment-13223282 ]
Allen Wittenauer commented on HADOOP-8025:
------------------------------------------
I think someone has the wrong jira tagged in their commit or Jenkins has gone insane.
> change default distcp log location to be /tmp rather than cwd
> -------------------------------------------------------------
>
> Key: HADOOP-8025
> URL: https://issues.apache.org/jira/browse/HADOOP-8025
> Project: Hadoop Common
> Issue Type: Improvement
> Affects Versions: 1.0.0
> Reporter: Allen Wittenauer
> Priority: Trivial
> Attachments: HADOOP-8025-branch-1.0.txt
>
>
> distcp loves to leave emtpy files around. this puts them in /tmp so at least they are easy to find and kill.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-8025) change default distcp log location
to be /tmp rather than cwd
Posted by "Hudson (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-8025?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13222766#comment-13222766 ]
Hudson commented on HADOOP-8025:
--------------------------------
Integrated in Hadoop-Hdfs-trunk-Commit #1911 (See [https://builds.apache.org/job/Hadoop-Hdfs-trunk-Commit/1911/])
HADOOP-8025. Amendment (Revision 1297276)
Result = SUCCESS
tucu : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1297276
Files :
* /hadoop/common/trunk/hadoop-client/pom.xml
> change default distcp log location to be /tmp rather than cwd
> -------------------------------------------------------------
>
> Key: HADOOP-8025
> URL: https://issues.apache.org/jira/browse/HADOOP-8025
> Project: Hadoop Common
> Issue Type: Improvement
> Affects Versions: 1.0.0
> Reporter: Allen Wittenauer
> Priority: Trivial
> Attachments: HADOOP-8025-branch-1.0.txt
>
>
> distcp loves to leave emtpy files around. this puts them in /tmp so at least they are easy to find and kill.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-8025) change default distcp log location
to be /tmp rather than cwd
Posted by "Brian Bockelman (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-8025?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13201756#comment-13201756 ]
Brian Bockelman commented on HADOOP-8025:
-----------------------------------------
Hi Allen,
Writing into /tmp is pretty scary to me due to things like symlink attacks. (If I want to find a security bug in a project, a simple place to start is to grep for usage of "/tmp"). It's not obvious to me whether or not the suggested patch opens distcp to symlink attacks.
If the proper flags are indeed used later on, I would suggest adding a comment to this patch pointing out why its not a problem.
Brian
PS - I'm probably a touch grouchy about this because I file tickets for user applications approximately monthly for leaving cruft in /tmp! Once it gets in the code, it never seems to escape.
> change default distcp log location to be /tmp rather than cwd
> -------------------------------------------------------------
>
> Key: HADOOP-8025
> URL: https://issues.apache.org/jira/browse/HADOOP-8025
> Project: Hadoop Common
> Issue Type: Improvement
> Affects Versions: 1.0.0
> Reporter: Allen Wittenauer
> Priority: Trivial
> Attachments: HADOOP-8025-branch-1.0.txt
>
>
> distcp loves to leave emtpy files around. this puts them in /tmp so at least they are easy to find and kill.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira