You are viewing a plain text version of this content. The canonical link for it is here.
Posted to common-dev@hadoop.apache.org by "Michael Andrews (JIRA)" <ji...@apache.org> on 2009/03/10 03:02:50 UTC
[jira] Created: (HADOOP-5445) HDFS Tmpreaper
HDFS Tmpreaper
--------------
Key: HADOOP-5445
URL: https://issues.apache.org/jira/browse/HADOOP-5445
Project: Hadoop Core
Issue Type: New Feature
Components: dfs
Environment: CentOs 4/5, Java 1.5, Hadoop 0.17.3
Reporter: Michael Andrews
Priority: Minor
Fix For: 0.17.3
Attachments: DateDelta.java, TmpReaper.java
Java implementation of tmpreaper utility for HDFS. Helps when you expect processes to die before they can clean up. I have perl unit tests that can be ported over to java or groovy if the hadoop team is interested in this utility. One issue is that the unit tests set the modification time of test files, which is unsupported in HDFS (as far as I can tell).
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-5445) HDFS Tmpreaper
Posted by "Michael Andrews (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-5445?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Michael Andrews updated HADOOP-5445:
------------------------------------
Attachment: DateDelta.java
TmpReaper.java
> HDFS Tmpreaper
> --------------
>
> Key: HADOOP-5445
> URL: https://issues.apache.org/jira/browse/HADOOP-5445
> Project: Hadoop Core
> Issue Type: New Feature
> Components: dfs
> Environment: CentOs 4/5, Java 1.5, Hadoop 0.17.3
> Reporter: Michael Andrews
> Priority: Minor
> Fix For: 0.17.3
>
> Attachments: DateDelta.java, TmpReaper.java
>
>
> Java implementation of tmpreaper utility for HDFS. Helps when you expect processes to die before they can clean up. I have perl unit tests that can be ported over to java or groovy if the hadoop team is interested in this utility. One issue is that the unit tests set the modification time of test files, which is unsupported in HDFS (as far as I can tell).
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-5445) HDFS Tmpreaper
Posted by "Johan Oskarsson (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-5445?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Johan Oskarsson updated HADOOP-5445:
------------------------------------
Fix Version/s: (was: 0.17.3)
Not a bugfix, won't go into 0.17.3
> HDFS Tmpreaper
> --------------
>
> Key: HADOOP-5445
> URL: https://issues.apache.org/jira/browse/HADOOP-5445
> Project: Hadoop Core
> Issue Type: New Feature
> Components: dfs
> Environment: CentOs 4/5, Java 1.5, Hadoop 0.17.3
> Reporter: Michael Andrews
> Priority: Minor
> Attachments: DateDelta.java, TmpReaper.java
>
>
> Java implementation of tmpreaper utility for HDFS. Helps when you expect processes to die before they can clean up. I have perl unit tests that can be ported over to java or groovy if the hadoop team is interested in this utility. One issue is that the unit tests set the modification time of test files, which is unsupported in HDFS (as far as I can tell).
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-5445) HDFS Tmpreaper
Posted by "Michael Andrews (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-5445?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12680947#action_12680947 ]
Michael Andrews commented on HADOOP-5445:
-----------------------------------------
This utility is meant to delete temporary files (typically in /tmp) that are older than a certain time. The user can specify which directories to search for old files in. Many unix systems run tmpreaper as cron job to clean up /tmp so that processes don't run out of temporary file handles.
(http://olympus.het.brown.edu/cgi-bin/man/man2html?tmpreaper+8)
The attached code is pretty much a clone of tmpreaper except that it operates on files stored in HDFS. Here is the usage message:
Usage:
tmpreaper [-help -verbose -test -force -debug] <delta> <dirs1> [<dirs2> ...]
options
-help (-h) Print this message
-verbose (-v) Be extra verbose
-test (-t) Don't actually remove any files
-force (-f) Override failsafe, e.g. delete files modified in the last
7 days
-debug (-d) Print out stack traces.
delta Amount and a unit of time, e.g. 30s,1m,5h,5d,6M,2y
dirs [dirs ...]
> HDFS Tmpreaper
> --------------
>
> Key: HADOOP-5445
> URL: https://issues.apache.org/jira/browse/HADOOP-5445
> Project: Hadoop Core
> Issue Type: New Feature
> Components: dfs
> Environment: CentOs 4/5, Java 1.5, Hadoop 0.17.3
> Reporter: Michael Andrews
> Priority: Minor
> Fix For: 0.17.3
>
> Attachments: DateDelta.java, TmpReaper.java
>
>
> Java implementation of tmpreaper utility for HDFS. Helps when you expect processes to die before they can clean up. I have perl unit tests that can be ported over to java or groovy if the hadoop team is interested in this utility. One issue is that the unit tests set the modification time of test files, which is unsupported in HDFS (as far as I can tell).
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-5445) HDFS Tmpreaper
Posted by "dhruba borthakur (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-5445?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12703444#action_12703444 ]
dhruba borthakur commented on HADOOP-5445:
------------------------------------------
> it is really helpful for a cronjob to come along and cleanup transient results that no one is using so that disk space can be recovered.
We have been doing precisely that. We have a cron job (outside of hadoop) to clean up left-over hdfs files periodically.
> HDFS Tmpreaper
> --------------
>
> Key: HADOOP-5445
> URL: https://issues.apache.org/jira/browse/HADOOP-5445
> Project: Hadoop Core
> Issue Type: New Feature
> Components: dfs
> Environment: CentOs 4/5, Java 1.5, Hadoop 0.17.3
> Reporter: Michael Andrews
> Priority: Minor
> Fix For: 0.17.3
>
> Attachments: DateDelta.java, TmpReaper.java
>
>
> Java implementation of tmpreaper utility for HDFS. Helps when you expect processes to die before they can clean up. I have perl unit tests that can be ported over to java or groovy if the hadoop team is interested in this utility. One issue is that the unit tests set the modification time of test files, which is unsupported in HDFS (as far as I can tell).
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-5445) HDFS Tmpreaper
Posted by "Michael Andrews (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-5445?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12700288#action_12700288 ]
Michael Andrews commented on HADOOP-5445:
-----------------------------------------
I agree that fuse HDFS would be perfect for this situation, and indeed this was one of the first options we looked into before developing this utility. We chose not to go with fuse because it seemed rather immature at the time (0.17) and also because it required a kernel module that was not standard in our distro at the time (CentOS 4.1). The time it took to develop this utility (2 days) was much less that the time it would have taken to create a new kickstart ISO and push this out to our production environment (red tape mostly), so we went with the more complicated but less time intensive route. I will definitely reevaluate the state of the fuse module, if we need to duplicate another standard unix tool.
> HDFS Tmpreaper
> --------------
>
> Key: HADOOP-5445
> URL: https://issues.apache.org/jira/browse/HADOOP-5445
> Project: Hadoop Core
> Issue Type: New Feature
> Components: dfs
> Environment: CentOs 4/5, Java 1.5, Hadoop 0.17.3
> Reporter: Michael Andrews
> Priority: Minor
> Fix For: 0.17.3
>
> Attachments: DateDelta.java, TmpReaper.java
>
>
> Java implementation of tmpreaper utility for HDFS. Helps when you expect processes to die before they can clean up. I have perl unit tests that can be ported over to java or groovy if the hadoop team is interested in this utility. One issue is that the unit tests set the modification time of test files, which is unsupported in HDFS (as far as I can tell).
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-5445) HDFS Tmpreaper
Posted by "Michael Andrews (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-5445?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12700259#action_12700259 ]
Michael Andrews commented on HADOOP-5445:
-----------------------------------------
Just wondering if there is any interest in this, it hasn't been commented on in a while.
The utility doesn't seem to do very much, but it does become very useful when your environment consists of many map/reduce jobs that are chained together in a pipeline. Each of our jobs potentially reports intermediate results and/or leaves the intermediate results for another job to pick up. Each of these results are transient and should probably be removed (as they can be rebuilt later), but need to be around long enough for other jobs to pickupt. In this context it is really helpful for a cronjob to come along and cleanup transient results that no one is using so that disk space can be recovered.
> HDFS Tmpreaper
> --------------
>
> Key: HADOOP-5445
> URL: https://issues.apache.org/jira/browse/HADOOP-5445
> Project: Hadoop Core
> Issue Type: New Feature
> Components: dfs
> Environment: CentOs 4/5, Java 1.5, Hadoop 0.17.3
> Reporter: Michael Andrews
> Priority: Minor
> Fix For: 0.17.3
>
> Attachments: DateDelta.java, TmpReaper.java
>
>
> Java implementation of tmpreaper utility for HDFS. Helps when you expect processes to die before they can clean up. I have perl unit tests that can be ported over to java or groovy if the hadoop team is interested in this utility. One issue is that the unit tests set the modification time of test files, which is unsupported in HDFS (as far as I can tell).
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-5445) HDFS Tmpreaper
Posted by "dhruba borthakur (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-5445?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12700267#action_12700267 ]
dhruba borthakur commented on HADOOP-5445:
------------------------------------------
This is a good tool, I agree. There are lots of unix-y tools that we can port to HDFS. One way to make them all run transparently on Hadoop file system is to use the fuse mount http://wiki.apache.org/hadoop/MountableHDFS. This allows a large chunk of unix-y tools to run on HDFS.
In our installation, we mount hdfs on one machine through fuse. Then we run some periodic (via cron) unix utilities that does cleaning/minitoring/reporting of HDFS in standard UNIX-filesystem-type ways. Will such a approach satisfy your needs?
> HDFS Tmpreaper
> --------------
>
> Key: HADOOP-5445
> URL: https://issues.apache.org/jira/browse/HADOOP-5445
> Project: Hadoop Core
> Issue Type: New Feature
> Components: dfs
> Environment: CentOs 4/5, Java 1.5, Hadoop 0.17.3
> Reporter: Michael Andrews
> Priority: Minor
> Fix For: 0.17.3
>
> Attachments: DateDelta.java, TmpReaper.java
>
>
> Java implementation of tmpreaper utility for HDFS. Helps when you expect processes to die before they can clean up. I have perl unit tests that can be ported over to java or groovy if the hadoop team is interested in this utility. One issue is that the unit tests set the modification time of test files, which is unsupported in HDFS (as far as I can tell).
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-5445) HDFS Tmpreaper
Posted by "dhruba borthakur (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-5445?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12680791#action_12680791 ]
dhruba borthakur commented on HADOOP-5445:
------------------------------------------
Can you pl explain what files does this utility delete?
Also, HDFS support access times and modification times starting with the 0.19 release.
> HDFS Tmpreaper
> --------------
>
> Key: HADOOP-5445
> URL: https://issues.apache.org/jira/browse/HADOOP-5445
> Project: Hadoop Core
> Issue Type: New Feature
> Components: dfs
> Environment: CentOs 4/5, Java 1.5, Hadoop 0.17.3
> Reporter: Michael Andrews
> Priority: Minor
> Fix For: 0.17.3
>
> Attachments: DateDelta.java, TmpReaper.java
>
>
> Java implementation of tmpreaper utility for HDFS. Helps when you expect processes to die before they can clean up. I have perl unit tests that can be ported over to java or groovy if the hadoop team is interested in this utility. One issue is that the unit tests set the modification time of test files, which is unsupported in HDFS (as far as I can tell).
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.