You are viewing a plain text version of this content. The canonical link for it is here.
Posted to common-dev@hadoop.apache.org by "Arun C Murthy (JIRA)" <ji...@apache.org> on 2007/10/25 10:27:51 UTC
[jira] Issue Comment Edited: (HADOOP-2098) File handles for log
files are still open in case of jobs with 0 maps
[ https://issues.apache.org/jira/browse/HADOOP-2098?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#action_12537539 ]
acmurthy edited comment on HADOOP-2098 at 10/25/07 1:26 AM:
-----------------------------------------------------------------
We should also check for zero input files in {{FileInputFormat.validateInput(JobConf)}} and throw an {{InvalidInputException}} in the first place.
Something along the lines of:
{noformat}
Index: /home/arun/dev/java/hadoop/HADOOP-1881/src/java/org/apache/hadoop/mapred/FileInputFormat.java
===================================================================
---src/java/org/apache/hadoop/mapred/FileInputFormat.java (revision 588140)
+++ src/java/org/apache/hadoop/mapred/FileInputFormat.java (working copy)
@@ -150,6 +150,9 @@
}
}
}
+ if (totalFiles == 0) {
+ result.add(new IOException("Found zero input files"));
+ }
if (!result.isEmpty()) {
throw new InvalidInputException(result);
}
{noformat}
was (Author: acmurthy):
We should also check for zero input files in {{FileInputFormat.validateInput(JobConf)}} and throw an {{InvalidInputException}} in the first place. Prevention is better than cure...
> File handles for log files are still open in case of jobs with 0 maps
> ---------------------------------------------------------------------
>
> Key: HADOOP-2098
> URL: https://issues.apache.org/jira/browse/HADOOP-2098
> Project: Hadoop
> Issue Type: Bug
> Components: mapred
> Affects Versions: 0.15.0
> Reporter: Amar Kamat
> Assignee: Amar Kamat
> Fix For: 0.16.0
>
> Attachments: HADOOP-2098.patch
>
>
> When a job with zero maps is submitted the handle for the log file for that job is still open and can be seen using {{lsof}}. This over time could lead to {{Too many open files Exception}}.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.