You are viewing a plain text version of this content. The canonical link for it is here.
Posted to common-dev@hadoop.apache.org by "Rodrigo Schmidt (JIRA)" <ji...@apache.org> on 2009/02/06 01:59:59 UTC
[jira] Created: (HADOOP-5180) FileNotFoundException when finishing
a profiled task that doesn't generate an output file
FileNotFoundException when finishing a profiled task that doesn't generate an output file
-----------------------------------------------------------------------------------------
Key: HADOOP-5180
URL: https://issues.apache.org/jira/browse/HADOOP-5180
Project: Hadoop Core
Issue Type: Bug
Components: mapred
Reporter: Rodrigo Schmidt
Priority: Trivial
When running a hadoop job with mapred.task.profile=true, hadoop assumes that mapred.task.profile.params contains a "%s" that can be changed for a file name that will be created by the profiler containing its text output. If the profiler doesn't generate such an output file, profiled tasks will raise a FileNotFound exception at the end of their execution:
09/02/05 15:39:10 INFO mapred.JobClient: map 100% reduce 0%
09/02/05 15:39:10 INFO mapred.JobClient: Communication problem with server: java.io.FileNotFoundException: http://machine1:50060/tasklog?plaintext=true&taskid=attempt_200901221838_0043_m_000000_0&filter=profile
at sun.net.www.protocol.http.HttpURLConnection.getInputStream(HttpURLConnection.java:1239)
at org.apache.hadoop.mapred.JobClient.downloadProfile(JobClient.java:1089)
at org.apache.hadoop.mapred.JobClient.runJob(JobClient.java:1166)
at JavaCat.main(JavaCat.java:46)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at org.apache.hadoop.util.RunJar.main(RunJar.java:165)
at org.apache.hadoop.mapred.JobShell.run(JobShell.java:54)
at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:65)
at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:79)
at org.apache.hadoop.mapred.JobShell.main(JobShell.java:68)
This happens because it's assumed users will use the default hprof profiler, which actually allows you to set the output file name. Not all profilers are like that and some don't have any option to create a text output log at all or don't allow you to change the name of the output file. In my case, I'm a profiler that doesn't generate text output and only allows you to set the output directory for the snapshot files it generates.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Assigned: (HADOOP-5180) FileNotFoundException when finishing
a profiled task that doesn't generate an output file
Posted by "Rodrigo Schmidt (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-5180?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Rodrigo Schmidt reassigned HADOOP-5180:
---------------------------------------
Assignee: Rodrigo Schmidt
> FileNotFoundException when finishing a profiled task that doesn't generate an output file
> -----------------------------------------------------------------------------------------
>
> Key: HADOOP-5180
> URL: https://issues.apache.org/jira/browse/HADOOP-5180
> Project: Hadoop Core
> Issue Type: Bug
> Components: mapred
> Reporter: Rodrigo Schmidt
> Assignee: Rodrigo Schmidt
> Priority: Trivial
>
> When running a hadoop job with mapred.task.profile=true, hadoop assumes that mapred.task.profile.params contains a "%s" that can be changed for a file name that will be created by the profiler containing its text output. If the profiler doesn't generate such an output file, profiled tasks will raise a FileNotFound exception at the end of their execution:
> 09/02/05 15:39:10 INFO mapred.JobClient: map 100% reduce 0%
> 09/02/05 15:39:10 INFO mapred.JobClient: Communication problem with server: java.io.FileNotFoundException: http://machine1:50060/tasklog?plaintext=true&taskid=attempt_200901221838_0043_m_000000_0&filter=profile
> at sun.net.www.protocol.http.HttpURLConnection.getInputStream(HttpURLConnection.java:1239)
> at org.apache.hadoop.mapred.JobClient.downloadProfile(JobClient.java:1089)
> at org.apache.hadoop.mapred.JobClient.runJob(JobClient.java:1166)
> at JavaCat.main(JavaCat.java:46)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:597)
> at org.apache.hadoop.util.RunJar.main(RunJar.java:165)
> at org.apache.hadoop.mapred.JobShell.run(JobShell.java:54)
> at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:65)
> at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:79)
> at org.apache.hadoop.mapred.JobShell.main(JobShell.java:68)
> This happens because it's assumed users will use the default hprof profiler, which actually allows you to set the output file name. Not all profilers are like that and some don't have any option to create a text output log at all or don't allow you to change the name of the output file. In my case, I'm a profiler that doesn't generate text output and only allows you to set the output directory for the snapshot files it generates.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.