You are viewing a plain text version of this content. The canonical link for it is here.
Posted to common-user@hadoop.apache.org by Razen Al Harbi <ra...@yahoo.com> on 2009/04/28 11:13:20 UTC
I need help
Hi all,
I am writing an application in which I create a forked process to execute a specific Map/Reduce job. The problem is that when I try to read the output stream of the forked process I get nothing and when I execute the same job manually it starts printing the output I am expecting. For clarification I will go through the simple code snippet:
Process p = rt.exec("hadoop jar GraphClean args");
BufferedReader reader = new BufferedReader(new InputStreamReader(p.getInputStream()));
String line = null;
check = true;
while(check){
line = reader.readLine();
if(line != null){// I know this will not finish it's only for testing.
System.out.println(line);
}
}
If I run this code nothing shows up. But if execute the command (hadoop jar GraphClean args) from the command line it works fine. I am using hadoop 0.19.0.
Thanks,
Razen
Re: I need help
Posted by Steve Loughran <st...@apache.org>.
Razen Al Harbi wrote:
> Hi all,
>
> I am writing an application in which I create a forked process to execute a specific Map/Reduce job. The problem is that when I try to read the output stream of the forked process I get nothing and when I execute the same job manually it starts printing the output I am expecting. For clarification I will go through the simple code snippet:
>
>
> Process p = rt.exec("hadoop jar GraphClean args");
> BufferedReader reader = new BufferedReader(new InputStreamReader(p.getInputStream()));
> String line = null;
> check = true;
> while(check){
> line = reader.readLine();
> if(line != null){// I know this will not finish it's only for testing.
> System.out.println(line);
> }
> }
>
> If I run this code nothing shows up. But if execute the command (hadoop jar GraphClean args) from the command line it works fine. I am using hadoop 0.19.0.
>
Why not just invoke the Hadoop job submission calls yourself, no need to
exec anything?
Look at org.apache.hadoop.util.RunJar to see what you need to do.
Avoid calling RunJar.main() directly as
- it calls System.exit() when it wants to exit with an error
- it adds shutdown hooks
-steve
Re: I need help
Posted by Steve Loughran <st...@apache.org>.
Razen Alharbi wrote:
> Thanks everybody,
>
> The issue was that hadoop writes all the outputs to stderr instead of stdout
> and i don't know why. I would really love to know why the usual hadoop job
> progress is written to stderr.
because there is a line in log4.properties telling it to do just that?
log4j.appender.console.target=System.err
--
Steve Loughran http://www.1060.org/blogxter/publish/5
Author: Ant in Action http://antbook.org/
Re: I need help
Posted by Razen Alharbi <ra...@yahoo.com>.
Thanks everybody,
The issue was that hadoop writes all the outputs to stderr instead of stdout
and i don't know why. I would really love to know why the usual hadoop job
progress is written to stderr.
Thanks again.
Razen
Razen Alharbi wrote:
>
> Hi all,
>
> I am writing an application in which I create a forked process to execute
> a specific Map/Reduce job. The problem is that when I try to read the
> output stream of the forked process I get nothing and when I execute the
> same job manually it starts printing the output I am expecting. For
> clarification I will go through the simple code snippet:
>
>
> Process p = rt.exec("hadoop jar GraphClean args");
> BufferedReader reader = new BufferedReader(new
> InputStreamReader(p.getInputStream()));
> String line = null;
> check = true;
> while(check){
> line = reader.readLine();
> if(line != null){// I know this will not finish it's only for testing.
> System.out.println(line);
> }
> }
>
> If I run this code nothing shows up. But if execute the command (hadoop
> jar GraphClean args) from the command line it works fine. I am using
> hadoop 0.19.0.
>
> Thanks,
>
> Razen
>
>
>
>
--
View this message in context: http://www.nabble.com/I-need-help-tp23273273p23307094.html
Sent from the Hadoop core-user mailing list archive at Nabble.com.
Re: I need help
Posted by "Edward J. Yoon" <ed...@apache.org>.
Hi,
Is that command available for all nodes? Did you try as below? ;)
Process proc = rt.exec("/bin/hostname");
..
output.collect(hostname, disk usage);
On Tue, Apr 28, 2009 at 6:13 PM, Razen Al Harbi <ra...@yahoo.com> wrote:
> Hi all,
>
> I am writing an application in which I create a forked process to execute a specific Map/Reduce job. The problem is that when I try to read the output stream of the forked process I get nothing and when I execute the same job manually it starts printing the output I am expecting. For clarification I will go through the simple code snippet:
>
>
> Process p = rt.exec("hadoop jar GraphClean args");
> BufferedReader reader = new BufferedReader(new InputStreamReader(p.getInputStream()));
> String line = null;
> check = true;
> while(check){
> line = reader.readLine();
> if(line != null){// I know this will not finish it's only for testing.
> System.out.println(line);
> }
> }
>
> If I run this code nothing shows up. But if execute the command (hadoop jar GraphClean args) from the command line it works fine. I am using hadoop 0.19.0.
>
> Thanks,
>
> Razen
>
>
>
--
Best Regards, Edward J. Yoon @ NHN, corp.
edwardyoon@apache.org
http://blog.udanax.org
Re: I need help
Posted by "Edward J. Yoon" <ed...@apache.org>.
Why not read the output result after job done? And, if you wanted see
the log4j log, you need to set the stdout option to log4jproperties.
On Wed, Apr 29, 2009 at 4:35 AM, Razen Alharbi <ra...@yahoo.com> wrote:
>
> Thanks for the reply,
>
> -Steve:
> I know that I can use the JobClient to run or submit jobs; however, for the
> time being I need to exec the job as a separate process.
>
> -Edward:
> The forked job is not executed from witin a map or reduce so I dont need to
> do data collection.
>
> It seems for some reason the output of the reduce tasks is not written to
> stdout because when I tried to direct the output to a tmp file using the
> following command (hadoop jar GraphClean args > tmp), nothing was written to
> the file and the output still goes to the screen.
>
> Regards,
>
> Razen
>
>
>
> Razen Alharbi wrote:
>>
>> Hi all,
>>
>> I am writing an application in which I create a forked process to execute
>> a specific Map/Reduce job. The problem is that when I try to read the
>> output stream of the forked process I get nothing and when I execute the
>> same job manually it starts printing the output I am expecting. For
>> clarification I will go through the simple code snippet:
>>
>>
>> Process p = rt.exec("hadoop jar GraphClean args");
>> BufferedReader reader = new BufferedReader(new
>> InputStreamReader(p.getInputStream()));
>> String line = null;
>> check = true;
>> while(check){
>> line = reader.readLine();
>> if(line != null){// I know this will not finish it's only for testing.
>> System.out.println(line);
>> }
>> }
>>
>> If I run this code nothing shows up. But if execute the command (hadoop
>> jar GraphClean args) from the command line it works fine. I am using
>> hadoop 0.19.0.
>>
>> Thanks,
>>
>> Razen
>>
>>
>>
>>
>
> --
> View this message in context: http://www.nabble.com/I-need-help-tp23273273p23284528.html
> Sent from the Hadoop core-user mailing list archive at Nabble.com.
>
>
--
Best Regards, Edward J. Yoon @ NHN, corp.
edwardyoon@apache.org
http://blog.udanax.org
Re: I need help
Posted by Razen Alharbi <ra...@yahoo.com>.
Thanks for the reply,
-Steve:
I know that I can use the JobClient to run or submit jobs; however, for the
time being I need to exec the job as a separate process.
-Edward:
The forked job is not executed from witin a map or reduce so I dont need to
do data collection.
It seems for some reason the output of the reduce tasks is not written to
stdout because when I tried to direct the output to a tmp file using the
following command (hadoop jar GraphClean args > tmp), nothing was written to
the file and the output still goes to the screen.
Regards,
Razen
Razen Alharbi wrote:
>
> Hi all,
>
> I am writing an application in which I create a forked process to execute
> a specific Map/Reduce job. The problem is that when I try to read the
> output stream of the forked process I get nothing and when I execute the
> same job manually it starts printing the output I am expecting. For
> clarification I will go through the simple code snippet:
>
>
> Process p = rt.exec("hadoop jar GraphClean args");
> BufferedReader reader = new BufferedReader(new
> InputStreamReader(p.getInputStream()));
> String line = null;
> check = true;
> while(check){
> line = reader.readLine();
> if(line != null){// I know this will not finish it's only for testing.
> System.out.println(line);
> }
> }
>
> If I run this code nothing shows up. But if execute the command (hadoop
> jar GraphClean args) from the command line it works fine. I am using
> hadoop 0.19.0.
>
> Thanks,
>
> Razen
>
>
>
>
--
View this message in context: http://www.nabble.com/I-need-help-tp23273273p23284528.html
Sent from the Hadoop core-user mailing list archive at Nabble.com.