You are viewing a plain text version of this content. The canonical link for it is here.
Posted to common-user@hadoop.apache.org by Daniel Yehdego <dt...@miners.utep.edu> on 2011/12/04 04:09:33 UTC
Hadoop Streaming
I have the following error in running hadoop streaming,
PipeMapRed\.waitOutputThreads(): subprocess failed with code 126 at org\.apache\.hadoop\.streaming\.PipeMapRed\.waitOutputThreads(PipeMapRed\.java:311) at org\.apache\.hadoop\.streaming\.PipeMapRed\.mapRedFinished(PipeMapRed\.java:545) at org\.apache\.hadoop\.streaming\.PipeMapper\.close(PipeMapper\.java:132) at org\.apache\.hadoop\.mapred\.MapRunner\.run(MapRunner\.java:57) at org\.apache\.hadoop\.streaming\.PipeMapRunner\.run(PipeMapRunner\.java:36) at org\.apache\.hadoop\.mapred\.MapTask\.runOldMapper(MapTask\.java:358) at org\.apache\.hadoop\.mapred\.MapTask\.run(MapTask\.java:307) at org\.apache\.hadoop\.mapred\.Child\.main(Child\.java:170)
I couldn't find out any other error information.
Any help ?
Re: Hadoop Streaming
Posted by Tom Melendez <to...@supertom.com>.
Oh, I see the line wrapped. My bad.
Either way, I think the NLineInputFormat is what you need. I'm
assuming you want one line of input to execute on one mapper.
Thanks,
Tom
On Sat, Dec 3, 2011 at 7:57 PM, Daniel Yehdego
<dt...@miners.utep.edu> wrote:
>
> TOM,
> What the HADOOP script do is ...read each line from the STDIN and execute the program pknotsRG. tmp.txt is a temporary file.
> the script is like this:
> #!/bin/sh
> rm -f temp.txt; while read line
> do echo $line >> temp.txt; done exec /data/yehdego/hadoop-0.20.2/PKNOTSRG/src/pknotsRG -k 0 -F temp.txt;
>
>> Date: Sat, 3 Dec 2011 19:49:46 -0800
>> Subject: Re: Hadoop Streaming
>> From: tom@supertom.com
>> To: common-user@hadoop.apache.org
>>
>> Hi Daniel,
>>
>> I see from your other thread that your HADOOP script has a line like:
>>
>> #!/bin/shrm -f temp.txt
>>
>> I'm not sure what that is, exactly. I suspect the -f is reading from
>> some file and the while loop you had listed read from stdin it seems.
>>
>> What does your input look like? I think what's happening is that you
>> might be expecting lines of input and you're getting splits. What
>> does your input look like?
>>
>> You might want to try this:
>> -inputformat org.apache.hadoop.mapred.lib.NLineInputFormat
>>
>> Thanks,
>>
>> Tom
>>
>>
>>
>>
>> On Sat, Dec 3, 2011 at 7:22 PM, Daniel Yehdego
>> <dt...@miners.utep.edu> wrote:
>> >
>> > Thanks Tom for your reply,
>> > I think my code is reading from stdin. Because I tried it locally using the following command and its running:
>> > $ bin/hadoop fs -cat /user/yehdego/Hadoop-Data-New/RF00171_A.bpseqL3G1_seg_Optimized_Method.txt | head -2 | ./HADOOP
>> >
>> > But when I tried streaming , it failed and gave me the error code 126.
>> >
>> >> Date: Sat, 3 Dec 2011 19:14:20 -0800
>> >> Subject: Re: Hadoop Streaming
>> >> From: tom@supertom.com
>> >> To: common-user@hadoop.apache.org
>> >>
>> >> So that code 126 should be kicked out by your program - do you know
>> >> what that means?
>> >>
>> >> Your code can read from stdin?
>> >>
>> >> Thanks,
>> >>
>> >> Tom
>> >>
>> >> On Sat, Dec 3, 2011 at 7:09 PM, Daniel Yehdego
>> >> <dt...@miners.utep.edu> wrote:
>> >> >
>> >> > I have the following error in running hadoop streaming,
>> >> > PipeMapRed\.waitOutputThreads(): subprocess failed with code 126 at org\.apache\.hadoop\.streaming\.PipeMapRed\.waitOutputThreads(PipeMapRed\.java:311) at org\.apache\.hadoop\.streaming\.PipeMapRed\.mapRedFinished(PipeMapRed\.java:545) at org\.apache\.hadoop\.streaming\.PipeMapper\.close(PipeMapper\.java:132) at org\.apache\.hadoop\.mapred\.MapRunner\.run(MapRunner\.java:57) at org\.apache\.hadoop\.streaming\.PipeMapRunner\.run(PipeMapRunner\.java:36) at org\.apache\.hadoop\.mapred\.MapTask\.runOldMapper(MapTask\.java:358) at org\.apache\.hadoop\.mapred\.MapTask\.run(MapTask\.java:307) at org\.apache\.hadoop\.mapred\.Child\.main(Child\.java:170)
>> >> > I couldn't find out any other error information.
>> >> > Any help ?
>> >> >
>> >
>
RE: Hadoop Streaming
Posted by Daniel Yehdego <dt...@miners.utep.edu>.
TOM,
What the HADOOP script do is ...read each line from the STDIN and execute the program pknotsRG. tmp.txt is a temporary file.
the script is like this:
#!/bin/sh
rm -f temp.txt; while read line
do echo $line >> temp.txt; done exec /data/yehdego/hadoop-0.20.2/PKNOTSRG/src/pknotsRG -k 0 -F temp.txt;
> Date: Sat, 3 Dec 2011 19:49:46 -0800
> Subject: Re: Hadoop Streaming
> From: tom@supertom.com
> To: common-user@hadoop.apache.org
>
> Hi Daniel,
>
> I see from your other thread that your HADOOP script has a line like:
>
> #!/bin/shrm -f temp.txt
>
> I'm not sure what that is, exactly. I suspect the -f is reading from
> some file and the while loop you had listed read from stdin it seems.
>
> What does your input look like? I think what's happening is that you
> might be expecting lines of input and you're getting splits. What
> does your input look like?
>
> You might want to try this:
> -inputformat org.apache.hadoop.mapred.lib.NLineInputFormat
>
> Thanks,
>
> Tom
>
>
>
>
> On Sat, Dec 3, 2011 at 7:22 PM, Daniel Yehdego
> <dt...@miners.utep.edu> wrote:
> >
> > Thanks Tom for your reply,
> > I think my code is reading from stdin. Because I tried it locally using the following command and its running:
> > $ bin/hadoop fs -cat /user/yehdego/Hadoop-Data-New/RF00171_A.bpseqL3G1_seg_Optimized_Method.txt | head -2 | ./HADOOP
> >
> > But when I tried streaming , it failed and gave me the error code 126.
> >
> >> Date: Sat, 3 Dec 2011 19:14:20 -0800
> >> Subject: Re: Hadoop Streaming
> >> From: tom@supertom.com
> >> To: common-user@hadoop.apache.org
> >>
> >> So that code 126 should be kicked out by your program - do you know
> >> what that means?
> >>
> >> Your code can read from stdin?
> >>
> >> Thanks,
> >>
> >> Tom
> >>
> >> On Sat, Dec 3, 2011 at 7:09 PM, Daniel Yehdego
> >> <dt...@miners.utep.edu> wrote:
> >> >
> >> > I have the following error in running hadoop streaming,
> >> > PipeMapRed\.waitOutputThreads(): subprocess failed with code 126 at org\.apache\.hadoop\.streaming\.PipeMapRed\.waitOutputThreads(PipeMapRed\.java:311) at org\.apache\.hadoop\.streaming\.PipeMapRed\.mapRedFinished(PipeMapRed\.java:545) at org\.apache\.hadoop\.streaming\.PipeMapper\.close(PipeMapper\.java:132) at org\.apache\.hadoop\.mapred\.MapRunner\.run(MapRunner\.java:57) at org\.apache\.hadoop\.streaming\.PipeMapRunner\.run(PipeMapRunner\.java:36) at org\.apache\.hadoop\.mapred\.MapTask\.runOldMapper(MapTask\.java:358) at org\.apache\.hadoop\.mapred\.MapTask\.run(MapTask\.java:307) at org\.apache\.hadoop\.mapred\.Child\.main(Child\.java:170)
> >> > I couldn't find out any other error information.
> >> > Any help ?
> >> >
> >
Re: Hadoop Streaming
Posted by Tom Melendez <to...@supertom.com>.
Hi Daniel,
I see from your other thread that your HADOOP script has a line like:
#!/bin/shrm -f temp.txt
I'm not sure what that is, exactly. I suspect the -f is reading from
some file and the while loop you had listed read from stdin it seems.
What does your input look like? I think what's happening is that you
might be expecting lines of input and you're getting splits. What
does your input look like?
You might want to try this:
-inputformat org.apache.hadoop.mapred.lib.NLineInputFormat
Thanks,
Tom
On Sat, Dec 3, 2011 at 7:22 PM, Daniel Yehdego
<dt...@miners.utep.edu> wrote:
>
> Thanks Tom for your reply,
> I think my code is reading from stdin. Because I tried it locally using the following command and its running:
> $ bin/hadoop fs -cat /user/yehdego/Hadoop-Data-New/RF00171_A.bpseqL3G1_seg_Optimized_Method.txt | head -2 | ./HADOOP
>
> But when I tried streaming , it failed and gave me the error code 126.
>
>> Date: Sat, 3 Dec 2011 19:14:20 -0800
>> Subject: Re: Hadoop Streaming
>> From: tom@supertom.com
>> To: common-user@hadoop.apache.org
>>
>> So that code 126 should be kicked out by your program - do you know
>> what that means?
>>
>> Your code can read from stdin?
>>
>> Thanks,
>>
>> Tom
>>
>> On Sat, Dec 3, 2011 at 7:09 PM, Daniel Yehdego
>> <dt...@miners.utep.edu> wrote:
>> >
>> > I have the following error in running hadoop streaming,
>> > PipeMapRed\.waitOutputThreads(): subprocess failed with code 126 at org\.apache\.hadoop\.streaming\.PipeMapRed\.waitOutputThreads(PipeMapRed\.java:311) at org\.apache\.hadoop\.streaming\.PipeMapRed\.mapRedFinished(PipeMapRed\.java:545) at org\.apache\.hadoop\.streaming\.PipeMapper\.close(PipeMapper\.java:132) at org\.apache\.hadoop\.mapred\.MapRunner\.run(MapRunner\.java:57) at org\.apache\.hadoop\.streaming\.PipeMapRunner\.run(PipeMapRunner\.java:36) at org\.apache\.hadoop\.mapred\.MapTask\.runOldMapper(MapTask\.java:358) at org\.apache\.hadoop\.mapred\.MapTask\.run(MapTask\.java:307) at org\.apache\.hadoop\.mapred\.Child\.main(Child\.java:170)
>> > I couldn't find out any other error information.
>> > Any help ?
>> >
>
RE: Hadoop Streaming
Posted by Daniel Yehdego <dt...@miners.utep.edu>.
Thanks Tom for your reply,
I think my code is reading from stdin. Because I tried it locally using the following command and its running:
$ bin/hadoop fs -cat /user/yehdego/Hadoop-Data-New/RF00171_A.bpseqL3G1_seg_Optimized_Method.txt | head -2 | ./HADOOP
But when I tried streaming , it failed and gave me the error code 126.
> Date: Sat, 3 Dec 2011 19:14:20 -0800
> Subject: Re: Hadoop Streaming
> From: tom@supertom.com
> To: common-user@hadoop.apache.org
>
> So that code 126 should be kicked out by your program - do you know
> what that means?
>
> Your code can read from stdin?
>
> Thanks,
>
> Tom
>
> On Sat, Dec 3, 2011 at 7:09 PM, Daniel Yehdego
> <dt...@miners.utep.edu> wrote:
> >
> > I have the following error in running hadoop streaming,
> > PipeMapRed\.waitOutputThreads(): subprocess failed with code 126 at org\.apache\.hadoop\.streaming\.PipeMapRed\.waitOutputThreads(PipeMapRed\.java:311) at org\.apache\.hadoop\.streaming\.PipeMapRed\.mapRedFinished(PipeMapRed\.java:545) at org\.apache\.hadoop\.streaming\.PipeMapper\.close(PipeMapper\.java:132) at org\.apache\.hadoop\.mapred\.MapRunner\.run(MapRunner\.java:57) at org\.apache\.hadoop\.streaming\.PipeMapRunner\.run(PipeMapRunner\.java:36) at org\.apache\.hadoop\.mapred\.MapTask\.runOldMapper(MapTask\.java:358) at org\.apache\.hadoop\.mapred\.MapTask\.run(MapTask\.java:307) at org\.apache\.hadoop\.mapred\.Child\.main(Child\.java:170)
> > I couldn't find out any other error information.
> > Any help ?
> >
Re: Hadoop Streaming
Posted by Tom Melendez <to...@supertom.com>.
So that code 126 should be kicked out by your program - do you know
what that means?
Your code can read from stdin?
Thanks,
Tom
On Sat, Dec 3, 2011 at 7:09 PM, Daniel Yehdego
<dt...@miners.utep.edu> wrote:
>
> I have the following error in running hadoop streaming,
> PipeMapRed\.waitOutputThreads(): subprocess failed with code 126 at org\.apache\.hadoop\.streaming\.PipeMapRed\.waitOutputThreads(PipeMapRed\.java:311) at org\.apache\.hadoop\.streaming\.PipeMapRed\.mapRedFinished(PipeMapRed\.java:545) at org\.apache\.hadoop\.streaming\.PipeMapper\.close(PipeMapper\.java:132) at org\.apache\.hadoop\.mapred\.MapRunner\.run(MapRunner\.java:57) at org\.apache\.hadoop\.streaming\.PipeMapRunner\.run(PipeMapRunner\.java:36) at org\.apache\.hadoop\.mapred\.MapTask\.runOldMapper(MapTask\.java:358) at org\.apache\.hadoop\.mapred\.MapTask\.run(MapTask\.java:307) at org\.apache\.hadoop\.mapred\.Child\.main(Child\.java:170)
> I couldn't find out any other error information.
> Any help ?
>