You are viewing a plain text version of this content. The canonical link for it is here.
Posted to common-user@hadoop.apache.org by ChaoChun Liang <cc...@gmail.com> on 2007/08/22 05:14:20 UTC
Fail to execute pipe example in Hadoop 0.14.0
Hi,
I followed the command, which described in README.txt under
~/src/examples/pipes
to execute wordcount-simple.
% bin/hadoop pipes -conf src/examples/pipes/conf/word.xml -input in-dir
-output out-dir
the program hangs as following
07/08/22 10:53:04 INFO mapred.FileInputFormat: Total input paths to
process : 1
07/08/22 10:53:05 INFO mapred.JobClient: Running job:
job_200708221046_0001
07/08/22 10:53:06 INFO mapred.JobClient: map 0% reduce 0%
the stderr under the map-log shows the message,
"Hadoop Pipes Exception: Illegal text protocol command"
1. Anyone meets the same problem?
2. there are three kinds of channels for socket communication,
"hadoop.pipes.command.port"
"hadoop.pipes.command.file" and stdin, which defined in runTask(). When
and how to use?
P.S. the error above looks like no proper commands for TextProtocol()
to get the next event?
3. Any guide line for *pipe*(ex. how to configure, how it works?)
ChaoChun
--
View this message in context: http://www.nabble.com/Fail-to-execute-pipe-example-in-Hadoop-0.14.0-tf4309130.html#a12267278
Sent from the Hadoop Users mailing list archive at Nabble.com.
Re: Fail to execute pipe example in Hadoop 0.14.0
Posted by ChaoChun Liang <cc...@gmail.com>.
>>Oh dear. Please file a bug on this.
I just jointed to Hadoop mail-list, how to file a bug?
>>Which version of Hadoop are you running?
>>In particular, are you running with or without HADOOP-1553?
My test version downloaded from
http://lucene.apache.org/hadoop/releases.html,
which according the release note, "Hadoop release 0.14.0 available"
by Doug Cutting Aug 22, 2007; 01:16am.
>>Do you have mapred.userlog.limit.kb set to 0 or non-0?
mapred.userlog.limit.kb set to 0
>>The C++ library has two modes the binary and text protocols. It
>>should always use the binary protocol, which it detects by having a
>>environment variable hadoop.pipes.command.port or
>>hadoop.pipes.command.file defined.
I tried to set "hadoop.pipes.command.port" in hadoop-site.xml and still not
work.
>>Since you clearly had the text
>>protocol running, the environment must have been messed up somehow.
>>The text protocol is really there to help me in debugging the
>>framework, although it means that if you run a pipes program from the
>>console you can interact with it.
Should all commands(ex. "start", "runMap", ...etc) transfer automatically by
Hadoop pipe if the "hadoop.pipes.command.port〞works correctly? Or users
need to write something to do that?
ChaoChun
--
View this message in context: http://www.nabble.com/Fail-to-execute-pipe-example-in-Hadoop-0.14.0-tf4309130.html#a12297808
Sent from the Hadoop Users mailing list archive at Nabble.com.
Re: Fail to execute pipe example in Hadoop 0.14.0
Posted by Owen O'Malley <oo...@yahoo-inc.com>.
On Aug 21, 2007, at 8:14 PM, ChaoChun Liang wrote:
> the stderr under the map-log shows the message,
> "Hadoop Pipes Exception: Illegal text protocol command"
Oh dear. Please file a bug on this. Which version of Hadoop are you
running? In particular, are you running with or without HADOOP-1553?
Do you have mapred.userlog.limit.kb set to 0 or non-0?
The C++ library has two modes the binary and text protocols. It
should always use the binary protocol, which it detects by having a
environment variable hadoop.pipes.command.port or
hadoop.pipes.command.file defined. Since you clearly had the text
protocol running, the environment must have been messed up somehow.
The text protocol is really there to help me in debugging the
framework, although it means that if you run a pipes program from the
console you can interact with it.
> 3. Any guide line for *pipe*(ex. how to configure, how it works?)
I plan on allocating time in the next couple weeks to write up some
of how to use and configure pipes. There is a little bit on how to
run the pipes program stand-alone for debugging at the bottom of the
page:
http://wiki.apache.org/lucene-hadoop/HowToDebugMapReducePrograms
-- Owen