You are viewing a plain text version of this content. The canonical link for it is here.
Posted to mapreduce-user@hadoop.apache.org by David Haykazyan <da...@onetick.com> on 2010/09/29 17:46:14 UTC
Hadoop pipes example hangs
Hi,
I am trying to run Hadoop pipes C++ example. I have successfully compiled it
with gcc using includes and libs under hadoop-0.21.0/c++/Linux-amd64-64. But
when I try to run it, it hangs.
Here is the command line:
~/tools/hadoop-0.21.0/bin/hadoop pipes -conf word.xml -input input/ -output
output
Here is my word.xml file:
<?xml version="1.0"?>
<configuration>
<property>
<name>hadoop.pipes.executable</name>
<value>/home/david/work/kuku</value>
</property>
<property>
<name>hadoop.pipes.java.recordreader</name>
<value>true</value>
</property>
<property>
<name>hadoop.pipes.java.recordwriter</name>
<value>true</value>
</property>
</configuration>
Here is the log when it runs:
10/09/29 20:33:53 INFO security.Groups: Group mapping
impl=org.apache.hadoop.security.ShellBasedUnixGroupsMapping;
cacheTimeout=300000
10/09/29 20:33:53 INFO jvm.JvmMetrics: Initializing JVM Metrics with
processName=JobTracker, sessionId=
10/09/29 20:33:53 WARN mapreduce.JobSubmitter: Use GenericOptionsParser for
parsing the arguments. Applications should implement Tool for the same.
10/09/29 20:33:53 WARN mapreduce.JobSubmitter: No job jar file set. User
classes may not be found. See Job or Job#setJar(String).
10/09/29 20:33:53 INFO mapred.FileInputFormat: Total input paths to process
: 5
10/09/29 20:33:53 WARN conf.Configuration: mapred.map.tasks is deprecated.
Instead, use mapreduce.job.maps
10/09/29 20:33:53 INFO mapreduce.JobSubmitter: number of splits:5
10/09/29 20:33:53 INFO mapreduce.JobSubmitter: adding the following
namenodes' delegation tokens:null
10/09/29 20:33:53 WARN security.TokenCache: Overwriting existing token
storage with # keys=0
10/09/29 20:33:53 INFO filecache.TrackerDistributedCacheManager: Doing chmod
on localdir :/tmp/hadoop-david/mapred/local/archive/1697460138282585864
10/09/29 20:33:53 INFO filecache.TrackerDistributedCacheManager: Cached
/home/david/work/kuku as
/tmp/hadoop-david/mapred/local/archive/1697460138282585864/file/home/david/w
ork/kuku
10/09/29 20:33:53 INFO mapreduce.Job: Running job: job_local_0001
10/09/29 20:33:53 INFO mapred.LocalJobRunner: Waiting for map tasks
10/09/29 20:33:53 INFO mapred.LocalJobRunner: Starting task:
attempt_local_0001_m_000000_0
10/09/29 20:33:53 INFO mapred.MapTask: numReduceTasks: 1
10/09/29 20:33:54 INFO mapred.MapTask: (EQUATOR) 0 kvi 26214396(104857584)
10/09/29 20:33:54 INFO mapred.MapTask: mapreduce.task.io.sort.mb: 100
10/09/29 20:33:54 INFO mapred.MapTask: soft limit at 83886080
10/09/29 20:33:54 INFO mapred.MapTask: bufstart = 0; bufvoid = 104857600
10/09/29 20:33:54 INFO mapred.MapTask: kvstart = 26214396; length = 6553600
10/09/29 20:33:54 INFO util.ProcessTree: setsid exited with exit code 0
10/09/29 20:33:54 INFO mapreduce.Job: map 0% reduce 0%
10/09/29 20:33:59 INFO mapred.LocalJobRunner:
file:/home/david/work/input/file04:0+27 > map
I have also tried to execute the compiled executable directly. It also hangs
and here's the backtrace output from gdb:
#0 0x0000003f410c439b in read () from /lib64/libc.so.6
#1 0x0000003f4106c797 in _IO_new_file_underflow () from /lib64/libc.so.6
#2 0x0000003f4106d15e in _IO_default_uflow_internal () from
/lib64/libc.so.6
#3 0x0000003f41068a1b in getc () from /lib64/libc.so.6
#4 0x000000000041370c in HadoopPipes::TextProtocol::readUpto ()
#5 0x000000000041376b in HadoopPipes::TextProtocol::nextEvent ()
#6 0x000000000040b63c in HadoopPipes::TaskContextImpl::waitForTask ()
#7 0x00000000004096d0 in HadoopPipes::runTask ()
#8 0x000000000040234a in main (argc=1, argv=0x7fff5b1667b8) at
wordcount-simple.cpp:30
Any ideas? Any help is appreciated. Thanks in advance.
Regards, David.