You are viewing a plain text version of this content. The canonical link for it is here.
Posted to mapreduce-user@hadoop.apache.org by David Haykazyan <da...@onetick.com> on 2010/09/29 17:46:14 UTC

Hadoop pipes example hangs

Hi,

 

I am trying to run Hadoop pipes C++ example. I have successfully compiled it
with gcc using includes and libs under hadoop-0.21.0/c++/Linux-amd64-64. But
when I try to run it, it hangs.

 

Here is the command line:

~/tools/hadoop-0.21.0/bin/hadoop pipes -conf word.xml -input input/ -output
output

 

Here is my word.xml file:

 

<?xml version="1.0"?>

<configuration>

  <property>

    <name>hadoop.pipes.executable</name>

    <value>/home/david/work/kuku</value>

  </property>

  <property>

    <name>hadoop.pipes.java.recordreader</name>

    <value>true</value>

  </property>

  <property>

    <name>hadoop.pipes.java.recordwriter</name>

    <value>true</value>

  </property>

</configuration>

 

Here is the log when it runs:

 

10/09/29 20:33:53 INFO security.Groups: Group mapping
impl=org.apache.hadoop.security.ShellBasedUnixGroupsMapping;
cacheTimeout=300000

10/09/29 20:33:53 INFO jvm.JvmMetrics: Initializing JVM Metrics with
processName=JobTracker, sessionId=

10/09/29 20:33:53 WARN mapreduce.JobSubmitter: Use GenericOptionsParser for
parsing the arguments. Applications should implement Tool for the same.

10/09/29 20:33:53 WARN mapreduce.JobSubmitter: No job jar file set.  User
classes may not be found. See Job or Job#setJar(String).

10/09/29 20:33:53 INFO mapred.FileInputFormat: Total input paths to process
: 5

10/09/29 20:33:53 WARN conf.Configuration: mapred.map.tasks is deprecated.
Instead, use mapreduce.job.maps

10/09/29 20:33:53 INFO mapreduce.JobSubmitter: number of splits:5

10/09/29 20:33:53 INFO mapreduce.JobSubmitter: adding the following
namenodes' delegation tokens:null

10/09/29 20:33:53 WARN security.TokenCache: Overwriting existing token
storage with # keys=0

10/09/29 20:33:53 INFO filecache.TrackerDistributedCacheManager: Doing chmod
on localdir :/tmp/hadoop-david/mapred/local/archive/1697460138282585864

10/09/29 20:33:53 INFO filecache.TrackerDistributedCacheManager: Cached
/home/david/work/kuku as
/tmp/hadoop-david/mapred/local/archive/1697460138282585864/file/home/david/w
ork/kuku

10/09/29 20:33:53 INFO mapreduce.Job: Running job: job_local_0001

10/09/29 20:33:53 INFO mapred.LocalJobRunner: Waiting for map tasks

10/09/29 20:33:53 INFO mapred.LocalJobRunner: Starting task:
attempt_local_0001_m_000000_0

10/09/29 20:33:53 INFO mapred.MapTask: numReduceTasks: 1

10/09/29 20:33:54 INFO mapred.MapTask: (EQUATOR) 0 kvi 26214396(104857584)

10/09/29 20:33:54 INFO mapred.MapTask: mapreduce.task.io.sort.mb: 100

10/09/29 20:33:54 INFO mapred.MapTask: soft limit at 83886080

10/09/29 20:33:54 INFO mapred.MapTask: bufstart = 0; bufvoid = 104857600

10/09/29 20:33:54 INFO mapred.MapTask: kvstart = 26214396; length = 6553600

10/09/29 20:33:54 INFO util.ProcessTree: setsid exited with exit code 0

10/09/29 20:33:54 INFO mapreduce.Job:  map 0% reduce 0%

10/09/29 20:33:59 INFO mapred.LocalJobRunner:
file:/home/david/work/input/file04:0+27 > map

 

I have also tried to execute the compiled executable directly. It also hangs
and here's the backtrace output from gdb:

 

#0  0x0000003f410c439b in read () from /lib64/libc.so.6

#1  0x0000003f4106c797 in _IO_new_file_underflow () from /lib64/libc.so.6

#2  0x0000003f4106d15e in _IO_default_uflow_internal () from
/lib64/libc.so.6

#3  0x0000003f41068a1b in getc () from /lib64/libc.so.6

#4  0x000000000041370c in HadoopPipes::TextProtocol::readUpto ()

#5  0x000000000041376b in HadoopPipes::TextProtocol::nextEvent ()

#6  0x000000000040b63c in HadoopPipes::TaskContextImpl::waitForTask ()

#7  0x00000000004096d0 in HadoopPipes::runTask ()

#8  0x000000000040234a in main (argc=1, argv=0x7fff5b1667b8) at
wordcount-simple.cpp:30

 

Any ideas? Any help is appreciated. Thanks in advance.

 

Regards, David.