Standalone operation of Hadoop Pipes Program

[Rajeev Jain <ra...@gmail.com>]

Hi,

Is it possible to run Hadoop Pipes C++ programs in standalone mode?

I'm using the following configuration file without success, please advise.
I've tested the binary under DFS (Pseudo Mode) and the results are correct.
My standalone mode error is reproduced below.

FYI, I'm using Hadoop 0.20 w/ Java 1.6.0_16

Thanks,

RJ

Configuration (wcount.xml):

<?xml version="1.0"?>
<configuration>
    <property>
        <name>hadoop.pipes.executable</name>
        <value>/home/rj/work/p1/wcount-cc</value>
    </property>
    <property>
        <name>hadoop.pipes.java.recordreader</name>
        <value>true</value>
    </property>
    <property>
        <name>hadoop.pipes.java.recordwriter</name>
        <value>true</value>
    </property>
</configuration>


Command Line:

[rj@scoobydoo p1]$ hadoop pipes -conf conf/wcount.xml -input input -out
put output


Run Output:

09/08/18 20:15:00 INFO jvm.JvmMetrics: Initializing JVM Metrics with
processName
=JobTracker, sessionId=
09/08/18 20:15:00 WARN mapred.JobClient: No job jar file set.  User classes
may
not be found. See JobConf(Class) or JobConf#setJar(String).
09/08/18 20:15:00 INFO mapred.FileInputFormat: Total input paths to process
: 1
09/08/18 20:15:00 INFO mapred.JobClient: Running job: job_local_0001
09/08/18 20:15:00 INFO mapred.FileInputFormat: Total input paths to process
: 1
09/08/18 20:15:00 INFO mapred.MapTask: numReduceTasks: 1
09/08/18 20:15:00 INFO mapred.MapTask: io.sort.mb = 100
09/08/18 20:15:01 INFO mapred.MapTask: data buffer = 79691776/99614720
09/08/18 20:15:01 INFO mapred.MapTask: record buffer = 262144/327680
09/08/18 20:15:01 WARN mapred.LocalJobRunner: job_local_0001
java.lang.NullPointerException
        at
org.apache.hadoop.mapred.pipes.Application.<init>(Application.java:91
)
        at
org.apache.hadoop.mapred.pipes.PipesMapRunner.run(PipesMapRunner.java
:68)
        at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:356)
        at org.apache.hadoop.mapred.MapTask.run(MapTask.java:305)
        at
org.apache.hadoop.mapred.LocalJobRunner$Job.run(LocalJobRunner.java:1
76)
09/08/18 20:15:01 INFO mapred.JobClient:  map 0% reduce 0%
09/08/18 20:15:01 INFO mapred.JobClient: Job complete: job_local_0001
09/08/18 20:15:01 INFO mapred.JobClient: Counters: 0
Exception in thread "main" java.io.IOException: Job failed!
        at org.apache.hadoop.mapred.JobClient.runJob(JobClient.java:1255)
        at
org.apache.hadoop.mapred.pipes.Submitter.runJob(Submitter.java:244)
        at org.apache.hadoop.mapred.pipes.Submitter.run(Submitter.java:480)
        at org.apache.hadoop.mapred.pipes.Submitter.main(Submitter.java:494)