pipes C++ and custom record reader in Java

[Mateusz Berezecki <ma...@gmail.com>]

Hello list readers,


I have a problem with pipes API and custom inputformat

My setup is as follows:

- InputFormat class written in Java
- mapper/reducer written in C++ using pipes API
- record reader written in C++

My InputFormat class does only have a dummy record reader which does nothing
so that I set up appropriate jobconf flags as to redirect all input records
to the C++ record reader.

I've managed to make this scenario work but this required altering CLASSPATH
in the hadoop's startup script.
Is there any way I could do this without altering this script?
I've seen that the Submitter.java has a following piece of code,
but it appears after processing -inputformat switch so the
jar file doesn't get added before loading inputformat class.

       // if they gave us a jar file, include it into the class path
      String jarFile = conf.getJar();
       if (jarFile != null) {
         ClassLoader loader =
          new URLClassLoader(new URL[]{ FileSystem.getLocal(conf).
                                         pathToFile(new
Path(jarFile)).toURL()});
         conf.setClassLoader(loader);

I was hoping that moving this block of code
and putting it just before processing -inputformat switch
and after processing -jar switch would produce the desired result
and I'd no longer see class not found error when providing a format
argument (i.e. -inputformat)

I packaged the jar containing the inputformat class properly,
I provided a full class name (i.e. including package name, etc. )

I run the code as follows:

./hadoop pipes -conf /tmp/mapredconf.xml \
    -program /mapred/applications/mapred \
    -input Test -inputformat org.hypertable.mapreduce.TableInputFormat \
    -output test_output \
    -jobconf hadoop.pipes.java.recordreader=false,hadoop.pipes.java.recordwriter=false
\
    -jar /opt/hypertable/0.9.0.7/java/lib/hypertable-0.9.0.7.jar

And it still does not work.
What do I do wrong? Or maybe, how do I do this right?

Mateusz