You are viewing a plain text version of this content. The canonical link for it is here.
Posted to common-user@hadoop.apache.org by Keith Wiley <kw...@keithwiley.com> on 2010/04/14 21:27:34 UTC
Pipes program with Java InputFormat/RecordReader
How do I use a nondefault Java InputFormat/RecordReader with a Pipes program. I realize I can set:
<property>
<name>hadoop.pipes.java.recordreader</name>
<value>true</value>
</property>
or alterntively "-D hadoop.pipes.java.recordreader=true"
...to get the default reader (and that works and my pipes program runs), but how do I use my own (Java) InputFormat/RecordReader? I tried jarring up my InputFormat and RecordReader, then passing the jar via -libjars, and specifying it with hadoop.pipes.java.recordreader thus:
<property>
<name>mapred.input.format.class</name>
<value>FitsHeaderInputFormat</value>
</property>
<property>
<name>hadoop.pipes.java.recordreader</name>
<value>FitsHDURecordReader</value>
</property>
$ pipes -libjars [PATH]/pipes.jar ...rest of command...
where pipes.jar contains:
$ jar tvf [PATH]/pipes.jar
0 Wed Apr 14 11:49:40 PDT 2010 META-INF/
60 Wed Apr 14 11:49:40 PDT 2010 META-INF/MANIFEST.MF
5167 Thu Apr 08 14:26:30 PDT 2010 FitsHDURecordReader.java
1816 Thu Apr 08 14:26:30 PDT 2010 FitsHeaderInputFormat.java
...but I get this error:
Exception in thread "main" java.lang.RuntimeException: java.lang.RuntimeException: java.lang.ClassNotFoundException: FitsHeaderInputFormat
at org.apache.hadoop.conf.Configuration.getClass(Configuration.java:900)
at org.apache.hadoop.mapred.JobConf.getInputFormat(JobConf.java:399)
at org.apache.hadoop.mapred.pipes.Submitter.setupPipesJob(Submitter.java:289)
at org.apache.hadoop.mapred.pipes.Submitter.runJob(Submitter.java:247)
at org.apache.hadoop.mapred.pipes.Submitter.run(Submitter.java:479)
at org.apache.hadoop.mapred.pipes.Submitter.main(Submitter.java:494)
Caused by: java.lang.RuntimeException: java.lang.ClassNotFoundException: FitsHeaderInputFormat
at org.apache.hadoop.conf.Configuration.getClass(Configuration.java:868)
at org.apache.hadoop.conf.Configuration.getClass(Configuration.java:892)
... 5 more
Caused by: java.lang.ClassNotFoundException: FitsHeaderInputFormat
at java.net.URLClassLoader$1.run(URLClassLoader.java:200)
at java.security.AccessController.doPrivileged(Native Method)
at java.net.URLClassLoader.findClass(URLClassLoader.java:188)
at java.lang.ClassLoader.loadClass(ClassLoader.java:315)
at java.lang.ClassLoader.loadClass(ClassLoader.java:250)
at java.lang.ClassLoader.loadClassInternal(ClassLoader.java:398)
at java.lang.Class.forName0(Native Method)
at java.lang.Class.forName(Class.java:247)
at org.apache.hadoop.conf.Configuration.getClassByName(Configuration.java:815)
at org.apache.hadoop.conf.Configuration.getClass(Configuration.java:866)
... 6 more
Lastly, I also tried specifying the class by the package at the top of the java file. Thus, where FitsHeaderInputFormat.java contains:
package org.uw.astro.util;
I tried:
<property>
<name>mapred.input.format.class</name>
<value>org.uw.astro.utilFitsHeaderInputFormat</value>
</property>
...but I get the exact same error. I am rapidly running out of ideas here.
Thanks for any help.
________________________________________________________________________________
Keith Wiley kwiley@keithwiley.com www.keithwiley.com
"What I primarily learned in grad school is how much I *don't* know.
Consequently, I left grad school with a higher ignorance to knowledge ratio than
when I entered."
-- Keith Wiley
________________________________________________________________________________
Re: Pipes program with Java InputFormat/RecordReader
Posted by Keith Wiley <kw...@keithwiley.com>.
On Apr 14, 2010, at 20:55 , Amareshwari Sri Ramadasu wrote:
> Hi Keith,
>
> My answers inline.
>
> On 4/15/10 12:57 AM, "Keith Wiley" <kw...@keithwiley.com> wrote:
>
> How do I use a nondefault Java InputFormat/RecordReader with a Pipes program. I realize I can set:
>
> <property>
> <name>hadoop.pipes.java.recordreader</name>
> <value>true</value>
> </property>
>
> or alterntively "-D hadoop.pipes.java.recordreader=true"
>
> ...to get the default reader (and that works and my pipes program runs), but how do I use my own (Java) InputFormat/RecordReader? I tried jarring up my InputFormat and RecordReader, then passing the jar via -libjars, and specifying it with hadoop.pipes.java.recordreader thus:
>
> <property>
> <name>mapred.input.format.class</name>
> <value>FitsHeaderInputFormat</value>
> </property>
>
> <property>
> <name>hadoop.pipes.java.recordreader</name>
> <value>FitsHDURecordReader</value>
> </property>
>
> >>>>> Here, the value for hadoop.pipes.java.recordreader should be true or false, not the recordreader itself.
I agree that the recordreader parameter is usually true or false, I was just trying stuff out...so which should it be if I am using my own recordreader, but which is in java? Does the hadoop.pipes.java.recordreader parameter state whether a java recordreader of any kind is being used with C++ pipes, or only whether the built-in default recordreader is being used? More to the point, what is the correct setting for this parameter in my situation, the one described in my original post?
> $ pipes -libjars [PATH]/pipes.jar ...rest of command...
>
> >>>>> Can you run the following command without specifying any other configuration ?
> $pipes -libjars [PATH]/pipes.jar inputformat FitsHeaderInputFormat .. rest of the command.
I assume you meant "-inputformat", with the hyphen. I still get:
Exception in thread "main" java.lang.ClassNotFoundException: FitsHeaderInputFormat
> where pipes.jar contains:
>
> $ jar tvf [PATH]/pipes.jar
> 0 Wed Apr 14 11:49:40 PDT 2010 META-INF/
> 60 Wed Apr 14 11:49:40 PDT 2010 META-INF/MANIFEST.MF
> 5167 Thu Apr 08 14:26:30 PDT 2010 FitsHDURecordReader.java
> 1816 Thu Apr 08 14:26:30 PDT 2010 FitsHeaderInputFormat.java
>
> ...but I get this error:
>
> Exception in thread "main" java.lang.RuntimeException: java.lang.RuntimeException: java.lang.ClassNotFoundException: FitsHeaderInputFormat
> at org.apache.hadoop.conf.Configuration.getClass(Configuration.java:900)
> at org.apache.hadoop.mapred.JobConf.getInputFormat(JobConf.java:399)
> at org.apache.hadoop.mapred.pipes.Submitter.setupPipesJob(Submitter.java:289)
> at org.apache.hadoop.mapred.pipes.Submitter.runJob(Submitter.java:247)
> at org.apache.hadoop.mapred.pipes.Submitter.run(Submitter.java:479)
> at org.apache.hadoop.mapred.pipes.Submitter.main(Submitter.java:494)
> Caused by: java.lang.RuntimeException: java.lang.ClassNotFoundException: FitsHeaderInputFormat
> at org.apache.hadoop.conf.Configuration.getClass(Configuration.java:868)
> at org.apache.hadoop.conf.Configuration.getClass(Configuration.java:892)
> ... 5 more
> Caused by: java.lang.ClassNotFoundException: FitsHeaderInputFormat
> at java.net.URLClassLoader$1.run(URLClassLoader.java:200)
> at java.security.AccessController.doPrivileged(Native Method)
> at java.net.URLClassLoader.findClass(URLClassLoader.java:188)
> at java.lang.ClassLoader.loadClass(ClassLoader.java:315)
> at java.lang.ClassLoader.loadClass(ClassLoader.java:250)
> at java.lang.ClassLoader.loadClassInternal(ClassLoader.java:398)
> at java.lang.Class.forName0(Native Method)
> at java.lang.Class.forName(Class.java:247)
> at org.apache.hadoop.conf.Configuration.getClassByName(Configuration.java:815)
> at org.apache.hadoop.conf.Configuration.getClass(Configuration.java:866)
> ... 6 more
>
> Lastly, I also tried specifying the class by the package at the top of the java file. Thus, where FitsHeaderInputFormat.java contains:
>
> package org.uw.astro.util;
>
> I tried:
>
> <property>
> <name>mapred.input.format.class</name>
> <value>org.uw.astro.utilFitsHeaderInputFormat</value>
> </property>
>
> >>>>> Here, the value should be org.uw.astro.util.FitsHeaderInputFormat. Looks like you missed '.' after util.
Nah, that's just a typo in the email, sorry. The .xml file didn't have that error in it, so that's not the cause of the problem.
Why do I keep getting a ClassNotFoundException on the input format class. There has got to be a way to get this to work.
Any other suggestions would really help. Thank you.
________________________________________________________________________________
Keith Wiley kwiley@keithwiley.com www.keithwiley.com
"What I primarily learned in grad school is how much I *don't* know.
Consequently, I left grad school with a higher ignorance to knowledge ratio than
when I entered."
-- Keith Wiley
________________________________________________________________________________
Re: Pipes program with Java InputFormat/RecordReader
Posted by Amareshwari Sri Ramadasu <am...@yahoo-inc.com>.
Hi Keith,
My answers inline.
On 4/15/10 12:57 AM, "Keith Wiley" <kw...@keithwiley.com> wrote:
How do I use a nondefault Java InputFormat/RecordReader with a Pipes program. I realize I can set:
<property>
<name>hadoop.pipes.java.recordreader</name>
<value>true</value>
</property>
or alterntively "-D hadoop.pipes.java.recordreader=true"
...to get the default reader (and that works and my pipes program runs), but how do I use my own (Java) InputFormat/RecordReader? I tried jarring up my InputFormat and RecordReader, then passing the jar via -libjars, and specifying it with hadoop.pipes.java.recordreader thus:
<property>
<name>mapred.input.format.class</name>
<value>FitsHeaderInputFormat</value>
</property>
<property>
<name>hadoop.pipes.java.recordreader</name>
<value>FitsHDURecordReader</value>
</property>
Here, the value for hadoop.pipes.java.recordreader should be true or false, not the recordreader itself.
$ pipes -libjars [PATH]/pipes.jar ...rest of command...
Can you run the following command without specifying any other configuration ?
$pipes -libjars [PATH]/pipes.jar inputformat FitsHeaderInputFormat .. rest of the command.
where pipes.jar contains:
$ jar tvf [PATH]/pipes.jar
0 Wed Apr 14 11:49:40 PDT 2010 META-INF/
60 Wed Apr 14 11:49:40 PDT 2010 META-INF/MANIFEST.MF
5167 Thu Apr 08 14:26:30 PDT 2010 FitsHDURecordReader.java
1816 Thu Apr 08 14:26:30 PDT 2010 FitsHeaderInputFormat.java
...but I get this error:
Exception in thread "main" java.lang.RuntimeException: java.lang.RuntimeException: java.lang.ClassNotFoundException: FitsHeaderInputFormat
at org.apache.hadoop.conf.Configuration.getClass(Configuration.java:900)
at org.apache.hadoop.mapred.JobConf.getInputFormat(JobConf.java:399)
at org.apache.hadoop.mapred.pipes.Submitter.setupPipesJob(Submitter.java:289)
at org.apache.hadoop.mapred.pipes.Submitter.runJob(Submitter.java:247)
at org.apache.hadoop.mapred.pipes.Submitter.run(Submitter.java:479)
at org.apache.hadoop.mapred.pipes.Submitter.main(Submitter.java:494)
Caused by: java.lang.RuntimeException: java.lang.ClassNotFoundException: FitsHeaderInputFormat
at org.apache.hadoop.conf.Configuration.getClass(Configuration.java:868)
at org.apache.hadoop.conf.Configuration.getClass(Configuration.java:892)
... 5 more
Caused by: java.lang.ClassNotFoundException: FitsHeaderInputFormat
at java.net.URLClassLoader$1.run(URLClassLoader.java:200)
at java.security.AccessController.doPrivileged(Native Method)
at java.net.URLClassLoader.findClass(URLClassLoader.java:188)
at java.lang.ClassLoader.loadClass(ClassLoader.java:315)
at java.lang.ClassLoader.loadClass(ClassLoader.java:250)
at java.lang.ClassLoader.loadClassInternal(ClassLoader.java:398)
at java.lang.Class.forName0(Native Method)
at java.lang.Class.forName(Class.java:247)
at org.apache.hadoop.conf.Configuration.getClassByName(Configuration.java:815)
at org.apache.hadoop.conf.Configuration.getClass(Configuration.java:866)
... 6 more
Lastly, I also tried specifying the class by the package at the top of the java file. Thus, where FitsHeaderInputFormat.java contains:
package org.uw.astro.util;
I tried:
<property>
<name>mapred.input.format.class</name>
<value>org.uw.astro.utilFitsHeaderInputFormat</value>
</property>
Here, the value should be org.uw.astro.util.FitsHeaderInputFormat. Looks like you missed '.' after util.
...but I get the exact same error. I am rapidly running out of ideas here.
Thanks for any help.
________________________________________________________________________________
Keith Wiley kwiley@keithwiley.com www.keithwiley.com
"What I primarily learned in grad school is how much I *don't* know.
Consequently, I left grad school with a higher ignorance to knowledge ratio than
when I entered."
-- Keith Wiley
________________________________________________________________________________