You are viewing a plain text version of this content. The canonical link for it is here.
Posted to common-user@hadoop.apache.org by springring <sp...@126.com> on 2013/03/02 09:18:40 UTC
WholeFileInputFormat with streaming
Hi,
I want to use:
hadoop jar <my dir>/hadoop-streaming-0.20.2-cdh3u3.jar -inputformat org.apache.hadoop.streaming.WholeFileInputFormat
so, I download code from :
https://github.com/tomwhite/hadoop-book/tree/master/ch07/src/main/java
WholeFileInputFormat.java
WholeFileRecordReader.java
and package the java file with :
package org.apache.hadoop.streaming;
solution A:
copy WholeFileInputFormat.java , WholeFileRecordReader.java to hadoop-0.20.2-cdh3u3/src/contrib/streaming/src/java/org/apache/hadoop/streaming/
then
javac -classpath /usr/lib/hadoop-0.20/hadoop-0.20.2-cdh3u3-core.jar:/usr/lib/hadoop-0.20:/usr/lib/hadoop-0.20/lib/* -d WFInputFormatClassNew hadoop-0.20.2-cdh3u3/src/contrib/streaming/src/java/org/apache/hadoop/streaming/*.java
there is a lot of error
solution B:
compile the java file WholeFileInputFormat.java , WholeFileRecordReader.java:
javac -classpath /usr/lib/hadoop-0.20/hadoop-0.20.2-cdh3u3-core.jar:/usr/lib/hadoop-0.20/*:/usr/lib/hadoop-0.20/lib/* -d WFInputFormatClass <two java file>
copy /usr/lib/hadoop/contrib/streaming/hadoop-streaming-0.20.2-cdh3u3.jar to <my dir>
then :
jar uf hadoop-streaming-0.20.2-cdh3u3.jar WFInputFormatClass/org/apache/hadoop/streaming/WholeFileRecordReader.class
jar uf hadoop-streaming-0.20.2-cdh3u3.jar WFInputFormatClass/org/apache/hadoop/streaming/WholeFileInputFormat.class
there is no error, but when I run:
hadoop jar <my dir>/hadoop-streaming-0.20.2-cdh3u3.jar -inputformat org.apache.hadoop.streaming.WholeFileInputFormat .......
there is error:
-inputformat : class not found : org.apache.hadoop.streaming.WholeFileInputFormat
what's wrong with the two solution? or is there any new solution?
thx.
Ring
Re: WholeFileInputFormat with streaming
Posted by Fatih Haltas <fa...@nyu.edu>.
Hi Ring,
Can you write output of "jar tf filename" command to see package
organization of jar, you created?
2 Mart 2013 Cumartesi tarihinde springring adlı kullanıcı şöyle yazdı:
> Hi,
>
> I want to use:
> hadoop jar <my dir>/hadoop-streaming-0.20.2-cdh3u3.jar -inputformat
> org.apache.hadoop.streaming.WholeFileInputFormat
>
> so, I download code from :
> https://github.com/tomwhite/hadoop-book/tree/master/ch07/src/main/java
> WholeFileInputFormat.java<https://github.com/tomwhite/hadoop-book/blob/master/ch07/src/main/java/WholeFileInputFormat.java>
> WholeFil! eRecordRe ader.java<https://github.com/tomwhite/hadoop-book/blob/master/ch07/src/main/java/WholeFileRecordReader.java>
>
> and package the java file with :
> package org.apache.hadoop.streaming;
>
> solution A:
> copy WholeFileInputFormat.java<https://github.com/tomwhite/hadoop-book/blob/master/ch07/src/main/java/WholeFileInputFormat.java>,
> WholeFileRecordReader.java<https://github.com/tomwhite/hadoop-book/blob/master/ch07/src/main/java/WholeFileRecordReader.java>
> to
> hadoop-0.20.2-cdh3u3/src/contrib/streaming/src/java/org/apache/hadoop/streaming/
> then
> javac -classpath
> /usr/lib/hadoop-0.20/hadoop-0.20.2-cdh3u3-core.jar:/usr/lib/hadoop-0.20:/usr/lib/hadoop-0.20/lib/*
> -d ! WFInputFo rmatClassNew
> hadoop-0.20.2-cdh3u3/src/contrib/streaming/src/java/org/apache/hadoop/streaming/*.java
>
> there is a lot of error
>
> solution B:
> compile the java file WholeFileInputFormat.java<https://github.com/tomwhite/hadoop-book/blob/master/ch07/src/main/java/WholeFileInputFormat.java>,
> WholeFileRecordReader.java<https://github.com/tomwhite/hadoop-book/blob/master/ch07/src/main/java/WholeFileRecordReader.java>
> :
> javac -classpath
> /usr/lib/hadoop-0.20/hadoop-0.20.2-cdh3u3-core.jar:/usr/lib/hadoop-0.20/*:/usr/lib/hadoop-0.20/lib/*
> -d WFInputFormatClass <two java file>
> copy /usr/l! ib/hadoop
> /contrib/streaming/hadoop-streaming-0.20.2-cdh3u3.jar to <my dir>
> then :
> jar uf hadoop-streaming-0.20.2-cdh3u3.jar
> WFInputFormatClass/org/apache/hadoop/streaming/WholeFileRecordReader.class
> jar uf hadoop-streaming-0.20.2-cdh3u3.jar
> WFInputFormatClass/org/apache/hadoop/streaming/WholeFileInputFormat.class
> there is no error, but when I run:
> hadoop jar <my dir>/hadoop-streaming-0.20.2-cdh3u3.jar -inputformat
> org.apache.hadoop.streaming.WholeFileInputFormat .......
> there is error:
> -inputformat : class not found :
> org.apache.hadoop.streaming.WholeFileInputFormat
>
> what's wrong with the two solution? or is there any new solution?
>
> thx.
>
> Ring
>
>
>
Re: WholeFileInputFormat with streaming
Posted by Fatih Haltas <fa...@nyu.edu>.
Hi Ring,
Can you write output of "jar tf filename" command to see package
organization of jar, you created?
2 Mart 2013 Cumartesi tarihinde springring adlı kullanıcı şöyle yazdı:
> Hi,
>
> I want to use:
> hadoop jar <my dir>/hadoop-streaming-0.20.2-cdh3u3.jar -inputformat
> org.apache.hadoop.streaming.WholeFileInputFormat
>
> so, I download code from :
> https://github.com/tomwhite/hadoop-book/tree/master/ch07/src/main/java
> WholeFileInputFormat.java<https://github.com/tomwhite/hadoop-book/blob/master/ch07/src/main/java/WholeFileInputFormat.java>
> WholeFil! eRecordRe ader.java<https://github.com/tomwhite/hadoop-book/blob/master/ch07/src/main/java/WholeFileRecordReader.java>
>
> and package the java file with :
> package org.apache.hadoop.streaming;
>
> solution A:
> copy WholeFileInputFormat.java<https://github.com/tomwhite/hadoop-book/blob/master/ch07/src/main/java/WholeFileInputFormat.java>,
> WholeFileRecordReader.java<https://github.com/tomwhite/hadoop-book/blob/master/ch07/src/main/java/WholeFileRecordReader.java>
> to
> hadoop-0.20.2-cdh3u3/src/contrib/streaming/src/java/org/apache/hadoop/streaming/
> then
> javac -classpath
> /usr/lib/hadoop-0.20/hadoop-0.20.2-cdh3u3-core.jar:/usr/lib/hadoop-0.20:/usr/lib/hadoop-0.20/lib/*
> -d ! WFInputFo rmatClassNew
> hadoop-0.20.2-cdh3u3/src/contrib/streaming/src/java/org/apache/hadoop/streaming/*.java
>
> there is a lot of error
>
> solution B:
> compile the java file WholeFileInputFormat.java<https://github.com/tomwhite/hadoop-book/blob/master/ch07/src/main/java/WholeFileInputFormat.java>,
> WholeFileRecordReader.java<https://github.com/tomwhite/hadoop-book/blob/master/ch07/src/main/java/WholeFileRecordReader.java>
> :
> javac -classpath
> /usr/lib/hadoop-0.20/hadoop-0.20.2-cdh3u3-core.jar:/usr/lib/hadoop-0.20/*:/usr/lib/hadoop-0.20/lib/*
> -d WFInputFormatClass <two java file>
> copy /usr/l! ib/hadoop
> /contrib/streaming/hadoop-streaming-0.20.2-cdh3u3.jar to <my dir>
> then :
> jar uf hadoop-streaming-0.20.2-cdh3u3.jar
> WFInputFormatClass/org/apache/hadoop/streaming/WholeFileRecordReader.class
> jar uf hadoop-streaming-0.20.2-cdh3u3.jar
> WFInputFormatClass/org/apache/hadoop/streaming/WholeFileInputFormat.class
> there is no error, but when I run:
> hadoop jar <my dir>/hadoop-streaming-0.20.2-cdh3u3.jar -inputformat
> org.apache.hadoop.streaming.WholeFileInputFormat .......
> there is error:
> -inputformat : class not found :
> org.apache.hadoop.streaming.WholeFileInputFormat
>
> what's wrong with the two solution? or is there any new solution?
>
> thx.
>
> Ring
>
>
>
Re: WholeFileInputFormat with streaming
Posted by Fatih Haltas <fa...@nyu.edu>.
Hi Ring,
Can you write output of "jar tf filename" command to see package
organization of jar, you created?
2 Mart 2013 Cumartesi tarihinde springring adlı kullanıcı şöyle yazdı:
> Hi,
>
> I want to use:
> hadoop jar <my dir>/hadoop-streaming-0.20.2-cdh3u3.jar -inputformat
> org.apache.hadoop.streaming.WholeFileInputFormat
>
> so, I download code from :
> https://github.com/tomwhite/hadoop-book/tree/master/ch07/src/main/java
> WholeFileInputFormat.java<https://github.com/tomwhite/hadoop-book/blob/master/ch07/src/main/java/WholeFileInputFormat.java>
> WholeFil! eRecordRe ader.java<https://github.com/tomwhite/hadoop-book/blob/master/ch07/src/main/java/WholeFileRecordReader.java>
>
> and package the java file with :
> package org.apache.hadoop.streaming;
>
> solution A:
> copy WholeFileInputFormat.java<https://github.com/tomwhite/hadoop-book/blob/master/ch07/src/main/java/WholeFileInputFormat.java>,
> WholeFileRecordReader.java<https://github.com/tomwhite/hadoop-book/blob/master/ch07/src/main/java/WholeFileRecordReader.java>
> to
> hadoop-0.20.2-cdh3u3/src/contrib/streaming/src/java/org/apache/hadoop/streaming/
> then
> javac -classpath
> /usr/lib/hadoop-0.20/hadoop-0.20.2-cdh3u3-core.jar:/usr/lib/hadoop-0.20:/usr/lib/hadoop-0.20/lib/*
> -d ! WFInputFo rmatClassNew
> hadoop-0.20.2-cdh3u3/src/contrib/streaming/src/java/org/apache/hadoop/streaming/*.java
>
> there is a lot of error
>
> solution B:
> compile the java file WholeFileInputFormat.java<https://github.com/tomwhite/hadoop-book/blob/master/ch07/src/main/java/WholeFileInputFormat.java>,
> WholeFileRecordReader.java<https://github.com/tomwhite/hadoop-book/blob/master/ch07/src/main/java/WholeFileRecordReader.java>
> :
> javac -classpath
> /usr/lib/hadoop-0.20/hadoop-0.20.2-cdh3u3-core.jar:/usr/lib/hadoop-0.20/*:/usr/lib/hadoop-0.20/lib/*
> -d WFInputFormatClass <two java file>
> copy /usr/l! ib/hadoop
> /contrib/streaming/hadoop-streaming-0.20.2-cdh3u3.jar to <my dir>
> then :
> jar uf hadoop-streaming-0.20.2-cdh3u3.jar
> WFInputFormatClass/org/apache/hadoop/streaming/WholeFileRecordReader.class
> jar uf hadoop-streaming-0.20.2-cdh3u3.jar
> WFInputFormatClass/org/apache/hadoop/streaming/WholeFileInputFormat.class
> there is no error, but when I run:
> hadoop jar <my dir>/hadoop-streaming-0.20.2-cdh3u3.jar -inputformat
> org.apache.hadoop.streaming.WholeFileInputFormat .......
> there is error:
> -inputformat : class not found :
> org.apache.hadoop.streaming.WholeFileInputFormat
>
> what's wrong with the two solution? or is there any new solution?
>
> thx.
>
> Ring
>
>
>
Re: WholeFileInputFormat with streaming
Posted by Fatih Haltas <fa...@nyu.edu>.
Hi Ring,
Can you write output of "jar tf filename" command to see package
organization of jar, you created?
2 Mart 2013 Cumartesi tarihinde springring adlı kullanıcı şöyle yazdı:
> Hi,
>
> I want to use:
> hadoop jar <my dir>/hadoop-streaming-0.20.2-cdh3u3.jar -inputformat
> org.apache.hadoop.streaming.WholeFileInputFormat
>
> so, I download code from :
> https://github.com/tomwhite/hadoop-book/tree/master/ch07/src/main/java
> WholeFileInputFormat.java<https://github.com/tomwhite/hadoop-book/blob/master/ch07/src/main/java/WholeFileInputFormat.java>
> WholeFil! eRecordRe ader.java<https://github.com/tomwhite/hadoop-book/blob/master/ch07/src/main/java/WholeFileRecordReader.java>
>
> and package the java file with :
> package org.apache.hadoop.streaming;
>
> solution A:
> copy WholeFileInputFormat.java<https://github.com/tomwhite/hadoop-book/blob/master/ch07/src/main/java/WholeFileInputFormat.java>,
> WholeFileRecordReader.java<https://github.com/tomwhite/hadoop-book/blob/master/ch07/src/main/java/WholeFileRecordReader.java>
> to
> hadoop-0.20.2-cdh3u3/src/contrib/streaming/src/java/org/apache/hadoop/streaming/
> then
> javac -classpath
> /usr/lib/hadoop-0.20/hadoop-0.20.2-cdh3u3-core.jar:/usr/lib/hadoop-0.20:/usr/lib/hadoop-0.20/lib/*
> -d ! WFInputFo rmatClassNew
> hadoop-0.20.2-cdh3u3/src/contrib/streaming/src/java/org/apache/hadoop/streaming/*.java
>
> there is a lot of error
>
> solution B:
> compile the java file WholeFileInputFormat.java<https://github.com/tomwhite/hadoop-book/blob/master/ch07/src/main/java/WholeFileInputFormat.java>,
> WholeFileRecordReader.java<https://github.com/tomwhite/hadoop-book/blob/master/ch07/src/main/java/WholeFileRecordReader.java>
> :
> javac -classpath
> /usr/lib/hadoop-0.20/hadoop-0.20.2-cdh3u3-core.jar:/usr/lib/hadoop-0.20/*:/usr/lib/hadoop-0.20/lib/*
> -d WFInputFormatClass <two java file>
> copy /usr/l! ib/hadoop
> /contrib/streaming/hadoop-streaming-0.20.2-cdh3u3.jar to <my dir>
> then :
> jar uf hadoop-streaming-0.20.2-cdh3u3.jar
> WFInputFormatClass/org/apache/hadoop/streaming/WholeFileRecordReader.class
> jar uf hadoop-streaming-0.20.2-cdh3u3.jar
> WFInputFormatClass/org/apache/hadoop/streaming/WholeFileInputFormat.class
> there is no error, but when I run:
> hadoop jar <my dir>/hadoop-streaming-0.20.2-cdh3u3.jar -inputformat
> org.apache.hadoop.streaming.WholeFileInputFormat .......
> there is error:
> -inputformat : class not found :
> org.apache.hadoop.streaming.WholeFileInputFormat
>
> what's wrong with the two solution? or is there any new solution?
>
> thx.
>
> Ring
>
>
>