You are viewing a plain text version of this content. The canonical link for it is here.
Posted to hdfs-user@hadoop.apache.org by Vikas Jadhav <vi...@gmail.com> on 2013/03/01 09:42:37 UTC
Buiding Hadoop code from source
Hello,
I want to to build hadoop code downloded from svn or git
can someone explicitly tell about these steps.
also i want to do some modifications using eclipse and again build that
code and install it on my machine
Thank You
--
*
*
*
Thanx and Regards*
* Vikas Jadhav*
Re: Buiding Hadoop code from source
Posted by YouPeng Yang <yy...@gmail.com>.
Hi Vikas
I use this link:
http://wiki.apache.org/hadoop/EclipseEnvironment
Things go well,except for the maven part,it needs to download the
dependecies from repositroy ,it gets stuck duing the download because of
some mirror urls are unavaiable,
so you need to changes somewhat avaiable.
another one:
http://wiki.apache.org/hadoop/HowToContribute
regards
2013/3/1 Ling Kun <lk...@gmail.com>
> Hi vikas:
> Have you asked google first? I searched in Google using your keyword,
> and found the following link.
>
> It may be helpful to you.
>
>
> yours,
> Ling Kun
>
>
> On Fri, Mar 1, 2013 at 4:42 PM, Vikas Jadhav <vi...@gmail.com>wrote:
>
>> Hello,
>>
>> I want to to build hadoop code downloded from svn or git
>>
>>
>> can someone explicitly tell about these steps.
>>
>> also i want to do some modifications using eclipse and again build that
>> code and install it on my machine
>>
>> Thank You
>>
>> --
>> *
>> *
>> *
>>
>> Thanx and Regards*
>> * Vikas Jadhav*
>>
>
>
>
> --
> http://www.lingcc.com
>
Re: Buiding Hadoop code from source
Posted by YouPeng Yang <yy...@gmail.com>.
Hi Vikas
I use this link:
http://wiki.apache.org/hadoop/EclipseEnvironment
Things go well,except for the maven part,it needs to download the
dependecies from repositroy ,it gets stuck duing the download because of
some mirror urls are unavaiable,
so you need to changes somewhat avaiable.
another one:
http://wiki.apache.org/hadoop/HowToContribute
regards
2013/3/1 Ling Kun <lk...@gmail.com>
> Hi vikas:
> Have you asked google first? I searched in Google using your keyword,
> and found the following link.
>
> It may be helpful to you.
>
>
> yours,
> Ling Kun
>
>
> On Fri, Mar 1, 2013 at 4:42 PM, Vikas Jadhav <vi...@gmail.com>wrote:
>
>> Hello,
>>
>> I want to to build hadoop code downloded from svn or git
>>
>>
>> can someone explicitly tell about these steps.
>>
>> also i want to do some modifications using eclipse and again build that
>> code and install it on my machine
>>
>> Thank You
>>
>> --
>> *
>> *
>> *
>>
>> Thanx and Regards*
>> * Vikas Jadhav*
>>
>
>
>
> --
> http://www.lingcc.com
>
Re: Buiding Hadoop code from source
Posted by YouPeng Yang <yy...@gmail.com>.
Hi Vikas
I use this link:
http://wiki.apache.org/hadoop/EclipseEnvironment
Things go well,except for the maven part,it needs to download the
dependecies from repositroy ,it gets stuck duing the download because of
some mirror urls are unavaiable,
so you need to changes somewhat avaiable.
another one:
http://wiki.apache.org/hadoop/HowToContribute
regards
2013/3/1 Ling Kun <lk...@gmail.com>
> Hi vikas:
> Have you asked google first? I searched in Google using your keyword,
> and found the following link.
>
> It may be helpful to you.
>
>
> yours,
> Ling Kun
>
>
> On Fri, Mar 1, 2013 at 4:42 PM, Vikas Jadhav <vi...@gmail.com>wrote:
>
>> Hello,
>>
>> I want to to build hadoop code downloded from svn or git
>>
>>
>> can someone explicitly tell about these steps.
>>
>> also i want to do some modifications using eclipse and again build that
>> code and install it on my machine
>>
>> Thank You
>>
>> --
>> *
>> *
>> *
>>
>> Thanx and Regards*
>> * Vikas Jadhav*
>>
>
>
>
> --
> http://www.lingcc.com
>
Re: Buiding Hadoop code from source
Posted by YouPeng Yang <yy...@gmail.com>.
Hi Vikas
I use this link:
http://wiki.apache.org/hadoop/EclipseEnvironment
Things go well,except for the maven part,it needs to download the
dependecies from repositroy ,it gets stuck duing the download because of
some mirror urls are unavaiable,
so you need to changes somewhat avaiable.
another one:
http://wiki.apache.org/hadoop/HowToContribute
regards
2013/3/1 Ling Kun <lk...@gmail.com>
> Hi vikas:
> Have you asked google first? I searched in Google using your keyword,
> and found the following link.
>
> It may be helpful to you.
>
>
> yours,
> Ling Kun
>
>
> On Fri, Mar 1, 2013 at 4:42 PM, Vikas Jadhav <vi...@gmail.com>wrote:
>
>> Hello,
>>
>> I want to to build hadoop code downloded from svn or git
>>
>>
>> can someone explicitly tell about these steps.
>>
>> also i want to do some modifications using eclipse and again build that
>> code and install it on my machine
>>
>> Thank You
>>
>> --
>> *
>> *
>> *
>>
>> Thanx and Regards*
>> * Vikas Jadhav*
>>
>
>
>
> --
> http://www.lingcc.com
>
Re: Buiding Hadoop code from source
Posted by Ling Kun <lk...@gmail.com>.
Hi vikas:
Have you asked google first? I searched in Google using your keyword, and
found the following link.
It may be helpful to you.
yours,
Ling Kun
On Fri, Mar 1, 2013 at 4:42 PM, Vikas Jadhav <vi...@gmail.com>wrote:
> Hello,
>
> I want to to build hadoop code downloded from svn or git
>
>
> can someone explicitly tell about these steps.
>
> also i want to do some modifications using eclipse and again build that
> code and install it on my machine
>
> Thank You
>
> --
> *
> *
> *
>
> Thanx and Regards*
> * Vikas Jadhav*
>
--
http://www.lingcc.com
Re: WholeFileInputFormat with streaming
Posted by Fatih Haltas <fa...@nyu.edu>.
Hi Ring,
Can you write output of "jar tf filename" command to see package
organization of jar, you created?
2 Mart 2013 Cumartesi tarihinde springring adlı kullanıcı şöyle yazdı:
> Hi,
>
> I want to use:
> hadoop jar <my dir>/hadoop-streaming-0.20.2-cdh3u3.jar -inputformat
> org.apache.hadoop.streaming.WholeFileInputFormat
>
> so, I download code from :
> https://github.com/tomwhite/hadoop-book/tree/master/ch07/src/main/java
> WholeFileInputFormat.java<https://github.com/tomwhite/hadoop-book/blob/master/ch07/src/main/java/WholeFileInputFormat.java>
> WholeFil! eRecordRe ader.java<https://github.com/tomwhite/hadoop-book/blob/master/ch07/src/main/java/WholeFileRecordReader.java>
>
> and package the java file with :
> package org.apache.hadoop.streaming;
>
> solution A:
> copy WholeFileInputFormat.java<https://github.com/tomwhite/hadoop-book/blob/master/ch07/src/main/java/WholeFileInputFormat.java>,
> WholeFileRecordReader.java<https://github.com/tomwhite/hadoop-book/blob/master/ch07/src/main/java/WholeFileRecordReader.java>
> to
> hadoop-0.20.2-cdh3u3/src/contrib/streaming/src/java/org/apache/hadoop/streaming/
> then
> javac -classpath
> /usr/lib/hadoop-0.20/hadoop-0.20.2-cdh3u3-core.jar:/usr/lib/hadoop-0.20:/usr/lib/hadoop-0.20/lib/*
> -d ! WFInputFo rmatClassNew
> hadoop-0.20.2-cdh3u3/src/contrib/streaming/src/java/org/apache/hadoop/streaming/*.java
>
> there is a lot of error
>
> solution B:
> compile the java file WholeFileInputFormat.java<https://github.com/tomwhite/hadoop-book/blob/master/ch07/src/main/java/WholeFileInputFormat.java>,
> WholeFileRecordReader.java<https://github.com/tomwhite/hadoop-book/blob/master/ch07/src/main/java/WholeFileRecordReader.java>
> :
> javac -classpath
> /usr/lib/hadoop-0.20/hadoop-0.20.2-cdh3u3-core.jar:/usr/lib/hadoop-0.20/*:/usr/lib/hadoop-0.20/lib/*
> -d WFInputFormatClass <two java file>
> copy /usr/l! ib/hadoop
> /contrib/streaming/hadoop-streaming-0.20.2-cdh3u3.jar to <my dir>
> then :
> jar uf hadoop-streaming-0.20.2-cdh3u3.jar
> WFInputFormatClass/org/apache/hadoop/streaming/WholeFileRecordReader.class
> jar uf hadoop-streaming-0.20.2-cdh3u3.jar
> WFInputFormatClass/org/apache/hadoop/streaming/WholeFileInputFormat.class
> there is no error, but when I run:
> hadoop jar <my dir>/hadoop-streaming-0.20.2-cdh3u3.jar -inputformat
> org.apache.hadoop.streaming.WholeFileInputFormat .......
> there is error:
> -inputformat : class not found :
> org.apache.hadoop.streaming.WholeFileInputFormat
>
> what's wrong with the two solution? or is there any new solution?
>
> thx.
>
> Ring
>
>
>
Re: WholeFileInputFormat with streaming
Posted by Fatih Haltas <fa...@nyu.edu>.
Hi Ring,
Can you write output of "jar tf filename" command to see package
organization of jar, you created?
2 Mart 2013 Cumartesi tarihinde springring adlı kullanıcı şöyle yazdı:
> Hi,
>
> I want to use:
> hadoop jar <my dir>/hadoop-streaming-0.20.2-cdh3u3.jar -inputformat
> org.apache.hadoop.streaming.WholeFileInputFormat
>
> so, I download code from :
> https://github.com/tomwhite/hadoop-book/tree/master/ch07/src/main/java
> WholeFileInputFormat.java<https://github.com/tomwhite/hadoop-book/blob/master/ch07/src/main/java/WholeFileInputFormat.java>
> WholeFil! eRecordRe ader.java<https://github.com/tomwhite/hadoop-book/blob/master/ch07/src/main/java/WholeFileRecordReader.java>
>
> and package the java file with :
> package org.apache.hadoop.streaming;
>
> solution A:
> copy WholeFileInputFormat.java<https://github.com/tomwhite/hadoop-book/blob/master/ch07/src/main/java/WholeFileInputFormat.java>,
> WholeFileRecordReader.java<https://github.com/tomwhite/hadoop-book/blob/master/ch07/src/main/java/WholeFileRecordReader.java>
> to
> hadoop-0.20.2-cdh3u3/src/contrib/streaming/src/java/org/apache/hadoop/streaming/
> then
> javac -classpath
> /usr/lib/hadoop-0.20/hadoop-0.20.2-cdh3u3-core.jar:/usr/lib/hadoop-0.20:/usr/lib/hadoop-0.20/lib/*
> -d ! WFInputFo rmatClassNew
> hadoop-0.20.2-cdh3u3/src/contrib/streaming/src/java/org/apache/hadoop/streaming/*.java
>
> there is a lot of error
>
> solution B:
> compile the java file WholeFileInputFormat.java<https://github.com/tomwhite/hadoop-book/blob/master/ch07/src/main/java/WholeFileInputFormat.java>,
> WholeFileRecordReader.java<https://github.com/tomwhite/hadoop-book/blob/master/ch07/src/main/java/WholeFileRecordReader.java>
> :
> javac -classpath
> /usr/lib/hadoop-0.20/hadoop-0.20.2-cdh3u3-core.jar:/usr/lib/hadoop-0.20/*:/usr/lib/hadoop-0.20/lib/*
> -d WFInputFormatClass <two java file>
> copy /usr/l! ib/hadoop
> /contrib/streaming/hadoop-streaming-0.20.2-cdh3u3.jar to <my dir>
> then :
> jar uf hadoop-streaming-0.20.2-cdh3u3.jar
> WFInputFormatClass/org/apache/hadoop/streaming/WholeFileRecordReader.class
> jar uf hadoop-streaming-0.20.2-cdh3u3.jar
> WFInputFormatClass/org/apache/hadoop/streaming/WholeFileInputFormat.class
> there is no error, but when I run:
> hadoop jar <my dir>/hadoop-streaming-0.20.2-cdh3u3.jar -inputformat
> org.apache.hadoop.streaming.WholeFileInputFormat .......
> there is error:
> -inputformat : class not found :
> org.apache.hadoop.streaming.WholeFileInputFormat
>
> what's wrong with the two solution? or is there any new solution?
>
> thx.
>
> Ring
>
>
>
Re: WholeFileInputFormat with streaming
Posted by Fatih Haltas <fa...@nyu.edu>.
Hi Ring,
Can you write output of "jar tf filename" command to see package
organization of jar, you created?
2 Mart 2013 Cumartesi tarihinde springring adlı kullanıcı şöyle yazdı:
> Hi,
>
> I want to use:
> hadoop jar <my dir>/hadoop-streaming-0.20.2-cdh3u3.jar -inputformat
> org.apache.hadoop.streaming.WholeFileInputFormat
>
> so, I download code from :
> https://github.com/tomwhite/hadoop-book/tree/master/ch07/src/main/java
> WholeFileInputFormat.java<https://github.com/tomwhite/hadoop-book/blob/master/ch07/src/main/java/WholeFileInputFormat.java>
> WholeFil! eRecordRe ader.java<https://github.com/tomwhite/hadoop-book/blob/master/ch07/src/main/java/WholeFileRecordReader.java>
>
> and package the java file with :
> package org.apache.hadoop.streaming;
>
> solution A:
> copy WholeFileInputFormat.java<https://github.com/tomwhite/hadoop-book/blob/master/ch07/src/main/java/WholeFileInputFormat.java>,
> WholeFileRecordReader.java<https://github.com/tomwhite/hadoop-book/blob/master/ch07/src/main/java/WholeFileRecordReader.java>
> to
> hadoop-0.20.2-cdh3u3/src/contrib/streaming/src/java/org/apache/hadoop/streaming/
> then
> javac -classpath
> /usr/lib/hadoop-0.20/hadoop-0.20.2-cdh3u3-core.jar:/usr/lib/hadoop-0.20:/usr/lib/hadoop-0.20/lib/*
> -d ! WFInputFo rmatClassNew
> hadoop-0.20.2-cdh3u3/src/contrib/streaming/src/java/org/apache/hadoop/streaming/*.java
>
> there is a lot of error
>
> solution B:
> compile the java file WholeFileInputFormat.java<https://github.com/tomwhite/hadoop-book/blob/master/ch07/src/main/java/WholeFileInputFormat.java>,
> WholeFileRecordReader.java<https://github.com/tomwhite/hadoop-book/blob/master/ch07/src/main/java/WholeFileRecordReader.java>
> :
> javac -classpath
> /usr/lib/hadoop-0.20/hadoop-0.20.2-cdh3u3-core.jar:/usr/lib/hadoop-0.20/*:/usr/lib/hadoop-0.20/lib/*
> -d WFInputFormatClass <two java file>
> copy /usr/l! ib/hadoop
> /contrib/streaming/hadoop-streaming-0.20.2-cdh3u3.jar to <my dir>
> then :
> jar uf hadoop-streaming-0.20.2-cdh3u3.jar
> WFInputFormatClass/org/apache/hadoop/streaming/WholeFileRecordReader.class
> jar uf hadoop-streaming-0.20.2-cdh3u3.jar
> WFInputFormatClass/org/apache/hadoop/streaming/WholeFileInputFormat.class
> there is no error, but when I run:
> hadoop jar <my dir>/hadoop-streaming-0.20.2-cdh3u3.jar -inputformat
> org.apache.hadoop.streaming.WholeFileInputFormat .......
> there is error:
> -inputformat : class not found :
> org.apache.hadoop.streaming.WholeFileInputFormat
>
> what's wrong with the two solution? or is there any new solution?
>
> thx.
>
> Ring
>
>
>
Re: WholeFileInputFormat with streaming
Posted by Fatih Haltas <fa...@nyu.edu>.
Hi Ring,
Can you write output of "jar tf filename" command to see package
organization of jar, you created?
2 Mart 2013 Cumartesi tarihinde springring adlı kullanıcı şöyle yazdı:
> Hi,
>
> I want to use:
> hadoop jar <my dir>/hadoop-streaming-0.20.2-cdh3u3.jar -inputformat
> org.apache.hadoop.streaming.WholeFileInputFormat
>
> so, I download code from :
> https://github.com/tomwhite/hadoop-book/tree/master/ch07/src/main/java
> WholeFileInputFormat.java<https://github.com/tomwhite/hadoop-book/blob/master/ch07/src/main/java/WholeFileInputFormat.java>
> WholeFil! eRecordRe ader.java<https://github.com/tomwhite/hadoop-book/blob/master/ch07/src/main/java/WholeFileRecordReader.java>
>
> and package the java file with :
> package org.apache.hadoop.streaming;
>
> solution A:
> copy WholeFileInputFormat.java<https://github.com/tomwhite/hadoop-book/blob/master/ch07/src/main/java/WholeFileInputFormat.java>,
> WholeFileRecordReader.java<https://github.com/tomwhite/hadoop-book/blob/master/ch07/src/main/java/WholeFileRecordReader.java>
> to
> hadoop-0.20.2-cdh3u3/src/contrib/streaming/src/java/org/apache/hadoop/streaming/
> then
> javac -classpath
> /usr/lib/hadoop-0.20/hadoop-0.20.2-cdh3u3-core.jar:/usr/lib/hadoop-0.20:/usr/lib/hadoop-0.20/lib/*
> -d ! WFInputFo rmatClassNew
> hadoop-0.20.2-cdh3u3/src/contrib/streaming/src/java/org/apache/hadoop/streaming/*.java
>
> there is a lot of error
>
> solution B:
> compile the java file WholeFileInputFormat.java<https://github.com/tomwhite/hadoop-book/blob/master/ch07/src/main/java/WholeFileInputFormat.java>,
> WholeFileRecordReader.java<https://github.com/tomwhite/hadoop-book/blob/master/ch07/src/main/java/WholeFileRecordReader.java>
> :
> javac -classpath
> /usr/lib/hadoop-0.20/hadoop-0.20.2-cdh3u3-core.jar:/usr/lib/hadoop-0.20/*:/usr/lib/hadoop-0.20/lib/*
> -d WFInputFormatClass <two java file>
> copy /usr/l! ib/hadoop
> /contrib/streaming/hadoop-streaming-0.20.2-cdh3u3.jar to <my dir>
> then :
> jar uf hadoop-streaming-0.20.2-cdh3u3.jar
> WFInputFormatClass/org/apache/hadoop/streaming/WholeFileRecordReader.class
> jar uf hadoop-streaming-0.20.2-cdh3u3.jar
> WFInputFormatClass/org/apache/hadoop/streaming/WholeFileInputFormat.class
> there is no error, but when I run:
> hadoop jar <my dir>/hadoop-streaming-0.20.2-cdh3u3.jar -inputformat
> org.apache.hadoop.streaming.WholeFileInputFormat .......
> there is error:
> -inputformat : class not found :
> org.apache.hadoop.streaming.WholeFileInputFormat
>
> what's wrong with the two solution? or is there any new solution?
>
> thx.
>
> Ring
>
>
>
WholeFileInputFormat with streaming
Posted by springring <sp...@126.com>.
Hi,
I want to use:
hadoop jar <my dir>/hadoop-streaming-0.20.2-cdh3u3.jar -inputformat org.apache.hadoop.streaming.WholeFileInputFormat
so, I download code from :
https://github.com/tomwhite/hadoop-book/tree/master/ch07/src/main/java
WholeFileInputFormat.java
WholeFileRecordReader.java
and package the java file with :
package org.apache.hadoop.streaming;
solution A:
copy WholeFileInputFormat.java , WholeFileRecordReader.java to hadoop-0.20.2-cdh3u3/src/contrib/streaming/src/java/org/apache/hadoop/streaming/
then
javac -classpath /usr/lib/hadoop-0.20/hadoop-0.20.2-cdh3u3-core.jar:/usr/lib/hadoop-0.20:/usr/lib/hadoop-0.20/lib/* -d WFInputFormatClassNew hadoop-0.20.2-cdh3u3/src/contrib/streaming/src/java/org/apache/hadoop/streaming/*.java
there is a lot of error
solution B:
compile the java file WholeFileInputFormat.java , WholeFileRecordReader.java:
javac -classpath /usr/lib/hadoop-0.20/hadoop-0.20.2-cdh3u3-core.jar:/usr/lib/hadoop-0.20/*:/usr/lib/hadoop-0.20/lib/* -d WFInputFormatClass <two java file>
copy /usr/lib/hadoop/contrib/streaming/hadoop-streaming-0.20.2-cdh3u3.jar to <my dir>
then :
jar uf hadoop-streaming-0.20.2-cdh3u3.jar WFInputFormatClass/org/apache/hadoop/streaming/WholeFileRecordReader.class
jar uf hadoop-streaming-0.20.2-cdh3u3.jar WFInputFormatClass/org/apache/hadoop/streaming/WholeFileInputFormat.class
there is no error, but when I run:
hadoop jar <my dir>/hadoop-streaming-0.20.2-cdh3u3.jar -inputformat org.apache.hadoop.streaming.WholeFileInputFormat .......
there is error:
-inputformat : class not found : org.apache.hadoop.streaming.WholeFileInputFormat
what's wrong with the two solution? or is there any new solution?
thx.
Ring
Re: Buiding Hadoop code from source
Posted by Ling Kun <lk...@gmail.com>.
Hi vikas:
Have you asked google first? I searched in Google using your keyword, and
found the following link.
It may be helpful to you.
yours,
Ling Kun
On Fri, Mar 1, 2013 at 4:42 PM, Vikas Jadhav <vi...@gmail.com>wrote:
> Hello,
>
> I want to to build hadoop code downloded from svn or git
>
>
> can someone explicitly tell about these steps.
>
> also i want to do some modifications using eclipse and again build that
> code and install it on my machine
>
> Thank You
>
> --
> *
> *
> *
>
> Thanx and Regards*
> * Vikas Jadhav*
>
--
http://www.lingcc.com
Re: Buiding Hadoop code from source
Posted by Ling Kun <lk...@gmail.com>.
Hi vikas:
Have you asked google first? I searched in Google using your keyword, and
found the following link.
It may be helpful to you.
yours,
Ling Kun
On Fri, Mar 1, 2013 at 4:42 PM, Vikas Jadhav <vi...@gmail.com>wrote:
> Hello,
>
> I want to to build hadoop code downloded from svn or git
>
>
> can someone explicitly tell about these steps.
>
> also i want to do some modifications using eclipse and again build that
> code and install it on my machine
>
> Thank You
>
> --
> *
> *
> *
>
> Thanx and Regards*
> * Vikas Jadhav*
>
--
http://www.lingcc.com
WholeFileInputFormat with streaming
Posted by springring <sp...@126.com>.
Hi,
I want to use:
hadoop jar <my dir>/hadoop-streaming-0.20.2-cdh3u3.jar -inputformat org.apache.hadoop.streaming.WholeFileInputFormat
so, I download code from :
https://github.com/tomwhite/hadoop-book/tree/master/ch07/src/main/java
WholeFileInputFormat.java
WholeFileRecordReader.java
and package the java file with :
package org.apache.hadoop.streaming;
solution A:
copy WholeFileInputFormat.java , WholeFileRecordReader.java to hadoop-0.20.2-cdh3u3/src/contrib/streaming/src/java/org/apache/hadoop/streaming/
then
javac -classpath /usr/lib/hadoop-0.20/hadoop-0.20.2-cdh3u3-core.jar:/usr/lib/hadoop-0.20:/usr/lib/hadoop-0.20/lib/* -d WFInputFormatClassNew hadoop-0.20.2-cdh3u3/src/contrib/streaming/src/java/org/apache/hadoop/streaming/*.java
there is a lot of error
solution B:
compile the java file WholeFileInputFormat.java , WholeFileRecordReader.java:
javac -classpath /usr/lib/hadoop-0.20/hadoop-0.20.2-cdh3u3-core.jar:/usr/lib/hadoop-0.20/*:/usr/lib/hadoop-0.20/lib/* -d WFInputFormatClass <two java file>
copy /usr/lib/hadoop/contrib/streaming/hadoop-streaming-0.20.2-cdh3u3.jar to <my dir>
then :
jar uf hadoop-streaming-0.20.2-cdh3u3.jar WFInputFormatClass/org/apache/hadoop/streaming/WholeFileRecordReader.class
jar uf hadoop-streaming-0.20.2-cdh3u3.jar WFInputFormatClass/org/apache/hadoop/streaming/WholeFileInputFormat.class
there is no error, but when I run:
hadoop jar <my dir>/hadoop-streaming-0.20.2-cdh3u3.jar -inputformat org.apache.hadoop.streaming.WholeFileInputFormat .......
there is error:
-inputformat : class not found : org.apache.hadoop.streaming.WholeFileInputFormat
what's wrong with the two solution? or is there any new solution?
thx.
Ring
WholeFileInputFormat with streaming
Posted by springring <sp...@126.com>.
Hi,
I want to use:
hadoop jar <my dir>/hadoop-streaming-0.20.2-cdh3u3.jar -inputformat org.apache.hadoop.streaming.WholeFileInputFormat
so, I download code from :
https://github.com/tomwhite/hadoop-book/tree/master/ch07/src/main/java
WholeFileInputFormat.java
WholeFileRecordReader.java
and package the java file with :
package org.apache.hadoop.streaming;
solution A:
copy WholeFileInputFormat.java , WholeFileRecordReader.java to hadoop-0.20.2-cdh3u3/src/contrib/streaming/src/java/org/apache/hadoop/streaming/
then
javac -classpath /usr/lib/hadoop-0.20/hadoop-0.20.2-cdh3u3-core.jar:/usr/lib/hadoop-0.20:/usr/lib/hadoop-0.20/lib/* -d WFInputFormatClassNew hadoop-0.20.2-cdh3u3/src/contrib/streaming/src/java/org/apache/hadoop/streaming/*.java
there is a lot of error
solution B:
compile the java file WholeFileInputFormat.java , WholeFileRecordReader.java:
javac -classpath /usr/lib/hadoop-0.20/hadoop-0.20.2-cdh3u3-core.jar:/usr/lib/hadoop-0.20/*:/usr/lib/hadoop-0.20/lib/* -d WFInputFormatClass <two java file>
copy /usr/lib/hadoop/contrib/streaming/hadoop-streaming-0.20.2-cdh3u3.jar to <my dir>
then :
jar uf hadoop-streaming-0.20.2-cdh3u3.jar WFInputFormatClass/org/apache/hadoop/streaming/WholeFileRecordReader.class
jar uf hadoop-streaming-0.20.2-cdh3u3.jar WFInputFormatClass/org/apache/hadoop/streaming/WholeFileInputFormat.class
there is no error, but when I run:
hadoop jar <my dir>/hadoop-streaming-0.20.2-cdh3u3.jar -inputformat org.apache.hadoop.streaming.WholeFileInputFormat .......
there is error:
-inputformat : class not found : org.apache.hadoop.streaming.WholeFileInputFormat
what's wrong with the two solution? or is there any new solution?
thx.
Ring
WholeFileInputFormat with streaming
Posted by springring <sp...@126.com>.
Hi,
I want to use:
hadoop jar <my dir>/hadoop-streaming-0.20.2-cdh3u3.jar -inputformat org.apache.hadoop.streaming.WholeFileInputFormat
so, I download code from :
https://github.com/tomwhite/hadoop-book/tree/master/ch07/src/main/java
WholeFileInputFormat.java
WholeFileRecordReader.java
and package the java file with :
package org.apache.hadoop.streaming;
solution A:
copy WholeFileInputFormat.java , WholeFileRecordReader.java to hadoop-0.20.2-cdh3u3/src/contrib/streaming/src/java/org/apache/hadoop/streaming/
then
javac -classpath /usr/lib/hadoop-0.20/hadoop-0.20.2-cdh3u3-core.jar:/usr/lib/hadoop-0.20:/usr/lib/hadoop-0.20/lib/* -d WFInputFormatClassNew hadoop-0.20.2-cdh3u3/src/contrib/streaming/src/java/org/apache/hadoop/streaming/*.java
there is a lot of error
solution B:
compile the java file WholeFileInputFormat.java , WholeFileRecordReader.java:
javac -classpath /usr/lib/hadoop-0.20/hadoop-0.20.2-cdh3u3-core.jar:/usr/lib/hadoop-0.20/*:/usr/lib/hadoop-0.20/lib/* -d WFInputFormatClass <two java file>
copy /usr/lib/hadoop/contrib/streaming/hadoop-streaming-0.20.2-cdh3u3.jar to <my dir>
then :
jar uf hadoop-streaming-0.20.2-cdh3u3.jar WFInputFormatClass/org/apache/hadoop/streaming/WholeFileRecordReader.class
jar uf hadoop-streaming-0.20.2-cdh3u3.jar WFInputFormatClass/org/apache/hadoop/streaming/WholeFileInputFormat.class
there is no error, but when I run:
hadoop jar <my dir>/hadoop-streaming-0.20.2-cdh3u3.jar -inputformat org.apache.hadoop.streaming.WholeFileInputFormat .......
there is error:
-inputformat : class not found : org.apache.hadoop.streaming.WholeFileInputFormat
what's wrong with the two solution? or is there any new solution?
thx.
Ring
Re: Buiding Hadoop code from source
Posted by Arpit Agarwal <aa...@hortonworks.com>.
Vikas, are you trying to build trunk? You will need Maven 3.0.
I recommend checking out the source tree and reading BUILDING.txt which has
detailed instructions, including how to build jar packages.
--Arpit
On Fri, Mar 1, 2013 at 9:49 AM, Vikas Jadhav <vi...@gmail.com>wrote:
> thank you all
> Working on ubuntu
>
> svn checkout
> http://svn.apache.org/repos/asf/hadoop/common/trunk/hadoop-trunk
>
> just want to know how to create jar of changed code and deploy total hadoop
> framework
>
> i followed
> 1) svn checkout
> http://svn.apache.org/repos/asf/hadoop/common/trunk/hadoop-trunk
>
> 2) ant
>
> 3) ant eclipse
>
> 4) set ANT_HOME=/usr/share/ant
>
>
>
>
>
> On Fri, Mar 1, 2013 at 8:56 PM, yiyu jia <ji...@gmail.com> wrote:
>
> > I write down my steps of compiling hadoop 1.1 here:
> >
> >
> http://yiyujia.blogspot.com/2012/10/steps-to-compile-hadoop-on-centos-63.html
> >
> > I basically follow articles on the apache wiki. I just add some steps
> that
> > not expilicitly mentioned like preparing googel protocolBuffer etc.
> >
> >
> > On Fri, Mar 1, 2013 at 8:48 AM, Noelle Jakusz <nj...@vmware.com>
> wrote:
> >
> > > +1
> > > Just getting started, I was looking for this too :)
> > >
> > > Thanks!
> > >
> > > ~~
> > > Noelle Jakusz
> > > Skype: noelle.jakusz
> > > Phone: (419) 862-5463<tel:(419)%20862-5463>
> > > http://about.me/noellejakusz
> > >
> > > On Mar 1, 2013, at 3:47 AM, "Agarwal, Nikhil" <
> Nikhil.Agarwal@netapp.com
> > > <ma...@netapp.com>> wrote:
> > >
> > > Hi Vikas,
> > >
> > > I you want to checkout the latest source code and compile and build
> then
> > I
> > > think Hadoop wiki would be the best source to see the steps.
> > >
> > > http://wiki.apache.org/hadoop/HowToContribute
> > >
> > > -----Original Message-----
> > > From: Vikas Jadhav [mailto:vikascjadhav87@gmail.com]
> > > Sent: Friday, March 01, 2013 2:13 PM
> > > To: hadoop-dev@lucene.apache.org<ma...@lucene.apache.org>;
> > > user@hadoop.apache.org<ma...@hadoop.apache.org>
> > > Subject: Buiding Hadoop code from source
> > >
> > > Hello,
> > >
> > > I want to to build hadoop code downloded from svn or git
> > >
> > >
> > > can someone explicitly tell about these steps.
> > >
> > > also i want to do some modifications using eclipse and again build that
> > > code and install it on my machine
> > >
> > > Thank You
> > >
> > > --
> > > *
> > > *
> > > *
> > >
> > > Thanx and Regards*
> > > * Vikas Jadhav*
> > >
> >
>
>
>
> --
> *
> *
> *
>
> Thanx and Regards*
> * Vikas Jadhav*
>
Re: Buiding Hadoop code from source
Posted by Vikas Jadhav <vi...@gmail.com>.
thank you all
Working on ubuntu
svn checkout http://svn.apache.org/repos/asf/hadoop/common/trunk/hadoop-trunk
just want to know how to create jar of changed code and deploy total hadoop
framework
i followed
1) svn checkout http://svn.apache.org/repos/asf/hadoop/common/trunk/hadoop-trunk
2) ant
3) ant eclipse
4) set ANT_HOME=/usr/share/ant
On Fri, Mar 1, 2013 at 8:56 PM, yiyu jia <ji...@gmail.com> wrote:
> I write down my steps of compiling hadoop 1.1 here:
>
> http://yiyujia.blogspot.com/2012/10/steps-to-compile-hadoop-on-centos-63.html
>
> I basically follow articles on the apache wiki. I just add some steps that
> not expilicitly mentioned like preparing googel protocolBuffer etc.
>
>
> On Fri, Mar 1, 2013 at 8:48 AM, Noelle Jakusz <nj...@vmware.com> wrote:
>
> > +1
> > Just getting started, I was looking for this too :)
> >
> > Thanks!
> >
> > ~~
> > Noelle Jakusz
> > Skype: noelle.jakusz
> > Phone: (419) 862-5463<tel:(419)%20862-5463>
> > http://about.me/noellejakusz
> >
> > On Mar 1, 2013, at 3:47 AM, "Agarwal, Nikhil" <Nikhil.Agarwal@netapp.com
> > <ma...@netapp.com>> wrote:
> >
> > Hi Vikas,
> >
> > I you want to checkout the latest source code and compile and build then
> I
> > think Hadoop wiki would be the best source to see the steps.
> >
> > http://wiki.apache.org/hadoop/HowToContribute
> >
> > -----Original Message-----
> > From: Vikas Jadhav [mailto:vikascjadhav87@gmail.com]
> > Sent: Friday, March 01, 2013 2:13 PM
> > To: hadoop-dev@lucene.apache.org<ma...@lucene.apache.org>;
> > user@hadoop.apache.org<ma...@hadoop.apache.org>
> > Subject: Buiding Hadoop code from source
> >
> > Hello,
> >
> > I want to to build hadoop code downloded from svn or git
> >
> >
> > can someone explicitly tell about these steps.
> >
> > also i want to do some modifications using eclipse and again build that
> > code and install it on my machine
> >
> > Thank You
> >
> > --
> > *
> > *
> > *
> >
> > Thanx and Regards*
> > * Vikas Jadhav*
> >
>
--
*
*
*
Thanx and Regards*
* Vikas Jadhav*
Re: Buiding Hadoop code from source
Posted by yiyu jia <ji...@gmail.com>.
I write down my steps of compiling hadoop 1.1 here:
http://yiyujia.blogspot.com/2012/10/steps-to-compile-hadoop-on-centos-63.html
I basically follow articles on the apache wiki. I just add some steps that
not expilicitly mentioned like preparing googel protocolBuffer etc.
On Fri, Mar 1, 2013 at 8:48 AM, Noelle Jakusz <nj...@vmware.com> wrote:
> +1
> Just getting started, I was looking for this too :)
>
> Thanks!
>
> ~~
> Noelle Jakusz
> Skype: noelle.jakusz
> Phone: (419) 862-5463<tel:(419)%20862-5463>
> http://about.me/noellejakusz
>
> On Mar 1, 2013, at 3:47 AM, "Agarwal, Nikhil" <Nikhil.Agarwal@netapp.com
> <ma...@netapp.com>> wrote:
>
> Hi Vikas,
>
> I you want to checkout the latest source code and compile and build then I
> think Hadoop wiki would be the best source to see the steps.
>
> http://wiki.apache.org/hadoop/HowToContribute
>
> -----Original Message-----
> From: Vikas Jadhav [mailto:vikascjadhav87@gmail.com]
> Sent: Friday, March 01, 2013 2:13 PM
> To: hadoop-dev@lucene.apache.org<ma...@lucene.apache.org>;
> user@hadoop.apache.org<ma...@hadoop.apache.org>
> Subject: Buiding Hadoop code from source
>
> Hello,
>
> I want to to build hadoop code downloded from svn or git
>
>
> can someone explicitly tell about these steps.
>
> also i want to do some modifications using eclipse and again build that
> code and install it on my machine
>
> Thank You
>
> --
> *
> *
> *
>
> Thanx and Regards*
> * Vikas Jadhav*
>
Re: Buiding Hadoop code from source
Posted by Noelle Jakusz <nj...@vmware.com>.
+1
Just getting started, I was looking for this too :)
Thanks!
~~
Noelle Jakusz
Skype: noelle.jakusz
Phone: (419) 862-5463<tel:(419)%20862-5463>
http://about.me/noellejakusz
On Mar 1, 2013, at 3:47 AM, "Agarwal, Nikhil" <Ni...@netapp.com>> wrote:
Hi Vikas,
I you want to checkout the latest source code and compile and build then I think Hadoop wiki would be the best source to see the steps.
http://wiki.apache.org/hadoop/HowToContribute
-----Original Message-----
From: Vikas Jadhav [mailto:vikascjadhav87@gmail.com]
Sent: Friday, March 01, 2013 2:13 PM
To: hadoop-dev@lucene.apache.org<ma...@lucene.apache.org>; user@hadoop.apache.org<ma...@hadoop.apache.org>
Subject: Buiding Hadoop code from source
Hello,
I want to to build hadoop code downloded from svn or git
can someone explicitly tell about these steps.
also i want to do some modifications using eclipse and again build that code and install it on my machine
Thank You
--
*
*
*
Thanx and Regards*
* Vikas Jadhav*
RE: Buiding Hadoop code from source
Posted by "Agarwal, Nikhil" <Ni...@netapp.com>.
Hi Vikas,
I you want to checkout the latest source code and compile and build then I think Hadoop wiki would be the best source to see the steps.
http://wiki.apache.org/hadoop/HowToContribute
-----Original Message-----
From: Vikas Jadhav [mailto:vikascjadhav87@gmail.com]
Sent: Friday, March 01, 2013 2:13 PM
To: hadoop-dev@lucene.apache.org; user@hadoop.apache.org
Subject: Buiding Hadoop code from source
Hello,
I want to to build hadoop code downloded from svn or git
can someone explicitly tell about these steps.
also i want to do some modifications using eclipse and again build that code and install it on my machine
Thank You
--
*
*
*
Thanx and Regards*
* Vikas Jadhav*
Re: Buiding Hadoop code from source
Posted by Ling Kun <lk...@gmail.com>.
Hi vikas:
Have you asked google first? I searched in Google using your keyword, and
found the following link.
It may be helpful to you.
yours,
Ling Kun
On Fri, Mar 1, 2013 at 4:42 PM, Vikas Jadhav <vi...@gmail.com>wrote:
> Hello,
>
> I want to to build hadoop code downloded from svn or git
>
>
> can someone explicitly tell about these steps.
>
> also i want to do some modifications using eclipse and again build that
> code and install it on my machine
>
> Thank You
>
> --
> *
> *
> *
>
> Thanx and Regards*
> * Vikas Jadhav*
>
--
http://www.lingcc.com
WholeFileInputFormat with streaming
Posted by springring <sp...@126.com>.
Hi,
I want to use:
hadoop jar <my dir>/hadoop-streaming-0.20.2-cdh3u3.jar -inputformat org.apache.hadoop.streaming.WholeFileInputFormat
so, I download code from :
https://github.com/tomwhite/hadoop-book/tree/master/ch07/src/main/java
WholeFileInputFormat.java
WholeFileRecordReader.java
and package the java file with :
package org.apache.hadoop.streaming;
solution A:
copy WholeFileInputFormat.java , WholeFileRecordReader.java to hadoop-0.20.2-cdh3u3/src/contrib/streaming/src/java/org/apache/hadoop/streaming/
then
javac -classpath /usr/lib/hadoop-0.20/hadoop-0.20.2-cdh3u3-core.jar:/usr/lib/hadoop-0.20:/usr/lib/hadoop-0.20/lib/* -d WFInputFormatClassNew hadoop-0.20.2-cdh3u3/src/contrib/streaming/src/java/org/apache/hadoop/streaming/*.java
there is a lot of error
solution B:
compile the java file WholeFileInputFormat.java , WholeFileRecordReader.java:
javac -classpath /usr/lib/hadoop-0.20/hadoop-0.20.2-cdh3u3-core.jar:/usr/lib/hadoop-0.20/*:/usr/lib/hadoop-0.20/lib/* -d WFInputFormatClass <two java file>
copy /usr/lib/hadoop/contrib/streaming/hadoop-streaming-0.20.2-cdh3u3.jar to <my dir>
then :
jar uf hadoop-streaming-0.20.2-cdh3u3.jar WFInputFormatClass/org/apache/hadoop/streaming/WholeFileRecordReader.class
jar uf hadoop-streaming-0.20.2-cdh3u3.jar WFInputFormatClass/org/apache/hadoop/streaming/WholeFileInputFormat.class
there is no error, but when I run:
hadoop jar <my dir>/hadoop-streaming-0.20.2-cdh3u3.jar -inputformat org.apache.hadoop.streaming.WholeFileInputFormat .......
there is error:
-inputformat : class not found : org.apache.hadoop.streaming.WholeFileInputFormat
what's wrong with the two solution? or is there any new solution?
thx.
Ring