You are viewing a plain text version of this content. The canonical link for it is here.
Posted to common-user@hadoop.apache.org by Matthew John <tm...@gmail.com> on 2010/09/08 05:14:04 UTC
How to rebuild Hadoop ??
Hi all,
I wrote some new writable files corresponding to my data input. I added
them to /src/org/......../io/ where all the writables reside. Similarly, I
also wrote input/output format files and a recordreader and added them to
src/mapred/...../mapred/ where all related files reside.
I want to run the Sort function (in examples) with these new classes
(writables, recreader, i/oformat). So I also modified the Sort to
incorporate these files and import them in the Sort.java file. After all
this, I gave a ant clean and then ant command to build everything fresh. But
nothing really happened I guess because when I run the program , it give
ClassNotFoundException for the classes I give as parameters in the command.
Some one please help me out !! How to modify the core/ files (incorporate
more core io/mapred files) in HADOOP !!
Thanks,
Matthew John
Re: How to rebuild Hadoop ??
Posted by Matthew John <tm...@gmail.com>.
Hey Guys ! ,
Finally my examples.jar got built :) :) .. It was just a small error -> dint
initialize the package for some of the newly written files :P ..
Now i will run the command :
bin/hadoop jar hadoop-0.20.2-examples.jar<new one> sort -inFormat
org.apache.hadoop.mapred.MetafileInputFormat -outFormat
org.apache.hadoop.mapred.MetafileOutputFormat -outKey
org.apache.hadoop.io.FpMetaId -outValue org.apache.hadoop.io.FpMetadata
fp_input fp_output
and see what happens !!
Thanks a lot for your time..
Matthew
Re: How to rebuild Hadoop ??
Posted by Matthew John <tm...@gmail.com>.
<target name="examples" depends="jar, compile-examples" description="Make
the Hadoop examples jar.">
<jar jarfile="${build.dir}/${final.name}-examples.jar"
basedir="${build.examples}">
<manifest>
<attribute name="Main-Class"
value="org/apache/hadoop/examples/ExampleDriver"/>
</manifest>
</jar>
</target>
this is part of the build.xml which creates the example.jar. Here (what I
understand) its given that it depends on jar which is the hadoop.core.jar. I
have a feeling its still depending on the older version of core.jar and so
its not able to find the classes which are not updated in the older version
of the core.jar. Therefore it gives ClassNotFound.....
I want to make a new Example jar which depends on the new core.jar. Please
guide me on that and let me know if my understanding is wrong.
Thanks,
Matthew John
Re: How to rebuild Hadoop ??
Posted by Amareshwari Sri Ramadasu <am...@yahoo-inc.com>.
If the jar passed in -libjars is in local filesystem, it is added to client classpath also.
On 9/8/10 3:57 PM, "Jeff Zhang" <zj...@gmail.com> wrote:
Matthew,
InputFormat will been used in client side, so you should combine the
hadoop-0.20.2.jar and hadoop-example.jar into one single jar
On Wed, Sep 8, 2010 at 2:58 AM, Matthew John <tm...@gmail.com> wrote:
> Hey Jeff ,
>
> I gave the command :
>
> bin/hadoop jar hadoop-0.20.2-examples.jar sort -libjars
> ./build/hadoop-0.20.3-dev-core.jar -inFormat
> org.apache.hadoop.mapred.MetafileInputFormat -outFormat
> org.apache.hadoop.mapred.MetafileOutputFormat -outKey
> org.apache.hadoop.io.FpMetaId -outValue org.apache.hadoop.io.FpMetadata
> fp_input fp_output
>
> where hadoop-0.20.3-dev-core.jar is the new core jar (using command ant jar)
> whereas hadoop-0.20.2-examples.jar is still the same old examples jar file
> (I couldnot make the new examples jar using ant examples since I doesnt
> have the latest dependencies on the new classes i have defined). the other
> parameters are the new classes I want to use for running Sort.
> I feel I should make the new examples jar but dont know how to .. :( :( ..
> please tell me how to give new core jar as a parameter to run the ant
> examples.
>
> I am getting the following errors when i ran the command.. :
>
> java.lang.ClassNotFoundException:
> org.apache.hadoop.mapred.MetafileInputFormat.. and so on ...
>
> Thanks,
>
> Matthew
>
--
Best Regards
Jeff Zhang
Re: How to rebuild Hadoop ??
Posted by Jeff Zhang <zj...@gmail.com>.
Matthew,
InputFormat will been used in client side, so you should combine the
hadoop-0.20.2.jar and hadoop-example.jar into one single jar
On Wed, Sep 8, 2010 at 2:58 AM, Matthew John <tm...@gmail.com> wrote:
> Hey Jeff ,
>
> I gave the command :
>
> bin/hadoop jar hadoop-0.20.2-examples.jar sort -libjars
> ./build/hadoop-0.20.3-dev-core.jar -inFormat
> org.apache.hadoop.mapred.MetafileInputFormat -outFormat
> org.apache.hadoop.mapred.MetafileOutputFormat -outKey
> org.apache.hadoop.io.FpMetaId -outValue org.apache.hadoop.io.FpMetadata
> fp_input fp_output
>
> where hadoop-0.20.3-dev-core.jar is the new core jar (using command ant jar)
> whereas hadoop-0.20.2-examples.jar is still the same old examples jar file
> (I couldnot make the new examples jar using ant examples since I doesnt
> have the latest dependencies on the new classes i have defined). the other
> parameters are the new classes I want to use for running Sort.
> I feel I should make the new examples jar but dont know how to .. :( :( ..
> please tell me how to give new core jar as a parameter to run the ant
> examples.
>
> I am getting the following errors when i ran the command.. :
>
> java.lang.ClassNotFoundException:
> org.apache.hadoop.mapred.MetafileInputFormat.. and so on ...
>
> Thanks,
>
> Matthew
>
--
Best Regards
Jeff Zhang
Re: How to rebuild Hadoop ??
Posted by Matthew John <tm...@gmail.com>.
Hey Jeff ,
I gave the command :
bin/hadoop jar hadoop-0.20.2-examples.jar sort -libjars
./build/hadoop-0.20.3-dev-core.jar -inFormat
org.apache.hadoop.mapred.MetafileInputFormat -outFormat
org.apache.hadoop.mapred.MetafileOutputFormat -outKey
org.apache.hadoop.io.FpMetaId -outValue org.apache.hadoop.io.FpMetadata
fp_input fp_output
where hadoop-0.20.3-dev-core.jar is the new core jar (using command ant jar)
whereas hadoop-0.20.2-examples.jar is still the same old examples jar file
(I couldnot make the new examples jar using ant examples since I doesnt
have the latest dependencies on the new classes i have defined). the other
parameters are the new classes I want to use for running Sort.
I feel I should make the new examples jar but dont know how to .. :( :( ..
please tell me how to give new core jar as a parameter to run the ant
examples.
I am getting the following errors when i ran the command.. :
java.lang.ClassNotFoundException:
org.apache.hadoop.mapred.MetafileInputFormat.. and so on ...
Thanks,
Matthew
Re: How to rebuild Hadoop ??
Posted by Jeff Zhang <zj...@gmail.com>.
Mattew,
Do you still put your source code in the io package and mapred package ?
If so, rebuild and hadoop core (seems you have done), and then invoke
the command
bin/hadoop -jar hadoop-0.20.2-example.jar sort -libjars
hadoop-core-0.20.2.jar(your new builded jar) other_arguments
PS: I do not recommend you to put your code mixed with hadoop code,
it's better to put it another places, and build an individual jar.
On Wed, Sep 8, 2010 at 1:12 AM, Matthew John <tm...@gmail.com> wrote:
> Thanks Jeff !
> Following what you have said, I build my hadoop core jar first (command ->
> ant jar). That created a hadoop-core.jar in the build. Now can you please
> tell me how to use this as dependable for the building of examples.jar.
> Because if I give ant example , it gives errors like the new classes I ve
> included in the core are not found. I suppose thats because its using the
> old hadoop-core.jar .
>
> Thanks,
>
> Matthew John
>
--
Best Regards
Jeff Zhang
Re: How to rebuild Hadoop ??
Posted by Matthew John <tm...@gmail.com>.
Thanks Jeff !
Following what you have said, I build my hadoop core jar first (command ->
ant jar). That created a hadoop-core.jar in the build. Now can you please
tell me how to use this as dependable for the building of examples.jar.
Because if I give ant example , it gives errors like the new classes I ve
included in the core are not found. I suppose thats because its using the
old hadoop-core.jar .
Thanks,
Matthew John
Re: How to rebuild Hadoop ??
Posted by Jeff Zhang <zj...@gmail.com>.
Do you run "ant example" ?
On Tue, Sep 7, 2010 at 10:29 PM, Matthew John
<tm...@gmail.com> wrote:
> Thanks a lot Jeff !
>
> The problem is that everytime I build (using ant ) there is a build folder
> created. But there is no examples.jar created inside that. I wanted to add
> some files into io package and mapred package. So I suppose I should put the
> files appropriately ( inside io and mapred folder respectively). I want to
> run the Sort in examples.jar using these added classes. I guess I can import
> these new files in the Sort code and build the entire thing again.
>
> But I am not able to figure out how to rebuild these core containing jar and
> examples jar with the modified sort.
>
--
Best Regards
Jeff Zhang
Re: How to rebuild Hadoop ??
Posted by Matthew John <tm...@gmail.com>.
Thanks a lot Jeff !
The problem is that everytime I build (using ant ) there is a build folder
created. But there is no examples.jar created inside that. I wanted to add
some files into io package and mapred package. So I suppose I should put the
files appropriately ( inside io and mapred folder respectively). I want to
run the Sort in examples.jar using these added classes. I guess I can import
these new files in the Sort code and build the entire thing again.
But I am not able to figure out how to rebuild these core containing jar and
examples jar with the modified sort.
Re: How to rebuild Hadoop ??
Posted by Jeff Zhang <zj...@gmail.com>.
Matthew,
You should put your code in the example source folder, and rebuild the
example. And use the new generated hadoop*version*example.jar in the
build folder.
PS: each mapred job needs a jar which contains the classes this job needs
On Tue, Sep 7, 2010 at 8:14 PM, Matthew John <tm...@gmail.com> wrote:
> Hi all,
>
>
> I wrote some new writable files corresponding to my data input. I added
> them to /src/org/......../io/ where all the writables reside. Similarly, I
> also wrote input/output format files and a recordreader and added them to
> src/mapred/...../mapred/ where all related files reside.
>
> I want to run the Sort function (in examples) with these new classes
> (writables, recreader, i/oformat). So I also modified the Sort to
> incorporate these files and import them in the Sort.java file. After all
> this, I gave a ant clean and then ant command to build everything fresh. But
> nothing really happened I guess because when I run the program , it give
> ClassNotFoundException for the classes I give as parameters in the command.
>
> Some one please help me out !! How to modify the core/ files (incorporate
> more core io/mapred files) in HADOOP !!
>
> Thanks,
>
> Matthew John
>
--
Best Regards
Jeff Zhang