You are viewing a plain text version of this content. The canonical link for it is here.
Posted to hdfs-user@hadoop.apache.org by Pantazis Deligiannis <ak...@gmail.com> on 2012/08/11 03:55:30 UTC
Compiling a mapreduce job under the new Apache Hadoop YARN
In the old Hadoop mapreduce v1 framework (e.g. 1.0.3) the way to compile a job was using the already provided "hadoop-core-HADOOP_VERSION.jar" through the following command:
javac -cp ".../hadoop-core-HADOOP_VERSION.jar" -d class_dir Example.java
In the newest releases for Hadoop mapreduce v2 YARN (e.g. 2.0.1-alpha or the cloudera 4.0) a similar core.jar is not provided, though, to be used during compilation. The closest i have managed to get is by using the "hadoop-common-2.0.1-alpha.jar" in the /share/hadoop/common dir but this still leaves a lot of compilation errors.
Is there some other specific way or using different jars for compiling a YARN job? Or maybe for some reason you need to build the source yourself in the new releases to get the core.jar? I have searched quite a lot but can't find a relevant answer.
Thank you very much!
Re: Compiling a mapreduce job under the new Apache Hadoop YARN
Posted by Pantazis Deligiannis <ak...@gmail.com>.
Worked perfectly!
Thanks a lot! :)
On 11 Aug 2012, at 16:54, Harsh J <ha...@cloudera.com> wrote:
> Hi Pantazis,
>
> It is better to use maven or other such tools to develop MR Java
> programs, as that handles dependencies for you. In maven you may use
> the hadoop-client POM.
>
> Alternatively, if you have a hadoop setup available, do this:
>
> $ javac -cp `hadoop classpath`:. -d class_dir Example.java
>
> The command "hadoop classpath" will add all requisite jars from the
> installed version to the java(c) classpath automatically.
>
> On Sat, Aug 11, 2012 at 7:25 AM, Pantazis Deligiannis
> <ak...@gmail.com> wrote:
>> In the old Hadoop mapreduce v1 framework (e.g. 1.0.3) the way to compile a job was using the already provided "hadoop-core-HADOOP_VERSION.jar" through the following command:
>>
>> javac -cp ".../hadoop-core-HADOOP_VERSION.jar" -d class_dir Example.java
>>
>> In the newest releases for Hadoop mapreduce v2 YARN (e.g. 2.0.1-alpha or the cloudera 4.0) a similar core.jar is not provided, though, to be used during compilation. The closest i have managed to get is by using the "hadoop-common-2.0.1-alpha.jar" in the /share/hadoop/common dir but this still leaves a lot of compilation errors.
>>
>> Is there some other specific way or using different jars for compiling a YARN job? Or maybe for some reason you need to build the source yourself in the new releases to get the core.jar? I have searched quite a lot but can't find a relevant answer.
>>
>> Thank you very much!
>
>
>
> --
> Harsh J
Re: Compiling a mapreduce job under the new Apache Hadoop YARN
Posted by Pantazis Deligiannis <ak...@gmail.com>.
Worked perfectly!
Thanks a lot! :)
On 11 Aug 2012, at 16:54, Harsh J <ha...@cloudera.com> wrote:
> Hi Pantazis,
>
> It is better to use maven or other such tools to develop MR Java
> programs, as that handles dependencies for you. In maven you may use
> the hadoop-client POM.
>
> Alternatively, if you have a hadoop setup available, do this:
>
> $ javac -cp `hadoop classpath`:. -d class_dir Example.java
>
> The command "hadoop classpath" will add all requisite jars from the
> installed version to the java(c) classpath automatically.
>
> On Sat, Aug 11, 2012 at 7:25 AM, Pantazis Deligiannis
> <ak...@gmail.com> wrote:
>> In the old Hadoop mapreduce v1 framework (e.g. 1.0.3) the way to compile a job was using the already provided "hadoop-core-HADOOP_VERSION.jar" through the following command:
>>
>> javac -cp ".../hadoop-core-HADOOP_VERSION.jar" -d class_dir Example.java
>>
>> In the newest releases for Hadoop mapreduce v2 YARN (e.g. 2.0.1-alpha or the cloudera 4.0) a similar core.jar is not provided, though, to be used during compilation. The closest i have managed to get is by using the "hadoop-common-2.0.1-alpha.jar" in the /share/hadoop/common dir but this still leaves a lot of compilation errors.
>>
>> Is there some other specific way or using different jars for compiling a YARN job? Or maybe for some reason you need to build the source yourself in the new releases to get the core.jar? I have searched quite a lot but can't find a relevant answer.
>>
>> Thank you very much!
>
>
>
> --
> Harsh J
Re: Compiling a mapreduce job under the new Apache Hadoop YARN
Posted by Pantazis Deligiannis <ak...@gmail.com>.
Worked perfectly!
Thanks a lot! :)
On 11 Aug 2012, at 16:54, Harsh J <ha...@cloudera.com> wrote:
> Hi Pantazis,
>
> It is better to use maven or other such tools to develop MR Java
> programs, as that handles dependencies for you. In maven you may use
> the hadoop-client POM.
>
> Alternatively, if you have a hadoop setup available, do this:
>
> $ javac -cp `hadoop classpath`:. -d class_dir Example.java
>
> The command "hadoop classpath" will add all requisite jars from the
> installed version to the java(c) classpath automatically.
>
> On Sat, Aug 11, 2012 at 7:25 AM, Pantazis Deligiannis
> <ak...@gmail.com> wrote:
>> In the old Hadoop mapreduce v1 framework (e.g. 1.0.3) the way to compile a job was using the already provided "hadoop-core-HADOOP_VERSION.jar" through the following command:
>>
>> javac -cp ".../hadoop-core-HADOOP_VERSION.jar" -d class_dir Example.java
>>
>> In the newest releases for Hadoop mapreduce v2 YARN (e.g. 2.0.1-alpha or the cloudera 4.0) a similar core.jar is not provided, though, to be used during compilation. The closest i have managed to get is by using the "hadoop-common-2.0.1-alpha.jar" in the /share/hadoop/common dir but this still leaves a lot of compilation errors.
>>
>> Is there some other specific way or using different jars for compiling a YARN job? Or maybe for some reason you need to build the source yourself in the new releases to get the core.jar? I have searched quite a lot but can't find a relevant answer.
>>
>> Thank you very much!
>
>
>
> --
> Harsh J
Re: Compiling a mapreduce job under the new Apache Hadoop YARN
Posted by Pantazis Deligiannis <ak...@gmail.com>.
Worked perfectly!
Thanks a lot! :)
On 11 Aug 2012, at 16:54, Harsh J <ha...@cloudera.com> wrote:
> Hi Pantazis,
>
> It is better to use maven or other such tools to develop MR Java
> programs, as that handles dependencies for you. In maven you may use
> the hadoop-client POM.
>
> Alternatively, if you have a hadoop setup available, do this:
>
> $ javac -cp `hadoop classpath`:. -d class_dir Example.java
>
> The command "hadoop classpath" will add all requisite jars from the
> installed version to the java(c) classpath automatically.
>
> On Sat, Aug 11, 2012 at 7:25 AM, Pantazis Deligiannis
> <ak...@gmail.com> wrote:
>> In the old Hadoop mapreduce v1 framework (e.g. 1.0.3) the way to compile a job was using the already provided "hadoop-core-HADOOP_VERSION.jar" through the following command:
>>
>> javac -cp ".../hadoop-core-HADOOP_VERSION.jar" -d class_dir Example.java
>>
>> In the newest releases for Hadoop mapreduce v2 YARN (e.g. 2.0.1-alpha or the cloudera 4.0) a similar core.jar is not provided, though, to be used during compilation. The closest i have managed to get is by using the "hadoop-common-2.0.1-alpha.jar" in the /share/hadoop/common dir but this still leaves a lot of compilation errors.
>>
>> Is there some other specific way or using different jars for compiling a YARN job? Or maybe for some reason you need to build the source yourself in the new releases to get the core.jar? I have searched quite a lot but can't find a relevant answer.
>>
>> Thank you very much!
>
>
>
> --
> Harsh J
Re: Compiling a mapreduce job under the new Apache Hadoop YARN
Posted by Harsh J <ha...@cloudera.com>.
Hi Pantazis,
It is better to use maven or other such tools to develop MR Java
programs, as that handles dependencies for you. In maven you may use
the hadoop-client POM.
Alternatively, if you have a hadoop setup available, do this:
$ javac -cp `hadoop classpath`:. -d class_dir Example.java
The command "hadoop classpath" will add all requisite jars from the
installed version to the java(c) classpath automatically.
On Sat, Aug 11, 2012 at 7:25 AM, Pantazis Deligiannis
<ak...@gmail.com> wrote:
> In the old Hadoop mapreduce v1 framework (e.g. 1.0.3) the way to compile a job was using the already provided "hadoop-core-HADOOP_VERSION.jar" through the following command:
>
> javac -cp ".../hadoop-core-HADOOP_VERSION.jar" -d class_dir Example.java
>
> In the newest releases for Hadoop mapreduce v2 YARN (e.g. 2.0.1-alpha or the cloudera 4.0) a similar core.jar is not provided, though, to be used during compilation. The closest i have managed to get is by using the "hadoop-common-2.0.1-alpha.jar" in the /share/hadoop/common dir but this still leaves a lot of compilation errors.
>
> Is there some other specific way or using different jars for compiling a YARN job? Or maybe for some reason you need to build the source yourself in the new releases to get the core.jar? I have searched quite a lot but can't find a relevant answer.
>
> Thank you very much!
--
Harsh J
Re: Compiling a mapreduce job under the new Apache Hadoop YARN
Posted by Harsh J <ha...@cloudera.com>.
Hi Pantazis,
It is better to use maven or other such tools to develop MR Java
programs, as that handles dependencies for you. In maven you may use
the hadoop-client POM.
Alternatively, if you have a hadoop setup available, do this:
$ javac -cp `hadoop classpath`:. -d class_dir Example.java
The command "hadoop classpath" will add all requisite jars from the
installed version to the java(c) classpath automatically.
On Sat, Aug 11, 2012 at 7:25 AM, Pantazis Deligiannis
<ak...@gmail.com> wrote:
> In the old Hadoop mapreduce v1 framework (e.g. 1.0.3) the way to compile a job was using the already provided "hadoop-core-HADOOP_VERSION.jar" through the following command:
>
> javac -cp ".../hadoop-core-HADOOP_VERSION.jar" -d class_dir Example.java
>
> In the newest releases for Hadoop mapreduce v2 YARN (e.g. 2.0.1-alpha or the cloudera 4.0) a similar core.jar is not provided, though, to be used during compilation. The closest i have managed to get is by using the "hadoop-common-2.0.1-alpha.jar" in the /share/hadoop/common dir but this still leaves a lot of compilation errors.
>
> Is there some other specific way or using different jars for compiling a YARN job? Or maybe for some reason you need to build the source yourself in the new releases to get the core.jar? I have searched quite a lot but can't find a relevant answer.
>
> Thank you very much!
--
Harsh J
Re: Compiling a mapreduce job under the new Apache Hadoop YARN
Posted by Harsh J <ha...@cloudera.com>.
Hi Pantazis,
It is better to use maven or other such tools to develop MR Java
programs, as that handles dependencies for you. In maven you may use
the hadoop-client POM.
Alternatively, if you have a hadoop setup available, do this:
$ javac -cp `hadoop classpath`:. -d class_dir Example.java
The command "hadoop classpath" will add all requisite jars from the
installed version to the java(c) classpath automatically.
On Sat, Aug 11, 2012 at 7:25 AM, Pantazis Deligiannis
<ak...@gmail.com> wrote:
> In the old Hadoop mapreduce v1 framework (e.g. 1.0.3) the way to compile a job was using the already provided "hadoop-core-HADOOP_VERSION.jar" through the following command:
>
> javac -cp ".../hadoop-core-HADOOP_VERSION.jar" -d class_dir Example.java
>
> In the newest releases for Hadoop mapreduce v2 YARN (e.g. 2.0.1-alpha or the cloudera 4.0) a similar core.jar is not provided, though, to be used during compilation. The closest i have managed to get is by using the "hadoop-common-2.0.1-alpha.jar" in the /share/hadoop/common dir but this still leaves a lot of compilation errors.
>
> Is there some other specific way or using different jars for compiling a YARN job? Or maybe for some reason you need to build the source yourself in the new releases to get the core.jar? I have searched quite a lot but can't find a relevant answer.
>
> Thank you very much!
--
Harsh J
Re: Compiling a mapreduce job under the new Apache Hadoop YARN
Posted by Harsh J <ha...@cloudera.com>.
Hi Pantazis,
It is better to use maven or other such tools to develop MR Java
programs, as that handles dependencies for you. In maven you may use
the hadoop-client POM.
Alternatively, if you have a hadoop setup available, do this:
$ javac -cp `hadoop classpath`:. -d class_dir Example.java
The command "hadoop classpath" will add all requisite jars from the
installed version to the java(c) classpath automatically.
On Sat, Aug 11, 2012 at 7:25 AM, Pantazis Deligiannis
<ak...@gmail.com> wrote:
> In the old Hadoop mapreduce v1 framework (e.g. 1.0.3) the way to compile a job was using the already provided "hadoop-core-HADOOP_VERSION.jar" through the following command:
>
> javac -cp ".../hadoop-core-HADOOP_VERSION.jar" -d class_dir Example.java
>
> In the newest releases for Hadoop mapreduce v2 YARN (e.g. 2.0.1-alpha or the cloudera 4.0) a similar core.jar is not provided, though, to be used during compilation. The closest i have managed to get is by using the "hadoop-common-2.0.1-alpha.jar" in the /share/hadoop/common dir but this still leaves a lot of compilation errors.
>
> Is there some other specific way or using different jars for compiling a YARN job? Or maybe for some reason you need to build the source yourself in the new releases to get the core.jar? I have searched quite a lot but can't find a relevant answer.
>
> Thank you very much!
--
Harsh J