You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@hadoop.apache.org by John Lilley <jo...@redpoint.net> on 2013/06/05 19:37:56 UTC
Hadoop JARs and Eclipse
Well, I've failed and given up on building Hadoop in Eclipse. Too many things go wrong with Maven plugins and m2e.
But Hadoop builds just fine using the command-line, and it runs using Sandy's development-node instructions.
My strategy now is
1) Tell Eclipse about all of the Hadoop JARs
2) Make a project with the sample ApplicationMaster distributed shell and build it.
Is there a shortcut for adding all relevant JARs to Eclipse? They ended up in a lot of separate folders after the build and I can't tell if there is one "target" containing all of the ones I need.
Thanks
John
RE: Hadoop JARs and Eclipse
Posted by John Lilley <jo...@redpoint.net>.
Never mind, answered my own question... I just make a new maven project and add a dependency to the pom.xml, I checked in ~/.m2 that the hadoop-client module is there.
john
-----Original Message-----
From: John Lilley [mailto:john.lilley@redpoint.net]
Sent: Thursday, June 06, 2013 11:06 AM
To: user@hadoop.apache.org
Subject: RE: Hadoop JARs and Eclipse
If I am running from built code that was installed with mvn install Do I need to change the pom.xml file to reflect that I am using local repository?
Thanks,
John
-----Original Message-----
From: Harsh J [mailto:harsh@cloudera.com]
Sent: Wednesday, June 05, 2013 11:43 AM
To: <us...@hadoop.apache.org>
Subject: Re: Hadoop JARs and Eclipse
If your goal is to simply build an application, then you can use a Maven project. Why do you require the whole of Hadoop's projects itself on Eclipse when you can simply have the dependencies via a maven pom.xml?
The following is what you can use in a simple maven app, to include all necessary dependencies automatically, for example:
<dependencies>
<dependency>
<groupId>org.apache.hadoop</groupId>
<artifactId>hadoop-client</artifactId>
<version>2.0.4-alpha</version>
</dependency>
</dependencies>
Then importing this project into Eclipse (using m2e plugin) would be equivalent to "Adding all relevant Hadoop JARs to Eclipse".
On Wed, Jun 5, 2013 at 11:07 PM, John Lilley <jo...@redpoint.net> wrote:
> Well, I've failed and given up on building Hadoop in Eclipse. Too
> many things go wrong with Maven plugins and m2e.
>
> But Hadoop builds just fine using the command-line, and it runs using
> Sandy's development-node instructions.
>
> My strategy now is
>
> 1) Tell Eclipse about all of the Hadoop JARs
>
> 2) Make a project with the sample ApplicationMaster distributed shell
> and build it.
>
> Is there a shortcut for adding all relevant JARs to Eclipse? They
> ended up in a lot of separate folders after the build and I can't tell
> if there is one "target" containing all of the ones I need.
>
> Thanks
>
> John
>
>
>
>
--
Harsh J
RE: Hadoop JARs and Eclipse
Posted by John Lilley <jo...@redpoint.net>.
Never mind, answered my own question... I just make a new maven project and add a dependency to the pom.xml, I checked in ~/.m2 that the hadoop-client module is there.
john
-----Original Message-----
From: John Lilley [mailto:john.lilley@redpoint.net]
Sent: Thursday, June 06, 2013 11:06 AM
To: user@hadoop.apache.org
Subject: RE: Hadoop JARs and Eclipse
If I am running from built code that was installed with mvn install Do I need to change the pom.xml file to reflect that I am using local repository?
Thanks,
John
-----Original Message-----
From: Harsh J [mailto:harsh@cloudera.com]
Sent: Wednesday, June 05, 2013 11:43 AM
To: <us...@hadoop.apache.org>
Subject: Re: Hadoop JARs and Eclipse
If your goal is to simply build an application, then you can use a Maven project. Why do you require the whole of Hadoop's projects itself on Eclipse when you can simply have the dependencies via a maven pom.xml?
The following is what you can use in a simple maven app, to include all necessary dependencies automatically, for example:
<dependencies>
<dependency>
<groupId>org.apache.hadoop</groupId>
<artifactId>hadoop-client</artifactId>
<version>2.0.4-alpha</version>
</dependency>
</dependencies>
Then importing this project into Eclipse (using m2e plugin) would be equivalent to "Adding all relevant Hadoop JARs to Eclipse".
On Wed, Jun 5, 2013 at 11:07 PM, John Lilley <jo...@redpoint.net> wrote:
> Well, I've failed and given up on building Hadoop in Eclipse. Too
> many things go wrong with Maven plugins and m2e.
>
> But Hadoop builds just fine using the command-line, and it runs using
> Sandy's development-node instructions.
>
> My strategy now is
>
> 1) Tell Eclipse about all of the Hadoop JARs
>
> 2) Make a project with the sample ApplicationMaster distributed shell
> and build it.
>
> Is there a shortcut for adding all relevant JARs to Eclipse? They
> ended up in a lot of separate folders after the build and I can't tell
> if there is one "target" containing all of the ones I need.
>
> Thanks
>
> John
>
>
>
>
--
Harsh J
RE: Hadoop JARs and Eclipse
Posted by John Lilley <jo...@redpoint.net>.
Never mind, answered my own question... I just make a new maven project and add a dependency to the pom.xml, I checked in ~/.m2 that the hadoop-client module is there.
john
-----Original Message-----
From: John Lilley [mailto:john.lilley@redpoint.net]
Sent: Thursday, June 06, 2013 11:06 AM
To: user@hadoop.apache.org
Subject: RE: Hadoop JARs and Eclipse
If I am running from built code that was installed with mvn install Do I need to change the pom.xml file to reflect that I am using local repository?
Thanks,
John
-----Original Message-----
From: Harsh J [mailto:harsh@cloudera.com]
Sent: Wednesday, June 05, 2013 11:43 AM
To: <us...@hadoop.apache.org>
Subject: Re: Hadoop JARs and Eclipse
If your goal is to simply build an application, then you can use a Maven project. Why do you require the whole of Hadoop's projects itself on Eclipse when you can simply have the dependencies via a maven pom.xml?
The following is what you can use in a simple maven app, to include all necessary dependencies automatically, for example:
<dependencies>
<dependency>
<groupId>org.apache.hadoop</groupId>
<artifactId>hadoop-client</artifactId>
<version>2.0.4-alpha</version>
</dependency>
</dependencies>
Then importing this project into Eclipse (using m2e plugin) would be equivalent to "Adding all relevant Hadoop JARs to Eclipse".
On Wed, Jun 5, 2013 at 11:07 PM, John Lilley <jo...@redpoint.net> wrote:
> Well, I've failed and given up on building Hadoop in Eclipse. Too
> many things go wrong with Maven plugins and m2e.
>
> But Hadoop builds just fine using the command-line, and it runs using
> Sandy's development-node instructions.
>
> My strategy now is
>
> 1) Tell Eclipse about all of the Hadoop JARs
>
> 2) Make a project with the sample ApplicationMaster distributed shell
> and build it.
>
> Is there a shortcut for adding all relevant JARs to Eclipse? They
> ended up in a lot of separate folders after the build and I can't tell
> if there is one "target" containing all of the ones I need.
>
> Thanks
>
> John
>
>
>
>
--
Harsh J
RE: Hadoop JARs and Eclipse
Posted by John Lilley <jo...@redpoint.net>.
Never mind, answered my own question... I just make a new maven project and add a dependency to the pom.xml, I checked in ~/.m2 that the hadoop-client module is there.
john
-----Original Message-----
From: John Lilley [mailto:john.lilley@redpoint.net]
Sent: Thursday, June 06, 2013 11:06 AM
To: user@hadoop.apache.org
Subject: RE: Hadoop JARs and Eclipse
If I am running from built code that was installed with mvn install Do I need to change the pom.xml file to reflect that I am using local repository?
Thanks,
John
-----Original Message-----
From: Harsh J [mailto:harsh@cloudera.com]
Sent: Wednesday, June 05, 2013 11:43 AM
To: <us...@hadoop.apache.org>
Subject: Re: Hadoop JARs and Eclipse
If your goal is to simply build an application, then you can use a Maven project. Why do you require the whole of Hadoop's projects itself on Eclipse when you can simply have the dependencies via a maven pom.xml?
The following is what you can use in a simple maven app, to include all necessary dependencies automatically, for example:
<dependencies>
<dependency>
<groupId>org.apache.hadoop</groupId>
<artifactId>hadoop-client</artifactId>
<version>2.0.4-alpha</version>
</dependency>
</dependencies>
Then importing this project into Eclipse (using m2e plugin) would be equivalent to "Adding all relevant Hadoop JARs to Eclipse".
On Wed, Jun 5, 2013 at 11:07 PM, John Lilley <jo...@redpoint.net> wrote:
> Well, I've failed and given up on building Hadoop in Eclipse. Too
> many things go wrong with Maven plugins and m2e.
>
> But Hadoop builds just fine using the command-line, and it runs using
> Sandy's development-node instructions.
>
> My strategy now is
>
> 1) Tell Eclipse about all of the Hadoop JARs
>
> 2) Make a project with the sample ApplicationMaster distributed shell
> and build it.
>
> Is there a shortcut for adding all relevant JARs to Eclipse? They
> ended up in a lot of separate folders after the build and I can't tell
> if there is one "target" containing all of the ones I need.
>
> Thanks
>
> John
>
>
>
>
--
Harsh J
RE: Hadoop JARs and Eclipse
Posted by John Lilley <jo...@redpoint.net>.
If I am running from built code that was installed with
mvn install
Do I need to change the pom.xml file to reflect that I am using local repository?
Thanks,
John
-----Original Message-----
From: Harsh J [mailto:harsh@cloudera.com]
Sent: Wednesday, June 05, 2013 11:43 AM
To: <us...@hadoop.apache.org>
Subject: Re: Hadoop JARs and Eclipse
If your goal is to simply build an application, then you can use a Maven project. Why do you require the whole of Hadoop's projects itself on Eclipse when you can simply have the dependencies via a maven pom.xml?
The following is what you can use in a simple maven app, to include all necessary dependencies automatically, for example:
<dependencies>
<dependency>
<groupId>org.apache.hadoop</groupId>
<artifactId>hadoop-client</artifactId>
<version>2.0.4-alpha</version>
</dependency>
</dependencies>
Then importing this project into Eclipse (using m2e plugin) would be equivalent to "Adding all relevant Hadoop JARs to Eclipse".
On Wed, Jun 5, 2013 at 11:07 PM, John Lilley <jo...@redpoint.net> wrote:
> Well, I've failed and given up on building Hadoop in Eclipse. Too
> many things go wrong with Maven plugins and m2e.
>
> But Hadoop builds just fine using the command-line, and it runs using
> Sandy's development-node instructions.
>
> My strategy now is
>
> 1) Tell Eclipse about all of the Hadoop JARs
>
> 2) Make a project with the sample ApplicationMaster distributed shell
> and build it.
>
> Is there a shortcut for adding all relevant JARs to Eclipse? They
> ended up in a lot of separate folders after the build and I can't tell
> if there is one "target" containing all of the ones I need.
>
> Thanks
>
> John
>
>
>
>
--
Harsh J
RE: Hadoop JARs and Eclipse
Posted by John Lilley <jo...@redpoint.net>.
Harsh,
Thanks so much for your thorough explanation.
John
-----Original Message-----
From: Harsh J [mailto:harsh@cloudera.com]
Sent: Wednesday, June 05, 2013 8:18 PM
To: <us...@hadoop.apache.org>
Subject: Re: Hadoop JARs and Eclipse
Hi John,
On Thu, Jun 6, 2013 at 1:21 AM, John Lilley <jo...@redpoint.net> wrote:
> -- From where will it fetch the Hadoop JARs?
>From the Maven Central repository (we publish our jars and dependencies are also available there), or a custom defined repository if you lack internet access.
> -- If I have built Hadoop from source, how do I tell it to find my just-built JARs instead of those from the official release site?
If you built Hadoop locally with "mvn install", then their version is added to your local repository (~/.m2) and can be "referred' to after a successful "mvn install".
> -- Is it necessary to build Hadoop from source so that the Eclipse debugger can locate source files and display them if I need to step in?
No, see below.
> -- Step into the source code of referenced Hadoop code in the debugger.
This one is easy with the Eclipse Maven IDE plugin, see http://stackoverflow.com/questions/310720/get-source-jar-files-attached-to-eclipse-for-maven-managed-dependencies.
This lets you view and also debug-step into sources of dependencies.
> -- Make changes to the Hadoop source and rebuild it if things are not working as expected (e.g. to add logging).
> -- Switch to building from SVN instead of a release snapshot, and apply patches.
If you're wanting to use the branch-2 tip or the trunk tip, then they carry their own version (2.1.0-beta-SNAPSHOT, or 3.0.0-SNAPSHOT and such). Referring these versions in the pom.xml of your project would be the first step (you may also have to enable discovery of snapshot releases?). Second would be to make the changes to the hadoop source and running a global "mvn install" to update the local repository
(~/.m2) versions to the patched ones. Subsequently, your project will be able to find it.
What am saying is you can do this without having to manage Hadoop as an eclipse project - just patch and build (plus install) on CLI, but mark your project to reference their exact SNAPSHOT version so it discovers the new built jars.
--
Harsh J
RE: Hadoop JARs and Eclipse
Posted by John Lilley <jo...@redpoint.net>.
Harsh,
Thanks so much for your thorough explanation.
John
-----Original Message-----
From: Harsh J [mailto:harsh@cloudera.com]
Sent: Wednesday, June 05, 2013 8:18 PM
To: <us...@hadoop.apache.org>
Subject: Re: Hadoop JARs and Eclipse
Hi John,
On Thu, Jun 6, 2013 at 1:21 AM, John Lilley <jo...@redpoint.net> wrote:
> -- From where will it fetch the Hadoop JARs?
>From the Maven Central repository (we publish our jars and dependencies are also available there), or a custom defined repository if you lack internet access.
> -- If I have built Hadoop from source, how do I tell it to find my just-built JARs instead of those from the official release site?
If you built Hadoop locally with "mvn install", then their version is added to your local repository (~/.m2) and can be "referred' to after a successful "mvn install".
> -- Is it necessary to build Hadoop from source so that the Eclipse debugger can locate source files and display them if I need to step in?
No, see below.
> -- Step into the source code of referenced Hadoop code in the debugger.
This one is easy with the Eclipse Maven IDE plugin, see http://stackoverflow.com/questions/310720/get-source-jar-files-attached-to-eclipse-for-maven-managed-dependencies.
This lets you view and also debug-step into sources of dependencies.
> -- Make changes to the Hadoop source and rebuild it if things are not working as expected (e.g. to add logging).
> -- Switch to building from SVN instead of a release snapshot, and apply patches.
If you're wanting to use the branch-2 tip or the trunk tip, then they carry their own version (2.1.0-beta-SNAPSHOT, or 3.0.0-SNAPSHOT and such). Referring these versions in the pom.xml of your project would be the first step (you may also have to enable discovery of snapshot releases?). Second would be to make the changes to the hadoop source and running a global "mvn install" to update the local repository
(~/.m2) versions to the patched ones. Subsequently, your project will be able to find it.
What am saying is you can do this without having to manage Hadoop as an eclipse project - just patch and build (plus install) on CLI, but mark your project to reference their exact SNAPSHOT version so it discovers the new built jars.
--
Harsh J
RE: Hadoop JARs and Eclipse
Posted by John Lilley <jo...@redpoint.net>.
Harsh,
Thanks so much for your thorough explanation.
John
-----Original Message-----
From: Harsh J [mailto:harsh@cloudera.com]
Sent: Wednesday, June 05, 2013 8:18 PM
To: <us...@hadoop.apache.org>
Subject: Re: Hadoop JARs and Eclipse
Hi John,
On Thu, Jun 6, 2013 at 1:21 AM, John Lilley <jo...@redpoint.net> wrote:
> -- From where will it fetch the Hadoop JARs?
>From the Maven Central repository (we publish our jars and dependencies are also available there), or a custom defined repository if you lack internet access.
> -- If I have built Hadoop from source, how do I tell it to find my just-built JARs instead of those from the official release site?
If you built Hadoop locally with "mvn install", then their version is added to your local repository (~/.m2) and can be "referred' to after a successful "mvn install".
> -- Is it necessary to build Hadoop from source so that the Eclipse debugger can locate source files and display them if I need to step in?
No, see below.
> -- Step into the source code of referenced Hadoop code in the debugger.
This one is easy with the Eclipse Maven IDE plugin, see http://stackoverflow.com/questions/310720/get-source-jar-files-attached-to-eclipse-for-maven-managed-dependencies.
This lets you view and also debug-step into sources of dependencies.
> -- Make changes to the Hadoop source and rebuild it if things are not working as expected (e.g. to add logging).
> -- Switch to building from SVN instead of a release snapshot, and apply patches.
If you're wanting to use the branch-2 tip or the trunk tip, then they carry their own version (2.1.0-beta-SNAPSHOT, or 3.0.0-SNAPSHOT and such). Referring these versions in the pom.xml of your project would be the first step (you may also have to enable discovery of snapshot releases?). Second would be to make the changes to the hadoop source and running a global "mvn install" to update the local repository
(~/.m2) versions to the patched ones. Subsequently, your project will be able to find it.
What am saying is you can do this without having to manage Hadoop as an eclipse project - just patch and build (plus install) on CLI, but mark your project to reference their exact SNAPSHOT version so it discovers the new built jars.
--
Harsh J
RE: Hadoop JARs and Eclipse
Posted by John Lilley <jo...@redpoint.net>.
Harsh,
Thanks so much for your thorough explanation.
John
-----Original Message-----
From: Harsh J [mailto:harsh@cloudera.com]
Sent: Wednesday, June 05, 2013 8:18 PM
To: <us...@hadoop.apache.org>
Subject: Re: Hadoop JARs and Eclipse
Hi John,
On Thu, Jun 6, 2013 at 1:21 AM, John Lilley <jo...@redpoint.net> wrote:
> -- From where will it fetch the Hadoop JARs?
>From the Maven Central repository (we publish our jars and dependencies are also available there), or a custom defined repository if you lack internet access.
> -- If I have built Hadoop from source, how do I tell it to find my just-built JARs instead of those from the official release site?
If you built Hadoop locally with "mvn install", then their version is added to your local repository (~/.m2) and can be "referred' to after a successful "mvn install".
> -- Is it necessary to build Hadoop from source so that the Eclipse debugger can locate source files and display them if I need to step in?
No, see below.
> -- Step into the source code of referenced Hadoop code in the debugger.
This one is easy with the Eclipse Maven IDE plugin, see http://stackoverflow.com/questions/310720/get-source-jar-files-attached-to-eclipse-for-maven-managed-dependencies.
This lets you view and also debug-step into sources of dependencies.
> -- Make changes to the Hadoop source and rebuild it if things are not working as expected (e.g. to add logging).
> -- Switch to building from SVN instead of a release snapshot, and apply patches.
If you're wanting to use the branch-2 tip or the trunk tip, then they carry their own version (2.1.0-beta-SNAPSHOT, or 3.0.0-SNAPSHOT and such). Referring these versions in the pom.xml of your project would be the first step (you may also have to enable discovery of snapshot releases?). Second would be to make the changes to the hadoop source and running a global "mvn install" to update the local repository
(~/.m2) versions to the patched ones. Subsequently, your project will be able to find it.
What am saying is you can do this without having to manage Hadoop as an eclipse project - just patch and build (plus install) on CLI, but mark your project to reference their exact SNAPSHOT version so it discovers the new built jars.
--
Harsh J
Re: Hadoop JARs and Eclipse
Posted by Harsh J <ha...@cloudera.com>.
Hi John,
On Thu, Jun 6, 2013 at 1:21 AM, John Lilley <jo...@redpoint.net> wrote:
> -- From where will it fetch the Hadoop JARs?
>From the Maven Central repository (we publish our jars and
dependencies are also available there), or a custom defined repository
if you lack internet access.
> -- If I have built Hadoop from source, how do I tell it to find my just-built JARs instead of those from the official release site?
If you built Hadoop locally with "mvn install", then their version is
added to your local repository (~/.m2) and can be "referred' to after
a successful "mvn install".
> -- Is it necessary to build Hadoop from source so that the Eclipse debugger can locate source files and display them if I need to step in?
No, see below.
> -- Step into the source code of referenced Hadoop code in the debugger.
This one is easy with the Eclipse Maven IDE plugin, see
http://stackoverflow.com/questions/310720/get-source-jar-files-attached-to-eclipse-for-maven-managed-dependencies.
This lets you view and also debug-step into sources of dependencies.
> -- Make changes to the Hadoop source and rebuild it if things are not working as expected (e.g. to add logging).
> -- Switch to building from SVN instead of a release snapshot, and apply patches.
If you're wanting to use the branch-2 tip or the trunk tip, then they
carry their own version (2.1.0-beta-SNAPSHOT, or 3.0.0-SNAPSHOT and
such). Referring these versions in the pom.xml of your project would
be the first step (you may also have to enable discovery of snapshot
releases?). Second would be to make the changes to the hadoop source
and running a global "mvn install" to update the local repository
(~/.m2) versions to the patched ones. Subsequently, your project will
be able to find it.
What am saying is you can do this without having to manage Hadoop as
an eclipse project - just patch and build (plus install) on CLI, but
mark your project to reference their exact SNAPSHOT version so it
discovers the new built jars.
--
Harsh J
Re: Hadoop JARs and Eclipse
Posted by Harsh J <ha...@cloudera.com>.
Hi John,
On Thu, Jun 6, 2013 at 1:21 AM, John Lilley <jo...@redpoint.net> wrote:
> -- From where will it fetch the Hadoop JARs?
>From the Maven Central repository (we publish our jars and
dependencies are also available there), or a custom defined repository
if you lack internet access.
> -- If I have built Hadoop from source, how do I tell it to find my just-built JARs instead of those from the official release site?
If you built Hadoop locally with "mvn install", then their version is
added to your local repository (~/.m2) and can be "referred' to after
a successful "mvn install".
> -- Is it necessary to build Hadoop from source so that the Eclipse debugger can locate source files and display them if I need to step in?
No, see below.
> -- Step into the source code of referenced Hadoop code in the debugger.
This one is easy with the Eclipse Maven IDE plugin, see
http://stackoverflow.com/questions/310720/get-source-jar-files-attached-to-eclipse-for-maven-managed-dependencies.
This lets you view and also debug-step into sources of dependencies.
> -- Make changes to the Hadoop source and rebuild it if things are not working as expected (e.g. to add logging).
> -- Switch to building from SVN instead of a release snapshot, and apply patches.
If you're wanting to use the branch-2 tip or the trunk tip, then they
carry their own version (2.1.0-beta-SNAPSHOT, or 3.0.0-SNAPSHOT and
such). Referring these versions in the pom.xml of your project would
be the first step (you may also have to enable discovery of snapshot
releases?). Second would be to make the changes to the hadoop source
and running a global "mvn install" to update the local repository
(~/.m2) versions to the patched ones. Subsequently, your project will
be able to find it.
What am saying is you can do this without having to manage Hadoop as
an eclipse project - just patch and build (plus install) on CLI, but
mark your project to reference their exact SNAPSHOT version so it
discovers the new built jars.
--
Harsh J
Re: Hadoop JARs and Eclipse
Posted by Harsh J <ha...@cloudera.com>.
Hi John,
On Thu, Jun 6, 2013 at 1:21 AM, John Lilley <jo...@redpoint.net> wrote:
> -- From where will it fetch the Hadoop JARs?
>From the Maven Central repository (we publish our jars and
dependencies are also available there), or a custom defined repository
if you lack internet access.
> -- If I have built Hadoop from source, how do I tell it to find my just-built JARs instead of those from the official release site?
If you built Hadoop locally with "mvn install", then their version is
added to your local repository (~/.m2) and can be "referred' to after
a successful "mvn install".
> -- Is it necessary to build Hadoop from source so that the Eclipse debugger can locate source files and display them if I need to step in?
No, see below.
> -- Step into the source code of referenced Hadoop code in the debugger.
This one is easy with the Eclipse Maven IDE plugin, see
http://stackoverflow.com/questions/310720/get-source-jar-files-attached-to-eclipse-for-maven-managed-dependencies.
This lets you view and also debug-step into sources of dependencies.
> -- Make changes to the Hadoop source and rebuild it if things are not working as expected (e.g. to add logging).
> -- Switch to building from SVN instead of a release snapshot, and apply patches.
If you're wanting to use the branch-2 tip or the trunk tip, then they
carry their own version (2.1.0-beta-SNAPSHOT, or 3.0.0-SNAPSHOT and
such). Referring these versions in the pom.xml of your project would
be the first step (you may also have to enable discovery of snapshot
releases?). Second would be to make the changes to the hadoop source
and running a global "mvn install" to update the local repository
(~/.m2) versions to the patched ones. Subsequently, your project will
be able to find it.
What am saying is you can do this without having to manage Hadoop as
an eclipse project - just patch and build (plus install) on CLI, but
mark your project to reference their exact SNAPSHOT version so it
discovers the new built jars.
--
Harsh J
Re: Hadoop JARs and Eclipse
Posted by Harsh J <ha...@cloudera.com>.
Hi John,
On Thu, Jun 6, 2013 at 1:21 AM, John Lilley <jo...@redpoint.net> wrote:
> -- From where will it fetch the Hadoop JARs?
>From the Maven Central repository (we publish our jars and
dependencies are also available there), or a custom defined repository
if you lack internet access.
> -- If I have built Hadoop from source, how do I tell it to find my just-built JARs instead of those from the official release site?
If you built Hadoop locally with "mvn install", then their version is
added to your local repository (~/.m2) and can be "referred' to after
a successful "mvn install".
> -- Is it necessary to build Hadoop from source so that the Eclipse debugger can locate source files and display them if I need to step in?
No, see below.
> -- Step into the source code of referenced Hadoop code in the debugger.
This one is easy with the Eclipse Maven IDE plugin, see
http://stackoverflow.com/questions/310720/get-source-jar-files-attached-to-eclipse-for-maven-managed-dependencies.
This lets you view and also debug-step into sources of dependencies.
> -- Make changes to the Hadoop source and rebuild it if things are not working as expected (e.g. to add logging).
> -- Switch to building from SVN instead of a release snapshot, and apply patches.
If you're wanting to use the branch-2 tip or the trunk tip, then they
carry their own version (2.1.0-beta-SNAPSHOT, or 3.0.0-SNAPSHOT and
such). Referring these versions in the pom.xml of your project would
be the first step (you may also have to enable discovery of snapshot
releases?). Second would be to make the changes to the hadoop source
and running a global "mvn install" to update the local repository
(~/.m2) versions to the patched ones. Subsequently, your project will
be able to find it.
What am saying is you can do this without having to manage Hadoop as
an eclipse project - just patch and build (plus install) on CLI, but
mark your project to reference their exact SNAPSHOT version so it
discovers the new built jars.
--
Harsh J
RE: Hadoop JARs and Eclipse
Posted by John Lilley <jo...@redpoint.net>.
Harsh,
Thanks for your response. I don't need the whole of Hadoop projects in Eclipse, I just need to be able to build Hadoop and step into its source files. Please help me better understand what you recommend as I have a very cursory understanding of maven and its interaction with Eclipse.
-- From where will it fetch the Hadoop JARs?
-- If I have built Hadoop from source, how do I tell it to find my just-built JARs instead of those from the official release site?
-- Is it necessary to build Hadoop from source so that the Eclipse debugger can locate source files and display them if I need to step in?
My goals are to build an application on Hadoop 2.0. Currently I am using the 2.0.4 alpha release. Eventually I this application will include an ApplicationMaster, auxiliary service, and a pile of embedded code and data. However, I don't believe that I will be able to use Hadoop releases as-is without trouble. I suspect that the following will be necessary at some point:
-- Step into the source code of referenced Hadoop code in the debugger.
-- Make changes to the Hadoop source and rebuild it if things are not working as expected (e.g. to add logging).
-- Switch to building from SVN instead of a release snapshot, and apply patches.
Thanks again,
John
-----Original Message-----
From: Harsh J [mailto:harsh@cloudera.com]
Sent: Wednesday, June 05, 2013 11:43 AM
To: <us...@hadoop.apache.org>
Subject: Re: Hadoop JARs and Eclipse
If your goal is to simply build an application, then you can use a Maven project. Why do you require the whole of Hadoop's projects itself on Eclipse when you can simply have the dependencies via a maven pom.xml?
The following is what you can use in a simple maven app, to include all necessary dependencies automatically, for example:
<dependencies>
<dependency>
<groupId>org.apache.hadoop</groupId>
<artifactId>hadoop-client</artifactId>
<version>2.0.4-alpha</version>
</dependency>
</dependencies>
Then importing this project into Eclipse (using m2e plugin) would be equivalent to "Adding all relevant Hadoop JARs to Eclipse".
On Wed, Jun 5, 2013 at 11:07 PM, John Lilley <jo...@redpoint.net> wrote:
> Well, I've failed and given up on building Hadoop in Eclipse. Too
> many things go wrong with Maven plugins and m2e.
>
> But Hadoop builds just fine using the command-line, and it runs using
> Sandy's development-node instructions.
>
> My strategy now is
>
> 1) Tell Eclipse about all of the Hadoop JARs
>
> 2) Make a project with the sample ApplicationMaster distributed shell
> and build it.
>
> Is there a shortcut for adding all relevant JARs to Eclipse? They
> ended up in a lot of separate folders after the build and I can't tell
> if there is one "target" containing all of the ones I need.
>
> Thanks
>
> John
>
>
>
>
--
Harsh J
RE: Hadoop JARs and Eclipse
Posted by John Lilley <jo...@redpoint.net>.
Harsh,
Thanks for your response. I don't need the whole of Hadoop projects in Eclipse, I just need to be able to build Hadoop and step into its source files. Please help me better understand what you recommend as I have a very cursory understanding of maven and its interaction with Eclipse.
-- From where will it fetch the Hadoop JARs?
-- If I have built Hadoop from source, how do I tell it to find my just-built JARs instead of those from the official release site?
-- Is it necessary to build Hadoop from source so that the Eclipse debugger can locate source files and display them if I need to step in?
My goals are to build an application on Hadoop 2.0. Currently I am using the 2.0.4 alpha release. Eventually I this application will include an ApplicationMaster, auxiliary service, and a pile of embedded code and data. However, I don't believe that I will be able to use Hadoop releases as-is without trouble. I suspect that the following will be necessary at some point:
-- Step into the source code of referenced Hadoop code in the debugger.
-- Make changes to the Hadoop source and rebuild it if things are not working as expected (e.g. to add logging).
-- Switch to building from SVN instead of a release snapshot, and apply patches.
Thanks again,
John
-----Original Message-----
From: Harsh J [mailto:harsh@cloudera.com]
Sent: Wednesday, June 05, 2013 11:43 AM
To: <us...@hadoop.apache.org>
Subject: Re: Hadoop JARs and Eclipse
If your goal is to simply build an application, then you can use a Maven project. Why do you require the whole of Hadoop's projects itself on Eclipse when you can simply have the dependencies via a maven pom.xml?
The following is what you can use in a simple maven app, to include all necessary dependencies automatically, for example:
<dependencies>
<dependency>
<groupId>org.apache.hadoop</groupId>
<artifactId>hadoop-client</artifactId>
<version>2.0.4-alpha</version>
</dependency>
</dependencies>
Then importing this project into Eclipse (using m2e plugin) would be equivalent to "Adding all relevant Hadoop JARs to Eclipse".
On Wed, Jun 5, 2013 at 11:07 PM, John Lilley <jo...@redpoint.net> wrote:
> Well, I've failed and given up on building Hadoop in Eclipse. Too
> many things go wrong with Maven plugins and m2e.
>
> But Hadoop builds just fine using the command-line, and it runs using
> Sandy's development-node instructions.
>
> My strategy now is
>
> 1) Tell Eclipse about all of the Hadoop JARs
>
> 2) Make a project with the sample ApplicationMaster distributed shell
> and build it.
>
> Is there a shortcut for adding all relevant JARs to Eclipse? They
> ended up in a lot of separate folders after the build and I can't tell
> if there is one "target" containing all of the ones I need.
>
> Thanks
>
> John
>
>
>
>
--
Harsh J
RE: Hadoop JARs and Eclipse
Posted by John Lilley <jo...@redpoint.net>.
If I am running from built code that was installed with
mvn install
Do I need to change the pom.xml file to reflect that I am using local repository?
Thanks,
John
-----Original Message-----
From: Harsh J [mailto:harsh@cloudera.com]
Sent: Wednesday, June 05, 2013 11:43 AM
To: <us...@hadoop.apache.org>
Subject: Re: Hadoop JARs and Eclipse
If your goal is to simply build an application, then you can use a Maven project. Why do you require the whole of Hadoop's projects itself on Eclipse when you can simply have the dependencies via a maven pom.xml?
The following is what you can use in a simple maven app, to include all necessary dependencies automatically, for example:
<dependencies>
<dependency>
<groupId>org.apache.hadoop</groupId>
<artifactId>hadoop-client</artifactId>
<version>2.0.4-alpha</version>
</dependency>
</dependencies>
Then importing this project into Eclipse (using m2e plugin) would be equivalent to "Adding all relevant Hadoop JARs to Eclipse".
On Wed, Jun 5, 2013 at 11:07 PM, John Lilley <jo...@redpoint.net> wrote:
> Well, I've failed and given up on building Hadoop in Eclipse. Too
> many things go wrong with Maven plugins and m2e.
>
> But Hadoop builds just fine using the command-line, and it runs using
> Sandy's development-node instructions.
>
> My strategy now is
>
> 1) Tell Eclipse about all of the Hadoop JARs
>
> 2) Make a project with the sample ApplicationMaster distributed shell
> and build it.
>
> Is there a shortcut for adding all relevant JARs to Eclipse? They
> ended up in a lot of separate folders after the build and I can't tell
> if there is one "target" containing all of the ones I need.
>
> Thanks
>
> John
>
>
>
>
--
Harsh J
RE: Hadoop JARs and Eclipse
Posted by John Lilley <jo...@redpoint.net>.
Harsh,
Thanks for your response. I don't need the whole of Hadoop projects in Eclipse, I just need to be able to build Hadoop and step into its source files. Please help me better understand what you recommend as I have a very cursory understanding of maven and its interaction with Eclipse.
-- From where will it fetch the Hadoop JARs?
-- If I have built Hadoop from source, how do I tell it to find my just-built JARs instead of those from the official release site?
-- Is it necessary to build Hadoop from source so that the Eclipse debugger can locate source files and display them if I need to step in?
My goals are to build an application on Hadoop 2.0. Currently I am using the 2.0.4 alpha release. Eventually I this application will include an ApplicationMaster, auxiliary service, and a pile of embedded code and data. However, I don't believe that I will be able to use Hadoop releases as-is without trouble. I suspect that the following will be necessary at some point:
-- Step into the source code of referenced Hadoop code in the debugger.
-- Make changes to the Hadoop source and rebuild it if things are not working as expected (e.g. to add logging).
-- Switch to building from SVN instead of a release snapshot, and apply patches.
Thanks again,
John
-----Original Message-----
From: Harsh J [mailto:harsh@cloudera.com]
Sent: Wednesday, June 05, 2013 11:43 AM
To: <us...@hadoop.apache.org>
Subject: Re: Hadoop JARs and Eclipse
If your goal is to simply build an application, then you can use a Maven project. Why do you require the whole of Hadoop's projects itself on Eclipse when you can simply have the dependencies via a maven pom.xml?
The following is what you can use in a simple maven app, to include all necessary dependencies automatically, for example:
<dependencies>
<dependency>
<groupId>org.apache.hadoop</groupId>
<artifactId>hadoop-client</artifactId>
<version>2.0.4-alpha</version>
</dependency>
</dependencies>
Then importing this project into Eclipse (using m2e plugin) would be equivalent to "Adding all relevant Hadoop JARs to Eclipse".
On Wed, Jun 5, 2013 at 11:07 PM, John Lilley <jo...@redpoint.net> wrote:
> Well, I've failed and given up on building Hadoop in Eclipse. Too
> many things go wrong with Maven plugins and m2e.
>
> But Hadoop builds just fine using the command-line, and it runs using
> Sandy's development-node instructions.
>
> My strategy now is
>
> 1) Tell Eclipse about all of the Hadoop JARs
>
> 2) Make a project with the sample ApplicationMaster distributed shell
> and build it.
>
> Is there a shortcut for adding all relevant JARs to Eclipse? They
> ended up in a lot of separate folders after the build and I can't tell
> if there is one "target" containing all of the ones I need.
>
> Thanks
>
> John
>
>
>
>
--
Harsh J
RE: Hadoop JARs and Eclipse
Posted by John Lilley <jo...@redpoint.net>.
Harsh,
Thanks for your response. I don't need the whole of Hadoop projects in Eclipse, I just need to be able to build Hadoop and step into its source files. Please help me better understand what you recommend as I have a very cursory understanding of maven and its interaction with Eclipse.
-- From where will it fetch the Hadoop JARs?
-- If I have built Hadoop from source, how do I tell it to find my just-built JARs instead of those from the official release site?
-- Is it necessary to build Hadoop from source so that the Eclipse debugger can locate source files and display them if I need to step in?
My goals are to build an application on Hadoop 2.0. Currently I am using the 2.0.4 alpha release. Eventually I this application will include an ApplicationMaster, auxiliary service, and a pile of embedded code and data. However, I don't believe that I will be able to use Hadoop releases as-is without trouble. I suspect that the following will be necessary at some point:
-- Step into the source code of referenced Hadoop code in the debugger.
-- Make changes to the Hadoop source and rebuild it if things are not working as expected (e.g. to add logging).
-- Switch to building from SVN instead of a release snapshot, and apply patches.
Thanks again,
John
-----Original Message-----
From: Harsh J [mailto:harsh@cloudera.com]
Sent: Wednesday, June 05, 2013 11:43 AM
To: <us...@hadoop.apache.org>
Subject: Re: Hadoop JARs and Eclipse
If your goal is to simply build an application, then you can use a Maven project. Why do you require the whole of Hadoop's projects itself on Eclipse when you can simply have the dependencies via a maven pom.xml?
The following is what you can use in a simple maven app, to include all necessary dependencies automatically, for example:
<dependencies>
<dependency>
<groupId>org.apache.hadoop</groupId>
<artifactId>hadoop-client</artifactId>
<version>2.0.4-alpha</version>
</dependency>
</dependencies>
Then importing this project into Eclipse (using m2e plugin) would be equivalent to "Adding all relevant Hadoop JARs to Eclipse".
On Wed, Jun 5, 2013 at 11:07 PM, John Lilley <jo...@redpoint.net> wrote:
> Well, I've failed and given up on building Hadoop in Eclipse. Too
> many things go wrong with Maven plugins and m2e.
>
> But Hadoop builds just fine using the command-line, and it runs using
> Sandy's development-node instructions.
>
> My strategy now is
>
> 1) Tell Eclipse about all of the Hadoop JARs
>
> 2) Make a project with the sample ApplicationMaster distributed shell
> and build it.
>
> Is there a shortcut for adding all relevant JARs to Eclipse? They
> ended up in a lot of separate folders after the build and I can't tell
> if there is one "target" containing all of the ones I need.
>
> Thanks
>
> John
>
>
>
>
--
Harsh J
RE: Hadoop JARs and Eclipse
Posted by John Lilley <jo...@redpoint.net>.
If I am running from built code that was installed with
mvn install
Do I need to change the pom.xml file to reflect that I am using local repository?
Thanks,
John
-----Original Message-----
From: Harsh J [mailto:harsh@cloudera.com]
Sent: Wednesday, June 05, 2013 11:43 AM
To: <us...@hadoop.apache.org>
Subject: Re: Hadoop JARs and Eclipse
If your goal is to simply build an application, then you can use a Maven project. Why do you require the whole of Hadoop's projects itself on Eclipse when you can simply have the dependencies via a maven pom.xml?
The following is what you can use in a simple maven app, to include all necessary dependencies automatically, for example:
<dependencies>
<dependency>
<groupId>org.apache.hadoop</groupId>
<artifactId>hadoop-client</artifactId>
<version>2.0.4-alpha</version>
</dependency>
</dependencies>
Then importing this project into Eclipse (using m2e plugin) would be equivalent to "Adding all relevant Hadoop JARs to Eclipse".
On Wed, Jun 5, 2013 at 11:07 PM, John Lilley <jo...@redpoint.net> wrote:
> Well, I've failed and given up on building Hadoop in Eclipse. Too
> many things go wrong with Maven plugins and m2e.
>
> But Hadoop builds just fine using the command-line, and it runs using
> Sandy's development-node instructions.
>
> My strategy now is
>
> 1) Tell Eclipse about all of the Hadoop JARs
>
> 2) Make a project with the sample ApplicationMaster distributed shell
> and build it.
>
> Is there a shortcut for adding all relevant JARs to Eclipse? They
> ended up in a lot of separate folders after the build and I can't tell
> if there is one "target" containing all of the ones I need.
>
> Thanks
>
> John
>
>
>
>
--
Harsh J
RE: Hadoop JARs and Eclipse
Posted by John Lilley <jo...@redpoint.net>.
If I am running from built code that was installed with
mvn install
Do I need to change the pom.xml file to reflect that I am using local repository?
Thanks,
John
-----Original Message-----
From: Harsh J [mailto:harsh@cloudera.com]
Sent: Wednesday, June 05, 2013 11:43 AM
To: <us...@hadoop.apache.org>
Subject: Re: Hadoop JARs and Eclipse
If your goal is to simply build an application, then you can use a Maven project. Why do you require the whole of Hadoop's projects itself on Eclipse when you can simply have the dependencies via a maven pom.xml?
The following is what you can use in a simple maven app, to include all necessary dependencies automatically, for example:
<dependencies>
<dependency>
<groupId>org.apache.hadoop</groupId>
<artifactId>hadoop-client</artifactId>
<version>2.0.4-alpha</version>
</dependency>
</dependencies>
Then importing this project into Eclipse (using m2e plugin) would be equivalent to "Adding all relevant Hadoop JARs to Eclipse".
On Wed, Jun 5, 2013 at 11:07 PM, John Lilley <jo...@redpoint.net> wrote:
> Well, I've failed and given up on building Hadoop in Eclipse. Too
> many things go wrong with Maven plugins and m2e.
>
> But Hadoop builds just fine using the command-line, and it runs using
> Sandy's development-node instructions.
>
> My strategy now is
>
> 1) Tell Eclipse about all of the Hadoop JARs
>
> 2) Make a project with the sample ApplicationMaster distributed shell
> and build it.
>
> Is there a shortcut for adding all relevant JARs to Eclipse? They
> ended up in a lot of separate folders after the build and I can't tell
> if there is one "target" containing all of the ones I need.
>
> Thanks
>
> John
>
>
>
>
--
Harsh J
Re: Hadoop JARs and Eclipse
Posted by Harsh J <ha...@cloudera.com>.
If your goal is to simply build an application, then you can use a
Maven project. Why do you require the whole of Hadoop's projects
itself on Eclipse when you can simply have the dependencies via a
maven pom.xml?
The following is what you can use in a simple maven app, to include
all necessary dependencies automatically, for example:
<dependencies>
<dependency>
<groupId>org.apache.hadoop</groupId>
<artifactId>hadoop-client</artifactId>
<version>2.0.4-alpha</version>
</dependency>
</dependencies>
Then importing this project into Eclipse (using m2e plugin) would be
equivalent to "Adding all relevant Hadoop JARs to Eclipse".
On Wed, Jun 5, 2013 at 11:07 PM, John Lilley <jo...@redpoint.net> wrote:
> Well, I’ve failed and given up on building Hadoop in Eclipse. Too many
> things go wrong with Maven plugins and m2e.
>
> But Hadoop builds just fine using the command-line, and it runs using
> Sandy’s development-node instructions.
>
> My strategy now is
>
> 1) Tell Eclipse about all of the Hadoop JARs
>
> 2) Make a project with the sample ApplicationMaster distributed shell
> and build it.
>
> Is there a shortcut for adding all relevant JARs to Eclipse? They ended up
> in a lot of separate folders after the build and I can’t tell if there is
> one “target” containing all of the ones I need.
>
> Thanks
>
> John
>
>
>
>
--
Harsh J
Re: Hadoop JARs and Eclipse
Posted by Harsh J <ha...@cloudera.com>.
If your goal is to simply build an application, then you can use a
Maven project. Why do you require the whole of Hadoop's projects
itself on Eclipse when you can simply have the dependencies via a
maven pom.xml?
The following is what you can use in a simple maven app, to include
all necessary dependencies automatically, for example:
<dependencies>
<dependency>
<groupId>org.apache.hadoop</groupId>
<artifactId>hadoop-client</artifactId>
<version>2.0.4-alpha</version>
</dependency>
</dependencies>
Then importing this project into Eclipse (using m2e plugin) would be
equivalent to "Adding all relevant Hadoop JARs to Eclipse".
On Wed, Jun 5, 2013 at 11:07 PM, John Lilley <jo...@redpoint.net> wrote:
> Well, I’ve failed and given up on building Hadoop in Eclipse. Too many
> things go wrong with Maven plugins and m2e.
>
> But Hadoop builds just fine using the command-line, and it runs using
> Sandy’s development-node instructions.
>
> My strategy now is
>
> 1) Tell Eclipse about all of the Hadoop JARs
>
> 2) Make a project with the sample ApplicationMaster distributed shell
> and build it.
>
> Is there a shortcut for adding all relevant JARs to Eclipse? They ended up
> in a lot of separate folders after the build and I can’t tell if there is
> one “target” containing all of the ones I need.
>
> Thanks
>
> John
>
>
>
>
--
Harsh J
Re: Hadoop JARs and Eclipse
Posted by Harsh J <ha...@cloudera.com>.
If your goal is to simply build an application, then you can use a
Maven project. Why do you require the whole of Hadoop's projects
itself on Eclipse when you can simply have the dependencies via a
maven pom.xml?
The following is what you can use in a simple maven app, to include
all necessary dependencies automatically, for example:
<dependencies>
<dependency>
<groupId>org.apache.hadoop</groupId>
<artifactId>hadoop-client</artifactId>
<version>2.0.4-alpha</version>
</dependency>
</dependencies>
Then importing this project into Eclipse (using m2e plugin) would be
equivalent to "Adding all relevant Hadoop JARs to Eclipse".
On Wed, Jun 5, 2013 at 11:07 PM, John Lilley <jo...@redpoint.net> wrote:
> Well, I’ve failed and given up on building Hadoop in Eclipse. Too many
> things go wrong with Maven plugins and m2e.
>
> But Hadoop builds just fine using the command-line, and it runs using
> Sandy’s development-node instructions.
>
> My strategy now is
>
> 1) Tell Eclipse about all of the Hadoop JARs
>
> 2) Make a project with the sample ApplicationMaster distributed shell
> and build it.
>
> Is there a shortcut for adding all relevant JARs to Eclipse? They ended up
> in a lot of separate folders after the build and I can’t tell if there is
> one “target” containing all of the ones I need.
>
> Thanks
>
> John
>
>
>
>
--
Harsh J
Re: Hadoop JARs and Eclipse
Posted by Harsh J <ha...@cloudera.com>.
If your goal is to simply build an application, then you can use a
Maven project. Why do you require the whole of Hadoop's projects
itself on Eclipse when you can simply have the dependencies via a
maven pom.xml?
The following is what you can use in a simple maven app, to include
all necessary dependencies automatically, for example:
<dependencies>
<dependency>
<groupId>org.apache.hadoop</groupId>
<artifactId>hadoop-client</artifactId>
<version>2.0.4-alpha</version>
</dependency>
</dependencies>
Then importing this project into Eclipse (using m2e plugin) would be
equivalent to "Adding all relevant Hadoop JARs to Eclipse".
On Wed, Jun 5, 2013 at 11:07 PM, John Lilley <jo...@redpoint.net> wrote:
> Well, I’ve failed and given up on building Hadoop in Eclipse. Too many
> things go wrong with Maven plugins and m2e.
>
> But Hadoop builds just fine using the command-line, and it runs using
> Sandy’s development-node instructions.
>
> My strategy now is
>
> 1) Tell Eclipse about all of the Hadoop JARs
>
> 2) Make a project with the sample ApplicationMaster distributed shell
> and build it.
>
> Is there a shortcut for adding all relevant JARs to Eclipse? They ended up
> in a lot of separate folders after the build and I can’t tell if there is
> one “target” containing all of the ones I need.
>
> Thanks
>
> John
>
>
>
>
--
Harsh J