You are viewing a plain text version of this content. The canonical link for it is here.
Posted to common-dev@hadoop.apache.org by Andrei Dragomir <ad...@adobe.com> on 2009/11/05 11:34:50 UTC
How to build and deploy Hadoop 0.21 ?
Hello everyone.
We ran into a bunch of issues with building and deploying hadoop 0.21.
It would be great to get some answers about how things should work, so
we can try to fix them.
1. When checking out the repositories, each of them can be built by
itself perfectly. BUT, if you look in hdfs it has mapreduce libraries,
and in mapreduce it has hdfs libraries. That's kind of a cross-
reference between projects.
Q: Is this dependence necessary ? Can we get rid of it ?
Q: if it's necessary, how does one build the jars with the latest
version of the source code ? how are the jars in the scm repository
created (hadoop-hdfs/lib/hadoop-mapred-0.21-dev.jar) as long as there
is a cross-reference ?
2. There are issues with the jar files and the webapps (dfshealth.jsp,
etc). Right now, the only way to have a hadoop functioning system is
to: build hdfs and mapreduce; copy everything from hdfs/build and
mapreduce/build to common/build.
Q: Is there a better way of doing this ? What needs to be fixed to
have the webapps in the jar files (like on 0.20). Are there JIRA
issues logged on this ?
We would really appreciate some answers at least related to where
hadoop is going with this build step, so we can help with patches /
fixes.
Thank you,
Andrei Dragomirt
Re: How to build and deploy Hadoop 0.21 ?
Posted by Aaron Kimball <aa...@cloudera.com>.
On Thu, Nov 5, 2009 at 2:34 AM, Andrei Dragomir <ad...@adobe.com> wrote:
> Hello everyone.
> We ran into a bunch of issues with building and deploying hadoop 0.21.
> It would be great to get some answers about how things should work, so
> we can try to fix them.
>
> 1. When checking out the repositories, each of them can be built by
> itself perfectly. BUT, if you look in hdfs it has mapreduce libraries,
> and in mapreduce it has hdfs libraries. That's kind of a cross-
> reference between projects.
> Q: Is this dependence necessary ? Can we get rid of it ?
>
Those are build-time dependencies. Ideally you'll ignore them post-build.
> Q: if it's necessary, how does one build the jars with the latest
> version of the source code ? how are the jars in the scm repository
> created (hadoop-hdfs/lib/hadoop-mapred-0.21-dev.jar) as long as there
> is a cross-reference ?
> 2. There are issues with the jar files and the webapps (dfshealth.jsp,
> etc). Right now, the only way to have a hadoop functioning system is
> to: build hdfs and mapreduce; copy everything from hdfs/build and
> mapreduce/build to common/build.
>
Yup.
> Q: Is there a better way of doing this ? What needs to be fixed to
> have the webapps in the jar files (like on 0.20). Are there JIRA
> issues logged on this ?
>
>
I have created a Makefile and some associated scripts that will build
everything and squash it together for you; see
https://issues.apache.org/jira/browse/HADOOP-6342
There is also a longer-term effort to use Maven to coordinate the three
subprojects, and use a local repository for inter-project development on a
single machine; see https://issues.apache.org/jira/browse/HADOOP-5107 for
progress there.
> We would really appreciate some answers at least related to where
> hadoop is going with this build step, so we can help with patches /
> fixes.
>
> Thank you,
> Andrei Dragomirt
>
Re: How to build and deploy Hadoop 0.21 ?
Posted by Aaron Kimball <aa...@cloudera.com>.
On Thu, Nov 5, 2009 at 2:34 AM, Andrei Dragomir <ad...@adobe.com> wrote:
> Hello everyone.
> We ran into a bunch of issues with building and deploying hadoop 0.21.
> It would be great to get some answers about how things should work, so
> we can try to fix them.
>
> 1. When checking out the repositories, each of them can be built by
> itself perfectly. BUT, if you look in hdfs it has mapreduce libraries,
> and in mapreduce it has hdfs libraries. That's kind of a cross-
> reference between projects.
> Q: Is this dependence necessary ? Can we get rid of it ?
>
Those are build-time dependencies. Ideally you'll ignore them post-build.
> Q: if it's necessary, how does one build the jars with the latest
> version of the source code ? how are the jars in the scm repository
> created (hadoop-hdfs/lib/hadoop-mapred-0.21-dev.jar) as long as there
> is a cross-reference ?
> 2. There are issues with the jar files and the webapps (dfshealth.jsp,
> etc). Right now, the only way to have a hadoop functioning system is
> to: build hdfs and mapreduce; copy everything from hdfs/build and
> mapreduce/build to common/build.
>
Yup.
> Q: Is there a better way of doing this ? What needs to be fixed to
> have the webapps in the jar files (like on 0.20). Are there JIRA
> issues logged on this ?
>
>
I have created a Makefile and some associated scripts that will build
everything and squash it together for you; see
https://issues.apache.org/jira/browse/HADOOP-6342
There is also a longer-term effort to use Maven to coordinate the three
subprojects, and use a local repository for inter-project development on a
single machine; see https://issues.apache.org/jira/browse/HADOOP-5107 for
progress there.
> We would really appreciate some answers at least related to where
> hadoop is going with this build step, so we can help with patches /
> fixes.
>
> Thank you,
> Andrei Dragomirt
>