You are viewing a plain text version of this content. The canonical link for it is here.
Posted to mapreduce-dev@hadoop.apache.org by Eli Collins <el...@cloudera.com> on 2011/08/19 00:19:16 UTC
Picking up local common changes in mr
Hey gang,
What's the new equivalent of resolvers=true in the new MR build? ie
how do you get a a local common change to get picked up by mr?
Thanks,
Eli
Re: Picking up local common changes in mr
Posted by Robert Evans <ev...@yahoo-inc.com>.
One thing to be aware of is that with -SNAPSHOT at the end of the version Maven will start looking at dates. So if you have a 0.23.0-SNAPSHOT that you personally modified/built in your .m2 repository and go to build something that depends on it. If the nightly build has pushed it to the apache repo after you built your version maven might download the "newer" version replacing your changes. If you changes impact multiple components then your choices are to always build the entire project (or at least the subset that has dependent changes) or always build with -o after your initial build/install.
--Bobby
On 8/19/11 11:41 AM, "Matt Foley" <mf...@hortonworks.com> wrote:
Thanks for the nice clear statement, Alejandro.
--Matt
On Thu, Aug 18, 2011 at 4:40 PM, Alejandro Abdelnur <tu...@cloudera.com>wrote:
> This is handled by maven reactor.
>
> When your run Maven in a multimodule project (like we have), all modules
> that are part of the build (from the dir where you are) down are used for
> the build/test/packaging, all modules that are not part of the build are
> picked up from .m2/repo.
>
> For example
>
> "cd trunk/hadoop-mapreduce;mvn compile" uses hadoop-common & hadoop-hdfs
> from m2/repo
>
> "cd trunk;mvn compile" uses hadoop-common, hadoop-hdfs, hadoop-mapreduce
> from the build.
>
> HTH
>
> Thxs.
>
> Alejandro
>
>
> On Thu, Aug 18, 2011 at 4:35 PM, Matt Foley <mf...@hortonworks.com>
> wrote:
>
> > Since we put all the effort into "un-splitting" the components, shouldn't
> > we
> > have a switch
> > that causes, eg, the MAPREDUCE build to pick up artifacts from COMMON and
> > HDFS builds
> > in specified sibling directories, without using m2 as an intermediary?
> >
> > Of course it should respect dependencies (via maven) so that if HDFS
> source
> > has been modified,
> > the HDFS artifacts will also be rebuilt before MAPREDUCE uses them :-)
> >
> > --Matt
> >
> > On Thu, Aug 18, 2011 at 3:30 PM, Giridharan Kesavan <
> > gkesavan@hortonworks.com> wrote:
> >
> > > Hello,
> > >
> > > Its the same -Dresolvers=internal for the ant build system; For the
> > > maven/yarn build system as long as you have the latest common jar in
> > > the m2 cache its going to resolve common from the maven cache. If not
> > > from the apache maven repo. You can force the builds to use the cache
> > > by adding -o option. (offline builds)
> > >
> > > Thanks,
> > > Giri
> > >
> > > On Thu, Aug 18, 2011 at 3:19 PM, Eli Collins <el...@cloudera.com> wrote:
> > > > Hey gang,
> > > >
> > > > What's the new equivalent of resolvers=true in the new MR build? ie
> > > > how do you get a a local common change to get picked up by mr?
> > > >
> > > > Thanks,
> > > > Eli
> > > >
> > >
> >
>
Re: Picking up local common changes in mr
Posted by Matt Foley <mf...@hortonworks.com>.
Thanks for the nice clear statement, Alejandro.
--Matt
On Thu, Aug 18, 2011 at 4:40 PM, Alejandro Abdelnur <tu...@cloudera.com>wrote:
> This is handled by maven reactor.
>
> When your run Maven in a multimodule project (like we have), all modules
> that are part of the build (from the dir where you are) down are used for
> the build/test/packaging, all modules that are not part of the build are
> picked up from .m2/repo.
>
> For example
>
> "cd trunk/hadoop-mapreduce;mvn compile" uses hadoop-common & hadoop-hdfs
> from m2/repo
>
> "cd trunk;mvn compile" uses hadoop-common, hadoop-hdfs, hadoop-mapreduce
> from the build.
>
> HTH
>
> Thxs.
>
> Alejandro
>
>
> On Thu, Aug 18, 2011 at 4:35 PM, Matt Foley <mf...@hortonworks.com>
> wrote:
>
> > Since we put all the effort into "un-splitting" the components, shouldn't
> > we
> > have a switch
> > that causes, eg, the MAPREDUCE build to pick up artifacts from COMMON and
> > HDFS builds
> > in specified sibling directories, without using m2 as an intermediary?
> >
> > Of course it should respect dependencies (via maven) so that if HDFS
> source
> > has been modified,
> > the HDFS artifacts will also be rebuilt before MAPREDUCE uses them :-)
> >
> > --Matt
> >
> > On Thu, Aug 18, 2011 at 3:30 PM, Giridharan Kesavan <
> > gkesavan@hortonworks.com> wrote:
> >
> > > Hello,
> > >
> > > Its the same -Dresolvers=internal for the ant build system; For the
> > > maven/yarn build system as long as you have the latest common jar in
> > > the m2 cache its going to resolve common from the maven cache. If not
> > > from the apache maven repo. You can force the builds to use the cache
> > > by adding -o option. (offline builds)
> > >
> > > Thanks,
> > > Giri
> > >
> > > On Thu, Aug 18, 2011 at 3:19 PM, Eli Collins <el...@cloudera.com> wrote:
> > > > Hey gang,
> > > >
> > > > What's the new equivalent of resolvers=true in the new MR build? ie
> > > > how do you get a a local common change to get picked up by mr?
> > > >
> > > > Thanks,
> > > > Eli
> > > >
> > >
> >
>
Re: Picking up local common changes in mr
Posted by Alejandro Abdelnur <tu...@cloudera.com>.
This is handled by maven reactor.
When your run Maven in a multimodule project (like we have), all modules
that are part of the build (from the dir where you are) down are used for
the build/test/packaging, all modules that are not part of the build are
picked up from .m2/repo.
For example
"cd trunk/hadoop-mapreduce;mvn compile" uses hadoop-common & hadoop-hdfs
from m2/repo
"cd trunk;mvn compile" uses hadoop-common, hadoop-hdfs, hadoop-mapreduce
from the build.
HTH
Thxs.
Alejandro
On Thu, Aug 18, 2011 at 4:35 PM, Matt Foley <mf...@hortonworks.com> wrote:
> Since we put all the effort into "un-splitting" the components, shouldn't
> we
> have a switch
> that causes, eg, the MAPREDUCE build to pick up artifacts from COMMON and
> HDFS builds
> in specified sibling directories, without using m2 as an intermediary?
>
> Of course it should respect dependencies (via maven) so that if HDFS source
> has been modified,
> the HDFS artifacts will also be rebuilt before MAPREDUCE uses them :-)
>
> --Matt
>
> On Thu, Aug 18, 2011 at 3:30 PM, Giridharan Kesavan <
> gkesavan@hortonworks.com> wrote:
>
> > Hello,
> >
> > Its the same -Dresolvers=internal for the ant build system; For the
> > maven/yarn build system as long as you have the latest common jar in
> > the m2 cache its going to resolve common from the maven cache. If not
> > from the apache maven repo. You can force the builds to use the cache
> > by adding -o option. (offline builds)
> >
> > Thanks,
> > Giri
> >
> > On Thu, Aug 18, 2011 at 3:19 PM, Eli Collins <el...@cloudera.com> wrote:
> > > Hey gang,
> > >
> > > What's the new equivalent of resolvers=true in the new MR build? ie
> > > how do you get a a local common change to get picked up by mr?
> > >
> > > Thanks,
> > > Eli
> > >
> >
>
Re: Picking up local common changes in mr
Posted by Matt Foley <mf...@hortonworks.com>.
Since we put all the effort into "un-splitting" the components, shouldn't we
have a switch
that causes, eg, the MAPREDUCE build to pick up artifacts from COMMON and
HDFS builds
in specified sibling directories, without using m2 as an intermediary?
Of course it should respect dependencies (via maven) so that if HDFS source
has been modified,
the HDFS artifacts will also be rebuilt before MAPREDUCE uses them :-)
--Matt
On Thu, Aug 18, 2011 at 3:30 PM, Giridharan Kesavan <
gkesavan@hortonworks.com> wrote:
> Hello,
>
> Its the same -Dresolvers=internal for the ant build system; For the
> maven/yarn build system as long as you have the latest common jar in
> the m2 cache its going to resolve common from the maven cache. If not
> from the apache maven repo. You can force the builds to use the cache
> by adding -o option. (offline builds)
>
> Thanks,
> Giri
>
> On Thu, Aug 18, 2011 at 3:19 PM, Eli Collins <el...@cloudera.com> wrote:
> > Hey gang,
> >
> > What's the new equivalent of resolvers=true in the new MR build? ie
> > how do you get a a local common change to get picked up by mr?
> >
> > Thanks,
> > Eli
> >
>
Re: Picking up local common changes in mr
Posted by Eli Collins <el...@cloudera.com>.
Blowing away the common dir in my local m2 repo fixed the issue, for
some reason it wasn't re-populating it.
Thanks,
Eli
On Thu, Aug 18, 2011 at 3:42 PM, Eli Collins <el...@cloudera.com> wrote:
> Thanks Giri. That doesn't seem to work from me. Eg if I make a change
> to FileUtil in hadoop-common then mvn-install -DskipTests, it gets
> populated into my local m2 repo but it is not picked up by mvn
> test-compile -Dresolvers=internal in hadoop-mapreduce (ie per below I
> can't introduce a new method that's used in a test). Does this work
> for you or am I doing something wrong?
>
> Thanks,
> Eli
>
> hadoop-mapreduce (trunk)$ git diff
> diff --git hadoop-common/src/main/java/org/apache/hadoop/fs/FileUtil.java hadoop
> index 8e7aa30..0471caf 100644
> --- hadoop-common/src/main/java/org/apache/hadoop/fs/FileUtil.java
> +++ hadoop-common/src/main/java/org/apache/hadoop/fs/FileUtil.java
>
> + public static boolean fullyDelete2(File dir) { }
> +
>
> diff --git hadoop-mapreduce/hadoop-yarn/hadoop-yarn-common/src/test/java/org/apa
> index 454ef2c..d029529 100644
> --- hadoop-mapreduce/hadoop-yarn/hadoop-yarn-common/src/test/java/org/apache/had
> +++ hadoop-mapreduce/hadoop-yarn/hadoop-yarn-common/src/test/java/org/apache/had
> @@ -1,3 +1,4 @@
> } finally {
> FileUtil.fullyDelete(procfsRootDir);
> + FileUtil.fullyDelete2(procfsRootDir);
> }
> }
>
>
>
>
> On Thu, Aug 18, 2011 at 3:30 PM, Giridharan Kesavan
> <gk...@hortonworks.com> wrote:
>> Hello,
>>
>> Its the same -Dresolvers=internal for the ant build system; For the
>> maven/yarn build system as long as you have the latest common jar in
>> the m2 cache its going to resolve common from the maven cache. If not
>> from the apache maven repo. You can force the builds to use the cache
>> by adding -o option. (offline builds)
>>
>> Thanks,
>> Giri
>>
>> On Thu, Aug 18, 2011 at 3:19 PM, Eli Collins <el...@cloudera.com> wrote:
>>> Hey gang,
>>>
>>> What's the new equivalent of resolvers=true in the new MR build? ie
>>> how do you get a a local common change to get picked up by mr?
>>>
>>> Thanks,
>>> Eli
>>>
>>
>
Re: Picking up local common changes in mr
Posted by Giridharan Kesavan <gk...@hortonworks.com>.
>> populated into my local m2 repo but it is not picked up by mvn
>> test-compile -Dresolvers=internal
mvn doenst understand -Dresolvers=internal. (this is an ant/ivy configuration).
by default maven resolves artifacts from the local cache and if not
available then resolves from the mvn repository.
you could try with ant veryclean
On Thu, Aug 18, 2011 at 3:42 PM, Eli Collins <el...@cloudera.com> wrote:
> Thanks Giri. That doesn't seem to work from me. Eg if I make a change
> to FileUtil in hadoop-common then mvn-install -DskipTests, it gets
> populated into my local m2 repo but it is not picked up by mvn
> test-compile -Dresolvers=internal in hadoop-mapreduce (ie per below I
> can't introduce a new method that's used in a test). Does this work
> for you or am I doing something wrong?
>
> Thanks,
> Eli
>
> hadoop-mapreduce (trunk)$ git diff
> diff --git hadoop-common/src/main/java/org/apache/hadoop/fs/FileUtil.java hadoop
> index 8e7aa30..0471caf 100644
> --- hadoop-common/src/main/java/org/apache/hadoop/fs/FileUtil.java
> +++ hadoop-common/src/main/java/org/apache/hadoop/fs/FileUtil.java
>
> + public static boolean fullyDelete2(File dir) { }
> +
>
> diff --git hadoop-mapreduce/hadoop-yarn/hadoop-yarn-common/src/test/java/org/apa
> index 454ef2c..d029529 100644
> --- hadoop-mapreduce/hadoop-yarn/hadoop-yarn-common/src/test/java/org/apache/had
> +++ hadoop-mapreduce/hadoop-yarn/hadoop-yarn-common/src/test/java/org/apache/had
> @@ -1,3 +1,4 @@
> } finally {
> FileUtil.fullyDelete(procfsRootDir);
> + FileUtil.fullyDelete2(procfsRootDir);
> }
> }
>
>
>
>
> On Thu, Aug 18, 2011 at 3:30 PM, Giridharan Kesavan
> <gk...@hortonworks.com> wrote:
>> Hello,
>>
>> Its the same -Dresolvers=internal for the ant build system; For the
>> maven/yarn build system as long as you have the latest common jar in
>> the m2 cache its going to resolve common from the maven cache. If not
>> from the apache maven repo. You can force the builds to use the cache
>> by adding -o option. (offline builds)
>>
>> Thanks,
>> Giri
>>
>> On Thu, Aug 18, 2011 at 3:19 PM, Eli Collins <el...@cloudera.com> wrote:
>>> Hey gang,
>>>
>>> What's the new equivalent of resolvers=true in the new MR build? ie
>>> how do you get a a local common change to get picked up by mr?
>>>
>>> Thanks,
>>> Eli
>>>
>>
>
Re: Picking up local common changes in mr
Posted by Eli Collins <el...@cloudera.com>.
Thanks Giri. That doesn't seem to work from me. Eg if I make a change
to FileUtil in hadoop-common then mvn-install -DskipTests, it gets
populated into my local m2 repo but it is not picked up by mvn
test-compile -Dresolvers=internal in hadoop-mapreduce (ie per below I
can't introduce a new method that's used in a test). Does this work
for you or am I doing something wrong?
Thanks,
Eli
hadoop-mapreduce (trunk)$ git diff
diff --git hadoop-common/src/main/java/org/apache/hadoop/fs/FileUtil.java hadoop
index 8e7aa30..0471caf 100644
--- hadoop-common/src/main/java/org/apache/hadoop/fs/FileUtil.java
+++ hadoop-common/src/main/java/org/apache/hadoop/fs/FileUtil.java
+ public static boolean fullyDelete2(File dir) { }
+
diff --git hadoop-mapreduce/hadoop-yarn/hadoop-yarn-common/src/test/java/org/apa
index 454ef2c..d029529 100644
--- hadoop-mapreduce/hadoop-yarn/hadoop-yarn-common/src/test/java/org/apache/had
+++ hadoop-mapreduce/hadoop-yarn/hadoop-yarn-common/src/test/java/org/apache/had
@@ -1,3 +1,4 @@
} finally {
FileUtil.fullyDelete(procfsRootDir);
+ FileUtil.fullyDelete2(procfsRootDir);
}
}
On Thu, Aug 18, 2011 at 3:30 PM, Giridharan Kesavan
<gk...@hortonworks.com> wrote:
> Hello,
>
> Its the same -Dresolvers=internal for the ant build system; For the
> maven/yarn build system as long as you have the latest common jar in
> the m2 cache its going to resolve common from the maven cache. If not
> from the apache maven repo. You can force the builds to use the cache
> by adding -o option. (offline builds)
>
> Thanks,
> Giri
>
> On Thu, Aug 18, 2011 at 3:19 PM, Eli Collins <el...@cloudera.com> wrote:
>> Hey gang,
>>
>> What's the new equivalent of resolvers=true in the new MR build? ie
>> how do you get a a local common change to get picked up by mr?
>>
>> Thanks,
>> Eli
>>
>
Re: Picking up local common changes in mr
Posted by Giridharan Kesavan <gk...@hortonworks.com>.
Hello,
Its the same -Dresolvers=internal for the ant build system; For the
maven/yarn build system as long as you have the latest common jar in
the m2 cache its going to resolve common from the maven cache. If not
from the apache maven repo. You can force the builds to use the cache
by adding -o option. (offline builds)
Thanks,
Giri
On Thu, Aug 18, 2011 at 3:19 PM, Eli Collins <el...@cloudera.com> wrote:
> Hey gang,
>
> What's the new equivalent of resolvers=true in the new MR build? ie
> how do you get a a local common change to get picked up by mr?
>
> Thanks,
> Eli
>