You are viewing a plain text version of this content. The canonical link for it is here.
Posted to mapreduce-dev@hadoop.apache.org by Alejandro Abdelnur <tu...@cloudera.com> on 2011/07/29 20:35:05 UTC
need some ivy help (for MAPREDUCE-2741)
I'm trying to get mapred to work using hadoop-commons generated by
HADOOP-6671.
I've tweaked all ivy.xml files in mapreduce to make lgo4j exclude
jms/jmx/jdmk.
I've able to run successfully 'ant jar'
But when running 'ant tar' things fail with the output at the end of this
message.
I've been trying -and failing- to figure out all morning how ivy:resolve
works and why is still pulling log4j excluded dependencies.
Any hint on why the exclusions are not be considered by the
'ivy-resolve-test' target?
Thxs.
Alejandro
ivy-resolve-test:
[ivy:resolve] downloading
http://repo1.maven.org/maven2/org/mortbay/jetty/jetty-servlet-tester/6.1.14/jetty-servlet-tester-6.1.14.jar...
[ivy:resolve] ..... (8kB)
[ivy:resolve] .. (0kB)
[ivy:resolve] [SUCCESSFUL ]
org.mortbay.jetty#jetty-servlet-tester;6.1.14!jetty-servlet-tester.jar
(908ms)
[ivy:resolve]
[ivy:resolve] :: problems summary ::
[ivy:resolve] :::: WARNINGS
[ivy:resolve] [FAILED ] javax.jms#jms;1.1!jms.jar: (0ms)
[ivy:resolve] ==== fs: tried
[ivy:resolve] /Users/tucu/.m2/repository/javax/jms/jms/1.1/jms-1.1.jar
[ivy:resolve] ==== apache-snapshot: tried
[ivy:resolve]
https://repository.apache.org/content/repositories/snapshots/javax/jms/jms/1.1/jms-1.1.jar
[ivy:resolve] ==== maven2: tried
[ivy:resolve] http://repo1.maven.org/maven2/javax/jms/jms/1.1/jms-1.1.jar
[ivy:resolve] [FAILED ] com.sun.jdmk#jmxtools;1.2.1!jmxtools.jar: (0ms)
[ivy:resolve] ==== fs: tried
[ivy:resolve] /Users/tucu/.m2/repository/com/sun/jdmk/jmxtools/1.2.1/jmxtools-1.2.1.jar
[ivy:resolve] ==== apache-snapshot: tried
[ivy:resolve]
https://repository.apache.org/content/repositories/snapshots/com/sun/jdmk/jmxtools/1.2.1/jmxtools-1.2.1.jar
[ivy:resolve] ==== maven2: tried
[ivy:resolve]
http://repo1.maven.org/maven2/com/sun/jdmk/jmxtools/1.2.1/jmxtools-1.2.1.jar
[ivy:resolve] [FAILED ] com.sun.jmx#jmxri;1.2.1!jmxri.jar: (0ms)
[ivy:resolve] ==== fs: tried
[ivy:resolve] /Users/tucu/.m2/repository/com/sun/jmx/jmxri/1.2.1/jmxri-1.2.1.jar
[ivy:resolve] ==== apache-snapshot: tried
[ivy:resolve]
https://repository.apache.org/content/repositories/snapshots/com/sun/jmx/jmxri/1.2.1/jmxri-1.2.1.jar
[ivy:resolve] ==== maven2: tried
[ivy:resolve]
http://repo1.maven.org/maven2/com/sun/jmx/jmxri/1.2.1/jmxri-1.2.1.jar
[ivy:resolve] ::::::::::::::::::::::::::::::::::::::::::::::
[ivy:resolve] :: FAILED DOWNLOADS ::
[ivy:resolve] :: ^ see resolution messages for details ^ ::
[ivy:resolve] ::::::::::::::::::::::::::::::::::::::::::::::
[ivy:resolve] :: javax.jms#jms;1.1!jms.jar
[ivy:resolve] :: com.sun.jdmk#jmxtools;1.2.1!jmxtools.jar
[ivy:resolve] :: com.sun.jmx#jmxri;1.2.1!jmxri.jar
[ivy:resolve] ::::::::::::::::::::::::::::::::::::::::::::::
[ivy:resolve]
[ivy:resolve] :: USE VERBOSE OR DEBUG MESSAGE LEVEL FOR MORE DETAILS
Re: need some ivy help (for MAPREDUCE-2741)
Posted by Steve Loughran <st...@apache.org>.
On 29/07/11 22:30, Alejandro Abdelnur wrote:
> Giri,
>
> It is 'compile->master' already.
>
> the download attempt is post compilation, in the 'ivy-resolve-test' target.
>
It'll be something in that config then which has an indirect dependency
on log4j -something that isn't in the other configurations.
>>>
>>> Alejandro
>>>
>>> ivy-resolve-test:
>>> [ivy:resolve] downloading
>>>
>>>
>> http://repo1.maven.org/maven2/org/mortbay/jetty/jetty-servlet-tester/6.1.14/jetty-servlet-tester-6.1.14.jar
>> .
>>> ..
>>> [ivy:resolve] ..... (8kB)
>>> [ivy:resolve] .. (0kB)
>>> [ivy:resolve] [SUCCESSFUL ]
>>> org.mortbay.jetty#jetty-servlet-tester;6.1.14!jetty-servlet-tester.jar
this depends on Jetty, Jetty depends on so much else.
Try setting up this to exclude jetty (referring to it elsewhere) and you
won't get the downstream things.
The alternative to "shotgun debugging" of transient dependencies is to
set up a local repository containing those artifacts (or force them in
to ~/m2), get the ivy report to find out what is asking for them, then
edit the parent in the ivy.xml file to exclude log4j. Oh, then rm the
patched in artifacts to make sure all is well again
>>> (908ms)
>>> [ivy:resolve]
>>> [ivy:resolve] :: problems summary ::
>>> [ivy:resolve] :::: WARNINGS
>>> [ivy:resolve] [FAILED ] javax.jms#jms;1.1!jms.jar: (0ms)
>>> [ivy:resolve] ==== fs: tried
>>> [ivy:resolve] /Users/tucu/.m2/repository/javax/jms/jms/1.1/jms-1.1.jar
>>> [ivy:resolve] ==== apache-snapshot: tried
>>> [ivy:resolve]
>>>
>>>
>> https://repository.apache.org/content/repositories/snapshots/javax/jms/jms/1.1/jms-1.1.jar
>>> [ivy:resolve] ==== maven2: tried
>>> [ivy:resolve]
>>> http://repo1.maven.org/maven2/javax/jms/jms/1.1/jms-1.1.jar
>>> [ivy:resolve] [FAILED ] com.sun.jdmk#jmxtools;1.2.1!jmxtools.jar:
>>> (0ms)
>>> [ivy:resolve] ==== fs: tried
>>> [ivy:resolve]
>>>
>> /Users/tucu/.m2/repository/com/sun/jdmk/jmxtools/1.2.1/jmxtools-1.2.1.jar
>>> [ivy:resolve] ==== apache-snapshot: tried
>>> [ivy:resolve]
>>>
>>>
>> https://repository.apache.org/content/repositories/snapshots/com/sun/jdmk/jmxtools/1.2.1/jmxtools-1.2.1.jar
>>> [ivy:resolve] ==== maven2: tried
>>> [ivy:resolve]
>>>
>>>
>> http://repo1.maven.org/maven2/com/sun/jdmk/jmxtools/1.2.1/jmxtools-1.2.1.jar
>>> [ivy:resolve] [FAILED ] com.sun.jmx#jmxri;1.2.1!jmxri.jar: (0ms)
>>> [ivy:resolve] ==== fs: tried
>>> [ivy:resolve]
>>> /Users/tucu/.m2/repository/com/sun/jmx/jmxri/1.2.1/jmxri-1.2.1.jar
>>> [ivy:resolve] ==== apache-snapshot: tried
>>> [ivy:resolve]
>>>
>>>
>> https://repository.apache.org/content/repositories/snapshots/com/sun/jmx/jmxri/1.2.1/jmxri-1.2.1.jar
>>> [ivy:resolve] ==== maven2: tried
>>> [ivy:resolve]
>>> http://repo1.maven.org/maven2/com/sun/jmx/jmxri/1.2.1/jmxri-1.2.1.jar
>>> [ivy:resolve] ::::::::::::::::::::::::::::::::::::::::::::::
>>> [ivy:resolve] :: FAILED DOWNLOADS ::
>>> [ivy:resolve] :: ^ see resolution messages for details ^ ::
>>> [ivy:resolve] ::::::::::::::::::::::::::::::::::::::::::::::
>>> [ivy:resolve] :: javax.jms#jms;1.1!jms.jar
>>> [ivy:resolve] :: com.sun.jdmk#jmxtools;1.2.1!jmxtools.jar
>>> [ivy:resolve] :: com.sun.jmx#jmxri;1.2.1!jmxri.jar
>>> [ivy:resolve] ::::::::::::::::::::::::::::::::::::::::::::::
>>> [ivy:resolve]
>>> [ivy:resolve] :: USE VERBOSE OR DEBUG MESSAGE LEVEL FOR MORE DETAILS
>>>
>>
>
Re: need some ivy help (for MAPREDUCE-2741)
Posted by Alejandro Abdelnur <tu...@cloudera.com>.
Giri,
It is 'compile->master' already.
the download attempt is post compilation, in the 'ivy-resolve-test' target.
Thxs.
Alejandro
On Fri, Jul 29, 2011 at 2:05 PM, Giridharan Kesavan <
gkesavan@hortonworks.com> wrote:
> Alejandro,
>
> conf="compile->master"
> Always retain the lo4j conf as compile->master this way ivy won't download
> the transient deps. When compile->default ivy would download the transient
> deps.
> jmx/jmx/jdmk are transient deps of log4j.
>
>
> -Giri
>
> On Fri, Jul 29, 2011 at 11:35 AM, Alejandro Abdelnur <tucu@cloudera.com
> >wrote:
>
> > I'm trying to get mapred to work using hadoop-commons generated by
> > HADOOP-6671.
> >
> > I've tweaked all ivy.xml files in mapreduce to make lgo4j exclude
> > jms/jmx/jdmk.
> >
> > I've able to run successfully 'ant jar'
> >
> > But when running 'ant tar' things fail with the output at the end of this
> > message.
> >
> > I've been trying -and failing- to figure out all morning how ivy:resolve
> > works and why is still pulling log4j excluded dependencies.
> >
> > Any hint on why the exclusions are not be considered by the
> > 'ivy-resolve-test' target?
> >
> > Thxs.
> >
> > Alejandro
> >
> > ivy-resolve-test:
> > [ivy:resolve] downloading
> >
> >
> http://repo1.maven.org/maven2/org/mortbay/jetty/jetty-servlet-tester/6.1.14/jetty-servlet-tester-6.1.14.jar
> .
> > ..
> > [ivy:resolve] ..... (8kB)
> > [ivy:resolve] .. (0kB)
> > [ivy:resolve] [SUCCESSFUL ]
> > org.mortbay.jetty#jetty-servlet-tester;6.1.14!jetty-servlet-tester.jar
> > (908ms)
> > [ivy:resolve]
> > [ivy:resolve] :: problems summary ::
> > [ivy:resolve] :::: WARNINGS
> > [ivy:resolve] [FAILED ] javax.jms#jms;1.1!jms.jar: (0ms)
> > [ivy:resolve] ==== fs: tried
> > [ivy:resolve] /Users/tucu/.m2/repository/javax/jms/jms/1.1/jms-1.1.jar
> > [ivy:resolve] ==== apache-snapshot: tried
> > [ivy:resolve]
> >
> >
> https://repository.apache.org/content/repositories/snapshots/javax/jms/jms/1.1/jms-1.1.jar
> > [ivy:resolve] ==== maven2: tried
> > [ivy:resolve]
> > http://repo1.maven.org/maven2/javax/jms/jms/1.1/jms-1.1.jar
> > [ivy:resolve] [FAILED ] com.sun.jdmk#jmxtools;1.2.1!jmxtools.jar:
> > (0ms)
> > [ivy:resolve] ==== fs: tried
> > [ivy:resolve]
> >
> /Users/tucu/.m2/repository/com/sun/jdmk/jmxtools/1.2.1/jmxtools-1.2.1.jar
> > [ivy:resolve] ==== apache-snapshot: tried
> > [ivy:resolve]
> >
> >
> https://repository.apache.org/content/repositories/snapshots/com/sun/jdmk/jmxtools/1.2.1/jmxtools-1.2.1.jar
> > [ivy:resolve] ==== maven2: tried
> > [ivy:resolve]
> >
> >
> http://repo1.maven.org/maven2/com/sun/jdmk/jmxtools/1.2.1/jmxtools-1.2.1.jar
> > [ivy:resolve] [FAILED ] com.sun.jmx#jmxri;1.2.1!jmxri.jar: (0ms)
> > [ivy:resolve] ==== fs: tried
> > [ivy:resolve]
> > /Users/tucu/.m2/repository/com/sun/jmx/jmxri/1.2.1/jmxri-1.2.1.jar
> > [ivy:resolve] ==== apache-snapshot: tried
> > [ivy:resolve]
> >
> >
> https://repository.apache.org/content/repositories/snapshots/com/sun/jmx/jmxri/1.2.1/jmxri-1.2.1.jar
> > [ivy:resolve] ==== maven2: tried
> > [ivy:resolve]
> > http://repo1.maven.org/maven2/com/sun/jmx/jmxri/1.2.1/jmxri-1.2.1.jar
> > [ivy:resolve] ::::::::::::::::::::::::::::::::::::::::::::::
> > [ivy:resolve] :: FAILED DOWNLOADS ::
> > [ivy:resolve] :: ^ see resolution messages for details ^ ::
> > [ivy:resolve] ::::::::::::::::::::::::::::::::::::::::::::::
> > [ivy:resolve] :: javax.jms#jms;1.1!jms.jar
> > [ivy:resolve] :: com.sun.jdmk#jmxtools;1.2.1!jmxtools.jar
> > [ivy:resolve] :: com.sun.jmx#jmxri;1.2.1!jmxri.jar
> > [ivy:resolve] ::::::::::::::::::::::::::::::::::::::::::::::
> > [ivy:resolve]
> > [ivy:resolve] :: USE VERBOSE OR DEBUG MESSAGE LEVEL FOR MORE DETAILS
> >
>
Re: need some ivy help (for MAPREDUCE-2741)
Posted by Giridharan Kesavan <gk...@hortonworks.com>.
Alejandro,
conf="compile->master"
Always retain the lo4j conf as compile->master this way ivy won't download
the transient deps. When compile->default ivy would download the transient
deps.
jmx/jmx/jdmk are transient deps of log4j.
-Giri
On Fri, Jul 29, 2011 at 11:35 AM, Alejandro Abdelnur <tu...@cloudera.com>wrote:
> I'm trying to get mapred to work using hadoop-commons generated by
> HADOOP-6671.
>
> I've tweaked all ivy.xml files in mapreduce to make lgo4j exclude
> jms/jmx/jdmk.
>
> I've able to run successfully 'ant jar'
>
> But when running 'ant tar' things fail with the output at the end of this
> message.
>
> I've been trying -and failing- to figure out all morning how ivy:resolve
> works and why is still pulling log4j excluded dependencies.
>
> Any hint on why the exclusions are not be considered by the
> 'ivy-resolve-test' target?
>
> Thxs.
>
> Alejandro
>
> ivy-resolve-test:
> [ivy:resolve] downloading
>
> http://repo1.maven.org/maven2/org/mortbay/jetty/jetty-servlet-tester/6.1.14/jetty-servlet-tester-6.1.14.jar.
> ..
> [ivy:resolve] ..... (8kB)
> [ivy:resolve] .. (0kB)
> [ivy:resolve] [SUCCESSFUL ]
> org.mortbay.jetty#jetty-servlet-tester;6.1.14!jetty-servlet-tester.jar
> (908ms)
> [ivy:resolve]
> [ivy:resolve] :: problems summary ::
> [ivy:resolve] :::: WARNINGS
> [ivy:resolve] [FAILED ] javax.jms#jms;1.1!jms.jar: (0ms)
> [ivy:resolve] ==== fs: tried
> [ivy:resolve] /Users/tucu/.m2/repository/javax/jms/jms/1.1/jms-1.1.jar
> [ivy:resolve] ==== apache-snapshot: tried
> [ivy:resolve]
>
> https://repository.apache.org/content/repositories/snapshots/javax/jms/jms/1.1/jms-1.1.jar
> [ivy:resolve] ==== maven2: tried
> [ivy:resolve]
> http://repo1.maven.org/maven2/javax/jms/jms/1.1/jms-1.1.jar
> [ivy:resolve] [FAILED ] com.sun.jdmk#jmxtools;1.2.1!jmxtools.jar:
> (0ms)
> [ivy:resolve] ==== fs: tried
> [ivy:resolve]
> /Users/tucu/.m2/repository/com/sun/jdmk/jmxtools/1.2.1/jmxtools-1.2.1.jar
> [ivy:resolve] ==== apache-snapshot: tried
> [ivy:resolve]
>
> https://repository.apache.org/content/repositories/snapshots/com/sun/jdmk/jmxtools/1.2.1/jmxtools-1.2.1.jar
> [ivy:resolve] ==== maven2: tried
> [ivy:resolve]
>
> http://repo1.maven.org/maven2/com/sun/jdmk/jmxtools/1.2.1/jmxtools-1.2.1.jar
> [ivy:resolve] [FAILED ] com.sun.jmx#jmxri;1.2.1!jmxri.jar: (0ms)
> [ivy:resolve] ==== fs: tried
> [ivy:resolve]
> /Users/tucu/.m2/repository/com/sun/jmx/jmxri/1.2.1/jmxri-1.2.1.jar
> [ivy:resolve] ==== apache-snapshot: tried
> [ivy:resolve]
>
> https://repository.apache.org/content/repositories/snapshots/com/sun/jmx/jmxri/1.2.1/jmxri-1.2.1.jar
> [ivy:resolve] ==== maven2: tried
> [ivy:resolve]
> http://repo1.maven.org/maven2/com/sun/jmx/jmxri/1.2.1/jmxri-1.2.1.jar
> [ivy:resolve] ::::::::::::::::::::::::::::::::::::::::::::::
> [ivy:resolve] :: FAILED DOWNLOADS ::
> [ivy:resolve] :: ^ see resolution messages for details ^ ::
> [ivy:resolve] ::::::::::::::::::::::::::::::::::::::::::::::
> [ivy:resolve] :: javax.jms#jms;1.1!jms.jar
> [ivy:resolve] :: com.sun.jdmk#jmxtools;1.2.1!jmxtools.jar
> [ivy:resolve] :: com.sun.jmx#jmxri;1.2.1!jmxri.jar
> [ivy:resolve] ::::::::::::::::::::::::::::::::::::::::::::::
> [ivy:resolve]
> [ivy:resolve] :: USE VERBOSE OR DEBUG MESSAGE LEVEL FOR MORE DETAILS
>