You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@tez.apache.org by Chris K Wensel <ch...@wensel.net> on 2015/02/26 19:59:37 UTC
Guava deps
I’m running Tez 0.6.0 with the default dependency of Hadoop 2.6 and am receiving a fair number of these
Exception in thread "org.apache.hadoop.util.JvmPauseMonitor$Monitor@74173125" java.lang.NoSuchMethodError: com.google.common.base.Stopwatch.elapsedMillis()J
at org.apache.hadoop.util.JvmPauseMonitor$Monitor.run(JvmPauseMonitor.java:184)
at java.lang.Thread.run(Thread.java:722)
of which result in a complete failure to run Tez
This is due to Tez/Yarn building against Guava 11.0.2,
org.apache.tez:tez-api:0.6.0
| +--- log4j:log4j:1.2.17
| +--- com.google.guava:guava:11.0.2 -> 16.0.1
but Curator requiring 16.0.1 and forcing the up version
org.apache.curator:curator-framework:2.6.0
| | | | | +--- org.apache.curator:curator-client:2.6.0
| | | | | | +--- org.slf4j:slf4j-api:1.7.6
| | | | | | +--- org.apache.zookeeper:zookeeper:3.4.6 (*)
| | | | | | \--- com.google.guava:guava:16.0.1
this makes me wonder how Tez 0.6.0 passes any tests (a side effect of maven?), let alone actually run on a cluster as curator is used by auth which is used by common
or is it a happy coincidence that auth doesn’t use any guava api’s not available in 11.0.2? thus it’s safe to force a downgrade (or put 11.0.2 first in the classpath).
ckw
—
Chris K Wensel
chris@wensel.net
Re: Guava deps
Posted by Chris K Wensel <ch...@wensel.net>.
It looks like this is coming down to Ivy vs Maven conflict management.
Maven chooses nearest, Ivy chooses latest.
I'm forcing to guava 11 to be consistent with how maven resolves dependencies.
ckw
> On Feb 26, 2015, at 4:01 PM, Siddharth Seth <ss...@apache.org> wrote:
>
> Chris,
>
> This is what I see by running a mvn dependency:tree on hadoop. Both seem to
> be pulling in guava-11. This is a little strange since curator changed to
> 16 in March 2014, and before that were on 14.
>
> Hadoop-2.6
> [INFO] +- org.apache.curator:curator-framework:jar:2.6.0:compile
> [INFO] | +- org.apache.curator:curator-client:jar:2.6.0:compile
> [INFO] | \- com.google.guava:guava:jar:11.0.2:compile
> [INFO] | \- com.google.code.findbugs:jsr305:jar:1.3.9:compile
>
> branch-2
> [INFO] +- org.apache.curator:curator-framework:jar:2.7.1:compile
> [INFO] | +- org.apache.curator:curator-client:jar:2.7.1:compile
> [INFO] | \- com.google.guava:guava:jar:11.0.2:compile
> [INFO] | \- com.google.code.findbugs:jsr305:jar:3.0.0:compile (version
> managed from 1.3.9)
>
>
> I'm guessing that maven does something here - either excluding the curator
> guava dependency from being pulled in by Hadoop, or it's fixing versions.
>
> The dist directory for Hadoop contains only the guava-11 jar. The same
> applies to Tez.
>
> I don't think auth will be using any dependencies from 16 given what the
> maven dependency tree gives.
> There's been a fair amount of discussion on Hadoop jiras to update the
> Guava version - I'm not sure what the resolution is though.
>
> From the way Hadoop gets only guava11 in it's classpath, I'd say it's safe
> to remove the Guava16 jar in your setup.
>
> Is first in classpath a valid assumption ? It does work - but I'm not sure
> there's any guarantees / contracts for this to actually work.
>
> On a related note, Tez may end up shading the guava version it uses - and
> remove it from dependencies that are exported.
>
>
> On Thu, Feb 26, 2015 at 10:59 AM, Chris K Wensel <ch...@wensel.net> wrote:
>
>> I’m running Tez 0.6.0 with the default dependency of Hadoop 2.6 and am
>> receiving a fair number of these
>>
>> Exception in thread
>> "org.apache.hadoop.util.JvmPauseMonitor$Monitor@74173125"
>> java.lang.NoSuchMethodError:
>> com.google.common.base.Stopwatch.elapsedMillis()J
>> at
>> org.apache.hadoop.util.JvmPauseMonitor$Monitor.run(JvmPauseMonitor.java:184)
>> at java.lang.Thread.run(Thread.java:722)
>>
>> of which result in a complete failure to run Tez
>>
>> This is due to Tez/Yarn building against Guava 11.0.2,
>>
>> org.apache.tez:tez-api:0.6.0
>> | +--- log4j:log4j:1.2.17
>> | +--- com.google.guava:guava:11.0.2 -> 16.0.1
>>
>> but Curator requiring 16.0.1 and forcing the up version
>>
>> org.apache.curator:curator-framework:2.6.0
>> | | | | | +--- org.apache.curator:curator-client:2.6.0
>> | | | | | | +--- org.slf4j:slf4j-api:1.7.6
>> | | | | | | +--- org.apache.zookeeper:zookeeper:3.4.6 (*)
>> | | | | | | \--- com.google.guava:guava:16.0.1
>>
>> this makes me wonder how Tez 0.6.0 passes any tests (a side effect of
>> maven?), let alone actually run on a cluster as curator is used by auth
>> which is used by common
>>
>> or is it a happy coincidence that auth doesn’t use any guava api’s not
>> available in 11.0.2? thus it’s safe to force a downgrade (or put 11.0.2
>> first in the classpath).
>>
>> ckw
>>
>> —
>> Chris K Wensel
>> chris@wensel.net
>>
>>
>>
>>
>>
—
Chris K Wensel
chris@wensel.net
Re: Guava deps
Posted by Siddharth Seth <ss...@apache.org>.
Chris,
This is what I see by running a mvn dependency:tree on hadoop. Both seem to
be pulling in guava-11. This is a little strange since curator changed to
16 in March 2014, and before that were on 14.
Hadoop-2.6
[INFO] +- org.apache.curator:curator-framework:jar:2.6.0:compile
[INFO] | +- org.apache.curator:curator-client:jar:2.6.0:compile
[INFO] | \- com.google.guava:guava:jar:11.0.2:compile
[INFO] | \- com.google.code.findbugs:jsr305:jar:1.3.9:compile
branch-2
[INFO] +- org.apache.curator:curator-framework:jar:2.7.1:compile
[INFO] | +- org.apache.curator:curator-client:jar:2.7.1:compile
[INFO] | \- com.google.guava:guava:jar:11.0.2:compile
[INFO] | \- com.google.code.findbugs:jsr305:jar:3.0.0:compile (version
managed from 1.3.9)
I'm guessing that maven does something here - either excluding the curator
guava dependency from being pulled in by Hadoop, or it's fixing versions.
The dist directory for Hadoop contains only the guava-11 jar. The same
applies to Tez.
I don't think auth will be using any dependencies from 16 given what the
maven dependency tree gives.
There's been a fair amount of discussion on Hadoop jiras to update the
Guava version - I'm not sure what the resolution is though.
>From the way Hadoop gets only guava11 in it's classpath, I'd say it's safe
to remove the Guava16 jar in your setup.
Is first in classpath a valid assumption ? It does work - but I'm not sure
there's any guarantees / contracts for this to actually work.
On a related note, Tez may end up shading the guava version it uses - and
remove it from dependencies that are exported.
On Thu, Feb 26, 2015 at 10:59 AM, Chris K Wensel <ch...@wensel.net> wrote:
> I’m running Tez 0.6.0 with the default dependency of Hadoop 2.6 and am
> receiving a fair number of these
>
> Exception in thread
> "org.apache.hadoop.util.JvmPauseMonitor$Monitor@74173125"
> java.lang.NoSuchMethodError:
> com.google.common.base.Stopwatch.elapsedMillis()J
> at
> org.apache.hadoop.util.JvmPauseMonitor$Monitor.run(JvmPauseMonitor.java:184)
> at java.lang.Thread.run(Thread.java:722)
>
> of which result in a complete failure to run Tez
>
> This is due to Tez/Yarn building against Guava 11.0.2,
>
> org.apache.tez:tez-api:0.6.0
> | +--- log4j:log4j:1.2.17
> | +--- com.google.guava:guava:11.0.2 -> 16.0.1
>
> but Curator requiring 16.0.1 and forcing the up version
>
> org.apache.curator:curator-framework:2.6.0
> | | | | | +--- org.apache.curator:curator-client:2.6.0
> | | | | | | +--- org.slf4j:slf4j-api:1.7.6
> | | | | | | +--- org.apache.zookeeper:zookeeper:3.4.6 (*)
> | | | | | | \--- com.google.guava:guava:16.0.1
>
> this makes me wonder how Tez 0.6.0 passes any tests (a side effect of
> maven?), let alone actually run on a cluster as curator is used by auth
> which is used by common
>
> or is it a happy coincidence that auth doesn’t use any guava api’s not
> available in 11.0.2? thus it’s safe to force a downgrade (or put 11.0.2
> first in the classpath).
>
> ckw
>
> —
> Chris K Wensel
> chris@wensel.net
>
>
>
>
>