You are viewing a plain text version of this content. The canonical link for it is here.

Posted to user@spark.apache.org by Todd <bi...@163.com> on 2016/05/23 03:48:58 UTC

How spark depends on Guava

Hi,
In the spark code, guava maven dependency scope is provided, my question is, how spark depends on guava during runtime? I looked into the spark-assembly-1.6.1-hadoop2.6.1.jar,and didn't find class entries like com.google.common.base.Preconditions etc...

Re:Re: How spark depends on Guava

Posted by Todd <bi...@163.com>.

Thanks Mat.
When you look at the spark assembly jar(such as spark-assembly-1.6.0-hadoop2.6.1.jar), you will find that there is very few classes belonging to guava library.
So i am wondering where guava library comes into play during run-time.






At 2016-05-23 15:42:51, "Mat Schaffer" <ma...@schaffer.me> wrote:

I got curious so I tried sbt dependencyTree. Looks like Guava comes into spark core from a couple places.




-Mat


matschaffer.com


On Mon, May 23, 2016 at 2:32 PM, Todd <bi...@163.com> wrote:



Can someone please take alook at my question?I am spark-shell local mode and yarn-client mode.Spark code uses guava library,spark should have guava in place during run time.


Thanks.





At 2016-05-23 11:48:58, "Todd" <bi...@163.com> wrote:

Hi,
In the spark code, guava maven dependency scope is provided, my question is, how spark depends on guava during runtime? I looked into the spark-assembly-1.6.1-hadoop2.6.1.jar,and didn't find class entries like com.google.common.base.Preconditions etc...

Re: How spark depends on Guava

Posted by Mat Schaffer <ma...@schaffer.me>.

I got curious so I tried sbt dependencyTree. Looks like Guava comes into
spark core from a couple places.


-Mat

matschaffer.com

On Mon, May 23, 2016 at 2:32 PM, Todd <bi...@163.com> wrote:

>
> Can someone please take alook at my question?I am spark-shell local mode
> and yarn-client mode.Spark code uses guava library,spark should have guava
> in place during run time.
>
> Thanks.
>
>
>
> At 2016-05-23 11:48:58, "Todd" <bi...@163.com> wrote:
>
> Hi,
> In the spark code, guava maven dependency scope is provided, my question
> is, how spark depends on guava during runtime? I looked into the
> spark-assembly-1.6.1-hadoop2.6.1.jar,and didn't find class entries like
> com.google.common.base.Preconditions etc...
>
>

Re: How spark depends on Guava

Posted by Steve Loughran <st...@hortonworks.com>.

On 23 May 2016, at 06:32, Todd <bi...@163.com>> wrote:


Can someone please take alook at my question?I am spark-shell local mode and yarn-client mode.Spark code uses guava library,spark should have guava in place during run time.

Thanks.



At 2016-05-23 11:48:58, "Todd" <bi...@163.com>> wrote:
Hi,
In the spark code, guava maven dependency scope is provided, my question is, how spark depends on guava during runtime? I looked into the spark-assembly-1.6.1-hadoop2.6.1.jar,and didn't find class entries like com.google.common.base.Preconditions etc...

Spark "shades" guava on import into the assembly; the libaries are moved to a new package and all internal references with it.

This is because guava is a nightmare of backwards compatibility.

if you want guava, decide which version you want and ask for it explicitly

Re:How spark depends on Guava

Posted by Todd <bi...@163.com>.


Can someone please take alook at my question?I am spark-shell local mode and yarn-client mode.Spark code uses guava library,spark should have guava in place during run time.


Thanks.





At 2016-05-23 11:48:58, "Todd" <bi...@163.com> wrote:

Hi,
In the spark code, guava maven dependency scope is provided, my question is, how spark depends on guava during runtime? I looked into the spark-assembly-1.6.1-hadoop2.6.1.jar,and didn't find class entries like com.google.common.base.Preconditions etc...

Re: How spark depends on Guava

Posted by Jacek Laskowski <ja...@japila.pl>.

Hi Todd,

It's used heavily for thread pool executors for one. Don't know about other
uses.

Jacek
On 23 May 2016 5:49 a.m., "Todd" <bi...@163.com> wrote:

> Hi,
> In the spark code, guava maven dependency scope is provided, my question
> is, how spark depends on guava during runtime? I looked into the
> spark-assembly-1.6.1-hadoop2.6.1.jar,and didn't find class entries like
> com.google.common.base.Preconditions etc...
>