You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@spark.apache.org by Shailesh Birari <sb...@gmail.com> on 2015/01/21 00:55:51 UTC

Spark 1.2 - com/google/common/base/Preconditions java.lang.NoClassDefFoundErro

Hello,

I recently upgraded my setup from Spark 1.1 to Spark 1.2.
My existing applications are working fine on ubuntu cluster.
But, when I try to execute Spark MLlib application from Eclipse (Windows
node) it gives java.lang.NoClassDefFoundError:
com/google/common/base/Preconditions exception.

Note, 
   1. With Spark 1.1 this was working fine.
   2. The Spark 1.2 jar files are linked in Eclipse project.
   3. Checked the jar -tf output and found the above com.google.common.base
is not present.

-----------------------------------------------------------------------------------------------------------------Exception
log:

Exception in thread "main" java.lang.NoClassDefFoundError:
com/google/common/base/Preconditions
	at
org.apache.spark.network.client.TransportClientFactory.<init>(TransportClientFactory.java:94)
	at
org.apache.spark.network.TransportContext.createClientFactory(TransportContext.java:77)
	at
org.apache.spark.network.netty.NettyBlockTransferService.init(NettyBlockTransferService.scala:62)
	at org.apache.spark.storage.BlockManager.initialize(BlockManager.scala:194)
	at org.apache.spark.SparkContext.<init>(SparkContext.scala:340)
	at
org.apache.spark.examples.mllib.TallSkinnySVD$.main(TallSkinnySVD.scala:74)
	at org.apache.spark.examples.mllib.TallSkinnySVD.main(TallSkinnySVD.scala)
Caused by: java.lang.ClassNotFoundException:
com.google.common.base.Preconditions
	at java.net.URLClassLoader$1.run(Unknown Source)
	at java.net.URLClassLoader$1.run(Unknown Source)
	at java.security.AccessController.doPrivileged(Native Method)
	at java.net.URLClassLoader.findClass(Unknown Source)
	at java.lang.ClassLoader.loadClass(Unknown Source)
	at sun.misc.Launcher$AppClassLoader.loadClass(Unknown Source)
	at java.lang.ClassLoader.loadClass(Unknown Source)
	... 7 more

-----------------------------------------------------------------------------------------------------------------

jar -tf output:


consb2@CONSB2A
/cygdrive/c/SB/spark-1.2.0-bin-hadoop2.4/spark-1.2.0-bin-hadoop2.4/lib
$ jar -tf spark-assembly-1.2.0-hadoop2.4.0.jar | grep Preconditions
org/spark-project/guava/common/base/Preconditions.class
org/spark-project/guava/common/math/MathPreconditions.class
com/clearspring/analytics/util/Preconditions.class
parquet/Preconditions.class
com/google/inject/internal/util/$Preconditions.class

---------------------------------------------------------------------------------------------------------------

Please help me in resolving this.

Thanks,
  Shailesh




--
View this message in context: http://apache-spark-user-list.1001560.n3.nabble.com/Spark-1-2-com-google-common-base-Preconditions-java-lang-NoClassDefFoundErro-tp21271.html
Sent from the Apache Spark User List mailing list archive at Nabble.com.

---------------------------------------------------------------------
To unsubscribe, e-mail: user-unsubscribe@spark.apache.org
For additional commands, e-mail: user-help@spark.apache.org


RE: Spark 1.2 - com/google/common/base/Preconditions java.lang.NoClassDefFoundErro

Posted by Bob Tiernay <bt...@hotmail.com>.
If using Maven, one simply use whatever version they prefer and at build time and the artifact using something like:
  <build>    <plugins>      <plugin>        <groupId>org.apache.maven.plugins</groupId>        <artifactId>maven-shade-plugin</artifactId>        <executions>          <execution>            <phase>package</phase>            <goals>              <goal>shade</goal>            </goals>            <configuration>
              <relocations>                <relocation>                  <pattern>com.google.common</pattern>                  <shadedPattern>org.shaded.google.common</shadedPattern>                </relocation>              </relocations>

            </configuration>          </execution>        </executions>      </plugin>    </plugins>  </build>




Of course this won't help during development if there are conflicts.

From: ilikerps@gmail.com
Date: Tue, 20 Jan 2015 18:26:32 -0800
Subject: Re: Spark 1.2 - com/google/common/base/Preconditions java.lang.NoClassDefFoundErro
To: sbirari77@gmail.com
CC: fnothaft@berkeley.edu; sowen@cloudera.com; user@spark.apache.org

Spark's network-common package depends on guava as a "provided" dependency in order to avoid conflicting with other libraries (e.g., Hadoop) that depend on specific versions. com/google/common/base/Preconditions has been present in Guava since version 2, so this is likely a "dependency not found" rather than "wrong version of dependency" issue.
To resolve this, please depend on some version of Guava (14.0.1 is guaranteed to work, as should any other version from the past few years).
On Tue, Jan 20, 2015 at 6:16 PM, Shailesh Birari <sb...@gmail.com> wrote:
Hi Frank,
Its a normal eclipse project where I added Scala and Spark libraries as user libraries.Though, I am not attaching any hadoop libraries, in my application code I have following line.
  System.setProperty("hadoop.home.dir", "C:\\SB\\HadoopWin")

This Hadoop home dir contains "winutils.exe" only.
Don't think that its an issue.
Please suggest.
Thanks,  Shailesh

On Wed, Jan 21, 2015 at 2:19 PM, Frank Austin Nothaft <fn...@berkeley.edu> wrote:
Shailesh,
To add, are you packaging Hadoop in your app? Hadoop will pull in Guava. Not sure if you are using Maven (or what) to build, but if you can pull up your builds dependency tree, you will likely find com.google.guava being brought in by one of your dependencies.
Regards,

Frank Austin Nothaftfnothaft@berkeley.edufnothaft@eecs.berkeley.edu202-340-0466



On Jan 20, 2015, at 5:13 PM, Shailesh Birari <sb...@gmail.com> wrote:
Hello,
I double checked the libraries. I am linking only with Spark 1.2.Along with Spark 1.2 jars I have Scala 2.10 jars and JRE 7 jars linked and nothing else.
Thanks,   Shailesh
On Wed, Jan 21, 2015 at 12:58 PM, Sean Owen <so...@cloudera.com> wrote:
Guava is shaded in Spark 1.2+. It looks like you are mixing versions

of Spark then, with some that still refer to unshaded Guava. Make sure

you are not packaging Spark with your app and that you don't have

other versions lying around.



On Tue, Jan 20, 2015 at 11:55 PM, Shailesh Birari <sb...@gmail.com> wrote:

> Hello,

>

> I recently upgraded my setup from Spark 1.1 to Spark 1.2.

> My existing applications are working fine on ubuntu cluster.

> But, when I try to execute Spark MLlib application from Eclipse (Windows

> node) it gives java.lang.NoClassDefFoundError:

> com/google/common/base/Preconditions exception.

>

> Note,

>    1. With Spark 1.1 this was working fine.

>    2. The Spark 1.2 jar files are linked in Eclipse project.

>    3. Checked the jar -tf output and found the above com.google.common.base

> is not present.

>

> -----------------------------------------------------------------------------------------------------------------Exception

> log:

>

> Exception in thread "main" java.lang.NoClassDefFoundError:

> com/google/common/base/Preconditions

>         at

> org.apache.spark.network.client.TransportClientFactory.<init>(TransportClientFactory.java:94)

>         at

> org.apache.spark.network.TransportContext.createClientFactory(TransportContext.java:77)

>         at

> org.apache.spark.network.netty.NettyBlockTransferService.init(NettyBlockTransferService.scala:62)

>         at org.apache.spark.storage.BlockManager.initialize(BlockManager.scala:194)

>         at org.apache.spark.SparkContext.<init>(SparkContext.scala:340)

>         at

> org.apache.spark.examples.mllib.TallSkinnySVD$.main(TallSkinnySVD.scala:74)

>         at org.apache.spark.examples.mllib.TallSkinnySVD.main(TallSkinnySVD.scala)

> Caused by: java.lang.ClassNotFoundException:

> com.google.common.base.Preconditions

>         at java.net.URLClassLoader$1.run(Unknown Source)

>         at java.net.URLClassLoader$1.run(Unknown Source)

>         at java.security.AccessController.doPrivileged(Native Method)

>         at java.net.URLClassLoader.findClass(Unknown Source)

>         at java.lang.ClassLoader.loadClass(Unknown Source)

>         at sun.misc.Launcher$AppClassLoader.loadClass(Unknown Source)

>         at java.lang.ClassLoader.loadClass(Unknown Source)

>         ... 7 more

>

> -----------------------------------------------------------------------------------------------------------------

>

> jar -tf output:

>

>

> consb2@CONSB2A

> /cygdrive/c/SB/spark-1.2.0-bin-hadoop2.4/spark-1.2.0-bin-hadoop2.4/lib

> $ jar -tf spark-assembly-1.2.0-hadoop2.4.0.jar | grep Preconditions

> org/spark-project/guava/common/base/Preconditions.class

> org/spark-project/guava/common/math/MathPreconditions.class

> com/clearspring/analytics/util/Preconditions.class

> parquet/Preconditions.class

> com/google/inject/internal/util/$Preconditions.class

>

> ---------------------------------------------------------------------------------------------------------------

>

> Please help me in resolving this.

>

> Thanks,

>   Shailesh

>

>

>

>

> --

> View this message in context: http://apache-spark-user-list.1001560.n3.nabble.com/Spark-1-2-com-google-common-base-Preconditions-java-lang-NoClassDefFoundErro-tp21271.html

> Sent from the Apache Spark User List mailing list archive at Nabble.com.

>

> ---------------------------------------------------------------------

> To unsubscribe, e-mail: user-unsubscribe@spark.apache.org

> For additional commands, e-mail: user-help@spark.apache.org

>







 		 	   		  

Re: Spark 1.2 - com/google/common/base/Preconditions java.lang.NoClassDefFoundErro

Posted by Shailesh Birari <sb...@gmail.com>.
Thanks Aaron.

Adding Guava jar resolves the issue.

Shailesh

On Wed, Jan 21, 2015 at 3:26 PM, Aaron Davidson <il...@gmail.com> wrote:

> Spark's network-common package depends on guava as a "provided" dependency
> in order to avoid conflicting with other libraries (e.g., Hadoop) that
> depend on specific versions. com/google/common/base/Preconditions has been
> present in Guava since version 2, so this is likely a "dependency not
> found" rather than "wrong version of dependency" issue.
>
> To resolve this, please depend on some version of Guava (14.0.1 is
> guaranteed to work, as should any other version from the past few years).
>
> On Tue, Jan 20, 2015 at 6:16 PM, Shailesh Birari <sb...@gmail.com>
> wrote:
>
>> Hi Frank,
>>
>> Its a normal eclipse project where I added Scala and Spark libraries as
>> user libraries.
>> Though, I am not attaching any hadoop libraries, in my application code I
>> have following line.
>>
>>   System.setProperty("hadoop.home.dir", "C:\\SB\\HadoopWin")
>>
>> This Hadoop home dir contains "winutils.exe" only.
>>
>> Don't think that its an issue.
>>
>> Please suggest.
>>
>> Thanks,
>>   Shailesh
>>
>>
>> On Wed, Jan 21, 2015 at 2:19 PM, Frank Austin Nothaft <
>> fnothaft@berkeley.edu> wrote:
>>
>>> Shailesh,
>>>
>>> To add, are you packaging Hadoop in your app? Hadoop will pull in Guava.
>>> Not sure if you are using Maven (or what) to build, but if you can pull up
>>> your builds dependency tree, you will likely find com.google.guava being
>>> brought in by one of your dependencies.
>>>
>>> Regards,
>>>
>>> Frank Austin Nothaft
>>> fnothaft@berkeley.edu
>>> fnothaft@eecs.berkeley.edu
>>> 202-340-0466
>>>
>>> On Jan 20, 2015, at 5:13 PM, Shailesh Birari <sb...@gmail.com>
>>> wrote:
>>>
>>> Hello,
>>>
>>> I double checked the libraries. I am linking only with Spark 1.2.
>>> Along with Spark 1.2 jars I have Scala 2.10 jars and JRE 7 jars linked
>>> and nothing else.
>>>
>>> Thanks,
>>>    Shailesh
>>>
>>> On Wed, Jan 21, 2015 at 12:58 PM, Sean Owen <so...@cloudera.com> wrote:
>>>
>>>> Guava is shaded in Spark 1.2+. It looks like you are mixing versions
>>>> of Spark then, with some that still refer to unshaded Guava. Make sure
>>>> you are not packaging Spark with your app and that you don't have
>>>> other versions lying around.
>>>>
>>>> On Tue, Jan 20, 2015 at 11:55 PM, Shailesh Birari <sb...@gmail.com>
>>>> wrote:
>>>> > Hello,
>>>> >
>>>> > I recently upgraded my setup from Spark 1.1 to Spark 1.2.
>>>> > My existing applications are working fine on ubuntu cluster.
>>>> > But, when I try to execute Spark MLlib application from Eclipse
>>>> (Windows
>>>> > node) it gives java.lang.NoClassDefFoundError:
>>>> > com/google/common/base/Preconditions exception.
>>>> >
>>>> > Note,
>>>> >    1. With Spark 1.1 this was working fine.
>>>> >    2. The Spark 1.2 jar files are linked in Eclipse project.
>>>> >    3. Checked the jar -tf output and found the above
>>>> com.google.common.base
>>>> > is not present.
>>>> >
>>>> >
>>>> -----------------------------------------------------------------------------------------------------------------Exception
>>>> > log:
>>>> >
>>>> > Exception in thread "main" java.lang.NoClassDefFoundError:
>>>> > com/google/common/base/Preconditions
>>>> >         at
>>>> >
>>>> org.apache.spark.network.client.TransportClientFactory.<init>(TransportClientFactory.java:94)
>>>> >         at
>>>> >
>>>> org.apache.spark.network.TransportContext.createClientFactory(TransportContext.java:77)
>>>> >         at
>>>> >
>>>> org.apache.spark.network.netty.NettyBlockTransferService.init(NettyBlockTransferService.scala:62)
>>>> >         at
>>>> org.apache.spark.storage.BlockManager.initialize(BlockManager.scala:194)
>>>> >         at
>>>> org.apache.spark.SparkContext.<init>(SparkContext.scala:340)
>>>> >         at
>>>> >
>>>> org.apache.spark.examples.mllib.TallSkinnySVD$.main(TallSkinnySVD.scala:74)
>>>> >         at
>>>> org.apache.spark.examples.mllib.TallSkinnySVD.main(TallSkinnySVD.scala)
>>>> > Caused by: java.lang.ClassNotFoundException:
>>>> > com.google.common.base.Preconditions
>>>> >         at java.net.URLClassLoader$1.run(Unknown Source)
>>>> >         at java.net.URLClassLoader$1.run(Unknown Source)
>>>> >         at java.security.AccessController.doPrivileged(Native Method)
>>>> >         at java.net.URLClassLoader.findClass(Unknown Source)
>>>> >         at java.lang.ClassLoader.loadClass(Unknown Source)
>>>> >         at sun.misc.Launcher$AppClassLoader.loadClass(Unknown Source)
>>>> >         at java.lang.ClassLoader.loadClass(Unknown Source)
>>>> >         ... 7 more
>>>> >
>>>> >
>>>> -----------------------------------------------------------------------------------------------------------------
>>>> >
>>>> > jar -tf output:
>>>> >
>>>> >
>>>> > consb2@CONSB2A
>>>> > /cygdrive/c/SB/spark-1.2.0-bin-hadoop2.4/spark-1.2.0-bin-hadoop2.4/lib
>>>> > $ jar -tf spark-assembly-1.2.0-hadoop2.4.0.jar | grep Preconditions
>>>> > org/spark-project/guava/common/base/Preconditions.class
>>>> > org/spark-project/guava/common/math/MathPreconditions.class
>>>> > com/clearspring/analytics/util/Preconditions.class
>>>> > parquet/Preconditions.class
>>>> > com/google/inject/internal/util/$Preconditions.class
>>>> >
>>>> >
>>>> ---------------------------------------------------------------------------------------------------------------
>>>> >
>>>> > Please help me in resolving this.
>>>> >
>>>> > Thanks,
>>>> >   Shailesh
>>>> >
>>>> >
>>>> >
>>>> >
>>>> > --
>>>> > View this message in context:
>>>> http://apache-spark-user-list.1001560.n3.nabble.com/Spark-1-2-com-google-common-base-Preconditions-java-lang-NoClassDefFoundErro-tp21271.html
>>>> > Sent from the Apache Spark User List mailing list archive at
>>>> Nabble.com.
>>>> >
>>>> > ---------------------------------------------------------------------
>>>> > To unsubscribe, e-mail: user-unsubscribe@spark.apache.org
>>>> > For additional commands, e-mail: user-help@spark.apache.org
>>>> >
>>>>
>>>
>>>
>>>
>>
>

Re: Spark 1.2 - com/google/common/base/Preconditions java.lang.NoClassDefFoundErro

Posted by Aaron Davidson <il...@gmail.com>.
Spark's network-common package depends on guava as a "provided" dependency
in order to avoid conflicting with other libraries (e.g., Hadoop) that
depend on specific versions. com/google/common/base/Preconditions has been
present in Guava since version 2, so this is likely a "dependency not
found" rather than "wrong version of dependency" issue.

To resolve this, please depend on some version of Guava (14.0.1 is
guaranteed to work, as should any other version from the past few years).

On Tue, Jan 20, 2015 at 6:16 PM, Shailesh Birari <sb...@gmail.com>
wrote:

> Hi Frank,
>
> Its a normal eclipse project where I added Scala and Spark libraries as
> user libraries.
> Though, I am not attaching any hadoop libraries, in my application code I
> have following line.
>
>   System.setProperty("hadoop.home.dir", "C:\\SB\\HadoopWin")
>
> This Hadoop home dir contains "winutils.exe" only.
>
> Don't think that its an issue.
>
> Please suggest.
>
> Thanks,
>   Shailesh
>
>
> On Wed, Jan 21, 2015 at 2:19 PM, Frank Austin Nothaft <
> fnothaft@berkeley.edu> wrote:
>
>> Shailesh,
>>
>> To add, are you packaging Hadoop in your app? Hadoop will pull in Guava.
>> Not sure if you are using Maven (or what) to build, but if you can pull up
>> your builds dependency tree, you will likely find com.google.guava being
>> brought in by one of your dependencies.
>>
>> Regards,
>>
>> Frank Austin Nothaft
>> fnothaft@berkeley.edu
>> fnothaft@eecs.berkeley.edu
>> 202-340-0466
>>
>> On Jan 20, 2015, at 5:13 PM, Shailesh Birari <sb...@gmail.com> wrote:
>>
>> Hello,
>>
>> I double checked the libraries. I am linking only with Spark 1.2.
>> Along with Spark 1.2 jars I have Scala 2.10 jars and JRE 7 jars linked
>> and nothing else.
>>
>> Thanks,
>>    Shailesh
>>
>> On Wed, Jan 21, 2015 at 12:58 PM, Sean Owen <so...@cloudera.com> wrote:
>>
>>> Guava is shaded in Spark 1.2+. It looks like you are mixing versions
>>> of Spark then, with some that still refer to unshaded Guava. Make sure
>>> you are not packaging Spark with your app and that you don't have
>>> other versions lying around.
>>>
>>> On Tue, Jan 20, 2015 at 11:55 PM, Shailesh Birari <sb...@gmail.com>
>>> wrote:
>>> > Hello,
>>> >
>>> > I recently upgraded my setup from Spark 1.1 to Spark 1.2.
>>> > My existing applications are working fine on ubuntu cluster.
>>> > But, when I try to execute Spark MLlib application from Eclipse
>>> (Windows
>>> > node) it gives java.lang.NoClassDefFoundError:
>>> > com/google/common/base/Preconditions exception.
>>> >
>>> > Note,
>>> >    1. With Spark 1.1 this was working fine.
>>> >    2. The Spark 1.2 jar files are linked in Eclipse project.
>>> >    3. Checked the jar -tf output and found the above
>>> com.google.common.base
>>> > is not present.
>>> >
>>> >
>>> -----------------------------------------------------------------------------------------------------------------Exception
>>> > log:
>>> >
>>> > Exception in thread "main" java.lang.NoClassDefFoundError:
>>> > com/google/common/base/Preconditions
>>> >         at
>>> >
>>> org.apache.spark.network.client.TransportClientFactory.<init>(TransportClientFactory.java:94)
>>> >         at
>>> >
>>> org.apache.spark.network.TransportContext.createClientFactory(TransportContext.java:77)
>>> >         at
>>> >
>>> org.apache.spark.network.netty.NettyBlockTransferService.init(NettyBlockTransferService.scala:62)
>>> >         at
>>> org.apache.spark.storage.BlockManager.initialize(BlockManager.scala:194)
>>> >         at org.apache.spark.SparkContext.<init>(SparkContext.scala:340)
>>> >         at
>>> >
>>> org.apache.spark.examples.mllib.TallSkinnySVD$.main(TallSkinnySVD.scala:74)
>>> >         at
>>> org.apache.spark.examples.mllib.TallSkinnySVD.main(TallSkinnySVD.scala)
>>> > Caused by: java.lang.ClassNotFoundException:
>>> > com.google.common.base.Preconditions
>>> >         at java.net.URLClassLoader$1.run(Unknown Source)
>>> >         at java.net.URLClassLoader$1.run(Unknown Source)
>>> >         at java.security.AccessController.doPrivileged(Native Method)
>>> >         at java.net.URLClassLoader.findClass(Unknown Source)
>>> >         at java.lang.ClassLoader.loadClass(Unknown Source)
>>> >         at sun.misc.Launcher$AppClassLoader.loadClass(Unknown Source)
>>> >         at java.lang.ClassLoader.loadClass(Unknown Source)
>>> >         ... 7 more
>>> >
>>> >
>>> -----------------------------------------------------------------------------------------------------------------
>>> >
>>> > jar -tf output:
>>> >
>>> >
>>> > consb2@CONSB2A
>>> > /cygdrive/c/SB/spark-1.2.0-bin-hadoop2.4/spark-1.2.0-bin-hadoop2.4/lib
>>> > $ jar -tf spark-assembly-1.2.0-hadoop2.4.0.jar | grep Preconditions
>>> > org/spark-project/guava/common/base/Preconditions.class
>>> > org/spark-project/guava/common/math/MathPreconditions.class
>>> > com/clearspring/analytics/util/Preconditions.class
>>> > parquet/Preconditions.class
>>> > com/google/inject/internal/util/$Preconditions.class
>>> >
>>> >
>>> ---------------------------------------------------------------------------------------------------------------
>>> >
>>> > Please help me in resolving this.
>>> >
>>> > Thanks,
>>> >   Shailesh
>>> >
>>> >
>>> >
>>> >
>>> > --
>>> > View this message in context:
>>> http://apache-spark-user-list.1001560.n3.nabble.com/Spark-1-2-com-google-common-base-Preconditions-java-lang-NoClassDefFoundErro-tp21271.html
>>> > Sent from the Apache Spark User List mailing list archive at
>>> Nabble.com.
>>> >
>>> > ---------------------------------------------------------------------
>>> > To unsubscribe, e-mail: user-unsubscribe@spark.apache.org
>>> > For additional commands, e-mail: user-help@spark.apache.org
>>> >
>>>
>>
>>
>>
>

Re: Spark 1.2 - com/google/common/base/Preconditions java.lang.NoClassDefFoundErro

Posted by Shailesh Birari <sb...@gmail.com>.
Hi Frank,

Its a normal eclipse project where I added Scala and Spark libraries as
user libraries.
Though, I am not attaching any hadoop libraries, in my application code I
have following line.

  System.setProperty("hadoop.home.dir", "C:\\SB\\HadoopWin")

This Hadoop home dir contains "winutils.exe" only.

Don't think that its an issue.

Please suggest.

Thanks,
  Shailesh


On Wed, Jan 21, 2015 at 2:19 PM, Frank Austin Nothaft <fnothaft@berkeley.edu
> wrote:

> Shailesh,
>
> To add, are you packaging Hadoop in your app? Hadoop will pull in Guava.
> Not sure if you are using Maven (or what) to build, but if you can pull up
> your builds dependency tree, you will likely find com.google.guava being
> brought in by one of your dependencies.
>
> Regards,
>
> Frank Austin Nothaft
> fnothaft@berkeley.edu
> fnothaft@eecs.berkeley.edu
> 202-340-0466
>
> On Jan 20, 2015, at 5:13 PM, Shailesh Birari <sb...@gmail.com> wrote:
>
> Hello,
>
> I double checked the libraries. I am linking only with Spark 1.2.
> Along with Spark 1.2 jars I have Scala 2.10 jars and JRE 7 jars linked and
> nothing else.
>
> Thanks,
>    Shailesh
>
> On Wed, Jan 21, 2015 at 12:58 PM, Sean Owen <so...@cloudera.com> wrote:
>
>> Guava is shaded in Spark 1.2+. It looks like you are mixing versions
>> of Spark then, with some that still refer to unshaded Guava. Make sure
>> you are not packaging Spark with your app and that you don't have
>> other versions lying around.
>>
>> On Tue, Jan 20, 2015 at 11:55 PM, Shailesh Birari <sb...@gmail.com>
>> wrote:
>> > Hello,
>> >
>> > I recently upgraded my setup from Spark 1.1 to Spark 1.2.
>> > My existing applications are working fine on ubuntu cluster.
>> > But, when I try to execute Spark MLlib application from Eclipse (Windows
>> > node) it gives java.lang.NoClassDefFoundError:
>> > com/google/common/base/Preconditions exception.
>> >
>> > Note,
>> >    1. With Spark 1.1 this was working fine.
>> >    2. The Spark 1.2 jar files are linked in Eclipse project.
>> >    3. Checked the jar -tf output and found the above
>> com.google.common.base
>> > is not present.
>> >
>> >
>> -----------------------------------------------------------------------------------------------------------------Exception
>> > log:
>> >
>> > Exception in thread "main" java.lang.NoClassDefFoundError:
>> > com/google/common/base/Preconditions
>> >         at
>> >
>> org.apache.spark.network.client.TransportClientFactory.<init>(TransportClientFactory.java:94)
>> >         at
>> >
>> org.apache.spark.network.TransportContext.createClientFactory(TransportContext.java:77)
>> >         at
>> >
>> org.apache.spark.network.netty.NettyBlockTransferService.init(NettyBlockTransferService.scala:62)
>> >         at
>> org.apache.spark.storage.BlockManager.initialize(BlockManager.scala:194)
>> >         at org.apache.spark.SparkContext.<init>(SparkContext.scala:340)
>> >         at
>> >
>> org.apache.spark.examples.mllib.TallSkinnySVD$.main(TallSkinnySVD.scala:74)
>> >         at
>> org.apache.spark.examples.mllib.TallSkinnySVD.main(TallSkinnySVD.scala)
>> > Caused by: java.lang.ClassNotFoundException:
>> > com.google.common.base.Preconditions
>> >         at java.net.URLClassLoader$1.run(Unknown Source)
>> >         at java.net.URLClassLoader$1.run(Unknown Source)
>> >         at java.security.AccessController.doPrivileged(Native Method)
>> >         at java.net.URLClassLoader.findClass(Unknown Source)
>> >         at java.lang.ClassLoader.loadClass(Unknown Source)
>> >         at sun.misc.Launcher$AppClassLoader.loadClass(Unknown Source)
>> >         at java.lang.ClassLoader.loadClass(Unknown Source)
>> >         ... 7 more
>> >
>> >
>> -----------------------------------------------------------------------------------------------------------------
>> >
>> > jar -tf output:
>> >
>> >
>> > consb2@CONSB2A
>> > /cygdrive/c/SB/spark-1.2.0-bin-hadoop2.4/spark-1.2.0-bin-hadoop2.4/lib
>> > $ jar -tf spark-assembly-1.2.0-hadoop2.4.0.jar | grep Preconditions
>> > org/spark-project/guava/common/base/Preconditions.class
>> > org/spark-project/guava/common/math/MathPreconditions.class
>> > com/clearspring/analytics/util/Preconditions.class
>> > parquet/Preconditions.class
>> > com/google/inject/internal/util/$Preconditions.class
>> >
>> >
>> ---------------------------------------------------------------------------------------------------------------
>> >
>> > Please help me in resolving this.
>> >
>> > Thanks,
>> >   Shailesh
>> >
>> >
>> >
>> >
>> > --
>> > View this message in context:
>> http://apache-spark-user-list.1001560.n3.nabble.com/Spark-1-2-com-google-common-base-Preconditions-java-lang-NoClassDefFoundErro-tp21271.html
>> > Sent from the Apache Spark User List mailing list archive at Nabble.com
>> .
>> >
>> > ---------------------------------------------------------------------
>> > To unsubscribe, e-mail: user-unsubscribe@spark.apache.org
>> > For additional commands, e-mail: user-help@spark.apache.org
>> >
>>
>
>
>

Re: Spark 1.2 - com/google/common/base/Preconditions java.lang.NoClassDefFoundErro

Posted by Frank Austin Nothaft <fn...@berkeley.edu>.
Shailesh,

To add, are you packaging Hadoop in your app? Hadoop will pull in Guava. Not sure if you are using Maven (or what) to build, but if you can pull up your builds dependency tree, you will likely find com.google.guava being brought in by one of your dependencies.

Regards,

Frank Austin Nothaft
fnothaft@berkeley.edu
fnothaft@eecs.berkeley.edu
202-340-0466

On Jan 20, 2015, at 5:13 PM, Shailesh Birari <sb...@gmail.com> wrote:

> Hello,
> 
> I double checked the libraries. I am linking only with Spark 1.2.
> Along with Spark 1.2 jars I have Scala 2.10 jars and JRE 7 jars linked and nothing else.
> 
> Thanks,
>    Shailesh
> 
> On Wed, Jan 21, 2015 at 12:58 PM, Sean Owen <so...@cloudera.com> wrote:
> Guava is shaded in Spark 1.2+. It looks like you are mixing versions
> of Spark then, with some that still refer to unshaded Guava. Make sure
> you are not packaging Spark with your app and that you don't have
> other versions lying around.
> 
> On Tue, Jan 20, 2015 at 11:55 PM, Shailesh Birari <sb...@gmail.com> wrote:
> > Hello,
> >
> > I recently upgraded my setup from Spark 1.1 to Spark 1.2.
> > My existing applications are working fine on ubuntu cluster.
> > But, when I try to execute Spark MLlib application from Eclipse (Windows
> > node) it gives java.lang.NoClassDefFoundError:
> > com/google/common/base/Preconditions exception.
> >
> > Note,
> >    1. With Spark 1.1 this was working fine.
> >    2. The Spark 1.2 jar files are linked in Eclipse project.
> >    3. Checked the jar -tf output and found the above com.google.common.base
> > is not present.
> >
> > -----------------------------------------------------------------------------------------------------------------Exception
> > log:
> >
> > Exception in thread "main" java.lang.NoClassDefFoundError:
> > com/google/common/base/Preconditions
> >         at
> > org.apache.spark.network.client.TransportClientFactory.<init>(TransportClientFactory.java:94)
> >         at
> > org.apache.spark.network.TransportContext.createClientFactory(TransportContext.java:77)
> >         at
> > org.apache.spark.network.netty.NettyBlockTransferService.init(NettyBlockTransferService.scala:62)
> >         at org.apache.spark.storage.BlockManager.initialize(BlockManager.scala:194)
> >         at org.apache.spark.SparkContext.<init>(SparkContext.scala:340)
> >         at
> > org.apache.spark.examples.mllib.TallSkinnySVD$.main(TallSkinnySVD.scala:74)
> >         at org.apache.spark.examples.mllib.TallSkinnySVD.main(TallSkinnySVD.scala)
> > Caused by: java.lang.ClassNotFoundException:
> > com.google.common.base.Preconditions
> >         at java.net.URLClassLoader$1.run(Unknown Source)
> >         at java.net.URLClassLoader$1.run(Unknown Source)
> >         at java.security.AccessController.doPrivileged(Native Method)
> >         at java.net.URLClassLoader.findClass(Unknown Source)
> >         at java.lang.ClassLoader.loadClass(Unknown Source)
> >         at sun.misc.Launcher$AppClassLoader.loadClass(Unknown Source)
> >         at java.lang.ClassLoader.loadClass(Unknown Source)
> >         ... 7 more
> >
> > -----------------------------------------------------------------------------------------------------------------
> >
> > jar -tf output:
> >
> >
> > consb2@CONSB2A
> > /cygdrive/c/SB/spark-1.2.0-bin-hadoop2.4/spark-1.2.0-bin-hadoop2.4/lib
> > $ jar -tf spark-assembly-1.2.0-hadoop2.4.0.jar | grep Preconditions
> > org/spark-project/guava/common/base/Preconditions.class
> > org/spark-project/guava/common/math/MathPreconditions.class
> > com/clearspring/analytics/util/Preconditions.class
> > parquet/Preconditions.class
> > com/google/inject/internal/util/$Preconditions.class
> >
> > ---------------------------------------------------------------------------------------------------------------
> >
> > Please help me in resolving this.
> >
> > Thanks,
> >   Shailesh
> >
> >
> >
> >
> > --
> > View this message in context: http://apache-spark-user-list.1001560.n3.nabble.com/Spark-1-2-com-google-common-base-Preconditions-java-lang-NoClassDefFoundErro-tp21271.html
> > Sent from the Apache Spark User List mailing list archive at Nabble.com.
> >
> > ---------------------------------------------------------------------
> > To unsubscribe, e-mail: user-unsubscribe@spark.apache.org
> > For additional commands, e-mail: user-help@spark.apache.org
> >
> 


Re: Spark 1.2 - com/google/common/base/Preconditions java.lang.NoClassDefFoundErro

Posted by Shailesh Birari <sb...@gmail.com>.
Hello,

I double checked the libraries. I am linking only with Spark 1.2.
Along with Spark 1.2 jars I have Scala 2.10 jars and JRE 7 jars linked and
nothing else.

Thanks,
   Shailesh

On Wed, Jan 21, 2015 at 12:58 PM, Sean Owen <so...@cloudera.com> wrote:

> Guava is shaded in Spark 1.2+. It looks like you are mixing versions
> of Spark then, with some that still refer to unshaded Guava. Make sure
> you are not packaging Spark with your app and that you don't have
> other versions lying around.
>
> On Tue, Jan 20, 2015 at 11:55 PM, Shailesh Birari <sb...@gmail.com>
> wrote:
> > Hello,
> >
> > I recently upgraded my setup from Spark 1.1 to Spark 1.2.
> > My existing applications are working fine on ubuntu cluster.
> > But, when I try to execute Spark MLlib application from Eclipse (Windows
> > node) it gives java.lang.NoClassDefFoundError:
> > com/google/common/base/Preconditions exception.
> >
> > Note,
> >    1. With Spark 1.1 this was working fine.
> >    2. The Spark 1.2 jar files are linked in Eclipse project.
> >    3. Checked the jar -tf output and found the above
> com.google.common.base
> > is not present.
> >
> >
> -----------------------------------------------------------------------------------------------------------------Exception
> > log:
> >
> > Exception in thread "main" java.lang.NoClassDefFoundError:
> > com/google/common/base/Preconditions
> >         at
> >
> org.apache.spark.network.client.TransportClientFactory.<init>(TransportClientFactory.java:94)
> >         at
> >
> org.apache.spark.network.TransportContext.createClientFactory(TransportContext.java:77)
> >         at
> >
> org.apache.spark.network.netty.NettyBlockTransferService.init(NettyBlockTransferService.scala:62)
> >         at
> org.apache.spark.storage.BlockManager.initialize(BlockManager.scala:194)
> >         at org.apache.spark.SparkContext.<init>(SparkContext.scala:340)
> >         at
> >
> org.apache.spark.examples.mllib.TallSkinnySVD$.main(TallSkinnySVD.scala:74)
> >         at
> org.apache.spark.examples.mllib.TallSkinnySVD.main(TallSkinnySVD.scala)
> > Caused by: java.lang.ClassNotFoundException:
> > com.google.common.base.Preconditions
> >         at java.net.URLClassLoader$1.run(Unknown Source)
> >         at java.net.URLClassLoader$1.run(Unknown Source)
> >         at java.security.AccessController.doPrivileged(Native Method)
> >         at java.net.URLClassLoader.findClass(Unknown Source)
> >         at java.lang.ClassLoader.loadClass(Unknown Source)
> >         at sun.misc.Launcher$AppClassLoader.loadClass(Unknown Source)
> >         at java.lang.ClassLoader.loadClass(Unknown Source)
> >         ... 7 more
> >
> >
> -----------------------------------------------------------------------------------------------------------------
> >
> > jar -tf output:
> >
> >
> > consb2@CONSB2A
> > /cygdrive/c/SB/spark-1.2.0-bin-hadoop2.4/spark-1.2.0-bin-hadoop2.4/lib
> > $ jar -tf spark-assembly-1.2.0-hadoop2.4.0.jar | grep Preconditions
> > org/spark-project/guava/common/base/Preconditions.class
> > org/spark-project/guava/common/math/MathPreconditions.class
> > com/clearspring/analytics/util/Preconditions.class
> > parquet/Preconditions.class
> > com/google/inject/internal/util/$Preconditions.class
> >
> >
> ---------------------------------------------------------------------------------------------------------------
> >
> > Please help me in resolving this.
> >
> > Thanks,
> >   Shailesh
> >
> >
> >
> >
> > --
> > View this message in context:
> http://apache-spark-user-list.1001560.n3.nabble.com/Spark-1-2-com-google-common-base-Preconditions-java-lang-NoClassDefFoundErro-tp21271.html
> > Sent from the Apache Spark User List mailing list archive at Nabble.com.
> >
> > ---------------------------------------------------------------------
> > To unsubscribe, e-mail: user-unsubscribe@spark.apache.org
> > For additional commands, e-mail: user-help@spark.apache.org
> >
>

Re: Spark 1.2 - com/google/common/base/Preconditions java.lang.NoClassDefFoundErro

Posted by Ted Yu <yu...@gmail.com>.
Please also see this thread:
http://search-hadoop.com/m/JW1q5De7pU1

On Tue, Jan 20, 2015 at 3:58 PM, Sean Owen <so...@cloudera.com> wrote:

> Guava is shaded in Spark 1.2+. It looks like you are mixing versions
> of Spark then, with some that still refer to unshaded Guava. Make sure
> you are not packaging Spark with your app and that you don't have
> other versions lying around.
>
> On Tue, Jan 20, 2015 at 11:55 PM, Shailesh Birari <sb...@gmail.com>
> wrote:
> > Hello,
> >
> > I recently upgraded my setup from Spark 1.1 to Spark 1.2.
> > My existing applications are working fine on ubuntu cluster.
> > But, when I try to execute Spark MLlib application from Eclipse (Windows
> > node) it gives java.lang.NoClassDefFoundError:
> > com/google/common/base/Preconditions exception.
> >
> > Note,
> >    1. With Spark 1.1 this was working fine.
> >    2. The Spark 1.2 jar files are linked in Eclipse project.
> >    3. Checked the jar -tf output and found the above
> com.google.common.base
> > is not present.
> >
> >
> -----------------------------------------------------------------------------------------------------------------Exception
> > log:
> >
> > Exception in thread "main" java.lang.NoClassDefFoundError:
> > com/google/common/base/Preconditions
> >         at
> >
> org.apache.spark.network.client.TransportClientFactory.<init>(TransportClientFactory.java:94)
> >         at
> >
> org.apache.spark.network.TransportContext.createClientFactory(TransportContext.java:77)
> >         at
> >
> org.apache.spark.network.netty.NettyBlockTransferService.init(NettyBlockTransferService.scala:62)
> >         at
> org.apache.spark.storage.BlockManager.initialize(BlockManager.scala:194)
> >         at org.apache.spark.SparkContext.<init>(SparkContext.scala:340)
> >         at
> >
> org.apache.spark.examples.mllib.TallSkinnySVD$.main(TallSkinnySVD.scala:74)
> >         at
> org.apache.spark.examples.mllib.TallSkinnySVD.main(TallSkinnySVD.scala)
> > Caused by: java.lang.ClassNotFoundException:
> > com.google.common.base.Preconditions
> >         at java.net.URLClassLoader$1.run(Unknown Source)
> >         at java.net.URLClassLoader$1.run(Unknown Source)
> >         at java.security.AccessController.doPrivileged(Native Method)
> >         at java.net.URLClassLoader.findClass(Unknown Source)
> >         at java.lang.ClassLoader.loadClass(Unknown Source)
> >         at sun.misc.Launcher$AppClassLoader.loadClass(Unknown Source)
> >         at java.lang.ClassLoader.loadClass(Unknown Source)
> >         ... 7 more
> >
> >
> -----------------------------------------------------------------------------------------------------------------
> >
> > jar -tf output:
> >
> >
> > consb2@CONSB2A
> > /cygdrive/c/SB/spark-1.2.0-bin-hadoop2.4/spark-1.2.0-bin-hadoop2.4/lib
> > $ jar -tf spark-assembly-1.2.0-hadoop2.4.0.jar | grep Preconditions
> > org/spark-project/guava/common/base/Preconditions.class
> > org/spark-project/guava/common/math/MathPreconditions.class
> > com/clearspring/analytics/util/Preconditions.class
> > parquet/Preconditions.class
> > com/google/inject/internal/util/$Preconditions.class
> >
> >
> ---------------------------------------------------------------------------------------------------------------
> >
> > Please help me in resolving this.
> >
> > Thanks,
> >   Shailesh
> >
> >
> >
> >
> > --
> > View this message in context:
> http://apache-spark-user-list.1001560.n3.nabble.com/Spark-1-2-com-google-common-base-Preconditions-java-lang-NoClassDefFoundErro-tp21271.html
> > Sent from the Apache Spark User List mailing list archive at Nabble.com.
> >
> > ---------------------------------------------------------------------
> > To unsubscribe, e-mail: user-unsubscribe@spark.apache.org
> > For additional commands, e-mail: user-help@spark.apache.org
> >
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: user-unsubscribe@spark.apache.org
> For additional commands, e-mail: user-help@spark.apache.org
>
>

Re: Spark 1.2 - com/google/common/base/Preconditions java.lang.NoClassDefFoundErro

Posted by Sean Owen <so...@cloudera.com>.
Guava is shaded in Spark 1.2+. It looks like you are mixing versions
of Spark then, with some that still refer to unshaded Guava. Make sure
you are not packaging Spark with your app and that you don't have
other versions lying around.

On Tue, Jan 20, 2015 at 11:55 PM, Shailesh Birari <sb...@gmail.com> wrote:
> Hello,
>
> I recently upgraded my setup from Spark 1.1 to Spark 1.2.
> My existing applications are working fine on ubuntu cluster.
> But, when I try to execute Spark MLlib application from Eclipse (Windows
> node) it gives java.lang.NoClassDefFoundError:
> com/google/common/base/Preconditions exception.
>
> Note,
>    1. With Spark 1.1 this was working fine.
>    2. The Spark 1.2 jar files are linked in Eclipse project.
>    3. Checked the jar -tf output and found the above com.google.common.base
> is not present.
>
> -----------------------------------------------------------------------------------------------------------------Exception
> log:
>
> Exception in thread "main" java.lang.NoClassDefFoundError:
> com/google/common/base/Preconditions
>         at
> org.apache.spark.network.client.TransportClientFactory.<init>(TransportClientFactory.java:94)
>         at
> org.apache.spark.network.TransportContext.createClientFactory(TransportContext.java:77)
>         at
> org.apache.spark.network.netty.NettyBlockTransferService.init(NettyBlockTransferService.scala:62)
>         at org.apache.spark.storage.BlockManager.initialize(BlockManager.scala:194)
>         at org.apache.spark.SparkContext.<init>(SparkContext.scala:340)
>         at
> org.apache.spark.examples.mllib.TallSkinnySVD$.main(TallSkinnySVD.scala:74)
>         at org.apache.spark.examples.mllib.TallSkinnySVD.main(TallSkinnySVD.scala)
> Caused by: java.lang.ClassNotFoundException:
> com.google.common.base.Preconditions
>         at java.net.URLClassLoader$1.run(Unknown Source)
>         at java.net.URLClassLoader$1.run(Unknown Source)
>         at java.security.AccessController.doPrivileged(Native Method)
>         at java.net.URLClassLoader.findClass(Unknown Source)
>         at java.lang.ClassLoader.loadClass(Unknown Source)
>         at sun.misc.Launcher$AppClassLoader.loadClass(Unknown Source)
>         at java.lang.ClassLoader.loadClass(Unknown Source)
>         ... 7 more
>
> -----------------------------------------------------------------------------------------------------------------
>
> jar -tf output:
>
>
> consb2@CONSB2A
> /cygdrive/c/SB/spark-1.2.0-bin-hadoop2.4/spark-1.2.0-bin-hadoop2.4/lib
> $ jar -tf spark-assembly-1.2.0-hadoop2.4.0.jar | grep Preconditions
> org/spark-project/guava/common/base/Preconditions.class
> org/spark-project/guava/common/math/MathPreconditions.class
> com/clearspring/analytics/util/Preconditions.class
> parquet/Preconditions.class
> com/google/inject/internal/util/$Preconditions.class
>
> ---------------------------------------------------------------------------------------------------------------
>
> Please help me in resolving this.
>
> Thanks,
>   Shailesh
>
>
>
>
> --
> View this message in context: http://apache-spark-user-list.1001560.n3.nabble.com/Spark-1-2-com-google-common-base-Preconditions-java-lang-NoClassDefFoundErro-tp21271.html
> Sent from the Apache Spark User List mailing list archive at Nabble.com.
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: user-unsubscribe@spark.apache.org
> For additional commands, e-mail: user-help@spark.apache.org
>

---------------------------------------------------------------------
To unsubscribe, e-mail: user-unsubscribe@spark.apache.org
For additional commands, e-mail: user-help@spark.apache.org