You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@spark.apache.org by "Jorge Machado (Jira)" <ji...@apache.org> on 2020/03/28 16:26:00 UTC
[jira] [Comment Edited] (SPARK-30272) Remove usage of Guava that
breaks in Guava 27
[ https://issues.apache.org/jira/browse/SPARK-30272?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17069933#comment-17069933 ]
Jorge Machado edited comment on SPARK-30272 at 3/28/20, 4:25 PM:
-----------------------------------------------------------------
Hey Sean,
This seems still to make problems for example:
{code:java}
> $ ./bin/run-example SparkPi 100 [±master ●]> $ ./bin/run-example SparkPi 100 [±master ●]20/03/28 17:21:13 WARN Utils: Your hostname, Jorges-MacBook-Pro.local resolves to a loopback address: 127.0.0.1; using 192.168.1.2 instead (on interface en0)20/03/28 17:21:13 WARN Utils: Set SPARK_LOCAL_IP if you need to bind to another address20/03/28 17:21:14 WARN NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicableUsing Spark's default log4j profile: org/apache/spark/log4j-defaults.properties20/03/28 17:21:14 INFO SparkContext: Running Spark version 3.1.0-SNAPSHOT20/03/28 17:21:14 INFO ResourceUtils: ==============================================================20/03/28 17:21:14 INFO ResourceUtils: No custom resources configured for spark.driver.20/03/28 17:21:14 INFO ResourceUtils: ==============================================================20/03/28 17:21:14 INFO SparkContext: Submitted application: Spark Pi20/03/28 17:21:14 INFO ResourceProfile: Default ResourceProfile created, executor resources: Map(cores -> name: cores, amount: 1, script: , vendor: , memory -> name: memory, amount: 1024, script: , vendor: ), task resources: Map(cpus -> name: cpus, amount: 1.0)20/03/28 17:21:14 INFO ResourceProfile: Limiting resource is cpu20/03/28 17:21:14 INFO ResourceProfileManager: Added ResourceProfile id: 020/03/28 17:21:14 INFO SecurityManager: Changing view acls to: jorge20/03/28 17:21:14 INFO SecurityManager: Changing modify acls to: jorge20/03/28 17:21:14 INFO SecurityManager: Changing view acls groups to:20/03/28 17:21:14 INFO SecurityManager: Changing modify acls groups to:20/03/28 17:21:14 INFO SecurityManager: SecurityManager: authentication disabled; ui acls disabled; users with view permissions: Set(jorge); groups with view permissions: Set(); users with modify permissions: Set(jorge); groups with modify permissions: Set()20/03/28 17:21:14 INFO Utils: Successfully started service 'sparkDriver' on port 58192.20/03/28 17:21:14 INFO SparkEnv: Registering MapOutputTracker20/03/28 17:21:14 INFO SparkEnv: Registering BlockManagerMaster20/03/28 17:21:14 INFO BlockManagerMasterEndpoint: Using org.apache.spark.storage.DefaultTopologyMapper for getting topology information20/03/28 17:21:14 INFO BlockManagerMasterEndpoint: BlockManagerMasterEndpoint up20/03/28 17:21:14 INFO SparkEnv: Registering BlockManagerMasterHeartbeat20/03/28 17:21:14 INFO DiskBlockManager: Created local directory at /private/var/folders/0h/5b7dw9p11l58hyk0_s0d3cnh0000gn/T/blockmgr-d9e88815-075e-4c9b-9cc8-21c72e97c86920/03/28 17:21:14 INFO MemoryStore: MemoryStore started with capacity 366.3 MiB20/03/28 17:21:14 INFO SparkEnv: Registering OutputCommitCoordinator20/03/28 17:21:15 INFO Utils: Successfully started service 'SparkUI' on port 4040.20/03/28 17:21:15 INFO SparkUI: Bound SparkUI to 0.0.0.0, and started at http://192.168.1.2:404020/03/28 17:21:15 INFO SparkContext: Added JAR file:///Users/jorge/Downloads/spark/dist/examples/jars/spark-examples_2.12-3.1.0-SNAPSHOT.jar at spark://192.168.1.2:58192/jars/spark-examples_2.12-3.1.0-SNAPSHOT.jar with timestamp 158541247516620/03/28 17:21:15 INFO SparkContext: Added JAR file:///Users/jorge/Downloads/spark/dist/examples/jars/scopt_2.12-3.7.1.jar at spark://192.168.1.2:58192/jars/scopt_2.12-3.7.1.jar with timestamp 158541247516620/03/28 17:21:15 INFO Executor: Starting executor ID driver on host 192.168.1.220/03/28 17:21:15 INFO Utils: Successfully started service 'org.apache.spark.network.netty.NettyBlockTransferService' on port 58193.20/03/28 17:21:15 INFO NettyBlockTransferService: Server created on 192.168.1.2:5819320/03/28 17:21:15 INFO BlockManager: Using org.apache.spark.storage.RandomBlockReplicationPolicy for block replication policyException in thread "main" java.lang.NoClassDefFoundError: org/sparkproject/guava/util/concurrent/internal/InternalFutureFailureAccess at java.lang.ClassLoader.defineClass1(Native Method) at java.lang.ClassLoader.defineClass(ClassLoader.java:763) at java.security.SecureClassLoader.defineClass(SecureClassLoader.java:142) at java.net.URLClassLoader.defineClass(URLClassLoader.java:468) at java.net.URLClassLoader.access$100(URLClassLoader.java:74) at java.net.URLClassLoader$1.run(URLClassLoader.java:369) at java.net.URLClassLoader$1.run(URLClassLoader.java:363) at java.security.AccessController.doPrivileged(Native Method) at java.net.URLClassLoader.findClass(URLClassLoader.java:362) at java.lang.ClassLoader.loadClass(ClassLoader.java:424) at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:349) at java.lang.ClassLoader.loadClass(ClassLoader.java:357) at java.lang.ClassLoader.defineClass1(Native Method) at java.lang.ClassLoader.defineClass(ClassLoader.java:763) at java.security.SecureClassLoader.defineClass(SecureClassLoader.java:142) at java.net.URLClassLoader.defineClass(URLClassLoader.java:468) at java.net.URLClassLoader.access$100(URLClassLoader.java:74) at java.net.URLClassLoader$1.run(URLClassLoader.java:369) at java.net.URLClassLoader$1.run(URLClassLoader.java:363) at java.security.AccessController.doPrivileged(Native Method) at java.net.URLClassLoader.findClass(URLClassLoader.java:362) at java.lang.ClassLoader.loadClass(ClassLoader.java:424) at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:349) at java.lang.ClassLoader.loadClass(ClassLoader.java:357) at java.lang.ClassLoader.defineClass1(Native Method) at java.lang.ClassLoader.defineClass(ClassLoader.java:763) at java.security.SecureClassLoader.defineClass(SecureClassLoader.java:142) at java.net.URLClassLoader.defineClass(URLClassLoader.java:468) at java.net.URLClassLoader.access$100(URLClassLoader.java:74) at java.net.URLClassLoader$1.run(URLClassLoader.java:369) at java.net.URLClassLoader$1.run(URLClassLoader.java:363) at java.security.AccessController.doPrivileged(Native Method) at java.net.URLClassLoader.findClass(URLClassLoader.java:362) at java.lang.ClassLoader.loadClass(ClassLoader.java:424) at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:349) at java.lang.ClassLoader.loadClass(ClassLoader.java:357) at org.sparkproject.guava.cache.LocalCache$LoadingValueReference.<init>(LocalCache.java:3472) at org.sparkproject.guava.cache.LocalCache$LoadingValueReference.<init>(LocalCache.java:3476) at org.sparkproject.guava.cache.LocalCache$Segment.lockedGetOrLoad(LocalCache.java:2134) at org.sparkproject.guava.cache.LocalCache$Segment.get(LocalCache.java:2045) at org.sparkproject.guava.cache.LocalCache.get(LocalCache.java:3951) at org.sparkproject.guava.cache.LocalCache.getOrLoad(LocalCache.java:3974) at org.sparkproject.guava.cache.LocalCache$LocalLoadingCache.get(LocalCache.java:4958) at org.apache.spark.storage.BlockManagerId$.getCachedBlockManagerId(BlockManagerId.scala:146) at org.apache.spark.storage.BlockManagerId$.apply(BlockManagerId.scala:127) at org.apache.spark.storage.BlockManager.initialize(BlockManager.scala:457) at org.apache.spark.SparkContext.<init>(SparkContext.scala:564) at org.apache.spark.SparkContext$.getOrCreate(SparkContext.scala:2570) at org.apache.spark.sql.SparkSession$Builder.$anonfun$getOrCreate$5(SparkSession.scala:936) at scala.Option.getOrElse(Option.scala:189) at org.apache.spark.sql.SparkSession$Builder.getOrCreate(SparkSession.scala:927) at org.apache.spark.examples.SparkPi$.main(SparkPi.scala:30) at org.apache.spark.examples.SparkPi.main(SparkPi.scala) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.apache.spark.deploy.JavaMainApplication.start(SparkApplication.scala:52) at org.apache.spark.deploy.SparkSubmit.org$apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:932) at org.apache.spark.deploy.SparkSubmit.doRunMain$1(SparkSubmit.scala:180) at org.apache.spark.deploy.SparkSubmit.submit(SparkSubmit.scala:203) at org.apache.spark.deploy.SparkSubmit.doSubmit(SparkSubmit.scala:90) at org.apache.spark.deploy.SparkSubmit$$anon$2.doSubmit(SparkSubmit.scala:1011) at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:1020) at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)Caused by: java.lang.ClassNotFoundException: org.sparkproject.guava.util.concurrent.internal.InternalFutureFailureAccess at java.net.URLClassLoader.findClass(URLClassLoader.java:382) at java.lang.ClassLoader.loadClass(ClassLoader.java:424) at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:349) at java.lang.ClassLoader.loadClass(ClassLoader.java:357) ... 65 more20/03/28 17:21:15 INFO DiskBlockManager: Shutdown hook called
{code}
I still see a lot of references to guava 14 on master is this normal ? Sorry for the question...
was (Author: jomach):
Hey Sean,
This seems still to make problems for example:
{code:java}
java.lang.NoClassDefFoundError: com/google/common/util/concurrent/internal/InternalFutureFailureAccess java.lang.NoClassDefFoundError: com/google/common/util/concurrent/internal/InternalFutureFailureAccess at java.lang.ClassLoader.defineClass1(Native Method) at java.lang.ClassLoader.defineClass(ClassLoader.java:757) at java.security.SecureClassLoader.defineClass(SecureClassLoader.java:142) at java.net.URLClassLoader.defineClass(URLClassLoader.java:468) at java.net.URLClassLoader.access$100(URLClassLoader.java:74) at java.net.URLClassLoader$1.run(URLClassLoader.java:369) at java.net.URLClassLoader$1.run(URLClassLoader.java:363) at java.security.AccessController.doPrivileged(Native Method) at java.net.URLClassLoader.findClass(URLClassLoader.java:362) at java.lang.ClassLoader.loadClass(ClassLoader.java:419) at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:352) at java.lang.ClassLoader.loadClass(ClassLoader.java:352) at java.lang.ClassLoader.defineClass1(Native Method) at java.lang.ClassLoader.defineClass(ClassLoader.java:757) at java.security.SecureClassLoader.defineClass(SecureClassLoader.java:142) at java.net.URLClassLoader.defineClass(URLClassLoader.java:468) at java.net.URLClassLoader.access$100(URLClassLoader.java:74) at java.net.URLClassLoader$1.run(URLClassLoader.java:369) at java.net.URLClassLoader$1.run(URLClassLoader.java:363) at java.security.AccessController.doPrivileged(Native Method) at java.net.URLClassLoader.findClass(URLClassLoader.java:362) at java.lang.ClassLoader.loadClass(ClassLoader.java:419) at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:352) at java.lang.ClassLoader.loadClass(ClassLoader.java:352) at java.lang.ClassLoader.defineClass1(Native Method) at java.lang.ClassLoader.defineClass(ClassLoader.java:757) at java.security.SecureClassLoader.defineClass(SecureClassLoader.java:142) at java.net.URLClassLoader.defineClass(URLClassLoader.java:468) at java.net.URLClassLoader.access$100(URLClassLoader.java:74) at java.net.URLClassLoader$1.run(URLClassLoader.java:369) at java.net.URLClassLoader$1.run(URLClassLoader.java:363) at java.security.AccessController.doPrivileged(Native Method) at java.net.URLClassLoader.findClass(URLClassLoader.java:362) at java.lang.ClassLoader.loadClass(ClassLoader.java:419) at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:352) at java.lang.ClassLoader.loadClass(ClassLoader.java:352) at com.google.common.cache.LocalCache$LoadingValueReference.<init>(LocalCache.java:3472) at com.google.common.cache.LocalCache$LoadingValueReference.<init>(LocalCache.java:3476) at com.google.common.cache.LocalCache$Segment.lockedGetOrLoad(LocalCache.java:2134) at com.google.common.cache.LocalCache$Segment.get(LocalCache.java:2045) at com.google.common.cache.LocalCache.get(LocalCache.java:3951) at com.google.common.cache.LocalCache.getOrLoad(LocalCache.java:3974) at com.google.common.cache.LocalCache$LocalLoadingCache.get(LocalCache.java:4958) at org.apache.hadoop.security.Groups.getGroups(Groups.java:228) at org.apache.hadoop.security.UserGroupInformation.getGroups(UserGroupInformation.java:1588) at org.apache.hadoop.security.UserGroupInformation.getPrimaryGroupName(UserGroupInformation.java:1453) at org.apache.hadoop.fs.azurebfs.AzureBlobFileSystemStore.<init>(AzureBlobFileSystemStore.java:147) at org.apache.hadoop.fs.azurebfs.AzureBlobFileSystem.initialize(AzureBlobFileSystem.java:104) at org.apache.hadoop.fs.FileSystem.createFileSystem(FileSystem.java:3303) at org.apache.hadoop.fs.FileSystem.access$200(FileSystem.java:124) at org.apache.hadoop.fs.FileSystem$Cache.getInternal(FileSystem.java:3352) at org.apache.hadoop.fs.FileSystem$Cache.get(FileSystem.java:3320) at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:479) at org.apache.hadoop.fs.Path.getFileSystem(Path.java:365) at org.apache.hadoop.mapreduce.lib.input.FileInputFormat.setInputPaths(FileInputFormat.java:522) at org.apache.hadoop.mapreduce.lib.input.FileInputFormat.setInputPaths(FileInputFormat.java:491) at org.apache.spark.SparkContext.$anonfun$newAPIHadoopFile$2(SparkContext.scala:1219) at org.apache.spark.rdd.RDDOperationScope$.withScope(RDDOperationScope.scala:151) at org.apache.spark.rdd.RDDOperationScope$.withScope(RDDOperationScope.scala:112) at org.apache.spark.SparkContext.withScope(SparkContext.scala:757) at org.apache.spark.SparkContext.newAPIHadoopFile(SparkContext.scala:1207) at org.apache.spark.api.java.JavaSparkContext.newAPIHadoopFile(JavaSparkContext.scala:484)
{code}
I still see a lot of references to guava 14 on master is this normal ? Sorry for the question...
> Remove usage of Guava that breaks in Guava 27
> ---------------------------------------------
>
> Key: SPARK-30272
> URL: https://issues.apache.org/jira/browse/SPARK-30272
> Project: Spark
> Issue Type: Improvement
> Components: Spark Core, SQL
> Affects Versions: 3.0.0
> Reporter: Sean R. Owen
> Assignee: Sean R. Owen
> Priority: Major
> Fix For: 3.0.0
>
>
> Background:
> https://issues.apache.org/jira/browse/SPARK-29250
> https://github.com/apache/spark/pull/25932
> Hadoop 3.2.1 will update Guava from 11 to 27. There are a number of methods that changed between those releases, typically just a rename, but, means one set of code can't work with both, while we want to work with Hadoop 2.x and 3.x. Among them:
> - Objects.toStringHelper was moved to MoreObjects; we can just use the Commons Lang3 equivalent
> - Objects.hashCode etc were renamed; use java.util.Objects equivalents
> - MoreExecutors.sameThreadExecutor() became directExecutor(); for same-thread execution we can use a dummy implementation of ExecutorService / Executor
> - TypeToken.isAssignableFrom become isSupertypeOf; work around with reflection
> There is probably more to the Guava issue than just this change, but it will make Spark itself work with more versions and reduce our exposure to Guava along the way anyway.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org