You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@spark.apache.org by "Sean Owen (JIRA)" <ji...@apache.org> on 2015/03/17 14:59:38 UTC
[jira] [Commented] (SPARK-6388) Spark 1.3 + Hadoop 2.6 Can't work
on Java 8_40
[ https://issues.apache.org/jira/browse/SPARK-6388?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14365142#comment-14365142 ]
Sean Owen commented on SPARK-6388:
----------------------------------
Hm, it's not clear what you mean here. You are just showing that you are building Spark. That can't fail with an Akka error.
Can you report the output of the build and of tests?
What is this log output you are attaching -- what are you running?
I am running Java 8_40 on Mac OS X and have no problems.
> Spark 1.3 + Hadoop 2.6 Can't work on Java 8_40
> ----------------------------------------------
>
> Key: SPARK-6388
> URL: https://issues.apache.org/jira/browse/SPARK-6388
> Project: Spark
> Issue Type: Bug
> Components: Block Manager, Spark Submit, YARN
> Affects Versions: 1.3.0
> Environment: 1. Linux version 3.16.0-30-generic (buildd@komainu) (gcc version 4.9.1 (Ubuntu 4.9.1-16ubuntu6) ) #40-Ubuntu SMP Mon Jan 12 22:06:37 UTC 2015
> 2. Oracle Java 8 update 40 for Linux X64
> 3. Scala 2.10.5
> 4. Hadoop 2.6 (pre-build version)
> Reporter: John
> Original Estimate: 24h
> Remaining Estimate: 24h
>
> I build Apache Spark 1.3 munally.
> ---------------------------
> JAVA_HOME=PATH_TO_JAVA8
> mvn clean package -Pyarn -Phadoop-2.4 -Dhadoop.version=2.6.0 -DskipTests
> ---------------------------
> Something goes wrong, akka always tell me
> ---------------------------
> 15/03/17 21:28:10 WARN remote.ReliableDeliverySupervisor: Association with remote system [akka.tcp://sparkYarnAM@Server2:42161] has failed, address is now gated for [5000] ms. Reason is: [Disassociated].
> ---------------------------
> I build another version of Spark 1.3 + Hadoop 2.6 under Java 7.
> Everything goes well.
> Logs
> ---------------------------
> 15/03/17 21:27:06 INFO spark.SparkContext: Running Spark version 1.3.0
> 15/03/17 21:27:07 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
> 15/03/17 21:27:08 INFO spark.SecurityManager: Changing view Servers to: hduser
> 15/03/17 21:27:08 INFO spark.SecurityManager: Changing modify Servers to: hduser
> 15/03/17 21:27:08 INFO spark.SecurityManager: SecurityManager: authentication disabled; ui Servers disabled; users with view permissions: Set(hduser); users with modify permissions: Set(hduser)
> 15/03/17 21:27:08 INFO slf4j.Slf4jLogger: Slf4jLogger started
> 15/03/17 21:27:08 INFO Remoting: Starting remoting
> 15/03/17 21:27:09 INFO Remoting: Remoting started; listening on addresses :[akka.tcp://sparkDriver@Server3:37951]
> 15/03/17 21:27:09 INFO util.Utils: Successfully started service 'sparkDriver' on port 37951.
> 15/03/17 21:27:09 INFO spark.SparkEnv: Registering MapOutputTracker
> 15/03/17 21:27:09 INFO spark.SparkEnv: Registering BlockManagerMaster
> 15/03/17 21:27:09 INFO storage.DiskBlockManager: Created local directory at /tmp/spark-0db692bb-cd02-40c8-a8f0-3813c6da18e2/blockmgr-a1d0ad23-ab76-4177-80a0-a6f982a64d80
> 15/03/17 21:27:09 INFO storage.MemoryStore: MemoryStore started with capacity 265.1 MB
> 15/03/17 21:27:09 INFO spark.HttpFileServer: HTTP File server directory is /tmp/spark-502ef3f8-b8cd-45cf-b1df-97df297cdb35/httpd-6303e24d-4b2b-4614-bb1d-74e8d331189b
> 15/03/17 21:27:09 INFO spark.HttpServer: Starting HTTP Server
> 15/03/17 21:27:09 INFO server.Server: jetty-8.y.z-SNAPSHOT
> 15/03/17 21:27:10 INFO server.AbstractConnector: Started SocketConnector@0.0.0.0:48000
> 15/03/17 21:27:10 INFO util.Utils: Successfully started service 'HTTP file server' on port 48000.
> 15/03/17 21:27:10 INFO spark.SparkEnv: Registering OutputCommitCoordinator
> 15/03/17 21:27:10 INFO server.Server: jetty-8.y.z-SNAPSHOT
> 15/03/17 21:27:10 INFO server.AbstractConnector: Started SelectChannelConnector@0.0.0.0:4040
> 15/03/17 21:27:10 INFO util.Utils: Successfully started service 'SparkUI' on port 4040.
> 15/03/17 21:27:10 INFO ui.SparkUI: Started SparkUI at http://Server3:4040
> 15/03/17 21:27:10 INFO spark.SparkContext: Added JAR file:/home/hduser/spark-java2.jar at http://192.168.11.42:48000/jars/spark-java2.jar with timestamp 1426598830307
> 15/03/17 21:27:10 INFO client.RMProxy: Connecting to ResourceManager at Server3/192.168.11.42:8050
> 15/03/17 21:27:11 INFO yarn.Client: Requesting a new application from cluster with 3 NodeManagers
> 15/03/17 21:27:11 INFO yarn.Client: Verifying our application has not requested more than the maximum memory capability of the cluster (8192 MB per container)
> 15/03/17 21:27:11 INFO yarn.Client: Will allocate AM container, with 896 MB memory including 384 MB overhead
> 15/03/17 21:27:11 INFO yarn.Client: Setting up container launch context for our AM
> 15/03/17 21:27:11 INFO yarn.Client: Preparing resources for our AM container
> 15/03/17 21:27:12 INFO yarn.Client: Uploading resource file:/home/hduser/spark-1.3.0/assembly/target/scala-2.10/spark-assembly-1.3.0-hadoop2.6.0.jar -> hdfs://Server3:9000/user/hduser/.sparkStaging/application_1426595477608_0002/spark-assembly-1.3.0-hadoop2.6.0.jar
> 15/03/17 21:27:21 INFO yarn.Client: Setting up the launch environment for our AM container
> 15/03/17 21:27:21 INFO spark.SecurityManager: Changing view Servers to: hduser
> 15/03/17 21:27:21 INFO spark.SecurityManager: Changing modify Servers to: hduser
> 15/03/17 21:27:21 INFO spark.SecurityManager: SecurityManager: authentication disabled; ui Servers disabled; users with view permissions: Set(hduser); users with modify permissions: Set(hduser)
> 15/03/17 21:27:21 INFO yarn.Client: Submitting application 2 to ResourceManager
> 15/03/17 21:27:22 INFO impl.YarnClientImpl: Submitted application application_1426595477608_0002
> 15/03/17 21:27:23 INFO yarn.Client: Application report for application_1426595477608_0002 (state: ACCEPTED)
> 15/03/17 21:27:23 INFO yarn.Client:
> client token: N/A
> diagnostics: N/A
> ApplicationMaster host: N/A
> ApplicationMaster RPC port: -1
> queue: default
> start time: 1426598841696
> final status: UNDEFINED
> tracking URL: http://Server3:8088/proxy/application_1426595477608_0002/
> user: hduser
> 15/03/17 21:27:24 INFO yarn.Client: Application report for application_1426595477608_0002 (state: ACCEPTED)
> 15/03/17 21:27:25 INFO yarn.Client: Application report for application_1426595477608_0002 (state: ACCEPTED)
> 15/03/17 21:27:26 INFO yarn.Client: Application report for application_1426595477608_0002 (state: ACCEPTED)
> 15/03/17 21:27:27 INFO yarn.Client: Application report for application_1426595477608_0002 (state: ACCEPTED)
> 15/03/17 21:27:28 INFO yarn.Client: Application report for application_1426595477608_0002 (state: ACCEPTED)
> 15/03/17 21:27:29 INFO yarn.Client: Application report for application_1426595477608_0002 (state: ACCEPTED)
> 15/03/17 21:27:30 INFO yarn.Client: Application report for application_1426595477608_0002 (state: ACCEPTED)
> 15/03/17 21:27:31 INFO yarn.Client: Application report for application_1426595477608_0002 (state: ACCEPTED)
> 15/03/17 21:27:32 INFO yarn.Client: Application report for application_1426595477608_0002 (state: ACCEPTED)
> 15/03/17 21:27:33 INFO yarn.Client: Application report for application_1426595477608_0002 (state: ACCEPTED)
> 15/03/17 21:27:34 INFO yarn.Client: Application report for application_1426595477608_0002 (state: ACCEPTED)
> 15/03/17 21:27:35 INFO yarn.Client: Application report for application_1426595477608_0002 (state: ACCEPTED)
> 15/03/17 21:27:36 INFO yarn.Client: Application report for application_1426595477608_0002 (state: ACCEPTED)
> 15/03/17 21:27:37 INFO yarn.Client: Application report for application_1426595477608_0002 (state: ACCEPTED)
> 15/03/17 21:27:38 INFO yarn.Client: Application report for application_1426595477608_0002 (state: ACCEPTED)
> 15/03/17 21:27:39 INFO yarn.Client: Application report for application_1426595477608_0002 (state: ACCEPTED)
> 15/03/17 21:27:40 INFO yarn.Client: Application report for application_1426595477608_0002 (state: ACCEPTED)
> 15/03/17 21:27:41 INFO yarn.Client: Application report for application_1426595477608_0002 (state: ACCEPTED)
> 15/03/17 21:27:42 INFO yarn.Client: Application report for application_1426595477608_0002 (state: ACCEPTED)
> 15/03/17 21:27:43 INFO yarn.Client: Application report for application_1426595477608_0002 (state: ACCEPTED)
> 15/03/17 21:27:44 INFO yarn.Client: Application report for application_1426595477608_0002 (state: ACCEPTED)
> 15/03/17 21:27:45 INFO yarn.Client: Application report for application_1426595477608_0002 (state: ACCEPTED)
> 15/03/17 21:27:46 INFO yarn.Client: Application report for application_1426595477608_0002 (state: ACCEPTED)
> 15/03/17 21:27:47 INFO yarn.Client: Application report for application_1426595477608_0002 (state: ACCEPTED)
> 15/03/17 21:27:48 INFO yarn.Client: Application report for application_1426595477608_0002 (state: ACCEPTED)
> 15/03/17 21:27:49 INFO yarn.Client: Application report for application_1426595477608_0002 (state: ACCEPTED)
> 15/03/17 21:27:50 INFO yarn.Client: Application report for application_1426595477608_0002 (state: ACCEPTED)
> 15/03/17 21:27:51 INFO yarn.Client: Application report for application_1426595477608_0002 (state: ACCEPTED)
> 15/03/17 21:27:52 INFO yarn.Client: Application report for application_1426595477608_0002 (state: ACCEPTED)
> 15/03/17 21:27:53 INFO yarn.Client: Application report for application_1426595477608_0002 (state: ACCEPTED)
> 15/03/17 21:27:54 INFO yarn.Client: Application report for application_1426595477608_0002 (state: ACCEPTED)
> 15/03/17 21:27:55 INFO yarn.Client: Application report for application_1426595477608_0002 (state: ACCEPTED)
> 15/03/17 21:27:56 INFO yarn.Client: Application report for application_1426595477608_0002 (state: ACCEPTED)
> 15/03/17 21:27:57 INFO yarn.Client: Application report for application_1426595477608_0002 (state: ACCEPTED)
> 15/03/17 21:27:58 INFO yarn.Client: Application report for application_1426595477608_0002 (state: ACCEPTED)
> 15/03/17 21:27:59 INFO yarn.Client: Application report for application_1426595477608_0002 (state: ACCEPTED)
> 15/03/17 21:28:00 INFO yarn.Client: Application report for application_1426595477608_0002 (state: ACCEPTED)
> 15/03/17 21:28:01 INFO yarn.Client: Application report for application_1426595477608_0002 (state: ACCEPTED)
> 15/03/17 21:28:02 INFO yarn.Client: Application report for application_1426595477608_0002 (state: ACCEPTED)
> 15/03/17 21:28:03 INFO yarn.Client: Application report for application_1426595477608_0002 (state: ACCEPTED)
> 15/03/17 21:28:04 INFO yarn.Client: Application report for application_1426595477608_0002 (state: ACCEPTED)
> 15/03/17 21:28:05 INFO yarn.Client: Application report for application_1426595477608_0002 (state: ACCEPTED)
> 15/03/17 21:28:06 INFO yarn.Client: Application report for application_1426595477608_0002 (state: ACCEPTED)
> 15/03/17 21:28:07 INFO yarn.Client: Application report for application_1426595477608_0002 (state: ACCEPTED)
> 15/03/17 21:28:08 INFO yarn.Client: Application report for application_1426595477608_0002 (state: ACCEPTED)
> 15/03/17 21:28:08 INFO cluster.YarnClientSchedulerBackend: ApplicationMaster registered as Actor[akka.tcp://sparkYarnAM@Server2:42161/user/YarnAM#1971185024]
> 15/03/17 21:28:08 INFO cluster.YarnClientSchedulerBackend: Add WebUI Filter. org.apache.hadoop.yarn.server.webproxy.amfilter.AmIpFilter, Map(PROXY_HOSTS -> Server2, PROXY_URI_BASES -> http://Server2:8088/proxy/application_1426595477608_0002), /proxy/application_1426595477608_0002
> 15/03/17 21:28:08 INFO ui.JettyUtils: Adding filter: org.apache.hadoop.yarn.server.webproxy.amfilter.AmIpFilter
> 15/03/17 21:28:09 INFO yarn.Client: Application report for application_1426595477608_0002 (state: RUNNING)
> 15/03/17 21:28:09 INFO yarn.Client:
> client token: N/A
> diagnostics: N/A
> ApplicationMaster host: Server2
> ApplicationMaster RPC port: 0
> queue: default
> start time: 1426598841696
> final status: UNDEFINED
> tracking URL: http://Server3:8088/proxy/application_1426595477608_0002/
> user: hduser
> 15/03/17 21:28:09 INFO cluster.YarnClientSchedulerBackend: Application application_1426595477608_0002 has started running.
> 15/03/17 21:28:09 INFO netty.NettyBlockTransferService: Server created on 41323
> 15/03/17 21:28:09 INFO storage.BlockManagerMaster: Trying to register BlockManager
> 15/03/17 21:28:09 INFO storage.BlockManagerMasterActor: Registering block manager Server3:41323 with 265.1 MB RAM, BlockManagerId(<driver>, Server3, 41323)
> 15/03/17 21:28:09 INFO storage.BlockManagerMaster: Registered BlockManager
> 15/03/17 21:28:09 INFO cluster.YarnClientSchedulerBackend: SchedulerBackend is ready for scheduling beginning after waiting maxRegisteredResourcesWaitingTime: 30000(ms)
> 15/03/17 21:28:09 INFO storage.MemoryStore: ensureFreeSpace(85091) called with curMem=0, maxMem=278019440
> 15/03/17 21:28:09 INFO storage.MemoryStore: Block broadcast_0 stored as values in memory (estimated size 83.1 KB, free 265.1 MB)
> 15/03/17 21:28:10 INFO storage.MemoryStore: ensureFreeSpace(36387) called with curMem=85091, maxMem=278019440
> 15/03/17 21:28:10 INFO storage.MemoryStore: Block broadcast_0_piece0 stored as bytes in memory (estimated size 35.5 KB, free 265.0 MB)
> 15/03/17 21:28:10 INFO storage.BlockManagerInfo: Added broadcast_0_piece0 in memory on Server3:41323 (size: 35.5 KB, free: 265.1 MB)
> 15/03/17 21:28:10 INFO storage.BlockManagerMaster: Updated info of block broadcast_0_piece0
> 15/03/17 21:28:10 INFO spark.SparkContext: Created broadcast 0 from textFile at Test1.java:30
> 15/03/17 21:28:10 INFO mapred.FileInputFormat: Total input paths to process : 1
> 15/03/17 21:28:10 INFO spark.SparkContext: Starting job: first at RowMatrix.scala:62
> 15/03/17 21:28:10 INFO scheduler.DAGScheduler: Got job 0 (first at RowMatrix.scala:62) with 1 output partitions (allowLocal=true)
> 15/03/17 21:28:10 INFO scheduler.DAGScheduler: Final stage: Stage 0(first at RowMatrix.scala:62)
> 15/03/17 21:28:10 INFO scheduler.DAGScheduler: Parents of final stage: List()
> 15/03/17 21:28:10 INFO scheduler.DAGScheduler: Missing parents: List()
> 15/03/17 21:28:10 INFO scheduler.DAGScheduler: Submitting Stage 0 (MapPartitionsRDD[2] at map at Test1.java:33), which has no missing parents
> 15/03/17 21:28:10 INFO storage.MemoryStore: ensureFreeSpace(3784) called with curMem=121478, maxMem=278019440
> 15/03/17 21:28:10 INFO storage.MemoryStore: Block broadcast_1 stored as values in memory (estimated size 3.7 KB, free 265.0 MB)
> 15/03/17 21:28:10 INFO storage.MemoryStore: ensureFreeSpace(2755) called with curMem=125262, maxMem=278019440
> 15/03/17 21:28:10 INFO storage.MemoryStore: Block broadcast_1_piece0 stored as bytes in memory (estimated size 2.7 KB, free 265.0 MB)
> 15/03/17 21:28:10 INFO storage.BlockManagerInfo: Added broadcast_1_piece0 in memory on Server3:41323 (size: 2.7 KB, free: 265.1 MB)
> 15/03/17 21:28:10 INFO storage.BlockManagerMaster: Updated info of block broadcast_1_piece0
> 15/03/17 21:28:10 INFO spark.SparkContext: Created broadcast 1 from broadcast at DAGScheduler.scala:839
> 15/03/17 21:28:10 INFO scheduler.DAGScheduler: Submitting 1 missing tasks from Stage 0 (MapPartitionsRDD[2] at map at Test1.java:33)
> 15/03/17 21:28:10 INFO cluster.YarnScheduler: Adding task set 0.0 with 1 tasks
> 15/03/17 21:28:10 WARN remote.ReliableDeliverySupervisor: Association with remote system [akka.tcp://sparkYarnAM@Server2:42161] has failed, address is now gated for [5000] ms. Reason is: [Disassociated].
> 15/03/17 21:28:14 INFO cluster.YarnClientSchedulerBackend: ApplicationMaster registered as Actor[akka.tcp://sparkYarnAM@Server2:34207/user/YarnAM#1946221926]
> 15/03/17 21:28:14 INFO cluster.YarnClientSchedulerBackend: Add WebUI Filter. org.apache.hadoop.yarn.server.webproxy.amfilter.AmIpFilter, Map(PROXY_HOSTS -> Server2, PROXY_URI_BASES -> http://Server2:8088/proxy/application_1426595477608_0002), /proxy/application_1426595477608_0002
> 15/03/17 21:28:14 INFO ui.JettyUtils: Adding filter: org.apache.hadoop.yarn.server.webproxy.amfilter.AmIpFilter
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org