You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@spark.apache.org by Stuti Awasthi <st...@hcl.com> on 2014/12/04 07:05:10 UTC
Issue in executing Spark Application from Eclipse
Hi All,
I have a standalone Spark(1.1) cluster on one machine and I have installed scala Eclipse IDE (scala 2.10) on my desktop. I am trying to execute a spark code to execute over my standalone cluster but getting errors.
Please guide me to resolve this.
Code:
val logFile = "<File Path present on desktop>" // Should be some file on your system
val conf = new SparkConf().setAppName("Simple Application").setMaster("spark://<IP>:<PORT>").setSparkHome("/home/stuti/Spark/spark-1.1.0-bin-hadoop1");
val sc = new SparkContext(conf)
println(sc.master) // Print correct master
val logData = sc.textFile(logFile, 2).cache()
println(logData.count) // throws error
Error :
Using Spark's default log4j profile: org/apache/spark/log4j-defaults.properties
14/12/04 11:05:38 INFO SecurityManager: Changing view acls to: stutiawasthi,
14/12/04 11:05:38 INFO SecurityManager: Changing modify acls to: stutiawasthi,
14/12/04 11:05:38 INFO SecurityManager: SecurityManager: authentication disabled; ui acls disabled; users with view permissions: Set(stutiawasthi, ); users with modify permissions: Set(stutiawasthi, )
14/12/04 11:05:39 INFO Slf4jLogger: Slf4jLogger started
14/12/04 11:05:39 INFO Remoting: Starting remoting
14/12/04 11:05:40 INFO Remoting: Remoting started; listening on addresses :[akka.tcp://sparkDriver@<HOSTNAME_DESKTOP>:62308]
14/12/04 11:05:40 INFO Remoting: Remoting now listens on addresses: [akka.tcp://sparkDriver@<HOSTNAME_DESKTOP>:62308]
14/12/04 11:05:40 INFO Utils: Successfully started service 'sparkDriver' on port 62308.
14/12/04 11:05:40 INFO SparkEnv: Registering MapOutputTracker
14/12/04 11:05:40 INFO SparkEnv: Registering BlockManagerMaster
14/12/04 11:05:40 INFO DiskBlockManager: Created local directory at C:\Users\STUTIA~1\AppData\Local\Temp\spark-local-20141204110540-ad60
14/12/04 11:05:40 INFO Utils: Successfully started service 'Connection manager for block manager' on port 62311.
14/12/04 11:05:40 INFO ConnectionManager: Bound socket to port 62311 with id = ConnectionManagerId(<HOSTNAME_DESKTOP>,62311)
14/12/04 11:05:41 INFO MemoryStore: MemoryStore started with capacity 133.6 MB
14/12/04 11:05:41 INFO BlockManagerMaster: Trying to register BlockManager
14/12/04 11:05:41 INFO BlockManagerMasterActor: Registering block manager <HOSTNAME_DESKTOP>:62311 with 133.6 MB RAM
14/12/04 11:05:41 INFO BlockManagerMaster: Registered BlockManager
14/12/04 11:05:41 INFO HttpFileServer: HTTP File server directory is C:\Users\STUTIA~1\AppData\Local\Temp\spark-b65e69f4-69b9-4bb2-b41f-67165909e4c7
14/12/04 11:05:41 INFO HttpServer: Starting HTTP Server
14/12/04 11:05:41 INFO Utils: Successfully started service 'HTTP file server' on port 62312.
14/12/04 11:05:42 INFO Utils: Successfully started service 'SparkUI' on port 4040.
14/12/04 11:05:42 INFO SparkUI: Started SparkUI at http://<HOSTNAME_DESKTOP>:4040
14/12/04 11:05:43 INFO AppClient$ClientActor: Connecting to master spark://10.112.67.80:7077...
14/12/04 11:05:43 INFO SparkDeploySchedulerBackend: SchedulerBackend is ready for scheduling beginning after reached minRegisteredResourcesRatio: 0.0
spark://10.112.67.80:7077
14/12/04 11:05:44 WARN SizeEstimator: Failed to check whether UseCompressedOops is set; assuming yes
14/12/04 11:05:45 INFO MemoryStore: ensureFreeSpace(31447) called with curMem=0, maxMem=140142182
14/12/04 11:05:45 INFO MemoryStore: Block broadcast_0 stored as values in memory (estimated size 30.7 KB, free 133.6 MB)
14/12/04 11:05:45 INFO MemoryStore: ensureFreeSpace(3631) called with curMem=31447, maxMem=140142182
14/12/04 11:05:45 INFO MemoryStore: Block broadcast_0_piece0 stored as bytes in memory (estimated size 3.5 KB, free 133.6 MB)
14/12/04 11:05:45 INFO BlockManagerInfo: Added broadcast_0_piece0 in memory on <HOSTNAME_DESKTOP>:62311 (size: 3.5 KB, free: 133.6 MB)
14/12/04 11:05:45 INFO BlockManagerMaster: Updated info of block broadcast_0_piece0
14/12/04 11:05:45 WARN NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
14/12/04 11:05:45 WARN LoadSnappy: Snappy native library not loaded
14/12/04 11:05:46 INFO FileInputFormat: Total input paths to process : 1
14/12/04 11:05:46 INFO SparkContext: Starting job: count at Test.scala:15
14/12/04 11:05:46 INFO DAGScheduler: Got job 0 (count at Test.scala:15) with 2 output partitions (allowLocal=false)
14/12/04 11:05:46 INFO DAGScheduler: Final stage: Stage 0(count at Test.scala:15)
14/12/04 11:05:46 INFO DAGScheduler: Parents of final stage: List()
14/12/04 11:05:46 INFO DAGScheduler: Missing parents: List()
14/12/04 11:05:46 INFO DAGScheduler: Submitting Stage 0 (D:/Workspace/Spark/Test/README MappedRDD[1] at textFile at Test.scala:14), which has no missing parents
14/12/04 11:05:46 INFO MemoryStore: ensureFreeSpace(2408) called with curMem=35078, maxMem=140142182
14/12/04 11:05:46 INFO MemoryStore: Block broadcast_1 stored as values in memory (estimated size 2.4 KB, free 133.6 MB)
14/12/04 11:05:46 INFO MemoryStore: ensureFreeSpace(1541) called with curMem=37486, maxMem=140142182
14/12/04 11:05:46 INFO MemoryStore: Block broadcast_1_piece0 stored as bytes in memory (estimated size 1541.0 B, free 133.6 MB)
14/12/04 11:05:46 INFO BlockManagerInfo: Added broadcast_1_piece0 in memory on <HOSTNAME_DESKTOP>:62311 (size: 1541.0 B, free: 133.6 MB)
14/12/04 11:05:46 INFO BlockManagerMaster: Updated info of block broadcast_1_piece0
14/12/04 11:05:46 INFO DAGScheduler: Submitting 2 missing tasks from Stage 0 (D:/Workspace/Spark/Test/README MappedRDD[1] at textFile at Test.scala:14)
14/12/04 11:05:46 INFO TaskSchedulerImpl: Adding task set 0.0 with 2 tasks
14/12/04 11:06:01 WARN TaskSchedulerImpl: Initial job has not accepted any resources; check your cluster UI to ensure that workers are registered and have sufficient memory
14/12/04 11:06:03 INFO AppClient$ClientActor: Connecting to master spark://10.112.67.80:7077...
14/12/04 11:06:16 WARN TaskSchedulerImpl: Initial job has not accepted any resources; check your cluster UI to ensure that workers are registered and have sufficient memory
14/12/04 11:06:23 INFO AppClient$ClientActor: Connecting to master spark://10.112.67.80:7077...
14/12/04 11:06:31 WARN TaskSchedulerImpl: Initial job has not accepted any resources; check your cluster UI to ensure that workers are registered and have sufficient memory
Thanks
Stuti Awasthi
::DISCLAIMER::
----------------------------------------------------------------------------------------------------------------------------------------------------
The contents of this e-mail and any attachment(s) are confidential and intended for the named recipient(s) only.
E-mail transmission is not guaranteed to be secure or error-free as information could be intercepted, corrupted,
lost, destroyed, arrive late or incomplete, or may contain viruses in transmission. The e mail and its contents
(with or without referred errors) shall therefore not attach any liability on the originator or HCL or its affiliates.
Views or opinions, if any, presented in this email are solely those of the author and may not necessarily reflect the
views or opinions of HCL or its affiliates. Any form of reproduction, dissemination, copying, disclosure, modification,
distribution and / or publication of this message without the prior written consent of authorized representative of
HCL is strictly prohibited. If you have received this email in error please delete it and notify the sender immediately.
Before opening any email and/or attachments, please check them for viruses and other defects.
----------------------------------------------------------------------------------------------------------------------------------------------------
Re: Issue in executing Spark Application from Eclipse
Posted by Andrew Or <an...@databricks.com>.
Hey Stuti,
Did you start your standalone Master and Workers? You can do this through
sbin/start-all.sh (see
http://spark.apache.org/docs/latest/spark-standalone.html). Otherwise, I
would recommend launching your application from the command line through
bin/spark-submit. I am not sure if we officially support launching Spark
applications from an IDE, because spark-submit handles very specific cases
of how we set up class paths and JVM memory etc.
-Andrew
2014-12-03 22:05 GMT-08:00 Stuti Awasthi <st...@hcl.com>:
> Hi All,
>
> I have a standalone Spark(1.1) cluster on one machine and I have installed
> scala Eclipse IDE (scala 2.10) on my desktop. I am trying to execute a
> spark code to execute over my standalone cluster but getting errors.
>
> Please guide me to resolve this.
>
>
>
> Code:
>
> val logFile = "<File Path present on desktop>" // Should be some file on
> your system
>
> val conf = new SparkConf().setAppName("Simple
> Application").setMaster("spark://<IP>:<PORT>").setSparkHome("/home/stuti/Spark/spark-1.1.0-bin-hadoop1");
>
> val sc = new SparkContext(conf)
>
>
> println(sc.master)
> // Print correct master
>
> val logData = sc.textFile(logFile, 2).cache()
>
>
> println(logData.count)
> // throws error
>
>
>
>
>
>
>
> Error :
>
> Using Spark's default log4j profile:
> org/apache/spark/log4j-defaults.properties
>
> 14/12/04 11:05:38 INFO SecurityManager: Changing view acls to:
> stutiawasthi,
>
> 14/12/04 11:05:38 INFO SecurityManager: Changing modify acls to:
> stutiawasthi,
>
> 14/12/04 11:05:38 INFO SecurityManager: SecurityManager: authentication
> disabled; ui acls disabled; users with view permissions: Set(stutiawasthi,
> ); users with modify permissions: Set(stutiawasthi, )
>
> 14/12/04 11:05:39 INFO Slf4jLogger: Slf4jLogger started
>
> 14/12/04 11:05:39 INFO Remoting: Starting remoting
>
> 14/12/04 11:05:40 INFO Remoting: Remoting started; listening on addresses
> :[akka.tcp://sparkDriver@<HOSTNAME_DESKTOP>:62308]
>
> 14/12/04 11:05:40 INFO Remoting: Remoting now listens on addresses:
> [akka.tcp://sparkDriver@<HOSTNAME_DESKTOP>:62308]
>
> 14/12/04 11:05:40 INFO Utils: Successfully started service 'sparkDriver'
> on port 62308.
>
> 14/12/04 11:05:40 INFO SparkEnv: Registering MapOutputTracker
>
> 14/12/04 11:05:40 INFO SparkEnv: Registering BlockManagerMaster
>
> 14/12/04 11:05:40 INFO DiskBlockManager: Created local directory at
> C:\Users\STUTIA~1\AppData\Local\Temp\spark-local-20141204110540-ad60
>
> 14/12/04 11:05:40 INFO Utils: Successfully started service 'Connection
> manager for block manager' on port 62311.
>
> 14/12/04 11:05:40 INFO ConnectionManager: Bound socket to port 62311 with
> id = ConnectionManagerId(<HOSTNAME_DESKTOP>,62311)
>
> 14/12/04 11:05:41 INFO MemoryStore: MemoryStore started with capacity
> 133.6 MB
>
> 14/12/04 11:05:41 INFO BlockManagerMaster: Trying to register BlockManager
>
> 14/12/04 11:05:41 INFO BlockManagerMasterActor: Registering block manager
> <HOSTNAME_DESKTOP>:62311 with 133.6 MB RAM
>
> 14/12/04 11:05:41 INFO BlockManagerMaster: Registered BlockManager
>
> 14/12/04 11:05:41 INFO HttpFileServer: HTTP File server directory is
> C:\Users\STUTIA~1\AppData\Local\Temp\spark-b65e69f4-69b9-4bb2-b41f-67165909e4c7
>
> 14/12/04 11:05:41 INFO HttpServer: Starting HTTP Server
>
> 14/12/04 11:05:41 INFO Utils: Successfully started service 'HTTP file
> server' on port 62312.
>
> 14/12/04 11:05:42 INFO Utils: Successfully started service 'SparkUI' on
> port 4040.
>
> 14/12/04 11:05:42 INFO SparkUI: Started SparkUI at http://
> <HOSTNAME_DESKTOP>:4040
>
> 14/12/04 11:05:43 INFO AppClient$ClientActor: Connecting to master
> spark://10.112.67.80:7077...
>
> 14/12/04 11:05:43 INFO SparkDeploySchedulerBackend: SchedulerBackend is
> ready for scheduling beginning after reached minRegisteredResourcesRatio:
> 0.0
>
> spark://10.112.67.80:7077
>
> 14/12/04 11:05:44 WARN SizeEstimator: Failed to check whether
> UseCompressedOops is set; assuming yes
>
> 14/12/04 11:05:45 INFO MemoryStore: ensureFreeSpace(31447) called with
> curMem=0, maxMem=140142182
>
> 14/12/04 11:05:45 INFO MemoryStore: Block broadcast_0 stored as values in
> memory (estimated size 30.7 KB, free 133.6 MB)
>
> 14/12/04 11:05:45 INFO MemoryStore: ensureFreeSpace(3631) called with
> curMem=31447, maxMem=140142182
>
> 14/12/04 11:05:45 INFO MemoryStore: Block broadcast_0_piece0 stored as
> bytes in memory (estimated size 3.5 KB, free 133.6 MB)
>
> 14/12/04 11:05:45 INFO BlockManagerInfo: Added broadcast_0_piece0 in
> memory on <HOSTNAME_DESKTOP>:62311 (size: 3.5 KB, free: 133.6 MB)
>
> 14/12/04 11:05:45 INFO BlockManagerMaster: Updated info of block
> broadcast_0_piece0
>
> 14/12/04 11:05:45 WARN NativeCodeLoader: Unable to load native-hadoop
> library for your platform... using builtin-java classes where applicable
>
> 14/12/04 11:05:45 WARN LoadSnappy: Snappy native library not loaded
>
> 14/12/04 11:05:46 INFO FileInputFormat: Total input paths to process : 1
>
> 14/12/04 11:05:46 INFO SparkContext: Starting job: count at Test.scala:15
>
> 14/12/04 11:05:46 INFO DAGScheduler: Got job 0 (count at Test.scala:15)
> with 2 output partitions (allowLocal=false)
>
> 14/12/04 11:05:46 INFO DAGScheduler: Final stage: Stage 0(count at
> Test.scala:15)
>
> 14/12/04 11:05:46 INFO DAGScheduler: Parents of final stage: List()
>
> 14/12/04 11:05:46 INFO DAGScheduler: Missing parents: List()
>
> 14/12/04 11:05:46 INFO DAGScheduler: Submitting Stage 0
> (D:/Workspace/Spark/Test/README MappedRDD[1] at textFile at Test.scala:14),
> which has no missing parents
>
> 14/12/04 11:05:46 INFO MemoryStore: ensureFreeSpace(2408) called with
> curMem=35078, maxMem=140142182
>
> 14/12/04 11:05:46 INFO MemoryStore: Block broadcast_1 stored as values in
> memory (estimated size 2.4 KB, free 133.6 MB)
>
> 14/12/04 11:05:46 INFO MemoryStore: ensureFreeSpace(1541) called with
> curMem=37486, maxMem=140142182
>
> 14/12/04 11:05:46 INFO MemoryStore: Block broadcast_1_piece0 stored as
> bytes in memory (estimated size 1541.0 B, free 133.6 MB)
>
> 14/12/04 11:05:46 INFO BlockManagerInfo: Added broadcast_1_piece0 in
> memory on <HOSTNAME_DESKTOP>:62311 (size: 1541.0 B, free: 133.6 MB)
>
> 14/12/04 11:05:46 INFO BlockManagerMaster: Updated info of block
> broadcast_1_piece0
>
> 14/12/04 11:05:46 INFO DAGScheduler: Submitting 2 missing tasks from Stage
> 0 (D:/Workspace/Spark/Test/README MappedRDD[1] at textFile at Test.scala:14)
>
> 14/12/04 11:05:46 INFO TaskSchedulerImpl: Adding task set 0.0 with 2 tasks
>
> 14/12/04 11:06:01 WARN TaskSchedulerImpl: Initial job has not accepted any
> resources; check your cluster UI to ensure that workers are registered and
> have sufficient memory
>
> 14/12/04 11:06:03 INFO AppClient$ClientActor: Connecting to master
> spark://10.112.67.80:7077...
>
> 14/12/04 11:06:16 WARN TaskSchedulerImpl: Initial job has not accepted any
> resources; check your cluster UI to ensure that workers are registered and
> have sufficient memory
>
> 14/12/04 11:06:23 INFO AppClient$ClientActor: Connecting to master
> spark://10.112.67.80:7077...
>
> 14/12/04 11:06:31 WARN TaskSchedulerImpl: Initial job has not accepted any
> resources; check your cluster UI to ensure that workers are registered and
> have sufficient memory
>
>
>
> Thanks
>
> Stuti Awasthi
>
>
>
>
>
> ::DISCLAIMER::
>
> ----------------------------------------------------------------------------------------------------------------------------------------------------
>
> The contents of this e-mail and any attachment(s) are confidential and
> intended for the named recipient(s) only.
> E-mail transmission is not guaranteed to be secure or error-free as
> information could be intercepted, corrupted,
> lost, destroyed, arrive late or incomplete, or may contain viruses in
> transmission. The e mail and its contents
> (with or without referred errors) shall therefore not attach any liability
> on the originator or HCL or its affiliates.
> Views or opinions, if any, presented in this email are solely those of the
> author and may not necessarily reflect the
> views or opinions of HCL or its affiliates. Any form of reproduction,
> dissemination, copying, disclosure, modification,
> distribution and / or publication of this message without the prior
> written consent of authorized representative of
> HCL is strictly prohibited. If you have received this email in error
> please delete it and notify the sender immediately.
> Before opening any email and/or attachments, please check them for viruses
> and other defects.
>
>
> ----------------------------------------------------------------------------------------------------------------------------------------------------
>
Re: Issue in executing Spark Application from Eclipse
Posted by Akhil Das <ak...@sigmoidanalytics.com>.
It seems you provided master url as spark://10.112.67.80:7077 , i think you
should give spark://ubuntu:7077 instead.
Thanks
Best Regards
On Thu, Dec 4, 2014 at 11:35 AM, Stuti Awasthi <st...@hcl.com> wrote:
> Hi All,
>
> I have a standalone Spark(1.1) cluster on one machine and I have installed
> scala Eclipse IDE (scala 2.10) on my desktop. I am trying to execute a
> spark code to execute over my standalone cluster but getting errors.
>
> Please guide me to resolve this.
>
>
>
> Code:
>
> val logFile = "<File Path present on desktop>" // Should be some file on
> your system
>
> val conf = new SparkConf().setAppName("Simple
> Application").setMaster("spark://<IP>:<PORT>").setSparkHome("/home/stuti/Spark/spark-1.1.0-bin-hadoop1");
>
> val sc = new SparkContext(conf)
>
>
> println(sc.master)
> // Print correct master
>
> val logData = sc.textFile(logFile, 2).cache()
>
>
> println(logData.count)
> // throws error
>
>
>
>
>
>
>
> Error :
>
> Using Spark's default log4j profile:
> org/apache/spark/log4j-defaults.properties
>
> 14/12/04 11:05:38 INFO SecurityManager: Changing view acls to:
> stutiawasthi,
>
> 14/12/04 11:05:38 INFO SecurityManager: Changing modify acls to:
> stutiawasthi,
>
> 14/12/04 11:05:38 INFO SecurityManager: SecurityManager: authentication
> disabled; ui acls disabled; users with view permissions: Set(stutiawasthi,
> ); users with modify permissions: Set(stutiawasthi, )
>
> 14/12/04 11:05:39 INFO Slf4jLogger: Slf4jLogger started
>
> 14/12/04 11:05:39 INFO Remoting: Starting remoting
>
> 14/12/04 11:05:40 INFO Remoting: Remoting started; listening on addresses
> :[akka.tcp://sparkDriver@<HOSTNAME_DESKTOP>:62308]
>
> 14/12/04 11:05:40 INFO Remoting: Remoting now listens on addresses:
> [akka.tcp://sparkDriver@<HOSTNAME_DESKTOP>:62308]
>
> 14/12/04 11:05:40 INFO Utils: Successfully started service 'sparkDriver'
> on port 62308.
>
> 14/12/04 11:05:40 INFO SparkEnv: Registering MapOutputTracker
>
> 14/12/04 11:05:40 INFO SparkEnv: Registering BlockManagerMaster
>
> 14/12/04 11:05:40 INFO DiskBlockManager: Created local directory at
> C:\Users\STUTIA~1\AppData\Local\Temp\spark-local-20141204110540-ad60
>
> 14/12/04 11:05:40 INFO Utils: Successfully started service 'Connection
> manager for block manager' on port 62311.
>
> 14/12/04 11:05:40 INFO ConnectionManager: Bound socket to port 62311 with
> id = ConnectionManagerId(<HOSTNAME_DESKTOP>,62311)
>
> 14/12/04 11:05:41 INFO MemoryStore: MemoryStore started with capacity
> 133.6 MB
>
> 14/12/04 11:05:41 INFO BlockManagerMaster: Trying to register BlockManager
>
> 14/12/04 11:05:41 INFO BlockManagerMasterActor: Registering block manager
> <HOSTNAME_DESKTOP>:62311 with 133.6 MB RAM
>
> 14/12/04 11:05:41 INFO BlockManagerMaster: Registered BlockManager
>
> 14/12/04 11:05:41 INFO HttpFileServer: HTTP File server directory is
> C:\Users\STUTIA~1\AppData\Local\Temp\spark-b65e69f4-69b9-4bb2-b41f-67165909e4c7
>
> 14/12/04 11:05:41 INFO HttpServer: Starting HTTP Server
>
> 14/12/04 11:05:41 INFO Utils: Successfully started service 'HTTP file
> server' on port 62312.
>
> 14/12/04 11:05:42 INFO Utils: Successfully started service 'SparkUI' on
> port 4040.
>
> 14/12/04 11:05:42 INFO SparkUI: Started SparkUI at http://
> <HOSTNAME_DESKTOP>:4040
>
> 14/12/04 11:05:43 INFO AppClient$ClientActor: Connecting to master
> spark://10.112.67.80:7077...
>
> 14/12/04 11:05:43 INFO SparkDeploySchedulerBackend: SchedulerBackend is
> ready for scheduling beginning after reached minRegisteredResourcesRatio:
> 0.0
>
> spark://10.112.67.80:7077
>
> 14/12/04 11:05:44 WARN SizeEstimator: Failed to check whether
> UseCompressedOops is set; assuming yes
>
> 14/12/04 11:05:45 INFO MemoryStore: ensureFreeSpace(31447) called with
> curMem=0, maxMem=140142182
>
> 14/12/04 11:05:45 INFO MemoryStore: Block broadcast_0 stored as values in
> memory (estimated size 30.7 KB, free 133.6 MB)
>
> 14/12/04 11:05:45 INFO MemoryStore: ensureFreeSpace(3631) called with
> curMem=31447, maxMem=140142182
>
> 14/12/04 11:05:45 INFO MemoryStore: Block broadcast_0_piece0 stored as
> bytes in memory (estimated size 3.5 KB, free 133.6 MB)
>
> 14/12/04 11:05:45 INFO BlockManagerInfo: Added broadcast_0_piece0 in
> memory on <HOSTNAME_DESKTOP>:62311 (size: 3.5 KB, free: 133.6 MB)
>
> 14/12/04 11:05:45 INFO BlockManagerMaster: Updated info of block
> broadcast_0_piece0
>
> 14/12/04 11:05:45 WARN NativeCodeLoader: Unable to load native-hadoop
> library for your platform... using builtin-java classes where applicable
>
> 14/12/04 11:05:45 WARN LoadSnappy: Snappy native library not loaded
>
> 14/12/04 11:05:46 INFO FileInputFormat: Total input paths to process : 1
>
> 14/12/04 11:05:46 INFO SparkContext: Starting job: count at Test.scala:15
>
> 14/12/04 11:05:46 INFO DAGScheduler: Got job 0 (count at Test.scala:15)
> with 2 output partitions (allowLocal=false)
>
> 14/12/04 11:05:46 INFO DAGScheduler: Final stage: Stage 0(count at
> Test.scala:15)
>
> 14/12/04 11:05:46 INFO DAGScheduler: Parents of final stage: List()
>
> 14/12/04 11:05:46 INFO DAGScheduler: Missing parents: List()
>
> 14/12/04 11:05:46 INFO DAGScheduler: Submitting Stage 0
> (D:/Workspace/Spark/Test/README MappedRDD[1] at textFile at Test.scala:14),
> which has no missing parents
>
> 14/12/04 11:05:46 INFO MemoryStore: ensureFreeSpace(2408) called with
> curMem=35078, maxMem=140142182
>
> 14/12/04 11:05:46 INFO MemoryStore: Block broadcast_1 stored as values in
> memory (estimated size 2.4 KB, free 133.6 MB)
>
> 14/12/04 11:05:46 INFO MemoryStore: ensureFreeSpace(1541) called with
> curMem=37486, maxMem=140142182
>
> 14/12/04 11:05:46 INFO MemoryStore: Block broadcast_1_piece0 stored as
> bytes in memory (estimated size 1541.0 B, free 133.6 MB)
>
> 14/12/04 11:05:46 INFO BlockManagerInfo: Added broadcast_1_piece0 in
> memory on <HOSTNAME_DESKTOP>:62311 (size: 1541.0 B, free: 133.6 MB)
>
> 14/12/04 11:05:46 INFO BlockManagerMaster: Updated info of block
> broadcast_1_piece0
>
> 14/12/04 11:05:46 INFO DAGScheduler: Submitting 2 missing tasks from Stage
> 0 (D:/Workspace/Spark/Test/README MappedRDD[1] at textFile at Test.scala:14)
>
> 14/12/04 11:05:46 INFO TaskSchedulerImpl: Adding task set 0.0 with 2 tasks
>
> 14/12/04 11:06:01 WARN TaskSchedulerImpl: Initial job has not accepted any
> resources; check your cluster UI to ensure that workers are registered and
> have sufficient memory
>
> 14/12/04 11:06:03 INFO AppClient$ClientActor: Connecting to master
> spark://10.112.67.80:7077...
>
> 14/12/04 11:06:16 WARN TaskSchedulerImpl: Initial job has not accepted any
> resources; check your cluster UI to ensure that workers are registered and
> have sufficient memory
>
> 14/12/04 11:06:23 INFO AppClient$ClientActor: Connecting to master
> spark://10.112.67.80:7077...
>
> 14/12/04 11:06:31 WARN TaskSchedulerImpl: Initial job has not accepted any
> resources; check your cluster UI to ensure that workers are registered and
> have sufficient memory
>
>
>
> Thanks
>
> Stuti Awasthi
>
>
>
>
>
> ::DISCLAIMER::
>
> ----------------------------------------------------------------------------------------------------------------------------------------------------
>
> The contents of this e-mail and any attachment(s) are confidential and
> intended for the named recipient(s) only.
> E-mail transmission is not guaranteed to be secure or error-free as
> information could be intercepted, corrupted,
> lost, destroyed, arrive late or incomplete, or may contain viruses in
> transmission. The e mail and its contents
> (with or without referred errors) shall therefore not attach any liability
> on the originator or HCL or its affiliates.
> Views or opinions, if any, presented in this email are solely those of the
> author and may not necessarily reflect the
> views or opinions of HCL or its affiliates. Any form of reproduction,
> dissemination, copying, disclosure, modification,
> distribution and / or publication of this message without the prior
> written consent of authorized representative of
> HCL is strictly prohibited. If you have received this email in error
> please delete it and notify the sender immediately.
> Before opening any email and/or attachments, please check them for viruses
> and other defects.
>
>
> ----------------------------------------------------------------------------------------------------------------------------------------------------
>
Re: Issue in executing Spark Application from Eclipse
Posted by Sonal Goyal <so...@gmail.com>.
Seems like there is an issue with your standalone cluster as can be seen
from the master logs. Are your DNS entries correct?
Best Regards,
Sonal
Founder, Nube Technologies <http://www.nubetech.co>
<http://in.linkedin.com/in/sonalgoyal>
On Thu, Dec 4, 2014 at 11:50 AM, Stuti Awasthi <st...@hcl.com> wrote:
> Thanks Sonal,
>
>
>
> Yes the cluster is up.
>
> · *URL:* spark://ubuntu:7077
>
> · *Workers:* 1
>
> · *Cores:* 1 Total, 0 Used
>
> · *Memory:* 512.0 MB Total, 0.0 B Used
>
> · *Applications:* 0 Running, 0 Completed
>
> · *Drivers:* 0 Running, 0 Completed
>
> · *Status:* ALIVE
>
> Spark Master Logs :
>
> 14/12/03 22:14:38 ERROR EndpointWriter: AssociationError
> [akka.tcp://sparkMaster@ubuntu:7077] -> [akka.tcp://sparkDriver@<Hostname_Desktop>:62152]:
> Error [Association failed with [akka.tcp://sparkDriver@<HOSTNAME_DESKTOP>:62152]]
> [
>
> akka.remote.EndpointAssociationException: Association failed with
> [akka.tcp://sparkDriver@<HOSTNAME_DESKTOP>:62152]
>
> *Caused by:
> akka.remote.transport.netty.NettyTransport$$anonfun$associate$1$$anon$2:
> connection timed out: <HOSTNAME_DESKTOP>/10.112.67.149:62128
> <http://10.112.67.149:62128>*
>
> ]
>
>
>
> Also Both machine are able to ping each other and are on LAN. Desktop
> machine on Windows and Spark on Ubuntu.
>
>
>
> Thanks
>
> Stuti Awasthi
>
>
>
> *From:* Sonal Goyal [mailto:sonalgoyal4@gmail.com]
> *Sent:* Thursday, December 04, 2014 11:44 AM
> *To:* Stuti Awasthi
> *Subject:* Re: Issue in executing Spark Application from Eclipse
>
>
>
> Have you checked your cluster UI? Are the worker processes up? what do the
> worker logs say?
>
>
> Best Regards,
> Sonal
> Founder, Nube Technologies <http://www.nubetech.co>
>
>
>
>
>
> On Thu, Dec 4, 2014 at 11:35 AM, Stuti Awasthi <st...@hcl.com>
> wrote:
>
> Hi All,
>
> I have a standalone Spark(1.1) cluster on one machine and I have installed
> scala Eclipse IDE (scala 2.10) on my desktop. I am trying to execute a
> spark code to execute over my standalone cluster but getting errors.
>
> Please guide me to resolve this.
>
>
>
> Code:
>
> val logFile = "<File Path present on desktop>" // Should be some file on
> your system
>
> val conf = new SparkConf().setAppName("Simple
> Application").setMaster("spark://<IP>:<PORT>").setSparkHome("/home/stuti/Spark/spark-1.1.0-bin-hadoop1");
>
> val sc = new SparkContext(conf)
>
>
> println(sc.master)
> // Print correct master
>
> val logData = sc.textFile(logFile, 2).cache()
>
>
> println(logData.count)
> // throws error
>
>
>
>
>
>
>
> Error :
>
> Using Spark's default log4j profile:
> org/apache/spark/log4j-defaults.properties
>
> 14/12/04 11:05:38 INFO SecurityManager: Changing view acls to:
> stutiawasthi,
>
> 14/12/04 11:05:38 INFO SecurityManager: Changing modify acls to:
> stutiawasthi,
>
> 14/12/04 11:05:38 INFO SecurityManager: SecurityManager: authentication
> disabled; ui acls disabled; users with view permissions: Set(stutiawasthi,
> ); users with modify permissions: Set(stutiawasthi, )
>
> 14/12/04 11:05:39 INFO Slf4jLogger: Slf4jLogger started
>
> 14/12/04 11:05:39 INFO Remoting: Starting remoting
>
> 14/12/04 11:05:40 INFO Remoting: Remoting started; listening on addresses
> :[akka.tcp://sparkDriver@<HOSTNAME_DESKTOP>:62308]
>
> 14/12/04 11:05:40 INFO Remoting: Remoting now listens on addresses:
> [akka.tcp://sparkDriver@<HOSTNAME_DESKTOP>:62308]
>
> 14/12/04 11:05:40 INFO Utils: Successfully started service 'sparkDriver'
> on port 62308.
>
> 14/12/04 11:05:40 INFO SparkEnv: Registering MapOutputTracker
>
> 14/12/04 11:05:40 INFO SparkEnv: Registering BlockManagerMaster
>
> 14/12/04 11:05:40 INFO DiskBlockManager: Created local directory at
> C:\Users\STUTIA~1\AppData\Local\Temp\spark-local-20141204110540-ad60
>
> 14/12/04 11:05:40 INFO Utils: Successfully started service 'Connection
> manager for block manager' on port 62311.
>
> 14/12/04 11:05:40 INFO ConnectionManager: Bound socket to port 62311 with
> id = ConnectionManagerId(<HOSTNAME_DESKTOP>,62311)
>
> 14/12/04 11:05:41 INFO MemoryStore: MemoryStore started with capacity
> 133.6 MB
>
> 14/12/04 11:05:41 INFO BlockManagerMaster: Trying to register BlockManager
>
> 14/12/04 11:05:41 INFO BlockManagerMasterActor: Registering block manager
> <HOSTNAME_DESKTOP>:62311 with 133.6 MB RAM
>
> 14/12/04 11:05:41 INFO BlockManagerMaster: Registered BlockManager
>
> 14/12/04 11:05:41 INFO HttpFileServer: HTTP File server directory is
> C:\Users\STUTIA~1\AppData\Local\Temp\spark-b65e69f4-69b9-4bb2-b41f-67165909e4c7
>
> 14/12/04 11:05:41 INFO HttpServer: Starting HTTP Server
>
> 14/12/04 11:05:41 INFO Utils: Successfully started service 'HTTP file
> server' on port 62312.
>
> 14/12/04 11:05:42 INFO Utils: Successfully started service 'SparkUI' on
> port 4040.
>
> 14/12/04 11:05:42 INFO SparkUI: Started SparkUI at
> http://<HOSTNAME_DESKTOP>:4040
>
> 14/12/04 11:05:43 INFO AppClient$ClientActor: Connecting to master
> spark://10.112.67.80:7077...
>
> 14/12/04 11:05:43 INFO SparkDeploySchedulerBackend: SchedulerBackend is
> ready for scheduling beginning after reached minRegisteredResourcesRatio:
> 0.0
>
> spark://10.112.67.80:7077
>
> 14/12/04 11:05:44 WARN SizeEstimator: Failed to check whether
> UseCompressedOops is set; assuming yes
>
> 14/12/04 11:05:45 INFO MemoryStore: ensureFreeSpace(31447) called with
> curMem=0, maxMem=140142182
>
> 14/12/04 11:05:45 INFO MemoryStore: Block broadcast_0 stored as values in
> memory (estimated size 30.7 KB, free 133.6 MB)
>
> 14/12/04 11:05:45 INFO MemoryStore: ensureFreeSpace(3631) called with
> curMem=31447, maxMem=140142182
>
> 14/12/04 11:05:45 INFO MemoryStore: Block broadcast_0_piece0 stored as
> bytes in memory (estimated size 3.5 KB, free 133.6 MB)
>
> 14/12/04 11:05:45 INFO BlockManagerInfo: Added broadcast_0_piece0 in
> memory on <HOSTNAME_DESKTOP>:62311 (size: 3.5 KB, free: 133.6 MB)
>
> 14/12/04 11:05:45 INFO BlockManagerMaster: Updated info of block
> broadcast_0_piece0
>
> 14/12/04 11:05:45 WARN NativeCodeLoader: Unable to load native-hadoop
> library for your platform... using builtin-java classes where applicable
>
> 14/12/04 11:05:45 WARN LoadSnappy: Snappy native library not loaded
>
> 14/12/04 11:05:46 INFO FileInputFormat: Total input paths to process : 1
>
> 14/12/04 11:05:46 INFO SparkContext: Starting job: count at Test.scala:15
>
> 14/12/04 11:05:46 INFO DAGScheduler: Got job 0 (count at Test.scala:15)
> with 2 output partitions (allowLocal=false)
>
> 14/12/04 11:05:46 INFO DAGScheduler: Final stage: Stage 0(count at
> Test.scala:15)
>
> 14/12/04 11:05:46 INFO DAGScheduler: Parents of final stage: List()
>
> 14/12/04 11:05:46 INFO DAGScheduler: Missing parents: List()
>
> 14/12/04 11:05:46 INFO DAGScheduler: Submitting Stage 0
> (D:/Workspace/Spark/Test/README MappedRDD[1] at textFile at Test.scala:14),
> which has no missing parents
>
> 14/12/04 11:05:46 INFO MemoryStore: ensureFreeSpace(2408) called with
> curMem=35078, maxMem=140142182
>
> 14/12/04 11:05:46 INFO MemoryStore: Block broadcast_1 stored as values in
> memory (estimated size 2.4 KB, free 133.6 MB)
>
> 14/12/04 11:05:46 INFO MemoryStore: ensureFreeSpace(1541) called with
> curMem=37486, maxMem=140142182
>
> 14/12/04 11:05:46 INFO MemoryStore: Block broadcast_1_piece0 stored as
> bytes in memory (estimated size 1541.0 B, free 133.6 MB)
>
> 14/12/04 11:05:46 INFO BlockManagerInfo: Added broadcast_1_piece0 in
> memory on <HOSTNAME_DESKTOP>:62311 (size: 1541.0 B, free: 133.6 MB)
>
> 14/12/04 11:05:46 INFO BlockManagerMaster: Updated info of block
> broadcast_1_piece0
>
> 14/12/04 11:05:46 INFO DAGScheduler: Submitting 2 missing tasks from Stage
> 0 (D:/Workspace/Spark/Test/README MappedRDD[1] at textFile at Test.scala:14)
>
> 14/12/04 11:05:46 INFO TaskSchedulerImpl: Adding task set 0.0 with 2 tasks
>
> 14/12/04 11:06:01 WARN TaskSchedulerImpl: Initial job has not accepted any
> resources; check your cluster UI to ensure that workers are registered and
> have sufficient memory
>
> 14/12/04 11:06:03 INFO AppClient$ClientActor: Connecting to master
> spark://10.112.67.80:7077...
>
> 14/12/04 11:06:16 WARN TaskSchedulerImpl: Initial job has not accepted any
> resources; check your cluster UI to ensure that workers are registered and
> have sufficient memory
>
> 14/12/04 11:06:23 INFO AppClient$ClientActor: Connecting to master
> spark://10.112.67.80:7077...
>
> 14/12/04 11:06:31 WARN TaskSchedulerImpl: Initial job has not accepted any
> resources; check your cluster UI to ensure that workers are registered and
> have sufficient memory
>
>
>
> Thanks
>
> Stuti Awasthi
>
>
>
>
>
> ::DISCLAIMER::
>
> ----------------------------------------------------------------------------------------------------------------------------------------------------
>
> The contents of this e-mail and any attachment(s) are confidential and
> intended for the named recipient(s) only.
> E-mail transmission is not guaranteed to be secure or error-free as
> information could be intercepted, corrupted,
> lost, destroyed, arrive late or incomplete, or may contain viruses in
> transmission. The e mail and its contents
> (with or without referred errors) shall therefore not attach any liability
> on the originator or HCL or its affiliates.
> Views or opinions, if any, presented in this email are solely those of the
> author and may not necessarily reflect the
> views or opinions of HCL or its affiliates. Any form of reproduction,
> dissemination, copying, disclosure, modification,
> distribution and / or publication of this message without the prior
> written consent of authorized representative of
> HCL is strictly prohibited. If you have received this email in error
> please delete it and notify the sender immediately.
> Before opening any email and/or attachments, please check them for viruses
> and other defects.
>
>
> ----------------------------------------------------------------------------------------------------------------------------------------------------
>
>
>