You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@spark.apache.org by Oleg Ruchovets <or...@gmail.com> on 2016/01/18 11:51:56 UTC
spark 1.6.0 on ec2 doesn't work
Hi ,
I try to follow the spartk 1.6.0 to install spark on EC2.
It doesn't work properly - got exceptions and at the end standalone spark
cluster installed.
here is log information:
Any suggestions?
Thanks
Oleg.
oleg@robinhood:~/install/spark-1.6.0-bin-hadoop2.6/ec2$ ./spark-ec2
--key-pair=CC-ES-Demo
--identity-file=/home/oleg/work/entity_extraction_framework/ec2_pem_key/CC-ES-Demo.pem
--region=us-east-1 --zone=us-east-1a --spot-price=0.05 -s 5
--spark-version=1.6.0 launch entity-extraction-spark-cluster
Setting up security groups...
Searching for existing cluster entity-extraction-spark-cluster in region
us-east-1...
Spark AMI: ami-5bb18832
Launching instances...
Requesting 5 slaves as spot instances with price $0.050
Waiting for spot instances to be granted...
0 of 5 slaves granted, waiting longer
0 of 5 slaves granted, waiting longer
0 of 5 slaves granted, waiting longer
0 of 5 slaves granted, waiting longer
0 of 5 slaves granted, waiting longer
0 of 5 slaves granted, waiting longer
0 of 5 slaves granted, waiting longer
0 of 5 slaves granted, waiting longer
0 of 5 slaves granted, waiting longer
All 5 slaves granted
Launched master in us-east-1a, regid = r-9384033f
Waiting for AWS to propagate instance metadata...
Waiting for cluster to enter 'ssh-ready' state..........
Warning: SSH connection error. (This could be temporary.)
Host: ec2-52-90-186-83.compute-1.amazonaws.com
SSH return code: 255
SSH output: ssh: connect to host ec2-52-90-186-83.compute-1.amazonaws.com
port 22: Connection refused
.
Warning: SSH connection error. (This could be temporary.)
Host: ec2-52-90-186-83.compute-1.amazonaws.com
SSH return code: 255
SSH output: ssh: connect to host ec2-52-90-186-83.compute-1.amazonaws.com
port 22: Connection refused
.
Warning: SSH connection error. (This could be temporary.)
Host: ec2-52-90-186-83.compute-1.amazonaws.com
SSH return code: 255
SSH output: ssh: connect to host ec2-52-90-186-83.compute-1.amazonaws.com
port 22: Connection refused
.
Cluster is now in 'ssh-ready' state. Waited 442 seconds.
Generating cluster's SSH key on master...
Warning: Permanently added
'ec2-52-90-186-83.compute-1.amazonaws.com,52.90.186.83'
(ECDSA) to the list of known hosts.
Connection to ec2-52-90-186-83.compute-1.amazonaws.com closed.
Warning: Permanently added
'ec2-52-90-186-83.compute-1.amazonaws.com,52.90.186.83'
(ECDSA) to the list of known hosts.
Transferring cluster's SSH key to slaves...
ec2-54-165-243-74.compute-1.amazonaws.com
Warning: Permanently added
'ec2-54-165-243-74.compute-1.amazonaws.com,54.165.243.74'
(ECDSA) to the list of known hosts.
ec2-54-88-245-107.compute-1.amazonaws.com
Warning: Permanently added
'ec2-54-88-245-107.compute-1.amazonaws.com,54.88.245.107'
(ECDSA) to the list of known hosts.
ec2-54-172-29-47.compute-1.amazonaws.com
Warning: Permanently added
'ec2-54-172-29-47.compute-1.amazonaws.com,54.172.29.47'
(ECDSA) to the list of known hosts.
ec2-54-165-131-210.compute-1.amazonaws.com
Warning: Permanently added
'ec2-54-165-131-210.compute-1.amazonaws.com,54.165.131.210'
(ECDSA) to the list of known hosts.
ec2-54-172-46-184.compute-1.amazonaws.com
Warning: Permanently added
'ec2-54-172-46-184.compute-1.amazonaws.com,54.172.46.184'
(ECDSA) to the list of known hosts.
Cloning spark-ec2 scripts from
https://github.com/amplab/spark-ec2/tree/branch-1.5 on master...
Warning: Permanently added
'ec2-52-90-186-83.compute-1.amazonaws.com,52.90.186.83'
(ECDSA) to the list of known hosts.
Cloning into 'spark-ec2'...
remote: Counting objects: 2068, done.
remote: Total 2068 (delta 0), reused 0 (delta 0), pack-reused 2068
Receiving objects: 100% (2068/2068), 349.76 KiB, done.
Resolving deltas: 100% (796/796), done.
Connection to ec2-52-90-186-83.compute-1.amazonaws.com closed.
Deploying files to master...
Warning: Permanently added
'ec2-52-90-186-83.compute-1.amazonaws.com,52.90.186.83'
(ECDSA) to the list of known hosts.
sending incremental file list
root/spark-ec2/ec2-variables.sh
sent 1,835 bytes received 40 bytes 416.67 bytes/sec
total size is 1,684 speedup is 0.90
Running setup on master...
Warning: Permanently added
'ec2-52-90-186-83.compute-1.amazonaws.com,52.90.186.83'
(ECDSA) to the list of known hosts.
Connection to ec2-52-90-186-83.compute-1.amazonaws.com closed.
Warning: Permanently added
'ec2-52-90-186-83.compute-1.amazonaws.com,52.90.186.83'
(ECDSA) to the list of known hosts.
Setting up Spark on ip-172-31-24-124.ec2.internal...
Setting executable permissions on scripts...
RSYNC'ing /root/spark-ec2 to other cluster nodes...
ec2-54-165-243-74.compute-1.amazonaws.com
Warning: Permanently added
'ec2-54-165-243-74.compute-1.amazonaws.com,172.31.19.61'
(ECDSA) to the list of known hosts.
ec2-54-88-245-107.compute-1.amazonaws.com
id_rsa
100% 1679 1.6KB/s 00:00
Warning: Permanently added
'ec2-54-88-245-107.compute-1.amazonaws.com,172.31.30.81'
(ECDSA) to the list of known hosts.
ec2-54-172-29-47.compute-1.amazonaws.com
id_rsa
100% 1679 1.6KB/s 00:00
Warning: Permanently added
'ec2-54-172-29-47.compute-1.amazonaws.com,172.31.29.54'
(ECDSA) to the list of known hosts.
ec2-54-165-131-210.compute-1.amazonaws.com
Warning: Permanently added
'ec2-54-165-131-210.compute-1.amazonaws.com,172.31.23.10'
(ECDSA) to the list of known hosts.
id_rsa
100% 1679 1.6KB/s 00:00
ec2-54-172-46-184.compute-1.amazonaws.com
Warning: Permanently added
'ec2-54-172-46-184.compute-1.amazonaws.com,172.31.30.167'
(ECDSA) to the list of known hosts.
id_rsa
100% 1679 1.6KB/s 00:00
id_rsa
100% 1679 1.6KB/s 00:00
[timing] rsync /root/spark-ec2: 00h 00m 01s
Running setup-slave on all cluster nodes to mount filesystems, etc...
[1] 08:08:10 [SUCCESS] ec2-52-90-186-83.compute-1.amazonaws.com
checking/fixing resolution of hostname
Setting up slave on ip-172-31-24-124.ec2.internal... of type m1.large
1024+0 records in
1024+0 records out
1073741824 bytes (1.1 GB) copied, 2.01407 s, 533 MB/s
mkswap: /mnt/swap: warning: don't erase bootbits sectors
on whole disk. Use -f to force.
Setting up swapspace version 1, size = 1048572 KiB
no label, UUID=b4d25f54-4732-40bb-8086-c78117cb58b2
Added 1024 MB swap file /mnt/swap
Stderr: Warning: Permanently added
'ec2-52-90-186-83.compute-1.amazonaws.com,172.31.24.124'
(ECDSA) to the list of known hosts.
Connection to ec2-52-90-186-83.compute-1.amazonaws.com closed.
[2] 08:08:24 [SUCCESS] ec2-54-165-243-74.compute-1.amazonaws.com
checking/fixing resolution of hostname
Setting up slave on ip-172-31-19-61.ec2.internal... of type m1.large
1024+0 records in
1024+0 records out
1073741824 bytes (1.1 GB) copied, 2.11705 s, 507 MB/s
mkswap: /mnt/swap: warning: don't erase bootbits sectors
on whole disk. Use -f to force.
Setting up swapspace version 1, size = 1048572 KiB
no label, UUID=928041a8-4d48-4c65-94e2-d9f84e14cad9
Added 1024 MB swap file /mnt/swap
Stderr: Connection to ec2-54-165-243-74.compute-1.amazonaws.com closed.
[3] 08:08:27 [SUCCESS] ec2-54-88-245-107.compute-1.amazonaws.com
checking/fixing resolution of hostname
Setting up slave on ip-172-31-30-81.ec2.internal... of type m1.large
1024+0 records in
1024+0 records out
1073741824 bytes (1.1 GB) copied, 2.21007 s, 486 MB/s
mkswap: /mnt/swap: warning: don't erase bootbits sectors
on whole disk. Use -f to force.
Setting up swapspace version 1, size = 1048572 KiB
no label, UUID=1e8c3d4c-7e27-4c35-acae-d83ec2ea9edb
Added 1024 MB swap file /mnt/swap
Stderr: Connection to ec2-54-88-245-107.compute-1.amazonaws.com closed.
[4] 08:08:32 [SUCCESS] ec2-54-172-29-47.compute-1.amazonaws.com
checking/fixing resolution of hostname
Setting up slave on ip-172-31-29-54.ec2.internal... of type m1.large
1024+0 records in
1024+0 records out
1073741824 bytes (1.1 GB) copied, 2.15544 s, 498 MB/s
mkswap: /mnt/swap: warning: don't erase bootbits sectors
on whole disk. Use -f to force.
Setting up swapspace version 1, size = 1048572 KiB
no label, UUID=7bd81d33-ae22-4973-810e-855535ecb743
Added 1024 MB swap file /mnt/swap
Stderr: Connection to ec2-54-172-29-47.compute-1.amazonaws.com closed.
[5] 08:08:34 [SUCCESS] ec2-54-165-131-210.compute-1.amazonaws.com
checking/fixing resolution of hostname
Setting up slave on ip-172-31-23-10.ec2.internal... of type m1.large
1024+0 records in
1024+0 records out
1073741824 bytes (1.1 GB) copied, 2.39186 s, 449 MB/s
mkswap: /mnt/swap: warning: don't erase bootbits sectors
on whole disk. Use -f to force.
Setting up swapspace version 1, size = 1048572 KiB
no label, UUID=abbdbe4d-f8e8-469b-90d2-c9d0a244b261
Added 1024 MB swap file /mnt/swap
Stderr: Connection to ec2-54-165-131-210.compute-1.amazonaws.com closed.
[6] 08:08:37 [SUCCESS] ec2-54-172-46-184.compute-1.amazonaws.com
checking/fixing resolution of hostname
Setting up slave on ip-172-31-30-167.ec2.internal... of type m1.large
1024+0 records in
1024+0 records out
1073741824 bytes (1.1 GB) copied, 2.1603 s, 497 MB/s
mkswap: /mnt/swap: warning: don't erase bootbits sectors
on whole disk. Use -f to force.
Setting up swapspace version 1, size = 1048572 KiB
no label, UUID=115ac0e9-c28c-4404-a648-826ece20815d
Added 1024 MB swap file /mnt/swap
Stderr: Connection to ec2-54-172-46-184.compute-1.amazonaws.com closed.
[timing] setup-slave: 00h 00m 45s
Initializing scala
Unpacking Scala
--2016-01-18 08:08:37--
http://s3.amazonaws.com/spark-related-packages/scala-2.10.3.tgz
Resolving s3.amazonaws.com (s3.amazonaws.com)... 54.231.13.224
Connecting to s3.amazonaws.com (s3.amazonaws.com)|54.231.13.224|:80...
connected.
HTTP request sent, awaiting response... 200 OK
Length: 30531249 (29M) [application/x-compressed]
Saving to: ‘scala-2.10.3.tgz’
100%[============================================================================================================================================>]
30,531,249 3.46MB/s in 10s
2016-01-18 08:08:47 (2.86 MB/s) - ‘scala-2.10.3.tgz’ saved
[30531249/30531249]
[timing] scala init: 00h 00m 11s
Initializing spark
--2016-01-18 08:08:48--
http://s3.amazonaws.com/spark-related-packages/spark-1.6.0-bin-hadoop1.tgz
Resolving s3.amazonaws.com (s3.amazonaws.com)... 54.231.81.220
Connecting to s3.amazonaws.com (s3.amazonaws.com)|54.231.81.220|:80...
connected.
HTTP request sent, awaiting response... 200 OK
Length: 243448482 (232M) [application/x-compressed]
Saving to: ‘spark-1.6.0-bin-hadoop1.tgz’
100%[============================================================================================================================================>]
243,448,482 65.6MB/s in 3.5s
2016-01-18 08:08:52 (65.6 MB/s) - ‘spark-1.6.0-bin-hadoop1.tgz’ saved
[243448482/243448482]
Unpacking Spark
[timing] spark init: 00h 00m 08s
Initializing ephemeral-hdfs
--2016-01-18 08:08:56--
http://s3.amazonaws.com/spark-related-packages/hadoop-1.0.4.tar.gz
Resolving s3.amazonaws.com (s3.amazonaws.com)... 54.231.17.48
Connecting to s3.amazonaws.com (s3.amazonaws.com)|54.231.17.48|:80...
connected.
HTTP request sent, awaiting response... 200 OK
Length: 62793050 (60M) [application/x-gzip]
Saving to: ‘hadoop-1.0.4.tar.gz’
100%[============================================================================================================================================>]
62,793,050 69.2MB/s in 0.9s
2016-01-18 08:08:57 (69.2 MB/s) - ‘hadoop-1.0.4.tar.gz’ saved
[62793050/62793050]
Unpacking Hadoop
RSYNC'ing /root/ephemeral-hdfs to slaves...
ec2-54-165-243-74.compute-1.amazonaws.com
Warning: Permanently added
'ec2-54-165-243-74.compute-1.amazonaws.com,172.31.19.61'
(ECDSA) to the list of known hosts.
ec2-54-88-245-107.compute-1.amazonaws.com
Warning: Permanently added
'ec2-54-88-245-107.compute-1.amazonaws.com,172.31.30.81'
(ECDSA) to the list of known hosts.
ec2-54-172-29-47.compute-1.amazonaws.com
Warning: Permanently added
'ec2-54-172-29-47.compute-1.amazonaws.com,172.31.29.54'
(ECDSA) to the list of known hosts.
ec2-54-165-131-210.compute-1.amazonaws.com
Warning: Permanently added
'ec2-54-165-131-210.compute-1.amazonaws.com,172.31.23.10'
(ECDSA) to the list of known hosts.
ec2-54-172-46-184.compute-1.amazonaws.com
Warning: Permanently added
'ec2-54-172-46-184.compute-1.amazonaws.com,172.31.30.167'
(ECDSA) to the list of known hosts.
[timing] ephemeral-hdfs init: 00h 00m 54s
Initializing persistent-hdfs
--2016-01-18 08:09:50--
http://s3.amazonaws.com/spark-related-packages/hadoop-1.0.4.tar.gz
Resolving s3.amazonaws.com (s3.amazonaws.com)... 54.231.49.236
Connecting to s3.amazonaws.com (s3.amazonaws.com)|54.231.49.236|:80...
connected.
HTTP request sent, awaiting response... 200 OK
Length: 62793050 (60M) [application/x-gzip]
Saving to: ‘hadoop-1.0.4.tar.gz’
100%[============================================================================================================================================>]
62,793,050 67.4MB/s in 0.9s
2016-01-18 08:09:51 (67.4 MB/s) - ‘hadoop-1.0.4.tar.gz’ saved
[62793050/62793050]
Unpacking Hadoop
RSYNC'ing /root/persistent-hdfs to slaves...
ec2-54-165-243-74.compute-1.amazonaws.com
ec2-54-88-245-107.compute-1.amazonaws.com
ec2-54-172-29-47.compute-1.amazonaws.com
ec2-54-165-131-210.compute-1.amazonaws.com
ec2-54-172-46-184.compute-1.amazonaws.com
[timing] persistent-hdfs init: 00h 00m 39s
Initializing spark-standalone
[timing] spark-standalone init: 00h 00m 00s
Initializing tachyon
--2016-01-18 08:10:29--
https://s3.amazonaws.com/Tachyon/tachyon-0.8.2-bin.tar.gz
Resolving s3.amazonaws.com (s3.amazonaws.com)... 54.231.81.67
Connecting to s3.amazonaws.com (s3.amazonaws.com)|54.231.81.67|:443...
connected.
HTTP request sent, awaiting response... 403 Forbidden
2016-01-18 08:10:29 ERROR 403: Forbidden.
ERROR: Unknown Tachyon version
tachyon/init.sh: line 60: return: -1: invalid option
return: usage: return [n]
Unpacking Tachyon
tar (child): tachyon-*.tar.gz: Cannot open: No such file or directory
tar (child): Error is not recoverable: exiting now
tar: Child returned status 2
tar: Error is not recoverable: exiting now
rm: cannot remove `tachyon-*.tar.gz': No such file or directory
ls: cannot access tachyon-*: No such file or directory
mv: missing destination file operand after `tachyon'
Try `mv --help' for more information.
[timing] tachyon init: 00h 00m 00s
Initializing rstudio
--2016-01-18 08:10:29--
http://download2.rstudio.org/rstudio-server-rhel-0.99.446-x86_64.rpm
Resolving download2.rstudio.org (download2.rstudio.org)... 54.192.18.169,
54.192.18.246, 54.192.18.133, ...
Connecting to download2.rstudio.org
(download2.rstudio.org)|54.192.18.169|:80...
connected.
HTTP request sent, awaiting response... 200 OK
Length: 35035164 (33M) [application/x-redhat-package-manager]
Saving to: ‘rstudio-server-rhel-0.99.446-x86_64.rpm’
100%[============================================================================================================================================>]
35,035,164 84.0MB/s in 0.4s
2016-01-18 08:10:29 (84.0 MB/s) - ‘rstudio-server-rhel-0.99.446-x86_64.rpm’
saved [35035164/35035164]
Loaded plugins: priorities, update-motd, upgrade-helper
Examining rstudio-server-rhel-0.99.446-x86_64.rpm:
rstudio-server-0.99.446-1.x86_64
Marking rstudio-server-rhel-0.99.446-x86_64.rpm to be installed
Resolving Dependencies
--> Running transaction check
---> Package rstudio-server.x86_64 0:0.99.446-1 will be installed
--> Finished Dependency Resolution
Dependencies Resolved
======================================================================================================================================================================================
Package Arch
Version Repository
Size
======================================================================================================================================================================================
Installing:
rstudio-server x86_64
0.99.446-1 /rstudio-server-rhel-0.99.446-x86_64
252 M
Transaction Summary
======================================================================================================================================================================================
Install 1 Package
Total size: 252 M
Installed size: 252 M
Downloading packages:
Running transaction check
Running transaction test
Transaction test succeeded
Running transaction
Installing : rstudio-server-0.99.446-1.x86_64
1/1
groupadd: group 'rstudio-server' already exists
rsession: no process killed
rstudio-server start/running, process 2535
Verifying : rstudio-server-0.99.446-1.x86_64
1/1
Installed:
rstudio-server.x86_64 0:0.99.446-1
Complete!
rstudio-server start/running, process 2570
[timing] rstudio init: 00h 00m 39s
Initializing ganglia
Connection to ec2-54-165-243-74.compute-1.amazonaws.com closed.
Connection to ec2-54-88-245-107.compute-1.amazonaws.com closed.
Connection to ec2-54-172-29-47.compute-1.amazonaws.com closed.
Connection to ec2-54-165-131-210.compute-1.amazonaws.com closed.
Connection to ec2-54-172-46-184.compute-1.amazonaws.com closed.
[timing] ganglia init: 00h 00m 02s
Creating local config files...
Connection to ec2-54-165-243-74.compute-1.amazonaws.com closed.
Connection to ec2-54-165-243-74.compute-1.amazonaws.com closed.
Configuring /etc/ganglia/gmond.conf
Configuring /etc/ganglia/gmetad.conf
Configuring /etc/httpd/conf.d/ganglia.conf
Configuring /etc/httpd/conf/httpd.conf
Configuring /root/mapreduce/hadoop.version
Configuring /root/mapreduce/conf/core-site.xml
Configuring /root/mapreduce/conf/slaves
Configuring /root/mapreduce/conf/mapred-site.xml
Configuring /root/mapreduce/conf/hdfs-site.xml
Configuring /root/mapreduce/conf/hadoop-env.sh
Configuring /root/mapreduce/conf/masters
Configuring /root/persistent-hdfs/conf/core-site.xml
Configuring /root/persistent-hdfs/conf/slaves
Configuring /root/persistent-hdfs/conf/mapred-site.xml
Configuring /root/persistent-hdfs/conf/hdfs-site.xml
Configuring /root/persistent-hdfs/conf/hadoop-env.sh
Configuring /root/persistent-hdfs/conf/masters
Configuring /root/ephemeral-hdfs/conf/core-site.xml
Configuring /root/ephemeral-hdfs/conf/yarn-site.xml
Configuring /root/ephemeral-hdfs/conf/slaves
Configuring /root/ephemeral-hdfs/conf/mapred-site.xml
Configuring /root/ephemeral-hdfs/conf/hadoop-metrics2.properties
Configuring /root/ephemeral-hdfs/conf/capacity-scheduler.xml
Configuring /root/ephemeral-hdfs/conf/yarn-env.sh
Configuring /root/ephemeral-hdfs/conf/hdfs-site.xml
Configuring /root/ephemeral-hdfs/conf/hadoop-env.sh
Configuring /root/ephemeral-hdfs/conf/masters
Configuring /root/spark/conf/core-site.xml
Configuring /root/spark/conf/spark-defaults.conf
Configuring /root/spark/conf/spark-env.sh
Configuring /root/tachyon/conf/slaves
Configuring /root/tachyon/conf/workers
Configuring /root/tachyon/conf/tachyon-env.sh
Deploying Spark config files...
RSYNC'ing /root/spark/conf to slaves...
ec2-54-165-243-74.compute-1.amazonaws.com
ec2-54-88-245-107.compute-1.amazonaws.com
ec2-54-172-29-47.compute-1.amazonaws.com
ec2-54-165-131-210.compute-1.amazonaws.com
ec2-54-172-46-184.compute-1.amazonaws.com
Setting up scala
RSYNC'ing /root/scala to slaves...
ec2-54-165-243-74.compute-1.amazonaws.com
ec2-54-88-245-107.compute-1.amazonaws.com
ec2-54-172-29-47.compute-1.amazonaws.com
ec2-54-165-131-210.compute-1.amazonaws.com
ec2-54-172-46-184.compute-1.amazonaws.com
[timing] scala setup: 00h 00m 09s
Setting up spark
RSYNC'ing /root/spark to slaves...
ec2-54-165-243-74.compute-1.amazonaws.com
ec2-54-88-245-107.compute-1.amazonaws.com
ec2-54-172-29-47.compute-1.amazonaws.com
ec2-54-165-131-210.compute-1.amazonaws.com
ec2-54-172-46-184.compute-1.amazonaws.com
[timing] spark setup: 00h 01m 07s
Setting up ephemeral-hdfs
ec2-54-165-243-74.compute-1.amazonaws.com
ec2-54-88-245-107.compute-1.amazonaws.com
Connection to ec2-54-165-243-74.compute-1.amazonaws.com closed.
ec2-54-172-29-47.compute-1.amazonaws.com
Connection to ec2-54-88-245-107.compute-1.amazonaws.com closed.
Connection to ec2-54-172-29-47.compute-1.amazonaws.com closed.
ec2-54-165-131-210.compute-1.amazonaws.com
Connection to ec2-54-165-131-210.compute-1.amazonaws.com closed.
ec2-54-172-46-184.compute-1.amazonaws.com
Connection to ec2-54-172-46-184.compute-1.amazonaws.com closed.
RSYNC'ing /root/ephemeral-hdfs/conf to slaves...
ec2-54-165-243-74.compute-1.amazonaws.com
ec2-54-88-245-107.compute-1.amazonaws.com
ec2-54-172-29-47.compute-1.amazonaws.com
ec2-54-165-131-210.compute-1.amazonaws.com
ec2-54-172-46-184.compute-1.amazonaws.com
Formatting ephemeral HDFS namenode...
Warning: $HADOOP_HOME is deprecated.
16/01/18 08:12:39 INFO namenode.NameNode: STARTUP_MSG:
/************************************************************
STARTUP_MSG: Starting NameNode
STARTUP_MSG: host = ip-172-31-24-124.ec2.internal/172.31.24.124
STARTUP_MSG: args = [-format]
STARTUP_MSG: version = 1.0.4
STARTUP_MSG: build =
https://svn.apache.org/repos/asf/hadoop/common/branches/branch-1.0 -r
1393290; compiled by 'hortonfo' on Wed Oct 3 05:13:58 UTC 2012
************************************************************/
16/01/18 08:12:39 INFO util.GSet: VM type = 64-bit
16/01/18 08:12:39 INFO util.GSet: 2% max memory = 17.78 MB
16/01/18 08:12:39 INFO util.GSet: capacity = 2^21 = 2097152 entries
16/01/18 08:12:39 INFO util.GSet: recommended=2097152, actual=2097152
16/01/18 08:12:39 INFO namenode.FSNamesystem: fsOwner=root
16/01/18 08:12:39 INFO namenode.FSNamesystem: supergroup=supergroup
16/01/18 08:12:39 INFO namenode.FSNamesystem: isPermissionEnabled=false
16/01/18 08:12:39 INFO namenode.FSNamesystem: dfs.block.invalidate.limit=100
16/01/18 08:12:39 INFO namenode.FSNamesystem: isAccessTokenEnabled=false
accessKeyUpdateInterval=0 min(s), accessTokenLifetime=0 min(s)
16/01/18 08:12:39 INFO namenode.NameNode: Caching file names occuring more
than 10 times
16/01/18 08:12:39 INFO common.Storage: Image file of size 110 saved in 0
seconds.
16/01/18 08:12:39 INFO common.Storage: Storage directory
/mnt/ephemeral-hdfs/dfs/name has been successfully formatted.
16/01/18 08:12:39 INFO namenode.NameNode: SHUTDOWN_MSG:
/************************************************************
SHUTDOWN_MSG: Shutting down NameNode at ip-172-31-24-124.ec2.internal/
172.31.24.124
************************************************************/
Starting ephemeral HDFS...
Warning: $HADOOP_HOME is deprecated.
starting namenode, logging to
/mnt/ephemeral-hdfs/logs/hadoop-root-namenode-ip-172-31-24-124.ec2.internal
Error: Could not find or load main class crayondata.com.log
ec2-54-172-29-47.compute-1.amazonaws.com: Warning: $HADOOP_HOME is
deprecated.
ec2-54-172-29-47.compute-1.amazonaws.com:
ec2-54-172-29-47.compute-1.amazonaws.com: starting datanode, logging to
/mnt/ephemeral-hdfs/logs/hadoop-root-datanode-ip-172-31-29-54.ec2.internal.out
ec2-54-172-46-184.compute-1.amazonaws.com: Warning: $HADOOP_HOME is
deprecated.
ec2-54-172-46-184.compute-1.amazonaws.com:
ec2-54-172-46-184.compute-1.amazonaws.com: starting datanode, logging to
/mnt/ephemeral-hdfs/logs/hadoop-root-datanode-ip-172-31-30-167.ec2.internal.out
ec2-54-165-131-210.compute-1.amazonaws.com: Warning: $HADOOP_HOME is
deprecated.
ec2-54-165-131-210.compute-1.amazonaws.com:
ec2-54-165-131-210.compute-1.amazonaws.com: starting datanode, logging to
/mnt/ephemeral-hdfs/logs/hadoop-root-datanode-ip-172-31-23-10.ec2.internal.out
ec2-54-88-245-107.compute-1.amazonaws.com: Warning: $HADOOP_HOME is
deprecated.
ec2-54-88-245-107.compute-1.amazonaws.com:
ec2-54-88-245-107.compute-1.amazonaws.com: starting datanode, logging to
/mnt/ephemeral-hdfs/logs/hadoop-root-datanode-ip-172-31-30-81.ec2.internal.out
ec2-54-165-243-74.compute-1.amazonaws.com: Warning: $HADOOP_HOME is
deprecated.
ec2-54-165-243-74.compute-1.amazonaws.com:
ec2-54-165-243-74.compute-1.amazonaws.com: starting datanode, logging to
/mnt/ephemeral-hdfs/logs/hadoop-root-datanode-ip-172-31-19-61.ec2.internal.out
ec2-52-90-186-83.compute-1.amazonaws.com: Warning: Permanently added '
ec2-52-90-186-83.compute-1.amazonaws.com,172.31.24.124' (ECDSA) to the list
of known hosts.
ec2-52-90-186-83.compute-1.amazonaws.com: Warning: $HADOOP_HOME is
deprecated.
ec2-52-90-186-83.compute-1.amazonaws.com:
ec2-52-90-186-83.compute-1.amazonaws.com: starting secondarynamenode,
logging to
/mnt/ephemeral-hdfs/logs/hadoop-root-secondarynamenode-ip-172-31-24-124.ec2.internal.out
[timing] ephemeral-hdfs setup: 00h 00m 12s
Setting up persistent-hdfs
Pseudo-terminal will not be allocated because stdin is not a terminal.
Pseudo-terminal will not be allocated because stdin is not a terminal.
Pseudo-terminal will not be allocated because stdin is not a terminal.
Pseudo-terminal will not be allocated because stdin is not a terminal.
Pseudo-terminal will not be allocated because stdin is not a terminal.
RSYNC'ing /root/persistent-hdfs/conf to slaves...
ec2-54-165-243-74.compute-1.amazonaws.com
ec2-54-88-245-107.compute-1.amazonaws.com
ec2-54-172-29-47.compute-1.amazonaws.com
ec2-54-165-131-210.compute-1.amazonaws.com
ec2-54-172-46-184.compute-1.amazonaws.com
Formatting persistent HDFS namenode...
Warning: $HADOOP_HOME is deprecated.
16/01/18 08:12:50 INFO namenode.NameNode: STARTUP_MSG:
/************************************************************
STARTUP_MSG: Starting NameNode
STARTUP_MSG: host = ip-172-31-24-124.ec2.internal/172.31.24.124
STARTUP_MSG: args = [-format]
STARTUP_MSG: version = 1.0.4
STARTUP_MSG: build =
https://svn.apache.org/repos/asf/hadoop/common/branches/branch-1.0 -r
1393290; compiled by 'hortonfo' on Wed Oct 3 05:13:58 UTC 2012
************************************************************/
16/01/18 08:12:50 INFO util.GSet: VM type = 64-bit
16/01/18 08:12:50 INFO util.GSet: 2% max memory = 17.78 MB
16/01/18 08:12:50 INFO util.GSet: capacity = 2^21 = 2097152 entries
16/01/18 08:12:50 INFO util.GSet: recommended=2097152, actual=2097152
16/01/18 08:12:50 INFO namenode.FSNamesystem: fsOwner=root
16/01/18 08:12:50 INFO namenode.FSNamesystem: supergroup=supergroup
16/01/18 08:12:50 INFO namenode.FSNamesystem: isPermissionEnabled=false
16/01/18 08:12:50 INFO namenode.FSNamesystem: dfs.block.invalidate.limit=100
16/01/18 08:12:50 INFO namenode.FSNamesystem: isAccessTokenEnabled=false
accessKeyUpdateInterval=0 min(s), accessTokenLifetime=0 min(s)
16/01/18 08:12:50 INFO namenode.NameNode: Caching file names occuring more
than 10 times
16/01/18 08:12:50 INFO common.Storage: Image file of size 110 saved in 0
seconds.
16/01/18 08:12:50 INFO common.Storage: Storage directory
/vol/persistent-hdfs/dfs/name has been successfully formatted.
16/01/18 08:12:50 INFO namenode.NameNode: SHUTDOWN_MSG:
/************************************************************
SHUTDOWN_MSG: Shutting down NameNode at ip-172-31-24-124.ec2.internal/
172.31.24.124
************************************************************/
Persistent HDFS installed, won't start by default...
[timing] persistent-hdfs setup: 00h 00m 06s
Setting up spark-standalone
RSYNC'ing /root/spark/conf to slaves...
ec2-54-165-243-74.compute-1.amazonaws.com
ec2-54-88-245-107.compute-1.amazonaws.com
ec2-54-172-29-47.compute-1.amazonaws.com
ec2-54-165-131-210.compute-1.amazonaws.com
ec2-54-172-46-184.compute-1.amazonaws.com
RSYNC'ing /root/spark-ec2 to slaves...
ec2-54-165-243-74.compute-1.amazonaws.com
ec2-54-88-245-107.compute-1.amazonaws.com
ec2-54-172-29-47.compute-1.amazonaws.com
ec2-54-165-131-210.compute-1.amazonaws.com
ec2-54-172-46-184.compute-1.amazonaws.com
ec2-54-165-243-74.compute-1.amazonaws.com: no
org.apache.spark.deploy.worker.Worker to stop
ec2-54-88-245-107.compute-1.amazonaws.com: no
org.apache.spark.deploy.worker.Worker to stop
ec2-54-172-29-47.compute-1.amazonaws.com: no
org.apache.spark.deploy.worker.Worker to stop
ec2-54-165-131-210.compute-1.amazonaws.com: no
org.apache.spark.deploy.worker.Worker to stop
ec2-54-172-46-184.compute-1.amazonaws.com: no
org.apache.spark.deploy.worker.Worker to stop
no org.apache.spark.deploy.master.Master to stop
starting org.apache.spark.deploy.master.Master, logging to
/root/spark/logs/spark-root-org.apache.spark.deploy.master.Master-1-ip-172-31-24-124.ec2.internal
crayondata.com.out
ec2-54-88-245-107.compute-1.amazonaws.com: starting
org.apache.spark.deploy.worker.Worker, logging to
/root/spark/logs/spark-root-org.apache.spark.deploy.worker.Worker-1-ip-172-31-30-81.ec2.internal.out
ec2-54-165-243-74.compute-1.amazonaws.com: starting
org.apache.spark.deploy.worker.Worker, logging to
/root/spark/logs/spark-root-org.apache.spark.deploy.worker.Worker-1-ip-172-31-19-61.ec2.internal.out
ec2-54-172-46-184.compute-1.amazonaws.com: starting
org.apache.spark.deploy.worker.Worker, logging to
/root/spark/logs/spark-root-org.apache.spark.deploy.worker.Worker-1-ip-172-31-30-167.ec2.internal.out
ec2-54-165-131-210.compute-1.amazonaws.com: starting
org.apache.spark.deploy.worker.Worker, logging to
/root/spark/logs/spark-root-org.apache.spark.deploy.worker.Worker-1-ip-172-31-23-10.ec2.internal.out
ec2-54-172-29-47.compute-1.amazonaws.com: starting
org.apache.spark.deploy.worker.Worker, logging to
/root/spark/logs/spark-root-org.apache.spark.deploy.worker.Worker-1-ip-172-31-29-54.ec2.internal.out
[timing] spark-standalone setup: 00h 00m 39s
Setting up tachyon
RSYNC'ing /root/tachyon to slaves...
ec2-54-165-243-74.compute-1.amazonaws.com
ec2-54-88-245-107.compute-1.amazonaws.com
ec2-54-172-29-47.compute-1.amazonaws.com
ec2-54-165-131-210.compute-1.amazonaws.com
ec2-54-172-46-184.compute-1.amazonaws.com
./tachyon/setup.sh: line 5: /root/tachyon/bin/tachyon: No such file or
directory
./tachyon/setup.sh: line 9: /root/tachyon/bin/tachyon-start.sh: No such
file or directory
[timing] tachyon setup: 00h 00m 04s
Setting up rstudio
spark-ec2/setup.sh: line 110: ./rstudio/setup.sh: No such file or directory
[timing] rstudio setup: 00h 00m 00s
Setting up ganglia
RSYNC'ing /etc/ganglia to slaves...
ec2-54-165-243-74.compute-1.amazonaws.com
ec2-54-88-245-107.compute-1.amazonaws.com
ec2-54-172-29-47.compute-1.amazonaws.com
ec2-54-165-131-210.compute-1.amazonaws.com
ec2-54-172-46-184.compute-1.amazonaws.com
Shutting down GANGLIA gmond: [FAILED]
Starting GANGLIA gmond: [ OK ]
Shutting down GANGLIA gmond: [FAILED]
Starting GANGLIA gmond: [ OK ]
Connection to ec2-54-165-243-74.compute-1.amazonaws.com closed.
Shutting down GANGLIA gmond: [FAILED]
Starting GANGLIA gmond: [ OK ]
Connection to ec2-54-88-245-107.compute-1.amazonaws.com closed.
Shutting down GANGLIA gmond: [FAILED]
Starting GANGLIA gmond: [ OK ]
Connection to ec2-54-172-29-47.compute-1.amazonaws.com closed.
Shutting down GANGLIA gmond: [FAILED]
Starting GANGLIA gmond: [ OK ]
Connection to ec2-54-165-131-210.compute-1.amazonaws.com closed.
Shutting down GANGLIA gmond: [FAILED]
Starting GANGLIA gmond: [ OK ]
Connection to ec2-54-172-46-184.compute-1.amazonaws.com closed.
Shutting down GANGLIA gmetad: [FAILED]
Starting GANGLIA gmetad: [ OK ]
Stopping httpd: [FAILED]
Starting httpd: httpd: Syntax error on line 154 of
/etc/httpd/conf/httpd.conf: Cannot load
/etc/httpd/modules/mod_authz_core.so into server:
/etc/httpd/modules/mod_authz_core.so: cannot open shared object file: No
such file or directory
[FAILED]
[timing] ganglia setup: 00h 00m 04s
Connection to ec2-52-90-186-83.compute-1.amazonaws.com closed.
Spark standalone cluster started at
http://ec2-52-90-186-83.compute-1.amazonaws.com:8080
Ganglia started at
http://ec2-52-90-186-83.compute-1.amazonaws.com:5080/ganglia
Done!
RE: spark 1.6.0 on ec2 doesn't work
Posted by vi...@wipro.com.
Have you verified the spark master/slaves are started correctly? Please check using netstat command and open ports mode. Are they listening? Binds to which address etc..
From: Oleg Ruchovets [mailto:oruchovets@gmail.com]
Sent: 19 January 2016 11:24
To: Peter Zhang <zh...@gmail.com>
Cc: Daniel Darabos <da...@lynxanalytics.com>; user <us...@spark.apache.org>
Subject: Re: spark 1.6.0 on ec2 doesn't work
I am running from $SPARK_HOME.
It looks like connection problem to port 9000. It is on master machine.
What is this process is spark tries to connect?
Should I start any framework , processes before executing spark?
Thanks
OIeg.
16/01/19 03:17:56 INFO ipc.Client: Retrying connect to server: ec2-54-88-242-197.compute-1.amazonaws.com/172.31.28.196:9000<http://ec2-54-88-242-197.compute-1.amazonaws.com/172.31.28.196:9000>. Already tried 1 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=10, sleepTime=1 SECONDS)
16/01/19 03:17:57 INFO ipc.Client: Retrying connect to server: ec2-54-88-242-197.compute-1.amazonaws.com/172.31.28.196:9000<http://ec2-54-88-242-197.compute-1.amazonaws.com/172.31.28.196:9000>. Already tried 2 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=10, sleepTime=1 SECONDS)
16/01/19 03:17:58 INFO ipc.Client: Retrying connect to server: ec2-54-88-242-197.compute-1.amazonaws.com/172.31.28.196:9000<http://ec2-54-88-242-197.compute-1.amazonaws.com/172.31.28.196:9000>. Already tried 3 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=10, sleepTime=1 SECONDS)
16/01/19 03:17:59 INFO ipc.Client: Retrying connect to server: ec2-54-88-242-197.compute-1.amazonaws.com/172.31.28.196:9000<http://ec2-54-88-242-197.compute-1.amazonaws.com/172.31.28.196:9000>. Already tried 4 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=10, sleepTime=1 SECONDS)
16/01/19 03:18:00 INFO ipc.Client: Retrying connect to server: ec2-54-88-242-197.compute-1.amazonaws.com/172.31.28.196:9000<http://ec2-54-88-242-197.compute-1.amazonaws.com/172.31.28.196:9000>. Already tried 5 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=10, sleepTime=1 SECONDS)
16/01/19 03:18:01 INFO ipc.Client: Retrying connect to server: ec2-54-88-242-197.compute-1.amazonaws.com/172.31.28.196:9000<http://ec2-54-88-242-197.compute-1.amazonaws.com/172.31.28.196:9000>. Already tried 6 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=10, sleepTime=1 SECONDS)
16/01/19 03:18:02 INFO ipc.Client: Retrying connect to server: ec2-54-88-242-197.compute-1.amazonaws.com/172.31.28.196:9000<http://ec2-54-88-242-197.compute-1.amazonaws.com/172.31.28.196:9000>. Already tried 7 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=10, sleepTime=1 SECONDS)
16/01/19 03:18:03 INFO ipc.Client: Retrying connect to server: ec2-54-88-242-197.compute-1.amazonaws.com/172.31.28.196:9000<http://ec2-54-88-242-197.compute-1.amazonaws.com/172.31.28.196:9000>. Already tried 8 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=10, sleepTime=1 SECONDS)
16/01/19 03:18:04 INFO ipc.Client: Retrying connect to server: ec2-54-88-242-197.compute-1.amazonaws.com/172.31.28.196:9000<http://ec2-54-88-242-197.compute-1.amazonaws.com/172.31.28.196:9000>. Already tried 9 time(s); retry
On Tue, Jan 19, 2016 at 1:13 PM, Peter Zhang <zh...@gmail.com>> wrote:
Could you run spark-shell at $SPARK_HOME DIR?
You can try to change you command run at $SPARK_HOME or, point to README.md with full path.
Peter Zhang
--
Google
Sent with Airmail
On January 19, 2016 at 11:26:14, Oleg Ruchovets (oruchovets@gmail.com<ma...@gmail.com>) wrote:
It looks spark is not working fine :
I followed this link ( http://spark.apache.org/docs/latest/ec2-scripts.html. ) and I see spot instances installed on EC2.
from spark shell I am counting lines and got connection exception.
scala> val lines = sc.textFile("README.md")
scala> lines.count()
scala> val lines = sc.textFile("README.md")
16/01/19 03:17:35 INFO storage.MemoryStore: Block broadcast_0 stored as values in memory (estimated size 26.5 KB, free 26.5 KB)
16/01/19 03:17:35 INFO storage.MemoryStore: Block broadcast_0_piece0 stored as bytes in memory (estimated size 5.6 KB, free 32.1 KB)
16/01/19 03:17:35 INFO storage.BlockManagerInfo: Added broadcast_0_piece0 in memory on 172.31.28.196:44028<http://172.31.28.196:44028> (size: 5.6 KB, free: 511.5 MB)
16/01/19 03:17:35 INFO spark.SparkContext: Created broadcast 0 from textFile at <console>:21
lines: org.apache.spark.rdd.RDD[String] = MapPartitionsRDD[1] at textFile at <console>:21
scala> lines.count()
16/01/19 03:17:55 INFO ipc.Client: Retrying connect to server: ec2-54-88-242-197.compute-1.amazonaws.com/172.31.28.196:9000<http://ec2-54-88-242-197.compute-1.amazonaws.com/172.31.28.196:9000>. Already tried 0 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=10, sleepTime=1 SECONDS)
16/01/19 03:17:56 INFO ipc.Client: Retrying connect to server: ec2-54-88-242-197.compute-1.amazonaws.com/172.31.28.196:9000<http://ec2-54-88-242-197.compute-1.amazonaws.com/172.31.28.196:9000>. Already tried 1 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=10, sleepTime=1 SECONDS)
16/01/19 03:17:57 INFO ipc.Client: Retrying connect to server: ec2-54-88-242-197.compute-1.amazonaws.com/172.31.28.196:9000<http://ec2-54-88-242-197.compute-1.amazonaws.com/172.31.28.196:9000>. Already tried 2 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=10, sleepTime=1 SECONDS)
16/01/19 03:17:58 INFO ipc.Client: Retrying connect to server: ec2-54-88-242-197.compute-1.amazonaws.com/172.31.28.196:9000<http://ec2-54-88-242-197.compute-1.amazonaws.com/172.31.28.196:9000>. Already tried 3 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=10, sleepTime=1 SECONDS)
16/01/19 03:17:59 INFO ipc.Client: Retrying connect to server: ec2-54-88-242-197.compute-1.amazonaws.com/172.31.28.196:9000<http://ec2-54-88-242-197.compute-1.amazonaws.com/172.31.28.196:9000>. Already tried 4 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=10, sleepTime=1 SECONDS)
16/01/19 03:18:00 INFO ipc.Client: Retrying connect to server: ec2-54-88-242-197.compute-1.amazonaws.com/172.31.28.196:9000<http://ec2-54-88-242-197.compute-1.amazonaws.com/172.31.28.196:9000>. Already tried 5 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=10, sleepTime=1 SECONDS)
16/01/19 03:18:01 INFO ipc.Client: Retrying connect to server: ec2-54-88-242-197.compute-1.amazonaws.com/172.31.28.196:9000<http://ec2-54-88-242-197.compute-1.amazonaws.com/172.31.28.196:9000>. Already tried 6 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=10, sleepTime=1 SECONDS)
16/01/19 03:18:02 INFO ipc.Client: Retrying connect to server: ec2-54-88-242-197.compute-1.amazonaws.com/172.31.28.196:9000<http://ec2-54-88-242-197.compute-1.amazonaws.com/172.31.28.196:9000>. Already tried 7 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=10, sleepTime=1 SECONDS)
16/01/19 03:18:03 INFO ipc.Client: Retrying connect to server: ec2-54-88-242-197.compute-1.amazonaws.com/172.31.28.196:9000<http://ec2-54-88-242-197.compute-1.amazonaws.com/172.31.28.196:9000>. Already tried 8 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=10, sleepTime=1 SECONDS)
16/01/19 03:18:04 INFO ipc.Client: Retrying connect to server: ec2-54-88-242-197.compute-1.amazonaws.com/172.31.28.196:9000<http://ec2-54-88-242-197.compute-1.amazonaws.com/172.31.28.196:9000>. Already tried 9 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=10, sleepTime=1 SECONDS)
java.lang.RuntimeException: java.net.ConnectException: Call to ec2-54-88-242-197.compute-1.amazonaws.com/172.31.28.196:9000<http://ec2-54-88-242-197.compute-1.amazonaws.com/172.31.28.196:9000> failed on connection exception: java.net.ConnectException: Connection refused
at org.apache.hadoop.mapred.JobConf.getWorkingDirectory(JobConf.java:567)
at org.apache.hadoop.mapred.FileInputFormat.setInputPaths(FileInputFormat.java:318)
at org.apache.hadoop.mapred.FileInputFormat.setInputPaths(FileInputFormat.java:291)
at org.apache.spark.SparkContext$$anonfun$hadoopFile$1$$anonfun$33.apply(SparkContext.scala:1015)
at org.apache.spark.SparkContext$$anonfun$hadoopFile$1$$anonfun$33.apply(SparkContext.scala:1015)
at org.apache.spark.rdd.HadoopRDD$$anonfun$getJobConf$6.apply(HadoopRDD.scala:176)
at org.apache.spark.rdd.HadoopRDD$$anonfun$getJobConf$6.apply(HadoopRDD.scala:176)
at scala.Option.map(Option.scala:145)
at org.apache.spark.rdd.HadoopRDD.getJobConf(HadoopRDD.scala:176)
at org.apache.spark.rdd.HadoopRDD.getPartitions(HadoopRDD.scala:195)
at org.apache.spark.rdd.RDD$$anonfun$partitions$2.apply(RDD.scala:239)
at org.apache.spark.rdd.RDD$$anonfun$partitions$2.apply(RDD.scala:237)
at scala.Option.getOrElse(Option.scala:120)
at org.apache.spark.rdd.RDD.partitions(RDD.scala:237)
at org.apache.spark.rdd.MapPartitionsRDD.getPartitions(MapPartitionsRDD.scala:35)
at org.apache.spark.rdd.RDD$$anonfun$partitions$2.apply(RDD.scala:239)
at org.apache.spark.rdd.RDD$$anonfun$partitions$2.apply(RDD.scala:237)
at scala.Option.getOrElse(Option.scala:120)
at org.apache.spark.rdd.RDD.partitions(RDD.scala:237)
at org.apache.spark.SparkContext.runJob(SparkContext.scala:1929)
at org.apache.spark.rdd.RDD.count(RDD.scala:1143)
at $iwC$$iwC$$iwC$$iwC$$iwC$$iwC.<init>(<console>:24)
at $iwC$$iwC$$iwC$$iwC$$iwC.<init>(<console>:29)
at $iwC$$iwC$$iwC$$iwC.<init>(<console>:31)
at $iwC$$iwC$$iwC.<init>(<console>:33)
at $iwC$$iwC.<init>(<console>:35)
at $iwC.<init>(<console>:37)
at <init>(<console>:39)
at .<init>(<console>:43)
at .<clinit>(<console>)
at .<init>(<console>:7)
at .<clinit>(<console>)
at $print(<console>)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at org.apache.spark.repl.SparkIMain$ReadEvalPrint.call(SparkIMain.scala:1065)
at org.apache.spark.repl.SparkIMain$Request.loadAndRun(SparkIMain.scala:1346)
at org.apache.spark.repl.SparkIMain.loadAndRunReq$1(SparkIMain.scala:840)
at org.apache.spark.repl.SparkIMain.interpret(SparkIMain.scala:871)
at org.apache.spark.repl.SparkIMain.interpret(SparkIMain.scala:819)
at org.apache.spark.repl.SparkILoop.reallyInterpret$1(SparkILoop.scala:857)
at org.apache.spark.repl.SparkILoop.interpretStartingWith(SparkILoop.scala:902)
at org.apache.spark.repl.SparkILoop.command(SparkILoop.scala:814)
at org.apache.spark.repl.SparkILoop.processLine$1(SparkILoop.scala:657)
at org.apache.spark.repl.SparkILoop.innerLoop$1(SparkILoop.scala:665)
at org.apache.spark.repl.SparkILoop.org<http://org.apache.spark.repl.SparkILoop.org>$apache$spark$repl$SparkILoop$$loop(SparkILoop.scala:670)
at org.apache.spark.repl.SparkILoop$$anonfun$org$apache$spark$repl$SparkILoop$$process$1.apply$mcZ$sp(SparkILoop.scala:997)
at org.apache.spark.repl.SparkILoop$$anonfun$org$apache$spark$repl$SparkILoop$$process$1.apply(SparkILoop.scala:945)
at org.apache.spark.repl.SparkILoop$$anonfun$org$apache$spark$repl$SparkILoop$$process$1.apply(SparkILoop.scala:945)
at scala.tools.nsc.util.ScalaClassLoader$.savingContextLoader(ScalaClassLoader.scala:135)
at org.apache.spark.repl.SparkILoop.org<http://org.apache.spark.repl.SparkILoop.org>$apache$spark$repl$SparkILoop$$process(SparkILoop.scala:945)
at org.apache.spark.repl.SparkILoop.process(SparkILoop.scala:1059)
at org.apache.spark.repl.Main$.main(Main.scala:31)
at org.apache.spark.repl.Main.main(Main.scala)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at org.apache.spark.deploy.SparkSubmit$.org$apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:731)
at org.apache.spark.deploy.SparkSubmit$.doRunMain$1(SparkSubmit.scala:181)
at org.apache.spark.deploy.SparkSubmit$.submit(SparkSubmit.scala:206)
at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:121)
at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)
Caused by: java.net.ConnectException: Call to ec2-54-88-242-197.compute-1.amazonaws.com/172.31.28.196:9000<http://ec2-54-88-242-197.compute-1.amazonaws.com/172.31.28.196:9000> failed on connection exception: java.net.ConnectException: Connection refused
at org.apache.hadoop.ipc.Client.wrapException(Client.java:1142)
at org.apache.hadoop.ipc.Client.call(Client.java:1118)
at org.apache.hadoop.ipc.RPC$Invoker.invoke(RPC.java:229)
at com.sun.proxy.$Proxy15.getProtocolVersion(Unknown Source)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:85)
at org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:62)
at com.sun.proxy.$Proxy15.getProtocolVersion(Unknown Source)
at org.apache.hadoop.ipc.RPC.checkVersion(RPC.java:422)
at org.apache.hadoop.hdfs.DFSClient.createNamenode(DFSClient.java:183)
at org.apache.hadoop.hdfs.DFSClient.<init>(DFSClient.java:281)
at org.apache.hadoop.hdfs.DFSClient.<init>(DFSClient.java:245)
at org.apache.hadoop.hdfs.DistributedFileSystem.initialize(DistributedFileSystem.java:100)
at org.apache.hadoop.fs.FileSystem.createFileSystem(FileSystem.java:1446)
at org.apache.hadoop.fs.FileSystem.access$200(FileSystem.java:67)
at org.apache.hadoop.fs.FileSystem$Cache.get(FileSystem.java:1464)
at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:263)
at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:124)
at org.apache.hadoop.mapred.JobConf.getWorkingDirectory(JobConf.java:563)
... 64 more
Caused by: java.net.ConnectException: Connection refused
at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:744)
at org.apache.hadoop.net.SocketIOWithTimeout.connect(SocketIOWithTimeout.java:206)
at org.apache.hadoop.net.NetUtils.connect(NetUtils.java:511)
at org.apache.hadoop.net.NetUtils.connect(NetUtils.java:481)
at org.apache.hadoop.ipc.Client$Connection.setupConnection(Client.java:457)
at org.apache.hadoop.ipc.Client$Connection.setupIOstreams(Client.java:583)
at org.apache.hadoop.ipc.Client$Connection.access$2200(Client.java:205)
at org.apache.hadoop.ipc.Client.getConnection(Client.java:1249)
at org.apache.hadoop.ipc.Client.call(Client.java:1093)
... 84 more
scala>
On Tue, Jan 19, 2016 at 1:22 AM, Daniel Darabos <da...@lynxanalytics.com>> wrote:
On Mon, Jan 18, 2016 at 5:24 PM, Oleg Ruchovets <or...@gmail.com>> wrote:
I thought script tries to install hadoop / hdfs also. And it looks like it failed. Installation is only standalone spark without hadoop. Is it correct behaviour?
Yes, it also sets up two HDFS clusters. Are they not working? Try to see if Spark is working by running some simple jobs on it. (See http://spark.apache.org/docs/latest/ec2-scripts.html.)
There is no program called Hadoop. If you mean YARN, then indeed the script does not set up YARN. It sets up standalone Spark.
Also errors in the log:
ERROR: Unknown Tachyon version
Error: Could not find or load main class crayondata.com.log
As long as Spark is working fine, you can ignore all output from the EC2 script :).
The information contained in this electronic message and any attachments to this message are intended for the exclusive use of the addressee(s) and may contain proprietary, confidential or privileged information. If you are not the intended recipient, you should not disseminate, distribute or copy this e-mail. Please notify the sender immediately and destroy all copies of this message and any attachments. WARNING: Computer viruses can be transmitted via email. The recipient should check this email and any attachments for the presence of viruses. The company accepts no liability for any damage caused by any virus transmitted by this email. www.wipro.com
Re: spark 1.6.0 on ec2 doesn't work
Posted by Oleg Ruchovets <or...@gmail.com>.
I am running from $SPARK_HOME.
It looks like connection problem to port 9000. It is on master machine.
What is this process is spark tries to connect?
Should I start any framework , processes before executing spark?
Thanks
OIeg.
16/01/19 03:17:56 INFO ipc.Client: Retrying connect to server:
ec2-54-88-242-197.compute-1.amazonaws.com/172.31.28.196:9000. Already tried
1 time(s); retry policy is
RetryUpToMaximumCountWithFixedSleep(maxRetries=10, sleepTime=1 SECONDS)
16/01/19 03:17:57 INFO ipc.Client: Retrying connect to server:
ec2-54-88-242-197.compute-1.amazonaws.com/172.31.28.196:9000. Already tried
2 time(s); retry policy is
RetryUpToMaximumCountWithFixedSleep(maxRetries=10, sleepTime=1 SECONDS)
16/01/19 03:17:58 INFO ipc.Client: Retrying connect to server:
ec2-54-88-242-197.compute-1.amazonaws.com/172.31.28.196:9000. Already tried
3 time(s); retry policy is
RetryUpToMaximumCountWithFixedSleep(maxRetries=10, sleepTime=1 SECONDS)
16/01/19 03:17:59 INFO ipc.Client: Retrying connect to server:
ec2-54-88-242-197.compute-1.amazonaws.com/172.31.28.196:9000. Already tried
4 time(s); retry policy is
RetryUpToMaximumCountWithFixedSleep(maxRetries=10, sleepTime=1 SECONDS)
16/01/19 03:18:00 INFO ipc.Client: Retrying connect to server:
ec2-54-88-242-197.compute-1.amazonaws.com/172.31.28.196:9000. Already tried
5 time(s); retry policy is
RetryUpToMaximumCountWithFixedSleep(maxRetries=10, sleepTime=1 SECONDS)
16/01/19 03:18:01 INFO ipc.Client: Retrying connect to server:
ec2-54-88-242-197.compute-1.amazonaws.com/172.31.28.196:9000. Already tried
6 time(s); retry policy is
RetryUpToMaximumCountWithFixedSleep(maxRetries=10, sleepTime=1 SECONDS)
16/01/19 03:18:02 INFO ipc.Client: Retrying connect to server:
ec2-54-88-242-197.compute-1.amazonaws.com/172.31.28.196:9000. Already tried
7 time(s); retry policy is
RetryUpToMaximumCountWithFixedSleep(maxRetries=10, sleepTime=1 SECONDS)
16/01/19 03:18:03 INFO ipc.Client: Retrying connect to server:
ec2-54-88-242-197.compute-1.amazonaws.com/172.31.28.196:9000. Already tried
8 time(s); retry policy is
RetryUpToMaximumCountWithFixedSleep(maxRetries=10, sleepTime=1 SECONDS)
16/01/19 03:18:04 INFO ipc.Client: Retrying connect to server:
ec2-54-88-242-197.compute-1.amazonaws.com/172.31.28.196:9000. Already tried
9 time(s); retry
On Tue, Jan 19, 2016 at 1:13 PM, Peter Zhang <zh...@gmail.com> wrote:
> Could you run spark-shell at $SPARK_HOME DIR?
>
> You can try to change you command run at $SPARK_HOME or, point to
> README.md with full path.
>
>
> Peter Zhang
> --
> Google
> Sent with Airmail
>
> On January 19, 2016 at 11:26:14, Oleg Ruchovets (oruchovets@gmail.com)
> wrote:
>
> It looks spark is not working fine :
>
> I followed this link (
> http://spark.apache.org/docs/latest/ec2-scripts.html. ) and I see spot
> instances installed on EC2.
>
> from spark shell I am counting lines and got connection exception.
> *scala> val lines = sc.textFile("README.md")*
> *scala> lines.count()*
>
>
>
> *scala> val lines = sc.textFile("README.md")*
>
> 16/01/19 03:17:35 INFO storage.MemoryStore: Block broadcast_0 stored as
> values in memory (estimated size 26.5 KB, free 26.5 KB)
> 16/01/19 03:17:35 INFO storage.MemoryStore: Block broadcast_0_piece0
> stored as bytes in memory (estimated size 5.6 KB, free 32.1 KB)
> 16/01/19 03:17:35 INFO storage.BlockManagerInfo: Added broadcast_0_piece0
> in memory on 172.31.28.196:44028 (size: 5.6 KB, free: 511.5 MB)
> 16/01/19 03:17:35 INFO spark.SparkContext: Created broadcast 0 from
> textFile at <console>:21
> lines: org.apache.spark.rdd.RDD[String] = MapPartitionsRDD[1] at textFile
> at <console>:21
>
> *scala> lines.count()*
>
> 16/01/19 03:17:55 INFO ipc.Client: Retrying connect to server:
> ec2-54-88-242-197.compute-1.amazonaws.com/172.31.28.196:9000. Already
> tried 0 time(s); retry policy is
> RetryUpToMaximumCountWithFixedSleep(maxRetries=10, sleepTime=1 SECONDS)
> 16/01/19 03:17:56 INFO ipc.Client: Retrying connect to server:
> ec2-54-88-242-197.compute-1.amazonaws.com/172.31.28.196:9000. Already
> tried 1 time(s); retry policy is
> RetryUpToMaximumCountWithFixedSleep(maxRetries=10, sleepTime=1 SECONDS)
> 16/01/19 03:17:57 INFO ipc.Client: Retrying connect to server:
> ec2-54-88-242-197.compute-1.amazonaws.com/172.31.28.196:9000. Already
> tried 2 time(s); retry policy is
> RetryUpToMaximumCountWithFixedSleep(maxRetries=10, sleepTime=1 SECONDS)
> 16/01/19 03:17:58 INFO ipc.Client: Retrying connect to server:
> ec2-54-88-242-197.compute-1.amazonaws.com/172.31.28.196:9000. Already
> tried 3 time(s); retry policy is
> RetryUpToMaximumCountWithFixedSleep(maxRetries=10, sleepTime=1 SECONDS)
> 16/01/19 03:17:59 INFO ipc.Client: Retrying connect to server:
> ec2-54-88-242-197.compute-1.amazonaws.com/172.31.28.196:9000. Already
> tried 4 time(s); retry policy is
> RetryUpToMaximumCountWithFixedSleep(maxRetries=10, sleepTime=1 SECONDS)
> 16/01/19 03:18:00 INFO ipc.Client: Retrying connect to server:
> ec2-54-88-242-197.compute-1.amazonaws.com/172.31.28.196:9000. Already
> tried 5 time(s); retry policy is
> RetryUpToMaximumCountWithFixedSleep(maxRetries=10, sleepTime=1 SECONDS)
> 16/01/19 03:18:01 INFO ipc.Client: Retrying connect to server:
> ec2-54-88-242-197.compute-1.amazonaws.com/172.31.28.196:9000. Already
> tried 6 time(s); retry policy is
> RetryUpToMaximumCountWithFixedSleep(maxRetries=10, sleepTime=1 SECONDS)
> 16/01/19 03:18:02 INFO ipc.Client: Retrying connect to server:
> ec2-54-88-242-197.compute-1.amazonaws.com/172.31.28.196:9000. Already
> tried 7 time(s); retry policy is
> RetryUpToMaximumCountWithFixedSleep(maxRetries=10, sleepTime=1 SECONDS)
> 16/01/19 03:18:03 INFO ipc.Client: Retrying connect to server:
> ec2-54-88-242-197.compute-1.amazonaws.com/172.31.28.196:9000. Already
> tried 8 time(s); retry policy is
> RetryUpToMaximumCountWithFixedSleep(maxRetries=10, sleepTime=1 SECONDS)
> 16/01/19 03:18:04 INFO ipc.Client: Retrying connect to server:
> ec2-54-88-242-197.compute-1.amazonaws.com/172.31.28.196:9000. Already
> tried 9 time(s); retry policy is
> RetryUpToMaximumCountWithFixedSleep(maxRetries=10, sleepTime=1 SECONDS)
> java.lang.RuntimeException: java.net.ConnectException: Call to
> ec2-54-88-242-197.compute-1.amazonaws.com/172.31.28.196:9000 failed on
> connection exception: java.net.ConnectException: Connection refused
> at org.apache.hadoop.mapred.JobConf.getWorkingDirectory(JobConf.java:567)
> at
> org.apache.hadoop.mapred.FileInputFormat.setInputPaths(FileInputFormat.java:318)
> at
> org.apache.hadoop.mapred.FileInputFormat.setInputPaths(FileInputFormat.java:291)
> at
> org.apache.spark.SparkContext$$anonfun$hadoopFile$1$$anonfun$33.apply(SparkContext.scala:1015)
> at
> org.apache.spark.SparkContext$$anonfun$hadoopFile$1$$anonfun$33.apply(SparkContext.scala:1015)
> at
> org.apache.spark.rdd.HadoopRDD$$anonfun$getJobConf$6.apply(HadoopRDD.scala:176)
> at
> org.apache.spark.rdd.HadoopRDD$$anonfun$getJobConf$6.apply(HadoopRDD.scala:176)
> at scala.Option.map(Option.scala:145)
> at org.apache.spark.rdd.HadoopRDD.getJobConf(HadoopRDD.scala:176)
> at org.apache.spark.rdd.HadoopRDD.getPartitions(HadoopRDD.scala:195)
> at org.apache.spark.rdd.RDD$$anonfun$partitions$2.apply(RDD.scala:239)
> at org.apache.spark.rdd.RDD$$anonfun$partitions$2.apply(RDD.scala:237)
> at scala.Option.getOrElse(Option.scala:120)
> at org.apache.spark.rdd.RDD.partitions(RDD.scala:237)
> at
> org.apache.spark.rdd.MapPartitionsRDD.getPartitions(MapPartitionsRDD.scala:35)
> at org.apache.spark.rdd.RDD$$anonfun$partitions$2.apply(RDD.scala:239)
> at org.apache.spark.rdd.RDD$$anonfun$partitions$2.apply(RDD.scala:237)
> at scala.Option.getOrElse(Option.scala:120)
> at org.apache.spark.rdd.RDD.partitions(RDD.scala:237)
> at org.apache.spark.SparkContext.runJob(SparkContext.scala:1929)
> at org.apache.spark.rdd.RDD.count(RDD.scala:1143)
> at $iwC$$iwC$$iwC$$iwC$$iwC$$iwC.<init>(<console>:24)
> at $iwC$$iwC$$iwC$$iwC$$iwC.<init>(<console>:29)
> at $iwC$$iwC$$iwC$$iwC.<init>(<console>:31)
> at $iwC$$iwC$$iwC.<init>(<console>:33)
> at $iwC$$iwC.<init>(<console>:35)
> at $iwC.<init>(<console>:37)
> at <init>(<console>:39)
> at .<init>(<console>:43)
> at .<clinit>(<console>)
> at .<init>(<console>:7)
> at .<clinit>(<console>)
> at $print(<console>)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:606)
> at
> org.apache.spark.repl.SparkIMain$ReadEvalPrint.call(SparkIMain.scala:1065)
> at
> org.apache.spark.repl.SparkIMain$Request.loadAndRun(SparkIMain.scala:1346)
> at org.apache.spark.repl.SparkIMain.loadAndRunReq$1(SparkIMain.scala:840)
> at org.apache.spark.repl.SparkIMain.interpret(SparkIMain.scala:871)
> at org.apache.spark.repl.SparkIMain.interpret(SparkIMain.scala:819)
> at org.apache.spark.repl.SparkILoop.reallyInterpret$1(SparkILoop.scala:857)
> at
> org.apache.spark.repl.SparkILoop.interpretStartingWith(SparkILoop.scala:902)
> at org.apache.spark.repl.SparkILoop.command(SparkILoop.scala:814)
> at org.apache.spark.repl.SparkILoop.processLine$1(SparkILoop.scala:657)
> at org.apache.spark.repl.SparkILoop.innerLoop$1(SparkILoop.scala:665)
> at org.apache.spark.repl.SparkILoop.org
> $apache$spark$repl$SparkILoop$$loop(SparkILoop.scala:670)
> at
> org.apache.spark.repl.SparkILoop$$anonfun$org$apache$spark$repl$SparkILoop$$process$1.apply$mcZ$sp(SparkILoop.scala:997)
> at
> org.apache.spark.repl.SparkILoop$$anonfun$org$apache$spark$repl$SparkILoop$$process$1.apply(SparkILoop.scala:945)
> at
> org.apache.spark.repl.SparkILoop$$anonfun$org$apache$spark$repl$SparkILoop$$process$1.apply(SparkILoop.scala:945)
> at
> scala.tools.nsc.util.ScalaClassLoader$.savingContextLoader(ScalaClassLoader.scala:135)
> at org.apache.spark.repl.SparkILoop.org
> $apache$spark$repl$SparkILoop$$process(SparkILoop.scala:945)
> at org.apache.spark.repl.SparkILoop.process(SparkILoop.scala:1059)
> at org.apache.spark.repl.Main$.main(Main.scala:31)
> at org.apache.spark.repl.Main.main(Main.scala)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:606)
> at
> org.apache.spark.deploy.SparkSubmit$.org$apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:731)
> at org.apache.spark.deploy.SparkSubmit$.doRunMain$1(SparkSubmit.scala:181)
> at org.apache.spark.deploy.SparkSubmit$.submit(SparkSubmit.scala:206)
> at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:121)
> at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)
> Caused by: java.net.ConnectException: Call to
> ec2-54-88-242-197.compute-1.amazonaws.com/172.31.28.196:9000 failed on
> connection exception: java.net.ConnectException: Connection refused
> at org.apache.hadoop.ipc.Client.wrapException(Client.java:1142)
> at org.apache.hadoop.ipc.Client.call(Client.java:1118)
> at org.apache.hadoop.ipc.RPC$Invoker.invoke(RPC.java:229)
> at com.sun.proxy.$Proxy15.getProtocolVersion(Unknown Source)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:606)
> at
> org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:85)
> at
> org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:62)
> at com.sun.proxy.$Proxy15.getProtocolVersion(Unknown Source)
> at org.apache.hadoop.ipc.RPC.checkVersion(RPC.java:422)
> at org.apache.hadoop.hdfs.DFSClient.createNamenode(DFSClient.java:183)
> at org.apache.hadoop.hdfs.DFSClient.<init>(DFSClient.java:281)
> at org.apache.hadoop.hdfs.DFSClient.<init>(DFSClient.java:245)
> at
> org.apache.hadoop.hdfs.DistributedFileSystem.initialize(DistributedFileSystem.java:100)
> at org.apache.hadoop.fs.FileSystem.createFileSystem(FileSystem.java:1446)
> at org.apache.hadoop.fs.FileSystem.access$200(FileSystem.java:67)
> at org.apache.hadoop.fs.FileSystem$Cache.get(FileSystem.java:1464)
> at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:263)
> at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:124)
> at org.apache.hadoop.mapred.JobConf.getWorkingDirectory(JobConf.java:563)
> ... 64 more
> Caused by: java.net.ConnectException: Connection refused
> at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
> at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:744)
> at
> org.apache.hadoop.net.SocketIOWithTimeout.connect(SocketIOWithTimeout.java:206)
> at org.apache.hadoop.net.NetUtils.connect(NetUtils.java:511)
> at org.apache.hadoop.net.NetUtils.connect(NetUtils.java:481)
> at org.apache.hadoop.ipc.Client$Connection.setupConnection(Client.java:457)
> at org.apache.hadoop.ipc.Client$Connection.setupIOstreams(Client.java:583)
> at org.apache.hadoop.ipc.Client$Connection.access$2200(Client.java:205)
> at org.apache.hadoop.ipc.Client.getConnection(Client.java:1249)
> at org.apache.hadoop.ipc.Client.call(Client.java:1093)
> ... 84 more
>
>
> scala>
>
>
>
>
> On Tue, Jan 19, 2016 at 1:22 AM, Daniel Darabos <
> daniel.darabos@lynxanalytics.com> wrote:
>
>>
>> On Mon, Jan 18, 2016 at 5:24 PM, Oleg Ruchovets <or...@gmail.com>
>> wrote:
>>
>>> I thought script tries to install hadoop / hdfs also. And it looks like
>>> it failed. Installation is only standalone spark without hadoop. Is it
>>> correct behaviour?
>>>
>>
>> Yes, it also sets up two HDFS clusters. Are they not working? Try to see
>> if Spark is working by running some simple jobs on it. (See
>> http://spark.apache.org/docs/latest/ec2-scripts.html.)
>>
>> There is no program called Hadoop. If you mean YARN, then indeed the
>> script does not set up YARN. It sets up standalone Spark.
>>
>>
>>> Also errors in the log:
>>> ERROR: Unknown Tachyon version
>>> Error: Could not find or load main class crayondata.com.log
>>>
>>
>> As long as Spark is working fine, you can ignore all output from the EC2
>> script :).
>>
>
>
Re: spark 1.6.0 on ec2 doesn't work
Posted by Peter Zhang <zh...@gmail.com>.
Could you run spark-shell at $SPARK_HOME DIR?
You can try to change you command run at $SPARK_HOME or, point to README.md with full path.
Peter Zhang
--
Google
Sent with Airmail
On January 19, 2016 at 11:26:14, Oleg Ruchovets (oruchovets@gmail.com) wrote:
It looks spark is not working fine :
I followed this link ( http://spark.apache.org/docs/latest/ec2-scripts.html. ) and I see spot instances installed on EC2.
from spark shell I am counting lines and got connection exception.
scala> val lines = sc.textFile("README.md")
scala> lines.count()
scala> val lines = sc.textFile("README.md")
16/01/19 03:17:35 INFO storage.MemoryStore: Block broadcast_0 stored as values in memory (estimated size 26.5 KB, free 26.5 KB)
16/01/19 03:17:35 INFO storage.MemoryStore: Block broadcast_0_piece0 stored as bytes in memory (estimated size 5.6 KB, free 32.1 KB)
16/01/19 03:17:35 INFO storage.BlockManagerInfo: Added broadcast_0_piece0 in memory on 172.31.28.196:44028 (size: 5.6 KB, free: 511.5 MB)
16/01/19 03:17:35 INFO spark.SparkContext: Created broadcast 0 from textFile at <console>:21
lines: org.apache.spark.rdd.RDD[String] = MapPartitionsRDD[1] at textFile at <console>:21
scala> lines.count()
16/01/19 03:17:55 INFO ipc.Client: Retrying connect to server: ec2-54-88-242-197.compute-1.amazonaws.com/172.31.28.196:9000. Already tried 0 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=10, sleepTime=1 SECONDS)
16/01/19 03:17:56 INFO ipc.Client: Retrying connect to server: ec2-54-88-242-197.compute-1.amazonaws.com/172.31.28.196:9000. Already tried 1 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=10, sleepTime=1 SECONDS)
16/01/19 03:17:57 INFO ipc.Client: Retrying connect to server: ec2-54-88-242-197.compute-1.amazonaws.com/172.31.28.196:9000. Already tried 2 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=10, sleepTime=1 SECONDS)
16/01/19 03:17:58 INFO ipc.Client: Retrying connect to server: ec2-54-88-242-197.compute-1.amazonaws.com/172.31.28.196:9000. Already tried 3 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=10, sleepTime=1 SECONDS)
16/01/19 03:17:59 INFO ipc.Client: Retrying connect to server: ec2-54-88-242-197.compute-1.amazonaws.com/172.31.28.196:9000. Already tried 4 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=10, sleepTime=1 SECONDS)
16/01/19 03:18:00 INFO ipc.Client: Retrying connect to server: ec2-54-88-242-197.compute-1.amazonaws.com/172.31.28.196:9000. Already tried 5 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=10, sleepTime=1 SECONDS)
16/01/19 03:18:01 INFO ipc.Client: Retrying connect to server: ec2-54-88-242-197.compute-1.amazonaws.com/172.31.28.196:9000. Already tried 6 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=10, sleepTime=1 SECONDS)
16/01/19 03:18:02 INFO ipc.Client: Retrying connect to server: ec2-54-88-242-197.compute-1.amazonaws.com/172.31.28.196:9000. Already tried 7 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=10, sleepTime=1 SECONDS)
16/01/19 03:18:03 INFO ipc.Client: Retrying connect to server: ec2-54-88-242-197.compute-1.amazonaws.com/172.31.28.196:9000. Already tried 8 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=10, sleepTime=1 SECONDS)
16/01/19 03:18:04 INFO ipc.Client: Retrying connect to server: ec2-54-88-242-197.compute-1.amazonaws.com/172.31.28.196:9000. Already tried 9 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=10, sleepTime=1 SECONDS)
java.lang.RuntimeException: java.net.ConnectException: Call to ec2-54-88-242-197.compute-1.amazonaws.com/172.31.28.196:9000 failed on connection exception: java.net.ConnectException: Connection refused
at org.apache.hadoop.mapred.JobConf.getWorkingDirectory(JobConf.java:567)
at org.apache.hadoop.mapred.FileInputFormat.setInputPaths(FileInputFormat.java:318)
at org.apache.hadoop.mapred.FileInputFormat.setInputPaths(FileInputFormat.java:291)
at org.apache.spark.SparkContext$$anonfun$hadoopFile$1$$anonfun$33.apply(SparkContext.scala:1015)
at org.apache.spark.SparkContext$$anonfun$hadoopFile$1$$anonfun$33.apply(SparkContext.scala:1015)
at org.apache.spark.rdd.HadoopRDD$$anonfun$getJobConf$6.apply(HadoopRDD.scala:176)
at org.apache.spark.rdd.HadoopRDD$$anonfun$getJobConf$6.apply(HadoopRDD.scala:176)
at scala.Option.map(Option.scala:145)
at org.apache.spark.rdd.HadoopRDD.getJobConf(HadoopRDD.scala:176)
at org.apache.spark.rdd.HadoopRDD.getPartitions(HadoopRDD.scala:195)
at org.apache.spark.rdd.RDD$$anonfun$partitions$2.apply(RDD.scala:239)
at org.apache.spark.rdd.RDD$$anonfun$partitions$2.apply(RDD.scala:237)
at scala.Option.getOrElse(Option.scala:120)
at org.apache.spark.rdd.RDD.partitions(RDD.scala:237)
at org.apache.spark.rdd.MapPartitionsRDD.getPartitions(MapPartitionsRDD.scala:35)
at org.apache.spark.rdd.RDD$$anonfun$partitions$2.apply(RDD.scala:239)
at org.apache.spark.rdd.RDD$$anonfun$partitions$2.apply(RDD.scala:237)
at scala.Option.getOrElse(Option.scala:120)
at org.apache.spark.rdd.RDD.partitions(RDD.scala:237)
at org.apache.spark.SparkContext.runJob(SparkContext.scala:1929)
at org.apache.spark.rdd.RDD.count(RDD.scala:1143)
at $iwC$$iwC$$iwC$$iwC$$iwC$$iwC.<init>(<console>:24)
at $iwC$$iwC$$iwC$$iwC$$iwC.<init>(<console>:29)
at $iwC$$iwC$$iwC$$iwC.<init>(<console>:31)
at $iwC$$iwC$$iwC.<init>(<console>:33)
at $iwC$$iwC.<init>(<console>:35)
at $iwC.<init>(<console>:37)
at <init>(<console>:39)
at .<init>(<console>:43)
at .<clinit>(<console>)
at .<init>(<console>:7)
at .<clinit>(<console>)
at $print(<console>)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at org.apache.spark.repl.SparkIMain$ReadEvalPrint.call(SparkIMain.scala:1065)
at org.apache.spark.repl.SparkIMain$Request.loadAndRun(SparkIMain.scala:1346)
at org.apache.spark.repl.SparkIMain.loadAndRunReq$1(SparkIMain.scala:840)
at org.apache.spark.repl.SparkIMain.interpret(SparkIMain.scala:871)
at org.apache.spark.repl.SparkIMain.interpret(SparkIMain.scala:819)
at org.apache.spark.repl.SparkILoop.reallyInterpret$1(SparkILoop.scala:857)
at org.apache.spark.repl.SparkILoop.interpretStartingWith(SparkILoop.scala:902)
at org.apache.spark.repl.SparkILoop.command(SparkILoop.scala:814)
at org.apache.spark.repl.SparkILoop.processLine$1(SparkILoop.scala:657)
at org.apache.spark.repl.SparkILoop.innerLoop$1(SparkILoop.scala:665)
at org.apache.spark.repl.SparkILoop.org$apache$spark$repl$SparkILoop$$loop(SparkILoop.scala:670)
at org.apache.spark.repl.SparkILoop$$anonfun$org$apache$spark$repl$SparkILoop$$process$1.apply$mcZ$sp(SparkILoop.scala:997)
at org.apache.spark.repl.SparkILoop$$anonfun$org$apache$spark$repl$SparkILoop$$process$1.apply(SparkILoop.scala:945)
at org.apache.spark.repl.SparkILoop$$anonfun$org$apache$spark$repl$SparkILoop$$process$1.apply(SparkILoop.scala:945)
at scala.tools.nsc.util.ScalaClassLoader$.savingContextLoader(ScalaClassLoader.scala:135)
at org.apache.spark.repl.SparkILoop.org$apache$spark$repl$SparkILoop$$process(SparkILoop.scala:945)
at org.apache.spark.repl.SparkILoop.process(SparkILoop.scala:1059)
at org.apache.spark.repl.Main$.main(Main.scala:31)
at org.apache.spark.repl.Main.main(Main.scala)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at org.apache.spark.deploy.SparkSubmit$.org$apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:731)
at org.apache.spark.deploy.SparkSubmit$.doRunMain$1(SparkSubmit.scala:181)
at org.apache.spark.deploy.SparkSubmit$.submit(SparkSubmit.scala:206)
at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:121)
at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)
Caused by: java.net.ConnectException: Call to ec2-54-88-242-197.compute-1.amazonaws.com/172.31.28.196:9000 failed on connection exception: java.net.ConnectException: Connection refused
at org.apache.hadoop.ipc.Client.wrapException(Client.java:1142)
at org.apache.hadoop.ipc.Client.call(Client.java:1118)
at org.apache.hadoop.ipc.RPC$Invoker.invoke(RPC.java:229)
at com.sun.proxy.$Proxy15.getProtocolVersion(Unknown Source)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:85)
at org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:62)
at com.sun.proxy.$Proxy15.getProtocolVersion(Unknown Source)
at org.apache.hadoop.ipc.RPC.checkVersion(RPC.java:422)
at org.apache.hadoop.hdfs.DFSClient.createNamenode(DFSClient.java:183)
at org.apache.hadoop.hdfs.DFSClient.<init>(DFSClient.java:281)
at org.apache.hadoop.hdfs.DFSClient.<init>(DFSClient.java:245)
at org.apache.hadoop.hdfs.DistributedFileSystem.initialize(DistributedFileSystem.java:100)
at org.apache.hadoop.fs.FileSystem.createFileSystem(FileSystem.java:1446)
at org.apache.hadoop.fs.FileSystem.access$200(FileSystem.java:67)
at org.apache.hadoop.fs.FileSystem$Cache.get(FileSystem.java:1464)
at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:263)
at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:124)
at org.apache.hadoop.mapred.JobConf.getWorkingDirectory(JobConf.java:563)
... 64 more
Caused by: java.net.ConnectException: Connection refused
at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:744)
at org.apache.hadoop.net.SocketIOWithTimeout.connect(SocketIOWithTimeout.java:206)
at org.apache.hadoop.net.NetUtils.connect(NetUtils.java:511)
at org.apache.hadoop.net.NetUtils.connect(NetUtils.java:481)
at org.apache.hadoop.ipc.Client$Connection.setupConnection(Client.java:457)
at org.apache.hadoop.ipc.Client$Connection.setupIOstreams(Client.java:583)
at org.apache.hadoop.ipc.Client$Connection.access$2200(Client.java:205)
at org.apache.hadoop.ipc.Client.getConnection(Client.java:1249)
at org.apache.hadoop.ipc.Client.call(Client.java:1093)
... 84 more
scala>
On Tue, Jan 19, 2016 at 1:22 AM, Daniel Darabos <da...@lynxanalytics.com> wrote:
On Mon, Jan 18, 2016 at 5:24 PM, Oleg Ruchovets <or...@gmail.com> wrote:
I thought script tries to install hadoop / hdfs also. And it looks like it failed. Installation is only standalone spark without hadoop. Is it correct behaviour?
Yes, it also sets up two HDFS clusters. Are they not working? Try to see if Spark is working by running some simple jobs on it. (See http://spark.apache.org/docs/latest/ec2-scripts.html.)
There is no program called Hadoop. If you mean YARN, then indeed the script does not set up YARN. It sets up standalone Spark.
Also errors in the log:
ERROR: Unknown Tachyon version
Error: Could not find or load main class crayondata.com.log
As long as Spark is working fine, you can ignore all output from the EC2 script :).
Re: spark 1.6.0 on ec2 doesn't work
Posted by Oleg Ruchovets <or...@gmail.com>.
It looks spark is not working fine :
I followed this link ( http://spark.apache.org/docs/latest/ec2-scripts.html.
) and I see spot instances installed on EC2.
from spark shell I am counting lines and got connection exception.
*scala> val lines = sc.textFile("README.md")*
*scala> lines.count()*
*scala> val lines = sc.textFile("README.md")*
16/01/19 03:17:35 INFO storage.MemoryStore: Block broadcast_0 stored as
values in memory (estimated size 26.5 KB, free 26.5 KB)
16/01/19 03:17:35 INFO storage.MemoryStore: Block broadcast_0_piece0 stored
as bytes in memory (estimated size 5.6 KB, free 32.1 KB)
16/01/19 03:17:35 INFO storage.BlockManagerInfo: Added broadcast_0_piece0
in memory on 172.31.28.196:44028 (size: 5.6 KB, free: 511.5 MB)
16/01/19 03:17:35 INFO spark.SparkContext: Created broadcast 0 from
textFile at <console>:21
lines: org.apache.spark.rdd.RDD[String] = MapPartitionsRDD[1] at textFile
at <console>:21
*scala> lines.count()*
16/01/19 03:17:55 INFO ipc.Client: Retrying connect to server:
ec2-54-88-242-197.compute-1.amazonaws.com/172.31.28.196:9000. Already tried
0 time(s); retry policy is
RetryUpToMaximumCountWithFixedSleep(maxRetries=10, sleepTime=1 SECONDS)
16/01/19 03:17:56 INFO ipc.Client: Retrying connect to server:
ec2-54-88-242-197.compute-1.amazonaws.com/172.31.28.196:9000. Already tried
1 time(s); retry policy is
RetryUpToMaximumCountWithFixedSleep(maxRetries=10, sleepTime=1 SECONDS)
16/01/19 03:17:57 INFO ipc.Client: Retrying connect to server:
ec2-54-88-242-197.compute-1.amazonaws.com/172.31.28.196:9000. Already tried
2 time(s); retry policy is
RetryUpToMaximumCountWithFixedSleep(maxRetries=10, sleepTime=1 SECONDS)
16/01/19 03:17:58 INFO ipc.Client: Retrying connect to server:
ec2-54-88-242-197.compute-1.amazonaws.com/172.31.28.196:9000. Already tried
3 time(s); retry policy is
RetryUpToMaximumCountWithFixedSleep(maxRetries=10, sleepTime=1 SECONDS)
16/01/19 03:17:59 INFO ipc.Client: Retrying connect to server:
ec2-54-88-242-197.compute-1.amazonaws.com/172.31.28.196:9000. Already tried
4 time(s); retry policy is
RetryUpToMaximumCountWithFixedSleep(maxRetries=10, sleepTime=1 SECONDS)
16/01/19 03:18:00 INFO ipc.Client: Retrying connect to server:
ec2-54-88-242-197.compute-1.amazonaws.com/172.31.28.196:9000. Already tried
5 time(s); retry policy is
RetryUpToMaximumCountWithFixedSleep(maxRetries=10, sleepTime=1 SECONDS)
16/01/19 03:18:01 INFO ipc.Client: Retrying connect to server:
ec2-54-88-242-197.compute-1.amazonaws.com/172.31.28.196:9000. Already tried
6 time(s); retry policy is
RetryUpToMaximumCountWithFixedSleep(maxRetries=10, sleepTime=1 SECONDS)
16/01/19 03:18:02 INFO ipc.Client: Retrying connect to server:
ec2-54-88-242-197.compute-1.amazonaws.com/172.31.28.196:9000. Already tried
7 time(s); retry policy is
RetryUpToMaximumCountWithFixedSleep(maxRetries=10, sleepTime=1 SECONDS)
16/01/19 03:18:03 INFO ipc.Client: Retrying connect to server:
ec2-54-88-242-197.compute-1.amazonaws.com/172.31.28.196:9000. Already tried
8 time(s); retry policy is
RetryUpToMaximumCountWithFixedSleep(maxRetries=10, sleepTime=1 SECONDS)
16/01/19 03:18:04 INFO ipc.Client: Retrying connect to server:
ec2-54-88-242-197.compute-1.amazonaws.com/172.31.28.196:9000. Already tried
9 time(s); retry policy is
RetryUpToMaximumCountWithFixedSleep(maxRetries=10, sleepTime=1 SECONDS)
java.lang.RuntimeException: java.net.ConnectException: Call to
ec2-54-88-242-197.compute-1.amazonaws.com/172.31.28.196:9000 failed on
connection exception: java.net.ConnectException: Connection refused
at org.apache.hadoop.mapred.JobConf.getWorkingDirectory(JobConf.java:567)
at
org.apache.hadoop.mapred.FileInputFormat.setInputPaths(FileInputFormat.java:318)
at
org.apache.hadoop.mapred.FileInputFormat.setInputPaths(FileInputFormat.java:291)
at
org.apache.spark.SparkContext$$anonfun$hadoopFile$1$$anonfun$33.apply(SparkContext.scala:1015)
at
org.apache.spark.SparkContext$$anonfun$hadoopFile$1$$anonfun$33.apply(SparkContext.scala:1015)
at
org.apache.spark.rdd.HadoopRDD$$anonfun$getJobConf$6.apply(HadoopRDD.scala:176)
at
org.apache.spark.rdd.HadoopRDD$$anonfun$getJobConf$6.apply(HadoopRDD.scala:176)
at scala.Option.map(Option.scala:145)
at org.apache.spark.rdd.HadoopRDD.getJobConf(HadoopRDD.scala:176)
at org.apache.spark.rdd.HadoopRDD.getPartitions(HadoopRDD.scala:195)
at org.apache.spark.rdd.RDD$$anonfun$partitions$2.apply(RDD.scala:239)
at org.apache.spark.rdd.RDD$$anonfun$partitions$2.apply(RDD.scala:237)
at scala.Option.getOrElse(Option.scala:120)
at org.apache.spark.rdd.RDD.partitions(RDD.scala:237)
at
org.apache.spark.rdd.MapPartitionsRDD.getPartitions(MapPartitionsRDD.scala:35)
at org.apache.spark.rdd.RDD$$anonfun$partitions$2.apply(RDD.scala:239)
at org.apache.spark.rdd.RDD$$anonfun$partitions$2.apply(RDD.scala:237)
at scala.Option.getOrElse(Option.scala:120)
at org.apache.spark.rdd.RDD.partitions(RDD.scala:237)
at org.apache.spark.SparkContext.runJob(SparkContext.scala:1929)
at org.apache.spark.rdd.RDD.count(RDD.scala:1143)
at $iwC$$iwC$$iwC$$iwC$$iwC$$iwC.<init>(<console>:24)
at $iwC$$iwC$$iwC$$iwC$$iwC.<init>(<console>:29)
at $iwC$$iwC$$iwC$$iwC.<init>(<console>:31)
at $iwC$$iwC$$iwC.<init>(<console>:33)
at $iwC$$iwC.<init>(<console>:35)
at $iwC.<init>(<console>:37)
at <init>(<console>:39)
at .<init>(<console>:43)
at .<clinit>(<console>)
at .<init>(<console>:7)
at .<clinit>(<console>)
at $print(<console>)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at
org.apache.spark.repl.SparkIMain$ReadEvalPrint.call(SparkIMain.scala:1065)
at
org.apache.spark.repl.SparkIMain$Request.loadAndRun(SparkIMain.scala:1346)
at org.apache.spark.repl.SparkIMain.loadAndRunReq$1(SparkIMain.scala:840)
at org.apache.spark.repl.SparkIMain.interpret(SparkIMain.scala:871)
at org.apache.spark.repl.SparkIMain.interpret(SparkIMain.scala:819)
at org.apache.spark.repl.SparkILoop.reallyInterpret$1(SparkILoop.scala:857)
at
org.apache.spark.repl.SparkILoop.interpretStartingWith(SparkILoop.scala:902)
at org.apache.spark.repl.SparkILoop.command(SparkILoop.scala:814)
at org.apache.spark.repl.SparkILoop.processLine$1(SparkILoop.scala:657)
at org.apache.spark.repl.SparkILoop.innerLoop$1(SparkILoop.scala:665)
at org.apache.spark.repl.SparkILoop.org
$apache$spark$repl$SparkILoop$$loop(SparkILoop.scala:670)
at
org.apache.spark.repl.SparkILoop$$anonfun$org$apache$spark$repl$SparkILoop$$process$1.apply$mcZ$sp(SparkILoop.scala:997)
at
org.apache.spark.repl.SparkILoop$$anonfun$org$apache$spark$repl$SparkILoop$$process$1.apply(SparkILoop.scala:945)
at
org.apache.spark.repl.SparkILoop$$anonfun$org$apache$spark$repl$SparkILoop$$process$1.apply(SparkILoop.scala:945)
at
scala.tools.nsc.util.ScalaClassLoader$.savingContextLoader(ScalaClassLoader.scala:135)
at org.apache.spark.repl.SparkILoop.org
$apache$spark$repl$SparkILoop$$process(SparkILoop.scala:945)
at org.apache.spark.repl.SparkILoop.process(SparkILoop.scala:1059)
at org.apache.spark.repl.Main$.main(Main.scala:31)
at org.apache.spark.repl.Main.main(Main.scala)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at
org.apache.spark.deploy.SparkSubmit$.org$apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:731)
at org.apache.spark.deploy.SparkSubmit$.doRunMain$1(SparkSubmit.scala:181)
at org.apache.spark.deploy.SparkSubmit$.submit(SparkSubmit.scala:206)
at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:121)
at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)
Caused by: java.net.ConnectException: Call to
ec2-54-88-242-197.compute-1.amazonaws.com/172.31.28.196:9000 failed on
connection exception: java.net.ConnectException: Connection refused
at org.apache.hadoop.ipc.Client.wrapException(Client.java:1142)
at org.apache.hadoop.ipc.Client.call(Client.java:1118)
at org.apache.hadoop.ipc.RPC$Invoker.invoke(RPC.java:229)
at com.sun.proxy.$Proxy15.getProtocolVersion(Unknown Source)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at
org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:85)
at
org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:62)
at com.sun.proxy.$Proxy15.getProtocolVersion(Unknown Source)
at org.apache.hadoop.ipc.RPC.checkVersion(RPC.java:422)
at org.apache.hadoop.hdfs.DFSClient.createNamenode(DFSClient.java:183)
at org.apache.hadoop.hdfs.DFSClient.<init>(DFSClient.java:281)
at org.apache.hadoop.hdfs.DFSClient.<init>(DFSClient.java:245)
at
org.apache.hadoop.hdfs.DistributedFileSystem.initialize(DistributedFileSystem.java:100)
at org.apache.hadoop.fs.FileSystem.createFileSystem(FileSystem.java:1446)
at org.apache.hadoop.fs.FileSystem.access$200(FileSystem.java:67)
at org.apache.hadoop.fs.FileSystem$Cache.get(FileSystem.java:1464)
at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:263)
at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:124)
at org.apache.hadoop.mapred.JobConf.getWorkingDirectory(JobConf.java:563)
... 64 more
Caused by: java.net.ConnectException: Connection refused
at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:744)
at
org.apache.hadoop.net.SocketIOWithTimeout.connect(SocketIOWithTimeout.java:206)
at org.apache.hadoop.net.NetUtils.connect(NetUtils.java:511)
at org.apache.hadoop.net.NetUtils.connect(NetUtils.java:481)
at org.apache.hadoop.ipc.Client$Connection.setupConnection(Client.java:457)
at org.apache.hadoop.ipc.Client$Connection.setupIOstreams(Client.java:583)
at org.apache.hadoop.ipc.Client$Connection.access$2200(Client.java:205)
at org.apache.hadoop.ipc.Client.getConnection(Client.java:1249)
at org.apache.hadoop.ipc.Client.call(Client.java:1093)
... 84 more
scala>
On Tue, Jan 19, 2016 at 1:22 AM, Daniel Darabos <
daniel.darabos@lynxanalytics.com> wrote:
>
> On Mon, Jan 18, 2016 at 5:24 PM, Oleg Ruchovets <or...@gmail.com>
> wrote:
>
>> I thought script tries to install hadoop / hdfs also. And it looks like
>> it failed. Installation is only standalone spark without hadoop. Is it
>> correct behaviour?
>>
>
> Yes, it also sets up two HDFS clusters. Are they not working? Try to see
> if Spark is working by running some simple jobs on it. (See
> http://spark.apache.org/docs/latest/ec2-scripts.html.)
>
> There is no program called Hadoop. If you mean YARN, then indeed the
> script does not set up YARN. It sets up standalone Spark.
>
>
>> Also errors in the log:
>> ERROR: Unknown Tachyon version
>> Error: Could not find or load main class crayondata.com.log
>>
>
> As long as Spark is working fine, you can ignore all output from the EC2
> script :).
>
Re: spark 1.6.0 on ec2 doesn't work
Posted by Daniel Darabos <da...@lynxanalytics.com>.
On Mon, Jan 18, 2016 at 5:24 PM, Oleg Ruchovets <or...@gmail.com>
wrote:
> I thought script tries to install hadoop / hdfs also. And it looks like it
> failed. Installation is only standalone spark without hadoop. Is it correct
> behaviour?
>
Yes, it also sets up two HDFS clusters. Are they not working? Try to see if
Spark is working by running some simple jobs on it. (See
http://spark.apache.org/docs/latest/ec2-scripts.html.)
There is no program called Hadoop. If you mean YARN, then indeed the script
does not set up YARN. It sets up standalone Spark.
> Also errors in the log:
> ERROR: Unknown Tachyon version
> Error: Could not find or load main class crayondata.com.log
>
As long as Spark is working fine, you can ignore all output from the EC2
script :).
Re: spark 1.6.0 on ec2 doesn't work
Posted by Oleg Ruchovets <or...@gmail.com>.
I thought script tries to install hadoop / hdfs also. And it looks like it
failed. Installation is only standalone spark without hadoop. Is it correct
behaviour?
Also errors in the log:
ERROR: Unknown Tachyon version
Error: Could not find or load main class crayondata.com.log
Thanks
Oleg.
Re: spark 1.6.0 on ec2 doesn't work
Posted by Daniel Darabos <da...@lynxanalytics.com>.
Hi,
How do you know it doesn't work? The log looks roughly normal to me. Is
Spark not running at the printed address? Can you not start jobs?
On Mon, Jan 18, 2016 at 11:51 AM, Oleg Ruchovets <or...@gmail.com>
wrote:
> Hi ,
> I try to follow the spartk 1.6.0 to install spark on EC2.
>
> It doesn't work properly - got exceptions and at the end standalone spark
> cluster installed.
>
The purpose of the script is to install a standalone Spark cluster. So
that's not an error :).
> here is log information:
>
> Any suggestions?
>
> Thanks
> Oleg.
>
> oleg@robinhood:~/install/spark-1.6.0-bin-hadoop2.6/ec2$ ./spark-ec2
> --key-pair=CC-ES-Demo
> --identity-file=/home/oleg/work/entity_extraction_framework/ec2_pem_key/CC-ES-Demo.pem
> --region=us-east-1 --zone=us-east-1a --spot-price=0.05 -s 5
> --spark-version=1.6.0 launch entity-extraction-spark-cluster
> Setting up security groups...
> Searching for existing cluster entity-extraction-spark-cluster in region
> us-east-1...
> Spark AMI: ami-5bb18832
> Launching instances...
> Requesting 5 slaves as spot instances with price $0.050
> Waiting for spot instances to be granted...
> 0 of 5 slaves granted, waiting longer
> 0 of 5 slaves granted, waiting longer
> 0 of 5 slaves granted, waiting longer
> 0 of 5 slaves granted, waiting longer
> 0 of 5 slaves granted, waiting longer
> 0 of 5 slaves granted, waiting longer
> 0 of 5 slaves granted, waiting longer
> 0 of 5 slaves granted, waiting longer
> 0 of 5 slaves granted, waiting longer
> All 5 slaves granted
> Launched master in us-east-1a, regid = r-9384033f
> Waiting for AWS to propagate instance metadata...
> Waiting for cluster to enter 'ssh-ready' state..........
>
> Warning: SSH connection error. (This could be temporary.)
> Host: ec2-52-90-186-83.compute-1.amazonaws.com
> SSH return code: 255
> SSH output: ssh: connect to host ec2-52-90-186-83.compute-1.amazonaws.com
> port 22: Connection refused
>
> .
>
> Warning: SSH connection error. (This could be temporary.)
> Host: ec2-52-90-186-83.compute-1.amazonaws.com
> SSH return code: 255
> SSH output: ssh: connect to host ec2-52-90-186-83.compute-1.amazonaws.com
> port 22: Connection refused
>
> .
>
> Warning: SSH connection error. (This could be temporary.)
> Host: ec2-52-90-186-83.compute-1.amazonaws.com
> SSH return code: 255
> SSH output: ssh: connect to host ec2-52-90-186-83.compute-1.amazonaws.com
> port 22: Connection refused
>
> .
> Cluster is now in 'ssh-ready' state. Waited 442 seconds.
> Generating cluster's SSH key on master...
> Warning: Permanently added 'ec2-52-90-186-83.compute-1.amazonaws.com,52.90.186.83'
> (ECDSA) to the list of known hosts.
> Connection to ec2-52-90-186-83.compute-1.amazonaws.com closed.
> Warning: Permanently added 'ec2-52-90-186-83.compute-1.amazonaws.com,52.90.186.83'
> (ECDSA) to the list of known hosts.
> Transferring cluster's SSH key to slaves...
> ec2-54-165-243-74.compute-1.amazonaws.com
> Warning: Permanently added 'ec2-54-165-243-74.compute-1.amazonaws.com,54.165.243.74'
> (ECDSA) to the list of known hosts.
> ec2-54-88-245-107.compute-1.amazonaws.com
> Warning: Permanently added 'ec2-54-88-245-107.compute-1.amazonaws.com,54.88.245.107'
> (ECDSA) to the list of known hosts.
> ec2-54-172-29-47.compute-1.amazonaws.com
> Warning: Permanently added 'ec2-54-172-29-47.compute-1.amazonaws.com,54.172.29.47'
> (ECDSA) to the list of known hosts.
> ec2-54-165-131-210.compute-1.amazonaws.com
> Warning: Permanently added 'ec2-54-165-131-210.compute-1.amazonaws.com,54.165.131.210'
> (ECDSA) to the list of known hosts.
> ec2-54-172-46-184.compute-1.amazonaws.com
> Warning: Permanently added 'ec2-54-172-46-184.compute-1.amazonaws.com,54.172.46.184'
> (ECDSA) to the list of known hosts.
> Cloning spark-ec2 scripts from
> https://github.com/amplab/spark-ec2/tree/branch-1.5 on master...
> Warning: Permanently added 'ec2-52-90-186-83.compute-1.amazonaws.com,52.90.186.83'
> (ECDSA) to the list of known hosts.
> Cloning into 'spark-ec2'...
> remote: Counting objects: 2068, done.
> remote: Total 2068 (delta 0), reused 0 (delta 0), pack-reused 2068
> Receiving objects: 100% (2068/2068), 349.76 KiB, done.
> Resolving deltas: 100% (796/796), done.
> Connection to ec2-52-90-186-83.compute-1.amazonaws.com closed.
> Deploying files to master...
> Warning: Permanently added 'ec2-52-90-186-83.compute-1.amazonaws.com,52.90.186.83'
> (ECDSA) to the list of known hosts.
> sending incremental file list
> root/spark-ec2/ec2-variables.sh
>
> sent 1,835 bytes received 40 bytes 416.67 bytes/sec
> total size is 1,684 speedup is 0.90
> Running setup on master...
> Warning: Permanently added 'ec2-52-90-186-83.compute-1.amazonaws.com,52.90.186.83'
> (ECDSA) to the list of known hosts.
> Connection to ec2-52-90-186-83.compute-1.amazonaws.com closed.
> Warning: Permanently added 'ec2-52-90-186-83.compute-1.amazonaws.com,52.90.186.83'
> (ECDSA) to the list of known hosts.
> Setting up Spark on ip-172-31-24-124.ec2.internal...
> Setting executable permissions on scripts...
> RSYNC'ing /root/spark-ec2 to other cluster nodes...
> ec2-54-165-243-74.compute-1.amazonaws.com
> Warning: Permanently added 'ec2-54-165-243-74.compute-1.amazonaws.com,172.31.19.61'
> (ECDSA) to the list of known hosts.
> ec2-54-88-245-107.compute-1.amazonaws.com
> id_rsa
>
> 100% 1679 1.6KB/s 00:00
> Warning: Permanently added 'ec2-54-88-245-107.compute-1.amazonaws.com,172.31.30.81'
> (ECDSA) to the list of known hosts.
> ec2-54-172-29-47.compute-1.amazonaws.com
> id_rsa
>
> 100% 1679 1.6KB/s 00:00
> Warning: Permanently added 'ec2-54-172-29-47.compute-1.amazonaws.com,172.31.29.54'
> (ECDSA) to the list of known hosts.
> ec2-54-165-131-210.compute-1.amazonaws.com
> Warning: Permanently added 'ec2-54-165-131-210.compute-1.amazonaws.com,172.31.23.10'
> (ECDSA) to the list of known hosts.
> id_rsa
>
> 100% 1679 1.6KB/s 00:00
> ec2-54-172-46-184.compute-1.amazonaws.com
> Warning: Permanently added 'ec2-54-172-46-184.compute-1.amazonaws.com,172.31.30.167'
> (ECDSA) to the list of known hosts.
> id_rsa
>
> 100% 1679 1.6KB/s 00:00
> id_rsa
>
> 100% 1679 1.6KB/s 00:00
> [timing] rsync /root/spark-ec2: 00h 00m 01s
> Running setup-slave on all cluster nodes to mount filesystems, etc...
> [1] 08:08:10 [SUCCESS] ec2-52-90-186-83.compute-1.amazonaws.com
> checking/fixing resolution of hostname
> Setting up slave on ip-172-31-24-124.ec2.internal... of type m1.large
> 1024+0 records in
> 1024+0 records out
> 1073741824 bytes (1.1 GB) copied, 2.01407 s, 533 MB/s
> mkswap: /mnt/swap: warning: don't erase bootbits sectors
> on whole disk. Use -f to force.
> Setting up swapspace version 1, size = 1048572 KiB
> no label, UUID=b4d25f54-4732-40bb-8086-c78117cb58b2
> Added 1024 MB swap file /mnt/swap
> Stderr: Warning: Permanently added '
> ec2-52-90-186-83.compute-1.amazonaws.com,172.31.24.124' (ECDSA) to the
> list of known hosts.
> Connection to ec2-52-90-186-83.compute-1.amazonaws.com closed.
> [2] 08:08:24 [SUCCESS] ec2-54-165-243-74.compute-1.amazonaws.com
> checking/fixing resolution of hostname
> Setting up slave on ip-172-31-19-61.ec2.internal... of type m1.large
> 1024+0 records in
> 1024+0 records out
> 1073741824 bytes (1.1 GB) copied, 2.11705 s, 507 MB/s
> mkswap: /mnt/swap: warning: don't erase bootbits sectors
> on whole disk. Use -f to force.
> Setting up swapspace version 1, size = 1048572 KiB
> no label, UUID=928041a8-4d48-4c65-94e2-d9f84e14cad9
> Added 1024 MB swap file /mnt/swap
> Stderr: Connection to ec2-54-165-243-74.compute-1.amazonaws.com closed.
> [3] 08:08:27 [SUCCESS] ec2-54-88-245-107.compute-1.amazonaws.com
> checking/fixing resolution of hostname
> Setting up slave on ip-172-31-30-81.ec2.internal... of type m1.large
> 1024+0 records in
> 1024+0 records out
> 1073741824 bytes (1.1 GB) copied, 2.21007 s, 486 MB/s
> mkswap: /mnt/swap: warning: don't erase bootbits sectors
> on whole disk. Use -f to force.
> Setting up swapspace version 1, size = 1048572 KiB
> no label, UUID=1e8c3d4c-7e27-4c35-acae-d83ec2ea9edb
> Added 1024 MB swap file /mnt/swap
> Stderr: Connection to ec2-54-88-245-107.compute-1.amazonaws.com closed.
> [4] 08:08:32 [SUCCESS] ec2-54-172-29-47.compute-1.amazonaws.com
> checking/fixing resolution of hostname
> Setting up slave on ip-172-31-29-54.ec2.internal... of type m1.large
> 1024+0 records in
> 1024+0 records out
> 1073741824 bytes (1.1 GB) copied, 2.15544 s, 498 MB/s
> mkswap: /mnt/swap: warning: don't erase bootbits sectors
> on whole disk. Use -f to force.
> Setting up swapspace version 1, size = 1048572 KiB
> no label, UUID=7bd81d33-ae22-4973-810e-855535ecb743
> Added 1024 MB swap file /mnt/swap
> Stderr: Connection to ec2-54-172-29-47.compute-1.amazonaws.com closed.
> [5] 08:08:34 [SUCCESS] ec2-54-165-131-210.compute-1.amazonaws.com
> checking/fixing resolution of hostname
> Setting up slave on ip-172-31-23-10.ec2.internal... of type m1.large
> 1024+0 records in
> 1024+0 records out
> 1073741824 bytes (1.1 GB) copied, 2.39186 s, 449 MB/s
> mkswap: /mnt/swap: warning: don't erase bootbits sectors
> on whole disk. Use -f to force.
> Setting up swapspace version 1, size = 1048572 KiB
> no label, UUID=abbdbe4d-f8e8-469b-90d2-c9d0a244b261
> Added 1024 MB swap file /mnt/swap
> Stderr: Connection to ec2-54-165-131-210.compute-1.amazonaws.com closed.
> [6] 08:08:37 [SUCCESS] ec2-54-172-46-184.compute-1.amazonaws.com
> checking/fixing resolution of hostname
> Setting up slave on ip-172-31-30-167.ec2.internal... of type m1.large
> 1024+0 records in
> 1024+0 records out
> 1073741824 bytes (1.1 GB) copied, 2.1603 s, 497 MB/s
> mkswap: /mnt/swap: warning: don't erase bootbits sectors
> on whole disk. Use -f to force.
> Setting up swapspace version 1, size = 1048572 KiB
> no label, UUID=115ac0e9-c28c-4404-a648-826ece20815d
> Added 1024 MB swap file /mnt/swap
> Stderr: Connection to ec2-54-172-46-184.compute-1.amazonaws.com closed.
> [timing] setup-slave: 00h 00m 45s
> Initializing scala
> Unpacking Scala
> --2016-01-18 08:08:37--
> http://s3.amazonaws.com/spark-related-packages/scala-2.10.3.tgz
> Resolving s3.amazonaws.com (s3.amazonaws.com)... 54.231.13.224
> Connecting to s3.amazonaws.com (s3.amazonaws.com)|54.231.13.224|:80...
> connected.
> HTTP request sent, awaiting response... 200 OK
> Length: 30531249 (29M) [application/x-compressed]
> Saving to: ‘scala-2.10.3.tgz’
>
> 100%[============================================================================================================================================>]
> 30,531,249 3.46MB/s in 10s
>
> 2016-01-18 08:08:47 (2.86 MB/s) - ‘scala-2.10.3.tgz’ saved
> [30531249/30531249]
>
> [timing] scala init: 00h 00m 11s
> Initializing spark
> --2016-01-18 08:08:48--
> http://s3.amazonaws.com/spark-related-packages/spark-1.6.0-bin-hadoop1.tgz
> Resolving s3.amazonaws.com (s3.amazonaws.com)... 54.231.81.220
> Connecting to s3.amazonaws.com (s3.amazonaws.com)|54.231.81.220|:80...
> connected.
> HTTP request sent, awaiting response... 200 OK
> Length: 243448482 (232M) [application/x-compressed]
> Saving to: ‘spark-1.6.0-bin-hadoop1.tgz’
>
> 100%[============================================================================================================================================>]
> 243,448,482 65.6MB/s in 3.5s
>
> 2016-01-18 08:08:52 (65.6 MB/s) - ‘spark-1.6.0-bin-hadoop1.tgz’ saved
> [243448482/243448482]
>
> Unpacking Spark
> [timing] spark init: 00h 00m 08s
> Initializing ephemeral-hdfs
> --2016-01-18 08:08:56--
> http://s3.amazonaws.com/spark-related-packages/hadoop-1.0.4.tar.gz
> Resolving s3.amazonaws.com (s3.amazonaws.com)... 54.231.17.48
> Connecting to s3.amazonaws.com (s3.amazonaws.com)|54.231.17.48|:80...
> connected.
> HTTP request sent, awaiting response... 200 OK
> Length: 62793050 (60M) [application/x-gzip]
> Saving to: ‘hadoop-1.0.4.tar.gz’
>
> 100%[============================================================================================================================================>]
> 62,793,050 69.2MB/s in 0.9s
>
> 2016-01-18 08:08:57 (69.2 MB/s) - ‘hadoop-1.0.4.tar.gz’ saved
> [62793050/62793050]
>
> Unpacking Hadoop
> RSYNC'ing /root/ephemeral-hdfs to slaves...
> ec2-54-165-243-74.compute-1.amazonaws.com
> Warning: Permanently added 'ec2-54-165-243-74.compute-1.amazonaws.com,172.31.19.61'
> (ECDSA) to the list of known hosts.
> ec2-54-88-245-107.compute-1.amazonaws.com
> Warning: Permanently added 'ec2-54-88-245-107.compute-1.amazonaws.com,172.31.30.81'
> (ECDSA) to the list of known hosts.
> ec2-54-172-29-47.compute-1.amazonaws.com
> Warning: Permanently added 'ec2-54-172-29-47.compute-1.amazonaws.com,172.31.29.54'
> (ECDSA) to the list of known hosts.
> ec2-54-165-131-210.compute-1.amazonaws.com
> Warning: Permanently added 'ec2-54-165-131-210.compute-1.amazonaws.com,172.31.23.10'
> (ECDSA) to the list of known hosts.
> ec2-54-172-46-184.compute-1.amazonaws.com
> Warning: Permanently added 'ec2-54-172-46-184.compute-1.amazonaws.com,172.31.30.167'
> (ECDSA) to the list of known hosts.
> [timing] ephemeral-hdfs init: 00h 00m 54s
> Initializing persistent-hdfs
> --2016-01-18 08:09:50--
> http://s3.amazonaws.com/spark-related-packages/hadoop-1.0.4.tar.gz
> Resolving s3.amazonaws.com (s3.amazonaws.com)... 54.231.49.236
> Connecting to s3.amazonaws.com (s3.amazonaws.com)|54.231.49.236|:80...
> connected.
> HTTP request sent, awaiting response... 200 OK
> Length: 62793050 (60M) [application/x-gzip]
> Saving to: ‘hadoop-1.0.4.tar.gz’
>
> 100%[============================================================================================================================================>]
> 62,793,050 67.4MB/s in 0.9s
>
> 2016-01-18 08:09:51 (67.4 MB/s) - ‘hadoop-1.0.4.tar.gz’ saved
> [62793050/62793050]
>
> Unpacking Hadoop
> RSYNC'ing /root/persistent-hdfs to slaves...
> ec2-54-165-243-74.compute-1.amazonaws.com
> ec2-54-88-245-107.compute-1.amazonaws.com
> ec2-54-172-29-47.compute-1.amazonaws.com
> ec2-54-165-131-210.compute-1.amazonaws.com
> ec2-54-172-46-184.compute-1.amazonaws.com
> [timing] persistent-hdfs init: 00h 00m 39s
> Initializing spark-standalone
> [timing] spark-standalone init: 00h 00m 00s
> Initializing tachyon
> --2016-01-18 08:10:29--
> https://s3.amazonaws.com/Tachyon/tachyon-0.8.2-bin.tar.gz
> Resolving s3.amazonaws.com (s3.amazonaws.com)... 54.231.81.67
> Connecting to s3.amazonaws.com (s3.amazonaws.com)|54.231.81.67|:443...
> connected.
> HTTP request sent, awaiting response... 403 Forbidden
> 2016-01-18 08:10:29 ERROR 403: Forbidden.
>
> ERROR: Unknown Tachyon version
> tachyon/init.sh: line 60: return: -1: invalid option
> return: usage: return [n]
> Unpacking Tachyon
> tar (child): tachyon-*.tar.gz: Cannot open: No such file or directory
> tar (child): Error is not recoverable: exiting now
> tar: Child returned status 2
> tar: Error is not recoverable: exiting now
> rm: cannot remove `tachyon-*.tar.gz': No such file or directory
> ls: cannot access tachyon-*: No such file or directory
> mv: missing destination file operand after `tachyon'
> Try `mv --help' for more information.
> [timing] tachyon init: 00h 00m 00s
> Initializing rstudio
> --2016-01-18 08:10:29--
> http://download2.rstudio.org/rstudio-server-rhel-0.99.446-x86_64.rpm
> Resolving download2.rstudio.org (download2.rstudio.org)... 54.192.18.169,
> 54.192.18.246, 54.192.18.133, ...
> Connecting to download2.rstudio.org (download2.rstudio.org)|54.192.18.169|:80...
> connected.
> HTTP request sent, awaiting response... 200 OK
> Length: 35035164 (33M) [application/x-redhat-package-manager]
> Saving to: ‘rstudio-server-rhel-0.99.446-x86_64.rpm’
>
> 100%[============================================================================================================================================>]
> 35,035,164 84.0MB/s in 0.4s
>
> 2016-01-18 08:10:29 (84.0 MB/s) -
> ‘rstudio-server-rhel-0.99.446-x86_64.rpm’ saved [35035164/35035164]
>
> Loaded plugins: priorities, update-motd, upgrade-helper
> Examining rstudio-server-rhel-0.99.446-x86_64.rpm:
> rstudio-server-0.99.446-1.x86_64
> Marking rstudio-server-rhel-0.99.446-x86_64.rpm to be installed
> Resolving Dependencies
> --> Running transaction check
> ---> Package rstudio-server.x86_64 0:0.99.446-1 will be installed
> --> Finished Dependency Resolution
>
> Dependencies Resolved
>
>
> ======================================================================================================================================================================================
> Package Arch
> Version Repository
> Size
>
> ======================================================================================================================================================================================
> Installing:
> rstudio-server x86_64
> 0.99.446-1 /rstudio-server-rhel-0.99.446-x86_64
> 252 M
>
> Transaction Summary
>
> ======================================================================================================================================================================================
> Install 1 Package
>
> Total size: 252 M
> Installed size: 252 M
> Downloading packages:
> Running transaction check
> Running transaction test
> Transaction test succeeded
> Running transaction
> Installing : rstudio-server-0.99.446-1.x86_64
>
> 1/1
> groupadd: group 'rstudio-server' already exists
> rsession: no process killed
> rstudio-server start/running, process 2535
> Verifying : rstudio-server-0.99.446-1.x86_64
>
> 1/1
>
> Installed:
> rstudio-server.x86_64 0:0.99.446-1
>
>
>
> Complete!
> rstudio-server start/running, process 2570
> [timing] rstudio init: 00h 00m 39s
> Initializing ganglia
> Connection to ec2-54-165-243-74.compute-1.amazonaws.com closed.
> Connection to ec2-54-88-245-107.compute-1.amazonaws.com closed.
> Connection to ec2-54-172-29-47.compute-1.amazonaws.com closed.
> Connection to ec2-54-165-131-210.compute-1.amazonaws.com closed.
> Connection to ec2-54-172-46-184.compute-1.amazonaws.com closed.
> [timing] ganglia init: 00h 00m 02s
> Creating local config files...
> Connection to ec2-54-165-243-74.compute-1.amazonaws.com closed.
> Connection to ec2-54-165-243-74.compute-1.amazonaws.com closed.
> Configuring /etc/ganglia/gmond.conf
> Configuring /etc/ganglia/gmetad.conf
> Configuring /etc/httpd/conf.d/ganglia.conf
> Configuring /etc/httpd/conf/httpd.conf
> Configuring /root/mapreduce/hadoop.version
> Configuring /root/mapreduce/conf/core-site.xml
> Configuring /root/mapreduce/conf/slaves
> Configuring /root/mapreduce/conf/mapred-site.xml
> Configuring /root/mapreduce/conf/hdfs-site.xml
> Configuring /root/mapreduce/conf/hadoop-env.sh
> Configuring /root/mapreduce/conf/masters
> Configuring /root/persistent-hdfs/conf/core-site.xml
> Configuring /root/persistent-hdfs/conf/slaves
> Configuring /root/persistent-hdfs/conf/mapred-site.xml
> Configuring /root/persistent-hdfs/conf/hdfs-site.xml
> Configuring /root/persistent-hdfs/conf/hadoop-env.sh
> Configuring /root/persistent-hdfs/conf/masters
> Configuring /root/ephemeral-hdfs/conf/core-site.xml
> Configuring /root/ephemeral-hdfs/conf/yarn-site.xml
> Configuring /root/ephemeral-hdfs/conf/slaves
> Configuring /root/ephemeral-hdfs/conf/mapred-site.xml
> Configuring /root/ephemeral-hdfs/conf/hadoop-metrics2.properties
> Configuring /root/ephemeral-hdfs/conf/capacity-scheduler.xml
> Configuring /root/ephemeral-hdfs/conf/yarn-env.sh
> Configuring /root/ephemeral-hdfs/conf/hdfs-site.xml
> Configuring /root/ephemeral-hdfs/conf/hadoop-env.sh
> Configuring /root/ephemeral-hdfs/conf/masters
> Configuring /root/spark/conf/core-site.xml
> Configuring /root/spark/conf/spark-defaults.conf
> Configuring /root/spark/conf/spark-env.sh
> Configuring /root/tachyon/conf/slaves
> Configuring /root/tachyon/conf/workers
> Configuring /root/tachyon/conf/tachyon-env.sh
> Deploying Spark config files...
> RSYNC'ing /root/spark/conf to slaves...
> ec2-54-165-243-74.compute-1.amazonaws.com
> ec2-54-88-245-107.compute-1.amazonaws.com
> ec2-54-172-29-47.compute-1.amazonaws.com
> ec2-54-165-131-210.compute-1.amazonaws.com
> ec2-54-172-46-184.compute-1.amazonaws.com
> Setting up scala
> RSYNC'ing /root/scala to slaves...
> ec2-54-165-243-74.compute-1.amazonaws.com
> ec2-54-88-245-107.compute-1.amazonaws.com
> ec2-54-172-29-47.compute-1.amazonaws.com
> ec2-54-165-131-210.compute-1.amazonaws.com
> ec2-54-172-46-184.compute-1.amazonaws.com
> [timing] scala setup: 00h 00m 09s
> Setting up spark
> RSYNC'ing /root/spark to slaves...
> ec2-54-165-243-74.compute-1.amazonaws.com
> ec2-54-88-245-107.compute-1.amazonaws.com
> ec2-54-172-29-47.compute-1.amazonaws.com
> ec2-54-165-131-210.compute-1.amazonaws.com
> ec2-54-172-46-184.compute-1.amazonaws.com
> [timing] spark setup: 00h 01m 07s
> Setting up ephemeral-hdfs
> ec2-54-165-243-74.compute-1.amazonaws.com
> ec2-54-88-245-107.compute-1.amazonaws.com
> Connection to ec2-54-165-243-74.compute-1.amazonaws.com closed.
> ec2-54-172-29-47.compute-1.amazonaws.com
> Connection to ec2-54-88-245-107.compute-1.amazonaws.com closed.
> Connection to ec2-54-172-29-47.compute-1.amazonaws.com closed.
> ec2-54-165-131-210.compute-1.amazonaws.com
> Connection to ec2-54-165-131-210.compute-1.amazonaws.com closed.
> ec2-54-172-46-184.compute-1.amazonaws.com
> Connection to ec2-54-172-46-184.compute-1.amazonaws.com closed.
> RSYNC'ing /root/ephemeral-hdfs/conf to slaves...
> ec2-54-165-243-74.compute-1.amazonaws.com
> ec2-54-88-245-107.compute-1.amazonaws.com
> ec2-54-172-29-47.compute-1.amazonaws.com
> ec2-54-165-131-210.compute-1.amazonaws.com
> ec2-54-172-46-184.compute-1.amazonaws.com
> Formatting ephemeral HDFS namenode...
> Warning: $HADOOP_HOME is deprecated.
>
> 16/01/18 08:12:39 INFO namenode.NameNode: STARTUP_MSG:
> /************************************************************
> STARTUP_MSG: Starting NameNode
> STARTUP_MSG: host = ip-172-31-24-124.ec2.internal/172.31.24.124
> STARTUP_MSG: args = [-format]
> STARTUP_MSG: version = 1.0.4
> STARTUP_MSG: build =
> https://svn.apache.org/repos/asf/hadoop/common/branches/branch-1.0 -r
> 1393290; compiled by 'hortonfo' on Wed Oct 3 05:13:58 UTC 2012
> ************************************************************/
> 16/01/18 08:12:39 INFO util.GSet: VM type = 64-bit
> 16/01/18 08:12:39 INFO util.GSet: 2% max memory = 17.78 MB
> 16/01/18 08:12:39 INFO util.GSet: capacity = 2^21 = 2097152 entries
> 16/01/18 08:12:39 INFO util.GSet: recommended=2097152, actual=2097152
> 16/01/18 08:12:39 INFO namenode.FSNamesystem: fsOwner=root
> 16/01/18 08:12:39 INFO namenode.FSNamesystem: supergroup=supergroup
> 16/01/18 08:12:39 INFO namenode.FSNamesystem: isPermissionEnabled=false
> 16/01/18 08:12:39 INFO namenode.FSNamesystem:
> dfs.block.invalidate.limit=100
> 16/01/18 08:12:39 INFO namenode.FSNamesystem: isAccessTokenEnabled=false
> accessKeyUpdateInterval=0 min(s), accessTokenLifetime=0 min(s)
> 16/01/18 08:12:39 INFO namenode.NameNode: Caching file names occuring more
> than 10 times
> 16/01/18 08:12:39 INFO common.Storage: Image file of size 110 saved in 0
> seconds.
> 16/01/18 08:12:39 INFO common.Storage: Storage directory
> /mnt/ephemeral-hdfs/dfs/name has been successfully formatted.
> 16/01/18 08:12:39 INFO namenode.NameNode: SHUTDOWN_MSG:
> /************************************************************
> SHUTDOWN_MSG: Shutting down NameNode at ip-172-31-24-124.ec2.internal/
> 172.31.24.124
> ************************************************************/
> Starting ephemeral HDFS...
> Warning: $HADOOP_HOME is deprecated.
>
> starting namenode, logging to
> /mnt/ephemeral-hdfs/logs/hadoop-root-namenode-ip-172-31-24-124.ec2.internal
> Error: Could not find or load main class crayondata.com.log
> ec2-54-172-29-47.compute-1.amazonaws.com: Warning: $HADOOP_HOME is
> deprecated.
> ec2-54-172-29-47.compute-1.amazonaws.com:
> ec2-54-172-29-47.compute-1.amazonaws.com: starting datanode, logging to
> /mnt/ephemeral-hdfs/logs/hadoop-root-datanode-ip-172-31-29-54.ec2.internal.out
> ec2-54-172-46-184.compute-1.amazonaws.com: Warning: $HADOOP_HOME is
> deprecated.
> ec2-54-172-46-184.compute-1.amazonaws.com:
> ec2-54-172-46-184.compute-1.amazonaws.com: starting datanode, logging to
> /mnt/ephemeral-hdfs/logs/hadoop-root-datanode-ip-172-31-30-167.ec2.internal.out
> ec2-54-165-131-210.compute-1.amazonaws.com: Warning: $HADOOP_HOME is
> deprecated.
> ec2-54-165-131-210.compute-1.amazonaws.com:
> ec2-54-165-131-210.compute-1.amazonaws.com: starting datanode, logging to
> /mnt/ephemeral-hdfs/logs/hadoop-root-datanode-ip-172-31-23-10.ec2.internal.out
> ec2-54-88-245-107.compute-1.amazonaws.com: Warning: $HADOOP_HOME is
> deprecated.
> ec2-54-88-245-107.compute-1.amazonaws.com:
> ec2-54-88-245-107.compute-1.amazonaws.com: starting datanode, logging to
> /mnt/ephemeral-hdfs/logs/hadoop-root-datanode-ip-172-31-30-81.ec2.internal.out
> ec2-54-165-243-74.compute-1.amazonaws.com: Warning: $HADOOP_HOME is
> deprecated.
> ec2-54-165-243-74.compute-1.amazonaws.com:
> ec2-54-165-243-74.compute-1.amazonaws.com: starting datanode, logging to
> /mnt/ephemeral-hdfs/logs/hadoop-root-datanode-ip-172-31-19-61.ec2.internal.out
> ec2-52-90-186-83.compute-1.amazonaws.com: Warning: Permanently added '
> ec2-52-90-186-83.compute-1.amazonaws.com,172.31.24.124' (ECDSA) to the
> list of known hosts.
> ec2-52-90-186-83.compute-1.amazonaws.com: Warning: $HADOOP_HOME is
> deprecated.
> ec2-52-90-186-83.compute-1.amazonaws.com:
> ec2-52-90-186-83.compute-1.amazonaws.com: starting secondarynamenode,
> logging to
> /mnt/ephemeral-hdfs/logs/hadoop-root-secondarynamenode-ip-172-31-24-124.ec2.internal.out
> [timing] ephemeral-hdfs setup: 00h 00m 12s
> Setting up persistent-hdfs
> Pseudo-terminal will not be allocated because stdin is not a terminal.
> Pseudo-terminal will not be allocated because stdin is not a terminal.
> Pseudo-terminal will not be allocated because stdin is not a terminal.
> Pseudo-terminal will not be allocated because stdin is not a terminal.
> Pseudo-terminal will not be allocated because stdin is not a terminal.
> RSYNC'ing /root/persistent-hdfs/conf to slaves...
> ec2-54-165-243-74.compute-1.amazonaws.com
> ec2-54-88-245-107.compute-1.amazonaws.com
> ec2-54-172-29-47.compute-1.amazonaws.com
> ec2-54-165-131-210.compute-1.amazonaws.com
> ec2-54-172-46-184.compute-1.amazonaws.com
> Formatting persistent HDFS namenode...
> Warning: $HADOOP_HOME is deprecated.
>
> 16/01/18 08:12:50 INFO namenode.NameNode: STARTUP_MSG:
> /************************************************************
> STARTUP_MSG: Starting NameNode
> STARTUP_MSG: host = ip-172-31-24-124.ec2.internal/172.31.24.124
> STARTUP_MSG: args = [-format]
> STARTUP_MSG: version = 1.0.4
> STARTUP_MSG: build =
> https://svn.apache.org/repos/asf/hadoop/common/branches/branch-1.0 -r
> 1393290; compiled by 'hortonfo' on Wed Oct 3 05:13:58 UTC 2012
> ************************************************************/
> 16/01/18 08:12:50 INFO util.GSet: VM type = 64-bit
> 16/01/18 08:12:50 INFO util.GSet: 2% max memory = 17.78 MB
> 16/01/18 08:12:50 INFO util.GSet: capacity = 2^21 = 2097152 entries
> 16/01/18 08:12:50 INFO util.GSet: recommended=2097152, actual=2097152
> 16/01/18 08:12:50 INFO namenode.FSNamesystem: fsOwner=root
> 16/01/18 08:12:50 INFO namenode.FSNamesystem: supergroup=supergroup
> 16/01/18 08:12:50 INFO namenode.FSNamesystem: isPermissionEnabled=false
> 16/01/18 08:12:50 INFO namenode.FSNamesystem:
> dfs.block.invalidate.limit=100
> 16/01/18 08:12:50 INFO namenode.FSNamesystem: isAccessTokenEnabled=false
> accessKeyUpdateInterval=0 min(s), accessTokenLifetime=0 min(s)
> 16/01/18 08:12:50 INFO namenode.NameNode: Caching file names occuring more
> than 10 times
> 16/01/18 08:12:50 INFO common.Storage: Image file of size 110 saved in 0
> seconds.
> 16/01/18 08:12:50 INFO common.Storage: Storage directory
> /vol/persistent-hdfs/dfs/name has been successfully formatted.
> 16/01/18 08:12:50 INFO namenode.NameNode: SHUTDOWN_MSG:
> /************************************************************
> SHUTDOWN_MSG: Shutting down NameNode at ip-172-31-24-124.ec2.internal/
> 172.31.24.124
> ************************************************************/
> Persistent HDFS installed, won't start by default...
> [timing] persistent-hdfs setup: 00h 00m 06s
> Setting up spark-standalone
> RSYNC'ing /root/spark/conf to slaves...
> ec2-54-165-243-74.compute-1.amazonaws.com
> ec2-54-88-245-107.compute-1.amazonaws.com
> ec2-54-172-29-47.compute-1.amazonaws.com
> ec2-54-165-131-210.compute-1.amazonaws.com
> ec2-54-172-46-184.compute-1.amazonaws.com
> RSYNC'ing /root/spark-ec2 to slaves...
> ec2-54-165-243-74.compute-1.amazonaws.com
> ec2-54-88-245-107.compute-1.amazonaws.com
> ec2-54-172-29-47.compute-1.amazonaws.com
> ec2-54-165-131-210.compute-1.amazonaws.com
> ec2-54-172-46-184.compute-1.amazonaws.com
> ec2-54-165-243-74.compute-1.amazonaws.com: no
> org.apache.spark.deploy.worker.Worker to stop
> ec2-54-88-245-107.compute-1.amazonaws.com: no
> org.apache.spark.deploy.worker.Worker to stop
> ec2-54-172-29-47.compute-1.amazonaws.com: no
> org.apache.spark.deploy.worker.Worker to stop
> ec2-54-165-131-210.compute-1.amazonaws.com: no
> org.apache.spark.deploy.worker.Worker to stop
> ec2-54-172-46-184.compute-1.amazonaws.com: no
> org.apache.spark.deploy.worker.Worker to stop
> no org.apache.spark.deploy.master.Master to stop
> starting org.apache.spark.deploy.master.Master, logging to
> /root/spark/logs/spark-root-org.apache.spark.deploy.master.Master-1-ip-172-31-24-124.ec2.internal
> crayondata.com.out
> ec2-54-88-245-107.compute-1.amazonaws.com: starting
> org.apache.spark.deploy.worker.Worker, logging to
> /root/spark/logs/spark-root-org.apache.spark.deploy.worker.Worker-1-ip-172-31-30-81.ec2.internal.out
> ec2-54-165-243-74.compute-1.amazonaws.com: starting
> org.apache.spark.deploy.worker.Worker, logging to
> /root/spark/logs/spark-root-org.apache.spark.deploy.worker.Worker-1-ip-172-31-19-61.ec2.internal.out
> ec2-54-172-46-184.compute-1.amazonaws.com: starting
> org.apache.spark.deploy.worker.Worker, logging to
> /root/spark/logs/spark-root-org.apache.spark.deploy.worker.Worker-1-ip-172-31-30-167.ec2.internal.out
> ec2-54-165-131-210.compute-1.amazonaws.com: starting
> org.apache.spark.deploy.worker.Worker, logging to
> /root/spark/logs/spark-root-org.apache.spark.deploy.worker.Worker-1-ip-172-31-23-10.ec2.internal.out
> ec2-54-172-29-47.compute-1.amazonaws.com: starting
> org.apache.spark.deploy.worker.Worker, logging to
> /root/spark/logs/spark-root-org.apache.spark.deploy.worker.Worker-1-ip-172-31-29-54.ec2.internal.out
> [timing] spark-standalone setup: 00h 00m 39s
> Setting up tachyon
> RSYNC'ing /root/tachyon to slaves...
> ec2-54-165-243-74.compute-1.amazonaws.com
> ec2-54-88-245-107.compute-1.amazonaws.com
> ec2-54-172-29-47.compute-1.amazonaws.com
> ec2-54-165-131-210.compute-1.amazonaws.com
> ec2-54-172-46-184.compute-1.amazonaws.com
> ./tachyon/setup.sh: line 5: /root/tachyon/bin/tachyon: No such file or
> directory
> ./tachyon/setup.sh: line 9: /root/tachyon/bin/tachyon-start.sh: No such
> file or directory
> [timing] tachyon setup: 00h 00m 04s
> Setting up rstudio
> spark-ec2/setup.sh: line 110: ./rstudio/setup.sh: No such file or directory
> [timing] rstudio setup: 00h 00m 00s
> Setting up ganglia
> RSYNC'ing /etc/ganglia to slaves...
> ec2-54-165-243-74.compute-1.amazonaws.com
> ec2-54-88-245-107.compute-1.amazonaws.com
> ec2-54-172-29-47.compute-1.amazonaws.com
> ec2-54-165-131-210.compute-1.amazonaws.com
> ec2-54-172-46-184.compute-1.amazonaws.com
> Shutting down GANGLIA gmond: [FAILED]
> Starting GANGLIA gmond: [ OK ]
> Shutting down GANGLIA gmond: [FAILED]
> Starting GANGLIA gmond: [ OK ]
> Connection to ec2-54-165-243-74.compute-1.amazonaws.com closed.
> Shutting down GANGLIA gmond: [FAILED]
> Starting GANGLIA gmond: [ OK ]
> Connection to ec2-54-88-245-107.compute-1.amazonaws.com closed.
> Shutting down GANGLIA gmond: [FAILED]
> Starting GANGLIA gmond: [ OK ]
> Connection to ec2-54-172-29-47.compute-1.amazonaws.com closed.
> Shutting down GANGLIA gmond: [FAILED]
> Starting GANGLIA gmond: [ OK ]
> Connection to ec2-54-165-131-210.compute-1.amazonaws.com closed.
> Shutting down GANGLIA gmond: [FAILED]
> Starting GANGLIA gmond: [ OK ]
> Connection to ec2-54-172-46-184.compute-1.amazonaws.com closed.
> Shutting down GANGLIA gmetad: [FAILED]
> Starting GANGLIA gmetad: [ OK ]
> Stopping httpd: [FAILED]
> Starting httpd: httpd: Syntax error on line 154 of
> /etc/httpd/conf/httpd.conf: Cannot load
> /etc/httpd/modules/mod_authz_core.so into server:
> /etc/httpd/modules/mod_authz_core.so: cannot open shared object file: No
> such file or directory
> [FAILED]
> [timing] ganglia setup: 00h 00m 04s
> Connection to ec2-52-90-186-83.compute-1.amazonaws.com closed.
> Spark standalone cluster started at
> http://ec2-52-90-186-83.compute-1.amazonaws.com:8080
> Ganglia started at
> http://ec2-52-90-186-83.compute-1.amazonaws.com:5080/ganglia
> Done!
>
>
Re: spark 1.6.0 on ec2 doesn't work
Posted by Calvin Jia <ji...@gmail.com>.
Hi Oleg,
The Tachyon related issue should be fixed.
Hope this helps,
Calvin
On Mon, Jan 18, 2016 at 2:51 AM, Oleg Ruchovets <or...@gmail.com>
wrote:
> Hi ,
> I try to follow the spartk 1.6.0 to install spark on EC2.
>
> It doesn't work properly - got exceptions and at the end standalone spark
> cluster installed.
> here is log information:
>
> Any suggestions?
>
> Thanks
> Oleg.
>
> oleg@robinhood:~/install/spark-1.6.0-bin-hadoop2.6/ec2$ ./spark-ec2
> --key-pair=CC-ES-Demo
> --identity-file=/home/oleg/work/entity_extraction_framework/ec2_pem_key/CC-ES-Demo.pem
> --region=us-east-1 --zone=us-east-1a --spot-price=0.05 -s 5
> --spark-version=1.6.0 launch entity-extraction-spark-cluster
> Setting up security groups...
> Searching for existing cluster entity-extraction-spark-cluster in region
> us-east-1...
> Spark AMI: ami-5bb18832
> Launching instances...
> Requesting 5 slaves as spot instances with price $0.050
> Waiting for spot instances to be granted...
> 0 of 5 slaves granted, waiting longer
> 0 of 5 slaves granted, waiting longer
> 0 of 5 slaves granted, waiting longer
> 0 of 5 slaves granted, waiting longer
> 0 of 5 slaves granted, waiting longer
> 0 of 5 slaves granted, waiting longer
> 0 of 5 slaves granted, waiting longer
> 0 of 5 slaves granted, waiting longer
> 0 of 5 slaves granted, waiting longer
> All 5 slaves granted
> Launched master in us-east-1a, regid = r-9384033f
> Waiting for AWS to propagate instance metadata...
> Waiting for cluster to enter 'ssh-ready' state..........
>
> Warning: SSH connection error. (This could be temporary.)
> Host: ec2-52-90-186-83.compute-1.amazonaws.com
> SSH return code: 255
> SSH output: ssh: connect to host ec2-52-90-186-83.compute-1.amazonaws.com
> port 22: Connection refused
>
> .
>
> Warning: SSH connection error. (This could be temporary.)
> Host: ec2-52-90-186-83.compute-1.amazonaws.com
> SSH return code: 255
> SSH output: ssh: connect to host ec2-52-90-186-83.compute-1.amazonaws.com
> port 22: Connection refused
>
> .
>
> Warning: SSH connection error. (This could be temporary.)
> Host: ec2-52-90-186-83.compute-1.amazonaws.com
> SSH return code: 255
> SSH output: ssh: connect to host ec2-52-90-186-83.compute-1.amazonaws.com
> port 22: Connection refused
>
> .
> Cluster is now in 'ssh-ready' state. Waited 442 seconds.
> Generating cluster's SSH key on master...
> Warning: Permanently added 'ec2-52-90-186-83.compute-1.amazonaws.com,52.90.186.83'
> (ECDSA) to the list of known hosts.
> Connection to ec2-52-90-186-83.compute-1.amazonaws.com closed.
> Warning: Permanently added 'ec2-52-90-186-83.compute-1.amazonaws.com,52.90.186.83'
> (ECDSA) to the list of known hosts.
> Transferring cluster's SSH key to slaves...
> ec2-54-165-243-74.compute-1.amazonaws.com
> Warning: Permanently added 'ec2-54-165-243-74.compute-1.amazonaws.com,54.165.243.74'
> (ECDSA) to the list of known hosts.
> ec2-54-88-245-107.compute-1.amazonaws.com
> Warning: Permanently added 'ec2-54-88-245-107.compute-1.amazonaws.com,54.88.245.107'
> (ECDSA) to the list of known hosts.
> ec2-54-172-29-47.compute-1.amazonaws.com
> Warning: Permanently added 'ec2-54-172-29-47.compute-1.amazonaws.com,54.172.29.47'
> (ECDSA) to the list of known hosts.
> ec2-54-165-131-210.compute-1.amazonaws.com
> Warning: Permanently added 'ec2-54-165-131-210.compute-1.amazonaws.com,54.165.131.210'
> (ECDSA) to the list of known hosts.
> ec2-54-172-46-184.compute-1.amazonaws.com
> Warning: Permanently added 'ec2-54-172-46-184.compute-1.amazonaws.com,54.172.46.184'
> (ECDSA) to the list of known hosts.
> Cloning spark-ec2 scripts from
> https://github.com/amplab/spark-ec2/tree/branch-1.5 on master...
> Warning: Permanently added 'ec2-52-90-186-83.compute-1.amazonaws.com,52.90.186.83'
> (ECDSA) to the list of known hosts.
> Cloning into 'spark-ec2'...
> remote: Counting objects: 2068, done.
> remote: Total 2068 (delta 0), reused 0 (delta 0), pack-reused 2068
> Receiving objects: 100% (2068/2068), 349.76 KiB, done.
> Resolving deltas: 100% (796/796), done.
> Connection to ec2-52-90-186-83.compute-1.amazonaws.com closed.
> Deploying files to master...
> Warning: Permanently added 'ec2-52-90-186-83.compute-1.amazonaws.com,52.90.186.83'
> (ECDSA) to the list of known hosts.
> sending incremental file list
> root/spark-ec2/ec2-variables.sh
>
> sent 1,835 bytes received 40 bytes 416.67 bytes/sec
> total size is 1,684 speedup is 0.90
> Running setup on master...
> Warning: Permanently added 'ec2-52-90-186-83.compute-1.amazonaws.com,52.90.186.83'
> (ECDSA) to the list of known hosts.
> Connection to ec2-52-90-186-83.compute-1.amazonaws.com closed.
> Warning: Permanently added 'ec2-52-90-186-83.compute-1.amazonaws.com,52.90.186.83'
> (ECDSA) to the list of known hosts.
> Setting up Spark on ip-172-31-24-124.ec2.internal...
> Setting executable permissions on scripts...
> RSYNC'ing /root/spark-ec2 to other cluster nodes...
> ec2-54-165-243-74.compute-1.amazonaws.com
> Warning: Permanently added 'ec2-54-165-243-74.compute-1.amazonaws.com,172.31.19.61'
> (ECDSA) to the list of known hosts.
> ec2-54-88-245-107.compute-1.amazonaws.com
> id_rsa
>
> 100% 1679 1.6KB/s 00:00
> Warning: Permanently added 'ec2-54-88-245-107.compute-1.amazonaws.com,172.31.30.81'
> (ECDSA) to the list of known hosts.
> ec2-54-172-29-47.compute-1.amazonaws.com
> id_rsa
>
> 100% 1679 1.6KB/s 00:00
> Warning: Permanently added 'ec2-54-172-29-47.compute-1.amazonaws.com,172.31.29.54'
> (ECDSA) to the list of known hosts.
> ec2-54-165-131-210.compute-1.amazonaws.com
> Warning: Permanently added 'ec2-54-165-131-210.compute-1.amazonaws.com,172.31.23.10'
> (ECDSA) to the list of known hosts.
> id_rsa
>
> 100% 1679 1.6KB/s 00:00
> ec2-54-172-46-184.compute-1.amazonaws.com
> Warning: Permanently added 'ec2-54-172-46-184.compute-1.amazonaws.com,172.31.30.167'
> (ECDSA) to the list of known hosts.
> id_rsa
>
> 100% 1679 1.6KB/s 00:00
> id_rsa
>
> 100% 1679 1.6KB/s 00:00
> [timing] rsync /root/spark-ec2: 00h 00m 01s
> Running setup-slave on all cluster nodes to mount filesystems, etc...
> [1] 08:08:10 [SUCCESS] ec2-52-90-186-83.compute-1.amazonaws.com
> checking/fixing resolution of hostname
> Setting up slave on ip-172-31-24-124.ec2.internal... of type m1.large
> 1024+0 records in
> 1024+0 records out
> 1073741824 bytes (1.1 GB) copied, 2.01407 s, 533 MB/s
> mkswap: /mnt/swap: warning: don't erase bootbits sectors
> on whole disk. Use -f to force.
> Setting up swapspace version 1, size = 1048572 KiB
> no label, UUID=b4d25f54-4732-40bb-8086-c78117cb58b2
> Added 1024 MB swap file /mnt/swap
> Stderr: Warning: Permanently added '
> ec2-52-90-186-83.compute-1.amazonaws.com,172.31.24.124' (ECDSA) to the
> list of known hosts.
> Connection to ec2-52-90-186-83.compute-1.amazonaws.com closed.
> [2] 08:08:24 [SUCCESS] ec2-54-165-243-74.compute-1.amazonaws.com
> checking/fixing resolution of hostname
> Setting up slave on ip-172-31-19-61.ec2.internal... of type m1.large
> 1024+0 records in
> 1024+0 records out
> 1073741824 bytes (1.1 GB) copied, 2.11705 s, 507 MB/s
> mkswap: /mnt/swap: warning: don't erase bootbits sectors
> on whole disk. Use -f to force.
> Setting up swapspace version 1, size = 1048572 KiB
> no label, UUID=928041a8-4d48-4c65-94e2-d9f84e14cad9
> Added 1024 MB swap file /mnt/swap
> Stderr: Connection to ec2-54-165-243-74.compute-1.amazonaws.com closed.
> [3] 08:08:27 [SUCCESS] ec2-54-88-245-107.compute-1.amazonaws.com
> checking/fixing resolution of hostname
> Setting up slave on ip-172-31-30-81.ec2.internal... of type m1.large
> 1024+0 records in
> 1024+0 records out
> 1073741824 bytes (1.1 GB) copied, 2.21007 s, 486 MB/s
> mkswap: /mnt/swap: warning: don't erase bootbits sectors
> on whole disk. Use -f to force.
> Setting up swapspace version 1, size = 1048572 KiB
> no label, UUID=1e8c3d4c-7e27-4c35-acae-d83ec2ea9edb
> Added 1024 MB swap file /mnt/swap
> Stderr: Connection to ec2-54-88-245-107.compute-1.amazonaws.com closed.
> [4] 08:08:32 [SUCCESS] ec2-54-172-29-47.compute-1.amazonaws.com
> checking/fixing resolution of hostname
> Setting up slave on ip-172-31-29-54.ec2.internal... of type m1.large
> 1024+0 records in
> 1024+0 records out
> 1073741824 bytes (1.1 GB) copied, 2.15544 s, 498 MB/s
> mkswap: /mnt/swap: warning: don't erase bootbits sectors
> on whole disk. Use -f to force.
> Setting up swapspace version 1, size = 1048572 KiB
> no label, UUID=7bd81d33-ae22-4973-810e-855535ecb743
> Added 1024 MB swap file /mnt/swap
> Stderr: Connection to ec2-54-172-29-47.compute-1.amazonaws.com closed.
> [5] 08:08:34 [SUCCESS] ec2-54-165-131-210.compute-1.amazonaws.com
> checking/fixing resolution of hostname
> Setting up slave on ip-172-31-23-10.ec2.internal... of type m1.large
> 1024+0 records in
> 1024+0 records out
> 1073741824 bytes (1.1 GB) copied, 2.39186 s, 449 MB/s
> mkswap: /mnt/swap: warning: don't erase bootbits sectors
> on whole disk. Use -f to force.
> Setting up swapspace version 1, size = 1048572 KiB
> no label, UUID=abbdbe4d-f8e8-469b-90d2-c9d0a244b261
> Added 1024 MB swap file /mnt/swap
> Stderr: Connection to ec2-54-165-131-210.compute-1.amazonaws.com closed.
> [6] 08:08:37 [SUCCESS] ec2-54-172-46-184.compute-1.amazonaws.com
> checking/fixing resolution of hostname
> Setting up slave on ip-172-31-30-167.ec2.internal... of type m1.large
> 1024+0 records in
> 1024+0 records out
> 1073741824 bytes (1.1 GB) copied, 2.1603 s, 497 MB/s
> mkswap: /mnt/swap: warning: don't erase bootbits sectors
> on whole disk. Use -f to force.
> Setting up swapspace version 1, size = 1048572 KiB
> no label, UUID=115ac0e9-c28c-4404-a648-826ece20815d
> Added 1024 MB swap file /mnt/swap
> Stderr: Connection to ec2-54-172-46-184.compute-1.amazonaws.com closed.
> [timing] setup-slave: 00h 00m 45s
> Initializing scala
> Unpacking Scala
> --2016-01-18 08:08:37--
> http://s3.amazonaws.com/spark-related-packages/scala-2.10.3.tgz
> Resolving s3.amazonaws.com (s3.amazonaws.com)... 54.231.13.224
> Connecting to s3.amazonaws.com (s3.amazonaws.com)|54.231.13.224|:80...
> connected.
> HTTP request sent, awaiting response... 200 OK
> Length: 30531249 (29M) [application/x-compressed]
> Saving to: ‘scala-2.10.3.tgz’
>
> 100%[============================================================================================================================================>]
> 30,531,249 3.46MB/s in 10s
>
> 2016-01-18 08:08:47 (2.86 MB/s) - ‘scala-2.10.3.tgz’ saved
> [30531249/30531249]
>
> [timing] scala init: 00h 00m 11s
> Initializing spark
> --2016-01-18 08:08:48--
> http://s3.amazonaws.com/spark-related-packages/spark-1.6.0-bin-hadoop1.tgz
> Resolving s3.amazonaws.com (s3.amazonaws.com)... 54.231.81.220
> Connecting to s3.amazonaws.com (s3.amazonaws.com)|54.231.81.220|:80...
> connected.
> HTTP request sent, awaiting response... 200 OK
> Length: 243448482 (232M) [application/x-compressed]
> Saving to: ‘spark-1.6.0-bin-hadoop1.tgz’
>
> 100%[============================================================================================================================================>]
> 243,448,482 65.6MB/s in 3.5s
>
> 2016-01-18 08:08:52 (65.6 MB/s) - ‘spark-1.6.0-bin-hadoop1.tgz’ saved
> [243448482/243448482]
>
> Unpacking Spark
> [timing] spark init: 00h 00m 08s
> Initializing ephemeral-hdfs
> --2016-01-18 08:08:56--
> http://s3.amazonaws.com/spark-related-packages/hadoop-1.0.4.tar.gz
> Resolving s3.amazonaws.com (s3.amazonaws.com)... 54.231.17.48
> Connecting to s3.amazonaws.com (s3.amazonaws.com)|54.231.17.48|:80...
> connected.
> HTTP request sent, awaiting response... 200 OK
> Length: 62793050 (60M) [application/x-gzip]
> Saving to: ‘hadoop-1.0.4.tar.gz’
>
> 100%[============================================================================================================================================>]
> 62,793,050 69.2MB/s in 0.9s
>
> 2016-01-18 08:08:57 (69.2 MB/s) - ‘hadoop-1.0.4.tar.gz’ saved
> [62793050/62793050]
>
> Unpacking Hadoop
> RSYNC'ing /root/ephemeral-hdfs to slaves...
> ec2-54-165-243-74.compute-1.amazonaws.com
> Warning: Permanently added 'ec2-54-165-243-74.compute-1.amazonaws.com,172.31.19.61'
> (ECDSA) to the list of known hosts.
> ec2-54-88-245-107.compute-1.amazonaws.com
> Warning: Permanently added 'ec2-54-88-245-107.compute-1.amazonaws.com,172.31.30.81'
> (ECDSA) to the list of known hosts.
> ec2-54-172-29-47.compute-1.amazonaws.com
> Warning: Permanently added 'ec2-54-172-29-47.compute-1.amazonaws.com,172.31.29.54'
> (ECDSA) to the list of known hosts.
> ec2-54-165-131-210.compute-1.amazonaws.com
> Warning: Permanently added 'ec2-54-165-131-210.compute-1.amazonaws.com,172.31.23.10'
> (ECDSA) to the list of known hosts.
> ec2-54-172-46-184.compute-1.amazonaws.com
> Warning: Permanently added 'ec2-54-172-46-184.compute-1.amazonaws.com,172.31.30.167'
> (ECDSA) to the list of known hosts.
> [timing] ephemeral-hdfs init: 00h 00m 54s
> Initializing persistent-hdfs
> --2016-01-18 08:09:50--
> http://s3.amazonaws.com/spark-related-packages/hadoop-1.0.4.tar.gz
> Resolving s3.amazonaws.com (s3.amazonaws.com)... 54.231.49.236
> Connecting to s3.amazonaws.com (s3.amazonaws.com)|54.231.49.236|:80...
> connected.
> HTTP request sent, awaiting response... 200 OK
> Length: 62793050 (60M) [application/x-gzip]
> Saving to: ‘hadoop-1.0.4.tar.gz’
>
> 100%[============================================================================================================================================>]
> 62,793,050 67.4MB/s in 0.9s
>
> 2016-01-18 08:09:51 (67.4 MB/s) - ‘hadoop-1.0.4.tar.gz’ saved
> [62793050/62793050]
>
> Unpacking Hadoop
> RSYNC'ing /root/persistent-hdfs to slaves...
> ec2-54-165-243-74.compute-1.amazonaws.com
> ec2-54-88-245-107.compute-1.amazonaws.com
> ec2-54-172-29-47.compute-1.amazonaws.com
> ec2-54-165-131-210.compute-1.amazonaws.com
> ec2-54-172-46-184.compute-1.amazonaws.com
> [timing] persistent-hdfs init: 00h 00m 39s
> Initializing spark-standalone
> [timing] spark-standalone init: 00h 00m 00s
> Initializing tachyon
> --2016-01-18 08:10:29--
> https://s3.amazonaws.com/Tachyon/tachyon-0.8.2-bin.tar.gz
> Resolving s3.amazonaws.com (s3.amazonaws.com)... 54.231.81.67
> Connecting to s3.amazonaws.com (s3.amazonaws.com)|54.231.81.67|:443...
> connected.
> HTTP request sent, awaiting response... 403 Forbidden
> 2016-01-18 08:10:29 ERROR 403: Forbidden.
>
> ERROR: Unknown Tachyon version
> tachyon/init.sh: line 60: return: -1: invalid option
> return: usage: return [n]
> Unpacking Tachyon
> tar (child): tachyon-*.tar.gz: Cannot open: No such file or directory
> tar (child): Error is not recoverable: exiting now
> tar: Child returned status 2
> tar: Error is not recoverable: exiting now
> rm: cannot remove `tachyon-*.tar.gz': No such file or directory
> ls: cannot access tachyon-*: No such file or directory
> mv: missing destination file operand after `tachyon'
> Try `mv --help' for more information.
> [timing] tachyon init: 00h 00m 00s
> Initializing rstudio
> --2016-01-18 08:10:29--
> http://download2.rstudio.org/rstudio-server-rhel-0.99.446-x86_64.rpm
> Resolving download2.rstudio.org (download2.rstudio.org)... 54.192.18.169,
> 54.192.18.246, 54.192.18.133, ...
> Connecting to download2.rstudio.org (download2.rstudio.org)|54.192.18.169|:80...
> connected.
> HTTP request sent, awaiting response... 200 OK
> Length: 35035164 (33M) [application/x-redhat-package-manager]
> Saving to: ‘rstudio-server-rhel-0.99.446-x86_64.rpm’
>
> 100%[============================================================================================================================================>]
> 35,035,164 84.0MB/s in 0.4s
>
> 2016-01-18 08:10:29 (84.0 MB/s) -
> ‘rstudio-server-rhel-0.99.446-x86_64.rpm’ saved [35035164/35035164]
>
> Loaded plugins: priorities, update-motd, upgrade-helper
> Examining rstudio-server-rhel-0.99.446-x86_64.rpm:
> rstudio-server-0.99.446-1.x86_64
> Marking rstudio-server-rhel-0.99.446-x86_64.rpm to be installed
> Resolving Dependencies
> --> Running transaction check
> ---> Package rstudio-server.x86_64 0:0.99.446-1 will be installed
> --> Finished Dependency Resolution
>
> Dependencies Resolved
>
>
> ======================================================================================================================================================================================
> Package Arch
> Version Repository
> Size
>
> ======================================================================================================================================================================================
> Installing:
> rstudio-server x86_64
> 0.99.446-1 /rstudio-server-rhel-0.99.446-x86_64
> 252 M
>
> Transaction Summary
>
> ======================================================================================================================================================================================
> Install 1 Package
>
> Total size: 252 M
> Installed size: 252 M
> Downloading packages:
> Running transaction check
> Running transaction test
> Transaction test succeeded
> Running transaction
> Installing : rstudio-server-0.99.446-1.x86_64
>
> 1/1
> groupadd: group 'rstudio-server' already exists
> rsession: no process killed
> rstudio-server start/running, process 2535
> Verifying : rstudio-server-0.99.446-1.x86_64
>
> 1/1
>
> Installed:
> rstudio-server.x86_64 0:0.99.446-1
>
>
>
> Complete!
> rstudio-server start/running, process 2570
> [timing] rstudio init: 00h 00m 39s
> Initializing ganglia
> Connection to ec2-54-165-243-74.compute-1.amazonaws.com closed.
> Connection to ec2-54-88-245-107.compute-1.amazonaws.com closed.
> Connection to ec2-54-172-29-47.compute-1.amazonaws.com closed.
> Connection to ec2-54-165-131-210.compute-1.amazonaws.com closed.
> Connection to ec2-54-172-46-184.compute-1.amazonaws.com closed.
> [timing] ganglia init: 00h 00m 02s
> Creating local config files...
> Connection to ec2-54-165-243-74.compute-1.amazonaws.com closed.
> Connection to ec2-54-165-243-74.compute-1.amazonaws.com closed.
> Configuring /etc/ganglia/gmond.conf
> Configuring /etc/ganglia/gmetad.conf
> Configuring /etc/httpd/conf.d/ganglia.conf
> Configuring /etc/httpd/conf/httpd.conf
> Configuring /root/mapreduce/hadoop.version
> Configuring /root/mapreduce/conf/core-site.xml
> Configuring /root/mapreduce/conf/slaves
> Configuring /root/mapreduce/conf/mapred-site.xml
> Configuring /root/mapreduce/conf/hdfs-site.xml
> Configuring /root/mapreduce/conf/hadoop-env.sh
> Configuring /root/mapreduce/conf/masters
> Configuring /root/persistent-hdfs/conf/core-site.xml
> Configuring /root/persistent-hdfs/conf/slaves
> Configuring /root/persistent-hdfs/conf/mapred-site.xml
> Configuring /root/persistent-hdfs/conf/hdfs-site.xml
> Configuring /root/persistent-hdfs/conf/hadoop-env.sh
> Configuring /root/persistent-hdfs/conf/masters
> Configuring /root/ephemeral-hdfs/conf/core-site.xml
> Configuring /root/ephemeral-hdfs/conf/yarn-site.xml
> Configuring /root/ephemeral-hdfs/conf/slaves
> Configuring /root/ephemeral-hdfs/conf/mapred-site.xml
> Configuring /root/ephemeral-hdfs/conf/hadoop-metrics2.properties
> Configuring /root/ephemeral-hdfs/conf/capacity-scheduler.xml
> Configuring /root/ephemeral-hdfs/conf/yarn-env.sh
> Configuring /root/ephemeral-hdfs/conf/hdfs-site.xml
> Configuring /root/ephemeral-hdfs/conf/hadoop-env.sh
> Configuring /root/ephemeral-hdfs/conf/masters
> Configuring /root/spark/conf/core-site.xml
> Configuring /root/spark/conf/spark-defaults.conf
> Configuring /root/spark/conf/spark-env.sh
> Configuring /root/tachyon/conf/slaves
> Configuring /root/tachyon/conf/workers
> Configuring /root/tachyon/conf/tachyon-env.sh
> Deploying Spark config files...
> RSYNC'ing /root/spark/conf to slaves...
> ec2-54-165-243-74.compute-1.amazonaws.com
> ec2-54-88-245-107.compute-1.amazonaws.com
> ec2-54-172-29-47.compute-1.amazonaws.com
> ec2-54-165-131-210.compute-1.amazonaws.com
> ec2-54-172-46-184.compute-1.amazonaws.com
> Setting up scala
> RSYNC'ing /root/scala to slaves...
> ec2-54-165-243-74.compute-1.amazonaws.com
> ec2-54-88-245-107.compute-1.amazonaws.com
> ec2-54-172-29-47.compute-1.amazonaws.com
> ec2-54-165-131-210.compute-1.amazonaws.com
> ec2-54-172-46-184.compute-1.amazonaws.com
> [timing] scala setup: 00h 00m 09s
> Setting up spark
> RSYNC'ing /root/spark to slaves...
> ec2-54-165-243-74.compute-1.amazonaws.com
> ec2-54-88-245-107.compute-1.amazonaws.com
> ec2-54-172-29-47.compute-1.amazonaws.com
> ec2-54-165-131-210.compute-1.amazonaws.com
> ec2-54-172-46-184.compute-1.amazonaws.com
> [timing] spark setup: 00h 01m 07s
> Setting up ephemeral-hdfs
> ec2-54-165-243-74.compute-1.amazonaws.com
> ec2-54-88-245-107.compute-1.amazonaws.com
> Connection to ec2-54-165-243-74.compute-1.amazonaws.com closed.
> ec2-54-172-29-47.compute-1.amazonaws.com
> Connection to ec2-54-88-245-107.compute-1.amazonaws.com closed.
> Connection to ec2-54-172-29-47.compute-1.amazonaws.com closed.
> ec2-54-165-131-210.compute-1.amazonaws.com
> Connection to ec2-54-165-131-210.compute-1.amazonaws.com closed.
> ec2-54-172-46-184.compute-1.amazonaws.com
> Connection to ec2-54-172-46-184.compute-1.amazonaws.com closed.
> RSYNC'ing /root/ephemeral-hdfs/conf to slaves...
> ec2-54-165-243-74.compute-1.amazonaws.com
> ec2-54-88-245-107.compute-1.amazonaws.com
> ec2-54-172-29-47.compute-1.amazonaws.com
> ec2-54-165-131-210.compute-1.amazonaws.com
> ec2-54-172-46-184.compute-1.amazonaws.com
> Formatting ephemeral HDFS namenode...
> Warning: $HADOOP_HOME is deprecated.
>
> 16/01/18 08:12:39 INFO namenode.NameNode: STARTUP_MSG:
> /************************************************************
> STARTUP_MSG: Starting NameNode
> STARTUP_MSG: host = ip-172-31-24-124.ec2.internal/172.31.24.124
> STARTUP_MSG: args = [-format]
> STARTUP_MSG: version = 1.0.4
> STARTUP_MSG: build =
> https://svn.apache.org/repos/asf/hadoop/common/branches/branch-1.0 -r
> 1393290; compiled by 'hortonfo' on Wed Oct 3 05:13:58 UTC 2012
> ************************************************************/
> 16/01/18 08:12:39 INFO util.GSet: VM type = 64-bit
> 16/01/18 08:12:39 INFO util.GSet: 2% max memory = 17.78 MB
> 16/01/18 08:12:39 INFO util.GSet: capacity = 2^21 = 2097152 entries
> 16/01/18 08:12:39 INFO util.GSet: recommended=2097152, actual=2097152
> 16/01/18 08:12:39 INFO namenode.FSNamesystem: fsOwner=root
> 16/01/18 08:12:39 INFO namenode.FSNamesystem: supergroup=supergroup
> 16/01/18 08:12:39 INFO namenode.FSNamesystem: isPermissionEnabled=false
> 16/01/18 08:12:39 INFO namenode.FSNamesystem:
> dfs.block.invalidate.limit=100
> 16/01/18 08:12:39 INFO namenode.FSNamesystem: isAccessTokenEnabled=false
> accessKeyUpdateInterval=0 min(s), accessTokenLifetime=0 min(s)
> 16/01/18 08:12:39 INFO namenode.NameNode: Caching file names occuring more
> than 10 times
> 16/01/18 08:12:39 INFO common.Storage: Image file of size 110 saved in 0
> seconds.
> 16/01/18 08:12:39 INFO common.Storage: Storage directory
> /mnt/ephemeral-hdfs/dfs/name has been successfully formatted.
> 16/01/18 08:12:39 INFO namenode.NameNode: SHUTDOWN_MSG:
> /************************************************************
> SHUTDOWN_MSG: Shutting down NameNode at ip-172-31-24-124.ec2.internal/
> 172.31.24.124
> ************************************************************/
> Starting ephemeral HDFS...
> Warning: $HADOOP_HOME is deprecated.
>
> starting namenode, logging to
> /mnt/ephemeral-hdfs/logs/hadoop-root-namenode-ip-172-31-24-124.ec2.internal
> Error: Could not find or load main class crayondata.com.log
> ec2-54-172-29-47.compute-1.amazonaws.com: Warning: $HADOOP_HOME is
> deprecated.
> ec2-54-172-29-47.compute-1.amazonaws.com:
> ec2-54-172-29-47.compute-1.amazonaws.com: starting datanode, logging to
> /mnt/ephemeral-hdfs/logs/hadoop-root-datanode-ip-172-31-29-54.ec2.internal.out
> ec2-54-172-46-184.compute-1.amazonaws.com: Warning: $HADOOP_HOME is
> deprecated.
> ec2-54-172-46-184.compute-1.amazonaws.com:
> ec2-54-172-46-184.compute-1.amazonaws.com: starting datanode, logging to
> /mnt/ephemeral-hdfs/logs/hadoop-root-datanode-ip-172-31-30-167.ec2.internal.out
> ec2-54-165-131-210.compute-1.amazonaws.com: Warning: $HADOOP_HOME is
> deprecated.
> ec2-54-165-131-210.compute-1.amazonaws.com:
> ec2-54-165-131-210.compute-1.amazonaws.com: starting datanode, logging to
> /mnt/ephemeral-hdfs/logs/hadoop-root-datanode-ip-172-31-23-10.ec2.internal.out
> ec2-54-88-245-107.compute-1.amazonaws.com: Warning: $HADOOP_HOME is
> deprecated.
> ec2-54-88-245-107.compute-1.amazonaws.com:
> ec2-54-88-245-107.compute-1.amazonaws.com: starting datanode, logging to
> /mnt/ephemeral-hdfs/logs/hadoop-root-datanode-ip-172-31-30-81.ec2.internal.out
> ec2-54-165-243-74.compute-1.amazonaws.com: Warning: $HADOOP_HOME is
> deprecated.
> ec2-54-165-243-74.compute-1.amazonaws.com:
> ec2-54-165-243-74.compute-1.amazonaws.com: starting datanode, logging to
> /mnt/ephemeral-hdfs/logs/hadoop-root-datanode-ip-172-31-19-61.ec2.internal.out
> ec2-52-90-186-83.compute-1.amazonaws.com: Warning: Permanently added '
> ec2-52-90-186-83.compute-1.amazonaws.com,172.31.24.124' (ECDSA) to the
> list of known hosts.
> ec2-52-90-186-83.compute-1.amazonaws.com: Warning: $HADOOP_HOME is
> deprecated.
> ec2-52-90-186-83.compute-1.amazonaws.com:
> ec2-52-90-186-83.compute-1.amazonaws.com: starting secondarynamenode,
> logging to
> /mnt/ephemeral-hdfs/logs/hadoop-root-secondarynamenode-ip-172-31-24-124.ec2.internal.out
> [timing] ephemeral-hdfs setup: 00h 00m 12s
> Setting up persistent-hdfs
> Pseudo-terminal will not be allocated because stdin is not a terminal.
> Pseudo-terminal will not be allocated because stdin is not a terminal.
> Pseudo-terminal will not be allocated because stdin is not a terminal.
> Pseudo-terminal will not be allocated because stdin is not a terminal.
> Pseudo-terminal will not be allocated because stdin is not a terminal.
> RSYNC'ing /root/persistent-hdfs/conf to slaves...
> ec2-54-165-243-74.compute-1.amazonaws.com
> ec2-54-88-245-107.compute-1.amazonaws.com
> ec2-54-172-29-47.compute-1.amazonaws.com
> ec2-54-165-131-210.compute-1.amazonaws.com
> ec2-54-172-46-184.compute-1.amazonaws.com
> Formatting persistent HDFS namenode...
> Warning: $HADOOP_HOME is deprecated.
>
> 16/01/18 08:12:50 INFO namenode.NameNode: STARTUP_MSG:
> /************************************************************
> STARTUP_MSG: Starting NameNode
> STARTUP_MSG: host = ip-172-31-24-124.ec2.internal/172.31.24.124
> STARTUP_MSG: args = [-format]
> STARTUP_MSG: version = 1.0.4
> STARTUP_MSG: build =
> https://svn.apache.org/repos/asf/hadoop/common/branches/branch-1.0 -r
> 1393290; compiled by 'hortonfo' on Wed Oct 3 05:13:58 UTC 2012
> ************************************************************/
> 16/01/18 08:12:50 INFO util.GSet: VM type = 64-bit
> 16/01/18 08:12:50 INFO util.GSet: 2% max memory = 17.78 MB
> 16/01/18 08:12:50 INFO util.GSet: capacity = 2^21 = 2097152 entries
> 16/01/18 08:12:50 INFO util.GSet: recommended=2097152, actual=2097152
> 16/01/18 08:12:50 INFO namenode.FSNamesystem: fsOwner=root
> 16/01/18 08:12:50 INFO namenode.FSNamesystem: supergroup=supergroup
> 16/01/18 08:12:50 INFO namenode.FSNamesystem: isPermissionEnabled=false
> 16/01/18 08:12:50 INFO namenode.FSNamesystem:
> dfs.block.invalidate.limit=100
> 16/01/18 08:12:50 INFO namenode.FSNamesystem: isAccessTokenEnabled=false
> accessKeyUpdateInterval=0 min(s), accessTokenLifetime=0 min(s)
> 16/01/18 08:12:50 INFO namenode.NameNode: Caching file names occuring more
> than 10 times
> 16/01/18 08:12:50 INFO common.Storage: Image file of size 110 saved in 0
> seconds.
> 16/01/18 08:12:50 INFO common.Storage: Storage directory
> /vol/persistent-hdfs/dfs/name has been successfully formatted.
> 16/01/18 08:12:50 INFO namenode.NameNode: SHUTDOWN_MSG:
> /************************************************************
> SHUTDOWN_MSG: Shutting down NameNode at ip-172-31-24-124.ec2.internal/
> 172.31.24.124
> ************************************************************/
> Persistent HDFS installed, won't start by default...
> [timing] persistent-hdfs setup: 00h 00m 06s
> Setting up spark-standalone
> RSYNC'ing /root/spark/conf to slaves...
> ec2-54-165-243-74.compute-1.amazonaws.com
> ec2-54-88-245-107.compute-1.amazonaws.com
> ec2-54-172-29-47.compute-1.amazonaws.com
> ec2-54-165-131-210.compute-1.amazonaws.com
> ec2-54-172-46-184.compute-1.amazonaws.com
> RSYNC'ing /root/spark-ec2 to slaves...
> ec2-54-165-243-74.compute-1.amazonaws.com
> ec2-54-88-245-107.compute-1.amazonaws.com
> ec2-54-172-29-47.compute-1.amazonaws.com
> ec2-54-165-131-210.compute-1.amazonaws.com
> ec2-54-172-46-184.compute-1.amazonaws.com
> ec2-54-165-243-74.compute-1.amazonaws.com: no
> org.apache.spark.deploy.worker.Worker to stop
> ec2-54-88-245-107.compute-1.amazonaws.com: no
> org.apache.spark.deploy.worker.Worker to stop
> ec2-54-172-29-47.compute-1.amazonaws.com: no
> org.apache.spark.deploy.worker.Worker to stop
> ec2-54-165-131-210.compute-1.amazonaws.com: no
> org.apache.spark.deploy.worker.Worker to stop
> ec2-54-172-46-184.compute-1.amazonaws.com: no
> org.apache.spark.deploy.worker.Worker to stop
> no org.apache.spark.deploy.master.Master to stop
> starting org.apache.spark.deploy.master.Master, logging to
> /root/spark/logs/spark-root-org.apache.spark.deploy.master.Master-1-ip-172-31-24-124.ec2.internal
> crayondata.com.out
> ec2-54-88-245-107.compute-1.amazonaws.com: starting
> org.apache.spark.deploy.worker.Worker, logging to
> /root/spark/logs/spark-root-org.apache.spark.deploy.worker.Worker-1-ip-172-31-30-81.ec2.internal.out
> ec2-54-165-243-74.compute-1.amazonaws.com: starting
> org.apache.spark.deploy.worker.Worker, logging to
> /root/spark/logs/spark-root-org.apache.spark.deploy.worker.Worker-1-ip-172-31-19-61.ec2.internal.out
> ec2-54-172-46-184.compute-1.amazonaws.com: starting
> org.apache.spark.deploy.worker.Worker, logging to
> /root/spark/logs/spark-root-org.apache.spark.deploy.worker.Worker-1-ip-172-31-30-167.ec2.internal.out
> ec2-54-165-131-210.compute-1.amazonaws.com: starting
> org.apache.spark.deploy.worker.Worker, logging to
> /root/spark/logs/spark-root-org.apache.spark.deploy.worker.Worker-1-ip-172-31-23-10.ec2.internal.out
> ec2-54-172-29-47.compute-1.amazonaws.com: starting
> org.apache.spark.deploy.worker.Worker, logging to
> /root/spark/logs/spark-root-org.apache.spark.deploy.worker.Worker-1-ip-172-31-29-54.ec2.internal.out
> [timing] spark-standalone setup: 00h 00m 39s
> Setting up tachyon
> RSYNC'ing /root/tachyon to slaves...
> ec2-54-165-243-74.compute-1.amazonaws.com
> ec2-54-88-245-107.compute-1.amazonaws.com
> ec2-54-172-29-47.compute-1.amazonaws.com
> ec2-54-165-131-210.compute-1.amazonaws.com
> ec2-54-172-46-184.compute-1.amazonaws.com
> ./tachyon/setup.sh: line 5: /root/tachyon/bin/tachyon: No such file or
> directory
> ./tachyon/setup.sh: line 9: /root/tachyon/bin/tachyon-start.sh: No such
> file or directory
> [timing] tachyon setup: 00h 00m 04s
> Setting up rstudio
> spark-ec2/setup.sh: line 110: ./rstudio/setup.sh: No such file or directory
> [timing] rstudio setup: 00h 00m 00s
> Setting up ganglia
> RSYNC'ing /etc/ganglia to slaves...
> ec2-54-165-243-74.compute-1.amazonaws.com
> ec2-54-88-245-107.compute-1.amazonaws.com
> ec2-54-172-29-47.compute-1.amazonaws.com
> ec2-54-165-131-210.compute-1.amazonaws.com
> ec2-54-172-46-184.compute-1.amazonaws.com
> Shutting down GANGLIA gmond: [FAILED]
> Starting GANGLIA gmond: [ OK ]
> Shutting down GANGLIA gmond: [FAILED]
> Starting GANGLIA gmond: [ OK ]
> Connection to ec2-54-165-243-74.compute-1.amazonaws.com closed.
> Shutting down GANGLIA gmond: [FAILED]
> Starting GANGLIA gmond: [ OK ]
> Connection to ec2-54-88-245-107.compute-1.amazonaws.com closed.
> Shutting down GANGLIA gmond: [FAILED]
> Starting GANGLIA gmond: [ OK ]
> Connection to ec2-54-172-29-47.compute-1.amazonaws.com closed.
> Shutting down GANGLIA gmond: [FAILED]
> Starting GANGLIA gmond: [ OK ]
> Connection to ec2-54-165-131-210.compute-1.amazonaws.com closed.
> Shutting down GANGLIA gmond: [FAILED]
> Starting GANGLIA gmond: [ OK ]
> Connection to ec2-54-172-46-184.compute-1.amazonaws.com closed.
> Shutting down GANGLIA gmetad: [FAILED]
> Starting GANGLIA gmetad: [ OK ]
> Stopping httpd: [FAILED]
> Starting httpd: httpd: Syntax error on line 154 of
> /etc/httpd/conf/httpd.conf: Cannot load
> /etc/httpd/modules/mod_authz_core.so into server:
> /etc/httpd/modules/mod_authz_core.so: cannot open shared object file: No
> such file or directory
> [FAILED]
> [timing] ganglia setup: 00h 00m 04s
> Connection to ec2-52-90-186-83.compute-1.amazonaws.com closed.
> Spark standalone cluster started at
> http://ec2-52-90-186-83.compute-1.amazonaws.com:8080
> Ganglia started at
> http://ec2-52-90-186-83.compute-1.amazonaws.com:5080/ganglia
> Done!
>
>