You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@spark.apache.org by Haripriya Ayyalasomayajula <ah...@gmail.com> on 2015/07/28 05:32:54 UTC
Spark on Mesos - Shut down failed while running spark-shell
Hi all,
I am running Spark 1.4.1 on mesos 0.23.0
While I am able to start spark-shell on the node with mesos-master running,
it works fine. But when I try to start spark-shell on mesos-slave nodes,
I'm encounter this error. I greatly appreciate any help.
15/07/27 22:14:44 INFO Utils: Successfully started service 'SparkUI' on
port 4040.
15/07/27 22:14:44 INFO SparkUI: Started SparkUI at http://10.142.0.140:4040
Warning: MESOS_NATIVE_LIBRARY is deprecated, use MESOS_NATIVE_JAVA_LIBRARY
instead. Future releases will not support JNI bindings via
MESOS_NATIVE_LIBRARY.
Warning: MESOS_NATIVE_LIBRARY is deprecated, use MESOS_NATIVE_JAVA_LIBRARY
instead. Future releases will not support JNI bindings via
MESOS_NATIVE_LIBRARY.
WARNING: Logging before InitGoogleLogging() is written to STDERR
W0727 22:14:45.091286 33441 sched.cpp:1326]
**************************************************
Scheduler driver bound to loopback interface! Cannot communicate with
remote master(s). You might want to set 'LIBPROCESS_IP' environment
variable to use a routable IP address.
**************************************************
2015-07-27 22:14:45,091:33222(0x7fff9e1fc700):ZOO_INFO@log_env@712: Client
environment:zookeeper.version=zookeeper C client 3.4.5
2015-07-27 22:14:45,091:33222(0x7fff9e1fc700):ZOO_INFO@log_env@716: Client
environment:host.name=nid00011
I0727 22:14:45.091995 33441 sched.cpp:157] Version: 0.23.0
2015-07-27 22:14:45,092:33222(0x7fff9e1fc700):ZOO_INFO@log_env@723: Client
environment:os.name=Linux
2015-07-27 22:14:45,092:33222(0x7fff9e1fc700):ZOO_INFO@log_env@724: Client
environment:os.arch=2.6.32-431.el6_1.0000.8785-cray_ari_athena_c_cos
2015-07-27 22:14:45,092:33222(0x7fff9e1fc700):ZOO_INFO@log_env@725: Client
environment:os.version=#1 SMP Wed Jun 24 19:34:50 UTC 2015
2015-07-27 22:14:45,092:33222(0x7fff9e1fc700):ZOO_INFO@log_env@733: Client
environment:user.name=root
2015-07-27 22:14:45,092:33222(0x7fff9e1fc700):ZOO_INFO@log_env@741: Client
environment:user.home=/root
2015-07-27 22:14:45,092:33222(0x7fff9e1fc700):ZOO_INFO@log_env@753: Client
environment:user.dir=/opt/spark-1.4.1/spark-source
2015-07-27 22:14:45,092:33222(0x7fff9e1fc700):ZOO_INFO@zookeeper_init@786:
Initiating client connection, host=192.168.0.10:2181 sessionTimeout=10000
watcher=0x7fffb561a8e0 sessionId=0
sessionPasswd=<null>context=0x7ffdd0000930 flags=0
2015-07-27 22:14:45,092:33222(0x7fff6ebfd700):ZOO_INFO@check_events@1703:
initiated connection to server [192.168.0.10:2181]
2015-07-27 22:14:45,096:33222(0x7fff6ebfd700):ZOO_INFO@check_events@1750:
session establishment complete on server [192.168.0.10:2181],
sessionId=0x14ed296a0fd000a, negotiated timeout=10000
I0727 22:14:45.096891 33479 group.cpp:313] Group process (group(1)@
127.0.0.1:45546) connected to ZooKeeper
I0727 22:14:45.096914 33479 group.cpp:787] Syncing group operations: queue
size (joins, cancels, datas) = (0, 0, 0)
I0727 22:14:45.096923 33479 group.cpp:385] Trying to create path '/mesos'
in ZooKeeper
I0727 22:14:45.099181 33471 detector.cpp:138] Detected a new leader:
(id='4')
I0727 22:14:45.099298 33483 group.cpp:656] Trying to get
'/mesos/info_0000000004' in ZooKeeper
W0727 22:14:45.100443 33453 detector.cpp:444] Leading master
master@127.0.0.1:5050 is using a Protobuf binary format when registering
with ZooKeeper (info): this will be deprecated as of Mesos 0.24 (see
MESOS-2340)
I0727 22:14:45.100544 33453 detector.cpp:481] A new leading master (UPID=
master@127.0.0.1:5050) is detected
I0727 22:14:45.100739 33478 sched.cpp:254] New master detected at
master@127.0.0.1:5050
I0727 22:14:45.101104 33478 sched.cpp:264] No credentials provided.
Attempting to register without authentication
E0727 22:14:45.101210 33490 socket.hpp:107] Shutdown failed on fd=88:
Transport endpoint is not connected [107]
E0727 22:14:45.101380 33490 socket.hpp:107] Shutdown failed on fd=89:
Transport endpoint is not connected [107]
E0727 22:14:46.643348 33490 socket.hpp:107] Shutdown failed on fd=88:
Transport endpoint is not connected [107]
E0727 22:14:47.111336 33490 socket.hpp:107] Shutdown failed on fd=88:
Transport endpoint is not connected [107]
15/07/27 22:14:50 INFO DiskBlockManager: Shutdown hook called
15/07/27 22:14:50 INFO Utils: path =
/tmp/spark-3f94442b-7873-463f-91dd-3ee62ed5b263/blockmgr-74a5ed25-025b-4186-b1d8-dc395f287a8f,
already present as root for deletion.
15/07/27 22:14:50 INFO Utils: Shutdown hook called
15/07/27 22:14:50 INFO Utils: Deleting directory
/tmp/spark-3f94442b-7873-463f-91dd-3ee62ed5b263/httpd-5d2a71e5-1d36-47f7-b122-31f1dd12a0f0
15/07/27 22:14:50 INFO Utils: Deleting directory
/tmp/spark-3f94442b-7873-463f-91dd-3ee62ed5b263
15/07/27 22:14:50 INFO Utils: Deleting directory
/tmp/spark-bfd6c444-5346-4315-9501-1baed4d500de
--
Regards,
Haripriya Ayyalasomayajula
Re: Spark on Mesos - Shut down failed while running spark-shell
Posted by Tim Chen <ti...@mesosphere.io>.
Hi Haripriya,
Your master has registered it's public ip to be 127.0.0.1:5050 which won't
be able to be reached by the slave node.
If mesos didn't pick up the right ip you can specifiy one yourself via the
--ip flag.
Tim
On Mon, Jul 27, 2015 at 8:32 PM, Haripriya Ayyalasomayajula <
aharipriya92@gmail.com> wrote:
> Hi all,
>
> I am running Spark 1.4.1 on mesos 0.23.0
>
>
> While I am able to start spark-shell on the node with mesos-master
> running, it works fine. But when I try to start spark-shell on mesos-slave
> nodes, I'm encounter this error. I greatly appreciate any help.
>
>
>
> 15/07/27 22:14:44 INFO Utils: Successfully started service 'SparkUI' on
> port 4040.
>
> 15/07/27 22:14:44 INFO SparkUI: Started SparkUI at
> http://10.142.0.140:4040
>
> Warning: MESOS_NATIVE_LIBRARY is deprecated, use MESOS_NATIVE_JAVA_LIBRARY
> instead. Future releases will not support JNI bindings via
> MESOS_NATIVE_LIBRARY.
>
> Warning: MESOS_NATIVE_LIBRARY is deprecated, use MESOS_NATIVE_JAVA_LIBRARY
> instead. Future releases will not support JNI bindings via
> MESOS_NATIVE_LIBRARY.
>
> WARNING: Logging before InitGoogleLogging() is written to STDERR
>
> W0727 22:14:45.091286 33441 sched.cpp:1326]
>
> **************************************************
>
> Scheduler driver bound to loopback interface! Cannot communicate with
> remote master(s). You might want to set 'LIBPROCESS_IP' environment
> variable to use a routable IP address.
>
> **************************************************
>
> 2015-07-27 22:14:45,091:33222(0x7fff9e1fc700):ZOO_INFO@log_env@712:
> Client environment:zookeeper.version=zookeeper C client 3.4.5
>
> 2015-07-27 22:14:45,091:33222(0x7fff9e1fc700):ZOO_INFO@log_env@716:
> Client environment:host.name=nid00011
>
> I0727 22:14:45.091995 33441 sched.cpp:157] Version: 0.23.0
>
> 2015-07-27 22:14:45,092:33222(0x7fff9e1fc700):ZOO_INFO@log_env@723:
> Client environment:os.name=Linux
>
> 2015-07-27 22:14:45,092:33222(0x7fff9e1fc700):ZOO_INFO@log_env@724:
> Client environment:os.arch=2.6.32-431.el6_1.0000.8785-cray_ari_athena_c_cos
>
> 2015-07-27 22:14:45,092:33222(0x7fff9e1fc700):ZOO_INFO@log_env@725:
> Client environment:os.version=#1 SMP Wed Jun 24 19:34:50 UTC 2015
>
> 2015-07-27 22:14:45,092:33222(0x7fff9e1fc700):ZOO_INFO@log_env@733:
> Client environment:user.name=root
>
> 2015-07-27 22:14:45,092:33222(0x7fff9e1fc700):ZOO_INFO@log_env@741:
> Client environment:user.home=/root
>
> 2015-07-27 22:14:45,092:33222(0x7fff9e1fc700):ZOO_INFO@log_env@753:
> Client environment:user.dir=/opt/spark-1.4.1/spark-source
>
> 2015-07-27 22:14:45,092:33222(0x7fff9e1fc700):ZOO_INFO@zookeeper_init@786:
> Initiating client connection, host=192.168.0.10:2181 sessionTimeout=10000
> watcher=0x7fffb561a8e0 sessionId=0
> sessionPasswd=<null>context=0x7ffdd0000930 flags=0
>
> 2015-07-27 22:14:45,092:33222(0x7fff6ebfd700):ZOO_INFO@check_events@1703:
> initiated connection to server [192.168.0.10:2181]
>
> 2015-07-27 22:14:45,096:33222(0x7fff6ebfd700):ZOO_INFO@check_events@1750:
> session establishment complete on server [192.168.0.10:2181],
> sessionId=0x14ed296a0fd000a, negotiated timeout=10000
>
> I0727 22:14:45.096891 33479 group.cpp:313] Group process (group(1)@
> 127.0.0.1:45546) connected to ZooKeeper
>
> I0727 22:14:45.096914 33479 group.cpp:787] Syncing group operations: queue
> size (joins, cancels, datas) = (0, 0, 0)
>
> I0727 22:14:45.096923 33479 group.cpp:385] Trying to create path '/mesos'
> in ZooKeeper
>
> I0727 22:14:45.099181 33471 detector.cpp:138] Detected a new leader:
> (id='4')
>
> I0727 22:14:45.099298 33483 group.cpp:656] Trying to get
> '/mesos/info_0000000004' in ZooKeeper
>
> W0727 22:14:45.100443 33453 detector.cpp:444] Leading master
> master@127.0.0.1:5050 is using a Protobuf binary format when registering
> with ZooKeeper (info): this will be deprecated as of Mesos 0.24 (see
> MESOS-2340)
>
> I0727 22:14:45.100544 33453 detector.cpp:481] A new leading master (UPID=
> master@127.0.0.1:5050) is detected
>
> I0727 22:14:45.100739 33478 sched.cpp:254] New master detected at
> master@127.0.0.1:5050
>
> I0727 22:14:45.101104 33478 sched.cpp:264] No credentials provided.
> Attempting to register without authentication
>
> E0727 22:14:45.101210 33490 socket.hpp:107] Shutdown failed on fd=88:
> Transport endpoint is not connected [107]
>
> E0727 22:14:45.101380 33490 socket.hpp:107] Shutdown failed on fd=89:
> Transport endpoint is not connected [107]
>
> E0727 22:14:46.643348 33490 socket.hpp:107] Shutdown failed on fd=88:
> Transport endpoint is not connected [107]
>
> E0727 22:14:47.111336 33490 socket.hpp:107] Shutdown failed on fd=88:
> Transport endpoint is not connected [107]
>
> 15/07/27 22:14:50 INFO DiskBlockManager: Shutdown hook called
>
> 15/07/27 22:14:50 INFO Utils: path =
> /tmp/spark-3f94442b-7873-463f-91dd-3ee62ed5b263/blockmgr-74a5ed25-025b-4186-b1d8-dc395f287a8f,
> already present as root for deletion.
>
> 15/07/27 22:14:50 INFO Utils: Shutdown hook called
>
> 15/07/27 22:14:50 INFO Utils: Deleting directory
> /tmp/spark-3f94442b-7873-463f-91dd-3ee62ed5b263/httpd-5d2a71e5-1d36-47f7-b122-31f1dd12a0f0
>
> 15/07/27 22:14:50 INFO Utils: Deleting directory
> /tmp/spark-3f94442b-7873-463f-91dd-3ee62ed5b263
>
> 15/07/27 22:14:50 INFO Utils: Deleting directory
> /tmp/spark-bfd6c444-5346-4315-9501-1baed4d500de
>
> --
> Regards,
> Haripriya Ayyalasomayajula
>
>