You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@mesos.apache.org by Rodrick Brown <ro...@orchard-app.com> on 2015/11/03 19:56:57 UTC
spark shuffle service failing to start on slaves
My slave instance runs as user “mesos” and I have the following directive set
$ cat /etc/mesos-slave/switch_user
false
When trying to start start-mesos-shuffle-service.sh process I I’m getting the following execution
Spark Command: /usr/java/jdk1.8.0_60/jre/bin/java -cp /opt/spark-1.5.0-bin-hadoop2.4/sbin/../conf/:/opt/spark-1.5.0-bin-hadoop2.4/lib/spark-assembly-1.5.0-hadoop2.4.0.jar:/opt/spark-1.5.0-bin-hadoop2.4/lib/datanucleus-core-3.2.10.jar:/opt/spark-1.5.0-bin-hadoop2.4/lib/datanucleus-rdbms-3.2.9.jar:/opt/spark-1.5.0-bin-hadoop2.4/lib/datanucleus-api-jdo-3.2.6.jar -Xms1g -Xmx1g org.apache.spark.deploy.mesos.MesosExternalShuffleService
========================================
log4j:WARN No appenders could be found for logger (org.apache.hadoop.metrics2.lib.MutableMetricsFactory).
log4j:WARN Please initialize the log4j system properly.
log4j:WARN See http://logging.apache.org/log4j/1.2/faq.html#noconfig for more info.
Using Spark's default log4j profile: org/apache/spark/log4j-defaults.properties
15/11/03 18:50:39 INFO SecurityManager: Changing view acls to: root
15/11/03 18:50:39 INFO SecurityManager: Changing modify acls to: root
15/11/03 18:50:39 INFO SecurityManager: SecurityManager: authentication disabled; ui acls disabled; users with view permissions: Set(root); users with modify permissions: Set(root)
15/11/03 18:50:40 INFO MesosExternalShuffleService: Starting shuffle service on port 7337 with useSasl = false
Exception in thread "main" java.net.BindException: Address already in use
at sun.nio.ch.Net.bind0(Native Method)
at sun.nio.ch.Net.bind(Net.java:433)
at sun.nio.ch.Net.bind(Net.java:425)
at sun.nio.ch.ServerSocketChannelImpl.bind(ServerSocketChannelImpl.java:223)
at sun.nio.ch.ServerSocketAdaptor.bind(ServerSocketAdaptor.java:74)
at io.netty.channel.socket.nio.NioServerSocketChannel.doBind(NioServerSocketChannel.java:125)
at io.netty.channel.AbstractChannel$AbstractUnsafe.bind(AbstractChannel.java:485)
at io.netty.channel.DefaultChannelPipeline$HeadContext.bind(DefaultChannelPipeline.java:1089)
at io.netty.channel.AbstractChannelHandlerContext.invokeBind(AbstractChannelHandlerContext.java:430)
at io.netty.channel.AbstractChannelHandlerContext.bind(AbstractChannelHandlerContext.java:415)
at io.netty.channel.DefaultChannelPipeline.bind(DefaultChannelPipeline.java:903)
at io.netty.channel.AbstractChannel.bind(AbstractChannel.java:198)
at io.netty.bootstrap.AbstractBootstrap$2.run(AbstractBootstrap.java:348)
at io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:357)
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:357)
at io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:111)
at java.lang.Thread.run(Thread.java:745)
--
<http://www.orchardplatform.com/>
Rodrick Brown / DevOPs Engineer
+1 917 445 6839 / rodrick@orchardplatform.com <ma...@orchardplatform.com>
Orchard Platform
101 5th Avenue, 4th Floor, New York, NY 10003
http://www.orchardplatform.com <http://www.orchardplatform.com/>
Orchard Blog <http://www.orchardplatform.com/blog/> | Marketplace Lending Meetup <http://www.meetup.com/Peer-to-Peer-Lending-P2P/>
--
*NOTICE TO RECIPIENTS*: This communication is confidential and intended for
the use of the addressee only. If you are not an intended recipient of this
communication, please delete it immediately and notify the sender by return
email. Unauthorized reading, dissemination, distribution or copying of this
communication is prohibited. This communication does not constitute an
offer to sell or a solicitation of an indication of interest to purchase
any loan, security or any other financial product or instrument, nor is it
an offer to sell or a solicitation of an indication of interest to purchase
any products or services to any persons who are prohibited from receiving
such information under applicable law. The contents of this communication
may not be accurate or complete and are subject to change without notice.
As such, Orchard App, Inc. (including its subsidiaries and affiliates,
"Orchard") makes no representation regarding the accuracy or completeness
of the information contained herein. The intended recipient is advised to
consult its own professional advisors, including those specializing in
legal, tax and accounting matters. Orchard does not provide legal, tax or
accounting advice.
Re: spark shuffle service failing to start on slaves
Posted by haosdent <ha...@gmail.com>.
Does change spark.shuffle.service.port to a unused port works for you?
On Wed, Nov 4, 2015 at 2:56 AM, Rodrick Brown <ro...@orchard-app.com>
wrote:
> My slave instance runs as user “mesos” and I have the following directive
> set
>
> $ cat /etc/mesos-slave/switch_user
> false
>
> When trying to start start-mesos-shuffle-service.sh process I I’m getting
> the following execution
>
> Spark Command: /usr/java/jdk1.8.0_60/jre/bin/java -cp
> /opt/spark-1.5.0-bin-hadoop2.4/sbin/../conf/:/opt/spark-1.5.0-bin-hadoop2.4/lib/spark-assembly-1.5.0-hadoop2.4.0.jar:/opt/spark-1.5.0-bin-hadoop2.4/lib/datanucleus-core-3.2.10.jar:/opt/spark-1.5.0-bin-hadoop2.4/lib/datanucleus-rdbms-3.2.9.jar:/opt/spark-1.5.0-bin-hadoop2.4/lib/datanucleus-api-jdo-3.2.6.jar
> -Xms1g -Xmx1g org.apache.spark.deploy.mesos.MesosExternalShuffleService
> ========================================
> log4j:WARN No appenders could be found for logger
> (org.apache.hadoop.metrics2.lib.MutableMetricsFactory).
> log4j:WARN Please initialize the log4j system properly.
> log4j:WARN See http://logging.apache.org/log4j/1.2/faq.html#noconfig for
> more info.
> Using Spark's default log4j profile:
> org/apache/spark/log4j-defaults.properties
> 15/11/03 18:50:39 INFO SecurityManager: Changing view acls to: root
> 15/11/03 18:50:39 INFO SecurityManager: Changing modify acls to: root
> 15/11/03 18:50:39 INFO SecurityManager: SecurityManager: authentication
> disabled; ui acls disabled; users with view permissions: Set(root); users
> with modify permissions: Set(root)
> 15/11/03 18:50:40 INFO MesosExternalShuffleService: Starting shuffle
> service on port 7337 with useSasl = false
> Exception in thread "main" java.net.BindException: Address already in use
> at sun.nio.ch.Net.bind0(Native Method)
> at sun.nio.ch.Net.bind(Net.java:433)
> at sun.nio.ch.Net.bind(Net.java:425)
> at
> sun.nio.ch.ServerSocketChannelImpl.bind(ServerSocketChannelImpl.java:223)
> at sun.nio.ch.ServerSocketAdaptor.bind(ServerSocketAdaptor.java:74)
> at
> io.netty.channel.socket.nio.NioServerSocketChannel.doBind(NioServerSocketChannel.java:125)
> at
> io.netty.channel.AbstractChannel$AbstractUnsafe.bind(AbstractChannel.java:485)
> at
> io.netty.channel.DefaultChannelPipeline$HeadContext.bind(DefaultChannelPipeline.java:1089)
> at
> io.netty.channel.AbstractChannelHandlerContext.invokeBind(AbstractChannelHandlerContext.java:430)
> at
> io.netty.channel.AbstractChannelHandlerContext.bind(AbstractChannelHandlerContext.java:415)
> at
> io.netty.channel.DefaultChannelPipeline.bind(DefaultChannelPipeline.java:903)
> at io.netty.channel.AbstractChannel.bind(AbstractChannel.java:198)
> at
> io.netty.bootstrap.AbstractBootstrap$2.run(AbstractBootstrap.java:348)
> at
> io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:357)
> at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:357)
> at
> io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:111)
> at java.lang.Thread.run(Thread.java:745)
>
> --
>
> [image: Orchard Platform] <http://www.orchardplatform.com/>
>
> Rodrick Brown / DevOPs Engineer
> +1 917 445 6839 / rodrick@orchardplatform.com
> <ch...@orchardplatform.com>
>
> Orchard Platform
> 101 5th Avenue, 4th Floor, New York, NY 10003
> http://www.orchardplatform.com
>
> Orchard Blog <http://www.orchardplatform.com/blog/> | Marketplace Lending
> Meetup <http://www.meetup.com/Peer-to-Peer-Lending-P2P/>
>
>
> *NOTICE TO RECIPIENTS*: This communication is confidential and intended
> for the use of the addressee only. If you are not an intended recipient of
> this communication, please delete it immediately and notify the sender by
> return email. Unauthorized reading, dissemination, distribution or copying
> of this communication is prohibited. This communication does not constitute
> an offer to sell or a solicitation of an indication of interest to purchase
> any loan, security or any other financial product or instrument, nor is it
> an offer to sell or a solicitation of an indication of interest to purchase
> any products or services to any persons who are prohibited from receiving
> such information under applicable law. The contents of this communication
> may not be accurate or complete and are subject to change without notice.
> As such, Orchard App, Inc. (including its subsidiaries and affiliates,
> "Orchard") makes no representation regarding the accuracy or completeness
> of the information contained herein. The intended recipient is advised to
> consult its own professional advisors, including those specializing in
> legal, tax and accounting matters. Orchard does not provide legal, tax or
> accounting advice.
>
--
Best Regards,
Haosdent Huang