You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@spark.apache.org by "Marcelo Vanzin (JIRA)" <ji...@apache.org> on 2016/07/12 20:23:20 UTC
[jira] [Created] (SPARK-16505) YARN shuffle service should throw
errors when it fails to start
Marcelo Vanzin created SPARK-16505:
--------------------------------------
Summary: YARN shuffle service should throw errors when it fails to start
Key: SPARK-16505
URL: https://issues.apache.org/jira/browse/SPARK-16505
Project: Spark
Issue Type: Bug
Components: YARN
Affects Versions: 2.0.0
Reporter: Marcelo Vanzin
Right now the YARN shuffle service will swallow errors that happen during startup and just log them:
{code}
try {
blockHandler = new ExternalShuffleBlockHandler(transportConf, registeredExecutorFile);
} catch (Exception e) {
logger.error("Failed to initialize external shuffle service", e);
}
{code}
This causes two undesirable things to happen:
- because {{blockHandler}} will remain {{null}} when an error happens, every request to the shuffle service will cause an NPE
- because the NM is running, containers may be assigned to that host, only to fail to register with the shuffle service.
Example of the first:
{noformat}
2016-05-25 15:01:12,198 ERROR org.apache.spark.network.TransportContext: Error while initializing Netty pipeline
java.lang.NullPointerException
at org.apache.spark.network.server.TransportRequestHandler.<init>(TransportRequestHandler.java:77)
at org.apache.spark.network.TransportContext.createChannelHandler(TransportContext.java:159)
at org.apache.spark.network.TransportContext.initializePipeline(TransportContext.java:135)
at org.apache.spark.network.server.TransportServer$1.initChannel(TransportServer.java:123)
at org.apache.spark.network.server.TransportServer$1.initChannel(TransportServer.java:116)
at io.netty.channel.ChannelInitializer.channelRegistered(ChannelInitializer.java:69)
{noformat}
Example of the second:
{noformat}
16/05/25 15:01:12 INFO storage.BlockManager: Registering executor with local external shuffle service.
16/05/25 15:01:12 ERROR client.TransportClient: Failed to send RPC 5736508221708472525 to qxhddn01.ascap.com/10.6.41.31:7337: java.nio.channels.ClosedChannelException
java.nio.channels.ClosedChannelException
16/05/25 15:01:12 ERROR storage.BlockManager: Failed to connect to external shuffle server, will retry 2 more times after waiting 5 seconds...
java.lang.RuntimeException: java.io.IOException: Failed to send RPC 5736508221708472525 to qxhddn01.ascap.com/10.6.41.31:7337: java.nio.channels.ClosedChannelException
at org.spark-project.guava.base.Throwables.propagate(Throwables.java:160)
at org.apache.spark.network.client.TransportClient.sendRpcSync(TransportClient.java:272)
{noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org