You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@spark.apache.org by "Thomas Graves (JIRA)" <ji...@apache.org> on 2016/07/14 14:43:20 UTC
[jira] [Resolved] (SPARK-16505) YARN shuffle service should throw
errors when it fails to start
[ https://issues.apache.org/jira/browse/SPARK-16505?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Thomas Graves resolved SPARK-16505.
-----------------------------------
Resolution: Fixed
Fix Version/s: 2.1.0
> YARN shuffle service should throw errors when it fails to start
> ---------------------------------------------------------------
>
> Key: SPARK-16505
> URL: https://issues.apache.org/jira/browse/SPARK-16505
> Project: Spark
> Issue Type: Bug
> Components: YARN
> Affects Versions: 2.0.0
> Reporter: Marcelo Vanzin
> Fix For: 2.1.0
>
>
> Right now the YARN shuffle service will swallow errors that happen during startup and just log them:
> {code}
> try {
> blockHandler = new ExternalShuffleBlockHandler(transportConf, registeredExecutorFile);
> } catch (Exception e) {
> logger.error("Failed to initialize external shuffle service", e);
> }
> {code}
> This causes two undesirable things to happen:
> - because {{blockHandler}} will remain {{null}} when an error happens, every request to the shuffle service will cause an NPE
> - because the NM is running, containers may be assigned to that host, only to fail to register with the shuffle service.
> Example of the first:
> {noformat}
> 2016-05-25 15:01:12,198 ERROR org.apache.spark.network.TransportContext: Error while initializing Netty pipeline
> java.lang.NullPointerException
> at org.apache.spark.network.server.TransportRequestHandler.<init>(TransportRequestHandler.java:77)
> at org.apache.spark.network.TransportContext.createChannelHandler(TransportContext.java:159)
> at org.apache.spark.network.TransportContext.initializePipeline(TransportContext.java:135)
> at org.apache.spark.network.server.TransportServer$1.initChannel(TransportServer.java:123)
> at org.apache.spark.network.server.TransportServer$1.initChannel(TransportServer.java:116)
> at io.netty.channel.ChannelInitializer.channelRegistered(ChannelInitializer.java:69)
> {noformat}
> Example of the second:
> {noformat}
> 16/05/25 15:01:12 INFO storage.BlockManager: Registering executor with local external shuffle service.
> 16/05/25 15:01:12 ERROR client.TransportClient: Failed to send RPC 5736508221708472525 to qxhddn01.ascap.com/10.6.41.31:7337: java.nio.channels.ClosedChannelException
> java.nio.channels.ClosedChannelException
> 16/05/25 15:01:12 ERROR storage.BlockManager: Failed to connect to external shuffle server, will retry 2 more times after waiting 5 seconds...
> java.lang.RuntimeException: java.io.IOException: Failed to send RPC 5736508221708472525 to qxhddn01.ascap.com/10.6.41.31:7337: java.nio.channels.ClosedChannelException
> at org.spark-project.guava.base.Throwables.propagate(Throwables.java:160)
> at org.apache.spark.network.client.TransportClient.sendRpcSync(TransportClient.java:272)
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org