[jira] [Created] (SPARK-21494) Spark 2.2.0 AES encryption not working with External shuffle

["Udit Mehrotra (JIRA)" <ji...@apache.org>]

Udit Mehrotra created SPARK-21494:
-------------------------------------

             Summary: Spark 2.2.0 AES encryption not working with External shuffle
                 Key: SPARK-21494
                 URL: https://issues.apache.org/jira/browse/SPARK-21494
             Project: Spark
          Issue Type: Bug
          Components: Block Manager, Shuffle
    Affects Versions: 2.2.0
         Environment: AWS EMR
            Reporter: Udit Mehrotra


Spark’s new AES based authentication mechanism does not seem to work when configured with external shuffle service on YARN. Here is the stack trace for the error we see in the driver logs:
ERROR YarnScheduler: Lost executor 40 on ip-10-167-104-125.ec2.internal: Unable to create executor due to Unable to register with external shuffle server due to: java.lang.IllegalArgumentException: Authentication failed.
                at org.apache.spark.network.crypto.AuthRpcHandler.receive(AuthRpcHandler.java:125)
                at org.apache.spark.network.server.TransportRequestHandler.processRpcRequest(TransportRequestHandler.java:157)
                at org.apache.spark.network.server.TransportRequestHandler.handle(TransportRequestHandler.java:105)
                at org.apache.spark.network.server.TransportChannelHandler.channelRead(TransportChannelHandler.java:118)
                at org.spark_project.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:357)
                at org.spark_project.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:343)
                at org.spark_project.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:336)
                at org.spark_project.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:287)
                at org.spark_project.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:357)
                at org.spark_project.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:343)
                at org.spark_project.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:336)
                at org.spark_project.io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102)
 
Here are the settings we are configuring in ‘spark-defaults’ and ‘yarn-site’:
spark.network.crypto.enabled true
spark.network.crypto.saslFallback false
spark.authenticate               true
 
Turning on DEBUG logs for class ‘org.apache.spark.network.crypto’ on both Spark and YARN side is not giving much information either about why authentication fails. The driver and node manager logs have been attached to the JIRA.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org