[jira] [Commented] (SPARK-6028) Provide an alternative RPC implementation based on the network transport module

["Charles Allen (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/SPARK-6028?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15364587#comment-15364587 ] 

Charles Allen commented on SPARK-6028:
--------------------------------------

I just tried out 1.6.1 upgrading from 1.5.2 running spark on mesos. Everything was fine in 1.5.2

None of the mesos backend executors launch anymore due to the following error (reported in failed tasks on the mesos slaves):

{code}
Exception in thread "main" java.lang.reflect.UndeclaredThrowableException
	at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1563)
	at org.apache.spark.deploy.SparkHadoopUtil.runAsSparkUser(SparkHadoopUtil.scala:68)
	at org.apache.spark.executor.CoarseGrainedExecutorBackend$.run(CoarseGrainedExecutorBackend.scala:157)
	at org.apache.spark.executor.CoarseGrainedExecutorBackend$.main(CoarseGrainedExecutorBackend.scala:259)
	at org.apache.spark.executor.CoarseGrainedExecutorBackend.main(CoarseGrainedExecutorBackend.scala)
Caused by: org.apache.spark.SparkException: Invalid Spark URL: akka.tcp://sparkDriver@HOST_REDACTED:43709/user/CoarseGrainedScheduler
	at org.apache.spark.rpc.netty.RpcEndpointAddress$.apply(RpcEndpointAddress.scala:62)
	at org.apache.spark.rpc.netty.NettyRpcEnv.asyncSetupEndpointRefByURI(NettyRpcEnv.scala:140)
	at org.apache.spark.rpc.RpcEnv.setupEndpointRefByURI(RpcEnv.scala:97)
	at org.apache.spark.executor.CoarseGrainedExecutorBackend$$anonfun$run$1.apply$mcV$sp(CoarseGrainedExecutorBackend.scala:171)
	at org.apache.spark.deploy.SparkHadoopUtil$$anon$1.run(SparkHadoopUtil.scala:69)
	at org.apache.spark.deploy.SparkHadoopUtil$$anon$1.run(SparkHadoopUtil.scala:68)
	at java.security.AccessController.doPrivileged(Native Method)
	at javax.security.auth.Subject.doAs(Subject.java:422)
	at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1548)
	... 4 more
{code}

> Provide an alternative RPC implementation based on the network transport module
> -------------------------------------------------------------------------------
>
>                 Key: SPARK-6028
>                 URL: https://issues.apache.org/jira/browse/SPARK-6028
>             Project: Spark
>          Issue Type: Sub-task
>          Components: Spark Core
>            Reporter: Reynold Xin
>            Assignee: Shixiong Zhu
>            Priority: Critical
>             Fix For: 1.6.0
>
>
> Network transport module implements a low level RPC interface. We can build a new RPC implementation on top of that to replace Akka's.
> Design document: https://docs.google.com/document/d/1CF5G6rGVQMKSyV_QKo4D2M-x6rxz5x1Ew7aK3Uq6u8c/edit?usp=sharing



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org