You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@spark.apache.org by "Adrian Bridgett (JIRA)" <ji...@apache.org> on 2016/11/28 15:23:58 UTC
[jira] [Commented] (SPARK-18611) mesos shuffle service on v2 isn't
compatible with spark v1
[ https://issues.apache.org/jira/browse/SPARK-18611?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15702213#comment-15702213 ]
Adrian Bridgett commented on SPARK-18611:
-----------------------------------------
dragos spotted this problem in his comment on https://github.com/apache/spark/pull/11272/files FYI.
> mesos shuffle service on v2 isn't compatible with spark v1
> ----------------------------------------------------------
>
> Key: SPARK-18611
> URL: https://issues.apache.org/jira/browse/SPARK-18611
> Project: Spark
> Issue Type: Bug
> Components: Shuffle
> Affects Versions: 1.6.3
> Reporter: Adrian Bridgett
> Priority: Minor
>
> In SPARK-12583 heartbeating was added to the shuffle service.
> This unfortunately changes registerDriver with a different signature so we can't use sparkv1 clients with sparkv2 mesos shuffle service.
> http://apache-spark-developers-list.1001551.n3.nabble.com/YARN-Shuffle-service-and-its-compatibility-td17222.html is a slightly different issue, but it does suggest that it's agreed that compatibility is a good idea in these cases.
> https://github.com/apache/spark/pull/13279 was created a while ago to backport this to 1.6.x but was rejected (too invasive). However I don't believe this issue (v1/v2 compatibility) was recognised back then so perhaps it's time to reconsider?
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org