You are viewing a plain text version of this content. The canonical link for it is here.

Posted to issues@spark.apache.org by "Shixiong Zhu (JIRA)" <ji...@apache.org> on 2017/07/10 20:45:00 UTC

[jira] [Created] (SPARK-21369) Don't use Scala classes in external shuffle service

Shixiong Zhu created SPARK-21369:
------------------------------------

             Summary: Don't use Scala classes in external shuffle service
                 Key: SPARK-21369
                 URL: https://issues.apache.org/jira/browse/SPARK-21369
             Project: Spark
          Issue Type: Bug
          Components: Shuffle, YARN
    Affects Versions: 2.2.0
            Reporter: Shixiong Zhu
            Assignee: Shixiong Zhu


Right now the external shuffle service uses Scala Tuple2. However, the Scala library won't be shaded into the yarn shuffle assembly jar. Then when the codes are called, it will throw ClassNotFoundException.

Right now it's safe by default because we disabled spark.reducer.maxReqSizeShuffleToMem. However,  to allow using spark.reducer.maxReqSizeShuffleToMem, we should remove all usages of Tuples.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org