You are viewing a plain text version of this content. The canonical link for it is here.

Posted to issues@spark.apache.org by "Hyukjin Kwon (JIRA)" <ji...@apache.org> on 2017/01/26 14:53:24 UTC

[jira] [Commented] (SPARK-5786) Documentation of Narrow Dependencies

    [ https://issues.apache.org/jira/browse/SPARK-5786?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15839788#comment-15839788 ] 

Hyukjin Kwon commented on SPARK-5786:
-------------------------------------

It seems they are documented, at least, in API docs, e.g., https://github.com/apache/spark/blob/4cb49412d1d7d10ffcc738475928c7de2bc59fd4/core/src/main/scala/org/apache/spark/scheduler/DAGScheduler.scala#L55-L61

> Documentation of Narrow Dependencies
> ------------------------------------
>
>                 Key: SPARK-5786
>                 URL: https://issues.apache.org/jira/browse/SPARK-5786
>             Project: Spark
>          Issue Type: Improvement
>          Components: Documentation
>            Reporter: Imran Rashid
>
> Narrow dependencies can really improve job performance by skipping shuffles entirely.  However aside from being mentioned in some early papers and during some meetups, they aren't explained (or even mentioned) in the docs.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org