You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@spark.apache.org by "Sean Owen (JIRA)" <ji...@apache.org> on 2017/02/08 11:24:42 UTC

[jira] [Resolved] (SPARK-19513) Split partition without shuffle(means repartition locally)

     [ https://issues.apache.org/jira/browse/SPARK-19513?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Sean Owen resolved SPARK-19513.
-------------------------------
    Resolution: Invalid

[~leocook] please read http://spark.apache.org/contributing.html before opening any JIRAs. I'm going to close this because a lot is wrong here, including 'affecting' a future version and unclear description.

But mostly, this is already possible with a coalesce where shuffle=false.

> Split partition without shuffle(means repartition locally)
> ----------------------------------------------------------
>
>                 Key: SPARK-19513
>                 URL: https://issues.apache.org/jira/browse/SPARK-19513
>             Project: Spark
>          Issue Type: New Feature
>          Components: Spark Core
>    Affects Versions: 3.0.0
>            Reporter: leocook
>
> provide a method, that can split partition without shuffle, just repartition locally.
> Sample: there is a partition with to much data, it will be OOM to spark task, 
> if we increase the num of partition,  we usually call repartition,but it will shuffle.
> Now I want to split partitions locally, just make less data in each partition。
> !http://7xriy2.com1.z0.glb.clouddn.com/spark%20split%20partition%20no%20shuffer.png!



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org