You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@spark.apache.org by shahid <sh...@trialx.com> on 2015/10/21 06:57:02 UTC
Can we split partition
Hi
I have a large partition(data skewed) i need to split it to no. of
partitions, repartitioning causes lot of shuffle. Can we do that..?
--
View this message in context: http://apache-spark-user-list.1001560.n3.nabble.com/Can-we-split-partition-tp25151.html
Sent from the Apache Spark User List mailing list archive at Nabble.com.
---------------------------------------------------------------------
To unsubscribe, e-mail: user-unsubscribe@spark.apache.org
For additional commands, e-mail: user-help@spark.apache.org
Re: Can we split partition
Posted by Akhil Das <ak...@sigmoidanalytics.com>.
Did you try coalesce? It doesn't shuffle the data around.
Thanks
Best Regards
On Wed, Oct 21, 2015 at 10:27 AM, shahid <sh...@trialx.com> wrote:
> Hi
>
> I have a large partition(data skewed) i need to split it to no. of
> partitions, repartitioning causes lot of shuffle. Can we do that..?
>
>
>
> --
> View this message in context:
> http://apache-spark-user-list.1001560.n3.nabble.com/Can-we-split-partition-tp25151.html
> Sent from the Apache Spark User List mailing list archive at Nabble.com.
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: user-unsubscribe@spark.apache.org
> For additional commands, e-mail: user-help@spark.apache.org
>
>