You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@spark.apache.org by WangJianfei <wa...@otcaix.iscas.ac.cn> on 2016/11/10 13:43:22 UTC
If we run sc.textfile(path,xxx) many times, will the elements be
the same in each partition
Hi Devs:
If i run sc.textFile(path,xxx) many times, will the elements be the
same(same element,same order)in each partitions?
My experiment show that it's the same, but which may not cover all the
cases. Thank you!
--
View this message in context: http://apache-spark-developers-list.1001551.n3.nabble.com/If-we-run-sc-textfile-path-xxx-many-times-will-the-elements-be-the-same-in-each-partition-tp19814.html
Sent from the Apache Spark Developers List mailing list archive at Nabble.com.
---------------------------------------------------------------------
To unsubscribe e-mail: dev-unsubscribe@spark.apache.org
Re: If we run sc.textfile(path,xxx) many times, will the elements be
the same in each partition
Posted by Prashant Sharma <sc...@gmail.com>.
+user -dev
Since the same hash based partitioner is in action by default. In my
understanding every time same partitioning will happen.
Thanks,
On Nov 10, 2016 7:13 PM, "WangJianfei" <wa...@otcaix.iscas.ac.cn>
wrote:
> Hi Devs:
> If i run sc.textFile(path,xxx) many times, will the elements be the
> same(same element,same order)in each partitions?
> My experiment show that it's the same, but which may not cover all the
> cases. Thank you!
>
>
>
> --
> View this message in context: http://apache-spark-
> developers-list.1001551.n3.nabble.com/If-we-run-sc-
> textfile-path-xxx-many-times-will-the-elements-be-the-same-
> in-each-partition-tp19814.html
> Sent from the Apache Spark Developers List mailing list archive at
> Nabble.com.
>
> ---------------------------------------------------------------------
> To unsubscribe e-mail: dev-unsubscribe@spark.apache.org
>
>