what is a partition? how it works?

[Joe L <se...@yahoo.com>]

I want to know as follows:

what is a partition? how it works?
how it is different from hadoop partition?

For example:
>>> sc.parallelize([1,2,3,4]).map(lambda x:
>>> (x,x)).partitionBy(2).glom().collect()
[[(2,2), (4,4)], [(1,1), (3,3)]]

from this, we will get 2 partitions but what does it mean? how do they
reside in memory in the cluster?

I am sorry for such a simple question but I couldn't find any specific
information about what happens underneath partitioning. 

Thank you, Joe



--
View this message in context: http://apache-spark-user-list.1001560.n3.nabble.com/what-is-a-partition-how-it-works-tp4325.html
Sent from the Apache Spark User List mailing list archive at Nabble.com.