You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@spark.apache.org by prabhastechie <pr...@gmail.com> on 2018/02/19 02:03:52 UTC
[SparkQL] how are RDDs partitioned and distributed in a standalone
cluster?
Say I have a main method with the following pseudo-code (to be run on a spark
standalone cluster):
main(args) {
RDD rdd
rdd1 = rdd.map(...)
// some other statements not using RDD
rdd2 = rdd.filter(...)
}
When executed, will each of the two statements involving RDDs (map and
filter) be individually partitioned and distributed on available cluster
nodes? And any statements not involving RDDs (or data frames) will typically
be executed on the driver?
Is that how spark take advantage of the cluster?
--
Sent from: http://apache-spark-user-list.1001560.n3.nabble.com/
---------------------------------------------------------------------
To unsubscribe e-mail: user-unsubscribe@spark.apache.org