You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@spark.apache.org by DaPsul <da...@gmx.de> on 2016/02/18 22:37:56 UTC
subtractByKey increases RDD size in memory - any ideas?
(copy from
http://stackoverflow.com/questions/35467128/spark-subtractbykey-increases-rdd-cached-memory-size)
I've found a very strange behavior for RDD's (spark 1.6.0 with scala 2.11):
When i use subtractByKey on an RDD the resulting RDD should be of equal or
smaller size. What i get is an RDD that takes even more space in memory:
//Initialize first RDD
val rdd1 = sc.parallelize(Array((1,1),(2,2),(3,3))).cache()
//dummy action to cache it => size according to webgui: 184 Bytes
rdd1.first
//Initialize RDD to subtract (empty RDD should result in no change for rdd1)
val rdd2 = sc.parallelize(Array[(Int,Int)]())
//perform subtraction
val rdd3 = rdd1.subtractByKey(rdd2).cache()
//dummy action to cache rdd3 => size according to webgui: 208 Bytes
rdd3.first
I frist realized this strange behaviour for an RDD of ~200k rows and size
1.3 GB that scaled up to more than 2 GB after subtraction
Edit: Tried the example above with more values(10k) => same behaviour. The
size increases by ~1.6 times. Also reduceByKey seems to have a similar
effect.
When i create an RDD by
sc.paralellize(rdd3.collect())
the size is the same as for rdd3, so the increased size carries over even if
it's extracted from RDD.
--
View this message in context: http://apache-spark-user-list.1001560.n3.nabble.com/subtractByKey-increases-RDD-size-in-memory-any-ideas-tp26272.html
Sent from the Apache Spark User List mailing list archive at Nabble.com.
---------------------------------------------------------------------
To unsubscribe, e-mail: user-unsubscribe@spark.apache.org
For additional commands, e-mail: user-help@spark.apache.org
Re: subtractByKey increases RDD size in memory - any ideas?
Posted by DaPsul <da...@gmx.de>.
That could be possible but if you extract the data and create a new RDD
the size is still bigger:
val data = rdd3.collect()
val rdd4 = sc.paralellize(data)
Am 19/02/16 um 02:32 schrieb Andrew Ehrlich:
> There could be clues in the different RDD subclasses; rdd1 is
> ParallelCollectionRDD but rdd3 is SubtractedRDD.
>
> On Thu, Feb 18, 2016 at 1:37 PM, DaPsul <dapsul@gmx.de
> <ma...@gmx.de>> wrote:
>
> (copy from
> http://stackoverflow.com/questions/35467128/spark-subtractbykey-increases-rdd-cached-memory-size)
>
> I've found a very strange behavior for RDD's (spark 1.6.0 with
> scala 2.11):
>
> When i use subtractByKey on an RDD the resulting RDD should be of
> equal or
> smaller size. What i get is an RDD that takes even more space in
> memory:
>
> //Initialize first RDD
> val rdd1 = sc.parallelize(Array((1,1),(2,2),(3,3))).cache()
>
> //dummy action to cache it => size according to webgui: 184 Bytes
> rdd1.first
>
> //Initialize RDD to subtract (empty RDD should result in no change
> for rdd1)
> val rdd2 = sc.parallelize(Array[(Int,Int)]())
>
> //perform subtraction
> val rdd3 = rdd1.subtractByKey(rdd2).cache()
>
> //dummy action to cache rdd3 => size according to webgui: 208 Bytes
> rdd3.first
>
> I frist realized this strange behaviour for an RDD of ~200k rows
> and size
> 1.3 GB that scaled up to more than 2 GB after subtraction
>
> Edit: Tried the example above with more values(10k) => same
> behaviour. The
> size increases by ~1.6 times. Also reduceByKey seems to have a similar
> effect.
>
> When i create an RDD by
>
> sc.paralellize(rdd3.collect())
>
> the size is the same as for rdd3, so the increased size carries
> over even if
> it's extracted from RDD.
>
>
>
>
> --
> View this message in context:
> http://apache-spark-user-list.1001560.n3.nabble.com/subtractByKey-increases-RDD-size-in-memory-any-ideas-tp26272.html
> Sent from the Apache Spark User List mailing list archive at
> Nabble.com.
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: user-unsubscribe@spark.apache.org
> <ma...@spark.apache.org>
> For additional commands, e-mail: user-help@spark.apache.org
> <ma...@spark.apache.org>
>
>
Re: subtractByKey increases RDD size in memory - any ideas?
Posted by Andrew Ehrlich <an...@aehrlich.com>.
There could be clues in the different RDD subclasses; rdd1 is
ParallelCollectionRDD but rdd3 is SubtractedRDD.
On Thu, Feb 18, 2016 at 1:37 PM, DaPsul <da...@gmx.de> wrote:
> (copy from
>
> http://stackoverflow.com/questions/35467128/spark-subtractbykey-increases-rdd-cached-memory-size
> )
>
> I've found a very strange behavior for RDD's (spark 1.6.0 with scala 2.11):
>
> When i use subtractByKey on an RDD the resulting RDD should be of equal or
> smaller size. What i get is an RDD that takes even more space in memory:
>
> //Initialize first RDD
> val rdd1 = sc.parallelize(Array((1,1),(2,2),(3,3))).cache()
>
> //dummy action to cache it => size according to webgui: 184 Bytes
> rdd1.first
>
> //Initialize RDD to subtract (empty RDD should result in no change for
> rdd1)
> val rdd2 = sc.parallelize(Array[(Int,Int)]())
>
> //perform subtraction
> val rdd3 = rdd1.subtractByKey(rdd2).cache()
>
> //dummy action to cache rdd3 => size according to webgui: 208 Bytes
> rdd3.first
>
> I frist realized this strange behaviour for an RDD of ~200k rows and size
> 1.3 GB that scaled up to more than 2 GB after subtraction
>
> Edit: Tried the example above with more values(10k) => same behaviour. The
> size increases by ~1.6 times. Also reduceByKey seems to have a similar
> effect.
>
> When i create an RDD by
>
> sc.paralellize(rdd3.collect())
>
> the size is the same as for rdd3, so the increased size carries over even
> if
> it's extracted from RDD.
>
>
>
>
> --
> View this message in context:
> http://apache-spark-user-list.1001560.n3.nabble.com/subtractByKey-increases-RDD-size-in-memory-any-ideas-tp26272.html
> Sent from the Apache Spark User List mailing list archive at Nabble.com.
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: user-unsubscribe@spark.apache.org
> For additional commands, e-mail: user-help@spark.apache.org
>
>