[jira] [Created] (SPARK-29319) Memory and disk usage not accurate when blocks are evicted and re-loaded in memory

["Marcelo Masiero Vanzin (Jira)" <ji...@apache.org>]

Marcelo Masiero Vanzin created SPARK-29319:
----------------------------------------------

             Summary: Memory and disk usage not accurate when blocks are evicted and re-loaded in memory
                 Key: SPARK-29319
                 URL: https://issues.apache.org/jira/browse/SPARK-29319
             Project: Spark
          Issue Type: Bug
          Components: Spark Core
    Affects Versions: 2.4.4, 3.0.0
            Reporter: Marcelo Masiero Vanzin


I found this while running more targeted tests for the underlying behavior of this code, triggered by SPARK-27468. I ran this code:

{code}
import java.util.Arrays
import org.apache.spark.rdd._
import org.apache.spark.storage._

def newCachedRDD(level: StorageLevel): RDD[Array[Long]] = {
  val rdd = sc.parallelize(1 to 64, 64).map { i =>
    val a = new Array[Long](1024 * 1024)
    Arrays.fill(a, i)
    a
  }
  rdd.persist(level)
  rdd.count()
  rdd
}

val r1 = newCachedRDD(level = StorageLevel.MEMORY_AND_DISK)
val r2 = newCachedRDD(level = StorageLevel.MEMORY_ONLY)
{code}

With {{./bin/spark-shell --master 'local-cluster[1,1,1024]'}}.

After it runs, you end up with the expected values: r1 has everything cached, only using disk, because all its memory blocks were evicted by r2; r2 has as many blocks as the memory can hold.

The problem shows up when you start playing with those RDDs again.

Calling {{r1.count()}} will cause all of r2's blocks to be evicted, since r1's blocks are loaded back in memory. But no block update is sent to the driver about that load, so the driver does not know that the blocks are now in memory. The UI will show that r1 is using 0 bytes of memory, and r2 disappears from the storage page (this last one as expected).

Calling {{r2.count()}} after that will cause r1's blocks to be evicted again. This will send updates to the driver, which will now double-count the disk usage. So if you keep doing this back and forth, r1's disk usage will keep growing in the UI, when in fact it doesn't change at all.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org