[jira] [Created] (SPARK-8503) SizeEstimator returns negative value for recursive data structures

["Ilya Rakitsin (JIRA)" <ji...@apache.org>]

Ilya Rakitsin created SPARK-8503:
------------------------------------

             Summary: SizeEstimator returns negative value for recursive data structures
                 Key: SPARK-8503
                 URL: https://issues.apache.org/jira/browse/SPARK-8503
             Project: Spark
          Issue Type: Bug
          Components: Spark Core
    Affects Versions: 1.3.1
            Reporter: Ilya Rakitsin


When estimating size of recursive data structures like graphs, with transient fields referencing one another, SizeEstimator may return negative value if the structure if big enough.

This then affects the logic of other components, e.g. SizeTracker#takeSample() and may lead to incorrect behavior and exceptions like:

java.lang.IllegalArgumentException: requirement failed: sizeInBytes was negative: -9223372036854691384
	at scala.Predef$.require(Predef.scala:233)
	at org.apache.spark.storage.BlockInfo.markReady(BlockInfo.scala:55)
	at org.apache.spark.storage.BlockManager.doPut(BlockManager.scala:810)
	at org.apache.spark.storage.BlockManager.putIterator(BlockManager.scala:637)
	at org.apache.spark.storage.BlockManager.putSingle(BlockManager.scala:991)
	at org.apache.spark.broadcast.TorrentBroadcast.writeBlocks(TorrentBroadcast.scala:98)
	at org.apache.spark.broadcast.TorrentBroadcast.<init>(TorrentBroadcast.scala:84)
	at org.apache.spark.broadcast.TorrentBroadcastFactory.newBroadcast(TorrentBroadcastFactory.scala:34)
	at org.apache.spark.broadcast.TorrentBroadcastFactory.newBroadcast(TorrentBroadcastFactory.scala:29)
	at org.apache.spark.broadcast.BroadcastManager.newBroadcast(BroadcastManager.scala:62)
	at org.apache.spark.SparkContext.broadcast(SparkContext.scala:1051)



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org