[jira] [Created] (SPARK-18986) ExternalAppendOnlyMap shouldn't fail when forced to spill before calling its iterator

["Liang-Chi Hsieh (JIRA)" <ji...@apache.org>]

Liang-Chi Hsieh created SPARK-18986:
---------------------------------------

             Summary: ExternalAppendOnlyMap shouldn't fail when forced to spill before calling its iterator
                 Key: SPARK-18986
                 URL: https://issues.apache.org/jira/browse/SPARK-18986
             Project: Spark
          Issue Type: Bug
          Components: SQL
            Reporter: Liang-Chi Hsieh


{{ExternalAppendOnlyMap.forceSpill}} now uses an assert to check if an iterator is not null in the map. However, the assertion is only true after the map is asked for iterator. Before it, if another memory consumer asks more memory than currently available, {{ExternalAppendOnlyMap.forceSpill}} is also be called too. In this case, we will see failure like this:

{code}
[info]   java.lang.AssertionError: assertion failed
[info]   at scala.Predef$.assert(Predef.scala:156)
[info]   at org.apache.spark.util.collection.ExternalAppendOnlyMap.forceSpill(ExternalAppendOnlyMap.scala:196)
[info]   at org.apache.spark.util.collection.Spillable.spill(Spillable.scala:111)
[info]   at org.apache.spark.util.collection.ExternalAppendOnlyMapSuite$$anonfun$13.apply$mcV$sp(ExternalAppendOnly
MapSuite.scala:294)
{code}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org