[jira] [Commented] (SPARK-18986) ExternalAppendOnlyMap shouldn't fail when forced to spill before calling its iterator

["Gili Nachum (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/SPARK-18986?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16081995#comment-16081995 ] 

Gili Nachum commented on SPARK-18986:
-------------------------------------

I'm currently using 2.11, and this was fixed in 2.2.0 (not released yet).
The workaround I used (might help others) was to increase the # of partitions to avoid the buggy spill altogether.

> ExternalAppendOnlyMap shouldn't fail when forced to spill before calling its iterator
> -------------------------------------------------------------------------------------
>
>                 Key: SPARK-18986
>                 URL: https://issues.apache.org/jira/browse/SPARK-18986
>             Project: Spark
>          Issue Type: Bug
>          Components: Spark Core
>            Reporter: Liang-Chi Hsieh
>            Assignee: Liang-Chi Hsieh
>             Fix For: 2.2.0
>
>
> {{ExternalAppendOnlyMap.forceSpill}} now uses an assert to check if an iterator is not null in the map. However, the assertion is only true after the map is asked for iterator. Before it, if another memory consumer asks more memory than currently available, {{ExternalAppendOnlyMap.forceSpill}} is also be called too. In this case, we will see failure like this:
> {code}
> [info]   java.lang.AssertionError: assertion failed
> [info]   at scala.Predef$.assert(Predef.scala:156)
> [info]   at org.apache.spark.util.collection.ExternalAppendOnlyMap.forceSpill(ExternalAppendOnlyMap.scala:196)
> [info]   at org.apache.spark.util.collection.Spillable.spill(Spillable.scala:111)
> [info]   at org.apache.spark.util.collection.ExternalAppendOnlyMapSuite$$anonfun$13.apply$mcV$sp(ExternalAppendOnly
> MapSuite.scala:294)
> {code}



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org