[jira] [Resolved] (SPARK-1149) Bad partitioners can cause Spark to hang

["Sean Owen (JIRA)" <ji...@apache.org>]

     [ https://issues.apache.org/jira/browse/SPARK-1149?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Sean Owen resolved SPARK-1149.
------------------------------
    Resolution: Fixed

Looks like Patrick merged this into master in March. It might have been fixed for ... 1.0?

> Bad partitioners can cause Spark to hang
> ----------------------------------------
>
>                 Key: SPARK-1149
>                 URL: https://issues.apache.org/jira/browse/SPARK-1149
>             Project: Spark
>          Issue Type: Bug
>          Components: Spark Core
>            Reporter: Bryn Keller
>            Priority: Minor
>
> While implementing a unit test for lookup, I accidentally created a situation where a partitioner returned a partition number that was outside its range. It should have returned 0 or 1, but in the last case, it returned a -1. 
> Rather than reporting the problem via an exception, Spark simply hangs during the unit test run.
> We should catch this bad behavior by partitioners and throw an exception.
> test("lookup with bad partitioner") {
>     val pairs = sc.parallelize(Array((1,2), (3,4), (5,6), (5,7)))
>     val p = new Partitioner {
>       def numPartitions: Int = 2
>       def getPartition(key: Any): Int = key.hashCode() % 2
>     }
>     val shuffled = pairs.partitionBy(p)
>     assert(shuffled.partitioner === Some(p))
>     assert(shuffled.lookup(1) === Seq(2))
>     assert(shuffled.lookup(5) === Seq(6,7))
>     assert(shuffled.lookup(-1) === Seq())
>   }



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org