[jira] [Created] (SPARK-21986) QuantileDiscretizer picks wrong split point for data with lots of 0's

["Barry Becker (JIRA)" <ji...@apache.org>]

Barry Becker created SPARK-21986:
------------------------------------

             Summary: QuantileDiscretizer picks wrong split point for data with lots of 0's
                 Key: SPARK-21986
                 URL: https://issues.apache.org/jira/browse/SPARK-21986
             Project: Spark
          Issue Type: Bug
          Components: MLlib
    Affects Versions: 2.1.1
            Reporter: Barry Becker


I have some simple test cases to help illustrate (see below).
I discovered this with data that had 96,000 rows, but can reproduce with much smaller data that has roughly the same distribution of values.

If I have data like
  Seq(0, 0, 0, 0, 0, 40, 0, 0, 45, 46, 0)

and ask for 3 buckets, then it does the right thing and yields splits of 
Seq(Double.NegativeInfinity, 0.0, 40.0, Double.PositiveInfinity)

However, if I add just one more zero, such that I have data like
 Seq(0, 0, 0, 0, 0, 0, 40, 0, 0, 45, 46, 0)
then it will do the wrong thing and give splits of 
  Seq(Double.NegativeInfinity, 0.0, Double.PositiveInfinity))

I'm not bothered that it gave fewer buckets than asked for (that is to be expected), but I am bothered that it picked 0.0 instead of 40 as the one split point.
The way it did it, now I have 1 bucket with all the data, and a second with none of the data.
Am I interpreting something wrong?
Here are my 2 test cases in scala:
{code}
class QuantileDiscretizerSuite extends FunSuite {

  test("Quantile discretizer on data with lots of 0") {
    verify(Seq(0, 0, 0, 0, 0, 0, 40, 0, 0, 45, 46, 0),
      Seq(Double.NegativeInfinity, 0.0, Double.PositiveInfinity))
  }

  test("Quantile discretizer on data with one less 0") {
    verify(Seq(0, 0, 0, 0, 0, 40, 0, 0, 45, 46, 0),
      Seq(Double.NegativeInfinity, 0.0, 40.0, Double.PositiveInfinity))
  }
  
  def verify(data: Seq[Int], expectedSplits: Seq[Double]): Unit = {
    val theData: Seq[(Int, Double)] = data.map {
      case x: Int => (x, 0.0)
      case _ => (0, 0.0)
    }

    val df = SPARK_SESSION.sqlContext.createDataFrame(theData).toDF("rawCol", "unused")

    val qb = new QuantileDiscretizer()
      .setInputCol("rawCol")
      .setOutputCol("binnedColumn")
      .setRelativeError(0.0)
      .setNumBuckets(3)
      .fit(df)

    assertResult(expectedSplits) {qb.getSplits}
  }
}
{code}



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org