[jira] [Created] (SPARK-22951) count() after dropDuplicates() on emptyDataFrame() returns incorrect value

["Michael Dreibelbis (JIRA)" <ji...@apache.org>]

Michael Dreibelbis created SPARK-22951:
------------------------------------------

             Summary: count() after dropDuplicates() on emptyDataFrame() returns incorrect value
                 Key: SPARK-22951
                 URL: https://issues.apache.org/jira/browse/SPARK-22951
             Project: Spark
          Issue Type: Bug
          Components: SQL
    Affects Versions: 1.6.2
            Reporter: Michael Dreibelbis


here is a minimal Spark Application to reproduce:

{code}
import org.apache.spark.sql.SQLContext
import org.apache.spark.{SparkConf, SparkContext}


object DropDupesApp extends App {
  
  override def main(args: Array[String]): Unit = {
    val conf = new SparkConf()
      .setAppName("test")
      .setMaster("local")
    val sc = new SparkContext(conf)
    val sql = SQLContext.getOrCreate(sc)
    assert(sql.emptyDataFrame.count == 0) // expected
    assert(sql.emptyDataFrame.dropDuplicates.count == 1) // unexpected
  }
  
}
{code}



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org