[jira] [Created] (PARQUET-1510) Dictionary filter skips null values when evaluating not-equals.

["Ryan Blue (JIRA)" <ji...@apache.org>]

Ryan Blue created PARQUET-1510:
----------------------------------

             Summary: Dictionary filter skips null values when evaluating not-equals.
                 Key: PARQUET-1510
                 URL: https://issues.apache.org/jira/browse/PARQUET-1510
             Project: Parquet
          Issue Type: Improvement
            Reporter: Ryan Blue


This was discovered in Spark, see SPARK-26677. From the Spark PR:

{code}
// Repeat the values to get dictionary encoding.
Seq(Some("A"), Some("A"), None).toDF.repartition(1).write.mode("overwrite").parquet("/tmp/foo")
spark.read.parquet("/tmp/foo").where("NOT (value <=> 'A')").show()
+-----+
|value|
+-----+
+-----+
{code}

{code}
// Use plain encoding.
Seq(Some("A"), None).toDF.repartition(1).write.mode("overwrite").parquet("/tmp/bar")
spark.read.parquet("/tmp/bar").where("NOT (value <=> 'A')").show()
+-----+
|value|
+-----+
| null|
+-----+
{code}

This is a correctness issue.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)