[jira] [Created] (SPARK-26812) PushProjectionThroughUnion nullability issue

["Bogdan Raducanu (JIRA)" <ji...@apache.org>]

Bogdan Raducanu created SPARK-26812:
---------------------------------------

             Summary: PushProjectionThroughUnion nullability issue
                 Key: SPARK-26812
                 URL: https://issues.apache.org/jira/browse/SPARK-26812
             Project: Spark
          Issue Type: Bug
          Components: SQL
    Affects Versions: 2.4.0
            Reporter: Bogdan Raducanu


Union output data types are the output data types of the first child.
However the other union children may have different values nullability.
This means that we can't always push down a project on the children.

To reproduce
{code}
Seq(Map("foo" -> "bar")).toDF("a").write.saveAsTable("table1")
sql("SELECT 1 AS b").write.saveAsTable("table2")
sql("CREATE OR REPLACE VIEW test1 AS SELECT map() AS a FROM table2 UNION ALL SELECT a FROM table1")
 sql("select * from test1").show
{code}

This fails becaus the plan is no longer resolved.
The plan is broken by the PushProjectionThroughUnion rule which pushed down a cast to map<string,string> with values nullability=true on a child with type map<string, string> with values nullability=false.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org