[jira] [Created] (SPARK-16076) Dataset - outer join nulls can sometimes combinate to default values

["Richard Marscher (JIRA)" <ji...@apache.org>]

Richard Marscher created SPARK-16076:
----------------------------------------

             Summary: Dataset - outer join nulls can sometimes combinate to default values
                 Key: SPARK-16076
                 URL: https://issues.apache.org/jira/browse/SPARK-16076
             Project: Spark
          Issue Type: Bug
          Components: SQL
    Affects Versions: 2.0.0
            Reporter: Richard Marscher


Recently joinWith behavior was tweaked to properly pass through missing rows in an outer join to preserve it as a null. But circumstantially calling some combinator functions afterwards can undo this and cause the data to revert to default values.

I have a reproducing Databricks notebook where two Dataset of type (Int, Int) are joined on the first element in the tuples. If a `map` call is made after the joinWith, the map call lifts the null values into default values for the given datatype (-1 for Int).

https://databricks-prod-cloudfront.cloud.databricks.com/public/4027ec902e239c93eaaa8714f173bcfc/160347920874755/4268263383756277/673639177603143/latest.html



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org