You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@spark.apache.org by "Hyukjin Kwon (JIRA)" <ji...@apache.org> on 2019/05/21 04:23:22 UTC
[jira] [Updated] (SPARK-16076) Dataset - outer join nulls can
sometimes combinate to default values
[ https://issues.apache.org/jira/browse/SPARK-16076?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Hyukjin Kwon updated SPARK-16076:
---------------------------------
Labels: bulk-closed (was: )
> Dataset - outer join nulls can sometimes combinate to default values
> --------------------------------------------------------------------
>
> Key: SPARK-16076
> URL: https://issues.apache.org/jira/browse/SPARK-16076
> Project: Spark
> Issue Type: Bug
> Components: SQL
> Affects Versions: 2.0.0
> Reporter: Richard Marscher
> Priority: Major
> Labels: bulk-closed
>
> Recently joinWith behavior was tweaked to properly pass through missing rows in an outer join to preserve it as a null. But circumstantially calling some combinator functions afterwards can undo this and cause the data to revert to default values.
> I have a reproducing Databricks notebook where two Dataset of type (Int, Int) are joined on the first element in the tuples. If a `map` call is made after the joinWith, the map call lifts the null values into default values for the given datatype (-1 for Int).
> https://databricks-prod-cloudfront.cloud.databricks.com/public/4027ec902e239c93eaaa8714f173bcfc/160347920874755/4268263383756277/673639177603143/latest.html
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org