You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@spark.apache.org by "Wenchen Fan (JIRA)" <ji...@apache.org> on 2016/06/21 11:38:58 UTC
[jira] [Commented] (SPARK-16076) Dataset - outer join nulls can
sometimes combinate to default values
[ https://issues.apache.org/jira/browse/SPARK-16076?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15341609#comment-15341609 ]
Wenchen Fan commented on SPARK-16076:
-------------------------------------
I think this is caused by https://issues.apache.org/jira/browse/SPARK-16097
> Dataset - outer join nulls can sometimes combinate to default values
> --------------------------------------------------------------------
>
> Key: SPARK-16076
> URL: https://issues.apache.org/jira/browse/SPARK-16076
> Project: Spark
> Issue Type: Bug
> Components: SQL
> Affects Versions: 2.0.0
> Reporter: Richard Marscher
>
> Recently joinWith behavior was tweaked to properly pass through missing rows in an outer join to preserve it as a null. But circumstantially calling some combinator functions afterwards can undo this and cause the data to revert to default values.
> I have a reproducing Databricks notebook where two Dataset of type (Int, Int) are joined on the first element in the tuples. If a `map` call is made after the joinWith, the map call lifts the null values into default values for the given datatype (-1 for Int).
> https://databricks-prod-cloudfront.cloud.databricks.com/public/4027ec902e239c93eaaa8714f173bcfc/160347920874755/4268263383756277/673639177603143/latest.html
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org