You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@spark.apache.org by "Reynold Xin (JIRA)" <ji...@apache.org> on 2015/07/08 02:58:05 UTC
[jira] [Resolved] (SPARK-8685) dataframe left joins are not working
as expected in pyspark
[ https://issues.apache.org/jira/browse/SPARK-8685?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Reynold Xin resolved SPARK-8685.
--------------------------------
Resolution: Fixed
Fix Version/s: 1.5.0
> dataframe left joins are not working as expected in pyspark
> -----------------------------------------------------------
>
> Key: SPARK-8685
> URL: https://issues.apache.org/jira/browse/SPARK-8685
> Project: Spark
> Issue Type: Sub-task
> Components: PySpark, SQL
> Affects Versions: 1.4.0
> Environment: ubuntu 14.04
> Reporter: axel dahl
> Assignee: Davies Liu
> Fix For: 1.5.0
>
>
> I have the following code:
> {code}
> from pyspark import SQLContext
> d1 = [{'name':'bob', 'country': 'usa', 'age': 1},
> {'name':'alice', 'country': 'jpn', 'age': 2},
> {'name':'carol', 'country': 'ire', 'age': 3}]
> d2 = [{'name':'bob', 'country': 'usa', 'colour':'red'},
> {'name':'carol', 'country': 'ire', 'colour':'green'}]
> r1 = sc.parallelize(d1)
> r2 = sc.parallelize(d2)
> sqlContext = SQLContext(sc)
> df1 = sqlContext.createDataFrame(d1)
> df2 = sqlContext.createDataFrame(d2)
> df1.join(df2, (df1.name == df2.name) & (df1.country == df2.country), 'left_outer').collect()
> {code}
> When I run it I get the following, (notice in the first row, all join keys are take from the right-side and so are blanked out):
> {code}
> [Row(age=2, country=None, name=None, colour=None, country=None, name=None),
> Row(age=1, country=u'usa', name=u'bob', colour=u'red', country=u'usa', name=u'bob'),
> Row(age=3, country=u'ire', name=u'carol', colour=u'green', country=u'ire', name=u'alice')]
> {code}
> I would expect to get (though ideally without duplicate columns):
> {code}
> [Row(age=2, country=u'ire', name=u'alice', colour=None, country=None, name=None),
> Row(age=1, country=u'usa', name=u'bob', colour=u'red', country=u'usa', name=u'bob'),
> Row(age=3, country=u'ire', name=u'carol', colour=u'green', country=u'ire', name=u'alice')]
> {code}
> The workaround for now is this rather clunky piece of code:
> {code}
> df2 = sqlContext.createDataFrame(d2).withColumnRenamed('name', 'name2').withColumnRenamed('country', 'country2')
> df1.join(df2, (df1.name == df2.name2) & (df1.country == df2.country2), 'left_outer').collect()
> {code}
> Also, {{.show()}} works
> {code}
> sqlContext = SQLContext(sc)
> df1 = sqlContext.createDataFrame(d1)
> df2 = sqlContext.createDataFrame(d2)
> df1.join(df2, (df1.name == df2.name) & (df1.country == df2.country), 'left_outer').show()
> +---+-------+-----+------+-------+-----+
> |age|country| name|colour|country| name|
> +---+-------+-----+------+-------+-----+
> | 3| ire|carol| green| ire|carol|
> | 2| jpn|alice| null| null| null|
> | 1| usa| bob| red| usa| bob|
> +---+-------+-----+------+-------+-----+
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org