[jira] [Resolved] (SPARK-24210) incorrect handling of boolean expressions when using column in expressions in pyspark.sql.DataFrame filter function

["Hyukjin Kwon (Jira)" <ji...@apache.org>]

     [ https://issues.apache.org/jira/browse/SPARK-24210?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Hyukjin Kwon resolved SPARK-24210.
----------------------------------
    Resolution: Incomplete

> incorrect handling of boolean expressions when using column in expressions in pyspark.sql.DataFrame filter function
> -------------------------------------------------------------------------------------------------------------------
>
>                 Key: SPARK-24210
>                 URL: https://issues.apache.org/jira/browse/SPARK-24210
>             Project: Spark
>          Issue Type: Bug
>          Components: PySpark
>    Affects Versions: 2.1.2
>            Reporter: Michael H
>            Priority: Major
>              Labels: bulk-closed
>
> {code:python}
> ex = spark.createDataFrame([
>     ('r0', 0, 0),
>     ('r1', 0, 1),
>     ('r2', 1, 0),
>     ('r3', 1, 1)]\
>   , "row: string, c1: int, c2: int")
> #KO: returns r1 and r3
> ex.filter(('c1 = 1') and ('c2 = 1')).show()
> #OK, raises an exception
> ex.filter(('c1 == 1') & ('c2 == 1')).show()
> #KO: returns r0 and r3
> ex.filter('c1 = 1 & c2 = 1').show()
> #KO: returns r0 and r3
> ex.filter('c1 == 1 & c2 == 1').show()
> #OK: returns r3 only
> ex.filter('c1 = 1 and c2 = 1').show()
> #OK: returns r3 only
> ex.filter('c1 == 1 and c2 == 1').show()
> {code}
> building the expressions using {code}ex.c1{code} or {code}ex['c1']{code} we don't have this.
> Issue seems related with
> https://github.com/apache/spark/pull/6961



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org