You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@spark.apache.org by "Apache Spark (JIRA)" <ji...@apache.org> on 2015/06/23 23:14:43 UTC
[jira] [Assigned] (SPARK-8568) Prevent accidental use of "and" and
"or" to build invalid expressions in Python
[ https://issues.apache.org/jira/browse/SPARK-8568?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Apache Spark reassigned SPARK-8568:
-----------------------------------
Assignee: Apache Spark (was: Davies Liu)
> Prevent accidental use of "and" and "or" to build invalid expressions in Python
> -------------------------------------------------------------------------------
>
> Key: SPARK-8568
> URL: https://issues.apache.org/jira/browse/SPARK-8568
> Project: Spark
> Issue Type: Sub-task
> Components: SQL
> Reporter: Reynold Xin
> Assignee: Apache Spark
> Priority: Critical
>
> In Spark DataFrames (and in Pandas as well), the correct way to construct a conjunctive expression is to use the bitwise and operator, i.e.: "(x > 5) & (y > 6)".
> However, a lot of users assume that they should be using the Python "and" keyword, i.e. doing "x > 5 and y > 6". Python's boolean evaluation logic converts "x > 5 and y > 6" into just "y > 6" (since "x > 5" is not None). This is super confusing & error prone.
> We should override __bool__ and __nonzero__ for Column to throw an exception if users call "and" and "or" on Column expressions.
> Background: see this blog post http://www.nodalpoint.com/unexpected-behavior-of-spark-dataframe-filter-method/
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org