You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by GitBox <gi...@apache.org> on 2019/03/18 03:45:01 UTC

[GitHub] [spark] HyukjinKwon opened a new pull request #24121: [SPARK-26979][PYTHON][FOLLOW-UP] Make binary math/string functions take string as columns as well

HyukjinKwon opened a new pull request #24121: [SPARK-26979][PYTHON][FOLLOW-UP] Make binary math/string functions take string as columns as well
URL: https://github.com/apache/spark/pull/24121
 
 
   ## What changes were proposed in this pull request?
   
   This is a followup of https://github.com/apache/spark/pull/23882 to handle binary math/string functions. For instance, see the cases below:
   
   **Before:**
   
   ```python
   >>> from pyspark.sql.functions import lit, ascii
   >>> spark.range(1).select(lit('a').alias("value")).select(ascii("value"))
   Traceback (most recent call last):
     File "<stdin>", line 1, in <module>
     File "/.../spark/python/pyspark/sql/functions.py", line 51, in _
       jc = getattr(sc._jvm.functions, name)(col._jc if isinstance(col, Column) else col)
     File "/.../spark/python/lib/py4j-0.10.8.1-src.zip/py4j/java_gateway.py", line 1286, in __call__
     File "/.../spark/python/pyspark/sql/utils.py", line 63, in deco
       return f(*a, **kw)
     File "/.../spark/python/lib/py4j-0.10.8.1-src.zip/py4j/protocol.py", line 332, in get_return_value
   py4j.protocol.Py4JError: An error occurred while calling z:org.apache.spark.sql.functions.ascii. Trace:
   py4j.Py4JException: Method ascii([class java.lang.String]) does not exist
   	at py4j.reflection.ReflectionEngine.getMethod(ReflectionEngine.java:318)
   	at py4j.reflection.ReflectionEngine.getMethod(ReflectionEngine.java:339)
   	at py4j.Gateway.invoke(Gateway.java:276)
   	at py4j.commands.AbstractCommand.invokeMethod(AbstractCommand.java:132)
   	at py4j.commands.CallCommand.execute(CallCommand.java:79)
   	at py4j.GatewayConnection.run(GatewayConnection.java:238)
   	at java.lang.Thread.run(Thread.java:748)
   ```
   
   ```python
   >>> from pyspark.sql.functions import atan2
   >>> spark.range(1).select(atan2("id", "id"))
   Traceback (most recent call last):
     File "<stdin>", line 1, in <module>
     File "/.../spark/python/pyspark/sql/functions.py", line 78, in _
       jc = getattr(sc._jvm.functions, name)(col1._jc if isinstance(col1, Column) else float(col1),
   ValueError: could not convert string to float: id
   ```
   
   **After:**
   
   ```python
   >>> from pyspark.sql.functions import lit, ascii
   >>> spark.range(1).select(lit('a').alias("value")).select(ascii("value"))
   DataFrame[ascii(value): int]
   ```
   
   ```python
   >>> from pyspark.sql.functions import atan2
   >>> spark.range(1).select(atan2("id", "id"))
   DataFrame[ATAN2(id, id): double]
   ```
   
   Note that,
   
   - This PR causes a slight behaviour changes for math functions. For instance, numbers as strings (e.g., `"1"`) were supported as arguments of binary math functions before. After this PR, it recognises it as column names. 
   
   - I also intentionally didn't document this behaviour changes since we're going ahead for Spark 3.0 and I don't think numbers as strings make much sense in math functions. 
   
   - There is another exception `when`, which takes string as literal values as below. This PR doeesn't fix this ambiguity.
     ```python
     >>> spark.range(1).select(when(lit(True), col("id"))).show()
     ```
   
     ```
     +--------------------------+
     |CASE WHEN true THEN id END|
     +--------------------------+
     |                         0|
     +--------------------------+
     ```
   
     ```python
     >>> spark.range(1).select(when(lit(True), "id")).show()
     ```
   
     ```
     +--------------------------+
     |CASE WHEN true THEN id END|
     +--------------------------+
     |                        id|
     +--------------------------+
     ```
   
   This PR also fixes as below:
   
   https://github.com/apache/spark/pull/23882 fixed it to:
   
   - Rename `_create_function` to `_create_name_function`
   - Define new `_create_function` to take strings as column names.
   
   This PR, I proposes to:
   
   - Revert `_create_name_function` name to `_create_function`.
   - Define new `_create_function_with_column`.
   
   ## How was this patch tested?
   
   Some unit tests were added for binary math / string functions.

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org