[GitHub] [spark] HyukjinKwon opened a new pull request #24838: [SPARK-27995][PYTHON] Note the difference between str of Python 2 and 3 at Arrow optimized toPandas

[GitBox <gi...@apache.org>]

HyukjinKwon opened a new pull request #24838: [SPARK-27995][PYTHON] Note the difference between str of Python 2 and 3 at Arrow optimized toPandas
URL: https://github.com/apache/spark/pull/24838
 
 
   ## What changes were proposed in this pull request?
   
   When Arrow optimization is enabled in Python 2.7, 
   
   ```python
   import pandas
   pdf = pandas.DataFrame(["test1", "test2"])
   df = spark.createDataFrame(pdf)
   df.show()
   ```
   
   I got the following output:
   
   ```
   +----------------+
   |               0|
   +----------------+
   |[74 65 73 74 31]|
   |[74 65 73 74 32]|
   +----------------+
   ```
   
   This looks because Python's `str` and `byte` are same. it does look right:
   
   ```python
   >>> str == bytes
   True
   >>> isinstance("a", bytes)
   True
   ```
   
   To cut it short:
   
   1. Python 2 treats `str` as `bytes`.
   2. PySpark added some special codes and hacks to recognizes `str` as string types.
   3. PyArrow / Pandas followed Python 2 difference
   
   
   To fix, we have two options:
   
   1. Fix it to match the behaviour to PySpark's
   2. Note the differences
   
    but Python 2 is deprecated anyway. I think it's better to just note it and for go option 2.
   
   ## How was this patch tested?
   
   Manually tested.
   

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org