[spark] branch master updated: [SPARK-27612][PYTHON] Use Python's default protocol instead of highest protocol

[gu...@apache.org]

This is an automated email from the ASF dual-hosted git repository.

gurwls223 pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/spark.git


The following commit(s) were added to refs/heads/master by this push:
     new 5c47924  [SPARK-27612][PYTHON] Use Python's default protocol instead of highest protocol
5c47924 is described below

commit 5c479243de8d88dbe6ac7ab8580826511af74ff5
Author: HyukjinKwon <gu...@apache.org>
AuthorDate: Fri May 3 14:40:13 2019 +0900

    [SPARK-27612][PYTHON] Use Python's default protocol instead of highest protocol
    
    ## What changes were proposed in this pull request?
    
    This PR partially reverts https://github.com/apache/spark/pull/20691
    
    After we changed the Python protocol to highest ones, seems like it introduced a correctness bug. This potentially affects all Python related code paths.
    
    I suspect a bug related to Pryolite (maybe opcodes `MEMOIZE`, `FRAME` and/or our `RowPickler`). I would like to stick to default protocol for now and investigate the issue separately.
    
    I will separately investigate later to bring highest protocol back.
    
    ## How was this patch tested?
    
    Unittest was added.
    
    ```bash
    ./run-tests --python-executables=python3.7 --testname "pyspark.sql.tests.test_serde SerdeTests.test_int_array_serialization"
    ```
    
    Closes #24519 from HyukjinKwon/SPARK-27612.
    
    Authored-by: HyukjinKwon <gu...@apache.org>
    Signed-off-by: HyukjinKwon <gu...@apache.org>
---
 python/pyspark/serializers.py          | 3 ++-
 python/pyspark/sql/tests/test_serde.py | 6 ++++++
 2 files changed, 8 insertions(+), 1 deletion(-)

diff --git a/python/pyspark/serializers.py b/python/pyspark/serializers.py
index 6058e94..5311087 100644
--- a/python/pyspark/serializers.py
+++ b/python/pyspark/serializers.py
@@ -62,11 +62,12 @@ import itertools
 if sys.version < '3':
     import cPickle as pickle
     from itertools import izip as zip, imap as map
+    pickle_protocol = 2
 else:
     import pickle
     basestring = unicode = str
     xrange = range
-pickle_protocol = pickle.HIGHEST_PROTOCOL
+    pickle_protocol = 3
 
 from pyspark import cloudpickle
 from pyspark.util import _exception_message
diff --git a/python/pyspark/sql/tests/test_serde.py b/python/pyspark/sql/tests/test_serde.py
index 8707f46..1c18e93 100644
--- a/python/pyspark/sql/tests/test_serde.py
+++ b/python/pyspark/sql/tests/test_serde.py
@@ -126,6 +126,12 @@ class SerdeTests(ReusedSQLTestCase):
         df = self.spark.createDataFrame(data, schema=schema)
         df.collect()
 
+    def test_int_array_serialization(self):
+        # Note that this test seems dependent on parallelism.
+        data = self.spark.sparkContext.parallelize([[1, 2, 3, 4]] * 100, numSlices=12)
+        df = self.spark.createDataFrame(data, "array<integer>")
+        self.assertEqual(len(list(filter(lambda r: None in r.value, df.collect()))), 0)
+
 
 if __name__ == "__main__":
     import unittest


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@spark.apache.org
For additional commands, e-mail: commits-help@spark.apache.org