You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@spark.apache.org by "Josh Rosen (JIRA)" <ji...@apache.org> on 2014/08/10 09:08:12 UTC
[jira] [Assigned] (SPARK-2954) PySpark MLlib serialization tests
fail on Python 2.6
[ https://issues.apache.org/jira/browse/SPARK-2954?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Josh Rosen reassigned SPARK-2954:
---------------------------------
Assignee: Josh Rosen
> PySpark MLlib serialization tests fail on Python 2.6
> ----------------------------------------------------
>
> Key: SPARK-2954
> URL: https://issues.apache.org/jira/browse/SPARK-2954
> Project: Spark
> Issue Type: Bug
> Components: PySpark
> Affects Versions: 1.1.0
> Reporter: Josh Rosen
> Assignee: Josh Rosen
>
> The PySpark MLlib tests currently fail on Python 2.6 due to problems unpacking data from bytearray using struct.unpack:
> {code}
> **********************************************************************
> File "pyspark/mllib/_common.py", line 181, in __main__._deserialize_double
> Failed example:
> _deserialize_double(_serialize_double(1L)) == 1.0
> Exception raised:
> Traceback (most recent call last):
> File "/System/Library/Frameworks/Python.framework/Versions/2.6/lib/python2.6/doctest.py", line 1253, in __run
> compileflags, 1) in test.globs
> File "<doctest __main__._deserialize_double[4]>", line 1, in <module>
> _deserialize_double(_serialize_double(1L)) == 1.0
> File "pyspark/mllib/_common.py", line 194, in _deserialize_double
> return struct.unpack("d", ba[offset:])[0]
> error: unpack requires a string argument of length 8
> **********************************************************************
> File "pyspark/mllib/_common.py", line 184, in __main__._deserialize_double
> Failed example:
> _deserialize_double(_serialize_double(sys.float_info.max)) == x
> Exception raised:
> Traceback (most recent call last):
> File "/System/Library/Frameworks/Python.framework/Versions/2.6/lib/python2.6/doctest.py", line 1253, in __run
> compileflags, 1) in test.globs
> File "<doctest __main__._deserialize_double[6]>", line 1, in <module>
> _deserialize_double(_serialize_double(sys.float_info.max)) == x
> File "pyspark/mllib/_common.py", line 194, in _deserialize_double
> return struct.unpack("d", ba[offset:])[0]
> error: unpack requires a string argument of length 8
> **********************************************************************
> File "pyspark/mllib/_common.py", line 187, in __main__._deserialize_double
> Failed example:
> _deserialize_double(_serialize_double(sys.float_info.max)) == y
> Exception raised:
> Traceback (most recent call last):
> File "/System/Library/Frameworks/Python.framework/Versions/2.6/lib/python2.6/doctest.py", line 1253, in __run
> compileflags, 1) in test.globs
> File "<doctest __main__._deserialize_double[8]>", line 1, in <module>
> _deserialize_double(_serialize_double(sys.float_info.max)) == y
> File "pyspark/mllib/_common.py", line 194, in _deserialize_double
> return struct.unpack("d", ba[offset:])[0]
> error: unpack requires a string argument of length 8
> **********************************************************************
> {code}
> It looks like one solution is to wrap the {{bytearray}} with {{buffer()}}: http://stackoverflow.com/a/15467046/590203
--
This message was sent by Atlassian JIRA
(v6.2#6252)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org