You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@spark.apache.org by cu...@apache.org on 2018/05/18 16:53:35 UTC
spark git commit: [SPARK-24303][PYTHON] Update cloudpickle to v0.4.4
Repository: spark
Updated Branches:
refs/heads/master 7b2dca5b1 -> 0cf59fcbe
[SPARK-24303][PYTHON] Update cloudpickle to v0.4.4
## What changes were proposed in this pull request?
cloudpickle 0.4.4 is released - https://github.com/cloudpipe/cloudpickle/releases/tag/v0.4.4
There's no invasive change - the main difference is that we are now able to pickle the root logger, which fix is pretty isolated.
## How was this patch tested?
Jenkins tests.
Author: hyukjinkwon <gu...@apache.org>
Closes #21350 from HyukjinKwon/SPARK-24303.
Project: http://git-wip-us.apache.org/repos/asf/spark/repo
Commit: http://git-wip-us.apache.org/repos/asf/spark/commit/0cf59fcb
Tree: http://git-wip-us.apache.org/repos/asf/spark/tree/0cf59fcb
Diff: http://git-wip-us.apache.org/repos/asf/spark/diff/0cf59fcb
Branch: refs/heads/master
Commit: 0cf59fcbe3799dd3c4469cbf8cd842d668a76f34
Parents: 7b2dca5
Author: hyukjinkwon <gu...@apache.org>
Authored: Fri May 18 09:53:24 2018 -0700
Committer: Bryan Cutler <cu...@gmail.com>
Committed: Fri May 18 09:53:24 2018 -0700
----------------------------------------------------------------------
python/pyspark/cloudpickle.py | 13 +++++++++----
1 file changed, 9 insertions(+), 4 deletions(-)
----------------------------------------------------------------------
http://git-wip-us.apache.org/repos/asf/spark/blob/0cf59fcb/python/pyspark/cloudpickle.py
----------------------------------------------------------------------
diff --git a/python/pyspark/cloudpickle.py b/python/pyspark/cloudpickle.py
index ea845b9..88519d7 100644
--- a/python/pyspark/cloudpickle.py
+++ b/python/pyspark/cloudpickle.py
@@ -272,7 +272,7 @@ class CloudPickler(Pickler):
if not PY3:
def save_buffer(self, obj):
self.save(str(obj))
- dispatch[buffer] = save_buffer
+ dispatch[buffer] = save_buffer # noqa: F821 'buffer' was removed in Python 3
def save_unsupported(self, obj):
raise pickle.PicklingError("Cannot pickle objects of type %s" % type(obj))
@@ -801,10 +801,10 @@ class CloudPickler(Pickler):
def save_not_implemented(self, obj):
self.save_reduce(_gen_not_implemented, ())
- if PY3:
- dispatch[io.TextIOWrapper] = save_file
- else:
+ try: # Python 2
dispatch[file] = save_file
+ except NameError: # Python 3
+ dispatch[io.TextIOWrapper] = save_file
dispatch[type(Ellipsis)] = save_ellipsis
dispatch[type(NotImplemented)] = save_not_implemented
@@ -819,6 +819,11 @@ class CloudPickler(Pickler):
dispatch[logging.Logger] = save_logger
+ def save_root_logger(self, obj):
+ self.save_reduce(logging.getLogger, (), obj=obj)
+
+ dispatch[logging.RootLogger] = save_root_logger
+
"""Special functions for Add-on libraries"""
def inject_addons(self):
"""Plug in system. Register additional pickling functions if modules already loaded"""
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@spark.apache.org
For additional commands, e-mail: commits-help@spark.apache.org