git commit: [SPARK-1687] [PySpark] fix unit tests related to pickable namedtuple

[jo...@apache.org]

Repository: spark
Updated Branches:
  refs/heads/master 8e7d5ba1a -> 9fd82dbbc


[SPARK-1687] [PySpark] fix unit tests related to pickable namedtuple

serializer is imported multiple times during doctests, so it's better to make _hijack_namedtuple() safe to be called multiple times.

Author: Davies Liu <da...@gmail.com>

Closes #1771 from davies/fix and squashes the following commits:

1a9e336 [Davies Liu] fix unit tests


Project: http://git-wip-us.apache.org/repos/asf/spark/repo
Commit: http://git-wip-us.apache.org/repos/asf/spark/commit/9fd82dbb
Tree: http://git-wip-us.apache.org/repos/asf/spark/tree/9fd82dbb
Diff: http://git-wip-us.apache.org/repos/asf/spark/diff/9fd82dbb

Branch: refs/heads/master
Commit: 9fd82dbbcb8b10debbe95f1acab53ae8b340f38e
Parents: 8e7d5ba
Author: Davies Liu <da...@gmail.com>
Authored: Mon Aug 4 15:54:52 2014 -0700
Committer: Josh Rosen <jo...@apache.org>
Committed: Mon Aug 4 15:54:52 2014 -0700

----------------------------------------------------------------------
 python/pyspark/serializers.py | 6 +++++-
 1 file changed, 5 insertions(+), 1 deletion(-)
----------------------------------------------------------------------


http://git-wip-us.apache.org/repos/asf/spark/blob/9fd82dbb/python/pyspark/serializers.py
----------------------------------------------------------------------
diff --git a/python/pyspark/serializers.py b/python/pyspark/serializers.py
index 1b52c14..a10f85b 100644
--- a/python/pyspark/serializers.py
+++ b/python/pyspark/serializers.py
@@ -297,8 +297,11 @@ def _hack_namedtuple(cls):
 
 def _hijack_namedtuple():
     """ Hack namedtuple() to make it picklable """
-    global _old_namedtuple # or it will put in closure
+    # hijack only one time
+    if hasattr(collections.namedtuple, "__hijack"):
+        return
 
+    global _old_namedtuple # or it will put in closure
     def _copy_func(f):
         return types.FunctionType(f.func_code, f.func_globals, f.func_name,
                 f.func_defaults, f.func_closure)
@@ -313,6 +316,7 @@ def _hijack_namedtuple():
     collections.namedtuple.func_globals["_old_namedtuple"] = _old_namedtuple
     collections.namedtuple.func_globals["_hack_namedtuple"] = _hack_namedtuple
     collections.namedtuple.func_code = namedtuple.func_code
+    collections.namedtuple.__hijack = 1
 
     # hack the cls already generated by namedtuple
     # those created in other module can be pickled as normal,


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@spark.apache.org
For additional commands, e-mail: commits-help@spark.apache.org