You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by GitBox <gi...@apache.org> on 2021/01/04 09:37:12 UTC
[GitHub] [spark] HyukjinKwon opened a new pull request #31007: [SPARK-33983][PYTHON] Update cloudpickle to v1.6.0
HyukjinKwon opened a new pull request #31007:
URL: https://github.com/apache/spark/pull/31007
### What changes were proposed in this pull request?
This PR proposes to upgrade cloudpickle from 1.5.0 to 1.6.0.
It virtually contains one fix:
https://github.com/cloudpipe/cloudpickle/commit/4510be850d55bc60decf86953324f98bc3199f9e
From a cursory look, this isn't a regression, and not even properly supported in Python:
```python
>>> import pickle
>>> pickle.dumps({}.keys())
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
TypeError: cannot pickle 'dict_keys' object
### Why are the changes needed?
To leverage bug fixes from the cloudpickle upstream.
### Does this PR introduce _any_ user-facing change?
No.
### How was this patch tested?
Jenkins build and GitHub actions build will test it out.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] MaxGekk commented on a change in pull request #31007: [SPARK-33983][PYTHON] Update cloudpickle to v1.6.0
Posted by GitBox <gi...@apache.org>.
MaxGekk commented on a change in pull request #31007:
URL: https://github.com/apache/spark/pull/31007#discussion_r551230196
##########
File path: python/pyspark/cloudpickle/cloudpickle.py
##########
@@ -236,7 +236,7 @@ def _extract_code_globals(co):
out_names = {names[oparg] for _, oparg in _walk_global_ops(co)}
# Declaring a function inside another one using the "def ..."
- # syntax generates a constant code object corresponding to the one
+ # syntax generates a constant code object corresonding to the one
Review comment:
typo?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] MaxGekk commented on a change in pull request #31007: [SPARK-33983][PYTHON] Update cloudpickle to v1.6.0
Posted by GitBox <gi...@apache.org>.
MaxGekk commented on a change in pull request #31007:
URL: https://github.com/apache/spark/pull/31007#discussion_r551230681
##########
File path: python/pyspark/cloudpickle/cloudpickle.py
##########
@@ -88,7 +88,7 @@ def g():
DEFAULT_PROTOCOL = pickle.HIGHEST_PROTOCOL
# Track the provenance of reconstructed dynamic classes to make it possible to
-# reconstruct instances from the matching singleton class definition when
+# recontruct instances from the matching singleton class definition when
Review comment:
typo?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] HyukjinKwon commented on a change in pull request #31007: [SPARK-33983][PYTHON] Update cloudpickle to v1.6.0
Posted by GitBox <gi...@apache.org>.
HyukjinKwon commented on a change in pull request #31007:
URL: https://github.com/apache/spark/pull/31007#discussion_r551245463
##########
File path: python/pyspark/cloudpickle/cloudpickle.py
##########
@@ -236,7 +236,7 @@ def _extract_code_globals(co):
out_names = {names[oparg] for _, oparg in _walk_global_ops(co)}
# Declaring a function inside another one using the "def ..."
- # syntax generates a constant code object corresponding to the one
+ # syntax generates a constant code object corresonding to the one
Review comment:
https://github.com/cloudpipe/cloudpickle/pull/406. Let's fix it back when we upgrade next time.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] HyukjinKwon commented on pull request #31007: [SPARK-33983][PYTHON] Update cloudpickle to v1.6.0
Posted by GitBox <gi...@apache.org>.
HyukjinKwon commented on pull request #31007:
URL: https://github.com/apache/spark/pull/31007#issuecomment-754281952
Thanks @MaxGekk and @dongjoon-hyun !
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] dongjoon-hyun closed pull request #31007: [SPARK-33983][PYTHON] Update cloudpickle to v1.6.0
Posted by GitBox <gi...@apache.org>.
dongjoon-hyun closed pull request #31007:
URL: https://github.com/apache/spark/pull/31007
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] HyukjinKwon commented on a change in pull request #31007: [SPARK-33983][PYTHON] Update cloudpickle to v1.6.0
Posted by GitBox <gi...@apache.org>.
HyukjinKwon commented on a change in pull request #31007:
URL: https://github.com/apache/spark/pull/31007#discussion_r551236353
##########
File path: python/pyspark/cloudpickle/cloudpickle.py
##########
@@ -236,7 +236,7 @@ def _extract_code_globals(co):
out_names = {names[oparg] for _, oparg in _walk_global_ops(co)}
# Declaring a function inside another one using the "def ..."
- # syntax generates a constant code object corresponding to the one
+ # syntax generates a constant code object corresonding to the one
Review comment:
Its a copy from cloudpickle. I think I would just keep it as is.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] MaxGekk commented on a change in pull request #31007: [SPARK-33983][PYTHON] Update cloudpickle to v1.6.0
Posted by GitBox <gi...@apache.org>.
MaxGekk commented on a change in pull request #31007:
URL: https://github.com/apache/spark/pull/31007#discussion_r551231283
##########
File path: python/pyspark/cloudpickle/cloudpickle.py
##########
@@ -699,7 +699,7 @@ def _make_skel_func(code, cell_count, base_globals=None):
"""
# This function is deprecated and should be removed in cloudpickle 1.7
warnings.warn(
- "A pickle file created using an old (<=1.4.1) version of cloudpickle "
+ "A pickle file created using an old (<=1.4.1) version of cloudpicke "
Review comment:
Could you re-check all changes in comments.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org