You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by dgingrich <gi...@git.apache.org> on 2017/02/08 01:08:29 UTC
[GitHub] spark pull request #16845: [SPARK-19505][Python] AttributeError on Exception...
GitHub user dgingrich opened a pull request:
https://github.com/apache/spark/pull/16845
[SPARK-19505][Python] AttributeError on Exception.message in Python3
## What changes were proposed in this pull request?
Switch from "e.message" to "str(e)" for Python 3 compatibility. Grepped for all occurrences of `.message` in `pyspark/` and these were the only occurrences.
## How was this patch tested?
- No regression of automated tests (`python/run-tests.py` passed, `dev/run-tests` had failures when pulling from `master` and same failures afterwards)
- Code inspection
- Manually tested the `e.message` -> `str(e)` change in the REPL:
Python 2:
```
$ python
Python 2.7.10 (default, Jul 30 2016, 19:40:32)
[GCC 4.2.1 Compatible Apple LLVM 8.0.0 (clang-800.0.34)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> e = Exception('test')
>>> str(e)
'test'
>>> e.message
'test'
```
Python 3:
```
$ python3
Python 3.4.5 (default, Sep 29 2016, 00:36:58)
[GCC 4.2.1 Compatible Apple LLVM 7.3.0 (clang-703.0.31)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> e = Exception('test')
>>> str(e)
'test'
>>> e.message
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
AttributeError: 'Exception' object has no attribute 'message'
```
Open to suggestions on adding better unit tests for the failure itself. Could do a conditional `unittest.mock` import (like for `unittest2`) and patch the underlying calls but that seemed like overkill.
## Legal
This is my original work and that I license the work to the project under the project\u2019s open source license.
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/dgingrich/spark topic-spark-19505-py3-exceptions
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/spark/pull/16845.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #16845
----
commit cb5c34d80b0f0551514d0153a543210ba1e7ea0e
Author: David Gingrich <da...@textio.com>
Date: 2017-02-08T00:14:21Z
Support Python 3 exception messages
* Switch from "e.message" to "str(e)"
----
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #16845: [SPARK-19505][Python] AttributeError on Exception.messag...
Posted by HyukjinKwon <gi...@git.apache.org>.
Github user HyukjinKwon commented on the issue:
https://github.com/apache/spark/pull/16845
Yes, up to my knowledge, `message` attribute is deprecated from 2.6 and removed in 3.x. It seems reasonable. Let me please @holdenk.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #16845: [SPARK-19505][Python] AttributeError on Exception.messag...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/16845
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/73446/
Test PASSed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #16845: [SPARK-19505][Python] AttributeError on Exception.messag...
Posted by holdenk <gi...@git.apache.org>.
Github user holdenk commented on the issue:
https://github.com/apache/spark/pull/16845
Let's see if I can ask Jenkins to test or if I need to get another list updates for that. Jenkins ok to test.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #16845: [SPARK-19505][Python] AttributeError on Exception.messag...
Posted by HyukjinKwon <gi...@git.apache.org>.
Github user HyukjinKwon commented on the issue:
https://github.com/apache/spark/pull/16845
(Actually, I personally prefer `str(e)` unless we can identify any actual case although it is a potential problem but maybe too narrow to introduce another util.. Maybe, I am being too cautious.)
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #16845: [SPARK-19505][Python] AttributeError on Exception.messag...
Posted by dgingrich <gi...@git.apache.org>.
Github user dgingrich commented on the issue:
https://github.com/apache/spark/pull/16845
Ping! Let me know if you need more work from me.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #16845: [SPARK-19505][Python] AttributeError on Exception.messag...
Posted by dgingrich <gi...@git.apache.org>.
Github user dgingrich commented on the issue:
https://github.com/apache/spark/pull/16845
IMO the helper is better since the Exception message is controlled by the user. I thought Holden was +1'ing the problem (not handling unicode) but re-reading he might have been +1'ing that it was fine. Let me know which way you prefer, it's trivial to switch back to using `str`.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #16845: [SPARK-19505][Python] AttributeError on Exception.messag...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/16845
Merged build finished. Test PASSed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #16845: [SPARK-19505][Python] AttributeError on Exception...
Posted by asfgit <gi...@git.apache.org>.
Github user asfgit closed the pull request at:
https://github.com/apache/spark/pull/16845
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #16845: [SPARK-19505][Python] AttributeError on Exception...
Posted by holdenk <gi...@git.apache.org>.
Github user holdenk commented on a diff in the pull request:
https://github.com/apache/spark/pull/16845#discussion_r103054615
--- Diff: python/pyspark/util.py ---
@@ -0,0 +1,45 @@
+# -*- coding: utf-8 -*-
+#
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+
+__all__ = []
+
+
+def _exception_message(excp):
+ """Return the message from an exception as either a str or unicode object. Supports both
+ Python 2 and Python 3.
+
+ >>> msg = "Exception message"
+ >>> excp = Exception(msg)
+ >>> msg == _exception_message(excp)
+ True
+
+ >>> msg = u"unic�de"
+ >>> excp = Exception(msg)
+ >>> msg == _exception_message(excp)
+ True
+ """
+ if hasattr(excp, "message"):
+ return excp.message
+ return str(excp)
+
+
+if __name__ == "__main__":
+ import doctest
+ (failure_count, test_count) = doctest.testmod()
--- End diff --
These tests won't be run by the default test script, we could fix this by updating the list of things to test - but since this function is only used in one place for now maybe it would make sense to just put this in cloudpickle.py anyways. What do you think?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #16845: [SPARK-19505][Python] AttributeError on Exception.messag...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/16845
**[Test build #75631 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/75631/testReport)** for PR 16845 at commit [`91dc9bf`](https://github.com/apache/spark/commit/91dc9bf10e0c52f4bb477bebe918688fb95d3707).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #16845: [SPARK-19505][Python] AttributeError on Exception.messag...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/16845
Can one of the admins verify this patch?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #16845: [SPARK-19505][Python] AttributeError on Exception...
Posted by holdenk <gi...@git.apache.org>.
Github user holdenk commented on a diff in the pull request:
https://github.com/apache/spark/pull/16845#discussion_r103532524
--- Diff: python/pyspark/util.py ---
@@ -0,0 +1,45 @@
+# -*- coding: utf-8 -*-
+#
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+
+__all__ = []
+
+
+def _exception_message(excp):
+ """Return the message from an exception as either a str or unicode object. Supports both
+ Python 2 and Python 3.
+
+ >>> msg = "Exception message"
+ >>> excp = Exception(msg)
+ >>> msg == _exception_message(excp)
+ True
+
+ >>> msg = u"unic�de"
+ >>> excp = Exception(msg)
+ >>> msg == _exception_message(excp)
+ True
+ """
+ if hasattr(excp, "message"):
+ return excp.message
+ return str(excp)
+
+
+if __name__ == "__main__":
+ import doctest
+ (failure_count, test_count) = doctest.testmod()
--- End diff --
That seems reasonable, in that case you will need to make sure this is called by the test scripts (take a look at `./dev/sparktestsupport/modules.py`).
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #16845: [SPARK-19505][Python] AttributeError on Exception.messag...
Posted by holdenk <gi...@git.apache.org>.
Github user holdenk commented on the issue:
https://github.com/apache/spark/pull/16845
Ok in theory I've been added, so lets see Jenkins test this please.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #16845: [SPARK-19505][Python] AttributeError on Exception...
Posted by dgingrich <gi...@git.apache.org>.
Github user dgingrich commented on a diff in the pull request:
https://github.com/apache/spark/pull/16845#discussion_r100175968
--- Diff: python/pyspark/broadcast.py ---
@@ -82,7 +82,7 @@ def dump(self, value, f):
except pickle.PickleError:
raise
except Exception as e:
- msg = "Could not serialize broadcast: " + e.__class__.__name__ + ": " + e.message
+ msg = "Could not serialize broadcast: " + e.__class__.__name__ + ": " + str(e)
--- End diff --
Good call, I forgot about Python 2's `str`/`unicode` behavior. It could be a problem if an exception includes user input as part of the message. IMO it's worth handling it rigorously since throwing in the `except:` hides the original error. I'll look at fixes tonight, in the worse case we could add a `get_exception_message()` helper.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #16845: [SPARK-19505][Python] AttributeError on Exception.messag...
Posted by holdenk <gi...@git.apache.org>.
Github user holdenk commented on the issue:
https://github.com/apache/spark/pull/16845
Let's make sure everything is still fine since we've made some changes in the testing infrastructure though - if it passes I'll merge this tonight (european time).
Jenkins retest this please.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #16845: [SPARK-19505][Python] AttributeError on Exception...
Posted by dgingrich <gi...@git.apache.org>.
Github user dgingrich commented on a diff in the pull request:
https://github.com/apache/spark/pull/16845#discussion_r105249885
--- Diff: python/pyspark/util.py ---
@@ -0,0 +1,45 @@
+# -*- coding: utf-8 -*-
+#
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+
+__all__ = []
+
+
+def _exception_message(excp):
+ """Return the message from an exception as either a str or unicode object. Supports both
+ Python 2 and Python 3.
+
+ >>> msg = "Exception message"
+ >>> excp = Exception(msg)
+ >>> msg == _exception_message(excp)
+ True
+
+ >>> msg = u"unic�de"
+ >>> excp = Exception(msg)
+ >>> msg == _exception_message(excp)
+ True
+ """
+ if hasattr(excp, "message"):
+ return excp.message
+ return str(excp)
+
+
+if __name__ == "__main__":
+ import doctest
+ (failure_count, test_count) = doctest.testmod()
--- End diff --
Thanks Holden, looking at this today (sorry for the delay, been busy at work).
FWIW the patch to add support for the utility function `six` was rejected: https://github.com/benjaminp/six/pull/177
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #16845: [SPARK-19505][Python] AttributeError on Exception.messag...
Posted by holdenk <gi...@git.apache.org>.
Github user holdenk commented on the issue:
https://github.com/apache/spark/pull/16845
+1 to @HyukjinKwon comment.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #16845: [SPARK-19505][Python] AttributeError on Exception...
Posted by holdenk <gi...@git.apache.org>.
Github user holdenk commented on a diff in the pull request:
https://github.com/apache/spark/pull/16845#discussion_r101346487
--- Diff: python/pyspark/broadcast.py ---
@@ -82,7 +83,8 @@ def dump(self, value, f):
except pickle.PickleError:
raise
except Exception as e:
- msg = "Could not serialize broadcast: " + e.__class__.__name__ + ": " + e.message
+ msg = ("Could not serialize broadcast: " + e.__class__.__name__ + ": "
--- End diff --
Would this maybe be easier to read using format rather than string concatenation?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #16845: [SPARK-19505][Python] AttributeError on Exception.messag...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/16845
**[Test build #73446 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/73446/testReport)** for PR 16845 at commit [`7b8ace4`](https://github.com/apache/spark/commit/7b8ace4eec126ebef289cac067fa80f0d680d278).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #16845: [SPARK-19505][Python] AttributeError on Exception.messag...
Posted by holdenk <gi...@git.apache.org>.
Github user holdenk commented on the issue:
https://github.com/apache/spark/pull/16845
Ok seems like I need to bug one of the other committers to ask Jenkins to test this, maybe @davies can do this in the meantime?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #16845: [SPARK-19505][Python] AttributeError on Exception.messag...
Posted by holdenk <gi...@git.apache.org>.
Github user holdenk commented on the issue:
https://github.com/apache/spark/pull/16845
Jenkins, retest this please.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #16845: [SPARK-19505][Python] AttributeError on Exception.messag...
Posted by holdenk <gi...@git.apache.org>.
Github user holdenk commented on the issue:
https://github.com/apache/spark/pull/16845
LGTM - sorry for the slowness.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #16845: [SPARK-19505][Python] AttributeError on Exception.messag...
Posted by HyukjinKwon <gi...@git.apache.org>.
Github user HyukjinKwon commented on the issue:
https://github.com/apache/spark/pull/16845
Yea, I think she is meant to decide this :).
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #16845: [SPARK-19505][Python] AttributeError on Exception.messag...
Posted by dgingrich <gi...@git.apache.org>.
Github user dgingrich commented on the issue:
https://github.com/apache/spark/pull/16845
Another alternate is to use [`six`](https://pythonhosted.org/six/). In the standard library `text_type`) works:
```python
# Python2.6
>>> import six
>>> six.text_type(Exception(u"unic�de"))
u'unic\xf6de'
```
I also submitted a patch to six to add an `exception_message` helper (https://github.com/benjaminp/six/pull/177) but I don't know if it will be accepted since `Exception.message` has been deprecated since 2.6.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #16845: [SPARK-19505][Python] AttributeError on Exception.messag...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/16845
**[Test build #73446 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/73446/testReport)** for PR 16845 at commit [`7b8ace4`](https://github.com/apache/spark/commit/7b8ace4eec126ebef289cac067fa80f0d680d278).
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #16845: [SPARK-19505][Python] AttributeError on Exception.messag...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/16845
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/75631/
Test PASSed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #16845: [SPARK-19505][Python] AttributeError on Exception.messag...
Posted by holdenk <gi...@git.apache.org>.
Github user holdenk commented on the issue:
https://github.com/apache/spark/pull/16845
Merged to master
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #16845: [SPARK-19505][Python] AttributeError on Exception.messag...
Posted by holdenk <gi...@git.apache.org>.
Github user holdenk commented on the issue:
https://github.com/apache/spark/pull/16845
So being able to format the messages in the 2.X line and 3 line seems like a reasonable improvement, even if we are using a deprecated feature in 2.6+ for the 2.X line. Let me know when you have a chance to update the PR so the tests are run.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #16845: [SPARK-19505][Python] AttributeError on Exception...
Posted by dgingrich <gi...@git.apache.org>.
Github user dgingrich commented on a diff in the pull request:
https://github.com/apache/spark/pull/16845#discussion_r103401522
--- Diff: python/pyspark/util.py ---
@@ -0,0 +1,45 @@
+# -*- coding: utf-8 -*-
+#
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+
+__all__ = []
+
+
+def _exception_message(excp):
+ """Return the message from an exception as either a str or unicode object. Supports both
+ Python 2 and Python 3.
+
+ >>> msg = "Exception message"
+ >>> excp = Exception(msg)
+ >>> msg == _exception_message(excp)
+ True
+
+ >>> msg = u"unic�de"
+ >>> excp = Exception(msg)
+ >>> msg == _exception_message(excp)
+ True
+ """
+ if hasattr(excp, "message"):
+ return excp.message
+ return str(excp)
+
+
+if __name__ == "__main__":
+ import doctest
+ (failure_count, test_count) = doctest.testmod()
--- End diff --
Well the util is called from both cloudpickle.py and broadcast.py, so I'd rather leave the helper here and add it the list of things to test. Leaving it as is for now but willing to change.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #16845: [SPARK-19505][Python] AttributeError on Exception.messag...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/16845
**[Test build #75631 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/75631/testReport)** for PR 16845 at commit [`91dc9bf`](https://github.com/apache/spark/commit/91dc9bf10e0c52f4bb477bebe918688fb95d3707).
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #16845: [SPARK-19505][Python] AttributeError on Exception.messag...
Posted by dgingrich <gi...@git.apache.org>.
Github user dgingrich commented on the issue:
https://github.com/apache/spark/pull/16845
Great, thanks Holden!
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #16845: [SPARK-19505][Python] AttributeError on Exception...
Posted by HyukjinKwon <gi...@git.apache.org>.
Github user HyukjinKwon commented on a diff in the pull request:
https://github.com/apache/spark/pull/16845#discussion_r99999008
--- Diff: python/pyspark/broadcast.py ---
@@ -82,7 +82,7 @@ def dump(self, value, f):
except pickle.PickleError:
raise
except Exception as e:
- msg = "Could not serialize broadcast: " + e.__class__.__name__ + ": " + e.message
+ msg = "Could not serialize broadcast: " + e.__class__.__name__ + ": " + str(e)
--- End diff --
BTW, we should keep in mind that
```python
>>> "" + Exception(u"j�rn").message
u'j\xf6rn'
>>> "" + str(Exception(u"j�rn"))
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
UnicodeEncodeError: 'ascii' codec can't encode character u'\xf6' in position 1: ordinal not in range(128)
```
this could cause such encoding problem. It'd be really rare case though I think. It seems this usage is already there in the code base so I assume it is fine.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #16845: [SPARK-19505][Python] AttributeError on Exception.messag...
Posted by dgingrich <gi...@git.apache.org>.
Github user dgingrich commented on the issue:
https://github.com/apache/spark/pull/16845
Added to `pyspark.util` to modules. Let me know if you want any other changes.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #16845: [SPARK-19505][Python] AttributeError on Exception.messag...
Posted by dgingrich <gi...@git.apache.org>.
Github user dgingrich commented on the issue:
https://github.com/apache/spark/pull/16845
Added a helper that falls back from `e.message` to `str(e)`. Also rebased to master.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #16845: [SPARK-19505][Python] AttributeError on Exception.messag...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/16845
Merged build finished. Test PASSed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #16845: [SPARK-19505][Python] AttributeError on Exception...
Posted by dgingrich <gi...@git.apache.org>.
Github user dgingrich commented on a diff in the pull request:
https://github.com/apache/spark/pull/16845#discussion_r103400340
--- Diff: python/pyspark/broadcast.py ---
@@ -82,7 +83,8 @@ def dump(self, value, f):
except pickle.PickleError:
raise
except Exception as e:
- msg = "Could not serialize broadcast: " + e.__class__.__name__ + ": " + e.message
+ msg = ("Could not serialize broadcast: " + e.__class__.__name__ + ": "
--- End diff --
Sure, I was trying to minimize changes but I prefer formatting with `%`. Switched.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #16845: [SPARK-19505][Python] AttributeError on Exception.messag...
Posted by HyukjinKwon <gi...@git.apache.org>.
Github user HyukjinKwon commented on the issue:
https://github.com/apache/spark/pull/16845
BTW, could anyone say "ok to test"? Anyhow it is true that `Exception` does not have `message` in Python 3 and we need a fix.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org