You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@spark.apache.org by sh...@apache.org on 2015/07/07 18:43:10 UTC
spark git commit: [SPARK-8821] [EC2] Switched to binary mode for file
reading
Repository: spark
Updated Branches:
refs/heads/branch-1.4 bf8b47d17 -> 83a621a5a
[SPARK-8821] [EC2] Switched to binary mode for file reading
Otherwise the script will crash with
- Downloading boto...
Traceback (most recent call last):
File "ec2/spark_ec2.py", line 148, in <module>
setup_external_libs(external_libs)
File "ec2/spark_ec2.py", line 128, in setup_external_libs
if hashlib.md5(tar.read()).hexdigest() != lib["md5"]:
File "/usr/lib/python3.4/codecs.py", line 319, in decode
(result, consumed) = self._buffer_decode(data, self.errors, final)
UnicodeDecodeError: 'utf-8' codec can't decode byte 0x8b in position 1: invalid start byte
In case of an utf8 env setting.
Author: Simon Hafner <ha...@gmail.com>
Closes #7215 from reactormonk/branch-1.4 and squashes the following commits:
e86957a [Simon Hafner] [SPARK-8821] [EC2] Switched to binary mode
Project: http://git-wip-us.apache.org/repos/asf/spark/repo
Commit: http://git-wip-us.apache.org/repos/asf/spark/commit/83a621a5
Tree: http://git-wip-us.apache.org/repos/asf/spark/tree/83a621a5
Diff: http://git-wip-us.apache.org/repos/asf/spark/diff/83a621a5
Branch: refs/heads/branch-1.4
Commit: 83a621a5a8f8a2991c4cfa687279589e5c623d46
Parents: bf8b47d
Author: Simon Hafner <ha...@gmail.com>
Authored: Tue Jul 7 09:42:59 2015 -0700
Committer: Shivaram Venkataraman <sh...@cs.berkeley.edu>
Committed: Tue Jul 7 09:42:59 2015 -0700
----------------------------------------------------------------------
ec2/spark_ec2.py | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
----------------------------------------------------------------------
http://git-wip-us.apache.org/repos/asf/spark/blob/83a621a5/ec2/spark_ec2.py
----------------------------------------------------------------------
diff --git a/ec2/spark_ec2.py b/ec2/spark_ec2.py
index 05fa47f..91f0a24 100755
--- a/ec2/spark_ec2.py
+++ b/ec2/spark_ec2.py
@@ -127,7 +127,7 @@ def setup_external_libs(libs):
)
with open(tgz_file_path, "wb") as tgz_file:
tgz_file.write(download_stream.read())
- with open(tgz_file_path) as tar:
+ with open(tgz_file_path, "rb") as tar:
if hashlib.md5(tar.read()).hexdigest() != lib["md5"]:
print("ERROR: Got wrong md5sum for {lib}.".format(lib=lib["name"]), file=stderr)
sys.exit(1)
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@spark.apache.org
For additional commands, e-mail: commits-help@spark.apache.org