You are viewing a plain text version of this content. The canonical link for it is here.
Posted to notifications@libcloud.apache.org by "Alexey (JIRA)" <ji...@apache.org> on 2016/01/12 07:31:39 UTC
[jira] [Updated] (LIBCLOUD-792) S3 SignatureDoesNotMatch
[ https://issues.apache.org/jira/browse/LIBCLOUD-792?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Alexey updated LIBCLOUD-792:
----------------------------
Flags: Patch
Description:
I noticed that *sometimes* upload fails with SignatureDoesNotMatch error. I tried to debug this, and I got the following notice in the error message from S3:
{code:xml}
<StringToSign>PUT\nE8F2ok2KBiJTfNy91PLz+A==\n\n1452529414\n/mybucket/11.png?partNumber=1&uploadId=DQGexmin_N3usw4giAtLWoRbCWWRWQij2a20xPO_BgSRENZRpgGqhgMm9goSPoHkmkheIMBrPcI_Z2xWZtqMmZcmekuVhpTBre4cuS.nfyP6DQuTPEuAYop5abOHbm2t</StringToSign>
{code}
and in the code libcloud/storage/drivers/s3.py:147 string_to_sign equals the following
{code:none}
string_to_sign = "PUT\nE8F2ok2KBiJTfNy91PLz+A==\n\n1452529414\n/mybucket/11.png?uploadId=DQGexmin_N3usw4giAtLWoRbCWWRWQij2a20xPO_BgSRENZRpgGqhgMm9goSPoHkmkheIMBrPcI_Z2xWZtqMmZcmekuVhpTBre4cuS.nfyP6DQuTPEuAYop5abOHbm2t&partNumber=1"
{code}
Obviously the order of GET parameters is not preserved. For me the following patch do the trick, it keeps parameters of multipart upload in order:
{code:none}
diff --git a/libcloud/storage/drivers/s3.py b/libcloud/storage/drivers/s3.py
index 77a86de..1b5a5c0 100644
--- a/libcloud/storage/drivers/s3.py
+++ b/libcloud/storage/drivers/s3.py
@@ -20,6 +20,8 @@ import sys
from hashlib import sha1
+from collections import OrderedDict
+
try:
from lxml.etree import Element, SubElement
except ImportError:
@@ -520,7 +522,9 @@ class BaseS3StorageDriver(StorageDriver):
bytes_transferred = 0
count = 1
chunks = []
- params = {'uploadId': upload_id}
+ params = OrderedDict()
+ params['partNumber'] = 0
+ params['uploadId'] = upload_id
# Read the input data in chunk sizes suitable for AWS
for data in read_in_chunks(iterator, chunk_size=CHUNK_SIZE,
{code}
was:
I noticed that *sometimes* upload fails with SignatureDoesNotMatch error. I tried to debug this, and I got the following notice in the error message from S3:
{code:xml}
<StringToSign>PUT\nE8F2ok2KBiJTfNy91PLz+A==\n\n1452529414\n/mybucket/11.png?partNumber=1&uploadId=DQGexmin_N3usw4giAtLWoRbCWWRWQij2a20xPO_BgSRENZRpgGqhgMm9goSPoHkmkheIMBrPcI_Z2xWZtqMmZcmekuVhpTBre4cuS.nfyP6DQuTPEuAYop5abOHbm2t</StringToSign>
{code}
and in the code libcloud/storage/drivers/s3.py:147 string_to_sign equals the following
{code:none}
string_to_sign = "PUT\nE8F2ok2KBiJTfNy91PLz+A==\n\n1452529414\n/mybucket/11.png?uploadId=DQGexmin_N3usw4giAtLWoRbCWWRWQij2a20xPO_BgSRENZRpgGqhgMm9goSPoHkmkheIMBrPcI_Z2xWZtqMmZcmekuVhpTBre4cuS.nfyP6DQuTPEuAYop5abOHbm2t&partNumber=1"
{code}
Obviously the order of GET parameters is not preserved. I have no idea how to hot-fix this issue (except replacing *params* dictionary with OrderedDict or array of tuples, but this is a pretty big change)
> S3 SignatureDoesNotMatch
> ------------------------
>
> Key: LIBCLOUD-792
> URL: https://issues.apache.org/jira/browse/LIBCLOUD-792
> Project: Libcloud
> Issue Type: Bug
> Components: Storage
> Reporter: Alexey
>
> I noticed that *sometimes* upload fails with SignatureDoesNotMatch error. I tried to debug this, and I got the following notice in the error message from S3:
> {code:xml}
> <StringToSign>PUT\nE8F2ok2KBiJTfNy91PLz+A==\n\n1452529414\n/mybucket/11.png?partNumber=1&uploadId=DQGexmin_N3usw4giAtLWoRbCWWRWQij2a20xPO_BgSRENZRpgGqhgMm9goSPoHkmkheIMBrPcI_Z2xWZtqMmZcmekuVhpTBre4cuS.nfyP6DQuTPEuAYop5abOHbm2t</StringToSign>
> {code}
> and in the code libcloud/storage/drivers/s3.py:147 string_to_sign equals the following
> {code:none}
> string_to_sign = "PUT\nE8F2ok2KBiJTfNy91PLz+A==\n\n1452529414\n/mybucket/11.png?uploadId=DQGexmin_N3usw4giAtLWoRbCWWRWQij2a20xPO_BgSRENZRpgGqhgMm9goSPoHkmkheIMBrPcI_Z2xWZtqMmZcmekuVhpTBre4cuS.nfyP6DQuTPEuAYop5abOHbm2t&partNumber=1"
> {code}
> Obviously the order of GET parameters is not preserved. For me the following patch do the trick, it keeps parameters of multipart upload in order:
> {code:none}
> diff --git a/libcloud/storage/drivers/s3.py b/libcloud/storage/drivers/s3.py
> index 77a86de..1b5a5c0 100644
> --- a/libcloud/storage/drivers/s3.py
> +++ b/libcloud/storage/drivers/s3.py
> @@ -20,6 +20,8 @@ import sys
> from hashlib import sha1
> +from collections import OrderedDict
> +
> try:
> from lxml.etree import Element, SubElement
> except ImportError:
> @@ -520,7 +522,9 @@ class BaseS3StorageDriver(StorageDriver):
> bytes_transferred = 0
> count = 1
> chunks = []
> - params = {'uploadId': upload_id}
> + params = OrderedDict()
> + params['partNumber'] = 0
> + params['uploadId'] = upload_id
> # Read the input data in chunk sizes suitable for AWS
> for data in read_in_chunks(iterator, chunk_size=CHUNK_SIZE,
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)