[jira] [Commented] (JCLOUDS-1368) GCS cannot upload multipart file larger than 1 GB

["ASF subversion and git services (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/JCLOUDS-1368?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16319143#comment-16319143 ] 

ASF subversion and git services commented on JCLOUDS-1368:
----------------------------------------------------------

Commit d05be89614b4539b2ac16e911439dcb3df2e23df in jclouds's branch refs/heads/master from [~gaul]
[ https://git-wip-us.apache.org/repos/asf?p=jclouds.git;h=d05be89 ]

JCLOUDS-1368: Correct use of slicing algorithm

MultipartUploadSlicingAlgorithm creates multiple equal-sized parts and
a remaining amount.  BaseBlobStore.putMultipartBlob used this
interface incorrectly, which could create more parts than intended
since the remaining size could be larger than the part size.  This
manifested with Google Cloud Storage which only allows 32 parts.


> GCS cannot upload multipart file larger than 1 GB
> -------------------------------------------------
>
>                 Key: JCLOUDS-1368
>                 URL: https://issues.apache.org/jira/browse/JCLOUDS-1368
>             Project: jclouds
>          Issue Type: Bug
>          Components: jclouds-blobstore
>    Affects Versions: 2.0.3
>            Reporter: Andrew Gaul
>            Assignee: Andrew Gaul
>              Labels: google-cloud-storage
>
> When uploading a file:
> {code:java}
> long size = 33L * (32L * 1024L * 1024L);
> ByteSource byteSource = TestUtils.randomByteSource().slice(0, size);
> blobStore.createContainerInLocation(null, containerName);
> Blob blob = blobStore.blobBuilder(blobName)
>       .payload(byteSource)
>       .contentLength(size)
>       .build();
> blobStore.putBlob(containerName, blob, PutOptions.Builder.multipart());
> {code}
> I see an error:
> {noformat}
> org.jclouds.http.HttpResponseException: command: POST https://www.googleapis.com/storage/v1/b/gaul-blobstore0/o/large-blob/compose HTTP/1.1 failed with response: HTTP/1.1 400 Bad Request; content: [{
>  "error": {
>   "errors": [
>    {
>     "domain": "global",
>     "reason": "invalid",
>     "message": "The number of source components provided (33) exceeds the maximum (32)"
>    }
>   ],
>   "code": 400,
>   "message": "The number of source components provided (33) exceeds the maximum (32)"
>  }
> }
> ]
> {noformat}
> This is due to the wonky {{MultipartUploadSlicingAlgorithm}} slicing into (31) 32 MB parts then creating a remainder 64 MB part that {{BaseBlobStore.putMultipartBlob}} does not account for.  This interface seems stupid and better to create (32) 33 MB parts.  Originally reported at:
> https://stackoverflow.com/questions/47048264/the-number-of-source-components-provided-78-exceeds-the-maximum-32



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)