[GitHub] [jclouds] gaul commented on a change in pull request #89: JCLOUDS-1521: Remove need for content length when using multipart upload

[GitBox <gi...@apache.org>]

gaul commented on a change in pull request #89:
URL: https://github.com/apache/jclouds/pull/89#discussion_r584266806



##########
File path: apis/openstack-swift/src/main/java/org/jclouds/openstack/swift/v1/blobstore/RegionScopedSwiftBlobStore.java
##########
@@ -638,22 +639,45 @@ protected String putMultipartBlob(String container, Blob blob, PutOptions overri
    @Beta
    protected String putMultipartBlob(String container, Blob blob, PutOptions overrides, ListeningExecutorService executor) {
       ArrayList<ListenableFuture<MultipartPart>> parts = new ArrayList<ListenableFuture<MultipartPart>>();
+      MultipartUpload mpu = initiateMultipartUpload(container, blob.getMetadata(), overrides);
+      Payload payload = blob.getPayload();
+      boolean repeatable = payload.isRepeatable();
+      MultipartUploadChunkSizeCalculator calculator = new MultipartUploadChunkSizeCalculator(
+          getMinimumMultipartPartSize(), getMaximumMultipartPartSize());
+      long partSize = calculator.getPartSize();
+      int partNumber = 1;
+      int read;
+
+      try (PushbackInputStream is = new PushbackInputStream(payload.openStream())) {
+         InputStream wrapper = new FilterInputStream(is) {
+            @Override
+            public long skip(long offset) throws IOException {
+               if (repeatable) {
+                  return in.skip(offset);
+               }
+               return offset;
+            }
 
-      long contentLength = checkNotNull(blob.getMetadata().getContentMetadata().getContentLength(),
-            "must provide content-length to use multi-part upload");
-      MultipartUploadSlicingAlgorithm algorithm = new MultipartUploadSlicingAlgorithm(
-            getMinimumMultipartPartSize(), getMaximumMultipartPartSize(), getMaximumNumberOfParts());
-      long partSize = algorithm.calculateChunkSize(contentLength);
-      MultipartUpload mpu = initiateMultipartUpload(container, blob.getMetadata(), partSize, overrides);
-      int partNumber = 0;
+            @Override
+            public void close() {
+               //do not close the underlying stream
+            }
+         };
+
+         while ((read = is.read()) > -1) {
+            is.unread(read);
+            Payload slice = Payloads.newInputStreamPayload(ByteStreams.limit(wrapper, partSize));

Review comment:
       It has been a while since I looked at this code, but does this buffer the entire stream in-memory when using a non-repeateable `Payload`?  




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org