[jclouds] JCLOUDS-984: Expose multipart upload component operations (#737)

[Andrew Gaul <no...@github.com>]

@danbroudy @kahing @timuralp @zack-shoylev Please review.  This commit exposes all the component multipart upload operations.  Unfortunately providers have different requirements for part sizes, aborting, adding metadata during initate or complete, and whether the provider offers composite objects (Swift) or merely partial uploads (S3).  I did my best here but give the new `BlobStore` methods extra scrutiny.  Still noodling over how to support this with the local blobstores but it will likely be similar to Swift.

@danbroudy @zack-shoylev I would appreciate your help with GCS and Swift, respectively.  I took a whack at the latter but it needs some more work around part sizes.
You can view, comment on, or merge this pull request online at:

  https://github.com/jclouds/jclouds/pull/737

-- Commit Summary --

  * JCLOUDS-894: Expose multipart component operations
  * List both committed and uncommitted Azure blocks
  * JCLOUDS-894: Expose local blobstore multipart operations
  * JCLOUDS-894: Expose Atmos multipart operations
  * Add S3 list multipart parts
  * JCLOUDS-894: Expose S3 multipart operations
  * JCLOUDS-894: Expose Azure multipart operations
  * JCLOUDS-894: Expose Swift multipart operations
  * JCLOUDS-894: Expose legacy Swift multipart operations

-- File Changes --

    M apis/atmos/src/main/java/org/jclouds/atmos/blobstore/AtmosBlobStore.java (39)
    M apis/openstack-swift/src/main/java/org/jclouds/openstack/swift/v1/blobstore/RegionScopedSwiftBlobStore.java (68)
    M apis/openstack-swift/src/main/java/org/jclouds/openstack/swift/v1/functions/ParseObjectFromResponse.java (3)
    M apis/s3/src/main/java/org/jclouds/s3/S3Client.java (9)
    M apis/s3/src/main/java/org/jclouds/s3/blobstore/S3BlobStore.java (57)
    A apis/s3/src/main/java/org/jclouds/s3/xml/PartIdsFromHttpResponse.java (79)
    M apis/s3/src/test/java/org/jclouds/s3/S3ClientLiveTest.java (4)
    A apis/s3/src/test/java/org/jclouds/s3/xml/PartIdsFromHttpResponseTest.java (56)
    A apis/s3/src/test/resources/multipart-upload-list-parts.xml (31)
    M apis/swift/src/main/java/org/jclouds/openstack/swift/blobstore/SwiftBlobStore.java (39)
    M blobstore/pom.xml (5)
    M blobstore/src/main/java/org/jclouds/blobstore/BlobStore.java (28)
    M blobstore/src/main/java/org/jclouds/blobstore/config/LocalBlobStore.java (50)
    A blobstore/src/main/java/org/jclouds/blobstore/domain/MultipartPart.java (31)
    A blobstore/src/main/java/org/jclouds/blobstore/domain/MultipartUpload.java (33)
    M blobstore/src/main/java/org/jclouds/blobstore/util/ForwardingBlobStore.java (39)
    M blobstore/src/main/java/org/jclouds/blobstore/util/ReadOnlyBlobStore.java (31)
    M blobstore/src/test/java/org/jclouds/blobstore/integration/internal/BaseBlobIntegrationTest.java (98)
    M providers/aws-s3/src/main/java/org/jclouds/aws/s3/blobstore/AWSS3BlobStore.java (4)
    M providers/azureblob/src/main/java/org/jclouds/azureblob/AzureBlobClient.java (2)
    M providers/azureblob/src/main/java/org/jclouds/azureblob/blobstore/AzureBlobStore.java (81)

-- Patch Links --

https://github.com/jclouds/jclouds/pull/737.patch
https://github.com/jclouds/jclouds/pull/737.diff

---
Reply to this email directly or view it on GitHub:
https://github.com/jclouds/jclouds/pull/737

Re: [jclouds] JCLOUDS-894: Expose multipart upload component operations (#737)

[Andrew Gaul <no...@github.com>]

Implemented `LocalBlobStore` in latest commit.

---
Reply to this email directly or view it on GitHub:
https://github.com/jclouds/jclouds/pull/737#issuecomment-97313428

Re: [jclouds] JCLOUDS-894: Expose multipart upload component operations (#737)

[Andrew Gaul <no...@github.com>]

Regarding test failures, I have not implemented the `LocalBlobStore` operations yet.

---
Reply to this email directly or view it on GitHub:
https://github.com/jclouds/jclouds/pull/737#issuecomment-97244871

Re: [jclouds] JCLOUDS-894: Expose multipart upload component operations (#737)

[Zack Shoylev <no...@github.com>]

> +      ImmutableList.Builder<Segment> builder = ImmutableList.builder();
> +      for (MultipartPart part : parts) {
> +         String path = mpu.containerName() + "/" + mpu.blobName() + "-" + part.partNumber();
> +         builder.add(Segment.builder().path(path).etag(part.partETag()).sizeBytes(part.partSize()).build());
> +      }
> +      Map<String, String> metadata = ImmutableMap.of();  // TODO: how to populate this from mpu.blobMetadata()?
> +      return api.getStaticLargeObjectApi(regionId, mpu.containerName()).replaceManifest(mpu.blobName(),
> +            builder.build(), metadata);
> +   }
> +
> +   @Override
> +   public MultipartPart uploadMultipartPart(MultipartUpload mpu, int partNumber, Payload payload) {
> +      String partName = mpu.blobName() + "-" + partNumber;
> +      String eTag = api.getObjectApi(regionId, mpu.containerName()).put(partName, payload);
> +      // TODO: only works for payloads where the caller specified the length.  How else can we get at the size?
> +      // ObjectApi.getWithoutBody does not return the size.

Actually content metadata returns the content-length without having to get the whole object.
http://docs.rackspace.com/files/api/v1/cf-devguide/content/HEAD_retrieveobjectmeta_v1__account___container___object__objectServicesOperations_d1e000.html

However, at the moment it gets filtered out I think. So this might require additional work first.

---
Reply to this email directly or view it on GitHub:
https://github.com/jclouds/jclouds/pull/737/files#r29293784

Re: [jclouds] JCLOUDS-894: Expose multipart upload component operations (#737)

[danbroudy <no...@github.com>]

Thanks for the heads up on this! I added the unimplemented placeholder like Atmos and will update in the future. 

---
Reply to this email directly or view it on GitHub:
https://github.com/jclouds/jclouds/pull/737#issuecomment-98610063

Re: [jclouds] JCLOUDS-894: Expose multipart upload component operations (#737)

[Zack Shoylev <no...@github.com>]

Some test failures

https://jclouds.ci.cloudbees.com/job/jclouds-pull-requests/1728/org.apache.jclouds$jclouds-blobstore/console

---
Reply to this email directly or view it on GitHub:
https://github.com/jclouds/jclouds/pull/737#issuecomment-97243820