You are viewing a plain text version of this content. The canonical link for it is here.
Posted to notifications@jclouds.apache.org by archupsg03 <no...@github.com> on 2017/05/23 11:45:12 UTC
[jclouds/jclouds] DLO - Lower Level Provider API changes (#1105)
You can view, comment on, or merge this pull request online at:
https://github.com/jclouds/jclouds/pull/1105
-- Commit Summary --
* DLO - Lower Level Provider API changes
-- File Changes --
M apis/openstack-swift/src/main/java/org/jclouds/openstack/swift/v1/SwiftApi.java (6)
A apis/openstack-swift/src/main/java/org/jclouds/openstack/swift/v1/blobstore/functions/ContentMetaDataToMap.java (45)
M apis/openstack-swift/src/main/java/org/jclouds/openstack/swift/v1/domain/Segment.java (2)
A apis/openstack-swift/src/main/java/org/jclouds/openstack/swift/v1/features/DynamicLargeObjectApi.java (106)
A apis/openstack-swift/src/test/java/org/jclouds/openstack/swift/v1/features/DynamicLargeObjectApiLiveTest.java (194)
A apis/openstack-swift/src/test/java/org/jclouds/openstack/swift/v1/features/DynamicLargeObjectApiMockTest.java (181)
-- Patch Links --
https://github.com/jclouds/jclouds/pull/1105.patch
https://github.com/jclouds/jclouds/pull/1105.diff
--
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/jclouds/jclouds/pull/1105
Re: [jclouds/jclouds] DLO - Lower Level Provider API changes (#1105)
Posted by archupsg03 <no...@github.com>.
archupsg03 commented on this pull request.
> + for (int i = partNumber; i < 3; partNumber++) {
+ String objName = String.format("%s/%s/%s", objectName, "dlo", partNumber);
+ String data = String.format("%s%s", "data", partNumber);
+ String etag = getApi().getDynamicLargeObjectApi(regionId, defaultContainerName).uploadPart(
+ defaultContainerName, objName, Payloads.newPayload(data), ImmutableMap.of("myfoo", "Bar"),
+ ImmutableMap.of("myfoo", "Bar"));
+ Segment s = new Segment(objName, etag, data.length());
+ assertNotNull(etag);
+ segmentList.add(s);
+ total_size += data.length();
+ }
+ String etagOfEtags = getApi().getDynamicLargeObjectApi(regionId, defaultContainerName).replaceManifest(
+ objectName, segmentList, ImmutableMap.of("MyFoo", "Bar"), ImmutableMap.of("MyFoo", "Bar"));
+
+ SwiftObject bigObject = getApi().getObjectApi(regionId, defaultContainerName).get(objectName);
+ assertThat(bigObject.getETag().equals(etagOfEtags));
Done
--
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/jclouds/jclouds/pull/1105#discussion_r118907933
Re: [jclouds/jclouds] DLO - Lower Level Provider API changes (#1105)
Posted by archupsg03 <no...@github.com>.
archupsg03 commented on this pull request.
> + super.setup();
+ for (String regionId : regions) {
+ boolean created = getApi().getContainerApi(regionId).create(defaultContainerName);
+ if (!created) {
+ deleteAllObjectsInContainer(regionId, defaultContainerName);
+ }
+ }
+
+ /* megOf1s = new byte[1024 * 1024];
+ megOf2s = new byte[1024 * 1024];
+
+ Arrays.fill(megOf1s, (byte) 1);
+ Arrays.fill(megOf2s, (byte) 2);*/
+
+ megOf1s = TestUtils.randomByteSource().slice(0, 1048576);
+ megOf2s = TestUtils.randomByteSource().slice(0, 1048576);
Done
--
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/jclouds/jclouds/pull/1105#discussion_r118909541
Re: [jclouds/jclouds] DLO - Lower Level Provider API changes (#1105)
Posted by Timur Alperovich <no...@github.com>.
timuralp commented on this pull request.
Here are the ones I spotted. If you can configure your editor to remove trailing white space, please clean them up. If you can't that's fine. Maybe that will get cleaned up before merging.
> + // configure the blobstore to use multipart uploading of the file
+ for (int partNumber = 0; partNumber < 3; partNumber++) {
+ String objName = String.format("%s/%s/%s", objectName, "dlo", partNumber);
+ String data = String.format("%s%s", "data", partNumber);
+ ByteSource payload = ByteSource.wrap(data.getBytes(Charsets.UTF_8));
+ Blob blob = blobStore.blobBuilder(objName)
+ .payload(payload)
+ .build();
+ String etag = blobStore.putBlob(defaultContainerName, blob);
+ assertNotNull(etag);
+ total_size += data.length();
+ }
+
+ getApi().getDynamicLargeObjectApi(regionId, defaultContainerName).putManifest(objectName,
+ ImmutableMap.of("myfoo", "Bar"));
+
There is trailing white space on this line. I don't know if there is a way to get github to highlight it.
> + SwiftObject object1s = getApi().getObjectApi(regionId, containerName).get(name);
+ assertThat(object1s.getETag()).isEqualTo(etag1s);
+ assertThat(object1s.getPayload().getContentMetadata().getContentLength()).isEqualTo(Long.valueOf(1024L * 1024L));
+ }
+
+ protected void deleteAllObjectsInContainerDLO(String regionId, final String containerName) {
+ ObjectList objects = getApi().getObjectApi(regionId, containerName).list(new ListContainerOptions());
+ if (objects == null) {
+ return;
+ }
+ List<String> pathsToDelete = Lists.transform(objects, new Function<SwiftObject, String>() {
+ public String apply(SwiftObject input) {
+ return containerName + "/" + input.getName();
+ }
+ });
+
Trailing white space here and line 134.
> + @SuppressWarnings("deprecation")
+ @Test
+ public void uploadLargeFile() throws Exception {
+ MockWebServer server = mockOpenStackServer();
+ server.enqueue(addCommonHeaders(new MockResponse().setBody(stringFromResource("/access.json"))));
+ server.enqueue(new MockResponse().setBody("").addHeader(ETAG, "89d903bc35dede724fd52c51437ff5fd"));
+ server.enqueue(new MockResponse().setBody("").addHeader(ETAG, "d41d8cd98f00b204e9800998ecf8427e"));
+ server.enqueue(addCommonHeaders(new MockResponse().addHeader("X-Object-Manifest", "myContainer/myObject")));
+
+ try {
+ SwiftApi api = api(server.getUrl("/").toString(), "openstack-swift");
+ assertEquals(api.getObjectApi("DFW", containerName).put(objectName.concat("1"), Payloads.newPayload("data1")),
+ "89d903bc35dede724fd52c51437ff5fd");
+ assertEquals(api.getDynamicLargeObjectApi("DFW", containerName).putManifest(objectName,
+ ImmutableMap.of("MyFoo", "Bar"), ImmutableMap.of("MyFoo", "Bar")), "d41d8cd98f00b204e9800998ecf8427e");
+
Trailing white space here.
> + MockWebServer server = mockOpenStackServer();
+ server.enqueue(addCommonHeaders(new MockResponse().setBody(stringFromResource("/access.json"))));
+ server.enqueue(new MockResponse().setBody("").addHeader(ETAG, "89d903bc35dede724fd52c51437ff5fd"));
+ server.enqueue(new MockResponse().setBody("").addHeader(ETAG, "d41d8cd98f00b204e9800998ecf8427e"));
+ server.enqueue(addCommonHeaders(new MockResponse().addHeader("X-Object-Manifest", "myContainer/myObject")));
+
+ try {
+ SwiftApi api = api(server.getUrl("/").toString(), "openstack-swift");
+ assertEquals(api.getObjectApi("DFW", containerName).put(objectName.concat("1"), Payloads.newPayload("data1")),
+ "89d903bc35dede724fd52c51437ff5fd");
+ assertEquals(api.getDynamicLargeObjectApi("DFW", containerName).putManifest(objectName,
+ ImmutableMap.of("MyFoo", "Bar"), ImmutableMap.of("MyFoo", "Bar")), "d41d8cd98f00b204e9800998ecf8427e");
+
+ assertEquals(server.getRequestCount(), 3);
+ assertAuthentication(server);
+
Trailing white space.
> +
+ try {
+ SwiftApi api = api(server.getUrl("/").toString(), "openstack-swift");
+ assertEquals(api.getObjectApi("DFW", containerName).put(objectName.concat("1"), Payloads.newPayload("data1")),
+ "89d903bc35dede724fd52c51437ff5fd");
+ assertEquals(api.getDynamicLargeObjectApi("DFW", containerName).putManifest(objectName,
+ ImmutableMap.of("MyFoo", "Bar"), ImmutableMap.of("MyFoo", "Bar")), "d41d8cd98f00b204e9800998ecf8427e");
+
+ assertEquals(server.getRequestCount(), 3);
+ assertAuthentication(server);
+
+ RecordedRequest uploadRequest = server.takeRequest();
+ assertEquals(uploadRequest.getRequestLine(),
+ "PUT /v1/MossoCloudFS_5bcf396e-39dd-45ff-93a1-712b9aba90a9/myContainer/myObjectTest1 HTTP/1.1");
+ assertEquals(new String(uploadRequest.getBody()), "data1");
+
Trailing white space.
> @@ -67,4 +68,9 @@ ObjectApi getObjectApi(@EndpointParam(parser = RegionToEndpoint.class) String re
@Path("/{containerName}")
StaticLargeObjectApi getStaticLargeObjectApi(@EndpointParam(parser = RegionToEndpoint.class) String region,
@PathParam("containerName") String containerName);
+
Trailing white space.
> + * @param headers
+ * Binds the map to headers, without prefixing/escaping the header
+ * name/key.
+ *
+ * @return {@link SwiftObject#getEtag()} of the object, which is the MD5
+ * checksum of the concatenated ETag values of the {@code segments}.
+ */
+ @Deprecated
+ @Named("dynamicLargeObject:putManifest")
+ @PUT
+ @ResponseParser(ETagHeader.class)
+ @Headers(keys = "X-Object-Manifest", values = "{containerName}/{objectName}/")
+ String putManifest(@PathParam("objectName") String objectName,
+ @BinderParam(BindObjectMetadataToHeaders.class) Map<String, String> metadata,
+ @BinderParam(BindToHeaders.class) Map<String, String> headers);
+
Trailing white space.
> + RegionScopedBlobStoreContext ctx = RegionScopedBlobStoreContext.class.cast(view);
+ BlobStore blobStore = ctx.getBlobStore();
+ String defaultContainerName = getContainerName();
+ // configure the blobstore to use multipart uploading of the file
+ for (int partNumber = 0; partNumber < 3; partNumber++) {
+ String objName = String.format("%s/%s/%s", objectName, "dlo", partNumber);
+ String data = String.format("%s%s", "data", partNumber);
+ ByteSource payload = ByteSource.wrap(data.getBytes(Charsets.UTF_8));
+ Blob blob = blobStore.blobBuilder(objName)
+ .payload(payload)
+ .build();
+ String etag = blobStore.putBlob(defaultContainerName, blob);
+ assertNotNull(etag);
+ total_size += data.length();
+ }
+
Trailing white space.
--
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/jclouds/jclouds/pull/1105#pullrequestreview-48715778
Re: [jclouds/jclouds] DLO - Lower Level Provider API changes (#1105)
Posted by archupsg03 <no...@github.com>.
@archupsg03 pushed 1 commit.
93b41da CheckStyle Exceptions
--
You are receiving this because you are subscribed to this thread.
View it on GitHub:
https://github.com/jclouds/jclouds/pull/1105/files/2fba0091268ac08364fb045d474b5ae0435b3301..93b41da44f42710ab1ed55a78ea05c88b5b2e7e1
Re: [jclouds/jclouds] DLO - Lower Level Provider API changes (#1105)
Posted by Andrew Gaul <no...@github.com>.
andrewgaul commented on this pull request.
> +@Consumes(APPLICATION_JSON)
+@Path("/{objectName}")
+public interface DynamicLargeObjectApi {
+ /**
+ * Creates or updates a dynamic large object's manifest.
+ *
+ * @param objectName
+ * corresponds to {@link SwiftObject#getName()}.
+ * @param metadata
+ * corresponds to {@link SwiftObject#getMetadata()}.
+ * @param headers
+ * Binds the map to headers, without prefixing/escaping the header
+ * name/key.
+ *
+ * @return {@link SwiftObject#getEtag()} of the object, which is the MD5
+ * checksum of the concatenated ETag values of the {@code segments}.
In addition to the `@Deprecated` annotation which javac consumes, we can document in Javadoc for humans to refers to the SLO API.
--
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/jclouds/jclouds/pull/1105#discussion_r135452928
Re: [jclouds/jclouds] DLO - Lower Level Provider API changes (#1105)
Posted by archupsg03 <no...@github.com>.
@andrewgaul @timuralp any other blockers for merging this PR
--
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/jclouds/jclouds/pull/1105#issuecomment-316906273
Re: [jclouds/jclouds] DLO - Lower Level Provider API changes (#1105)
Posted by archupsg03 <no...@github.com>.
archupsg03 commented on this pull request.
> + * @param metadata
+ * corresponds to {@link SwiftObject#getMetadata()}.
+ * @param headers
+ * Binds the map to headers, without prefixing/escaping the header
+ * name/key.
+ *
+ * @return {@link SwiftObject#getEtag()} of the object, which is the MD5
+ * checksum of the concatenated ETag values of the {@code segments}.
+ */
+
+ @Deprecated
+ @Named("dynamicLargeObject:uploadLargeFile")
+ @PUT
+ @Headers(keys = EXPECT, values = "100-continue")
+ @ResponseParser(ETagHeader.class)
+ String uploadPart(@PathParam("containerName") String container, @PathParam("objectName") String objectName,
Sorry. Did not went through this comment. Yeah it looks like not needed. I added those methods from Andrew's comment
--
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/jclouds/jclouds/pull/1105#discussion_r120548716
Re: [jclouds/jclouds] DLO - Lower Level Provider API changes (#1105)
Posted by Timur Alperovich <no...@github.com>.
timuralp commented on this pull request.
> + * @param metadata
+ * corresponds to {@link SwiftObject#getMetadata()}.
+ * @param headers
+ * Binds the map to headers, without prefixing/escaping the header
+ * name/key.
+ *
+ * @return {@link SwiftObject#getEtag()} of the object, which is the MD5
+ * checksum of the concatenated ETag values of the {@code segments}.
+ */
+ @Deprecated
+ @Named("dynamicLargeObject:replaceManifest")
+ @PUT
+ @ResponseParser(ETagHeader.class)
+ @Headers(keys = "X-Object-Manifest", values = "{containerName}/{objectName}/")
+ String replaceManifest(@PathParam("objectName") String objectName,
+ @BinderParam(BindToJsonPayload.class) Collection<Segment> segments,
@archupsg03 I probably was not clear on this. I was suggesting that using the Swift PutOptions class and setting the correct header (X-Object-Manifest) would be sufficient with the existing PutObject API. However, talking to @andrewgaul, I understand the preference for the separate API class, as it makes the DLO separation from the regular object API obvious.
The X-Object-Manifest header is mandatory in the API, so that header must be set -- that is correct.
In terms of your question on behavior, for DLO, Swift's behavior depends on the number of segments. If the number of segments is less than the full listing, Swift computes DLO ETag as an md5 sum of segment ETags. Otherwise, the ETag is not included. [1]
I'll try to look into the prior jclouds DLO documentation to try to understand your question about the 1.9 implementation.
1. https://github.com/openstack/swift/blob/master/swift/common/middleware/dlo.py#L307
--
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/jclouds/jclouds/pull/1105#discussion_r120170979
Re: [jclouds/jclouds] DLO - Lower Level Provider API changes (#1105)
Posted by archupsg03 <no...@github.com>.
archupsg03 commented on this pull request.
> + @SuppressWarnings("deprecation")
+ @Test
+ public void uploadLargeFile(String regionId) throws IOException, InterruptedException {
+ /*
+ * createRandomFile(SIZE, bigFile); Payload payLoad = new
+ * FilePayload(bigFile);
+ */
+ List<Segment> list1 = new ArrayList<Segment>();
+ int partNumber = 1;
+ int total_size = 0;
+ // configure the blobstore to use multipart uploading of the file
+ /*
+ * payLoad.getContentMetadata().setContentLength(bigFile.length());
+ * BasePayloadSlicer slicer = new BasePayloadSlicer();
+ */
+ while (partNumber < 3) {
Done
--
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/jclouds/jclouds/pull/1105#discussion_r118183857
Re: [jclouds/jclouds] DLO - Lower Level Provider API changes (#1105)
Posted by Andrew Gaul <no...@github.com>.
> Line number 87 in DynamicLargetObjectApiLiveTest is call to ObjectApi.put methods which was existing and this is a basic call. It may be because of some other issue
Please investigate this.
--
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/jclouds/jclouds/pull/1105#issuecomment-325824177
Re: [jclouds/jclouds] DLO - Lower Level Provider API changes (#1105)
Posted by Andrew Gaul <no...@github.com>.
@archupsg03 Sorry I was not available the last month. Thanks @timuralp for stepping up! This looks close to ready, please address review comments so we can merge this. Should we defer DLO delete for a subsequent pull request?
--
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/jclouds/jclouds/pull/1105#issuecomment-313892958
Re: [jclouds/jclouds] DLO - Lower Level Provider API changes (#1105)
Posted by archupsg03 <no...@github.com>.
archupsg03 commented on this pull request.
> + * ordered parts which will be concatenated upon download.
+ * @param metadata
+ * corresponds to {@link SwiftObject#getMetadata()}.
+ * @param headers
+ * Binds the map to headers, without prefixing/escaping the header
+ * name/key.
+ *
+ * @return {@link SwiftObject#getEtag()} of the object, which is the MD5
+ * checksum of the concatenated ETag values of the {@code segments}.
+ */
+ @Deprecated
+ @Named("dynamicLargeObject:replaceManifest")
+ @PUT
+ @ResponseParser(ETagHeader.class)
+ @Headers(keys = "X-Object-Manifest", values = "{containerName}/{objectName}/")
+ String replaceManifest(@PathParam("containerName") String containerName, @PathParam("objectName") String objectName,
Done
--
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/jclouds/jclouds/pull/1105#discussion_r118183180
Re: [jclouds/jclouds] DLO - Lower Level Provider API changes (#1105)
Posted by archupsg03 <no...@github.com>.
archupsg03 commented on this pull request.
> + * @param objectName
+ * corresponds to {@link SwiftObject#getName()}.
+ * @param segments
+ * ordered parts which will be concatenated upon download.
+ * @param metadata
+ * corresponds to {@link SwiftObject#getMetadata()}.
+ * @param headers
+ * Binds the map to headers, without prefixing/escaping the header
+ * name/key.
+ *
+ * @return {@link SwiftObject#getEtag()} of the object, which is the MD5
+ * checksum of the concatenated ETag values of the {@code segments}.
+ */
+
+ @Deprecated
+ @Named("dynamicLargeObject: uploadLargeFile")
Done
--
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/jclouds/jclouds/pull/1105#discussion_r118183231
Re: [jclouds/jclouds] DLO - Lower Level Provider API changes (#1105)
Posted by archupsg03 <no...@github.com>.
archupsg03 commented on this pull request.
> + @Override
+ @BeforeClass(groups = "live")
+ public void setup() {
+ super.setup();
+ for (String regionId : regions) {
+ boolean created = getApi().getContainerApi(regionId).create(defaultContainerName);
+ if (!created) {
+ deleteAllObjectsInContainer(regionId, defaultContainerName);
+ }
+ }
+
+ /* megOf1s = new byte[1024 * 1024];
+ megOf2s = new byte[1024 * 1024];
+
+ Arrays.fill(megOf1s, (byte) 1);
+ Arrays.fill(megOf2s, (byte) 2);*/
Done
--
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/jclouds/jclouds/pull/1105#discussion_r118909356
Re: [jclouds/jclouds] DLO - Lower Level Provider API changes (#1105)
Posted by archupsg03 <no...@github.com>.
@archupsg03 pushed 1 commit.
b8c04a0 DLO - method name change
--
You are receiving this because you are subscribed to this thread.
View it on GitHub:
https://github.com/jclouds/jclouds/pull/1105/files/9906c00c68933634126014e60f1bfd05ade7f897..b8c04a07eae6abd285d13f04d32b0c1ae4a6b2a4
Re: [jclouds/jclouds] DLO - Lower Level Provider API changes (#1105)
Posted by Andrew Gaul <no...@github.com>.
andrewgaul requested changes on this pull request.
@archupsg03 I have more comments but let's address the first set. As you make changes, please reply "Done" to each individual comment.
@timuralp please review `DynamicLargeObjectApi`.
> + @BinderParam(SetPayload.class) Payload blob,
+ @BinderParam(BindObjectMetadataToHeaders.class) Map<String, String> metadata,
+ @BinderParam(BindToHeaders.class) Map<String, String> headers);
+
+ /**
+ * Delete objects.
+ *
+ * @param objectName
+ * corresponds to {@link SwiftObject#getName()}.
+ */
+
+ @Deprecated
+ @Named("dynamicLargeObject: removeLargeFile")
+ @DELETE
+ @Fallback(VoidOnNotFoundOr404.class)
+ void removeLargeFile(@PathParam("containerName") String container, @PathParam("objectName") String name);
What does this do differently than `ObjectApi.delete`?
> + * ordered parts which will be concatenated upon download.
+ * @param metadata
+ * corresponds to {@link SwiftObject#getMetadata()}.
+ * @param headers
+ * Binds the map to headers, without prefixing/escaping the header
+ * name/key.
+ *
+ * @return {@link SwiftObject#getEtag()} of the object, which is the MD5
+ * checksum of the concatenated ETag values of the {@code segments}.
+ */
+ @Deprecated
+ @Named("dynamicLargeObject:replaceManifest")
+ @PUT
+ @ResponseParser(ETagHeader.class)
+ @Headers(keys = "X-Object-Manifest", values = "{containerName}/{objectName}/")
+ String replaceManifest(@PathParam("containerName") String containerName, @PathParam("objectName") String objectName,
Why do these methods take a `containerName` parameter but equivalent `StaticLargeObjectApi` methods do not?
> + @Test
+ public void testReplaceManifest() throws Exception {
+ for (String regionId : regions) {
+ assertReplaceManifest(regionId, defaultContainerName, defaultName);
+ uploadLargeFile(regionId);
+ removeLargeFile(regionId);
+ }
+ }
+
+ @SuppressWarnings("deprecation")
+ @Test
+ public void uploadLargeFile(String regionId) throws IOException, InterruptedException {
+ /*
+ * createRandomFile(SIZE, bigFile); Payload payLoad = new
+ * FilePayload(bigFile);
+ */
Remove commented-out code.
> + * @param metadata
+ * corresponds to {@link SwiftObject#getMetadata()}.
+ * @param headers
+ * Binds the map to headers, without prefixing/escaping the header
+ * name/key.
+ *
+ * @return {@link SwiftObject#getEtag()} of the object, which is the MD5
+ * checksum of the concatenated ETag values of the {@code segments}.
+ */
+ @Deprecated
+ @Named("dynamicLargeObject:replaceManifest")
+ @PUT
+ @ResponseParser(ETagHeader.class)
+ @Headers(keys = "X-Object-Manifest", values = "{containerName}/{objectName}/")
+ String replaceManifest(@PathParam("containerName") String containerName, @PathParam("objectName") String objectName,
+ @BinderParam(BindToJsonPayload.class) List<Segment> segments,
Could `segments` be a `Collection`?
> +
+import com.google.common.collect.ImmutableMap;
+import com.google.common.collect.ImmutableMap.Builder;
+
+/**
+ * Updates Content Metadata.
+ *
+ * Required to update the Content Length of Object File as Sum of Content length
+ * of Segments
+ *
+ * @param contentMetadata
+ *
+ * @return Map with Content Metadata.
+ */
+
+public class ContentMetaDataToMap {
What does this do and where is it used?
> +import com.google.common.collect.ImmutableMap.Builder;
+
+/**
+ * Updates Content Metadata.
+ *
+ * Required to update the Content Length of Object File as Sum of Content length
+ * of Segments
+ *
+ * @param contentMetadata
+ *
+ * @return Map with Content Metadata.
+ */
+
+public class ContentMetaDataToMap {
+
+ public ImmutableMap<String, String> getContentMetadataForManifest(ContentMetadata contentMetadata) {
Return `Map`.
> + * @param objectName
+ * corresponds to {@link SwiftObject#getName()}.
+ * @param segments
+ * ordered parts which will be concatenated upon download.
+ * @param metadata
+ * corresponds to {@link SwiftObject#getMetadata()}.
+ * @param headers
+ * Binds the map to headers, without prefixing/escaping the header
+ * name/key.
+ *
+ * @return {@link SwiftObject#getEtag()} of the object, which is the MD5
+ * checksum of the concatenated ETag values of the {@code segments}.
+ */
+
+ @Deprecated
+ @Named("dynamicLargeObject: uploadLargeFile")
Remove space after colon (repeated below).
> + * @param metadata
+ * corresponds to {@link SwiftObject#getMetadata()}.
+ * @param headers
+ * Binds the map to headers, without prefixing/escaping the header
+ * name/key.
+ *
+ * @return {@link SwiftObject#getEtag()} of the object, which is the MD5
+ * checksum of the concatenated ETag values of the {@code segments}.
+ */
+
+ @Deprecated
+ @Named("dynamicLargeObject: uploadLargeFile")
+ @PUT
+ @Headers(keys = EXPECT, values = "100-continue")
+ @ResponseParser(ETagHeader.class)
+ String uploadLargeFile(@PathParam("containerName") String container, @PathParam("objectName") String objectName,
I don't understand this API. Shouldn't callers upload single-part objects of the form, "obj1", "obj2", then call a method which creates the DLO with the name "obj"?
> + getApi().getObjectApi(regionId, containerName).delete(name);
+ }
+ }
+
+ @Override
+ @BeforeClass(groups = "live")
+ public void setup() {
+ super.setup();
+ for (String regionId : regions) {
+ boolean created = getApi().getContainerApi(regionId).create(defaultContainerName);
+ if (!created) {
+ deleteAllObjectsInContainer(regionId, defaultContainerName);
+ }
+ }
+
+ megOf1s = new byte[1024 * 1024];
Use `TestUtils.randomByteSource`.
> + assertEquals(bigObject.getETag(), etagOfEtags);
+ assertEquals(bigObject.getPayload().getContentMetadata().getContentLength(), Long.valueOf(2 * 1024 * 1024));
+ assertEquals(bigObject.getMetadata(), ImmutableMap.of("myfoo", "Bar"));
+
+ // segments are visible
+ assertEquals(getApi().getContainerApi(regionId).get(containerName).getObjectCount(), Long.valueOf(3));
+ }
+
+ protected void assertMegabyteAndETagMatches(String regionId, String containerName, String name, String etag1s) {
+ SwiftObject object1s = getApi().getObjectApi(regionId, containerName).get(name);
+ assertEquals(object1s.getETag(), etag1s);
+ assertEquals(object1s.getPayload().getContentMetadata().getContentLength(), Long.valueOf(1024 * 1024));
+ }
+
+ protected void deleteAllObjectsInContainerDLO(String regionId, final String containerName) {
+ Uninterruptibles.sleepUninterruptibly(10, TimeUnit.SECONDS);
Remove sleep.
> +
+import com.google.common.base.Function;
+import com.google.common.collect.ImmutableList;
+import com.google.common.collect.ImmutableMap;
+import com.google.common.collect.Lists;
+import com.google.common.io.ByteSource;
+import com.google.common.util.concurrent.Uninterruptibles;
+
+@Test(groups = "live", testName = "DynamicLargeObjectApiLiveTest", singleThreaded = true)
+public class DynamicLargeObjectApiLiveTest extends BaseSwiftApiLiveTest {
+
+ private String defaultName = getClass().getSimpleName();
+ private String defaultContainerName = getClass().getSimpleName() + "Container";
+ private byte[] megOf1s;
+ private byte[] megOf2s;
+ String objectName = "myObject";
Add `private` access.
> + @SuppressWarnings("deprecation")
+ @Test
+ public void uploadLargeFile(String regionId) throws IOException, InterruptedException {
+ /*
+ * createRandomFile(SIZE, bigFile); Payload payLoad = new
+ * FilePayload(bigFile);
+ */
+ List<Segment> list1 = new ArrayList<Segment>();
+ int partNumber = 1;
+ int total_size = 0;
+ // configure the blobstore to use multipart uploading of the file
+ /*
+ * payLoad.getContentMetadata().setContentLength(bigFile.length());
+ * BasePayloadSlicer slicer = new BasePayloadSlicer();
+ */
+ while (partNumber < 3) {
Prefer `for` loop.
> + @Test
+ public void uploadLargeFile(String regionId) throws IOException, InterruptedException {
+ /*
+ * createRandomFile(SIZE, bigFile); Payload payLoad = new
+ * FilePayload(bigFile);
+ */
+ List<Segment> list1 = new ArrayList<Segment>();
+ int partNumber = 1;
+ int total_size = 0;
+ // configure the blobstore to use multipart uploading of the file
+ /*
+ * payLoad.getContentMetadata().setContentLength(bigFile.length());
+ * BasePayloadSlicer slicer = new BasePayloadSlicer();
+ */
+ while (partNumber < 3) {
+ String objName = objectName + "/dlo/" + String.valueOf(partNumber);
Remove call to `String.valueOf`.
> + // configure the blobstore to use multipart uploading of the file
+ /*
+ * payLoad.getContentMetadata().setContentLength(bigFile.length());
+ * BasePayloadSlicer slicer = new BasePayloadSlicer();
+ */
+ while (partNumber < 3) {
+ String objName = objectName + "/dlo/" + String.valueOf(partNumber);
+ String data = "data".concat(String.valueOf(partNumber));
+ String etag = getApi().getDynamicLargeObjectApi(regionId, defaultContainerName).uploadLargeFile(
+ defaultContainerName, objName, Payloads.newPayload(data), ImmutableMap.of("myfoo", "Bar"),
+ ImmutableMap.of("myfoo", "Bar"));
+ Segment s = new Segment(objName, etag, data.length());
+ assertNotNull(etag);
+ list1.add(s);
+ partNumber++;
+ total_size = total_size + data.length();
Prefer `+=`.
> + String objName = objectName + "/dlo/" + String.valueOf(partNumber);
+ String data = "data".concat(String.valueOf(partNumber));
+ String etag = getApi().getDynamicLargeObjectApi(regionId, defaultContainerName).uploadLargeFile(
+ defaultContainerName, objName, Payloads.newPayload(data), ImmutableMap.of("myfoo", "Bar"),
+ ImmutableMap.of("myfoo", "Bar"));
+ Segment s = new Segment(objName, etag, data.length());
+ assertNotNull(etag);
+ list1.add(s);
+ partNumber++;
+ total_size = total_size + data.length();
+ }
+ String etagOfEtags = getApi().getDynamicLargeObjectApi(regionId, defaultContainerName).replaceManifest(
+ defaultContainerName, objectName, list1, ImmutableMap.of("MyFoo", "Bar"), ImmutableMap.of("MyFoo", "Bar"));
+
+ SwiftObject bigObject = getApi().getObjectApi(regionId, defaultContainerName).get(objectName);
+ assertEquals(bigObject.getETag(), etagOfEtags);
Prefer `assertThat`. Repeated throughout.
--
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/jclouds/jclouds/pull/1105#pullrequestreview-39905447
Re: [jclouds/jclouds] DLO - Lower Level Provider API changes (#1105)
Posted by archupsg03 <no...@github.com>.
archupsg03 commented on this pull request.
> + // configure the blobstore to use multipart uploading of the file
+ /*
+ * payLoad.getContentMetadata().setContentLength(bigFile.length());
+ * BasePayloadSlicer slicer = new BasePayloadSlicer();
+ */
+ while (partNumber < 3) {
+ String objName = objectName + "/dlo/" + String.valueOf(partNumber);
+ String data = "data".concat(String.valueOf(partNumber));
+ String etag = getApi().getDynamicLargeObjectApi(regionId, defaultContainerName).uploadLargeFile(
+ defaultContainerName, objName, Payloads.newPayload(data), ImmutableMap.of("myfoo", "Bar"),
+ ImmutableMap.of("myfoo", "Bar"));
+ Segment s = new Segment(objName, etag, data.length());
+ assertNotNull(etag);
+ list1.add(s);
+ partNumber++;
+ total_size = total_size + data.length();
Done
--
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/jclouds/jclouds/pull/1105#discussion_r118183888
Re: [jclouds/jclouds] DLO - Lower Level Provider API changes (#1105)
Posted by archupsg03 <no...@github.com>.
@timuralp Done with almost all the changes. can you review once
--
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/jclouds/jclouds/pull/1105#issuecomment-308945363
Re: [jclouds/jclouds] DLO - Lower Level Provider API changes (#1105)
Posted by archupsg03 <no...@github.com>.
archupsg03 commented on this pull request.
> +
+import com.google.common.collect.ImmutableMap;
+import com.google.common.collect.ImmutableMap.Builder;
+
+/**
+ * Updates Content Metadata.
+ *
+ * Required to update the Content Length of Object File as Sum of Content length
+ * of Segments
+ *
+ * @param contentMetadata
+ *
+ * @return Map with Content Metadata.
+ */
+
+public class ContentMetaDataToMap {
Removed this file, since its decided as not need based on Comment https://github.com/jclouds/jclouds/pull/1105#discussion_r120260682
--
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/jclouds/jclouds/pull/1105#discussion_r120318602
Re: [jclouds/jclouds] DLO - Lower Level Provider API changes (#1105)
Posted by archupsg03 <no...@github.com>.
archupsg03 commented on this pull request.
> + * @param metadata
+ * corresponds to {@link SwiftObject#getMetadata()}.
+ * @param headers
+ * Binds the map to headers, without prefixing/escaping the header
+ * name/key.
+ *
+ * @return {@link SwiftObject#getEtag()} of the object, which is the MD5
+ * checksum of the concatenated ETag values of the {@code segments}.
+ */
+ @Deprecated
+ @Named("dynamicLargeObject:replaceManifest")
+ @PUT
+ @ResponseParser(ETagHeader.class)
+ @Headers(keys = "X-Object-Manifest", values = "{containerName}/{objectName}/")
+ String replaceManifest(@PathParam("objectName") String objectName,
+ @BinderParam(BindToJsonPayload.class) Collection<Segment> segments,
@timuralp
My thought is to have 2 methods for putManifest resembling 1.9, like
String putObjectManifest(@PathParam("container") String container, @PathParam("name") String name); ----> returns etag of empty string
String putObjectManifest(@PathParam("container") String container, @PathParam("name") @ParamParser(ObjectName.class)
@BinderParam(BindSwiftObjectMetadataToRequest.class) SwiftObject object);
returns etag of manifest content.
Users who were using 1.9, may switch to latest version. We should be able to have the backward compatibility, which is not achievable now.
--
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/jclouds/jclouds/pull/1105#discussion_r120260682
Re: [jclouds/jclouds] DLO - Lower Level Provider API changes (#1105)
Posted by Andrew Gaul <no...@github.com>.
I decline to share my Rackspace credentials which are linked to my credit card with some random Internet person.
--
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/jclouds/jclouds/pull/1105#issuecomment-331255445
Re: [jclouds/jclouds] DLO - Lower Level Provider API changes (#1105)
Posted by archupsg03 <no...@github.com>.
Thanks @timuralp . Were you able to run test sucessfully ?
Sure we ll wait for @andrewgaul
--
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/jclouds/jclouds/pull/1105#issuecomment-311243633
Re: [jclouds/jclouds] DLO - Lower Level Provider API changes (#1105)
Posted by Timur Alperovich <no...@github.com>.
Great! Tests pass for me. Unfortunately, they take a long time, but I think that's related to the jclouds handling of the containers to be created for testing. @archupsg03 there are some small nits with white space changes in the commits. Before merging, can you make sure to remove these? I'll let @andrewgaul comment on any other changes he'd like to see.
--
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/jclouds/jclouds/pull/1105#issuecomment-313310088
Re: [jclouds/jclouds] DLO - Lower Level Provider API changes (#1105)
Posted by WebFolder OÜ <no...@github.com>.
@archupsg03 Could you please fix those minor issues.
--
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/jclouds/jclouds/pull/1105#issuecomment-324098618
Re: [jclouds/jclouds] DLO - Lower Level Provider API changes (#1105)
Posted by archupsg03 <no...@github.com>.
@andrewgaul, Since we are not working with rackspace cloud, we dont have a setup to test against it. If possible can you share rackspace cloud credentials, so that I can try to figure out the issue
--
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/jclouds/jclouds/pull/1105#issuecomment-331144591
Re: [jclouds/jclouds] DLO - Lower Level Provider API changes (#1105)
Posted by archupsg03 <no...@github.com>.
archupsg03 commented on this pull request.
> + @Test
+ public void testReplaceManifest() throws Exception {
+ for (String regionId : regions) {
+ assertReplaceManifest(regionId, defaultContainerName, defaultName);
+ uploadLargeFile(regionId);
+ removeLargeFile(regionId);
+ }
+ }
+
+ @SuppressWarnings("deprecation")
+ @Test
+ public void uploadLargeFile(String regionId) throws IOException, InterruptedException {
+ /*
+ * createRandomFile(SIZE, bigFile); Payload payLoad = new
+ * FilePayload(bigFile);
+ */
Done
--
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/jclouds/jclouds/pull/1105#discussion_r118183843
Re: [jclouds/jclouds] DLO - Lower Level Provider API changes (#1105)
Posted by archupsg03 <no...@github.com>.
archupsg03 commented on this pull request.
> + @PUT
+ @ResponseParser(ETagHeader.class)
+ @Headers(keys = "X-Object-Manifest", values = "{containerName}/{objectName}/")
+ String putManifest(@PathParam("objectName") String objectName,
+ @BinderParam(BindObjectMetadataToHeaders.class) Map<String, String> metadata,
+ @BinderParam(BindToHeaders.class) Map<String, String> headers);
+
+ /**
+ * Creates or updates a dynamic large object's manifest.
+ *
+ * @param objectName
+ * corresponds to {@link SwiftObject#getName()}.
+ * @param metadata
+ * corresponds to {@link SwiftObject#getMetadata()}.
+ *
+ * @return {@link SwiftObject#getEtag()} of the object, which is the MD5
I am getting etag of empty string. Same like what you have mentioned. Its copy paste error.
--
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/jclouds/jclouds/pull/1105#discussion_r120588498
Re: [jclouds/jclouds] DLO - Lower Level Provider API changes (#1105)
Posted by archupsg03 <no...@github.com>.
archupsg03 commented on this pull request.
> + @Test
+ public void uploadLargeFile(String regionId) throws IOException, InterruptedException {
+ /*
+ * createRandomFile(SIZE, bigFile); Payload payLoad = new
+ * FilePayload(bigFile);
+ */
+ List<Segment> list1 = new ArrayList<Segment>();
+ int partNumber = 1;
+ int total_size = 0;
+ // configure the blobstore to use multipart uploading of the file
+ /*
+ * payLoad.getContentMetadata().setContentLength(bigFile.length());
+ * BasePayloadSlicer slicer = new BasePayloadSlicer();
+ */
+ while (partNumber < 3) {
+ String objName = objectName + "/dlo/" + String.valueOf(partNumber);
Done
--
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/jclouds/jclouds/pull/1105#discussion_r118183874
Re: [jclouds/jclouds] DLO - Lower Level Provider API changes (#1105)
Posted by archupsg03 <no...@github.com>.
archupsg03 commented on this pull request.
> + * ordered parts which will be concatenated upon download.
+ * @param metadata
+ * corresponds to {@link SwiftObject#getMetadata()}.
+ * @param headers
+ * Binds the map to headers, without prefixing/escaping the header
+ * name/key.
+ *
+ * @return {@link SwiftObject#getEtag()} of the object, which is the MD5
+ * checksum of the concatenated ETag values of the {@code segments}.
+ */
+ @Deprecated
+ @Named("dynamicLargeObject:replaceManifest")
+ @PUT
+ @ResponseParser(ETagHeader.class)
+ @Headers(keys = "X-Object-Manifest", values = "{containerName}/{objectName}/")
+ String replaceManifest(@PathParam("objectName") String objectName,
Changed it to putManifets
--
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/jclouds/jclouds/pull/1105#discussion_r118906526
Re: [jclouds/jclouds] DLO - Lower Level Provider API changes (#1105)
Posted by archupsg03 <no...@github.com>.
archupsg03 commented on this pull request.
> + String objName = objectName + "/dlo/" + String.valueOf(partNumber);
+ String data = "data".concat(String.valueOf(partNumber));
+ String etag = getApi().getDynamicLargeObjectApi(regionId, defaultContainerName).uploadLargeFile(
+ defaultContainerName, objName, Payloads.newPayload(data), ImmutableMap.of("myfoo", "Bar"),
+ ImmutableMap.of("myfoo", "Bar"));
+ Segment s = new Segment(objName, etag, data.length());
+ assertNotNull(etag);
+ list1.add(s);
+ partNumber++;
+ total_size = total_size + data.length();
+ }
+ String etagOfEtags = getApi().getDynamicLargeObjectApi(regionId, defaultContainerName).replaceManifest(
+ defaultContainerName, objectName, list1, ImmutableMap.of("MyFoo", "Bar"), ImmutableMap.of("MyFoo", "Bar"));
+
+ SwiftObject bigObject = getApi().getObjectApi(regionId, defaultContainerName).get(objectName);
+ assertEquals(bigObject.getETag(), etagOfEtags);
Done
--
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/jclouds/jclouds/pull/1105#discussion_r118183899
Re: [jclouds/jclouds] DLO - Lower Level Provider API changes (#1105)
Posted by archupsg03 <no...@github.com>.
@archupsg03 pushed 1 commit.
2ee67e1 Minor comments addressed
--
You are receiving this because you are subscribed to this thread.
View it on GitHub:
https://github.com/jclouds/jclouds/pull/1105/files/42f0273b6c704ee3b544ffce10266aae9dd40d72..2ee67e1ef0030f223fb7e31ccf58e9eb1e46d768
Re: [jclouds/jclouds] DLO - Lower Level Provider API changes (#1105)
Posted by archupsg03 <no...@github.com>.
archupsg03 commented on this pull request.
> + SwiftObject object1s = getApi().getObjectApi(regionId, containerName).get(name);
+ assertThat(object1s.getETag().equals(etag1s));
+ assertThat(object1s.getPayload().getContentMetadata().getContentLength().equals(Long.valueOf(1024 * 1024)));
+ }
+
+ protected void deleteAllObjectsInContainerDLO(String regionId, final String containerName) {
+ ObjectList objects = getApi().getObjectApi(regionId, containerName).list(new ListContainerOptions());
+ if (objects == null) {
+ return;
+ }
+ List<String> pathsToDelete = Lists.transform(objects, new Function<SwiftObject, String>() {
+ public String apply(SwiftObject input) {
+ return containerName + "/" + input.getName();
+ }
+ });
+ if (!pathsToDelete.isEmpty()) {
Its valid check I feel
--
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/jclouds/jclouds/pull/1105#discussion_r118909260
Re: [jclouds/jclouds] DLO - Lower Level Provider API changes (#1105)
Posted by archupsg03 <no...@github.com>.
archupsg03 commented on this pull request.
> + public void setup() {
+ super.setup();
+ for (String regionId : regions) {
+ boolean created = getApi().getContainerApi(regionId).create(defaultContainerName);
+ if (!created) {
+ deleteAllObjectsInContainer(regionId, defaultContainerName);
+ }
+ }
+
+ /* megOf1s = new byte[1024 * 1024];
+ megOf2s = new byte[1024 * 1024];
+
+ Arrays.fill(megOf1s, (byte) 1);
+ Arrays.fill(megOf2s, (byte) 2);*/
+
+ megOf1s = TestUtils.randomByteSource().slice(0, 1048576);
Done
--
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/jclouds/jclouds/pull/1105#discussion_r118909529
Re: [jclouds/jclouds] DLO - Lower Level Provider API changes (#1105)
Posted by archupsg03 <no...@github.com>.
archupsg03 commented on this pull request.
> + * @param metadata
+ * corresponds to {@link SwiftObject#getMetadata()}.
+ * @param headers
+ * Binds the map to headers, without prefixing/escaping the header
+ * name/key.
+ *
+ * @return {@link SwiftObject#getEtag()} of the object, which is the MD5
+ * checksum of the concatenated ETag values of the {@code segments}.
+ */
+ @Deprecated
+ @Named("dynamicLargeObject:replaceManifest")
+ @PUT
+ @ResponseParser(ETagHeader.class)
+ @Headers(keys = "X-Object-Manifest", values = "{containerName}/{objectName}/")
+ String replaceManifest(@PathParam("containerName") String containerName, @PathParam("objectName") String objectName,
+ @BinderParam(BindToJsonPayload.class) List<Segment> segments,
Done
--
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/jclouds/jclouds/pull/1105#discussion_r118183203
Re: [jclouds/jclouds] DLO - Lower Level Provider API changes (#1105)
Posted by archupsg03 <no...@github.com>.
@andrewgaul Removed trailing whitespaces that was visible to me :(. If you are still spacing any issues I think its better you can change accordingly.
--
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/jclouds/jclouds/pull/1105#issuecomment-314002576
Re: [jclouds/jclouds] DLO - Lower Level Provider API changes (#1105)
Posted by archupsg03 <no...@github.com>.
archupsg03 commented on this pull request.
> + assertThat(getApi().getContainerApi(regionId).get(defaultContainerName).getObjectCount()).isEqualTo(Long.valueOf(3));
+ }
+
+ @SuppressWarnings("deprecation")
+ protected void assertReplaceManifest(String regionId, String containerName, String name) {
+ ObjectApi objectApi = getApi().getObjectApi(regionId, containerName);
+
+ String etag1s = objectApi.put(name + "/1", newByteSourcePayload(megOf1s));
+ awaitConsistency();
+ assertMegabyteAndETagMatches(regionId, containerName, name + "/1", etag1s);
+
+ String etag2s = objectApi.put(name + "/2", newByteSourcePayload(megOf2s));
+ awaitConsistency();
+ assertMegabyteAndETagMatches(regionId, containerName, name + "/2", etag2s);
+
+ List<Segment> segments = ImmutableList.<Segment> builder()
Done
--
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/jclouds/jclouds/pull/1105#discussion_r120589552
Re: [jclouds/jclouds] DLO - Lower Level Provider API changes (#1105)
Posted by archupsg03 <no...@github.com>.
archupsg03 commented on this pull request.
> + @BinderParam(SetPayload.class) Payload blob,
+ @BinderParam(BindObjectMetadataToHeaders.class) Map<String, String> metadata,
+ @BinderParam(BindToHeaders.class) Map<String, String> headers);
+
+ /**
+ * Delete objects.
+ *
+ * @param objectName
+ * corresponds to {@link SwiftObject#getName()}.
+ */
+
+ @Deprecated
+ @Named("dynamicLargeObject: removeLargeFile")
+ @DELETE
+ @Fallback(VoidOnNotFoundOr404.class)
+ void removeLargeFile(@PathParam("containerName") String container, @PathParam("objectName") String name);
My point of view was like, user using DLO API's to upload object, should be able to delete with same DLO API's, instead of creating new ObjectApi for deleting purpose.
If you mandate the user to use ObjectApi, then this change is not required
--
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/jclouds/jclouds/pull/1105#discussion_r118158588
Re: [jclouds/jclouds] DLO - Lower Level Provider API changes (#1105)
Posted by archupsg03 <no...@github.com>.
@archupsg03 pushed 1 commit.
05ba514 Made testcase work for latest changes
--
You are receiving this because you are subscribed to this thread.
View it on GitHub:
https://github.com/jclouds/jclouds/pull/1105/files/998b6a926c52797549393348fdf44f7b0da3d557..05ba51403a8d855a541c4676b49a4ce45b6e3b59
Re: [jclouds/jclouds] DLO - Lower Level Provider API changes (#1105)
Posted by archupsg03 <no...@github.com>.
Regarding the rackspace issue, I am not sure of it, and its a basic and existing call. @timuralp Any comment on this
--
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/jclouds/jclouds/pull/1105#issuecomment-327658754
Re: [jclouds/jclouds] DLO - Lower Level Provider API changes (#1105)
Posted by archupsg03 <no...@github.com>.
archupsg03 commented on this pull request.
> +
+import org.jclouds.io.Payloads;
+import org.jclouds.openstack.swift.v1.SwiftApi;
+import org.jclouds.openstack.v2_0.internal.BaseOpenStackMockTest;
+import org.testng.annotations.Test;
+
+import com.google.common.collect.ImmutableMap;
+import com.google.common.net.HttpHeaders;
+import com.squareup.okhttp.mockwebserver.MockResponse;
+import com.squareup.okhttp.mockwebserver.MockWebServer;
+import com.squareup.okhttp.mockwebserver.RecordedRequest;
+
+@Test(groups = "unit", testName = "DynamicLargeObjectApiMockTest")
+public class DynamicLargeObjectApiMockTest extends BaseOpenStackMockTest<SwiftApi> {
+
+ String containerName = "myContainer";
Error on changing to private visibility.
--
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/jclouds/jclouds/pull/1105#discussion_r135258638
Re: [jclouds/jclouds] DLO - Lower Level Provider API changes (#1105)
Posted by archupsg03 <no...@github.com>.
archupsg03 commented on this pull request.
> + @Deprecated
+ @Named("dynamicLargeObject:replaceManifest")
+ @PUT
+ @ResponseParser(ETagHeader.class)
+ @Headers(keys = "X-Object-Manifest", values = "{containerName}/{objectName}/")
+ String replaceManifest(@PathParam("objectName") String objectName,
+ @BinderParam(BindToJsonPayload.class) Collection<Segment> segments,
+ @BinderParam(BindObjectMetadataToHeaders.class) Map<String, String> metadata,
+ @BinderParam(BindToHeaders.class) Map<String, String> headers);
+
+ /**
+ * Creates object segments.
+ *
+ * @param objectName
+ * corresponds to {@link SwiftObject#getName()}.
+ * @param segments
Done
--
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/jclouds/jclouds/pull/1105#discussion_r120338892
Re: [jclouds/jclouds] DLO - Lower Level Provider API changes (#1105)
Posted by Timur Alperovich <no...@github.com>.
timuralp commented on this pull request.
> + * @param metadata
+ * corresponds to {@link SwiftObject#getMetadata()}.
+ * @param headers
+ * Binds the map to headers, without prefixing/escaping the header
+ * name/key.
+ *
+ * @return {@link SwiftObject#getEtag()} of the object, which is the MD5
+ * checksum of the concatenated ETag values of the {@code segments}.
+ */
+
+ @Deprecated
+ @Named("dynamicLargeObject:uploadLargeFile")
+ @PUT
+ @Headers(keys = EXPECT, values = "100-continue")
+ @ResponseParser(ETagHeader.class)
+ String uploadPart(@PathParam("containerName") String container, @PathParam("objectName") String objectName,
Sorry -- I called the method wrong. It's `put()` in `ObjectApi` or `putBlob()` in the `RegionScopedSwiftBlobStore` class. I suspect consumers should call `putBlob()` and then upload the manifest through the API you're adding. You can probably call either one in the tests.
--
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/jclouds/jclouds/pull/1105#discussion_r120803915
Re: [jclouds/jclouds] DLO - Lower Level Provider API changes (#1105)
Posted by Andrew Gaul <no...@github.com>.
Please just sign up for a Rackspace account which takes a few minutes. I have spent hours reviewing code with you.
--
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/jclouds/jclouds/pull/1105#issuecomment-331348481
Re: [jclouds/jclouds] DLO - Lower Level Provider API changes (#1105)
Posted by archupsg03 <no...@github.com>.
I am doubting that container doesnot exist in rackspace cloud, so why we are facing the issue. Can you make sure that the container used for DLO exist and then give a try
--
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/jclouds/jclouds/pull/1105#issuecomment-331347852
Re: [jclouds/jclouds] DLO - Lower Level Provider API changes (#1105)
Posted by archupsg03 <no...@github.com>.
@archupsg03 pushed 1 commit.
42f0273 Removed Trailing white spaces
--
You are receiving this because you are subscribed to this thread.
View it on GitHub:
https://github.com/jclouds/jclouds/pull/1105/files/3aa1552342bf030eef0838a43cc354cff7dcc7b9..42f0273b6c704ee3b544ffce10266aae9dd40d72
Re: [jclouds/jclouds] DLO - Lower Level Provider API changes (#1105)
Posted by archupsg03 <no...@github.com>.
@andrewgaul . Sure will work on it by this week end.
One more clarification here, incase of swift MPU say you have 3 parts and due to some reason if the uploading of one part failed, current behaviour is it tries continuing with rest of the parts. But expectation is not to proceed if there is a failure in uploading path. If so what is the way to stop all the submitted task to the executor on exception. Am referring to RegionScopedSwiftBlobStore call to completeMultipartUplaod -> Futures.getUnchecked(list)
--
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/jclouds/jclouds/pull/1105#issuecomment-324305111
Re: [jclouds/jclouds] DLO - Lower Level Provider API changes (#1105)
Posted by archupsg03 <no...@github.com>.
archupsg03 commented on this pull request.
> +
+ awaitConsistency();
+
+ SwiftObject bigObject = getApi().getObjectApi(regionId, containerName).get(name);
+ assertThat(bigObject.getETag().equals(etagOfEtags));
+ assertThat(bigObject.getPayload().getContentMetadata().getContentLength().equals(Long.valueOf(2 * 1024 * 1024)));
+ assertThat(bigObject.getMetadata().equals(ImmutableMap.of("myfoo", "Bar")));
+
+ // segments are visible
+ assertThat(getApi().getContainerApi(regionId).get(containerName).getObjectCount().equals(Long.valueOf(3)));
+ }
+
+ protected void assertMegabyteAndETagMatches(String regionId, String containerName, String name, String etag1s) {
+ SwiftObject object1s = getApi().getObjectApi(regionId, containerName).get(name);
+ assertThat(object1s.getETag().equals(etag1s));
+ assertThat(object1s.getPayload().getContentMetadata().getContentLength().equals(Long.valueOf(1024 * 1024)));
Done
--
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/jclouds/jclouds/pull/1105#discussion_r118908231
Re: [jclouds/jclouds] DLO - Lower Level Provider API changes (#1105)
Posted by archupsg03 <no...@github.com>.
archupsg03 commented on this pull request.
> + * @param metadata
+ * corresponds to {@link SwiftObject#getMetadata()}.
+ * @param headers
+ * Binds the map to headers, without prefixing/escaping the header
+ * name/key.
+ *
+ * @return {@link SwiftObject#getEtag()} of the object, which is the MD5
+ * checksum of the concatenated ETag values of the {@code segments}.
+ */
+
+ @Deprecated
+ @Named("dynamicLargeObject: uploadLargeFile")
+ @PUT
+ @Headers(keys = EXPECT, values = "100-continue")
+ @ResponseParser(ETagHeader.class)
+ String uploadLargeFile(@PathParam("containerName") String container, @PathParam("objectName") String objectName,
Chnage method name to Upload Part. Done
--
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/jclouds/jclouds/pull/1105#discussion_r118183760
Re: [jclouds/jclouds] DLO - Lower Level Provider API changes (#1105)
Posted by archupsg03 <no...@github.com>.
archupsg03 commented on this pull request.
> + * @param metadata
+ * corresponds to {@link SwiftObject#getMetadata()}.
+ * @param headers
+ * Binds the map to headers, without prefixing/escaping the header
+ * name/key.
+ *
+ * @return {@link SwiftObject#getEtag()} of the object, which is the MD5
+ * checksum of the concatenated ETag values of the {@code segments}.
+ */
+ @Deprecated
+ @Named("dynamicLargeObject:replaceManifest")
+ @PUT
+ @ResponseParser(ETagHeader.class)
+ @Headers(keys = "X-Object-Manifest", values = "{containerName}/{objectName}/")
+ String replaceManifest(@PathParam("objectName") String objectName,
+ @BinderParam(BindToJsonPayload.class) Collection<Segment> segments,
@timuralp Any comments on this ?
--
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/jclouds/jclouds/pull/1105#discussion_r120037701
Re: [jclouds/jclouds] DLO - Lower Level Provider API changes (#1105)
Posted by Timur Alperovich <no...@github.com>.
timuralp commented on this pull request.
> + SwiftObject object1s = getApi().getObjectApi(regionId, containerName).get(name);
+ assertThat(object1s.getETag().equals(etag1s));
+ assertThat(object1s.getPayload().getContentMetadata().getContentLength().equals(Long.valueOf(1024 * 1024)));
+ }
+
+ protected void deleteAllObjectsInContainerDLO(String regionId, final String containerName) {
+ ObjectList objects = getApi().getObjectApi(regionId, containerName).list(new ListContainerOptions());
+ if (objects == null) {
+ return;
+ }
+ List<String> pathsToDelete = Lists.transform(objects, new Function<SwiftObject, String>() {
+ public String apply(SwiftObject input) {
+ return containerName + "/" + input.getName();
+ }
+ });
+ if (!pathsToDelete.isEmpty()) {
@archupsg03 can you remove this `if` clause? It's not a useful check. If the list is empty, the loop is a NOOP.
> @@ -37,7 +37,7 @@
@Named("size_bytes")
private final long sizeBytes;
- private Segment(String path, String etag, long sizeBytes) {
+ public Segment(String path, String etag, long sizeBytes) {
@archupsg03 I think we can remove this change?
> + * @param metadata
+ * corresponds to {@link SwiftObject#getMetadata()}.
+ * @param headers
+ * Binds the map to headers, without prefixing/escaping the header
+ * name/key.
+ *
+ * @return {@link SwiftObject#getEtag()} of the object, which is the MD5
+ * checksum of the concatenated ETag values of the {@code segments}.
+ */
+
+ @Deprecated
+ @Named("dynamicLargeObject:uploadLargeFile")
+ @PUT
+ @Headers(keys = EXPECT, values = "100-continue")
+ @ResponseParser(ETagHeader.class)
+ String uploadPart(@PathParam("containerName") String container, @PathParam("objectName") String objectName,
Sorry -- I called the method wrong. It's put() in ObjectApi or putBlob() in the RegionScopedSwiftBlobStore class. I suspect consumers should call putBlob() and then upload the manifest through the API you're adding. You can probably call either one in the tests.
> + assertThat(object1s.getPayload().getContentMetadata().getContentLength()).isEqualTo(Long.valueOf(1024L * 1024L));
+ }
+
+ protected void deleteAllObjectsInContainerDLO(String regionId, final String containerName) {
+ ObjectList objects = getApi().getObjectApi(regionId, containerName).list(new ListContainerOptions());
+ if (objects == null) {
+ return;
+ }
+ List<String> pathsToDelete = Lists.transform(objects, new Function<SwiftObject, String>() {
+ public String apply(SwiftObject input) {
+ return containerName + "/" + input.getName();
+ }
+ });
+ if (!pathsToDelete.isEmpty()) {
+ for (String name : pathsToDelete)
+ getApi().getObjectApi(regionId, containerName).delete(name);
The object API takes in the containerName. Does the containerName still need to be in the object Name, as well? In the above List transformation, we prepend containerName to every object -- that seems wrong or I'm missing something.
--
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/jclouds/jclouds/pull/1105#pullrequestreview-42796224
Re: [jclouds/jclouds] DLO - Lower Level Provider API changes (#1105)
Posted by archupsg03 <no...@github.com>.
@archupsg03 pushed 1 commit.
9906c00 DLO - Comments Addressed1
--
You are receiving this because you are subscribed to this thread.
View it on GitHub:
https://github.com/jclouds/jclouds/pull/1105/files/0dff0ae919f6c11550300940aba5f3b7e96f7a77..9906c00c68933634126014e60f1bfd05ade7f897
Re: [jclouds/jclouds] DLO - Lower Level Provider API changes (#1105)
Posted by archupsg03 <no...@github.com>.
archupsg03 commented on this pull request.
> + getApi().getObjectApi(regionId, containerName).delete(name);
+ }
+ }
+
+ @Override
+ @BeforeClass(groups = "live")
+ public void setup() {
+ super.setup();
+ for (String regionId : regions) {
+ boolean created = getApi().getContainerApi(regionId).create(defaultContainerName);
+ if (!created) {
+ deleteAllObjectsInContainer(regionId, defaultContainerName);
+ }
+ }
+
+ megOf1s = new byte[1024 * 1024];
Done
--
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/jclouds/jclouds/pull/1105#discussion_r118183922
Re: [jclouds/jclouds] DLO - Lower Level Provider API changes (#1105)
Posted by archupsg03 <no...@github.com>.
archupsg03 commented on this pull request.
> +import com.google.common.collect.ImmutableMap.Builder;
+
+/**
+ * Updates Content Metadata.
+ *
+ * Required to update the Content Length of Object File as Sum of Content length
+ * of Segments
+ *
+ * @param contentMetadata
+ *
+ * @return Map with Content Metadata.
+ */
+
+public class ContentMetaDataToMap {
+
+ public ImmutableMap<String, String> getContentMetadataForManifest(ContentMetadata contentMetadata) {
Not required
--
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/jclouds/jclouds/pull/1105#discussion_r120318780
Re: [jclouds/jclouds] DLO - Lower Level Provider API changes (#1105)
Posted by Timur Alperovich <no...@github.com>.
timuralp commented on this pull request.
> + * @param metadata
+ * corresponds to {@link SwiftObject#getMetadata()}.
+ * @param headers
+ * Binds the map to headers, without prefixing/escaping the header
+ * name/key.
+ *
+ * @return {@link SwiftObject#getEtag()} of the object, which is the MD5
+ * checksum of the concatenated ETag values of the {@code segments}.
+ */
+ @Deprecated
+ @Named("dynamicLargeObject:replaceManifest")
+ @PUT
+ @ResponseParser(ETagHeader.class)
+ @Headers(keys = "X-Object-Manifest", values = "{containerName}/{objectName}/")
+ String replaceManifest(@PathParam("objectName") String objectName,
+ @BinderParam(BindToJsonPayload.class) Collection<Segment> segments,
Swift does not return the DLO ETag as a response to PUT. At least I do not observe this with docker-swift and Swift 2.14 (latest release):
```
DEBUG:swiftclient:REQ: curl -i http://localhost:32768/v1/AUTH_test/test/s3proxy.jar -X PUT -H "x-object-meta-mtime: 1440388916.150022" -H "Content-Length: 0" -H "x-object-manifest: test_segments/s3proxy.jar/1440388916.150022/7663104/1000000/" -H "Content-Type: " -H "X-Auth-Token: AUTH_tkadf8c03346214185ba387e3888893305"
DEBUG:swiftclient:RESP STATUS: 201 Created
DEBUG:swiftclient:RESP HEADERS: {u'Content-Length': u'0', u'Last-Modified': u'Tue, 06 Jun 2017 06:09:31 GMT', u'Etag': u'd41d8cd98f00b204e9800998ecf8427e', u'X-Trans-Id': u'tx9825d3a9fadb4c50bf4f8-005936471a', u'Date': u'Tue, 06 Jun 2017 06:09:30 GMT', u'Content-Type': u'text/html; charset=UTF-8', u'X-Openstack-Request-Id': u'tx9825d3a9fadb4c50bf4f8-005936471a'}
```
The ETag above is of the empty string and jclouds should expose exactly that as a response to a putManifest request.
In jclouds 1.9, the interaction you're describing is the end of performing an MPU where a 0-sized DLO manifest is uploaded (0-sized object with the X-Object-Manifest header). The two methods that are implemented are for the case of specifying manifest metadata through an existing Blob or through the object name. I don't think jclouds 2.0 should replicate either of the approaches.
In jclouds 2.0, you could introduce two DLO methods, akin to the SLO setup, where one has only the metadata parameter (```@BinderParam(BindObjectMetadataToHeaders.class) Map<String, String> metadata```), while the other one includes a headers parameter (```@BinderParam(BindToHeaders.class) Map<String, String> headers```). That would be inline with the current code.
We should not attempt to satisfy old clients, as the prior API has been marked deprecated and jclouds should prefer to use SLO whenever possible. DLO is an option only for clients who explicitly have to use it to interact with old data or if they have some other special reason.
--
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/jclouds/jclouds/pull/1105#discussion_r120275386
Re: [jclouds/jclouds] DLO - Lower Level Provider API changes (#1105)
Posted by archupsg03 <no...@github.com>.
@archupsg03 pushed 1 commit.
0dff0ae DLO - Comments Added
--
You are receiving this because you are subscribed to this thread.
View it on GitHub:
https://github.com/jclouds/jclouds/pull/1105/files/28ec26db280a6efdf17a9085bbc6d2efb47e00a7..0dff0ae919f6c11550300940aba5f3b7e96f7a77
Re: [jclouds/jclouds] DLO - Lower Level Provider API changes (#1105)
Posted by archupsg03 <no...@github.com>.
archupsg03 commented on this pull request.
> +
+import org.jclouds.io.Payloads;
+import org.jclouds.openstack.swift.v1.SwiftApi;
+import org.jclouds.openstack.v2_0.internal.BaseOpenStackMockTest;
+import org.testng.annotations.Test;
+
+import com.google.common.collect.ImmutableMap;
+import com.google.common.net.HttpHeaders;
+import com.squareup.okhttp.mockwebserver.MockResponse;
+import com.squareup.okhttp.mockwebserver.MockWebServer;
+import com.squareup.okhttp.mockwebserver.RecordedRequest;
+
+@Test(groups = "unit", testName = "DynamicLargeObjectApiMockTest")
+public class DynamicLargeObjectApiMockTest extends BaseOpenStackMockTest<SwiftApi> {
+
+ String containerName = "myContainer";
Made it final. But private is not possible here
--
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/jclouds/jclouds/pull/1105#discussion_r135676989
Re: [jclouds/jclouds] DLO - Lower Level Provider API changes (#1105)
Posted by archupsg03 <no...@github.com>.
archupsg03 commented on this pull request.
> + * use it and give us feedback. Based on that feedback, minor changes to the
+ * interfaces may happen. This code will replace
+ * org.jclouds.openstack.swift.SwiftClient in jclouds 2.0 and it is recommended
+ * you adopt it sooner than later.
+ */
+@Beta
+@RequestFilters(AuthenticateRequest.class)
+@Consumes(APPLICATION_JSON)
+@Path("/{objectName}")
+public interface DynamicLargeObjectApi {
+ /**
+ * Creates or updates a dynamic large object's manifest.
+ *
+ * @param objectName
+ * corresponds to {@link SwiftObject#getName()}.
+ * @param segments
Removed segment parameter based on #1105
--
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/jclouds/jclouds/pull/1105#discussion_r120322633
Re: [jclouds/jclouds] DLO - Lower Level Provider API changes (#1105)
Posted by Andrew Gaul <no...@github.com>.
I added the classes to test this against Rackspace, an OpenStack provider:
https://github.com/andrewgaul/jclouds/tree/Lower_Level_Provider_API
These tests skip and fail:
```
[pool-2-thread-2] Test uploadLargeFile(org.jclouds.rackspace.cloudfiles.us.features.CloudFilesUSDynamicLargeObjectApiLiveTest) skipped.
Test suite progress: tests succeeded: 0, failed: 0, skipped: 1.
Starting test testReplaceManifest(org.jclouds.rackspace.cloudfiles.us.features.CloudFilesUSDynamicLargeObjectApiLiveTest)
[pool-2-thread-1] Test testReplaceManifest(org.jclouds.rackspace.cloudfiles.us.features.CloudFilesUSDynamicLargeObjectApiLiveTest) failed.
Test suite progress: tests succeeded: 0, failed: 1, skipped: 1.
Tests run: 2, Failures: 1, Errors: 0, Skipped: 1, Time elapsed: 119.459 sec <<< FAILURE! - in org.jclouds.rackspace.cloudfiles.us.features.CloudFilesUSDynamicLargeObjectApiLiveTest
testReplaceManifest(org.jclouds.rackspace.cloudfiles.us.features.CloudFilesUSDynamicLargeObjectApiLiveTest) Time elapsed: 96.624 sec <<< FAILURE!
org.jclouds.http.HttpResponseException: Server rejected operation connecting to PUT https://storage101.syd2.clouddrive.com/v1/MossoCloudFS_27bdd8b2-bd25-4e71-8782-9df11aac95a2/gaul-blobstore2/CloudFilesUSDynamicLargeObjectApiLiveTest/1 HTTP/1.1
at org.jclouds.http.internal.BaseHttpCommandExecutorService.invoke(BaseHttpCommandExecutorService.java:122)
at org.jclouds.rest.internal.InvokeHttpMethod.invoke(InvokeHttpMethod.java:90)
at org.jclouds.rest.internal.InvokeHttpMethod.apply(InvokeHttpMethod.java:73)
at org.jclouds.rest.internal.InvokeHttpMethod.apply(InvokeHttpMethod.java:44)
at org.jclouds.reflect.FunctionalReflection$FunctionalInvocationHandler.handleInvocation(FunctionalReflection.java:117)
at com.google.common.reflect.AbstractInvocationHandler.invoke(AbstractInvocationHandler.java:87)
at com.sun.proxy.$Proxy72.put(Unknown Source)
at org.jclouds.openstack.swift.v1.features.DynamicLargeObjectApiLiveTest.assertReplaceManifest(DynamicLargeObjectApiLiveTest.java:87)
at org.jclouds.openstack.swift.v1.features.DynamicLargeObjectApiLiveTest.testReplaceManifest(DynamicLargeObjectApiLiveTest.java:50)
```
--
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/jclouds/jclouds/pull/1105#issuecomment-325575491
Re: [jclouds/jclouds] DLO - Lower Level Provider API changes (#1105)
Posted by Timur Alperovich <no...@github.com>.
timuralp commented on this pull request.
@archupsg03 please try running the live tests. I encountered a few issues when try to do that.
> + public void testReplaceManifest() throws Exception {
+ for (String regionId : regions) {
+ assertReplaceManifest(regionId, defaultContainerName, defaultName);
+ uploadLargeFile(regionId);
+ }
+ }
+
+ @SuppressWarnings("deprecation")
+ @Test
+ public void uploadLargeFile(String regionId) throws IOException, InterruptedException {
+ int partNumber = 1;
+ int total_size = 0;
+ RegionScopedBlobStoreContext ctx = RegionScopedBlobStoreContext.class.cast(view);
+ BlobStore blobStore = ctx.getBlobStore();
+ // configure the blobstore to use multipart uploading of the file
+ for (int i = partNumber; i <= 3; partNumber++) {
This looks like a bug. Wouldn't `i` be incremented? As is, I have a test that never completes, but creates 1000s of parts.
> + public String apply(SwiftObject input) {
+ return containerName + "/" + input.getName();
+ }
+ });
+
+ for (String name : pathsToDelete)
+ getApi().getObjectApi(regionId, containerName).delete(name);
+
+ }
+
+ @Override
+ @BeforeClass(groups = "live")
+ public void setup() {
+ super.setup();
+ for (String regionId : regions) {
+ boolean created = getApi().getContainerApi(regionId).create(defaultContainerName);
You shouldn't have to do this. The jclouds tests provide containers. Check out this method and the S3 provider on how to use them: `blobstore/src/test/java/[...]/integration/internal/BaseBlobStoreIntegrationTest.java:434`. This a copy/paste from the SLO test, but we shouldn't use the same pattern.
> + RegionScopedBlobStoreContext ctx = RegionScopedBlobStoreContext.class.cast(view);
+ BlobStore blobStore = ctx.getBlobStore();
+ // configure the blobstore to use multipart uploading of the file
+ for (int i = partNumber; i <= 3; partNumber++) {
+ String objName = String.format("%s/%s/%s", objectName, "dlo", partNumber);
+ String data = String.format("%s%s", "data", partNumber);
+ ByteSource payload = ByteSource.wrap(data.getBytes(Charsets.UTF_8));
+ Blob blob = blobStore.blobBuilder(objName)
+ .payload(payload)
+ .build();
+ String etag = blobStore.putBlob(defaultContainerName, blob);
+ assertNotNull(etag);
+ total_size += data.length();
+ }
+
+ SwiftObject bigObject = getApi().getObjectApi(regionId, defaultContainerName).get(objectName);
After fixing the above bug, I get a null pointer exception here.
--
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/jclouds/jclouds/pull/1105#pullrequestreview-47052241
Re: [jclouds/jclouds] DLO - Lower Level Provider API changes (#1105)
Posted by archupsg03 <no...@github.com>.
archupsg03 commented on this pull request.
> + * @param metadata
+ * corresponds to {@link SwiftObject#getMetadata()}.
+ * @param headers
+ * Binds the map to headers, without prefixing/escaping the header
+ * name/key.
+ *
+ * @return {@link SwiftObject#getEtag()} of the object, which is the MD5
+ * checksum of the concatenated ETag values of the {@code segments}.
+ */
+ @Deprecated
+ @Named("dynamicLargeObject:replaceManifest")
+ @PUT
+ @ResponseParser(ETagHeader.class)
+ @Headers(keys = "X-Object-Manifest", values = "{containerName}/{objectName}/")
+ String replaceManifest(@PathParam("objectName") String objectName,
+ @BinderParam(BindToJsonPayload.class) Collection<Segment> segments,
@timuralp
As per swift documentation [1], PUT request for manifest requires the objectName alone.
I tried by removing parameters except objectName, the manifest get uploaded properly, but I am facing issue in etag. Swift stat for manifest outputs the md5 of concatenated etags while etag returned in the response from swift is md5 of empty string.
Need suggestions here. How to address this issue.
--
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/jclouds/jclouds/pull/1105#discussion_r118897904
Re: [jclouds/jclouds] DLO - Lower Level Provider API changes (#1105)
Posted by Andrew Gaul <no...@github.com>.
Sorry how do these issues relate to your problem?
Please do not make substantial edits to comments which are better addressed as new comments. GitHub does not generate mails for these and it ruins the flow of the discussion.
--
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/jclouds/jclouds/pull/1105#issuecomment-331347461
Re: [jclouds/jclouds] DLO - Lower Level Provider API changes (#1105)
Posted by archupsg03 <no...@github.com>.
archupsg03 commented on this pull request.
> +@Consumes(APPLICATION_JSON)
+@Path("/{objectName}")
+public interface DynamicLargeObjectApi {
+ /**
+ * Creates or updates a dynamic large object's manifest.
+ *
+ * @param objectName
+ * corresponds to {@link SwiftObject#getName()}.
+ * @param metadata
+ * corresponds to {@link SwiftObject#getMetadata()}.
+ * @param headers
+ * Binds the map to headers, without prefixing/escaping the header
+ * name/key.
+ *
+ * @return {@link SwiftObject#getEtag()} of the object, which is the MD5
+ * checksum of the concatenated ETag values of the {@code segments}.
Not getting this comment
--
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/jclouds/jclouds/pull/1105#discussion_r135259481
Re: [jclouds/jclouds] DLO - Lower Level Provider API changes (#1105)
Posted by archupsg03 <no...@github.com>.
archupsg03 commented on this pull request.
> + * checksum of the concatenated ETag values of the {@code segments}.
+ */
+ @Deprecated
+ @Named("dynamicLargeObject:replaceManifest")
+ @PUT
+ @ResponseParser(ETagHeader.class)
+ @Headers(keys = "X-Object-Manifest", values = "{containerName}/{objectName}/")
+ String replaceManifest(@PathParam("objectName") String objectName,
+ @BinderParam(BindToJsonPayload.class) Collection<Segment> segments,
+ @BinderParam(BindObjectMetadataToHeaders.class) Map<String, String> metadata,
+ @BinderParam(BindToHeaders.class) Map<String, String> headers);
+
+ /**
+ * Creates object segments.
+ *
+ * @param objectName
Done
--
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/jclouds/jclouds/pull/1105#discussion_r120338594
Re: [jclouds/jclouds] DLO - Lower Level Provider API changes (#1105)
Posted by archupsg03 <no...@github.com>.
@archupsg03 pushed 1 commit.
2fba009 Minor change
--
You are receiving this because you are subscribed to this thread.
View it on GitHub:
https://github.com/jclouds/jclouds/pull/1105/files/b1ea32f70e1469d50b0e7298015a304c62f258f4..2fba0091268ac08364fb045d474b5ae0435b3301
Re: [jclouds/jclouds] DLO - Lower Level Provider API changes (#1105)
Posted by archupsg03 <no...@github.com>.
archupsg03 commented on this pull request.
> + @BinderParam(SetPayload.class) Payload blob,
+ @BinderParam(BindObjectMetadataToHeaders.class) Map<String, String> metadata,
+ @BinderParam(BindToHeaders.class) Map<String, String> headers);
+
+ /**
+ * Delete objects.
+ *
+ * @param objectName
+ * corresponds to {@link SwiftObject#getName()}.
+ */
+
+ @Deprecated
+ @Named("dynamicLargeObject: removeLargeFile")
+ @DELETE
+ @Fallback(VoidOnNotFoundOr404.class)
+ void removeLargeFile(@PathParam("containerName") String container, @PathParam("objectName") String name);
Removed removeLargeFile API.Done
--
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/jclouds/jclouds/pull/1105#discussion_r118183824
Re: [jclouds/jclouds] DLO - Lower Level Provider API changes (#1105)
Posted by archupsg03 <no...@github.com>.
archupsg03 commented on this pull request.
> + * @param metadata
+ * corresponds to {@link SwiftObject#getMetadata()}.
+ * @param headers
+ * Binds the map to headers, without prefixing/escaping the header
+ * name/key.
+ *
+ * @return {@link SwiftObject#getEtag()} of the object, which is the MD5
+ * checksum of the concatenated ETag values of the {@code segments}.
+ */
+
+ @Deprecated
+ @Named("dynamicLargeObject: uploadLargeFile")
+ @PUT
+ @Headers(keys = EXPECT, values = "100-continue")
+ @ResponseParser(ETagHeader.class)
+ String uploadLargeFile(@PathParam("containerName") String container, @PathParam("objectName") String objectName,
The method name is misleading. It does the same as you mention in the comment. Will change method name to uploadPart
--
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/jclouds/jclouds/pull/1105#discussion_r118158465
Re: [jclouds/jclouds] DLO - Lower Level Provider API changes (#1105)
Posted by archupsg03 <no...@github.com>.
@timuralp Have fixed the NPE exception and tests run successfully for me. Can you give a try ?
--
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/jclouds/jclouds/pull/1105#issuecomment-312554318
Re: [jclouds/jclouds] DLO - Lower Level Provider API changes (#1105)
Posted by Andrew Gaul <no...@github.com>.
I do not understand your comment about `completeMultipartUpload` which is not part of this pull request.
--
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/jclouds/jclouds/pull/1105#issuecomment-324798349
Re: [jclouds/jclouds] DLO - Lower Level Provider API changes (#1105)
Posted by Andrew Gaul <no...@github.com>.
andrewgaul requested changes on this pull request.
Please address @timuralp comments which suggest a misimplementation of the API with the segments parameter. Will add more comments after you address this set.
> +
+import com.google.common.collect.ImmutableMap;
+import com.google.common.collect.ImmutableMap.Builder;
+
+/**
+ * Updates Content Metadata.
+ *
+ * Required to update the Content Length of Object File as Sum of Content length
+ * of Segments
+ *
+ * @param contentMetadata
+ *
+ * @return Map with Content Metadata.
+ */
+
+public class ContentMetaDataToMap {
But if you don't use it anywhere...?
> + * @param metadata
+ * corresponds to {@link SwiftObject#getMetadata()}.
+ * @param headers
+ * Binds the map to headers, without prefixing/escaping the header
+ * name/key.
+ *
+ * @return {@link SwiftObject#getEtag()} of the object, which is the MD5
+ * checksum of the concatenated ETag values of the {@code segments}.
+ */
+
+ @Deprecated
+ @Named("dynamicLargeObject:uploadLargeFile")
+ @PUT
+ @Headers(keys = EXPECT, values = "100-continue")
+ @ResponseParser(ETagHeader.class)
+ String uploadPart(@PathParam("containerName") String container, @PathParam("objectName") String objectName,
Agree this seems useless.
This pull request should scope exactly to creating and getting DLO objects since the Swift API offers only this. The portable abstraction will largely not interact with it, with the exception of getting legacy DLO objects.
> + for (int i = partNumber; i < 3; partNumber++) {
+ String objName = String.format("%s/%s/%s", objectName, "dlo", partNumber);
+ String data = String.format("%s%s", "data", partNumber);
+ String etag = getApi().getDynamicLargeObjectApi(regionId, defaultContainerName).uploadPart(
+ defaultContainerName, objName, Payloads.newPayload(data), ImmutableMap.of("myfoo", "Bar"),
+ ImmutableMap.of("myfoo", "Bar"));
+ Segment s = new Segment(objName, etag, data.length());
+ assertNotNull(etag);
+ segmentList.add(s);
+ total_size += data.length();
+ }
+ String etagOfEtags = getApi().getDynamicLargeObjectApi(regionId, defaultContainerName).replaceManifest(
+ objectName, segmentList, ImmutableMap.of("MyFoo", "Bar"), ImmutableMap.of("MyFoo", "Bar"));
+
+ SwiftObject bigObject = getApi().getObjectApi(regionId, defaultContainerName).get(objectName);
+ assertThat(bigObject.getETag().equals(etagOfEtags));
You misused `assertThat` -- all of these checks are no-ops. You need to write `assertThat(...).isEqualTo` or `.isTrue`.
> + @Override
+ @BeforeClass(groups = "live")
+ public void setup() {
+ super.setup();
+ for (String regionId : regions) {
+ boolean created = getApi().getContainerApi(regionId).create(defaultContainerName);
+ if (!created) {
+ deleteAllObjectsInContainer(regionId, defaultContainerName);
+ }
+ }
+
+ /* megOf1s = new byte[1024 * 1024];
+ megOf2s = new byte[1024 * 1024];
+
+ Arrays.fill(megOf1s, (byte) 1);
+ Arrays.fill(megOf2s, (byte) 2);*/
Again, remove commented code!
> + super.setup();
+ for (String regionId : regions) {
+ boolean created = getApi().getContainerApi(regionId).create(defaultContainerName);
+ if (!created) {
+ deleteAllObjectsInContainer(regionId, defaultContainerName);
+ }
+ }
+
+ /* megOf1s = new byte[1024 * 1024];
+ megOf2s = new byte[1024 * 1024];
+
+ Arrays.fill(megOf1s, (byte) 1);
+ Arrays.fill(megOf2s, (byte) 2);*/
+
+ megOf1s = TestUtils.randomByteSource().slice(0, 1048576);
+ megOf2s = TestUtils.randomByteSource().slice(0, 1048576);
Prefer 1024 * 1024.
> +
+ awaitConsistency();
+
+ SwiftObject bigObject = getApi().getObjectApi(regionId, containerName).get(name);
+ assertThat(bigObject.getETag().equals(etagOfEtags));
+ assertThat(bigObject.getPayload().getContentMetadata().getContentLength().equals(Long.valueOf(2 * 1024 * 1024)));
+ assertThat(bigObject.getMetadata().equals(ImmutableMap.of("myfoo", "Bar")));
+
+ // segments are visible
+ assertThat(getApi().getContainerApi(regionId).get(containerName).getObjectCount().equals(Long.valueOf(3)));
+ }
+
+ protected void assertMegabyteAndETagMatches(String regionId, String containerName, String name, String etag1s) {
+ SwiftObject object1s = getApi().getObjectApi(regionId, containerName).get(name);
+ assertThat(object1s.getETag().equals(etag1s));
+ assertThat(object1s.getPayload().getContentMetadata().getContentLength().equals(Long.valueOf(1024 * 1024)));
Prefer 1024L * 1024L over `valueOf`.
> + public void setup() {
+ super.setup();
+ for (String regionId : regions) {
+ boolean created = getApi().getContainerApi(regionId).create(defaultContainerName);
+ if (!created) {
+ deleteAllObjectsInContainer(regionId, defaultContainerName);
+ }
+ }
+
+ /* megOf1s = new byte[1024 * 1024];
+ megOf2s = new byte[1024 * 1024];
+
+ Arrays.fill(megOf1s, (byte) 1);
+ Arrays.fill(megOf2s, (byte) 2);*/
+
+ megOf1s = TestUtils.randomByteSource().slice(0, 1048576);
Move initialization of these constants to their declarations and mark `static final`.
> + SwiftObject object1s = getApi().getObjectApi(regionId, containerName).get(name);
+ assertThat(object1s.getETag().equals(etag1s));
+ assertThat(object1s.getPayload().getContentMetadata().getContentLength().equals(Long.valueOf(1024 * 1024)));
+ }
+
+ protected void deleteAllObjectsInContainerDLO(String regionId, final String containerName) {
+ ObjectList objects = getApi().getObjectApi(regionId, containerName).list(new ListContainerOptions());
+ if (objects == null) {
+ return;
+ }
+ List<String> pathsToDelete = Lists.transform(objects, new Function<SwiftObject, String>() {
+ public String apply(SwiftObject input) {
+ return containerName + "/" + input.getName();
+ }
+ });
+ if (!pathsToDelete.isEmpty()) {
Remove useless conditional.
> + // segments are visible
+ assertThat(getApi().getContainerApi(regionId).get(containerName).getObjectCount().equals(Long.valueOf(3)));
+ }
+
+ protected void assertMegabyteAndETagMatches(String regionId, String containerName, String name, String etag1s) {
+ SwiftObject object1s = getApi().getObjectApi(regionId, containerName).get(name);
+ assertThat(object1s.getETag().equals(etag1s));
+ assertThat(object1s.getPayload().getContentMetadata().getContentLength().equals(Long.valueOf(1024 * 1024)));
+ }
+
+ protected void deleteAllObjectsInContainerDLO(String regionId, final String containerName) {
+ ObjectList objects = getApi().getObjectApi(regionId, containerName).list(new ListContainerOptions());
+ if (objects == null) {
+ return;
+ }
+ List<String> pathsToDelete = Lists.transform(objects, new Function<SwiftObject, String>() {
Why do you need this `Function` if you loop over the paths below?
--
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/jclouds/jclouds/pull/1105#pullrequestreview-40140287
Re: [jclouds/jclouds] DLO - Lower Level Provider API changes (#1105)
Posted by archupsg03 <no...@github.com>.
archupsg03 commented on this pull request.
> +
+import com.google.common.collect.ImmutableMap;
+import com.google.common.collect.ImmutableMap.Builder;
+
+/**
+ * Updates Content Metadata.
+ *
+ * Required to update the Content Length of Object File as Sum of Content length
+ * of Segments
+ *
+ * @param contentMetadata
+ *
+ * @return Map with Content Metadata.
+ */
+
+public class ContentMetaDataToMap {
Replace Manifest request an argument with Content-metadata
blob.getMetadata().getContentMetadata() return
[cacheControl=null, contentDisposition=null, contentEncoding=null, contentLanguage=null, contentLength=36701504, contentMD5=null, contentType=application/unknown, expires=null]
But we need Map{content-type=application/unknown}
Static Large Object API also does something similar to this.
Suggestion needed here :(
--
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/jclouds/jclouds/pull/1105#discussion_r118170303
Re: [jclouds/jclouds] DLO - Lower Level Provider API changes (#1105)
Posted by archupsg03 <no...@github.com>.
@archupsg03 pushed 1 commit.
3aa1552 Null Pointer Exception in DLO Live Test fixed
--
You are receiving this because you are subscribed to this thread.
View it on GitHub:
https://github.com/jclouds/jclouds/pull/1105/files/93b41da44f42710ab1ed55a78ea05c88b5b2e7e1..3aa1552342bf030eef0838a43cc354cff7dcc7b9
Re: [jclouds/jclouds] DLO - Lower Level Provider API changes (#1105)
Posted by Timur Alperovich <no...@github.com>.
@archupsg03 thanks! This looks pretty good. I will try to run the tests this weekend to verify. @andrewgaul is still traveling for next couple of weeks, so we'll have to wait until the first week of July or so to merge this and for his comments on the PR.
--
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/jclouds/jclouds/pull/1105#issuecomment-310580163
Re: [jclouds/jclouds] DLO - Lower Level Provider API changes (#1105)
Posted by Andrew Gaul <no...@github.com>.
andrewgaul commented on this pull request.
> + // segments are visible
+ assertThat(getApi().getContainerApi(regionId).get(containerName).getObjectCount().equals(Long.valueOf(3)));
+ }
+
+ protected void assertMegabyteAndETagMatches(String regionId, String containerName, String name, String etag1s) {
+ SwiftObject object1s = getApi().getObjectApi(regionId, containerName).get(name);
+ assertThat(object1s.getETag().equals(etag1s));
+ assertThat(object1s.getPayload().getContentMetadata().getContentLength().equals(Long.valueOf(1024 * 1024)));
+ }
+
+ protected void deleteAllObjectsInContainerDLO(String regionId, final String containerName) {
+ ObjectList objects = getApi().getObjectApi(regionId, containerName).list(new ListContainerOptions());
+ if (objects == null) {
+ return;
+ }
+ List<String> pathsToDelete = Lists.transform(objects, new Function<SwiftObject, String>() {
Why do you mix imperative and functional styles? You can more clearly write:
```java
for (SwiftObject object : objects) {
String name = containerName + "/" + input.getName();
getApi().getObjectApi(regionId, containerName).delete(name);
}
```
> +import org.jclouds.rest.annotations.BinderParam;
+import org.jclouds.rest.annotations.Headers;
+import org.jclouds.rest.annotations.RequestFilters;
+import org.jclouds.rest.annotations.ResponseParser;
+
+import com.google.common.annotations.Beta;
+
+/**
+ * Provides access to the OpenStack Object Storage (Swift) Static Large Object
+ * API features.
+ * <p/>
+ * This API is new to jclouds and hence is in Beta. That means we need people to
+ * use it and give us feedback. Based on that feedback, minor changes to the
+ * interfaces may happen. This code will replace
+ * org.jclouds.openstack.swift.SwiftClient in jclouds 2.0 and it is recommended
+ * you adopt it sooner than later.
Remove useless comment.
> +@Consumes(APPLICATION_JSON)
+@Path("/{objectName}")
+public interface DynamicLargeObjectApi {
+ /**
+ * Creates or updates a dynamic large object's manifest.
+ *
+ * @param objectName
+ * corresponds to {@link SwiftObject#getName()}.
+ * @param metadata
+ * corresponds to {@link SwiftObject#getMetadata()}.
+ * @param headers
+ * Binds the map to headers, without prefixing/escaping the header
+ * name/key.
+ *
+ * @return {@link SwiftObject#getEtag()} of the object, which is the MD5
+ * checksum of the concatenated ETag values of the {@code segments}.
Add `@deprecated` Javadoc pointing users to SLO API.
> +import org.jclouds.openstack.swift.v1.options.ListContainerOptions;
+import org.jclouds.utils.TestUtils;
+import org.testng.annotations.Test;
+
+import com.google.common.base.Charsets;
+import com.google.common.base.Function;
+import com.google.common.collect.ImmutableMap;
+import com.google.common.collect.Lists;
+import com.google.common.io.ByteSource;
+
+@Test(groups = "live", testName = "DynamicLargeObjectApiLiveTest", singleThreaded = true)
+public class DynamicLargeObjectApiLiveTest extends BaseSwiftApiLiveTest {
+
+ private String defaultName = getClass().getSimpleName();
+ private static final ByteSource megOf1s = TestUtils.randomByteSource().slice(0, 1024 * 1024);;
+ private static final ByteSource megOf2s = TestUtils.randomByteSource().slice(0, 1024 * 1024);;
Extra semicolons.
> + assertReplaceManifest(regionId, defaultName);
+ uploadLargeFile(regionId);
+ }
+ }
+
+ @SuppressWarnings("deprecation")
+ @Test
+ public void uploadLargeFile(String regionId) throws IOException, InterruptedException {
+ int total_size = 0;
+ RegionScopedBlobStoreContext ctx = RegionScopedBlobStoreContext.class.cast(view);
+ BlobStore blobStore = ctx.getBlobStore();
+ String defaultContainerName = getContainerName();
+ // configure the blobstore to use multipart uploading of the file
+ for (int partNumber = 0; partNumber < 3; partNumber++) {
+ String objName = String.format("%s/%s/%s", objectName, "dlo", partNumber);
+ String data = String.format("%s%s", "data", partNumber);
Simplify to:
```java
String data = "data" + partNumber;
```
> + private static final ByteSource megOf1s = TestUtils.randomByteSource().slice(0, 1024 * 1024);;
+ private static final ByteSource megOf2s = TestUtils.randomByteSource().slice(0, 1024 * 1024);;
+ private String objectName = "myObject";
+
+ @Test
+ public void testReplaceManifest() throws Exception {
+ for (String regionId : regions) {
+ assertReplaceManifest(regionId, defaultName);
+ uploadLargeFile(regionId);
+ }
+ }
+
+ @SuppressWarnings("deprecation")
+ @Test
+ public void uploadLargeFile(String regionId) throws IOException, InterruptedException {
+ int total_size = 0;
Prefer `long totalSize` which will eliminate the `Long.valueOf` call below.
> +
+import org.jclouds.io.Payloads;
+import org.jclouds.openstack.swift.v1.SwiftApi;
+import org.jclouds.openstack.v2_0.internal.BaseOpenStackMockTest;
+import org.testng.annotations.Test;
+
+import com.google.common.collect.ImmutableMap;
+import com.google.common.net.HttpHeaders;
+import com.squareup.okhttp.mockwebserver.MockResponse;
+import com.squareup.okhttp.mockwebserver.MockWebServer;
+import com.squareup.okhttp.mockwebserver.RecordedRequest;
+
+@Test(groups = "unit", testName = "DynamicLargeObjectApiMockTest")
+public class DynamicLargeObjectApiMockTest extends BaseOpenStackMockTest<SwiftApi> {
+
+ String containerName = "myContainer";
`private` visibility.
--
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/jclouds/jclouds/pull/1105#pullrequestreview-48767678
Re: [jclouds/jclouds] DLO - Lower Level Provider API changes (#1105)
Posted by archupsg03 <no...@github.com>.
archupsg03 commented on this pull request.
> +@RequestFilters(AuthenticateRequest.class)
+@Consumes(APPLICATION_JSON)
+@Path("/{objectName}")
+public interface DynamicLargeObjectApi {
+ /**
+ * Creates or updates a dynamic large object's manifest.
+ *
+ * @param objectName
+ * corresponds to {@link SwiftObject#getName()}.
+ * @param metadata
+ * corresponds to {@link SwiftObject#getMetadata()}.
+ * @param headers
+ * Binds the map to headers, without prefixing/escaping the header
+ * name/key.
+ *
+ * @return {@link SwiftObject#getEtag()} of the object, which is the MD5
I am getting etag of empty string. Same like what you have mentioned. Its copy paste error.
--
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/jclouds/jclouds/pull/1105#discussion_r120588488
Re: [jclouds/jclouds] JCLOUDS-1281: DLO - Lower Level Provider API
changes (#1105)
Posted by Andrew Gaul <no...@github.com>.
I rebased and squashed your commit and pushed to master as eaf3c779dce63ddc808d5ff8fe3ed9aade10b2f2. I included some test fixes and pushed to master as 9dc85ecaabb23bdc0b1d6b3b6650f22673631203.
--
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/jclouds/jclouds/pull/1105#issuecomment-342586832
Re: [jclouds/jclouds] DLO - Lower Level Provider API changes (#1105)
Posted by Andrew Gaul <no...@github.com>.
@archupsg03 Do you plan to continue work on this pull request or should I close it?
--
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/jclouds/jclouds/pull/1105#issuecomment-327648825
Re: [jclouds/jclouds] DLO - Lower Level Provider API changes (#1105)
Posted by Timur Alperovich <no...@github.com>.
timuralp commented on this pull request.
I think this is getting better. Please addres the outstanding comments.
> @@ -37,7 +37,7 @@
@Named("size_bytes")
private final long sizeBytes;
- private Segment(String path, String etag, long sizeBytes) {
+ public Segment(String path, String etag, long sizeBytes) {
Is this change still necessary? Is it only used in tests?
> +@RequestFilters(AuthenticateRequest.class)
+@Consumes(APPLICATION_JSON)
+@Path("/{objectName}")
+public interface DynamicLargeObjectApi {
+ /**
+ * Creates or updates a dynamic large object's manifest.
+ *
+ * @param objectName
+ * corresponds to {@link SwiftObject#getName()}.
+ * @param metadata
+ * corresponds to {@link SwiftObject#getMetadata()}.
+ * @param headers
+ * Binds the map to headers, without prefixing/escaping the header
+ * name/key.
+ *
+ * @return {@link SwiftObject#getEtag()} of the object, which is the MD5
As a result of a PUT of a 0-sized object with X-Object-Manifest header set, I'm observing that Swift returns an ETag of the 0-sized object. Are you observing a different behavior in your tests?
> + @PUT
+ @ResponseParser(ETagHeader.class)
+ @Headers(keys = "X-Object-Manifest", values = "{containerName}/{objectName}/")
+ String putManifest(@PathParam("objectName") String objectName,
+ @BinderParam(BindObjectMetadataToHeaders.class) Map<String, String> metadata,
+ @BinderParam(BindToHeaders.class) Map<String, String> headers);
+
+ /**
+ * Creates or updates a dynamic large object's manifest.
+ *
+ * @param objectName
+ * corresponds to {@link SwiftObject#getName()}.
+ * @param metadata
+ * corresponds to {@link SwiftObject#getMetadata()}.
+ *
+ * @return {@link SwiftObject#getEtag()} of the object, which is the MD5
Same comment as above about the ETag applies here.
> + * @param metadata
+ * corresponds to {@link SwiftObject#getMetadata()}.
+ * @param headers
+ * Binds the map to headers, without prefixing/escaping the header
+ * name/key.
+ *
+ * @return {@link SwiftObject#getEtag()} of the object, which is the MD5
+ * checksum of the concatenated ETag values of the {@code segments}.
+ */
+
+ @Deprecated
+ @Named("dynamicLargeObject:uploadLargeFile")
+ @PUT
+ @Headers(keys = EXPECT, values = "100-continue")
+ @ResponseParser(ETagHeader.class)
+ String uploadPart(@PathParam("containerName") String container, @PathParam("objectName") String objectName,
@archupsg03 can you address the questions above? I'm still unclear on why this method is required. The point is that callers can and should call `putObject()` to upload new or replace existing parts of the DLO. Could you explain how you expect this method to be used and why it would be preferred to calling `putObject()`?
> + private static final ByteSource megOf1s = TestUtils.randomByteSource().slice(0, 1024 * 1024);;
+ private static final ByteSource megOf2s = TestUtils.randomByteSource().slice(0, 1024 * 1024);;
+ private String objectName = "myObject";
+
+ @Test
+ public void testReplaceManifest() throws Exception {
+ for (String regionId : regions) {
+ assertReplaceManifest(regionId, defaultContainerName, defaultName);
+ uploadLargeFile(regionId);
+ }
+ }
+
+ @SuppressWarnings("deprecation")
+ @Test
+ public void uploadLargeFile(String regionId) throws IOException, InterruptedException {
+ List<Segment> segmentList = new ArrayList<Segment>();
This is no longer used in the test. Can we remove this code entirely? On the other hand, it would be useful to upload some blobs and verify that the ETag is returned as expected from a HEAD.
> + assertThat(getApi().getContainerApi(regionId).get(defaultContainerName).getObjectCount()).isEqualTo(Long.valueOf(3));
+ }
+
+ @SuppressWarnings("deprecation")
+ protected void assertReplaceManifest(String regionId, String containerName, String name) {
+ ObjectApi objectApi = getApi().getObjectApi(regionId, containerName);
+
+ String etag1s = objectApi.put(name + "/1", newByteSourcePayload(megOf1s));
+ awaitConsistency();
+ assertMegabyteAndETagMatches(regionId, containerName, name + "/1", etag1s);
+
+ String etag2s = objectApi.put(name + "/2", newByteSourcePayload(megOf2s));
+ awaitConsistency();
+ assertMegabyteAndETagMatches(regionId, containerName, name + "/2", etag2s);
+
+ List<Segment> segments = ImmutableList.<Segment> builder()
I don't think these segments are ever uploaded?
> + SwiftObject object1s = getApi().getObjectApi(regionId, containerName).get(name);
+ assertThat(object1s.getETag().equals(etag1s));
+ assertThat(object1s.getPayload().getContentMetadata().getContentLength().equals(Long.valueOf(1024 * 1024)));
+ }
+
+ protected void deleteAllObjectsInContainerDLO(String regionId, final String containerName) {
+ ObjectList objects = getApi().getObjectApi(regionId, containerName).list(new ListContainerOptions());
+ if (objects == null) {
+ return;
+ }
+ List<String> pathsToDelete = Lists.transform(objects, new Function<SwiftObject, String>() {
+ public String apply(SwiftObject input) {
+ return containerName + "/" + input.getName();
+ }
+ });
+ if (!pathsToDelete.isEmpty()) {
If `pathsToDelete` is empty, the for loop would be a NOOP.
--
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/jclouds/jclouds/pull/1105#pullrequestreview-42512966
Re: [jclouds/jclouds] DLO - Lower Level Provider API changes (#1105)
Posted by archupsg03 <no...@github.com>.
ok no problem. But can you check whether the issue is because of one of the 2 Jira's mentioned in above comment.
--
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/jclouds/jclouds/pull/1105#issuecomment-331346672
Re: [jclouds/jclouds] DLO - Lower Level Provider API changes (#1105)
Posted by Andrew Gaul <no...@github.com>.
andrewgaul commented on this pull request.
> +
+import org.jclouds.io.Payloads;
+import org.jclouds.openstack.swift.v1.SwiftApi;
+import org.jclouds.openstack.v2_0.internal.BaseOpenStackMockTest;
+import org.testng.annotations.Test;
+
+import com.google.common.collect.ImmutableMap;
+import com.google.common.net.HttpHeaders;
+import com.squareup.okhttp.mockwebserver.MockResponse;
+import com.squareup.okhttp.mockwebserver.MockWebServer;
+import com.squareup.okhttp.mockwebserver.RecordedRequest;
+
+@Test(groups = "unit", testName = "DynamicLargeObjectApiMockTest")
+public class DynamicLargeObjectApiMockTest extends BaseOpenStackMockTest<SwiftApi> {
+
+ String containerName = "myContainer";
How is that possible? You do not extend `DynamicLargeObjectApiMockTest` anywhere. Also this class should be `final`.
--
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/jclouds/jclouds/pull/1105#discussion_r135453138
Re: [jclouds/jclouds] DLO - Lower Level Provider API changes (#1105)
Posted by Timur Alperovich <no...@github.com>.
timuralp commented on this pull request.
It's a little difficult for me to review this in isolation, without the DLO abstraction that is using this PR. I would expect very few new methods added here -- specifically, only replaceManifest (as you have it, or putManifest that I suggested), which has the DLO header. Outside of that, the other methods don't make sense to me as-is (e.g. uploadPart is the same putObject, as far as I can tell).
> + * @param metadata
+ * corresponds to {@link SwiftObject#getMetadata()}.
+ * @param headers
+ * Binds the map to headers, without prefixing/escaping the header
+ * name/key.
+ *
+ * @return {@link SwiftObject#getEtag()} of the object, which is the MD5
+ * checksum of the concatenated ETag values of the {@code segments}.
+ */
+ @Deprecated
+ @Named("dynamicLargeObject:replaceManifest")
+ @PUT
+ @ResponseParser(ETagHeader.class)
+ @Headers(keys = "X-Object-Manifest", values = "{containerName}/{objectName}/")
+ String replaceManifest(@PathParam("objectName") String objectName,
+ @BinderParam(BindToJsonPayload.class) Collection<Segment> segments,
With DLO, manifest header specifies a segment prefix. How does the segments collection get used? This seems like a copy of the SLO API, but I don't think it actually makes sense here -- in the SLO case, the actual JSON manifest object is constructed from the segments.
> + * ordered parts which will be concatenated upon download.
+ * @param metadata
+ * corresponds to {@link SwiftObject#getMetadata()}.
+ * @param headers
+ * Binds the map to headers, without prefixing/escaping the header
+ * name/key.
+ *
+ * @return {@link SwiftObject#getEtag()} of the object, which is the MD5
+ * checksum of the concatenated ETag values of the {@code segments}.
+ */
+ @Deprecated
+ @Named("dynamicLargeObject:replaceManifest")
+ @PUT
+ @ResponseParser(ETagHeader.class)
+ @Headers(keys = "X-Object-Manifest", values = "{containerName}/{objectName}/")
+ String replaceManifest(@PathParam("objectName") String objectName,
I understand that this is the same as the SLO API, but the replaceManifest name seems like a misnomer. Maybe `putManifest` would make more sense? Don't have a strong preference on this, though.
> + * use it and give us feedback. Based on that feedback, minor changes to the
+ * interfaces may happen. This code will replace
+ * org.jclouds.openstack.swift.SwiftClient in jclouds 2.0 and it is recommended
+ * you adopt it sooner than later.
+ */
+@Beta
+@RequestFilters(AuthenticateRequest.class)
+@Consumes(APPLICATION_JSON)
+@Path("/{objectName}")
+public interface DynamicLargeObjectApi {
+ /**
+ * Creates or updates a dynamic large object's manifest.
+ *
+ * @param objectName
+ * corresponds to {@link SwiftObject#getName()}.
+ * @param segments
This comment does not make sense in the DLO context. See a comment below about the segments parameter.
> + * @param metadata
+ * corresponds to {@link SwiftObject#getMetadata()}.
+ * @param headers
+ * Binds the map to headers, without prefixing/escaping the header
+ * name/key.
+ *
+ * @return {@link SwiftObject#getEtag()} of the object, which is the MD5
+ * checksum of the concatenated ETag values of the {@code segments}.
+ */
+
+ @Deprecated
+ @Named("dynamicLargeObject:uploadLargeFile")
+ @PUT
+ @Headers(keys = EXPECT, values = "100-continue")
+ @ResponseParser(ETagHeader.class)
+ String uploadPart(@PathParam("containerName") String container, @PathParam("objectName") String objectName,
Is this any different from `PutObject`? I think a higher level wrapper is required to make DLO operations make sense, but I don't think this API method is required. Maybe you could explain how you envision `uploadPart` to fit with the higher level primitives.
> + @Deprecated
+ @Named("dynamicLargeObject:replaceManifest")
+ @PUT
+ @ResponseParser(ETagHeader.class)
+ @Headers(keys = "X-Object-Manifest", values = "{containerName}/{objectName}/")
+ String replaceManifest(@PathParam("objectName") String objectName,
+ @BinderParam(BindToJsonPayload.class) Collection<Segment> segments,
+ @BinderParam(BindObjectMetadataToHeaders.class) Map<String, String> metadata,
+ @BinderParam(BindToHeaders.class) Map<String, String> headers);
+
+ /**
+ * Creates object segments.
+ *
+ * @param objectName
+ * corresponds to {@link SwiftObject#getName()}.
+ * @param segments
Copy/paste comment error? I don't see a segments parameter.
> + * checksum of the concatenated ETag values of the {@code segments}.
+ */
+ @Deprecated
+ @Named("dynamicLargeObject:replaceManifest")
+ @PUT
+ @ResponseParser(ETagHeader.class)
+ @Headers(keys = "X-Object-Manifest", values = "{containerName}/{objectName}/")
+ String replaceManifest(@PathParam("objectName") String objectName,
+ @BinderParam(BindToJsonPayload.class) Collection<Segment> segments,
+ @BinderParam(BindObjectMetadataToHeaders.class) Map<String, String> metadata,
+ @BinderParam(BindToHeaders.class) Map<String, String> headers);
+
+ /**
+ * Creates object segments.
+ *
+ * @param objectName
I don't see an objectName parameter. However, containerName and blob are not documented.
--
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/jclouds/jclouds/pull/1105#pullrequestreview-40093201
Re: [jclouds/jclouds] DLO - Lower Level Provider API changes (#1105)
Posted by archupsg03 <no...@github.com>.
archupsg03 commented on this pull request.
> + // segments are visible
+ assertThat(getApi().getContainerApi(regionId).get(containerName).getObjectCount().equals(Long.valueOf(3)));
+ }
+
+ protected void assertMegabyteAndETagMatches(String regionId, String containerName, String name, String etag1s) {
+ SwiftObject object1s = getApi().getObjectApi(regionId, containerName).get(name);
+ assertThat(object1s.getETag().equals(etag1s));
+ assertThat(object1s.getPayload().getContentMetadata().getContentLength().equals(Long.valueOf(1024 * 1024)));
+ }
+
+ protected void deleteAllObjectsInContainerDLO(String regionId, final String containerName) {
+ ObjectList objects = getApi().getObjectApi(regionId, containerName).list(new ListContainerOptions());
+ if (objects == null) {
+ return;
+ }
+ List<String> pathsToDelete = Lists.transform(objects, new Function<SwiftObject, String>() {
Before looping, the object by their name for delete, we need list of object in container.
--
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/jclouds/jclouds/pull/1105#discussion_r118909055
Re: [jclouds/jclouds] DLO - Lower Level Provider API changes (#1105)
Posted by archupsg03 <no...@github.com>.
archupsg03 commented on this pull request.
> + assertEquals(bigObject.getETag(), etagOfEtags);
+ assertEquals(bigObject.getPayload().getContentMetadata().getContentLength(), Long.valueOf(2 * 1024 * 1024));
+ assertEquals(bigObject.getMetadata(), ImmutableMap.of("myfoo", "Bar"));
+
+ // segments are visible
+ assertEquals(getApi().getContainerApi(regionId).get(containerName).getObjectCount(), Long.valueOf(3));
+ }
+
+ protected void assertMegabyteAndETagMatches(String regionId, String containerName, String name, String etag1s) {
+ SwiftObject object1s = getApi().getObjectApi(regionId, containerName).get(name);
+ assertEquals(object1s.getETag(), etag1s);
+ assertEquals(object1s.getPayload().getContentMetadata().getContentLength(), Long.valueOf(1024 * 1024));
+ }
+
+ protected void deleteAllObjectsInContainerDLO(String regionId, final String containerName) {
+ Uninterruptibles.sleepUninterruptibly(10, TimeUnit.SECONDS);
Done
--
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/jclouds/jclouds/pull/1105#discussion_r118183909
Re: [jclouds/jclouds] DLO - Lower Level Provider API changes (#1105)
Posted by archupsg03 <no...@github.com>.
archupsg03 commented on this pull request.
> + * @param metadata
+ * corresponds to {@link SwiftObject#getMetadata()}.
+ * @param headers
+ * Binds the map to headers, without prefixing/escaping the header
+ * name/key.
+ *
+ * @return {@link SwiftObject#getEtag()} of the object, which is the MD5
+ * checksum of the concatenated ETag values of the {@code segments}.
+ */
+ @Deprecated
+ @Named("dynamicLargeObject:replaceManifest")
+ @PUT
+ @ResponseParser(ETagHeader.class)
+ @Headers(keys = "X-Object-Manifest", values = "{containerName}/{objectName}/")
+ String replaceManifest(@PathParam("objectName") String objectName,
+ @BinderParam(BindToJsonPayload.class) Collection<Segment> segments,
@timuralp Have made changes accordingly. Can you have a look ? Suggestions/Comments are welcome
--
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/jclouds/jclouds/pull/1105#discussion_r120339105
Re: [jclouds/jclouds] DLO - Lower Level Provider API changes (#1105)
Posted by archupsg03 <no...@github.com>.
archupsg03 commented on this pull request.
> +
+import com.google.common.base.Function;
+import com.google.common.collect.ImmutableList;
+import com.google.common.collect.ImmutableMap;
+import com.google.common.collect.Lists;
+import com.google.common.io.ByteSource;
+import com.google.common.util.concurrent.Uninterruptibles;
+
+@Test(groups = "live", testName = "DynamicLargeObjectApiLiveTest", singleThreaded = true)
+public class DynamicLargeObjectApiLiveTest extends BaseSwiftApiLiveTest {
+
+ private String defaultName = getClass().getSimpleName();
+ private String defaultContainerName = getClass().getSimpleName() + "Container";
+ private byte[] megOf1s;
+ private byte[] megOf2s;
+ String objectName = "myObject";
Done
--
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/jclouds/jclouds/pull/1105#discussion_r118183835
Re: [jclouds/jclouds] DLO - Lower Level Provider API changes (#1105)
Posted by archupsg03 <no...@github.com>.
@timuralp Am not facing any checkstyle exception in the final commit. Can you just point out to few white space
--
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/jclouds/jclouds/pull/1105#issuecomment-313354178
Re: [jclouds/jclouds] DLO - Lower Level Provider API changes (#1105)
Posted by Andrew Gaul <no...@github.com>.
The tests fail since they create a container name in the default region then try to create DLOs in a different region.
--
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/jclouds/jclouds/pull/1105#issuecomment-342550648
Re: [jclouds/jclouds] DLO - Lower Level Provider API changes (#1105)
Posted by archupsg03 <no...@github.com>.
Line number 87 in DynamicLargetObjectApiLiveTest is call to ObjectApi.put methods which was existing and this is a basic call. It may be because of some other issue
--
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/jclouds/jclouds/pull/1105#issuecomment-325577963
Re: [jclouds/jclouds] DLO - Lower Level Provider API changes (#1105)
Posted by Andrew Gaul <no...@github.com>.
andrewgaul commented on this pull request.
> + @BinderParam(SetPayload.class) Payload blob,
+ @BinderParam(BindObjectMetadataToHeaders.class) Map<String, String> metadata,
+ @BinderParam(BindToHeaders.class) Map<String, String> headers);
+
+ /**
+ * Delete objects.
+ *
+ * @param objectName
+ * corresponds to {@link SwiftObject#getName()}.
+ */
+
+ @Deprecated
+ @Named("dynamicLargeObject: removeLargeFile")
+ @DELETE
+ @Fallback(VoidOnNotFoundOr404.class)
+ void removeLargeFile(@PathParam("containerName") String container, @PathParam("objectName") String name);
The jclouds provider API should reflect the REST API. Thus this method is not needed.
--
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/jclouds/jclouds/pull/1105#discussion_r118165470
Re: [jclouds/jclouds] JCLOUDS-1281: DLO - Lower Level Provider API
changes (#1105)
Posted by Andrew Gaul <no...@github.com>.
Closed #1105.
--
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/jclouds/jclouds/pull/1105#event-1330377191
Re: [jclouds/jclouds] DLO - Lower Level Provider API changes (#1105)
Posted by archupsg03 <no...@github.com>.
@archupsg03 pushed 1 commit.
998b6a9 DLO Manifest Comments addressed
--
You are receiving this because you are subscribed to this thread.
View it on GitHub:
https://github.com/jclouds/jclouds/pull/1105/files/b8c04a07eae6abd285d13f04d32b0c1ae4a6b2a4..998b6a926c52797549393348fdf44f7b0da3d557