You are viewing a plain text version of this content. The canonical link for it is here.
Posted to notifications@jclouds.apache.org by Chris Custine <no...@github.com> on 2014/06/04 21:52:35 UTC
[jclouds] JCLOUDS-584 - Update HP Cloud object storage provider
(#394)
https://issues.apache.org/jira/browse/JCLOUDS-584
You can merge this Pull Request by running:
git pull https://github.com/ccustine/jclouds fixes/JCLOUDS-584
Or you can view, comment on it, or merge it online at:
https://github.com/jclouds/jclouds/pull/394
-- Commit Summary --
* HP Cloud Object Storage Fixes for 13.5
-- File Changes --
M blobstore/src/test/java/org/jclouds/blobstore/integration/internal/BaseContainerLiveTest.java (8)
M providers/hpcloud-objectstorage/pom.xml (4)
M providers/hpcloud-objectstorage/src/main/java/org/jclouds/hpcloud/objectstorage/HPCloudObjectStorageApiMetadata.java (28)
M providers/hpcloud-objectstorage/src/main/java/org/jclouds/hpcloud/objectstorage/HPCloudObjectStorageProviderMetadata.java (9)
M providers/hpcloud-objectstorage/src/main/java/org/jclouds/hpcloud/objectstorage/blobstore/HPCloudObjectStorageAsyncBlobStore.java (22)
M providers/hpcloud-objectstorage/src/main/java/org/jclouds/hpcloud/objectstorage/blobstore/HPCloudObjectStorageBlobStore.java (7)
M providers/hpcloud-objectstorage/src/main/java/org/jclouds/hpcloud/objectstorage/config/HPCloudObjectStorageRestClientModule.java (39)
M providers/hpcloud-objectstorage/src/main/java/org/jclouds/hpcloud/objectstorage/domain/CDNContainer.java (4)
M providers/hpcloud-objectstorage/src/test/java/org/jclouds/hpcloud/objectstorage/blobstore/HPCloudObjectStorageBlobSignerExpectTest.java (26)
M providers/hpcloud-objectstorage/src/test/java/org/jclouds/hpcloud/objectstorage/blobstore/HPCloudObjectStorageBlobStoreExpectTest.java (2)
M providers/hpcloud-objectstorage/src/test/java/org/jclouds/hpcloud/objectstorage/blobstore/integration/HPCloudObjectStorageBlobIntegrationLiveTest.java (15)
M providers/hpcloud-objectstorage/src/test/java/org/jclouds/hpcloud/objectstorage/blobstore/integration/HPCloudObjectStorageContainerLiveTest.java (10)
M providers/hpcloud-objectstorage/src/test/java/org/jclouds/hpcloud/objectstorage/blobstore/integration/HPCloudObjectStorageServiceIntegrationLiveTest.java (7)
M providers/hpcloud-objectstorage/src/test/resources/keystoneAuthResponseNoCDN.json (381)
M providers/hpcloud-objectstorage/src/test/resources/keystoneAuthResponseWithCDN.json (411)
M providers/hpcloud-objectstorage/src/test/resources/test_list_cdn.json (6)
-- Patch Links --
https://github.com/jclouds/jclouds/pull/394.patch
https://github.com/jclouds/jclouds/pull/394.diff
---
Reply to this email directly or view it on GitHub:
https://github.com/jclouds/jclouds/pull/394
Re: [jclouds] JCLOUDS-584 - Update HP Cloud object storage provider
(#394)
Posted by CloudBees pull request builder plugin <no...@github.com>.
[jclouds-java-7-pull-requests #1398](https://jclouds.ci.cloudbees.com/job/jclouds-java-7-pull-requests/1398/) SUCCESS
This pull request looks good
---
Reply to this email directly or view it on GitHub:
https://github.com/jclouds/jclouds/pull/394#issuecomment-47012238
Re: [jclouds] JCLOUDS-584 - Update HP Cloud object storage provider
(#394)
Posted by Andrew Phillips <no...@github.com>.
> @@ -69,12 +69,7 @@ protected HPCloudObjectStorageBlobStore(BlobStoreContext context, BlobUtils blob
>
> @Override
> public boolean createContainerInLocation(Location location, String container, CreateContainerOptions options) {
> - try {
> - return createContainerInLocation(location, container);
> - } finally {
> - if (options.isPublicRead())
> - enableAndCache.apply(container);
> - }
> + return options.isPublicRead() ? enableAndCache.apply(container) != null : createContainerInLocation(location, container);
> I will clarify this with a comment and rename the enableAndCache method.
Thanks for explaining, Chris! Given that, I think a short comment "enabling CDN on the container also creates it if it does not exist" would be fine for me.
---
Reply to this email directly or view it on GitHub:
https://github.com/jclouds/jclouds/pull/394/files#r13494428
Re: [jclouds] JCLOUDS-584 - Update HP Cloud object storage provider
(#394)
Posted by BuildHive <no...@github.com>.
[jclouds » jclouds #1243](https://buildhive.cloudbees.com/job/jclouds/job/jclouds/1243/) SUCCESS
This pull request looks good
[(what's this?)](https://www.cloudbees.com/what-is-buildhive)
---
Reply to this email directly or view it on GitHub:
https://github.com/jclouds/jclouds/pull/394#issuecomment-46510415
Re: [jclouds] JCLOUDS-584 - Update HP Cloud object storage provider
(#394)
Posted by Andrew Phillips <no...@github.com>.
> + public static class HPCloudObjectStorageEndpointModule extends KeystoneAuthenticationModule {
> + @Provides
> + @Singleton
> + @Storage
> + protected Supplier<URI> provideStorageUrl(RegionIdToURISupplier.Factory factory,
> + @ApiVersion String apiVersion,
> + @Named(LocationConstants.PROPERTY_REGION) String region) {
> +
> + Supplier<Map<String, Supplier<URI>>> endpointsSupplier = factory.createForApiTypeAndVersion(ServiceType.OBJECT_STORE, null);
> +
> + if (region.isEmpty()) {
> + return getLastValueInMap(endpointsSupplier);
> + } else {
> + return getValueInMapOrNull(endpointsSupplier, region);
> + }
> + }
Duplicate of the code above? Any way to factor this out? And will this ever return `null` (from the `getValueInMapOrNull` call)? If so, mark the return value as `@Nullable`?
---
Reply to this email directly or view it on GitHub:
https://github.com/jclouds/jclouds/pull/394/files#r13473150
Re: [jclouds] JCLOUDS-584 - Update HP Cloud object storage provider
(#394)
Posted by Chris Custine <no...@github.com>.
> }, userExecutor);
> - return returnVal;
> + }
Copied from previous comment:
"Yes, a byproduct of enabling CDN on the container (what enableAndCache does) is if the container doesn't exist, it gets created. So it only makes one call instead of two, and it also fixes an issue with resolving the HP Cloud object storage endpoint versions in the catalog where it would create the container in region-a and try to enable the CDN in region-b. I can document this issue in more detail but I have pointed this issue out to HP and hope they fix it.
I will clarify this with a comment and rename the enableAndCache method."
---
Reply to this email directly or view it on GitHub:
https://github.com/jclouds/jclouds/pull/394/files#r14144761
Re: [jclouds] JCLOUDS-584 - Update HP Cloud object storage provider
(#394)
Posted by Andrew Phillips <no...@github.com>.
> @@ -24,4 +25,13 @@
> public HPCloudObjectStorageContainerLiveTest() {
> provider = "hpcloud-objectstorage";
> }
> +
> + @Override
> + @Test
> + public void testPublicAccessInNonDefaultLocation() { throw new SkipException("Locations are ignored"); }
[minor] Do we prefer `SkipException` or `@Test(enabled = false)`?
---
Reply to this email directly or view it on GitHub:
https://github.com/jclouds/jclouds/pull/394/files#r14159808
Re: [jclouds] JCLOUDS-584 - Update HP Cloud object storage provider
(#394)
Posted by Andrew Phillips <no...@github.com>.
> @@ -72,6 +76,10 @@ public boolean apply(@Nullable StorageMetadata input) {
>
> assert metadata.getPublicUri() != null : metadata;
>
> + SocketOpen socketOpen = context.utils().injector().getInstance(SocketOpen.class);
> + Predicate<HostAndPort> socketTester = retry(socketOpen, 1200, 10, SECONDS);
> + assert socketTester.apply(HostAndPort.fromHost(metadata.getPublicUri().getHost()).withDefaultPort(80)) : metadata.getPublicUri();
Use `assertTrue`? I know we still have `assert`s here and there in the codebase, but in general I think we're trying to use the TestNG assertions in tests now. Thanks!
---
Reply to this email directly or view it on GitHub:
https://github.com/jclouds/jclouds/pull/394/files#r13472909
Re: [jclouds] JCLOUDS-584 - Update HP Cloud object storage provider
(#394)
Posted by CloudBees pull request builder plugin <no...@github.com>.
[jclouds-java-7-pull-requests #1329](https://jclouds.ci.cloudbees.com/job/jclouds-java-7-pull-requests/1329/) SUCCESS
This pull request looks good
---
Reply to this email directly or view it on GitHub:
https://github.com/jclouds/jclouds/pull/394#issuecomment-45146630
Re: [jclouds] JCLOUDS-584 - Update HP Cloud object storage provider
(#394)
Posted by Chris Custine <no...@github.com>.
Added new tests for provideCDNUrl and provideStorageUrl and fixed new checkstyle dings, all pointed out by @demobox. Squashed and rebased on master with Jira issue in comment. Let me know if there is anything else I should fix up.
---
Reply to this email directly or view it on GitHub:
https://github.com/jclouds/jclouds/pull/394#issuecomment-45594315
Re: [jclouds] JCLOUDS-584 - Update HP Cloud object storage provider
(#394)
Posted by CloudBees pull request builder plugin <no...@github.com>.
[jclouds-java-7-pull-requests #1342](https://jclouds.ci.cloudbees.com/job/jclouds-java-7-pull-requests/1342/) SUCCESS
This pull request looks good
---
Reply to this email directly or view it on GitHub:
https://github.com/jclouds/jclouds/pull/394#issuecomment-45598131
Re: [jclouds] JCLOUDS-584 - Update HP Cloud object storage provider
(#394)
Posted by BuildHive <no...@github.com>.
[jclouds » jclouds #1205](https://buildhive.cloudbees.com/job/jclouds/job/jclouds/1205/) UNSTABLE
Looks like there's a problem with this pull request
[(what's this?)](https://www.cloudbees.com/what-is-buildhive)
---
Reply to this email directly or view it on GitHub:
https://github.com/jclouds/jclouds/pull/394#issuecomment-45598165
Re: [jclouds] JCLOUDS-584 - Update HP Cloud object storage provider
(#394)
Posted by CloudBees pull request builder plugin <no...@github.com>.
[jclouds-pull-requests #871](https://jclouds.ci.cloudbees.com/job/jclouds-pull-requests/871/) SUCCESS
This pull request looks good
---
Reply to this email directly or view it on GitHub:
https://github.com/jclouds/jclouds/pull/394#issuecomment-45597472
Re: [jclouds] JCLOUDS-584 - Update HP Cloud object storage provider
(#394)
Posted by Andrew Gaul <no...@github.com>.
> }, userExecutor);
> - return returnVal;
> + }
@ccustine Could you address this comment?
---
Reply to this email directly or view it on GitHub:
https://github.com/jclouds/jclouds/pull/394/files#r14132935
Re: [jclouds] JCLOUDS-584 - Update HP Cloud object storage provider
(#394)
Posted by Andrew Phillips <no...@github.com>.
> @@ -69,12 +69,7 @@ protected HPCloudObjectStorageBlobStore(BlobStoreContext context, BlobUtils blob
>
> @Override
> public boolean createContainerInLocation(Location location, String container, CreateContainerOptions options) {
> - try {
> - return createContainerInLocation(location, container);
> - } finally {
> - if (options.isPublicRead())
> - enableAndCache.apply(container);
> - }
> + return options.isPublicRead() ? enableAndCache.apply(container) != null : createContainerInLocation(location, container);
I guess this answers my above question...`isPublicRead` does _indeed_ mean that `createContainerInLocation` should not be called? Can we add a comment to indicate that that's intentional? Or, perhaps even better, rename `enableAndCache` to indicate that it also creates the container (or does it?)?
---
Reply to this email directly or view it on GitHub:
https://github.com/jclouds/jclouds/pull/394/files#r13473122
Re: [jclouds] JCLOUDS-584 - Update HP Cloud object storage provider
(#394)
Posted by Andrew Phillips <no...@github.com>.
I may have missed something, but I can't see any obvious tests for the new logic in `HPCloudObjectStorageRestClientModule`? Add a unit test for that?
---
Reply to this email directly or view it on GitHub:
https://github.com/jclouds/jclouds/pull/394#issuecomment-45293686
Re: [jclouds] JCLOUDS-584 - Update HP Cloud object storage provider
(#394)
Posted by BuildHive <no...@github.com>.
[jclouds » jclouds #1186](https://buildhive.cloudbees.com/job/jclouds/job/jclouds/1186/) SUCCESS
This pull request looks good
[(what's this?)](https://www.cloudbees.com/what-is-buildhive)
---
Reply to this email directly or view it on GitHub:
https://github.com/jclouds/jclouds/pull/394#issuecomment-45155401
Re: [jclouds] JCLOUDS-584 - Update HP Cloud object storage provider
(#394)
Posted by Shri Javadekar <no...@github.com>.
Hey Chris,
Thanks for adding support for the 13.5 API. Could you list some points (maybe in the commit message itself) about what the actual changes are? Will make it easier to understand the code changes.
---
Reply to this email directly or view it on GitHub:
https://github.com/jclouds/jclouds/pull/394#issuecomment-45181404
Re: [jclouds] JCLOUDS-584 - Update HP Cloud object storage provider
(#394)
Posted by Andrew Gaul <no...@github.com>.
:+1: @jdaggett can you merge this and the backport?
---
Reply to this email directly or view it on GitHub:
https://github.com/jclouds/jclouds/pull/394#issuecomment-47008539
Re: [jclouds] JCLOUDS-584 - Update HP Cloud object storage provider
(#394)
Posted by Jeremy Daggett <no...@github.com>.
Closed #394.
---
Reply to this email directly or view it on GitHub:
https://github.com/jclouds/jclouds/pull/394#event-134704376
Re: [jclouds] JCLOUDS-584 - Update HP Cloud object storage provider
(#394)
Posted by Chris Custine <no...@github.com>.
Just FYI to anyone reviewing this, I believe #392 is a pre-requisite for live tests to pass with this PR.
---
Reply to this email directly or view it on GitHub:
https://github.com/jclouds/jclouds/pull/394#issuecomment-45667503
Re: [jclouds] JCLOUDS-584 - Update HP Cloud object storage provider
(#394)
Posted by Everett Toews <no...@github.com>.
@ccustine Thanks a ton for updating this. It's great to see you here. :)
I see that the pom.xml file didn't get updated to use the dependency for the new Swift API.
<dependency>
<groupId>org.apache.jclouds.labs</groupId>
<artifactId>openstack-swift</artifactId>
<version>${project.parent.version}</version>
</dependency>
Is that an update planned for the future?
---
Reply to this email directly or view it on GitHub:
https://github.com/jclouds/jclouds/pull/394#issuecomment-45554962
Re: [jclouds] JCLOUDS-584 - Update HP Cloud object storage provider
(#394)
Posted by CloudBees pull request builder plugin <no...@github.com>.
[jclouds-java-7-pull-requests #1374](https://jclouds.ci.cloudbees.com/job/jclouds-java-7-pull-requests/1374/) SUCCESS
This pull request looks good
---
Reply to this email directly or view it on GitHub:
https://github.com/jclouds/jclouds/pull/394#issuecomment-46505998
Re: [jclouds] JCLOUDS-584 - Update HP Cloud object storage provider
(#394)
Posted by Andrew Phillips <no...@github.com>.
> jclouds-java-7-pull-requests #1329 SUCCESS
Some new [Checkstyle violations](https://jclouds.ci.cloudbees.com/job/jclouds-java-7-pull-requests/1329/org.apache.jclouds.provider$hpcloud-objectstorage/violations/). Could you have a look at those, @ccustine?
---
Reply to this email directly or view it on GitHub:
https://github.com/jclouds/jclouds/pull/394#issuecomment-45293276
Re: [jclouds] JCLOUDS-584 - Update HP Cloud object storage provider
(#394)
Posted by Andrew Phillips <no...@github.com>.
> + fail("static test Strings do not parse to URI: " + ex.getMessage());
> + }
> +
> + expect(mockSupplier.get())
> + .andReturn(endpoints)
> + .anyTimes();
> + expect(mockFactory.createForApiTypeAndVersion(ServiceType.OBJECT_STORE, null))
> + .andReturn(mockSupplier)
> + .anyTimes();
> + expect(mockCDNFactory.createForApiTypeAndVersion(HPExtensionServiceType.CDN, apiVersion))
> + .andReturn(mockSupplier)
> + .anyTimes();
> +
> + replay(mockSupplier);
> + replay(mockFactory);
> + replay(mockCDNFactory);
[minor] `replay(a, b, c)`
---
Reply to this email directly or view it on GitHub:
https://github.com/jclouds/jclouds/pull/394/files#r14159832
Re: [jclouds] JCLOUDS-584 - Update HP Cloud object storage provider
(#394)
Posted by Chris Custine <no...@github.com>.
@everett-toews Yes, this is a refactoring of the current provider and I am aware that the new provider in labs will be replacing this one at some point. The plan for now was to refactor the existing code so that it will be possible to backport to 1.7.4. Myself and @Phanatic are planning to port to the new provider in openstack-labs (possibly for 1.8 depending on timing) as time permits.
---
Reply to this email directly or view it on GitHub:
https://github.com/jclouds/jclouds/pull/394#issuecomment-45558177
Re: [jclouds] JCLOUDS-584 - Update HP Cloud object storage provider
(#394)
Posted by Andrew Phillips <no...@github.com>.
> }, userExecutor);
> - return returnVal;
> + }
I may be missing something here, but now, if `isPublicRead` is true, `createContainerInLocation` is never called? Is that intentional?
---
Reply to this email directly or view it on GitHub:
https://github.com/jclouds/jclouds/pull/394/files#r13473077
Re: [jclouds] JCLOUDS-584 - Update HP Cloud object storage provider
(#394)
Posted by Jeremy Daggett <no...@github.com>.
+1 Looks good @ccustine Thanks!
---
Reply to this email directly or view it on GitHub:
https://github.com/jclouds/jclouds/pull/394#issuecomment-46979111
Re: [jclouds] JCLOUDS-584 - Update HP Cloud object storage provider
(#394)
Posted by CloudBees pull request builder plugin <no...@github.com>.
[jclouds-pull-requests #858](https://jclouds.ci.cloudbees.com/job/jclouds-pull-requests/858/) FAILURE
Looks like there's a problem with this pull request
---
Reply to this email directly or view it on GitHub:
https://github.com/jclouds/jclouds/pull/394#issuecomment-45142680
Re: [jclouds] JCLOUDS-584 - Update HP Cloud object storage provider
(#394)
Posted by CloudBees pull request builder plugin <no...@github.com>.
[jclouds-pull-requests #903](https://jclouds.ci.cloudbees.com/job/jclouds-pull-requests/903/) SUCCESS
This pull request looks good
---
Reply to this email directly or view it on GitHub:
https://github.com/jclouds/jclouds/pull/394#issuecomment-46505714
Re: [jclouds] JCLOUDS-584 - Update HP Cloud object storage provider
(#394)
Posted by Chris Custine <no...@github.com>.
> @@ -24,4 +25,13 @@
> public HPCloudObjectStorageContainerLiveTest() {
> provider = "hpcloud-objectstorage";
> }
> +
> + @Override
> + @Test
> + public void testPublicAccessInNonDefaultLocation() { throw new SkipException("Locations are ignored"); }
@demobox Whatever you guys prefer... I tend to use SkipException because it nags me on every build and reminds me how many tests I am avoiding proper fixes for :-) I think this particular case will be fixed by the openstack-swift work done by @jdaggett once I can port this provider to it.
---
Reply to this email directly or view it on GitHub:
https://github.com/jclouds/jclouds/pull/394/files#r14164388
Re: [jclouds] JCLOUDS-584 - Update HP Cloud object storage provider
(#394)
Posted by BuildHive <no...@github.com>.
[jclouds » jclouds #1275](https://buildhive.cloudbees.com/job/jclouds/job/jclouds/1275/) SUCCESS
This pull request looks good
[(what's this?)](https://www.cloudbees.com/what-is-buildhive)
---
Reply to this email directly or view it on GitHub:
https://github.com/jclouds/jclouds/pull/394#issuecomment-47013069
Re: [jclouds] JCLOUDS-584 - Update HP Cloud object storage provider
(#394)
Posted by CloudBees pull request builder plugin <no...@github.com>.
[jclouds-pull-requests #927](https://jclouds.ci.cloudbees.com/job/jclouds-pull-requests/927/) SUCCESS
This pull request looks good
---
Reply to this email directly or view it on GitHub:
https://github.com/jclouds/jclouds/pull/394#issuecomment-47011555
Re: [jclouds] JCLOUDS-584 - Update HP Cloud object storage provider
(#394)
Posted by Andrew Phillips <no...@github.com>.
> @@ -36,9 +36,7 @@
> <test.hpcloud-objectstorage.api-version>1.0</test.hpcloud-objectstorage.api-version>
> <test.hpcloud-objectstorage.build-version />
> <!--
> - Identity must be tenantName:userName. Note that the following
> - credentials are the HP Cloud username and password, not the access
> - key id and secret key.
> + Identity must be tenantName:accessKey. Credential is accessKey secret.
Is that "accessKey secret" or "accessKey", which is a secret? I.e. is secret some other value that needs to be provided (I hope that makes some kind of sense)?
Perhaps add quotes, e.g.
Identity must be "tenantName:accessKey". Credential is the accessKey secret.
?
---
Reply to this email directly or view it on GitHub:
https://github.com/jclouds/jclouds/pull/394/files#r13472949
Re: [jclouds] JCLOUDS-584 - Update HP Cloud object storage provider
(#394)
Posted by Chris Custine <no...@github.com>.
@shrinandj I added a comment on the actual Jira issue at https://issues.apache.org/jira/browse/JCLOUDS-584 just pointing out the major changes. Let me know if you have questions about specific parts of the code by adding line comments here in the PR and I will address them.
---
Reply to this email directly or view it on GitHub:
https://github.com/jclouds/jclouds/pull/394#issuecomment-45240277
Re: [jclouds] JCLOUDS-584 - Update HP Cloud object storage provider
(#394)
Posted by Chris Custine <no...@github.com>.
> }, userExecutor);
> - return returnVal;
> + }
@andrewgaul The answer got lost in the outdated diff below... not sure how to avoid this but if anyone has any hints let me know. Click on the "Show outdated diff" below this comment and we covered this question below. I also updated the code and commented it to make it more clear.
---
Reply to this email directly or view it on GitHub:
https://github.com/jclouds/jclouds/pull/394/files#r14144684
Re: [jclouds] JCLOUDS-584 - Update HP Cloud object storage provider
(#394)
Posted by Andrew Phillips <no...@github.com>.
+1, looks good to me, too. Thanks, @ccustine! As @everett-toews said: good to see you around!
---
Reply to this email directly or view it on GitHub:
https://github.com/jclouds/jclouds/pull/394#issuecomment-47039729
Re: [jclouds] JCLOUDS-584 - Update HP Cloud object storage provider
(#394)
Posted by Chris Custine <no...@github.com>.
> @@ -69,12 +69,7 @@ protected HPCloudObjectStorageBlobStore(BlobStoreContext context, BlobUtils blob
>
> @Override
> public boolean createContainerInLocation(Location location, String container, CreateContainerOptions options) {
> - try {
> - return createContainerInLocation(location, container);
> - } finally {
> - if (options.isPublicRead())
> - enableAndCache.apply(container);
> - }
> + return options.isPublicRead() ? enableAndCache.apply(container) != null : createContainerInLocation(location, container);
Yes, a byproduct of enabling CDN on the container (what enableAndCache does) is if the container doesn't exist, it gets created. So it only makes one call instead of two, and it also fixes an issue with resolving the HP Cloud object storage endpoint versions in the catalog where it would create the container in region-a and try to enable the CDN in region-b. I can document this issue in more detail but I have pointed this issue out to HP and hope they fix it.
I will clarify this with a comment and rename the enableAndCache method.
---
Reply to this email directly or view it on GitHub:
https://github.com/jclouds/jclouds/pull/394/files#r13482545