You are viewing a plain text version of this content. The canonical link for it is here.
Posted to notifications@jclouds.apache.org by Chris Custine <no...@github.com> on 2014/06/04 21:52:35 UTC

[jclouds] JCLOUDS-584 - Update HP Cloud object storage provider (#394)

https://issues.apache.org/jira/browse/JCLOUDS-584
You can merge this Pull Request by running:

  git pull https://github.com/ccustine/jclouds fixes/JCLOUDS-584

Or you can view, comment on it, or merge it online at:

  https://github.com/jclouds/jclouds/pull/394

-- Commit Summary --

  * HP Cloud Object Storage Fixes for 13.5

-- File Changes --

    M blobstore/src/test/java/org/jclouds/blobstore/integration/internal/BaseContainerLiveTest.java (8)
    M providers/hpcloud-objectstorage/pom.xml (4)
    M providers/hpcloud-objectstorage/src/main/java/org/jclouds/hpcloud/objectstorage/HPCloudObjectStorageApiMetadata.java (28)
    M providers/hpcloud-objectstorage/src/main/java/org/jclouds/hpcloud/objectstorage/HPCloudObjectStorageProviderMetadata.java (9)
    M providers/hpcloud-objectstorage/src/main/java/org/jclouds/hpcloud/objectstorage/blobstore/HPCloudObjectStorageAsyncBlobStore.java (22)
    M providers/hpcloud-objectstorage/src/main/java/org/jclouds/hpcloud/objectstorage/blobstore/HPCloudObjectStorageBlobStore.java (7)
    M providers/hpcloud-objectstorage/src/main/java/org/jclouds/hpcloud/objectstorage/config/HPCloudObjectStorageRestClientModule.java (39)
    M providers/hpcloud-objectstorage/src/main/java/org/jclouds/hpcloud/objectstorage/domain/CDNContainer.java (4)
    M providers/hpcloud-objectstorage/src/test/java/org/jclouds/hpcloud/objectstorage/blobstore/HPCloudObjectStorageBlobSignerExpectTest.java (26)
    M providers/hpcloud-objectstorage/src/test/java/org/jclouds/hpcloud/objectstorage/blobstore/HPCloudObjectStorageBlobStoreExpectTest.java (2)
    M providers/hpcloud-objectstorage/src/test/java/org/jclouds/hpcloud/objectstorage/blobstore/integration/HPCloudObjectStorageBlobIntegrationLiveTest.java (15)
    M providers/hpcloud-objectstorage/src/test/java/org/jclouds/hpcloud/objectstorage/blobstore/integration/HPCloudObjectStorageContainerLiveTest.java (10)
    M providers/hpcloud-objectstorage/src/test/java/org/jclouds/hpcloud/objectstorage/blobstore/integration/HPCloudObjectStorageServiceIntegrationLiveTest.java (7)
    M providers/hpcloud-objectstorage/src/test/resources/keystoneAuthResponseNoCDN.json (381)
    M providers/hpcloud-objectstorage/src/test/resources/keystoneAuthResponseWithCDN.json (411)
    M providers/hpcloud-objectstorage/src/test/resources/test_list_cdn.json (6)

-- Patch Links --

https://github.com/jclouds/jclouds/pull/394.patch
https://github.com/jclouds/jclouds/pull/394.diff

---
Reply to this email directly or view it on GitHub:
https://github.com/jclouds/jclouds/pull/394

Re: [jclouds] JCLOUDS-584 - Update HP Cloud object storage provider (#394)

Posted by CloudBees pull request builder plugin <no...@github.com>.
[jclouds-java-7-pull-requests #1398](https://jclouds.ci.cloudbees.com/job/jclouds-java-7-pull-requests/1398/) SUCCESS
This pull request looks good

---
Reply to this email directly or view it on GitHub:
https://github.com/jclouds/jclouds/pull/394#issuecomment-47012238

Re: [jclouds] JCLOUDS-584 - Update HP Cloud object storage provider (#394)

Posted by Andrew Phillips <no...@github.com>.
> @@ -69,12 +69,7 @@ protected HPCloudObjectStorageBlobStore(BlobStoreContext context, BlobUtils blob
>  
>     @Override
>     public boolean createContainerInLocation(Location location, String container, CreateContainerOptions options) {
> -      try {
> -         return createContainerInLocation(location, container);
> -      } finally {
> -         if (options.isPublicRead())
> -            enableAndCache.apply(container);
> -      }
> +      return options.isPublicRead() ? enableAndCache.apply(container) != null : createContainerInLocation(location, container);

> I will clarify this with a comment and rename the enableAndCache method.

Thanks for explaining, Chris! Given that, I think a short comment "enabling CDN on the container also creates it if it does not exist" would be fine for me.

---
Reply to this email directly or view it on GitHub:
https://github.com/jclouds/jclouds/pull/394/files#r13494428

Re: [jclouds] JCLOUDS-584 - Update HP Cloud object storage provider (#394)

Posted by BuildHive <no...@github.com>.
[jclouds » jclouds #1243](https://buildhive.cloudbees.com/job/jclouds/job/jclouds/1243/) SUCCESS
This pull request looks good
[(what's this?)](https://www.cloudbees.com/what-is-buildhive)

---
Reply to this email directly or view it on GitHub:
https://github.com/jclouds/jclouds/pull/394#issuecomment-46510415

Re: [jclouds] JCLOUDS-584 - Update HP Cloud object storage provider (#394)

Posted by Andrew Phillips <no...@github.com>.
> +   public static class HPCloudObjectStorageEndpointModule extends KeystoneAuthenticationModule {
> +      @Provides
> +      @Singleton
> +      @Storage
> +      protected Supplier<URI> provideStorageUrl(RegionIdToURISupplier.Factory factory,
> +                                                @ApiVersion String apiVersion,
> +                                                @Named(LocationConstants.PROPERTY_REGION) String region) {
> +
> +         Supplier<Map<String, Supplier<URI>>> endpointsSupplier = factory.createForApiTypeAndVersion(ServiceType.OBJECT_STORE, null);
> +
> +         if (region.isEmpty()) {
> +            return getLastValueInMap(endpointsSupplier);
> +         } else {
> +            return getValueInMapOrNull(endpointsSupplier, region);
> +         }
> +      }

Duplicate of the code above? Any way to factor this out? And will this ever return `null` (from the `getValueInMapOrNull` call)? If so, mark the return value as `@Nullable`?

---
Reply to this email directly or view it on GitHub:
https://github.com/jclouds/jclouds/pull/394/files#r13473150

Re: [jclouds] JCLOUDS-584 - Update HP Cloud object storage provider (#394)

Posted by Chris Custine <no...@github.com>.
>           }, userExecutor);
> -      return returnVal;
> +      }

Copied from previous comment:

"Yes, a byproduct of enabling CDN on the container (what enableAndCache does) is if the container doesn't exist, it gets created. So it only makes one call instead of two, and it also fixes an issue with resolving the HP Cloud object storage endpoint versions in the catalog where it would create the container in region-a and try to enable the CDN in region-b. I can document this issue in more detail but I have pointed this issue out to HP and hope they fix it.

I will clarify this with a comment and rename the enableAndCache method."

---
Reply to this email directly or view it on GitHub:
https://github.com/jclouds/jclouds/pull/394/files#r14144761

Re: [jclouds] JCLOUDS-584 - Update HP Cloud object storage provider (#394)

Posted by Andrew Phillips <no...@github.com>.
> @@ -24,4 +25,13 @@
>     public HPCloudObjectStorageContainerLiveTest() {
>        provider = "hpcloud-objectstorage";
>     }
> +
> +   @Override
> +   @Test
> +   public void testPublicAccessInNonDefaultLocation() { throw new SkipException("Locations are ignored"); }

[minor] Do we prefer `SkipException` or `@Test(enabled = false)`?

---
Reply to this email directly or view it on GitHub:
https://github.com/jclouds/jclouds/pull/394/files#r14159808

Re: [jclouds] JCLOUDS-584 - Update HP Cloud object storage provider (#394)

Posted by Andrew Phillips <no...@github.com>.
> @@ -72,6 +76,10 @@ public boolean apply(@Nullable StorageMetadata input) {
>  
>           assert metadata.getPublicUri() != null : metadata;
>  
> +         SocketOpen socketOpen = context.utils().injector().getInstance(SocketOpen.class);
> +         Predicate<HostAndPort> socketTester = retry(socketOpen, 1200, 10, SECONDS);
> +         assert socketTester.apply(HostAndPort.fromHost(metadata.getPublicUri().getHost()).withDefaultPort(80)) : metadata.getPublicUri();

Use `assertTrue`? I know we still have `assert`s here and there in the codebase, but in general I think we're trying to use the TestNG assertions in tests now. Thanks!

---
Reply to this email directly or view it on GitHub:
https://github.com/jclouds/jclouds/pull/394/files#r13472909

Re: [jclouds] JCLOUDS-584 - Update HP Cloud object storage provider (#394)

Posted by CloudBees pull request builder plugin <no...@github.com>.
[jclouds-java-7-pull-requests #1329](https://jclouds.ci.cloudbees.com/job/jclouds-java-7-pull-requests/1329/) SUCCESS
This pull request looks good

---
Reply to this email directly or view it on GitHub:
https://github.com/jclouds/jclouds/pull/394#issuecomment-45146630

Re: [jclouds] JCLOUDS-584 - Update HP Cloud object storage provider (#394)

Posted by Chris Custine <no...@github.com>.
Added new tests for provideCDNUrl and provideStorageUrl and fixed new checkstyle dings, all pointed out by @demobox.  Squashed and rebased on master with Jira issue in comment. Let me know if there is anything else I should fix up.

---
Reply to this email directly or view it on GitHub:
https://github.com/jclouds/jclouds/pull/394#issuecomment-45594315

Re: [jclouds] JCLOUDS-584 - Update HP Cloud object storage provider (#394)

Posted by CloudBees pull request builder plugin <no...@github.com>.
[jclouds-java-7-pull-requests #1342](https://jclouds.ci.cloudbees.com/job/jclouds-java-7-pull-requests/1342/) SUCCESS
This pull request looks good

---
Reply to this email directly or view it on GitHub:
https://github.com/jclouds/jclouds/pull/394#issuecomment-45598131

Re: [jclouds] JCLOUDS-584 - Update HP Cloud object storage provider (#394)

Posted by BuildHive <no...@github.com>.
[jclouds » jclouds #1205](https://buildhive.cloudbees.com/job/jclouds/job/jclouds/1205/) UNSTABLE
Looks like there's a problem with this pull request
[(what's this?)](https://www.cloudbees.com/what-is-buildhive)

---
Reply to this email directly or view it on GitHub:
https://github.com/jclouds/jclouds/pull/394#issuecomment-45598165

Re: [jclouds] JCLOUDS-584 - Update HP Cloud object storage provider (#394)

Posted by CloudBees pull request builder plugin <no...@github.com>.
[jclouds-pull-requests #871](https://jclouds.ci.cloudbees.com/job/jclouds-pull-requests/871/) SUCCESS
This pull request looks good

---
Reply to this email directly or view it on GitHub:
https://github.com/jclouds/jclouds/pull/394#issuecomment-45597472

Re: [jclouds] JCLOUDS-584 - Update HP Cloud object storage provider (#394)

Posted by Andrew Gaul <no...@github.com>.
>           }, userExecutor);
> -      return returnVal;
> +      }

@ccustine Could you address this comment?

---
Reply to this email directly or view it on GitHub:
https://github.com/jclouds/jclouds/pull/394/files#r14132935

Re: [jclouds] JCLOUDS-584 - Update HP Cloud object storage provider (#394)

Posted by Andrew Phillips <no...@github.com>.
> @@ -69,12 +69,7 @@ protected HPCloudObjectStorageBlobStore(BlobStoreContext context, BlobUtils blob
>  
>     @Override
>     public boolean createContainerInLocation(Location location, String container, CreateContainerOptions options) {
> -      try {
> -         return createContainerInLocation(location, container);
> -      } finally {
> -         if (options.isPublicRead())
> -            enableAndCache.apply(container);
> -      }
> +      return options.isPublicRead() ? enableAndCache.apply(container) != null : createContainerInLocation(location, container);

I guess this answers my above question...`isPublicRead` does _indeed_ mean that `createContainerInLocation` should not be called? Can we add a comment to indicate that that's intentional? Or, perhaps even better, rename `enableAndCache` to indicate that it also creates the container (or does it?)?

---
Reply to this email directly or view it on GitHub:
https://github.com/jclouds/jclouds/pull/394/files#r13473122

Re: [jclouds] JCLOUDS-584 - Update HP Cloud object storage provider (#394)

Posted by Andrew Phillips <no...@github.com>.
I may have missed something, but I can't see any obvious tests for the new logic in `HPCloudObjectStorageRestClientModule`? Add a unit test for that?

---
Reply to this email directly or view it on GitHub:
https://github.com/jclouds/jclouds/pull/394#issuecomment-45293686

Re: [jclouds] JCLOUDS-584 - Update HP Cloud object storage provider (#394)

Posted by BuildHive <no...@github.com>.
[jclouds » jclouds #1186](https://buildhive.cloudbees.com/job/jclouds/job/jclouds/1186/) SUCCESS
This pull request looks good
[(what's this?)](https://www.cloudbees.com/what-is-buildhive)

---
Reply to this email directly or view it on GitHub:
https://github.com/jclouds/jclouds/pull/394#issuecomment-45155401

Re: [jclouds] JCLOUDS-584 - Update HP Cloud object storage provider (#394)

Posted by Shri Javadekar <no...@github.com>.
Hey Chris,

Thanks for adding support for the 13.5 API. Could you list some points (maybe in the commit message itself) about what the actual changes are? Will make it easier to understand the code changes.

---
Reply to this email directly or view it on GitHub:
https://github.com/jclouds/jclouds/pull/394#issuecomment-45181404

Re: [jclouds] JCLOUDS-584 - Update HP Cloud object storage provider (#394)

Posted by Andrew Gaul <no...@github.com>.
:+1: @jdaggett can you merge this and the backport?

---
Reply to this email directly or view it on GitHub:
https://github.com/jclouds/jclouds/pull/394#issuecomment-47008539

Re: [jclouds] JCLOUDS-584 - Update HP Cloud object storage provider (#394)

Posted by Jeremy Daggett <no...@github.com>.
Closed #394.

---
Reply to this email directly or view it on GitHub:
https://github.com/jclouds/jclouds/pull/394#event-134704376

Re: [jclouds] JCLOUDS-584 - Update HP Cloud object storage provider (#394)

Posted by Chris Custine <no...@github.com>.
Just FYI to anyone reviewing this, I believe #392 is a pre-requisite for live tests to pass with this PR.  

---
Reply to this email directly or view it on GitHub:
https://github.com/jclouds/jclouds/pull/394#issuecomment-45667503

Re: [jclouds] JCLOUDS-584 - Update HP Cloud object storage provider (#394)

Posted by Everett Toews <no...@github.com>.
@ccustine Thanks a ton for updating this. It's great to see you here. :)

I see that the pom.xml file didn't get updated to use the dependency for the new Swift API. 

    <dependency>
      <groupId>org.apache.jclouds.labs</groupId>
      <artifactId>openstack-swift</artifactId>
      <version>${project.parent.version}</version>
    </dependency>

Is that an update planned for the future?

---
Reply to this email directly or view it on GitHub:
https://github.com/jclouds/jclouds/pull/394#issuecomment-45554962

Re: [jclouds] JCLOUDS-584 - Update HP Cloud object storage provider (#394)

Posted by CloudBees pull request builder plugin <no...@github.com>.
[jclouds-java-7-pull-requests #1374](https://jclouds.ci.cloudbees.com/job/jclouds-java-7-pull-requests/1374/) SUCCESS
This pull request looks good

---
Reply to this email directly or view it on GitHub:
https://github.com/jclouds/jclouds/pull/394#issuecomment-46505998

Re: [jclouds] JCLOUDS-584 - Update HP Cloud object storage provider (#394)

Posted by Andrew Phillips <no...@github.com>.
> jclouds-java-7-pull-requests #1329 SUCCESS

Some new [Checkstyle violations](https://jclouds.ci.cloudbees.com/job/jclouds-java-7-pull-requests/1329/org.apache.jclouds.provider$hpcloud-objectstorage/violations/). Could you have a look at those, @ccustine?

---
Reply to this email directly or view it on GitHub:
https://github.com/jclouds/jclouds/pull/394#issuecomment-45293276

Re: [jclouds] JCLOUDS-584 - Update HP Cloud object storage provider (#394)

Posted by Andrew Phillips <no...@github.com>.
> +         fail("static test Strings do not parse to URI: " + ex.getMessage());
> +      }
> +
> +      expect(mockSupplier.get())
> +         .andReturn(endpoints)
> +         .anyTimes();
> +      expect(mockFactory.createForApiTypeAndVersion(ServiceType.OBJECT_STORE, null))
> +         .andReturn(mockSupplier)
> +         .anyTimes();
> +      expect(mockCDNFactory.createForApiTypeAndVersion(HPExtensionServiceType.CDN, apiVersion))
> +         .andReturn(mockSupplier)
> +         .anyTimes();
> +
> +      replay(mockSupplier);
> +      replay(mockFactory);
> +      replay(mockCDNFactory);

[minor] `replay(a, b, c)`

---
Reply to this email directly or view it on GitHub:
https://github.com/jclouds/jclouds/pull/394/files#r14159832

Re: [jclouds] JCLOUDS-584 - Update HP Cloud object storage provider (#394)

Posted by Chris Custine <no...@github.com>.
@everett-toews Yes, this is a refactoring of the current provider and I am aware that the new provider in labs will be replacing this one at some point.  The plan for now was to refactor the existing code so that it will be possible to backport to 1.7.4.  Myself and @Phanatic are planning to port to the new provider in openstack-labs (possibly for 1.8 depending on timing) as time permits.

---
Reply to this email directly or view it on GitHub:
https://github.com/jclouds/jclouds/pull/394#issuecomment-45558177

Re: [jclouds] JCLOUDS-584 - Update HP Cloud object storage provider (#394)

Posted by Andrew Phillips <no...@github.com>.
>           }, userExecutor);
> -      return returnVal;
> +      }

I may be missing something here, but now, if `isPublicRead` is true, `createContainerInLocation` is never called? Is that intentional?

---
Reply to this email directly or view it on GitHub:
https://github.com/jclouds/jclouds/pull/394/files#r13473077

Re: [jclouds] JCLOUDS-584 - Update HP Cloud object storage provider (#394)

Posted by Jeremy Daggett <no...@github.com>.
+1 Looks good @ccustine Thanks!

---
Reply to this email directly or view it on GitHub:
https://github.com/jclouds/jclouds/pull/394#issuecomment-46979111

Re: [jclouds] JCLOUDS-584 - Update HP Cloud object storage provider (#394)

Posted by CloudBees pull request builder plugin <no...@github.com>.
[jclouds-pull-requests #858](https://jclouds.ci.cloudbees.com/job/jclouds-pull-requests/858/) FAILURE
Looks like there's a problem with this pull request

---
Reply to this email directly or view it on GitHub:
https://github.com/jclouds/jclouds/pull/394#issuecomment-45142680

Re: [jclouds] JCLOUDS-584 - Update HP Cloud object storage provider (#394)

Posted by CloudBees pull request builder plugin <no...@github.com>.
[jclouds-pull-requests #903](https://jclouds.ci.cloudbees.com/job/jclouds-pull-requests/903/) SUCCESS
This pull request looks good

---
Reply to this email directly or view it on GitHub:
https://github.com/jclouds/jclouds/pull/394#issuecomment-46505714

Re: [jclouds] JCLOUDS-584 - Update HP Cloud object storage provider (#394)

Posted by Chris Custine <no...@github.com>.
> @@ -24,4 +25,13 @@
>     public HPCloudObjectStorageContainerLiveTest() {
>        provider = "hpcloud-objectstorage";
>     }
> +
> +   @Override
> +   @Test
> +   public void testPublicAccessInNonDefaultLocation() { throw new SkipException("Locations are ignored"); }

@demobox Whatever you guys prefer...  I tend to use SkipException because it nags me on every build and reminds me how many tests I am avoiding proper fixes for :-)  I think this particular case will be fixed by the openstack-swift work done by @jdaggett once I can port this provider to it.

---
Reply to this email directly or view it on GitHub:
https://github.com/jclouds/jclouds/pull/394/files#r14164388

Re: [jclouds] JCLOUDS-584 - Update HP Cloud object storage provider (#394)

Posted by BuildHive <no...@github.com>.
[jclouds » jclouds #1275](https://buildhive.cloudbees.com/job/jclouds/job/jclouds/1275/) SUCCESS
This pull request looks good
[(what's this?)](https://www.cloudbees.com/what-is-buildhive)

---
Reply to this email directly or view it on GitHub:
https://github.com/jclouds/jclouds/pull/394#issuecomment-47013069

Re: [jclouds] JCLOUDS-584 - Update HP Cloud object storage provider (#394)

Posted by CloudBees pull request builder plugin <no...@github.com>.
[jclouds-pull-requests #927](https://jclouds.ci.cloudbees.com/job/jclouds-pull-requests/927/) SUCCESS
This pull request looks good

---
Reply to this email directly or view it on GitHub:
https://github.com/jclouds/jclouds/pull/394#issuecomment-47011555

Re: [jclouds] JCLOUDS-584 - Update HP Cloud object storage provider (#394)

Posted by Andrew Phillips <no...@github.com>.
> @@ -36,9 +36,7 @@
>      <test.hpcloud-objectstorage.api-version>1.0</test.hpcloud-objectstorage.api-version>
>      <test.hpcloud-objectstorage.build-version />
>      <!--
> -        Identity must be tenantName:userName.  Note that the following
> -        credentials are the HP Cloud username and password, not the access
> -        key id and secret key.
> +        Identity must be tenantName:accessKey.  Credential is accessKey secret.

Is that "accessKey secret" or "accessKey", which is a secret? I.e. is secret some other value that needs to be provided (I hope that makes some kind of sense)?

Perhaps add quotes, e.g.

Identity must be "tenantName:accessKey". Credential is the accessKey secret.

?

---
Reply to this email directly or view it on GitHub:
https://github.com/jclouds/jclouds/pull/394/files#r13472949

Re: [jclouds] JCLOUDS-584 - Update HP Cloud object storage provider (#394)

Posted by Chris Custine <no...@github.com>.
@shrinandj I added a comment on the actual Jira issue at https://issues.apache.org/jira/browse/JCLOUDS-584 just pointing out the major changes. Let me know if you have questions about specific parts of the code by adding line comments here in the PR and I will address them.

---
Reply to this email directly or view it on GitHub:
https://github.com/jclouds/jclouds/pull/394#issuecomment-45240277

Re: [jclouds] JCLOUDS-584 - Update HP Cloud object storage provider (#394)

Posted by Chris Custine <no...@github.com>.
>           }, userExecutor);
> -      return returnVal;
> +      }

@andrewgaul The answer got lost in the outdated diff below...  not sure how to avoid this but if anyone has any hints let me know.  Click on the "Show outdated diff" below this comment and we covered this question below.  I also updated the code and commented it to make it more clear.

---
Reply to this email directly or view it on GitHub:
https://github.com/jclouds/jclouds/pull/394/files#r14144684

Re: [jclouds] JCLOUDS-584 - Update HP Cloud object storage provider (#394)

Posted by Andrew Phillips <no...@github.com>.
+1, looks good to me, too. Thanks, @ccustine! As @everett-toews said: good to see you around!

---
Reply to this email directly or view it on GitHub:
https://github.com/jclouds/jclouds/pull/394#issuecomment-47039729

Re: [jclouds] JCLOUDS-584 - Update HP Cloud object storage provider (#394)

Posted by Chris Custine <no...@github.com>.
> @@ -69,12 +69,7 @@ protected HPCloudObjectStorageBlobStore(BlobStoreContext context, BlobUtils blob
>  
>     @Override
>     public boolean createContainerInLocation(Location location, String container, CreateContainerOptions options) {
> -      try {
> -         return createContainerInLocation(location, container);
> -      } finally {
> -         if (options.isPublicRead())
> -            enableAndCache.apply(container);
> -      }
> +      return options.isPublicRead() ? enableAndCache.apply(container) != null : createContainerInLocation(location, container);

Yes, a byproduct of enabling CDN on the container (what enableAndCache does) is if the container doesn't exist, it gets created.  So it only makes one call instead of two, and it also fixes an issue with resolving the HP Cloud object storage endpoint versions in the catalog where it would create the container in region-a and try to enable the CDN in region-b.  I can document this issue in more detail but I have pointed this issue out to HP and hope they fix it.

I will clarify this with a comment and rename the enableAndCache method.

---
Reply to this email directly or view it on GitHub:
https://github.com/jclouds/jclouds/pull/394/files#r13482545