You are viewing a plain text version of this content. The canonical link for it is here.
Posted to notifications@jclouds.apache.org by Gordon Jones <no...@github.com> on 2017/03/24 15:32:28 UTC

[jclouds/jclouds-labs] Adding Dimension Data Infrastructure API (#377)

The next pull request from DD, this one adds the infrastructure API
You can view, comment on, or merge this pull request online at:

  https://github.com/jclouds/jclouds-labs/pull/377

-- Commit Summary --

  * Adding Dimension Data Infrastructure API

-- File Changes --

    M dimensiondata/src/main/java/org/jclouds/dimensiondata/cloudcontrol/DimensionDataCloudControlApi.java (4)
    A dimensiondata/src/main/java/org/jclouds/dimensiondata/cloudcontrol/features/InfrastructureApi.java (79)
    A dimensiondata/src/main/java/org/jclouds/dimensiondata/cloudcontrol/filters/DatacenterIdListDatacentersFilter.java (45)
    A dimensiondata/src/main/java/org/jclouds/dimensiondata/cloudcontrol/parsers/ParseDatacenters.java (62)
    A dimensiondata/src/main/java/org/jclouds/dimensiondata/cloudcontrol/parsers/ParseOperatingSystems.java (63)
    M dimensiondata/src/test/java/org/jclouds/dimensiondata/cloudcontrol/features/AccountApiMockTest.java (5)
    A dimensiondata/src/test/java/org/jclouds/dimensiondata/cloudcontrol/features/InfrastructureApiLiveTest.java (53)
    A dimensiondata/src/test/java/org/jclouds/dimensiondata/cloudcontrol/features/InfrastructureApiMockTest.java (114)
    A dimensiondata/src/test/java/org/jclouds/dimensiondata/cloudcontrol/internal/BaseAccountAwareCloudControlMockTest.java (38)
    M dimensiondata/src/test/java/org/jclouds/dimensiondata/cloudcontrol/internal/BaseDimensionDataCloudControlMockTest.java (11)
    A dimensiondata/src/test/java/org/jclouds/dimensiondata/cloudcontrol/options/PaginationOptionsTest.java (84)
    A dimensiondata/src/test/resources/datacenters.json (247)
    A dimensiondata/src/test/resources/operatingSystems.json (173)

-- Patch Links --

https://github.com/jclouds/jclouds-labs/pull/377.patch
https://github.com/jclouds/jclouds-labs/pull/377.diff

-- 
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/jclouds/jclouds-labs/pull/377

Re: [jclouds/jclouds-labs] Adding Dimension Data Infrastructure API (#377)

Posted by Gordon Jones <no...@github.com>.
gatj98 commented on this pull request.



> +
+      assertEquals(250, operatingSystems.getPageSize());
+      assertEquals(33, operatingSystems.getTotalCount());
+      assertEquals(33, operatingSystems.getPageCount());
+      assertEquals(1, operatingSystems.getPageNumber());
+
+      Uris.UriBuilder uriBuilder = Uris
+            .uriBuilder("/caas/2.4/6ac1e746-b1ea-4da5-a24e-caf1a978789d/infrastructure/operatingSystem");
+      uriBuilder.addQuery("datacenterId", "NA9");
+      uriBuilder.addQuery(paginationOptions.buildQueryParameters());
+      assertSent(HttpMethod.GET, uriBuilder.toString());
+   }
+
+   public void testListOperatingSystems404() {
+      server.enqueue(response404());
+      assertTrue(api.getInfrastructureApi().listOperatingSystems("NA9").concat().isEmpty());

:+1:

-- 
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/jclouds/jclouds-labs/pull/377#discussion_r108030548

Re: [jclouds/jclouds-labs] Adding Dimension Data Infrastructure API (#377)

Posted by Gordon Jones <no...@github.com>.
rebuild please

-- 
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/jclouds/jclouds-labs/pull/377#issuecomment-289057282

Re: [jclouds/jclouds-labs] Adding Dimension Data Infrastructure API (#377)

Posted by Gordon Jones <no...@github.com>.
gatj98 commented on this pull request.



> @@ -52,12 +51,12 @@ public void testListDatacentersWithPagination() throws Exception {
       server.enqueue(jsonResponse("/datacenters-page2.json"));
       Iterable<Datacenter> datacenters = api.getInfrastructureApi().listDatacenters().concat();
 
-      assertEquals(server.getRequestCount(), 2);
-      assertEquals(size(datacenters), 2);
+      assertEquals(server.getRequestCount(), 2); // 1 for account, 1 for page 1
+      assertEquals(size(datacenters), 2); // Causes all pages to be requested
       assertEquals(server.getRequestCount(), 3); // 1 for account details, 2 pages

Am not 100% happy with this as I feel it needs code comments to explain what the test is doing.

Maybe a helper method of the form "consumeIterableAndAssertPagesRequested(Iterable iterable, int expectedExtraPagesRequested)" would be better?



-- 
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/jclouds/jclouds-labs/pull/377#pullrequestreview-29177096

Re: [jclouds/jclouds-labs] Adding Dimension Data Infrastructure API (#377)

Posted by Gordon Jones <no...@github.com>.
gatj98 commented on this pull request.



> + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.jclouds.dimensiondata.cloudcontrol.options;
+
+import com.google.common.collect.Multimap;
+import org.testng.annotations.BeforeMethod;
+import org.testng.annotations.Test;
+
+import java.util.Collection;
+
+import static org.testng.Assert.assertEquals;
+import static org.testng.Assert.assertTrue;
+
+@Test(groups = "unit", testName = "PaginationOptionsTest", singleThreaded = true)

Initially it wasn't single threaded, but push request build failed for one of the tests at line 81 where the size of the value collection is checked. I figured this was because the threads were accessing the paginationOptions field at the same time. 

Given it's such a quick test didn't really see the benefit of multi threading so I made it single threaded rather than putting 'PaginationOptions paginationOptions = new PaginationOptions();' into each test method and ditching the instance fields. Will change it though

-- 
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/jclouds/jclouds-labs/pull/377#discussion_r108030417

Re: [jclouds/jclouds-labs] Adding Dimension Data Infrastructure API (#377)

Posted by Ignasi Barrera <no...@github.com>.
nacx commented on this pull request.



> +
+import com.google.common.base.Supplier;
+import org.jclouds.http.HttpException;
+import org.jclouds.http.HttpRequest;
+import org.jclouds.http.HttpRequestFilter;
+import org.jclouds.location.Zone;
+
+import javax.inject.Inject;
+import java.util.Set;
+
+/**
+ * Adds set of Datacenter IDs as set in jclouds.zones JVM property.
+ */
+public class DatacenterIdListDatacentersFilter implements HttpRequestFilter {
+   @Inject
+   @Zone

Makes sense. In that case, take into account that the default Region/Zone suppliers (those bound with the `@Region` and `@Zone` qualifiers) take the values from configuration properties (as in [ProfitBricks](https://github.com/jclouds/jclouds/blob/master/providers/profitbricks/src/main/java/org/jclouds/profitbricks/ProfitBricksProviderMetadata.java#L64-L74).
In this case, I think it would make sense to configure all the regions via properties (including their endpoints with the `jclouds.region.<regionname>.endpoint` property) and then bind specific Zone suppliers to get them using the API, as done in [EC2](https://github.com/jclouds/jclouds/blob/master/apis/ec2/src/main/java/org/jclouds/ec2/config/BaseEC2HttpApiModule.java#L53-L57).
WDYT?

-- 
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/jclouds/jclouds-labs/pull/377#discussion_r108097826

Re: [jclouds/jclouds-labs] Adding Dimension Data Infrastructure API (#377)

Posted by Gordon Jones <no...@github.com>.
gatj98 commented on this pull request.



> +
+import com.google.common.base.Supplier;
+import org.jclouds.http.HttpException;
+import org.jclouds.http.HttpRequest;
+import org.jclouds.http.HttpRequestFilter;
+import org.jclouds.location.Zone;
+
+import javax.inject.Inject;
+import java.util.Set;
+
+/**
+ * Adds set of Datacenter IDs as set in jclouds.zones JVM property.
+ */
+public class DatacenterIdListDatacentersFilter implements HttpRequestFilter {
+   @Inject
+   @Zone

Good Q - we were having a discussion about this ourselves the other day. Your input would be appreciated @nacx!

DD has a number of 'Geos', each has it's own cloud control API endpoint, and consists of 1 or more datacenters. At present we see our provider as being for a single Geo (as it's the simplest thing to do to start with), but maybe we should aim to have a single provider to cover them all in time. There is no global API endpoint, so when creating a compute service instance we'd have to call out to multiple Geos to find out where the user is based (their home Geo) and which 'shaddow' Geos they also have access to. We do have an API that can be called on any Geo to find a users home Geo that does this fan-out, but I'm not sure if it's publicly published or not.

Anyway, to answer your Q I think the mapping should be:
Region -> Geo (only 1 for now - we prob need to do something to model this?)
Zone -> Datacenter

Does that sound ok?

-- 
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/jclouds/jclouds-labs/pull/377#discussion_r108029942

Re: [jclouds/jclouds-labs] Adding Dimension Data Infrastructure API (#377)

Posted by Ignasi Barrera <no...@github.com>.
nacx approved this pull request.





-- 
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/jclouds/jclouds-labs/pull/377#pullrequestreview-29195619

Re: [jclouds/jclouds-labs] Adding Dimension Data Infrastructure API (#377)

Posted by Ignasi Barrera <no...@github.com>.
Closed #377.

-- 
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/jclouds/jclouds-labs/pull/377#event-1016756171

Re: [jclouds/jclouds-labs] Adding Dimension Data Infrastructure API (#377)

Posted by Gordon Jones <no...@github.com>.
Hi @nacx 
Thanks for your comments - have reworked the pagination tests in InfrastructureApiMockTest, and saw it fail because of the bug you spotted with the optional.

Have added an IHookable in BaseDimensionDataCloudControlMockTest that checks after each test runs that we've assertted all of the requests received by the mock web server - hopefully this will help us write better tests going forward.

-- 
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/jclouds/jclouds-labs/pull/377#issuecomment-289446793

Re: [jclouds/jclouds-labs] Adding Dimension Data Infrastructure API (#377)

Posted by Gordon Jones <no...@github.com>.
@gatj98 pushed 2 commits.

b88dab1  Fixed bug in InfrastructureApi pagination, and mock tests improved
1db79ae  Implemented IHookable to ensure that all requests received by mock web server have been asserted.


-- 
You are receiving this because you are subscribed to this thread.
View it on GitHub:
https://github.com/jclouds/jclouds-labs/pull/377/files/2b451dde38e261232692e1fa45dc1f9485322258..1db79ae7733e79128f5a3dc202c7558b5eb12161

Re: [jclouds/jclouds-labs] Adding Dimension Data Infrastructure API (#377)

Posted by Gordon Jones <no...@github.com>.
gatj98 commented on this pull request.



> +/**
+ * Mock tests for the {@link InfrastructureApi} class.
+ */
+@Test(groups = "unit", testName = "InfrastructureApiMockTest", singleThreaded = true)
+public class InfrastructureApiMockTest extends BaseAccountAwareCloudControlMockTest {
+
+   public void testListDatacenters() throws Exception {
+      server.enqueue(jsonResponse("/datacenters.json"));
+      Iterable<Datacenter> datacenters = api.getInfrastructureApi().listDatacenters().concat();
+
+      assertEquals(size(datacenters), 2); // Force the PagedIterable to advance
+      assertEquals(server.getRequestCount(), 2);
+
+      Uris.UriBuilder uriBuilder = Uris
+            .uriBuilder("/caas/2.4/6ac1e746-b1ea-4da5-a24e-caf1a978789d/infrastructure/datacenter");
+      Set<String> zones = ctx.utils().injector().getInstance(ZoneIdsSupplier.class).get();

Will look at this in conjunction with the region/zone config discussed above


-- 
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/jclouds/jclouds-labs/pull/377#discussion_r108117990

Re: [jclouds/jclouds-labs] Adding Dimension Data Infrastructure API (#377)

Posted by Gordon Jones <no...@github.com>.
@gatj98 pushed 1 commit.

2b451dd  Pull request changes, minus region/zone stuff


-- 
You are receiving this because you are subscribed to this thread.
View it on GitHub:
https://github.com/jclouds/jclouds-labs/pull/377/files/2861dd3efc90ca9ac433ca017374ae5adb86564f..2b451dde38e261232692e1fa45dc1f9485322258

Re: [jclouds/jclouds-labs] Adding Dimension Data Infrastructure API (#377)

Posted by Gordon Jones <no...@github.com>.
gatj98 commented on this pull request.



> +import org.jclouds.dimensiondata.cloudcontrol.DimensionDataCloudControlApi;
+import org.jclouds.dimensiondata.cloudcontrol.domain.Datacenter;
+import org.jclouds.dimensiondata.cloudcontrol.domain.Datacenters;
+import org.jclouds.dimensiondata.cloudcontrol.options.PaginationOptions;
+import org.jclouds.http.functions.ParseJson;
+import org.jclouds.json.Json;
+
+import javax.inject.Inject;
+import javax.inject.Singleton;
+import java.util.List;
+
+@Singleton
+public class ParseDatacenters extends ParseJson<Datacenters> {
+
+    @Inject
+    public ParseDatacenters(Json json) {

:+1:

-- 
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/jclouds/jclouds-labs/pull/377#discussion_r108109208

Re: [jclouds/jclouds-labs] Adding Dimension Data Infrastructure API (#377)

Posted by Ignasi Barrera <no...@github.com>.
nacx requested changes on this pull request.

Thanks @gatj98!
This looks very good, and it is great that you figured out how to implement the pagination (not easy at all) and that you implemented the mock tests accordingly.
Good job!

> +
+import com.google.common.base.Supplier;
+import org.jclouds.http.HttpException;
+import org.jclouds.http.HttpRequest;
+import org.jclouds.http.HttpRequestFilter;
+import org.jclouds.location.Zone;
+
+import javax.inject.Inject;
+import java.util.Set;
+
+/**
+ * Adds set of Datacenter IDs as set in jclouds.zones JVM property.
+ */
+public class DatacenterIdListDatacentersFilter implements HttpRequestFilter {
+   @Inject
+   @Zone

Locations in jclouds are defined in a hierarchy such as: Provider (unique; DimensionData) > Region > Zone.
Are we going to model the locations as zones, and a Datacenter will be one zone? In that case, what is going to be a region? Otherwise change the annotation to `@Region` and adopt that if it is more appropriate (just asking).

> +import org.jclouds.dimensiondata.cloudcontrol.DimensionDataCloudControlApi;
+import org.jclouds.dimensiondata.cloudcontrol.domain.Datacenter;
+import org.jclouds.dimensiondata.cloudcontrol.domain.Datacenters;
+import org.jclouds.dimensiondata.cloudcontrol.options.PaginationOptions;
+import org.jclouds.http.functions.ParseJson;
+import org.jclouds.json.Json;
+
+import javax.inject.Inject;
+import javax.inject.Singleton;
+import java.util.List;
+
+@Singleton
+public class ParseDatacenters extends ParseJson<Datacenters> {
+
+    @Inject
+    public ParseDatacenters(Json json) {

Remove the `public` modifier from all injection constructors.

Also, move all these parser classes that support the pagination iterables as inner classes of the API interface. It's the preferred approach now, to limit their visibility.

> +    public static class ToPagedIterable extends ArgsToPagedIterable<OperatingSystem, ToPagedIterable> {
+
+        private DimensionDataCloudControlApi api;
+
+        @Inject
+        public ToPagedIterable(DimensionDataCloudControlApi api) {
+            this.api = api;
+        }
+
+        @Override
+        protected Function<Object, IterableWithMarker<OperatingSystem>> markerToNextForArgs(List<Object> args) {
+            return new Function<Object, IterableWithMarker<OperatingSystem>>() {
+                @Override
+                public IterableWithMarker<OperatingSystem> apply(Object input) {
+                    PaginationOptions paginationOptions = PaginationOptions.class.cast(input);
+                    return api.getInfrastructureApi().listOperatingSystems(getArgs(request).get(0).toString(), paginationOptions);

No need for `getArgs(request)`; the `args` parameter in the method is already that; just use it.

> +    public static class ToPagedIterable extends ArgsToPagedIterable<OperatingSystem, ToPagedIterable> {
+
+        private DimensionDataCloudControlApi api;
+
+        @Inject
+        public ToPagedIterable(DimensionDataCloudControlApi api) {
+            this.api = api;
+        }
+
+        @Override
+        protected Function<Object, IterableWithMarker<OperatingSystem>> markerToNextForArgs(List<Object> args) {
+            return new Function<Object, IterableWithMarker<OperatingSystem>>() {
+                @Override
+                public IterableWithMarker<OperatingSystem> apply(Object input) {
+                    PaginationOptions paginationOptions = PaginationOptions.class.cast(input);
+                    return api.getInfrastructureApi().listOperatingSystems(getArgs(request).get(0).toString(), paginationOptions);

BTW, since in most cases, pagination methods take one argument (we've found it common in many APIs) jclouds provides the convenience class `Arg0ToPagedIterable`. It is the same but instead of getting the list of args, you just get one optional arg parameter (the first one passed to the original method). Perhaps you could consider extending that class if it is more convenient, but not a big deal :)

> +
+import org.jclouds.dimensiondata.cloudcontrol.domain.Datacenter;
+import org.jclouds.dimensiondata.cloudcontrol.domain.OperatingSystem;
+import org.jclouds.dimensiondata.cloudcontrol.internal.BaseDimensionDataCloudControlApiLiveTest;
+import org.testng.annotations.Test;
+
+import java.util.List;
+
+import static org.testng.Assert.assertNotNull;
+
+@Test(groups = "live", testName = "InfrastructureApiLiveTest", singleThreaded = true)
+public class InfrastructureApiLiveTest extends BaseDimensionDataCloudControlApiLiveTest {
+
+    @Test
+    public void testListDatacenters() {
+        List<Datacenter> datacenters = api().listDatacenters().concat().toList();

Better not call `toList()`. That will copy all elements into a list, losing the benefits of lazy loading each page. Not a big deal in the tests, but let's use pagination properly (this also serves as example for users).

> +      PaginatedCollection<Datacenter> datacenters = api.getInfrastructureApi().listDatacenters(paginationOptions);
+
+      assertEquals(250, datacenters.getPageSize());
+      assertEquals(2, datacenters.getTotalCount());
+      assertEquals(2, datacenters.getPageCount());
+      assertEquals(1, datacenters.getPageNumber());
+
+      Uris.UriBuilder uriBuilder = Uris
+            .uriBuilder("/caas/2.4/6ac1e746-b1ea-4da5-a24e-caf1a978789d/infrastructure/datacenter");
+      uriBuilder.addQuery(paginationOptions.buildQueryParameters());
+      assertSent(HttpMethod.GET, uriBuilder.toString());
+   }
+
+   public void testListDatacenters404() {
+      server.enqueue(response404());
+      assertTrue(api.getInfrastructureApi().listDatacenters().concat().isEmpty());

Also verify the sent request and that there is only on request generated.

> +
+      assertEquals(250, operatingSystems.getPageSize());
+      assertEquals(33, operatingSystems.getTotalCount());
+      assertEquals(33, operatingSystems.getPageCount());
+      assertEquals(1, operatingSystems.getPageNumber());
+
+      Uris.UriBuilder uriBuilder = Uris
+            .uriBuilder("/caas/2.4/6ac1e746-b1ea-4da5-a24e-caf1a978789d/infrastructure/operatingSystem");
+      uriBuilder.addQuery("datacenterId", "NA9");
+      uriBuilder.addQuery(paginationOptions.buildQueryParameters());
+      assertSent(HttpMethod.GET, uriBuilder.toString());
+   }
+
+   public void testListOperatingSystems404() {
+      server.enqueue(response404());
+      assertTrue(api.getInfrastructureApi().listOperatingSystems("NA9").concat().isEmpty());

Also verify the generated request.

> +/**
+ * Mock tests for the {@link InfrastructureApi} class.
+ */
+@Test(groups = "unit", testName = "InfrastructureApiMockTest", singleThreaded = true)
+public class InfrastructureApiMockTest extends BaseAccountAwareCloudControlMockTest {
+
+   public void testListDatacenters() throws Exception {
+      server.enqueue(jsonResponse("/datacenters.json"));
+      Iterable<Datacenter> datacenters = api.getInfrastructureApi().listDatacenters().concat();
+
+      assertEquals(size(datacenters), 2); // Force the PagedIterable to advance
+      assertEquals(server.getRequestCount(), 2);
+
+      Uris.UriBuilder uriBuilder = Uris
+            .uriBuilder("/caas/2.4/6ac1e746-b1ea-4da5-a24e-caf1a978789d/infrastructure/datacenter");
+      Set<String> zones = ctx.utils().injector().getInstance(ZoneIdsSupplier.class).get();

Is there a value configured there for this? Let's make sure (perhaps in the base class) to configure some zones so this test properly verifies the behaviour.

> + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.jclouds.dimensiondata.cloudcontrol.options;
+
+import com.google.common.collect.Multimap;
+import org.testng.annotations.BeforeMethod;
+import org.testng.annotations.Test;
+
+import java.util.Collection;
+
+import static org.testng.Assert.assertEquals;
+import static org.testng.Assert.assertTrue;
+
+@Test(groups = "unit", testName = "PaginationOptionsTest", singleThreaded = true)

No need for this to be single-threaded. Most single-threaded tests are due to the fact that mockwebserver starts the server in a before method. Tests that don't use MWS don't need to be single-threaded in general.

-- 
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/jclouds/jclouds-labs/pull/377#pullrequestreview-29018949

Re: [jclouds/jclouds-labs] Adding Dimension Data Infrastructure API (#377)

Posted by Gordon Jones <no...@github.com>.
Thanks @nacx - will tweak those and squash as requested

-- 
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/jclouds/jclouds-labs/pull/377#issuecomment-289456827

Re: [jclouds/jclouds-labs] Adding Dimension Data Infrastructure API (#377)

Posted by Gordon Jones <no...@github.com>.
@gatj98 pushed 2 commits.

d4be043  TestNG assertEquals() takes (actual, expected) - unlike JUnit
2861dd3  Making PaginationOptionsTest single threaded


-- 
You are receiving this because you are subscribed to this thread.
View it on GitHub:
https://github.com/jclouds/jclouds-labs/pull/377/files/12532d085d58c6611c8da7e800802230fbcc9a62..2861dd3efc90ca9ac433ca017374ae5adb86564f

Re: [jclouds/jclouds-labs] Adding Dimension Data Infrastructure API (#377)

Posted by Gordon Jones <no...@github.com>.
Ok that's squashed now @nacx - slight false start with a checkstyle error, but should be sorted now :)

-- 
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/jclouds/jclouds-labs/pull/377#issuecomment-289461537

Re: [jclouds/jclouds-labs] Adding Dimension Data Infrastructure API (#377)

Posted by Ignasi Barrera <no...@github.com>.
nacx requested changes on this pull request.

Just one comment. Thanks for the quick update!

> +
+         private DimensionDataCloudControlApi api;
+
+         @Inject
+         ToPagedIterable(DimensionDataCloudControlApi api) {
+            this.api = api;
+         }
+
+         @Override
+         protected Function<Object, IterableWithMarker<OperatingSystem>> markerToNextForArg0(
+               final Optional<Object> arg) {
+            return new Function<Object, IterableWithMarker<OperatingSystem>>() {
+               @Override
+               public IterableWithMarker<OperatingSystem> apply(Object input) {
+                  PaginationOptions paginationOptions = PaginationOptions.class.cast(input);
+                  return api.getInfrastructureApi().listOperatingSystems(arg.toString(), paginationOptions);

Take care! `arg.get().toString()`. Otherwise you're passing the `Optional` object.

> +
+         private DimensionDataCloudControlApi api;
+
+         @Inject
+         ToPagedIterable(DimensionDataCloudControlApi api) {
+            this.api = api;
+         }
+
+         @Override
+         protected Function<Object, IterableWithMarker<OperatingSystem>> markerToNextForArg0(
+               final Optional<Object> arg) {
+            return new Function<Object, IterableWithMarker<OperatingSystem>>() {
+               @Override
+               public IterableWithMarker<OperatingSystem> apply(Object input) {
+                  PaginationOptions paginationOptions = PaginationOptions.class.cast(input);
+                  return api.getInfrastructureApi().listOperatingSystems(arg.toString(), paginationOptions);

This should have been spotted by the Mock test. Let's modify the mock tests that use pagination and verify *all* the requests being sent. The second one should contain the pagination parameters and we should also verify that request.

-- 
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/jclouds/jclouds-labs/pull/377#pullrequestreview-29142280

Re: [jclouds/jclouds-labs] Adding Dimension Data Infrastructure API (#377)

Posted by Gordon Jones <no...@github.com>.
gatj98 commented on this pull request.



> +      PaginatedCollection<Datacenter> datacenters = api.getInfrastructureApi().listDatacenters(paginationOptions);
+
+      assertEquals(250, datacenters.getPageSize());
+      assertEquals(2, datacenters.getTotalCount());
+      assertEquals(2, datacenters.getPageCount());
+      assertEquals(1, datacenters.getPageNumber());
+
+      Uris.UriBuilder uriBuilder = Uris
+            .uriBuilder("/caas/2.4/6ac1e746-b1ea-4da5-a24e-caf1a978789d/infrastructure/datacenter");
+      uriBuilder.addQuery(paginationOptions.buildQueryParameters());
+      assertSent(HttpMethod.GET, uriBuilder.toString());
+   }
+
+   public void testListDatacenters404() {
+      server.enqueue(response404());
+      assertTrue(api.getInfrastructureApi().listDatacenters().concat().isEmpty());

👍 

-- 
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/jclouds/jclouds-labs/pull/377#discussion_r108030552

Re: [jclouds/jclouds-labs] Adding Dimension Data Infrastructure API (#377)

Posted by Ignasi Barrera <no...@github.com>.
>Looking at the region/zone stuff now - might be better if we do a separate pull request for that?

Sure!

-- 
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/jclouds/jclouds-labs/pull/377#issuecomment-289407593

Re: [jclouds/jclouds-labs] Adding Dimension Data Infrastructure API (#377)

Posted by Gordon Jones <no...@github.com>.
gatj98 commented on this pull request.



> +public class BaseAccountAwareCloudControlMockTest extends BaseDimensionDataCloudControlMockTest {
+
+   private boolean accountRetrieved;
+
+   @Override
+   protected void applyAdditionalServerConfig() {
+      server.enqueue(jsonResponse("/account.json"));
+   }
+
+   @BeforeMethod
+   public void reset() {
+      accountRetrieved = false;
+   }
+
+   protected Uris.UriBuilder addPageNumberToUriBuilder(Uris.UriBuilder uriBuilder, int pageNumber) {
+      uriBuilder.addQuery(PaginationOptions.Builder.pageNumber(pageNumber).buildQueryParameters());

Fair point.
Actually this method should prob be on BaseDimensionDataCloudControlMockTest rather than this class

-- 
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/jclouds/jclouds-labs/pull/377#discussion_r108170169

Re: [jclouds/jclouds-labs] Adding Dimension Data Infrastructure API (#377)

Posted by Ignasi Barrera <no...@github.com>.
Pushed to [master](http://git-wip-us.apache.org/repos/asf/jclouds-labs/commit/84ebe3d3) and [2.0.x](http://git-wip-us.apache.org/repos/asf/jclouds-labs/commit/42dc110f). Thanks @gatj98!

-- 
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/jclouds/jclouds-labs/pull/377#issuecomment-289467697

Re: [jclouds/jclouds-labs] Adding Dimension Data Infrastructure API (#377)

Posted by Ignasi Barrera <no...@github.com>.
nacx commented on this pull request.

Changes look good! Just one minor and one trivial comment left. Mind directly squashing the commits when addressing them so I can cleanly merge the PR?

> +public class BaseAccountAwareCloudControlMockTest extends BaseDimensionDataCloudControlMockTest {
+
+   private boolean accountRetrieved;
+
+   @Override
+   protected void applyAdditionalServerConfig() {
+      server.enqueue(jsonResponse("/account.json"));
+   }
+
+   @BeforeMethod
+   public void reset() {
+      accountRetrieved = false;
+   }
+
+   protected Uris.UriBuilder addPageNumberToUriBuilder(Uris.UriBuilder uriBuilder, int pageNumber) {
+      uriBuilder.addQuery(PaginationOptions.Builder.pageNumber(pageNumber).buildQueryParameters());

It's better not to use the pagination class. I'd say it would be better to directly call `uriBuilder.addQuery("param-name", pageNumber)`, to catch any (accidental or not) issue if the pagination options class is changed in the future.

>     }
 
    @AfterMethod(alwaysRun = true)
-   public void stop() throws IOException {
+   public void stop(ITestContext testContext) throws Exception {

Is this parameter actually needed?

-- 
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/jclouds/jclouds-labs/pull/377#pullrequestreview-29183161

Re: [jclouds/jclouds-labs] Adding Dimension Data Infrastructure API (#377)

Posted by Ignasi Barrera <no...@github.com>.
nacx commented on this pull request.



> @@ -52,12 +51,12 @@ public void testListDatacentersWithPagination() throws Exception {
       server.enqueue(jsonResponse("/datacenters-page2.json"));
       Iterable<Datacenter> datacenters = api.getInfrastructureApi().listDatacenters().concat();
 
-      assertEquals(server.getRequestCount(), 2);
-      assertEquals(size(datacenters), 2);
+      assertEquals(server.getRequestCount(), 2); // 1 for account, 1 for page 1
+      assertEquals(size(datacenters), 2); // Causes all pages to be requested
       assertEquals(server.getRequestCount(), 3); // 1 for account details, 2 pages

Sounds good :)

-- 
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/jclouds/jclouds-labs/pull/377#discussion_r108163601

Re: [jclouds/jclouds-labs] Adding Dimension Data Infrastructure API (#377)

Posted by Ignasi Barrera <no...@github.com>.
nacx commented on this pull request.



> + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.jclouds.dimensiondata.cloudcontrol.options;
+
+import com.google.common.collect.Multimap;
+import org.testng.annotations.BeforeMethod;
+import org.testng.annotations.Test;
+
+import java.util.Collection;
+
+import static org.testng.Assert.assertEquals;
+import static org.testng.Assert.assertTrue;
+
+@Test(groups = "unit", testName = "PaginationOptionsTest", singleThreaded = true)

Instead of making it single-threaded, I'd rather declare the `PaginationOptions` in each method. It's a very simple object and won't introduce much overhead (and you're already *creating* it before every method, so it should be OK to move the declaration and creation to each test method instead of having a class variable).

-- 
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/jclouds/jclouds-labs/pull/377#discussion_r108098247

Re: [jclouds/jclouds-labs] Adding Dimension Data Infrastructure API (#377)

Posted by Gordon Jones <no...@github.com>.
@gatj98 pushed 1 commit.

5e0e063  Added helper method to consume iterable and count page requests made


-- 
You are receiving this because you are subscribed to this thread.
View it on GitHub:
https://github.com/jclouds/jclouds-labs/pull/377/files/54c0d8e3888ff4362616e043b04e28dd16b07bd0..5e0e0632cc527bd0577814b40848ca80354c6adf

Re: [jclouds/jclouds-labs] Adding Dimension Data Infrastructure API (#377)

Posted by Gordon Jones <no...@github.com>.
@gatj98 pushed 1 commit.

54c0d8e  Moved addPagesToUriBuilder to base test class


-- 
You are receiving this because you are subscribed to this thread.
View it on GitHub:
https://github.com/jclouds/jclouds-labs/pull/377/files/1db79ae7733e79128f5a3dc202c7558b5eb12161..54c0d8e3888ff4362616e043b04e28dd16b07bd0

Re: [jclouds/jclouds-labs] Adding Dimension Data Infrastructure API (#377)

Posted by Ignasi Barrera <no...@github.com>.
>Have added an IHookable in BaseDimensionDataCloudControlMockTest that checks after each test runs that we've assertted all of the requests received by the mock web server - hopefully this will help us write better tests going forward.

And this looks like a good pattern we can propagate to all other providers. Thanks!

-- 
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/jclouds/jclouds-labs/pull/377#issuecomment-289448459

Re: [jclouds/jclouds-labs] Adding Dimension Data Infrastructure API (#377)

Posted by Gordon Jones <no...@github.com>.
gatj98 commented on this pull request.



> @@ -52,12 +51,12 @@ public void testListDatacentersWithPagination() throws Exception {
       server.enqueue(jsonResponse("/datacenters-page2.json"));
       Iterable<Datacenter> datacenters = api.getInfrastructureApi().listDatacenters().concat();
 
-      assertEquals(server.getRequestCount(), 2);
-      assertEquals(size(datacenters), 2);
+      assertEquals(server.getRequestCount(), 2); // 1 for account, 1 for page 1
+      assertEquals(size(datacenters), 2); // Causes all pages to be requested
       assertEquals(server.getRequestCount(), 3); // 1 for account details, 2 pages

Cool - added that in latest commit

-- 
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/jclouds/jclouds-labs/pull/377#discussion_r108165802

Re: [jclouds/jclouds-labs] Adding Dimension Data Infrastructure API (#377)

Posted by Gordon Jones <no...@github.com>.
gatj98 commented on this pull request.



> +    public static class ToPagedIterable extends ArgsToPagedIterable<OperatingSystem, ToPagedIterable> {
+
+        private DimensionDataCloudControlApi api;
+
+        @Inject
+        public ToPagedIterable(DimensionDataCloudControlApi api) {
+            this.api = api;
+        }
+
+        @Override
+        protected Function<Object, IterableWithMarker<OperatingSystem>> markerToNextForArgs(List<Object> args) {
+            return new Function<Object, IterableWithMarker<OperatingSystem>>() {
+                @Override
+                public IterableWithMarker<OperatingSystem> apply(Object input) {
+                    PaginationOptions paginationOptions = PaginationOptions.class.cast(input);
+                    return api.getInfrastructureApi().listOperatingSystems(getArgs(request).get(0).toString(), paginationOptions);

:+1: 

-- 
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/jclouds/jclouds-labs/pull/377#discussion_r108113115

Re: [jclouds/jclouds-labs] Adding Dimension Data Infrastructure API (#377)

Posted by Gordon Jones <no...@github.com>.
gatj98 commented on this pull request.



>     }
 
    @AfterMethod(alwaysRun = true)
-   public void stop() throws IOException {
+   public void stop(ITestContext testContext) throws Exception {

Nuts - no it's not - put it in whilst trying something else and forgot it was there!

-- 
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/jclouds/jclouds-labs/pull/377#discussion_r108169763

Re: [jclouds/jclouds-labs] Adding Dimension Data Infrastructure API (#377)

Posted by Gordon Jones <no...@github.com>.
Thanks for the comments @nacx - new commit with the changes (apart from region/zone stuff) made.
Looking at the region/zone stuff now - might be better if we do a separate pull request for that?

-- 
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/jclouds/jclouds-labs/pull/377#issuecomment-289390916