You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@openwhisk.apache.org by Ben Browning <bb...@redhat.com> on 2018/06/21 18:24:20 UTC
Let's maintain and test our Swagger spec
Our Swagger spec
(https://github.com/apache/incubator-openwhisk/blob/92a64c291156a2cd3d6b304babc2a193a46d0699/core/controller/src/main/resources/apiv1swagger.json)
is incomplete and doesn't accurately reflect the actual Controller
API. It's manually updated without a full test suite which means it's
easy for changes in the API to happen without the spec getting
updated.
An accurate Swagger spec will not only better document the OpenWhisk
API but also allow autogenerated clients in multiple languages to
supplement or eventually replace some of the existing client
implementations we have today. It also paves way for future compatible
server implementations, whether they be rewrites of the existing
Controller or stub test harnesses to facilitate end-to-end testing on
a developer's laptop.
As I'm already working with autogenerating code from the Swagger spec
for other purposes, I'm happy to take the lead on this effort. I'd
like to take a two-pronged approach for a test suite:
* Generate a server stub from the spec and ensure the wsk CLI can
communicate with it.
* Generate a client stub from the spec and ensure it can communicate
with the existing API.
There are a lot of details to figure out from those two statements.
And, this approach won't guarantee 100% correctness of the spec. The
only way to do that would be to generate all supported clients and the
Controller API from the spec. But, this should get us started in the
right direction.
If anyone's gone down this path before and has some wisdom to share,
please speak up!
Thanks,
Ben
Re: Let's maintain and test our Swagger spec
Posted by Rodric Rabbah <ro...@gmail.com>.
That's fantastic! Thanks Ben for taking this on... so long overdue.
On Thu, Jul 5, 2018 at 9:20 PM, Ben Browning <bb...@redhat.com> wrote:
> After several failed attempts, I have an approach that I believe will
> work well enough for our needs. I pushed some in-progress code and
> swagger spec fixes to a branch on a fork just so the general approach
> can be shared -
> https://github.com/projectodd/incubator-openwhisk/commit/
> 5adb6be27188dcdc5b9519b57c11802a431b3e9c
>
> This approach uses the swagger-request-validator Java library. I'm
> wiring that up to validate all requests and responses from the
> WskRest* tests transparently, which you can see in
> https://github.com/projectodd/incubator-openwhisk/commit/
> 5adb6be27188dcdc5b9519b57c11802a431b3e9c#diff-
> 5e514c6c3c27b9210d85a08d4ed3ed35
> (with some cruft and debug output that I'll clean up before submitting
> a PR)
>
> This means that almost all of those tests are now failing due to
> failed swagger spec validations. For now I'm going to concentrate on
> making the spec match the reality and try to get all the existing
> tests green with the validation in place and submit a PR. After that,
> we can patch up any known but uncaught holes by increasing the test
> coverage.
>
> Ben
>
>
>
> On Thu, Jun 28, 2018 at 11:23 PM, Carlos Santana <cs...@gmail.com>
> wrote:
> > Thanks Ben for the quick update on this task
> >
> > -cs
> >
> > On Thu, Jun 28, 2018 at 5:15 PM Ben Browning <bb...@redhat.com>
> wrote:
> >
> >> After doing some preliminary poking at this, I believe we'll want to
> >> use either a tool like https://github.com/google/oatts to generate a
> >> test suite from our Swagger spec OR use swagger-codegen to generate a
> >> Scala client from our Swagger spec and try to plug that into the
> >> existing WskRest tests.
> >>
> >> Using the oatts tool doesn't really fit well with the test setup in
> >> the existing incubator-openwhisk repo (where the API spec lives)
> >> because that generates Node.js tests.
> >>
> >> So, I'm leaning towards the second option, which is wiring in
> >> generation of a Scala client into the gradle build and having the
> >> current WskRest test client use this generated Scala client for
> >> testing instead of directly invoking URLs.
> >>
> >> However, last time I played with Scala code generated from Swagger
> >> specs it wasn't that usable. So, a bit more experimentation will
> >> validate whether this option is viable or if other alternatives need
> >> to be considered. I already have a handful of bugs in the API spec
> >> that need to be fixed but I'm waiting to fix and push those until I
> >> can get some kind of testing wired up to reproduce the bugs and verify
> >> the fix.
> >>
> >> Ben
> >>
> >>
> >> On Thu, Jun 21, 2018 at 3:04 PM, Carlos Santana <cs...@gmail.com>
> >> wrote:
> >> > Thanks Ben for looking into this, having a good API doc/spec and
> matching
> >> > tests is very need it.
> >> >
> >> > +1
> >> >
> >> > -cs
> >> >
> >> > On Thu, Jun 21, 2018 at 2:25 PM Ben Browning <bb...@redhat.com>
> >> wrote:
> >> >
> >> >> Our Swagger spec
> >> >> (
> >> >>
> >> https://github.com/apache/incubator-openwhisk/blob/
> 92a64c291156a2cd3d6b304babc2a193a46d0699/core/controller/
> src/main/resources/apiv1swagger.json
> >> >> )
> >> >> is incomplete and doesn't accurately reflect the actual Controller
> >> >> API. It's manually updated without a full test suite which means it's
> >> >> easy for changes in the API to happen without the spec getting
> >> >> updated.
> >> >>
> >> >> An accurate Swagger spec will not only better document the OpenWhisk
> >> >> API but also allow autogenerated clients in multiple languages to
> >> >> supplement or eventually replace some of the existing client
> >> >> implementations we have today. It also paves way for future
> compatible
> >> >> server implementations, whether they be rewrites of the existing
> >> >> Controller or stub test harnesses to facilitate end-to-end testing on
> >> >> a developer's laptop.
> >> >>
> >> >> As I'm already working with autogenerating code from the Swagger spec
> >> >> for other purposes, I'm happy to take the lead on this effort. I'd
> >> >> like to take a two-pronged approach for a test suite:
> >> >>
> >> >> * Generate a server stub from the spec and ensure the wsk CLI can
> >> >> communicate with it.
> >> >>
> >> >> * Generate a client stub from the spec and ensure it can communicate
> >> >> with the existing API.
> >> >>
> >> >> There are a lot of details to figure out from those two statements.
> >> >> And, this approach won't guarantee 100% correctness of the spec. The
> >> >> only way to do that would be to generate all supported clients and
> the
> >> >> Controller API from the spec. But, this should get us started in the
> >> >> right direction.
> >> >>
> >> >> If anyone's gone down this path before and has some wisdom to share,
> >> >> please speak up!
> >> >>
> >> >> Thanks,
> >> >>
> >> >> Ben
> >> >>
> >>
>
Re: Let's maintain and test our Swagger spec
Posted by James Thomas <jt...@gmail.com>.
Excellent work Ben, this will really help everyone building integrations
based upon the platform API.
Being able to auto-generate SDKs for most langauges is also something that
would really benefit the project.
Great stuff!
On 13 July 2018 at 22:32, Ben Browning <bb...@redhat.com> wrote:
> I've submitted a PR for the Swagger spec changes and validations at
> https://github.com/apache/incubator-openwhisk/pull/3878
>
> For this first iteration, I just focused on making the Swagger spec
> match the behavior of our existing tests. As Rodric pointed out, that
> may not exactly match the REST interface, but it at least gets us
> closer.
>
> As a future second iteration, I'd like to investigate generating and
> maintaining a new Java OpenWhisk client from the spec (see
> https://github.com/apache/incubator-openwhisk/issues/2466). Using the
> spec to generate and test a functioning Java client should help find
> and fix more spec issues and pave the way for potentially moving other
> hand-generated client libraries over to being at least partially
> generated from the API spec.
>
> Ben
>
>
> On Fri, Jul 6, 2018 at 12:55 AM, Rodric Rabbah <ro...@gmail.com> wrote:
> > Ben, you might want to take note of this PR https://github.com/apache/
> > incubator-openwhisk/pull/3840
> > which removes a number of tests suites (redundant with unit tests). Also
> > I've found that the WskRestOperations implementation in some places fixes
> > behaviors to match the CLI instead of strictly implementing the
> > narrower/strict REST interface. This may mean you have to edit tests (or
> > remove some that are only valid for clients).
> >
> > -r
> >
> > On Thu, Jul 5, 2018 at 9:20 PM, Ben Browning <bb...@redhat.com>
> wrote:
> >
> >> After several failed attempts, I have an approach that I believe will
> >> work well enough for our needs. I pushed some in-progress code and
> >> swagger spec fixes to a branch on a fork just so the general approach
> >> can be shared -
> >> https://github.com/projectodd/incubator-openwhisk/commit/5ad
> >> b6be27188dcdc5b9519b57c11802a431b3e9c
> >>
> >> This approach uses the swagger-request-validator Java library. I'm
> >> wiring that up to validate all requests and responses from the
> >> WskRest* tests transparently, which you can see in
> >> https://github.com/projectodd/incubator-openwhisk/commit/5ad
> >> b6be27188dcdc5b9519b57c11802a431b3e9c#diff-5e514c6c3c27b9210
> >> d85a08d4ed3ed35
> >> (with some cruft and debug output that I'll clean up before submitting
> >> a PR)
> >>
> >> This means that almost all of those tests are now failing due to
> >> failed swagger spec validations. For now I'm going to concentrate on
> >> making the spec match the reality and try to get all the existing
> >> tests green with the validation in place and submit a PR. After that,
> >> we can patch up any known but uncaught holes by increasing the test
> >> coverage.
> >>
> >> Ben
> >>
> >>
> >>
> >> On Thu, Jun 28, 2018 at 11:23 PM, Carlos Santana <cs...@gmail.com>
> >> wrote:
> >> > Thanks Ben for the quick update on this task
> >> >
> >> > -cs
> >> >
> >> > On Thu, Jun 28, 2018 at 5:15 PM Ben Browning <bb...@redhat.com>
> >> wrote:
> >> >
> >> >> After doing some preliminary poking at this, I believe we'll want to
> >> >> use either a tool like https://github.com/google/oatts to generate a
> >> >> test suite from our Swagger spec OR use swagger-codegen to generate a
> >> >> Scala client from our Swagger spec and try to plug that into the
> >> >> existing WskRest tests.
> >> >>
> >> >> Using the oatts tool doesn't really fit well with the test setup in
> >> >> the existing incubator-openwhisk repo (where the API spec lives)
> >> >> because that generates Node.js tests.
> >> >>
> >> >> So, I'm leaning towards the second option, which is wiring in
> >> >> generation of a Scala client into the gradle build and having the
> >> >> current WskRest test client use this generated Scala client for
> >> >> testing instead of directly invoking URLs.
> >> >>
> >> >> However, last time I played with Scala code generated from Swagger
> >> >> specs it wasn't that usable. So, a bit more experimentation will
> >> >> validate whether this option is viable or if other alternatives need
> >> >> to be considered. I already have a handful of bugs in the API spec
> >> >> that need to be fixed but I'm waiting to fix and push those until I
> >> >> can get some kind of testing wired up to reproduce the bugs and
> verify
> >> >> the fix.
> >> >>
> >> >> Ben
> >> >>
> >> >>
> >> >> On Thu, Jun 21, 2018 at 3:04 PM, Carlos Santana <
> csantana23@gmail.com>
> >> >> wrote:
> >> >> > Thanks Ben for looking into this, having a good API doc/spec and
> >> matching
> >> >> > tests is very need it.
> >> >> >
> >> >> > +1
> >> >> >
> >> >> > -cs
> >> >> >
> >> >> > On Thu, Jun 21, 2018 at 2:25 PM Ben Browning <bb...@redhat.com>
> >> >> wrote:
> >> >> >
> >> >> >> Our Swagger spec
> >> >> >> (
> >> >> >>
> >> >> https://github.com/apache/incubator-openwhisk/blob/92a64c291
> >> 156a2cd3d6b304babc2a193a46d0699/core/controller/src/main/
> >> resources/apiv1swagger.json
> >> >> >> )
> >> >> >> is incomplete and doesn't accurately reflect the actual Controller
> >> >> >> API. It's manually updated without a full test suite which means
> it's
> >> >> >> easy for changes in the API to happen without the spec getting
> >> >> >> updated.
> >> >> >>
> >> >> >> An accurate Swagger spec will not only better document the
> OpenWhisk
> >> >> >> API but also allow autogenerated clients in multiple languages to
> >> >> >> supplement or eventually replace some of the existing client
> >> >> >> implementations we have today. It also paves way for future
> >> compatible
> >> >> >> server implementations, whether they be rewrites of the existing
> >> >> >> Controller or stub test harnesses to facilitate end-to-end
> testing on
> >> >> >> a developer's laptop.
> >> >> >>
> >> >> >> As I'm already working with autogenerating code from the Swagger
> spec
> >> >> >> for other purposes, I'm happy to take the lead on this effort. I'd
> >> >> >> like to take a two-pronged approach for a test suite:
> >> >> >>
> >> >> >> * Generate a server stub from the spec and ensure the wsk CLI can
> >> >> >> communicate with it.
> >> >> >>
> >> >> >> * Generate a client stub from the spec and ensure it can
> communicate
> >> >> >> with the existing API.
> >> >> >>
> >> >> >> There are a lot of details to figure out from those two
> statements.
> >> >> >> And, this approach won't guarantee 100% correctness of the spec.
> The
> >> >> >> only way to do that would be to generate all supported clients and
> >> the
> >> >> >> Controller API from the spec. But, this should get us started in
> the
> >> >> >> right direction.
> >> >> >>
> >> >> >> If anyone's gone down this path before and has some wisdom to
> share,
> >> >> >> please speak up!
> >> >> >>
> >> >> >> Thanks,
> >> >> >>
> >> >> >> Ben
> >> >> >>
> >> >>
> >>
>
--
Regards,
James Thomas
Re: Let's maintain and test our Swagger spec
Posted by Ben Browning <bb...@redhat.com>.
I've submitted a PR for the Swagger spec changes and validations at
https://github.com/apache/incubator-openwhisk/pull/3878
For this first iteration, I just focused on making the Swagger spec
match the behavior of our existing tests. As Rodric pointed out, that
may not exactly match the REST interface, but it at least gets us
closer.
As a future second iteration, I'd like to investigate generating and
maintaining a new Java OpenWhisk client from the spec (see
https://github.com/apache/incubator-openwhisk/issues/2466). Using the
spec to generate and test a functioning Java client should help find
and fix more spec issues and pave the way for potentially moving other
hand-generated client libraries over to being at least partially
generated from the API spec.
Ben
On Fri, Jul 6, 2018 at 12:55 AM, Rodric Rabbah <ro...@gmail.com> wrote:
> Ben, you might want to take note of this PR https://github.com/apache/
> incubator-openwhisk/pull/3840
> which removes a number of tests suites (redundant with unit tests). Also
> I've found that the WskRestOperations implementation in some places fixes
> behaviors to match the CLI instead of strictly implementing the
> narrower/strict REST interface. This may mean you have to edit tests (or
> remove some that are only valid for clients).
>
> -r
>
> On Thu, Jul 5, 2018 at 9:20 PM, Ben Browning <bb...@redhat.com> wrote:
>
>> After several failed attempts, I have an approach that I believe will
>> work well enough for our needs. I pushed some in-progress code and
>> swagger spec fixes to a branch on a fork just so the general approach
>> can be shared -
>> https://github.com/projectodd/incubator-openwhisk/commit/5ad
>> b6be27188dcdc5b9519b57c11802a431b3e9c
>>
>> This approach uses the swagger-request-validator Java library. I'm
>> wiring that up to validate all requests and responses from the
>> WskRest* tests transparently, which you can see in
>> https://github.com/projectodd/incubator-openwhisk/commit/5ad
>> b6be27188dcdc5b9519b57c11802a431b3e9c#diff-5e514c6c3c27b9210
>> d85a08d4ed3ed35
>> (with some cruft and debug output that I'll clean up before submitting
>> a PR)
>>
>> This means that almost all of those tests are now failing due to
>> failed swagger spec validations. For now I'm going to concentrate on
>> making the spec match the reality and try to get all the existing
>> tests green with the validation in place and submit a PR. After that,
>> we can patch up any known but uncaught holes by increasing the test
>> coverage.
>>
>> Ben
>>
>>
>>
>> On Thu, Jun 28, 2018 at 11:23 PM, Carlos Santana <cs...@gmail.com>
>> wrote:
>> > Thanks Ben for the quick update on this task
>> >
>> > -cs
>> >
>> > On Thu, Jun 28, 2018 at 5:15 PM Ben Browning <bb...@redhat.com>
>> wrote:
>> >
>> >> After doing some preliminary poking at this, I believe we'll want to
>> >> use either a tool like https://github.com/google/oatts to generate a
>> >> test suite from our Swagger spec OR use swagger-codegen to generate a
>> >> Scala client from our Swagger spec and try to plug that into the
>> >> existing WskRest tests.
>> >>
>> >> Using the oatts tool doesn't really fit well with the test setup in
>> >> the existing incubator-openwhisk repo (where the API spec lives)
>> >> because that generates Node.js tests.
>> >>
>> >> So, I'm leaning towards the second option, which is wiring in
>> >> generation of a Scala client into the gradle build and having the
>> >> current WskRest test client use this generated Scala client for
>> >> testing instead of directly invoking URLs.
>> >>
>> >> However, last time I played with Scala code generated from Swagger
>> >> specs it wasn't that usable. So, a bit more experimentation will
>> >> validate whether this option is viable or if other alternatives need
>> >> to be considered. I already have a handful of bugs in the API spec
>> >> that need to be fixed but I'm waiting to fix and push those until I
>> >> can get some kind of testing wired up to reproduce the bugs and verify
>> >> the fix.
>> >>
>> >> Ben
>> >>
>> >>
>> >> On Thu, Jun 21, 2018 at 3:04 PM, Carlos Santana <cs...@gmail.com>
>> >> wrote:
>> >> > Thanks Ben for looking into this, having a good API doc/spec and
>> matching
>> >> > tests is very need it.
>> >> >
>> >> > +1
>> >> >
>> >> > -cs
>> >> >
>> >> > On Thu, Jun 21, 2018 at 2:25 PM Ben Browning <bb...@redhat.com>
>> >> wrote:
>> >> >
>> >> >> Our Swagger spec
>> >> >> (
>> >> >>
>> >> https://github.com/apache/incubator-openwhisk/blob/92a64c291
>> 156a2cd3d6b304babc2a193a46d0699/core/controller/src/main/
>> resources/apiv1swagger.json
>> >> >> )
>> >> >> is incomplete and doesn't accurately reflect the actual Controller
>> >> >> API. It's manually updated without a full test suite which means it's
>> >> >> easy for changes in the API to happen without the spec getting
>> >> >> updated.
>> >> >>
>> >> >> An accurate Swagger spec will not only better document the OpenWhisk
>> >> >> API but also allow autogenerated clients in multiple languages to
>> >> >> supplement or eventually replace some of the existing client
>> >> >> implementations we have today. It also paves way for future
>> compatible
>> >> >> server implementations, whether they be rewrites of the existing
>> >> >> Controller or stub test harnesses to facilitate end-to-end testing on
>> >> >> a developer's laptop.
>> >> >>
>> >> >> As I'm already working with autogenerating code from the Swagger spec
>> >> >> for other purposes, I'm happy to take the lead on this effort. I'd
>> >> >> like to take a two-pronged approach for a test suite:
>> >> >>
>> >> >> * Generate a server stub from the spec and ensure the wsk CLI can
>> >> >> communicate with it.
>> >> >>
>> >> >> * Generate a client stub from the spec and ensure it can communicate
>> >> >> with the existing API.
>> >> >>
>> >> >> There are a lot of details to figure out from those two statements.
>> >> >> And, this approach won't guarantee 100% correctness of the spec. The
>> >> >> only way to do that would be to generate all supported clients and
>> the
>> >> >> Controller API from the spec. But, this should get us started in the
>> >> >> right direction.
>> >> >>
>> >> >> If anyone's gone down this path before and has some wisdom to share,
>> >> >> please speak up!
>> >> >>
>> >> >> Thanks,
>> >> >>
>> >> >> Ben
>> >> >>
>> >>
>>
Re: Let's maintain and test our Swagger spec
Posted by Rodric Rabbah <ro...@gmail.com>.
Ben, you might want to take note of this PR https://github.com/apache/
incubator-openwhisk/pull/3840
which removes a number of tests suites (redundant with unit tests). Also
I've found that the WskRestOperations implementation in some places fixes
behaviors to match the CLI instead of strictly implementing the
narrower/strict REST interface. This may mean you have to edit tests (or
remove some that are only valid for clients).
-r
On Thu, Jul 5, 2018 at 9:20 PM, Ben Browning <bb...@redhat.com> wrote:
> After several failed attempts, I have an approach that I believe will
> work well enough for our needs. I pushed some in-progress code and
> swagger spec fixes to a branch on a fork just so the general approach
> can be shared -
> https://github.com/projectodd/incubator-openwhisk/commit/5ad
> b6be27188dcdc5b9519b57c11802a431b3e9c
>
> This approach uses the swagger-request-validator Java library. I'm
> wiring that up to validate all requests and responses from the
> WskRest* tests transparently, which you can see in
> https://github.com/projectodd/incubator-openwhisk/commit/5ad
> b6be27188dcdc5b9519b57c11802a431b3e9c#diff-5e514c6c3c27b9210
> d85a08d4ed3ed35
> (with some cruft and debug output that I'll clean up before submitting
> a PR)
>
> This means that almost all of those tests are now failing due to
> failed swagger spec validations. For now I'm going to concentrate on
> making the spec match the reality and try to get all the existing
> tests green with the validation in place and submit a PR. After that,
> we can patch up any known but uncaught holes by increasing the test
> coverage.
>
> Ben
>
>
>
> On Thu, Jun 28, 2018 at 11:23 PM, Carlos Santana <cs...@gmail.com>
> wrote:
> > Thanks Ben for the quick update on this task
> >
> > -cs
> >
> > On Thu, Jun 28, 2018 at 5:15 PM Ben Browning <bb...@redhat.com>
> wrote:
> >
> >> After doing some preliminary poking at this, I believe we'll want to
> >> use either a tool like https://github.com/google/oatts to generate a
> >> test suite from our Swagger spec OR use swagger-codegen to generate a
> >> Scala client from our Swagger spec and try to plug that into the
> >> existing WskRest tests.
> >>
> >> Using the oatts tool doesn't really fit well with the test setup in
> >> the existing incubator-openwhisk repo (where the API spec lives)
> >> because that generates Node.js tests.
> >>
> >> So, I'm leaning towards the second option, which is wiring in
> >> generation of a Scala client into the gradle build and having the
> >> current WskRest test client use this generated Scala client for
> >> testing instead of directly invoking URLs.
> >>
> >> However, last time I played with Scala code generated from Swagger
> >> specs it wasn't that usable. So, a bit more experimentation will
> >> validate whether this option is viable or if other alternatives need
> >> to be considered. I already have a handful of bugs in the API spec
> >> that need to be fixed but I'm waiting to fix and push those until I
> >> can get some kind of testing wired up to reproduce the bugs and verify
> >> the fix.
> >>
> >> Ben
> >>
> >>
> >> On Thu, Jun 21, 2018 at 3:04 PM, Carlos Santana <cs...@gmail.com>
> >> wrote:
> >> > Thanks Ben for looking into this, having a good API doc/spec and
> matching
> >> > tests is very need it.
> >> >
> >> > +1
> >> >
> >> > -cs
> >> >
> >> > On Thu, Jun 21, 2018 at 2:25 PM Ben Browning <bb...@redhat.com>
> >> wrote:
> >> >
> >> >> Our Swagger spec
> >> >> (
> >> >>
> >> https://github.com/apache/incubator-openwhisk/blob/92a64c291
> 156a2cd3d6b304babc2a193a46d0699/core/controller/src/main/
> resources/apiv1swagger.json
> >> >> )
> >> >> is incomplete and doesn't accurately reflect the actual Controller
> >> >> API. It's manually updated without a full test suite which means it's
> >> >> easy for changes in the API to happen without the spec getting
> >> >> updated.
> >> >>
> >> >> An accurate Swagger spec will not only better document the OpenWhisk
> >> >> API but also allow autogenerated clients in multiple languages to
> >> >> supplement or eventually replace some of the existing client
> >> >> implementations we have today. It also paves way for future
> compatible
> >> >> server implementations, whether they be rewrites of the existing
> >> >> Controller or stub test harnesses to facilitate end-to-end testing on
> >> >> a developer's laptop.
> >> >>
> >> >> As I'm already working with autogenerating code from the Swagger spec
> >> >> for other purposes, I'm happy to take the lead on this effort. I'd
> >> >> like to take a two-pronged approach for a test suite:
> >> >>
> >> >> * Generate a server stub from the spec and ensure the wsk CLI can
> >> >> communicate with it.
> >> >>
> >> >> * Generate a client stub from the spec and ensure it can communicate
> >> >> with the existing API.
> >> >>
> >> >> There are a lot of details to figure out from those two statements.
> >> >> And, this approach won't guarantee 100% correctness of the spec. The
> >> >> only way to do that would be to generate all supported clients and
> the
> >> >> Controller API from the spec. But, this should get us started in the
> >> >> right direction.
> >> >>
> >> >> If anyone's gone down this path before and has some wisdom to share,
> >> >> please speak up!
> >> >>
> >> >> Thanks,
> >> >>
> >> >> Ben
> >> >>
> >>
>
Re: Let's maintain and test our Swagger spec
Posted by Carlos Santana <cs...@gmail.com>.
TDD FTW !
- Carlos Santana
@csantanapr
> On Jul 6, 2018, at 2:20 AM, Ben Browning <bb...@redhat.com> wrote:
>
> After several failed attempts, I have an approach that I believe will
> work well enough for our needs. I pushed some in-progress code and
> swagger spec fixes to a branch on a fork just so the general approach
> can be shared -
> https://github.com/projectodd/incubator-openwhisk/commit/5adb6be27188dcdc5b9519b57c11802a431b3e9c
>
> This approach uses the swagger-request-validator Java library. I'm
> wiring that up to validate all requests and responses from the
> WskRest* tests transparently, which you can see in
> https://github.com/projectodd/incubator-openwhisk/commit/5adb6be27188dcdc5b9519b57c11802a431b3e9c#diff-5e514c6c3c27b9210d85a08d4ed3ed35
> (with some cruft and debug output that I'll clean up before submitting
> a PR)
>
> This means that almost all of those tests are now failing due to
> failed swagger spec validations. For now I'm going to concentrate on
> making the spec match the reality and try to get all the existing
> tests green with the validation in place and submit a PR. After that,
> we can patch up any known but uncaught holes by increasing the test
> coverage.
>
> Ben
>
>
>
>> On Thu, Jun 28, 2018 at 11:23 PM, Carlos Santana <cs...@gmail.com> wrote:
>> Thanks Ben for the quick update on this task
>>
>> -cs
>>
>>> On Thu, Jun 28, 2018 at 5:15 PM Ben Browning <bb...@redhat.com> wrote:
>>>
>>> After doing some preliminary poking at this, I believe we'll want to
>>> use either a tool like https://github.com/google/oatts to generate a
>>> test suite from our Swagger spec OR use swagger-codegen to generate a
>>> Scala client from our Swagger spec and try to plug that into the
>>> existing WskRest tests.
>>>
>>> Using the oatts tool doesn't really fit well with the test setup in
>>> the existing incubator-openwhisk repo (where the API spec lives)
>>> because that generates Node.js tests.
>>>
>>> So, I'm leaning towards the second option, which is wiring in
>>> generation of a Scala client into the gradle build and having the
>>> current WskRest test client use this generated Scala client for
>>> testing instead of directly invoking URLs.
>>>
>>> However, last time I played with Scala code generated from Swagger
>>> specs it wasn't that usable. So, a bit more experimentation will
>>> validate whether this option is viable or if other alternatives need
>>> to be considered. I already have a handful of bugs in the API spec
>>> that need to be fixed but I'm waiting to fix and push those until I
>>> can get some kind of testing wired up to reproduce the bugs and verify
>>> the fix.
>>>
>>> Ben
>>>
>>>
>>> On Thu, Jun 21, 2018 at 3:04 PM, Carlos Santana <cs...@gmail.com>
>>> wrote:
>>>> Thanks Ben for looking into this, having a good API doc/spec and matching
>>>> tests is very need it.
>>>>
>>>> +1
>>>>
>>>> -cs
>>>>
>>>> On Thu, Jun 21, 2018 at 2:25 PM Ben Browning <bb...@redhat.com>
>>> wrote:
>>>>
>>>>> Our Swagger spec
>>>>> (
>>>>>
>>> https://github.com/apache/incubator-openwhisk/blob/92a64c291156a2cd3d6b304babc2a193a46d0699/core/controller/src/main/resources/apiv1swagger.json
>>>>> )
>>>>> is incomplete and doesn't accurately reflect the actual Controller
>>>>> API. It's manually updated without a full test suite which means it's
>>>>> easy for changes in the API to happen without the spec getting
>>>>> updated.
>>>>>
>>>>> An accurate Swagger spec will not only better document the OpenWhisk
>>>>> API but also allow autogenerated clients in multiple languages to
>>>>> supplement or eventually replace some of the existing client
>>>>> implementations we have today. It also paves way for future compatible
>>>>> server implementations, whether they be rewrites of the existing
>>>>> Controller or stub test harnesses to facilitate end-to-end testing on
>>>>> a developer's laptop.
>>>>>
>>>>> As I'm already working with autogenerating code from the Swagger spec
>>>>> for other purposes, I'm happy to take the lead on this effort. I'd
>>>>> like to take a two-pronged approach for a test suite:
>>>>>
>>>>> * Generate a server stub from the spec and ensure the wsk CLI can
>>>>> communicate with it.
>>>>>
>>>>> * Generate a client stub from the spec and ensure it can communicate
>>>>> with the existing API.
>>>>>
>>>>> There are a lot of details to figure out from those two statements.
>>>>> And, this approach won't guarantee 100% correctness of the spec. The
>>>>> only way to do that would be to generate all supported clients and the
>>>>> Controller API from the spec. But, this should get us started in the
>>>>> right direction.
>>>>>
>>>>> If anyone's gone down this path before and has some wisdom to share,
>>>>> please speak up!
>>>>>
>>>>> Thanks,
>>>>>
>>>>> Ben
>>>>>
>>>
Re: Let's maintain and test our Swagger spec
Posted by Ben Browning <bb...@redhat.com>.
After several failed attempts, I have an approach that I believe will
work well enough for our needs. I pushed some in-progress code and
swagger spec fixes to a branch on a fork just so the general approach
can be shared -
https://github.com/projectodd/incubator-openwhisk/commit/5adb6be27188dcdc5b9519b57c11802a431b3e9c
This approach uses the swagger-request-validator Java library. I'm
wiring that up to validate all requests and responses from the
WskRest* tests transparently, which you can see in
https://github.com/projectodd/incubator-openwhisk/commit/5adb6be27188dcdc5b9519b57c11802a431b3e9c#diff-5e514c6c3c27b9210d85a08d4ed3ed35
(with some cruft and debug output that I'll clean up before submitting
a PR)
This means that almost all of those tests are now failing due to
failed swagger spec validations. For now I'm going to concentrate on
making the spec match the reality and try to get all the existing
tests green with the validation in place and submit a PR. After that,
we can patch up any known but uncaught holes by increasing the test
coverage.
Ben
On Thu, Jun 28, 2018 at 11:23 PM, Carlos Santana <cs...@gmail.com> wrote:
> Thanks Ben for the quick update on this task
>
> -cs
>
> On Thu, Jun 28, 2018 at 5:15 PM Ben Browning <bb...@redhat.com> wrote:
>
>> After doing some preliminary poking at this, I believe we'll want to
>> use either a tool like https://github.com/google/oatts to generate a
>> test suite from our Swagger spec OR use swagger-codegen to generate a
>> Scala client from our Swagger spec and try to plug that into the
>> existing WskRest tests.
>>
>> Using the oatts tool doesn't really fit well with the test setup in
>> the existing incubator-openwhisk repo (where the API spec lives)
>> because that generates Node.js tests.
>>
>> So, I'm leaning towards the second option, which is wiring in
>> generation of a Scala client into the gradle build and having the
>> current WskRest test client use this generated Scala client for
>> testing instead of directly invoking URLs.
>>
>> However, last time I played with Scala code generated from Swagger
>> specs it wasn't that usable. So, a bit more experimentation will
>> validate whether this option is viable or if other alternatives need
>> to be considered. I already have a handful of bugs in the API spec
>> that need to be fixed but I'm waiting to fix and push those until I
>> can get some kind of testing wired up to reproduce the bugs and verify
>> the fix.
>>
>> Ben
>>
>>
>> On Thu, Jun 21, 2018 at 3:04 PM, Carlos Santana <cs...@gmail.com>
>> wrote:
>> > Thanks Ben for looking into this, having a good API doc/spec and matching
>> > tests is very need it.
>> >
>> > +1
>> >
>> > -cs
>> >
>> > On Thu, Jun 21, 2018 at 2:25 PM Ben Browning <bb...@redhat.com>
>> wrote:
>> >
>> >> Our Swagger spec
>> >> (
>> >>
>> https://github.com/apache/incubator-openwhisk/blob/92a64c291156a2cd3d6b304babc2a193a46d0699/core/controller/src/main/resources/apiv1swagger.json
>> >> )
>> >> is incomplete and doesn't accurately reflect the actual Controller
>> >> API. It's manually updated without a full test suite which means it's
>> >> easy for changes in the API to happen without the spec getting
>> >> updated.
>> >>
>> >> An accurate Swagger spec will not only better document the OpenWhisk
>> >> API but also allow autogenerated clients in multiple languages to
>> >> supplement or eventually replace some of the existing client
>> >> implementations we have today. It also paves way for future compatible
>> >> server implementations, whether they be rewrites of the existing
>> >> Controller or stub test harnesses to facilitate end-to-end testing on
>> >> a developer's laptop.
>> >>
>> >> As I'm already working with autogenerating code from the Swagger spec
>> >> for other purposes, I'm happy to take the lead on this effort. I'd
>> >> like to take a two-pronged approach for a test suite:
>> >>
>> >> * Generate a server stub from the spec and ensure the wsk CLI can
>> >> communicate with it.
>> >>
>> >> * Generate a client stub from the spec and ensure it can communicate
>> >> with the existing API.
>> >>
>> >> There are a lot of details to figure out from those two statements.
>> >> And, this approach won't guarantee 100% correctness of the spec. The
>> >> only way to do that would be to generate all supported clients and the
>> >> Controller API from the spec. But, this should get us started in the
>> >> right direction.
>> >>
>> >> If anyone's gone down this path before and has some wisdom to share,
>> >> please speak up!
>> >>
>> >> Thanks,
>> >>
>> >> Ben
>> >>
>>
Re: Let's maintain and test our Swagger spec
Posted by Carlos Santana <cs...@gmail.com>.
Thanks Ben for the quick update on this task
-cs
On Thu, Jun 28, 2018 at 5:15 PM Ben Browning <bb...@redhat.com> wrote:
> After doing some preliminary poking at this, I believe we'll want to
> use either a tool like https://github.com/google/oatts to generate a
> test suite from our Swagger spec OR use swagger-codegen to generate a
> Scala client from our Swagger spec and try to plug that into the
> existing WskRest tests.
>
> Using the oatts tool doesn't really fit well with the test setup in
> the existing incubator-openwhisk repo (where the API spec lives)
> because that generates Node.js tests.
>
> So, I'm leaning towards the second option, which is wiring in
> generation of a Scala client into the gradle build and having the
> current WskRest test client use this generated Scala client for
> testing instead of directly invoking URLs.
>
> However, last time I played with Scala code generated from Swagger
> specs it wasn't that usable. So, a bit more experimentation will
> validate whether this option is viable or if other alternatives need
> to be considered. I already have a handful of bugs in the API spec
> that need to be fixed but I'm waiting to fix and push those until I
> can get some kind of testing wired up to reproduce the bugs and verify
> the fix.
>
> Ben
>
>
> On Thu, Jun 21, 2018 at 3:04 PM, Carlos Santana <cs...@gmail.com>
> wrote:
> > Thanks Ben for looking into this, having a good API doc/spec and matching
> > tests is very need it.
> >
> > +1
> >
> > -cs
> >
> > On Thu, Jun 21, 2018 at 2:25 PM Ben Browning <bb...@redhat.com>
> wrote:
> >
> >> Our Swagger spec
> >> (
> >>
> https://github.com/apache/incubator-openwhisk/blob/92a64c291156a2cd3d6b304babc2a193a46d0699/core/controller/src/main/resources/apiv1swagger.json
> >> )
> >> is incomplete and doesn't accurately reflect the actual Controller
> >> API. It's manually updated without a full test suite which means it's
> >> easy for changes in the API to happen without the spec getting
> >> updated.
> >>
> >> An accurate Swagger spec will not only better document the OpenWhisk
> >> API but also allow autogenerated clients in multiple languages to
> >> supplement or eventually replace some of the existing client
> >> implementations we have today. It also paves way for future compatible
> >> server implementations, whether they be rewrites of the existing
> >> Controller or stub test harnesses to facilitate end-to-end testing on
> >> a developer's laptop.
> >>
> >> As I'm already working with autogenerating code from the Swagger spec
> >> for other purposes, I'm happy to take the lead on this effort. I'd
> >> like to take a two-pronged approach for a test suite:
> >>
> >> * Generate a server stub from the spec and ensure the wsk CLI can
> >> communicate with it.
> >>
> >> * Generate a client stub from the spec and ensure it can communicate
> >> with the existing API.
> >>
> >> There are a lot of details to figure out from those two statements.
> >> And, this approach won't guarantee 100% correctness of the spec. The
> >> only way to do that would be to generate all supported clients and the
> >> Controller API from the spec. But, this should get us started in the
> >> right direction.
> >>
> >> If anyone's gone down this path before and has some wisdom to share,
> >> please speak up!
> >>
> >> Thanks,
> >>
> >> Ben
> >>
>
Re: Let's maintain and test our Swagger spec
Posted by Ben Browning <bb...@redhat.com>.
After doing some preliminary poking at this, I believe we'll want to
use either a tool like https://github.com/google/oatts to generate a
test suite from our Swagger spec OR use swagger-codegen to generate a
Scala client from our Swagger spec and try to plug that into the
existing WskRest tests.
Using the oatts tool doesn't really fit well with the test setup in
the existing incubator-openwhisk repo (where the API spec lives)
because that generates Node.js tests.
So, I'm leaning towards the second option, which is wiring in
generation of a Scala client into the gradle build and having the
current WskRest test client use this generated Scala client for
testing instead of directly invoking URLs.
However, last time I played with Scala code generated from Swagger
specs it wasn't that usable. So, a bit more experimentation will
validate whether this option is viable or if other alternatives need
to be considered. I already have a handful of bugs in the API spec
that need to be fixed but I'm waiting to fix and push those until I
can get some kind of testing wired up to reproduce the bugs and verify
the fix.
Ben
On Thu, Jun 21, 2018 at 3:04 PM, Carlos Santana <cs...@gmail.com> wrote:
> Thanks Ben for looking into this, having a good API doc/spec and matching
> tests is very need it.
>
> +1
>
> -cs
>
> On Thu, Jun 21, 2018 at 2:25 PM Ben Browning <bb...@redhat.com> wrote:
>
>> Our Swagger spec
>> (
>> https://github.com/apache/incubator-openwhisk/blob/92a64c291156a2cd3d6b304babc2a193a46d0699/core/controller/src/main/resources/apiv1swagger.json
>> )
>> is incomplete and doesn't accurately reflect the actual Controller
>> API. It's manually updated without a full test suite which means it's
>> easy for changes in the API to happen without the spec getting
>> updated.
>>
>> An accurate Swagger spec will not only better document the OpenWhisk
>> API but also allow autogenerated clients in multiple languages to
>> supplement or eventually replace some of the existing client
>> implementations we have today. It also paves way for future compatible
>> server implementations, whether they be rewrites of the existing
>> Controller or stub test harnesses to facilitate end-to-end testing on
>> a developer's laptop.
>>
>> As I'm already working with autogenerating code from the Swagger spec
>> for other purposes, I'm happy to take the lead on this effort. I'd
>> like to take a two-pronged approach for a test suite:
>>
>> * Generate a server stub from the spec and ensure the wsk CLI can
>> communicate with it.
>>
>> * Generate a client stub from the spec and ensure it can communicate
>> with the existing API.
>>
>> There are a lot of details to figure out from those two statements.
>> And, this approach won't guarantee 100% correctness of the spec. The
>> only way to do that would be to generate all supported clients and the
>> Controller API from the spec. But, this should get us started in the
>> right direction.
>>
>> If anyone's gone down this path before and has some wisdom to share,
>> please speak up!
>>
>> Thanks,
>>
>> Ben
>>
Re: Let's maintain and test our Swagger spec
Posted by Carlos Santana <cs...@gmail.com>.
Thanks Ben for looking into this, having a good API doc/spec and matching
tests is very need it.
+1
-cs
On Thu, Jun 21, 2018 at 2:25 PM Ben Browning <bb...@redhat.com> wrote:
> Our Swagger spec
> (
> https://github.com/apache/incubator-openwhisk/blob/92a64c291156a2cd3d6b304babc2a193a46d0699/core/controller/src/main/resources/apiv1swagger.json
> )
> is incomplete and doesn't accurately reflect the actual Controller
> API. It's manually updated without a full test suite which means it's
> easy for changes in the API to happen without the spec getting
> updated.
>
> An accurate Swagger spec will not only better document the OpenWhisk
> API but also allow autogenerated clients in multiple languages to
> supplement or eventually replace some of the existing client
> implementations we have today. It also paves way for future compatible
> server implementations, whether they be rewrites of the existing
> Controller or stub test harnesses to facilitate end-to-end testing on
> a developer's laptop.
>
> As I'm already working with autogenerating code from the Swagger spec
> for other purposes, I'm happy to take the lead on this effort. I'd
> like to take a two-pronged approach for a test suite:
>
> * Generate a server stub from the spec and ensure the wsk CLI can
> communicate with it.
>
> * Generate a client stub from the spec and ensure it can communicate
> with the existing API.
>
> There are a lot of details to figure out from those two statements.
> And, this approach won't guarantee 100% correctness of the spec. The
> only way to do that would be to generate all supported clients and the
> Controller API from the spec. But, this should get us started in the
> right direction.
>
> If anyone's gone down this path before and has some wisdom to share,
> please speak up!
>
> Thanks,
>
> Ben
>