You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@trafficcontrol.apache.org by Rawlin Peters <ra...@gmail.com> on 2019/02/04 16:41:28 UTC
Traffic Ops Mock for testing
Hey all,
@ocket8888 and I would like to propose implementing a minimal Traffic
Ops "mock" that could be used for integration testing things like
Traffic Ops clients (basically, anything that interacts with the
Traffic Ops API but doesn't need to verify the actual Traffic Ops
implementation, just the external interface) without having to stand
up a real instance of Traffic Ops.
This Mock TO would basically just be a minimal Go HTTP server that
receives requests against the TO API endpoints, unmarshalls JSON
request data into the endpoint's corresponding Go TC struct, marshalls
the Go TC struct back into JSON, and returns it back to the client.
So for most POST/PUT endpoints, the Mock TO would basically just be an
echo server, and for DELETEs it could just return some
statically-defined response that matches the API implementation.
Ideally, the DELETE could just reuse the corresponding DELETE success
response from TO, but we don't know how tightly we should integrate
this Mock TO with the Real TO just yet. For GET endpoints, we could
have Mock TO return some statically-defined JSON response for the
endpoint (be it via literal JSON files read off the filesystem or just
predefined TC Go structs that would marshal into the expected JSON),
but it would be nice to also have the Mock TO be able to dynamically
generate a GET response using reflection against the Go struct. With
dynamic generation of the response, we wouldn't have to maintain
static JSON files against the current version of the API for
everything, but we'd still be able to define static JSON responses for
specific test scenarios.
For example, in order to test ORT, you might want to hit the Mock TO
but define a static response for a configfile endpoint in order to
verify that ORT can create the file from a given response.
In order to test the Python and Java TO clients, we don't necessarily
need to define static JSON responses from Mock TO, since for those we
don't care about the actual values but more so about the structure of
the request and response through the various stages between the client
and the server (marshalling, unmarshalling, etc).
We're currently getting great test coverage of the Go TO client due to
being used for the Go API integration tests, but the Python and Java
TO clients are getting left behind. It would be nice to at least be
able to verify that there haven't been any breaking-API changes that
would actually break TO clients in the repo. For example, if there is
an API regression in a Go-rewrite of a Perl endpoint that breaks the
Python TO client, we can easily find and fix that regression before
the PR is merged. Currently, the only way we can find out if the
Python or Java TO clients have been broken is through normal usage
after breaking-PRs have been merged, which is not great.
A Mock TO would enable us to test all these things, keep the in-repo
TO clients from breaking, and help us prevent breaking the Traffic Ops
API contract accidentally. Mock TO would use the same Go structs as
the Real TO, so Mock TO wouldn't have to maintain its own separate
structs and would automatically stay in sync with changes to the
official TO structs. It would essentially always stay up to date with
the latest version of the structs, which we think is important for
something like this. Otherwise, it could quickly become out of sync
with what it's supposed to mock, and its value as a test tool would be
degraded.
A major advantage of this Mock TO is that it could be used for easy
integration testing where you don't necessarily want to set up and
configure a real instance of Traffic Ops, which is currently a major
pain and honestly a bit overkill for things like just verifying that a
TO client can properly handle requests/responses using the current
version of the API.
Some goals we would have for Mock TO:
- be integrated enough with Real TO so that it "automatically" stays
up to date with Real TO's interface as much as possible
- require very little maintenance (i.e. as new endpoints are added or
existing endpoints are modified, those changes would be picked up
"automatically" in Mock TO)
- virtually no required configuration (probably just a port to listen on)
- require no prerequisite data in order to run and serve mock requests
Please let us know if there's something specific you'd also like to
see in a Mock TO or have other ideas for mocking TO that you'd like to
share. We'd like for Mock TO to be as pain-free and useful as
possible.
- @ocket8888 and Rawlin
Re: [EXTERNAL] Traffic Ops Mock for testing
Posted by Jeremy Mitchell <mi...@gmail.com>.
I agree. I think Swagger is a nice goal for api 2.x. Would probably be
pretty messy integrating with 1.x. In my experience, tools like consistency
and api 1.x is hardly consistent.
On Mon, Feb 4, 2019 at 1:53 PM Rawlin Peters <ra...@gmail.com>
wrote:
> Swagger would be a good tool for this, but the amount of work involved
> in Swagger-izing the existing TO API would mean that we most likely
> wouldn't be able to get a Mock TO for quite a long time, and we could
> benefit from something like this in the meantime.
>
> I agree that we should probably integrate Swagger into TO API 2.0 from
> the start, but I'm not sure we should really spend much effort on
> Swagger-izing the existing TO API since that would mean less capacity
> to work on a better version of the API with Swagger built-in from the
> start.
>
> - Rawlin
>
> On Mon, Feb 4, 2019 at 1:31 PM Fieck, Brennan <Br...@comcast.com>
> wrote:
> >
> > Swagger is meant to be used to design the API which can then spit out a
> spec (as far as I can tell). I don't know how well that integrates with our
> existing API, or even our existing documentation. But supposing it's fairly
> easy to do, it means creating a separate model for every API response we
> have, and that means maintaining all of those models twice. If we're going
> to integrate Swagger I think it's better to wait until API v2 so that we
> can use it to make the model and maintain it exactly once.
> > ________________________________________
> > From: Gray, Jonathan <Jo...@comcast.com>
> > Sent: Monday, February 4, 2019 1:09 PM
> > To: dev@trafficcontrol.apache.org
> > Subject: Re: [EXTERNAL] Traffic Ops Mock for testing
> >
> > What do you think about instead of rolling a separate tool we just
> finish the swagger documentation that's partially integrated. Then you can
> just feed that swagger catalog into any languages stub code generator.
> Bonus is that you get client stubs too for free.
> >
> > Jonathan G
> >
> >
> > On 2/4/19, 1:03 PM, "Fieck, Brennan" <Br...@comcast.com> wrote:
> >
> > Just to be perfectly clear: there should be no database, just a
> single service - an HTTP server - that responds to requests for the Traffic
> Ops API request paths with responses that are structurally and
> syntactically valid for the respective request path.
> > ________________________________________
> > From: Rawlin Peters <ra...@gmail.com>
> > Sent: Monday, February 4, 2019 10:35 AM
> > To: dev@trafficcontrol.apache.org
> > Subject: Re: [EXTERNAL] Traffic Ops Mock for testing
> >
> > The idea is that Mock TO will use the same exact Go structs as TO,
> and
> > by default Mock TO will dynamically generate a response from the
> given
> > endpoint using reflection on the endpoint's corresponding Go struct.
> > No dataset would need to be maintained for Mock TO unless you wanted
> > to write specific test cases in which you provide a predefined static
> > response for Mock TO to return (be it via predefined Go structs or
> > just JSON files). The mechanism for choosing dynamic vs static
> > responses would have to be designed still, but that's the general
> > idea.
> >
> > - Rawlin
> >
> > On Mon, Feb 4, 2019 at 10:25 AM Gray, Jonathan
> > <Jo...@comcast.com> wrote:
> > >
> > > How do you plan to build/maintain the dataset behind such a tool?
> Do you intend to use a fixed TO dataset backed by postgres, or just fixed
> responses to various combinations of parameters.
> > >
> > > Jonathan G
> > >
> > > On 2/4/19, 9:41 AM, "Rawlin Peters" <ra...@gmail.com>
> wrote:
> > >
> > > Hey all,
> > >
> > > @ocket8888 and I would like to propose implementing a minimal
> Traffic
> > > Ops "mock" that could be used for integration testing things
> like
> > > Traffic Ops clients (basically, anything that interacts with
> the
> > > Traffic Ops API but doesn't need to verify the actual Traffic
> Ops
> > > implementation, just the external interface) without having to
> stand
> > > up a real instance of Traffic Ops.
> > >
> > > This Mock TO would basically just be a minimal Go HTTP server
> that
> > > receives requests against the TO API endpoints, unmarshalls
> JSON
> > > request data into the endpoint's corresponding Go TC struct,
> marshalls
> > > the Go TC struct back into JSON, and returns it back to the
> client.
> > >
> > > So for most POST/PUT endpoints, the Mock TO would basically
> just be an
> > > echo server, and for DELETEs it could just return some
> > > statically-defined response that matches the API
> implementation.
> > > Ideally, the DELETE could just reuse the corresponding DELETE
> success
> > > response from TO, but we don't know how tightly we should
> integrate
> > > this Mock TO with the Real TO just yet. For GET endpoints, we
> could
> > > have Mock TO return some statically-defined JSON response for
> the
> > > endpoint (be it via literal JSON files read off the filesystem
> or just
> > > predefined TC Go structs that would marshal into the expected
> JSON),
> > > but it would be nice to also have the Mock TO be able to
> dynamically
> > > generate a GET response using reflection against the Go
> struct. With
> > > dynamic generation of the response, we wouldn't have to
> maintain
> > > static JSON files against the current version of the API for
> > > everything, but we'd still be able to define static JSON
> responses for
> > > specific test scenarios.
> > >
> > > For example, in order to test ORT, you might want to hit the
> Mock TO
> > > but define a static response for a configfile endpoint in
> order to
> > > verify that ORT can create the file from a given response.
> > >
> > > In order to test the Python and Java TO clients, we don't
> necessarily
> > > need to define static JSON responses from Mock TO, since for
> those we
> > > don't care about the actual values but more so about the
> structure of
> > > the request and response through the various stages between
> the client
> > > and the server (marshalling, unmarshalling, etc).
> > >
> > > We're currently getting great test coverage of the Go TO
> client due to
> > > being used for the Go API integration tests, but the Python
> and Java
> > > TO clients are getting left behind. It would be nice to at
> least be
> > > able to verify that there haven't been any breaking-API
> changes that
> > > would actually break TO clients in the repo. For example, if
> there is
> > > an API regression in a Go-rewrite of a Perl endpoint that
> breaks the
> > > Python TO client, we can easily find and fix that regression
> before
> > > the PR is merged. Currently, the only way we can find out if
> the
> > > Python or Java TO clients have been broken is through normal
> usage
> > > after breaking-PRs have been merged, which is not great.
> > >
> > > A Mock TO would enable us to test all these things, keep the
> in-repo
> > > TO clients from breaking, and help us prevent breaking the
> Traffic Ops
> > > API contract accidentally. Mock TO would use the same Go
> structs as
> > > the Real TO, so Mock TO wouldn't have to maintain its own
> separate
> > > structs and would automatically stay in sync with changes to
> the
> > > official TO structs. It would essentially always stay up to
> date with
> > > the latest version of the structs, which we think is important
> for
> > > something like this. Otherwise, it could quickly become out of
> sync
> > > with what it's supposed to mock, and its value as a test tool
> would be
> > > degraded.
> > >
> > > A major advantage of this Mock TO is that it could be used for
> easy
> > > integration testing where you don't necessarily want to set up
> and
> > > configure a real instance of Traffic Ops, which is currently a
> major
> > > pain and honestly a bit overkill for things like just
> verifying that a
> > > TO client can properly handle requests/responses using the
> current
> > > version of the API.
> > >
> > > Some goals we would have for Mock TO:
> > > - be integrated enough with Real TO so that it "automatically"
> stays
> > > up to date with Real TO's interface as much as possible
> > > - require very little maintenance (i.e. as new endpoints are
> added or
> > > existing endpoints are modified, those changes would be picked
> up
> > > "automatically" in Mock TO)
> > > - virtually no required configuration (probably just a port to
> listen on)
> > > - require no prerequisite data in order to run and serve mock
> requests
> > >
> > > Please let us know if there's something specific you'd also
> like to
> > > see in a Mock TO or have other ideas for mocking TO that you'd
> like to
> > > share. We'd like for Mock TO to be as pain-free and useful as
> > > possible.
> > >
> > > - @ocket8888 and Rawlin
> > >
> > >
> >
> >
>
Re: [EXTERNAL] Traffic Ops Mock for testing
Posted by Rawlin Peters <ra...@gmail.com>.
Swagger would be a good tool for this, but the amount of work involved
in Swagger-izing the existing TO API would mean that we most likely
wouldn't be able to get a Mock TO for quite a long time, and we could
benefit from something like this in the meantime.
I agree that we should probably integrate Swagger into TO API 2.0 from
the start, but I'm not sure we should really spend much effort on
Swagger-izing the existing TO API since that would mean less capacity
to work on a better version of the API with Swagger built-in from the
start.
- Rawlin
On Mon, Feb 4, 2019 at 1:31 PM Fieck, Brennan <Br...@comcast.com> wrote:
>
> Swagger is meant to be used to design the API which can then spit out a spec (as far as I can tell). I don't know how well that integrates with our existing API, or even our existing documentation. But supposing it's fairly easy to do, it means creating a separate model for every API response we have, and that means maintaining all of those models twice. If we're going to integrate Swagger I think it's better to wait until API v2 so that we can use it to make the model and maintain it exactly once.
> ________________________________________
> From: Gray, Jonathan <Jo...@comcast.com>
> Sent: Monday, February 4, 2019 1:09 PM
> To: dev@trafficcontrol.apache.org
> Subject: Re: [EXTERNAL] Traffic Ops Mock for testing
>
> What do you think about instead of rolling a separate tool we just finish the swagger documentation that's partially integrated. Then you can just feed that swagger catalog into any languages stub code generator. Bonus is that you get client stubs too for free.
>
> Jonathan G
>
>
> On 2/4/19, 1:03 PM, "Fieck, Brennan" <Br...@comcast.com> wrote:
>
> Just to be perfectly clear: there should be no database, just a single service - an HTTP server - that responds to requests for the Traffic Ops API request paths with responses that are structurally and syntactically valid for the respective request path.
> ________________________________________
> From: Rawlin Peters <ra...@gmail.com>
> Sent: Monday, February 4, 2019 10:35 AM
> To: dev@trafficcontrol.apache.org
> Subject: Re: [EXTERNAL] Traffic Ops Mock for testing
>
> The idea is that Mock TO will use the same exact Go structs as TO, and
> by default Mock TO will dynamically generate a response from the given
> endpoint using reflection on the endpoint's corresponding Go struct.
> No dataset would need to be maintained for Mock TO unless you wanted
> to write specific test cases in which you provide a predefined static
> response for Mock TO to return (be it via predefined Go structs or
> just JSON files). The mechanism for choosing dynamic vs static
> responses would have to be designed still, but that's the general
> idea.
>
> - Rawlin
>
> On Mon, Feb 4, 2019 at 10:25 AM Gray, Jonathan
> <Jo...@comcast.com> wrote:
> >
> > How do you plan to build/maintain the dataset behind such a tool? Do you intend to use a fixed TO dataset backed by postgres, or just fixed responses to various combinations of parameters.
> >
> > Jonathan G
> >
> > On 2/4/19, 9:41 AM, "Rawlin Peters" <ra...@gmail.com> wrote:
> >
> > Hey all,
> >
> > @ocket8888 and I would like to propose implementing a minimal Traffic
> > Ops "mock" that could be used for integration testing things like
> > Traffic Ops clients (basically, anything that interacts with the
> > Traffic Ops API but doesn't need to verify the actual Traffic Ops
> > implementation, just the external interface) without having to stand
> > up a real instance of Traffic Ops.
> >
> > This Mock TO would basically just be a minimal Go HTTP server that
> > receives requests against the TO API endpoints, unmarshalls JSON
> > request data into the endpoint's corresponding Go TC struct, marshalls
> > the Go TC struct back into JSON, and returns it back to the client.
> >
> > So for most POST/PUT endpoints, the Mock TO would basically just be an
> > echo server, and for DELETEs it could just return some
> > statically-defined response that matches the API implementation.
> > Ideally, the DELETE could just reuse the corresponding DELETE success
> > response from TO, but we don't know how tightly we should integrate
> > this Mock TO with the Real TO just yet. For GET endpoints, we could
> > have Mock TO return some statically-defined JSON response for the
> > endpoint (be it via literal JSON files read off the filesystem or just
> > predefined TC Go structs that would marshal into the expected JSON),
> > but it would be nice to also have the Mock TO be able to dynamically
> > generate a GET response using reflection against the Go struct. With
> > dynamic generation of the response, we wouldn't have to maintain
> > static JSON files against the current version of the API for
> > everything, but we'd still be able to define static JSON responses for
> > specific test scenarios.
> >
> > For example, in order to test ORT, you might want to hit the Mock TO
> > but define a static response for a configfile endpoint in order to
> > verify that ORT can create the file from a given response.
> >
> > In order to test the Python and Java TO clients, we don't necessarily
> > need to define static JSON responses from Mock TO, since for those we
> > don't care about the actual values but more so about the structure of
> > the request and response through the various stages between the client
> > and the server (marshalling, unmarshalling, etc).
> >
> > We're currently getting great test coverage of the Go TO client due to
> > being used for the Go API integration tests, but the Python and Java
> > TO clients are getting left behind. It would be nice to at least be
> > able to verify that there haven't been any breaking-API changes that
> > would actually break TO clients in the repo. For example, if there is
> > an API regression in a Go-rewrite of a Perl endpoint that breaks the
> > Python TO client, we can easily find and fix that regression before
> > the PR is merged. Currently, the only way we can find out if the
> > Python or Java TO clients have been broken is through normal usage
> > after breaking-PRs have been merged, which is not great.
> >
> > A Mock TO would enable us to test all these things, keep the in-repo
> > TO clients from breaking, and help us prevent breaking the Traffic Ops
> > API contract accidentally. Mock TO would use the same Go structs as
> > the Real TO, so Mock TO wouldn't have to maintain its own separate
> > structs and would automatically stay in sync with changes to the
> > official TO structs. It would essentially always stay up to date with
> > the latest version of the structs, which we think is important for
> > something like this. Otherwise, it could quickly become out of sync
> > with what it's supposed to mock, and its value as a test tool would be
> > degraded.
> >
> > A major advantage of this Mock TO is that it could be used for easy
> > integration testing where you don't necessarily want to set up and
> > configure a real instance of Traffic Ops, which is currently a major
> > pain and honestly a bit overkill for things like just verifying that a
> > TO client can properly handle requests/responses using the current
> > version of the API.
> >
> > Some goals we would have for Mock TO:
> > - be integrated enough with Real TO so that it "automatically" stays
> > up to date with Real TO's interface as much as possible
> > - require very little maintenance (i.e. as new endpoints are added or
> > existing endpoints are modified, those changes would be picked up
> > "automatically" in Mock TO)
> > - virtually no required configuration (probably just a port to listen on)
> > - require no prerequisite data in order to run and serve mock requests
> >
> > Please let us know if there's something specific you'd also like to
> > see in a Mock TO or have other ideas for mocking TO that you'd like to
> > share. We'd like for Mock TO to be as pain-free and useful as
> > possible.
> >
> > - @ocket8888 and Rawlin
> >
> >
>
>
Re: [EXTERNAL] Traffic Ops Mock for testing
Posted by "Fieck, Brennan" <Br...@comcast.com>.
Swagger is meant to be used to design the API which can then spit out a spec (as far as I can tell). I don't know how well that integrates with our existing API, or even our existing documentation. But supposing it's fairly easy to do, it means creating a separate model for every API response we have, and that means maintaining all of those models twice. If we're going to integrate Swagger I think it's better to wait until API v2 so that we can use it to make the model and maintain it exactly once.
________________________________________
From: Gray, Jonathan <Jo...@comcast.com>
Sent: Monday, February 4, 2019 1:09 PM
To: dev@trafficcontrol.apache.org
Subject: Re: [EXTERNAL] Traffic Ops Mock for testing
What do you think about instead of rolling a separate tool we just finish the swagger documentation that's partially integrated. Then you can just feed that swagger catalog into any languages stub code generator. Bonus is that you get client stubs too for free.
Jonathan G
On 2/4/19, 1:03 PM, "Fieck, Brennan" <Br...@comcast.com> wrote:
Just to be perfectly clear: there should be no database, just a single service - an HTTP server - that responds to requests for the Traffic Ops API request paths with responses that are structurally and syntactically valid for the respective request path.
________________________________________
From: Rawlin Peters <ra...@gmail.com>
Sent: Monday, February 4, 2019 10:35 AM
To: dev@trafficcontrol.apache.org
Subject: Re: [EXTERNAL] Traffic Ops Mock for testing
The idea is that Mock TO will use the same exact Go structs as TO, and
by default Mock TO will dynamically generate a response from the given
endpoint using reflection on the endpoint's corresponding Go struct.
No dataset would need to be maintained for Mock TO unless you wanted
to write specific test cases in which you provide a predefined static
response for Mock TO to return (be it via predefined Go structs or
just JSON files). The mechanism for choosing dynamic vs static
responses would have to be designed still, but that's the general
idea.
- Rawlin
On Mon, Feb 4, 2019 at 10:25 AM Gray, Jonathan
<Jo...@comcast.com> wrote:
>
> How do you plan to build/maintain the dataset behind such a tool? Do you intend to use a fixed TO dataset backed by postgres, or just fixed responses to various combinations of parameters.
>
> Jonathan G
>
> On 2/4/19, 9:41 AM, "Rawlin Peters" <ra...@gmail.com> wrote:
>
> Hey all,
>
> @ocket8888 and I would like to propose implementing a minimal Traffic
> Ops "mock" that could be used for integration testing things like
> Traffic Ops clients (basically, anything that interacts with the
> Traffic Ops API but doesn't need to verify the actual Traffic Ops
> implementation, just the external interface) without having to stand
> up a real instance of Traffic Ops.
>
> This Mock TO would basically just be a minimal Go HTTP server that
> receives requests against the TO API endpoints, unmarshalls JSON
> request data into the endpoint's corresponding Go TC struct, marshalls
> the Go TC struct back into JSON, and returns it back to the client.
>
> So for most POST/PUT endpoints, the Mock TO would basically just be an
> echo server, and for DELETEs it could just return some
> statically-defined response that matches the API implementation.
> Ideally, the DELETE could just reuse the corresponding DELETE success
> response from TO, but we don't know how tightly we should integrate
> this Mock TO with the Real TO just yet. For GET endpoints, we could
> have Mock TO return some statically-defined JSON response for the
> endpoint (be it via literal JSON files read off the filesystem or just
> predefined TC Go structs that would marshal into the expected JSON),
> but it would be nice to also have the Mock TO be able to dynamically
> generate a GET response using reflection against the Go struct. With
> dynamic generation of the response, we wouldn't have to maintain
> static JSON files against the current version of the API for
> everything, but we'd still be able to define static JSON responses for
> specific test scenarios.
>
> For example, in order to test ORT, you might want to hit the Mock TO
> but define a static response for a configfile endpoint in order to
> verify that ORT can create the file from a given response.
>
> In order to test the Python and Java TO clients, we don't necessarily
> need to define static JSON responses from Mock TO, since for those we
> don't care about the actual values but more so about the structure of
> the request and response through the various stages between the client
> and the server (marshalling, unmarshalling, etc).
>
> We're currently getting great test coverage of the Go TO client due to
> being used for the Go API integration tests, but the Python and Java
> TO clients are getting left behind. It would be nice to at least be
> able to verify that there haven't been any breaking-API changes that
> would actually break TO clients in the repo. For example, if there is
> an API regression in a Go-rewrite of a Perl endpoint that breaks the
> Python TO client, we can easily find and fix that regression before
> the PR is merged. Currently, the only way we can find out if the
> Python or Java TO clients have been broken is through normal usage
> after breaking-PRs have been merged, which is not great.
>
> A Mock TO would enable us to test all these things, keep the in-repo
> TO clients from breaking, and help us prevent breaking the Traffic Ops
> API contract accidentally. Mock TO would use the same Go structs as
> the Real TO, so Mock TO wouldn't have to maintain its own separate
> structs and would automatically stay in sync with changes to the
> official TO structs. It would essentially always stay up to date with
> the latest version of the structs, which we think is important for
> something like this. Otherwise, it could quickly become out of sync
> with what it's supposed to mock, and its value as a test tool would be
> degraded.
>
> A major advantage of this Mock TO is that it could be used for easy
> integration testing where you don't necessarily want to set up and
> configure a real instance of Traffic Ops, which is currently a major
> pain and honestly a bit overkill for things like just verifying that a
> TO client can properly handle requests/responses using the current
> version of the API.
>
> Some goals we would have for Mock TO:
> - be integrated enough with Real TO so that it "automatically" stays
> up to date with Real TO's interface as much as possible
> - require very little maintenance (i.e. as new endpoints are added or
> existing endpoints are modified, those changes would be picked up
> "automatically" in Mock TO)
> - virtually no required configuration (probably just a port to listen on)
> - require no prerequisite data in order to run and serve mock requests
>
> Please let us know if there's something specific you'd also like to
> see in a Mock TO or have other ideas for mocking TO that you'd like to
> share. We'd like for Mock TO to be as pain-free and useful as
> possible.
>
> - @ocket8888 and Rawlin
>
>
Re: [EXTERNAL] Traffic Ops Mock for testing
Posted by "Gray, Jonathan" <Jo...@comcast.com>.
What do you think about instead of rolling a separate tool we just finish the swagger documentation that's partially integrated. Then you can just feed that swagger catalog into any languages stub code generator. Bonus is that you get client stubs too for free.
Jonathan G
On 2/4/19, 1:03 PM, "Fieck, Brennan" <Br...@comcast.com> wrote:
Just to be perfectly clear: there should be no database, just a single service - an HTTP server - that responds to requests for the Traffic Ops API request paths with responses that are structurally and syntactically valid for the respective request path.
________________________________________
From: Rawlin Peters <ra...@gmail.com>
Sent: Monday, February 4, 2019 10:35 AM
To: dev@trafficcontrol.apache.org
Subject: Re: [EXTERNAL] Traffic Ops Mock for testing
The idea is that Mock TO will use the same exact Go structs as TO, and
by default Mock TO will dynamically generate a response from the given
endpoint using reflection on the endpoint's corresponding Go struct.
No dataset would need to be maintained for Mock TO unless you wanted
to write specific test cases in which you provide a predefined static
response for Mock TO to return (be it via predefined Go structs or
just JSON files). The mechanism for choosing dynamic vs static
responses would have to be designed still, but that's the general
idea.
- Rawlin
On Mon, Feb 4, 2019 at 10:25 AM Gray, Jonathan
<Jo...@comcast.com> wrote:
>
> How do you plan to build/maintain the dataset behind such a tool? Do you intend to use a fixed TO dataset backed by postgres, or just fixed responses to various combinations of parameters.
>
> Jonathan G
>
> On 2/4/19, 9:41 AM, "Rawlin Peters" <ra...@gmail.com> wrote:
>
> Hey all,
>
> @ocket8888 and I would like to propose implementing a minimal Traffic
> Ops "mock" that could be used for integration testing things like
> Traffic Ops clients (basically, anything that interacts with the
> Traffic Ops API but doesn't need to verify the actual Traffic Ops
> implementation, just the external interface) without having to stand
> up a real instance of Traffic Ops.
>
> This Mock TO would basically just be a minimal Go HTTP server that
> receives requests against the TO API endpoints, unmarshalls JSON
> request data into the endpoint's corresponding Go TC struct, marshalls
> the Go TC struct back into JSON, and returns it back to the client.
>
> So for most POST/PUT endpoints, the Mock TO would basically just be an
> echo server, and for DELETEs it could just return some
> statically-defined response that matches the API implementation.
> Ideally, the DELETE could just reuse the corresponding DELETE success
> response from TO, but we don't know how tightly we should integrate
> this Mock TO with the Real TO just yet. For GET endpoints, we could
> have Mock TO return some statically-defined JSON response for the
> endpoint (be it via literal JSON files read off the filesystem or just
> predefined TC Go structs that would marshal into the expected JSON),
> but it would be nice to also have the Mock TO be able to dynamically
> generate a GET response using reflection against the Go struct. With
> dynamic generation of the response, we wouldn't have to maintain
> static JSON files against the current version of the API for
> everything, but we'd still be able to define static JSON responses for
> specific test scenarios.
>
> For example, in order to test ORT, you might want to hit the Mock TO
> but define a static response for a configfile endpoint in order to
> verify that ORT can create the file from a given response.
>
> In order to test the Python and Java TO clients, we don't necessarily
> need to define static JSON responses from Mock TO, since for those we
> don't care about the actual values but more so about the structure of
> the request and response through the various stages between the client
> and the server (marshalling, unmarshalling, etc).
>
> We're currently getting great test coverage of the Go TO client due to
> being used for the Go API integration tests, but the Python and Java
> TO clients are getting left behind. It would be nice to at least be
> able to verify that there haven't been any breaking-API changes that
> would actually break TO clients in the repo. For example, if there is
> an API regression in a Go-rewrite of a Perl endpoint that breaks the
> Python TO client, we can easily find and fix that regression before
> the PR is merged. Currently, the only way we can find out if the
> Python or Java TO clients have been broken is through normal usage
> after breaking-PRs have been merged, which is not great.
>
> A Mock TO would enable us to test all these things, keep the in-repo
> TO clients from breaking, and help us prevent breaking the Traffic Ops
> API contract accidentally. Mock TO would use the same Go structs as
> the Real TO, so Mock TO wouldn't have to maintain its own separate
> structs and would automatically stay in sync with changes to the
> official TO structs. It would essentially always stay up to date with
> the latest version of the structs, which we think is important for
> something like this. Otherwise, it could quickly become out of sync
> with what it's supposed to mock, and its value as a test tool would be
> degraded.
>
> A major advantage of this Mock TO is that it could be used for easy
> integration testing where you don't necessarily want to set up and
> configure a real instance of Traffic Ops, which is currently a major
> pain and honestly a bit overkill for things like just verifying that a
> TO client can properly handle requests/responses using the current
> version of the API.
>
> Some goals we would have for Mock TO:
> - be integrated enough with Real TO so that it "automatically" stays
> up to date with Real TO's interface as much as possible
> - require very little maintenance (i.e. as new endpoints are added or
> existing endpoints are modified, those changes would be picked up
> "automatically" in Mock TO)
> - virtually no required configuration (probably just a port to listen on)
> - require no prerequisite data in order to run and serve mock requests
>
> Please let us know if there's something specific you'd also like to
> see in a Mock TO or have other ideas for mocking TO that you'd like to
> share. We'd like for Mock TO to be as pain-free and useful as
> possible.
>
> - @ocket8888 and Rawlin
>
>
Re: [EXTERNAL] Traffic Ops Mock for testing
Posted by "Fieck, Brennan" <Br...@comcast.com>.
Just to be perfectly clear: there should be no database, just a single service - an HTTP server - that responds to requests for the Traffic Ops API request paths with responses that are structurally and syntactically valid for the respective request path.
________________________________________
From: Rawlin Peters <ra...@gmail.com>
Sent: Monday, February 4, 2019 10:35 AM
To: dev@trafficcontrol.apache.org
Subject: Re: [EXTERNAL] Traffic Ops Mock for testing
The idea is that Mock TO will use the same exact Go structs as TO, and
by default Mock TO will dynamically generate a response from the given
endpoint using reflection on the endpoint's corresponding Go struct.
No dataset would need to be maintained for Mock TO unless you wanted
to write specific test cases in which you provide a predefined static
response for Mock TO to return (be it via predefined Go structs or
just JSON files). The mechanism for choosing dynamic vs static
responses would have to be designed still, but that's the general
idea.
- Rawlin
On Mon, Feb 4, 2019 at 10:25 AM Gray, Jonathan
<Jo...@comcast.com> wrote:
>
> How do you plan to build/maintain the dataset behind such a tool? Do you intend to use a fixed TO dataset backed by postgres, or just fixed responses to various combinations of parameters.
>
> Jonathan G
>
> On 2/4/19, 9:41 AM, "Rawlin Peters" <ra...@gmail.com> wrote:
>
> Hey all,
>
> @ocket8888 and I would like to propose implementing a minimal Traffic
> Ops "mock" that could be used for integration testing things like
> Traffic Ops clients (basically, anything that interacts with the
> Traffic Ops API but doesn't need to verify the actual Traffic Ops
> implementation, just the external interface) without having to stand
> up a real instance of Traffic Ops.
>
> This Mock TO would basically just be a minimal Go HTTP server that
> receives requests against the TO API endpoints, unmarshalls JSON
> request data into the endpoint's corresponding Go TC struct, marshalls
> the Go TC struct back into JSON, and returns it back to the client.
>
> So for most POST/PUT endpoints, the Mock TO would basically just be an
> echo server, and for DELETEs it could just return some
> statically-defined response that matches the API implementation.
> Ideally, the DELETE could just reuse the corresponding DELETE success
> response from TO, but we don't know how tightly we should integrate
> this Mock TO with the Real TO just yet. For GET endpoints, we could
> have Mock TO return some statically-defined JSON response for the
> endpoint (be it via literal JSON files read off the filesystem or just
> predefined TC Go structs that would marshal into the expected JSON),
> but it would be nice to also have the Mock TO be able to dynamically
> generate a GET response using reflection against the Go struct. With
> dynamic generation of the response, we wouldn't have to maintain
> static JSON files against the current version of the API for
> everything, but we'd still be able to define static JSON responses for
> specific test scenarios.
>
> For example, in order to test ORT, you might want to hit the Mock TO
> but define a static response for a configfile endpoint in order to
> verify that ORT can create the file from a given response.
>
> In order to test the Python and Java TO clients, we don't necessarily
> need to define static JSON responses from Mock TO, since for those we
> don't care about the actual values but more so about the structure of
> the request and response through the various stages between the client
> and the server (marshalling, unmarshalling, etc).
>
> We're currently getting great test coverage of the Go TO client due to
> being used for the Go API integration tests, but the Python and Java
> TO clients are getting left behind. It would be nice to at least be
> able to verify that there haven't been any breaking-API changes that
> would actually break TO clients in the repo. For example, if there is
> an API regression in a Go-rewrite of a Perl endpoint that breaks the
> Python TO client, we can easily find and fix that regression before
> the PR is merged. Currently, the only way we can find out if the
> Python or Java TO clients have been broken is through normal usage
> after breaking-PRs have been merged, which is not great.
>
> A Mock TO would enable us to test all these things, keep the in-repo
> TO clients from breaking, and help us prevent breaking the Traffic Ops
> API contract accidentally. Mock TO would use the same Go structs as
> the Real TO, so Mock TO wouldn't have to maintain its own separate
> structs and would automatically stay in sync with changes to the
> official TO structs. It would essentially always stay up to date with
> the latest version of the structs, which we think is important for
> something like this. Otherwise, it could quickly become out of sync
> with what it's supposed to mock, and its value as a test tool would be
> degraded.
>
> A major advantage of this Mock TO is that it could be used for easy
> integration testing where you don't necessarily want to set up and
> configure a real instance of Traffic Ops, which is currently a major
> pain and honestly a bit overkill for things like just verifying that a
> TO client can properly handle requests/responses using the current
> version of the API.
>
> Some goals we would have for Mock TO:
> - be integrated enough with Real TO so that it "automatically" stays
> up to date with Real TO's interface as much as possible
> - require very little maintenance (i.e. as new endpoints are added or
> existing endpoints are modified, those changes would be picked up
> "automatically" in Mock TO)
> - virtually no required configuration (probably just a port to listen on)
> - require no prerequisite data in order to run and serve mock requests
>
> Please let us know if there's something specific you'd also like to
> see in a Mock TO or have other ideas for mocking TO that you'd like to
> share. We'd like for Mock TO to be as pain-free and useful as
> possible.
>
> - @ocket8888 and Rawlin
>
>
Re: [EXTERNAL] Traffic Ops Mock for testing
Posted by Rawlin Peters <ra...@gmail.com>.
The idea is that Mock TO will use the same exact Go structs as TO, and
by default Mock TO will dynamically generate a response from the given
endpoint using reflection on the endpoint's corresponding Go struct.
No dataset would need to be maintained for Mock TO unless you wanted
to write specific test cases in which you provide a predefined static
response for Mock TO to return (be it via predefined Go structs or
just JSON files). The mechanism for choosing dynamic vs static
responses would have to be designed still, but that's the general
idea.
- Rawlin
On Mon, Feb 4, 2019 at 10:25 AM Gray, Jonathan
<Jo...@comcast.com> wrote:
>
> How do you plan to build/maintain the dataset behind such a tool? Do you intend to use a fixed TO dataset backed by postgres, or just fixed responses to various combinations of parameters.
>
> Jonathan G
>
> On 2/4/19, 9:41 AM, "Rawlin Peters" <ra...@gmail.com> wrote:
>
> Hey all,
>
> @ocket8888 and I would like to propose implementing a minimal Traffic
> Ops "mock" that could be used for integration testing things like
> Traffic Ops clients (basically, anything that interacts with the
> Traffic Ops API but doesn't need to verify the actual Traffic Ops
> implementation, just the external interface) without having to stand
> up a real instance of Traffic Ops.
>
> This Mock TO would basically just be a minimal Go HTTP server that
> receives requests against the TO API endpoints, unmarshalls JSON
> request data into the endpoint's corresponding Go TC struct, marshalls
> the Go TC struct back into JSON, and returns it back to the client.
>
> So for most POST/PUT endpoints, the Mock TO would basically just be an
> echo server, and for DELETEs it could just return some
> statically-defined response that matches the API implementation.
> Ideally, the DELETE could just reuse the corresponding DELETE success
> response from TO, but we don't know how tightly we should integrate
> this Mock TO with the Real TO just yet. For GET endpoints, we could
> have Mock TO return some statically-defined JSON response for the
> endpoint (be it via literal JSON files read off the filesystem or just
> predefined TC Go structs that would marshal into the expected JSON),
> but it would be nice to also have the Mock TO be able to dynamically
> generate a GET response using reflection against the Go struct. With
> dynamic generation of the response, we wouldn't have to maintain
> static JSON files against the current version of the API for
> everything, but we'd still be able to define static JSON responses for
> specific test scenarios.
>
> For example, in order to test ORT, you might want to hit the Mock TO
> but define a static response for a configfile endpoint in order to
> verify that ORT can create the file from a given response.
>
> In order to test the Python and Java TO clients, we don't necessarily
> need to define static JSON responses from Mock TO, since for those we
> don't care about the actual values but more so about the structure of
> the request and response through the various stages between the client
> and the server (marshalling, unmarshalling, etc).
>
> We're currently getting great test coverage of the Go TO client due to
> being used for the Go API integration tests, but the Python and Java
> TO clients are getting left behind. It would be nice to at least be
> able to verify that there haven't been any breaking-API changes that
> would actually break TO clients in the repo. For example, if there is
> an API regression in a Go-rewrite of a Perl endpoint that breaks the
> Python TO client, we can easily find and fix that regression before
> the PR is merged. Currently, the only way we can find out if the
> Python or Java TO clients have been broken is through normal usage
> after breaking-PRs have been merged, which is not great.
>
> A Mock TO would enable us to test all these things, keep the in-repo
> TO clients from breaking, and help us prevent breaking the Traffic Ops
> API contract accidentally. Mock TO would use the same Go structs as
> the Real TO, so Mock TO wouldn't have to maintain its own separate
> structs and would automatically stay in sync with changes to the
> official TO structs. It would essentially always stay up to date with
> the latest version of the structs, which we think is important for
> something like this. Otherwise, it could quickly become out of sync
> with what it's supposed to mock, and its value as a test tool would be
> degraded.
>
> A major advantage of this Mock TO is that it could be used for easy
> integration testing where you don't necessarily want to set up and
> configure a real instance of Traffic Ops, which is currently a major
> pain and honestly a bit overkill for things like just verifying that a
> TO client can properly handle requests/responses using the current
> version of the API.
>
> Some goals we would have for Mock TO:
> - be integrated enough with Real TO so that it "automatically" stays
> up to date with Real TO's interface as much as possible
> - require very little maintenance (i.e. as new endpoints are added or
> existing endpoints are modified, those changes would be picked up
> "automatically" in Mock TO)
> - virtually no required configuration (probably just a port to listen on)
> - require no prerequisite data in order to run and serve mock requests
>
> Please let us know if there's something specific you'd also like to
> see in a Mock TO or have other ideas for mocking TO that you'd like to
> share. We'd like for Mock TO to be as pain-free and useful as
> possible.
>
> - @ocket8888 and Rawlin
>
>
Re: [EXTERNAL] Traffic Ops Mock for testing
Posted by "Gray, Jonathan" <Jo...@comcast.com>.
How do you plan to build/maintain the dataset behind such a tool? Do you intend to use a fixed TO dataset backed by postgres, or just fixed responses to various combinations of parameters.
Jonathan G
On 2/4/19, 9:41 AM, "Rawlin Peters" <ra...@gmail.com> wrote:
Hey all,
@ocket8888 and I would like to propose implementing a minimal Traffic
Ops "mock" that could be used for integration testing things like
Traffic Ops clients (basically, anything that interacts with the
Traffic Ops API but doesn't need to verify the actual Traffic Ops
implementation, just the external interface) without having to stand
up a real instance of Traffic Ops.
This Mock TO would basically just be a minimal Go HTTP server that
receives requests against the TO API endpoints, unmarshalls JSON
request data into the endpoint's corresponding Go TC struct, marshalls
the Go TC struct back into JSON, and returns it back to the client.
So for most POST/PUT endpoints, the Mock TO would basically just be an
echo server, and for DELETEs it could just return some
statically-defined response that matches the API implementation.
Ideally, the DELETE could just reuse the corresponding DELETE success
response from TO, but we don't know how tightly we should integrate
this Mock TO with the Real TO just yet. For GET endpoints, we could
have Mock TO return some statically-defined JSON response for the
endpoint (be it via literal JSON files read off the filesystem or just
predefined TC Go structs that would marshal into the expected JSON),
but it would be nice to also have the Mock TO be able to dynamically
generate a GET response using reflection against the Go struct. With
dynamic generation of the response, we wouldn't have to maintain
static JSON files against the current version of the API for
everything, but we'd still be able to define static JSON responses for
specific test scenarios.
For example, in order to test ORT, you might want to hit the Mock TO
but define a static response for a configfile endpoint in order to
verify that ORT can create the file from a given response.
In order to test the Python and Java TO clients, we don't necessarily
need to define static JSON responses from Mock TO, since for those we
don't care about the actual values but more so about the structure of
the request and response through the various stages between the client
and the server (marshalling, unmarshalling, etc).
We're currently getting great test coverage of the Go TO client due to
being used for the Go API integration tests, but the Python and Java
TO clients are getting left behind. It would be nice to at least be
able to verify that there haven't been any breaking-API changes that
would actually break TO clients in the repo. For example, if there is
an API regression in a Go-rewrite of a Perl endpoint that breaks the
Python TO client, we can easily find and fix that regression before
the PR is merged. Currently, the only way we can find out if the
Python or Java TO clients have been broken is through normal usage
after breaking-PRs have been merged, which is not great.
A Mock TO would enable us to test all these things, keep the in-repo
TO clients from breaking, and help us prevent breaking the Traffic Ops
API contract accidentally. Mock TO would use the same Go structs as
the Real TO, so Mock TO wouldn't have to maintain its own separate
structs and would automatically stay in sync with changes to the
official TO structs. It would essentially always stay up to date with
the latest version of the structs, which we think is important for
something like this. Otherwise, it could quickly become out of sync
with what it's supposed to mock, and its value as a test tool would be
degraded.
A major advantage of this Mock TO is that it could be used for easy
integration testing where you don't necessarily want to set up and
configure a real instance of Traffic Ops, which is currently a major
pain and honestly a bit overkill for things like just verifying that a
TO client can properly handle requests/responses using the current
version of the API.
Some goals we would have for Mock TO:
- be integrated enough with Real TO so that it "automatically" stays
up to date with Real TO's interface as much as possible
- require very little maintenance (i.e. as new endpoints are added or
existing endpoints are modified, those changes would be picked up
"automatically" in Mock TO)
- virtually no required configuration (probably just a port to listen on)
- require no prerequisite data in order to run and serve mock requests
Please let us know if there's something specific you'd also like to
see in a Mock TO or have other ideas for mocking TO that you'd like to
share. We'd like for Mock TO to be as pain-free and useful as
possible.
- @ocket8888 and Rawlin