You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@trafficcontrol.apache.org by Dewayne Richardson <de...@gmail.com> on 2017/08/09 18:55:52 UTC
Traffic Ops Golang Rewrite
Sorry for the TL;DR, but a lot of information needed to be conveyed. So,
based upon the TO Rewrite discussions Rob Butts and I have been working on
a Golang proxy (Mark Torluemke, affectionately calls "the Vampire Proxy")
that initially implements the */monitoring* endpoint to lay down the
foundation for rewriting more Traffic Ops endpoints in Go. The goal is to
do a straight rewrite in Go (that implements any old API's as well as any
new ones that follows the /api/1.2 format). The intent is to make this
proxy 100% backward compatible (including any HTTP header requirements) to
keep the existing TO API clients from breaking.
This PR is significant because when postinstall runs (after this PR is
merged) it switches the ports according to the following:
*TO Port Change Overview*
Port *443* will be owned by Golang proxy
Port *60443* will now be owned by the Mojolicious/Perl Hypnotoad service
See */opt/traffic_ops/app/conf/cdn.conf* for a new property
*traffic_ops_golang_port
=> '443'*.
*Important Operational Changes:*
*traffic_ops service*
The Golang Proxy Service is now combined with the *traffic_ops* service, so
when traffic_ops is restarted so is the Golang Proxy Service. Since the
Golang service is a proxy any APIs that are not implemented in the Golang
Service will be forwarded to the existing TO Perl API. Also, If the API is
implemented in the Golang Service the response is serviced by the Proxy
(where it will access the Postgres database as needed).
*traffic_ops logs*
*access.log - *old access.log is now renamed to *perl_access.log*, and the
Go proxy now takes over the *access.log* while logging in the *exact* format
as before (no monitoring or tooling changes are required)
*traffic_ops_golang.log* - this is a new file where any errors/debug will
be logged from the Go proxy.
*perl_access.log - the existing Mojolicious access.log gets a new name*
*traffic_ops.log* - existing Mojolicious debug file (no change)
There was a lot of debate and discussion about how to move forward and this
approach was less impactful to operations (which basically means less work
to move toward Go). Overtime, the goal is to do a rewrite of all of the
relevant endpoints that are in Mojolicious into Go (with a heavy focus on
modularity and unit testing, for a future release with Micro Services).
*What about the Qwilt contributions of API Gateway, Capabilities, and
Tenancy wasn't that a thing? *
Yes, and it still is. As for the API Gateway we will start "absorbing" the
code that *Amir* Yeshurun, so graciously contributed for the APIGW and
JWT. For the API Capabilities that *Naama Shoresh* also generously
contributed (all capabilities will be directly accessed from those
capability tables), please see the *Roadmap* below. And last but not
least, once once we are confident that tenancy works as designed, we will
begin porting all the tenancy hooks as well as any apis related to tenancy
(tenant apis, user apis, delivery service apis). All of the features that
Qwilt built will be used in some form or fashion.
*How do I jump in?*
We are going to take a "Biggest Bang for our Buck" approach to the TPv2 UI
and TO Golang API. The following roadmap is an initial plan of attack (of
course subject to debate).
*Roadmap*
- The TO API is READ heavy so all READ endpoints will be rewritten first
(avoiding Tenancy endpoints, like Users and Delivery Services)
- Circle back to the CREATE, UPDATE, DELETEs where more Golang foundation
will have to be developed (avoiding Tenancy endpoints)
- Implement Capabilities in the new TPv2 to help manage the
Roles/Capabilities with ease
- Evaluate Capabilities and design the api_capabilities data around that
- Hook in capabilities (once the capabilities design has settled and
database is correct in production)
- Evaluating the existing Tenancy Rules
- Rewrite Tenancy in Golang
- Tackle the 'Tenancy' endpoints in Golang, /deliveryservices, /users etc.
*Reasons to get excited!*
- *Improved PR's: *(Dan Kirkwood is helping us automate Jenkins/CI for
better PR quality so components like Documentation, Unit Testing are first
class citizens. The more required items checked off the PR checklist makes
PR merging MUCH easier.
- *Testing:* We have plans in the works to move testing to the forefront
of our development discussions. The hope is a fresh approach to Unit
Testing (in a modern Golang world) where Go tools for topics like "Code
Coverage" will help us improve our automated testing, and allow us to
"Refactor Mercilessly"
- *Performance:* My initial testing for the new* /monitoring* Go endpoint
shows an average 20x faster than the Mojolicous Perl equivalent running on
the same host. More efficiency endpoints mean more "scale" and less
production $$.
I will work to produce a list of initial endpoints and publish to the dev
list (along with the JIRA Tickets with the requirements to help manage the
priorities). The following PR (which I plan on merging today, if not
tomorrow) has a section that highlights the foundational features that the
existing TO API had that we had to build in Go, ie: logging, config
files, etc.
See the high level features implemented in the section *Things left before
this can be merged:*
TC JIRA for this Change:
https://issues.apache.org/jira/browse/TC-447
https://github.com/apache/incubator-trafficcontrol/pull/729
Expect more information about our progress and we hope to be as transparent
as possible,
-Dewayne Richardson
Re: Traffic Ops Golang Rewrite
Posted by Nir Sopher <ni...@qwilt.com>.
Got you. 10x,
Nir
On Mon, Aug 14, 2017 at 5:37 PM, Dewayne Richardson <de...@gmail.com>
wrote:
> The goal is to hit all the low hanging fruit first to discover all the
> refactor points and infrastructure before we attempt to bite off the bigger
> more impactful areas of Traffic Control like Tenancy and Capabilities.
>
> -Dew
>
> On Mon, Aug 14, 2017 at 5:22 AM, Nir Sopher <ni...@qwilt.com> wrote:
>
> > +1!
> >
> > As far as tenancy is concerned, the main logic is held in the
> > "Utils::Tenant" class.
> > I would be happy to use this class as an entry point to the Golang TO
> > development.
> >
> > BTW, the UT coverage of the tenancy checks is quite extensive.
> > Are we going to migrate the UTs as well, write new UTs in Golang, or keep
> > working with the Perl UTs?
> >
> > Nir
> >
> > On Wed, Aug 9, 2017 at 11:14 PM, Durfey, Ryan <Ry...@comcast.com>
> > wrote:
> >
> > > Great write up Dewayne. Given the length and forward looking nature of
> > > the topic I created a wiki page around this under traffic ops. Please
> > > continue discussion in the email thread and I will summarize any
> changes
> > > there.
> > >
> > > https://cwiki.apache.org/confluence/display/TC/Golang+
> > > Traffic+Ops+Replacement+-+Vampire+Proxy
> > >
> > > Ryan Durfey M | 303-524-5099
> > > CDN Support (24x7): 866-405-2993 or cdn_support@comcast.com<mailto:
> > > cdn_support@comcast.com>
> > >
> > >
> > > From: Dewayne Richardson <de...@gmail.com>
> > > Reply-To: "dev@trafficcontrol.incubator.apache.org" <
> > > dev@trafficcontrol.incubator.apache.org>
> > > Date: Wednesday, August 9, 2017 at 1:03 PM
> > > To: "dev@trafficcontrol.incubator.apache.org" <
> > > dev@trafficcontrol.incubator.apache.org>
> > > Subject: Traffic Ops Golang Rewrite
> > >
> > > Sorry for the TL;DR, but a lot of information needed to be conveyed.
> So,
> > > based upon the TO Rewrite discussions Rob Butts and I have been working
> > on
> > > a Golang proxy (Mark Torluemke, affectionately calls "the Vampire
> Proxy")
> > > that initially implements the */monitoring* endpoint to lay down the
> > > foundation for rewriting more Traffic Ops endpoints in Go. The goal is
> > to
> > > do a straight rewrite in Go (that implements any old API's as well as
> any
> > > new ones that follows the /api/1.2 format). The intent is to make this
> > > proxy 100% backward compatible (including any HTTP header requirements)
> > to
> > > keep the existing TO API clients from breaking.
> > >
> > > This PR is significant because when postinstall runs (after this PR is
> > > merged) it switches the ports according to the following:
> > >
> > > *TO Port Change Overview*
> > > Port *443* will be owned by Golang proxy
> > >
> > > Port *60443* will now be owned by the Mojolicious/Perl Hypnotoad
> service
> > >
> > > See */opt/traffic_ops/app/conf/cdn.conf* for a new property
> > > *traffic_ops_golang_port
> > > => '443'*.
> > >
> > > *Important Operational Changes:*
> > >
> > > *traffic_ops service*
> > > The Golang Proxy Service is now combined with the *traffic_ops*
> service,
> > so
> > > when traffic_ops is restarted so is the Golang Proxy Service. Since
> the
> > > Golang service is a proxy any APIs that are not implemented in the
> Golang
> > > Service will be forwarded to the existing TO Perl API. Also, If the
> API
> > is
> > > implemented in the Golang Service the response is serviced by the Proxy
> > > (where it will access the Postgres database as needed).
> > >
> > > *traffic_ops logs*
> > > *access.log - *old access.log is now renamed to *perl_access.log*, and
> > the
> > > Go proxy now takes over the *access.log* while logging in the *exact*
> > > format
> > > as before (no monitoring or tooling changes are required)
> > > *traffic_ops_golang.log* - this is a new file where any errors/debug
> will
> > > be logged from the Go proxy.
> > > *perl_access.log - the existing Mojolicious access.log gets a new name*
> > > *traffic_ops.log* - existing Mojolicious debug file (no change)
> > >
> > > There was a lot of debate and discussion about how to move forward and
> > this
> > > approach was less impactful to operations (which basically means less
> > work
> > > to move toward Go). Overtime, the goal is to do a rewrite of all of
> the
> > > relevant endpoints that are in Mojolicious into Go (with a heavy focus
> on
> > > modularity and unit testing, for a future release with Micro Services).
> > >
> > > *What about the Qwilt contributions of API Gateway, Capabilities, and
> > > Tenancy wasn't that a thing? *
> > > Yes, and it still is. As for the API Gateway we will start "absorbing"
> > the
> > > code that *Amir* Yeshurun, so graciously contributed for the APIGW and
> > > JWT. For the API Capabilities that *Naama Shoresh* also generously
> > > contributed (all capabilities will be directly accessed from those
> > > capability tables), please see the *Roadmap* below. And last but not
> > > least, once once we are confident that tenancy works as designed, we
> will
> > > begin porting all the tenancy hooks as well as any apis related to
> > tenancy
> > > (tenant apis, user apis, delivery service apis). All of the features
> that
> > > Qwilt built will be used in some form or fashion.
> > >
> > > *How do I jump in?*
> > > We are going to take a "Biggest Bang for our Buck" approach to the TPv2
> > UI
> > > and TO Golang API. The following roadmap is an initial plan of attack
> > (of
> > > course subject to debate).
> > >
> > > *Roadmap*
> > > - The TO API is READ heavy so all READ endpoints will be rewritten
> first
> > > (avoiding Tenancy endpoints, like Users and Delivery Services)
> > > - Circle back to the CREATE, UPDATE, DELETEs where more Golang
> foundation
> > > will have to be developed (avoiding Tenancy endpoints)
> > > - Implement Capabilities in the new TPv2 to help manage the
> > > Roles/Capabilities with ease
> > > - Evaluate Capabilities and design the api_capabilities data around
> that
> > > - Hook in capabilities (once the capabilities design has settled and
> > > database is correct in production)
> > > - Evaluating the existing Tenancy Rules
> > > - Rewrite Tenancy in Golang
> > > - Tackle the 'Tenancy' endpoints in Golang, /deliveryservices, /users
> > etc.
> > >
> > > *Reasons to get excited!*
> > >
> > > - *Improved PR's: *(Dan Kirkwood is helping us automate Jenkins/CI for
> > > better PR quality so components like Documentation, Unit Testing are
> > first
> > > class citizens. The more required items checked off the PR checklist
> > makes
> > > PR merging MUCH easier.
> > >
> > > - *Testing:* We have plans in the works to move testing to the
> forefront
> > > of our development discussions. The hope is a fresh approach to Unit
> > > Testing (in a modern Golang world) where Go tools for topics like "Code
> > > Coverage" will help us improve our automated testing, and allow us to
> > > "Refactor Mercilessly"
> > >
> > > - *Performance:* My initial testing for the new* /monitoring* Go
> endpoint
> > > shows an average 20x faster than the Mojolicous Perl equivalent running
> > on
> > > the same host. More efficiency endpoints mean more "scale" and less
> > > production $$.
> > >
> > > I will work to produce a list of initial endpoints and publish to the
> dev
> > > list (along with the JIRA Tickets with the requirements to help manage
> > the
> > > priorities). The following PR (which I plan on merging today, if not
> > > tomorrow) has a section that highlights the foundational features that
> > the
> > > existing TO API had that we had to build in Go, ie: logging, config
> > > files, etc.
> > >
> > > See the high level features implemented in the section *Things left
> > before
> > > this can be merged:*
> > >
> > > TC JIRA for this Change:
> > > https://issues.apache.org/jira/browse/TC-447
> > > https://github.com/apache/incubator-trafficcontrol/pull/729
> > >
> > > Expect more information about our progress and we hope to be as
> > transparent
> > > as possible,
> > >
> > > -Dewayne Richardson
> > >
> > >
> >
>
Re: Traffic Ops Golang Rewrite
Posted by Dewayne Richardson <de...@gmail.com>.
The goal is to hit all the low hanging fruit first to discover all the
refactor points and infrastructure before we attempt to bite off the bigger
more impactful areas of Traffic Control like Tenancy and Capabilities.
-Dew
On Mon, Aug 14, 2017 at 5:22 AM, Nir Sopher <ni...@qwilt.com> wrote:
> +1!
>
> As far as tenancy is concerned, the main logic is held in the
> "Utils::Tenant" class.
> I would be happy to use this class as an entry point to the Golang TO
> development.
>
> BTW, the UT coverage of the tenancy checks is quite extensive.
> Are we going to migrate the UTs as well, write new UTs in Golang, or keep
> working with the Perl UTs?
>
> Nir
>
> On Wed, Aug 9, 2017 at 11:14 PM, Durfey, Ryan <Ry...@comcast.com>
> wrote:
>
> > Great write up Dewayne. Given the length and forward looking nature of
> > the topic I created a wiki page around this under traffic ops. Please
> > continue discussion in the email thread and I will summarize any changes
> > there.
> >
> > https://cwiki.apache.org/confluence/display/TC/Golang+
> > Traffic+Ops+Replacement+-+Vampire+Proxy
> >
> > Ryan Durfey M | 303-524-5099
> > CDN Support (24x7): 866-405-2993 or cdn_support@comcast.com<mailto:
> > cdn_support@comcast.com>
> >
> >
> > From: Dewayne Richardson <de...@gmail.com>
> > Reply-To: "dev@trafficcontrol.incubator.apache.org" <
> > dev@trafficcontrol.incubator.apache.org>
> > Date: Wednesday, August 9, 2017 at 1:03 PM
> > To: "dev@trafficcontrol.incubator.apache.org" <
> > dev@trafficcontrol.incubator.apache.org>
> > Subject: Traffic Ops Golang Rewrite
> >
> > Sorry for the TL;DR, but a lot of information needed to be conveyed. So,
> > based upon the TO Rewrite discussions Rob Butts and I have been working
> on
> > a Golang proxy (Mark Torluemke, affectionately calls "the Vampire Proxy")
> > that initially implements the */monitoring* endpoint to lay down the
> > foundation for rewriting more Traffic Ops endpoints in Go. The goal is
> to
> > do a straight rewrite in Go (that implements any old API's as well as any
> > new ones that follows the /api/1.2 format). The intent is to make this
> > proxy 100% backward compatible (including any HTTP header requirements)
> to
> > keep the existing TO API clients from breaking.
> >
> > This PR is significant because when postinstall runs (after this PR is
> > merged) it switches the ports according to the following:
> >
> > *TO Port Change Overview*
> > Port *443* will be owned by Golang proxy
> >
> > Port *60443* will now be owned by the Mojolicious/Perl Hypnotoad service
> >
> > See */opt/traffic_ops/app/conf/cdn.conf* for a new property
> > *traffic_ops_golang_port
> > => '443'*.
> >
> > *Important Operational Changes:*
> >
> > *traffic_ops service*
> > The Golang Proxy Service is now combined with the *traffic_ops* service,
> so
> > when traffic_ops is restarted so is the Golang Proxy Service. Since the
> > Golang service is a proxy any APIs that are not implemented in the Golang
> > Service will be forwarded to the existing TO Perl API. Also, If the API
> is
> > implemented in the Golang Service the response is serviced by the Proxy
> > (where it will access the Postgres database as needed).
> >
> > *traffic_ops logs*
> > *access.log - *old access.log is now renamed to *perl_access.log*, and
> the
> > Go proxy now takes over the *access.log* while logging in the *exact*
> > format
> > as before (no monitoring or tooling changes are required)
> > *traffic_ops_golang.log* - this is a new file where any errors/debug will
> > be logged from the Go proxy.
> > *perl_access.log - the existing Mojolicious access.log gets a new name*
> > *traffic_ops.log* - existing Mojolicious debug file (no change)
> >
> > There was a lot of debate and discussion about how to move forward and
> this
> > approach was less impactful to operations (which basically means less
> work
> > to move toward Go). Overtime, the goal is to do a rewrite of all of the
> > relevant endpoints that are in Mojolicious into Go (with a heavy focus on
> > modularity and unit testing, for a future release with Micro Services).
> >
> > *What about the Qwilt contributions of API Gateway, Capabilities, and
> > Tenancy wasn't that a thing? *
> > Yes, and it still is. As for the API Gateway we will start "absorbing"
> the
> > code that *Amir* Yeshurun, so graciously contributed for the APIGW and
> > JWT. For the API Capabilities that *Naama Shoresh* also generously
> > contributed (all capabilities will be directly accessed from those
> > capability tables), please see the *Roadmap* below. And last but not
> > least, once once we are confident that tenancy works as designed, we will
> > begin porting all the tenancy hooks as well as any apis related to
> tenancy
> > (tenant apis, user apis, delivery service apis). All of the features that
> > Qwilt built will be used in some form or fashion.
> >
> > *How do I jump in?*
> > We are going to take a "Biggest Bang for our Buck" approach to the TPv2
> UI
> > and TO Golang API. The following roadmap is an initial plan of attack
> (of
> > course subject to debate).
> >
> > *Roadmap*
> > - The TO API is READ heavy so all READ endpoints will be rewritten first
> > (avoiding Tenancy endpoints, like Users and Delivery Services)
> > - Circle back to the CREATE, UPDATE, DELETEs where more Golang foundation
> > will have to be developed (avoiding Tenancy endpoints)
> > - Implement Capabilities in the new TPv2 to help manage the
> > Roles/Capabilities with ease
> > - Evaluate Capabilities and design the api_capabilities data around that
> > - Hook in capabilities (once the capabilities design has settled and
> > database is correct in production)
> > - Evaluating the existing Tenancy Rules
> > - Rewrite Tenancy in Golang
> > - Tackle the 'Tenancy' endpoints in Golang, /deliveryservices, /users
> etc.
> >
> > *Reasons to get excited!*
> >
> > - *Improved PR's: *(Dan Kirkwood is helping us automate Jenkins/CI for
> > better PR quality so components like Documentation, Unit Testing are
> first
> > class citizens. The more required items checked off the PR checklist
> makes
> > PR merging MUCH easier.
> >
> > - *Testing:* We have plans in the works to move testing to the forefront
> > of our development discussions. The hope is a fresh approach to Unit
> > Testing (in a modern Golang world) where Go tools for topics like "Code
> > Coverage" will help us improve our automated testing, and allow us to
> > "Refactor Mercilessly"
> >
> > - *Performance:* My initial testing for the new* /monitoring* Go endpoint
> > shows an average 20x faster than the Mojolicous Perl equivalent running
> on
> > the same host. More efficiency endpoints mean more "scale" and less
> > production $$.
> >
> > I will work to produce a list of initial endpoints and publish to the dev
> > list (along with the JIRA Tickets with the requirements to help manage
> the
> > priorities). The following PR (which I plan on merging today, if not
> > tomorrow) has a section that highlights the foundational features that
> the
> > existing TO API had that we had to build in Go, ie: logging, config
> > files, etc.
> >
> > See the high level features implemented in the section *Things left
> before
> > this can be merged:*
> >
> > TC JIRA for this Change:
> > https://issues.apache.org/jira/browse/TC-447
> > https://github.com/apache/incubator-trafficcontrol/pull/729
> >
> > Expect more information about our progress and we hope to be as
> transparent
> > as possible,
> >
> > -Dewayne Richardson
> >
> >
>
Re: Traffic Ops Golang Rewrite
Posted by Nir Sopher <ni...@qwilt.com>.
+1!
As far as tenancy is concerned, the main logic is held in the
"Utils::Tenant" class.
I would be happy to use this class as an entry point to the Golang TO
development.
BTW, the UT coverage of the tenancy checks is quite extensive.
Are we going to migrate the UTs as well, write new UTs in Golang, or keep
working with the Perl UTs?
Nir
On Wed, Aug 9, 2017 at 11:14 PM, Durfey, Ryan <Ry...@comcast.com>
wrote:
> Great write up Dewayne. Given the length and forward looking nature of
> the topic I created a wiki page around this under traffic ops. Please
> continue discussion in the email thread and I will summarize any changes
> there.
>
> https://cwiki.apache.org/confluence/display/TC/Golang+
> Traffic+Ops+Replacement+-+Vampire+Proxy
>
> Ryan Durfey M | 303-524-5099
> CDN Support (24x7): 866-405-2993 or cdn_support@comcast.com<mailto:
> cdn_support@comcast.com>
>
>
> From: Dewayne Richardson <de...@gmail.com>
> Reply-To: "dev@trafficcontrol.incubator.apache.org" <
> dev@trafficcontrol.incubator.apache.org>
> Date: Wednesday, August 9, 2017 at 1:03 PM
> To: "dev@trafficcontrol.incubator.apache.org" <
> dev@trafficcontrol.incubator.apache.org>
> Subject: Traffic Ops Golang Rewrite
>
> Sorry for the TL;DR, but a lot of information needed to be conveyed. So,
> based upon the TO Rewrite discussions Rob Butts and I have been working on
> a Golang proxy (Mark Torluemke, affectionately calls "the Vampire Proxy")
> that initially implements the */monitoring* endpoint to lay down the
> foundation for rewriting more Traffic Ops endpoints in Go. The goal is to
> do a straight rewrite in Go (that implements any old API's as well as any
> new ones that follows the /api/1.2 format). The intent is to make this
> proxy 100% backward compatible (including any HTTP header requirements) to
> keep the existing TO API clients from breaking.
>
> This PR is significant because when postinstall runs (after this PR is
> merged) it switches the ports according to the following:
>
> *TO Port Change Overview*
> Port *443* will be owned by Golang proxy
>
> Port *60443* will now be owned by the Mojolicious/Perl Hypnotoad service
>
> See */opt/traffic_ops/app/conf/cdn.conf* for a new property
> *traffic_ops_golang_port
> => '443'*.
>
> *Important Operational Changes:*
>
> *traffic_ops service*
> The Golang Proxy Service is now combined with the *traffic_ops* service, so
> when traffic_ops is restarted so is the Golang Proxy Service. Since the
> Golang service is a proxy any APIs that are not implemented in the Golang
> Service will be forwarded to the existing TO Perl API. Also, If the API is
> implemented in the Golang Service the response is serviced by the Proxy
> (where it will access the Postgres database as needed).
>
> *traffic_ops logs*
> *access.log - *old access.log is now renamed to *perl_access.log*, and the
> Go proxy now takes over the *access.log* while logging in the *exact*
> format
> as before (no monitoring or tooling changes are required)
> *traffic_ops_golang.log* - this is a new file where any errors/debug will
> be logged from the Go proxy.
> *perl_access.log - the existing Mojolicious access.log gets a new name*
> *traffic_ops.log* - existing Mojolicious debug file (no change)
>
> There was a lot of debate and discussion about how to move forward and this
> approach was less impactful to operations (which basically means less work
> to move toward Go). Overtime, the goal is to do a rewrite of all of the
> relevant endpoints that are in Mojolicious into Go (with a heavy focus on
> modularity and unit testing, for a future release with Micro Services).
>
> *What about the Qwilt contributions of API Gateway, Capabilities, and
> Tenancy wasn't that a thing? *
> Yes, and it still is. As for the API Gateway we will start "absorbing" the
> code that *Amir* Yeshurun, so graciously contributed for the APIGW and
> JWT. For the API Capabilities that *Naama Shoresh* also generously
> contributed (all capabilities will be directly accessed from those
> capability tables), please see the *Roadmap* below. And last but not
> least, once once we are confident that tenancy works as designed, we will
> begin porting all the tenancy hooks as well as any apis related to tenancy
> (tenant apis, user apis, delivery service apis). All of the features that
> Qwilt built will be used in some form or fashion.
>
> *How do I jump in?*
> We are going to take a "Biggest Bang for our Buck" approach to the TPv2 UI
> and TO Golang API. The following roadmap is an initial plan of attack (of
> course subject to debate).
>
> *Roadmap*
> - The TO API is READ heavy so all READ endpoints will be rewritten first
> (avoiding Tenancy endpoints, like Users and Delivery Services)
> - Circle back to the CREATE, UPDATE, DELETEs where more Golang foundation
> will have to be developed (avoiding Tenancy endpoints)
> - Implement Capabilities in the new TPv2 to help manage the
> Roles/Capabilities with ease
> - Evaluate Capabilities and design the api_capabilities data around that
> - Hook in capabilities (once the capabilities design has settled and
> database is correct in production)
> - Evaluating the existing Tenancy Rules
> - Rewrite Tenancy in Golang
> - Tackle the 'Tenancy' endpoints in Golang, /deliveryservices, /users etc.
>
> *Reasons to get excited!*
>
> - *Improved PR's: *(Dan Kirkwood is helping us automate Jenkins/CI for
> better PR quality so components like Documentation, Unit Testing are first
> class citizens. The more required items checked off the PR checklist makes
> PR merging MUCH easier.
>
> - *Testing:* We have plans in the works to move testing to the forefront
> of our development discussions. The hope is a fresh approach to Unit
> Testing (in a modern Golang world) where Go tools for topics like "Code
> Coverage" will help us improve our automated testing, and allow us to
> "Refactor Mercilessly"
>
> - *Performance:* My initial testing for the new* /monitoring* Go endpoint
> shows an average 20x faster than the Mojolicous Perl equivalent running on
> the same host. More efficiency endpoints mean more "scale" and less
> production $$.
>
> I will work to produce a list of initial endpoints and publish to the dev
> list (along with the JIRA Tickets with the requirements to help manage the
> priorities). The following PR (which I plan on merging today, if not
> tomorrow) has a section that highlights the foundational features that the
> existing TO API had that we had to build in Go, ie: logging, config
> files, etc.
>
> See the high level features implemented in the section *Things left before
> this can be merged:*
>
> TC JIRA for this Change:
> https://issues.apache.org/jira/browse/TC-447
> https://github.com/apache/incubator-trafficcontrol/pull/729
>
> Expect more information about our progress and we hope to be as transparent
> as possible,
>
> -Dewayne Richardson
>
>
Re: Traffic Ops Golang Rewrite
Posted by "Durfey, Ryan" <Ry...@comcast.com>.
Great write up Dewayne. Given the length and forward looking nature of the topic I created a wiki page around this under traffic ops. Please continue discussion in the email thread and I will summarize any changes there.
https://cwiki.apache.org/confluence/display/TC/Golang+Traffic+Ops+Replacement+-+Vampire+Proxy
Ryan Durfey M | 303-524-5099
CDN Support (24x7): 866-405-2993 or cdn_support@comcast.com<ma...@comcast.com>
From: Dewayne Richardson <de...@gmail.com>
Reply-To: "dev@trafficcontrol.incubator.apache.org" <de...@trafficcontrol.incubator.apache.org>
Date: Wednesday, August 9, 2017 at 1:03 PM
To: "dev@trafficcontrol.incubator.apache.org" <de...@trafficcontrol.incubator.apache.org>
Subject: Traffic Ops Golang Rewrite
Sorry for the TL;DR, but a lot of information needed to be conveyed. So,
based upon the TO Rewrite discussions Rob Butts and I have been working on
a Golang proxy (Mark Torluemke, affectionately calls "the Vampire Proxy")
that initially implements the */monitoring* endpoint to lay down the
foundation for rewriting more Traffic Ops endpoints in Go. The goal is to
do a straight rewrite in Go (that implements any old API's as well as any
new ones that follows the /api/1.2 format). The intent is to make this
proxy 100% backward compatible (including any HTTP header requirements) to
keep the existing TO API clients from breaking.
This PR is significant because when postinstall runs (after this PR is
merged) it switches the ports according to the following:
*TO Port Change Overview*
Port *443* will be owned by Golang proxy
Port *60443* will now be owned by the Mojolicious/Perl Hypnotoad service
See */opt/traffic_ops/app/conf/cdn.conf* for a new property
*traffic_ops_golang_port
=> '443'*.
*Important Operational Changes:*
*traffic_ops service*
The Golang Proxy Service is now combined with the *traffic_ops* service, so
when traffic_ops is restarted so is the Golang Proxy Service. Since the
Golang service is a proxy any APIs that are not implemented in the Golang
Service will be forwarded to the existing TO Perl API. Also, If the API is
implemented in the Golang Service the response is serviced by the Proxy
(where it will access the Postgres database as needed).
*traffic_ops logs*
*access.log - *old access.log is now renamed to *perl_access.log*, and the
Go proxy now takes over the *access.log* while logging in the *exact* format
as before (no monitoring or tooling changes are required)
*traffic_ops_golang.log* - this is a new file where any errors/debug will
be logged from the Go proxy.
*perl_access.log - the existing Mojolicious access.log gets a new name*
*traffic_ops.log* - existing Mojolicious debug file (no change)
There was a lot of debate and discussion about how to move forward and this
approach was less impactful to operations (which basically means less work
to move toward Go). Overtime, the goal is to do a rewrite of all of the
relevant endpoints that are in Mojolicious into Go (with a heavy focus on
modularity and unit testing, for a future release with Micro Services).
*What about the Qwilt contributions of API Gateway, Capabilities, and
Tenancy wasn't that a thing? *
Yes, and it still is. As for the API Gateway we will start "absorbing" the
code that *Amir* Yeshurun, so graciously contributed for the APIGW and
JWT. For the API Capabilities that *Naama Shoresh* also generously
contributed (all capabilities will be directly accessed from those
capability tables), please see the *Roadmap* below. And last but not
least, once once we are confident that tenancy works as designed, we will
begin porting all the tenancy hooks as well as any apis related to tenancy
(tenant apis, user apis, delivery service apis). All of the features that
Qwilt built will be used in some form or fashion.
*How do I jump in?*
We are going to take a "Biggest Bang for our Buck" approach to the TPv2 UI
and TO Golang API. The following roadmap is an initial plan of attack (of
course subject to debate).
*Roadmap*
- The TO API is READ heavy so all READ endpoints will be rewritten first
(avoiding Tenancy endpoints, like Users and Delivery Services)
- Circle back to the CREATE, UPDATE, DELETEs where more Golang foundation
will have to be developed (avoiding Tenancy endpoints)
- Implement Capabilities in the new TPv2 to help manage the
Roles/Capabilities with ease
- Evaluate Capabilities and design the api_capabilities data around that
- Hook in capabilities (once the capabilities design has settled and
database is correct in production)
- Evaluating the existing Tenancy Rules
- Rewrite Tenancy in Golang
- Tackle the 'Tenancy' endpoints in Golang, /deliveryservices, /users etc.
*Reasons to get excited!*
- *Improved PR's: *(Dan Kirkwood is helping us automate Jenkins/CI for
better PR quality so components like Documentation, Unit Testing are first
class citizens. The more required items checked off the PR checklist makes
PR merging MUCH easier.
- *Testing:* We have plans in the works to move testing to the forefront
of our development discussions. The hope is a fresh approach to Unit
Testing (in a modern Golang world) where Go tools for topics like "Code
Coverage" will help us improve our automated testing, and allow us to
"Refactor Mercilessly"
- *Performance:* My initial testing for the new* /monitoring* Go endpoint
shows an average 20x faster than the Mojolicous Perl equivalent running on
the same host. More efficiency endpoints mean more "scale" and less
production $$.
I will work to produce a list of initial endpoints and publish to the dev
list (along with the JIRA Tickets with the requirements to help manage the
priorities). The following PR (which I plan on merging today, if not
tomorrow) has a section that highlights the foundational features that the
existing TO API had that we had to build in Go, ie: logging, config
files, etc.
See the high level features implemented in the section *Things left before
this can be merged:*
TC JIRA for this Change:
https://issues.apache.org/jira/browse/TC-447
https://github.com/apache/incubator-trafficcontrol/pull/729
Expect more information about our progress and we hope to be as transparent
as possible,
-Dewayne Richardson