You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@trafficcontrol.apache.org by "Hoppal, Michael" <Mi...@comcast.com> on 2019/11/12 15:43:06 UTC
Traffic Ops Route Deprecation Strategy
As the Traffic Ops API is being rewritten from Perl to Golang there has been several routes that have been deprecated and probably more to come.
In the deprecation efforts I have seen two strategies:
* The route IS NOT rewritten from Perl to Golang and a deprecation notice is added to the alert response in the Perl handler
* The route IS rewritten from Perl to Golang and a deprecation notice is added to the alert response in the Golang handler
I think we should have consistency in our approach and wanted to get people’s thoughts.
I would vote that we do not rewrite a deprecated route from Perl to Golang.
Thanks,
Michael
Re: Traffic Ops Route Deprecation Strategy
Posted by Lavanya Bathina <la...@gmail.com>.
+1 on do not rewrite a deprecate route to GO!
On Nov 12, 2019, at 8:43 AM, Hoppal, Michael <Mi...@comcast.com> wrote:
As the Traffic Ops API is being rewritten from Perl to Golang there has been several routes that have been deprecated and probably more to come.
In the deprecation efforts I have seen two strategies:
* The route IS NOT rewritten from Perl to Golang and a deprecation notice is added to the alert response in the Perl handler
* The route IS rewritten from Perl to Golang and a deprecation notice is added to the alert response in the Golang handler
I think we should have consistency in our approach and wanted to get people’s thoughts.
I would vote that we do not rewrite a deprecated route from Perl to Golang.
Thanks,
Michael
Re: [EXTERNAL] Re: Traffic Ops Route Deprecation Strategy
Posted by "Gray, Jonathan" <Jo...@comcast.com>.
Agreed with Rob. API versioning aside, how does that look from our native SDK implementations? Do we have native client code for those routes? Do we have project code dependent on that native code?
Jonathan G
On 11/12/19, 9:05 AM, "Robert O Butts" <ro...@apache.org> wrote:
Whether we rewrite a route in Go is an implementation detail. To the
interface, to users, it doesn't matter whether a route is rewritten or not.
But our API follows Semantic Versioning, in order to not break users. We
can't just remove endpoints that some of us don't use, and assume other
people don't, maybe people who never speak up on the mailing list. We'll
never gain a big userbase if we keep breaking users.
Per the _project_ SemVer, once we have API 2.0, we can deprecate API 1.x,
and in the next major _project_ release, remove API 1.x. Irrespective of
Perl or Go.
My big concern is, API 2.0 is a big project. How long has the rewrite to Go
taken? Do we really believe designing and implementing a completely new API
will be any less time?
I don't want killing Perl to have to wait on that.
I know it feels like a waste to rewrite routes that you don't use, and
probably few people do. But that's the cost of a stable project. How many
"deprecated" routes are there? If it comes down to taking the development
time to rewrite them so we can kill Perl faster, or leaving Perl around, I
vote we just do the work and kill Perl.
>If we don't rewrite them, then Perl will last until API 2.0 has been
designed, released and then *another full major release cycle*. That's way
too long to have two codebases for the same component, IMO, especially
since the rewrite is already 50% complete.
+1
On Tue, Nov 12, 2019 at 8:54 AM ocket 8888 <oc...@gmail.com> wrote:
> I vote that by and large we DO rewrite them, with exceptions for routes
> that just plain don't work, even in Perl. Those are few, though.
>
> If we don't rewrite them, then Perl will last until API 2.0 has been
> designed, released and then *another full major release cycle*. That's way
> too long to have two codebases for the same component, IMO, especially
> since the rewrite is already 50% complete.
>
> On Tue, Nov 12, 2019 at 8:43 AM Hoppal, Michael <
> Michael_Hoppal@comcast.com>
> wrote:
>
> > As the Traffic Ops API is being rewritten from Perl to Golang there has
> > been several routes that have been deprecated and probably more to come.
> >
> > In the deprecation efforts I have seen two strategies:
> >
> >
> > * The route IS NOT rewritten from Perl to Golang and a deprecation
> > notice is added to the alert response in the Perl handler
> > * The route IS rewritten from Perl to Golang and a deprecation notice
> > is added to the alert response in the Golang handler
> >
> > I think we should have consistency in our approach and wanted to get
> > people’s thoughts.
> >
> > I would vote that we do not rewrite a deprecated route from Perl to
> Golang.
> >
> > Thanks,
> >
> > Michael
> >
>
Re: [EXTERNAL] Re: Traffic Ops Route Deprecation Strategy
Posted by Rawlin Peters <ra...@apache.org>.
Yes, API v2.0 would be all Go, and in TC 5.0 we would drop support for
API 1.x and remove the Perl implementation.
- Rawlin
On Tue, Dec 3, 2019 at 11:04 AM Schmidt, Andrew (Contractor)
<An...@comcast.com> wrote:
>
> +1 Makes sense to me. Is 2.0 the target for dropping the PERL implementation?
>
> On 12/2/19, 12:19 PM, "Gray, Jonathan" <Jo...@comcast.com> wrote:
>
> +1
>
> I think the only point of contention was if the API v2.0 would be in ATC v4.0 or not. Either way is fine with me, and more time to know this is coming makes sense.
>
> Jonathan G
>
>
> On 12/2/19, 11:32 AM, "Rawlin Peters" <ra...@apache.org> wrote:
>
> Hey all,
>
> A bunch of us got together and discussed this issue offline, and we
> got consensus to go with Option 1 - Graduate current API from 1.x to
> 2.x - allowing us to leave behind routes in 1.x that are
> dangerous/deprecated. Deprecated endpoints will not be rewritten from
> Perl to Go.
>
> So, we will graduate the latest minor version of existing 1.x Go
> endpoints to 2.0. For example, today we have
> /api/1.2/deliveryservices, /api/1.3/deliveryservices, and
> /api/1.4/deliveryservices. We will take the existing handler for
> /api/1.4/deliveryservices and promote it to /api/2.0/deliveryservices.
> We will follow this example for other 1.x Go endpoints with multiple
> minor versions as well.
>
> API endpoints that are deemed broken, dangerous, or deprecated will
> not be promoted to API 2.0. This will likely include the ~7 endpoints
> that I've mentioned in this thread already, but we haven't yet
> compiled the full list of endpoints that won't be promoted to 2.0. We
> will come up with that list (along with its criteria) soon and bring
> that list to the mailing list for everyone to look over and agree
> upon. Deprecated endpoints will be updated to return deprecation
> alerts where applicable.
>
> This API graduation will likely miss the boat for 4.0 but will
> hopefully land in 4.1 or 4.2 depending on how long it takes. So this
> would be the tentative timetable:
>
> TC 4.0:
> - API 1.x supported, some deprecation notices
>
> TC 4.1:
> - API 1.x still supported, deprecation notices added to endpoints not
> graduated to 2.0
> - API 2.0 supported, consisting of 1.x endpoints that were graduated
> - starting with this release, you need to start migrating external
> clients off of 1.x over to 2.0
>
> TC 4.2:
> - internal clients (e.g. TM, TR, etc) will be migrated off API 1.x
> over to 2.0. Doing this step after 4.1 adds confidence that 1.x is
> still supported alongside 2.0 in order to provide a smooth migration
> period for API clients.
>
> TC 5.0:
> - API 1.x no longer supported, only API 2.x is supported
>
> Those who took part in that offline discussion, please keep me honest
> here with a +1 if we did in fact come to a rough consensus on this
> option. If you weren't part of that offline discussion, feel free to
> voice your vote on this issue as well (I believe a bunch of people
> were already +1 on this option a few weeks ago).
>
> - Rawlin
>
> On Tue, Nov 19, 2019 at 12:23 PM Robert O Butts <ro...@apache.org> wrote:
> >
> > > - /riak/stats
> >
> > Looking at the Perl, this looks easy to rewrite. It's just proxying a
> > request to Riak.
> >
> > +1 on rewriting this, for API stability, even if we don't "value" the
> > endpoint as a project.
> >
> > >- /cachegroup_fallbacks
> >
> > Likewise +1 on rewriting. Not as trivial as /riak/stats, but it also looks
> > easy to me, looks like a standard CRUD endpoint. +1 on leaving a
> > "deprecated, please prefer X" message in, and +1 in removing in the big API
> > rewrite/refactor/2.0.
> >
> > > - /capabilities (all methods except for GET) -- capabilities are
> > built-in, should not be changed at runtime
> > > - /api_capabilities (all methods except for GET) -- same reasons as
> > /capabilities
> >
> > I actually don't agree with this - because they're useful to plugins. IMO
> > it should be possible to add and modify Capabilities and API Capabilities,
> > because they can be used by plugins which add new endpoints, to control
> > access to them. If there's disagreement there, I'd vote we take that to a
> > separate thread.
> >
> > But in terms of the endpoints themselves, the Perl also looks easy to
> > rewrite, looks like standard CRUD. -1 on deprecating or removing, +1 on
> > rewriting.
> >
> > > - /cdns/{name}/configs/routing -- this was a half-baked attempt at
> > redoing CrConfig.json -- cannot possibly be of use to anyone in its current
> > state
> >
> > Ugh. I agree with the sentiment; and this one looks slightly less trivial.
> > But I personally don't see it as an unreasonable amount of work. For the
> > sake of API stability, I'm personally willing to volunteer to rewrite this
> > one, including API tests to make fixing easy if the schema changes (which
> > it appears to have in master). And hopefully it can just exist until the
> > fabled API Rewrite. Is that acceptable?
> >
> > > - the ATS config generation endpoints -- Rob actually proposed to remove
> > these from 1.x due to danger of being out of sync w/ cache-side config
> > generator
> >
> > I'm personally on the fence about this one. Like I said in that thread, I'm
> > the last one to want to break users, and I don't take this lightly. If,
> > after I made the case for it, I'm the only one, I might be inclined to
> > retract my proposal, and just continue to maintain them. Maybe some more
> > API tests would help with that. I think I'm the primary dev on the ATS
> > config gen at this point anyway.
> >
> > > - ATS Config Files Stuff Aren't these all already in Go anyway? Whether
> > or not they should be removed from the API aside, I don't think anything
> > actually needs to hit Perl anymore.
> >
> > Correct.
> >
> > > - /hwinfo/dtdata
> >
> > I think the 'render' call here is just rendering the JSON. But, the
> > endpoint itself is just hanging and not returning for me, I'm not sure why.
> > I'd like to volunteer to look into this endpoint as well. I think you might
> > be right, this one might be unreasonable to rewrite. But if it is
> > reasonable, I'd like to just do it. I'd like to at least look into it.
> >
> >
> > On Tue, Nov 19, 2019 at 12:05 PM ocket 8888 <oc...@gmail.com> wrote:
> >
> > > I'm generally +1 for option 2
> > >
> > > regarding those specific endpoints:
> > >
> > > - /riak/stats if this is easy and doesn't expose anything dangerous, we
> > > should do it IMO.
> > > - /cachegroup_fallbacks should be pretty trivial, so I still say do it
> > > - /capabilities since these aren't actually used for anything, I think it's
> > > pretty safe (and easy) to rewrite. Plus there's already a PR open for
> > > exactly that
> > > - /api_capabilities dropping this one could be problematic, just because
> > > people may depend on it to tell them what endpoints exist. To be clear, I
> > > think it ought to be deprecated, but unfortunately I just don't think it's
> > > terribly unlikely that this is built into scripts and tools. The actual
> > > code for the rewrite should be easy too, although maintaining it is a pain.
> > >
> > > - ATS Config Files Stuff Aren't these all already in Go anyway? Whether or
> > > not they should be removed from the API aside, I don't think anything
> > > actually needs to hit Perl anymore.
> > >
> > > - /cdn/{{name}}/configs/routing CRConfigs are large and complex, so if this
> > > never worked anyway maybe we should just add a Go handler for it that
> > > returns a '410 Gone' response (or '501 Not Implemented' is the other one
> > > I've heard suggested) and leave the actual logic behind in the Perl
> > > - /hwinfo/dtdata if this calls into Mojolicious rendering, returns a
> > > non-json response, and/or contains information that in either content or
> > > encoding is meaningless to any sane client then it should also return a
> > > 410/501 and get left behind in Perl.
> > >
> > > On Tue, Nov 19, 2019 at 11:08 AM Rawlin Peters <ra...@apache.org> wrote:
> > >
> > > > While I appreciate and mostly agree with your essay on the importance
> > > > of not breaking users (or rather, breaking them in predictable ways),
> > > > I for one would really like to meet at some middle ground with option
> > > > 2. I'm not asking to remove 50 high-usage endpoints from 1.x. IIRC
> > > > I've only specifically proposed "leaving behind" a few specific Perl
> > > > endpoints that I believe serve no purpose, have no valid reason to be
> > > > used, and/or are dangerous to the system:
> > > >
> > > > - /riak/stats -- informational, was used for debugging the initial
> > > > Riak integration. This one might be pretty easy to rewrite since it
> > > > just calls a corresponding riak API and returns the result via TO, so
> > > > I could be on board with rewriting it if necessary.
> > > > - /cachegroup_fallbacks -- these endpoints were released in 3.0 before
> > > > we managed to augment the existing /cachegroups API to handle
> > > > fallbacks, so I could get on board with rewriting these if we really
> > > > think it's important.
> > > > - /capabilities (all methods except for GET) -- capabilities are
> > > > built-in, should not be changed at runtime
> > > > - /api_capabilities (all methods except for GET) -- same reasons as
> > > > /capabilities
> > > > - /cdns/{name}/configs/routing -- this was a half-baked attempt at
> > > > redoing CrConfig.json -- cannot possibly be of use to anyone in its
> > > > current state
> > > > - the ATS config generation endpoints -- Rob actually proposed to
> > > > remove these from 1.x due to danger of being out of sync w/ cache-side
> > > > config generator
> > > > - /hwinfo/dtdata -- AFAIK the "dtdata" endpoints were only
> > > > used/required by the old Perl Mojolicious UI, no valid reason to be
> > > > using this as an external API client. In fact, I'm not even sure _how_
> > > > this endpoint can be used outside of the Mojolicious framework/UI. It
> > > > calls the Mojolicious `render` function, which usually means rendering
> > > > an embedded Perl HTML template given some data. This is one I would
> > > > consider completely unreasonable to rewrite to Go.
> > > >
> > > > Can we get on board with leaving at least these 1.x endpoints behind
> > > > in the Perl? Doing a cursory skim over the remaining 1.x endpoints to
> > > > rewrite, it _seems_ like everything else would fall into the "simpler
> > > > to just rewrite rather than gain consensus to remove" category.
> > > >
> > > > - Rawlin
> > > >
> > > > On Tue, Nov 19, 2019 at 9:19 AM Robert O Butts <ro...@apache.org> wrote:
> > > > >
> > > > > >Option 1 - Graduate current API from 1.x to 2.x - allowing us to leave
> > > > > behind routes in 1.x that are dangerous or deprecated
> > > > > >Option 2 - Rewrite all endpoints from Perl to Golang in current
> > > version
> > > > as
> > > > > we are doing now.
> > > > >
> > > > > +1 on either Option 1 or Option 2, without a strong preference. Option
> > > 1
> > > > > has advantages of letting us also safely remove the Cache Config
> > > > endpoints
> > > > > and implement Layered Profiles. But Option 2 also has the advantage of
> > > > not
> > > > > requiring users to upgrade.
> > > > >
> > > > > > Option 3 - Only rewrite endpoints from Perl to Golang in current
> > > > version
> > > > > that we agree are not dangerous or should deprecate
> > > > >
> > > > > -1. As before, it breaks users unpredictably.
> > > > >
> > > > > >It will literally outright break clients (*all* of them, not just
> > > > *some*)
> > > > >
> > > > > This is technically correct: technically, new versions 'break' all
> > > users.
> > > > > The difference is, it does it in a predictable, safe way that users can
> > > > > handle, inspect, and safely upgrade.
> > > > >
> > > > > >Can someone please explain to me why (A) artificially graduating the
> > > > > current 1.x API to 2.x, and removing the deprecated endpoints from 1.x,
> > > > is
> > > > > better for API clients overall than (B) just removing deprecated
> > > > endpoints
> > > > > from 1.x? Existing 1.x clients would gain absolutely nothing from being
> > > > > forced to update to 2.x like this.
> > > > >
> > > > > >for zero gain in value
> > > > >
> > > > > Users would gain something. There is a very specific gain.
> > > > >
> > > > > The gain is that the breakage is predictable. People frequently upgrade
> > > > > minor versions automatically, either running commands without thinking,
> > > > or
> > > > > even automating the entire process.
> > > > >
> > > > > For TC, this applies to both CDN operators, and their end-users.
> > > > >
> > > > > When the minor versions contain breaking changes, that automatic
> > > upgrade
> > > > > then breaks. In this case, any scripts or tools they have using the
> > > > removed
> > > > > endpoints will break. If those scripts are important, this could bring
> > > > down
> > > > > their CDN, or break their Delivery Service, or otherwise cause an
> > > outage.
> > > > >
> > > > > Contrawise, users expect breaking changes in Major versions, and don't
> > > > > typically automatically upgrade them. Rather, they'll manually inspect
> > > > the
> > > > > update, do testing, and check the changelog for breaking changes.
> > > > >
> > > > > It's not a technical thing, it's a human thing. Humans automatically
> > > > > upgrade minor versions, and write scripts to do so, and dependency
> > > files
> > > > to
> > > > > allow it, because the software industry taught people that minor
> > > versions
> > > > > shouldn't break anything.
> > > > >
> > > > > If we start breaking minor versions, initially users will experience
> > > > > breakage. Not all users, and not all catastrophic. But some users will
> > > > > experience catastrophic outages. And it will be our fault as a project.
> > > > And
> > > > > after enough times, they'll fork TC, or find another product, or write
> > > > > their own; and stop participating in the community, or never start.
> > > > >
> > > > > Then, once they've learned that we can't be trusted not to break
> > > things,
> > > > > they'll experience the 2nd problem: where nothing can be safely
> > > upgraded,
> > > > > and the smallest change requires long and expensive testing to verify a
> > > > > single minor version or even patch upgrade, and still risk missing
> > > > > something and bringing their CDN down. To be honest, we're pretty much
> > > > > already here. It already costs our dev and ops teams unbelievable
> > > amounts
> > > > > of time and person-hours of testing for every minor upgrade. That time
> > > > far
> > > > > exceeds the development time to just rewrite the small endpoints we
> > > don't
> > > > > like anymore (and tangentially, also the time to write the Integration
> > > > and
> > > > > E2E tests TC desperately needs).
> > > > >
> > > > > ATC has so much potential. There aren't many OSS CDNs out there, and
> > > > > there's certainly the demand. We have the potential to become a huge
> > > > > project, which for us existing users, translates into potentially an
> > > > > enormous amount of free development. But we'll never get there if we
> > > keep
> > > > > breaking things.
> > > > >
> > > > > And for my particular organization (Comcast), we have a hard enough
> > > time
> > > > > upgrading. Our upgrades, especially Traffic Ops, are extremely painful,
> > > > and
> > > > > require enormous amounts of manual testing. Even then, we frequently
> > > have
> > > > > issues that require rollback, sometimes even an outage; even after
> > > those
> > > > > unbelievably amounts of developer and operator time testing. We
> > > > desperately
> > > > > need a system that is safer and easier to upgrade. Breaking changes are
> > > > > always going to require more testing; but we can at least make minor
> > > > > versions safe to upgrade.
> > > > >
> > > > > > I know I am changing my initial stance but I am +1 on rewriting all
> > > the
> > > > > Perl endpoints to Golang and not breaking the API contract we have in
> > > > > place. Most of the API routes to rewrite are rather simple to port and
> > > I
> > > > > think would be faster than getting consensus on actually deprecating it
> > > > > from the mailing list.
> > > > >
> > > > > I'm inclined to agree with this. The things I've seen marked
> > > "deprecated"
> > > > > seem to mostly be small, and look like they'd be pretty easy to
> > > rewrite.
> > > > > But I haven't seen the whole list. Is there a list somewhere, so the
> > > > > community can see it to better decide?
> > > > >
> > > > >
> > > > > On Tue, Nov 19, 2019 at 7:24 AM Hoppal, Michael <
> > > > Michael_Hoppal@comcast.com>
> > > > > wrote:
> > > > >
> > > > > > I would love to drive this to some type of consensus on rewrite
> > > > strategy
> > > > > > as we have a good amount of resources (and growing) working on the
> > > > rewrite
> > > > > > and I want to keep that momentum.
> > > > > >
> > > > > > Sounds like there is three options from the email thread (please add
> > > > any I
> > > > > > missed)
> > > > > >
> > > > > > Option 1 - Graduate current API from 1.x to 2.x - allowing us to
> > > leave
> > > > > > behind routes in 1.x that are dangerous or deprecated
> > > > > > Option 2 - Rewrite all endpoints from Perl to Golang in current
> > > > version as
> > > > > > we are doing now.
> > > > > > Option 3 - Only rewrite endpoints from Perl to Golang in current
> > > > version
> > > > > > that we agree are not dangerous or should deprecate
> > > > > >
> > > > > > I am +1 for option 2 (which I know is different from my original
> > > > stance).
> > > > > >
> > > > > > Which direction would others like to go?
> > > > > >
> > > > > > Thanks,
> > > > > >
> > > > > > Michael
> > > > > >
> > > > > >
> > > > > > I know I am changing my initial stance but I am +1 on rewriting all
> > > the
> > > > > > Perl endpoints to Golang and not breaking the API contract we have in
> > > > > > place. Most of the API routes to rewrite are rather simple to port
> > > and
> > > > I
> > > > > > think would be faster than getting consensus on actually deprecating
> > > it
> > > > > > from the mailing list.
> > > > > >
> > > > > > On 11/16/19, 3:31 PM, "Rawlin Peters" <ra...@apache.org> wrote:
> > > > > >
> > > > > > > Moving to 2.x means that it's my fault as a consumer of the API
> > > > if I
> > > > > > was still using the deprecated endpoint (and that change should be
> > > > part of
> > > > > > a larger upgrade plan). Simply removing the deprecated routes,
> > > > > > retroactively, means I can't rely on having a transition period to
> > > get
> > > > to
> > > > > > 2.0 outside of a large completely down upgrade window.
> > > > > >
> > > > > > There would be a transition period either way. Endpoints would be
> > > > > > deprecated in ATC 4.0 and removed in 5.0. The release notes for
> > > 4.0
> > > > > > would clearly list out the deprecated endpoints. If they have
> > > > > > replacements, there would be instructions on how to transition to
> > > > the
> > > > > > replacement. Deprecated endpoints would return alerts. After
> > > > upgrading
> > > > > > to 4.0, operators should be well aware that certain endpoints are
> > > > > > being removed in 5.0, and it's on them to start using new
> > > > endpoints,
> > > > > > change workflows, etc.
> > > > > >
> > > > > > By forcing operators to upgrade all of their clients to start
> > > using
> > > > > > API 2.x, it just adds to their burden of upgrading when that time
> > > > > > comes. Rather than surgically fixing the clients of the 2% of
> > > > > > deprecated API endpoints before upgrading to ATC 5.0, operators
> > > > would
> > > > > > have to fix every single client they have to use API 2.x before
> > > > > > upgrading to 5.0. Is that exercise really worth it? Most people
> > > > would
> > > > > > probably be blissfully unaware that the "POST
> > > > /api/1.1/capabilities"
> > > > > > endpoint disappeared (among other deprecated endpoints). Less
> > > > footguns
> > > > > > in the system, and they didn't even have to update their clients.
> > > > > >
> > > > > > Anyways, circling back to Michael's original question, it sounds
> > > > like
> > > > > > either option we choose, we at least have consensus on this
> > > > > > deprecation strategy:
> > > > > > - The route IS NOT rewritten from Perl to Golang and a
> > > deprecation
> > > > > > notice is added to the alert response in the Perl handler
> > > > > >
> > > > > > If we're content to force all API clients to update their clients
> > > > to
> > > > > > 2.x for zero gain in value, at least we won't be wasting work by
> > > > > > rewriting deprecated Perl endpoints to Go.
> > > > > >
> > > > > > - Rawlin
> > > > > >
> > > > > > On Fri, Nov 15, 2019 at 7:06 PM Gray, Jonathan
> > > > > > <Jo...@comcast.com> wrote:
> > > > > > >
> > > > > > > Moving to 2.x means that it's my fault as a consumer of the API
> > > > if I
> > > > > > was still using the deprecated endpoint (and that change should be
> > > > part of
> > > > > > a larger upgrade plan). Simply removing the deprecated routes,
> > > > > > retroactively, means I can't rely on having a transition period to
> > > get
> > > > to
> > > > > > 2.0 outside of a large completely down upgrade window. Depending on
> > > > how
> > > > > > the API is used, even in our own components such as TM, that could
> > > > prove
> > > > > > disastrous. For the older minor revs like 1.1/1.2, I could live with
> > > > > > dropping those as part of a major project release (including upgrade
> > > > > > instructions outlining what releases you must be on beforehand).
> > > > Currently
> > > > > > 1.3 I view as stable enough to code to or use a native client library
> > > > from,
> > > > > > and 1.4 is my risk/overhead to take for new stuff as it's still
> > > > evolving
> > > > > > and improving. Going to 1.5 or 2.0 moves that forward. The path of
> > > > least
> > > > > > surprise for humans is to expect big shifts on those major revs.
> > > > Removal
> > > > > > falls under that umbrella and gets as a consumer not to strictly
> > > focus
> > > > on
> > > > > > endpoint->endpoint moves, but rather workflow shifts and maybe now I
> > > > have
> > > > > > to use the API a different way than before. I believe getting this
> > > > right
> > > > > > is something that would greatly help us understand does X component
> > > at
> > > > > > version 1 work with Y component at version 2, because now you're
> > > > asking do
> > > > > > components X and Y both speak protocol A. With major releases we can
> > > > work
> > > > > > through that each time, but it's error prone and requires experts in
> > > > all
> > > > > > products to agree upon. If you do nothing but consume major
> > > releases,
> > > > and
> > > > > > then conduct your own impact evaluation having read the upgrade
> > > > > > instructions thoroughly, maybe you'd be able to smoothly transition.
> > > > > > >
> > > > > > > Jonathan G
> > > > > > >
> > > > > > >
> > > > > > > On 11/15/19, 5:24 PM, "Rawlin Peters" <ra...@apache.org>
> > > wrote:
> > > > > > >
> > > > > > > Can someone please explain to me why (A) artificially
> > > > graduating
> > > > > > the
> > > > > > > current 1.x API to 2.x, and removing the deprecated
> > > endpoints
> > > > > > from
> > > > > > > 1.x, is better for API clients overall than (B) just
> > > removing
> > > > > > > deprecated endpoints from 1.x? Existing 1.x clients would
> > > > gain
> > > > > > > absolutely nothing from being forced to update to 2.x like
> > > > this.
> > > > > > >
> > > > > > > The way I see the two choices:
> > > > > > > A) break every single API client out there
> > > > > > > B) break only clients of APIs that we've decided upon as a
> > > > > > community
> > > > > > > to be deprecated
> > > > > > >
> > > > > > > IMO, when we have to break things, we should try break the
> > > > least
> > > > > > > amount of clients possible. All I'm looking for is a
> > > valuable
> > > > > > reason
> > > > > > > to choose option A over option B, that isn't just "because
> > > > > > SemVer".
> > > > > > >
> > > > > > > - Rawlin
> > > > > > >
> > > > > > > On Wed, Nov 13, 2019 at 3:24 PM Schmidt, Andrew
> > > (Contractor)
> > > > > > > <An...@comcast.com> wrote:
> > > > > > > >
> > > > > > > > If we decide to get rid of the API versioning it makes
> > > > sense
> > > > > > to me that we would do that on a major version upgrade, probably to
> > > > the ATC
> > > > > > version. But I think it would be too early to do it on the 2.0 API
> > > > version.
> > > > > > I like this compromise +1. I was wondering why we were stuck on the
> > > > idea
> > > > > > that 2.0 had to be an API rewrite. That seems like it’s a long way
> > > > out. 2.0
> > > > > > = deprecations and fixes. 3.0 or 5.0 = api rewrite and maybe version
> > > > > > convergence.
> > > > > > > >
> > > > > > > > On 11/13/19, 11:52 AM, "ocket8888@gmail.com" <
> > > > > > ocket8888@gmail.com> wrote:
> > > > > > > >
> > > > > > > > Lemme lead with: making breaking changes on a major
> > > > > > version boundary
> > > > > > > > gets +1 from me - no reason the API needs to be fully
> > > > > > redesigned for
> > > > > > > > any particular number.
> > > > > > > >
> > > > > > > > But I just gotta bring this up again: if we want to
> > > be
> > > > > > able to
> > > > > > > > deprecate an endpoint for a major release and remove
> > > > it in
> > > > > > the next
> > > > > > > > without needing to deal with the messiness of API
> > > > > > versions, we could
> > > > > > > > always just get rid of API versions. Major ATC
> > > > revisions
> > > > > > should be
> > > > > > > > easier to come by with our new release strategy.
> > > > > > > >
> > > > > > > > On Wed, 2019-11-13 at 10:23 -0700, Rawlin Peters
> > > wrote:
> > > > > > > > > > It is a compromise - it's an inconvenience to
> > > > users,
> > > > > > forcing them
> > > > > > > > > > to fix
> > > > > > > > > scripts to upgrade. Which isn't good for user
> > > > experience
> > > > > > either; but
> > > > > > > > > it
> > > > > > > > > doesn't outright break people. It's a compromise
> > > I'm
> > > > > > willing to live
> > > > > > > > > with.
> > > > > > > > >
> > > > > > > > > It will literally outright break clients (*all* of
> > > > them,
> > > > > > not just
> > > > > > > > > *some*). We have been down this road before, so
> > > I'll
> > > > > > just repeat what
> > > > > > > > > I said when this exact conversation came up months
> > > > > > ago[1]:
> > > > > > > > >
> > > > > > > > > "So it basically boils down to:
> > > > > > > > > A) promote Go 1.x routes to 2.x and eventually
> > > break
> > > > ALL
> > > > > > users of the
> > > > > > > > > 1.x API when the time comes to remove 1.x
> > > > > > > > > B) deprecate and remove some 1.x Perl routes and
> > > > > > POSSIBLY break SOME
> > > > > > > > > users"
> > > > > > > > >
> > > > > > > > > Either way we slice and dice it with versioning,
> > > > > > removing an endpoint
> > > > > > > > > means that clients of that endpoint will break. By
> > > > > > graduating only
> > > > > > > > > the
> > > > > > > > > "good stuff" from 1.x to 2.x, and removing
> > > endpoints
> > > > > > from 1.x, we
> > > > > > > > > will
> > > > > > > > > unnecessarily break all 1.x clients when the time
> > > > comes
> > > > > > to remove the
> > > > > > > > > rest of 1.x. Instead, we can choose to break only
> > > > > > clients of 1.x APIs
> > > > > > > > > that we already deemed dangerous, unnecessary,
> > > > useless,
> > > > > > etc. enough
> > > > > > > > > to
> > > > > > > > > deprecate.
> > > > > > > > >
> > > > > > > > > Why break 100% of our API clients when we can
> > > choose
> > > > to
> > > > > > break only a
> > > > > > > > > small number of clients that happen to use
> > > > > > > > > dangerous/unnecessary/useless APIs? Why would we
> > > > break
> > > > > > our API
> > > > > > > > > clients
> > > > > > > > > unnecessarily for no added value? There is no sense
> > > > in
> > > > > > blindly
> > > > > > > > > adhering to Semantic Versioning just for the sake
> > > of
> > > > > > adhering to
> > > > > > > > > Semantic Versioning. Instead, use it as a
> > > guideline,
> > > > and
> > > > > > break the
> > > > > > > > > rules where it makes the most sense to do so.
> > > > > > > > >
> > > > > > > > > - Rawlin
> > > > > > > > >
> > > > > > > > > [1]
> > > > > > > > >
> > > > > >
> > > >
> > > https://urldefense.com/v3/__https://lists.apache.org/thread.html/15c6bfcdce0ac00a3a8612a96c3d668a8a5a16c6ca9401de9473d31c@*3Cdev.trafficcontrol.apache.org*3E__;JSU!rx_L75ITgOQ!V_b3E9iEYqNlZus18f-sVuofwuyxapCthg3LdO4NxyMCU39b7mdf4klP5KNR7XZ7d3Uc$
> > > > > > > > >
> > > > > > > > > On Wed, Nov 13, 2019 at 9:43 AM Robert O Butts <
> > > > > > rob@apache.org>
> > > > > > > > > wrote:
> > > > > > > > > > > But if it's really valuable to remove these
> > > > > > endpoints, maybe it's
> > > > > > > > > > appropriate to bump the major version and ask
> > > > > > everybody to update
> > > > > > > > > > their
> > > > > > > > > > clients?
> > > > > > > > > >
> > > > > > > > > > +1
> > > > > > > > > >
> > > > > > > > > > This sounds like the best compromise to me. It
> > > > > > addresses both sides
> > > > > > > > > > - it
> > > > > > > > > > lets us remove endpoints, and it also prevents
> > > > > > breaking users.
> > > > > > > > > >
> > > > > > > > > > It is a compromise - it's an inconvenience to
> > > > users,
> > > > > > forcing them
> > > > > > > > > > to fix
> > > > > > > > > > scripts to upgrade. Which isn't good for user
> > > > > > experience either;
> > > > > > > > > > but it
> > > > > > > > > > doesn't outright break people. It's a compromise
> > > > I'm
> > > > > > willing to
> > > > > > > > > > live with.
> > > > > > > > > >
> > > > > > > > > > This actually lets us do at least 3 things:
> > > > > > > > > > 1. Remove "deprecated" endpoints.
> > > > > > > > > > 2. Remove the unmaintained dangerously-large
> > > Cache
> > > > > > Config
> > > > > > > > > > endpoints.
> > > > > > > > > > 3. Lets us do Layered Profiles, changing Servers
> > > > and
> > > > > > DSes to have
> > > > > > > > > > arrays of
> > > > > > > > > > Profiles instead of a single Profile, without
> > > hacky
> > > > > > backward-
> > > > > > > > > > compatibility
> > > > > > > > > > attempts.
> > > > > > > > > >
> > > > > > > > > > To clarify, `/api/2.0` would be essentially the
> > > > > > current API, just
> > > > > > > > > > with the
> > > > > > > > > > above breaking changes (possibly others) which
> > > > require
> > > > > > a SemVer
> > > > > > > > > > major
> > > > > > > > > > version increase. And the big API
> > > rewrite/redesign
> > > > > > we've been
> > > > > > > > > > calling "2.0"
> > > > > > > > > > would become `/api/3.0` or some future version.
> > > > > > > > > >
> > > > > > > > > > It's not perfect, but it seems to address the
> > > most
> > > > > > concerns on all
> > > > > > > > > > sides.
> > > > > > > > > > Thoughts? Objections?
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > > On Wed, Nov 13, 2019 at 8:45 AM Chris Lemmons <
> > > > > > alficles@gmail.com>
> > > > > > > > > > wrote:
> > > > > > > > > >
> > > > > > > > > > > For the endpoints, I agree in part and disagree
> > > > in
> > > > > > part.
> > > > > > > > > > >
> > > > > > > > > > > Agree for these two:
> > > > > > > > > > > - the endpoints are dangerous and pose a risk
> > > to
> > > > the
> > > > > > overall
> > > > > > > > > > > Traffic
> > > > > > > > > > > Control system
> > > > > > > > > > > - the task of rewriting the endpoints from Perl
> > > > to
> > > > > > Go represents
> > > > > > > > > > > an
> > > > > > > > > > > unreasonable amount of work for the project
> > > > > > > > > > >
> > > > > > > > > > > And I would add:
> > > > > > > > > > > - the endpoint does not serve its intended
> > > > purpose
> > > > > > and cannot
> > > > > > > > > > > reasonably
> > > > > > > > > > > be used
> > > > > > > > > > > - the endpoint has been made unsafe or unusable
> > > > by
> > > > > > new features
> > > > > > > > > > > added to TC
> > > > > > > > > > >
> > > > > > > > > > > Disagree for these two:
> > > > > > > > > > > - the endpoints have been obsoleted by other
> > > > > > endpoints
> > > > > > > > > > > - the endpoints don't seem to serve a true
> > > > purpose
> > > > > > or don't seem
> > > > > > > > > > > to be
> > > > > > > > > > > valuable to the project anymore
> > > > > > > > > > >
> > > > > > > > > > > For the most part, obsoleted endpoints can be
> > > > > > maintained with a
> > > > > > > > > > > shim
> > > > > > > > > > > into the new code. If they would require "an
> > > > > > unreasonable amount
> > > > > > > > > > > of
> > > > > > > > > > > work", it might make sense to remove them,
> > > then.
> > > > > > > > > > >
> > > > > > > > > > > Things that don't seem valuable usually fall
> > > into
> > > > > > another
> > > > > > > > > > > category
> > > > > > > > > > > like "cannot reasonably be used", but if the
> > > only
> > > > > > reason for
> > > > > > > > > > > deprecation is that the feature is going away,
> > > we
> > > > > > should really
> > > > > > > > > > > support it according to our promise.
> > > > > > > > > > >
> > > > > > > > > > > But if it's really valuable to remove these
> > > > > > endpoints, maybe it's
> > > > > > > > > > > appropriate to bump the major version and ask
> > > > > > everybody to update
> > > > > > > > > > > their clients?
> > > > > > > > > > >
> > > > > > > > > > > On Tue, Nov 12, 2019 at 5:27 PM Rawlin Peters <
> > > > > > rawlin@apache.org>
> > > > > > > > > > > wrote:
> > > > > > > > > > > > The real crux of the issue here is whether or
> > > > not
> > > > > > we agree as a
> > > > > > > > > > > > project that we can move forward with this
> > > plan
> > > > > > that I believe
> > > > > > > > > > > > to be a
> > > > > > > > > > > > good compromise that meets most concerns but
> > > > > > "breaks the
> > > > > > > > > > > > Semantic
> > > > > > > > > > > > Versioning promise":
> > > > > > > > > > > > 1) don't rewrite deprecated endpoints from
> > > > Perl to
> > > > > > Go
> > > > > > > > > > > > 2) in the Perl handler, add a deprecation
> > > > notice
> > > > > > to the
> > > > > > > > > > > > "alerts"
> > > > > > > > > > > > section of the response
> > > > > > > > > > > > 3) once a deprecated API has been deprecated
> > > > for
> > > > > > one major
> > > > > > > > > > > > release,
> > > > > > > > > > > > delete it from the API in the subsequent
> > > major
> > > > > > release (e.g. if
> > > > > > > > > > > > deprecation notices are added and released in
> > > > ATC
> > > > > > 4.x, we can
> > > > > > > > > > > > remove
> > > > > > > > > > > > the deprecated endpoint in ATC 5.x)
> > > > > > > > > > > >
> > > > > > > > > > > > Rob, from your latest proposal to remove the
> > > > ATS
> > > > > > config API
> > > > > > > > > > > > endpoints
> > > > > > > > > > > > from 1.x, I thought we were past this issue
> > > > > > already and could
> > > > > > > > > > > > finally
> > > > > > > > > > > > agree to remove deprecated endpoints from 1.x
> > > > > > (following steps
> > > > > > > > > > > > 1-3
> > > > > > > > > > > > above) without requiring an all-new 2.x API.
> > > > Are
> > > > > > we not
> > > > > > > > > > > > actually past
> > > > > > > > > > > > that? You came up with valid reasons for
> > > > applying
> > > > > > these steps
> > > > > > > > > > > > to the
> > > > > > > > > > > > ATS config API endpoints, and I think it's
> > > > fair to
> > > > > > say that we
> > > > > > > > > > > > can
> > > > > > > > > > > > come up with valid reasons for applying these
> > > > > > steps to other
> > > > > > > > > > > > deprecated API endpoints as well.
> > > > > > > > > > > >
> > > > > > > > > > > > It seems like the real problem is actually
> > > > > > agreeing on which
> > > > > > > > > > > > _specific_ endpoints we should be able to
> > > > follow
> > > > > > steps 1-3 for.
> > > > > > > > > > > > IMO I
> > > > > > > > > > > > think it should apply to any 1.x endpoints
> > > > that we
> > > > > > have valid
> > > > > > > > > > > > reasons
> > > > > > > > > > > > for not carrying forward to Go, including but
> > > > not
> > > > > > limited to:
> > > > > > > > > > > > - the endpoints are dangerous and pose a risk
> > > > to
> > > > > > the overall
> > > > > > > > > > > > Traffic
> > > > > > > > > > > > Control system
> > > > > > > > > > > > - the endpoints have been obsoleted by other
> > > > > > endpoints
> > > > > > > > > > > > - the endpoints don't seem to serve a true
> > > > purpose
> > > > > > or don't
> > > > > > > > > > > > seem to be
> > > > > > > > > > > > valuable to the project anymore
> > > > > > > > > > > > - the task of rewriting the endpoints from
> > > > Perl to
> > > > > > Go
> > > > > > > > > > > > represents an
> > > > > > > > > > > > unreasonable amount of work for the project
> > > > > > > > > > > >
> > > > > > > > > > > > Obviously most of these reasons are
> > > > subjective, so
> > > > > > we'd have to
> > > > > > > > > > > > come
> > > > > > > > > > > > to some level of consensus on deprecating
> > > > > > particular endpoints
> > > > > > > > > > > > first,
> > > > > > > > > > > > but we should be able to follow steps 1-3 for
> > > > > > whatever
> > > > > > > > > > > > endpoints we
> > > > > > > > > > > > decide.
> > > > > > > > > > > >
> > > > > > > > > > > > Making breaking changes shouldn't be taken
> > > > > > lightly, but we need
> > > > > > > > > > > > a path
> > > > > > > > > > > > forward that allows some calculated, breaking
> > > > > > changes where
> > > > > > > > > > > > necessary.
> > > > > > > > > > > > Otherwise, we are unnecessarily holding the
> > > > > > project back.
> > > > > > > > > > > >
> > > > > > > > > > > > - Rawlin
> > > > > > > > > > > >
> > > > > > > > > > > > On Tue, Nov 12, 2019 at 11:44 AM Jeremy
> > > > Mitchell <
> > > > > > > > > > > > mitchell852@gmail.com>
> > > > > > > > > > > wrote:
> > > > > > > > > > > > > Yeah, technically a "rewrite" of 1.x means
> > > > > > shifting the
> > > > > > > > > > > > > implementation
> > > > > > > > > > > from
> > > > > > > > > > > > > Perl to Go for ALL of 1.x. If you leave a
> > > > subset
> > > > > > of 1.x in
> > > > > > > > > > > > > Perl, then
> > > > > > > > > > > you
> > > > > > > > > > > > > have to call it a "partial rewrite".
> > > > > > > > > > > > >
> > > > > > > > > > > > > So, assuming we want to do a full rewrite
> > > (no
> > > > > > more perl for
> > > > > > > > > > > > > the api
> > > > > > > > > > > > > whatsoever), I think that means every 1.x
> > > api
> > > > > > endpoint (good
> > > > > > > > > > > > > or bad)
> > > > > > > > > > > needs
> > > > > > > > > > > > > to be rewritten to Go so we can remove the
> > > > > > dependency on
> > > > > > > > > > > > > perl.
> > > > > > > > > > > > >
> > > > > > > > > > > > > However, I still like marking many as
> > > > > > "deprecated" so we
> > > > > > > > > > > > > train users
> > > > > > > > > > > to use
> > > > > > > > > > > > > "better" endpoints that we envision will
> > > > exist
> > > > > > in 2.x and
> > > > > > > > > > > > > beyond.
> > > > > > > > > > > > >
> > > > > > > > > > > > > Jeremy
> > > > > > > > > > > > >
> > > > > > > > > > > > > On Tue, Nov 12, 2019 at 11:22 AM Hoppal,
> > > > Michael
> > > > > > <
> > > > > > > > > > > Michael_Hoppal@comcast.com>
> > > > > > > > > > > > > wrote:
> > > > > > > > > > > > >
> > > > > > > > > > > > > > We do not have a lot of deprecated routes
> > > > as
> > > > > > of now as we
> > > > > > > > > > > > > > focused on
> > > > > > > > > > > > > > routes that are more used first.
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > I know on a lot of the remaining rewrites
> > > > > > there has been
> > > > > > > > > > > > > > discussion
> > > > > > > > > > > on the
> > > > > > > > > > > > > > matching Github issues on potential
> > > > > > deprecation (I cannot
> > > > > > > > > > > > > > think of
> > > > > > > > > > > the
> > > > > > > > > > > > > > exact count but enough to warrant this
> > > > email
> > > > > > __)
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > It is key to note that I am not arguing
> > > for
> > > > > > removal of
> > > > > > > > > > > > > > endpoints
> > > > > > > > > > > within
> > > > > > > > > > > > > > the current API version but if there is
> > > an
> > > > > > endpoint we
> > > > > > > > > > > > > > agree should
> > > > > > > > > > > be
> > > > > > > > > > > > > > deprecated going forward to put a message
> > > > > > within the
> > > > > > > > > > > > > > response and on
> > > > > > > > > > > the
> > > > > > > > > > > > > > docs.
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > And thanks for the input sounds like you
> > > > are
> > > > > > +1 on
> > > > > > > > > > > > > > rewriting all
> > > > > > > > > > > routes to
> > > > > > > > > > > > > > Golang as Brennan is.
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > On 11/12/19, 9:06 AM, "Robert O Butts" <
> > > > > > rob@apache.org>
> > > > > > > > > > > > > > wrote:
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > Whether we rewrite a route in Go is
> > > an
> > > > > > implementation
> > > > > > > > > > > > > > detail. To
> > > > > > > > > > > the
> > > > > > > > > > > > > > interface, to users, it doesn't
> > > matter
> > > > > > whether a route
> > > > > > > > > > > > > > is
> > > > > > > > > > > rewritten or
> > > > > > > > > > > > > > not.
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > But our API follows Semantic
> > > > Versioning,
> > > > > > in order to
> > > > > > > > > > > > > > not break
> > > > > > > > > > > users.
> > > > > > > > > > > > > > We
> > > > > > > > > > > > > > can't just remove endpoints that some
> > > > of
> > > > > > us don't use,
> > > > > > > > > > > > > > and
> > > > > > > > > > > assume other
> > > > > > > > > > > > > > people don't, maybe people who never
> > > > speak
> > > > > > up on the
> > > > > > > > > > > > > > mailing
> > > > > > > > > > > list.
> > > > > > > > > > > > > > We'll
> > > > > > > > > > > > > > never gain a big userbase if we keep
> > > > > > breaking users.
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > Per the _project_ SemVer, once we
> > > have
> > > > API
> > > > > > 2.0, we can
> > > > > > > > > > > > > > deprecate
> > > > > > > > > > > API
> > > > > > > > > > > > > > 1.x,
> > > > > > > > > > > > > > and in the next major _project_
> > > > release,
> > > > > > remove API
> > > > > > > > > > > > > > 1.x.
> > > > > > > > > > > Irrespective
> > > > > > > > > > > > > > of
> > > > > > > > > > > > > > Perl or Go.
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > My big concern is, API 2.0 is a big
> > > > > > project. How long
> > > > > > > > > > > > > > has the
> > > > > > > > > > > rewrite
> > > > > > > > > > > > > > to Go
> > > > > > > > > > > > > > taken? Do we really believe designing
> > > > and
> > > > > > implementing
> > > > > > > > > > > > > > a
> > > > > > > > > > > completely
> > > > > > > > > > > > > > new API
> > > > > > > > > > > > > > will be any less time?
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > I don't want killing Perl to have to
> > > > wait
> > > > > > on that.
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > I know it feels like a waste to
> > > rewrite
> > > > > > routes that you
> > > > > > > > > > > > > > don't
> > > > > > > > > > > use, and
> > > > > > > > > > > > > > probably few people do. But that's
> > > the
> > > > > > cost of a stable
> > > > > > > > > > > > > > project.
> > > > > > > > > > > How
> > > > > > > > > > > > > > many
> > > > > > > > > > > > > > "deprecated" routes are there? If it
> > > > comes
> > > > > > down to
> > > > > > > > > > > > > > taking the
> > > > > > > > > > > > > > development
> > > > > > > > > > > > > > time to rewrite them so we can kill
> > > > Perl
> > > > > > faster, or
> > > > > > > > > > > > > > leaving Perl
> > > > > > > > > > > > > > around, I
> > > > > > > > > > > > > > vote we just do the work and kill
> > > Perl.
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > >If we don't rewrite them, then Perl
> > > > will
> > > > > > last until
> > > > > > > > > > > > > > API 2.0 has
> > > > > > > > > > > been
> > > > > > > > > > > > > > designed, released and then *another
> > > > full
> > > > > > major release
> > > > > > > > > > > > > > cycle*.
> > > > > > > > > > > That's
> > > > > > > > > > > > > > way
> > > > > > > > > > > > > > too long to have two codebases for
> > > the
> > > > > > same component,
> > > > > > > > > > > > > > IMO,
> > > > > > > > > > > especially
> > > > > > > > > > > > > > since the rewrite is already 50%
> > > > complete.
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > +1
> > > > > > > > > > > > > >
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > On Tue, Nov 12, 2019 at 8:54 AM ocket
> > > > 8888
> > > > > > <
> > > > > > > > > > > > > > ocket8888@gmail.com>
> > > > > > > > > > > > > > wrote:
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > > I vote that by and large we DO
> > > > rewrite
> > > > > > them, with
> > > > > > > > > > > > > > exceptions
> > > > > > > > > > > for
> > > > > > > > > > > > > > routes
> > > > > > > > > > > > > > > that just plain don't work, even in
> > > > > > Perl. Those are
> > > > > > > > > > > > > > few,
> > > > > > > > > > > though.
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > If we don't rewrite them, then Perl
> > > > will
> > > > > > last until
> > > > > > > > > > > > > > API 2.0
> > > > > > > > > > > has been
> > > > > > > > > > > > > > > designed, released and then
> > > *another
> > > > > > full major
> > > > > > > > > > > > > > release cycle*.
> > > > > > > > > > > > > > That's way
> > > > > > > > > > > > > > > too long to have two codebases for
> > > > the
> > > > > > same
> > > > > > > > > > > > > > component, IMO,
> > > > > > > > > > > > > > especially
> > > > > > > > > > > > > > > since the rewrite is already 50%
> > > > > > complete.
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > On Tue, Nov 12, 2019 at 8:43 AM
> > > > Hoppal,
> > > > > > Michael <
> > > > > > > > > > > > > > > Michael_Hoppal@comcast.com>
> > > > > > > > > > > > > > > wrote:
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > As the Traffic Ops API is being
> > > > > > rewritten from Perl
> > > > > > > > > > > > > > to Golang
> > > > > > > > > > > > > > there has
> > > > > > > > > > > > > > > > been several routes that have
> > > been
> > > > > > deprecated and
> > > > > > > > > > > > > > probably
> > > > > > > > > > > more to
> > > > > > > > > > > > > > come.
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > In the deprecation efforts I have
> > > > seen
> > > > > > two
> > > > > > > > > > > > > > strategies:
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > * The route IS NOT rewritten
> > > > from
> > > > > > Perl to
> > > > > > > > > > > > > > Golang and a
> > > > > > > > > > > > > > deprecation
> > > > > > > > > > > > > > > > notice is added to the alert
> > > > response
> > > > > > in the Perl
> > > > > > > > > > > > > > handler
> > > > > > > > > > > > > > > > * The route IS rewritten from
> > > > Perl
> > > > > > to Golang
> > > > > > > > > > > > > > and a
> > > > > > > > > > > deprecation
> > > > > > > > > > > > > > notice
> > > > > > > > > > > > > > > > is added to the alert response in
> > > > the
> > > > > > Golang
> > > > > > > > > > > > > > handler
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > I think we should have
> > > consistency
> > > > in
> > > > > > our approach
> > > > > > > > > > > > > > and
> > > > > > > > > > > wanted to
> > > > > > > > > > > > > > get
> > > > > > > > > > > > > > > > people’s thoughts.
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > I would vote that we do not
> > > > rewrite a
> > > > > > deprecated
> > > > > > > > > > > > > > route from
> > > > > > > > > > > Perl to
> > > > > > > > > > > > > > > Golang.
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > Thanks,
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > Michael
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > >
> > > > > > > > > > > > > >
> > > > > > > > > > > > > >
> > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > > >
> > > > > >
> > > > > >
> > > > > >
> > > >
> > >
>
>
>
>
Re: [EXTERNAL] Re: Traffic Ops Route Deprecation Strategy
Posted by "Schmidt, Andrew (Contractor)" <An...@comcast.com>.
+1 Makes sense to me. Is 2.0 the target for dropping the PERL implementation?
On 12/2/19, 12:19 PM, "Gray, Jonathan" <Jo...@comcast.com> wrote:
+1
I think the only point of contention was if the API v2.0 would be in ATC v4.0 or not. Either way is fine with me, and more time to know this is coming makes sense.
Jonathan G
On 12/2/19, 11:32 AM, "Rawlin Peters" <ra...@apache.org> wrote:
Hey all,
A bunch of us got together and discussed this issue offline, and we
got consensus to go with Option 1 - Graduate current API from 1.x to
2.x - allowing us to leave behind routes in 1.x that are
dangerous/deprecated. Deprecated endpoints will not be rewritten from
Perl to Go.
So, we will graduate the latest minor version of existing 1.x Go
endpoints to 2.0. For example, today we have
/api/1.2/deliveryservices, /api/1.3/deliveryservices, and
/api/1.4/deliveryservices. We will take the existing handler for
/api/1.4/deliveryservices and promote it to /api/2.0/deliveryservices.
We will follow this example for other 1.x Go endpoints with multiple
minor versions as well.
API endpoints that are deemed broken, dangerous, or deprecated will
not be promoted to API 2.0. This will likely include the ~7 endpoints
that I've mentioned in this thread already, but we haven't yet
compiled the full list of endpoints that won't be promoted to 2.0. We
will come up with that list (along with its criteria) soon and bring
that list to the mailing list for everyone to look over and agree
upon. Deprecated endpoints will be updated to return deprecation
alerts where applicable.
This API graduation will likely miss the boat for 4.0 but will
hopefully land in 4.1 or 4.2 depending on how long it takes. So this
would be the tentative timetable:
TC 4.0:
- API 1.x supported, some deprecation notices
TC 4.1:
- API 1.x still supported, deprecation notices added to endpoints not
graduated to 2.0
- API 2.0 supported, consisting of 1.x endpoints that were graduated
- starting with this release, you need to start migrating external
clients off of 1.x over to 2.0
TC 4.2:
- internal clients (e.g. TM, TR, etc) will be migrated off API 1.x
over to 2.0. Doing this step after 4.1 adds confidence that 1.x is
still supported alongside 2.0 in order to provide a smooth migration
period for API clients.
TC 5.0:
- API 1.x no longer supported, only API 2.x is supported
Those who took part in that offline discussion, please keep me honest
here with a +1 if we did in fact come to a rough consensus on this
option. If you weren't part of that offline discussion, feel free to
voice your vote on this issue as well (I believe a bunch of people
were already +1 on this option a few weeks ago).
- Rawlin
On Tue, Nov 19, 2019 at 12:23 PM Robert O Butts <ro...@apache.org> wrote:
>
> > - /riak/stats
>
> Looking at the Perl, this looks easy to rewrite. It's just proxying a
> request to Riak.
>
> +1 on rewriting this, for API stability, even if we don't "value" the
> endpoint as a project.
>
> >- /cachegroup_fallbacks
>
> Likewise +1 on rewriting. Not as trivial as /riak/stats, but it also looks
> easy to me, looks like a standard CRUD endpoint. +1 on leaving a
> "deprecated, please prefer X" message in, and +1 in removing in the big API
> rewrite/refactor/2.0.
>
> > - /capabilities (all methods except for GET) -- capabilities are
> built-in, should not be changed at runtime
> > - /api_capabilities (all methods except for GET) -- same reasons as
> /capabilities
>
> I actually don't agree with this - because they're useful to plugins. IMO
> it should be possible to add and modify Capabilities and API Capabilities,
> because they can be used by plugins which add new endpoints, to control
> access to them. If there's disagreement there, I'd vote we take that to a
> separate thread.
>
> But in terms of the endpoints themselves, the Perl also looks easy to
> rewrite, looks like standard CRUD. -1 on deprecating or removing, +1 on
> rewriting.
>
> > - /cdns/{name}/configs/routing -- this was a half-baked attempt at
> redoing CrConfig.json -- cannot possibly be of use to anyone in its current
> state
>
> Ugh. I agree with the sentiment; and this one looks slightly less trivial.
> But I personally don't see it as an unreasonable amount of work. For the
> sake of API stability, I'm personally willing to volunteer to rewrite this
> one, including API tests to make fixing easy if the schema changes (which
> it appears to have in master). And hopefully it can just exist until the
> fabled API Rewrite. Is that acceptable?
>
> > - the ATS config generation endpoints -- Rob actually proposed to remove
> these from 1.x due to danger of being out of sync w/ cache-side config
> generator
>
> I'm personally on the fence about this one. Like I said in that thread, I'm
> the last one to want to break users, and I don't take this lightly. If,
> after I made the case for it, I'm the only one, I might be inclined to
> retract my proposal, and just continue to maintain them. Maybe some more
> API tests would help with that. I think I'm the primary dev on the ATS
> config gen at this point anyway.
>
> > - ATS Config Files Stuff Aren't these all already in Go anyway? Whether
> or not they should be removed from the API aside, I don't think anything
> actually needs to hit Perl anymore.
>
> Correct.
>
> > - /hwinfo/dtdata
>
> I think the 'render' call here is just rendering the JSON. But, the
> endpoint itself is just hanging and not returning for me, I'm not sure why.
> I'd like to volunteer to look into this endpoint as well. I think you might
> be right, this one might be unreasonable to rewrite. But if it is
> reasonable, I'd like to just do it. I'd like to at least look into it.
>
>
> On Tue, Nov 19, 2019 at 12:05 PM ocket 8888 <oc...@gmail.com> wrote:
>
> > I'm generally +1 for option 2
> >
> > regarding those specific endpoints:
> >
> > - /riak/stats if this is easy and doesn't expose anything dangerous, we
> > should do it IMO.
> > - /cachegroup_fallbacks should be pretty trivial, so I still say do it
> > - /capabilities since these aren't actually used for anything, I think it's
> > pretty safe (and easy) to rewrite. Plus there's already a PR open for
> > exactly that
> > - /api_capabilities dropping this one could be problematic, just because
> > people may depend on it to tell them what endpoints exist. To be clear, I
> > think it ought to be deprecated, but unfortunately I just don't think it's
> > terribly unlikely that this is built into scripts and tools. The actual
> > code for the rewrite should be easy too, although maintaining it is a pain.
> >
> > - ATS Config Files Stuff Aren't these all already in Go anyway? Whether or
> > not they should be removed from the API aside, I don't think anything
> > actually needs to hit Perl anymore.
> >
> > - /cdn/{{name}}/configs/routing CRConfigs are large and complex, so if this
> > never worked anyway maybe we should just add a Go handler for it that
> > returns a '410 Gone' response (or '501 Not Implemented' is the other one
> > I've heard suggested) and leave the actual logic behind in the Perl
> > - /hwinfo/dtdata if this calls into Mojolicious rendering, returns a
> > non-json response, and/or contains information that in either content or
> > encoding is meaningless to any sane client then it should also return a
> > 410/501 and get left behind in Perl.
> >
> > On Tue, Nov 19, 2019 at 11:08 AM Rawlin Peters <ra...@apache.org> wrote:
> >
> > > While I appreciate and mostly agree with your essay on the importance
> > > of not breaking users (or rather, breaking them in predictable ways),
> > > I for one would really like to meet at some middle ground with option
> > > 2. I'm not asking to remove 50 high-usage endpoints from 1.x. IIRC
> > > I've only specifically proposed "leaving behind" a few specific Perl
> > > endpoints that I believe serve no purpose, have no valid reason to be
> > > used, and/or are dangerous to the system:
> > >
> > > - /riak/stats -- informational, was used for debugging the initial
> > > Riak integration. This one might be pretty easy to rewrite since it
> > > just calls a corresponding riak API and returns the result via TO, so
> > > I could be on board with rewriting it if necessary.
> > > - /cachegroup_fallbacks -- these endpoints were released in 3.0 before
> > > we managed to augment the existing /cachegroups API to handle
> > > fallbacks, so I could get on board with rewriting these if we really
> > > think it's important.
> > > - /capabilities (all methods except for GET) -- capabilities are
> > > built-in, should not be changed at runtime
> > > - /api_capabilities (all methods except for GET) -- same reasons as
> > > /capabilities
> > > - /cdns/{name}/configs/routing -- this was a half-baked attempt at
> > > redoing CrConfig.json -- cannot possibly be of use to anyone in its
> > > current state
> > > - the ATS config generation endpoints -- Rob actually proposed to
> > > remove these from 1.x due to danger of being out of sync w/ cache-side
> > > config generator
> > > - /hwinfo/dtdata -- AFAIK the "dtdata" endpoints were only
> > > used/required by the old Perl Mojolicious UI, no valid reason to be
> > > using this as an external API client. In fact, I'm not even sure _how_
> > > this endpoint can be used outside of the Mojolicious framework/UI. It
> > > calls the Mojolicious `render` function, which usually means rendering
> > > an embedded Perl HTML template given some data. This is one I would
> > > consider completely unreasonable to rewrite to Go.
> > >
> > > Can we get on board with leaving at least these 1.x endpoints behind
> > > in the Perl? Doing a cursory skim over the remaining 1.x endpoints to
> > > rewrite, it _seems_ like everything else would fall into the "simpler
> > > to just rewrite rather than gain consensus to remove" category.
> > >
> > > - Rawlin
> > >
> > > On Tue, Nov 19, 2019 at 9:19 AM Robert O Butts <ro...@apache.org> wrote:
> > > >
> > > > >Option 1 - Graduate current API from 1.x to 2.x - allowing us to leave
> > > > behind routes in 1.x that are dangerous or deprecated
> > > > >Option 2 - Rewrite all endpoints from Perl to Golang in current
> > version
> > > as
> > > > we are doing now.
> > > >
> > > > +1 on either Option 1 or Option 2, without a strong preference. Option
> > 1
> > > > has advantages of letting us also safely remove the Cache Config
> > > endpoints
> > > > and implement Layered Profiles. But Option 2 also has the advantage of
> > > not
> > > > requiring users to upgrade.
> > > >
> > > > > Option 3 - Only rewrite endpoints from Perl to Golang in current
> > > version
> > > > that we agree are not dangerous or should deprecate
> > > >
> > > > -1. As before, it breaks users unpredictably.
> > > >
> > > > >It will literally outright break clients (*all* of them, not just
> > > *some*)
> > > >
> > > > This is technically correct: technically, new versions 'break' all
> > users.
> > > > The difference is, it does it in a predictable, safe way that users can
> > > > handle, inspect, and safely upgrade.
> > > >
> > > > >Can someone please explain to me why (A) artificially graduating the
> > > > current 1.x API to 2.x, and removing the deprecated endpoints from 1.x,
> > > is
> > > > better for API clients overall than (B) just removing deprecated
> > > endpoints
> > > > from 1.x? Existing 1.x clients would gain absolutely nothing from being
> > > > forced to update to 2.x like this.
> > > >
> > > > >for zero gain in value
> > > >
> > > > Users would gain something. There is a very specific gain.
> > > >
> > > > The gain is that the breakage is predictable. People frequently upgrade
> > > > minor versions automatically, either running commands without thinking,
> > > or
> > > > even automating the entire process.
> > > >
> > > > For TC, this applies to both CDN operators, and their end-users.
> > > >
> > > > When the minor versions contain breaking changes, that automatic
> > upgrade
> > > > then breaks. In this case, any scripts or tools they have using the
> > > removed
> > > > endpoints will break. If those scripts are important, this could bring
> > > down
> > > > their CDN, or break their Delivery Service, or otherwise cause an
> > outage.
> > > >
> > > > Contrawise, users expect breaking changes in Major versions, and don't
> > > > typically automatically upgrade them. Rather, they'll manually inspect
> > > the
> > > > update, do testing, and check the changelog for breaking changes.
> > > >
> > > > It's not a technical thing, it's a human thing. Humans automatically
> > > > upgrade minor versions, and write scripts to do so, and dependency
> > files
> > > to
> > > > allow it, because the software industry taught people that minor
> > versions
> > > > shouldn't break anything.
> > > >
> > > > If we start breaking minor versions, initially users will experience
> > > > breakage. Not all users, and not all catastrophic. But some users will
> > > > experience catastrophic outages. And it will be our fault as a project.
> > > And
> > > > after enough times, they'll fork TC, or find another product, or write
> > > > their own; and stop participating in the community, or never start.
> > > >
> > > > Then, once they've learned that we can't be trusted not to break
> > things,
> > > > they'll experience the 2nd problem: where nothing can be safely
> > upgraded,
> > > > and the smallest change requires long and expensive testing to verify a
> > > > single minor version or even patch upgrade, and still risk missing
> > > > something and bringing their CDN down. To be honest, we're pretty much
> > > > already here. It already costs our dev and ops teams unbelievable
> > amounts
> > > > of time and person-hours of testing for every minor upgrade. That time
> > > far
> > > > exceeds the development time to just rewrite the small endpoints we
> > don't
> > > > like anymore (and tangentially, also the time to write the Integration
> > > and
> > > > E2E tests TC desperately needs).
> > > >
> > > > ATC has so much potential. There aren't many OSS CDNs out there, and
> > > > there's certainly the demand. We have the potential to become a huge
> > > > project, which for us existing users, translates into potentially an
> > > > enormous amount of free development. But we'll never get there if we
> > keep
> > > > breaking things.
> > > >
> > > > And for my particular organization (Comcast), we have a hard enough
> > time
> > > > upgrading. Our upgrades, especially Traffic Ops, are extremely painful,
> > > and
> > > > require enormous amounts of manual testing. Even then, we frequently
> > have
> > > > issues that require rollback, sometimes even an outage; even after
> > those
> > > > unbelievably amounts of developer and operator time testing. We
> > > desperately
> > > > need a system that is safer and easier to upgrade. Breaking changes are
> > > > always going to require more testing; but we can at least make minor
> > > > versions safe to upgrade.
> > > >
> > > > > I know I am changing my initial stance but I am +1 on rewriting all
> > the
> > > > Perl endpoints to Golang and not breaking the API contract we have in
> > > > place. Most of the API routes to rewrite are rather simple to port and
> > I
> > > > think would be faster than getting consensus on actually deprecating it
> > > > from the mailing list.
> > > >
> > > > I'm inclined to agree with this. The things I've seen marked
> > "deprecated"
> > > > seem to mostly be small, and look like they'd be pretty easy to
> > rewrite.
> > > > But I haven't seen the whole list. Is there a list somewhere, so the
> > > > community can see it to better decide?
> > > >
> > > >
> > > > On Tue, Nov 19, 2019 at 7:24 AM Hoppal, Michael <
> > > Michael_Hoppal@comcast.com>
> > > > wrote:
> > > >
> > > > > I would love to drive this to some type of consensus on rewrite
> > > strategy
> > > > > as we have a good amount of resources (and growing) working on the
> > > rewrite
> > > > > and I want to keep that momentum.
> > > > >
> > > > > Sounds like there is three options from the email thread (please add
> > > any I
> > > > > missed)
> > > > >
> > > > > Option 1 - Graduate current API from 1.x to 2.x - allowing us to
> > leave
> > > > > behind routes in 1.x that are dangerous or deprecated
> > > > > Option 2 - Rewrite all endpoints from Perl to Golang in current
> > > version as
> > > > > we are doing now.
> > > > > Option 3 - Only rewrite endpoints from Perl to Golang in current
> > > version
> > > > > that we agree are not dangerous or should deprecate
> > > > >
> > > > > I am +1 for option 2 (which I know is different from my original
> > > stance).
> > > > >
> > > > > Which direction would others like to go?
> > > > >
> > > > > Thanks,
> > > > >
> > > > > Michael
> > > > >
> > > > >
> > > > > I know I am changing my initial stance but I am +1 on rewriting all
> > the
> > > > > Perl endpoints to Golang and not breaking the API contract we have in
> > > > > place. Most of the API routes to rewrite are rather simple to port
> > and
> > > I
> > > > > think would be faster than getting consensus on actually deprecating
> > it
> > > > > from the mailing list.
> > > > >
> > > > > On 11/16/19, 3:31 PM, "Rawlin Peters" <ra...@apache.org> wrote:
> > > > >
> > > > > > Moving to 2.x means that it's my fault as a consumer of the API
> > > if I
> > > > > was still using the deprecated endpoint (and that change should be
> > > part of
> > > > > a larger upgrade plan). Simply removing the deprecated routes,
> > > > > retroactively, means I can't rely on having a transition period to
> > get
> > > to
> > > > > 2.0 outside of a large completely down upgrade window.
> > > > >
> > > > > There would be a transition period either way. Endpoints would be
> > > > > deprecated in ATC 4.0 and removed in 5.0. The release notes for
> > 4.0
> > > > > would clearly list out the deprecated endpoints. If they have
> > > > > replacements, there would be instructions on how to transition to
> > > the
> > > > > replacement. Deprecated endpoints would return alerts. After
> > > upgrading
> > > > > to 4.0, operators should be well aware that certain endpoints are
> > > > > being removed in 5.0, and it's on them to start using new
> > > endpoints,
> > > > > change workflows, etc.
> > > > >
> > > > > By forcing operators to upgrade all of their clients to start
> > using
> > > > > API 2.x, it just adds to their burden of upgrading when that time
> > > > > comes. Rather than surgically fixing the clients of the 2% of
> > > > > deprecated API endpoints before upgrading to ATC 5.0, operators
> > > would
> > > > > have to fix every single client they have to use API 2.x before
> > > > > upgrading to 5.0. Is that exercise really worth it? Most people
> > > would
> > > > > probably be blissfully unaware that the "POST
> > > /api/1.1/capabilities"
> > > > > endpoint disappeared (among other deprecated endpoints). Less
> > > footguns
> > > > > in the system, and they didn't even have to update their clients.
> > > > >
> > > > > Anyways, circling back to Michael's original question, it sounds
> > > like
> > > > > either option we choose, we at least have consensus on this
> > > > > deprecation strategy:
> > > > > - The route IS NOT rewritten from Perl to Golang and a
> > deprecation
> > > > > notice is added to the alert response in the Perl handler
> > > > >
> > > > > If we're content to force all API clients to update their clients
> > > to
> > > > > 2.x for zero gain in value, at least we won't be wasting work by
> > > > > rewriting deprecated Perl endpoints to Go.
> > > > >
> > > > > - Rawlin
> > > > >
> > > > > On Fri, Nov 15, 2019 at 7:06 PM Gray, Jonathan
> > > > > <Jo...@comcast.com> wrote:
> > > > > >
> > > > > > Moving to 2.x means that it's my fault as a consumer of the API
> > > if I
> > > > > was still using the deprecated endpoint (and that change should be
> > > part of
> > > > > a larger upgrade plan). Simply removing the deprecated routes,
> > > > > retroactively, means I can't rely on having a transition period to
> > get
> > > to
> > > > > 2.0 outside of a large completely down upgrade window. Depending on
> > > how
> > > > > the API is used, even in our own components such as TM, that could
> > > prove
> > > > > disastrous. For the older minor revs like 1.1/1.2, I could live with
> > > > > dropping those as part of a major project release (including upgrade
> > > > > instructions outlining what releases you must be on beforehand).
> > > Currently
> > > > > 1.3 I view as stable enough to code to or use a native client library
> > > from,
> > > > > and 1.4 is my risk/overhead to take for new stuff as it's still
> > > evolving
> > > > > and improving. Going to 1.5 or 2.0 moves that forward. The path of
> > > least
> > > > > surprise for humans is to expect big shifts on those major revs.
> > > Removal
> > > > > falls under that umbrella and gets as a consumer not to strictly
> > focus
> > > on
> > > > > endpoint->endpoint moves, but rather workflow shifts and maybe now I
> > > have
> > > > > to use the API a different way than before. I believe getting this
> > > right
> > > > > is something that would greatly help us understand does X component
> > at
> > > > > version 1 work with Y component at version 2, because now you're
> > > asking do
> > > > > components X and Y both speak protocol A. With major releases we can
> > > work
> > > > > through that each time, but it's error prone and requires experts in
> > > all
> > > > > products to agree upon. If you do nothing but consume major
> > releases,
> > > and
> > > > > then conduct your own impact evaluation having read the upgrade
> > > > > instructions thoroughly, maybe you'd be able to smoothly transition.
> > > > > >
> > > > > > Jonathan G
> > > > > >
> > > > > >
> > > > > > On 11/15/19, 5:24 PM, "Rawlin Peters" <ra...@apache.org>
> > wrote:
> > > > > >
> > > > > > Can someone please explain to me why (A) artificially
> > > graduating
> > > > > the
> > > > > > current 1.x API to 2.x, and removing the deprecated
> > endpoints
> > > > > from
> > > > > > 1.x, is better for API clients overall than (B) just
> > removing
> > > > > > deprecated endpoints from 1.x? Existing 1.x clients would
> > > gain
> > > > > > absolutely nothing from being forced to update to 2.x like
> > > this.
> > > > > >
> > > > > > The way I see the two choices:
> > > > > > A) break every single API client out there
> > > > > > B) break only clients of APIs that we've decided upon as a
> > > > > community
> > > > > > to be deprecated
> > > > > >
> > > > > > IMO, when we have to break things, we should try break the
> > > least
> > > > > > amount of clients possible. All I'm looking for is a
> > valuable
> > > > > reason
> > > > > > to choose option A over option B, that isn't just "because
> > > > > SemVer".
> > > > > >
> > > > > > - Rawlin
> > > > > >
> > > > > > On Wed, Nov 13, 2019 at 3:24 PM Schmidt, Andrew
> > (Contractor)
> > > > > > <An...@comcast.com> wrote:
> > > > > > >
> > > > > > > If we decide to get rid of the API versioning it makes
> > > sense
> > > > > to me that we would do that on a major version upgrade, probably to
> > > the ATC
> > > > > version. But I think it would be too early to do it on the 2.0 API
> > > version.
> > > > > I like this compromise +1. I was wondering why we were stuck on the
> > > idea
> > > > > that 2.0 had to be an API rewrite. That seems like it’s a long way
> > > out. 2.0
> > > > > = deprecations and fixes. 3.0 or 5.0 = api rewrite and maybe version
> > > > > convergence.
> > > > > > >
> > > > > > > On 11/13/19, 11:52 AM, "ocket8888@gmail.com" <
> > > > > ocket8888@gmail.com> wrote:
> > > > > > >
> > > > > > > Lemme lead with: making breaking changes on a major
> > > > > version boundary
> > > > > > > gets +1 from me - no reason the API needs to be fully
> > > > > redesigned for
> > > > > > > any particular number.
> > > > > > >
> > > > > > > But I just gotta bring this up again: if we want to
> > be
> > > > > able to
> > > > > > > deprecate an endpoint for a major release and remove
> > > it in
> > > > > the next
> > > > > > > without needing to deal with the messiness of API
> > > > > versions, we could
> > > > > > > always just get rid of API versions. Major ATC
> > > revisions
> > > > > should be
> > > > > > > easier to come by with our new release strategy.
> > > > > > >
> > > > > > > On Wed, 2019-11-13 at 10:23 -0700, Rawlin Peters
> > wrote:
> > > > > > > > > It is a compromise - it's an inconvenience to
> > > users,
> > > > > forcing them
> > > > > > > > > to fix
> > > > > > > > scripts to upgrade. Which isn't good for user
> > > experience
> > > > > either; but
> > > > > > > > it
> > > > > > > > doesn't outright break people. It's a compromise
> > I'm
> > > > > willing to live
> > > > > > > > with.
> > > > > > > >
> > > > > > > > It will literally outright break clients (*all* of
> > > them,
> > > > > not just
> > > > > > > > *some*). We have been down this road before, so
> > I'll
> > > > > just repeat what
> > > > > > > > I said when this exact conversation came up months
> > > > > ago[1]:
> > > > > > > >
> > > > > > > > "So it basically boils down to:
> > > > > > > > A) promote Go 1.x routes to 2.x and eventually
> > break
> > > ALL
> > > > > users of the
> > > > > > > > 1.x API when the time comes to remove 1.x
> > > > > > > > B) deprecate and remove some 1.x Perl routes and
> > > > > POSSIBLY break SOME
> > > > > > > > users"
> > > > > > > >
> > > > > > > > Either way we slice and dice it with versioning,
> > > > > removing an endpoint
> > > > > > > > means that clients of that endpoint will break. By
> > > > > graduating only
> > > > > > > > the
> > > > > > > > "good stuff" from 1.x to 2.x, and removing
> > endpoints
> > > > > from 1.x, we
> > > > > > > > will
> > > > > > > > unnecessarily break all 1.x clients when the time
> > > comes
> > > > > to remove the
> > > > > > > > rest of 1.x. Instead, we can choose to break only
> > > > > clients of 1.x APIs
> > > > > > > > that we already deemed dangerous, unnecessary,
> > > useless,
> > > > > etc. enough
> > > > > > > > to
> > > > > > > > deprecate.
> > > > > > > >
> > > > > > > > Why break 100% of our API clients when we can
> > choose
> > > to
> > > > > break only a
> > > > > > > > small number of clients that happen to use
> > > > > > > > dangerous/unnecessary/useless APIs? Why would we
> > > break
> > > > > our API
> > > > > > > > clients
> > > > > > > > unnecessarily for no added value? There is no sense
> > > in
> > > > > blindly
> > > > > > > > adhering to Semantic Versioning just for the sake
> > of
> > > > > adhering to
> > > > > > > > Semantic Versioning. Instead, use it as a
> > guideline,
> > > and
> > > > > break the
> > > > > > > > rules where it makes the most sense to do so.
> > > > > > > >
> > > > > > > > - Rawlin
> > > > > > > >
> > > > > > > > [1]
> > > > > > > >
> > > > >
> > >
> > https://urldefense.com/v3/__https://lists.apache.org/thread.html/15c6bfcdce0ac00a3a8612a96c3d668a8a5a16c6ca9401de9473d31c@*3Cdev.trafficcontrol.apache.org*3E__;JSU!rx_L75ITgOQ!V_b3E9iEYqNlZus18f-sVuofwuyxapCthg3LdO4NxyMCU39b7mdf4klP5KNR7XZ7d3Uc$
> > > > > > > >
> > > > > > > > On Wed, Nov 13, 2019 at 9:43 AM Robert O Butts <
> > > > > rob@apache.org>
> > > > > > > > wrote:
> > > > > > > > > > But if it's really valuable to remove these
> > > > > endpoints, maybe it's
> > > > > > > > > appropriate to bump the major version and ask
> > > > > everybody to update
> > > > > > > > > their
> > > > > > > > > clients?
> > > > > > > > >
> > > > > > > > > +1
> > > > > > > > >
> > > > > > > > > This sounds like the best compromise to me. It
> > > > > addresses both sides
> > > > > > > > > - it
> > > > > > > > > lets us remove endpoints, and it also prevents
> > > > > breaking users.
> > > > > > > > >
> > > > > > > > > It is a compromise - it's an inconvenience to
> > > users,
> > > > > forcing them
> > > > > > > > > to fix
> > > > > > > > > scripts to upgrade. Which isn't good for user
> > > > > experience either;
> > > > > > > > > but it
> > > > > > > > > doesn't outright break people. It's a compromise
> > > I'm
> > > > > willing to
> > > > > > > > > live with.
> > > > > > > > >
> > > > > > > > > This actually lets us do at least 3 things:
> > > > > > > > > 1. Remove "deprecated" endpoints.
> > > > > > > > > 2. Remove the unmaintained dangerously-large
> > Cache
> > > > > Config
> > > > > > > > > endpoints.
> > > > > > > > > 3. Lets us do Layered Profiles, changing Servers
> > > and
> > > > > DSes to have
> > > > > > > > > arrays of
> > > > > > > > > Profiles instead of a single Profile, without
> > hacky
> > > > > backward-
> > > > > > > > > compatibility
> > > > > > > > > attempts.
> > > > > > > > >
> > > > > > > > > To clarify, `/api/2.0` would be essentially the
> > > > > current API, just
> > > > > > > > > with the
> > > > > > > > > above breaking changes (possibly others) which
> > > require
> > > > > a SemVer
> > > > > > > > > major
> > > > > > > > > version increase. And the big API
> > rewrite/redesign
> > > > > we've been
> > > > > > > > > calling "2.0"
> > > > > > > > > would become `/api/3.0` or some future version.
> > > > > > > > >
> > > > > > > > > It's not perfect, but it seems to address the
> > most
> > > > > concerns on all
> > > > > > > > > sides.
> > > > > > > > > Thoughts? Objections?
> > > > > > > > >
> > > > > > > > >
> > > > > > > > > On Wed, Nov 13, 2019 at 8:45 AM Chris Lemmons <
> > > > > alficles@gmail.com>
> > > > > > > > > wrote:
> > > > > > > > >
> > > > > > > > > > For the endpoints, I agree in part and disagree
> > > in
> > > > > part.
> > > > > > > > > >
> > > > > > > > > > Agree for these two:
> > > > > > > > > > - the endpoints are dangerous and pose a risk
> > to
> > > the
> > > > > overall
> > > > > > > > > > Traffic
> > > > > > > > > > Control system
> > > > > > > > > > - the task of rewriting the endpoints from Perl
> > > to
> > > > > Go represents
> > > > > > > > > > an
> > > > > > > > > > unreasonable amount of work for the project
> > > > > > > > > >
> > > > > > > > > > And I would add:
> > > > > > > > > > - the endpoint does not serve its intended
> > > purpose
> > > > > and cannot
> > > > > > > > > > reasonably
> > > > > > > > > > be used
> > > > > > > > > > - the endpoint has been made unsafe or unusable
> > > by
> > > > > new features
> > > > > > > > > > added to TC
> > > > > > > > > >
> > > > > > > > > > Disagree for these two:
> > > > > > > > > > - the endpoints have been obsoleted by other
> > > > > endpoints
> > > > > > > > > > - the endpoints don't seem to serve a true
> > > purpose
> > > > > or don't seem
> > > > > > > > > > to be
> > > > > > > > > > valuable to the project anymore
> > > > > > > > > >
> > > > > > > > > > For the most part, obsoleted endpoints can be
> > > > > maintained with a
> > > > > > > > > > shim
> > > > > > > > > > into the new code. If they would require "an
> > > > > unreasonable amount
> > > > > > > > > > of
> > > > > > > > > > work", it might make sense to remove them,
> > then.
> > > > > > > > > >
> > > > > > > > > > Things that don't seem valuable usually fall
> > into
> > > > > another
> > > > > > > > > > category
> > > > > > > > > > like "cannot reasonably be used", but if the
> > only
> > > > > reason for
> > > > > > > > > > deprecation is that the feature is going away,
> > we
> > > > > should really
> > > > > > > > > > support it according to our promise.
> > > > > > > > > >
> > > > > > > > > > But if it's really valuable to remove these
> > > > > endpoints, maybe it's
> > > > > > > > > > appropriate to bump the major version and ask
> > > > > everybody to update
> > > > > > > > > > their clients?
> > > > > > > > > >
> > > > > > > > > > On Tue, Nov 12, 2019 at 5:27 PM Rawlin Peters <
> > > > > rawlin@apache.org>
> > > > > > > > > > wrote:
> > > > > > > > > > > The real crux of the issue here is whether or
> > > not
> > > > > we agree as a
> > > > > > > > > > > project that we can move forward with this
> > plan
> > > > > that I believe
> > > > > > > > > > > to be a
> > > > > > > > > > > good compromise that meets most concerns but
> > > > > "breaks the
> > > > > > > > > > > Semantic
> > > > > > > > > > > Versioning promise":
> > > > > > > > > > > 1) don't rewrite deprecated endpoints from
> > > Perl to
> > > > > Go
> > > > > > > > > > > 2) in the Perl handler, add a deprecation
> > > notice
> > > > > to the
> > > > > > > > > > > "alerts"
> > > > > > > > > > > section of the response
> > > > > > > > > > > 3) once a deprecated API has been deprecated
> > > for
> > > > > one major
> > > > > > > > > > > release,
> > > > > > > > > > > delete it from the API in the subsequent
> > major
> > > > > release (e.g. if
> > > > > > > > > > > deprecation notices are added and released in
> > > ATC
> > > > > 4.x, we can
> > > > > > > > > > > remove
> > > > > > > > > > > the deprecated endpoint in ATC 5.x)
> > > > > > > > > > >
> > > > > > > > > > > Rob, from your latest proposal to remove the
> > > ATS
> > > > > config API
> > > > > > > > > > > endpoints
> > > > > > > > > > > from 1.x, I thought we were past this issue
> > > > > already and could
> > > > > > > > > > > finally
> > > > > > > > > > > agree to remove deprecated endpoints from 1.x
> > > > > (following steps
> > > > > > > > > > > 1-3
> > > > > > > > > > > above) without requiring an all-new 2.x API.
> > > Are
> > > > > we not
> > > > > > > > > > > actually past
> > > > > > > > > > > that? You came up with valid reasons for
> > > applying
> > > > > these steps
> > > > > > > > > > > to the
> > > > > > > > > > > ATS config API endpoints, and I think it's
> > > fair to
> > > > > say that we
> > > > > > > > > > > can
> > > > > > > > > > > come up with valid reasons for applying these
> > > > > steps to other
> > > > > > > > > > > deprecated API endpoints as well.
> > > > > > > > > > >
> > > > > > > > > > > It seems like the real problem is actually
> > > > > agreeing on which
> > > > > > > > > > > _specific_ endpoints we should be able to
> > > follow
> > > > > steps 1-3 for.
> > > > > > > > > > > IMO I
> > > > > > > > > > > think it should apply to any 1.x endpoints
> > > that we
> > > > > have valid
> > > > > > > > > > > reasons
> > > > > > > > > > > for not carrying forward to Go, including but
> > > not
> > > > > limited to:
> > > > > > > > > > > - the endpoints are dangerous and pose a risk
> > > to
> > > > > the overall
> > > > > > > > > > > Traffic
> > > > > > > > > > > Control system
> > > > > > > > > > > - the endpoints have been obsoleted by other
> > > > > endpoints
> > > > > > > > > > > - the endpoints don't seem to serve a true
> > > purpose
> > > > > or don't
> > > > > > > > > > > seem to be
> > > > > > > > > > > valuable to the project anymore
> > > > > > > > > > > - the task of rewriting the endpoints from
> > > Perl to
> > > > > Go
> > > > > > > > > > > represents an
> > > > > > > > > > > unreasonable amount of work for the project
> > > > > > > > > > >
> > > > > > > > > > > Obviously most of these reasons are
> > > subjective, so
> > > > > we'd have to
> > > > > > > > > > > come
> > > > > > > > > > > to some level of consensus on deprecating
> > > > > particular endpoints
> > > > > > > > > > > first,
> > > > > > > > > > > but we should be able to follow steps 1-3 for
> > > > > whatever
> > > > > > > > > > > endpoints we
> > > > > > > > > > > decide.
> > > > > > > > > > >
> > > > > > > > > > > Making breaking changes shouldn't be taken
> > > > > lightly, but we need
> > > > > > > > > > > a path
> > > > > > > > > > > forward that allows some calculated, breaking
> > > > > changes where
> > > > > > > > > > > necessary.
> > > > > > > > > > > Otherwise, we are unnecessarily holding the
> > > > > project back.
> > > > > > > > > > >
> > > > > > > > > > > - Rawlin
> > > > > > > > > > >
> > > > > > > > > > > On Tue, Nov 12, 2019 at 11:44 AM Jeremy
> > > Mitchell <
> > > > > > > > > > > mitchell852@gmail.com>
> > > > > > > > > > wrote:
> > > > > > > > > > > > Yeah, technically a "rewrite" of 1.x means
> > > > > shifting the
> > > > > > > > > > > > implementation
> > > > > > > > > > from
> > > > > > > > > > > > Perl to Go for ALL of 1.x. If you leave a
> > > subset
> > > > > of 1.x in
> > > > > > > > > > > > Perl, then
> > > > > > > > > > you
> > > > > > > > > > > > have to call it a "partial rewrite".
> > > > > > > > > > > >
> > > > > > > > > > > > So, assuming we want to do a full rewrite
> > (no
> > > > > more perl for
> > > > > > > > > > > > the api
> > > > > > > > > > > > whatsoever), I think that means every 1.x
> > api
> > > > > endpoint (good
> > > > > > > > > > > > or bad)
> > > > > > > > > > needs
> > > > > > > > > > > > to be rewritten to Go so we can remove the
> > > > > dependency on
> > > > > > > > > > > > perl.
> > > > > > > > > > > >
> > > > > > > > > > > > However, I still like marking many as
> > > > > "deprecated" so we
> > > > > > > > > > > > train users
> > > > > > > > > > to use
> > > > > > > > > > > > "better" endpoints that we envision will
> > > exist
> > > > > in 2.x and
> > > > > > > > > > > > beyond.
> > > > > > > > > > > >
> > > > > > > > > > > > Jeremy
> > > > > > > > > > > >
> > > > > > > > > > > > On Tue, Nov 12, 2019 at 11:22 AM Hoppal,
> > > Michael
> > > > > <
> > > > > > > > > > Michael_Hoppal@comcast.com>
> > > > > > > > > > > > wrote:
> > > > > > > > > > > >
> > > > > > > > > > > > > We do not have a lot of deprecated routes
> > > as
> > > > > of now as we
> > > > > > > > > > > > > focused on
> > > > > > > > > > > > > routes that are more used first.
> > > > > > > > > > > > >
> > > > > > > > > > > > > I know on a lot of the remaining rewrites
> > > > > there has been
> > > > > > > > > > > > > discussion
> > > > > > > > > > on the
> > > > > > > > > > > > > matching Github issues on potential
> > > > > deprecation (I cannot
> > > > > > > > > > > > > think of
> > > > > > > > > > the
> > > > > > > > > > > > > exact count but enough to warrant this
> > > email
> > > > > __)
> > > > > > > > > > > > >
> > > > > > > > > > > > > It is key to note that I am not arguing
> > for
> > > > > removal of
> > > > > > > > > > > > > endpoints
> > > > > > > > > > within
> > > > > > > > > > > > > the current API version but if there is
> > an
> > > > > endpoint we
> > > > > > > > > > > > > agree should
> > > > > > > > > > be
> > > > > > > > > > > > > deprecated going forward to put a message
> > > > > within the
> > > > > > > > > > > > > response and on
> > > > > > > > > > the
> > > > > > > > > > > > > docs.
> > > > > > > > > > > > >
> > > > > > > > > > > > > And thanks for the input sounds like you
> > > are
> > > > > +1 on
> > > > > > > > > > > > > rewriting all
> > > > > > > > > > routes to
> > > > > > > > > > > > > Golang as Brennan is.
> > > > > > > > > > > > >
> > > > > > > > > > > > > On 11/12/19, 9:06 AM, "Robert O Butts" <
> > > > > rob@apache.org>
> > > > > > > > > > > > > wrote:
> > > > > > > > > > > > >
> > > > > > > > > > > > > Whether we rewrite a route in Go is
> > an
> > > > > implementation
> > > > > > > > > > > > > detail. To
> > > > > > > > > > the
> > > > > > > > > > > > > interface, to users, it doesn't
> > matter
> > > > > whether a route
> > > > > > > > > > > > > is
> > > > > > > > > > rewritten or
> > > > > > > > > > > > > not.
> > > > > > > > > > > > >
> > > > > > > > > > > > > But our API follows Semantic
> > > Versioning,
> > > > > in order to
> > > > > > > > > > > > > not break
> > > > > > > > > > users.
> > > > > > > > > > > > > We
> > > > > > > > > > > > > can't just remove endpoints that some
> > > of
> > > > > us don't use,
> > > > > > > > > > > > > and
> > > > > > > > > > assume other
> > > > > > > > > > > > > people don't, maybe people who never
> > > speak
> > > > > up on the
> > > > > > > > > > > > > mailing
> > > > > > > > > > list.
> > > > > > > > > > > > > We'll
> > > > > > > > > > > > > never gain a big userbase if we keep
> > > > > breaking users.
> > > > > > > > > > > > >
> > > > > > > > > > > > > Per the _project_ SemVer, once we
> > have
> > > API
> > > > > 2.0, we can
> > > > > > > > > > > > > deprecate
> > > > > > > > > > API
> > > > > > > > > > > > > 1.x,
> > > > > > > > > > > > > and in the next major _project_
> > > release,
> > > > > remove API
> > > > > > > > > > > > > 1.x.
> > > > > > > > > > Irrespective
> > > > > > > > > > > > > of
> > > > > > > > > > > > > Perl or Go.
> > > > > > > > > > > > >
> > > > > > > > > > > > > My big concern is, API 2.0 is a big
> > > > > project. How long
> > > > > > > > > > > > > has the
> > > > > > > > > > rewrite
> > > > > > > > > > > > > to Go
> > > > > > > > > > > > > taken? Do we really believe designing
> > > and
> > > > > implementing
> > > > > > > > > > > > > a
> > > > > > > > > > completely
> > > > > > > > > > > > > new API
> > > > > > > > > > > > > will be any less time?
> > > > > > > > > > > > >
> > > > > > > > > > > > > I don't want killing Perl to have to
> > > wait
> > > > > on that.
> > > > > > > > > > > > >
> > > > > > > > > > > > > I know it feels like a waste to
> > rewrite
> > > > > routes that you
> > > > > > > > > > > > > don't
> > > > > > > > > > use, and
> > > > > > > > > > > > > probably few people do. But that's
> > the
> > > > > cost of a stable
> > > > > > > > > > > > > project.
> > > > > > > > > > How
> > > > > > > > > > > > > many
> > > > > > > > > > > > > "deprecated" routes are there? If it
> > > comes
> > > > > down to
> > > > > > > > > > > > > taking the
> > > > > > > > > > > > > development
> > > > > > > > > > > > > time to rewrite them so we can kill
> > > Perl
> > > > > faster, or
> > > > > > > > > > > > > leaving Perl
> > > > > > > > > > > > > around, I
> > > > > > > > > > > > > vote we just do the work and kill
> > Perl.
> > > > > > > > > > > > >
> > > > > > > > > > > > > >If we don't rewrite them, then Perl
> > > will
> > > > > last until
> > > > > > > > > > > > > API 2.0 has
> > > > > > > > > > been
> > > > > > > > > > > > > designed, released and then *another
> > > full
> > > > > major release
> > > > > > > > > > > > > cycle*.
> > > > > > > > > > That's
> > > > > > > > > > > > > way
> > > > > > > > > > > > > too long to have two codebases for
> > the
> > > > > same component,
> > > > > > > > > > > > > IMO,
> > > > > > > > > > especially
> > > > > > > > > > > > > since the rewrite is already 50%
> > > complete.
> > > > > > > > > > > > >
> > > > > > > > > > > > > +1
> > > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > > > > > On Tue, Nov 12, 2019 at 8:54 AM ocket
> > > 8888
> > > > > <
> > > > > > > > > > > > > ocket8888@gmail.com>
> > > > > > > > > > > > > wrote:
> > > > > > > > > > > > >
> > > > > > > > > > > > > > I vote that by and large we DO
> > > rewrite
> > > > > them, with
> > > > > > > > > > > > > exceptions
> > > > > > > > > > for
> > > > > > > > > > > > > routes
> > > > > > > > > > > > > > that just plain don't work, even in
> > > > > Perl. Those are
> > > > > > > > > > > > > few,
> > > > > > > > > > though.
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > If we don't rewrite them, then Perl
> > > will
> > > > > last until
> > > > > > > > > > > > > API 2.0
> > > > > > > > > > has been
> > > > > > > > > > > > > > designed, released and then
> > *another
> > > > > full major
> > > > > > > > > > > > > release cycle*.
> > > > > > > > > > > > > That's way
> > > > > > > > > > > > > > too long to have two codebases for
> > > the
> > > > > same
> > > > > > > > > > > > > component, IMO,
> > > > > > > > > > > > > especially
> > > > > > > > > > > > > > since the rewrite is already 50%
> > > > > complete.
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > On Tue, Nov 12, 2019 at 8:43 AM
> > > Hoppal,
> > > > > Michael <
> > > > > > > > > > > > > > Michael_Hoppal@comcast.com>
> > > > > > > > > > > > > > wrote:
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > > As the Traffic Ops API is being
> > > > > rewritten from Perl
> > > > > > > > > > > > > to Golang
> > > > > > > > > > > > > there has
> > > > > > > > > > > > > > > been several routes that have
> > been
> > > > > deprecated and
> > > > > > > > > > > > > probably
> > > > > > > > > > more to
> > > > > > > > > > > > > come.
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > In the deprecation efforts I have
> > > seen
> > > > > two
> > > > > > > > > > > > > strategies:
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > * The route IS NOT rewritten
> > > from
> > > > > Perl to
> > > > > > > > > > > > > Golang and a
> > > > > > > > > > > > > deprecation
> > > > > > > > > > > > > > > notice is added to the alert
> > > response
> > > > > in the Perl
> > > > > > > > > > > > > handler
> > > > > > > > > > > > > > > * The route IS rewritten from
> > > Perl
> > > > > to Golang
> > > > > > > > > > > > > and a
> > > > > > > > > > deprecation
> > > > > > > > > > > > > notice
> > > > > > > > > > > > > > > is added to the alert response in
> > > the
> > > > > Golang
> > > > > > > > > > > > > handler
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > I think we should have
> > consistency
> > > in
> > > > > our approach
> > > > > > > > > > > > > and
> > > > > > > > > > wanted to
> > > > > > > > > > > > > get
> > > > > > > > > > > > > > > people’s thoughts.
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > I would vote that we do not
> > > rewrite a
> > > > > deprecated
> > > > > > > > > > > > > route from
> > > > > > > > > > Perl to
> > > > > > > > > > > > > > Golang.
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > Thanks,
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > Michael
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > >
> > > > > > >
> > > > > >
> > > > > >
> > > > >
> > > > >
> > > > >
> > >
> >
Re: [EXTERNAL] Re: Traffic Ops Route Deprecation Strategy
Posted by "Gray, Jonathan" <Jo...@comcast.com>.
+1
I think the only point of contention was if the API v2.0 would be in ATC v4.0 or not. Either way is fine with me, and more time to know this is coming makes sense.
Jonathan G
On 12/2/19, 11:32 AM, "Rawlin Peters" <ra...@apache.org> wrote:
Hey all,
A bunch of us got together and discussed this issue offline, and we
got consensus to go with Option 1 - Graduate current API from 1.x to
2.x - allowing us to leave behind routes in 1.x that are
dangerous/deprecated. Deprecated endpoints will not be rewritten from
Perl to Go.
So, we will graduate the latest minor version of existing 1.x Go
endpoints to 2.0. For example, today we have
/api/1.2/deliveryservices, /api/1.3/deliveryservices, and
/api/1.4/deliveryservices. We will take the existing handler for
/api/1.4/deliveryservices and promote it to /api/2.0/deliveryservices.
We will follow this example for other 1.x Go endpoints with multiple
minor versions as well.
API endpoints that are deemed broken, dangerous, or deprecated will
not be promoted to API 2.0. This will likely include the ~7 endpoints
that I've mentioned in this thread already, but we haven't yet
compiled the full list of endpoints that won't be promoted to 2.0. We
will come up with that list (along with its criteria) soon and bring
that list to the mailing list for everyone to look over and agree
upon. Deprecated endpoints will be updated to return deprecation
alerts where applicable.
This API graduation will likely miss the boat for 4.0 but will
hopefully land in 4.1 or 4.2 depending on how long it takes. So this
would be the tentative timetable:
TC 4.0:
- API 1.x supported, some deprecation notices
TC 4.1:
- API 1.x still supported, deprecation notices added to endpoints not
graduated to 2.0
- API 2.0 supported, consisting of 1.x endpoints that were graduated
- starting with this release, you need to start migrating external
clients off of 1.x over to 2.0
TC 4.2:
- internal clients (e.g. TM, TR, etc) will be migrated off API 1.x
over to 2.0. Doing this step after 4.1 adds confidence that 1.x is
still supported alongside 2.0 in order to provide a smooth migration
period for API clients.
TC 5.0:
- API 1.x no longer supported, only API 2.x is supported
Those who took part in that offline discussion, please keep me honest
here with a +1 if we did in fact come to a rough consensus on this
option. If you weren't part of that offline discussion, feel free to
voice your vote on this issue as well (I believe a bunch of people
were already +1 on this option a few weeks ago).
- Rawlin
On Tue, Nov 19, 2019 at 12:23 PM Robert O Butts <ro...@apache.org> wrote:
>
> > - /riak/stats
>
> Looking at the Perl, this looks easy to rewrite. It's just proxying a
> request to Riak.
>
> +1 on rewriting this, for API stability, even if we don't "value" the
> endpoint as a project.
>
> >- /cachegroup_fallbacks
>
> Likewise +1 on rewriting. Not as trivial as /riak/stats, but it also looks
> easy to me, looks like a standard CRUD endpoint. +1 on leaving a
> "deprecated, please prefer X" message in, and +1 in removing in the big API
> rewrite/refactor/2.0.
>
> > - /capabilities (all methods except for GET) -- capabilities are
> built-in, should not be changed at runtime
> > - /api_capabilities (all methods except for GET) -- same reasons as
> /capabilities
>
> I actually don't agree with this - because they're useful to plugins. IMO
> it should be possible to add and modify Capabilities and API Capabilities,
> because they can be used by plugins which add new endpoints, to control
> access to them. If there's disagreement there, I'd vote we take that to a
> separate thread.
>
> But in terms of the endpoints themselves, the Perl also looks easy to
> rewrite, looks like standard CRUD. -1 on deprecating or removing, +1 on
> rewriting.
>
> > - /cdns/{name}/configs/routing -- this was a half-baked attempt at
> redoing CrConfig.json -- cannot possibly be of use to anyone in its current
> state
>
> Ugh. I agree with the sentiment; and this one looks slightly less trivial.
> But I personally don't see it as an unreasonable amount of work. For the
> sake of API stability, I'm personally willing to volunteer to rewrite this
> one, including API tests to make fixing easy if the schema changes (which
> it appears to have in master). And hopefully it can just exist until the
> fabled API Rewrite. Is that acceptable?
>
> > - the ATS config generation endpoints -- Rob actually proposed to remove
> these from 1.x due to danger of being out of sync w/ cache-side config
> generator
>
> I'm personally on the fence about this one. Like I said in that thread, I'm
> the last one to want to break users, and I don't take this lightly. If,
> after I made the case for it, I'm the only one, I might be inclined to
> retract my proposal, and just continue to maintain them. Maybe some more
> API tests would help with that. I think I'm the primary dev on the ATS
> config gen at this point anyway.
>
> > - ATS Config Files Stuff Aren't these all already in Go anyway? Whether
> or not they should be removed from the API aside, I don't think anything
> actually needs to hit Perl anymore.
>
> Correct.
>
> > - /hwinfo/dtdata
>
> I think the 'render' call here is just rendering the JSON. But, the
> endpoint itself is just hanging and not returning for me, I'm not sure why.
> I'd like to volunteer to look into this endpoint as well. I think you might
> be right, this one might be unreasonable to rewrite. But if it is
> reasonable, I'd like to just do it. I'd like to at least look into it.
>
>
> On Tue, Nov 19, 2019 at 12:05 PM ocket 8888 <oc...@gmail.com> wrote:
>
> > I'm generally +1 for option 2
> >
> > regarding those specific endpoints:
> >
> > - /riak/stats if this is easy and doesn't expose anything dangerous, we
> > should do it IMO.
> > - /cachegroup_fallbacks should be pretty trivial, so I still say do it
> > - /capabilities since these aren't actually used for anything, I think it's
> > pretty safe (and easy) to rewrite. Plus there's already a PR open for
> > exactly that
> > - /api_capabilities dropping this one could be problematic, just because
> > people may depend on it to tell them what endpoints exist. To be clear, I
> > think it ought to be deprecated, but unfortunately I just don't think it's
> > terribly unlikely that this is built into scripts and tools. The actual
> > code for the rewrite should be easy too, although maintaining it is a pain.
> >
> > - ATS Config Files Stuff Aren't these all already in Go anyway? Whether or
> > not they should be removed from the API aside, I don't think anything
> > actually needs to hit Perl anymore.
> >
> > - /cdn/{{name}}/configs/routing CRConfigs are large and complex, so if this
> > never worked anyway maybe we should just add a Go handler for it that
> > returns a '410 Gone' response (or '501 Not Implemented' is the other one
> > I've heard suggested) and leave the actual logic behind in the Perl
> > - /hwinfo/dtdata if this calls into Mojolicious rendering, returns a
> > non-json response, and/or contains information that in either content or
> > encoding is meaningless to any sane client then it should also return a
> > 410/501 and get left behind in Perl.
> >
> > On Tue, Nov 19, 2019 at 11:08 AM Rawlin Peters <ra...@apache.org> wrote:
> >
> > > While I appreciate and mostly agree with your essay on the importance
> > > of not breaking users (or rather, breaking them in predictable ways),
> > > I for one would really like to meet at some middle ground with option
> > > 2. I'm not asking to remove 50 high-usage endpoints from 1.x. IIRC
> > > I've only specifically proposed "leaving behind" a few specific Perl
> > > endpoints that I believe serve no purpose, have no valid reason to be
> > > used, and/or are dangerous to the system:
> > >
> > > - /riak/stats -- informational, was used for debugging the initial
> > > Riak integration. This one might be pretty easy to rewrite since it
> > > just calls a corresponding riak API and returns the result via TO, so
> > > I could be on board with rewriting it if necessary.
> > > - /cachegroup_fallbacks -- these endpoints were released in 3.0 before
> > > we managed to augment the existing /cachegroups API to handle
> > > fallbacks, so I could get on board with rewriting these if we really
> > > think it's important.
> > > - /capabilities (all methods except for GET) -- capabilities are
> > > built-in, should not be changed at runtime
> > > - /api_capabilities (all methods except for GET) -- same reasons as
> > > /capabilities
> > > - /cdns/{name}/configs/routing -- this was a half-baked attempt at
> > > redoing CrConfig.json -- cannot possibly be of use to anyone in its
> > > current state
> > > - the ATS config generation endpoints -- Rob actually proposed to
> > > remove these from 1.x due to danger of being out of sync w/ cache-side
> > > config generator
> > > - /hwinfo/dtdata -- AFAIK the "dtdata" endpoints were only
> > > used/required by the old Perl Mojolicious UI, no valid reason to be
> > > using this as an external API client. In fact, I'm not even sure _how_
> > > this endpoint can be used outside of the Mojolicious framework/UI. It
> > > calls the Mojolicious `render` function, which usually means rendering
> > > an embedded Perl HTML template given some data. This is one I would
> > > consider completely unreasonable to rewrite to Go.
> > >
> > > Can we get on board with leaving at least these 1.x endpoints behind
> > > in the Perl? Doing a cursory skim over the remaining 1.x endpoints to
> > > rewrite, it _seems_ like everything else would fall into the "simpler
> > > to just rewrite rather than gain consensus to remove" category.
> > >
> > > - Rawlin
> > >
> > > On Tue, Nov 19, 2019 at 9:19 AM Robert O Butts <ro...@apache.org> wrote:
> > > >
> > > > >Option 1 - Graduate current API from 1.x to 2.x - allowing us to leave
> > > > behind routes in 1.x that are dangerous or deprecated
> > > > >Option 2 - Rewrite all endpoints from Perl to Golang in current
> > version
> > > as
> > > > we are doing now.
> > > >
> > > > +1 on either Option 1 or Option 2, without a strong preference. Option
> > 1
> > > > has advantages of letting us also safely remove the Cache Config
> > > endpoints
> > > > and implement Layered Profiles. But Option 2 also has the advantage of
> > > not
> > > > requiring users to upgrade.
> > > >
> > > > > Option 3 - Only rewrite endpoints from Perl to Golang in current
> > > version
> > > > that we agree are not dangerous or should deprecate
> > > >
> > > > -1. As before, it breaks users unpredictably.
> > > >
> > > > >It will literally outright break clients (*all* of them, not just
> > > *some*)
> > > >
> > > > This is technically correct: technically, new versions 'break' all
> > users.
> > > > The difference is, it does it in a predictable, safe way that users can
> > > > handle, inspect, and safely upgrade.
> > > >
> > > > >Can someone please explain to me why (A) artificially graduating the
> > > > current 1.x API to 2.x, and removing the deprecated endpoints from 1.x,
> > > is
> > > > better for API clients overall than (B) just removing deprecated
> > > endpoints
> > > > from 1.x? Existing 1.x clients would gain absolutely nothing from being
> > > > forced to update to 2.x like this.
> > > >
> > > > >for zero gain in value
> > > >
> > > > Users would gain something. There is a very specific gain.
> > > >
> > > > The gain is that the breakage is predictable. People frequently upgrade
> > > > minor versions automatically, either running commands without thinking,
> > > or
> > > > even automating the entire process.
> > > >
> > > > For TC, this applies to both CDN operators, and their end-users.
> > > >
> > > > When the minor versions contain breaking changes, that automatic
> > upgrade
> > > > then breaks. In this case, any scripts or tools they have using the
> > > removed
> > > > endpoints will break. If those scripts are important, this could bring
> > > down
> > > > their CDN, or break their Delivery Service, or otherwise cause an
> > outage.
> > > >
> > > > Contrawise, users expect breaking changes in Major versions, and don't
> > > > typically automatically upgrade them. Rather, they'll manually inspect
> > > the
> > > > update, do testing, and check the changelog for breaking changes.
> > > >
> > > > It's not a technical thing, it's a human thing. Humans automatically
> > > > upgrade minor versions, and write scripts to do so, and dependency
> > files
> > > to
> > > > allow it, because the software industry taught people that minor
> > versions
> > > > shouldn't break anything.
> > > >
> > > > If we start breaking minor versions, initially users will experience
> > > > breakage. Not all users, and not all catastrophic. But some users will
> > > > experience catastrophic outages. And it will be our fault as a project.
> > > And
> > > > after enough times, they'll fork TC, or find another product, or write
> > > > their own; and stop participating in the community, or never start.
> > > >
> > > > Then, once they've learned that we can't be trusted not to break
> > things,
> > > > they'll experience the 2nd problem: where nothing can be safely
> > upgraded,
> > > > and the smallest change requires long and expensive testing to verify a
> > > > single minor version or even patch upgrade, and still risk missing
> > > > something and bringing their CDN down. To be honest, we're pretty much
> > > > already here. It already costs our dev and ops teams unbelievable
> > amounts
> > > > of time and person-hours of testing for every minor upgrade. That time
> > > far
> > > > exceeds the development time to just rewrite the small endpoints we
> > don't
> > > > like anymore (and tangentially, also the time to write the Integration
> > > and
> > > > E2E tests TC desperately needs).
> > > >
> > > > ATC has so much potential. There aren't many OSS CDNs out there, and
> > > > there's certainly the demand. We have the potential to become a huge
> > > > project, which for us existing users, translates into potentially an
> > > > enormous amount of free development. But we'll never get there if we
> > keep
> > > > breaking things.
> > > >
> > > > And for my particular organization (Comcast), we have a hard enough
> > time
> > > > upgrading. Our upgrades, especially Traffic Ops, are extremely painful,
> > > and
> > > > require enormous amounts of manual testing. Even then, we frequently
> > have
> > > > issues that require rollback, sometimes even an outage; even after
> > those
> > > > unbelievably amounts of developer and operator time testing. We
> > > desperately
> > > > need a system that is safer and easier to upgrade. Breaking changes are
> > > > always going to require more testing; but we can at least make minor
> > > > versions safe to upgrade.
> > > >
> > > > > I know I am changing my initial stance but I am +1 on rewriting all
> > the
> > > > Perl endpoints to Golang and not breaking the API contract we have in
> > > > place. Most of the API routes to rewrite are rather simple to port and
> > I
> > > > think would be faster than getting consensus on actually deprecating it
> > > > from the mailing list.
> > > >
> > > > I'm inclined to agree with this. The things I've seen marked
> > "deprecated"
> > > > seem to mostly be small, and look like they'd be pretty easy to
> > rewrite.
> > > > But I haven't seen the whole list. Is there a list somewhere, so the
> > > > community can see it to better decide?
> > > >
> > > >
> > > > On Tue, Nov 19, 2019 at 7:24 AM Hoppal, Michael <
> > > Michael_Hoppal@comcast.com>
> > > > wrote:
> > > >
> > > > > I would love to drive this to some type of consensus on rewrite
> > > strategy
> > > > > as we have a good amount of resources (and growing) working on the
> > > rewrite
> > > > > and I want to keep that momentum.
> > > > >
> > > > > Sounds like there is three options from the email thread (please add
> > > any I
> > > > > missed)
> > > > >
> > > > > Option 1 - Graduate current API from 1.x to 2.x - allowing us to
> > leave
> > > > > behind routes in 1.x that are dangerous or deprecated
> > > > > Option 2 - Rewrite all endpoints from Perl to Golang in current
> > > version as
> > > > > we are doing now.
> > > > > Option 3 - Only rewrite endpoints from Perl to Golang in current
> > > version
> > > > > that we agree are not dangerous or should deprecate
> > > > >
> > > > > I am +1 for option 2 (which I know is different from my original
> > > stance).
> > > > >
> > > > > Which direction would others like to go?
> > > > >
> > > > > Thanks,
> > > > >
> > > > > Michael
> > > > >
> > > > >
> > > > > I know I am changing my initial stance but I am +1 on rewriting all
> > the
> > > > > Perl endpoints to Golang and not breaking the API contract we have in
> > > > > place. Most of the API routes to rewrite are rather simple to port
> > and
> > > I
> > > > > think would be faster than getting consensus on actually deprecating
> > it
> > > > > from the mailing list.
> > > > >
> > > > > On 11/16/19, 3:31 PM, "Rawlin Peters" <ra...@apache.org> wrote:
> > > > >
> > > > > > Moving to 2.x means that it's my fault as a consumer of the API
> > > if I
> > > > > was still using the deprecated endpoint (and that change should be
> > > part of
> > > > > a larger upgrade plan). Simply removing the deprecated routes,
> > > > > retroactively, means I can't rely on having a transition period to
> > get
> > > to
> > > > > 2.0 outside of a large completely down upgrade window.
> > > > >
> > > > > There would be a transition period either way. Endpoints would be
> > > > > deprecated in ATC 4.0 and removed in 5.0. The release notes for
> > 4.0
> > > > > would clearly list out the deprecated endpoints. If they have
> > > > > replacements, there would be instructions on how to transition to
> > > the
> > > > > replacement. Deprecated endpoints would return alerts. After
> > > upgrading
> > > > > to 4.0, operators should be well aware that certain endpoints are
> > > > > being removed in 5.0, and it's on them to start using new
> > > endpoints,
> > > > > change workflows, etc.
> > > > >
> > > > > By forcing operators to upgrade all of their clients to start
> > using
> > > > > API 2.x, it just adds to their burden of upgrading when that time
> > > > > comes. Rather than surgically fixing the clients of the 2% of
> > > > > deprecated API endpoints before upgrading to ATC 5.0, operators
> > > would
> > > > > have to fix every single client they have to use API 2.x before
> > > > > upgrading to 5.0. Is that exercise really worth it? Most people
> > > would
> > > > > probably be blissfully unaware that the "POST
> > > /api/1.1/capabilities"
> > > > > endpoint disappeared (among other deprecated endpoints). Less
> > > footguns
> > > > > in the system, and they didn't even have to update their clients.
> > > > >
> > > > > Anyways, circling back to Michael's original question, it sounds
> > > like
> > > > > either option we choose, we at least have consensus on this
> > > > > deprecation strategy:
> > > > > - The route IS NOT rewritten from Perl to Golang and a
> > deprecation
> > > > > notice is added to the alert response in the Perl handler
> > > > >
> > > > > If we're content to force all API clients to update their clients
> > > to
> > > > > 2.x for zero gain in value, at least we won't be wasting work by
> > > > > rewriting deprecated Perl endpoints to Go.
> > > > >
> > > > > - Rawlin
> > > > >
> > > > > On Fri, Nov 15, 2019 at 7:06 PM Gray, Jonathan
> > > > > <Jo...@comcast.com> wrote:
> > > > > >
> > > > > > Moving to 2.x means that it's my fault as a consumer of the API
> > > if I
> > > > > was still using the deprecated endpoint (and that change should be
> > > part of
> > > > > a larger upgrade plan). Simply removing the deprecated routes,
> > > > > retroactively, means I can't rely on having a transition period to
> > get
> > > to
> > > > > 2.0 outside of a large completely down upgrade window. Depending on
> > > how
> > > > > the API is used, even in our own components such as TM, that could
> > > prove
> > > > > disastrous. For the older minor revs like 1.1/1.2, I could live with
> > > > > dropping those as part of a major project release (including upgrade
> > > > > instructions outlining what releases you must be on beforehand).
> > > Currently
> > > > > 1.3 I view as stable enough to code to or use a native client library
> > > from,
> > > > > and 1.4 is my risk/overhead to take for new stuff as it's still
> > > evolving
> > > > > and improving. Going to 1.5 or 2.0 moves that forward. The path of
> > > least
> > > > > surprise for humans is to expect big shifts on those major revs.
> > > Removal
> > > > > falls under that umbrella and gets as a consumer not to strictly
> > focus
> > > on
> > > > > endpoint->endpoint moves, but rather workflow shifts and maybe now I
> > > have
> > > > > to use the API a different way than before. I believe getting this
> > > right
> > > > > is something that would greatly help us understand does X component
> > at
> > > > > version 1 work with Y component at version 2, because now you're
> > > asking do
> > > > > components X and Y both speak protocol A. With major releases we can
> > > work
> > > > > through that each time, but it's error prone and requires experts in
> > > all
> > > > > products to agree upon. If you do nothing but consume major
> > releases,
> > > and
> > > > > then conduct your own impact evaluation having read the upgrade
> > > > > instructions thoroughly, maybe you'd be able to smoothly transition.
> > > > > >
> > > > > > Jonathan G
> > > > > >
> > > > > >
> > > > > > On 11/15/19, 5:24 PM, "Rawlin Peters" <ra...@apache.org>
> > wrote:
> > > > > >
> > > > > > Can someone please explain to me why (A) artificially
> > > graduating
> > > > > the
> > > > > > current 1.x API to 2.x, and removing the deprecated
> > endpoints
> > > > > from
> > > > > > 1.x, is better for API clients overall than (B) just
> > removing
> > > > > > deprecated endpoints from 1.x? Existing 1.x clients would
> > > gain
> > > > > > absolutely nothing from being forced to update to 2.x like
> > > this.
> > > > > >
> > > > > > The way I see the two choices:
> > > > > > A) break every single API client out there
> > > > > > B) break only clients of APIs that we've decided upon as a
> > > > > community
> > > > > > to be deprecated
> > > > > >
> > > > > > IMO, when we have to break things, we should try break the
> > > least
> > > > > > amount of clients possible. All I'm looking for is a
> > valuable
> > > > > reason
> > > > > > to choose option A over option B, that isn't just "because
> > > > > SemVer".
> > > > > >
> > > > > > - Rawlin
> > > > > >
> > > > > > On Wed, Nov 13, 2019 at 3:24 PM Schmidt, Andrew
> > (Contractor)
> > > > > > <An...@comcast.com> wrote:
> > > > > > >
> > > > > > > If we decide to get rid of the API versioning it makes
> > > sense
> > > > > to me that we would do that on a major version upgrade, probably to
> > > the ATC
> > > > > version. But I think it would be too early to do it on the 2.0 API
> > > version.
> > > > > I like this compromise +1. I was wondering why we were stuck on the
> > > idea
> > > > > that 2.0 had to be an API rewrite. That seems like it’s a long way
> > > out. 2.0
> > > > > = deprecations and fixes. 3.0 or 5.0 = api rewrite and maybe version
> > > > > convergence.
> > > > > > >
> > > > > > > On 11/13/19, 11:52 AM, "ocket8888@gmail.com" <
> > > > > ocket8888@gmail.com> wrote:
> > > > > > >
> > > > > > > Lemme lead with: making breaking changes on a major
> > > > > version boundary
> > > > > > > gets +1 from me - no reason the API needs to be fully
> > > > > redesigned for
> > > > > > > any particular number.
> > > > > > >
> > > > > > > But I just gotta bring this up again: if we want to
> > be
> > > > > able to
> > > > > > > deprecate an endpoint for a major release and remove
> > > it in
> > > > > the next
> > > > > > > without needing to deal with the messiness of API
> > > > > versions, we could
> > > > > > > always just get rid of API versions. Major ATC
> > > revisions
> > > > > should be
> > > > > > > easier to come by with our new release strategy.
> > > > > > >
> > > > > > > On Wed, 2019-11-13 at 10:23 -0700, Rawlin Peters
> > wrote:
> > > > > > > > > It is a compromise - it's an inconvenience to
> > > users,
> > > > > forcing them
> > > > > > > > > to fix
> > > > > > > > scripts to upgrade. Which isn't good for user
> > > experience
> > > > > either; but
> > > > > > > > it
> > > > > > > > doesn't outright break people. It's a compromise
> > I'm
> > > > > willing to live
> > > > > > > > with.
> > > > > > > >
> > > > > > > > It will literally outright break clients (*all* of
> > > them,
> > > > > not just
> > > > > > > > *some*). We have been down this road before, so
> > I'll
> > > > > just repeat what
> > > > > > > > I said when this exact conversation came up months
> > > > > ago[1]:
> > > > > > > >
> > > > > > > > "So it basically boils down to:
> > > > > > > > A) promote Go 1.x routes to 2.x and eventually
> > break
> > > ALL
> > > > > users of the
> > > > > > > > 1.x API when the time comes to remove 1.x
> > > > > > > > B) deprecate and remove some 1.x Perl routes and
> > > > > POSSIBLY break SOME
> > > > > > > > users"
> > > > > > > >
> > > > > > > > Either way we slice and dice it with versioning,
> > > > > removing an endpoint
> > > > > > > > means that clients of that endpoint will break. By
> > > > > graduating only
> > > > > > > > the
> > > > > > > > "good stuff" from 1.x to 2.x, and removing
> > endpoints
> > > > > from 1.x, we
> > > > > > > > will
> > > > > > > > unnecessarily break all 1.x clients when the time
> > > comes
> > > > > to remove the
> > > > > > > > rest of 1.x. Instead, we can choose to break only
> > > > > clients of 1.x APIs
> > > > > > > > that we already deemed dangerous, unnecessary,
> > > useless,
> > > > > etc. enough
> > > > > > > > to
> > > > > > > > deprecate.
> > > > > > > >
> > > > > > > > Why break 100% of our API clients when we can
> > choose
> > > to
> > > > > break only a
> > > > > > > > small number of clients that happen to use
> > > > > > > > dangerous/unnecessary/useless APIs? Why would we
> > > break
> > > > > our API
> > > > > > > > clients
> > > > > > > > unnecessarily for no added value? There is no sense
> > > in
> > > > > blindly
> > > > > > > > adhering to Semantic Versioning just for the sake
> > of
> > > > > adhering to
> > > > > > > > Semantic Versioning. Instead, use it as a
> > guideline,
> > > and
> > > > > break the
> > > > > > > > rules where it makes the most sense to do so.
> > > > > > > >
> > > > > > > > - Rawlin
> > > > > > > >
> > > > > > > > [1]
> > > > > > > >
> > > > >
> > >
> > https://urldefense.com/v3/__https://lists.apache.org/thread.html/15c6bfcdce0ac00a3a8612a96c3d668a8a5a16c6ca9401de9473d31c@*3Cdev.trafficcontrol.apache.org*3E__;JSU!rx_L75ITgOQ!V_b3E9iEYqNlZus18f-sVuofwuyxapCthg3LdO4NxyMCU39b7mdf4klP5KNR7XZ7d3Uc$
> > > > > > > >
> > > > > > > > On Wed, Nov 13, 2019 at 9:43 AM Robert O Butts <
> > > > > rob@apache.org>
> > > > > > > > wrote:
> > > > > > > > > > But if it's really valuable to remove these
> > > > > endpoints, maybe it's
> > > > > > > > > appropriate to bump the major version and ask
> > > > > everybody to update
> > > > > > > > > their
> > > > > > > > > clients?
> > > > > > > > >
> > > > > > > > > +1
> > > > > > > > >
> > > > > > > > > This sounds like the best compromise to me. It
> > > > > addresses both sides
> > > > > > > > > - it
> > > > > > > > > lets us remove endpoints, and it also prevents
> > > > > breaking users.
> > > > > > > > >
> > > > > > > > > It is a compromise - it's an inconvenience to
> > > users,
> > > > > forcing them
> > > > > > > > > to fix
> > > > > > > > > scripts to upgrade. Which isn't good for user
> > > > > experience either;
> > > > > > > > > but it
> > > > > > > > > doesn't outright break people. It's a compromise
> > > I'm
> > > > > willing to
> > > > > > > > > live with.
> > > > > > > > >
> > > > > > > > > This actually lets us do at least 3 things:
> > > > > > > > > 1. Remove "deprecated" endpoints.
> > > > > > > > > 2. Remove the unmaintained dangerously-large
> > Cache
> > > > > Config
> > > > > > > > > endpoints.
> > > > > > > > > 3. Lets us do Layered Profiles, changing Servers
> > > and
> > > > > DSes to have
> > > > > > > > > arrays of
> > > > > > > > > Profiles instead of a single Profile, without
> > hacky
> > > > > backward-
> > > > > > > > > compatibility
> > > > > > > > > attempts.
> > > > > > > > >
> > > > > > > > > To clarify, `/api/2.0` would be essentially the
> > > > > current API, just
> > > > > > > > > with the
> > > > > > > > > above breaking changes (possibly others) which
> > > require
> > > > > a SemVer
> > > > > > > > > major
> > > > > > > > > version increase. And the big API
> > rewrite/redesign
> > > > > we've been
> > > > > > > > > calling "2.0"
> > > > > > > > > would become `/api/3.0` or some future version.
> > > > > > > > >
> > > > > > > > > It's not perfect, but it seems to address the
> > most
> > > > > concerns on all
> > > > > > > > > sides.
> > > > > > > > > Thoughts? Objections?
> > > > > > > > >
> > > > > > > > >
> > > > > > > > > On Wed, Nov 13, 2019 at 8:45 AM Chris Lemmons <
> > > > > alficles@gmail.com>
> > > > > > > > > wrote:
> > > > > > > > >
> > > > > > > > > > For the endpoints, I agree in part and disagree
> > > in
> > > > > part.
> > > > > > > > > >
> > > > > > > > > > Agree for these two:
> > > > > > > > > > - the endpoints are dangerous and pose a risk
> > to
> > > the
> > > > > overall
> > > > > > > > > > Traffic
> > > > > > > > > > Control system
> > > > > > > > > > - the task of rewriting the endpoints from Perl
> > > to
> > > > > Go represents
> > > > > > > > > > an
> > > > > > > > > > unreasonable amount of work for the project
> > > > > > > > > >
> > > > > > > > > > And I would add:
> > > > > > > > > > - the endpoint does not serve its intended
> > > purpose
> > > > > and cannot
> > > > > > > > > > reasonably
> > > > > > > > > > be used
> > > > > > > > > > - the endpoint has been made unsafe or unusable
> > > by
> > > > > new features
> > > > > > > > > > added to TC
> > > > > > > > > >
> > > > > > > > > > Disagree for these two:
> > > > > > > > > > - the endpoints have been obsoleted by other
> > > > > endpoints
> > > > > > > > > > - the endpoints don't seem to serve a true
> > > purpose
> > > > > or don't seem
> > > > > > > > > > to be
> > > > > > > > > > valuable to the project anymore
> > > > > > > > > >
> > > > > > > > > > For the most part, obsoleted endpoints can be
> > > > > maintained with a
> > > > > > > > > > shim
> > > > > > > > > > into the new code. If they would require "an
> > > > > unreasonable amount
> > > > > > > > > > of
> > > > > > > > > > work", it might make sense to remove them,
> > then.
> > > > > > > > > >
> > > > > > > > > > Things that don't seem valuable usually fall
> > into
> > > > > another
> > > > > > > > > > category
> > > > > > > > > > like "cannot reasonably be used", but if the
> > only
> > > > > reason for
> > > > > > > > > > deprecation is that the feature is going away,
> > we
> > > > > should really
> > > > > > > > > > support it according to our promise.
> > > > > > > > > >
> > > > > > > > > > But if it's really valuable to remove these
> > > > > endpoints, maybe it's
> > > > > > > > > > appropriate to bump the major version and ask
> > > > > everybody to update
> > > > > > > > > > their clients?
> > > > > > > > > >
> > > > > > > > > > On Tue, Nov 12, 2019 at 5:27 PM Rawlin Peters <
> > > > > rawlin@apache.org>
> > > > > > > > > > wrote:
> > > > > > > > > > > The real crux of the issue here is whether or
> > > not
> > > > > we agree as a
> > > > > > > > > > > project that we can move forward with this
> > plan
> > > > > that I believe
> > > > > > > > > > > to be a
> > > > > > > > > > > good compromise that meets most concerns but
> > > > > "breaks the
> > > > > > > > > > > Semantic
> > > > > > > > > > > Versioning promise":
> > > > > > > > > > > 1) don't rewrite deprecated endpoints from
> > > Perl to
> > > > > Go
> > > > > > > > > > > 2) in the Perl handler, add a deprecation
> > > notice
> > > > > to the
> > > > > > > > > > > "alerts"
> > > > > > > > > > > section of the response
> > > > > > > > > > > 3) once a deprecated API has been deprecated
> > > for
> > > > > one major
> > > > > > > > > > > release,
> > > > > > > > > > > delete it from the API in the subsequent
> > major
> > > > > release (e.g. if
> > > > > > > > > > > deprecation notices are added and released in
> > > ATC
> > > > > 4.x, we can
> > > > > > > > > > > remove
> > > > > > > > > > > the deprecated endpoint in ATC 5.x)
> > > > > > > > > > >
> > > > > > > > > > > Rob, from your latest proposal to remove the
> > > ATS
> > > > > config API
> > > > > > > > > > > endpoints
> > > > > > > > > > > from 1.x, I thought we were past this issue
> > > > > already and could
> > > > > > > > > > > finally
> > > > > > > > > > > agree to remove deprecated endpoints from 1.x
> > > > > (following steps
> > > > > > > > > > > 1-3
> > > > > > > > > > > above) without requiring an all-new 2.x API.
> > > Are
> > > > > we not
> > > > > > > > > > > actually past
> > > > > > > > > > > that? You came up with valid reasons for
> > > applying
> > > > > these steps
> > > > > > > > > > > to the
> > > > > > > > > > > ATS config API endpoints, and I think it's
> > > fair to
> > > > > say that we
> > > > > > > > > > > can
> > > > > > > > > > > come up with valid reasons for applying these
> > > > > steps to other
> > > > > > > > > > > deprecated API endpoints as well.
> > > > > > > > > > >
> > > > > > > > > > > It seems like the real problem is actually
> > > > > agreeing on which
> > > > > > > > > > > _specific_ endpoints we should be able to
> > > follow
> > > > > steps 1-3 for.
> > > > > > > > > > > IMO I
> > > > > > > > > > > think it should apply to any 1.x endpoints
> > > that we
> > > > > have valid
> > > > > > > > > > > reasons
> > > > > > > > > > > for not carrying forward to Go, including but
> > > not
> > > > > limited to:
> > > > > > > > > > > - the endpoints are dangerous and pose a risk
> > > to
> > > > > the overall
> > > > > > > > > > > Traffic
> > > > > > > > > > > Control system
> > > > > > > > > > > - the endpoints have been obsoleted by other
> > > > > endpoints
> > > > > > > > > > > - the endpoints don't seem to serve a true
> > > purpose
> > > > > or don't
> > > > > > > > > > > seem to be
> > > > > > > > > > > valuable to the project anymore
> > > > > > > > > > > - the task of rewriting the endpoints from
> > > Perl to
> > > > > Go
> > > > > > > > > > > represents an
> > > > > > > > > > > unreasonable amount of work for the project
> > > > > > > > > > >
> > > > > > > > > > > Obviously most of these reasons are
> > > subjective, so
> > > > > we'd have to
> > > > > > > > > > > come
> > > > > > > > > > > to some level of consensus on deprecating
> > > > > particular endpoints
> > > > > > > > > > > first,
> > > > > > > > > > > but we should be able to follow steps 1-3 for
> > > > > whatever
> > > > > > > > > > > endpoints we
> > > > > > > > > > > decide.
> > > > > > > > > > >
> > > > > > > > > > > Making breaking changes shouldn't be taken
> > > > > lightly, but we need
> > > > > > > > > > > a path
> > > > > > > > > > > forward that allows some calculated, breaking
> > > > > changes where
> > > > > > > > > > > necessary.
> > > > > > > > > > > Otherwise, we are unnecessarily holding the
> > > > > project back.
> > > > > > > > > > >
> > > > > > > > > > > - Rawlin
> > > > > > > > > > >
> > > > > > > > > > > On Tue, Nov 12, 2019 at 11:44 AM Jeremy
> > > Mitchell <
> > > > > > > > > > > mitchell852@gmail.com>
> > > > > > > > > > wrote:
> > > > > > > > > > > > Yeah, technically a "rewrite" of 1.x means
> > > > > shifting the
> > > > > > > > > > > > implementation
> > > > > > > > > > from
> > > > > > > > > > > > Perl to Go for ALL of 1.x. If you leave a
> > > subset
> > > > > of 1.x in
> > > > > > > > > > > > Perl, then
> > > > > > > > > > you
> > > > > > > > > > > > have to call it a "partial rewrite".
> > > > > > > > > > > >
> > > > > > > > > > > > So, assuming we want to do a full rewrite
> > (no
> > > > > more perl for
> > > > > > > > > > > > the api
> > > > > > > > > > > > whatsoever), I think that means every 1.x
> > api
> > > > > endpoint (good
> > > > > > > > > > > > or bad)
> > > > > > > > > > needs
> > > > > > > > > > > > to be rewritten to Go so we can remove the
> > > > > dependency on
> > > > > > > > > > > > perl.
> > > > > > > > > > > >
> > > > > > > > > > > > However, I still like marking many as
> > > > > "deprecated" so we
> > > > > > > > > > > > train users
> > > > > > > > > > to use
> > > > > > > > > > > > "better" endpoints that we envision will
> > > exist
> > > > > in 2.x and
> > > > > > > > > > > > beyond.
> > > > > > > > > > > >
> > > > > > > > > > > > Jeremy
> > > > > > > > > > > >
> > > > > > > > > > > > On Tue, Nov 12, 2019 at 11:22 AM Hoppal,
> > > Michael
> > > > > <
> > > > > > > > > > Michael_Hoppal@comcast.com>
> > > > > > > > > > > > wrote:
> > > > > > > > > > > >
> > > > > > > > > > > > > We do not have a lot of deprecated routes
> > > as
> > > > > of now as we
> > > > > > > > > > > > > focused on
> > > > > > > > > > > > > routes that are more used first.
> > > > > > > > > > > > >
> > > > > > > > > > > > > I know on a lot of the remaining rewrites
> > > > > there has been
> > > > > > > > > > > > > discussion
> > > > > > > > > > on the
> > > > > > > > > > > > > matching Github issues on potential
> > > > > deprecation (I cannot
> > > > > > > > > > > > > think of
> > > > > > > > > > the
> > > > > > > > > > > > > exact count but enough to warrant this
> > > email
> > > > > __)
> > > > > > > > > > > > >
> > > > > > > > > > > > > It is key to note that I am not arguing
> > for
> > > > > removal of
> > > > > > > > > > > > > endpoints
> > > > > > > > > > within
> > > > > > > > > > > > > the current API version but if there is
> > an
> > > > > endpoint we
> > > > > > > > > > > > > agree should
> > > > > > > > > > be
> > > > > > > > > > > > > deprecated going forward to put a message
> > > > > within the
> > > > > > > > > > > > > response and on
> > > > > > > > > > the
> > > > > > > > > > > > > docs.
> > > > > > > > > > > > >
> > > > > > > > > > > > > And thanks for the input sounds like you
> > > are
> > > > > +1 on
> > > > > > > > > > > > > rewriting all
> > > > > > > > > > routes to
> > > > > > > > > > > > > Golang as Brennan is.
> > > > > > > > > > > > >
> > > > > > > > > > > > > On 11/12/19, 9:06 AM, "Robert O Butts" <
> > > > > rob@apache.org>
> > > > > > > > > > > > > wrote:
> > > > > > > > > > > > >
> > > > > > > > > > > > > Whether we rewrite a route in Go is
> > an
> > > > > implementation
> > > > > > > > > > > > > detail. To
> > > > > > > > > > the
> > > > > > > > > > > > > interface, to users, it doesn't
> > matter
> > > > > whether a route
> > > > > > > > > > > > > is
> > > > > > > > > > rewritten or
> > > > > > > > > > > > > not.
> > > > > > > > > > > > >
> > > > > > > > > > > > > But our API follows Semantic
> > > Versioning,
> > > > > in order to
> > > > > > > > > > > > > not break
> > > > > > > > > > users.
> > > > > > > > > > > > > We
> > > > > > > > > > > > > can't just remove endpoints that some
> > > of
> > > > > us don't use,
> > > > > > > > > > > > > and
> > > > > > > > > > assume other
> > > > > > > > > > > > > people don't, maybe people who never
> > > speak
> > > > > up on the
> > > > > > > > > > > > > mailing
> > > > > > > > > > list.
> > > > > > > > > > > > > We'll
> > > > > > > > > > > > > never gain a big userbase if we keep
> > > > > breaking users.
> > > > > > > > > > > > >
> > > > > > > > > > > > > Per the _project_ SemVer, once we
> > have
> > > API
> > > > > 2.0, we can
> > > > > > > > > > > > > deprecate
> > > > > > > > > > API
> > > > > > > > > > > > > 1.x,
> > > > > > > > > > > > > and in the next major _project_
> > > release,
> > > > > remove API
> > > > > > > > > > > > > 1.x.
> > > > > > > > > > Irrespective
> > > > > > > > > > > > > of
> > > > > > > > > > > > > Perl or Go.
> > > > > > > > > > > > >
> > > > > > > > > > > > > My big concern is, API 2.0 is a big
> > > > > project. How long
> > > > > > > > > > > > > has the
> > > > > > > > > > rewrite
> > > > > > > > > > > > > to Go
> > > > > > > > > > > > > taken? Do we really believe designing
> > > and
> > > > > implementing
> > > > > > > > > > > > > a
> > > > > > > > > > completely
> > > > > > > > > > > > > new API
> > > > > > > > > > > > > will be any less time?
> > > > > > > > > > > > >
> > > > > > > > > > > > > I don't want killing Perl to have to
> > > wait
> > > > > on that.
> > > > > > > > > > > > >
> > > > > > > > > > > > > I know it feels like a waste to
> > rewrite
> > > > > routes that you
> > > > > > > > > > > > > don't
> > > > > > > > > > use, and
> > > > > > > > > > > > > probably few people do. But that's
> > the
> > > > > cost of a stable
> > > > > > > > > > > > > project.
> > > > > > > > > > How
> > > > > > > > > > > > > many
> > > > > > > > > > > > > "deprecated" routes are there? If it
> > > comes
> > > > > down to
> > > > > > > > > > > > > taking the
> > > > > > > > > > > > > development
> > > > > > > > > > > > > time to rewrite them so we can kill
> > > Perl
> > > > > faster, or
> > > > > > > > > > > > > leaving Perl
> > > > > > > > > > > > > around, I
> > > > > > > > > > > > > vote we just do the work and kill
> > Perl.
> > > > > > > > > > > > >
> > > > > > > > > > > > > >If we don't rewrite them, then Perl
> > > will
> > > > > last until
> > > > > > > > > > > > > API 2.0 has
> > > > > > > > > > been
> > > > > > > > > > > > > designed, released and then *another
> > > full
> > > > > major release
> > > > > > > > > > > > > cycle*.
> > > > > > > > > > That's
> > > > > > > > > > > > > way
> > > > > > > > > > > > > too long to have two codebases for
> > the
> > > > > same component,
> > > > > > > > > > > > > IMO,
> > > > > > > > > > especially
> > > > > > > > > > > > > since the rewrite is already 50%
> > > complete.
> > > > > > > > > > > > >
> > > > > > > > > > > > > +1
> > > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > > > > > On Tue, Nov 12, 2019 at 8:54 AM ocket
> > > 8888
> > > > > <
> > > > > > > > > > > > > ocket8888@gmail.com>
> > > > > > > > > > > > > wrote:
> > > > > > > > > > > > >
> > > > > > > > > > > > > > I vote that by and large we DO
> > > rewrite
> > > > > them, with
> > > > > > > > > > > > > exceptions
> > > > > > > > > > for
> > > > > > > > > > > > > routes
> > > > > > > > > > > > > > that just plain don't work, even in
> > > > > Perl. Those are
> > > > > > > > > > > > > few,
> > > > > > > > > > though.
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > If we don't rewrite them, then Perl
> > > will
> > > > > last until
> > > > > > > > > > > > > API 2.0
> > > > > > > > > > has been
> > > > > > > > > > > > > > designed, released and then
> > *another
> > > > > full major
> > > > > > > > > > > > > release cycle*.
> > > > > > > > > > > > > That's way
> > > > > > > > > > > > > > too long to have two codebases for
> > > the
> > > > > same
> > > > > > > > > > > > > component, IMO,
> > > > > > > > > > > > > especially
> > > > > > > > > > > > > > since the rewrite is already 50%
> > > > > complete.
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > On Tue, Nov 12, 2019 at 8:43 AM
> > > Hoppal,
> > > > > Michael <
> > > > > > > > > > > > > > Michael_Hoppal@comcast.com>
> > > > > > > > > > > > > > wrote:
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > > As the Traffic Ops API is being
> > > > > rewritten from Perl
> > > > > > > > > > > > > to Golang
> > > > > > > > > > > > > there has
> > > > > > > > > > > > > > > been several routes that have
> > been
> > > > > deprecated and
> > > > > > > > > > > > > probably
> > > > > > > > > > more to
> > > > > > > > > > > > > come.
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > In the deprecation efforts I have
> > > seen
> > > > > two
> > > > > > > > > > > > > strategies:
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > * The route IS NOT rewritten
> > > from
> > > > > Perl to
> > > > > > > > > > > > > Golang and a
> > > > > > > > > > > > > deprecation
> > > > > > > > > > > > > > > notice is added to the alert
> > > response
> > > > > in the Perl
> > > > > > > > > > > > > handler
> > > > > > > > > > > > > > > * The route IS rewritten from
> > > Perl
> > > > > to Golang
> > > > > > > > > > > > > and a
> > > > > > > > > > deprecation
> > > > > > > > > > > > > notice
> > > > > > > > > > > > > > > is added to the alert response in
> > > the
> > > > > Golang
> > > > > > > > > > > > > handler
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > I think we should have
> > consistency
> > > in
> > > > > our approach
> > > > > > > > > > > > > and
> > > > > > > > > > wanted to
> > > > > > > > > > > > > get
> > > > > > > > > > > > > > > people’s thoughts.
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > I would vote that we do not
> > > rewrite a
> > > > > deprecated
> > > > > > > > > > > > > route from
> > > > > > > > > > Perl to
> > > > > > > > > > > > > > Golang.
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > Thanks,
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > Michael
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > >
> > > > > > >
> > > > > >
> > > > > >
> > > > >
> > > > >
> > > > >
> > >
> >
Re: [EXTERNAL] Re: Traffic Ops Route Deprecation Strategy
Posted by Rawlin Peters <ra...@apache.org>.
Hey all,
A bunch of us got together and discussed this issue offline, and we
got consensus to go with Option 1 - Graduate current API from 1.x to
2.x - allowing us to leave behind routes in 1.x that are
dangerous/deprecated. Deprecated endpoints will not be rewritten from
Perl to Go.
So, we will graduate the latest minor version of existing 1.x Go
endpoints to 2.0. For example, today we have
/api/1.2/deliveryservices, /api/1.3/deliveryservices, and
/api/1.4/deliveryservices. We will take the existing handler for
/api/1.4/deliveryservices and promote it to /api/2.0/deliveryservices.
We will follow this example for other 1.x Go endpoints with multiple
minor versions as well.
API endpoints that are deemed broken, dangerous, or deprecated will
not be promoted to API 2.0. This will likely include the ~7 endpoints
that I've mentioned in this thread already, but we haven't yet
compiled the full list of endpoints that won't be promoted to 2.0. We
will come up with that list (along with its criteria) soon and bring
that list to the mailing list for everyone to look over and agree
upon. Deprecated endpoints will be updated to return deprecation
alerts where applicable.
This API graduation will likely miss the boat for 4.0 but will
hopefully land in 4.1 or 4.2 depending on how long it takes. So this
would be the tentative timetable:
TC 4.0:
- API 1.x supported, some deprecation notices
TC 4.1:
- API 1.x still supported, deprecation notices added to endpoints not
graduated to 2.0
- API 2.0 supported, consisting of 1.x endpoints that were graduated
- starting with this release, you need to start migrating external
clients off of 1.x over to 2.0
TC 4.2:
- internal clients (e.g. TM, TR, etc) will be migrated off API 1.x
over to 2.0. Doing this step after 4.1 adds confidence that 1.x is
still supported alongside 2.0 in order to provide a smooth migration
period for API clients.
TC 5.0:
- API 1.x no longer supported, only API 2.x is supported
Those who took part in that offline discussion, please keep me honest
here with a +1 if we did in fact come to a rough consensus on this
option. If you weren't part of that offline discussion, feel free to
voice your vote on this issue as well (I believe a bunch of people
were already +1 on this option a few weeks ago).
- Rawlin
On Tue, Nov 19, 2019 at 12:23 PM Robert O Butts <ro...@apache.org> wrote:
>
> > - /riak/stats
>
> Looking at the Perl, this looks easy to rewrite. It's just proxying a
> request to Riak.
>
> +1 on rewriting this, for API stability, even if we don't "value" the
> endpoint as a project.
>
> >- /cachegroup_fallbacks
>
> Likewise +1 on rewriting. Not as trivial as /riak/stats, but it also looks
> easy to me, looks like a standard CRUD endpoint. +1 on leaving a
> "deprecated, please prefer X" message in, and +1 in removing in the big API
> rewrite/refactor/2.0.
>
> > - /capabilities (all methods except for GET) -- capabilities are
> built-in, should not be changed at runtime
> > - /api_capabilities (all methods except for GET) -- same reasons as
> /capabilities
>
> I actually don't agree with this - because they're useful to plugins. IMO
> it should be possible to add and modify Capabilities and API Capabilities,
> because they can be used by plugins which add new endpoints, to control
> access to them. If there's disagreement there, I'd vote we take that to a
> separate thread.
>
> But in terms of the endpoints themselves, the Perl also looks easy to
> rewrite, looks like standard CRUD. -1 on deprecating or removing, +1 on
> rewriting.
>
> > - /cdns/{name}/configs/routing -- this was a half-baked attempt at
> redoing CrConfig.json -- cannot possibly be of use to anyone in its current
> state
>
> Ugh. I agree with the sentiment; and this one looks slightly less trivial.
> But I personally don't see it as an unreasonable amount of work. For the
> sake of API stability, I'm personally willing to volunteer to rewrite this
> one, including API tests to make fixing easy if the schema changes (which
> it appears to have in master). And hopefully it can just exist until the
> fabled API Rewrite. Is that acceptable?
>
> > - the ATS config generation endpoints -- Rob actually proposed to remove
> these from 1.x due to danger of being out of sync w/ cache-side config
> generator
>
> I'm personally on the fence about this one. Like I said in that thread, I'm
> the last one to want to break users, and I don't take this lightly. If,
> after I made the case for it, I'm the only one, I might be inclined to
> retract my proposal, and just continue to maintain them. Maybe some more
> API tests would help with that. I think I'm the primary dev on the ATS
> config gen at this point anyway.
>
> > - ATS Config Files Stuff Aren't these all already in Go anyway? Whether
> or not they should be removed from the API aside, I don't think anything
> actually needs to hit Perl anymore.
>
> Correct.
>
> > - /hwinfo/dtdata
>
> I think the 'render' call here is just rendering the JSON. But, the
> endpoint itself is just hanging and not returning for me, I'm not sure why.
> I'd like to volunteer to look into this endpoint as well. I think you might
> be right, this one might be unreasonable to rewrite. But if it is
> reasonable, I'd like to just do it. I'd like to at least look into it.
>
>
> On Tue, Nov 19, 2019 at 12:05 PM ocket 8888 <oc...@gmail.com> wrote:
>
> > I'm generally +1 for option 2
> >
> > regarding those specific endpoints:
> >
> > - /riak/stats if this is easy and doesn't expose anything dangerous, we
> > should do it IMO.
> > - /cachegroup_fallbacks should be pretty trivial, so I still say do it
> > - /capabilities since these aren't actually used for anything, I think it's
> > pretty safe (and easy) to rewrite. Plus there's already a PR open for
> > exactly that
> > - /api_capabilities dropping this one could be problematic, just because
> > people may depend on it to tell them what endpoints exist. To be clear, I
> > think it ought to be deprecated, but unfortunately I just don't think it's
> > terribly unlikely that this is built into scripts and tools. The actual
> > code for the rewrite should be easy too, although maintaining it is a pain.
> >
> > - ATS Config Files Stuff Aren't these all already in Go anyway? Whether or
> > not they should be removed from the API aside, I don't think anything
> > actually needs to hit Perl anymore.
> >
> > - /cdn/{{name}}/configs/routing CRConfigs are large and complex, so if this
> > never worked anyway maybe we should just add a Go handler for it that
> > returns a '410 Gone' response (or '501 Not Implemented' is the other one
> > I've heard suggested) and leave the actual logic behind in the Perl
> > - /hwinfo/dtdata if this calls into Mojolicious rendering, returns a
> > non-json response, and/or contains information that in either content or
> > encoding is meaningless to any sane client then it should also return a
> > 410/501 and get left behind in Perl.
> >
> > On Tue, Nov 19, 2019 at 11:08 AM Rawlin Peters <ra...@apache.org> wrote:
> >
> > > While I appreciate and mostly agree with your essay on the importance
> > > of not breaking users (or rather, breaking them in predictable ways),
> > > I for one would really like to meet at some middle ground with option
> > > 2. I'm not asking to remove 50 high-usage endpoints from 1.x. IIRC
> > > I've only specifically proposed "leaving behind" a few specific Perl
> > > endpoints that I believe serve no purpose, have no valid reason to be
> > > used, and/or are dangerous to the system:
> > >
> > > - /riak/stats -- informational, was used for debugging the initial
> > > Riak integration. This one might be pretty easy to rewrite since it
> > > just calls a corresponding riak API and returns the result via TO, so
> > > I could be on board with rewriting it if necessary.
> > > - /cachegroup_fallbacks -- these endpoints were released in 3.0 before
> > > we managed to augment the existing /cachegroups API to handle
> > > fallbacks, so I could get on board with rewriting these if we really
> > > think it's important.
> > > - /capabilities (all methods except for GET) -- capabilities are
> > > built-in, should not be changed at runtime
> > > - /api_capabilities (all methods except for GET) -- same reasons as
> > > /capabilities
> > > - /cdns/{name}/configs/routing -- this was a half-baked attempt at
> > > redoing CrConfig.json -- cannot possibly be of use to anyone in its
> > > current state
> > > - the ATS config generation endpoints -- Rob actually proposed to
> > > remove these from 1.x due to danger of being out of sync w/ cache-side
> > > config generator
> > > - /hwinfo/dtdata -- AFAIK the "dtdata" endpoints were only
> > > used/required by the old Perl Mojolicious UI, no valid reason to be
> > > using this as an external API client. In fact, I'm not even sure _how_
> > > this endpoint can be used outside of the Mojolicious framework/UI. It
> > > calls the Mojolicious `render` function, which usually means rendering
> > > an embedded Perl HTML template given some data. This is one I would
> > > consider completely unreasonable to rewrite to Go.
> > >
> > > Can we get on board with leaving at least these 1.x endpoints behind
> > > in the Perl? Doing a cursory skim over the remaining 1.x endpoints to
> > > rewrite, it _seems_ like everything else would fall into the "simpler
> > > to just rewrite rather than gain consensus to remove" category.
> > >
> > > - Rawlin
> > >
> > > On Tue, Nov 19, 2019 at 9:19 AM Robert O Butts <ro...@apache.org> wrote:
> > > >
> > > > >Option 1 - Graduate current API from 1.x to 2.x - allowing us to leave
> > > > behind routes in 1.x that are dangerous or deprecated
> > > > >Option 2 - Rewrite all endpoints from Perl to Golang in current
> > version
> > > as
> > > > we are doing now.
> > > >
> > > > +1 on either Option 1 or Option 2, without a strong preference. Option
> > 1
> > > > has advantages of letting us also safely remove the Cache Config
> > > endpoints
> > > > and implement Layered Profiles. But Option 2 also has the advantage of
> > > not
> > > > requiring users to upgrade.
> > > >
> > > > > Option 3 - Only rewrite endpoints from Perl to Golang in current
> > > version
> > > > that we agree are not dangerous or should deprecate
> > > >
> > > > -1. As before, it breaks users unpredictably.
> > > >
> > > > >It will literally outright break clients (*all* of them, not just
> > > *some*)
> > > >
> > > > This is technically correct: technically, new versions 'break' all
> > users.
> > > > The difference is, it does it in a predictable, safe way that users can
> > > > handle, inspect, and safely upgrade.
> > > >
> > > > >Can someone please explain to me why (A) artificially graduating the
> > > > current 1.x API to 2.x, and removing the deprecated endpoints from 1.x,
> > > is
> > > > better for API clients overall than (B) just removing deprecated
> > > endpoints
> > > > from 1.x? Existing 1.x clients would gain absolutely nothing from being
> > > > forced to update to 2.x like this.
> > > >
> > > > >for zero gain in value
> > > >
> > > > Users would gain something. There is a very specific gain.
> > > >
> > > > The gain is that the breakage is predictable. People frequently upgrade
> > > > minor versions automatically, either running commands without thinking,
> > > or
> > > > even automating the entire process.
> > > >
> > > > For TC, this applies to both CDN operators, and their end-users.
> > > >
> > > > When the minor versions contain breaking changes, that automatic
> > upgrade
> > > > then breaks. In this case, any scripts or tools they have using the
> > > removed
> > > > endpoints will break. If those scripts are important, this could bring
> > > down
> > > > their CDN, or break their Delivery Service, or otherwise cause an
> > outage.
> > > >
> > > > Contrawise, users expect breaking changes in Major versions, and don't
> > > > typically automatically upgrade them. Rather, they'll manually inspect
> > > the
> > > > update, do testing, and check the changelog for breaking changes.
> > > >
> > > > It's not a technical thing, it's a human thing. Humans automatically
> > > > upgrade minor versions, and write scripts to do so, and dependency
> > files
> > > to
> > > > allow it, because the software industry taught people that minor
> > versions
> > > > shouldn't break anything.
> > > >
> > > > If we start breaking minor versions, initially users will experience
> > > > breakage. Not all users, and not all catastrophic. But some users will
> > > > experience catastrophic outages. And it will be our fault as a project.
> > > And
> > > > after enough times, they'll fork TC, or find another product, or write
> > > > their own; and stop participating in the community, or never start.
> > > >
> > > > Then, once they've learned that we can't be trusted not to break
> > things,
> > > > they'll experience the 2nd problem: where nothing can be safely
> > upgraded,
> > > > and the smallest change requires long and expensive testing to verify a
> > > > single minor version or even patch upgrade, and still risk missing
> > > > something and bringing their CDN down. To be honest, we're pretty much
> > > > already here. It already costs our dev and ops teams unbelievable
> > amounts
> > > > of time and person-hours of testing for every minor upgrade. That time
> > > far
> > > > exceeds the development time to just rewrite the small endpoints we
> > don't
> > > > like anymore (and tangentially, also the time to write the Integration
> > > and
> > > > E2E tests TC desperately needs).
> > > >
> > > > ATC has so much potential. There aren't many OSS CDNs out there, and
> > > > there's certainly the demand. We have the potential to become a huge
> > > > project, which for us existing users, translates into potentially an
> > > > enormous amount of free development. But we'll never get there if we
> > keep
> > > > breaking things.
> > > >
> > > > And for my particular organization (Comcast), we have a hard enough
> > time
> > > > upgrading. Our upgrades, especially Traffic Ops, are extremely painful,
> > > and
> > > > require enormous amounts of manual testing. Even then, we frequently
> > have
> > > > issues that require rollback, sometimes even an outage; even after
> > those
> > > > unbelievably amounts of developer and operator time testing. We
> > > desperately
> > > > need a system that is safer and easier to upgrade. Breaking changes are
> > > > always going to require more testing; but we can at least make minor
> > > > versions safe to upgrade.
> > > >
> > > > > I know I am changing my initial stance but I am +1 on rewriting all
> > the
> > > > Perl endpoints to Golang and not breaking the API contract we have in
> > > > place. Most of the API routes to rewrite are rather simple to port and
> > I
> > > > think would be faster than getting consensus on actually deprecating it
> > > > from the mailing list.
> > > >
> > > > I'm inclined to agree with this. The things I've seen marked
> > "deprecated"
> > > > seem to mostly be small, and look like they'd be pretty easy to
> > rewrite.
> > > > But I haven't seen the whole list. Is there a list somewhere, so the
> > > > community can see it to better decide?
> > > >
> > > >
> > > > On Tue, Nov 19, 2019 at 7:24 AM Hoppal, Michael <
> > > Michael_Hoppal@comcast.com>
> > > > wrote:
> > > >
> > > > > I would love to drive this to some type of consensus on rewrite
> > > strategy
> > > > > as we have a good amount of resources (and growing) working on the
> > > rewrite
> > > > > and I want to keep that momentum.
> > > > >
> > > > > Sounds like there is three options from the email thread (please add
> > > any I
> > > > > missed)
> > > > >
> > > > > Option 1 - Graduate current API from 1.x to 2.x - allowing us to
> > leave
> > > > > behind routes in 1.x that are dangerous or deprecated
> > > > > Option 2 - Rewrite all endpoints from Perl to Golang in current
> > > version as
> > > > > we are doing now.
> > > > > Option 3 - Only rewrite endpoints from Perl to Golang in current
> > > version
> > > > > that we agree are not dangerous or should deprecate
> > > > >
> > > > > I am +1 for option 2 (which I know is different from my original
> > > stance).
> > > > >
> > > > > Which direction would others like to go?
> > > > >
> > > > > Thanks,
> > > > >
> > > > > Michael
> > > > >
> > > > >
> > > > > I know I am changing my initial stance but I am +1 on rewriting all
> > the
> > > > > Perl endpoints to Golang and not breaking the API contract we have in
> > > > > place. Most of the API routes to rewrite are rather simple to port
> > and
> > > I
> > > > > think would be faster than getting consensus on actually deprecating
> > it
> > > > > from the mailing list.
> > > > >
> > > > > On 11/16/19, 3:31 PM, "Rawlin Peters" <ra...@apache.org> wrote:
> > > > >
> > > > > > Moving to 2.x means that it's my fault as a consumer of the API
> > > if I
> > > > > was still using the deprecated endpoint (and that change should be
> > > part of
> > > > > a larger upgrade plan). Simply removing the deprecated routes,
> > > > > retroactively, means I can't rely on having a transition period to
> > get
> > > to
> > > > > 2.0 outside of a large completely down upgrade window.
> > > > >
> > > > > There would be a transition period either way. Endpoints would be
> > > > > deprecated in ATC 4.0 and removed in 5.0. The release notes for
> > 4.0
> > > > > would clearly list out the deprecated endpoints. If they have
> > > > > replacements, there would be instructions on how to transition to
> > > the
> > > > > replacement. Deprecated endpoints would return alerts. After
> > > upgrading
> > > > > to 4.0, operators should be well aware that certain endpoints are
> > > > > being removed in 5.0, and it's on them to start using new
> > > endpoints,
> > > > > change workflows, etc.
> > > > >
> > > > > By forcing operators to upgrade all of their clients to start
> > using
> > > > > API 2.x, it just adds to their burden of upgrading when that time
> > > > > comes. Rather than surgically fixing the clients of the 2% of
> > > > > deprecated API endpoints before upgrading to ATC 5.0, operators
> > > would
> > > > > have to fix every single client they have to use API 2.x before
> > > > > upgrading to 5.0. Is that exercise really worth it? Most people
> > > would
> > > > > probably be blissfully unaware that the "POST
> > > /api/1.1/capabilities"
> > > > > endpoint disappeared (among other deprecated endpoints). Less
> > > footguns
> > > > > in the system, and they didn't even have to update their clients.
> > > > >
> > > > > Anyways, circling back to Michael's original question, it sounds
> > > like
> > > > > either option we choose, we at least have consensus on this
> > > > > deprecation strategy:
> > > > > - The route IS NOT rewritten from Perl to Golang and a
> > deprecation
> > > > > notice is added to the alert response in the Perl handler
> > > > >
> > > > > If we're content to force all API clients to update their clients
> > > to
> > > > > 2.x for zero gain in value, at least we won't be wasting work by
> > > > > rewriting deprecated Perl endpoints to Go.
> > > > >
> > > > > - Rawlin
> > > > >
> > > > > On Fri, Nov 15, 2019 at 7:06 PM Gray, Jonathan
> > > > > <Jo...@comcast.com> wrote:
> > > > > >
> > > > > > Moving to 2.x means that it's my fault as a consumer of the API
> > > if I
> > > > > was still using the deprecated endpoint (and that change should be
> > > part of
> > > > > a larger upgrade plan). Simply removing the deprecated routes,
> > > > > retroactively, means I can't rely on having a transition period to
> > get
> > > to
> > > > > 2.0 outside of a large completely down upgrade window. Depending on
> > > how
> > > > > the API is used, even in our own components such as TM, that could
> > > prove
> > > > > disastrous. For the older minor revs like 1.1/1.2, I could live with
> > > > > dropping those as part of a major project release (including upgrade
> > > > > instructions outlining what releases you must be on beforehand).
> > > Currently
> > > > > 1.3 I view as stable enough to code to or use a native client library
> > > from,
> > > > > and 1.4 is my risk/overhead to take for new stuff as it's still
> > > evolving
> > > > > and improving. Going to 1.5 or 2.0 moves that forward. The path of
> > > least
> > > > > surprise for humans is to expect big shifts on those major revs.
> > > Removal
> > > > > falls under that umbrella and gets as a consumer not to strictly
> > focus
> > > on
> > > > > endpoint->endpoint moves, but rather workflow shifts and maybe now I
> > > have
> > > > > to use the API a different way than before. I believe getting this
> > > right
> > > > > is something that would greatly help us understand does X component
> > at
> > > > > version 1 work with Y component at version 2, because now you're
> > > asking do
> > > > > components X and Y both speak protocol A. With major releases we can
> > > work
> > > > > through that each time, but it's error prone and requires experts in
> > > all
> > > > > products to agree upon. If you do nothing but consume major
> > releases,
> > > and
> > > > > then conduct your own impact evaluation having read the upgrade
> > > > > instructions thoroughly, maybe you'd be able to smoothly transition.
> > > > > >
> > > > > > Jonathan G
> > > > > >
> > > > > >
> > > > > > On 11/15/19, 5:24 PM, "Rawlin Peters" <ra...@apache.org>
> > wrote:
> > > > > >
> > > > > > Can someone please explain to me why (A) artificially
> > > graduating
> > > > > the
> > > > > > current 1.x API to 2.x, and removing the deprecated
> > endpoints
> > > > > from
> > > > > > 1.x, is better for API clients overall than (B) just
> > removing
> > > > > > deprecated endpoints from 1.x? Existing 1.x clients would
> > > gain
> > > > > > absolutely nothing from being forced to update to 2.x like
> > > this.
> > > > > >
> > > > > > The way I see the two choices:
> > > > > > A) break every single API client out there
> > > > > > B) break only clients of APIs that we've decided upon as a
> > > > > community
> > > > > > to be deprecated
> > > > > >
> > > > > > IMO, when we have to break things, we should try break the
> > > least
> > > > > > amount of clients possible. All I'm looking for is a
> > valuable
> > > > > reason
> > > > > > to choose option A over option B, that isn't just "because
> > > > > SemVer".
> > > > > >
> > > > > > - Rawlin
> > > > > >
> > > > > > On Wed, Nov 13, 2019 at 3:24 PM Schmidt, Andrew
> > (Contractor)
> > > > > > <An...@comcast.com> wrote:
> > > > > > >
> > > > > > > If we decide to get rid of the API versioning it makes
> > > sense
> > > > > to me that we would do that on a major version upgrade, probably to
> > > the ATC
> > > > > version. But I think it would be too early to do it on the 2.0 API
> > > version.
> > > > > I like this compromise +1. I was wondering why we were stuck on the
> > > idea
> > > > > that 2.0 had to be an API rewrite. That seems like it’s a long way
> > > out. 2.0
> > > > > = deprecations and fixes. 3.0 or 5.0 = api rewrite and maybe version
> > > > > convergence.
> > > > > > >
> > > > > > > On 11/13/19, 11:52 AM, "ocket8888@gmail.com" <
> > > > > ocket8888@gmail.com> wrote:
> > > > > > >
> > > > > > > Lemme lead with: making breaking changes on a major
> > > > > version boundary
> > > > > > > gets +1 from me - no reason the API needs to be fully
> > > > > redesigned for
> > > > > > > any particular number.
> > > > > > >
> > > > > > > But I just gotta bring this up again: if we want to
> > be
> > > > > able to
> > > > > > > deprecate an endpoint for a major release and remove
> > > it in
> > > > > the next
> > > > > > > without needing to deal with the messiness of API
> > > > > versions, we could
> > > > > > > always just get rid of API versions. Major ATC
> > > revisions
> > > > > should be
> > > > > > > easier to come by with our new release strategy.
> > > > > > >
> > > > > > > On Wed, 2019-11-13 at 10:23 -0700, Rawlin Peters
> > wrote:
> > > > > > > > > It is a compromise - it's an inconvenience to
> > > users,
> > > > > forcing them
> > > > > > > > > to fix
> > > > > > > > scripts to upgrade. Which isn't good for user
> > > experience
> > > > > either; but
> > > > > > > > it
> > > > > > > > doesn't outright break people. It's a compromise
> > I'm
> > > > > willing to live
> > > > > > > > with.
> > > > > > > >
> > > > > > > > It will literally outright break clients (*all* of
> > > them,
> > > > > not just
> > > > > > > > *some*). We have been down this road before, so
> > I'll
> > > > > just repeat what
> > > > > > > > I said when this exact conversation came up months
> > > > > ago[1]:
> > > > > > > >
> > > > > > > > "So it basically boils down to:
> > > > > > > > A) promote Go 1.x routes to 2.x and eventually
> > break
> > > ALL
> > > > > users of the
> > > > > > > > 1.x API when the time comes to remove 1.x
> > > > > > > > B) deprecate and remove some 1.x Perl routes and
> > > > > POSSIBLY break SOME
> > > > > > > > users"
> > > > > > > >
> > > > > > > > Either way we slice and dice it with versioning,
> > > > > removing an endpoint
> > > > > > > > means that clients of that endpoint will break. By
> > > > > graduating only
> > > > > > > > the
> > > > > > > > "good stuff" from 1.x to 2.x, and removing
> > endpoints
> > > > > from 1.x, we
> > > > > > > > will
> > > > > > > > unnecessarily break all 1.x clients when the time
> > > comes
> > > > > to remove the
> > > > > > > > rest of 1.x. Instead, we can choose to break only
> > > > > clients of 1.x APIs
> > > > > > > > that we already deemed dangerous, unnecessary,
> > > useless,
> > > > > etc. enough
> > > > > > > > to
> > > > > > > > deprecate.
> > > > > > > >
> > > > > > > > Why break 100% of our API clients when we can
> > choose
> > > to
> > > > > break only a
> > > > > > > > small number of clients that happen to use
> > > > > > > > dangerous/unnecessary/useless APIs? Why would we
> > > break
> > > > > our API
> > > > > > > > clients
> > > > > > > > unnecessarily for no added value? There is no sense
> > > in
> > > > > blindly
> > > > > > > > adhering to Semantic Versioning just for the sake
> > of
> > > > > adhering to
> > > > > > > > Semantic Versioning. Instead, use it as a
> > guideline,
> > > and
> > > > > break the
> > > > > > > > rules where it makes the most sense to do so.
> > > > > > > >
> > > > > > > > - Rawlin
> > > > > > > >
> > > > > > > > [1]
> > > > > > > >
> > > > >
> > >
> > https://urldefense.com/v3/__https://lists.apache.org/thread.html/15c6bfcdce0ac00a3a8612a96c3d668a8a5a16c6ca9401de9473d31c@*3Cdev.trafficcontrol.apache.org*3E__;JSU!rx_L75ITgOQ!V_b3E9iEYqNlZus18f-sVuofwuyxapCthg3LdO4NxyMCU39b7mdf4klP5KNR7XZ7d3Uc$
> > > > > > > >
> > > > > > > > On Wed, Nov 13, 2019 at 9:43 AM Robert O Butts <
> > > > > rob@apache.org>
> > > > > > > > wrote:
> > > > > > > > > > But if it's really valuable to remove these
> > > > > endpoints, maybe it's
> > > > > > > > > appropriate to bump the major version and ask
> > > > > everybody to update
> > > > > > > > > their
> > > > > > > > > clients?
> > > > > > > > >
> > > > > > > > > +1
> > > > > > > > >
> > > > > > > > > This sounds like the best compromise to me. It
> > > > > addresses both sides
> > > > > > > > > - it
> > > > > > > > > lets us remove endpoints, and it also prevents
> > > > > breaking users.
> > > > > > > > >
> > > > > > > > > It is a compromise - it's an inconvenience to
> > > users,
> > > > > forcing them
> > > > > > > > > to fix
> > > > > > > > > scripts to upgrade. Which isn't good for user
> > > > > experience either;
> > > > > > > > > but it
> > > > > > > > > doesn't outright break people. It's a compromise
> > > I'm
> > > > > willing to
> > > > > > > > > live with.
> > > > > > > > >
> > > > > > > > > This actually lets us do at least 3 things:
> > > > > > > > > 1. Remove "deprecated" endpoints.
> > > > > > > > > 2. Remove the unmaintained dangerously-large
> > Cache
> > > > > Config
> > > > > > > > > endpoints.
> > > > > > > > > 3. Lets us do Layered Profiles, changing Servers
> > > and
> > > > > DSes to have
> > > > > > > > > arrays of
> > > > > > > > > Profiles instead of a single Profile, without
> > hacky
> > > > > backward-
> > > > > > > > > compatibility
> > > > > > > > > attempts.
> > > > > > > > >
> > > > > > > > > To clarify, `/api/2.0` would be essentially the
> > > > > current API, just
> > > > > > > > > with the
> > > > > > > > > above breaking changes (possibly others) which
> > > require
> > > > > a SemVer
> > > > > > > > > major
> > > > > > > > > version increase. And the big API
> > rewrite/redesign
> > > > > we've been
> > > > > > > > > calling "2.0"
> > > > > > > > > would become `/api/3.0` or some future version.
> > > > > > > > >
> > > > > > > > > It's not perfect, but it seems to address the
> > most
> > > > > concerns on all
> > > > > > > > > sides.
> > > > > > > > > Thoughts? Objections?
> > > > > > > > >
> > > > > > > > >
> > > > > > > > > On Wed, Nov 13, 2019 at 8:45 AM Chris Lemmons <
> > > > > alficles@gmail.com>
> > > > > > > > > wrote:
> > > > > > > > >
> > > > > > > > > > For the endpoints, I agree in part and disagree
> > > in
> > > > > part.
> > > > > > > > > >
> > > > > > > > > > Agree for these two:
> > > > > > > > > > - the endpoints are dangerous and pose a risk
> > to
> > > the
> > > > > overall
> > > > > > > > > > Traffic
> > > > > > > > > > Control system
> > > > > > > > > > - the task of rewriting the endpoints from Perl
> > > to
> > > > > Go represents
> > > > > > > > > > an
> > > > > > > > > > unreasonable amount of work for the project
> > > > > > > > > >
> > > > > > > > > > And I would add:
> > > > > > > > > > - the endpoint does not serve its intended
> > > purpose
> > > > > and cannot
> > > > > > > > > > reasonably
> > > > > > > > > > be used
> > > > > > > > > > - the endpoint has been made unsafe or unusable
> > > by
> > > > > new features
> > > > > > > > > > added to TC
> > > > > > > > > >
> > > > > > > > > > Disagree for these two:
> > > > > > > > > > - the endpoints have been obsoleted by other
> > > > > endpoints
> > > > > > > > > > - the endpoints don't seem to serve a true
> > > purpose
> > > > > or don't seem
> > > > > > > > > > to be
> > > > > > > > > > valuable to the project anymore
> > > > > > > > > >
> > > > > > > > > > For the most part, obsoleted endpoints can be
> > > > > maintained with a
> > > > > > > > > > shim
> > > > > > > > > > into the new code. If they would require "an
> > > > > unreasonable amount
> > > > > > > > > > of
> > > > > > > > > > work", it might make sense to remove them,
> > then.
> > > > > > > > > >
> > > > > > > > > > Things that don't seem valuable usually fall
> > into
> > > > > another
> > > > > > > > > > category
> > > > > > > > > > like "cannot reasonably be used", but if the
> > only
> > > > > reason for
> > > > > > > > > > deprecation is that the feature is going away,
> > we
> > > > > should really
> > > > > > > > > > support it according to our promise.
> > > > > > > > > >
> > > > > > > > > > But if it's really valuable to remove these
> > > > > endpoints, maybe it's
> > > > > > > > > > appropriate to bump the major version and ask
> > > > > everybody to update
> > > > > > > > > > their clients?
> > > > > > > > > >
> > > > > > > > > > On Tue, Nov 12, 2019 at 5:27 PM Rawlin Peters <
> > > > > rawlin@apache.org>
> > > > > > > > > > wrote:
> > > > > > > > > > > The real crux of the issue here is whether or
> > > not
> > > > > we agree as a
> > > > > > > > > > > project that we can move forward with this
> > plan
> > > > > that I believe
> > > > > > > > > > > to be a
> > > > > > > > > > > good compromise that meets most concerns but
> > > > > "breaks the
> > > > > > > > > > > Semantic
> > > > > > > > > > > Versioning promise":
> > > > > > > > > > > 1) don't rewrite deprecated endpoints from
> > > Perl to
> > > > > Go
> > > > > > > > > > > 2) in the Perl handler, add a deprecation
> > > notice
> > > > > to the
> > > > > > > > > > > "alerts"
> > > > > > > > > > > section of the response
> > > > > > > > > > > 3) once a deprecated API has been deprecated
> > > for
> > > > > one major
> > > > > > > > > > > release,
> > > > > > > > > > > delete it from the API in the subsequent
> > major
> > > > > release (e.g. if
> > > > > > > > > > > deprecation notices are added and released in
> > > ATC
> > > > > 4.x, we can
> > > > > > > > > > > remove
> > > > > > > > > > > the deprecated endpoint in ATC 5.x)
> > > > > > > > > > >
> > > > > > > > > > > Rob, from your latest proposal to remove the
> > > ATS
> > > > > config API
> > > > > > > > > > > endpoints
> > > > > > > > > > > from 1.x, I thought we were past this issue
> > > > > already and could
> > > > > > > > > > > finally
> > > > > > > > > > > agree to remove deprecated endpoints from 1.x
> > > > > (following steps
> > > > > > > > > > > 1-3
> > > > > > > > > > > above) without requiring an all-new 2.x API.
> > > Are
> > > > > we not
> > > > > > > > > > > actually past
> > > > > > > > > > > that? You came up with valid reasons for
> > > applying
> > > > > these steps
> > > > > > > > > > > to the
> > > > > > > > > > > ATS config API endpoints, and I think it's
> > > fair to
> > > > > say that we
> > > > > > > > > > > can
> > > > > > > > > > > come up with valid reasons for applying these
> > > > > steps to other
> > > > > > > > > > > deprecated API endpoints as well.
> > > > > > > > > > >
> > > > > > > > > > > It seems like the real problem is actually
> > > > > agreeing on which
> > > > > > > > > > > _specific_ endpoints we should be able to
> > > follow
> > > > > steps 1-3 for.
> > > > > > > > > > > IMO I
> > > > > > > > > > > think it should apply to any 1.x endpoints
> > > that we
> > > > > have valid
> > > > > > > > > > > reasons
> > > > > > > > > > > for not carrying forward to Go, including but
> > > not
> > > > > limited to:
> > > > > > > > > > > - the endpoints are dangerous and pose a risk
> > > to
> > > > > the overall
> > > > > > > > > > > Traffic
> > > > > > > > > > > Control system
> > > > > > > > > > > - the endpoints have been obsoleted by other
> > > > > endpoints
> > > > > > > > > > > - the endpoints don't seem to serve a true
> > > purpose
> > > > > or don't
> > > > > > > > > > > seem to be
> > > > > > > > > > > valuable to the project anymore
> > > > > > > > > > > - the task of rewriting the endpoints from
> > > Perl to
> > > > > Go
> > > > > > > > > > > represents an
> > > > > > > > > > > unreasonable amount of work for the project
> > > > > > > > > > >
> > > > > > > > > > > Obviously most of these reasons are
> > > subjective, so
> > > > > we'd have to
> > > > > > > > > > > come
> > > > > > > > > > > to some level of consensus on deprecating
> > > > > particular endpoints
> > > > > > > > > > > first,
> > > > > > > > > > > but we should be able to follow steps 1-3 for
> > > > > whatever
> > > > > > > > > > > endpoints we
> > > > > > > > > > > decide.
> > > > > > > > > > >
> > > > > > > > > > > Making breaking changes shouldn't be taken
> > > > > lightly, but we need
> > > > > > > > > > > a path
> > > > > > > > > > > forward that allows some calculated, breaking
> > > > > changes where
> > > > > > > > > > > necessary.
> > > > > > > > > > > Otherwise, we are unnecessarily holding the
> > > > > project back.
> > > > > > > > > > >
> > > > > > > > > > > - Rawlin
> > > > > > > > > > >
> > > > > > > > > > > On Tue, Nov 12, 2019 at 11:44 AM Jeremy
> > > Mitchell <
> > > > > > > > > > > mitchell852@gmail.com>
> > > > > > > > > > wrote:
> > > > > > > > > > > > Yeah, technically a "rewrite" of 1.x means
> > > > > shifting the
> > > > > > > > > > > > implementation
> > > > > > > > > > from
> > > > > > > > > > > > Perl to Go for ALL of 1.x. If you leave a
> > > subset
> > > > > of 1.x in
> > > > > > > > > > > > Perl, then
> > > > > > > > > > you
> > > > > > > > > > > > have to call it a "partial rewrite".
> > > > > > > > > > > >
> > > > > > > > > > > > So, assuming we want to do a full rewrite
> > (no
> > > > > more perl for
> > > > > > > > > > > > the api
> > > > > > > > > > > > whatsoever), I think that means every 1.x
> > api
> > > > > endpoint (good
> > > > > > > > > > > > or bad)
> > > > > > > > > > needs
> > > > > > > > > > > > to be rewritten to Go so we can remove the
> > > > > dependency on
> > > > > > > > > > > > perl.
> > > > > > > > > > > >
> > > > > > > > > > > > However, I still like marking many as
> > > > > "deprecated" so we
> > > > > > > > > > > > train users
> > > > > > > > > > to use
> > > > > > > > > > > > "better" endpoints that we envision will
> > > exist
> > > > > in 2.x and
> > > > > > > > > > > > beyond.
> > > > > > > > > > > >
> > > > > > > > > > > > Jeremy
> > > > > > > > > > > >
> > > > > > > > > > > > On Tue, Nov 12, 2019 at 11:22 AM Hoppal,
> > > Michael
> > > > > <
> > > > > > > > > > Michael_Hoppal@comcast.com>
> > > > > > > > > > > > wrote:
> > > > > > > > > > > >
> > > > > > > > > > > > > We do not have a lot of deprecated routes
> > > as
> > > > > of now as we
> > > > > > > > > > > > > focused on
> > > > > > > > > > > > > routes that are more used first.
> > > > > > > > > > > > >
> > > > > > > > > > > > > I know on a lot of the remaining rewrites
> > > > > there has been
> > > > > > > > > > > > > discussion
> > > > > > > > > > on the
> > > > > > > > > > > > > matching Github issues on potential
> > > > > deprecation (I cannot
> > > > > > > > > > > > > think of
> > > > > > > > > > the
> > > > > > > > > > > > > exact count but enough to warrant this
> > > email
> > > > > __)
> > > > > > > > > > > > >
> > > > > > > > > > > > > It is key to note that I am not arguing
> > for
> > > > > removal of
> > > > > > > > > > > > > endpoints
> > > > > > > > > > within
> > > > > > > > > > > > > the current API version but if there is
> > an
> > > > > endpoint we
> > > > > > > > > > > > > agree should
> > > > > > > > > > be
> > > > > > > > > > > > > deprecated going forward to put a message
> > > > > within the
> > > > > > > > > > > > > response and on
> > > > > > > > > > the
> > > > > > > > > > > > > docs.
> > > > > > > > > > > > >
> > > > > > > > > > > > > And thanks for the input sounds like you
> > > are
> > > > > +1 on
> > > > > > > > > > > > > rewriting all
> > > > > > > > > > routes to
> > > > > > > > > > > > > Golang as Brennan is.
> > > > > > > > > > > > >
> > > > > > > > > > > > > On 11/12/19, 9:06 AM, "Robert O Butts" <
> > > > > rob@apache.org>
> > > > > > > > > > > > > wrote:
> > > > > > > > > > > > >
> > > > > > > > > > > > > Whether we rewrite a route in Go is
> > an
> > > > > implementation
> > > > > > > > > > > > > detail. To
> > > > > > > > > > the
> > > > > > > > > > > > > interface, to users, it doesn't
> > matter
> > > > > whether a route
> > > > > > > > > > > > > is
> > > > > > > > > > rewritten or
> > > > > > > > > > > > > not.
> > > > > > > > > > > > >
> > > > > > > > > > > > > But our API follows Semantic
> > > Versioning,
> > > > > in order to
> > > > > > > > > > > > > not break
> > > > > > > > > > users.
> > > > > > > > > > > > > We
> > > > > > > > > > > > > can't just remove endpoints that some
> > > of
> > > > > us don't use,
> > > > > > > > > > > > > and
> > > > > > > > > > assume other
> > > > > > > > > > > > > people don't, maybe people who never
> > > speak
> > > > > up on the
> > > > > > > > > > > > > mailing
> > > > > > > > > > list.
> > > > > > > > > > > > > We'll
> > > > > > > > > > > > > never gain a big userbase if we keep
> > > > > breaking users.
> > > > > > > > > > > > >
> > > > > > > > > > > > > Per the _project_ SemVer, once we
> > have
> > > API
> > > > > 2.0, we can
> > > > > > > > > > > > > deprecate
> > > > > > > > > > API
> > > > > > > > > > > > > 1.x,
> > > > > > > > > > > > > and in the next major _project_
> > > release,
> > > > > remove API
> > > > > > > > > > > > > 1.x.
> > > > > > > > > > Irrespective
> > > > > > > > > > > > > of
> > > > > > > > > > > > > Perl or Go.
> > > > > > > > > > > > >
> > > > > > > > > > > > > My big concern is, API 2.0 is a big
> > > > > project. How long
> > > > > > > > > > > > > has the
> > > > > > > > > > rewrite
> > > > > > > > > > > > > to Go
> > > > > > > > > > > > > taken? Do we really believe designing
> > > and
> > > > > implementing
> > > > > > > > > > > > > a
> > > > > > > > > > completely
> > > > > > > > > > > > > new API
> > > > > > > > > > > > > will be any less time?
> > > > > > > > > > > > >
> > > > > > > > > > > > > I don't want killing Perl to have to
> > > wait
> > > > > on that.
> > > > > > > > > > > > >
> > > > > > > > > > > > > I know it feels like a waste to
> > rewrite
> > > > > routes that you
> > > > > > > > > > > > > don't
> > > > > > > > > > use, and
> > > > > > > > > > > > > probably few people do. But that's
> > the
> > > > > cost of a stable
> > > > > > > > > > > > > project.
> > > > > > > > > > How
> > > > > > > > > > > > > many
> > > > > > > > > > > > > "deprecated" routes are there? If it
> > > comes
> > > > > down to
> > > > > > > > > > > > > taking the
> > > > > > > > > > > > > development
> > > > > > > > > > > > > time to rewrite them so we can kill
> > > Perl
> > > > > faster, or
> > > > > > > > > > > > > leaving Perl
> > > > > > > > > > > > > around, I
> > > > > > > > > > > > > vote we just do the work and kill
> > Perl.
> > > > > > > > > > > > >
> > > > > > > > > > > > > >If we don't rewrite them, then Perl
> > > will
> > > > > last until
> > > > > > > > > > > > > API 2.0 has
> > > > > > > > > > been
> > > > > > > > > > > > > designed, released and then *another
> > > full
> > > > > major release
> > > > > > > > > > > > > cycle*.
> > > > > > > > > > That's
> > > > > > > > > > > > > way
> > > > > > > > > > > > > too long to have two codebases for
> > the
> > > > > same component,
> > > > > > > > > > > > > IMO,
> > > > > > > > > > especially
> > > > > > > > > > > > > since the rewrite is already 50%
> > > complete.
> > > > > > > > > > > > >
> > > > > > > > > > > > > +1
> > > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > > > > > On Tue, Nov 12, 2019 at 8:54 AM ocket
> > > 8888
> > > > > <
> > > > > > > > > > > > > ocket8888@gmail.com>
> > > > > > > > > > > > > wrote:
> > > > > > > > > > > > >
> > > > > > > > > > > > > > I vote that by and large we DO
> > > rewrite
> > > > > them, with
> > > > > > > > > > > > > exceptions
> > > > > > > > > > for
> > > > > > > > > > > > > routes
> > > > > > > > > > > > > > that just plain don't work, even in
> > > > > Perl. Those are
> > > > > > > > > > > > > few,
> > > > > > > > > > though.
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > If we don't rewrite them, then Perl
> > > will
> > > > > last until
> > > > > > > > > > > > > API 2.0
> > > > > > > > > > has been
> > > > > > > > > > > > > > designed, released and then
> > *another
> > > > > full major
> > > > > > > > > > > > > release cycle*.
> > > > > > > > > > > > > That's way
> > > > > > > > > > > > > > too long to have two codebases for
> > > the
> > > > > same
> > > > > > > > > > > > > component, IMO,
> > > > > > > > > > > > > especially
> > > > > > > > > > > > > > since the rewrite is already 50%
> > > > > complete.
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > On Tue, Nov 12, 2019 at 8:43 AM
> > > Hoppal,
> > > > > Michael <
> > > > > > > > > > > > > > Michael_Hoppal@comcast.com>
> > > > > > > > > > > > > > wrote:
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > > As the Traffic Ops API is being
> > > > > rewritten from Perl
> > > > > > > > > > > > > to Golang
> > > > > > > > > > > > > there has
> > > > > > > > > > > > > > > been several routes that have
> > been
> > > > > deprecated and
> > > > > > > > > > > > > probably
> > > > > > > > > > more to
> > > > > > > > > > > > > come.
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > In the deprecation efforts I have
> > > seen
> > > > > two
> > > > > > > > > > > > > strategies:
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > * The route IS NOT rewritten
> > > from
> > > > > Perl to
> > > > > > > > > > > > > Golang and a
> > > > > > > > > > > > > deprecation
> > > > > > > > > > > > > > > notice is added to the alert
> > > response
> > > > > in the Perl
> > > > > > > > > > > > > handler
> > > > > > > > > > > > > > > * The route IS rewritten from
> > > Perl
> > > > > to Golang
> > > > > > > > > > > > > and a
> > > > > > > > > > deprecation
> > > > > > > > > > > > > notice
> > > > > > > > > > > > > > > is added to the alert response in
> > > the
> > > > > Golang
> > > > > > > > > > > > > handler
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > I think we should have
> > consistency
> > > in
> > > > > our approach
> > > > > > > > > > > > > and
> > > > > > > > > > wanted to
> > > > > > > > > > > > > get
> > > > > > > > > > > > > > > people’s thoughts.
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > I would vote that we do not
> > > rewrite a
> > > > > deprecated
> > > > > > > > > > > > > route from
> > > > > > > > > > Perl to
> > > > > > > > > > > > > > Golang.
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > Thanks,
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > Michael
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > >
> > > > > > >
> > > > > >
> > > > > >
> > > > >
> > > > >
> > > > >
> > >
> >
Re: [EXTERNAL] Re: Traffic Ops Route Deprecation Strategy
Posted by Robert O Butts <ro...@apache.org>.
> - /riak/stats
Looking at the Perl, this looks easy to rewrite. It's just proxying a
request to Riak.
+1 on rewriting this, for API stability, even if we don't "value" the
endpoint as a project.
>- /cachegroup_fallbacks
Likewise +1 on rewriting. Not as trivial as /riak/stats, but it also looks
easy to me, looks like a standard CRUD endpoint. +1 on leaving a
"deprecated, please prefer X" message in, and +1 in removing in the big API
rewrite/refactor/2.0.
> - /capabilities (all methods except for GET) -- capabilities are
built-in, should not be changed at runtime
> - /api_capabilities (all methods except for GET) -- same reasons as
/capabilities
I actually don't agree with this - because they're useful to plugins. IMO
it should be possible to add and modify Capabilities and API Capabilities,
because they can be used by plugins which add new endpoints, to control
access to them. If there's disagreement there, I'd vote we take that to a
separate thread.
But in terms of the endpoints themselves, the Perl also looks easy to
rewrite, looks like standard CRUD. -1 on deprecating or removing, +1 on
rewriting.
> - /cdns/{name}/configs/routing -- this was a half-baked attempt at
redoing CrConfig.json -- cannot possibly be of use to anyone in its current
state
Ugh. I agree with the sentiment; and this one looks slightly less trivial.
But I personally don't see it as an unreasonable amount of work. For the
sake of API stability, I'm personally willing to volunteer to rewrite this
one, including API tests to make fixing easy if the schema changes (which
it appears to have in master). And hopefully it can just exist until the
fabled API Rewrite. Is that acceptable?
> - the ATS config generation endpoints -- Rob actually proposed to remove
these from 1.x due to danger of being out of sync w/ cache-side config
generator
I'm personally on the fence about this one. Like I said in that thread, I'm
the last one to want to break users, and I don't take this lightly. If,
after I made the case for it, I'm the only one, I might be inclined to
retract my proposal, and just continue to maintain them. Maybe some more
API tests would help with that. I think I'm the primary dev on the ATS
config gen at this point anyway.
> - ATS Config Files Stuff Aren't these all already in Go anyway? Whether
or not they should be removed from the API aside, I don't think anything
actually needs to hit Perl anymore.
Correct.
> - /hwinfo/dtdata
I think the 'render' call here is just rendering the JSON. But, the
endpoint itself is just hanging and not returning for me, I'm not sure why.
I'd like to volunteer to look into this endpoint as well. I think you might
be right, this one might be unreasonable to rewrite. But if it is
reasonable, I'd like to just do it. I'd like to at least look into it.
On Tue, Nov 19, 2019 at 12:05 PM ocket 8888 <oc...@gmail.com> wrote:
> I'm generally +1 for option 2
>
> regarding those specific endpoints:
>
> - /riak/stats if this is easy and doesn't expose anything dangerous, we
> should do it IMO.
> - /cachegroup_fallbacks should be pretty trivial, so I still say do it
> - /capabilities since these aren't actually used for anything, I think it's
> pretty safe (and easy) to rewrite. Plus there's already a PR open for
> exactly that
> - /api_capabilities dropping this one could be problematic, just because
> people may depend on it to tell them what endpoints exist. To be clear, I
> think it ought to be deprecated, but unfortunately I just don't think it's
> terribly unlikely that this is built into scripts and tools. The actual
> code for the rewrite should be easy too, although maintaining it is a pain.
>
> - ATS Config Files Stuff Aren't these all already in Go anyway? Whether or
> not they should be removed from the API aside, I don't think anything
> actually needs to hit Perl anymore.
>
> - /cdn/{{name}}/configs/routing CRConfigs are large and complex, so if this
> never worked anyway maybe we should just add a Go handler for it that
> returns a '410 Gone' response (or '501 Not Implemented' is the other one
> I've heard suggested) and leave the actual logic behind in the Perl
> - /hwinfo/dtdata if this calls into Mojolicious rendering, returns a
> non-json response, and/or contains information that in either content or
> encoding is meaningless to any sane client then it should also return a
> 410/501 and get left behind in Perl.
>
> On Tue, Nov 19, 2019 at 11:08 AM Rawlin Peters <ra...@apache.org> wrote:
>
> > While I appreciate and mostly agree with your essay on the importance
> > of not breaking users (or rather, breaking them in predictable ways),
> > I for one would really like to meet at some middle ground with option
> > 2. I'm not asking to remove 50 high-usage endpoints from 1.x. IIRC
> > I've only specifically proposed "leaving behind" a few specific Perl
> > endpoints that I believe serve no purpose, have no valid reason to be
> > used, and/or are dangerous to the system:
> >
> > - /riak/stats -- informational, was used for debugging the initial
> > Riak integration. This one might be pretty easy to rewrite since it
> > just calls a corresponding riak API and returns the result via TO, so
> > I could be on board with rewriting it if necessary.
> > - /cachegroup_fallbacks -- these endpoints were released in 3.0 before
> > we managed to augment the existing /cachegroups API to handle
> > fallbacks, so I could get on board with rewriting these if we really
> > think it's important.
> > - /capabilities (all methods except for GET) -- capabilities are
> > built-in, should not be changed at runtime
> > - /api_capabilities (all methods except for GET) -- same reasons as
> > /capabilities
> > - /cdns/{name}/configs/routing -- this was a half-baked attempt at
> > redoing CrConfig.json -- cannot possibly be of use to anyone in its
> > current state
> > - the ATS config generation endpoints -- Rob actually proposed to
> > remove these from 1.x due to danger of being out of sync w/ cache-side
> > config generator
> > - /hwinfo/dtdata -- AFAIK the "dtdata" endpoints were only
> > used/required by the old Perl Mojolicious UI, no valid reason to be
> > using this as an external API client. In fact, I'm not even sure _how_
> > this endpoint can be used outside of the Mojolicious framework/UI. It
> > calls the Mojolicious `render` function, which usually means rendering
> > an embedded Perl HTML template given some data. This is one I would
> > consider completely unreasonable to rewrite to Go.
> >
> > Can we get on board with leaving at least these 1.x endpoints behind
> > in the Perl? Doing a cursory skim over the remaining 1.x endpoints to
> > rewrite, it _seems_ like everything else would fall into the "simpler
> > to just rewrite rather than gain consensus to remove" category.
> >
> > - Rawlin
> >
> > On Tue, Nov 19, 2019 at 9:19 AM Robert O Butts <ro...@apache.org> wrote:
> > >
> > > >Option 1 - Graduate current API from 1.x to 2.x - allowing us to leave
> > > behind routes in 1.x that are dangerous or deprecated
> > > >Option 2 - Rewrite all endpoints from Perl to Golang in current
> version
> > as
> > > we are doing now.
> > >
> > > +1 on either Option 1 or Option 2, without a strong preference. Option
> 1
> > > has advantages of letting us also safely remove the Cache Config
> > endpoints
> > > and implement Layered Profiles. But Option 2 also has the advantage of
> > not
> > > requiring users to upgrade.
> > >
> > > > Option 3 - Only rewrite endpoints from Perl to Golang in current
> > version
> > > that we agree are not dangerous or should deprecate
> > >
> > > -1. As before, it breaks users unpredictably.
> > >
> > > >It will literally outright break clients (*all* of them, not just
> > *some*)
> > >
> > > This is technically correct: technically, new versions 'break' all
> users.
> > > The difference is, it does it in a predictable, safe way that users can
> > > handle, inspect, and safely upgrade.
> > >
> > > >Can someone please explain to me why (A) artificially graduating the
> > > current 1.x API to 2.x, and removing the deprecated endpoints from 1.x,
> > is
> > > better for API clients overall than (B) just removing deprecated
> > endpoints
> > > from 1.x? Existing 1.x clients would gain absolutely nothing from being
> > > forced to update to 2.x like this.
> > >
> > > >for zero gain in value
> > >
> > > Users would gain something. There is a very specific gain.
> > >
> > > The gain is that the breakage is predictable. People frequently upgrade
> > > minor versions automatically, either running commands without thinking,
> > or
> > > even automating the entire process.
> > >
> > > For TC, this applies to both CDN operators, and their end-users.
> > >
> > > When the minor versions contain breaking changes, that automatic
> upgrade
> > > then breaks. In this case, any scripts or tools they have using the
> > removed
> > > endpoints will break. If those scripts are important, this could bring
> > down
> > > their CDN, or break their Delivery Service, or otherwise cause an
> outage.
> > >
> > > Contrawise, users expect breaking changes in Major versions, and don't
> > > typically automatically upgrade them. Rather, they'll manually inspect
> > the
> > > update, do testing, and check the changelog for breaking changes.
> > >
> > > It's not a technical thing, it's a human thing. Humans automatically
> > > upgrade minor versions, and write scripts to do so, and dependency
> files
> > to
> > > allow it, because the software industry taught people that minor
> versions
> > > shouldn't break anything.
> > >
> > > If we start breaking minor versions, initially users will experience
> > > breakage. Not all users, and not all catastrophic. But some users will
> > > experience catastrophic outages. And it will be our fault as a project.
> > And
> > > after enough times, they'll fork TC, or find another product, or write
> > > their own; and stop participating in the community, or never start.
> > >
> > > Then, once they've learned that we can't be trusted not to break
> things,
> > > they'll experience the 2nd problem: where nothing can be safely
> upgraded,
> > > and the smallest change requires long and expensive testing to verify a
> > > single minor version or even patch upgrade, and still risk missing
> > > something and bringing their CDN down. To be honest, we're pretty much
> > > already here. It already costs our dev and ops teams unbelievable
> amounts
> > > of time and person-hours of testing for every minor upgrade. That time
> > far
> > > exceeds the development time to just rewrite the small endpoints we
> don't
> > > like anymore (and tangentially, also the time to write the Integration
> > and
> > > E2E tests TC desperately needs).
> > >
> > > ATC has so much potential. There aren't many OSS CDNs out there, and
> > > there's certainly the demand. We have the potential to become a huge
> > > project, which for us existing users, translates into potentially an
> > > enormous amount of free development. But we'll never get there if we
> keep
> > > breaking things.
> > >
> > > And for my particular organization (Comcast), we have a hard enough
> time
> > > upgrading. Our upgrades, especially Traffic Ops, are extremely painful,
> > and
> > > require enormous amounts of manual testing. Even then, we frequently
> have
> > > issues that require rollback, sometimes even an outage; even after
> those
> > > unbelievably amounts of developer and operator time testing. We
> > desperately
> > > need a system that is safer and easier to upgrade. Breaking changes are
> > > always going to require more testing; but we can at least make minor
> > > versions safe to upgrade.
> > >
> > > > I know I am changing my initial stance but I am +1 on rewriting all
> the
> > > Perl endpoints to Golang and not breaking the API contract we have in
> > > place. Most of the API routes to rewrite are rather simple to port and
> I
> > > think would be faster than getting consensus on actually deprecating it
> > > from the mailing list.
> > >
> > > I'm inclined to agree with this. The things I've seen marked
> "deprecated"
> > > seem to mostly be small, and look like they'd be pretty easy to
> rewrite.
> > > But I haven't seen the whole list. Is there a list somewhere, so the
> > > community can see it to better decide?
> > >
> > >
> > > On Tue, Nov 19, 2019 at 7:24 AM Hoppal, Michael <
> > Michael_Hoppal@comcast.com>
> > > wrote:
> > >
> > > > I would love to drive this to some type of consensus on rewrite
> > strategy
> > > > as we have a good amount of resources (and growing) working on the
> > rewrite
> > > > and I want to keep that momentum.
> > > >
> > > > Sounds like there is three options from the email thread (please add
> > any I
> > > > missed)
> > > >
> > > > Option 1 - Graduate current API from 1.x to 2.x - allowing us to
> leave
> > > > behind routes in 1.x that are dangerous or deprecated
> > > > Option 2 - Rewrite all endpoints from Perl to Golang in current
> > version as
> > > > we are doing now.
> > > > Option 3 - Only rewrite endpoints from Perl to Golang in current
> > version
> > > > that we agree are not dangerous or should deprecate
> > > >
> > > > I am +1 for option 2 (which I know is different from my original
> > stance).
> > > >
> > > > Which direction would others like to go?
> > > >
> > > > Thanks,
> > > >
> > > > Michael
> > > >
> > > >
> > > > I know I am changing my initial stance but I am +1 on rewriting all
> the
> > > > Perl endpoints to Golang and not breaking the API contract we have in
> > > > place. Most of the API routes to rewrite are rather simple to port
> and
> > I
> > > > think would be faster than getting consensus on actually deprecating
> it
> > > > from the mailing list.
> > > >
> > > > On 11/16/19, 3:31 PM, "Rawlin Peters" <ra...@apache.org> wrote:
> > > >
> > > > > Moving to 2.x means that it's my fault as a consumer of the API
> > if I
> > > > was still using the deprecated endpoint (and that change should be
> > part of
> > > > a larger upgrade plan). Simply removing the deprecated routes,
> > > > retroactively, means I can't rely on having a transition period to
> get
> > to
> > > > 2.0 outside of a large completely down upgrade window.
> > > >
> > > > There would be a transition period either way. Endpoints would be
> > > > deprecated in ATC 4.0 and removed in 5.0. The release notes for
> 4.0
> > > > would clearly list out the deprecated endpoints. If they have
> > > > replacements, there would be instructions on how to transition to
> > the
> > > > replacement. Deprecated endpoints would return alerts. After
> > upgrading
> > > > to 4.0, operators should be well aware that certain endpoints are
> > > > being removed in 5.0, and it's on them to start using new
> > endpoints,
> > > > change workflows, etc.
> > > >
> > > > By forcing operators to upgrade all of their clients to start
> using
> > > > API 2.x, it just adds to their burden of upgrading when that time
> > > > comes. Rather than surgically fixing the clients of the 2% of
> > > > deprecated API endpoints before upgrading to ATC 5.0, operators
> > would
> > > > have to fix every single client they have to use API 2.x before
> > > > upgrading to 5.0. Is that exercise really worth it? Most people
> > would
> > > > probably be blissfully unaware that the "POST
> > /api/1.1/capabilities"
> > > > endpoint disappeared (among other deprecated endpoints). Less
> > footguns
> > > > in the system, and they didn't even have to update their clients.
> > > >
> > > > Anyways, circling back to Michael's original question, it sounds
> > like
> > > > either option we choose, we at least have consensus on this
> > > > deprecation strategy:
> > > > - The route IS NOT rewritten from Perl to Golang and a
> deprecation
> > > > notice is added to the alert response in the Perl handler
> > > >
> > > > If we're content to force all API clients to update their clients
> > to
> > > > 2.x for zero gain in value, at least we won't be wasting work by
> > > > rewriting deprecated Perl endpoints to Go.
> > > >
> > > > - Rawlin
> > > >
> > > > On Fri, Nov 15, 2019 at 7:06 PM Gray, Jonathan
> > > > <Jo...@comcast.com> wrote:
> > > > >
> > > > > Moving to 2.x means that it's my fault as a consumer of the API
> > if I
> > > > was still using the deprecated endpoint (and that change should be
> > part of
> > > > a larger upgrade plan). Simply removing the deprecated routes,
> > > > retroactively, means I can't rely on having a transition period to
> get
> > to
> > > > 2.0 outside of a large completely down upgrade window. Depending on
> > how
> > > > the API is used, even in our own components such as TM, that could
> > prove
> > > > disastrous. For the older minor revs like 1.1/1.2, I could live with
> > > > dropping those as part of a major project release (including upgrade
> > > > instructions outlining what releases you must be on beforehand).
> > Currently
> > > > 1.3 I view as stable enough to code to or use a native client library
> > from,
> > > > and 1.4 is my risk/overhead to take for new stuff as it's still
> > evolving
> > > > and improving. Going to 1.5 or 2.0 moves that forward. The path of
> > least
> > > > surprise for humans is to expect big shifts on those major revs.
> > Removal
> > > > falls under that umbrella and gets as a consumer not to strictly
> focus
> > on
> > > > endpoint->endpoint moves, but rather workflow shifts and maybe now I
> > have
> > > > to use the API a different way than before. I believe getting this
> > right
> > > > is something that would greatly help us understand does X component
> at
> > > > version 1 work with Y component at version 2, because now you're
> > asking do
> > > > components X and Y both speak protocol A. With major releases we can
> > work
> > > > through that each time, but it's error prone and requires experts in
> > all
> > > > products to agree upon. If you do nothing but consume major
> releases,
> > and
> > > > then conduct your own impact evaluation having read the upgrade
> > > > instructions thoroughly, maybe you'd be able to smoothly transition.
> > > > >
> > > > > Jonathan G
> > > > >
> > > > >
> > > > > On 11/15/19, 5:24 PM, "Rawlin Peters" <ra...@apache.org>
> wrote:
> > > > >
> > > > > Can someone please explain to me why (A) artificially
> > graduating
> > > > the
> > > > > current 1.x API to 2.x, and removing the deprecated
> endpoints
> > > > from
> > > > > 1.x, is better for API clients overall than (B) just
> removing
> > > > > deprecated endpoints from 1.x? Existing 1.x clients would
> > gain
> > > > > absolutely nothing from being forced to update to 2.x like
> > this.
> > > > >
> > > > > The way I see the two choices:
> > > > > A) break every single API client out there
> > > > > B) break only clients of APIs that we've decided upon as a
> > > > community
> > > > > to be deprecated
> > > > >
> > > > > IMO, when we have to break things, we should try break the
> > least
> > > > > amount of clients possible. All I'm looking for is a
> valuable
> > > > reason
> > > > > to choose option A over option B, that isn't just "because
> > > > SemVer".
> > > > >
> > > > > - Rawlin
> > > > >
> > > > > On Wed, Nov 13, 2019 at 3:24 PM Schmidt, Andrew
> (Contractor)
> > > > > <An...@comcast.com> wrote:
> > > > > >
> > > > > > If we decide to get rid of the API versioning it makes
> > sense
> > > > to me that we would do that on a major version upgrade, probably to
> > the ATC
> > > > version. But I think it would be too early to do it on the 2.0 API
> > version.
> > > > I like this compromise +1. I was wondering why we were stuck on the
> > idea
> > > > that 2.0 had to be an API rewrite. That seems like it’s a long way
> > out. 2.0
> > > > = deprecations and fixes. 3.0 or 5.0 = api rewrite and maybe version
> > > > convergence.
> > > > > >
> > > > > > On 11/13/19, 11:52 AM, "ocket8888@gmail.com" <
> > > > ocket8888@gmail.com> wrote:
> > > > > >
> > > > > > Lemme lead with: making breaking changes on a major
> > > > version boundary
> > > > > > gets +1 from me - no reason the API needs to be fully
> > > > redesigned for
> > > > > > any particular number.
> > > > > >
> > > > > > But I just gotta bring this up again: if we want to
> be
> > > > able to
> > > > > > deprecate an endpoint for a major release and remove
> > it in
> > > > the next
> > > > > > without needing to deal with the messiness of API
> > > > versions, we could
> > > > > > always just get rid of API versions. Major ATC
> > revisions
> > > > should be
> > > > > > easier to come by with our new release strategy.
> > > > > >
> > > > > > On Wed, 2019-11-13 at 10:23 -0700, Rawlin Peters
> wrote:
> > > > > > > > It is a compromise - it's an inconvenience to
> > users,
> > > > forcing them
> > > > > > > > to fix
> > > > > > > scripts to upgrade. Which isn't good for user
> > experience
> > > > either; but
> > > > > > > it
> > > > > > > doesn't outright break people. It's a compromise
> I'm
> > > > willing to live
> > > > > > > with.
> > > > > > >
> > > > > > > It will literally outright break clients (*all* of
> > them,
> > > > not just
> > > > > > > *some*). We have been down this road before, so
> I'll
> > > > just repeat what
> > > > > > > I said when this exact conversation came up months
> > > > ago[1]:
> > > > > > >
> > > > > > > "So it basically boils down to:
> > > > > > > A) promote Go 1.x routes to 2.x and eventually
> break
> > ALL
> > > > users of the
> > > > > > > 1.x API when the time comes to remove 1.x
> > > > > > > B) deprecate and remove some 1.x Perl routes and
> > > > POSSIBLY break SOME
> > > > > > > users"
> > > > > > >
> > > > > > > Either way we slice and dice it with versioning,
> > > > removing an endpoint
> > > > > > > means that clients of that endpoint will break. By
> > > > graduating only
> > > > > > > the
> > > > > > > "good stuff" from 1.x to 2.x, and removing
> endpoints
> > > > from 1.x, we
> > > > > > > will
> > > > > > > unnecessarily break all 1.x clients when the time
> > comes
> > > > to remove the
> > > > > > > rest of 1.x. Instead, we can choose to break only
> > > > clients of 1.x APIs
> > > > > > > that we already deemed dangerous, unnecessary,
> > useless,
> > > > etc. enough
> > > > > > > to
> > > > > > > deprecate.
> > > > > > >
> > > > > > > Why break 100% of our API clients when we can
> choose
> > to
> > > > break only a
> > > > > > > small number of clients that happen to use
> > > > > > > dangerous/unnecessary/useless APIs? Why would we
> > break
> > > > our API
> > > > > > > clients
> > > > > > > unnecessarily for no added value? There is no sense
> > in
> > > > blindly
> > > > > > > adhering to Semantic Versioning just for the sake
> of
> > > > adhering to
> > > > > > > Semantic Versioning. Instead, use it as a
> guideline,
> > and
> > > > break the
> > > > > > > rules where it makes the most sense to do so.
> > > > > > >
> > > > > > > - Rawlin
> > > > > > >
> > > > > > > [1]
> > > > > > >
> > > >
> >
> https://urldefense.com/v3/__https://lists.apache.org/thread.html/15c6bfcdce0ac00a3a8612a96c3d668a8a5a16c6ca9401de9473d31c@*3Cdev.trafficcontrol.apache.org*3E__;JSU!rx_L75ITgOQ!V_b3E9iEYqNlZus18f-sVuofwuyxapCthg3LdO4NxyMCU39b7mdf4klP5KNR7XZ7d3Uc$
> > > > > > >
> > > > > > > On Wed, Nov 13, 2019 at 9:43 AM Robert O Butts <
> > > > rob@apache.org>
> > > > > > > wrote:
> > > > > > > > > But if it's really valuable to remove these
> > > > endpoints, maybe it's
> > > > > > > > appropriate to bump the major version and ask
> > > > everybody to update
> > > > > > > > their
> > > > > > > > clients?
> > > > > > > >
> > > > > > > > +1
> > > > > > > >
> > > > > > > > This sounds like the best compromise to me. It
> > > > addresses both sides
> > > > > > > > - it
> > > > > > > > lets us remove endpoints, and it also prevents
> > > > breaking users.
> > > > > > > >
> > > > > > > > It is a compromise - it's an inconvenience to
> > users,
> > > > forcing them
> > > > > > > > to fix
> > > > > > > > scripts to upgrade. Which isn't good for user
> > > > experience either;
> > > > > > > > but it
> > > > > > > > doesn't outright break people. It's a compromise
> > I'm
> > > > willing to
> > > > > > > > live with.
> > > > > > > >
> > > > > > > > This actually lets us do at least 3 things:
> > > > > > > > 1. Remove "deprecated" endpoints.
> > > > > > > > 2. Remove the unmaintained dangerously-large
> Cache
> > > > Config
> > > > > > > > endpoints.
> > > > > > > > 3. Lets us do Layered Profiles, changing Servers
> > and
> > > > DSes to have
> > > > > > > > arrays of
> > > > > > > > Profiles instead of a single Profile, without
> hacky
> > > > backward-
> > > > > > > > compatibility
> > > > > > > > attempts.
> > > > > > > >
> > > > > > > > To clarify, `/api/2.0` would be essentially the
> > > > current API, just
> > > > > > > > with the
> > > > > > > > above breaking changes (possibly others) which
> > require
> > > > a SemVer
> > > > > > > > major
> > > > > > > > version increase. And the big API
> rewrite/redesign
> > > > we've been
> > > > > > > > calling "2.0"
> > > > > > > > would become `/api/3.0` or some future version.
> > > > > > > >
> > > > > > > > It's not perfect, but it seems to address the
> most
> > > > concerns on all
> > > > > > > > sides.
> > > > > > > > Thoughts? Objections?
> > > > > > > >
> > > > > > > >
> > > > > > > > On Wed, Nov 13, 2019 at 8:45 AM Chris Lemmons <
> > > > alficles@gmail.com>
> > > > > > > > wrote:
> > > > > > > >
> > > > > > > > > For the endpoints, I agree in part and disagree
> > in
> > > > part.
> > > > > > > > >
> > > > > > > > > Agree for these two:
> > > > > > > > > - the endpoints are dangerous and pose a risk
> to
> > the
> > > > overall
> > > > > > > > > Traffic
> > > > > > > > > Control system
> > > > > > > > > - the task of rewriting the endpoints from Perl
> > to
> > > > Go represents
> > > > > > > > > an
> > > > > > > > > unreasonable amount of work for the project
> > > > > > > > >
> > > > > > > > > And I would add:
> > > > > > > > > - the endpoint does not serve its intended
> > purpose
> > > > and cannot
> > > > > > > > > reasonably
> > > > > > > > > be used
> > > > > > > > > - the endpoint has been made unsafe or unusable
> > by
> > > > new features
> > > > > > > > > added to TC
> > > > > > > > >
> > > > > > > > > Disagree for these two:
> > > > > > > > > - the endpoints have been obsoleted by other
> > > > endpoints
> > > > > > > > > - the endpoints don't seem to serve a true
> > purpose
> > > > or don't seem
> > > > > > > > > to be
> > > > > > > > > valuable to the project anymore
> > > > > > > > >
> > > > > > > > > For the most part, obsoleted endpoints can be
> > > > maintained with a
> > > > > > > > > shim
> > > > > > > > > into the new code. If they would require "an
> > > > unreasonable amount
> > > > > > > > > of
> > > > > > > > > work", it might make sense to remove them,
> then.
> > > > > > > > >
> > > > > > > > > Things that don't seem valuable usually fall
> into
> > > > another
> > > > > > > > > category
> > > > > > > > > like "cannot reasonably be used", but if the
> only
> > > > reason for
> > > > > > > > > deprecation is that the feature is going away,
> we
> > > > should really
> > > > > > > > > support it according to our promise.
> > > > > > > > >
> > > > > > > > > But if it's really valuable to remove these
> > > > endpoints, maybe it's
> > > > > > > > > appropriate to bump the major version and ask
> > > > everybody to update
> > > > > > > > > their clients?
> > > > > > > > >
> > > > > > > > > On Tue, Nov 12, 2019 at 5:27 PM Rawlin Peters <
> > > > rawlin@apache.org>
> > > > > > > > > wrote:
> > > > > > > > > > The real crux of the issue here is whether or
> > not
> > > > we agree as a
> > > > > > > > > > project that we can move forward with this
> plan
> > > > that I believe
> > > > > > > > > > to be a
> > > > > > > > > > good compromise that meets most concerns but
> > > > "breaks the
> > > > > > > > > > Semantic
> > > > > > > > > > Versioning promise":
> > > > > > > > > > 1) don't rewrite deprecated endpoints from
> > Perl to
> > > > Go
> > > > > > > > > > 2) in the Perl handler, add a deprecation
> > notice
> > > > to the
> > > > > > > > > > "alerts"
> > > > > > > > > > section of the response
> > > > > > > > > > 3) once a deprecated API has been deprecated
> > for
> > > > one major
> > > > > > > > > > release,
> > > > > > > > > > delete it from the API in the subsequent
> major
> > > > release (e.g. if
> > > > > > > > > > deprecation notices are added and released in
> > ATC
> > > > 4.x, we can
> > > > > > > > > > remove
> > > > > > > > > > the deprecated endpoint in ATC 5.x)
> > > > > > > > > >
> > > > > > > > > > Rob, from your latest proposal to remove the
> > ATS
> > > > config API
> > > > > > > > > > endpoints
> > > > > > > > > > from 1.x, I thought we were past this issue
> > > > already and could
> > > > > > > > > > finally
> > > > > > > > > > agree to remove deprecated endpoints from 1.x
> > > > (following steps
> > > > > > > > > > 1-3
> > > > > > > > > > above) without requiring an all-new 2.x API.
> > Are
> > > > we not
> > > > > > > > > > actually past
> > > > > > > > > > that? You came up with valid reasons for
> > applying
> > > > these steps
> > > > > > > > > > to the
> > > > > > > > > > ATS config API endpoints, and I think it's
> > fair to
> > > > say that we
> > > > > > > > > > can
> > > > > > > > > > come up with valid reasons for applying these
> > > > steps to other
> > > > > > > > > > deprecated API endpoints as well.
> > > > > > > > > >
> > > > > > > > > > It seems like the real problem is actually
> > > > agreeing on which
> > > > > > > > > > _specific_ endpoints we should be able to
> > follow
> > > > steps 1-3 for.
> > > > > > > > > > IMO I
> > > > > > > > > > think it should apply to any 1.x endpoints
> > that we
> > > > have valid
> > > > > > > > > > reasons
> > > > > > > > > > for not carrying forward to Go, including but
> > not
> > > > limited to:
> > > > > > > > > > - the endpoints are dangerous and pose a risk
> > to
> > > > the overall
> > > > > > > > > > Traffic
> > > > > > > > > > Control system
> > > > > > > > > > - the endpoints have been obsoleted by other
> > > > endpoints
> > > > > > > > > > - the endpoints don't seem to serve a true
> > purpose
> > > > or don't
> > > > > > > > > > seem to be
> > > > > > > > > > valuable to the project anymore
> > > > > > > > > > - the task of rewriting the endpoints from
> > Perl to
> > > > Go
> > > > > > > > > > represents an
> > > > > > > > > > unreasonable amount of work for the project
> > > > > > > > > >
> > > > > > > > > > Obviously most of these reasons are
> > subjective, so
> > > > we'd have to
> > > > > > > > > > come
> > > > > > > > > > to some level of consensus on deprecating
> > > > particular endpoints
> > > > > > > > > > first,
> > > > > > > > > > but we should be able to follow steps 1-3 for
> > > > whatever
> > > > > > > > > > endpoints we
> > > > > > > > > > decide.
> > > > > > > > > >
> > > > > > > > > > Making breaking changes shouldn't be taken
> > > > lightly, but we need
> > > > > > > > > > a path
> > > > > > > > > > forward that allows some calculated, breaking
> > > > changes where
> > > > > > > > > > necessary.
> > > > > > > > > > Otherwise, we are unnecessarily holding the
> > > > project back.
> > > > > > > > > >
> > > > > > > > > > - Rawlin
> > > > > > > > > >
> > > > > > > > > > On Tue, Nov 12, 2019 at 11:44 AM Jeremy
> > Mitchell <
> > > > > > > > > > mitchell852@gmail.com>
> > > > > > > > > wrote:
> > > > > > > > > > > Yeah, technically a "rewrite" of 1.x means
> > > > shifting the
> > > > > > > > > > > implementation
> > > > > > > > > from
> > > > > > > > > > > Perl to Go for ALL of 1.x. If you leave a
> > subset
> > > > of 1.x in
> > > > > > > > > > > Perl, then
> > > > > > > > > you
> > > > > > > > > > > have to call it a "partial rewrite".
> > > > > > > > > > >
> > > > > > > > > > > So, assuming we want to do a full rewrite
> (no
> > > > more perl for
> > > > > > > > > > > the api
> > > > > > > > > > > whatsoever), I think that means every 1.x
> api
> > > > endpoint (good
> > > > > > > > > > > or bad)
> > > > > > > > > needs
> > > > > > > > > > > to be rewritten to Go so we can remove the
> > > > dependency on
> > > > > > > > > > > perl.
> > > > > > > > > > >
> > > > > > > > > > > However, I still like marking many as
> > > > "deprecated" so we
> > > > > > > > > > > train users
> > > > > > > > > to use
> > > > > > > > > > > "better" endpoints that we envision will
> > exist
> > > > in 2.x and
> > > > > > > > > > > beyond.
> > > > > > > > > > >
> > > > > > > > > > > Jeremy
> > > > > > > > > > >
> > > > > > > > > > > On Tue, Nov 12, 2019 at 11:22 AM Hoppal,
> > Michael
> > > > <
> > > > > > > > > Michael_Hoppal@comcast.com>
> > > > > > > > > > > wrote:
> > > > > > > > > > >
> > > > > > > > > > > > We do not have a lot of deprecated routes
> > as
> > > > of now as we
> > > > > > > > > > > > focused on
> > > > > > > > > > > > routes that are more used first.
> > > > > > > > > > > >
> > > > > > > > > > > > I know on a lot of the remaining rewrites
> > > > there has been
> > > > > > > > > > > > discussion
> > > > > > > > > on the
> > > > > > > > > > > > matching Github issues on potential
> > > > deprecation (I cannot
> > > > > > > > > > > > think of
> > > > > > > > > the
> > > > > > > > > > > > exact count but enough to warrant this
> > email
> > > > __)
> > > > > > > > > > > >
> > > > > > > > > > > > It is key to note that I am not arguing
> for
> > > > removal of
> > > > > > > > > > > > endpoints
> > > > > > > > > within
> > > > > > > > > > > > the current API version but if there is
> an
> > > > endpoint we
> > > > > > > > > > > > agree should
> > > > > > > > > be
> > > > > > > > > > > > deprecated going forward to put a message
> > > > within the
> > > > > > > > > > > > response and on
> > > > > > > > > the
> > > > > > > > > > > > docs.
> > > > > > > > > > > >
> > > > > > > > > > > > And thanks for the input sounds like you
> > are
> > > > +1 on
> > > > > > > > > > > > rewriting all
> > > > > > > > > routes to
> > > > > > > > > > > > Golang as Brennan is.
> > > > > > > > > > > >
> > > > > > > > > > > > On 11/12/19, 9:06 AM, "Robert O Butts" <
> > > > rob@apache.org>
> > > > > > > > > > > > wrote:
> > > > > > > > > > > >
> > > > > > > > > > > > Whether we rewrite a route in Go is
> an
> > > > implementation
> > > > > > > > > > > > detail. To
> > > > > > > > > the
> > > > > > > > > > > > interface, to users, it doesn't
> matter
> > > > whether a route
> > > > > > > > > > > > is
> > > > > > > > > rewritten or
> > > > > > > > > > > > not.
> > > > > > > > > > > >
> > > > > > > > > > > > But our API follows Semantic
> > Versioning,
> > > > in order to
> > > > > > > > > > > > not break
> > > > > > > > > users.
> > > > > > > > > > > > We
> > > > > > > > > > > > can't just remove endpoints that some
> > of
> > > > us don't use,
> > > > > > > > > > > > and
> > > > > > > > > assume other
> > > > > > > > > > > > people don't, maybe people who never
> > speak
> > > > up on the
> > > > > > > > > > > > mailing
> > > > > > > > > list.
> > > > > > > > > > > > We'll
> > > > > > > > > > > > never gain a big userbase if we keep
> > > > breaking users.
> > > > > > > > > > > >
> > > > > > > > > > > > Per the _project_ SemVer, once we
> have
> > API
> > > > 2.0, we can
> > > > > > > > > > > > deprecate
> > > > > > > > > API
> > > > > > > > > > > > 1.x,
> > > > > > > > > > > > and in the next major _project_
> > release,
> > > > remove API
> > > > > > > > > > > > 1.x.
> > > > > > > > > Irrespective
> > > > > > > > > > > > of
> > > > > > > > > > > > Perl or Go.
> > > > > > > > > > > >
> > > > > > > > > > > > My big concern is, API 2.0 is a big
> > > > project. How long
> > > > > > > > > > > > has the
> > > > > > > > > rewrite
> > > > > > > > > > > > to Go
> > > > > > > > > > > > taken? Do we really believe designing
> > and
> > > > implementing
> > > > > > > > > > > > a
> > > > > > > > > completely
> > > > > > > > > > > > new API
> > > > > > > > > > > > will be any less time?
> > > > > > > > > > > >
> > > > > > > > > > > > I don't want killing Perl to have to
> > wait
> > > > on that.
> > > > > > > > > > > >
> > > > > > > > > > > > I know it feels like a waste to
> rewrite
> > > > routes that you
> > > > > > > > > > > > don't
> > > > > > > > > use, and
> > > > > > > > > > > > probably few people do. But that's
> the
> > > > cost of a stable
> > > > > > > > > > > > project.
> > > > > > > > > How
> > > > > > > > > > > > many
> > > > > > > > > > > > "deprecated" routes are there? If it
> > comes
> > > > down to
> > > > > > > > > > > > taking the
> > > > > > > > > > > > development
> > > > > > > > > > > > time to rewrite them so we can kill
> > Perl
> > > > faster, or
> > > > > > > > > > > > leaving Perl
> > > > > > > > > > > > around, I
> > > > > > > > > > > > vote we just do the work and kill
> Perl.
> > > > > > > > > > > >
> > > > > > > > > > > > >If we don't rewrite them, then Perl
> > will
> > > > last until
> > > > > > > > > > > > API 2.0 has
> > > > > > > > > been
> > > > > > > > > > > > designed, released and then *another
> > full
> > > > major release
> > > > > > > > > > > > cycle*.
> > > > > > > > > That's
> > > > > > > > > > > > way
> > > > > > > > > > > > too long to have two codebases for
> the
> > > > same component,
> > > > > > > > > > > > IMO,
> > > > > > > > > especially
> > > > > > > > > > > > since the rewrite is already 50%
> > complete.
> > > > > > > > > > > >
> > > > > > > > > > > > +1
> > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > > > On Tue, Nov 12, 2019 at 8:54 AM ocket
> > 8888
> > > > <
> > > > > > > > > > > > ocket8888@gmail.com>
> > > > > > > > > > > > wrote:
> > > > > > > > > > > >
> > > > > > > > > > > > > I vote that by and large we DO
> > rewrite
> > > > them, with
> > > > > > > > > > > > exceptions
> > > > > > > > > for
> > > > > > > > > > > > routes
> > > > > > > > > > > > > that just plain don't work, even in
> > > > Perl. Those are
> > > > > > > > > > > > few,
> > > > > > > > > though.
> > > > > > > > > > > > >
> > > > > > > > > > > > > If we don't rewrite them, then Perl
> > will
> > > > last until
> > > > > > > > > > > > API 2.0
> > > > > > > > > has been
> > > > > > > > > > > > > designed, released and then
> *another
> > > > full major
> > > > > > > > > > > > release cycle*.
> > > > > > > > > > > > That's way
> > > > > > > > > > > > > too long to have two codebases for
> > the
> > > > same
> > > > > > > > > > > > component, IMO,
> > > > > > > > > > > > especially
> > > > > > > > > > > > > since the rewrite is already 50%
> > > > complete.
> > > > > > > > > > > > >
> > > > > > > > > > > > > On Tue, Nov 12, 2019 at 8:43 AM
> > Hoppal,
> > > > Michael <
> > > > > > > > > > > > > Michael_Hoppal@comcast.com>
> > > > > > > > > > > > > wrote:
> > > > > > > > > > > > >
> > > > > > > > > > > > > > As the Traffic Ops API is being
> > > > rewritten from Perl
> > > > > > > > > > > > to Golang
> > > > > > > > > > > > there has
> > > > > > > > > > > > > > been several routes that have
> been
> > > > deprecated and
> > > > > > > > > > > > probably
> > > > > > > > > more to
> > > > > > > > > > > > come.
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > In the deprecation efforts I have
> > seen
> > > > two
> > > > > > > > > > > > strategies:
> > > > > > > > > > > > > >
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > * The route IS NOT rewritten
> > from
> > > > Perl to
> > > > > > > > > > > > Golang and a
> > > > > > > > > > > > deprecation
> > > > > > > > > > > > > > notice is added to the alert
> > response
> > > > in the Perl
> > > > > > > > > > > > handler
> > > > > > > > > > > > > > * The route IS rewritten from
> > Perl
> > > > to Golang
> > > > > > > > > > > > and a
> > > > > > > > > deprecation
> > > > > > > > > > > > notice
> > > > > > > > > > > > > > is added to the alert response in
> > the
> > > > Golang
> > > > > > > > > > > > handler
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > I think we should have
> consistency
> > in
> > > > our approach
> > > > > > > > > > > > and
> > > > > > > > > wanted to
> > > > > > > > > > > > get
> > > > > > > > > > > > > > people’s thoughts.
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > I would vote that we do not
> > rewrite a
> > > > deprecated
> > > > > > > > > > > > route from
> > > > > > > > > Perl to
> > > > > > > > > > > > > Golang.
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > Thanks,
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > Michael
> > > > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > >
> > > > > >
> > > > >
> > > > >
> > > >
> > > >
> > > >
> >
>
Re: [EXTERNAL] Re: Traffic Ops Route Deprecation Strategy
Posted by ocket 8888 <oc...@gmail.com>.
I'm generally +1 for option 2
regarding those specific endpoints:
- /riak/stats if this is easy and doesn't expose anything dangerous, we
should do it IMO.
- /cachegroup_fallbacks should be pretty trivial, so I still say do it
- /capabilities since these aren't actually used for anything, I think it's
pretty safe (and easy) to rewrite. Plus there's already a PR open for
exactly that
- /api_capabilities dropping this one could be problematic, just because
people may depend on it to tell them what endpoints exist. To be clear, I
think it ought to be deprecated, but unfortunately I just don't think it's
terribly unlikely that this is built into scripts and tools. The actual
code for the rewrite should be easy too, although maintaining it is a pain.
- ATS Config Files Stuff Aren't these all already in Go anyway? Whether or
not they should be removed from the API aside, I don't think anything
actually needs to hit Perl anymore.
- /cdn/{{name}}/configs/routing CRConfigs are large and complex, so if this
never worked anyway maybe we should just add a Go handler for it that
returns a '410 Gone' response (or '501 Not Implemented' is the other one
I've heard suggested) and leave the actual logic behind in the Perl
- /hwinfo/dtdata if this calls into Mojolicious rendering, returns a
non-json response, and/or contains information that in either content or
encoding is meaningless to any sane client then it should also return a
410/501 and get left behind in Perl.
On Tue, Nov 19, 2019 at 11:08 AM Rawlin Peters <ra...@apache.org> wrote:
> While I appreciate and mostly agree with your essay on the importance
> of not breaking users (or rather, breaking them in predictable ways),
> I for one would really like to meet at some middle ground with option
> 2. I'm not asking to remove 50 high-usage endpoints from 1.x. IIRC
> I've only specifically proposed "leaving behind" a few specific Perl
> endpoints that I believe serve no purpose, have no valid reason to be
> used, and/or are dangerous to the system:
>
> - /riak/stats -- informational, was used for debugging the initial
> Riak integration. This one might be pretty easy to rewrite since it
> just calls a corresponding riak API and returns the result via TO, so
> I could be on board with rewriting it if necessary.
> - /cachegroup_fallbacks -- these endpoints were released in 3.0 before
> we managed to augment the existing /cachegroups API to handle
> fallbacks, so I could get on board with rewriting these if we really
> think it's important.
> - /capabilities (all methods except for GET) -- capabilities are
> built-in, should not be changed at runtime
> - /api_capabilities (all methods except for GET) -- same reasons as
> /capabilities
> - /cdns/{name}/configs/routing -- this was a half-baked attempt at
> redoing CrConfig.json -- cannot possibly be of use to anyone in its
> current state
> - the ATS config generation endpoints -- Rob actually proposed to
> remove these from 1.x due to danger of being out of sync w/ cache-side
> config generator
> - /hwinfo/dtdata -- AFAIK the "dtdata" endpoints were only
> used/required by the old Perl Mojolicious UI, no valid reason to be
> using this as an external API client. In fact, I'm not even sure _how_
> this endpoint can be used outside of the Mojolicious framework/UI. It
> calls the Mojolicious `render` function, which usually means rendering
> an embedded Perl HTML template given some data. This is one I would
> consider completely unreasonable to rewrite to Go.
>
> Can we get on board with leaving at least these 1.x endpoints behind
> in the Perl? Doing a cursory skim over the remaining 1.x endpoints to
> rewrite, it _seems_ like everything else would fall into the "simpler
> to just rewrite rather than gain consensus to remove" category.
>
> - Rawlin
>
> On Tue, Nov 19, 2019 at 9:19 AM Robert O Butts <ro...@apache.org> wrote:
> >
> > >Option 1 - Graduate current API from 1.x to 2.x - allowing us to leave
> > behind routes in 1.x that are dangerous or deprecated
> > >Option 2 - Rewrite all endpoints from Perl to Golang in current version
> as
> > we are doing now.
> >
> > +1 on either Option 1 or Option 2, without a strong preference. Option 1
> > has advantages of letting us also safely remove the Cache Config
> endpoints
> > and implement Layered Profiles. But Option 2 also has the advantage of
> not
> > requiring users to upgrade.
> >
> > > Option 3 - Only rewrite endpoints from Perl to Golang in current
> version
> > that we agree are not dangerous or should deprecate
> >
> > -1. As before, it breaks users unpredictably.
> >
> > >It will literally outright break clients (*all* of them, not just
> *some*)
> >
> > This is technically correct: technically, new versions 'break' all users.
> > The difference is, it does it in a predictable, safe way that users can
> > handle, inspect, and safely upgrade.
> >
> > >Can someone please explain to me why (A) artificially graduating the
> > current 1.x API to 2.x, and removing the deprecated endpoints from 1.x,
> is
> > better for API clients overall than (B) just removing deprecated
> endpoints
> > from 1.x? Existing 1.x clients would gain absolutely nothing from being
> > forced to update to 2.x like this.
> >
> > >for zero gain in value
> >
> > Users would gain something. There is a very specific gain.
> >
> > The gain is that the breakage is predictable. People frequently upgrade
> > minor versions automatically, either running commands without thinking,
> or
> > even automating the entire process.
> >
> > For TC, this applies to both CDN operators, and their end-users.
> >
> > When the minor versions contain breaking changes, that automatic upgrade
> > then breaks. In this case, any scripts or tools they have using the
> removed
> > endpoints will break. If those scripts are important, this could bring
> down
> > their CDN, or break their Delivery Service, or otherwise cause an outage.
> >
> > Contrawise, users expect breaking changes in Major versions, and don't
> > typically automatically upgrade them. Rather, they'll manually inspect
> the
> > update, do testing, and check the changelog for breaking changes.
> >
> > It's not a technical thing, it's a human thing. Humans automatically
> > upgrade minor versions, and write scripts to do so, and dependency files
> to
> > allow it, because the software industry taught people that minor versions
> > shouldn't break anything.
> >
> > If we start breaking minor versions, initially users will experience
> > breakage. Not all users, and not all catastrophic. But some users will
> > experience catastrophic outages. And it will be our fault as a project.
> And
> > after enough times, they'll fork TC, or find another product, or write
> > their own; and stop participating in the community, or never start.
> >
> > Then, once they've learned that we can't be trusted not to break things,
> > they'll experience the 2nd problem: where nothing can be safely upgraded,
> > and the smallest change requires long and expensive testing to verify a
> > single minor version or even patch upgrade, and still risk missing
> > something and bringing their CDN down. To be honest, we're pretty much
> > already here. It already costs our dev and ops teams unbelievable amounts
> > of time and person-hours of testing for every minor upgrade. That time
> far
> > exceeds the development time to just rewrite the small endpoints we don't
> > like anymore (and tangentially, also the time to write the Integration
> and
> > E2E tests TC desperately needs).
> >
> > ATC has so much potential. There aren't many OSS CDNs out there, and
> > there's certainly the demand. We have the potential to become a huge
> > project, which for us existing users, translates into potentially an
> > enormous amount of free development. But we'll never get there if we keep
> > breaking things.
> >
> > And for my particular organization (Comcast), we have a hard enough time
> > upgrading. Our upgrades, especially Traffic Ops, are extremely painful,
> and
> > require enormous amounts of manual testing. Even then, we frequently have
> > issues that require rollback, sometimes even an outage; even after those
> > unbelievably amounts of developer and operator time testing. We
> desperately
> > need a system that is safer and easier to upgrade. Breaking changes are
> > always going to require more testing; but we can at least make minor
> > versions safe to upgrade.
> >
> > > I know I am changing my initial stance but I am +1 on rewriting all the
> > Perl endpoints to Golang and not breaking the API contract we have in
> > place. Most of the API routes to rewrite are rather simple to port and I
> > think would be faster than getting consensus on actually deprecating it
> > from the mailing list.
> >
> > I'm inclined to agree with this. The things I've seen marked "deprecated"
> > seem to mostly be small, and look like they'd be pretty easy to rewrite.
> > But I haven't seen the whole list. Is there a list somewhere, so the
> > community can see it to better decide?
> >
> >
> > On Tue, Nov 19, 2019 at 7:24 AM Hoppal, Michael <
> Michael_Hoppal@comcast.com>
> > wrote:
> >
> > > I would love to drive this to some type of consensus on rewrite
> strategy
> > > as we have a good amount of resources (and growing) working on the
> rewrite
> > > and I want to keep that momentum.
> > >
> > > Sounds like there is three options from the email thread (please add
> any I
> > > missed)
> > >
> > > Option 1 - Graduate current API from 1.x to 2.x - allowing us to leave
> > > behind routes in 1.x that are dangerous or deprecated
> > > Option 2 - Rewrite all endpoints from Perl to Golang in current
> version as
> > > we are doing now.
> > > Option 3 - Only rewrite endpoints from Perl to Golang in current
> version
> > > that we agree are not dangerous or should deprecate
> > >
> > > I am +1 for option 2 (which I know is different from my original
> stance).
> > >
> > > Which direction would others like to go?
> > >
> > > Thanks,
> > >
> > > Michael
> > >
> > >
> > > I know I am changing my initial stance but I am +1 on rewriting all the
> > > Perl endpoints to Golang and not breaking the API contract we have in
> > > place. Most of the API routes to rewrite are rather simple to port and
> I
> > > think would be faster than getting consensus on actually deprecating it
> > > from the mailing list.
> > >
> > > On 11/16/19, 3:31 PM, "Rawlin Peters" <ra...@apache.org> wrote:
> > >
> > > > Moving to 2.x means that it's my fault as a consumer of the API
> if I
> > > was still using the deprecated endpoint (and that change should be
> part of
> > > a larger upgrade plan). Simply removing the deprecated routes,
> > > retroactively, means I can't rely on having a transition period to get
> to
> > > 2.0 outside of a large completely down upgrade window.
> > >
> > > There would be a transition period either way. Endpoints would be
> > > deprecated in ATC 4.0 and removed in 5.0. The release notes for 4.0
> > > would clearly list out the deprecated endpoints. If they have
> > > replacements, there would be instructions on how to transition to
> the
> > > replacement. Deprecated endpoints would return alerts. After
> upgrading
> > > to 4.0, operators should be well aware that certain endpoints are
> > > being removed in 5.0, and it's on them to start using new
> endpoints,
> > > change workflows, etc.
> > >
> > > By forcing operators to upgrade all of their clients to start using
> > > API 2.x, it just adds to their burden of upgrading when that time
> > > comes. Rather than surgically fixing the clients of the 2% of
> > > deprecated API endpoints before upgrading to ATC 5.0, operators
> would
> > > have to fix every single client they have to use API 2.x before
> > > upgrading to 5.0. Is that exercise really worth it? Most people
> would
> > > probably be blissfully unaware that the "POST
> /api/1.1/capabilities"
> > > endpoint disappeared (among other deprecated endpoints). Less
> footguns
> > > in the system, and they didn't even have to update their clients.
> > >
> > > Anyways, circling back to Michael's original question, it sounds
> like
> > > either option we choose, we at least have consensus on this
> > > deprecation strategy:
> > > - The route IS NOT rewritten from Perl to Golang and a deprecation
> > > notice is added to the alert response in the Perl handler
> > >
> > > If we're content to force all API clients to update their clients
> to
> > > 2.x for zero gain in value, at least we won't be wasting work by
> > > rewriting deprecated Perl endpoints to Go.
> > >
> > > - Rawlin
> > >
> > > On Fri, Nov 15, 2019 at 7:06 PM Gray, Jonathan
> > > <Jo...@comcast.com> wrote:
> > > >
> > > > Moving to 2.x means that it's my fault as a consumer of the API
> if I
> > > was still using the deprecated endpoint (and that change should be
> part of
> > > a larger upgrade plan). Simply removing the deprecated routes,
> > > retroactively, means I can't rely on having a transition period to get
> to
> > > 2.0 outside of a large completely down upgrade window. Depending on
> how
> > > the API is used, even in our own components such as TM, that could
> prove
> > > disastrous. For the older minor revs like 1.1/1.2, I could live with
> > > dropping those as part of a major project release (including upgrade
> > > instructions outlining what releases you must be on beforehand).
> Currently
> > > 1.3 I view as stable enough to code to or use a native client library
> from,
> > > and 1.4 is my risk/overhead to take for new stuff as it's still
> evolving
> > > and improving. Going to 1.5 or 2.0 moves that forward. The path of
> least
> > > surprise for humans is to expect big shifts on those major revs.
> Removal
> > > falls under that umbrella and gets as a consumer not to strictly focus
> on
> > > endpoint->endpoint moves, but rather workflow shifts and maybe now I
> have
> > > to use the API a different way than before. I believe getting this
> right
> > > is something that would greatly help us understand does X component at
> > > version 1 work with Y component at version 2, because now you're
> asking do
> > > components X and Y both speak protocol A. With major releases we can
> work
> > > through that each time, but it's error prone and requires experts in
> all
> > > products to agree upon. If you do nothing but consume major releases,
> and
> > > then conduct your own impact evaluation having read the upgrade
> > > instructions thoroughly, maybe you'd be able to smoothly transition.
> > > >
> > > > Jonathan G
> > > >
> > > >
> > > > On 11/15/19, 5:24 PM, "Rawlin Peters" <ra...@apache.org> wrote:
> > > >
> > > > Can someone please explain to me why (A) artificially
> graduating
> > > the
> > > > current 1.x API to 2.x, and removing the deprecated endpoints
> > > from
> > > > 1.x, is better for API clients overall than (B) just removing
> > > > deprecated endpoints from 1.x? Existing 1.x clients would
> gain
> > > > absolutely nothing from being forced to update to 2.x like
> this.
> > > >
> > > > The way I see the two choices:
> > > > A) break every single API client out there
> > > > B) break only clients of APIs that we've decided upon as a
> > > community
> > > > to be deprecated
> > > >
> > > > IMO, when we have to break things, we should try break the
> least
> > > > amount of clients possible. All I'm looking for is a valuable
> > > reason
> > > > to choose option A over option B, that isn't just "because
> > > SemVer".
> > > >
> > > > - Rawlin
> > > >
> > > > On Wed, Nov 13, 2019 at 3:24 PM Schmidt, Andrew (Contractor)
> > > > <An...@comcast.com> wrote:
> > > > >
> > > > > If we decide to get rid of the API versioning it makes
> sense
> > > to me that we would do that on a major version upgrade, probably to
> the ATC
> > > version. But I think it would be too early to do it on the 2.0 API
> version.
> > > I like this compromise +1. I was wondering why we were stuck on the
> idea
> > > that 2.0 had to be an API rewrite. That seems like it’s a long way
> out. 2.0
> > > = deprecations and fixes. 3.0 or 5.0 = api rewrite and maybe version
> > > convergence.
> > > > >
> > > > > On 11/13/19, 11:52 AM, "ocket8888@gmail.com" <
> > > ocket8888@gmail.com> wrote:
> > > > >
> > > > > Lemme lead with: making breaking changes on a major
> > > version boundary
> > > > > gets +1 from me - no reason the API needs to be fully
> > > redesigned for
> > > > > any particular number.
> > > > >
> > > > > But I just gotta bring this up again: if we want to be
> > > able to
> > > > > deprecate an endpoint for a major release and remove
> it in
> > > the next
> > > > > without needing to deal with the messiness of API
> > > versions, we could
> > > > > always just get rid of API versions. Major ATC
> revisions
> > > should be
> > > > > easier to come by with our new release strategy.
> > > > >
> > > > > On Wed, 2019-11-13 at 10:23 -0700, Rawlin Peters wrote:
> > > > > > > It is a compromise - it's an inconvenience to
> users,
> > > forcing them
> > > > > > > to fix
> > > > > > scripts to upgrade. Which isn't good for user
> experience
> > > either; but
> > > > > > it
> > > > > > doesn't outright break people. It's a compromise I'm
> > > willing to live
> > > > > > with.
> > > > > >
> > > > > > It will literally outright break clients (*all* of
> them,
> > > not just
> > > > > > *some*). We have been down this road before, so I'll
> > > just repeat what
> > > > > > I said when this exact conversation came up months
> > > ago[1]:
> > > > > >
> > > > > > "So it basically boils down to:
> > > > > > A) promote Go 1.x routes to 2.x and eventually break
> ALL
> > > users of the
> > > > > > 1.x API when the time comes to remove 1.x
> > > > > > B) deprecate and remove some 1.x Perl routes and
> > > POSSIBLY break SOME
> > > > > > users"
> > > > > >
> > > > > > Either way we slice and dice it with versioning,
> > > removing an endpoint
> > > > > > means that clients of that endpoint will break. By
> > > graduating only
> > > > > > the
> > > > > > "good stuff" from 1.x to 2.x, and removing endpoints
> > > from 1.x, we
> > > > > > will
> > > > > > unnecessarily break all 1.x clients when the time
> comes
> > > to remove the
> > > > > > rest of 1.x. Instead, we can choose to break only
> > > clients of 1.x APIs
> > > > > > that we already deemed dangerous, unnecessary,
> useless,
> > > etc. enough
> > > > > > to
> > > > > > deprecate.
> > > > > >
> > > > > > Why break 100% of our API clients when we can choose
> to
> > > break only a
> > > > > > small number of clients that happen to use
> > > > > > dangerous/unnecessary/useless APIs? Why would we
> break
> > > our API
> > > > > > clients
> > > > > > unnecessarily for no added value? There is no sense
> in
> > > blindly
> > > > > > adhering to Semantic Versioning just for the sake of
> > > adhering to
> > > > > > Semantic Versioning. Instead, use it as a guideline,
> and
> > > break the
> > > > > > rules where it makes the most sense to do so.
> > > > > >
> > > > > > - Rawlin
> > > > > >
> > > > > > [1]
> > > > > >
> > >
> https://urldefense.com/v3/__https://lists.apache.org/thread.html/15c6bfcdce0ac00a3a8612a96c3d668a8a5a16c6ca9401de9473d31c@*3Cdev.trafficcontrol.apache.org*3E__;JSU!rx_L75ITgOQ!V_b3E9iEYqNlZus18f-sVuofwuyxapCthg3LdO4NxyMCU39b7mdf4klP5KNR7XZ7d3Uc$
> > > > > >
> > > > > > On Wed, Nov 13, 2019 at 9:43 AM Robert O Butts <
> > > rob@apache.org>
> > > > > > wrote:
> > > > > > > > But if it's really valuable to remove these
> > > endpoints, maybe it's
> > > > > > > appropriate to bump the major version and ask
> > > everybody to update
> > > > > > > their
> > > > > > > clients?
> > > > > > >
> > > > > > > +1
> > > > > > >
> > > > > > > This sounds like the best compromise to me. It
> > > addresses both sides
> > > > > > > - it
> > > > > > > lets us remove endpoints, and it also prevents
> > > breaking users.
> > > > > > >
> > > > > > > It is a compromise - it's an inconvenience to
> users,
> > > forcing them
> > > > > > > to fix
> > > > > > > scripts to upgrade. Which isn't good for user
> > > experience either;
> > > > > > > but it
> > > > > > > doesn't outright break people. It's a compromise
> I'm
> > > willing to
> > > > > > > live with.
> > > > > > >
> > > > > > > This actually lets us do at least 3 things:
> > > > > > > 1. Remove "deprecated" endpoints.
> > > > > > > 2. Remove the unmaintained dangerously-large Cache
> > > Config
> > > > > > > endpoints.
> > > > > > > 3. Lets us do Layered Profiles, changing Servers
> and
> > > DSes to have
> > > > > > > arrays of
> > > > > > > Profiles instead of a single Profile, without hacky
> > > backward-
> > > > > > > compatibility
> > > > > > > attempts.
> > > > > > >
> > > > > > > To clarify, `/api/2.0` would be essentially the
> > > current API, just
> > > > > > > with the
> > > > > > > above breaking changes (possibly others) which
> require
> > > a SemVer
> > > > > > > major
> > > > > > > version increase. And the big API rewrite/redesign
> > > we've been
> > > > > > > calling "2.0"
> > > > > > > would become `/api/3.0` or some future version.
> > > > > > >
> > > > > > > It's not perfect, but it seems to address the most
> > > concerns on all
> > > > > > > sides.
> > > > > > > Thoughts? Objections?
> > > > > > >
> > > > > > >
> > > > > > > On Wed, Nov 13, 2019 at 8:45 AM Chris Lemmons <
> > > alficles@gmail.com>
> > > > > > > wrote:
> > > > > > >
> > > > > > > > For the endpoints, I agree in part and disagree
> in
> > > part.
> > > > > > > >
> > > > > > > > Agree for these two:
> > > > > > > > - the endpoints are dangerous and pose a risk to
> the
> > > overall
> > > > > > > > Traffic
> > > > > > > > Control system
> > > > > > > > - the task of rewriting the endpoints from Perl
> to
> > > Go represents
> > > > > > > > an
> > > > > > > > unreasonable amount of work for the project
> > > > > > > >
> > > > > > > > And I would add:
> > > > > > > > - the endpoint does not serve its intended
> purpose
> > > and cannot
> > > > > > > > reasonably
> > > > > > > > be used
> > > > > > > > - the endpoint has been made unsafe or unusable
> by
> > > new features
> > > > > > > > added to TC
> > > > > > > >
> > > > > > > > Disagree for these two:
> > > > > > > > - the endpoints have been obsoleted by other
> > > endpoints
> > > > > > > > - the endpoints don't seem to serve a true
> purpose
> > > or don't seem
> > > > > > > > to be
> > > > > > > > valuable to the project anymore
> > > > > > > >
> > > > > > > > For the most part, obsoleted endpoints can be
> > > maintained with a
> > > > > > > > shim
> > > > > > > > into the new code. If they would require "an
> > > unreasonable amount
> > > > > > > > of
> > > > > > > > work", it might make sense to remove them, then.
> > > > > > > >
> > > > > > > > Things that don't seem valuable usually fall into
> > > another
> > > > > > > > category
> > > > > > > > like "cannot reasonably be used", but if the only
> > > reason for
> > > > > > > > deprecation is that the feature is going away, we
> > > should really
> > > > > > > > support it according to our promise.
> > > > > > > >
> > > > > > > > But if it's really valuable to remove these
> > > endpoints, maybe it's
> > > > > > > > appropriate to bump the major version and ask
> > > everybody to update
> > > > > > > > their clients?
> > > > > > > >
> > > > > > > > On Tue, Nov 12, 2019 at 5:27 PM Rawlin Peters <
> > > rawlin@apache.org>
> > > > > > > > wrote:
> > > > > > > > > The real crux of the issue here is whether or
> not
> > > we agree as a
> > > > > > > > > project that we can move forward with this plan
> > > that I believe
> > > > > > > > > to be a
> > > > > > > > > good compromise that meets most concerns but
> > > "breaks the
> > > > > > > > > Semantic
> > > > > > > > > Versioning promise":
> > > > > > > > > 1) don't rewrite deprecated endpoints from
> Perl to
> > > Go
> > > > > > > > > 2) in the Perl handler, add a deprecation
> notice
> > > to the
> > > > > > > > > "alerts"
> > > > > > > > > section of the response
> > > > > > > > > 3) once a deprecated API has been deprecated
> for
> > > one major
> > > > > > > > > release,
> > > > > > > > > delete it from the API in the subsequent major
> > > release (e.g. if
> > > > > > > > > deprecation notices are added and released in
> ATC
> > > 4.x, we can
> > > > > > > > > remove
> > > > > > > > > the deprecated endpoint in ATC 5.x)
> > > > > > > > >
> > > > > > > > > Rob, from your latest proposal to remove the
> ATS
> > > config API
> > > > > > > > > endpoints
> > > > > > > > > from 1.x, I thought we were past this issue
> > > already and could
> > > > > > > > > finally
> > > > > > > > > agree to remove deprecated endpoints from 1.x
> > > (following steps
> > > > > > > > > 1-3
> > > > > > > > > above) without requiring an all-new 2.x API.
> Are
> > > we not
> > > > > > > > > actually past
> > > > > > > > > that? You came up with valid reasons for
> applying
> > > these steps
> > > > > > > > > to the
> > > > > > > > > ATS config API endpoints, and I think it's
> fair to
> > > say that we
> > > > > > > > > can
> > > > > > > > > come up with valid reasons for applying these
> > > steps to other
> > > > > > > > > deprecated API endpoints as well.
> > > > > > > > >
> > > > > > > > > It seems like the real problem is actually
> > > agreeing on which
> > > > > > > > > _specific_ endpoints we should be able to
> follow
> > > steps 1-3 for.
> > > > > > > > > IMO I
> > > > > > > > > think it should apply to any 1.x endpoints
> that we
> > > have valid
> > > > > > > > > reasons
> > > > > > > > > for not carrying forward to Go, including but
> not
> > > limited to:
> > > > > > > > > - the endpoints are dangerous and pose a risk
> to
> > > the overall
> > > > > > > > > Traffic
> > > > > > > > > Control system
> > > > > > > > > - the endpoints have been obsoleted by other
> > > endpoints
> > > > > > > > > - the endpoints don't seem to serve a true
> purpose
> > > or don't
> > > > > > > > > seem to be
> > > > > > > > > valuable to the project anymore
> > > > > > > > > - the task of rewriting the endpoints from
> Perl to
> > > Go
> > > > > > > > > represents an
> > > > > > > > > unreasonable amount of work for the project
> > > > > > > > >
> > > > > > > > > Obviously most of these reasons are
> subjective, so
> > > we'd have to
> > > > > > > > > come
> > > > > > > > > to some level of consensus on deprecating
> > > particular endpoints
> > > > > > > > > first,
> > > > > > > > > but we should be able to follow steps 1-3 for
> > > whatever
> > > > > > > > > endpoints we
> > > > > > > > > decide.
> > > > > > > > >
> > > > > > > > > Making breaking changes shouldn't be taken
> > > lightly, but we need
> > > > > > > > > a path
> > > > > > > > > forward that allows some calculated, breaking
> > > changes where
> > > > > > > > > necessary.
> > > > > > > > > Otherwise, we are unnecessarily holding the
> > > project back.
> > > > > > > > >
> > > > > > > > > - Rawlin
> > > > > > > > >
> > > > > > > > > On Tue, Nov 12, 2019 at 11:44 AM Jeremy
> Mitchell <
> > > > > > > > > mitchell852@gmail.com>
> > > > > > > > wrote:
> > > > > > > > > > Yeah, technically a "rewrite" of 1.x means
> > > shifting the
> > > > > > > > > > implementation
> > > > > > > > from
> > > > > > > > > > Perl to Go for ALL of 1.x. If you leave a
> subset
> > > of 1.x in
> > > > > > > > > > Perl, then
> > > > > > > > you
> > > > > > > > > > have to call it a "partial rewrite".
> > > > > > > > > >
> > > > > > > > > > So, assuming we want to do a full rewrite (no
> > > more perl for
> > > > > > > > > > the api
> > > > > > > > > > whatsoever), I think that means every 1.x api
> > > endpoint (good
> > > > > > > > > > or bad)
> > > > > > > > needs
> > > > > > > > > > to be rewritten to Go so we can remove the
> > > dependency on
> > > > > > > > > > perl.
> > > > > > > > > >
> > > > > > > > > > However, I still like marking many as
> > > "deprecated" so we
> > > > > > > > > > train users
> > > > > > > > to use
> > > > > > > > > > "better" endpoints that we envision will
> exist
> > > in 2.x and
> > > > > > > > > > beyond.
> > > > > > > > > >
> > > > > > > > > > Jeremy
> > > > > > > > > >
> > > > > > > > > > On Tue, Nov 12, 2019 at 11:22 AM Hoppal,
> Michael
> > > <
> > > > > > > > Michael_Hoppal@comcast.com>
> > > > > > > > > > wrote:
> > > > > > > > > >
> > > > > > > > > > > We do not have a lot of deprecated routes
> as
> > > of now as we
> > > > > > > > > > > focused on
> > > > > > > > > > > routes that are more used first.
> > > > > > > > > > >
> > > > > > > > > > > I know on a lot of the remaining rewrites
> > > there has been
> > > > > > > > > > > discussion
> > > > > > > > on the
> > > > > > > > > > > matching Github issues on potential
> > > deprecation (I cannot
> > > > > > > > > > > think of
> > > > > > > > the
> > > > > > > > > > > exact count but enough to warrant this
> email
> > > __)
> > > > > > > > > > >
> > > > > > > > > > > It is key to note that I am not arguing for
> > > removal of
> > > > > > > > > > > endpoints
> > > > > > > > within
> > > > > > > > > > > the current API version but if there is an
> > > endpoint we
> > > > > > > > > > > agree should
> > > > > > > > be
> > > > > > > > > > > deprecated going forward to put a message
> > > within the
> > > > > > > > > > > response and on
> > > > > > > > the
> > > > > > > > > > > docs.
> > > > > > > > > > >
> > > > > > > > > > > And thanks for the input sounds like you
> are
> > > +1 on
> > > > > > > > > > > rewriting all
> > > > > > > > routes to
> > > > > > > > > > > Golang as Brennan is.
> > > > > > > > > > >
> > > > > > > > > > > On 11/12/19, 9:06 AM, "Robert O Butts" <
> > > rob@apache.org>
> > > > > > > > > > > wrote:
> > > > > > > > > > >
> > > > > > > > > > > Whether we rewrite a route in Go is an
> > > implementation
> > > > > > > > > > > detail. To
> > > > > > > > the
> > > > > > > > > > > interface, to users, it doesn't matter
> > > whether a route
> > > > > > > > > > > is
> > > > > > > > rewritten or
> > > > > > > > > > > not.
> > > > > > > > > > >
> > > > > > > > > > > But our API follows Semantic
> Versioning,
> > > in order to
> > > > > > > > > > > not break
> > > > > > > > users.
> > > > > > > > > > > We
> > > > > > > > > > > can't just remove endpoints that some
> of
> > > us don't use,
> > > > > > > > > > > and
> > > > > > > > assume other
> > > > > > > > > > > people don't, maybe people who never
> speak
> > > up on the
> > > > > > > > > > > mailing
> > > > > > > > list.
> > > > > > > > > > > We'll
> > > > > > > > > > > never gain a big userbase if we keep
> > > breaking users.
> > > > > > > > > > >
> > > > > > > > > > > Per the _project_ SemVer, once we have
> API
> > > 2.0, we can
> > > > > > > > > > > deprecate
> > > > > > > > API
> > > > > > > > > > > 1.x,
> > > > > > > > > > > and in the next major _project_
> release,
> > > remove API
> > > > > > > > > > > 1.x.
> > > > > > > > Irrespective
> > > > > > > > > > > of
> > > > > > > > > > > Perl or Go.
> > > > > > > > > > >
> > > > > > > > > > > My big concern is, API 2.0 is a big
> > > project. How long
> > > > > > > > > > > has the
> > > > > > > > rewrite
> > > > > > > > > > > to Go
> > > > > > > > > > > taken? Do we really believe designing
> and
> > > implementing
> > > > > > > > > > > a
> > > > > > > > completely
> > > > > > > > > > > new API
> > > > > > > > > > > will be any less time?
> > > > > > > > > > >
> > > > > > > > > > > I don't want killing Perl to have to
> wait
> > > on that.
> > > > > > > > > > >
> > > > > > > > > > > I know it feels like a waste to rewrite
> > > routes that you
> > > > > > > > > > > don't
> > > > > > > > use, and
> > > > > > > > > > > probably few people do. But that's the
> > > cost of a stable
> > > > > > > > > > > project.
> > > > > > > > How
> > > > > > > > > > > many
> > > > > > > > > > > "deprecated" routes are there? If it
> comes
> > > down to
> > > > > > > > > > > taking the
> > > > > > > > > > > development
> > > > > > > > > > > time to rewrite them so we can kill
> Perl
> > > faster, or
> > > > > > > > > > > leaving Perl
> > > > > > > > > > > around, I
> > > > > > > > > > > vote we just do the work and kill Perl.
> > > > > > > > > > >
> > > > > > > > > > > >If we don't rewrite them, then Perl
> will
> > > last until
> > > > > > > > > > > API 2.0 has
> > > > > > > > been
> > > > > > > > > > > designed, released and then *another
> full
> > > major release
> > > > > > > > > > > cycle*.
> > > > > > > > That's
> > > > > > > > > > > way
> > > > > > > > > > > too long to have two codebases for the
> > > same component,
> > > > > > > > > > > IMO,
> > > > > > > > especially
> > > > > > > > > > > since the rewrite is already 50%
> complete.
> > > > > > > > > > >
> > > > > > > > > > > +1
> > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > > > On Tue, Nov 12, 2019 at 8:54 AM ocket
> 8888
> > > <
> > > > > > > > > > > ocket8888@gmail.com>
> > > > > > > > > > > wrote:
> > > > > > > > > > >
> > > > > > > > > > > > I vote that by and large we DO
> rewrite
> > > them, with
> > > > > > > > > > > exceptions
> > > > > > > > for
> > > > > > > > > > > routes
> > > > > > > > > > > > that just plain don't work, even in
> > > Perl. Those are
> > > > > > > > > > > few,
> > > > > > > > though.
> > > > > > > > > > > >
> > > > > > > > > > > > If we don't rewrite them, then Perl
> will
> > > last until
> > > > > > > > > > > API 2.0
> > > > > > > > has been
> > > > > > > > > > > > designed, released and then *another
> > > full major
> > > > > > > > > > > release cycle*.
> > > > > > > > > > > That's way
> > > > > > > > > > > > too long to have two codebases for
> the
> > > same
> > > > > > > > > > > component, IMO,
> > > > > > > > > > > especially
> > > > > > > > > > > > since the rewrite is already 50%
> > > complete.
> > > > > > > > > > > >
> > > > > > > > > > > > On Tue, Nov 12, 2019 at 8:43 AM
> Hoppal,
> > > Michael <
> > > > > > > > > > > > Michael_Hoppal@comcast.com>
> > > > > > > > > > > > wrote:
> > > > > > > > > > > >
> > > > > > > > > > > > > As the Traffic Ops API is being
> > > rewritten from Perl
> > > > > > > > > > > to Golang
> > > > > > > > > > > there has
> > > > > > > > > > > > > been several routes that have been
> > > deprecated and
> > > > > > > > > > > probably
> > > > > > > > more to
> > > > > > > > > > > come.
> > > > > > > > > > > > >
> > > > > > > > > > > > > In the deprecation efforts I have
> seen
> > > two
> > > > > > > > > > > strategies:
> > > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > > > > > * The route IS NOT rewritten
> from
> > > Perl to
> > > > > > > > > > > Golang and a
> > > > > > > > > > > deprecation
> > > > > > > > > > > > > notice is added to the alert
> response
> > > in the Perl
> > > > > > > > > > > handler
> > > > > > > > > > > > > * The route IS rewritten from
> Perl
> > > to Golang
> > > > > > > > > > > and a
> > > > > > > > deprecation
> > > > > > > > > > > notice
> > > > > > > > > > > > > is added to the alert response in
> the
> > > Golang
> > > > > > > > > > > handler
> > > > > > > > > > > > >
> > > > > > > > > > > > > I think we should have consistency
> in
> > > our approach
> > > > > > > > > > > and
> > > > > > > > wanted to
> > > > > > > > > > > get
> > > > > > > > > > > > > people’s thoughts.
> > > > > > > > > > > > >
> > > > > > > > > > > > > I would vote that we do not
> rewrite a
> > > deprecated
> > > > > > > > > > > route from
> > > > > > > > Perl to
> > > > > > > > > > > > Golang.
> > > > > > > > > > > > >
> > > > > > > > > > > > > Thanks,
> > > > > > > > > > > > >
> > > > > > > > > > > > > Michael
> > > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > > >
> > > > >
> > > > >
> > > >
> > > >
> > >
> > >
> > >
>
Re: [EXTERNAL] Re: Traffic Ops Route Deprecation Strategy
Posted by Rawlin Peters <ra...@apache.org>.
While I appreciate and mostly agree with your essay on the importance
of not breaking users (or rather, breaking them in predictable ways),
I for one would really like to meet at some middle ground with option
2. I'm not asking to remove 50 high-usage endpoints from 1.x. IIRC
I've only specifically proposed "leaving behind" a few specific Perl
endpoints that I believe serve no purpose, have no valid reason to be
used, and/or are dangerous to the system:
- /riak/stats -- informational, was used for debugging the initial
Riak integration. This one might be pretty easy to rewrite since it
just calls a corresponding riak API and returns the result via TO, so
I could be on board with rewriting it if necessary.
- /cachegroup_fallbacks -- these endpoints were released in 3.0 before
we managed to augment the existing /cachegroups API to handle
fallbacks, so I could get on board with rewriting these if we really
think it's important.
- /capabilities (all methods except for GET) -- capabilities are
built-in, should not be changed at runtime
- /api_capabilities (all methods except for GET) -- same reasons as
/capabilities
- /cdns/{name}/configs/routing -- this was a half-baked attempt at
redoing CrConfig.json -- cannot possibly be of use to anyone in its
current state
- the ATS config generation endpoints -- Rob actually proposed to
remove these from 1.x due to danger of being out of sync w/ cache-side
config generator
- /hwinfo/dtdata -- AFAIK the "dtdata" endpoints were only
used/required by the old Perl Mojolicious UI, no valid reason to be
using this as an external API client. In fact, I'm not even sure _how_
this endpoint can be used outside of the Mojolicious framework/UI. It
calls the Mojolicious `render` function, which usually means rendering
an embedded Perl HTML template given some data. This is one I would
consider completely unreasonable to rewrite to Go.
Can we get on board with leaving at least these 1.x endpoints behind
in the Perl? Doing a cursory skim over the remaining 1.x endpoints to
rewrite, it _seems_ like everything else would fall into the "simpler
to just rewrite rather than gain consensus to remove" category.
- Rawlin
On Tue, Nov 19, 2019 at 9:19 AM Robert O Butts <ro...@apache.org> wrote:
>
> >Option 1 - Graduate current API from 1.x to 2.x - allowing us to leave
> behind routes in 1.x that are dangerous or deprecated
> >Option 2 - Rewrite all endpoints from Perl to Golang in current version as
> we are doing now.
>
> +1 on either Option 1 or Option 2, without a strong preference. Option 1
> has advantages of letting us also safely remove the Cache Config endpoints
> and implement Layered Profiles. But Option 2 also has the advantage of not
> requiring users to upgrade.
>
> > Option 3 - Only rewrite endpoints from Perl to Golang in current version
> that we agree are not dangerous or should deprecate
>
> -1. As before, it breaks users unpredictably.
>
> >It will literally outright break clients (*all* of them, not just *some*)
>
> This is technically correct: technically, new versions 'break' all users.
> The difference is, it does it in a predictable, safe way that users can
> handle, inspect, and safely upgrade.
>
> >Can someone please explain to me why (A) artificially graduating the
> current 1.x API to 2.x, and removing the deprecated endpoints from 1.x, is
> better for API clients overall than (B) just removing deprecated endpoints
> from 1.x? Existing 1.x clients would gain absolutely nothing from being
> forced to update to 2.x like this.
>
> >for zero gain in value
>
> Users would gain something. There is a very specific gain.
>
> The gain is that the breakage is predictable. People frequently upgrade
> minor versions automatically, either running commands without thinking, or
> even automating the entire process.
>
> For TC, this applies to both CDN operators, and their end-users.
>
> When the minor versions contain breaking changes, that automatic upgrade
> then breaks. In this case, any scripts or tools they have using the removed
> endpoints will break. If those scripts are important, this could bring down
> their CDN, or break their Delivery Service, or otherwise cause an outage.
>
> Contrawise, users expect breaking changes in Major versions, and don't
> typically automatically upgrade them. Rather, they'll manually inspect the
> update, do testing, and check the changelog for breaking changes.
>
> It's not a technical thing, it's a human thing. Humans automatically
> upgrade minor versions, and write scripts to do so, and dependency files to
> allow it, because the software industry taught people that minor versions
> shouldn't break anything.
>
> If we start breaking minor versions, initially users will experience
> breakage. Not all users, and not all catastrophic. But some users will
> experience catastrophic outages. And it will be our fault as a project. And
> after enough times, they'll fork TC, or find another product, or write
> their own; and stop participating in the community, or never start.
>
> Then, once they've learned that we can't be trusted not to break things,
> they'll experience the 2nd problem: where nothing can be safely upgraded,
> and the smallest change requires long and expensive testing to verify a
> single minor version or even patch upgrade, and still risk missing
> something and bringing their CDN down. To be honest, we're pretty much
> already here. It already costs our dev and ops teams unbelievable amounts
> of time and person-hours of testing for every minor upgrade. That time far
> exceeds the development time to just rewrite the small endpoints we don't
> like anymore (and tangentially, also the time to write the Integration and
> E2E tests TC desperately needs).
>
> ATC has so much potential. There aren't many OSS CDNs out there, and
> there's certainly the demand. We have the potential to become a huge
> project, which for us existing users, translates into potentially an
> enormous amount of free development. But we'll never get there if we keep
> breaking things.
>
> And for my particular organization (Comcast), we have a hard enough time
> upgrading. Our upgrades, especially Traffic Ops, are extremely painful, and
> require enormous amounts of manual testing. Even then, we frequently have
> issues that require rollback, sometimes even an outage; even after those
> unbelievably amounts of developer and operator time testing. We desperately
> need a system that is safer and easier to upgrade. Breaking changes are
> always going to require more testing; but we can at least make minor
> versions safe to upgrade.
>
> > I know I am changing my initial stance but I am +1 on rewriting all the
> Perl endpoints to Golang and not breaking the API contract we have in
> place. Most of the API routes to rewrite are rather simple to port and I
> think would be faster than getting consensus on actually deprecating it
> from the mailing list.
>
> I'm inclined to agree with this. The things I've seen marked "deprecated"
> seem to mostly be small, and look like they'd be pretty easy to rewrite.
> But I haven't seen the whole list. Is there a list somewhere, so the
> community can see it to better decide?
>
>
> On Tue, Nov 19, 2019 at 7:24 AM Hoppal, Michael <Mi...@comcast.com>
> wrote:
>
> > I would love to drive this to some type of consensus on rewrite strategy
> > as we have a good amount of resources (and growing) working on the rewrite
> > and I want to keep that momentum.
> >
> > Sounds like there is three options from the email thread (please add any I
> > missed)
> >
> > Option 1 - Graduate current API from 1.x to 2.x - allowing us to leave
> > behind routes in 1.x that are dangerous or deprecated
> > Option 2 - Rewrite all endpoints from Perl to Golang in current version as
> > we are doing now.
> > Option 3 - Only rewrite endpoints from Perl to Golang in current version
> > that we agree are not dangerous or should deprecate
> >
> > I am +1 for option 2 (which I know is different from my original stance).
> >
> > Which direction would others like to go?
> >
> > Thanks,
> >
> > Michael
> >
> >
> > I know I am changing my initial stance but I am +1 on rewriting all the
> > Perl endpoints to Golang and not breaking the API contract we have in
> > place. Most of the API routes to rewrite are rather simple to port and I
> > think would be faster than getting consensus on actually deprecating it
> > from the mailing list.
> >
> > On 11/16/19, 3:31 PM, "Rawlin Peters" <ra...@apache.org> wrote:
> >
> > > Moving to 2.x means that it's my fault as a consumer of the API if I
> > was still using the deprecated endpoint (and that change should be part of
> > a larger upgrade plan). Simply removing the deprecated routes,
> > retroactively, means I can't rely on having a transition period to get to
> > 2.0 outside of a large completely down upgrade window.
> >
> > There would be a transition period either way. Endpoints would be
> > deprecated in ATC 4.0 and removed in 5.0. The release notes for 4.0
> > would clearly list out the deprecated endpoints. If they have
> > replacements, there would be instructions on how to transition to the
> > replacement. Deprecated endpoints would return alerts. After upgrading
> > to 4.0, operators should be well aware that certain endpoints are
> > being removed in 5.0, and it's on them to start using new endpoints,
> > change workflows, etc.
> >
> > By forcing operators to upgrade all of their clients to start using
> > API 2.x, it just adds to their burden of upgrading when that time
> > comes. Rather than surgically fixing the clients of the 2% of
> > deprecated API endpoints before upgrading to ATC 5.0, operators would
> > have to fix every single client they have to use API 2.x before
> > upgrading to 5.0. Is that exercise really worth it? Most people would
> > probably be blissfully unaware that the "POST /api/1.1/capabilities"
> > endpoint disappeared (among other deprecated endpoints). Less footguns
> > in the system, and they didn't even have to update their clients.
> >
> > Anyways, circling back to Michael's original question, it sounds like
> > either option we choose, we at least have consensus on this
> > deprecation strategy:
> > - The route IS NOT rewritten from Perl to Golang and a deprecation
> > notice is added to the alert response in the Perl handler
> >
> > If we're content to force all API clients to update their clients to
> > 2.x for zero gain in value, at least we won't be wasting work by
> > rewriting deprecated Perl endpoints to Go.
> >
> > - Rawlin
> >
> > On Fri, Nov 15, 2019 at 7:06 PM Gray, Jonathan
> > <Jo...@comcast.com> wrote:
> > >
> > > Moving to 2.x means that it's my fault as a consumer of the API if I
> > was still using the deprecated endpoint (and that change should be part of
> > a larger upgrade plan). Simply removing the deprecated routes,
> > retroactively, means I can't rely on having a transition period to get to
> > 2.0 outside of a large completely down upgrade window. Depending on how
> > the API is used, even in our own components such as TM, that could prove
> > disastrous. For the older minor revs like 1.1/1.2, I could live with
> > dropping those as part of a major project release (including upgrade
> > instructions outlining what releases you must be on beforehand). Currently
> > 1.3 I view as stable enough to code to or use a native client library from,
> > and 1.4 is my risk/overhead to take for new stuff as it's still evolving
> > and improving. Going to 1.5 or 2.0 moves that forward. The path of least
> > surprise for humans is to expect big shifts on those major revs. Removal
> > falls under that umbrella and gets as a consumer not to strictly focus on
> > endpoint->endpoint moves, but rather workflow shifts and maybe now I have
> > to use the API a different way than before. I believe getting this right
> > is something that would greatly help us understand does X component at
> > version 1 work with Y component at version 2, because now you're asking do
> > components X and Y both speak protocol A. With major releases we can work
> > through that each time, but it's error prone and requires experts in all
> > products to agree upon. If you do nothing but consume major releases, and
> > then conduct your own impact evaluation having read the upgrade
> > instructions thoroughly, maybe you'd be able to smoothly transition.
> > >
> > > Jonathan G
> > >
> > >
> > > On 11/15/19, 5:24 PM, "Rawlin Peters" <ra...@apache.org> wrote:
> > >
> > > Can someone please explain to me why (A) artificially graduating
> > the
> > > current 1.x API to 2.x, and removing the deprecated endpoints
> > from
> > > 1.x, is better for API clients overall than (B) just removing
> > > deprecated endpoints from 1.x? Existing 1.x clients would gain
> > > absolutely nothing from being forced to update to 2.x like this.
> > >
> > > The way I see the two choices:
> > > A) break every single API client out there
> > > B) break only clients of APIs that we've decided upon as a
> > community
> > > to be deprecated
> > >
> > > IMO, when we have to break things, we should try break the least
> > > amount of clients possible. All I'm looking for is a valuable
> > reason
> > > to choose option A over option B, that isn't just "because
> > SemVer".
> > >
> > > - Rawlin
> > >
> > > On Wed, Nov 13, 2019 at 3:24 PM Schmidt, Andrew (Contractor)
> > > <An...@comcast.com> wrote:
> > > >
> > > > If we decide to get rid of the API versioning it makes sense
> > to me that we would do that on a major version upgrade, probably to the ATC
> > version. But I think it would be too early to do it on the 2.0 API version.
> > I like this compromise +1. I was wondering why we were stuck on the idea
> > that 2.0 had to be an API rewrite. That seems like it’s a long way out. 2.0
> > = deprecations and fixes. 3.0 or 5.0 = api rewrite and maybe version
> > convergence.
> > > >
> > > > On 11/13/19, 11:52 AM, "ocket8888@gmail.com" <
> > ocket8888@gmail.com> wrote:
> > > >
> > > > Lemme lead with: making breaking changes on a major
> > version boundary
> > > > gets +1 from me - no reason the API needs to be fully
> > redesigned for
> > > > any particular number.
> > > >
> > > > But I just gotta bring this up again: if we want to be
> > able to
> > > > deprecate an endpoint for a major release and remove it in
> > the next
> > > > without needing to deal with the messiness of API
> > versions, we could
> > > > always just get rid of API versions. Major ATC revisions
> > should be
> > > > easier to come by with our new release strategy.
> > > >
> > > > On Wed, 2019-11-13 at 10:23 -0700, Rawlin Peters wrote:
> > > > > > It is a compromise - it's an inconvenience to users,
> > forcing them
> > > > > > to fix
> > > > > scripts to upgrade. Which isn't good for user experience
> > either; but
> > > > > it
> > > > > doesn't outright break people. It's a compromise I'm
> > willing to live
> > > > > with.
> > > > >
> > > > > It will literally outright break clients (*all* of them,
> > not just
> > > > > *some*). We have been down this road before, so I'll
> > just repeat what
> > > > > I said when this exact conversation came up months
> > ago[1]:
> > > > >
> > > > > "So it basically boils down to:
> > > > > A) promote Go 1.x routes to 2.x and eventually break ALL
> > users of the
> > > > > 1.x API when the time comes to remove 1.x
> > > > > B) deprecate and remove some 1.x Perl routes and
> > POSSIBLY break SOME
> > > > > users"
> > > > >
> > > > > Either way we slice and dice it with versioning,
> > removing an endpoint
> > > > > means that clients of that endpoint will break. By
> > graduating only
> > > > > the
> > > > > "good stuff" from 1.x to 2.x, and removing endpoints
> > from 1.x, we
> > > > > will
> > > > > unnecessarily break all 1.x clients when the time comes
> > to remove the
> > > > > rest of 1.x. Instead, we can choose to break only
> > clients of 1.x APIs
> > > > > that we already deemed dangerous, unnecessary, useless,
> > etc. enough
> > > > > to
> > > > > deprecate.
> > > > >
> > > > > Why break 100% of our API clients when we can choose to
> > break only a
> > > > > small number of clients that happen to use
> > > > > dangerous/unnecessary/useless APIs? Why would we break
> > our API
> > > > > clients
> > > > > unnecessarily for no added value? There is no sense in
> > blindly
> > > > > adhering to Semantic Versioning just for the sake of
> > adhering to
> > > > > Semantic Versioning. Instead, use it as a guideline, and
> > break the
> > > > > rules where it makes the most sense to do so.
> > > > >
> > > > > - Rawlin
> > > > >
> > > > > [1]
> > > > >
> > https://urldefense.com/v3/__https://lists.apache.org/thread.html/15c6bfcdce0ac00a3a8612a96c3d668a8a5a16c6ca9401de9473d31c@*3Cdev.trafficcontrol.apache.org*3E__;JSU!rx_L75ITgOQ!V_b3E9iEYqNlZus18f-sVuofwuyxapCthg3LdO4NxyMCU39b7mdf4klP5KNR7XZ7d3Uc$
> > > > >
> > > > > On Wed, Nov 13, 2019 at 9:43 AM Robert O Butts <
> > rob@apache.org>
> > > > > wrote:
> > > > > > > But if it's really valuable to remove these
> > endpoints, maybe it's
> > > > > > appropriate to bump the major version and ask
> > everybody to update
> > > > > > their
> > > > > > clients?
> > > > > >
> > > > > > +1
> > > > > >
> > > > > > This sounds like the best compromise to me. It
> > addresses both sides
> > > > > > - it
> > > > > > lets us remove endpoints, and it also prevents
> > breaking users.
> > > > > >
> > > > > > It is a compromise - it's an inconvenience to users,
> > forcing them
> > > > > > to fix
> > > > > > scripts to upgrade. Which isn't good for user
> > experience either;
> > > > > > but it
> > > > > > doesn't outright break people. It's a compromise I'm
> > willing to
> > > > > > live with.
> > > > > >
> > > > > > This actually lets us do at least 3 things:
> > > > > > 1. Remove "deprecated" endpoints.
> > > > > > 2. Remove the unmaintained dangerously-large Cache
> > Config
> > > > > > endpoints.
> > > > > > 3. Lets us do Layered Profiles, changing Servers and
> > DSes to have
> > > > > > arrays of
> > > > > > Profiles instead of a single Profile, without hacky
> > backward-
> > > > > > compatibility
> > > > > > attempts.
> > > > > >
> > > > > > To clarify, `/api/2.0` would be essentially the
> > current API, just
> > > > > > with the
> > > > > > above breaking changes (possibly others) which require
> > a SemVer
> > > > > > major
> > > > > > version increase. And the big API rewrite/redesign
> > we've been
> > > > > > calling "2.0"
> > > > > > would become `/api/3.0` or some future version.
> > > > > >
> > > > > > It's not perfect, but it seems to address the most
> > concerns on all
> > > > > > sides.
> > > > > > Thoughts? Objections?
> > > > > >
> > > > > >
> > > > > > On Wed, Nov 13, 2019 at 8:45 AM Chris Lemmons <
> > alficles@gmail.com>
> > > > > > wrote:
> > > > > >
> > > > > > > For the endpoints, I agree in part and disagree in
> > part.
> > > > > > >
> > > > > > > Agree for these two:
> > > > > > > - the endpoints are dangerous and pose a risk to the
> > overall
> > > > > > > Traffic
> > > > > > > Control system
> > > > > > > - the task of rewriting the endpoints from Perl to
> > Go represents
> > > > > > > an
> > > > > > > unreasonable amount of work for the project
> > > > > > >
> > > > > > > And I would add:
> > > > > > > - the endpoint does not serve its intended purpose
> > and cannot
> > > > > > > reasonably
> > > > > > > be used
> > > > > > > - the endpoint has been made unsafe or unusable by
> > new features
> > > > > > > added to TC
> > > > > > >
> > > > > > > Disagree for these two:
> > > > > > > - the endpoints have been obsoleted by other
> > endpoints
> > > > > > > - the endpoints don't seem to serve a true purpose
> > or don't seem
> > > > > > > to be
> > > > > > > valuable to the project anymore
> > > > > > >
> > > > > > > For the most part, obsoleted endpoints can be
> > maintained with a
> > > > > > > shim
> > > > > > > into the new code. If they would require "an
> > unreasonable amount
> > > > > > > of
> > > > > > > work", it might make sense to remove them, then.
> > > > > > >
> > > > > > > Things that don't seem valuable usually fall into
> > another
> > > > > > > category
> > > > > > > like "cannot reasonably be used", but if the only
> > reason for
> > > > > > > deprecation is that the feature is going away, we
> > should really
> > > > > > > support it according to our promise.
> > > > > > >
> > > > > > > But if it's really valuable to remove these
> > endpoints, maybe it's
> > > > > > > appropriate to bump the major version and ask
> > everybody to update
> > > > > > > their clients?
> > > > > > >
> > > > > > > On Tue, Nov 12, 2019 at 5:27 PM Rawlin Peters <
> > rawlin@apache.org>
> > > > > > > wrote:
> > > > > > > > The real crux of the issue here is whether or not
> > we agree as a
> > > > > > > > project that we can move forward with this plan
> > that I believe
> > > > > > > > to be a
> > > > > > > > good compromise that meets most concerns but
> > "breaks the
> > > > > > > > Semantic
> > > > > > > > Versioning promise":
> > > > > > > > 1) don't rewrite deprecated endpoints from Perl to
> > Go
> > > > > > > > 2) in the Perl handler, add a deprecation notice
> > to the
> > > > > > > > "alerts"
> > > > > > > > section of the response
> > > > > > > > 3) once a deprecated API has been deprecated for
> > one major
> > > > > > > > release,
> > > > > > > > delete it from the API in the subsequent major
> > release (e.g. if
> > > > > > > > deprecation notices are added and released in ATC
> > 4.x, we can
> > > > > > > > remove
> > > > > > > > the deprecated endpoint in ATC 5.x)
> > > > > > > >
> > > > > > > > Rob, from your latest proposal to remove the ATS
> > config API
> > > > > > > > endpoints
> > > > > > > > from 1.x, I thought we were past this issue
> > already and could
> > > > > > > > finally
> > > > > > > > agree to remove deprecated endpoints from 1.x
> > (following steps
> > > > > > > > 1-3
> > > > > > > > above) without requiring an all-new 2.x API. Are
> > we not
> > > > > > > > actually past
> > > > > > > > that? You came up with valid reasons for applying
> > these steps
> > > > > > > > to the
> > > > > > > > ATS config API endpoints, and I think it's fair to
> > say that we
> > > > > > > > can
> > > > > > > > come up with valid reasons for applying these
> > steps to other
> > > > > > > > deprecated API endpoints as well.
> > > > > > > >
> > > > > > > > It seems like the real problem is actually
> > agreeing on which
> > > > > > > > _specific_ endpoints we should be able to follow
> > steps 1-3 for.
> > > > > > > > IMO I
> > > > > > > > think it should apply to any 1.x endpoints that we
> > have valid
> > > > > > > > reasons
> > > > > > > > for not carrying forward to Go, including but not
> > limited to:
> > > > > > > > - the endpoints are dangerous and pose a risk to
> > the overall
> > > > > > > > Traffic
> > > > > > > > Control system
> > > > > > > > - the endpoints have been obsoleted by other
> > endpoints
> > > > > > > > - the endpoints don't seem to serve a true purpose
> > or don't
> > > > > > > > seem to be
> > > > > > > > valuable to the project anymore
> > > > > > > > - the task of rewriting the endpoints from Perl to
> > Go
> > > > > > > > represents an
> > > > > > > > unreasonable amount of work for the project
> > > > > > > >
> > > > > > > > Obviously most of these reasons are subjective, so
> > we'd have to
> > > > > > > > come
> > > > > > > > to some level of consensus on deprecating
> > particular endpoints
> > > > > > > > first,
> > > > > > > > but we should be able to follow steps 1-3 for
> > whatever
> > > > > > > > endpoints we
> > > > > > > > decide.
> > > > > > > >
> > > > > > > > Making breaking changes shouldn't be taken
> > lightly, but we need
> > > > > > > > a path
> > > > > > > > forward that allows some calculated, breaking
> > changes where
> > > > > > > > necessary.
> > > > > > > > Otherwise, we are unnecessarily holding the
> > project back.
> > > > > > > >
> > > > > > > > - Rawlin
> > > > > > > >
> > > > > > > > On Tue, Nov 12, 2019 at 11:44 AM Jeremy Mitchell <
> > > > > > > > mitchell852@gmail.com>
> > > > > > > wrote:
> > > > > > > > > Yeah, technically a "rewrite" of 1.x means
> > shifting the
> > > > > > > > > implementation
> > > > > > > from
> > > > > > > > > Perl to Go for ALL of 1.x. If you leave a subset
> > of 1.x in
> > > > > > > > > Perl, then
> > > > > > > you
> > > > > > > > > have to call it a "partial rewrite".
> > > > > > > > >
> > > > > > > > > So, assuming we want to do a full rewrite (no
> > more perl for
> > > > > > > > > the api
> > > > > > > > > whatsoever), I think that means every 1.x api
> > endpoint (good
> > > > > > > > > or bad)
> > > > > > > needs
> > > > > > > > > to be rewritten to Go so we can remove the
> > dependency on
> > > > > > > > > perl.
> > > > > > > > >
> > > > > > > > > However, I still like marking many as
> > "deprecated" so we
> > > > > > > > > train users
> > > > > > > to use
> > > > > > > > > "better" endpoints that we envision will exist
> > in 2.x and
> > > > > > > > > beyond.
> > > > > > > > >
> > > > > > > > > Jeremy
> > > > > > > > >
> > > > > > > > > On Tue, Nov 12, 2019 at 11:22 AM Hoppal, Michael
> > <
> > > > > > > Michael_Hoppal@comcast.com>
> > > > > > > > > wrote:
> > > > > > > > >
> > > > > > > > > > We do not have a lot of deprecated routes as
> > of now as we
> > > > > > > > > > focused on
> > > > > > > > > > routes that are more used first.
> > > > > > > > > >
> > > > > > > > > > I know on a lot of the remaining rewrites
> > there has been
> > > > > > > > > > discussion
> > > > > > > on the
> > > > > > > > > > matching Github issues on potential
> > deprecation (I cannot
> > > > > > > > > > think of
> > > > > > > the
> > > > > > > > > > exact count but enough to warrant this email
> > __)
> > > > > > > > > >
> > > > > > > > > > It is key to note that I am not arguing for
> > removal of
> > > > > > > > > > endpoints
> > > > > > > within
> > > > > > > > > > the current API version but if there is an
> > endpoint we
> > > > > > > > > > agree should
> > > > > > > be
> > > > > > > > > > deprecated going forward to put a message
> > within the
> > > > > > > > > > response and on
> > > > > > > the
> > > > > > > > > > docs.
> > > > > > > > > >
> > > > > > > > > > And thanks for the input sounds like you are
> > +1 on
> > > > > > > > > > rewriting all
> > > > > > > routes to
> > > > > > > > > > Golang as Brennan is.
> > > > > > > > > >
> > > > > > > > > > On 11/12/19, 9:06 AM, "Robert O Butts" <
> > rob@apache.org>
> > > > > > > > > > wrote:
> > > > > > > > > >
> > > > > > > > > > Whether we rewrite a route in Go is an
> > implementation
> > > > > > > > > > detail. To
> > > > > > > the
> > > > > > > > > > interface, to users, it doesn't matter
> > whether a route
> > > > > > > > > > is
> > > > > > > rewritten or
> > > > > > > > > > not.
> > > > > > > > > >
> > > > > > > > > > But our API follows Semantic Versioning,
> > in order to
> > > > > > > > > > not break
> > > > > > > users.
> > > > > > > > > > We
> > > > > > > > > > can't just remove endpoints that some of
> > us don't use,
> > > > > > > > > > and
> > > > > > > assume other
> > > > > > > > > > people don't, maybe people who never speak
> > up on the
> > > > > > > > > > mailing
> > > > > > > list.
> > > > > > > > > > We'll
> > > > > > > > > > never gain a big userbase if we keep
> > breaking users.
> > > > > > > > > >
> > > > > > > > > > Per the _project_ SemVer, once we have API
> > 2.0, we can
> > > > > > > > > > deprecate
> > > > > > > API
> > > > > > > > > > 1.x,
> > > > > > > > > > and in the next major _project_ release,
> > remove API
> > > > > > > > > > 1.x.
> > > > > > > Irrespective
> > > > > > > > > > of
> > > > > > > > > > Perl or Go.
> > > > > > > > > >
> > > > > > > > > > My big concern is, API 2.0 is a big
> > project. How long
> > > > > > > > > > has the
> > > > > > > rewrite
> > > > > > > > > > to Go
> > > > > > > > > > taken? Do we really believe designing and
> > implementing
> > > > > > > > > > a
> > > > > > > completely
> > > > > > > > > > new API
> > > > > > > > > > will be any less time?
> > > > > > > > > >
> > > > > > > > > > I don't want killing Perl to have to wait
> > on that.
> > > > > > > > > >
> > > > > > > > > > I know it feels like a waste to rewrite
> > routes that you
> > > > > > > > > > don't
> > > > > > > use, and
> > > > > > > > > > probably few people do. But that's the
> > cost of a stable
> > > > > > > > > > project.
> > > > > > > How
> > > > > > > > > > many
> > > > > > > > > > "deprecated" routes are there? If it comes
> > down to
> > > > > > > > > > taking the
> > > > > > > > > > development
> > > > > > > > > > time to rewrite them so we can kill Perl
> > faster, or
> > > > > > > > > > leaving Perl
> > > > > > > > > > around, I
> > > > > > > > > > vote we just do the work and kill Perl.
> > > > > > > > > >
> > > > > > > > > > >If we don't rewrite them, then Perl will
> > last until
> > > > > > > > > > API 2.0 has
> > > > > > > been
> > > > > > > > > > designed, released and then *another full
> > major release
> > > > > > > > > > cycle*.
> > > > > > > That's
> > > > > > > > > > way
> > > > > > > > > > too long to have two codebases for the
> > same component,
> > > > > > > > > > IMO,
> > > > > > > especially
> > > > > > > > > > since the rewrite is already 50% complete.
> > > > > > > > > >
> > > > > > > > > > +1
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > > On Tue, Nov 12, 2019 at 8:54 AM ocket 8888
> > <
> > > > > > > > > > ocket8888@gmail.com>
> > > > > > > > > > wrote:
> > > > > > > > > >
> > > > > > > > > > > I vote that by and large we DO rewrite
> > them, with
> > > > > > > > > > exceptions
> > > > > > > for
> > > > > > > > > > routes
> > > > > > > > > > > that just plain don't work, even in
> > Perl. Those are
> > > > > > > > > > few,
> > > > > > > though.
> > > > > > > > > > >
> > > > > > > > > > > If we don't rewrite them, then Perl will
> > last until
> > > > > > > > > > API 2.0
> > > > > > > has been
> > > > > > > > > > > designed, released and then *another
> > full major
> > > > > > > > > > release cycle*.
> > > > > > > > > > That's way
> > > > > > > > > > > too long to have two codebases for the
> > same
> > > > > > > > > > component, IMO,
> > > > > > > > > > especially
> > > > > > > > > > > since the rewrite is already 50%
> > complete.
> > > > > > > > > > >
> > > > > > > > > > > On Tue, Nov 12, 2019 at 8:43 AM Hoppal,
> > Michael <
> > > > > > > > > > > Michael_Hoppal@comcast.com>
> > > > > > > > > > > wrote:
> > > > > > > > > > >
> > > > > > > > > > > > As the Traffic Ops API is being
> > rewritten from Perl
> > > > > > > > > > to Golang
> > > > > > > > > > there has
> > > > > > > > > > > > been several routes that have been
> > deprecated and
> > > > > > > > > > probably
> > > > > > > more to
> > > > > > > > > > come.
> > > > > > > > > > > >
> > > > > > > > > > > > In the deprecation efforts I have seen
> > two
> > > > > > > > > > strategies:
> > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > > > * The route IS NOT rewritten from
> > Perl to
> > > > > > > > > > Golang and a
> > > > > > > > > > deprecation
> > > > > > > > > > > > notice is added to the alert response
> > in the Perl
> > > > > > > > > > handler
> > > > > > > > > > > > * The route IS rewritten from Perl
> > to Golang
> > > > > > > > > > and a
> > > > > > > deprecation
> > > > > > > > > > notice
> > > > > > > > > > > > is added to the alert response in the
> > Golang
> > > > > > > > > > handler
> > > > > > > > > > > >
> > > > > > > > > > > > I think we should have consistency in
> > our approach
> > > > > > > > > > and
> > > > > > > wanted to
> > > > > > > > > > get
> > > > > > > > > > > > people’s thoughts.
> > > > > > > > > > > >
> > > > > > > > > > > > I would vote that we do not rewrite a
> > deprecated
> > > > > > > > > > route from
> > > > > > > Perl to
> > > > > > > > > > > Golang.
> > > > > > > > > > > >
> > > > > > > > > > > > Thanks,
> > > > > > > > > > > >
> > > > > > > > > > > > Michael
> > > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > >
> > > >
> > > >
> > >
> > >
> >
> >
> >
Re: [EXTERNAL] Re: Traffic Ops Route Deprecation Strategy
Posted by "Schmidt, Andrew (Contractor)" <An...@comcast.com>.
+1 on Option 2. I agree with Rob on all that. Since I'm not doing any of the rewrites myself it fine with me if everyone wants to rewrite all the endpoints we'll probably be deprecating in the future. If we do go with Option 2 then I think that frees us up to have a completely separate discussion in a new thread about moving from API version 1.x to ?. Maybe 2.0 maybe API = ATC version.
On 11/19/19, 9:19 AM, "Robert O Butts" <ro...@apache.org> wrote:
>Option 1 - Graduate current API from 1.x to 2.x - allowing us to leave
behind routes in 1.x that are dangerous or deprecated
>Option 2 - Rewrite all endpoints from Perl to Golang in current version as
we are doing now.
+1 on either Option 1 or Option 2, without a strong preference. Option 1
has advantages of letting us also safely remove the Cache Config endpoints
and implement Layered Profiles. But Option 2 also has the advantage of not
requiring users to upgrade.
> Option 3 - Only rewrite endpoints from Perl to Golang in current version
that we agree are not dangerous or should deprecate
-1. As before, it breaks users unpredictably.
>It will literally outright break clients (*all* of them, not just *some*)
This is technically correct: technically, new versions 'break' all users.
The difference is, it does it in a predictable, safe way that users can
handle, inspect, and safely upgrade.
>Can someone please explain to me why (A) artificially graduating the
current 1.x API to 2.x, and removing the deprecated endpoints from 1.x, is
better for API clients overall than (B) just removing deprecated endpoints
from 1.x? Existing 1.x clients would gain absolutely nothing from being
forced to update to 2.x like this.
>for zero gain in value
Users would gain something. There is a very specific gain.
The gain is that the breakage is predictable. People frequently upgrade
minor versions automatically, either running commands without thinking, or
even automating the entire process.
For TC, this applies to both CDN operators, and their end-users.
When the minor versions contain breaking changes, that automatic upgrade
then breaks. In this case, any scripts or tools they have using the removed
endpoints will break. If those scripts are important, this could bring down
their CDN, or break their Delivery Service, or otherwise cause an outage.
Contrawise, users expect breaking changes in Major versions, and don't
typically automatically upgrade them. Rather, they'll manually inspect the
update, do testing, and check the changelog for breaking changes.
It's not a technical thing, it's a human thing. Humans automatically
upgrade minor versions, and write scripts to do so, and dependency files to
allow it, because the software industry taught people that minor versions
shouldn't break anything.
If we start breaking minor versions, initially users will experience
breakage. Not all users, and not all catastrophic. But some users will
experience catastrophic outages. And it will be our fault as a project. And
after enough times, they'll fork TC, or find another product, or write
their own; and stop participating in the community, or never start.
Then, once they've learned that we can't be trusted not to break things,
they'll experience the 2nd problem: where nothing can be safely upgraded,
and the smallest change requires long and expensive testing to verify a
single minor version or even patch upgrade, and still risk missing
something and bringing their CDN down. To be honest, we're pretty much
already here. It already costs our dev and ops teams unbelievable amounts
of time and person-hours of testing for every minor upgrade. That time far
exceeds the development time to just rewrite the small endpoints we don't
like anymore (and tangentially, also the time to write the Integration and
E2E tests TC desperately needs).
ATC has so much potential. There aren't many OSS CDNs out there, and
there's certainly the demand. We have the potential to become a huge
project, which for us existing users, translates into potentially an
enormous amount of free development. But we'll never get there if we keep
breaking things.
And for my particular organization (Comcast), we have a hard enough time
upgrading. Our upgrades, especially Traffic Ops, are extremely painful, and
require enormous amounts of manual testing. Even then, we frequently have
issues that require rollback, sometimes even an outage; even after those
unbelievably amounts of developer and operator time testing. We desperately
need a system that is safer and easier to upgrade. Breaking changes are
always going to require more testing; but we can at least make minor
versions safe to upgrade.
> I know I am changing my initial stance but I am +1 on rewriting all the
Perl endpoints to Golang and not breaking the API contract we have in
place. Most of the API routes to rewrite are rather simple to port and I
think would be faster than getting consensus on actually deprecating it
from the mailing list.
I'm inclined to agree with this. The things I've seen marked "deprecated"
seem to mostly be small, and look like they'd be pretty easy to rewrite.
But I haven't seen the whole list. Is there a list somewhere, so the
community can see it to better decide?
On Tue, Nov 19, 2019 at 7:24 AM Hoppal, Michael <Mi...@comcast.com>
wrote:
> I would love to drive this to some type of consensus on rewrite strategy
> as we have a good amount of resources (and growing) working on the rewrite
> and I want to keep that momentum.
>
> Sounds like there is three options from the email thread (please add any I
> missed)
>
> Option 1 - Graduate current API from 1.x to 2.x - allowing us to leave
> behind routes in 1.x that are dangerous or deprecated
> Option 2 - Rewrite all endpoints from Perl to Golang in current version as
> we are doing now.
> Option 3 - Only rewrite endpoints from Perl to Golang in current version
> that we agree are not dangerous or should deprecate
>
> I am +1 for option 2 (which I know is different from my original stance).
>
> Which direction would others like to go?
>
> Thanks,
>
> Michael
>
>
> I know I am changing my initial stance but I am +1 on rewriting all the
> Perl endpoints to Golang and not breaking the API contract we have in
> place. Most of the API routes to rewrite are rather simple to port and I
> think would be faster than getting consensus on actually deprecating it
> from the mailing list.
>
> On 11/16/19, 3:31 PM, "Rawlin Peters" <ra...@apache.org> wrote:
>
> > Moving to 2.x means that it's my fault as a consumer of the API if I
> was still using the deprecated endpoint (and that change should be part of
> a larger upgrade plan). Simply removing the deprecated routes,
> retroactively, means I can't rely on having a transition period to get to
> 2.0 outside of a large completely down upgrade window.
>
> There would be a transition period either way. Endpoints would be
> deprecated in ATC 4.0 and removed in 5.0. The release notes for 4.0
> would clearly list out the deprecated endpoints. If they have
> replacements, there would be instructions on how to transition to the
> replacement. Deprecated endpoints would return alerts. After upgrading
> to 4.0, operators should be well aware that certain endpoints are
> being removed in 5.0, and it's on them to start using new endpoints,
> change workflows, etc.
>
> By forcing operators to upgrade all of their clients to start using
> API 2.x, it just adds to their burden of upgrading when that time
> comes. Rather than surgically fixing the clients of the 2% of
> deprecated API endpoints before upgrading to ATC 5.0, operators would
> have to fix every single client they have to use API 2.x before
> upgrading to 5.0. Is that exercise really worth it? Most people would
> probably be blissfully unaware that the "POST /api/1.1/capabilities"
> endpoint disappeared (among other deprecated endpoints). Less footguns
> in the system, and they didn't even have to update their clients.
>
> Anyways, circling back to Michael's original question, it sounds like
> either option we choose, we at least have consensus on this
> deprecation strategy:
> - The route IS NOT rewritten from Perl to Golang and a deprecation
> notice is added to the alert response in the Perl handler
>
> If we're content to force all API clients to update their clients to
> 2.x for zero gain in value, at least we won't be wasting work by
> rewriting deprecated Perl endpoints to Go.
>
> - Rawlin
>
> On Fri, Nov 15, 2019 at 7:06 PM Gray, Jonathan
> <Jo...@comcast.com> wrote:
> >
> > Moving to 2.x means that it's my fault as a consumer of the API if I
> was still using the deprecated endpoint (and that change should be part of
> a larger upgrade plan). Simply removing the deprecated routes,
> retroactively, means I can't rely on having a transition period to get to
> 2.0 outside of a large completely down upgrade window. Depending on how
> the API is used, even in our own components such as TM, that could prove
> disastrous. For the older minor revs like 1.1/1.2, I could live with
> dropping those as part of a major project release (including upgrade
> instructions outlining what releases you must be on beforehand). Currently
> 1.3 I view as stable enough to code to or use a native client library from,
> and 1.4 is my risk/overhead to take for new stuff as it's still evolving
> and improving. Going to 1.5 or 2.0 moves that forward. The path of least
> surprise for humans is to expect big shifts on those major revs. Removal
> falls under that umbrella and gets as a consumer not to strictly focus on
> endpoint->endpoint moves, but rather workflow shifts and maybe now I have
> to use the API a different way than before. I believe getting this right
> is something that would greatly help us understand does X component at
> version 1 work with Y component at version 2, because now you're asking do
> components X and Y both speak protocol A. With major releases we can work
> through that each time, but it's error prone and requires experts in all
> products to agree upon. If you do nothing but consume major releases, and
> then conduct your own impact evaluation having read the upgrade
> instructions thoroughly, maybe you'd be able to smoothly transition.
> >
> > Jonathan G
> >
> >
> > On 11/15/19, 5:24 PM, "Rawlin Peters" <ra...@apache.org> wrote:
> >
> > Can someone please explain to me why (A) artificially graduating
> the
> > current 1.x API to 2.x, and removing the deprecated endpoints
> from
> > 1.x, is better for API clients overall than (B) just removing
> > deprecated endpoints from 1.x? Existing 1.x clients would gain
> > absolutely nothing from being forced to update to 2.x like this.
> >
> > The way I see the two choices:
> > A) break every single API client out there
> > B) break only clients of APIs that we've decided upon as a
> community
> > to be deprecated
> >
> > IMO, when we have to break things, we should try break the least
> > amount of clients possible. All I'm looking for is a valuable
> reason
> > to choose option A over option B, that isn't just "because
> SemVer".
> >
> > - Rawlin
> >
> > On Wed, Nov 13, 2019 at 3:24 PM Schmidt, Andrew (Contractor)
> > <An...@comcast.com> wrote:
> > >
> > > If we decide to get rid of the API versioning it makes sense
> to me that we would do that on a major version upgrade, probably to the ATC
> version. But I think it would be too early to do it on the 2.0 API version.
> I like this compromise +1. I was wondering why we were stuck on the idea
> that 2.0 had to be an API rewrite. That seems like it’s a long way out. 2.0
> = deprecations and fixes. 3.0 or 5.0 = api rewrite and maybe version
> convergence.
> > >
> > > On 11/13/19, 11:52 AM, "ocket8888@gmail.com" <
> ocket8888@gmail.com> wrote:
> > >
> > > Lemme lead with: making breaking changes on a major
> version boundary
> > > gets +1 from me - no reason the API needs to be fully
> redesigned for
> > > any particular number.
> > >
> > > But I just gotta bring this up again: if we want to be
> able to
> > > deprecate an endpoint for a major release and remove it in
> the next
> > > without needing to deal with the messiness of API
> versions, we could
> > > always just get rid of API versions. Major ATC revisions
> should be
> > > easier to come by with our new release strategy.
> > >
> > > On Wed, 2019-11-13 at 10:23 -0700, Rawlin Peters wrote:
> > > > > It is a compromise - it's an inconvenience to users,
> forcing them
> > > > > to fix
> > > > scripts to upgrade. Which isn't good for user experience
> either; but
> > > > it
> > > > doesn't outright break people. It's a compromise I'm
> willing to live
> > > > with.
> > > >
> > > > It will literally outright break clients (*all* of them,
> not just
> > > > *some*). We have been down this road before, so I'll
> just repeat what
> > > > I said when this exact conversation came up months
> ago[1]:
> > > >
> > > > "So it basically boils down to:
> > > > A) promote Go 1.x routes to 2.x and eventually break ALL
> users of the
> > > > 1.x API when the time comes to remove 1.x
> > > > B) deprecate and remove some 1.x Perl routes and
> POSSIBLY break SOME
> > > > users"
> > > >
> > > > Either way we slice and dice it with versioning,
> removing an endpoint
> > > > means that clients of that endpoint will break. By
> graduating only
> > > > the
> > > > "good stuff" from 1.x to 2.x, and removing endpoints
> from 1.x, we
> > > > will
> > > > unnecessarily break all 1.x clients when the time comes
> to remove the
> > > > rest of 1.x. Instead, we can choose to break only
> clients of 1.x APIs
> > > > that we already deemed dangerous, unnecessary, useless,
> etc. enough
> > > > to
> > > > deprecate.
> > > >
> > > > Why break 100% of our API clients when we can choose to
> break only a
> > > > small number of clients that happen to use
> > > > dangerous/unnecessary/useless APIs? Why would we break
> our API
> > > > clients
> > > > unnecessarily for no added value? There is no sense in
> blindly
> > > > adhering to Semantic Versioning just for the sake of
> adhering to
> > > > Semantic Versioning. Instead, use it as a guideline, and
> break the
> > > > rules where it makes the most sense to do so.
> > > >
> > > > - Rawlin
> > > >
> > > > [1]
> > > >
> https://urldefense.com/v3/__https://lists.apache.org/thread.html/15c6bfcdce0ac00a3a8612a96c3d668a8a5a16c6ca9401de9473d31c@*3Cdev.trafficcontrol.apache.org*3E__;JSU!rx_L75ITgOQ!V_b3E9iEYqNlZus18f-sVuofwuyxapCthg3LdO4NxyMCU39b7mdf4klP5KNR7XZ7d3Uc$
> > > >
> > > > On Wed, Nov 13, 2019 at 9:43 AM Robert O Butts <
> rob@apache.org>
> > > > wrote:
> > > > > > But if it's really valuable to remove these
> endpoints, maybe it's
> > > > > appropriate to bump the major version and ask
> everybody to update
> > > > > their
> > > > > clients?
> > > > >
> > > > > +1
> > > > >
> > > > > This sounds like the best compromise to me. It
> addresses both sides
> > > > > - it
> > > > > lets us remove endpoints, and it also prevents
> breaking users.
> > > > >
> > > > > It is a compromise - it's an inconvenience to users,
> forcing them
> > > > > to fix
> > > > > scripts to upgrade. Which isn't good for user
> experience either;
> > > > > but it
> > > > > doesn't outright break people. It's a compromise I'm
> willing to
> > > > > live with.
> > > > >
> > > > > This actually lets us do at least 3 things:
> > > > > 1. Remove "deprecated" endpoints.
> > > > > 2. Remove the unmaintained dangerously-large Cache
> Config
> > > > > endpoints.
> > > > > 3. Lets us do Layered Profiles, changing Servers and
> DSes to have
> > > > > arrays of
> > > > > Profiles instead of a single Profile, without hacky
> backward-
> > > > > compatibility
> > > > > attempts.
> > > > >
> > > > > To clarify, `/api/2.0` would be essentially the
> current API, just
> > > > > with the
> > > > > above breaking changes (possibly others) which require
> a SemVer
> > > > > major
> > > > > version increase. And the big API rewrite/redesign
> we've been
> > > > > calling "2.0"
> > > > > would become `/api/3.0` or some future version.
> > > > >
> > > > > It's not perfect, but it seems to address the most
> concerns on all
> > > > > sides.
> > > > > Thoughts? Objections?
> > > > >
> > > > >
> > > > > On Wed, Nov 13, 2019 at 8:45 AM Chris Lemmons <
> alficles@gmail.com>
> > > > > wrote:
> > > > >
> > > > > > For the endpoints, I agree in part and disagree in
> part.
> > > > > >
> > > > > > Agree for these two:
> > > > > > - the endpoints are dangerous and pose a risk to the
> overall
> > > > > > Traffic
> > > > > > Control system
> > > > > > - the task of rewriting the endpoints from Perl to
> Go represents
> > > > > > an
> > > > > > unreasonable amount of work for the project
> > > > > >
> > > > > > And I would add:
> > > > > > - the endpoint does not serve its intended purpose
> and cannot
> > > > > > reasonably
> > > > > > be used
> > > > > > - the endpoint has been made unsafe or unusable by
> new features
> > > > > > added to TC
> > > > > >
> > > > > > Disagree for these two:
> > > > > > - the endpoints have been obsoleted by other
> endpoints
> > > > > > - the endpoints don't seem to serve a true purpose
> or don't seem
> > > > > > to be
> > > > > > valuable to the project anymore
> > > > > >
> > > > > > For the most part, obsoleted endpoints can be
> maintained with a
> > > > > > shim
> > > > > > into the new code. If they would require "an
> unreasonable amount
> > > > > > of
> > > > > > work", it might make sense to remove them, then.
> > > > > >
> > > > > > Things that don't seem valuable usually fall into
> another
> > > > > > category
> > > > > > like "cannot reasonably be used", but if the only
> reason for
> > > > > > deprecation is that the feature is going away, we
> should really
> > > > > > support it according to our promise.
> > > > > >
> > > > > > But if it's really valuable to remove these
> endpoints, maybe it's
> > > > > > appropriate to bump the major version and ask
> everybody to update
> > > > > > their clients?
> > > > > >
> > > > > > On Tue, Nov 12, 2019 at 5:27 PM Rawlin Peters <
> rawlin@apache.org>
> > > > > > wrote:
> > > > > > > The real crux of the issue here is whether or not
> we agree as a
> > > > > > > project that we can move forward with this plan
> that I believe
> > > > > > > to be a
> > > > > > > good compromise that meets most concerns but
> "breaks the
> > > > > > > Semantic
> > > > > > > Versioning promise":
> > > > > > > 1) don't rewrite deprecated endpoints from Perl to
> Go
> > > > > > > 2) in the Perl handler, add a deprecation notice
> to the
> > > > > > > "alerts"
> > > > > > > section of the response
> > > > > > > 3) once a deprecated API has been deprecated for
> one major
> > > > > > > release,
> > > > > > > delete it from the API in the subsequent major
> release (e.g. if
> > > > > > > deprecation notices are added and released in ATC
> 4.x, we can
> > > > > > > remove
> > > > > > > the deprecated endpoint in ATC 5.x)
> > > > > > >
> > > > > > > Rob, from your latest proposal to remove the ATS
> config API
> > > > > > > endpoints
> > > > > > > from 1.x, I thought we were past this issue
> already and could
> > > > > > > finally
> > > > > > > agree to remove deprecated endpoints from 1.x
> (following steps
> > > > > > > 1-3
> > > > > > > above) without requiring an all-new 2.x API. Are
> we not
> > > > > > > actually past
> > > > > > > that? You came up with valid reasons for applying
> these steps
> > > > > > > to the
> > > > > > > ATS config API endpoints, and I think it's fair to
> say that we
> > > > > > > can
> > > > > > > come up with valid reasons for applying these
> steps to other
> > > > > > > deprecated API endpoints as well.
> > > > > > >
> > > > > > > It seems like the real problem is actually
> agreeing on which
> > > > > > > _specific_ endpoints we should be able to follow
> steps 1-3 for.
> > > > > > > IMO I
> > > > > > > think it should apply to any 1.x endpoints that we
> have valid
> > > > > > > reasons
> > > > > > > for not carrying forward to Go, including but not
> limited to:
> > > > > > > - the endpoints are dangerous and pose a risk to
> the overall
> > > > > > > Traffic
> > > > > > > Control system
> > > > > > > - the endpoints have been obsoleted by other
> endpoints
> > > > > > > - the endpoints don't seem to serve a true purpose
> or don't
> > > > > > > seem to be
> > > > > > > valuable to the project anymore
> > > > > > > - the task of rewriting the endpoints from Perl to
> Go
> > > > > > > represents an
> > > > > > > unreasonable amount of work for the project
> > > > > > >
> > > > > > > Obviously most of these reasons are subjective, so
> we'd have to
> > > > > > > come
> > > > > > > to some level of consensus on deprecating
> particular endpoints
> > > > > > > first,
> > > > > > > but we should be able to follow steps 1-3 for
> whatever
> > > > > > > endpoints we
> > > > > > > decide.
> > > > > > >
> > > > > > > Making breaking changes shouldn't be taken
> lightly, but we need
> > > > > > > a path
> > > > > > > forward that allows some calculated, breaking
> changes where
> > > > > > > necessary.
> > > > > > > Otherwise, we are unnecessarily holding the
> project back.
> > > > > > >
> > > > > > > - Rawlin
> > > > > > >
> > > > > > > On Tue, Nov 12, 2019 at 11:44 AM Jeremy Mitchell <
> > > > > > > mitchell852@gmail.com>
> > > > > > wrote:
> > > > > > > > Yeah, technically a "rewrite" of 1.x means
> shifting the
> > > > > > > > implementation
> > > > > > from
> > > > > > > > Perl to Go for ALL of 1.x. If you leave a subset
> of 1.x in
> > > > > > > > Perl, then
> > > > > > you
> > > > > > > > have to call it a "partial rewrite".
> > > > > > > >
> > > > > > > > So, assuming we want to do a full rewrite (no
> more perl for
> > > > > > > > the api
> > > > > > > > whatsoever), I think that means every 1.x api
> endpoint (good
> > > > > > > > or bad)
> > > > > > needs
> > > > > > > > to be rewritten to Go so we can remove the
> dependency on
> > > > > > > > perl.
> > > > > > > >
> > > > > > > > However, I still like marking many as
> "deprecated" so we
> > > > > > > > train users
> > > > > > to use
> > > > > > > > "better" endpoints that we envision will exist
> in 2.x and
> > > > > > > > beyond.
> > > > > > > >
> > > > > > > > Jeremy
> > > > > > > >
> > > > > > > > On Tue, Nov 12, 2019 at 11:22 AM Hoppal, Michael
> <
> > > > > > Michael_Hoppal@comcast.com>
> > > > > > > > wrote:
> > > > > > > >
> > > > > > > > > We do not have a lot of deprecated routes as
> of now as we
> > > > > > > > > focused on
> > > > > > > > > routes that are more used first.
> > > > > > > > >
> > > > > > > > > I know on a lot of the remaining rewrites
> there has been
> > > > > > > > > discussion
> > > > > > on the
> > > > > > > > > matching Github issues on potential
> deprecation (I cannot
> > > > > > > > > think of
> > > > > > the
> > > > > > > > > exact count but enough to warrant this email
> __)
> > > > > > > > >
> > > > > > > > > It is key to note that I am not arguing for
> removal of
> > > > > > > > > endpoints
> > > > > > within
> > > > > > > > > the current API version but if there is an
> endpoint we
> > > > > > > > > agree should
> > > > > > be
> > > > > > > > > deprecated going forward to put a message
> within the
> > > > > > > > > response and on
> > > > > > the
> > > > > > > > > docs.
> > > > > > > > >
> > > > > > > > > And thanks for the input sounds like you are
> +1 on
> > > > > > > > > rewriting all
> > > > > > routes to
> > > > > > > > > Golang as Brennan is.
> > > > > > > > >
> > > > > > > > > On 11/12/19, 9:06 AM, "Robert O Butts" <
> rob@apache.org>
> > > > > > > > > wrote:
> > > > > > > > >
> > > > > > > > > Whether we rewrite a route in Go is an
> implementation
> > > > > > > > > detail. To
> > > > > > the
> > > > > > > > > interface, to users, it doesn't matter
> whether a route
> > > > > > > > > is
> > > > > > rewritten or
> > > > > > > > > not.
> > > > > > > > >
> > > > > > > > > But our API follows Semantic Versioning,
> in order to
> > > > > > > > > not break
> > > > > > users.
> > > > > > > > > We
> > > > > > > > > can't just remove endpoints that some of
> us don't use,
> > > > > > > > > and
> > > > > > assume other
> > > > > > > > > people don't, maybe people who never speak
> up on the
> > > > > > > > > mailing
> > > > > > list.
> > > > > > > > > We'll
> > > > > > > > > never gain a big userbase if we keep
> breaking users.
> > > > > > > > >
> > > > > > > > > Per the _project_ SemVer, once we have API
> 2.0, we can
> > > > > > > > > deprecate
> > > > > > API
> > > > > > > > > 1.x,
> > > > > > > > > and in the next major _project_ release,
> remove API
> > > > > > > > > 1.x.
> > > > > > Irrespective
> > > > > > > > > of
> > > > > > > > > Perl or Go.
> > > > > > > > >
> > > > > > > > > My big concern is, API 2.0 is a big
> project. How long
> > > > > > > > > has the
> > > > > > rewrite
> > > > > > > > > to Go
> > > > > > > > > taken? Do we really believe designing and
> implementing
> > > > > > > > > a
> > > > > > completely
> > > > > > > > > new API
> > > > > > > > > will be any less time?
> > > > > > > > >
> > > > > > > > > I don't want killing Perl to have to wait
> on that.
> > > > > > > > >
> > > > > > > > > I know it feels like a waste to rewrite
> routes that you
> > > > > > > > > don't
> > > > > > use, and
> > > > > > > > > probably few people do. But that's the
> cost of a stable
> > > > > > > > > project.
> > > > > > How
> > > > > > > > > many
> > > > > > > > > "deprecated" routes are there? If it comes
> down to
> > > > > > > > > taking the
> > > > > > > > > development
> > > > > > > > > time to rewrite them so we can kill Perl
> faster, or
> > > > > > > > > leaving Perl
> > > > > > > > > around, I
> > > > > > > > > vote we just do the work and kill Perl.
> > > > > > > > >
> > > > > > > > > >If we don't rewrite them, then Perl will
> last until
> > > > > > > > > API 2.0 has
> > > > > > been
> > > > > > > > > designed, released and then *another full
> major release
> > > > > > > > > cycle*.
> > > > > > That's
> > > > > > > > > way
> > > > > > > > > too long to have two codebases for the
> same component,
> > > > > > > > > IMO,
> > > > > > especially
> > > > > > > > > since the rewrite is already 50% complete.
> > > > > > > > >
> > > > > > > > > +1
> > > > > > > > >
> > > > > > > > >
> > > > > > > > > On Tue, Nov 12, 2019 at 8:54 AM ocket 8888
> <
> > > > > > > > > ocket8888@gmail.com>
> > > > > > > > > wrote:
> > > > > > > > >
> > > > > > > > > > I vote that by and large we DO rewrite
> them, with
> > > > > > > > > exceptions
> > > > > > for
> > > > > > > > > routes
> > > > > > > > > > that just plain don't work, even in
> Perl. Those are
> > > > > > > > > few,
> > > > > > though.
> > > > > > > > > >
> > > > > > > > > > If we don't rewrite them, then Perl will
> last until
> > > > > > > > > API 2.0
> > > > > > has been
> > > > > > > > > > designed, released and then *another
> full major
> > > > > > > > > release cycle*.
> > > > > > > > > That's way
> > > > > > > > > > too long to have two codebases for the
> same
> > > > > > > > > component, IMO,
> > > > > > > > > especially
> > > > > > > > > > since the rewrite is already 50%
> complete.
> > > > > > > > > >
> > > > > > > > > > On Tue, Nov 12, 2019 at 8:43 AM Hoppal,
> Michael <
> > > > > > > > > > Michael_Hoppal@comcast.com>
> > > > > > > > > > wrote:
> > > > > > > > > >
> > > > > > > > > > > As the Traffic Ops API is being
> rewritten from Perl
> > > > > > > > > to Golang
> > > > > > > > > there has
> > > > > > > > > > > been several routes that have been
> deprecated and
> > > > > > > > > probably
> > > > > > more to
> > > > > > > > > come.
> > > > > > > > > > >
> > > > > > > > > > > In the deprecation efforts I have seen
> two
> > > > > > > > > strategies:
> > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > > > * The route IS NOT rewritten from
> Perl to
> > > > > > > > > Golang and a
> > > > > > > > > deprecation
> > > > > > > > > > > notice is added to the alert response
> in the Perl
> > > > > > > > > handler
> > > > > > > > > > > * The route IS rewritten from Perl
> to Golang
> > > > > > > > > and a
> > > > > > deprecation
> > > > > > > > > notice
> > > > > > > > > > > is added to the alert response in the
> Golang
> > > > > > > > > handler
> > > > > > > > > > >
> > > > > > > > > > > I think we should have consistency in
> our approach
> > > > > > > > > and
> > > > > > wanted to
> > > > > > > > > get
> > > > > > > > > > > people’s thoughts.
> > > > > > > > > > >
> > > > > > > > > > > I would vote that we do not rewrite a
> deprecated
> > > > > > > > > route from
> > > > > > Perl to
> > > > > > > > > > Golang.
> > > > > > > > > > >
> > > > > > > > > > > Thanks,
> > > > > > > > > > >
> > > > > > > > > > > Michael
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > > > >
> > > > > > > > >
> > >
> > >
> >
> >
>
>
>
Re: [EXTERNAL] Re: Traffic Ops Route Deprecation Strategy
Posted by Robert O Butts <ro...@apache.org>.
>Option 1 - Graduate current API from 1.x to 2.x - allowing us to leave
behind routes in 1.x that are dangerous or deprecated
>Option 2 - Rewrite all endpoints from Perl to Golang in current version as
we are doing now.
+1 on either Option 1 or Option 2, without a strong preference. Option 1
has advantages of letting us also safely remove the Cache Config endpoints
and implement Layered Profiles. But Option 2 also has the advantage of not
requiring users to upgrade.
> Option 3 - Only rewrite endpoints from Perl to Golang in current version
that we agree are not dangerous or should deprecate
-1. As before, it breaks users unpredictably.
>It will literally outright break clients (*all* of them, not just *some*)
This is technically correct: technically, new versions 'break' all users.
The difference is, it does it in a predictable, safe way that users can
handle, inspect, and safely upgrade.
>Can someone please explain to me why (A) artificially graduating the
current 1.x API to 2.x, and removing the deprecated endpoints from 1.x, is
better for API clients overall than (B) just removing deprecated endpoints
from 1.x? Existing 1.x clients would gain absolutely nothing from being
forced to update to 2.x like this.
>for zero gain in value
Users would gain something. There is a very specific gain.
The gain is that the breakage is predictable. People frequently upgrade
minor versions automatically, either running commands without thinking, or
even automating the entire process.
For TC, this applies to both CDN operators, and their end-users.
When the minor versions contain breaking changes, that automatic upgrade
then breaks. In this case, any scripts or tools they have using the removed
endpoints will break. If those scripts are important, this could bring down
their CDN, or break their Delivery Service, or otherwise cause an outage.
Contrawise, users expect breaking changes in Major versions, and don't
typically automatically upgrade them. Rather, they'll manually inspect the
update, do testing, and check the changelog for breaking changes.
It's not a technical thing, it's a human thing. Humans automatically
upgrade minor versions, and write scripts to do so, and dependency files to
allow it, because the software industry taught people that minor versions
shouldn't break anything.
If we start breaking minor versions, initially users will experience
breakage. Not all users, and not all catastrophic. But some users will
experience catastrophic outages. And it will be our fault as a project. And
after enough times, they'll fork TC, or find another product, or write
their own; and stop participating in the community, or never start.
Then, once they've learned that we can't be trusted not to break things,
they'll experience the 2nd problem: where nothing can be safely upgraded,
and the smallest change requires long and expensive testing to verify a
single minor version or even patch upgrade, and still risk missing
something and bringing their CDN down. To be honest, we're pretty much
already here. It already costs our dev and ops teams unbelievable amounts
of time and person-hours of testing for every minor upgrade. That time far
exceeds the development time to just rewrite the small endpoints we don't
like anymore (and tangentially, also the time to write the Integration and
E2E tests TC desperately needs).
ATC has so much potential. There aren't many OSS CDNs out there, and
there's certainly the demand. We have the potential to become a huge
project, which for us existing users, translates into potentially an
enormous amount of free development. But we'll never get there if we keep
breaking things.
And for my particular organization (Comcast), we have a hard enough time
upgrading. Our upgrades, especially Traffic Ops, are extremely painful, and
require enormous amounts of manual testing. Even then, we frequently have
issues that require rollback, sometimes even an outage; even after those
unbelievably amounts of developer and operator time testing. We desperately
need a system that is safer and easier to upgrade. Breaking changes are
always going to require more testing; but we can at least make minor
versions safe to upgrade.
> I know I am changing my initial stance but I am +1 on rewriting all the
Perl endpoints to Golang and not breaking the API contract we have in
place. Most of the API routes to rewrite are rather simple to port and I
think would be faster than getting consensus on actually deprecating it
from the mailing list.
I'm inclined to agree with this. The things I've seen marked "deprecated"
seem to mostly be small, and look like they'd be pretty easy to rewrite.
But I haven't seen the whole list. Is there a list somewhere, so the
community can see it to better decide?
On Tue, Nov 19, 2019 at 7:24 AM Hoppal, Michael <Mi...@comcast.com>
wrote:
> I would love to drive this to some type of consensus on rewrite strategy
> as we have a good amount of resources (and growing) working on the rewrite
> and I want to keep that momentum.
>
> Sounds like there is three options from the email thread (please add any I
> missed)
>
> Option 1 - Graduate current API from 1.x to 2.x - allowing us to leave
> behind routes in 1.x that are dangerous or deprecated
> Option 2 - Rewrite all endpoints from Perl to Golang in current version as
> we are doing now.
> Option 3 - Only rewrite endpoints from Perl to Golang in current version
> that we agree are not dangerous or should deprecate
>
> I am +1 for option 2 (which I know is different from my original stance).
>
> Which direction would others like to go?
>
> Thanks,
>
> Michael
>
>
> I know I am changing my initial stance but I am +1 on rewriting all the
> Perl endpoints to Golang and not breaking the API contract we have in
> place. Most of the API routes to rewrite are rather simple to port and I
> think would be faster than getting consensus on actually deprecating it
> from the mailing list.
>
> On 11/16/19, 3:31 PM, "Rawlin Peters" <ra...@apache.org> wrote:
>
> > Moving to 2.x means that it's my fault as a consumer of the API if I
> was still using the deprecated endpoint (and that change should be part of
> a larger upgrade plan). Simply removing the deprecated routes,
> retroactively, means I can't rely on having a transition period to get to
> 2.0 outside of a large completely down upgrade window.
>
> There would be a transition period either way. Endpoints would be
> deprecated in ATC 4.0 and removed in 5.0. The release notes for 4.0
> would clearly list out the deprecated endpoints. If they have
> replacements, there would be instructions on how to transition to the
> replacement. Deprecated endpoints would return alerts. After upgrading
> to 4.0, operators should be well aware that certain endpoints are
> being removed in 5.0, and it's on them to start using new endpoints,
> change workflows, etc.
>
> By forcing operators to upgrade all of their clients to start using
> API 2.x, it just adds to their burden of upgrading when that time
> comes. Rather than surgically fixing the clients of the 2% of
> deprecated API endpoints before upgrading to ATC 5.0, operators would
> have to fix every single client they have to use API 2.x before
> upgrading to 5.0. Is that exercise really worth it? Most people would
> probably be blissfully unaware that the "POST /api/1.1/capabilities"
> endpoint disappeared (among other deprecated endpoints). Less footguns
> in the system, and they didn't even have to update their clients.
>
> Anyways, circling back to Michael's original question, it sounds like
> either option we choose, we at least have consensus on this
> deprecation strategy:
> - The route IS NOT rewritten from Perl to Golang and a deprecation
> notice is added to the alert response in the Perl handler
>
> If we're content to force all API clients to update their clients to
> 2.x for zero gain in value, at least we won't be wasting work by
> rewriting deprecated Perl endpoints to Go.
>
> - Rawlin
>
> On Fri, Nov 15, 2019 at 7:06 PM Gray, Jonathan
> <Jo...@comcast.com> wrote:
> >
> > Moving to 2.x means that it's my fault as a consumer of the API if I
> was still using the deprecated endpoint (and that change should be part of
> a larger upgrade plan). Simply removing the deprecated routes,
> retroactively, means I can't rely on having a transition period to get to
> 2.0 outside of a large completely down upgrade window. Depending on how
> the API is used, even in our own components such as TM, that could prove
> disastrous. For the older minor revs like 1.1/1.2, I could live with
> dropping those as part of a major project release (including upgrade
> instructions outlining what releases you must be on beforehand). Currently
> 1.3 I view as stable enough to code to or use a native client library from,
> and 1.4 is my risk/overhead to take for new stuff as it's still evolving
> and improving. Going to 1.5 or 2.0 moves that forward. The path of least
> surprise for humans is to expect big shifts on those major revs. Removal
> falls under that umbrella and gets as a consumer not to strictly focus on
> endpoint->endpoint moves, but rather workflow shifts and maybe now I have
> to use the API a different way than before. I believe getting this right
> is something that would greatly help us understand does X component at
> version 1 work with Y component at version 2, because now you're asking do
> components X and Y both speak protocol A. With major releases we can work
> through that each time, but it's error prone and requires experts in all
> products to agree upon. If you do nothing but consume major releases, and
> then conduct your own impact evaluation having read the upgrade
> instructions thoroughly, maybe you'd be able to smoothly transition.
> >
> > Jonathan G
> >
> >
> > On 11/15/19, 5:24 PM, "Rawlin Peters" <ra...@apache.org> wrote:
> >
> > Can someone please explain to me why (A) artificially graduating
> the
> > current 1.x API to 2.x, and removing the deprecated endpoints
> from
> > 1.x, is better for API clients overall than (B) just removing
> > deprecated endpoints from 1.x? Existing 1.x clients would gain
> > absolutely nothing from being forced to update to 2.x like this.
> >
> > The way I see the two choices:
> > A) break every single API client out there
> > B) break only clients of APIs that we've decided upon as a
> community
> > to be deprecated
> >
> > IMO, when we have to break things, we should try break the least
> > amount of clients possible. All I'm looking for is a valuable
> reason
> > to choose option A over option B, that isn't just "because
> SemVer".
> >
> > - Rawlin
> >
> > On Wed, Nov 13, 2019 at 3:24 PM Schmidt, Andrew (Contractor)
> > <An...@comcast.com> wrote:
> > >
> > > If we decide to get rid of the API versioning it makes sense
> to me that we would do that on a major version upgrade, probably to the ATC
> version. But I think it would be too early to do it on the 2.0 API version.
> I like this compromise +1. I was wondering why we were stuck on the idea
> that 2.0 had to be an API rewrite. That seems like it’s a long way out. 2.0
> = deprecations and fixes. 3.0 or 5.0 = api rewrite and maybe version
> convergence.
> > >
> > > On 11/13/19, 11:52 AM, "ocket8888@gmail.com" <
> ocket8888@gmail.com> wrote:
> > >
> > > Lemme lead with: making breaking changes on a major
> version boundary
> > > gets +1 from me - no reason the API needs to be fully
> redesigned for
> > > any particular number.
> > >
> > > But I just gotta bring this up again: if we want to be
> able to
> > > deprecate an endpoint for a major release and remove it in
> the next
> > > without needing to deal with the messiness of API
> versions, we could
> > > always just get rid of API versions. Major ATC revisions
> should be
> > > easier to come by with our new release strategy.
> > >
> > > On Wed, 2019-11-13 at 10:23 -0700, Rawlin Peters wrote:
> > > > > It is a compromise - it's an inconvenience to users,
> forcing them
> > > > > to fix
> > > > scripts to upgrade. Which isn't good for user experience
> either; but
> > > > it
> > > > doesn't outright break people. It's a compromise I'm
> willing to live
> > > > with.
> > > >
> > > > It will literally outright break clients (*all* of them,
> not just
> > > > *some*). We have been down this road before, so I'll
> just repeat what
> > > > I said when this exact conversation came up months
> ago[1]:
> > > >
> > > > "So it basically boils down to:
> > > > A) promote Go 1.x routes to 2.x and eventually break ALL
> users of the
> > > > 1.x API when the time comes to remove 1.x
> > > > B) deprecate and remove some 1.x Perl routes and
> POSSIBLY break SOME
> > > > users"
> > > >
> > > > Either way we slice and dice it with versioning,
> removing an endpoint
> > > > means that clients of that endpoint will break. By
> graduating only
> > > > the
> > > > "good stuff" from 1.x to 2.x, and removing endpoints
> from 1.x, we
> > > > will
> > > > unnecessarily break all 1.x clients when the time comes
> to remove the
> > > > rest of 1.x. Instead, we can choose to break only
> clients of 1.x APIs
> > > > that we already deemed dangerous, unnecessary, useless,
> etc. enough
> > > > to
> > > > deprecate.
> > > >
> > > > Why break 100% of our API clients when we can choose to
> break only a
> > > > small number of clients that happen to use
> > > > dangerous/unnecessary/useless APIs? Why would we break
> our API
> > > > clients
> > > > unnecessarily for no added value? There is no sense in
> blindly
> > > > adhering to Semantic Versioning just for the sake of
> adhering to
> > > > Semantic Versioning. Instead, use it as a guideline, and
> break the
> > > > rules where it makes the most sense to do so.
> > > >
> > > > - Rawlin
> > > >
> > > > [1]
> > > >
> https://urldefense.com/v3/__https://lists.apache.org/thread.html/15c6bfcdce0ac00a3a8612a96c3d668a8a5a16c6ca9401de9473d31c@*3Cdev.trafficcontrol.apache.org*3E__;JSU!rx_L75ITgOQ!V_b3E9iEYqNlZus18f-sVuofwuyxapCthg3LdO4NxyMCU39b7mdf4klP5KNR7XZ7d3Uc$
> > > >
> > > > On Wed, Nov 13, 2019 at 9:43 AM Robert O Butts <
> rob@apache.org>
> > > > wrote:
> > > > > > But if it's really valuable to remove these
> endpoints, maybe it's
> > > > > appropriate to bump the major version and ask
> everybody to update
> > > > > their
> > > > > clients?
> > > > >
> > > > > +1
> > > > >
> > > > > This sounds like the best compromise to me. It
> addresses both sides
> > > > > - it
> > > > > lets us remove endpoints, and it also prevents
> breaking users.
> > > > >
> > > > > It is a compromise - it's an inconvenience to users,
> forcing them
> > > > > to fix
> > > > > scripts to upgrade. Which isn't good for user
> experience either;
> > > > > but it
> > > > > doesn't outright break people. It's a compromise I'm
> willing to
> > > > > live with.
> > > > >
> > > > > This actually lets us do at least 3 things:
> > > > > 1. Remove "deprecated" endpoints.
> > > > > 2. Remove the unmaintained dangerously-large Cache
> Config
> > > > > endpoints.
> > > > > 3. Lets us do Layered Profiles, changing Servers and
> DSes to have
> > > > > arrays of
> > > > > Profiles instead of a single Profile, without hacky
> backward-
> > > > > compatibility
> > > > > attempts.
> > > > >
> > > > > To clarify, `/api/2.0` would be essentially the
> current API, just
> > > > > with the
> > > > > above breaking changes (possibly others) which require
> a SemVer
> > > > > major
> > > > > version increase. And the big API rewrite/redesign
> we've been
> > > > > calling "2.0"
> > > > > would become `/api/3.0` or some future version.
> > > > >
> > > > > It's not perfect, but it seems to address the most
> concerns on all
> > > > > sides.
> > > > > Thoughts? Objections?
> > > > >
> > > > >
> > > > > On Wed, Nov 13, 2019 at 8:45 AM Chris Lemmons <
> alficles@gmail.com>
> > > > > wrote:
> > > > >
> > > > > > For the endpoints, I agree in part and disagree in
> part.
> > > > > >
> > > > > > Agree for these two:
> > > > > > - the endpoints are dangerous and pose a risk to the
> overall
> > > > > > Traffic
> > > > > > Control system
> > > > > > - the task of rewriting the endpoints from Perl to
> Go represents
> > > > > > an
> > > > > > unreasonable amount of work for the project
> > > > > >
> > > > > > And I would add:
> > > > > > - the endpoint does not serve its intended purpose
> and cannot
> > > > > > reasonably
> > > > > > be used
> > > > > > - the endpoint has been made unsafe or unusable by
> new features
> > > > > > added to TC
> > > > > >
> > > > > > Disagree for these two:
> > > > > > - the endpoints have been obsoleted by other
> endpoints
> > > > > > - the endpoints don't seem to serve a true purpose
> or don't seem
> > > > > > to be
> > > > > > valuable to the project anymore
> > > > > >
> > > > > > For the most part, obsoleted endpoints can be
> maintained with a
> > > > > > shim
> > > > > > into the new code. If they would require "an
> unreasonable amount
> > > > > > of
> > > > > > work", it might make sense to remove them, then.
> > > > > >
> > > > > > Things that don't seem valuable usually fall into
> another
> > > > > > category
> > > > > > like "cannot reasonably be used", but if the only
> reason for
> > > > > > deprecation is that the feature is going away, we
> should really
> > > > > > support it according to our promise.
> > > > > >
> > > > > > But if it's really valuable to remove these
> endpoints, maybe it's
> > > > > > appropriate to bump the major version and ask
> everybody to update
> > > > > > their clients?
> > > > > >
> > > > > > On Tue, Nov 12, 2019 at 5:27 PM Rawlin Peters <
> rawlin@apache.org>
> > > > > > wrote:
> > > > > > > The real crux of the issue here is whether or not
> we agree as a
> > > > > > > project that we can move forward with this plan
> that I believe
> > > > > > > to be a
> > > > > > > good compromise that meets most concerns but
> "breaks the
> > > > > > > Semantic
> > > > > > > Versioning promise":
> > > > > > > 1) don't rewrite deprecated endpoints from Perl to
> Go
> > > > > > > 2) in the Perl handler, add a deprecation notice
> to the
> > > > > > > "alerts"
> > > > > > > section of the response
> > > > > > > 3) once a deprecated API has been deprecated for
> one major
> > > > > > > release,
> > > > > > > delete it from the API in the subsequent major
> release (e.g. if
> > > > > > > deprecation notices are added and released in ATC
> 4.x, we can
> > > > > > > remove
> > > > > > > the deprecated endpoint in ATC 5.x)
> > > > > > >
> > > > > > > Rob, from your latest proposal to remove the ATS
> config API
> > > > > > > endpoints
> > > > > > > from 1.x, I thought we were past this issue
> already and could
> > > > > > > finally
> > > > > > > agree to remove deprecated endpoints from 1.x
> (following steps
> > > > > > > 1-3
> > > > > > > above) without requiring an all-new 2.x API. Are
> we not
> > > > > > > actually past
> > > > > > > that? You came up with valid reasons for applying
> these steps
> > > > > > > to the
> > > > > > > ATS config API endpoints, and I think it's fair to
> say that we
> > > > > > > can
> > > > > > > come up with valid reasons for applying these
> steps to other
> > > > > > > deprecated API endpoints as well.
> > > > > > >
> > > > > > > It seems like the real problem is actually
> agreeing on which
> > > > > > > _specific_ endpoints we should be able to follow
> steps 1-3 for.
> > > > > > > IMO I
> > > > > > > think it should apply to any 1.x endpoints that we
> have valid
> > > > > > > reasons
> > > > > > > for not carrying forward to Go, including but not
> limited to:
> > > > > > > - the endpoints are dangerous and pose a risk to
> the overall
> > > > > > > Traffic
> > > > > > > Control system
> > > > > > > - the endpoints have been obsoleted by other
> endpoints
> > > > > > > - the endpoints don't seem to serve a true purpose
> or don't
> > > > > > > seem to be
> > > > > > > valuable to the project anymore
> > > > > > > - the task of rewriting the endpoints from Perl to
> Go
> > > > > > > represents an
> > > > > > > unreasonable amount of work for the project
> > > > > > >
> > > > > > > Obviously most of these reasons are subjective, so
> we'd have to
> > > > > > > come
> > > > > > > to some level of consensus on deprecating
> particular endpoints
> > > > > > > first,
> > > > > > > but we should be able to follow steps 1-3 for
> whatever
> > > > > > > endpoints we
> > > > > > > decide.
> > > > > > >
> > > > > > > Making breaking changes shouldn't be taken
> lightly, but we need
> > > > > > > a path
> > > > > > > forward that allows some calculated, breaking
> changes where
> > > > > > > necessary.
> > > > > > > Otherwise, we are unnecessarily holding the
> project back.
> > > > > > >
> > > > > > > - Rawlin
> > > > > > >
> > > > > > > On Tue, Nov 12, 2019 at 11:44 AM Jeremy Mitchell <
> > > > > > > mitchell852@gmail.com>
> > > > > > wrote:
> > > > > > > > Yeah, technically a "rewrite" of 1.x means
> shifting the
> > > > > > > > implementation
> > > > > > from
> > > > > > > > Perl to Go for ALL of 1.x. If you leave a subset
> of 1.x in
> > > > > > > > Perl, then
> > > > > > you
> > > > > > > > have to call it a "partial rewrite".
> > > > > > > >
> > > > > > > > So, assuming we want to do a full rewrite (no
> more perl for
> > > > > > > > the api
> > > > > > > > whatsoever), I think that means every 1.x api
> endpoint (good
> > > > > > > > or bad)
> > > > > > needs
> > > > > > > > to be rewritten to Go so we can remove the
> dependency on
> > > > > > > > perl.
> > > > > > > >
> > > > > > > > However, I still like marking many as
> "deprecated" so we
> > > > > > > > train users
> > > > > > to use
> > > > > > > > "better" endpoints that we envision will exist
> in 2.x and
> > > > > > > > beyond.
> > > > > > > >
> > > > > > > > Jeremy
> > > > > > > >
> > > > > > > > On Tue, Nov 12, 2019 at 11:22 AM Hoppal, Michael
> <
> > > > > > Michael_Hoppal@comcast.com>
> > > > > > > > wrote:
> > > > > > > >
> > > > > > > > > We do not have a lot of deprecated routes as
> of now as we
> > > > > > > > > focused on
> > > > > > > > > routes that are more used first.
> > > > > > > > >
> > > > > > > > > I know on a lot of the remaining rewrites
> there has been
> > > > > > > > > discussion
> > > > > > on the
> > > > > > > > > matching Github issues on potential
> deprecation (I cannot
> > > > > > > > > think of
> > > > > > the
> > > > > > > > > exact count but enough to warrant this email
> __)
> > > > > > > > >
> > > > > > > > > It is key to note that I am not arguing for
> removal of
> > > > > > > > > endpoints
> > > > > > within
> > > > > > > > > the current API version but if there is an
> endpoint we
> > > > > > > > > agree should
> > > > > > be
> > > > > > > > > deprecated going forward to put a message
> within the
> > > > > > > > > response and on
> > > > > > the
> > > > > > > > > docs.
> > > > > > > > >
> > > > > > > > > And thanks for the input sounds like you are
> +1 on
> > > > > > > > > rewriting all
> > > > > > routes to
> > > > > > > > > Golang as Brennan is.
> > > > > > > > >
> > > > > > > > > On 11/12/19, 9:06 AM, "Robert O Butts" <
> rob@apache.org>
> > > > > > > > > wrote:
> > > > > > > > >
> > > > > > > > > Whether we rewrite a route in Go is an
> implementation
> > > > > > > > > detail. To
> > > > > > the
> > > > > > > > > interface, to users, it doesn't matter
> whether a route
> > > > > > > > > is
> > > > > > rewritten or
> > > > > > > > > not.
> > > > > > > > >
> > > > > > > > > But our API follows Semantic Versioning,
> in order to
> > > > > > > > > not break
> > > > > > users.
> > > > > > > > > We
> > > > > > > > > can't just remove endpoints that some of
> us don't use,
> > > > > > > > > and
> > > > > > assume other
> > > > > > > > > people don't, maybe people who never speak
> up on the
> > > > > > > > > mailing
> > > > > > list.
> > > > > > > > > We'll
> > > > > > > > > never gain a big userbase if we keep
> breaking users.
> > > > > > > > >
> > > > > > > > > Per the _project_ SemVer, once we have API
> 2.0, we can
> > > > > > > > > deprecate
> > > > > > API
> > > > > > > > > 1.x,
> > > > > > > > > and in the next major _project_ release,
> remove API
> > > > > > > > > 1.x.
> > > > > > Irrespective
> > > > > > > > > of
> > > > > > > > > Perl or Go.
> > > > > > > > >
> > > > > > > > > My big concern is, API 2.0 is a big
> project. How long
> > > > > > > > > has the
> > > > > > rewrite
> > > > > > > > > to Go
> > > > > > > > > taken? Do we really believe designing and
> implementing
> > > > > > > > > a
> > > > > > completely
> > > > > > > > > new API
> > > > > > > > > will be any less time?
> > > > > > > > >
> > > > > > > > > I don't want killing Perl to have to wait
> on that.
> > > > > > > > >
> > > > > > > > > I know it feels like a waste to rewrite
> routes that you
> > > > > > > > > don't
> > > > > > use, and
> > > > > > > > > probably few people do. But that's the
> cost of a stable
> > > > > > > > > project.
> > > > > > How
> > > > > > > > > many
> > > > > > > > > "deprecated" routes are there? If it comes
> down to
> > > > > > > > > taking the
> > > > > > > > > development
> > > > > > > > > time to rewrite them so we can kill Perl
> faster, or
> > > > > > > > > leaving Perl
> > > > > > > > > around, I
> > > > > > > > > vote we just do the work and kill Perl.
> > > > > > > > >
> > > > > > > > > >If we don't rewrite them, then Perl will
> last until
> > > > > > > > > API 2.0 has
> > > > > > been
> > > > > > > > > designed, released and then *another full
> major release
> > > > > > > > > cycle*.
> > > > > > That's
> > > > > > > > > way
> > > > > > > > > too long to have two codebases for the
> same component,
> > > > > > > > > IMO,
> > > > > > especially
> > > > > > > > > since the rewrite is already 50% complete.
> > > > > > > > >
> > > > > > > > > +1
> > > > > > > > >
> > > > > > > > >
> > > > > > > > > On Tue, Nov 12, 2019 at 8:54 AM ocket 8888
> <
> > > > > > > > > ocket8888@gmail.com>
> > > > > > > > > wrote:
> > > > > > > > >
> > > > > > > > > > I vote that by and large we DO rewrite
> them, with
> > > > > > > > > exceptions
> > > > > > for
> > > > > > > > > routes
> > > > > > > > > > that just plain don't work, even in
> Perl. Those are
> > > > > > > > > few,
> > > > > > though.
> > > > > > > > > >
> > > > > > > > > > If we don't rewrite them, then Perl will
> last until
> > > > > > > > > API 2.0
> > > > > > has been
> > > > > > > > > > designed, released and then *another
> full major
> > > > > > > > > release cycle*.
> > > > > > > > > That's way
> > > > > > > > > > too long to have two codebases for the
> same
> > > > > > > > > component, IMO,
> > > > > > > > > especially
> > > > > > > > > > since the rewrite is already 50%
> complete.
> > > > > > > > > >
> > > > > > > > > > On Tue, Nov 12, 2019 at 8:43 AM Hoppal,
> Michael <
> > > > > > > > > > Michael_Hoppal@comcast.com>
> > > > > > > > > > wrote:
> > > > > > > > > >
> > > > > > > > > > > As the Traffic Ops API is being
> rewritten from Perl
> > > > > > > > > to Golang
> > > > > > > > > there has
> > > > > > > > > > > been several routes that have been
> deprecated and
> > > > > > > > > probably
> > > > > > more to
> > > > > > > > > come.
> > > > > > > > > > >
> > > > > > > > > > > In the deprecation efforts I have seen
> two
> > > > > > > > > strategies:
> > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > > > * The route IS NOT rewritten from
> Perl to
> > > > > > > > > Golang and a
> > > > > > > > > deprecation
> > > > > > > > > > > notice is added to the alert response
> in the Perl
> > > > > > > > > handler
> > > > > > > > > > > * The route IS rewritten from Perl
> to Golang
> > > > > > > > > and a
> > > > > > deprecation
> > > > > > > > > notice
> > > > > > > > > > > is added to the alert response in the
> Golang
> > > > > > > > > handler
> > > > > > > > > > >
> > > > > > > > > > > I think we should have consistency in
> our approach
> > > > > > > > > and
> > > > > > wanted to
> > > > > > > > > get
> > > > > > > > > > > people’s thoughts.
> > > > > > > > > > >
> > > > > > > > > > > I would vote that we do not rewrite a
> deprecated
> > > > > > > > > route from
> > > > > > Perl to
> > > > > > > > > > Golang.
> > > > > > > > > > >
> > > > > > > > > > > Thanks,
> > > > > > > > > > >
> > > > > > > > > > > Michael
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > > > >
> > > > > > > > >
> > >
> > >
> >
> >
>
>
>
Re: [EXTERNAL] Re: Traffic Ops Route Deprecation Strategy
Posted by "Hoppal, Michael" <Mi...@comcast.com>.
I would love to drive this to some type of consensus on rewrite strategy as we have a good amount of resources (and growing) working on the rewrite and I want to keep that momentum.
Sounds like there is three options from the email thread (please add any I missed)
Option 1 - Graduate current API from 1.x to 2.x - allowing us to leave behind routes in 1.x that are dangerous or deprecated
Option 2 - Rewrite all endpoints from Perl to Golang in current version as we are doing now.
Option 3 - Only rewrite endpoints from Perl to Golang in current version that we agree are not dangerous or should deprecate
I am +1 for option 2 (which I know is different from my original stance).
Which direction would others like to go?
Thanks,
Michael
I know I am changing my initial stance but I am +1 on rewriting all the Perl endpoints to Golang and not breaking the API contract we have in place. Most of the API routes to rewrite are rather simple to port and I think would be faster than getting consensus on actually deprecating it from the mailing list.
On 11/16/19, 3:31 PM, "Rawlin Peters" <ra...@apache.org> wrote:
> Moving to 2.x means that it's my fault as a consumer of the API if I was still using the deprecated endpoint (and that change should be part of a larger upgrade plan). Simply removing the deprecated routes, retroactively, means I can't rely on having a transition period to get to 2.0 outside of a large completely down upgrade window.
There would be a transition period either way. Endpoints would be
deprecated in ATC 4.0 and removed in 5.0. The release notes for 4.0
would clearly list out the deprecated endpoints. If they have
replacements, there would be instructions on how to transition to the
replacement. Deprecated endpoints would return alerts. After upgrading
to 4.0, operators should be well aware that certain endpoints are
being removed in 5.0, and it's on them to start using new endpoints,
change workflows, etc.
By forcing operators to upgrade all of their clients to start using
API 2.x, it just adds to their burden of upgrading when that time
comes. Rather than surgically fixing the clients of the 2% of
deprecated API endpoints before upgrading to ATC 5.0, operators would
have to fix every single client they have to use API 2.x before
upgrading to 5.0. Is that exercise really worth it? Most people would
probably be blissfully unaware that the "POST /api/1.1/capabilities"
endpoint disappeared (among other deprecated endpoints). Less footguns
in the system, and they didn't even have to update their clients.
Anyways, circling back to Michael's original question, it sounds like
either option we choose, we at least have consensus on this
deprecation strategy:
- The route IS NOT rewritten from Perl to Golang and a deprecation
notice is added to the alert response in the Perl handler
If we're content to force all API clients to update their clients to
2.x for zero gain in value, at least we won't be wasting work by
rewriting deprecated Perl endpoints to Go.
- Rawlin
On Fri, Nov 15, 2019 at 7:06 PM Gray, Jonathan
<Jo...@comcast.com> wrote:
>
> Moving to 2.x means that it's my fault as a consumer of the API if I was still using the deprecated endpoint (and that change should be part of a larger upgrade plan). Simply removing the deprecated routes, retroactively, means I can't rely on having a transition period to get to 2.0 outside of a large completely down upgrade window. Depending on how the API is used, even in our own components such as TM, that could prove disastrous. For the older minor revs like 1.1/1.2, I could live with dropping those as part of a major project release (including upgrade instructions outlining what releases you must be on beforehand). Currently 1.3 I view as stable enough to code to or use a native client library from, and 1.4 is my risk/overhead to take for new stuff as it's still evolving and improving. Going to 1.5 or 2.0 moves that forward. The path of least surprise for humans is to expect big shifts on those major revs. Removal falls under that umbrella and gets as a consumer not to strictly focus on endpoint->endpoint moves, but rather workflow shifts and maybe now I have to use the API a different way than before. I believe getting this right is something that would greatly help us understand does X component at version 1 work with Y component at version 2, because now you're asking do components X and Y both speak protocol A. With major releases we can work through that each time, but it's error prone and requires experts in all products to agree upon. If you do nothing but consume major releases, and then conduct your own impact evaluation having read the upgrade instructions thoroughly, maybe you'd be able to smoothly transition.
>
> Jonathan G
>
>
> On 11/15/19, 5:24 PM, "Rawlin Peters" <ra...@apache.org> wrote:
>
> Can someone please explain to me why (A) artificially graduating the
> current 1.x API to 2.x, and removing the deprecated endpoints from
> 1.x, is better for API clients overall than (B) just removing
> deprecated endpoints from 1.x? Existing 1.x clients would gain
> absolutely nothing from being forced to update to 2.x like this.
>
> The way I see the two choices:
> A) break every single API client out there
> B) break only clients of APIs that we've decided upon as a community
> to be deprecated
>
> IMO, when we have to break things, we should try break the least
> amount of clients possible. All I'm looking for is a valuable reason
> to choose option A over option B, that isn't just "because SemVer".
>
> - Rawlin
>
> On Wed, Nov 13, 2019 at 3:24 PM Schmidt, Andrew (Contractor)
> <An...@comcast.com> wrote:
> >
> > If we decide to get rid of the API versioning it makes sense to me that we would do that on a major version upgrade, probably to the ATC version. But I think it would be too early to do it on the 2.0 API version. I like this compromise +1. I was wondering why we were stuck on the idea that 2.0 had to be an API rewrite. That seems like it’s a long way out. 2.0 = deprecations and fixes. 3.0 or 5.0 = api rewrite and maybe version convergence.
> >
> > On 11/13/19, 11:52 AM, "ocket8888@gmail.com" <oc...@gmail.com> wrote:
> >
> > Lemme lead with: making breaking changes on a major version boundary
> > gets +1 from me - no reason the API needs to be fully redesigned for
> > any particular number.
> >
> > But I just gotta bring this up again: if we want to be able to
> > deprecate an endpoint for a major release and remove it in the next
> > without needing to deal with the messiness of API versions, we could
> > always just get rid of API versions. Major ATC revisions should be
> > easier to come by with our new release strategy.
> >
> > On Wed, 2019-11-13 at 10:23 -0700, Rawlin Peters wrote:
> > > > It is a compromise - it's an inconvenience to users, forcing them
> > > > to fix
> > > scripts to upgrade. Which isn't good for user experience either; but
> > > it
> > > doesn't outright break people. It's a compromise I'm willing to live
> > > with.
> > >
> > > It will literally outright break clients (*all* of them, not just
> > > *some*). We have been down this road before, so I'll just repeat what
> > > I said when this exact conversation came up months ago[1]:
> > >
> > > "So it basically boils down to:
> > > A) promote Go 1.x routes to 2.x and eventually break ALL users of the
> > > 1.x API when the time comes to remove 1.x
> > > B) deprecate and remove some 1.x Perl routes and POSSIBLY break SOME
> > > users"
> > >
> > > Either way we slice and dice it with versioning, removing an endpoint
> > > means that clients of that endpoint will break. By graduating only
> > > the
> > > "good stuff" from 1.x to 2.x, and removing endpoints from 1.x, we
> > > will
> > > unnecessarily break all 1.x clients when the time comes to remove the
> > > rest of 1.x. Instead, we can choose to break only clients of 1.x APIs
> > > that we already deemed dangerous, unnecessary, useless, etc. enough
> > > to
> > > deprecate.
> > >
> > > Why break 100% of our API clients when we can choose to break only a
> > > small number of clients that happen to use
> > > dangerous/unnecessary/useless APIs? Why would we break our API
> > > clients
> > > unnecessarily for no added value? There is no sense in blindly
> > > adhering to Semantic Versioning just for the sake of adhering to
> > > Semantic Versioning. Instead, use it as a guideline, and break the
> > > rules where it makes the most sense to do so.
> > >
> > > - Rawlin
> > >
> > > [1]
> > > https://urldefense.com/v3/__https://lists.apache.org/thread.html/15c6bfcdce0ac00a3a8612a96c3d668a8a5a16c6ca9401de9473d31c@*3Cdev.trafficcontrol.apache.org*3E__;JSU!rx_L75ITgOQ!V_b3E9iEYqNlZus18f-sVuofwuyxapCthg3LdO4NxyMCU39b7mdf4klP5KNR7XZ7d3Uc$
> > >
> > > On Wed, Nov 13, 2019 at 9:43 AM Robert O Butts <ro...@apache.org>
> > > wrote:
> > > > > But if it's really valuable to remove these endpoints, maybe it's
> > > > appropriate to bump the major version and ask everybody to update
> > > > their
> > > > clients?
> > > >
> > > > +1
> > > >
> > > > This sounds like the best compromise to me. It addresses both sides
> > > > - it
> > > > lets us remove endpoints, and it also prevents breaking users.
> > > >
> > > > It is a compromise - it's an inconvenience to users, forcing them
> > > > to fix
> > > > scripts to upgrade. Which isn't good for user experience either;
> > > > but it
> > > > doesn't outright break people. It's a compromise I'm willing to
> > > > live with.
> > > >
> > > > This actually lets us do at least 3 things:
> > > > 1. Remove "deprecated" endpoints.
> > > > 2. Remove the unmaintained dangerously-large Cache Config
> > > > endpoints.
> > > > 3. Lets us do Layered Profiles, changing Servers and DSes to have
> > > > arrays of
> > > > Profiles instead of a single Profile, without hacky backward-
> > > > compatibility
> > > > attempts.
> > > >
> > > > To clarify, `/api/2.0` would be essentially the current API, just
> > > > with the
> > > > above breaking changes (possibly others) which require a SemVer
> > > > major
> > > > version increase. And the big API rewrite/redesign we've been
> > > > calling "2.0"
> > > > would become `/api/3.0` or some future version.
> > > >
> > > > It's not perfect, but it seems to address the most concerns on all
> > > > sides.
> > > > Thoughts? Objections?
> > > >
> > > >
> > > > On Wed, Nov 13, 2019 at 8:45 AM Chris Lemmons <al...@gmail.com>
> > > > wrote:
> > > >
> > > > > For the endpoints, I agree in part and disagree in part.
> > > > >
> > > > > Agree for these two:
> > > > > - the endpoints are dangerous and pose a risk to the overall
> > > > > Traffic
> > > > > Control system
> > > > > - the task of rewriting the endpoints from Perl to Go represents
> > > > > an
> > > > > unreasonable amount of work for the project
> > > > >
> > > > > And I would add:
> > > > > - the endpoint does not serve its intended purpose and cannot
> > > > > reasonably
> > > > > be used
> > > > > - the endpoint has been made unsafe or unusable by new features
> > > > > added to TC
> > > > >
> > > > > Disagree for these two:
> > > > > - the endpoints have been obsoleted by other endpoints
> > > > > - the endpoints don't seem to serve a true purpose or don't seem
> > > > > to be
> > > > > valuable to the project anymore
> > > > >
> > > > > For the most part, obsoleted endpoints can be maintained with a
> > > > > shim
> > > > > into the new code. If they would require "an unreasonable amount
> > > > > of
> > > > > work", it might make sense to remove them, then.
> > > > >
> > > > > Things that don't seem valuable usually fall into another
> > > > > category
> > > > > like "cannot reasonably be used", but if the only reason for
> > > > > deprecation is that the feature is going away, we should really
> > > > > support it according to our promise.
> > > > >
> > > > > But if it's really valuable to remove these endpoints, maybe it's
> > > > > appropriate to bump the major version and ask everybody to update
> > > > > their clients?
> > > > >
> > > > > On Tue, Nov 12, 2019 at 5:27 PM Rawlin Peters <ra...@apache.org>
> > > > > wrote:
> > > > > > The real crux of the issue here is whether or not we agree as a
> > > > > > project that we can move forward with this plan that I believe
> > > > > > to be a
> > > > > > good compromise that meets most concerns but "breaks the
> > > > > > Semantic
> > > > > > Versioning promise":
> > > > > > 1) don't rewrite deprecated endpoints from Perl to Go
> > > > > > 2) in the Perl handler, add a deprecation notice to the
> > > > > > "alerts"
> > > > > > section of the response
> > > > > > 3) once a deprecated API has been deprecated for one major
> > > > > > release,
> > > > > > delete it from the API in the subsequent major release (e.g. if
> > > > > > deprecation notices are added and released in ATC 4.x, we can
> > > > > > remove
> > > > > > the deprecated endpoint in ATC 5.x)
> > > > > >
> > > > > > Rob, from your latest proposal to remove the ATS config API
> > > > > > endpoints
> > > > > > from 1.x, I thought we were past this issue already and could
> > > > > > finally
> > > > > > agree to remove deprecated endpoints from 1.x (following steps
> > > > > > 1-3
> > > > > > above) without requiring an all-new 2.x API. Are we not
> > > > > > actually past
> > > > > > that? You came up with valid reasons for applying these steps
> > > > > > to the
> > > > > > ATS config API endpoints, and I think it's fair to say that we
> > > > > > can
> > > > > > come up with valid reasons for applying these steps to other
> > > > > > deprecated API endpoints as well.
> > > > > >
> > > > > > It seems like the real problem is actually agreeing on which
> > > > > > _specific_ endpoints we should be able to follow steps 1-3 for.
> > > > > > IMO I
> > > > > > think it should apply to any 1.x endpoints that we have valid
> > > > > > reasons
> > > > > > for not carrying forward to Go, including but not limited to:
> > > > > > - the endpoints are dangerous and pose a risk to the overall
> > > > > > Traffic
> > > > > > Control system
> > > > > > - the endpoints have been obsoleted by other endpoints
> > > > > > - the endpoints don't seem to serve a true purpose or don't
> > > > > > seem to be
> > > > > > valuable to the project anymore
> > > > > > - the task of rewriting the endpoints from Perl to Go
> > > > > > represents an
> > > > > > unreasonable amount of work for the project
> > > > > >
> > > > > > Obviously most of these reasons are subjective, so we'd have to
> > > > > > come
> > > > > > to some level of consensus on deprecating particular endpoints
> > > > > > first,
> > > > > > but we should be able to follow steps 1-3 for whatever
> > > > > > endpoints we
> > > > > > decide.
> > > > > >
> > > > > > Making breaking changes shouldn't be taken lightly, but we need
> > > > > > a path
> > > > > > forward that allows some calculated, breaking changes where
> > > > > > necessary.
> > > > > > Otherwise, we are unnecessarily holding the project back.
> > > > > >
> > > > > > - Rawlin
> > > > > >
> > > > > > On Tue, Nov 12, 2019 at 11:44 AM Jeremy Mitchell <
> > > > > > mitchell852@gmail.com>
> > > > > wrote:
> > > > > > > Yeah, technically a "rewrite" of 1.x means shifting the
> > > > > > > implementation
> > > > > from
> > > > > > > Perl to Go for ALL of 1.x. If you leave a subset of 1.x in
> > > > > > > Perl, then
> > > > > you
> > > > > > > have to call it a "partial rewrite".
> > > > > > >
> > > > > > > So, assuming we want to do a full rewrite (no more perl for
> > > > > > > the api
> > > > > > > whatsoever), I think that means every 1.x api endpoint (good
> > > > > > > or bad)
> > > > > needs
> > > > > > > to be rewritten to Go so we can remove the dependency on
> > > > > > > perl.
> > > > > > >
> > > > > > > However, I still like marking many as "deprecated" so we
> > > > > > > train users
> > > > > to use
> > > > > > > "better" endpoints that we envision will exist in 2.x and
> > > > > > > beyond.
> > > > > > >
> > > > > > > Jeremy
> > > > > > >
> > > > > > > On Tue, Nov 12, 2019 at 11:22 AM Hoppal, Michael <
> > > > > Michael_Hoppal@comcast.com>
> > > > > > > wrote:
> > > > > > >
> > > > > > > > We do not have a lot of deprecated routes as of now as we
> > > > > > > > focused on
> > > > > > > > routes that are more used first.
> > > > > > > >
> > > > > > > > I know on a lot of the remaining rewrites there has been
> > > > > > > > discussion
> > > > > on the
> > > > > > > > matching Github issues on potential deprecation (I cannot
> > > > > > > > think of
> > > > > the
> > > > > > > > exact count but enough to warrant this email __)
> > > > > > > >
> > > > > > > > It is key to note that I am not arguing for removal of
> > > > > > > > endpoints
> > > > > within
> > > > > > > > the current API version but if there is an endpoint we
> > > > > > > > agree should
> > > > > be
> > > > > > > > deprecated going forward to put a message within the
> > > > > > > > response and on
> > > > > the
> > > > > > > > docs.
> > > > > > > >
> > > > > > > > And thanks for the input sounds like you are +1 on
> > > > > > > > rewriting all
> > > > > routes to
> > > > > > > > Golang as Brennan is.
> > > > > > > >
> > > > > > > > On 11/12/19, 9:06 AM, "Robert O Butts" <ro...@apache.org>
> > > > > > > > wrote:
> > > > > > > >
> > > > > > > > Whether we rewrite a route in Go is an implementation
> > > > > > > > detail. To
> > > > > the
> > > > > > > > interface, to users, it doesn't matter whether a route
> > > > > > > > is
> > > > > rewritten or
> > > > > > > > not.
> > > > > > > >
> > > > > > > > But our API follows Semantic Versioning, in order to
> > > > > > > > not break
> > > > > users.
> > > > > > > > We
> > > > > > > > can't just remove endpoints that some of us don't use,
> > > > > > > > and
> > > > > assume other
> > > > > > > > people don't, maybe people who never speak up on the
> > > > > > > > mailing
> > > > > list.
> > > > > > > > We'll
> > > > > > > > never gain a big userbase if we keep breaking users.
> > > > > > > >
> > > > > > > > Per the _project_ SemVer, once we have API 2.0, we can
> > > > > > > > deprecate
> > > > > API
> > > > > > > > 1.x,
> > > > > > > > and in the next major _project_ release, remove API
> > > > > > > > 1.x.
> > > > > Irrespective
> > > > > > > > of
> > > > > > > > Perl or Go.
> > > > > > > >
> > > > > > > > My big concern is, API 2.0 is a big project. How long
> > > > > > > > has the
> > > > > rewrite
> > > > > > > > to Go
> > > > > > > > taken? Do we really believe designing and implementing
> > > > > > > > a
> > > > > completely
> > > > > > > > new API
> > > > > > > > will be any less time?
> > > > > > > >
> > > > > > > > I don't want killing Perl to have to wait on that.
> > > > > > > >
> > > > > > > > I know it feels like a waste to rewrite routes that you
> > > > > > > > don't
> > > > > use, and
> > > > > > > > probably few people do. But that's the cost of a stable
> > > > > > > > project.
> > > > > How
> > > > > > > > many
> > > > > > > > "deprecated" routes are there? If it comes down to
> > > > > > > > taking the
> > > > > > > > development
> > > > > > > > time to rewrite them so we can kill Perl faster, or
> > > > > > > > leaving Perl
> > > > > > > > around, I
> > > > > > > > vote we just do the work and kill Perl.
> > > > > > > >
> > > > > > > > >If we don't rewrite them, then Perl will last until
> > > > > > > > API 2.0 has
> > > > > been
> > > > > > > > designed, released and then *another full major release
> > > > > > > > cycle*.
> > > > > That's
> > > > > > > > way
> > > > > > > > too long to have two codebases for the same component,
> > > > > > > > IMO,
> > > > > especially
> > > > > > > > since the rewrite is already 50% complete.
> > > > > > > >
> > > > > > > > +1
> > > > > > > >
> > > > > > > >
> > > > > > > > On Tue, Nov 12, 2019 at 8:54 AM ocket 8888 <
> > > > > > > > ocket8888@gmail.com>
> > > > > > > > wrote:
> > > > > > > >
> > > > > > > > > I vote that by and large we DO rewrite them, with
> > > > > > > > exceptions
> > > > > for
> > > > > > > > routes
> > > > > > > > > that just plain don't work, even in Perl. Those are
> > > > > > > > few,
> > > > > though.
> > > > > > > > >
> > > > > > > > > If we don't rewrite them, then Perl will last until
> > > > > > > > API 2.0
> > > > > has been
> > > > > > > > > designed, released and then *another full major
> > > > > > > > release cycle*.
> > > > > > > > That's way
> > > > > > > > > too long to have two codebases for the same
> > > > > > > > component, IMO,
> > > > > > > > especially
> > > > > > > > > since the rewrite is already 50% complete.
> > > > > > > > >
> > > > > > > > > On Tue, Nov 12, 2019 at 8:43 AM Hoppal, Michael <
> > > > > > > > > Michael_Hoppal@comcast.com>
> > > > > > > > > wrote:
> > > > > > > > >
> > > > > > > > > > As the Traffic Ops API is being rewritten from Perl
> > > > > > > > to Golang
> > > > > > > > there has
> > > > > > > > > > been several routes that have been deprecated and
> > > > > > > > probably
> > > > > more to
> > > > > > > > come.
> > > > > > > > > >
> > > > > > > > > > In the deprecation efforts I have seen two
> > > > > > > > strategies:
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > > * The route IS NOT rewritten from Perl to
> > > > > > > > Golang and a
> > > > > > > > deprecation
> > > > > > > > > > notice is added to the alert response in the Perl
> > > > > > > > handler
> > > > > > > > > > * The route IS rewritten from Perl to Golang
> > > > > > > > and a
> > > > > deprecation
> > > > > > > > notice
> > > > > > > > > > is added to the alert response in the Golang
> > > > > > > > handler
> > > > > > > > > >
> > > > > > > > > > I think we should have consistency in our approach
> > > > > > > > and
> > > > > wanted to
> > > > > > > > get
> > > > > > > > > > people’s thoughts.
> > > > > > > > > >
> > > > > > > > > > I would vote that we do not rewrite a deprecated
> > > > > > > > route from
> > > > > Perl to
> > > > > > > > > Golang.
> > > > > > > > > >
> > > > > > > > > > Thanks,
> > > > > > > > > >
> > > > > > > > > > Michael
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > > >
> > > > > > > >
> >
> >
>
>
Re: [EXTERNAL] Re: Traffic Ops Route Deprecation Strategy
Posted by Rawlin Peters <ra...@apache.org>.
> Moving to 2.x means that it's my fault as a consumer of the API if I was still using the deprecated endpoint (and that change should be part of a larger upgrade plan). Simply removing the deprecated routes, retroactively, means I can't rely on having a transition period to get to 2.0 outside of a large completely down upgrade window.
There would be a transition period either way. Endpoints would be
deprecated in ATC 4.0 and removed in 5.0. The release notes for 4.0
would clearly list out the deprecated endpoints. If they have
replacements, there would be instructions on how to transition to the
replacement. Deprecated endpoints would return alerts. After upgrading
to 4.0, operators should be well aware that certain endpoints are
being removed in 5.0, and it's on them to start using new endpoints,
change workflows, etc.
By forcing operators to upgrade all of their clients to start using
API 2.x, it just adds to their burden of upgrading when that time
comes. Rather than surgically fixing the clients of the 2% of
deprecated API endpoints before upgrading to ATC 5.0, operators would
have to fix every single client they have to use API 2.x before
upgrading to 5.0. Is that exercise really worth it? Most people would
probably be blissfully unaware that the "POST /api/1.1/capabilities"
endpoint disappeared (among other deprecated endpoints). Less footguns
in the system, and they didn't even have to update their clients.
Anyways, circling back to Michael's original question, it sounds like
either option we choose, we at least have consensus on this
deprecation strategy:
- The route IS NOT rewritten from Perl to Golang and a deprecation
notice is added to the alert response in the Perl handler
If we're content to force all API clients to update their clients to
2.x for zero gain in value, at least we won't be wasting work by
rewriting deprecated Perl endpoints to Go.
- Rawlin
On Fri, Nov 15, 2019 at 7:06 PM Gray, Jonathan
<Jo...@comcast.com> wrote:
>
> Moving to 2.x means that it's my fault as a consumer of the API if I was still using the deprecated endpoint (and that change should be part of a larger upgrade plan). Simply removing the deprecated routes, retroactively, means I can't rely on having a transition period to get to 2.0 outside of a large completely down upgrade window. Depending on how the API is used, even in our own components such as TM, that could prove disastrous. For the older minor revs like 1.1/1.2, I could live with dropping those as part of a major project release (including upgrade instructions outlining what releases you must be on beforehand). Currently 1.3 I view as stable enough to code to or use a native client library from, and 1.4 is my risk/overhead to take for new stuff as it's still evolving and improving. Going to 1.5 or 2.0 moves that forward. The path of least surprise for humans is to expect big shifts on those major revs. Removal falls under that umbrella and gets as a consumer not to strictly focus on endpoint->endpoint moves, but rather workflow shifts and maybe now I have to use the API a different way than before. I believe getting this right is something that would greatly help us understand does X component at version 1 work with Y component at version 2, because now you're asking do components X and Y both speak protocol A. With major releases we can work through that each time, but it's error prone and requires experts in all products to agree upon. If you do nothing but consume major releases, and then conduct your own impact evaluation having read the upgrade instructions thoroughly, maybe you'd be able to smoothly transition.
>
> Jonathan G
>
>
> On 11/15/19, 5:24 PM, "Rawlin Peters" <ra...@apache.org> wrote:
>
> Can someone please explain to me why (A) artificially graduating the
> current 1.x API to 2.x, and removing the deprecated endpoints from
> 1.x, is better for API clients overall than (B) just removing
> deprecated endpoints from 1.x? Existing 1.x clients would gain
> absolutely nothing from being forced to update to 2.x like this.
>
> The way I see the two choices:
> A) break every single API client out there
> B) break only clients of APIs that we've decided upon as a community
> to be deprecated
>
> IMO, when we have to break things, we should try break the least
> amount of clients possible. All I'm looking for is a valuable reason
> to choose option A over option B, that isn't just "because SemVer".
>
> - Rawlin
>
> On Wed, Nov 13, 2019 at 3:24 PM Schmidt, Andrew (Contractor)
> <An...@comcast.com> wrote:
> >
> > If we decide to get rid of the API versioning it makes sense to me that we would do that on a major version upgrade, probably to the ATC version. But I think it would be too early to do it on the 2.0 API version. I like this compromise +1. I was wondering why we were stuck on the idea that 2.0 had to be an API rewrite. That seems like it’s a long way out. 2.0 = deprecations and fixes. 3.0 or 5.0 = api rewrite and maybe version convergence.
> >
> > On 11/13/19, 11:52 AM, "ocket8888@gmail.com" <oc...@gmail.com> wrote:
> >
> > Lemme lead with: making breaking changes on a major version boundary
> > gets +1 from me - no reason the API needs to be fully redesigned for
> > any particular number.
> >
> > But I just gotta bring this up again: if we want to be able to
> > deprecate an endpoint for a major release and remove it in the next
> > without needing to deal with the messiness of API versions, we could
> > always just get rid of API versions. Major ATC revisions should be
> > easier to come by with our new release strategy.
> >
> > On Wed, 2019-11-13 at 10:23 -0700, Rawlin Peters wrote:
> > > > It is a compromise - it's an inconvenience to users, forcing them
> > > > to fix
> > > scripts to upgrade. Which isn't good for user experience either; but
> > > it
> > > doesn't outright break people. It's a compromise I'm willing to live
> > > with.
> > >
> > > It will literally outright break clients (*all* of them, not just
> > > *some*). We have been down this road before, so I'll just repeat what
> > > I said when this exact conversation came up months ago[1]:
> > >
> > > "So it basically boils down to:
> > > A) promote Go 1.x routes to 2.x and eventually break ALL users of the
> > > 1.x API when the time comes to remove 1.x
> > > B) deprecate and remove some 1.x Perl routes and POSSIBLY break SOME
> > > users"
> > >
> > > Either way we slice and dice it with versioning, removing an endpoint
> > > means that clients of that endpoint will break. By graduating only
> > > the
> > > "good stuff" from 1.x to 2.x, and removing endpoints from 1.x, we
> > > will
> > > unnecessarily break all 1.x clients when the time comes to remove the
> > > rest of 1.x. Instead, we can choose to break only clients of 1.x APIs
> > > that we already deemed dangerous, unnecessary, useless, etc. enough
> > > to
> > > deprecate.
> > >
> > > Why break 100% of our API clients when we can choose to break only a
> > > small number of clients that happen to use
> > > dangerous/unnecessary/useless APIs? Why would we break our API
> > > clients
> > > unnecessarily for no added value? There is no sense in blindly
> > > adhering to Semantic Versioning just for the sake of adhering to
> > > Semantic Versioning. Instead, use it as a guideline, and break the
> > > rules where it makes the most sense to do so.
> > >
> > > - Rawlin
> > >
> > > [1]
> > > https://urldefense.com/v3/__https://lists.apache.org/thread.html/15c6bfcdce0ac00a3a8612a96c3d668a8a5a16c6ca9401de9473d31c@*3Cdev.trafficcontrol.apache.org*3E__;JSU!rx_L75ITgOQ!V_b3E9iEYqNlZus18f-sVuofwuyxapCthg3LdO4NxyMCU39b7mdf4klP5KNR7XZ7d3Uc$
> > >
> > > On Wed, Nov 13, 2019 at 9:43 AM Robert O Butts <ro...@apache.org>
> > > wrote:
> > > > > But if it's really valuable to remove these endpoints, maybe it's
> > > > appropriate to bump the major version and ask everybody to update
> > > > their
> > > > clients?
> > > >
> > > > +1
> > > >
> > > > This sounds like the best compromise to me. It addresses both sides
> > > > - it
> > > > lets us remove endpoints, and it also prevents breaking users.
> > > >
> > > > It is a compromise - it's an inconvenience to users, forcing them
> > > > to fix
> > > > scripts to upgrade. Which isn't good for user experience either;
> > > > but it
> > > > doesn't outright break people. It's a compromise I'm willing to
> > > > live with.
> > > >
> > > > This actually lets us do at least 3 things:
> > > > 1. Remove "deprecated" endpoints.
> > > > 2. Remove the unmaintained dangerously-large Cache Config
> > > > endpoints.
> > > > 3. Lets us do Layered Profiles, changing Servers and DSes to have
> > > > arrays of
> > > > Profiles instead of a single Profile, without hacky backward-
> > > > compatibility
> > > > attempts.
> > > >
> > > > To clarify, `/api/2.0` would be essentially the current API, just
> > > > with the
> > > > above breaking changes (possibly others) which require a SemVer
> > > > major
> > > > version increase. And the big API rewrite/redesign we've been
> > > > calling "2.0"
> > > > would become `/api/3.0` or some future version.
> > > >
> > > > It's not perfect, but it seems to address the most concerns on all
> > > > sides.
> > > > Thoughts? Objections?
> > > >
> > > >
> > > > On Wed, Nov 13, 2019 at 8:45 AM Chris Lemmons <al...@gmail.com>
> > > > wrote:
> > > >
> > > > > For the endpoints, I agree in part and disagree in part.
> > > > >
> > > > > Agree for these two:
> > > > > - the endpoints are dangerous and pose a risk to the overall
> > > > > Traffic
> > > > > Control system
> > > > > - the task of rewriting the endpoints from Perl to Go represents
> > > > > an
> > > > > unreasonable amount of work for the project
> > > > >
> > > > > And I would add:
> > > > > - the endpoint does not serve its intended purpose and cannot
> > > > > reasonably
> > > > > be used
> > > > > - the endpoint has been made unsafe or unusable by new features
> > > > > added to TC
> > > > >
> > > > > Disagree for these two:
> > > > > - the endpoints have been obsoleted by other endpoints
> > > > > - the endpoints don't seem to serve a true purpose or don't seem
> > > > > to be
> > > > > valuable to the project anymore
> > > > >
> > > > > For the most part, obsoleted endpoints can be maintained with a
> > > > > shim
> > > > > into the new code. If they would require "an unreasonable amount
> > > > > of
> > > > > work", it might make sense to remove them, then.
> > > > >
> > > > > Things that don't seem valuable usually fall into another
> > > > > category
> > > > > like "cannot reasonably be used", but if the only reason for
> > > > > deprecation is that the feature is going away, we should really
> > > > > support it according to our promise.
> > > > >
> > > > > But if it's really valuable to remove these endpoints, maybe it's
> > > > > appropriate to bump the major version and ask everybody to update
> > > > > their clients?
> > > > >
> > > > > On Tue, Nov 12, 2019 at 5:27 PM Rawlin Peters <ra...@apache.org>
> > > > > wrote:
> > > > > > The real crux of the issue here is whether or not we agree as a
> > > > > > project that we can move forward with this plan that I believe
> > > > > > to be a
> > > > > > good compromise that meets most concerns but "breaks the
> > > > > > Semantic
> > > > > > Versioning promise":
> > > > > > 1) don't rewrite deprecated endpoints from Perl to Go
> > > > > > 2) in the Perl handler, add a deprecation notice to the
> > > > > > "alerts"
> > > > > > section of the response
> > > > > > 3) once a deprecated API has been deprecated for one major
> > > > > > release,
> > > > > > delete it from the API in the subsequent major release (e.g. if
> > > > > > deprecation notices are added and released in ATC 4.x, we can
> > > > > > remove
> > > > > > the deprecated endpoint in ATC 5.x)
> > > > > >
> > > > > > Rob, from your latest proposal to remove the ATS config API
> > > > > > endpoints
> > > > > > from 1.x, I thought we were past this issue already and could
> > > > > > finally
> > > > > > agree to remove deprecated endpoints from 1.x (following steps
> > > > > > 1-3
> > > > > > above) without requiring an all-new 2.x API. Are we not
> > > > > > actually past
> > > > > > that? You came up with valid reasons for applying these steps
> > > > > > to the
> > > > > > ATS config API endpoints, and I think it's fair to say that we
> > > > > > can
> > > > > > come up with valid reasons for applying these steps to other
> > > > > > deprecated API endpoints as well.
> > > > > >
> > > > > > It seems like the real problem is actually agreeing on which
> > > > > > _specific_ endpoints we should be able to follow steps 1-3 for.
> > > > > > IMO I
> > > > > > think it should apply to any 1.x endpoints that we have valid
> > > > > > reasons
> > > > > > for not carrying forward to Go, including but not limited to:
> > > > > > - the endpoints are dangerous and pose a risk to the overall
> > > > > > Traffic
> > > > > > Control system
> > > > > > - the endpoints have been obsoleted by other endpoints
> > > > > > - the endpoints don't seem to serve a true purpose or don't
> > > > > > seem to be
> > > > > > valuable to the project anymore
> > > > > > - the task of rewriting the endpoints from Perl to Go
> > > > > > represents an
> > > > > > unreasonable amount of work for the project
> > > > > >
> > > > > > Obviously most of these reasons are subjective, so we'd have to
> > > > > > come
> > > > > > to some level of consensus on deprecating particular endpoints
> > > > > > first,
> > > > > > but we should be able to follow steps 1-3 for whatever
> > > > > > endpoints we
> > > > > > decide.
> > > > > >
> > > > > > Making breaking changes shouldn't be taken lightly, but we need
> > > > > > a path
> > > > > > forward that allows some calculated, breaking changes where
> > > > > > necessary.
> > > > > > Otherwise, we are unnecessarily holding the project back.
> > > > > >
> > > > > > - Rawlin
> > > > > >
> > > > > > On Tue, Nov 12, 2019 at 11:44 AM Jeremy Mitchell <
> > > > > > mitchell852@gmail.com>
> > > > > wrote:
> > > > > > > Yeah, technically a "rewrite" of 1.x means shifting the
> > > > > > > implementation
> > > > > from
> > > > > > > Perl to Go for ALL of 1.x. If you leave a subset of 1.x in
> > > > > > > Perl, then
> > > > > you
> > > > > > > have to call it a "partial rewrite".
> > > > > > >
> > > > > > > So, assuming we want to do a full rewrite (no more perl for
> > > > > > > the api
> > > > > > > whatsoever), I think that means every 1.x api endpoint (good
> > > > > > > or bad)
> > > > > needs
> > > > > > > to be rewritten to Go so we can remove the dependency on
> > > > > > > perl.
> > > > > > >
> > > > > > > However, I still like marking many as "deprecated" so we
> > > > > > > train users
> > > > > to use
> > > > > > > "better" endpoints that we envision will exist in 2.x and
> > > > > > > beyond.
> > > > > > >
> > > > > > > Jeremy
> > > > > > >
> > > > > > > On Tue, Nov 12, 2019 at 11:22 AM Hoppal, Michael <
> > > > > Michael_Hoppal@comcast.com>
> > > > > > > wrote:
> > > > > > >
> > > > > > > > We do not have a lot of deprecated routes as of now as we
> > > > > > > > focused on
> > > > > > > > routes that are more used first.
> > > > > > > >
> > > > > > > > I know on a lot of the remaining rewrites there has been
> > > > > > > > discussion
> > > > > on the
> > > > > > > > matching Github issues on potential deprecation (I cannot
> > > > > > > > think of
> > > > > the
> > > > > > > > exact count but enough to warrant this email __)
> > > > > > > >
> > > > > > > > It is key to note that I am not arguing for removal of
> > > > > > > > endpoints
> > > > > within
> > > > > > > > the current API version but if there is an endpoint we
> > > > > > > > agree should
> > > > > be
> > > > > > > > deprecated going forward to put a message within the
> > > > > > > > response and on
> > > > > the
> > > > > > > > docs.
> > > > > > > >
> > > > > > > > And thanks for the input sounds like you are +1 on
> > > > > > > > rewriting all
> > > > > routes to
> > > > > > > > Golang as Brennan is.
> > > > > > > >
> > > > > > > > On 11/12/19, 9:06 AM, "Robert O Butts" <ro...@apache.org>
> > > > > > > > wrote:
> > > > > > > >
> > > > > > > > Whether we rewrite a route in Go is an implementation
> > > > > > > > detail. To
> > > > > the
> > > > > > > > interface, to users, it doesn't matter whether a route
> > > > > > > > is
> > > > > rewritten or
> > > > > > > > not.
> > > > > > > >
> > > > > > > > But our API follows Semantic Versioning, in order to
> > > > > > > > not break
> > > > > users.
> > > > > > > > We
> > > > > > > > can't just remove endpoints that some of us don't use,
> > > > > > > > and
> > > > > assume other
> > > > > > > > people don't, maybe people who never speak up on the
> > > > > > > > mailing
> > > > > list.
> > > > > > > > We'll
> > > > > > > > never gain a big userbase if we keep breaking users.
> > > > > > > >
> > > > > > > > Per the _project_ SemVer, once we have API 2.0, we can
> > > > > > > > deprecate
> > > > > API
> > > > > > > > 1.x,
> > > > > > > > and in the next major _project_ release, remove API
> > > > > > > > 1.x.
> > > > > Irrespective
> > > > > > > > of
> > > > > > > > Perl or Go.
> > > > > > > >
> > > > > > > > My big concern is, API 2.0 is a big project. How long
> > > > > > > > has the
> > > > > rewrite
> > > > > > > > to Go
> > > > > > > > taken? Do we really believe designing and implementing
> > > > > > > > a
> > > > > completely
> > > > > > > > new API
> > > > > > > > will be any less time?
> > > > > > > >
> > > > > > > > I don't want killing Perl to have to wait on that.
> > > > > > > >
> > > > > > > > I know it feels like a waste to rewrite routes that you
> > > > > > > > don't
> > > > > use, and
> > > > > > > > probably few people do. But that's the cost of a stable
> > > > > > > > project.
> > > > > How
> > > > > > > > many
> > > > > > > > "deprecated" routes are there? If it comes down to
> > > > > > > > taking the
> > > > > > > > development
> > > > > > > > time to rewrite them so we can kill Perl faster, or
> > > > > > > > leaving Perl
> > > > > > > > around, I
> > > > > > > > vote we just do the work and kill Perl.
> > > > > > > >
> > > > > > > > >If we don't rewrite them, then Perl will last until
> > > > > > > > API 2.0 has
> > > > > been
> > > > > > > > designed, released and then *another full major release
> > > > > > > > cycle*.
> > > > > That's
> > > > > > > > way
> > > > > > > > too long to have two codebases for the same component,
> > > > > > > > IMO,
> > > > > especially
> > > > > > > > since the rewrite is already 50% complete.
> > > > > > > >
> > > > > > > > +1
> > > > > > > >
> > > > > > > >
> > > > > > > > On Tue, Nov 12, 2019 at 8:54 AM ocket 8888 <
> > > > > > > > ocket8888@gmail.com>
> > > > > > > > wrote:
> > > > > > > >
> > > > > > > > > I vote that by and large we DO rewrite them, with
> > > > > > > > exceptions
> > > > > for
> > > > > > > > routes
> > > > > > > > > that just plain don't work, even in Perl. Those are
> > > > > > > > few,
> > > > > though.
> > > > > > > > >
> > > > > > > > > If we don't rewrite them, then Perl will last until
> > > > > > > > API 2.0
> > > > > has been
> > > > > > > > > designed, released and then *another full major
> > > > > > > > release cycle*.
> > > > > > > > That's way
> > > > > > > > > too long to have two codebases for the same
> > > > > > > > component, IMO,
> > > > > > > > especially
> > > > > > > > > since the rewrite is already 50% complete.
> > > > > > > > >
> > > > > > > > > On Tue, Nov 12, 2019 at 8:43 AM Hoppal, Michael <
> > > > > > > > > Michael_Hoppal@comcast.com>
> > > > > > > > > wrote:
> > > > > > > > >
> > > > > > > > > > As the Traffic Ops API is being rewritten from Perl
> > > > > > > > to Golang
> > > > > > > > there has
> > > > > > > > > > been several routes that have been deprecated and
> > > > > > > > probably
> > > > > more to
> > > > > > > > come.
> > > > > > > > > >
> > > > > > > > > > In the deprecation efforts I have seen two
> > > > > > > > strategies:
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > > * The route IS NOT rewritten from Perl to
> > > > > > > > Golang and a
> > > > > > > > deprecation
> > > > > > > > > > notice is added to the alert response in the Perl
> > > > > > > > handler
> > > > > > > > > > * The route IS rewritten from Perl to Golang
> > > > > > > > and a
> > > > > deprecation
> > > > > > > > notice
> > > > > > > > > > is added to the alert response in the Golang
> > > > > > > > handler
> > > > > > > > > >
> > > > > > > > > > I think we should have consistency in our approach
> > > > > > > > and
> > > > > wanted to
> > > > > > > > get
> > > > > > > > > > people’s thoughts.
> > > > > > > > > >
> > > > > > > > > > I would vote that we do not rewrite a deprecated
> > > > > > > > route from
> > > > > Perl to
> > > > > > > > > Golang.
> > > > > > > > > >
> > > > > > > > > > Thanks,
> > > > > > > > > >
> > > > > > > > > > Michael
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > > >
> > > > > > > >
> >
> >
>
>
Re: [EXTERNAL] Re: Traffic Ops Route Deprecation Strategy
Posted by "Gray, Jonathan" <Jo...@comcast.com>.
Moving to 2.x means that it's my fault as a consumer of the API if I was still using the deprecated endpoint (and that change should be part of a larger upgrade plan). Simply removing the deprecated routes, retroactively, means I can't rely on having a transition period to get to 2.0 outside of a large completely down upgrade window. Depending on how the API is used, even in our own components such as TM, that could prove disastrous. For the older minor revs like 1.1/1.2, I could live with dropping those as part of a major project release (including upgrade instructions outlining what releases you must be on beforehand). Currently 1.3 I view as stable enough to code to or use a native client library from, and 1.4 is my risk/overhead to take for new stuff as it's still evolving and improving. Going to 1.5 or 2.0 moves that forward. The path of least surprise for humans is to expect big shifts on those major revs. Removal falls under that umbrella and gets as a consumer not to strictly focus on endpoint->endpoint moves, but rather workflow shifts and maybe now I have to use the API a different way than before. I believe getting this right is something that would greatly help us understand does X component at version 1 work with Y component at version 2, because now you're asking do components X and Y both speak protocol A. With major releases we can work through that each time, but it's error prone and requires experts in all products to agree upon. If you do nothing but consume major releases, and then conduct your own impact evaluation having read the upgrade instructions thoroughly, maybe you'd be able to smoothly transition.
Jonathan G
On 11/15/19, 5:24 PM, "Rawlin Peters" <ra...@apache.org> wrote:
Can someone please explain to me why (A) artificially graduating the
current 1.x API to 2.x, and removing the deprecated endpoints from
1.x, is better for API clients overall than (B) just removing
deprecated endpoints from 1.x? Existing 1.x clients would gain
absolutely nothing from being forced to update to 2.x like this.
The way I see the two choices:
A) break every single API client out there
B) break only clients of APIs that we've decided upon as a community
to be deprecated
IMO, when we have to break things, we should try break the least
amount of clients possible. All I'm looking for is a valuable reason
to choose option A over option B, that isn't just "because SemVer".
- Rawlin
On Wed, Nov 13, 2019 at 3:24 PM Schmidt, Andrew (Contractor)
<An...@comcast.com> wrote:
>
> If we decide to get rid of the API versioning it makes sense to me that we would do that on a major version upgrade, probably to the ATC version. But I think it would be too early to do it on the 2.0 API version. I like this compromise +1. I was wondering why we were stuck on the idea that 2.0 had to be an API rewrite. That seems like it’s a long way out. 2.0 = deprecations and fixes. 3.0 or 5.0 = api rewrite and maybe version convergence.
>
> On 11/13/19, 11:52 AM, "ocket8888@gmail.com" <oc...@gmail.com> wrote:
>
> Lemme lead with: making breaking changes on a major version boundary
> gets +1 from me - no reason the API needs to be fully redesigned for
> any particular number.
>
> But I just gotta bring this up again: if we want to be able to
> deprecate an endpoint for a major release and remove it in the next
> without needing to deal with the messiness of API versions, we could
> always just get rid of API versions. Major ATC revisions should be
> easier to come by with our new release strategy.
>
> On Wed, 2019-11-13 at 10:23 -0700, Rawlin Peters wrote:
> > > It is a compromise - it's an inconvenience to users, forcing them
> > > to fix
> > scripts to upgrade. Which isn't good for user experience either; but
> > it
> > doesn't outright break people. It's a compromise I'm willing to live
> > with.
> >
> > It will literally outright break clients (*all* of them, not just
> > *some*). We have been down this road before, so I'll just repeat what
> > I said when this exact conversation came up months ago[1]:
> >
> > "So it basically boils down to:
> > A) promote Go 1.x routes to 2.x and eventually break ALL users of the
> > 1.x API when the time comes to remove 1.x
> > B) deprecate and remove some 1.x Perl routes and POSSIBLY break SOME
> > users"
> >
> > Either way we slice and dice it with versioning, removing an endpoint
> > means that clients of that endpoint will break. By graduating only
> > the
> > "good stuff" from 1.x to 2.x, and removing endpoints from 1.x, we
> > will
> > unnecessarily break all 1.x clients when the time comes to remove the
> > rest of 1.x. Instead, we can choose to break only clients of 1.x APIs
> > that we already deemed dangerous, unnecessary, useless, etc. enough
> > to
> > deprecate.
> >
> > Why break 100% of our API clients when we can choose to break only a
> > small number of clients that happen to use
> > dangerous/unnecessary/useless APIs? Why would we break our API
> > clients
> > unnecessarily for no added value? There is no sense in blindly
> > adhering to Semantic Versioning just for the sake of adhering to
> > Semantic Versioning. Instead, use it as a guideline, and break the
> > rules where it makes the most sense to do so.
> >
> > - Rawlin
> >
> > [1]
> > https://urldefense.com/v3/__https://lists.apache.org/thread.html/15c6bfcdce0ac00a3a8612a96c3d668a8a5a16c6ca9401de9473d31c@*3Cdev.trafficcontrol.apache.org*3E__;JSU!rx_L75ITgOQ!V_b3E9iEYqNlZus18f-sVuofwuyxapCthg3LdO4NxyMCU39b7mdf4klP5KNR7XZ7d3Uc$
> >
> > On Wed, Nov 13, 2019 at 9:43 AM Robert O Butts <ro...@apache.org>
> > wrote:
> > > > But if it's really valuable to remove these endpoints, maybe it's
> > > appropriate to bump the major version and ask everybody to update
> > > their
> > > clients?
> > >
> > > +1
> > >
> > > This sounds like the best compromise to me. It addresses both sides
> > > - it
> > > lets us remove endpoints, and it also prevents breaking users.
> > >
> > > It is a compromise - it's an inconvenience to users, forcing them
> > > to fix
> > > scripts to upgrade. Which isn't good for user experience either;
> > > but it
> > > doesn't outright break people. It's a compromise I'm willing to
> > > live with.
> > >
> > > This actually lets us do at least 3 things:
> > > 1. Remove "deprecated" endpoints.
> > > 2. Remove the unmaintained dangerously-large Cache Config
> > > endpoints.
> > > 3. Lets us do Layered Profiles, changing Servers and DSes to have
> > > arrays of
> > > Profiles instead of a single Profile, without hacky backward-
> > > compatibility
> > > attempts.
> > >
> > > To clarify, `/api/2.0` would be essentially the current API, just
> > > with the
> > > above breaking changes (possibly others) which require a SemVer
> > > major
> > > version increase. And the big API rewrite/redesign we've been
> > > calling "2.0"
> > > would become `/api/3.0` or some future version.
> > >
> > > It's not perfect, but it seems to address the most concerns on all
> > > sides.
> > > Thoughts? Objections?
> > >
> > >
> > > On Wed, Nov 13, 2019 at 8:45 AM Chris Lemmons <al...@gmail.com>
> > > wrote:
> > >
> > > > For the endpoints, I agree in part and disagree in part.
> > > >
> > > > Agree for these two:
> > > > - the endpoints are dangerous and pose a risk to the overall
> > > > Traffic
> > > > Control system
> > > > - the task of rewriting the endpoints from Perl to Go represents
> > > > an
> > > > unreasonable amount of work for the project
> > > >
> > > > And I would add:
> > > > - the endpoint does not serve its intended purpose and cannot
> > > > reasonably
> > > > be used
> > > > - the endpoint has been made unsafe or unusable by new features
> > > > added to TC
> > > >
> > > > Disagree for these two:
> > > > - the endpoints have been obsoleted by other endpoints
> > > > - the endpoints don't seem to serve a true purpose or don't seem
> > > > to be
> > > > valuable to the project anymore
> > > >
> > > > For the most part, obsoleted endpoints can be maintained with a
> > > > shim
> > > > into the new code. If they would require "an unreasonable amount
> > > > of
> > > > work", it might make sense to remove them, then.
> > > >
> > > > Things that don't seem valuable usually fall into another
> > > > category
> > > > like "cannot reasonably be used", but if the only reason for
> > > > deprecation is that the feature is going away, we should really
> > > > support it according to our promise.
> > > >
> > > > But if it's really valuable to remove these endpoints, maybe it's
> > > > appropriate to bump the major version and ask everybody to update
> > > > their clients?
> > > >
> > > > On Tue, Nov 12, 2019 at 5:27 PM Rawlin Peters <ra...@apache.org>
> > > > wrote:
> > > > > The real crux of the issue here is whether or not we agree as a
> > > > > project that we can move forward with this plan that I believe
> > > > > to be a
> > > > > good compromise that meets most concerns but "breaks the
> > > > > Semantic
> > > > > Versioning promise":
> > > > > 1) don't rewrite deprecated endpoints from Perl to Go
> > > > > 2) in the Perl handler, add a deprecation notice to the
> > > > > "alerts"
> > > > > section of the response
> > > > > 3) once a deprecated API has been deprecated for one major
> > > > > release,
> > > > > delete it from the API in the subsequent major release (e.g. if
> > > > > deprecation notices are added and released in ATC 4.x, we can
> > > > > remove
> > > > > the deprecated endpoint in ATC 5.x)
> > > > >
> > > > > Rob, from your latest proposal to remove the ATS config API
> > > > > endpoints
> > > > > from 1.x, I thought we were past this issue already and could
> > > > > finally
> > > > > agree to remove deprecated endpoints from 1.x (following steps
> > > > > 1-3
> > > > > above) without requiring an all-new 2.x API. Are we not
> > > > > actually past
> > > > > that? You came up with valid reasons for applying these steps
> > > > > to the
> > > > > ATS config API endpoints, and I think it's fair to say that we
> > > > > can
> > > > > come up with valid reasons for applying these steps to other
> > > > > deprecated API endpoints as well.
> > > > >
> > > > > It seems like the real problem is actually agreeing on which
> > > > > _specific_ endpoints we should be able to follow steps 1-3 for.
> > > > > IMO I
> > > > > think it should apply to any 1.x endpoints that we have valid
> > > > > reasons
> > > > > for not carrying forward to Go, including but not limited to:
> > > > > - the endpoints are dangerous and pose a risk to the overall
> > > > > Traffic
> > > > > Control system
> > > > > - the endpoints have been obsoleted by other endpoints
> > > > > - the endpoints don't seem to serve a true purpose or don't
> > > > > seem to be
> > > > > valuable to the project anymore
> > > > > - the task of rewriting the endpoints from Perl to Go
> > > > > represents an
> > > > > unreasonable amount of work for the project
> > > > >
> > > > > Obviously most of these reasons are subjective, so we'd have to
> > > > > come
> > > > > to some level of consensus on deprecating particular endpoints
> > > > > first,
> > > > > but we should be able to follow steps 1-3 for whatever
> > > > > endpoints we
> > > > > decide.
> > > > >
> > > > > Making breaking changes shouldn't be taken lightly, but we need
> > > > > a path
> > > > > forward that allows some calculated, breaking changes where
> > > > > necessary.
> > > > > Otherwise, we are unnecessarily holding the project back.
> > > > >
> > > > > - Rawlin
> > > > >
> > > > > On Tue, Nov 12, 2019 at 11:44 AM Jeremy Mitchell <
> > > > > mitchell852@gmail.com>
> > > > wrote:
> > > > > > Yeah, technically a "rewrite" of 1.x means shifting the
> > > > > > implementation
> > > > from
> > > > > > Perl to Go for ALL of 1.x. If you leave a subset of 1.x in
> > > > > > Perl, then
> > > > you
> > > > > > have to call it a "partial rewrite".
> > > > > >
> > > > > > So, assuming we want to do a full rewrite (no more perl for
> > > > > > the api
> > > > > > whatsoever), I think that means every 1.x api endpoint (good
> > > > > > or bad)
> > > > needs
> > > > > > to be rewritten to Go so we can remove the dependency on
> > > > > > perl.
> > > > > >
> > > > > > However, I still like marking many as "deprecated" so we
> > > > > > train users
> > > > to use
> > > > > > "better" endpoints that we envision will exist in 2.x and
> > > > > > beyond.
> > > > > >
> > > > > > Jeremy
> > > > > >
> > > > > > On Tue, Nov 12, 2019 at 11:22 AM Hoppal, Michael <
> > > > Michael_Hoppal@comcast.com>
> > > > > > wrote:
> > > > > >
> > > > > > > We do not have a lot of deprecated routes as of now as we
> > > > > > > focused on
> > > > > > > routes that are more used first.
> > > > > > >
> > > > > > > I know on a lot of the remaining rewrites there has been
> > > > > > > discussion
> > > > on the
> > > > > > > matching Github issues on potential deprecation (I cannot
> > > > > > > think of
> > > > the
> > > > > > > exact count but enough to warrant this email __)
> > > > > > >
> > > > > > > It is key to note that I am not arguing for removal of
> > > > > > > endpoints
> > > > within
> > > > > > > the current API version but if there is an endpoint we
> > > > > > > agree should
> > > > be
> > > > > > > deprecated going forward to put a message within the
> > > > > > > response and on
> > > > the
> > > > > > > docs.
> > > > > > >
> > > > > > > And thanks for the input sounds like you are +1 on
> > > > > > > rewriting all
> > > > routes to
> > > > > > > Golang as Brennan is.
> > > > > > >
> > > > > > > On 11/12/19, 9:06 AM, "Robert O Butts" <ro...@apache.org>
> > > > > > > wrote:
> > > > > > >
> > > > > > > Whether we rewrite a route in Go is an implementation
> > > > > > > detail. To
> > > > the
> > > > > > > interface, to users, it doesn't matter whether a route
> > > > > > > is
> > > > rewritten or
> > > > > > > not.
> > > > > > >
> > > > > > > But our API follows Semantic Versioning, in order to
> > > > > > > not break
> > > > users.
> > > > > > > We
> > > > > > > can't just remove endpoints that some of us don't use,
> > > > > > > and
> > > > assume other
> > > > > > > people don't, maybe people who never speak up on the
> > > > > > > mailing
> > > > list.
> > > > > > > We'll
> > > > > > > never gain a big userbase if we keep breaking users.
> > > > > > >
> > > > > > > Per the _project_ SemVer, once we have API 2.0, we can
> > > > > > > deprecate
> > > > API
> > > > > > > 1.x,
> > > > > > > and in the next major _project_ release, remove API
> > > > > > > 1.x.
> > > > Irrespective
> > > > > > > of
> > > > > > > Perl or Go.
> > > > > > >
> > > > > > > My big concern is, API 2.0 is a big project. How long
> > > > > > > has the
> > > > rewrite
> > > > > > > to Go
> > > > > > > taken? Do we really believe designing and implementing
> > > > > > > a
> > > > completely
> > > > > > > new API
> > > > > > > will be any less time?
> > > > > > >
> > > > > > > I don't want killing Perl to have to wait on that.
> > > > > > >
> > > > > > > I know it feels like a waste to rewrite routes that you
> > > > > > > don't
> > > > use, and
> > > > > > > probably few people do. But that's the cost of a stable
> > > > > > > project.
> > > > How
> > > > > > > many
> > > > > > > "deprecated" routes are there? If it comes down to
> > > > > > > taking the
> > > > > > > development
> > > > > > > time to rewrite them so we can kill Perl faster, or
> > > > > > > leaving Perl
> > > > > > > around, I
> > > > > > > vote we just do the work and kill Perl.
> > > > > > >
> > > > > > > >If we don't rewrite them, then Perl will last until
> > > > > > > API 2.0 has
> > > > been
> > > > > > > designed, released and then *another full major release
> > > > > > > cycle*.
> > > > That's
> > > > > > > way
> > > > > > > too long to have two codebases for the same component,
> > > > > > > IMO,
> > > > especially
> > > > > > > since the rewrite is already 50% complete.
> > > > > > >
> > > > > > > +1
> > > > > > >
> > > > > > >
> > > > > > > On Tue, Nov 12, 2019 at 8:54 AM ocket 8888 <
> > > > > > > ocket8888@gmail.com>
> > > > > > > wrote:
> > > > > > >
> > > > > > > > I vote that by and large we DO rewrite them, with
> > > > > > > exceptions
> > > > for
> > > > > > > routes
> > > > > > > > that just plain don't work, even in Perl. Those are
> > > > > > > few,
> > > > though.
> > > > > > > >
> > > > > > > > If we don't rewrite them, then Perl will last until
> > > > > > > API 2.0
> > > > has been
> > > > > > > > designed, released and then *another full major
> > > > > > > release cycle*.
> > > > > > > That's way
> > > > > > > > too long to have two codebases for the same
> > > > > > > component, IMO,
> > > > > > > especially
> > > > > > > > since the rewrite is already 50% complete.
> > > > > > > >
> > > > > > > > On Tue, Nov 12, 2019 at 8:43 AM Hoppal, Michael <
> > > > > > > > Michael_Hoppal@comcast.com>
> > > > > > > > wrote:
> > > > > > > >
> > > > > > > > > As the Traffic Ops API is being rewritten from Perl
> > > > > > > to Golang
> > > > > > > there has
> > > > > > > > > been several routes that have been deprecated and
> > > > > > > probably
> > > > more to
> > > > > > > come.
> > > > > > > > >
> > > > > > > > > In the deprecation efforts I have seen two
> > > > > > > strategies:
> > > > > > > > >
> > > > > > > > >
> > > > > > > > > * The route IS NOT rewritten from Perl to
> > > > > > > Golang and a
> > > > > > > deprecation
> > > > > > > > > notice is added to the alert response in the Perl
> > > > > > > handler
> > > > > > > > > * The route IS rewritten from Perl to Golang
> > > > > > > and a
> > > > deprecation
> > > > > > > notice
> > > > > > > > > is added to the alert response in the Golang
> > > > > > > handler
> > > > > > > > >
> > > > > > > > > I think we should have consistency in our approach
> > > > > > > and
> > > > wanted to
> > > > > > > get
> > > > > > > > > people’s thoughts.
> > > > > > > > >
> > > > > > > > > I would vote that we do not rewrite a deprecated
> > > > > > > route from
> > > > Perl to
> > > > > > > > Golang.
> > > > > > > > >
> > > > > > > > > Thanks,
> > > > > > > > >
> > > > > > > > > Michael
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > > >
> > > > > > >
>
>
Re: [EXTERNAL] Re: Traffic Ops Route Deprecation Strategy
Posted by Rawlin Peters <ra...@apache.org>.
Can someone please explain to me why (A) artificially graduating the
current 1.x API to 2.x, and removing the deprecated endpoints from
1.x, is better for API clients overall than (B) just removing
deprecated endpoints from 1.x? Existing 1.x clients would gain
absolutely nothing from being forced to update to 2.x like this.
The way I see the two choices:
A) break every single API client out there
B) break only clients of APIs that we've decided upon as a community
to be deprecated
IMO, when we have to break things, we should try break the least
amount of clients possible. All I'm looking for is a valuable reason
to choose option A over option B, that isn't just "because SemVer".
- Rawlin
On Wed, Nov 13, 2019 at 3:24 PM Schmidt, Andrew (Contractor)
<An...@comcast.com> wrote:
>
> If we decide to get rid of the API versioning it makes sense to me that we would do that on a major version upgrade, probably to the ATC version. But I think it would be too early to do it on the 2.0 API version. I like this compromise +1. I was wondering why we were stuck on the idea that 2.0 had to be an API rewrite. That seems like it’s a long way out. 2.0 = deprecations and fixes. 3.0 or 5.0 = api rewrite and maybe version convergence.
>
> On 11/13/19, 11:52 AM, "ocket8888@gmail.com" <oc...@gmail.com> wrote:
>
> Lemme lead with: making breaking changes on a major version boundary
> gets +1 from me - no reason the API needs to be fully redesigned for
> any particular number.
>
> But I just gotta bring this up again: if we want to be able to
> deprecate an endpoint for a major release and remove it in the next
> without needing to deal with the messiness of API versions, we could
> always just get rid of API versions. Major ATC revisions should be
> easier to come by with our new release strategy.
>
> On Wed, 2019-11-13 at 10:23 -0700, Rawlin Peters wrote:
> > > It is a compromise - it's an inconvenience to users, forcing them
> > > to fix
> > scripts to upgrade. Which isn't good for user experience either; but
> > it
> > doesn't outright break people. It's a compromise I'm willing to live
> > with.
> >
> > It will literally outright break clients (*all* of them, not just
> > *some*). We have been down this road before, so I'll just repeat what
> > I said when this exact conversation came up months ago[1]:
> >
> > "So it basically boils down to:
> > A) promote Go 1.x routes to 2.x and eventually break ALL users of the
> > 1.x API when the time comes to remove 1.x
> > B) deprecate and remove some 1.x Perl routes and POSSIBLY break SOME
> > users"
> >
> > Either way we slice and dice it with versioning, removing an endpoint
> > means that clients of that endpoint will break. By graduating only
> > the
> > "good stuff" from 1.x to 2.x, and removing endpoints from 1.x, we
> > will
> > unnecessarily break all 1.x clients when the time comes to remove the
> > rest of 1.x. Instead, we can choose to break only clients of 1.x APIs
> > that we already deemed dangerous, unnecessary, useless, etc. enough
> > to
> > deprecate.
> >
> > Why break 100% of our API clients when we can choose to break only a
> > small number of clients that happen to use
> > dangerous/unnecessary/useless APIs? Why would we break our API
> > clients
> > unnecessarily for no added value? There is no sense in blindly
> > adhering to Semantic Versioning just for the sake of adhering to
> > Semantic Versioning. Instead, use it as a guideline, and break the
> > rules where it makes the most sense to do so.
> >
> > - Rawlin
> >
> > [1]
> > https://lists.apache.org/thread.html/15c6bfcdce0ac00a3a8612a96c3d668a8a5a16c6ca9401de9473d31c@%3Cdev.trafficcontrol.apache.org%3E
> >
> > On Wed, Nov 13, 2019 at 9:43 AM Robert O Butts <ro...@apache.org>
> > wrote:
> > > > But if it's really valuable to remove these endpoints, maybe it's
> > > appropriate to bump the major version and ask everybody to update
> > > their
> > > clients?
> > >
> > > +1
> > >
> > > This sounds like the best compromise to me. It addresses both sides
> > > - it
> > > lets us remove endpoints, and it also prevents breaking users.
> > >
> > > It is a compromise - it's an inconvenience to users, forcing them
> > > to fix
> > > scripts to upgrade. Which isn't good for user experience either;
> > > but it
> > > doesn't outright break people. It's a compromise I'm willing to
> > > live with.
> > >
> > > This actually lets us do at least 3 things:
> > > 1. Remove "deprecated" endpoints.
> > > 2. Remove the unmaintained dangerously-large Cache Config
> > > endpoints.
> > > 3. Lets us do Layered Profiles, changing Servers and DSes to have
> > > arrays of
> > > Profiles instead of a single Profile, without hacky backward-
> > > compatibility
> > > attempts.
> > >
> > > To clarify, `/api/2.0` would be essentially the current API, just
> > > with the
> > > above breaking changes (possibly others) which require a SemVer
> > > major
> > > version increase. And the big API rewrite/redesign we've been
> > > calling "2.0"
> > > would become `/api/3.0` or some future version.
> > >
> > > It's not perfect, but it seems to address the most concerns on all
> > > sides.
> > > Thoughts? Objections?
> > >
> > >
> > > On Wed, Nov 13, 2019 at 8:45 AM Chris Lemmons <al...@gmail.com>
> > > wrote:
> > >
> > > > For the endpoints, I agree in part and disagree in part.
> > > >
> > > > Agree for these two:
> > > > - the endpoints are dangerous and pose a risk to the overall
> > > > Traffic
> > > > Control system
> > > > - the task of rewriting the endpoints from Perl to Go represents
> > > > an
> > > > unreasonable amount of work for the project
> > > >
> > > > And I would add:
> > > > - the endpoint does not serve its intended purpose and cannot
> > > > reasonably
> > > > be used
> > > > - the endpoint has been made unsafe or unusable by new features
> > > > added to TC
> > > >
> > > > Disagree for these two:
> > > > - the endpoints have been obsoleted by other endpoints
> > > > - the endpoints don't seem to serve a true purpose or don't seem
> > > > to be
> > > > valuable to the project anymore
> > > >
> > > > For the most part, obsoleted endpoints can be maintained with a
> > > > shim
> > > > into the new code. If they would require "an unreasonable amount
> > > > of
> > > > work", it might make sense to remove them, then.
> > > >
> > > > Things that don't seem valuable usually fall into another
> > > > category
> > > > like "cannot reasonably be used", but if the only reason for
> > > > deprecation is that the feature is going away, we should really
> > > > support it according to our promise.
> > > >
> > > > But if it's really valuable to remove these endpoints, maybe it's
> > > > appropriate to bump the major version and ask everybody to update
> > > > their clients?
> > > >
> > > > On Tue, Nov 12, 2019 at 5:27 PM Rawlin Peters <ra...@apache.org>
> > > > wrote:
> > > > > The real crux of the issue here is whether or not we agree as a
> > > > > project that we can move forward with this plan that I believe
> > > > > to be a
> > > > > good compromise that meets most concerns but "breaks the
> > > > > Semantic
> > > > > Versioning promise":
> > > > > 1) don't rewrite deprecated endpoints from Perl to Go
> > > > > 2) in the Perl handler, add a deprecation notice to the
> > > > > "alerts"
> > > > > section of the response
> > > > > 3) once a deprecated API has been deprecated for one major
> > > > > release,
> > > > > delete it from the API in the subsequent major release (e.g. if
> > > > > deprecation notices are added and released in ATC 4.x, we can
> > > > > remove
> > > > > the deprecated endpoint in ATC 5.x)
> > > > >
> > > > > Rob, from your latest proposal to remove the ATS config API
> > > > > endpoints
> > > > > from 1.x, I thought we were past this issue already and could
> > > > > finally
> > > > > agree to remove deprecated endpoints from 1.x (following steps
> > > > > 1-3
> > > > > above) without requiring an all-new 2.x API. Are we not
> > > > > actually past
> > > > > that? You came up with valid reasons for applying these steps
> > > > > to the
> > > > > ATS config API endpoints, and I think it's fair to say that we
> > > > > can
> > > > > come up with valid reasons for applying these steps to other
> > > > > deprecated API endpoints as well.
> > > > >
> > > > > It seems like the real problem is actually agreeing on which
> > > > > _specific_ endpoints we should be able to follow steps 1-3 for.
> > > > > IMO I
> > > > > think it should apply to any 1.x endpoints that we have valid
> > > > > reasons
> > > > > for not carrying forward to Go, including but not limited to:
> > > > > - the endpoints are dangerous and pose a risk to the overall
> > > > > Traffic
> > > > > Control system
> > > > > - the endpoints have been obsoleted by other endpoints
> > > > > - the endpoints don't seem to serve a true purpose or don't
> > > > > seem to be
> > > > > valuable to the project anymore
> > > > > - the task of rewriting the endpoints from Perl to Go
> > > > > represents an
> > > > > unreasonable amount of work for the project
> > > > >
> > > > > Obviously most of these reasons are subjective, so we'd have to
> > > > > come
> > > > > to some level of consensus on deprecating particular endpoints
> > > > > first,
> > > > > but we should be able to follow steps 1-3 for whatever
> > > > > endpoints we
> > > > > decide.
> > > > >
> > > > > Making breaking changes shouldn't be taken lightly, but we need
> > > > > a path
> > > > > forward that allows some calculated, breaking changes where
> > > > > necessary.
> > > > > Otherwise, we are unnecessarily holding the project back.
> > > > >
> > > > > - Rawlin
> > > > >
> > > > > On Tue, Nov 12, 2019 at 11:44 AM Jeremy Mitchell <
> > > > > mitchell852@gmail.com>
> > > > wrote:
> > > > > > Yeah, technically a "rewrite" of 1.x means shifting the
> > > > > > implementation
> > > > from
> > > > > > Perl to Go for ALL of 1.x. If you leave a subset of 1.x in
> > > > > > Perl, then
> > > > you
> > > > > > have to call it a "partial rewrite".
> > > > > >
> > > > > > So, assuming we want to do a full rewrite (no more perl for
> > > > > > the api
> > > > > > whatsoever), I think that means every 1.x api endpoint (good
> > > > > > or bad)
> > > > needs
> > > > > > to be rewritten to Go so we can remove the dependency on
> > > > > > perl.
> > > > > >
> > > > > > However, I still like marking many as "deprecated" so we
> > > > > > train users
> > > > to use
> > > > > > "better" endpoints that we envision will exist in 2.x and
> > > > > > beyond.
> > > > > >
> > > > > > Jeremy
> > > > > >
> > > > > > On Tue, Nov 12, 2019 at 11:22 AM Hoppal, Michael <
> > > > Michael_Hoppal@comcast.com>
> > > > > > wrote:
> > > > > >
> > > > > > > We do not have a lot of deprecated routes as of now as we
> > > > > > > focused on
> > > > > > > routes that are more used first.
> > > > > > >
> > > > > > > I know on a lot of the remaining rewrites there has been
> > > > > > > discussion
> > > > on the
> > > > > > > matching Github issues on potential deprecation (I cannot
> > > > > > > think of
> > > > the
> > > > > > > exact count but enough to warrant this email __)
> > > > > > >
> > > > > > > It is key to note that I am not arguing for removal of
> > > > > > > endpoints
> > > > within
> > > > > > > the current API version but if there is an endpoint we
> > > > > > > agree should
> > > > be
> > > > > > > deprecated going forward to put a message within the
> > > > > > > response and on
> > > > the
> > > > > > > docs.
> > > > > > >
> > > > > > > And thanks for the input sounds like you are +1 on
> > > > > > > rewriting all
> > > > routes to
> > > > > > > Golang as Brennan is.
> > > > > > >
> > > > > > > On 11/12/19, 9:06 AM, "Robert O Butts" <ro...@apache.org>
> > > > > > > wrote:
> > > > > > >
> > > > > > > Whether we rewrite a route in Go is an implementation
> > > > > > > detail. To
> > > > the
> > > > > > > interface, to users, it doesn't matter whether a route
> > > > > > > is
> > > > rewritten or
> > > > > > > not.
> > > > > > >
> > > > > > > But our API follows Semantic Versioning, in order to
> > > > > > > not break
> > > > users.
> > > > > > > We
> > > > > > > can't just remove endpoints that some of us don't use,
> > > > > > > and
> > > > assume other
> > > > > > > people don't, maybe people who never speak up on the
> > > > > > > mailing
> > > > list.
> > > > > > > We'll
> > > > > > > never gain a big userbase if we keep breaking users.
> > > > > > >
> > > > > > > Per the _project_ SemVer, once we have API 2.0, we can
> > > > > > > deprecate
> > > > API
> > > > > > > 1.x,
> > > > > > > and in the next major _project_ release, remove API
> > > > > > > 1.x.
> > > > Irrespective
> > > > > > > of
> > > > > > > Perl or Go.
> > > > > > >
> > > > > > > My big concern is, API 2.0 is a big project. How long
> > > > > > > has the
> > > > rewrite
> > > > > > > to Go
> > > > > > > taken? Do we really believe designing and implementing
> > > > > > > a
> > > > completely
> > > > > > > new API
> > > > > > > will be any less time?
> > > > > > >
> > > > > > > I don't want killing Perl to have to wait on that.
> > > > > > >
> > > > > > > I know it feels like a waste to rewrite routes that you
> > > > > > > don't
> > > > use, and
> > > > > > > probably few people do. But that's the cost of a stable
> > > > > > > project.
> > > > How
> > > > > > > many
> > > > > > > "deprecated" routes are there? If it comes down to
> > > > > > > taking the
> > > > > > > development
> > > > > > > time to rewrite them so we can kill Perl faster, or
> > > > > > > leaving Perl
> > > > > > > around, I
> > > > > > > vote we just do the work and kill Perl.
> > > > > > >
> > > > > > > >If we don't rewrite them, then Perl will last until
> > > > > > > API 2.0 has
> > > > been
> > > > > > > designed, released and then *another full major release
> > > > > > > cycle*.
> > > > That's
> > > > > > > way
> > > > > > > too long to have two codebases for the same component,
> > > > > > > IMO,
> > > > especially
> > > > > > > since the rewrite is already 50% complete.
> > > > > > >
> > > > > > > +1
> > > > > > >
> > > > > > >
> > > > > > > On Tue, Nov 12, 2019 at 8:54 AM ocket 8888 <
> > > > > > > ocket8888@gmail.com>
> > > > > > > wrote:
> > > > > > >
> > > > > > > > I vote that by and large we DO rewrite them, with
> > > > > > > exceptions
> > > > for
> > > > > > > routes
> > > > > > > > that just plain don't work, even in Perl. Those are
> > > > > > > few,
> > > > though.
> > > > > > > >
> > > > > > > > If we don't rewrite them, then Perl will last until
> > > > > > > API 2.0
> > > > has been
> > > > > > > > designed, released and then *another full major
> > > > > > > release cycle*.
> > > > > > > That's way
> > > > > > > > too long to have two codebases for the same
> > > > > > > component, IMO,
> > > > > > > especially
> > > > > > > > since the rewrite is already 50% complete.
> > > > > > > >
> > > > > > > > On Tue, Nov 12, 2019 at 8:43 AM Hoppal, Michael <
> > > > > > > > Michael_Hoppal@comcast.com>
> > > > > > > > wrote:
> > > > > > > >
> > > > > > > > > As the Traffic Ops API is being rewritten from Perl
> > > > > > > to Golang
> > > > > > > there has
> > > > > > > > > been several routes that have been deprecated and
> > > > > > > probably
> > > > more to
> > > > > > > come.
> > > > > > > > >
> > > > > > > > > In the deprecation efforts I have seen two
> > > > > > > strategies:
> > > > > > > > >
> > > > > > > > >
> > > > > > > > > * The route IS NOT rewritten from Perl to
> > > > > > > Golang and a
> > > > > > > deprecation
> > > > > > > > > notice is added to the alert response in the Perl
> > > > > > > handler
> > > > > > > > > * The route IS rewritten from Perl to Golang
> > > > > > > and a
> > > > deprecation
> > > > > > > notice
> > > > > > > > > is added to the alert response in the Golang
> > > > > > > handler
> > > > > > > > >
> > > > > > > > > I think we should have consistency in our approach
> > > > > > > and
> > > > wanted to
> > > > > > > get
> > > > > > > > > people’s thoughts.
> > > > > > > > >
> > > > > > > > > I would vote that we do not rewrite a deprecated
> > > > > > > route from
> > > > Perl to
> > > > > > > > Golang.
> > > > > > > > >
> > > > > > > > > Thanks,
> > > > > > > > >
> > > > > > > > > Michael
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > > >
> > > > > > >
>
>
Re: [EXTERNAL] Re: Traffic Ops Route Deprecation Strategy
Posted by "Schmidt, Andrew (Contractor)" <An...@comcast.com>.
If we decide to get rid of the API versioning it makes sense to me that we would do that on a major version upgrade, probably to the ATC version. But I think it would be too early to do it on the 2.0 API version. I like this compromise +1. I was wondering why we were stuck on the idea that 2.0 had to be an API rewrite. That seems like it’s a long way out. 2.0 = deprecations and fixes. 3.0 or 5.0 = api rewrite and maybe version convergence.
On 11/13/19, 11:52 AM, "ocket8888@gmail.com" <oc...@gmail.com> wrote:
Lemme lead with: making breaking changes on a major version boundary
gets +1 from me - no reason the API needs to be fully redesigned for
any particular number.
But I just gotta bring this up again: if we want to be able to
deprecate an endpoint for a major release and remove it in the next
without needing to deal with the messiness of API versions, we could
always just get rid of API versions. Major ATC revisions should be
easier to come by with our new release strategy.
On Wed, 2019-11-13 at 10:23 -0700, Rawlin Peters wrote:
> > It is a compromise - it's an inconvenience to users, forcing them
> > to fix
> scripts to upgrade. Which isn't good for user experience either; but
> it
> doesn't outright break people. It's a compromise I'm willing to live
> with.
>
> It will literally outright break clients (*all* of them, not just
> *some*). We have been down this road before, so I'll just repeat what
> I said when this exact conversation came up months ago[1]:
>
> "So it basically boils down to:
> A) promote Go 1.x routes to 2.x and eventually break ALL users of the
> 1.x API when the time comes to remove 1.x
> B) deprecate and remove some 1.x Perl routes and POSSIBLY break SOME
> users"
>
> Either way we slice and dice it with versioning, removing an endpoint
> means that clients of that endpoint will break. By graduating only
> the
> "good stuff" from 1.x to 2.x, and removing endpoints from 1.x, we
> will
> unnecessarily break all 1.x clients when the time comes to remove the
> rest of 1.x. Instead, we can choose to break only clients of 1.x APIs
> that we already deemed dangerous, unnecessary, useless, etc. enough
> to
> deprecate.
>
> Why break 100% of our API clients when we can choose to break only a
> small number of clients that happen to use
> dangerous/unnecessary/useless APIs? Why would we break our API
> clients
> unnecessarily for no added value? There is no sense in blindly
> adhering to Semantic Versioning just for the sake of adhering to
> Semantic Versioning. Instead, use it as a guideline, and break the
> rules where it makes the most sense to do so.
>
> - Rawlin
>
> [1]
> https://lists.apache.org/thread.html/15c6bfcdce0ac00a3a8612a96c3d668a8a5a16c6ca9401de9473d31c@%3Cdev.trafficcontrol.apache.org%3E
>
> On Wed, Nov 13, 2019 at 9:43 AM Robert O Butts <ro...@apache.org>
> wrote:
> > > But if it's really valuable to remove these endpoints, maybe it's
> > appropriate to bump the major version and ask everybody to update
> > their
> > clients?
> >
> > +1
> >
> > This sounds like the best compromise to me. It addresses both sides
> > - it
> > lets us remove endpoints, and it also prevents breaking users.
> >
> > It is a compromise - it's an inconvenience to users, forcing them
> > to fix
> > scripts to upgrade. Which isn't good for user experience either;
> > but it
> > doesn't outright break people. It's a compromise I'm willing to
> > live with.
> >
> > This actually lets us do at least 3 things:
> > 1. Remove "deprecated" endpoints.
> > 2. Remove the unmaintained dangerously-large Cache Config
> > endpoints.
> > 3. Lets us do Layered Profiles, changing Servers and DSes to have
> > arrays of
> > Profiles instead of a single Profile, without hacky backward-
> > compatibility
> > attempts.
> >
> > To clarify, `/api/2.0` would be essentially the current API, just
> > with the
> > above breaking changes (possibly others) which require a SemVer
> > major
> > version increase. And the big API rewrite/redesign we've been
> > calling "2.0"
> > would become `/api/3.0` or some future version.
> >
> > It's not perfect, but it seems to address the most concerns on all
> > sides.
> > Thoughts? Objections?
> >
> >
> > On Wed, Nov 13, 2019 at 8:45 AM Chris Lemmons <al...@gmail.com>
> > wrote:
> >
> > > For the endpoints, I agree in part and disagree in part.
> > >
> > > Agree for these two:
> > > - the endpoints are dangerous and pose a risk to the overall
> > > Traffic
> > > Control system
> > > - the task of rewriting the endpoints from Perl to Go represents
> > > an
> > > unreasonable amount of work for the project
> > >
> > > And I would add:
> > > - the endpoint does not serve its intended purpose and cannot
> > > reasonably
> > > be used
> > > - the endpoint has been made unsafe or unusable by new features
> > > added to TC
> > >
> > > Disagree for these two:
> > > - the endpoints have been obsoleted by other endpoints
> > > - the endpoints don't seem to serve a true purpose or don't seem
> > > to be
> > > valuable to the project anymore
> > >
> > > For the most part, obsoleted endpoints can be maintained with a
> > > shim
> > > into the new code. If they would require "an unreasonable amount
> > > of
> > > work", it might make sense to remove them, then.
> > >
> > > Things that don't seem valuable usually fall into another
> > > category
> > > like "cannot reasonably be used", but if the only reason for
> > > deprecation is that the feature is going away, we should really
> > > support it according to our promise.
> > >
> > > But if it's really valuable to remove these endpoints, maybe it's
> > > appropriate to bump the major version and ask everybody to update
> > > their clients?
> > >
> > > On Tue, Nov 12, 2019 at 5:27 PM Rawlin Peters <ra...@apache.org>
> > > wrote:
> > > > The real crux of the issue here is whether or not we agree as a
> > > > project that we can move forward with this plan that I believe
> > > > to be a
> > > > good compromise that meets most concerns but "breaks the
> > > > Semantic
> > > > Versioning promise":
> > > > 1) don't rewrite deprecated endpoints from Perl to Go
> > > > 2) in the Perl handler, add a deprecation notice to the
> > > > "alerts"
> > > > section of the response
> > > > 3) once a deprecated API has been deprecated for one major
> > > > release,
> > > > delete it from the API in the subsequent major release (e.g. if
> > > > deprecation notices are added and released in ATC 4.x, we can
> > > > remove
> > > > the deprecated endpoint in ATC 5.x)
> > > >
> > > > Rob, from your latest proposal to remove the ATS config API
> > > > endpoints
> > > > from 1.x, I thought we were past this issue already and could
> > > > finally
> > > > agree to remove deprecated endpoints from 1.x (following steps
> > > > 1-3
> > > > above) without requiring an all-new 2.x API. Are we not
> > > > actually past
> > > > that? You came up with valid reasons for applying these steps
> > > > to the
> > > > ATS config API endpoints, and I think it's fair to say that we
> > > > can
> > > > come up with valid reasons for applying these steps to other
> > > > deprecated API endpoints as well.
> > > >
> > > > It seems like the real problem is actually agreeing on which
> > > > _specific_ endpoints we should be able to follow steps 1-3 for.
> > > > IMO I
> > > > think it should apply to any 1.x endpoints that we have valid
> > > > reasons
> > > > for not carrying forward to Go, including but not limited to:
> > > > - the endpoints are dangerous and pose a risk to the overall
> > > > Traffic
> > > > Control system
> > > > - the endpoints have been obsoleted by other endpoints
> > > > - the endpoints don't seem to serve a true purpose or don't
> > > > seem to be
> > > > valuable to the project anymore
> > > > - the task of rewriting the endpoints from Perl to Go
> > > > represents an
> > > > unreasonable amount of work for the project
> > > >
> > > > Obviously most of these reasons are subjective, so we'd have to
> > > > come
> > > > to some level of consensus on deprecating particular endpoints
> > > > first,
> > > > but we should be able to follow steps 1-3 for whatever
> > > > endpoints we
> > > > decide.
> > > >
> > > > Making breaking changes shouldn't be taken lightly, but we need
> > > > a path
> > > > forward that allows some calculated, breaking changes where
> > > > necessary.
> > > > Otherwise, we are unnecessarily holding the project back.
> > > >
> > > > - Rawlin
> > > >
> > > > On Tue, Nov 12, 2019 at 11:44 AM Jeremy Mitchell <
> > > > mitchell852@gmail.com>
> > > wrote:
> > > > > Yeah, technically a "rewrite" of 1.x means shifting the
> > > > > implementation
> > > from
> > > > > Perl to Go for ALL of 1.x. If you leave a subset of 1.x in
> > > > > Perl, then
> > > you
> > > > > have to call it a "partial rewrite".
> > > > >
> > > > > So, assuming we want to do a full rewrite (no more perl for
> > > > > the api
> > > > > whatsoever), I think that means every 1.x api endpoint (good
> > > > > or bad)
> > > needs
> > > > > to be rewritten to Go so we can remove the dependency on
> > > > > perl.
> > > > >
> > > > > However, I still like marking many as "deprecated" so we
> > > > > train users
> > > to use
> > > > > "better" endpoints that we envision will exist in 2.x and
> > > > > beyond.
> > > > >
> > > > > Jeremy
> > > > >
> > > > > On Tue, Nov 12, 2019 at 11:22 AM Hoppal, Michael <
> > > Michael_Hoppal@comcast.com>
> > > > > wrote:
> > > > >
> > > > > > We do not have a lot of deprecated routes as of now as we
> > > > > > focused on
> > > > > > routes that are more used first.
> > > > > >
> > > > > > I know on a lot of the remaining rewrites there has been
> > > > > > discussion
> > > on the
> > > > > > matching Github issues on potential deprecation (I cannot
> > > > > > think of
> > > the
> > > > > > exact count but enough to warrant this email __)
> > > > > >
> > > > > > It is key to note that I am not arguing for removal of
> > > > > > endpoints
> > > within
> > > > > > the current API version but if there is an endpoint we
> > > > > > agree should
> > > be
> > > > > > deprecated going forward to put a message within the
> > > > > > response and on
> > > the
> > > > > > docs.
> > > > > >
> > > > > > And thanks for the input sounds like you are +1 on
> > > > > > rewriting all
> > > routes to
> > > > > > Golang as Brennan is.
> > > > > >
> > > > > > On 11/12/19, 9:06 AM, "Robert O Butts" <ro...@apache.org>
> > > > > > wrote:
> > > > > >
> > > > > > Whether we rewrite a route in Go is an implementation
> > > > > > detail. To
> > > the
> > > > > > interface, to users, it doesn't matter whether a route
> > > > > > is
> > > rewritten or
> > > > > > not.
> > > > > >
> > > > > > But our API follows Semantic Versioning, in order to
> > > > > > not break
> > > users.
> > > > > > We
> > > > > > can't just remove endpoints that some of us don't use,
> > > > > > and
> > > assume other
> > > > > > people don't, maybe people who never speak up on the
> > > > > > mailing
> > > list.
> > > > > > We'll
> > > > > > never gain a big userbase if we keep breaking users.
> > > > > >
> > > > > > Per the _project_ SemVer, once we have API 2.0, we can
> > > > > > deprecate
> > > API
> > > > > > 1.x,
> > > > > > and in the next major _project_ release, remove API
> > > > > > 1.x.
> > > Irrespective
> > > > > > of
> > > > > > Perl or Go.
> > > > > >
> > > > > > My big concern is, API 2.0 is a big project. How long
> > > > > > has the
> > > rewrite
> > > > > > to Go
> > > > > > taken? Do we really believe designing and implementing
> > > > > > a
> > > completely
> > > > > > new API
> > > > > > will be any less time?
> > > > > >
> > > > > > I don't want killing Perl to have to wait on that.
> > > > > >
> > > > > > I know it feels like a waste to rewrite routes that you
> > > > > > don't
> > > use, and
> > > > > > probably few people do. But that's the cost of a stable
> > > > > > project.
> > > How
> > > > > > many
> > > > > > "deprecated" routes are there? If it comes down to
> > > > > > taking the
> > > > > > development
> > > > > > time to rewrite them so we can kill Perl faster, or
> > > > > > leaving Perl
> > > > > > around, I
> > > > > > vote we just do the work and kill Perl.
> > > > > >
> > > > > > >If we don't rewrite them, then Perl will last until
> > > > > > API 2.0 has
> > > been
> > > > > > designed, released and then *another full major release
> > > > > > cycle*.
> > > That's
> > > > > > way
> > > > > > too long to have two codebases for the same component,
> > > > > > IMO,
> > > especially
> > > > > > since the rewrite is already 50% complete.
> > > > > >
> > > > > > +1
> > > > > >
> > > > > >
> > > > > > On Tue, Nov 12, 2019 at 8:54 AM ocket 8888 <
> > > > > > ocket8888@gmail.com>
> > > > > > wrote:
> > > > > >
> > > > > > > I vote that by and large we DO rewrite them, with
> > > > > > exceptions
> > > for
> > > > > > routes
> > > > > > > that just plain don't work, even in Perl. Those are
> > > > > > few,
> > > though.
> > > > > > >
> > > > > > > If we don't rewrite them, then Perl will last until
> > > > > > API 2.0
> > > has been
> > > > > > > designed, released and then *another full major
> > > > > > release cycle*.
> > > > > > That's way
> > > > > > > too long to have two codebases for the same
> > > > > > component, IMO,
> > > > > > especially
> > > > > > > since the rewrite is already 50% complete.
> > > > > > >
> > > > > > > On Tue, Nov 12, 2019 at 8:43 AM Hoppal, Michael <
> > > > > > > Michael_Hoppal@comcast.com>
> > > > > > > wrote:
> > > > > > >
> > > > > > > > As the Traffic Ops API is being rewritten from Perl
> > > > > > to Golang
> > > > > > there has
> > > > > > > > been several routes that have been deprecated and
> > > > > > probably
> > > more to
> > > > > > come.
> > > > > > > >
> > > > > > > > In the deprecation efforts I have seen two
> > > > > > strategies:
> > > > > > > >
> > > > > > > >
> > > > > > > > * The route IS NOT rewritten from Perl to
> > > > > > Golang and a
> > > > > > deprecation
> > > > > > > > notice is added to the alert response in the Perl
> > > > > > handler
> > > > > > > > * The route IS rewritten from Perl to Golang
> > > > > > and a
> > > deprecation
> > > > > > notice
> > > > > > > > is added to the alert response in the Golang
> > > > > > handler
> > > > > > > >
> > > > > > > > I think we should have consistency in our approach
> > > > > > and
> > > wanted to
> > > > > > get
> > > > > > > > people’s thoughts.
> > > > > > > >
> > > > > > > > I would vote that we do not rewrite a deprecated
> > > > > > route from
> > > Perl to
> > > > > > > Golang.
> > > > > > > >
> > > > > > > > Thanks,
> > > > > > > >
> > > > > > > > Michael
> > > > > > > >
> > > > > > >
> > > > > >
> > > > > >
> > > > > >
Re: [EXTERNAL] Re: Traffic Ops Route Deprecation Strategy
Posted by oc...@gmail.com.
Lemme lead with: making breaking changes on a major version boundary
gets +1 from me - no reason the API needs to be fully redesigned for
any particular number.
But I just gotta bring this up again: if we want to be able to
deprecate an endpoint for a major release and remove it in the next
without needing to deal with the messiness of API versions, we could
always just get rid of API versions. Major ATC revisions should be
easier to come by with our new release strategy.
On Wed, 2019-11-13 at 10:23 -0700, Rawlin Peters wrote:
> > It is a compromise - it's an inconvenience to users, forcing them
> > to fix
> scripts to upgrade. Which isn't good for user experience either; but
> it
> doesn't outright break people. It's a compromise I'm willing to live
> with.
>
> It will literally outright break clients (*all* of them, not just
> *some*). We have been down this road before, so I'll just repeat what
> I said when this exact conversation came up months ago[1]:
>
> "So it basically boils down to:
> A) promote Go 1.x routes to 2.x and eventually break ALL users of the
> 1.x API when the time comes to remove 1.x
> B) deprecate and remove some 1.x Perl routes and POSSIBLY break SOME
> users"
>
> Either way we slice and dice it with versioning, removing an endpoint
> means that clients of that endpoint will break. By graduating only
> the
> "good stuff" from 1.x to 2.x, and removing endpoints from 1.x, we
> will
> unnecessarily break all 1.x clients when the time comes to remove the
> rest of 1.x. Instead, we can choose to break only clients of 1.x APIs
> that we already deemed dangerous, unnecessary, useless, etc. enough
> to
> deprecate.
>
> Why break 100% of our API clients when we can choose to break only a
> small number of clients that happen to use
> dangerous/unnecessary/useless APIs? Why would we break our API
> clients
> unnecessarily for no added value? There is no sense in blindly
> adhering to Semantic Versioning just for the sake of adhering to
> Semantic Versioning. Instead, use it as a guideline, and break the
> rules where it makes the most sense to do so.
>
> - Rawlin
>
> [1]
> https://lists.apache.org/thread.html/15c6bfcdce0ac00a3a8612a96c3d668a8a5a16c6ca9401de9473d31c@%3Cdev.trafficcontrol.apache.org%3E
>
> On Wed, Nov 13, 2019 at 9:43 AM Robert O Butts <ro...@apache.org>
> wrote:
> > > But if it's really valuable to remove these endpoints, maybe it's
> > appropriate to bump the major version and ask everybody to update
> > their
> > clients?
> >
> > +1
> >
> > This sounds like the best compromise to me. It addresses both sides
> > - it
> > lets us remove endpoints, and it also prevents breaking users.
> >
> > It is a compromise - it's an inconvenience to users, forcing them
> > to fix
> > scripts to upgrade. Which isn't good for user experience either;
> > but it
> > doesn't outright break people. It's a compromise I'm willing to
> > live with.
> >
> > This actually lets us do at least 3 things:
> > 1. Remove "deprecated" endpoints.
> > 2. Remove the unmaintained dangerously-large Cache Config
> > endpoints.
> > 3. Lets us do Layered Profiles, changing Servers and DSes to have
> > arrays of
> > Profiles instead of a single Profile, without hacky backward-
> > compatibility
> > attempts.
> >
> > To clarify, `/api/2.0` would be essentially the current API, just
> > with the
> > above breaking changes (possibly others) which require a SemVer
> > major
> > version increase. And the big API rewrite/redesign we've been
> > calling "2.0"
> > would become `/api/3.0` or some future version.
> >
> > It's not perfect, but it seems to address the most concerns on all
> > sides.
> > Thoughts? Objections?
> >
> >
> > On Wed, Nov 13, 2019 at 8:45 AM Chris Lemmons <al...@gmail.com>
> > wrote:
> >
> > > For the endpoints, I agree in part and disagree in part.
> > >
> > > Agree for these two:
> > > - the endpoints are dangerous and pose a risk to the overall
> > > Traffic
> > > Control system
> > > - the task of rewriting the endpoints from Perl to Go represents
> > > an
> > > unreasonable amount of work for the project
> > >
> > > And I would add:
> > > - the endpoint does not serve its intended purpose and cannot
> > > reasonably
> > > be used
> > > - the endpoint has been made unsafe or unusable by new features
> > > added to TC
> > >
> > > Disagree for these two:
> > > - the endpoints have been obsoleted by other endpoints
> > > - the endpoints don't seem to serve a true purpose or don't seem
> > > to be
> > > valuable to the project anymore
> > >
> > > For the most part, obsoleted endpoints can be maintained with a
> > > shim
> > > into the new code. If they would require "an unreasonable amount
> > > of
> > > work", it might make sense to remove them, then.
> > >
> > > Things that don't seem valuable usually fall into another
> > > category
> > > like "cannot reasonably be used", but if the only reason for
> > > deprecation is that the feature is going away, we should really
> > > support it according to our promise.
> > >
> > > But if it's really valuable to remove these endpoints, maybe it's
> > > appropriate to bump the major version and ask everybody to update
> > > their clients?
> > >
> > > On Tue, Nov 12, 2019 at 5:27 PM Rawlin Peters <ra...@apache.org>
> > > wrote:
> > > > The real crux of the issue here is whether or not we agree as a
> > > > project that we can move forward with this plan that I believe
> > > > to be a
> > > > good compromise that meets most concerns but "breaks the
> > > > Semantic
> > > > Versioning promise":
> > > > 1) don't rewrite deprecated endpoints from Perl to Go
> > > > 2) in the Perl handler, add a deprecation notice to the
> > > > "alerts"
> > > > section of the response
> > > > 3) once a deprecated API has been deprecated for one major
> > > > release,
> > > > delete it from the API in the subsequent major release (e.g. if
> > > > deprecation notices are added and released in ATC 4.x, we can
> > > > remove
> > > > the deprecated endpoint in ATC 5.x)
> > > >
> > > > Rob, from your latest proposal to remove the ATS config API
> > > > endpoints
> > > > from 1.x, I thought we were past this issue already and could
> > > > finally
> > > > agree to remove deprecated endpoints from 1.x (following steps
> > > > 1-3
> > > > above) without requiring an all-new 2.x API. Are we not
> > > > actually past
> > > > that? You came up with valid reasons for applying these steps
> > > > to the
> > > > ATS config API endpoints, and I think it's fair to say that we
> > > > can
> > > > come up with valid reasons for applying these steps to other
> > > > deprecated API endpoints as well.
> > > >
> > > > It seems like the real problem is actually agreeing on which
> > > > _specific_ endpoints we should be able to follow steps 1-3 for.
> > > > IMO I
> > > > think it should apply to any 1.x endpoints that we have valid
> > > > reasons
> > > > for not carrying forward to Go, including but not limited to:
> > > > - the endpoints are dangerous and pose a risk to the overall
> > > > Traffic
> > > > Control system
> > > > - the endpoints have been obsoleted by other endpoints
> > > > - the endpoints don't seem to serve a true purpose or don't
> > > > seem to be
> > > > valuable to the project anymore
> > > > - the task of rewriting the endpoints from Perl to Go
> > > > represents an
> > > > unreasonable amount of work for the project
> > > >
> > > > Obviously most of these reasons are subjective, so we'd have to
> > > > come
> > > > to some level of consensus on deprecating particular endpoints
> > > > first,
> > > > but we should be able to follow steps 1-3 for whatever
> > > > endpoints we
> > > > decide.
> > > >
> > > > Making breaking changes shouldn't be taken lightly, but we need
> > > > a path
> > > > forward that allows some calculated, breaking changes where
> > > > necessary.
> > > > Otherwise, we are unnecessarily holding the project back.
> > > >
> > > > - Rawlin
> > > >
> > > > On Tue, Nov 12, 2019 at 11:44 AM Jeremy Mitchell <
> > > > mitchell852@gmail.com>
> > > wrote:
> > > > > Yeah, technically a "rewrite" of 1.x means shifting the
> > > > > implementation
> > > from
> > > > > Perl to Go for ALL of 1.x. If you leave a subset of 1.x in
> > > > > Perl, then
> > > you
> > > > > have to call it a "partial rewrite".
> > > > >
> > > > > So, assuming we want to do a full rewrite (no more perl for
> > > > > the api
> > > > > whatsoever), I think that means every 1.x api endpoint (good
> > > > > or bad)
> > > needs
> > > > > to be rewritten to Go so we can remove the dependency on
> > > > > perl.
> > > > >
> > > > > However, I still like marking many as "deprecated" so we
> > > > > train users
> > > to use
> > > > > "better" endpoints that we envision will exist in 2.x and
> > > > > beyond.
> > > > >
> > > > > Jeremy
> > > > >
> > > > > On Tue, Nov 12, 2019 at 11:22 AM Hoppal, Michael <
> > > Michael_Hoppal@comcast.com>
> > > > > wrote:
> > > > >
> > > > > > We do not have a lot of deprecated routes as of now as we
> > > > > > focused on
> > > > > > routes that are more used first.
> > > > > >
> > > > > > I know on a lot of the remaining rewrites there has been
> > > > > > discussion
> > > on the
> > > > > > matching Github issues on potential deprecation (I cannot
> > > > > > think of
> > > the
> > > > > > exact count but enough to warrant this email __)
> > > > > >
> > > > > > It is key to note that I am not arguing for removal of
> > > > > > endpoints
> > > within
> > > > > > the current API version but if there is an endpoint we
> > > > > > agree should
> > > be
> > > > > > deprecated going forward to put a message within the
> > > > > > response and on
> > > the
> > > > > > docs.
> > > > > >
> > > > > > And thanks for the input sounds like you are +1 on
> > > > > > rewriting all
> > > routes to
> > > > > > Golang as Brennan is.
> > > > > >
> > > > > > On 11/12/19, 9:06 AM, "Robert O Butts" <ro...@apache.org>
> > > > > > wrote:
> > > > > >
> > > > > > Whether we rewrite a route in Go is an implementation
> > > > > > detail. To
> > > the
> > > > > > interface, to users, it doesn't matter whether a route
> > > > > > is
> > > rewritten or
> > > > > > not.
> > > > > >
> > > > > > But our API follows Semantic Versioning, in order to
> > > > > > not break
> > > users.
> > > > > > We
> > > > > > can't just remove endpoints that some of us don't use,
> > > > > > and
> > > assume other
> > > > > > people don't, maybe people who never speak up on the
> > > > > > mailing
> > > list.
> > > > > > We'll
> > > > > > never gain a big userbase if we keep breaking users.
> > > > > >
> > > > > > Per the _project_ SemVer, once we have API 2.0, we can
> > > > > > deprecate
> > > API
> > > > > > 1.x,
> > > > > > and in the next major _project_ release, remove API
> > > > > > 1.x.
> > > Irrespective
> > > > > > of
> > > > > > Perl or Go.
> > > > > >
> > > > > > My big concern is, API 2.0 is a big project. How long
> > > > > > has the
> > > rewrite
> > > > > > to Go
> > > > > > taken? Do we really believe designing and implementing
> > > > > > a
> > > completely
> > > > > > new API
> > > > > > will be any less time?
> > > > > >
> > > > > > I don't want killing Perl to have to wait on that.
> > > > > >
> > > > > > I know it feels like a waste to rewrite routes that you
> > > > > > don't
> > > use, and
> > > > > > probably few people do. But that's the cost of a stable
> > > > > > project.
> > > How
> > > > > > many
> > > > > > "deprecated" routes are there? If it comes down to
> > > > > > taking the
> > > > > > development
> > > > > > time to rewrite them so we can kill Perl faster, or
> > > > > > leaving Perl
> > > > > > around, I
> > > > > > vote we just do the work and kill Perl.
> > > > > >
> > > > > > >If we don't rewrite them, then Perl will last until
> > > > > > API 2.0 has
> > > been
> > > > > > designed, released and then *another full major release
> > > > > > cycle*.
> > > That's
> > > > > > way
> > > > > > too long to have two codebases for the same component,
> > > > > > IMO,
> > > especially
> > > > > > since the rewrite is already 50% complete.
> > > > > >
> > > > > > +1
> > > > > >
> > > > > >
> > > > > > On Tue, Nov 12, 2019 at 8:54 AM ocket 8888 <
> > > > > > ocket8888@gmail.com>
> > > > > > wrote:
> > > > > >
> > > > > > > I vote that by and large we DO rewrite them, with
> > > > > > exceptions
> > > for
> > > > > > routes
> > > > > > > that just plain don't work, even in Perl. Those are
> > > > > > few,
> > > though.
> > > > > > >
> > > > > > > If we don't rewrite them, then Perl will last until
> > > > > > API 2.0
> > > has been
> > > > > > > designed, released and then *another full major
> > > > > > release cycle*.
> > > > > > That's way
> > > > > > > too long to have two codebases for the same
> > > > > > component, IMO,
> > > > > > especially
> > > > > > > since the rewrite is already 50% complete.
> > > > > > >
> > > > > > > On Tue, Nov 12, 2019 at 8:43 AM Hoppal, Michael <
> > > > > > > Michael_Hoppal@comcast.com>
> > > > > > > wrote:
> > > > > > >
> > > > > > > > As the Traffic Ops API is being rewritten from Perl
> > > > > > to Golang
> > > > > > there has
> > > > > > > > been several routes that have been deprecated and
> > > > > > probably
> > > more to
> > > > > > come.
> > > > > > > >
> > > > > > > > In the deprecation efforts I have seen two
> > > > > > strategies:
> > > > > > > >
> > > > > > > >
> > > > > > > > * The route IS NOT rewritten from Perl to
> > > > > > Golang and a
> > > > > > deprecation
> > > > > > > > notice is added to the alert response in the Perl
> > > > > > handler
> > > > > > > > * The route IS rewritten from Perl to Golang
> > > > > > and a
> > > deprecation
> > > > > > notice
> > > > > > > > is added to the alert response in the Golang
> > > > > > handler
> > > > > > > >
> > > > > > > > I think we should have consistency in our approach
> > > > > > and
> > > wanted to
> > > > > > get
> > > > > > > > people’s thoughts.
> > > > > > > >
> > > > > > > > I would vote that we do not rewrite a deprecated
> > > > > > route from
> > > Perl to
> > > > > > > Golang.
> > > > > > > >
> > > > > > > > Thanks,
> > > > > > > >
> > > > > > > > Michael
> > > > > > > >
> > > > > > >
> > > > > >
> > > > > >
> > > > > >
Re: [EXTERNAL] Re: Traffic Ops Route Deprecation Strategy
Posted by Rawlin Peters <ra...@apache.org>.
> It is a compromise - it's an inconvenience to users, forcing them to fix
scripts to upgrade. Which isn't good for user experience either; but it
doesn't outright break people. It's a compromise I'm willing to live with.
It will literally outright break clients (*all* of them, not just
*some*). We have been down this road before, so I'll just repeat what
I said when this exact conversation came up months ago[1]:
"So it basically boils down to:
A) promote Go 1.x routes to 2.x and eventually break ALL users of the
1.x API when the time comes to remove 1.x
B) deprecate and remove some 1.x Perl routes and POSSIBLY break SOME users"
Either way we slice and dice it with versioning, removing an endpoint
means that clients of that endpoint will break. By graduating only the
"good stuff" from 1.x to 2.x, and removing endpoints from 1.x, we will
unnecessarily break all 1.x clients when the time comes to remove the
rest of 1.x. Instead, we can choose to break only clients of 1.x APIs
that we already deemed dangerous, unnecessary, useless, etc. enough to
deprecate.
Why break 100% of our API clients when we can choose to break only a
small number of clients that happen to use
dangerous/unnecessary/useless APIs? Why would we break our API clients
unnecessarily for no added value? There is no sense in blindly
adhering to Semantic Versioning just for the sake of adhering to
Semantic Versioning. Instead, use it as a guideline, and break the
rules where it makes the most sense to do so.
- Rawlin
[1] https://lists.apache.org/thread.html/15c6bfcdce0ac00a3a8612a96c3d668a8a5a16c6ca9401de9473d31c@%3Cdev.trafficcontrol.apache.org%3E
On Wed, Nov 13, 2019 at 9:43 AM Robert O Butts <ro...@apache.org> wrote:
>
> > But if it's really valuable to remove these endpoints, maybe it's
> appropriate to bump the major version and ask everybody to update their
> clients?
>
> +1
>
> This sounds like the best compromise to me. It addresses both sides - it
> lets us remove endpoints, and it also prevents breaking users.
>
> It is a compromise - it's an inconvenience to users, forcing them to fix
> scripts to upgrade. Which isn't good for user experience either; but it
> doesn't outright break people. It's a compromise I'm willing to live with.
>
> This actually lets us do at least 3 things:
> 1. Remove "deprecated" endpoints.
> 2. Remove the unmaintained dangerously-large Cache Config endpoints.
> 3. Lets us do Layered Profiles, changing Servers and DSes to have arrays of
> Profiles instead of a single Profile, without hacky backward-compatibility
> attempts.
>
> To clarify, `/api/2.0` would be essentially the current API, just with the
> above breaking changes (possibly others) which require a SemVer major
> version increase. And the big API rewrite/redesign we've been calling "2.0"
> would become `/api/3.0` or some future version.
>
> It's not perfect, but it seems to address the most concerns on all sides.
> Thoughts? Objections?
>
>
> On Wed, Nov 13, 2019 at 8:45 AM Chris Lemmons <al...@gmail.com> wrote:
>
> > For the endpoints, I agree in part and disagree in part.
> >
> > Agree for these two:
> > - the endpoints are dangerous and pose a risk to the overall Traffic
> > Control system
> > - the task of rewriting the endpoints from Perl to Go represents an
> > unreasonable amount of work for the project
> >
> > And I would add:
> > - the endpoint does not serve its intended purpose and cannot reasonably
> > be used
> > - the endpoint has been made unsafe or unusable by new features added to TC
> >
> > Disagree for these two:
> > - the endpoints have been obsoleted by other endpoints
> > - the endpoints don't seem to serve a true purpose or don't seem to be
> > valuable to the project anymore
> >
> > For the most part, obsoleted endpoints can be maintained with a shim
> > into the new code. If they would require "an unreasonable amount of
> > work", it might make sense to remove them, then.
> >
> > Things that don't seem valuable usually fall into another category
> > like "cannot reasonably be used", but if the only reason for
> > deprecation is that the feature is going away, we should really
> > support it according to our promise.
> >
> > But if it's really valuable to remove these endpoints, maybe it's
> > appropriate to bump the major version and ask everybody to update
> > their clients?
> >
> > On Tue, Nov 12, 2019 at 5:27 PM Rawlin Peters <ra...@apache.org> wrote:
> > >
> > > The real crux of the issue here is whether or not we agree as a
> > > project that we can move forward with this plan that I believe to be a
> > > good compromise that meets most concerns but "breaks the Semantic
> > > Versioning promise":
> > > 1) don't rewrite deprecated endpoints from Perl to Go
> > > 2) in the Perl handler, add a deprecation notice to the "alerts"
> > > section of the response
> > > 3) once a deprecated API has been deprecated for one major release,
> > > delete it from the API in the subsequent major release (e.g. if
> > > deprecation notices are added and released in ATC 4.x, we can remove
> > > the deprecated endpoint in ATC 5.x)
> > >
> > > Rob, from your latest proposal to remove the ATS config API endpoints
> > > from 1.x, I thought we were past this issue already and could finally
> > > agree to remove deprecated endpoints from 1.x (following steps 1-3
> > > above) without requiring an all-new 2.x API. Are we not actually past
> > > that? You came up with valid reasons for applying these steps to the
> > > ATS config API endpoints, and I think it's fair to say that we can
> > > come up with valid reasons for applying these steps to other
> > > deprecated API endpoints as well.
> > >
> > > It seems like the real problem is actually agreeing on which
> > > _specific_ endpoints we should be able to follow steps 1-3 for. IMO I
> > > think it should apply to any 1.x endpoints that we have valid reasons
> > > for not carrying forward to Go, including but not limited to:
> > > - the endpoints are dangerous and pose a risk to the overall Traffic
> > > Control system
> > > - the endpoints have been obsoleted by other endpoints
> > > - the endpoints don't seem to serve a true purpose or don't seem to be
> > > valuable to the project anymore
> > > - the task of rewriting the endpoints from Perl to Go represents an
> > > unreasonable amount of work for the project
> > >
> > > Obviously most of these reasons are subjective, so we'd have to come
> > > to some level of consensus on deprecating particular endpoints first,
> > > but we should be able to follow steps 1-3 for whatever endpoints we
> > > decide.
> > >
> > > Making breaking changes shouldn't be taken lightly, but we need a path
> > > forward that allows some calculated, breaking changes where necessary.
> > > Otherwise, we are unnecessarily holding the project back.
> > >
> > > - Rawlin
> > >
> > > On Tue, Nov 12, 2019 at 11:44 AM Jeremy Mitchell <mi...@gmail.com>
> > wrote:
> > > >
> > > > Yeah, technically a "rewrite" of 1.x means shifting the implementation
> > from
> > > > Perl to Go for ALL of 1.x. If you leave a subset of 1.x in Perl, then
> > you
> > > > have to call it a "partial rewrite".
> > > >
> > > > So, assuming we want to do a full rewrite (no more perl for the api
> > > > whatsoever), I think that means every 1.x api endpoint (good or bad)
> > needs
> > > > to be rewritten to Go so we can remove the dependency on perl.
> > > >
> > > > However, I still like marking many as "deprecated" so we train users
> > to use
> > > > "better" endpoints that we envision will exist in 2.x and beyond.
> > > >
> > > > Jeremy
> > > >
> > > > On Tue, Nov 12, 2019 at 11:22 AM Hoppal, Michael <
> > Michael_Hoppal@comcast.com>
> > > > wrote:
> > > >
> > > > > We do not have a lot of deprecated routes as of now as we focused on
> > > > > routes that are more used first.
> > > > >
> > > > > I know on a lot of the remaining rewrites there has been discussion
> > on the
> > > > > matching Github issues on potential deprecation (I cannot think of
> > the
> > > > > exact count but enough to warrant this email __)
> > > > >
> > > > > It is key to note that I am not arguing for removal of endpoints
> > within
> > > > > the current API version but if there is an endpoint we agree should
> > be
> > > > > deprecated going forward to put a message within the response and on
> > the
> > > > > docs.
> > > > >
> > > > > And thanks for the input sounds like you are +1 on rewriting all
> > routes to
> > > > > Golang as Brennan is.
> > > > >
> > > > > On 11/12/19, 9:06 AM, "Robert O Butts" <ro...@apache.org> wrote:
> > > > >
> > > > > Whether we rewrite a route in Go is an implementation detail. To
> > the
> > > > > interface, to users, it doesn't matter whether a route is
> > rewritten or
> > > > > not.
> > > > >
> > > > > But our API follows Semantic Versioning, in order to not break
> > users.
> > > > > We
> > > > > can't just remove endpoints that some of us don't use, and
> > assume other
> > > > > people don't, maybe people who never speak up on the mailing
> > list.
> > > > > We'll
> > > > > never gain a big userbase if we keep breaking users.
> > > > >
> > > > > Per the _project_ SemVer, once we have API 2.0, we can deprecate
> > API
> > > > > 1.x,
> > > > > and in the next major _project_ release, remove API 1.x.
> > Irrespective
> > > > > of
> > > > > Perl or Go.
> > > > >
> > > > > My big concern is, API 2.0 is a big project. How long has the
> > rewrite
> > > > > to Go
> > > > > taken? Do we really believe designing and implementing a
> > completely
> > > > > new API
> > > > > will be any less time?
> > > > >
> > > > > I don't want killing Perl to have to wait on that.
> > > > >
> > > > > I know it feels like a waste to rewrite routes that you don't
> > use, and
> > > > > probably few people do. But that's the cost of a stable project.
> > How
> > > > > many
> > > > > "deprecated" routes are there? If it comes down to taking the
> > > > > development
> > > > > time to rewrite them so we can kill Perl faster, or leaving Perl
> > > > > around, I
> > > > > vote we just do the work and kill Perl.
> > > > >
> > > > > >If we don't rewrite them, then Perl will last until API 2.0 has
> > been
> > > > > designed, released and then *another full major release cycle*.
> > That's
> > > > > way
> > > > > too long to have two codebases for the same component, IMO,
> > especially
> > > > > since the rewrite is already 50% complete.
> > > > >
> > > > > +1
> > > > >
> > > > >
> > > > > On Tue, Nov 12, 2019 at 8:54 AM ocket 8888 <oc...@gmail.com>
> > > > > wrote:
> > > > >
> > > > > > I vote that by and large we DO rewrite them, with exceptions
> > for
> > > > > routes
> > > > > > that just plain don't work, even in Perl. Those are few,
> > though.
> > > > > >
> > > > > > If we don't rewrite them, then Perl will last until API 2.0
> > has been
> > > > > > designed, released and then *another full major release cycle*.
> > > > > That's way
> > > > > > too long to have two codebases for the same component, IMO,
> > > > > especially
> > > > > > since the rewrite is already 50% complete.
> > > > > >
> > > > > > On Tue, Nov 12, 2019 at 8:43 AM Hoppal, Michael <
> > > > > > Michael_Hoppal@comcast.com>
> > > > > > wrote:
> > > > > >
> > > > > > > As the Traffic Ops API is being rewritten from Perl to Golang
> > > > > there has
> > > > > > > been several routes that have been deprecated and probably
> > more to
> > > > > come.
> > > > > > >
> > > > > > > In the deprecation efforts I have seen two strategies:
> > > > > > >
> > > > > > >
> > > > > > > * The route IS NOT rewritten from Perl to Golang and a
> > > > > deprecation
> > > > > > > notice is added to the alert response in the Perl handler
> > > > > > > * The route IS rewritten from Perl to Golang and a
> > deprecation
> > > > > notice
> > > > > > > is added to the alert response in the Golang handler
> > > > > > >
> > > > > > > I think we should have consistency in our approach and
> > wanted to
> > > > > get
> > > > > > > people’s thoughts.
> > > > > > >
> > > > > > > I would vote that we do not rewrite a deprecated route from
> > Perl to
> > > > > > Golang.
> > > > > > >
> > > > > > > Thanks,
> > > > > > >
> > > > > > > Michael
> > > > > > >
> > > > > >
> > > > >
> > > > >
> > > > >
> >
Re: [EXTERNAL] Re: Traffic Ops Route Deprecation Strategy
Posted by Robert O Butts <ro...@apache.org>.
> But if it's really valuable to remove these endpoints, maybe it's
appropriate to bump the major version and ask everybody to update their
clients?
+1
This sounds like the best compromise to me. It addresses both sides - it
lets us remove endpoints, and it also prevents breaking users.
It is a compromise - it's an inconvenience to users, forcing them to fix
scripts to upgrade. Which isn't good for user experience either; but it
doesn't outright break people. It's a compromise I'm willing to live with.
This actually lets us do at least 3 things:
1. Remove "deprecated" endpoints.
2. Remove the unmaintained dangerously-large Cache Config endpoints.
3. Lets us do Layered Profiles, changing Servers and DSes to have arrays of
Profiles instead of a single Profile, without hacky backward-compatibility
attempts.
To clarify, `/api/2.0` would be essentially the current API, just with the
above breaking changes (possibly others) which require a SemVer major
version increase. And the big API rewrite/redesign we've been calling "2.0"
would become `/api/3.0` or some future version.
It's not perfect, but it seems to address the most concerns on all sides.
Thoughts? Objections?
On Wed, Nov 13, 2019 at 8:45 AM Chris Lemmons <al...@gmail.com> wrote:
> For the endpoints, I agree in part and disagree in part.
>
> Agree for these two:
> - the endpoints are dangerous and pose a risk to the overall Traffic
> Control system
> - the task of rewriting the endpoints from Perl to Go represents an
> unreasonable amount of work for the project
>
> And I would add:
> - the endpoint does not serve its intended purpose and cannot reasonably
> be used
> - the endpoint has been made unsafe or unusable by new features added to TC
>
> Disagree for these two:
> - the endpoints have been obsoleted by other endpoints
> - the endpoints don't seem to serve a true purpose or don't seem to be
> valuable to the project anymore
>
> For the most part, obsoleted endpoints can be maintained with a shim
> into the new code. If they would require "an unreasonable amount of
> work", it might make sense to remove them, then.
>
> Things that don't seem valuable usually fall into another category
> like "cannot reasonably be used", but if the only reason for
> deprecation is that the feature is going away, we should really
> support it according to our promise.
>
> But if it's really valuable to remove these endpoints, maybe it's
> appropriate to bump the major version and ask everybody to update
> their clients?
>
> On Tue, Nov 12, 2019 at 5:27 PM Rawlin Peters <ra...@apache.org> wrote:
> >
> > The real crux of the issue here is whether or not we agree as a
> > project that we can move forward with this plan that I believe to be a
> > good compromise that meets most concerns but "breaks the Semantic
> > Versioning promise":
> > 1) don't rewrite deprecated endpoints from Perl to Go
> > 2) in the Perl handler, add a deprecation notice to the "alerts"
> > section of the response
> > 3) once a deprecated API has been deprecated for one major release,
> > delete it from the API in the subsequent major release (e.g. if
> > deprecation notices are added and released in ATC 4.x, we can remove
> > the deprecated endpoint in ATC 5.x)
> >
> > Rob, from your latest proposal to remove the ATS config API endpoints
> > from 1.x, I thought we were past this issue already and could finally
> > agree to remove deprecated endpoints from 1.x (following steps 1-3
> > above) without requiring an all-new 2.x API. Are we not actually past
> > that? You came up with valid reasons for applying these steps to the
> > ATS config API endpoints, and I think it's fair to say that we can
> > come up with valid reasons for applying these steps to other
> > deprecated API endpoints as well.
> >
> > It seems like the real problem is actually agreeing on which
> > _specific_ endpoints we should be able to follow steps 1-3 for. IMO I
> > think it should apply to any 1.x endpoints that we have valid reasons
> > for not carrying forward to Go, including but not limited to:
> > - the endpoints are dangerous and pose a risk to the overall Traffic
> > Control system
> > - the endpoints have been obsoleted by other endpoints
> > - the endpoints don't seem to serve a true purpose or don't seem to be
> > valuable to the project anymore
> > - the task of rewriting the endpoints from Perl to Go represents an
> > unreasonable amount of work for the project
> >
> > Obviously most of these reasons are subjective, so we'd have to come
> > to some level of consensus on deprecating particular endpoints first,
> > but we should be able to follow steps 1-3 for whatever endpoints we
> > decide.
> >
> > Making breaking changes shouldn't be taken lightly, but we need a path
> > forward that allows some calculated, breaking changes where necessary.
> > Otherwise, we are unnecessarily holding the project back.
> >
> > - Rawlin
> >
> > On Tue, Nov 12, 2019 at 11:44 AM Jeremy Mitchell <mi...@gmail.com>
> wrote:
> > >
> > > Yeah, technically a "rewrite" of 1.x means shifting the implementation
> from
> > > Perl to Go for ALL of 1.x. If you leave a subset of 1.x in Perl, then
> you
> > > have to call it a "partial rewrite".
> > >
> > > So, assuming we want to do a full rewrite (no more perl for the api
> > > whatsoever), I think that means every 1.x api endpoint (good or bad)
> needs
> > > to be rewritten to Go so we can remove the dependency on perl.
> > >
> > > However, I still like marking many as "deprecated" so we train users
> to use
> > > "better" endpoints that we envision will exist in 2.x and beyond.
> > >
> > > Jeremy
> > >
> > > On Tue, Nov 12, 2019 at 11:22 AM Hoppal, Michael <
> Michael_Hoppal@comcast.com>
> > > wrote:
> > >
> > > > We do not have a lot of deprecated routes as of now as we focused on
> > > > routes that are more used first.
> > > >
> > > > I know on a lot of the remaining rewrites there has been discussion
> on the
> > > > matching Github issues on potential deprecation (I cannot think of
> the
> > > > exact count but enough to warrant this email __)
> > > >
> > > > It is key to note that I am not arguing for removal of endpoints
> within
> > > > the current API version but if there is an endpoint we agree should
> be
> > > > deprecated going forward to put a message within the response and on
> the
> > > > docs.
> > > >
> > > > And thanks for the input sounds like you are +1 on rewriting all
> routes to
> > > > Golang as Brennan is.
> > > >
> > > > On 11/12/19, 9:06 AM, "Robert O Butts" <ro...@apache.org> wrote:
> > > >
> > > > Whether we rewrite a route in Go is an implementation detail. To
> the
> > > > interface, to users, it doesn't matter whether a route is
> rewritten or
> > > > not.
> > > >
> > > > But our API follows Semantic Versioning, in order to not break
> users.
> > > > We
> > > > can't just remove endpoints that some of us don't use, and
> assume other
> > > > people don't, maybe people who never speak up on the mailing
> list.
> > > > We'll
> > > > never gain a big userbase if we keep breaking users.
> > > >
> > > > Per the _project_ SemVer, once we have API 2.0, we can deprecate
> API
> > > > 1.x,
> > > > and in the next major _project_ release, remove API 1.x.
> Irrespective
> > > > of
> > > > Perl or Go.
> > > >
> > > > My big concern is, API 2.0 is a big project. How long has the
> rewrite
> > > > to Go
> > > > taken? Do we really believe designing and implementing a
> completely
> > > > new API
> > > > will be any less time?
> > > >
> > > > I don't want killing Perl to have to wait on that.
> > > >
> > > > I know it feels like a waste to rewrite routes that you don't
> use, and
> > > > probably few people do. But that's the cost of a stable project.
> How
> > > > many
> > > > "deprecated" routes are there? If it comes down to taking the
> > > > development
> > > > time to rewrite them so we can kill Perl faster, or leaving Perl
> > > > around, I
> > > > vote we just do the work and kill Perl.
> > > >
> > > > >If we don't rewrite them, then Perl will last until API 2.0 has
> been
> > > > designed, released and then *another full major release cycle*.
> That's
> > > > way
> > > > too long to have two codebases for the same component, IMO,
> especially
> > > > since the rewrite is already 50% complete.
> > > >
> > > > +1
> > > >
> > > >
> > > > On Tue, Nov 12, 2019 at 8:54 AM ocket 8888 <oc...@gmail.com>
> > > > wrote:
> > > >
> > > > > I vote that by and large we DO rewrite them, with exceptions
> for
> > > > routes
> > > > > that just plain don't work, even in Perl. Those are few,
> though.
> > > > >
> > > > > If we don't rewrite them, then Perl will last until API 2.0
> has been
> > > > > designed, released and then *another full major release cycle*.
> > > > That's way
> > > > > too long to have two codebases for the same component, IMO,
> > > > especially
> > > > > since the rewrite is already 50% complete.
> > > > >
> > > > > On Tue, Nov 12, 2019 at 8:43 AM Hoppal, Michael <
> > > > > Michael_Hoppal@comcast.com>
> > > > > wrote:
> > > > >
> > > > > > As the Traffic Ops API is being rewritten from Perl to Golang
> > > > there has
> > > > > > been several routes that have been deprecated and probably
> more to
> > > > come.
> > > > > >
> > > > > > In the deprecation efforts I have seen two strategies:
> > > > > >
> > > > > >
> > > > > > * The route IS NOT rewritten from Perl to Golang and a
> > > > deprecation
> > > > > > notice is added to the alert response in the Perl handler
> > > > > > * The route IS rewritten from Perl to Golang and a
> deprecation
> > > > notice
> > > > > > is added to the alert response in the Golang handler
> > > > > >
> > > > > > I think we should have consistency in our approach and
> wanted to
> > > > get
> > > > > > people’s thoughts.
> > > > > >
> > > > > > I would vote that we do not rewrite a deprecated route from
> Perl to
> > > > > Golang.
> > > > > >
> > > > > > Thanks,
> > > > > >
> > > > > > Michael
> > > > > >
> > > > >
> > > >
> > > >
> > > >
>
Re: [EXTERNAL] Re: Traffic Ops Route Deprecation Strategy
Posted by Chris Lemmons <al...@gmail.com>.
For the endpoints, I agree in part and disagree in part.
Agree for these two:
- the endpoints are dangerous and pose a risk to the overall Traffic
Control system
- the task of rewriting the endpoints from Perl to Go represents an
unreasonable amount of work for the project
And I would add:
- the endpoint does not serve its intended purpose and cannot reasonably be used
- the endpoint has been made unsafe or unusable by new features added to TC
Disagree for these two:
- the endpoints have been obsoleted by other endpoints
- the endpoints don't seem to serve a true purpose or don't seem to be
valuable to the project anymore
For the most part, obsoleted endpoints can be maintained with a shim
into the new code. If they would require "an unreasonable amount of
work", it might make sense to remove them, then.
Things that don't seem valuable usually fall into another category
like "cannot reasonably be used", but if the only reason for
deprecation is that the feature is going away, we should really
support it according to our promise.
But if it's really valuable to remove these endpoints, maybe it's
appropriate to bump the major version and ask everybody to update
their clients?
On Tue, Nov 12, 2019 at 5:27 PM Rawlin Peters <ra...@apache.org> wrote:
>
> The real crux of the issue here is whether or not we agree as a
> project that we can move forward with this plan that I believe to be a
> good compromise that meets most concerns but "breaks the Semantic
> Versioning promise":
> 1) don't rewrite deprecated endpoints from Perl to Go
> 2) in the Perl handler, add a deprecation notice to the "alerts"
> section of the response
> 3) once a deprecated API has been deprecated for one major release,
> delete it from the API in the subsequent major release (e.g. if
> deprecation notices are added and released in ATC 4.x, we can remove
> the deprecated endpoint in ATC 5.x)
>
> Rob, from your latest proposal to remove the ATS config API endpoints
> from 1.x, I thought we were past this issue already and could finally
> agree to remove deprecated endpoints from 1.x (following steps 1-3
> above) without requiring an all-new 2.x API. Are we not actually past
> that? You came up with valid reasons for applying these steps to the
> ATS config API endpoints, and I think it's fair to say that we can
> come up with valid reasons for applying these steps to other
> deprecated API endpoints as well.
>
> It seems like the real problem is actually agreeing on which
> _specific_ endpoints we should be able to follow steps 1-3 for. IMO I
> think it should apply to any 1.x endpoints that we have valid reasons
> for not carrying forward to Go, including but not limited to:
> - the endpoints are dangerous and pose a risk to the overall Traffic
> Control system
> - the endpoints have been obsoleted by other endpoints
> - the endpoints don't seem to serve a true purpose or don't seem to be
> valuable to the project anymore
> - the task of rewriting the endpoints from Perl to Go represents an
> unreasonable amount of work for the project
>
> Obviously most of these reasons are subjective, so we'd have to come
> to some level of consensus on deprecating particular endpoints first,
> but we should be able to follow steps 1-3 for whatever endpoints we
> decide.
>
> Making breaking changes shouldn't be taken lightly, but we need a path
> forward that allows some calculated, breaking changes where necessary.
> Otherwise, we are unnecessarily holding the project back.
>
> - Rawlin
>
> On Tue, Nov 12, 2019 at 11:44 AM Jeremy Mitchell <mi...@gmail.com> wrote:
> >
> > Yeah, technically a "rewrite" of 1.x means shifting the implementation from
> > Perl to Go for ALL of 1.x. If you leave a subset of 1.x in Perl, then you
> > have to call it a "partial rewrite".
> >
> > So, assuming we want to do a full rewrite (no more perl for the api
> > whatsoever), I think that means every 1.x api endpoint (good or bad) needs
> > to be rewritten to Go so we can remove the dependency on perl.
> >
> > However, I still like marking many as "deprecated" so we train users to use
> > "better" endpoints that we envision will exist in 2.x and beyond.
> >
> > Jeremy
> >
> > On Tue, Nov 12, 2019 at 11:22 AM Hoppal, Michael <Mi...@comcast.com>
> > wrote:
> >
> > > We do not have a lot of deprecated routes as of now as we focused on
> > > routes that are more used first.
> > >
> > > I know on a lot of the remaining rewrites there has been discussion on the
> > > matching Github issues on potential deprecation (I cannot think of the
> > > exact count but enough to warrant this email __)
> > >
> > > It is key to note that I am not arguing for removal of endpoints within
> > > the current API version but if there is an endpoint we agree should be
> > > deprecated going forward to put a message within the response and on the
> > > docs.
> > >
> > > And thanks for the input sounds like you are +1 on rewriting all routes to
> > > Golang as Brennan is.
> > >
> > > On 11/12/19, 9:06 AM, "Robert O Butts" <ro...@apache.org> wrote:
> > >
> > > Whether we rewrite a route in Go is an implementation detail. To the
> > > interface, to users, it doesn't matter whether a route is rewritten or
> > > not.
> > >
> > > But our API follows Semantic Versioning, in order to not break users.
> > > We
> > > can't just remove endpoints that some of us don't use, and assume other
> > > people don't, maybe people who never speak up on the mailing list.
> > > We'll
> > > never gain a big userbase if we keep breaking users.
> > >
> > > Per the _project_ SemVer, once we have API 2.0, we can deprecate API
> > > 1.x,
> > > and in the next major _project_ release, remove API 1.x. Irrespective
> > > of
> > > Perl or Go.
> > >
> > > My big concern is, API 2.0 is a big project. How long has the rewrite
> > > to Go
> > > taken? Do we really believe designing and implementing a completely
> > > new API
> > > will be any less time?
> > >
> > > I don't want killing Perl to have to wait on that.
> > >
> > > I know it feels like a waste to rewrite routes that you don't use, and
> > > probably few people do. But that's the cost of a stable project. How
> > > many
> > > "deprecated" routes are there? If it comes down to taking the
> > > development
> > > time to rewrite them so we can kill Perl faster, or leaving Perl
> > > around, I
> > > vote we just do the work and kill Perl.
> > >
> > > >If we don't rewrite them, then Perl will last until API 2.0 has been
> > > designed, released and then *another full major release cycle*. That's
> > > way
> > > too long to have two codebases for the same component, IMO, especially
> > > since the rewrite is already 50% complete.
> > >
> > > +1
> > >
> > >
> > > On Tue, Nov 12, 2019 at 8:54 AM ocket 8888 <oc...@gmail.com>
> > > wrote:
> > >
> > > > I vote that by and large we DO rewrite them, with exceptions for
> > > routes
> > > > that just plain don't work, even in Perl. Those are few, though.
> > > >
> > > > If we don't rewrite them, then Perl will last until API 2.0 has been
> > > > designed, released and then *another full major release cycle*.
> > > That's way
> > > > too long to have two codebases for the same component, IMO,
> > > especially
> > > > since the rewrite is already 50% complete.
> > > >
> > > > On Tue, Nov 12, 2019 at 8:43 AM Hoppal, Michael <
> > > > Michael_Hoppal@comcast.com>
> > > > wrote:
> > > >
> > > > > As the Traffic Ops API is being rewritten from Perl to Golang
> > > there has
> > > > > been several routes that have been deprecated and probably more to
> > > come.
> > > > >
> > > > > In the deprecation efforts I have seen two strategies:
> > > > >
> > > > >
> > > > > * The route IS NOT rewritten from Perl to Golang and a
> > > deprecation
> > > > > notice is added to the alert response in the Perl handler
> > > > > * The route IS rewritten from Perl to Golang and a deprecation
> > > notice
> > > > > is added to the alert response in the Golang handler
> > > > >
> > > > > I think we should have consistency in our approach and wanted to
> > > get
> > > > > people’s thoughts.
> > > > >
> > > > > I would vote that we do not rewrite a deprecated route from Perl to
> > > > Golang.
> > > > >
> > > > > Thanks,
> > > > >
> > > > > Michael
> > > > >
> > > >
> > >
> > >
> > >
Re: [EXTERNAL] Re: Traffic Ops Route Deprecation Strategy
Posted by Rawlin Peters <ra...@apache.org>.
The real crux of the issue here is whether or not we agree as a
project that we can move forward with this plan that I believe to be a
good compromise that meets most concerns but "breaks the Semantic
Versioning promise":
1) don't rewrite deprecated endpoints from Perl to Go
2) in the Perl handler, add a deprecation notice to the "alerts"
section of the response
3) once a deprecated API has been deprecated for one major release,
delete it from the API in the subsequent major release (e.g. if
deprecation notices are added and released in ATC 4.x, we can remove
the deprecated endpoint in ATC 5.x)
Rob, from your latest proposal to remove the ATS config API endpoints
from 1.x, I thought we were past this issue already and could finally
agree to remove deprecated endpoints from 1.x (following steps 1-3
above) without requiring an all-new 2.x API. Are we not actually past
that? You came up with valid reasons for applying these steps to the
ATS config API endpoints, and I think it's fair to say that we can
come up with valid reasons for applying these steps to other
deprecated API endpoints as well.
It seems like the real problem is actually agreeing on which
_specific_ endpoints we should be able to follow steps 1-3 for. IMO I
think it should apply to any 1.x endpoints that we have valid reasons
for not carrying forward to Go, including but not limited to:
- the endpoints are dangerous and pose a risk to the overall Traffic
Control system
- the endpoints have been obsoleted by other endpoints
- the endpoints don't seem to serve a true purpose or don't seem to be
valuable to the project anymore
- the task of rewriting the endpoints from Perl to Go represents an
unreasonable amount of work for the project
Obviously most of these reasons are subjective, so we'd have to come
to some level of consensus on deprecating particular endpoints first,
but we should be able to follow steps 1-3 for whatever endpoints we
decide.
Making breaking changes shouldn't be taken lightly, but we need a path
forward that allows some calculated, breaking changes where necessary.
Otherwise, we are unnecessarily holding the project back.
- Rawlin
On Tue, Nov 12, 2019 at 11:44 AM Jeremy Mitchell <mi...@gmail.com> wrote:
>
> Yeah, technically a "rewrite" of 1.x means shifting the implementation from
> Perl to Go for ALL of 1.x. If you leave a subset of 1.x in Perl, then you
> have to call it a "partial rewrite".
>
> So, assuming we want to do a full rewrite (no more perl for the api
> whatsoever), I think that means every 1.x api endpoint (good or bad) needs
> to be rewritten to Go so we can remove the dependency on perl.
>
> However, I still like marking many as "deprecated" so we train users to use
> "better" endpoints that we envision will exist in 2.x and beyond.
>
> Jeremy
>
> On Tue, Nov 12, 2019 at 11:22 AM Hoppal, Michael <Mi...@comcast.com>
> wrote:
>
> > We do not have a lot of deprecated routes as of now as we focused on
> > routes that are more used first.
> >
> > I know on a lot of the remaining rewrites there has been discussion on the
> > matching Github issues on potential deprecation (I cannot think of the
> > exact count but enough to warrant this email __)
> >
> > It is key to note that I am not arguing for removal of endpoints within
> > the current API version but if there is an endpoint we agree should be
> > deprecated going forward to put a message within the response and on the
> > docs.
> >
> > And thanks for the input sounds like you are +1 on rewriting all routes to
> > Golang as Brennan is.
> >
> > On 11/12/19, 9:06 AM, "Robert O Butts" <ro...@apache.org> wrote:
> >
> > Whether we rewrite a route in Go is an implementation detail. To the
> > interface, to users, it doesn't matter whether a route is rewritten or
> > not.
> >
> > But our API follows Semantic Versioning, in order to not break users.
> > We
> > can't just remove endpoints that some of us don't use, and assume other
> > people don't, maybe people who never speak up on the mailing list.
> > We'll
> > never gain a big userbase if we keep breaking users.
> >
> > Per the _project_ SemVer, once we have API 2.0, we can deprecate API
> > 1.x,
> > and in the next major _project_ release, remove API 1.x. Irrespective
> > of
> > Perl or Go.
> >
> > My big concern is, API 2.0 is a big project. How long has the rewrite
> > to Go
> > taken? Do we really believe designing and implementing a completely
> > new API
> > will be any less time?
> >
> > I don't want killing Perl to have to wait on that.
> >
> > I know it feels like a waste to rewrite routes that you don't use, and
> > probably few people do. But that's the cost of a stable project. How
> > many
> > "deprecated" routes are there? If it comes down to taking the
> > development
> > time to rewrite them so we can kill Perl faster, or leaving Perl
> > around, I
> > vote we just do the work and kill Perl.
> >
> > >If we don't rewrite them, then Perl will last until API 2.0 has been
> > designed, released and then *another full major release cycle*. That's
> > way
> > too long to have two codebases for the same component, IMO, especially
> > since the rewrite is already 50% complete.
> >
> > +1
> >
> >
> > On Tue, Nov 12, 2019 at 8:54 AM ocket 8888 <oc...@gmail.com>
> > wrote:
> >
> > > I vote that by and large we DO rewrite them, with exceptions for
> > routes
> > > that just plain don't work, even in Perl. Those are few, though.
> > >
> > > If we don't rewrite them, then Perl will last until API 2.0 has been
> > > designed, released and then *another full major release cycle*.
> > That's way
> > > too long to have two codebases for the same component, IMO,
> > especially
> > > since the rewrite is already 50% complete.
> > >
> > > On Tue, Nov 12, 2019 at 8:43 AM Hoppal, Michael <
> > > Michael_Hoppal@comcast.com>
> > > wrote:
> > >
> > > > As the Traffic Ops API is being rewritten from Perl to Golang
> > there has
> > > > been several routes that have been deprecated and probably more to
> > come.
> > > >
> > > > In the deprecation efforts I have seen two strategies:
> > > >
> > > >
> > > > * The route IS NOT rewritten from Perl to Golang and a
> > deprecation
> > > > notice is added to the alert response in the Perl handler
> > > > * The route IS rewritten from Perl to Golang and a deprecation
> > notice
> > > > is added to the alert response in the Golang handler
> > > >
> > > > I think we should have consistency in our approach and wanted to
> > get
> > > > people’s thoughts.
> > > >
> > > > I would vote that we do not rewrite a deprecated route from Perl to
> > > Golang.
> > > >
> > > > Thanks,
> > > >
> > > > Michael
> > > >
> > >
> >
> >
> >
Re: [EXTERNAL] Re: Traffic Ops Route Deprecation Strategy
Posted by Jeremy Mitchell <mi...@gmail.com>.
Yeah, technically a "rewrite" of 1.x means shifting the implementation from
Perl to Go for ALL of 1.x. If you leave a subset of 1.x in Perl, then you
have to call it a "partial rewrite".
So, assuming we want to do a full rewrite (no more perl for the api
whatsoever), I think that means every 1.x api endpoint (good or bad) needs
to be rewritten to Go so we can remove the dependency on perl.
However, I still like marking many as "deprecated" so we train users to use
"better" endpoints that we envision will exist in 2.x and beyond.
Jeremy
On Tue, Nov 12, 2019 at 11:22 AM Hoppal, Michael <Mi...@comcast.com>
wrote:
> We do not have a lot of deprecated routes as of now as we focused on
> routes that are more used first.
>
> I know on a lot of the remaining rewrites there has been discussion on the
> matching Github issues on potential deprecation (I cannot think of the
> exact count but enough to warrant this email __)
>
> It is key to note that I am not arguing for removal of endpoints within
> the current API version but if there is an endpoint we agree should be
> deprecated going forward to put a message within the response and on the
> docs.
>
> And thanks for the input sounds like you are +1 on rewriting all routes to
> Golang as Brennan is.
>
> On 11/12/19, 9:06 AM, "Robert O Butts" <ro...@apache.org> wrote:
>
> Whether we rewrite a route in Go is an implementation detail. To the
> interface, to users, it doesn't matter whether a route is rewritten or
> not.
>
> But our API follows Semantic Versioning, in order to not break users.
> We
> can't just remove endpoints that some of us don't use, and assume other
> people don't, maybe people who never speak up on the mailing list.
> We'll
> never gain a big userbase if we keep breaking users.
>
> Per the _project_ SemVer, once we have API 2.0, we can deprecate API
> 1.x,
> and in the next major _project_ release, remove API 1.x. Irrespective
> of
> Perl or Go.
>
> My big concern is, API 2.0 is a big project. How long has the rewrite
> to Go
> taken? Do we really believe designing and implementing a completely
> new API
> will be any less time?
>
> I don't want killing Perl to have to wait on that.
>
> I know it feels like a waste to rewrite routes that you don't use, and
> probably few people do. But that's the cost of a stable project. How
> many
> "deprecated" routes are there? If it comes down to taking the
> development
> time to rewrite them so we can kill Perl faster, or leaving Perl
> around, I
> vote we just do the work and kill Perl.
>
> >If we don't rewrite them, then Perl will last until API 2.0 has been
> designed, released and then *another full major release cycle*. That's
> way
> too long to have two codebases for the same component, IMO, especially
> since the rewrite is already 50% complete.
>
> +1
>
>
> On Tue, Nov 12, 2019 at 8:54 AM ocket 8888 <oc...@gmail.com>
> wrote:
>
> > I vote that by and large we DO rewrite them, with exceptions for
> routes
> > that just plain don't work, even in Perl. Those are few, though.
> >
> > If we don't rewrite them, then Perl will last until API 2.0 has been
> > designed, released and then *another full major release cycle*.
> That's way
> > too long to have two codebases for the same component, IMO,
> especially
> > since the rewrite is already 50% complete.
> >
> > On Tue, Nov 12, 2019 at 8:43 AM Hoppal, Michael <
> > Michael_Hoppal@comcast.com>
> > wrote:
> >
> > > As the Traffic Ops API is being rewritten from Perl to Golang
> there has
> > > been several routes that have been deprecated and probably more to
> come.
> > >
> > > In the deprecation efforts I have seen two strategies:
> > >
> > >
> > > * The route IS NOT rewritten from Perl to Golang and a
> deprecation
> > > notice is added to the alert response in the Perl handler
> > > * The route IS rewritten from Perl to Golang and a deprecation
> notice
> > > is added to the alert response in the Golang handler
> > >
> > > I think we should have consistency in our approach and wanted to
> get
> > > people’s thoughts.
> > >
> > > I would vote that we do not rewrite a deprecated route from Perl to
> > Golang.
> > >
> > > Thanks,
> > >
> > > Michael
> > >
> >
>
>
>
Re: [EXTERNAL] Re: Traffic Ops Route Deprecation Strategy
Posted by "Hoppal, Michael" <Mi...@comcast.com>.
We do not have a lot of deprecated routes as of now as we focused on routes that are more used first.
I know on a lot of the remaining rewrites there has been discussion on the matching Github issues on potential deprecation (I cannot think of the exact count but enough to warrant this email __)
It is key to note that I am not arguing for removal of endpoints within the current API version but if there is an endpoint we agree should be deprecated going forward to put a message within the response and on the docs.
And thanks for the input sounds like you are +1 on rewriting all routes to Golang as Brennan is.
On 11/12/19, 9:06 AM, "Robert O Butts" <ro...@apache.org> wrote:
Whether we rewrite a route in Go is an implementation detail. To the
interface, to users, it doesn't matter whether a route is rewritten or not.
But our API follows Semantic Versioning, in order to not break users. We
can't just remove endpoints that some of us don't use, and assume other
people don't, maybe people who never speak up on the mailing list. We'll
never gain a big userbase if we keep breaking users.
Per the _project_ SemVer, once we have API 2.0, we can deprecate API 1.x,
and in the next major _project_ release, remove API 1.x. Irrespective of
Perl or Go.
My big concern is, API 2.0 is a big project. How long has the rewrite to Go
taken? Do we really believe designing and implementing a completely new API
will be any less time?
I don't want killing Perl to have to wait on that.
I know it feels like a waste to rewrite routes that you don't use, and
probably few people do. But that's the cost of a stable project. How many
"deprecated" routes are there? If it comes down to taking the development
time to rewrite them so we can kill Perl faster, or leaving Perl around, I
vote we just do the work and kill Perl.
>If we don't rewrite them, then Perl will last until API 2.0 has been
designed, released and then *another full major release cycle*. That's way
too long to have two codebases for the same component, IMO, especially
since the rewrite is already 50% complete.
+1
On Tue, Nov 12, 2019 at 8:54 AM ocket 8888 <oc...@gmail.com> wrote:
> I vote that by and large we DO rewrite them, with exceptions for routes
> that just plain don't work, even in Perl. Those are few, though.
>
> If we don't rewrite them, then Perl will last until API 2.0 has been
> designed, released and then *another full major release cycle*. That's way
> too long to have two codebases for the same component, IMO, especially
> since the rewrite is already 50% complete.
>
> On Tue, Nov 12, 2019 at 8:43 AM Hoppal, Michael <
> Michael_Hoppal@comcast.com>
> wrote:
>
> > As the Traffic Ops API is being rewritten from Perl to Golang there has
> > been several routes that have been deprecated and probably more to come.
> >
> > In the deprecation efforts I have seen two strategies:
> >
> >
> > * The route IS NOT rewritten from Perl to Golang and a deprecation
> > notice is added to the alert response in the Perl handler
> > * The route IS rewritten from Perl to Golang and a deprecation notice
> > is added to the alert response in the Golang handler
> >
> > I think we should have consistency in our approach and wanted to get
> > people’s thoughts.
> >
> > I would vote that we do not rewrite a deprecated route from Perl to
> Golang.
> >
> > Thanks,
> >
> > Michael
> >
>
Re: Traffic Ops Route Deprecation Strategy
Posted by Robert O Butts <ro...@apache.org>.
Whether we rewrite a route in Go is an implementation detail. To the
interface, to users, it doesn't matter whether a route is rewritten or not.
But our API follows Semantic Versioning, in order to not break users. We
can't just remove endpoints that some of us don't use, and assume other
people don't, maybe people who never speak up on the mailing list. We'll
never gain a big userbase if we keep breaking users.
Per the _project_ SemVer, once we have API 2.0, we can deprecate API 1.x,
and in the next major _project_ release, remove API 1.x. Irrespective of
Perl or Go.
My big concern is, API 2.0 is a big project. How long has the rewrite to Go
taken? Do we really believe designing and implementing a completely new API
will be any less time?
I don't want killing Perl to have to wait on that.
I know it feels like a waste to rewrite routes that you don't use, and
probably few people do. But that's the cost of a stable project. How many
"deprecated" routes are there? If it comes down to taking the development
time to rewrite them so we can kill Perl faster, or leaving Perl around, I
vote we just do the work and kill Perl.
>If we don't rewrite them, then Perl will last until API 2.0 has been
designed, released and then *another full major release cycle*. That's way
too long to have two codebases for the same component, IMO, especially
since the rewrite is already 50% complete.
+1
On Tue, Nov 12, 2019 at 8:54 AM ocket 8888 <oc...@gmail.com> wrote:
> I vote that by and large we DO rewrite them, with exceptions for routes
> that just plain don't work, even in Perl. Those are few, though.
>
> If we don't rewrite them, then Perl will last until API 2.0 has been
> designed, released and then *another full major release cycle*. That's way
> too long to have two codebases for the same component, IMO, especially
> since the rewrite is already 50% complete.
>
> On Tue, Nov 12, 2019 at 8:43 AM Hoppal, Michael <
> Michael_Hoppal@comcast.com>
> wrote:
>
> > As the Traffic Ops API is being rewritten from Perl to Golang there has
> > been several routes that have been deprecated and probably more to come.
> >
> > In the deprecation efforts I have seen two strategies:
> >
> >
> > * The route IS NOT rewritten from Perl to Golang and a deprecation
> > notice is added to the alert response in the Perl handler
> > * The route IS rewritten from Perl to Golang and a deprecation notice
> > is added to the alert response in the Golang handler
> >
> > I think we should have consistency in our approach and wanted to get
> > people’s thoughts.
> >
> > I would vote that we do not rewrite a deprecated route from Perl to
> Golang.
> >
> > Thanks,
> >
> > Michael
> >
>
Re: Traffic Ops Route Deprecation Strategy
Posted by ocket 8888 <oc...@gmail.com>.
I vote that by and large we DO rewrite them, with exceptions for routes
that just plain don't work, even in Perl. Those are few, though.
If we don't rewrite them, then Perl will last until API 2.0 has been
designed, released and then *another full major release cycle*. That's way
too long to have two codebases for the same component, IMO, especially
since the rewrite is already 50% complete.
On Tue, Nov 12, 2019 at 8:43 AM Hoppal, Michael <Mi...@comcast.com>
wrote:
> As the Traffic Ops API is being rewritten from Perl to Golang there has
> been several routes that have been deprecated and probably more to come.
>
> In the deprecation efforts I have seen two strategies:
>
>
> * The route IS NOT rewritten from Perl to Golang and a deprecation
> notice is added to the alert response in the Perl handler
> * The route IS rewritten from Perl to Golang and a deprecation notice
> is added to the alert response in the Golang handler
>
> I think we should have consistency in our approach and wanted to get
> people’s thoughts.
>
> I would vote that we do not rewrite a deprecated route from Perl to Golang.
>
> Thanks,
>
> Michael
>