You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@trafficcontrol.apache.org by Dan Kirkwood <da...@apache.org> on 2019/05/06 22:54:25 UTC
new feature: Consistent Hash with Query Parameters
Hi all.. I've just submitted a PR as a draft -
https://github.com/apache/trafficcontrol/pull/3552 that adds a new field to
a deliveryservice. `consistentHashQueryParams` lists the query parameter
keys that may be used on a deliveryservice to be included in addition to
the url path when the consistent hash value is calculated. This would
allow, for example, the same path to be used for an asset in a different
format, but not require the same cache to be used for each.
I submitted as a draft for now to allow testing on it. Right now, the db,
traffic ops, and traffic router changes are included. Traffic portal
changes are to follow.
Comment welcome..
Dan
Re: [EXTERNAL] new feature: Consistent Hash with Query Parameters
Posted by Rawlin Peters <ra...@gmail.com>.
> trred is very similar to the X-TC-Steering-Option Header - which could be collapsed to the same thing entirely if and when CLIENT_STEERING eats STEERING.
We're getting a little off-topic here, but `trred` is for allowing the
client to choose whether or not the response should be a 302 or a 200,
and the "x-tc-steering-option" header is for allowing the client to
specifically request one of the targets of a steering delivery
service. We should discuss how to header-ize the "trred" and "format"
query parameters in a separate thread IMO (after that work gets
prioritized).
- Rawlin
On Tue, May 7, 2019 at 12:09 PM Fieck, Brennan
<Br...@comcast.com> wrote:
>
> Both fakeClientIPAddress and trred already have HTTP Header equivalents, fwiw.
> More specifically, the fakeClientIPAddress query parameter is exactly the same as the X-MM-Client-IP Header,
> and trred is very similar to the X-TC-Steering-Option Header - which could be collapsed to the same thing entirely if and when CLIENT_STEERING eats STEERING.
>
> The most problematic one is also the one most likely to be used by an origin, unfortunately.
>
> > ... that needs to be investigated.
> Yeah, I was hoping to get an answer to that with my question. Maybe I should ask in the "users" list instead/as well? Probably a separate topic either way.
>
> > deprecation of these query parameters should be a distinct development effort
> You're right. But this seemed like a good time to bring it up, because this consistent hashing behavior showcases the issue of a collision between query parameters used by origins and those used by Traffic Router very nicely.
>
> Also, real quick:
> > That also means we will need to account for them in this
> > feature and ignore them when determining the string to use with
> > consistent hashing and cache selection
> Happy to report that by fiat of a database constraint placed on the allowed values, consistent hashing already in the draft PR will ignore reserved query parameters.
>
> ________________________________________
> From: Rawlin Peters <ra...@gmail.com>
> Sent: Tuesday, May 7, 2019 11:12 AM
> To: dev@trafficcontrol.apache.org
> Subject: Re: [EXTERNAL] new feature: Consistent Hash with Query Parameters
>
> +1 on the feature and general approach of the draft PR. I also share
> Jeff's concerns about the deprecation of query parameters reserved by
> Traffic Router. It's one thing to deprecate things that we mostly
> control in Traffic Control (e.g. a field in the Traffic Ops API), but
> a much harder task to deprecate things that would affect CDN clients.
> In a lot of cases, we can't really control CDN clients, so we need to
> be much more careful about those kinds of deprecations in order to
> keep them from impacting service. If this is a behavior that clients
> depend on, we can't really deprecate it until we have something for
> them to transition to instead (e.g. a headers-based approach). Both
> approaches would have to be supported for some period of time before
> we officially remove the reserved query parameters from Traffic
> Router.
>
> - Rawlin
>
> On Tue, May 7, 2019 at 9:17 AM Jeff Elsloo <el...@apache.org> wrote:
> >
> > +1 on this feature, but I'm -1 on tying the fate of this feature to
> > deprecation of the use of query parameters in Traffic Router.
> >
> > While I really want to see deprecation of the use of query parameters
> > in Traffic Router, we have some use cases that currently require them,
> > in particular `ttred`. I don't know how prevalent the other two are
> > (format, fakeClientIpAddress) in practice, but that needs to be
> > investigated.
> >
> > Because of this, we need to do some work to determine which query
> > parameters can be eliminated and what we need to do to continue
> > supporting our existing use cases. If we must continue to support
> > them, we might be able to find a slightly better approach that
> > constrains their use to specific delivery services. For that reason,
> > deprecation of these query parameters should be a distinct development
> > effort. That also means we will need to account for them in this
> > feature and ignore them when determining the string to use with
> > consistent hashing and cache selection.
> > --
> > Thanks,
> > Jeff
> >
> > On Tue, May 7, 2019 at 9:01 AM Fieck, Brennan <Br...@comcast.com> wrote:
> > >
> > > Personally I'd like to see this accompanied by an official deprecation for the "reserved" Query Parameters used by Traffic Router
> > >
> > > Is there someone out there using query parameters like e.g. `format` and `fakeClientIPAddress` who would for some reason be unable to use HTTP Headers instead?
> > > ________________________________________
> > > From: Dan Kirkwood <da...@apache.org>
> > > Sent: Monday, May 6, 2019 4:54 PM
> > > To: dev@trafficcontrol.apache.org
> > > Subject: [EXTERNAL] new feature: Consistent Hash with Query Parameters
> > >
> > > Hi all.. I've just submitted a PR as a draft -
> > > https://github.com/apache/trafficcontrol/pull/3552 that adds a new field to
> > > a deliveryservice. `consistentHashQueryParams` lists the query parameter
> > > keys that may be used on a deliveryservice to be included in addition to
> > > the url path when the consistent hash value is calculated. This would
> > > allow, for example, the same path to be used for an asset in a different
> > > format, but not require the same cache to be used for each.
> > >
> > > I submitted as a draft for now to allow testing on it. Right now, the db,
> > > traffic ops, and traffic router changes are included. Traffic portal
> > > changes are to follow.
> > >
> > > Comment welcome..
> > >
> > > Dan
Re: [EXTERNAL] new feature: Consistent Hash with Query Parameters
Posted by "Fieck, Brennan" <Br...@comcast.com>.
Both fakeClientIPAddress and trred already have HTTP Header equivalents, fwiw.
More specifically, the fakeClientIPAddress query parameter is exactly the same as the X-MM-Client-IP Header,
and trred is very similar to the X-TC-Steering-Option Header - which could be collapsed to the same thing entirely if and when CLIENT_STEERING eats STEERING.
The most problematic one is also the one most likely to be used by an origin, unfortunately.
> ... that needs to be investigated.
Yeah, I was hoping to get an answer to that with my question. Maybe I should ask in the "users" list instead/as well? Probably a separate topic either way.
> deprecation of these query parameters should be a distinct development effort
You're right. But this seemed like a good time to bring it up, because this consistent hashing behavior showcases the issue of a collision between query parameters used by origins and those used by Traffic Router very nicely.
Also, real quick:
> That also means we will need to account for them in this
> feature and ignore them when determining the string to use with
> consistent hashing and cache selection
Happy to report that by fiat of a database constraint placed on the allowed values, consistent hashing already in the draft PR will ignore reserved query parameters.
________________________________________
From: Rawlin Peters <ra...@gmail.com>
Sent: Tuesday, May 7, 2019 11:12 AM
To: dev@trafficcontrol.apache.org
Subject: Re: [EXTERNAL] new feature: Consistent Hash with Query Parameters
+1 on the feature and general approach of the draft PR. I also share
Jeff's concerns about the deprecation of query parameters reserved by
Traffic Router. It's one thing to deprecate things that we mostly
control in Traffic Control (e.g. a field in the Traffic Ops API), but
a much harder task to deprecate things that would affect CDN clients.
In a lot of cases, we can't really control CDN clients, so we need to
be much more careful about those kinds of deprecations in order to
keep them from impacting service. If this is a behavior that clients
depend on, we can't really deprecate it until we have something for
them to transition to instead (e.g. a headers-based approach). Both
approaches would have to be supported for some period of time before
we officially remove the reserved query parameters from Traffic
Router.
- Rawlin
On Tue, May 7, 2019 at 9:17 AM Jeff Elsloo <el...@apache.org> wrote:
>
> +1 on this feature, but I'm -1 on tying the fate of this feature to
> deprecation of the use of query parameters in Traffic Router.
>
> While I really want to see deprecation of the use of query parameters
> in Traffic Router, we have some use cases that currently require them,
> in particular `ttred`. I don't know how prevalent the other two are
> (format, fakeClientIpAddress) in practice, but that needs to be
> investigated.
>
> Because of this, we need to do some work to determine which query
> parameters can be eliminated and what we need to do to continue
> supporting our existing use cases. If we must continue to support
> them, we might be able to find a slightly better approach that
> constrains their use to specific delivery services. For that reason,
> deprecation of these query parameters should be a distinct development
> effort. That also means we will need to account for them in this
> feature and ignore them when determining the string to use with
> consistent hashing and cache selection.
> --
> Thanks,
> Jeff
>
> On Tue, May 7, 2019 at 9:01 AM Fieck, Brennan <Br...@comcast.com> wrote:
> >
> > Personally I'd like to see this accompanied by an official deprecation for the "reserved" Query Parameters used by Traffic Router
> >
> > Is there someone out there using query parameters like e.g. `format` and `fakeClientIPAddress` who would for some reason be unable to use HTTP Headers instead?
> > ________________________________________
> > From: Dan Kirkwood <da...@apache.org>
> > Sent: Monday, May 6, 2019 4:54 PM
> > To: dev@trafficcontrol.apache.org
> > Subject: [EXTERNAL] new feature: Consistent Hash with Query Parameters
> >
> > Hi all.. I've just submitted a PR as a draft -
> > https://github.com/apache/trafficcontrol/pull/3552 that adds a new field to
> > a deliveryservice. `consistentHashQueryParams` lists the query parameter
> > keys that may be used on a deliveryservice to be included in addition to
> > the url path when the consistent hash value is calculated. This would
> > allow, for example, the same path to be used for an asset in a different
> > format, but not require the same cache to be used for each.
> >
> > I submitted as a draft for now to allow testing on it. Right now, the db,
> > traffic ops, and traffic router changes are included. Traffic portal
> > changes are to follow.
> >
> > Comment welcome..
> >
> > Dan
Re: [EXTERNAL] new feature: Consistent Hash with Query Parameters
Posted by Rawlin Peters <ra...@gmail.com>.
+1 on the feature and general approach of the draft PR. I also share
Jeff's concerns about the deprecation of query parameters reserved by
Traffic Router. It's one thing to deprecate things that we mostly
control in Traffic Control (e.g. a field in the Traffic Ops API), but
a much harder task to deprecate things that would affect CDN clients.
In a lot of cases, we can't really control CDN clients, so we need to
be much more careful about those kinds of deprecations in order to
keep them from impacting service. If this is a behavior that clients
depend on, we can't really deprecate it until we have something for
them to transition to instead (e.g. a headers-based approach). Both
approaches would have to be supported for some period of time before
we officially remove the reserved query parameters from Traffic
Router.
- Rawlin
On Tue, May 7, 2019 at 9:17 AM Jeff Elsloo <el...@apache.org> wrote:
>
> +1 on this feature, but I'm -1 on tying the fate of this feature to
> deprecation of the use of query parameters in Traffic Router.
>
> While I really want to see deprecation of the use of query parameters
> in Traffic Router, we have some use cases that currently require them,
> in particular `ttred`. I don't know how prevalent the other two are
> (format, fakeClientIpAddress) in practice, but that needs to be
> investigated.
>
> Because of this, we need to do some work to determine which query
> parameters can be eliminated and what we need to do to continue
> supporting our existing use cases. If we must continue to support
> them, we might be able to find a slightly better approach that
> constrains their use to specific delivery services. For that reason,
> deprecation of these query parameters should be a distinct development
> effort. That also means we will need to account for them in this
> feature and ignore them when determining the string to use with
> consistent hashing and cache selection.
> --
> Thanks,
> Jeff
>
> On Tue, May 7, 2019 at 9:01 AM Fieck, Brennan <Br...@comcast.com> wrote:
> >
> > Personally I'd like to see this accompanied by an official deprecation for the "reserved" Query Parameters used by Traffic Router
> >
> > Is there someone out there using query parameters like e.g. `format` and `fakeClientIPAddress` who would for some reason be unable to use HTTP Headers instead?
> > ________________________________________
> > From: Dan Kirkwood <da...@apache.org>
> > Sent: Monday, May 6, 2019 4:54 PM
> > To: dev@trafficcontrol.apache.org
> > Subject: [EXTERNAL] new feature: Consistent Hash with Query Parameters
> >
> > Hi all.. I've just submitted a PR as a draft -
> > https://github.com/apache/trafficcontrol/pull/3552 that adds a new field to
> > a deliveryservice. `consistentHashQueryParams` lists the query parameter
> > keys that may be used on a deliveryservice to be included in addition to
> > the url path when the consistent hash value is calculated. This would
> > allow, for example, the same path to be used for an asset in a different
> > format, but not require the same cache to be used for each.
> >
> > I submitted as a draft for now to allow testing on it. Right now, the db,
> > traffic ops, and traffic router changes are included. Traffic portal
> > changes are to follow.
> >
> > Comment welcome..
> >
> > Dan
Re: [EXTERNAL] new feature: Consistent Hash with Query Parameters
Posted by Jeff Elsloo <el...@apache.org>.
+1 on this feature, but I'm -1 on tying the fate of this feature to
deprecation of the use of query parameters in Traffic Router.
While I really want to see deprecation of the use of query parameters
in Traffic Router, we have some use cases that currently require them,
in particular `ttred`. I don't know how prevalent the other two are
(format, fakeClientIpAddress) in practice, but that needs to be
investigated.
Because of this, we need to do some work to determine which query
parameters can be eliminated and what we need to do to continue
supporting our existing use cases. If we must continue to support
them, we might be able to find a slightly better approach that
constrains their use to specific delivery services. For that reason,
deprecation of these query parameters should be a distinct development
effort. That also means we will need to account for them in this
feature and ignore them when determining the string to use with
consistent hashing and cache selection.
--
Thanks,
Jeff
On Tue, May 7, 2019 at 9:01 AM Fieck, Brennan <Br...@comcast.com> wrote:
>
> Personally I'd like to see this accompanied by an official deprecation for the "reserved" Query Parameters used by Traffic Router
>
> Is there someone out there using query parameters like e.g. `format` and `fakeClientIPAddress` who would for some reason be unable to use HTTP Headers instead?
> ________________________________________
> From: Dan Kirkwood <da...@apache.org>
> Sent: Monday, May 6, 2019 4:54 PM
> To: dev@trafficcontrol.apache.org
> Subject: [EXTERNAL] new feature: Consistent Hash with Query Parameters
>
> Hi all.. I've just submitted a PR as a draft -
> https://github.com/apache/trafficcontrol/pull/3552 that adds a new field to
> a deliveryservice. `consistentHashQueryParams` lists the query parameter
> keys that may be used on a deliveryservice to be included in addition to
> the url path when the consistent hash value is calculated. This would
> allow, for example, the same path to be used for an asset in a different
> format, but not require the same cache to be used for each.
>
> I submitted as a draft for now to allow testing on it. Right now, the db,
> traffic ops, and traffic router changes are included. Traffic portal
> changes are to follow.
>
> Comment welcome..
>
> Dan
Re: [EXTERNAL] new feature: Consistent Hash with Query Parameters
Posted by "Fieck, Brennan" <Br...@comcast.com>.
Personally I'd like to see this accompanied by an official deprecation for the "reserved" Query Parameters used by Traffic Router
Is there someone out there using query parameters like e.g. `format` and `fakeClientIPAddress` who would for some reason be unable to use HTTP Headers instead?
________________________________________
From: Dan Kirkwood <da...@apache.org>
Sent: Monday, May 6, 2019 4:54 PM
To: dev@trafficcontrol.apache.org
Subject: [EXTERNAL] new feature: Consistent Hash with Query Parameters
Hi all.. I've just submitted a PR as a draft -
https://github.com/apache/trafficcontrol/pull/3552 that adds a new field to
a deliveryservice. `consistentHashQueryParams` lists the query parameter
keys that may be used on a deliveryservice to be included in addition to
the url path when the consistent hash value is calculated. This would
allow, for example, the same path to be used for an asset in a different
format, but not require the same cache to be used for each.
I submitted as a draft for now to allow testing on it. Right now, the db,
traffic ops, and traffic router changes are included. Traffic portal
changes are to follow.
Comment welcome..
Dan