Re: [EXTERNAL] Remap.config - inactive delivery services, missing https port ?

[Dave Neuman <ne...@apache.org>]

The inactive DS in Remaps has been an open issue since before we were open
source! :)
Here is the current issue in github:
https://github.com/apache/trafficcontrol/issues/905
Derek makes a good point, I believe it was meant as a status for the
CrConfig and not ORT and it could potentially cause issues if the order of
operations is not followed when adding new delivery services.  You are
right though, this does pose a *potential* risk with ANY_MAP services.
My gut instinct is that we will be making some changes in this area soon
(Voluspa/Self-Service/etc), so maybe we hold off on changing that behavior
for now until we make a formal decision on the direction we are heading in.

As for #2, I do believe that it is a bug that should be fixed.

On Tue, Oct 30, 2018 at 5:00 PM Gelinas, Derek <De...@comcast.com>
wrote:

> One concern I have about the first problem is that, when a delivery
> service is created, setting the service to active will activate it on the
> routers without the remap rules being in place on the caches.  Inactive
> really means “not on traffic router.”  That’s not functionality we
> necessarily want to lose until such a time as we can get configs out to
> caches more or less instantly.
>
>
>
> Derek
>
>
>
> *From: *Robert Butts <ro...@apache.org>
> *Reply-To: *"users@trafficcontrol.apache.org" <
> users@trafficcontrol.apache.org>
> *Date: *Tuesday, October 30, 2018 at 6:54 PM
> *To: *"users@trafficcontrol.apache.org" <us...@trafficcontrol.apache.org>
> *Subject: *[EXTERNAL] Remap.config - inactive delivery services, missing
> https port ?
>
>
>
> So, looking to rewrite the Perl remap.config in Go, I noticed a couple
> things, which I'm not 100% sure are bugs and not intended behavior.
>
>
>
> 1.
>
>
>
> Inactive delivery services are still being added to the remap rules. This
> is a result of the Perl ORM query neither selecting nor filtering the
> active column, and the code likewise not filtering. The ORM Schema which
> ends up being used for the remap rules is here, which you can see neither
> selects nor filters the `active` column:
>
>
> https://github.com/apache/trafficcontrol/blob/55d6eff3eab42bdf7f51ca12ec5cd92b9e7d3498/traffic_ops/app/lib/Schema/Result/DeliveryServiceInfoForServerList.pm#L41
>
>
>
> This seems particularly dangerous, especially if someone were to set an
> ANY_MAP DS to inactive thinking that disabled its usage, they could
> inadvertently proxy the internet.
>
>
>
> Does anyone know a valid reason for inactive delivery services to be
> inserted into the remaps? Does anyone have inactive delivery services they
> need to be inserted into remaps? If not, I'm going to consider it a bug,
> and fix it.
>
>
>
> 2.
>
>
>
> Nonstandard HTTP ports in the Server table's `tcp_port` column are being
> added to the "from" in the remap, but nonstandard HTTPS ports in the
> `https_port` are not. For example:
>
>
>
> `my $map_from = "http://" . $hname . $re . $ds_domain . $portstr . "/";`
>
> `$map_from = "https://" . $hname . $re . $ds_domain . "/";`
>
>
>
>
> https://github.com/apache/trafficcontrol/blob/c7c95bf51fec8f2c8ac74089d29f73a9c6bf8b2b/traffic_ops/app/lib/API/Configs/ApacheTrafficServer.pm#L966
>
>
> https://github.com/apache/trafficcontrol/blob/c7c95bf51fec8f2c8ac74089d29f73a9c6bf8b2b/traffic_ops/app/lib/API/Configs/ApacheTrafficServer.pm#L971
>
>
>
> Does anyone know if this is intentional? Is anyone using nonstandard
> `server.https_port` values on cache servers, which they need to _not_ be
> inserted in HTTPS remaps? If not, I'm going to consider it a bug, and fix
> it.
>
>
>
> Thanks,
>
>
>

Re: [EXTERNAL] Remap.config - inactive delivery services, missing https port ?

[Robert Butts <ro...@apache.org>]

Ok, if there's a good reason, I'll leave it. Just wasn't sure.

I'm not sure I fully understand though.

>when a delivery service is created, setting the service to active will
activate it on the routers without the remap rules being in place on the
caches.  Inactive really means “not on traffic router.”  That’s not
functionality we necessarily want to lose until such a time as we can get
configs out to caches more or less instantly.

Isn't that the case for any change? Don't you always have to
queue-then-snap, to add things, so they go to the cache first? I'm not
seeing what advantage it is to leave the DS inactive, Snap whenever, then
when you want to start routing, assuming your DS has since been Queued in
the state you want it, Snap; versus having an inactive DS, getting it to
the state you want, queuing, then snapping.

I guess it's a little faster? If, for some reason, you had a DS ready to
go, and suddenly needed to enable it as quickly as possible, you only have
to Snap?

Not sure I'm convinced it's worth the danger of having "disabled" remap
rules that can be abused. But I can see the "feature". Not sure I see that
big a use case, though.

>this does pose a *potential* risk with ANY_MAP services

Well, I don't think it's just ANY_MAP. For any DS, it means someone clever
could use it, when it hasn't been intentionally enabled. Seems like a
problem for Self-Service as well. What if a tenant on a multi-tenant CDN,
who's paying the CDN operator by-the-byte, sets up a DS and leaves it
inactive? Or sets a previously active DS to inactive? Someone malicious
could still send traffic thru the edge, traffic which the tenant didn't
want, and now either has to pay for, or the CDN operator has to pay for.
Either way it seems like a billing nightmare.

I think @nir has some good points on that Github issue:

> better define the "active" field, allowing it to have 3 modes (no
deployment, caches-only, and full-deployment).
> such fields should be in the hands of the CDN owner and not the DS owner

That seems reasonable to me. I wouldn't put it on the CDN owner though, why
not put it on the DS, and let the tenant decide? I'd propose a 3-phase
slider UI component, "inactive >> caches >> routed", probably automatically
going all the way to "routed" when clicked, so you have to manually move it
back to "caches" if you ever want that.

But, I agree with waiting, with everything else we have in motion right
now. But this is something we need to figure out, especially for Self
Service.

> we should make it clear that 'inactive' really does mean "not on Traffic
Router"; I'm not sure our documentation really reflects that right now.

+1


On Thu, Nov 1, 2018 at 9:04 AM Rawlin Peters <ra...@gmail.com>
wrote:

> At one point in time, active ANY_MAP DSes were being added to the
> CRConfig. That was a bug I fixed a while ago in the Perl, and I think
> in our CDN our ANY_MAP DSes were all set to 'inactive' at the time.
> That still appears to be the case today too. So yeah, we should make
> it clear that 'inactive' really does mean "not on Traffic Router"; I'm
> not sure our documentation really reflects that right now.
>
> - Rawlin
> On Thu, Nov 1, 2018 at 8:45 AM Dave Neuman <ne...@apache.org> wrote:
> >
> > The inactive DS in Remaps has been an open issue since before we were
> open source! :)
> > Here is the current issue in github:
> https://github.com/apache/trafficcontrol/issues/905
> > Derek makes a good point, I believe it was meant as a status for the
> CrConfig and not ORT and it could potentially cause issues if the order of
> operations is not followed when adding new delivery services.  You are
> right though, this does pose a potential risk with ANY_MAP services.
> > My gut instinct is that we will be making some changes in this area soon
> (Voluspa/Self-Service/etc), so maybe we hold off on changing that behavior
> for now until we make a formal decision on the direction we are heading in.
> >
> > As for #2, I do believe that it is a bug that should be fixed.
> >
> > On Tue, Oct 30, 2018 at 5:00 PM Gelinas, Derek <
> Derek_Gelinas@comcast.com> wrote:
> >>
> >> One concern I have about the first problem is that, when a delivery
> service is created, setting the service to active will activate it on the
> routers without the remap rules being in place on the caches.  Inactive
> really means “not on traffic router.”  That’s not functionality we
> necessarily want to lose until such a time as we can get configs out to
> caches more or less instantly.
> >>
> >>
> >>
> >> Derek
> >>
> >>
> >>
> >> From: Robert Butts <ro...@apache.org>
> >> Reply-To: "users@trafficcontrol.apache.org" <
> users@trafficcontrol.apache.org>
> >> Date: Tuesday, October 30, 2018 at 6:54 PM
> >> To: "users@trafficcontrol.apache.org" <us...@trafficcontrol.apache.org>
> >> Subject: [EXTERNAL] Remap.config - inactive delivery services, missing
> https port ?
> >>
> >>
> >>
> >> So, looking to rewrite the Perl remap.config in Go, I noticed a couple
> things, which I'm not 100% sure are bugs and not intended behavior.
> >>
> >>
> >>
> >> 1.
> >>
> >>
> >>
> >> Inactive delivery services are still being added to the remap rules.
> This is a result of the Perl ORM query neither selecting nor filtering the
> active column, and the code likewise not filtering. The ORM Schema which
> ends up being used for the remap rules is here, which you can see neither
> selects nor filters the `active` column:
> >>
> >>
> https://github.com/apache/trafficcontrol/blob/55d6eff3eab42bdf7f51ca12ec5cd92b9e7d3498/traffic_ops/app/lib/Schema/Result/DeliveryServiceInfoForServerList.pm#L41
> >>
> >>
> >>
> >> This seems particularly dangerous, especially if someone were to set an
> ANY_MAP DS to inactive thinking that disabled its usage, they could
> inadvertently proxy the internet.
> >>
> >>
> >>
> >> Does anyone know a valid reason for inactive delivery services to be
> inserted into the remaps? Does anyone have inactive delivery services they
> need to be inserted into remaps? If not, I'm going to consider it a bug,
> and fix it.
> >>
> >>
> >>
> >> 2.
> >>
> >>
> >>
> >> Nonstandard HTTP ports in the Server table's `tcp_port` column are
> being added to the "from" in the remap, but nonstandard HTTPS ports in the
> `https_port` are not. For example:
> >>
> >>
> >>
> >> `my $map_from = "http://" . $hname . $re . $ds_domain . $portstr .
> "/";`
> >>
> >> `$map_from = "https://" . $hname . $re . $ds_domain . "/";`
> >>
> >>
> >>
> >>
> https://github.com/apache/trafficcontrol/blob/c7c95bf51fec8f2c8ac74089d29f73a9c6bf8b2b/traffic_ops/app/lib/API/Configs/ApacheTrafficServer.pm#L966
> >>
> >>
> https://github.com/apache/trafficcontrol/blob/c7c95bf51fec8f2c8ac74089d29f73a9c6bf8b2b/traffic_ops/app/lib/API/Configs/ApacheTrafficServer.pm#L971
> >>
> >>
> >>
> >> Does anyone know if this is intentional? Is anyone using nonstandard
> `server.https_port` values on cache servers, which they need to _not_ be
> inserted in HTTPS remaps? If not, I'm going to consider it a bug, and fix
> it.
> >>
> >>
> >>
> >> Thanks,
> >>
> >>
>

Re: [EXTERNAL] Remap.config - inactive delivery services, missing https port ?

[Rawlin Peters <ra...@gmail.com>]

At one point in time, active ANY_MAP DSes were being added to the
CRConfig. That was a bug I fixed a while ago in the Perl, and I think
in our CDN our ANY_MAP DSes were all set to 'inactive' at the time.
That still appears to be the case today too. So yeah, we should make
it clear that 'inactive' really does mean "not on Traffic Router"; I'm
not sure our documentation really reflects that right now.

- Rawlin
On Thu, Nov 1, 2018 at 8:45 AM Dave Neuman <ne...@apache.org> wrote:
>
> The inactive DS in Remaps has been an open issue since before we were open source! :)
> Here is the current issue in github:  https://github.com/apache/trafficcontrol/issues/905
> Derek makes a good point, I believe it was meant as a status for the CrConfig and not ORT and it could potentially cause issues if the order of operations is not followed when adding new delivery services.  You are right though, this does pose a potential risk with ANY_MAP services.
> My gut instinct is that we will be making some changes in this area soon (Voluspa/Self-Service/etc), so maybe we hold off on changing that behavior for now until we make a formal decision on the direction we are heading in.
>
> As for #2, I do believe that it is a bug that should be fixed.
>
> On Tue, Oct 30, 2018 at 5:00 PM Gelinas, Derek <De...@comcast.com> wrote:
>>
>> One concern I have about the first problem is that, when a delivery service is created, setting the service to active will activate it on the routers without the remap rules being in place on the caches.  Inactive really means “not on traffic router.”  That’s not functionality we necessarily want to lose until such a time as we can get configs out to caches more or less instantly.
>>
>>
>>
>> Derek
>>
>>
>>
>> From: Robert Butts <ro...@apache.org>
>> Reply-To: "users@trafficcontrol.apache.org" <us...@trafficcontrol.apache.org>
>> Date: Tuesday, October 30, 2018 at 6:54 PM
>> To: "users@trafficcontrol.apache.org" <us...@trafficcontrol.apache.org>
>> Subject: [EXTERNAL] Remap.config - inactive delivery services, missing https port ?
>>
>>
>>
>> So, looking to rewrite the Perl remap.config in Go, I noticed a couple things, which I'm not 100% sure are bugs and not intended behavior.
>>
>>
>>
>> 1.
>>
>>
>>
>> Inactive delivery services are still being added to the remap rules. This is a result of the Perl ORM query neither selecting nor filtering the active column, and the code likewise not filtering. The ORM Schema which ends up being used for the remap rules is here, which you can see neither selects nor filters the `active` column:
>>
>> https://github.com/apache/trafficcontrol/blob/55d6eff3eab42bdf7f51ca12ec5cd92b9e7d3498/traffic_ops/app/lib/Schema/Result/DeliveryServiceInfoForServerList.pm#L41
>>
>>
>>
>> This seems particularly dangerous, especially if someone were to set an ANY_MAP DS to inactive thinking that disabled its usage, they could inadvertently proxy the internet.
>>
>>
>>
>> Does anyone know a valid reason for inactive delivery services to be inserted into the remaps? Does anyone have inactive delivery services they need to be inserted into remaps? If not, I'm going to consider it a bug, and fix it.
>>
>>
>>
>> 2.
>>
>>
>>
>> Nonstandard HTTP ports in the Server table's `tcp_port` column are being added to the "from" in the remap, but nonstandard HTTPS ports in the `https_port` are not. For example:
>>
>>
>>
>> `my $map_from = "http://" . $hname . $re . $ds_domain . $portstr . "/";`
>>
>> `$map_from = "https://" . $hname . $re . $ds_domain . "/";`
>>
>>
>>
>> https://github.com/apache/trafficcontrol/blob/c7c95bf51fec8f2c8ac74089d29f73a9c6bf8b2b/traffic_ops/app/lib/API/Configs/ApacheTrafficServer.pm#L966
>>
>> https://github.com/apache/trafficcontrol/blob/c7c95bf51fec8f2c8ac74089d29f73a9c6bf8b2b/traffic_ops/app/lib/API/Configs/ApacheTrafficServer.pm#L971
>>
>>
>>
>> Does anyone know if this is intentional? Is anyone using nonstandard `server.https_port` values on cache servers, which they need to _not_ be inserted in HTTPS remaps? If not, I'm going to consider it a bug, and fix it.
>>
>>
>>
>> Thanks,
>>
>>