You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@trafficcontrol.apache.org by John Rushford <jj...@gmail.com> on 2019/02/01 05:30:30 UTC
Re: [EXTERNAL] Re: Origins assigned to Multipe Delivery Services
produces indeterminate parent.config
origin virtual hosting relies on the Host header. So if the request from the client to the reverse
proxy has a Host header and maintain_pristine_host_hdr is not enabled, remap
will rewrite the Host header along with the request url. So in this case, remapping to
a CNAME should work.
> On Jan 29, 2019, at 6:02 PM, Steve Malenfant <sm...@gmail.com> wrote:
>
> CNAME might not necessary work if the origin is using virtual hosting. It
> would need to be configured as well. Might work with some origin? But not
> all of them.
>
> On Tue, Jan 29, 2019 at 7:40 PM Fieck, Brennan <Br...@comcast.com>
> wrote:
>
>> The issue is inherent with the structure of ATS configuration files. If I
>> understand correctly - someone correct me if I don't -
>> you can't have more than one rule for a "primary destination" in the
>> parent.config. Or, you can, but only one of them will
>> wind up catching all of your matches. Which means that if you have
>> multiple Delivery Services that use the same origin FQDN
>> (dest_domain) then you get proper behavior out of ATS if and only if the
>> resulting rules would be identical. I'd highly
>> recommend that you not move forward using duplicate origin FQDNs on your
>> Delivery Services, because it probably won't
>> do what you want.
>>
>> I'm not sure what you mean about the cachekeys. The Delivery Service URLs
>> should still be unique, so you shouldn't run
>> into a collision in the cache keys.
>>
>> Origins are an object. They're also a cachegroup type. And a profile type.
>> And a server type. And a URL field on a Delivery Service.
>>
>> The solution that's been used in the past to use the same actual origin
>> machine to service multiple Delivery Services safely
>> is to add a "Canonical Name" DNS record to the ATC internal zone for each
>> Delivery Service. The idea is to have a unique name
>> for each Delivery Service, but they all resolve to the same IP address.
>> ________________________________________
>> From: Steve Malenfant <sm...@gmail.com>
>> Sent: Tuesday, January 29, 2019 4:32 PM
>> To: dev@trafficcontrol.apache.org
>> Subject: Re: [EXTERNAL] Re: Origins assigned to Multipe Delivery Services
>> produces indeterminate parent.config
>>
>> We use and will use the same origin for multiple DS. It’s an inherent part
>> of the design of some part of our delivery.
>>
>> Wouldn’t be less of a problem using cachekeys for different delivery
>> services?
>>
>> I would rather see origin as an object and not just a simple URL. I’ve seen
>> this in the past in other systems.
>>
>> I don’t believe we can create multiple http services endpoint for a single
>> DS like we can for DNS type delivery service. This would solve it for us.
>>
>> Steve
>>
>> On Thu, Jan 24, 2019 at 11:22 AM Gray, Jonathan <Jonathan_Gray@comcast.com
>>>
>> wrote:
>>
>>> Even that has a strong element of confusion when in a self-service
>>> multi-tenancy world. Multiple tenants can use the same origins for
>> things
>>> like akami, aws, internal load balancers and any communication about the
>>> origin already being used will be confusing.
>>>
>>> Jonathan G
>>>
>>>
>>> On 1/23/19, 7:37 PM, "Gelinas, Derek" <De...@comcast.com>
>> wrote:
>>>
>>> Perhaps we could solve both issues and just allow it to happen on DS
>>> config but throw up a message when it is saved that says "hey we're going
>>> to do this but it's really not without an element of risk and ats isn't
>>> designed to work properly with multiple identical origins. We recommend
>> you
>>> use an alternative fqdn blah blah blah."
>>>
>>> Something along those lines, anyway.
>>>
>>> Derek
>>>
>>>> On Jan 23, 2019, at 7:18 PM, Rawlin Peters <
>> rawlin.peters@gmail.com>
>>> wrote:
>>>>
>>>> Simply prohibiting duplicate origin FQDNs has been vetoed now, so
>> we
>>>> can't really move forward with that unless the vetoers change their
>>>> minds and remove their -1s.
>>>>
>>>> Currently that leaves us with checking at DS create/update whether
>> or
>>>> not it would conflict with a shared origin due to different DS
>> types,
>>>> qstring config, etc. I am -1 on that idea since I think it will be
>>>> difficult to maintain and enumerate all the possible conflicting
>>>> cases, will present confusing error messages to the user to which
>> the
>>>> solution would just be "create a CNAME record" anyways, and that
>> the
>>>> ATS parent.config should really be per-remap rather than "global".
>> I
>>>> know there has been some discussion in the ATS community about
>> making
>>>> parent.config per-remap, but I don't have much information there.
>>>>
>>>> That is to say, I don't think there is a great solution to this
>>>> problem right now that we can all agree upon, but maybe our efforts
>>>> would be better spent in the ATS community by making a per-remap
>>>> parent.config. Then the duplicate origin problem in Traffic Control
>>>> would go away.
>>>>
>>>> Another option could be to have some kind of generic rules engine
>>> that
>>>> would allow an organization to create some set of rules that DSes
>>>> would have to abide by. For example, no duplicate origins, an
>> origin
>>>> can't be in a list of known malicious domains, you can't use the DS
>>>> type "HTTP_NO_CACHE", maxDnsAnswers must be < 5, TTLs must be > 30
>>> and
>>>> < 6000, protocol must be HTTPS, intitialDispersion must be < 10,
>> DSCP
>>>> must be 40, etc. It seems like for something truly self-service
>> you'd
>>>> probably need to set up some rules for your users to abide by that
>>>> might not necessarily apply to ALL organizations. Just a thought. I
>>>> don't think the duplicate origins problem alone would warrant
>>>> something like that, but if it also provided a solution for the
>>>> self-service effort, maybe it would be worth it.
>>>>
>>>> - Rawlin
>>>>
>>>> On Wed, Jan 23, 2019 at 4:15 AM Gelinas, Derek
>>>> <De...@comcast.com> wrote:
>>>>>
>>>>> That's the problem. We either need to be smart enough to recognize
>>> that in our config and warn the user or just prevent it and suggest to
>> the
>>> user that they use a different fqdn.
>>>>>
>>>>> My vote is for preventing it at this point.
>>>>>
>>>>>> On Jan 23, 2019, at 12:50 AM, Gray, Jonathan <
>>> Jonathan_Gray@comcast.com> wrote:
>>>>>>
>>>>>> It's not just differing types, it's anything that affects the
>>> parent.config which also includes the, qstring configuration, MSO
>>> parents/config, etc. as well.
>>>>>>
>>>>>> Jonathan G
>>>>>>
>>>>>>
>>>>>> On 1/22/19, 2:33 PM, "Dave Neuman" <ne...@apache.org> wrote:
>>>>>>
>>>>>> -1 on the one DS to Origin limitation. I think there are legit
>>> use cases
>>>>>> for it.
>>>>>> I would be +1 on the idea if we include type though. Two DSs
>>> can share the
>>>>>> same origin as long as they are of the same type (DNS, HTTP,
>>> HTTP_LIVE),
>>>>>> etc. You shouldn't be able to have two DSs with the same
>> origin
>>> but
>>>>>> different types.
>>>>>>
>>>>>> On Mon, Jan 14, 2019 at 7:53 AM Fieck, Brennan <
>>> Brennan_Fieck@comcast.com>
>>>>>> wrote:
>>>>>>
>>>>>>> I don't see this as a complicated use case or a barrier to
>> entry,
>>> an
>>>>>>> extremely basic introduction to ATC would only make use of one
>> DS
>>>>>>> (shameless CDN-in-a-Box plug).
>>>>>>> It seems far more complicated for a user to be using many DSes
>> to
>>> serve a
>>>>>>> single origin - we can easily make it clear in the docs that the
>>> two have a
>>>>>>> 1:1 relationship, with a footnote about CNAME records for users
>>> with the
>>>>>>> knowledge, resources, and need for such workarounds.
>>>>>>> ________________________________________
>>>>>>> From: Mark Torluemke <mt...@apache.org>
>>>>>>> Sent: Friday, January 11, 2019 1:21 PM
>>>>>>> To: dev@trafficcontrol.apache.org
>>>>>>> Subject: Re: [EXTERNAL] Re: Origins assigned to Multipe Delivery
>>> Services
>>>>>>> produces indeterminate parent.config
>>>>>>>
>>>>>>>>> 1. Prohibit creating new delivery services that would share an
>>>>>>> existing origin and prohibit updating a delivery service to a
>>> shared
>>>>>>> origin
>>>>>>> In case my position has been lost, I'm still -1 on this. :) IMO,
>>> we
>>>>>>> shouldn't let the few complicated use cases increase the
>> barrier
>>> to entry
>>>>>>> (to using ATC) for the masses.
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> On Fri, Jan 11, 2019 at 11:53 AM Rawlin Peters <
>>> rawlin.peters@gmail.com>
>>>>>>> wrote:
>>>>>>>
>>>>>>>> Alright, I'm trying to sum up this discussion so far since it
>>> seems
>>>>>>>> like everyone went on vacation and didn't really get a chance
>> to
>>> wrap
>>>>>>>> this one up:
>>>>>>>> - duplicate origins cause undefined behavior
>>>>>>>> - we need a way to migrate to a future that is free of
>> duplicate
>>>>>>>> origins in Traffic Control
>>>>>>>> - we need a visual and easy way to determine if Traffic Ops
>>> currently
>>>>>>>> contains duplicate origins, so that operators are incentivized
>>> to fix
>>>>>>>> them rather than let it slide indefinitely
>>>>>>>> - operators should have a fair amount of time to fix their
>>> duplicate
>>>>>>>> origins
>>>>>>>>
>>>>>>>> I believe this is what we've mostly agreed upon but haven't
>>> clearly voted
>>>>>>>> on:
>>>>>>>>
>>>>>>>> In release N we will:
>>>>>>>> 1. Prohibit creating new delivery services that would share an
>>>>>>>> existing origin and prohibit updating a delivery service to a
>>> shared
>>>>>>>> origin
>>>>>>>> 2. Add some kind of visual indicator that duplicate origins
>> are a
>>>>>>>> problem that need to be fixed before release N+1; otherwise, an
>>>>>>>> upgrade to N+1 will be prohibited.
>>>>>>>>
>>>>>>>> In release N+1 we will:
>>>>>>>> 3. Include a DB migration that adds a uniqueness constraint on
>>> origin
>>>>>>>> FQDN, removing the API-level checks for that.
>>>>>>>> 4. Prevent an upgrade to N+1 if duplicate origins are found
>> (this
>>>>>>>> might occur as a byproduct of step 3).
>>>>>>>>
>>>>>>>> I am +1 on this plan and believe this would hit on all the
>>> summarized
>>>>>>>> points above. Please provide a clear vote on this plan so that
>>> we can
>>>>>>>> dive deeper in the details (i.e. what release 'N' is, the best
>>> visual
>>>>>>>> indicator for step 2, and a friendly way to handle step 4).
>>>>>>>>
>>>>>>>> Thanks,
>>>>>>>> Rawlin
>>>>>>>>
>>>>>>>> On Wed, Dec 19, 2018 at 3:39 PM Jeremy Mitchell <
>>> mitchell852@gmail.com>
>>>>>>>> wrote:
>>>>>>>>>
>>>>>>>>> Not sure TP is the right place for a warning like "clean up
>> this
>>>>>>>>> 'duplicate' origin or your next upgrade will fail". Most users
>>> of our
>>>>>>>>> system will be like "not my problem".
>>>>>>>>>
>>>>>>>>> On Wed, Dec 19, 2018 at 11:58 AM Fieck, Brennan <
>>>>>>>> Brennan_Fieck@comcast.com>
>>>>>>>>> wrote:
>>>>>>>>>
>>>>>>>>>> Probably. It would impact load times by a bit, but the page
>>> for an
>>>>>>>>>> individual object is not our bottleneck.
>>>>>>>>>> ________________________________________
>>>>>>>>>> From: Robert Butts <ro...@apache.org>
>>>>>>>>>> Sent: Wednesday, December 19, 2018 11:50 AM
>>>>>>>>>> To: dev@trafficcontrol.apache.org
>>>>>>>>>> Subject: Re: [EXTERNAL] Re: Origins assigned to Multipe
>>> Delivery
>>>>>>>> Services
>>>>>>>>>> produces indeterminate parent.config
>>>>>>>>>>
>>>>>>>>>>> - Including a warning on startup and an API constraint
>>> preventing
>>>>>>>> adding
>>>>>>>>>> more bad data in the next 3.0.0 Release Candidate
>>>>>>>>>>> - Adding a database constraint immediately into master that
>>> won't
>>>>>>> be
>>>>>>>>>> cherry-picked into 3.0.0 but should be included in 3.1.0
>>>>>>>>>>
>>>>>>>>>> +1
>>>>>>>>>>
>>>>>>>>>> I understand Jonathan's objection, but at some point, we have
>>> to be
>>>>>>>> able to
>>>>>>>>>> move forward. This is a good compromise: deprecate, then
>>> remove. That
>>>>>>>> gives
>>>>>>>>>> people a full major version to fix their data.
>>>>>>>>>>
>>>>>>>>>> I would be ideal if it were more than just a logged warning,
>>> though.
>>>>>>>> Can we
>>>>>>>>>> add a big red banner in Traffic Portal, on the Delivery
>>> Service page
>>>>>>>> for
>>>>>>>>>> any DS with a duplicate origin, telling users to fix it, and
>>> that
>>>>>>> they
>>>>>>>>>> won't be able to upgrade to the next major version until it's
>>> fixed?
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> On Wed, Dec 19, 2018 at 10:57 AM Fieck, Brennan <
>>>>>>>> Brennan_Fieck@comcast.com
>>>>>>>>>>>
>>>>>>>>>> wrote:
>>>>>>>>>>
>>>>>>>>>>> So it seems like nobody has a problem with the "how" -
>>> disallowing
>>>>>>>>>>> duplicate origin FQDNs on Delivery Services - but we never
>>> reached
>>>>>>> a
>>>>>>>>>>> consensus on "when".
>>>>>>>>>>>
>>>>>>>>>>> I stand by my previous proposal:
>>>>>>>>>>> - Including a warning on startup and an API constraint
>>> preventing
>>>>>>>> adding
>>>>>>>>>>> more bad data in the next 3.0.0 Release Candidate
>>>>>>>>>>> - Adding a database constraint immediately into master that
>>> won't
>>>>>>> be
>>>>>>>>>>> cherry-picked into 3.0.0 but should be included in 3.1.0
>>>>>>>>>>> ________________________________________
>>>>>>>>>>> From: Rawlin Peters <ra...@gmail.com>
>>>>>>>>>>> Sent: Tuesday, December 18, 2018 4:59 PM
>>>>>>>>>>> To: dev@trafficcontrol.apache.org
>>>>>>>>>>> Subject: Re: [EXTERNAL] Re: Origins assigned to Multipe
>>> Delivery
>>>>>>>> Services
>>>>>>>>>>> produces indeterminate parent.config
>>>>>>>>>>>
>>>>>>>>>>> Also, building more around DS types will make it even harder
>>> to get
>>>>>>>>>>> away from DS types in the future too, which I know is
>>> something
>>>>>>> we've
>>>>>>>>>>> discussed on this mailing list before. It also adds to the
>>> overhead
>>>>>>>> of
>>>>>>>>>>> Delivery Service Topologies, since a lot of the DS types
>> won't
>>>>>>>>>>> carryover into that world.
>>>>>>>>>>>
>>>>>>>>>>> - Rawlin
>>>>>>>>>>>
>>>>>>>>>>> On Tue, Dec 18, 2018 at 2:42 PM Fieck, Brennan
>>>>>>>>>>> <Br...@comcast.com> wrote:
>>>>>>>>>>>>
>>>>>>>>>>>> +1.
>>>>>>>>>>>> If there's a simple way to work around duplicate origins
>>> being
>>>>>>>>>>> prohibited,
>>>>>>>>>>>> then we should rely on that instead of "enumerating all
>> those
>>>>>>>> possible
>>>>>>>>>>> conflicting
>>>>>>>>>>>> settings, which are not only highly complex and confusing,
>>> but
>>>>>>> also
>>>>>>>>>>> further
>>>>>>>>>>>> entrench us in only supporting ATS as a caching proxy
>>> (hurting
>>>>>>>> efforts
>>>>>>>>>> to
>>>>>>>>>>>> integrate e.g. Grove, nginx etc.)
>>>>>>>>>>>> ________________________________________
>>>>>>>>>>>> From: Rawlin Peters <ra...@gmail.com>
>>>>>>>>>>>> Sent: Tuesday, December 18, 2018 2:20 PM
>>>>>>>>>>>> To: dev@trafficcontrol.apache.org
>>>>>>>>>>>> Subject: Re: [EXTERNAL] Re: Origins assigned to Multipe
>>> Delivery
>>>>>>>>>>> Services produces indeterminate parent.config
>>>>>>>>>>>>
>>>>>>>>>>>> There are a number of different DS settings at play that
>> can
>>>>>>>>>>>> potentially cause conflicts. The question is: do we want to
>>> get
>>>>>>>> into
>>>>>>>>>>>> the business of enumerating all those possible conflicting
>>>>>>>> settings or
>>>>>>>>>>>> just simply prohibit duplicate origins altogether? I think
>>> we can
>>>>>>>> dig
>>>>>>>>>>>> in and get that "sufficiently advanced sql query" to check
>>> for
>>>>>>>>>>>> conflicting origins, but is that something we want to carry
>>> along
>>>>>>>> for
>>>>>>>>>>>> the foreseeable future? Aren't CNAMEs relatively cheaper
>> than
>>>>>>>>>>>> developing and maintaining that code and the mental
>> overhead
>>>>>>>> required
>>>>>>>>>>>> in understanding why you're getting an error that says your
>>>>>>>> requested
>>>>>>>>>>>> DS would cause an origin conflict? I think at the point
>>> you've
>>>>>>>>>>>> requested a DS that would create a conflict, you've chosen
>>> those
>>>>>>>>>>>> settings for a reason and would probably prefer to just
>>>>>>> create/use
>>>>>>>> a
>>>>>>>>>>>> CNAME in your new DS and keep the rest of your settings the
>>> same.
>>>>>>>>>>>>
>>>>>>>>>>>> Thinking in terms of errors, I'm imagining:
>>>>>>>>>>>> "cannot create delivery service: origin fqdn '
>>> foo.example.com'
>>>>>>>> already
>>>>>>>>>>> in use"
>>>>>>>>>>>> vs
>>>>>>>>>>>> "cannot create delivery service: origin fqdn '
>>> foo.example.com'
>>>>>>>> already
>>>>>>>>>>>> in use as type DNS_LIVE_NATNL, which is incompatible with
>>> your
>>>>>>>> chosen
>>>>>>>>>>>> type of HTTP_NO_CACHE"
>>>>>>>>>>>>
>>>>>>>>>>>> At that point you'd probably say to yourself, "uh, I need
>>>>>>>>>>>> HTTP_NO_CACHE, so what am I supposed to do now?"
>>>>>>>>>>>>
>>>>>>>>>>>> As a lazy developer I'm +1 on prohibiting duplicate origin
>>> fqdns
>>>>>>>>>>>> because the resulting code will be simpler, but I think
>>>>>>> eliminating
>>>>>>>>>>>> the mental overhead for operators could be worthwhile too.
>>> If we
>>>>>>>> can
>>>>>>>>>>>> agree on an end state of prohibiting duplicate origins
>>>>>>> altogether,
>>>>>>>> we
>>>>>>>>>>>> can start working on a design to smoothly transition us to
>>> that
>>>>>>>> point.
>>>>>>>>>>>> Are we willing to live with "just CNAME your origin fqdn"
>> as
>>> the
>>>>>>>>>>>> standard solution to duplicates?
>>>>>>>>>>>>
>>>>>>>>>>>> - Rawlin
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> On Tue, Dec 18, 2018 at 1:27 PM Gelinas, Derek
>>>>>>>>>>>> <De...@comcast.com> wrote:
>>>>>>>>>>>>>
>>>>>>>>>>>>> The only situation in which they can share origins is if
>> a)
>>> the
>>>>>>>>>>> origins are shared in an MSO configuration but still have
>>> different
>>>>>>>>>> defined
>>>>>>>>>>> origin fields in the delivery service, or if they're
>> assigned
>>> to
>>>>>>>>>> completely
>>>>>>>>>>> different cachegroups. It's when two delivery services
>> share
>>> the
>>>>>>>> same
>>>>>>>>>>> edges that there's an issue, because you end up with
>>> parent.config
>>>>>>>>>> issues.
>>>>>>>>>>> Actually you could even get away with it in mids as long as
>>> you
>>>>>>>> weren't
>>>>>>>>>>> doing anything like MSO to it.
>>>>>>>>>>>>>
>>>>>>>>>>>>> Could get messy real fast, though. Best to just create a
>>>>>>> second
>>>>>>>>>> FQDN.
>>>>>>>>>>>>>
>>>>>>>>>>>>> Derek
>>>>>>>>>>>>>
>>>>>>>>>>>>> On 12/18/18, 3:23 PM, "Fieck, Brennan" <
>>>>>>>> Brennan_Fieck@comcast.com>
>>>>>>>>>>> wrote:
>>>>>>>>>>>>>
>>>>>>>>>>>>> So no two Delivery Services may share an origin
>>> *regardless
>>>>>>>> of
>>>>>>>>>>> cache hierarchy* ? I've been told that DNS Delivery Services
>>> can
>>>>>>>> have the
>>>>>>>>>>> same origin as HTTP Delivery Services because they obey the
>>> same
>>>>>>>> cache
>>>>>>>>>>> hierarchy. You're saying that would still produce invalid
>>> output
>>>>>>>> and/or
>>>>>>>>>> is
>>>>>>>>>>> explicitly disallowed by ATS?
>>>>>>>>>>>>> ________________________________________
>>>>>>>>>>>>> From: Robert Butts <ro...@apache.org>
>>>>>>>>>>>>> Sent: Tuesday, December 18, 2018 1:09 PM
>>>>>>>>>>>>> To: dev@trafficcontrol.apache.org
>>>>>>>>>>>>> Subject: Re: [EXTERNAL] Re: Origins assigned to Multipe
>>>>>>>> Delivery
>>>>>>>>>>> Services produces indeterminate parent.config
>>>>>>>>>>>>>
>>>>>>>>>>>>>> can you give an example of what parent.config looks like
>>>>>>>> when 2
>>>>>>>>>>> ds's share
>>>>>>>>>>>>> an origin and have different a different topology?
>>>>>>>>>>>>>
>>>>>>>>>>>>> Answering because I encountered this directly, when
>>>>>>> rewriting
>>>>>>>>>>> parent.config.
>>>>>>>>>>>>>
>>>>>>>>>>>>> For example: Suppose you have one Delivery Service:
>>>>>>>>>>>>> XML_ID: foo
>>>>>>>>>>>>> Type: HTPT_LIVE_NATL
>>>>>>>>>>>>> Query String Handling: 1 - ignore in cache key, and pass
>>> up
>>>>>>>>>>>>> Origin Server Base URL: http://foo.example.net
>>>>>>>>>>>>>
>>>>>>>>>>>>> And another Delivery Service:
>>>>>>>>>>>>> XML_ID: bar
>>>>>>>>>>>>> Type: HTPT_LIVE_NATL
>>>>>>>>>>>>> Query String Handling: 1 - ignore in cache key, and pass
>>> up
>>>>>>>>>>>>> Origin Server Base URL: http://foo.example.net
>>>>>>>>>>>>>
>>>>>>>>>>>>> ATS only supports unique `dest_domain` entries in
>>>>>>>> parent.config.
>>>>>>>>>>> Therefore,
>>>>>>>>>>>>> the parent.config generated for a server assigned to
>> both
>>>>>>> of
>>>>>>>>>> these
>>>>>>>>>>> Delivery
>>>>>>>>>>>>> Services with either be:
>>>>>>>>>>>>>
>>>>>>>>>>>>> dest_domain=foo.example.net port=80 go_direct=true
>>>>>>>>>>>>>
>>>>>>>>>>>>> Or
>>>>>>>>>>>>>
>>>>>>>>>>>>> dest_domain=foo.example.net port=80 go_direct=false
>>>>>>>>>>> qstring=consider
>>>>>>>>>>>>>
>>>>>>>>>>>>> Right now, it's arbitrary which Perl Traffic Ops
>> inserts,
>>>>>>> and
>>>>>>>>>> Perl
>>>>>>>>>>> provides
>>>>>>>>>>>>> no warning or error of any kind (the pending Go
>>>>>>>> parent.config PR
>>>>>>>>>>> logs an
>>>>>>>>>>>>> error).
>>>>>>>>>>>>>
>>>>>>>>>>>>> Whichever is arbitrarily inserted, the resulting remaps
>>> for
>>>>>>>> the
>>>>>>>>>>> other
>>>>>>>>>>>>> delivery service will be wrong. Either `foo` requests
>> will
>>>>>>>> drop
>>>>>>>>>>> the query
>>>>>>>>>>>>> string when they shouldn't, and go to the mid when they
>>>>>>>>>> shouldn't;
>>>>>>>>>>> or `bar`
>>>>>>>>>>>>> requests will use the query string and skip the mid when
>>> it
>>>>>>>>>>> shouldn't.
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>> Does that make sense? The only correct solution, is to
>>>>>>>> somehow
>>>>>>>>>>> prevent
>>>>>>>>>>>>> different DSes having the same origin, and tell tenants
>>>>>>> they
>>>>>>>> must
>>>>>>>>>>> use
>>>>>>>>>>>>> CNAMEs if they need.
>>>>>>>>>>>>>
>>>>>>>>>>>>> This isn't a bug in Traffic Control. ATS fundamentally
>>>>>>>> doesn't
>>>>>>>>>>> support
>>>>>>>>>>>>> multiple remap rules with the same parent FQDN with
>>>>>>> different
>>>>>>>>>>>>> configurations. Hence, Traffic Control needs to prohibit
>>>>>>>> that.
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>> On Tue, Dec 18, 2018 at 12:24 PM Jeremy Mitchell <
>>>>>>>>>>> mitchell852@gmail.com>
>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>
>>>>>>>>>>>>>> brennan,
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> can you give an example of what parent.config looks like
>>>>>>>> when 2
>>>>>>>>>>> ds's share
>>>>>>>>>>>>>> an origin and have different a different topology?
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> jeremy
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> On Tue, Dec 18, 2018 at 11:39 AM Fieck, Brennan <
>>>>>>>>>>> Brennan_Fieck@comcast.com
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> To be clear, the "Warning" I'm talking about would
>>>>>>>> happen at
>>>>>>>>>>> startup, but
>>>>>>>>>>>>>>> I'd like a UI-only constraint to come with that to
>>>>>>>> disallow
>>>>>>>>>>> using the API
>>>>>>>>>>>>>>> to bind the same origin to multiple Delivery Services
>>>>>>>> with
>>>>>>>>>>> varying
>>>>>>>>>>>>>>> topography requirements. It wouldn't change the
>>>>>>> existing
>>>>>>>>>> data,
>>>>>>>>>>> but
>>>>>>>>>>>>>> prevent
>>>>>>>>>>>>>>> users from creating more bad data.
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> "warning" doesn't really sufficiently describe that, my
>>>>>>>> bad.
>>>>>>>>>>>>>>> ________________________________________
>>>>>>>>>>>>>>> From: Fieck, Brennan <Br...@comcast.com>
>>>>>>>>>>>>>>> Sent: Tuesday, December 18, 2018 11:24 AM
>>>>>>>>>>>>>>> To: dev@trafficcontrol.apache.org
>>>>>>>>>>>>>>> Subject: Re: [EXTERNAL] Re: Origins assigned to Multipe
>>>>>>>>>>> Delivery Services
>>>>>>>>>>>>>>> produces indeterminate parent.config
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> Well the cost of fixing this bug is a constraint on the
>>>>>>>> data.
>>>>>>>>>>> Unless we
>>>>>>>>>>>>>>> make it a UI-only constraint - which I'm personally
>>>>>>>> against -
>>>>>>>>>>> there must
>>>>>>>>>>>>>> be
>>>>>>>>>>>>>>> some point in the future where ATC cannot reasonably be
>>>>>>>>>>> expected to work
>>>>>>>>>>>>>>> with data that violates that constraint. The question
>>>>>>> is
>>>>>>>> when
>>>>>>>>>>> that should
>>>>>>>>>>>>>>> occur, which should likely happen at a minor version
>>>>>>>> release.
>>>>>>>>>>> Minor not
>>>>>>>>>>>>>>> major because it doesn't involve a change in data
>>>>>>>> structures,
>>>>>>>>>>> merely
>>>>>>>>>>>>>>> relationships between them - in my opinion that's a
>>>>>>> minor
>>>>>>>>>>> version change
>>>>>>>>>>>>>>> but that's definitely up for debate. With several
>>>>>>> release
>>>>>>>>>>> candidates for
>>>>>>>>>>>>>>> 3.0.0 that _doesn't_ include this restriction already
>>>>>>> in
>>>>>>>> the
>>>>>>>>>>> wild, I
>>>>>>>>>>>>>>> wouldn't recommend putting it in there. That means to
>>>>>>>> fix the
>>>>>>>>>>> bug as soon
>>>>>>>>>>>>>>> as possible it should go in 3.1.0 which should be the
>>>>>>>> target
>>>>>>>>>>> of "master"
>>>>>>>>>>>>>>> after the 3.0.0 release is cut from it.
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> So I'd recommend immediately implementing the
>>>>>>> constraint
>>>>>>>> in
>>>>>>>>>>> master with a
>>>>>>>>>>>>>>> refusal to upgrade with bad data, and backport a
>>>>>>> warning
>>>>>>>>>> about
>>>>>>>>>>> the future
>>>>>>>>>>>>>>> behavior into 3.0.0 or as part of a 3.0.1 provided we
>>>>>>> had
>>>>>>>>>> more
>>>>>>>>>>> changes
>>>>>>>>>>>>>> that
>>>>>>>>>>>>>>> would warrant a micro version bump.
>>>>>>>>>>>>>>> ________________________________________
>>>>>>>>>>>>>>> From: Gray, Jonathan <Jo...@comcast.com>
>>>>>>>>>>>>>>> Sent: Tuesday, December 18, 2018 9:34 AM
>>>>>>>>>>>>>>> To: dev@trafficcontrol.apache.org
>>>>>>>>>>>>>>> Subject: Re: [EXTERNAL] Re: Origins assigned to Multipe
>>>>>>>>>>> Delivery Services
>>>>>>>>>>>>>>> produces indeterminate parent.config
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> -1 Holding an ATC upgrade hostage to data cleanup seems
>>>>>>>> like
>>>>>>>>>> a
>>>>>>>>>>> bad idea.
>>>>>>>>>>>>>>> The issue isn't great, but it's also not new. We
>>>>>>> should
>>>>>>>>>> allow
>>>>>>>>>>> teams to
>>>>>>>>>>>>>> fix
>>>>>>>>>>>>>>> their data at their normal paces if it doesn't create
>>>>>>>>>>> significant
>>>>>>>>>>>>>> overhead
>>>>>>>>>>>>>>> or an inherant blocker for new functionality or
>>>>>>>> correction of
>>>>>>>>>>> other major
>>>>>>>>>>>>>>> problems imho.
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> Jonathan G
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> On 12/18/18, 9:28 AM, "Fieck, Brennan" <
>>>>>>>>>>> Brennan_Fieck@comcast.com>
>>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> Another option is we could detect collisions at
>>>>>>>> startup
>>>>>>>>>>> and simply
>>>>>>>>>>>>>>> refuse to continue with the upgrade until the data is
>>>>>>>> fixed.
>>>>>>>>>>> That would
>>>>>>>>>>>>>>> allow people using the now-unsupported data format to
>>>>>>>>>> continue
>>>>>>>>>>> to use
>>>>>>>>>>>>>> their
>>>>>>>>>>>>>>> old versions of Traffic Ops without wrecking their
>>>>>>>> database,
>>>>>>>>>>> but also
>>>>>>>>>>>>>>> provide an incentive to clean up the data.
>>>>>>>>>>>>>>> ________________________________________
>>>>>>>>>>>>>>> From: Gray, Jonathan <Jo...@comcast.com>
>>>>>>>>>>>>>>> Sent: Tuesday, December 18, 2018 5:12 AM
>>>>>>>>>>>>>>> To: dev@trafficcontrol.apache.org
>>>>>>>>>>>>>>> Subject: Re: [EXTERNAL] Re: Origins assigned to
>>>>>>>> Multipe
>>>>>>>>>>> Delivery
>>>>>>>>>>>>>>> Services produces indeterminate parent.config
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> I'm generally a fan of constrain your data in your
>>>>>>>>>>> database, but not
>>>>>>>>>>>>>>> necessarily exclusively. I see this as a one-way
>>>>>>>>>>> cleanup/conversion so
>>>>>>>>>>>>>> it
>>>>>>>>>>>>>>> doesn't need to be configurable; otherwise you have to
>>>>>>>> ask
>>>>>>>>>> the
>>>>>>>>>>> question
>>>>>>>>>>>>>>> what happens if someone turns it off. That said,
>>>>>>>> something
>>>>>>>>>> in
>>>>>>>>>>> the UI
>>>>>>>>>>>>>> layer
>>>>>>>>>>>>>>> would be nice to prevent spending significant
>>>>>>> quantities
>>>>>>>> of
>>>>>>>>>>> time
>>>>>>>>>>>>>> building a
>>>>>>>>>>>>>>> complex DS only to have it fail to post for reasons
>>>>>>> that
>>>>>>>>>> could
>>>>>>>>>>> have been
>>>>>>>>>>>>>>> known earlier.
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> The way my brain works in this case:
>>>>>>>>>>>>>>> If !unique_constraint_exists_query()
>>>>>>>>>>>>>>> If has_duplicates_query()
>>>>>>>>>>>>>>> show_warning()
>>>>>>>>>>>>>>> else
>>>>>>>>>>>>>>> add_unique_constraint()
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> to which the API and UI configuration could also
>>>>>>>> make use
>>>>>>>>>>> of
>>>>>>>>>>>>>>> unique_constraint_exists_query() to drive additional
>>>>>>>> layer
>>>>>>>>>>> constraints if
>>>>>>>>>>>>>>> desired.
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> Jonathan G
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> On 12/17/18, 1:11 PM, "Rawlin Peters" <
>>>>>>>>>>> rawlin.peters@gmail.com>
>>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> That is an interesting idea...detect at TO
>>>>>>>> startup
>>>>>>>>>>> whether or not
>>>>>>>>>>>>>>> there are duplicate origins and operate in a
>>>>>>>> "prevent
>>>>>>>>>>> duplicate
>>>>>>>>>>>>>>> origins" state if no duplicates are found or
>>>>>>>> "prevent
>>>>>>>>>>> conflicting
>>>>>>>>>>>>>>> DS
>>>>>>>>>>>>>>> topologies" state if duplicates are found? So
>>>>>>>> once
>>>>>>>>>>> operators have
>>>>>>>>>>>>>>> replaced all the duplicate origins with CNAMEs,
>>>>>>>> TO
>>>>>>>>>> will
>>>>>>>>>>>>>> essentially
>>>>>>>>>>>>>>> operate in a "prohibit all duplicate origins"
>>>>>>>> state.
>>>>>>>>>>> That would
>>>>>>>>>>>>>>> probably make for a simpler transition, but I'd
>>>>>>>> want
>>>>>>>>>>> to remove
>>>>>>>>>>>>>> that
>>>>>>>>>>>>>>> logic in a following release that strictly
>>>>>>>> prohibits
>>>>>>>>>>> duplicate
>>>>>>>>>>>>>>> origins
>>>>>>>>>>>>>>> (assuming that the community agrees we should
>>>>>>>>>> prohibit
>>>>>>>>>>> duplicate
>>>>>>>>>>>>>>> origins altogether).
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> As for DB constraints vs UI, I was thinking
>>>>>>> those
>>>>>>>>>>> DS-type
>>>>>>>>>>>>>>> constraints
>>>>>>>>>>>>>>> I pointed out would live in the API. It would
>>>>>>>>>>> basically be added
>>>>>>>>>>>>>>> validation in the deliveryservices POST/PUT
>>>>>>>> endpoint
>>>>>>>>>>> that checks
>>>>>>>>>>>>>>> the
>>>>>>>>>>>>>>> DB for existing DSes that conflict with the
>>>>>>>> requested
>>>>>>>>>>> DS.
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> - Rawlin
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> On Mon, Dec 17, 2018 at 12:35 PM Gray, Jonathan
>>>>>>>>>>>>>>> <Jo...@comcast.com> wrote:
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> These kinds of conditions should be
>>>>>>> detectable
>>>>>>>>>> with a
>>>>>>>>>>>>>>> sufficiently advanced SQL query. Is it possible to add
>>>>>>>> the
>>>>>>>>>>> constraint if
>>>>>>>>>>>>>>> it passes and emit a warning during TO startup
>>>>>>> otherwise?
>>>>>>>>>>> That would let
>>>>>>>>>>>>>>> you know the condition exists at startup but not
>>>>>>> getting
>>>>>>>> in
>>>>>>>>>>> your way and
>>>>>>>>>>>>>>> keep you out of trouble once you've cleaned up. We
>>>>>>> made
>>>>>>>> a
>>>>>>>>>>> mistake early
>>>>>>>>>>>>>>> on, but this would acknowledge it was bad and encourage
>>>>>>>> it to
>>>>>>>>>>> be fixed at
>>>>>>>>>>>>>>> the speed of operations teams. Also this puts the
>>>>>>>> constraint
>>>>>>>>>>> in the
>>>>>>>>>>>>>>> database rather than the UI which is really where the
>>>>>>>>>>> contention is for
>>>>>>>>>>>>>>> usability.
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> Jonathan G
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> On 12/17/18, 11:38 AM, "Rawlin Peters" <
>>>>>>>>>>>>>> rawlin.peters@gmail.com>
>>>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> We occasionally discuss this issue but
>>>>>>>> haven't
>>>>>>>>>>> tackled it
>>>>>>>>>>>>>>> yet. I think
>>>>>>>>>>>>>>>> the main issue is just that duplicate
>>>>>>>> origins
>>>>>>>>>>> have been
>>>>>>>>>>>>>>> allowed since
>>>>>>>>>>>>>>>> the beginning, and now everyone's Traffic
>>>>>>>> Ops
>>>>>>>>>>> could be
>>>>>>>>>>>>>>> littered with
>>>>>>>>>>>>>>>> duplicate origins. Also, depending on the
>>>>>>>>>> config
>>>>>>>>>>> of the
>>>>>>>>>>>>>>> duplicate
>>>>>>>>>>>>>>>> delivery services, the origins might not
>>>>>>>> be in
>>>>>>>>>>> conflict at
>>>>>>>>>>>>>>> all (if
>>>>>>>>>>>>>>>> they don't have different topology
>>>>>>>>>> constraints).
>>>>>>>>>>> I would
>>>>>>>>>>>>>>> love for us
>>>>>>>>>>>>>>>> to just add a uniqueness constraint, but
>>>>>>>> there
>>>>>>>>>>> would need
>>>>>>>>>>>>>> to
>>>>>>>>>>>>>>> be a fair
>>>>>>>>>>>>>>>> amount of warning to the community before
>>>>>>>> doing
>>>>>>>>>>> so and
>>>>>>>>>>>>>> might
>>>>>>>>>>>>>>>> invalidate a significant amount of valid
>>>>>>>> use
>>>>>>>>>>> cases.
>>>>>>>>>>>>>>> Operators would
>>>>>>>>>>>>>>>> need time to make DNS CNAME records for
>>>>>>> the
>>>>>>>>>>> duplicate
>>>>>>>>>>>>>>> origins and
>>>>>>>>>>>>>>>> update their DSes to use the different
>>>>>>>> CNAMEs.
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> I think as a good first step to
>>>>>>>> eliminating the
>>>>>>>>>>> use of
>>>>>>>>>>>>>>> duplicate
>>>>>>>>>>>>>>>> origins altogether, we should identify
>>>>>>>> which
>>>>>>>>>>> "topology
>>>>>>>>>>>>>>> constraints"
>>>>>>>>>>>>>>>> actually cause conflicting config when
>>>>>>> used
>>>>>>>>>> with
>>>>>>>>>>> duplicate
>>>>>>>>>>>>>>> origins and
>>>>>>>>>>>>>>>> prevent creating DSes with duplicate
>>>>>>>> origins
>>>>>>>>>> _if
>>>>>>>>>>> it would
>>>>>>>>>>>>>>> cause a
>>>>>>>>>>>>>>>> conflict with an existing DS that uses
>>>>>>> the
>>>>>>>> same
>>>>>>>>>>> origin_.
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> For instance, I believe an HTTP and
>>>>>>>> DNS-type DS
>>>>>>>>>>> can live
>>>>>>>>>>>>>>> happily
>>>>>>>>>>>>>>>> side-by-side using the same origin
>>>>>>>> (probably
>>>>>>>>>>> need different
>>>>>>>>>>>>>>>> routing_names?), but scenarios like HTTP
>>>>>>>> and
>>>>>>>>>>> HTTP_LIVE, or
>>>>>>>>>>>>>>> DNS and
>>>>>>>>>>>>>>>> HTTP_NO_CACHE sharing the same origin
>>>>>>> will
>>>>>>>>>> cause
>>>>>>>>>>> conflicts
>>>>>>>>>>>>>>> for sure.
>>>>>>>>>>>>>>>> So maybe we can start by making sure the
>>>>>>> DS
>>>>>>>>>>> types "match"
>>>>>>>>>>>>>>> when using
>>>>>>>>>>>>>>>> the same origin:
>>>>>>>>>>>>>>>> HTTP + DNS: possibly good, if they have
>>>>>>>>>>> different routing
>>>>>>>>>>>>>>> names?
>>>>>>>>>>>>>>>> HTTP_LIVE + HTTP_LIVE_NATNL: bad
>>>>>>>>>>>>>>>> HTTP_NO_CACHE + [any other type]: bad
>>>>>>>>>>>>>>>> HTTP_LIVE + HTTP: bad
>>>>>>>>>>>>>>>> etc.
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> There are most likely other conflict
>>>>>>>> scenarios
>>>>>>>>>>> that don't
>>>>>>>>>>>>>>> involve the
>>>>>>>>>>>>>>>> DS types, but I think this would be a
>>>>>>> good
>>>>>>>>>>> start. In the
>>>>>>>>>>>>>>> future with
>>>>>>>>>>>>>>>> Delivery Service Topologies (aka Flexible
>>>>>>>>>>> Cachegroups aka
>>>>>>>>>>>>>>> Bring Your
>>>>>>>>>>>>>>>> Own Topology), we might be able to
>>>>>>> prohibit
>>>>>>>>>>> assigning a DS
>>>>>>>>>>>>>>> to a
>>>>>>>>>>>>>>>> Topology if the DS's origin is already
>>>>>>>> used by
>>>>>>>>>>> another DS
>>>>>>>>>>>>>> in
>>>>>>>>>>>>>>> a
>>>>>>>>>>>>>>>> different Topology.
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> - Rawlin
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> On Mon, Dec 17, 2018 at 10:52 AM Fieck,
>>>>>>>> Brennan
>>>>>>>>>>>>>>>> <Br...@comcast.com> wrote:
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> As some of you may be aware,
>>>>>>>> `parent.config`
>>>>>>>>>>> files
>>>>>>>>>>>>>>> generated by Traffic Ops can vary wildly when an origin
>>>>>>>> is
>>>>>>>>>>> assigned to
>>>>>>>>>>>>>>> multiple Delivery Services. This results in undefined
>>>>>>>>>>> behavior. I'm told
>>>>>>>>>>>>>>> that the conflict only happens when two Delivery
>>>>>>> Services
>>>>>>>>>> with
>>>>>>>>>>> different
>>>>>>>>>>>>>>> "topology requirements" use the same origin, whatever
>>>>>>>> that
>>>>>>>>>>> means (content
>>>>>>>>>>>>>>> routing type?). Regardless, the issue should be
>>>>>>>> addressed.
>>>>>>>>>> The
>>>>>>>>>>> obvious
>>>>>>>>>>>>>>> solution is to put in place a database constraint that
>>>>>>>>>>> prevents an origin
>>>>>>>>>>>>>>> from being assigned to more that one Delivery Service
>>>>>>>> with
>>>>>>>>>> API
>>>>>>>>>>> checks in
>>>>>>>>>>>>>>> place that would provide helpful error messages when an
>>>>>>>>>>> attempt is made
>>>>>>>>>>>>>> to
>>>>>>>>>>>>>>> violate the constraint. However, would that mess with
>>>>>>>> things
>>>>>>>>>>> like
>>>>>>>>>>>>>>> Multi-Site Origin? Or is it just not viable for some
>>>>>>>> other
>>>>>>>>>>> reason? If it
>>>>>>>>>>>>>> is
>>>>>>>>>>>>>>> a good solution, I'm prepared to work on a fix that
>>>>>>>> utilizes
>>>>>>>>>>> it.
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>
>>>>>>>
>>>>>>
>>>>>>
>>>
>>>
>>>
>>
Re: [EXTERNAL] Re: Origins assigned to Multipe Delivery Services
produces indeterminate parent.config
Posted by John Rushford <jj...@gmail.com>.
This is the records.config setting: proxy.config.url_remap.pristine_host_hdr. The vhost
config on the origin would need to match the CNAME.
> On Jan 31, 2019, at 10:30 PM, John Rushford <jj...@gmail.com> wrote:
>
> origin virtual hosting relies on the Host header. So if the request from the client to the reverse
> proxy has a Host header and maintain_pristine_host_hdr is not enabled, remap
> will rewrite the Host header along with the request url. So in this case, remapping to
> a CNAME should work.
>
>> On Jan 29, 2019, at 6:02 PM, Steve Malenfant <sm...@gmail.com> wrote:
>>
>> CNAME might not necessary work if the origin is using virtual hosting. It
>> would need to be configured as well. Might work with some origin? But not
>> all of them.
>>
>> On Tue, Jan 29, 2019 at 7:40 PM Fieck, Brennan <Br...@comcast.com>
>> wrote:
>>
>>> The issue is inherent with the structure of ATS configuration files. If I
>>> understand correctly - someone correct me if I don't -
>>> you can't have more than one rule for a "primary destination" in the
>>> parent.config. Or, you can, but only one of them will
>>> wind up catching all of your matches. Which means that if you have
>>> multiple Delivery Services that use the same origin FQDN
>>> (dest_domain) then you get proper behavior out of ATS if and only if the
>>> resulting rules would be identical. I'd highly
>>> recommend that you not move forward using duplicate origin FQDNs on your
>>> Delivery Services, because it probably won't
>>> do what you want.
>>>
>>> I'm not sure what you mean about the cachekeys. The Delivery Service URLs
>>> should still be unique, so you shouldn't run
>>> into a collision in the cache keys.
>>>
>>> Origins are an object. They're also a cachegroup type. And a profile type.
>>> And a server type. And a URL field on a Delivery Service.
>>>
>>> The solution that's been used in the past to use the same actual origin
>>> machine to service multiple Delivery Services safely
>>> is to add a "Canonical Name" DNS record to the ATC internal zone for each
>>> Delivery Service. The idea is to have a unique name
>>> for each Delivery Service, but they all resolve to the same IP address.
>>> ________________________________________
>>> From: Steve Malenfant <sm...@gmail.com>
>>> Sent: Tuesday, January 29, 2019 4:32 PM
>>> To: dev@trafficcontrol.apache.org
>>> Subject: Re: [EXTERNAL] Re: Origins assigned to Multipe Delivery Services
>>> produces indeterminate parent.config
>>>
>>> We use and will use the same origin for multiple DS. It’s an inherent part
>>> of the design of some part of our delivery.
>>>
>>> Wouldn’t be less of a problem using cachekeys for different delivery
>>> services?
>>>
>>> I would rather see origin as an object and not just a simple URL. I’ve seen
>>> this in the past in other systems.
>>>
>>> I don’t believe we can create multiple http services endpoint for a single
>>> DS like we can for DNS type delivery service. This would solve it for us.
>>>
>>> Steve
>>>
>>> On Thu, Jan 24, 2019 at 11:22 AM Gray, Jonathan <Jonathan_Gray@comcast.com
>>>>
>>> wrote:
>>>
>>>> Even that has a strong element of confusion when in a self-service
>>>> multi-tenancy world. Multiple tenants can use the same origins for
>>> things
>>>> like akami, aws, internal load balancers and any communication about the
>>>> origin already being used will be confusing.
>>>>
>>>> Jonathan G
>>>>
>>>>
>>>> On 1/23/19, 7:37 PM, "Gelinas, Derek" <De...@comcast.com>
>>> wrote:
>>>>
>>>> Perhaps we could solve both issues and just allow it to happen on DS
>>>> config but throw up a message when it is saved that says "hey we're going
>>>> to do this but it's really not without an element of risk and ats isn't
>>>> designed to work properly with multiple identical origins. We recommend
>>> you
>>>> use an alternative fqdn blah blah blah."
>>>>
>>>> Something along those lines, anyway.
>>>>
>>>> Derek
>>>>
>>>>> On Jan 23, 2019, at 7:18 PM, Rawlin Peters <
>>> rawlin.peters@gmail.com>
>>>> wrote:
>>>>>
>>>>> Simply prohibiting duplicate origin FQDNs has been vetoed now, so
>>> we
>>>>> can't really move forward with that unless the vetoers change their
>>>>> minds and remove their -1s.
>>>>>
>>>>> Currently that leaves us with checking at DS create/update whether
>>> or
>>>>> not it would conflict with a shared origin due to different DS
>>> types,
>>>>> qstring config, etc. I am -1 on that idea since I think it will be
>>>>> difficult to maintain and enumerate all the possible conflicting
>>>>> cases, will present confusing error messages to the user to which
>>> the
>>>>> solution would just be "create a CNAME record" anyways, and that
>>> the
>>>>> ATS parent.config should really be per-remap rather than "global".
>>> I
>>>>> know there has been some discussion in the ATS community about
>>> making
>>>>> parent.config per-remap, but I don't have much information there.
>>>>>
>>>>> That is to say, I don't think there is a great solution to this
>>>>> problem right now that we can all agree upon, but maybe our efforts
>>>>> would be better spent in the ATS community by making a per-remap
>>>>> parent.config. Then the duplicate origin problem in Traffic Control
>>>>> would go away.
>>>>>
>>>>> Another option could be to have some kind of generic rules engine
>>>> that
>>>>> would allow an organization to create some set of rules that DSes
>>>>> would have to abide by. For example, no duplicate origins, an
>>> origin
>>>>> can't be in a list of known malicious domains, you can't use the DS
>>>>> type "HTTP_NO_CACHE", maxDnsAnswers must be < 5, TTLs must be > 30
>>>> and
>>>>> < 6000, protocol must be HTTPS, intitialDispersion must be < 10,
>>> DSCP
>>>>> must be 40, etc. It seems like for something truly self-service
>>> you'd
>>>>> probably need to set up some rules for your users to abide by that
>>>>> might not necessarily apply to ALL organizations. Just a thought. I
>>>>> don't think the duplicate origins problem alone would warrant
>>>>> something like that, but if it also provided a solution for the
>>>>> self-service effort, maybe it would be worth it.
>>>>>
>>>>> - Rawlin
>>>>>
>>>>> On Wed, Jan 23, 2019 at 4:15 AM Gelinas, Derek
>>>>> <De...@comcast.com> wrote:
>>>>>>
>>>>>> That's the problem. We either need to be smart enough to recognize
>>>> that in our config and warn the user or just prevent it and suggest to
>>> the
>>>> user that they use a different fqdn.
>>>>>>
>>>>>> My vote is for preventing it at this point.
>>>>>>
>>>>>>> On Jan 23, 2019, at 12:50 AM, Gray, Jonathan <
>>>> Jonathan_Gray@comcast.com> wrote:
>>>>>>>
>>>>>>> It's not just differing types, it's anything that affects the
>>>> parent.config which also includes the, qstring configuration, MSO
>>>> parents/config, etc. as well.
>>>>>>>
>>>>>>> Jonathan G
>>>>>>>
>>>>>>>
>>>>>>> On 1/22/19, 2:33 PM, "Dave Neuman" <ne...@apache.org> wrote:
>>>>>>>
>>>>>>> -1 on the one DS to Origin limitation. I think there are legit
>>>> use cases
>>>>>>> for it.
>>>>>>> I would be +1 on the idea if we include type though. Two DSs
>>>> can share the
>>>>>>> same origin as long as they are of the same type (DNS, HTTP,
>>>> HTTP_LIVE),
>>>>>>> etc. You shouldn't be able to have two DSs with the same
>>> origin
>>>> but
>>>>>>> different types.
>>>>>>>
>>>>>>> On Mon, Jan 14, 2019 at 7:53 AM Fieck, Brennan <
>>>> Brennan_Fieck@comcast.com>
>>>>>>> wrote:
>>>>>>>
>>>>>>>> I don't see this as a complicated use case or a barrier to
>>> entry,
>>>> an
>>>>>>>> extremely basic introduction to ATC would only make use of one
>>> DS
>>>>>>>> (shameless CDN-in-a-Box plug).
>>>>>>>> It seems far more complicated for a user to be using many DSes
>>> to
>>>> serve a
>>>>>>>> single origin - we can easily make it clear in the docs that the
>>>> two have a
>>>>>>>> 1:1 relationship, with a footnote about CNAME records for users
>>>> with the
>>>>>>>> knowledge, resources, and need for such workarounds.
>>>>>>>> ________________________________________
>>>>>>>> From: Mark Torluemke <mt...@apache.org>
>>>>>>>> Sent: Friday, January 11, 2019 1:21 PM
>>>>>>>> To: dev@trafficcontrol.apache.org
>>>>>>>> Subject: Re: [EXTERNAL] Re: Origins assigned to Multipe Delivery
>>>> Services
>>>>>>>> produces indeterminate parent.config
>>>>>>>>
>>>>>>>>>> 1. Prohibit creating new delivery services that would share an
>>>>>>>> existing origin and prohibit updating a delivery service to a
>>>> shared
>>>>>>>> origin
>>>>>>>> In case my position has been lost, I'm still -1 on this. :) IMO,
>>>> we
>>>>>>>> shouldn't let the few complicated use cases increase the
>>> barrier
>>>> to entry
>>>>>>>> (to using ATC) for the masses.
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> On Fri, Jan 11, 2019 at 11:53 AM Rawlin Peters <
>>>> rawlin.peters@gmail.com>
>>>>>>>> wrote:
>>>>>>>>
>>>>>>>>> Alright, I'm trying to sum up this discussion so far since it
>>>> seems
>>>>>>>>> like everyone went on vacation and didn't really get a chance
>>> to
>>>> wrap
>>>>>>>>> this one up:
>>>>>>>>> - duplicate origins cause undefined behavior
>>>>>>>>> - we need a way to migrate to a future that is free of
>>> duplicate
>>>>>>>>> origins in Traffic Control
>>>>>>>>> - we need a visual and easy way to determine if Traffic Ops
>>>> currently
>>>>>>>>> contains duplicate origins, so that operators are incentivized
>>>> to fix
>>>>>>>>> them rather than let it slide indefinitely
>>>>>>>>> - operators should have a fair amount of time to fix their
>>>> duplicate
>>>>>>>>> origins
>>>>>>>>>
>>>>>>>>> I believe this is what we've mostly agreed upon but haven't
>>>> clearly voted
>>>>>>>>> on:
>>>>>>>>>
>>>>>>>>> In release N we will:
>>>>>>>>> 1. Prohibit creating new delivery services that would share an
>>>>>>>>> existing origin and prohibit updating a delivery service to a
>>>> shared
>>>>>>>>> origin
>>>>>>>>> 2. Add some kind of visual indicator that duplicate origins
>>> are a
>>>>>>>>> problem that need to be fixed before release N+1; otherwise, an
>>>>>>>>> upgrade to N+1 will be prohibited.
>>>>>>>>>
>>>>>>>>> In release N+1 we will:
>>>>>>>>> 3. Include a DB migration that adds a uniqueness constraint on
>>>> origin
>>>>>>>>> FQDN, removing the API-level checks for that.
>>>>>>>>> 4. Prevent an upgrade to N+1 if duplicate origins are found
>>> (this
>>>>>>>>> might occur as a byproduct of step 3).
>>>>>>>>>
>>>>>>>>> I am +1 on this plan and believe this would hit on all the
>>>> summarized
>>>>>>>>> points above. Please provide a clear vote on this plan so that
>>>> we can
>>>>>>>>> dive deeper in the details (i.e. what release 'N' is, the best
>>>> visual
>>>>>>>>> indicator for step 2, and a friendly way to handle step 4).
>>>>>>>>>
>>>>>>>>> Thanks,
>>>>>>>>> Rawlin
>>>>>>>>>
>>>>>>>>> On Wed, Dec 19, 2018 at 3:39 PM Jeremy Mitchell <
>>>> mitchell852@gmail.com>
>>>>>>>>> wrote:
>>>>>>>>>>
>>>>>>>>>> Not sure TP is the right place for a warning like "clean up
>>> this
>>>>>>>>>> 'duplicate' origin or your next upgrade will fail". Most users
>>>> of our
>>>>>>>>>> system will be like "not my problem".
>>>>>>>>>>
>>>>>>>>>> On Wed, Dec 19, 2018 at 11:58 AM Fieck, Brennan <
>>>>>>>>> Brennan_Fieck@comcast.com>
>>>>>>>>>> wrote:
>>>>>>>>>>
>>>>>>>>>>> Probably. It would impact load times by a bit, but the page
>>>> for an
>>>>>>>>>>> individual object is not our bottleneck.
>>>>>>>>>>> ________________________________________
>>>>>>>>>>> From: Robert Butts <ro...@apache.org>
>>>>>>>>>>> Sent: Wednesday, December 19, 2018 11:50 AM
>>>>>>>>>>> To: dev@trafficcontrol.apache.org
>>>>>>>>>>> Subject: Re: [EXTERNAL] Re: Origins assigned to Multipe
>>>> Delivery
>>>>>>>>> Services
>>>>>>>>>>> produces indeterminate parent.config
>>>>>>>>>>>
>>>>>>>>>>>> - Including a warning on startup and an API constraint
>>>> preventing
>>>>>>>>> adding
>>>>>>>>>>> more bad data in the next 3.0.0 Release Candidate
>>>>>>>>>>>> - Adding a database constraint immediately into master that
>>>> won't
>>>>>>>> be
>>>>>>>>>>> cherry-picked into 3.0.0 but should be included in 3.1.0
>>>>>>>>>>>
>>>>>>>>>>> +1
>>>>>>>>>>>
>>>>>>>>>>> I understand Jonathan's objection, but at some point, we have
>>>> to be
>>>>>>>>> able to
>>>>>>>>>>> move forward. This is a good compromise: deprecate, then
>>>> remove. That
>>>>>>>>> gives
>>>>>>>>>>> people a full major version to fix their data.
>>>>>>>>>>>
>>>>>>>>>>> I would be ideal if it were more than just a logged warning,
>>>> though.
>>>>>>>>> Can we
>>>>>>>>>>> add a big red banner in Traffic Portal, on the Delivery
>>>> Service page
>>>>>>>>> for
>>>>>>>>>>> any DS with a duplicate origin, telling users to fix it, and
>>>> that
>>>>>>>> they
>>>>>>>>>>> won't be able to upgrade to the next major version until it's
>>>> fixed?
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> On Wed, Dec 19, 2018 at 10:57 AM Fieck, Brennan <
>>>>>>>>> Brennan_Fieck@comcast.com
>>>>>>>>>>>>
>>>>>>>>>>> wrote:
>>>>>>>>>>>
>>>>>>>>>>>> So it seems like nobody has a problem with the "how" -
>>>> disallowing
>>>>>>>>>>>> duplicate origin FQDNs on Delivery Services - but we never
>>>> reached
>>>>>>>> a
>>>>>>>>>>>> consensus on "when".
>>>>>>>>>>>>
>>>>>>>>>>>> I stand by my previous proposal:
>>>>>>>>>>>> - Including a warning on startup and an API constraint
>>>> preventing
>>>>>>>>> adding
>>>>>>>>>>>> more bad data in the next 3.0.0 Release Candidate
>>>>>>>>>>>> - Adding a database constraint immediately into master that
>>>> won't
>>>>>>>> be
>>>>>>>>>>>> cherry-picked into 3.0.0 but should be included in 3.1.0
>>>>>>>>>>>> ________________________________________
>>>>>>>>>>>> From: Rawlin Peters <ra...@gmail.com>
>>>>>>>>>>>> Sent: Tuesday, December 18, 2018 4:59 PM
>>>>>>>>>>>> To: dev@trafficcontrol.apache.org
>>>>>>>>>>>> Subject: Re: [EXTERNAL] Re: Origins assigned to Multipe
>>>> Delivery
>>>>>>>>> Services
>>>>>>>>>>>> produces indeterminate parent.config
>>>>>>>>>>>>
>>>>>>>>>>>> Also, building more around DS types will make it even harder
>>>> to get
>>>>>>>>>>>> away from DS types in the future too, which I know is
>>>> something
>>>>>>>> we've
>>>>>>>>>>>> discussed on this mailing list before. It also adds to the
>>>> overhead
>>>>>>>>> of
>>>>>>>>>>>> Delivery Service Topologies, since a lot of the DS types
>>> won't
>>>>>>>>>>>> carryover into that world.
>>>>>>>>>>>>
>>>>>>>>>>>> - Rawlin
>>>>>>>>>>>>
>>>>>>>>>>>> On Tue, Dec 18, 2018 at 2:42 PM Fieck, Brennan
>>>>>>>>>>>> <Br...@comcast.com> wrote:
>>>>>>>>>>>>>
>>>>>>>>>>>>> +1.
>>>>>>>>>>>>> If there's a simple way to work around duplicate origins
>>>> being
>>>>>>>>>>>> prohibited,
>>>>>>>>>>>>> then we should rely on that instead of "enumerating all
>>> those
>>>>>>>>> possible
>>>>>>>>>>>> conflicting
>>>>>>>>>>>>> settings, which are not only highly complex and confusing,
>>>> but
>>>>>>>> also
>>>>>>>>>>>> further
>>>>>>>>>>>>> entrench us in only supporting ATS as a caching proxy
>>>> (hurting
>>>>>>>>> efforts
>>>>>>>>>>> to
>>>>>>>>>>>>> integrate e.g. Grove, nginx etc.)
>>>>>>>>>>>>> ________________________________________
>>>>>>>>>>>>> From: Rawlin Peters <ra...@gmail.com>
>>>>>>>>>>>>> Sent: Tuesday, December 18, 2018 2:20 PM
>>>>>>>>>>>>> To: dev@trafficcontrol.apache.org
>>>>>>>>>>>>> Subject: Re: [EXTERNAL] Re: Origins assigned to Multipe
>>>> Delivery
>>>>>>>>>>>> Services produces indeterminate parent.config
>>>>>>>>>>>>>
>>>>>>>>>>>>> There are a number of different DS settings at play that
>>> can
>>>>>>>>>>>>> potentially cause conflicts. The question is: do we want to
>>>> get
>>>>>>>>> into
>>>>>>>>>>>>> the business of enumerating all those possible conflicting
>>>>>>>>> settings or
>>>>>>>>>>>>> just simply prohibit duplicate origins altogether? I think
>>>> we can
>>>>>>>>> dig
>>>>>>>>>>>>> in and get that "sufficiently advanced sql query" to check
>>>> for
>>>>>>>>>>>>> conflicting origins, but is that something we want to carry
>>>> along
>>>>>>>>> for
>>>>>>>>>>>>> the foreseeable future? Aren't CNAMEs relatively cheaper
>>> than
>>>>>>>>>>>>> developing and maintaining that code and the mental
>>> overhead
>>>>>>>>> required
>>>>>>>>>>>>> in understanding why you're getting an error that says your
>>>>>>>>> requested
>>>>>>>>>>>>> DS would cause an origin conflict? I think at the point
>>>> you've
>>>>>>>>>>>>> requested a DS that would create a conflict, you've chosen
>>>> those
>>>>>>>>>>>>> settings for a reason and would probably prefer to just
>>>>>>>> create/use
>>>>>>>>> a
>>>>>>>>>>>>> CNAME in your new DS and keep the rest of your settings the
>>>> same.
>>>>>>>>>>>>>
>>>>>>>>>>>>> Thinking in terms of errors, I'm imagining:
>>>>>>>>>>>>> "cannot create delivery service: origin fqdn '
>>>> foo.example.com'
>>>>>>>>> already
>>>>>>>>>>>> in use"
>>>>>>>>>>>>> vs
>>>>>>>>>>>>> "cannot create delivery service: origin fqdn '
>>>> foo.example.com'
>>>>>>>>> already
>>>>>>>>>>>>> in use as type DNS_LIVE_NATNL, which is incompatible with
>>>> your
>>>>>>>>> chosen
>>>>>>>>>>>>> type of HTTP_NO_CACHE"
>>>>>>>>>>>>>
>>>>>>>>>>>>> At that point you'd probably say to yourself, "uh, I need
>>>>>>>>>>>>> HTTP_NO_CACHE, so what am I supposed to do now?"
>>>>>>>>>>>>>
>>>>>>>>>>>>> As a lazy developer I'm +1 on prohibiting duplicate origin
>>>> fqdns
>>>>>>>>>>>>> because the resulting code will be simpler, but I think
>>>>>>>> eliminating
>>>>>>>>>>>>> the mental overhead for operators could be worthwhile too.
>>>> If we
>>>>>>>>> can
>>>>>>>>>>>>> agree on an end state of prohibiting duplicate origins
>>>>>>>> altogether,
>>>>>>>>> we
>>>>>>>>>>>>> can start working on a design to smoothly transition us to
>>>> that
>>>>>>>>> point.
>>>>>>>>>>>>> Are we willing to live with "just CNAME your origin fqdn"
>>> as
>>>> the
>>>>>>>>>>>>> standard solution to duplicates?
>>>>>>>>>>>>>
>>>>>>>>>>>>> - Rawlin
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>> On Tue, Dec 18, 2018 at 1:27 PM Gelinas, Derek
>>>>>>>>>>>>> <De...@comcast.com> wrote:
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> The only situation in which they can share origins is if
>>> a)
>>>> the
>>>>>>>>>>>> origins are shared in an MSO configuration but still have
>>>> different
>>>>>>>>>>> defined
>>>>>>>>>>>> origin fields in the delivery service, or if they're
>>> assigned
>>>> to
>>>>>>>>>>> completely
>>>>>>>>>>>> different cachegroups. It's when two delivery services
>>> share
>>>> the
>>>>>>>>> same
>>>>>>>>>>>> edges that there's an issue, because you end up with
>>>> parent.config
>>>>>>>>>>> issues.
>>>>>>>>>>>> Actually you could even get away with it in mids as long as
>>>> you
>>>>>>>>> weren't
>>>>>>>>>>>> doing anything like MSO to it.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Could get messy real fast, though. Best to just create a
>>>>>>>> second
>>>>>>>>>>> FQDN.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Derek
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> On 12/18/18, 3:23 PM, "Fieck, Brennan" <
>>>>>>>>> Brennan_Fieck@comcast.com>
>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> So no two Delivery Services may share an origin
>>>> *regardless
>>>>>>>>> of
>>>>>>>>>>>> cache hierarchy* ? I've been told that DNS Delivery Services
>>>> can
>>>>>>>>> have the
>>>>>>>>>>>> same origin as HTTP Delivery Services because they obey the
>>>> same
>>>>>>>>> cache
>>>>>>>>>>>> hierarchy. You're saying that would still produce invalid
>>>> output
>>>>>>>>> and/or
>>>>>>>>>>> is
>>>>>>>>>>>> explicitly disallowed by ATS?
>>>>>>>>>>>>>> ________________________________________
>>>>>>>>>>>>>> From: Robert Butts <ro...@apache.org>
>>>>>>>>>>>>>> Sent: Tuesday, December 18, 2018 1:09 PM
>>>>>>>>>>>>>> To: dev@trafficcontrol.apache.org
>>>>>>>>>>>>>> Subject: Re: [EXTERNAL] Re: Origins assigned to Multipe
>>>>>>>>> Delivery
>>>>>>>>>>>> Services produces indeterminate parent.config
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> can you give an example of what parent.config looks like
>>>>>>>>> when 2
>>>>>>>>>>>> ds's share
>>>>>>>>>>>>>> an origin and have different a different topology?
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Answering because I encountered this directly, when
>>>>>>>> rewriting
>>>>>>>>>>>> parent.config.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> For example: Suppose you have one Delivery Service:
>>>>>>>>>>>>>> XML_ID: foo
>>>>>>>>>>>>>> Type: HTPT_LIVE_NATL
>>>>>>>>>>>>>> Query String Handling: 1 - ignore in cache key, and pass
>>>> up
>>>>>>>>>>>>>> Origin Server Base URL: http://foo.example.net
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> And another Delivery Service:
>>>>>>>>>>>>>> XML_ID: bar
>>>>>>>>>>>>>> Type: HTPT_LIVE_NATL
>>>>>>>>>>>>>> Query String Handling: 1 - ignore in cache key, and pass
>>>> up
>>>>>>>>>>>>>> Origin Server Base URL: http://foo.example.net
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> ATS only supports unique `dest_domain` entries in
>>>>>>>>> parent.config.
>>>>>>>>>>>> Therefore,
>>>>>>>>>>>>>> the parent.config generated for a server assigned to
>>> both
>>>>>>>> of
>>>>>>>>>>> these
>>>>>>>>>>>> Delivery
>>>>>>>>>>>>>> Services with either be:
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> dest_domain=foo.example.net port=80 go_direct=true
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Or
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> dest_domain=foo.example.net port=80 go_direct=false
>>>>>>>>>>>> qstring=consider
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Right now, it's arbitrary which Perl Traffic Ops
>>> inserts,
>>>>>>>> and
>>>>>>>>>>> Perl
>>>>>>>>>>>> provides
>>>>>>>>>>>>>> no warning or error of any kind (the pending Go
>>>>>>>>> parent.config PR
>>>>>>>>>>>> logs an
>>>>>>>>>>>>>> error).
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Whichever is arbitrarily inserted, the resulting remaps
>>>> for
>>>>>>>>> the
>>>>>>>>>>>> other
>>>>>>>>>>>>>> delivery service will be wrong. Either `foo` requests
>>> will
>>>>>>>>> drop
>>>>>>>>>>>> the query
>>>>>>>>>>>>>> string when they shouldn't, and go to the mid when they
>>>>>>>>>>> shouldn't;
>>>>>>>>>>>> or `bar`
>>>>>>>>>>>>>> requests will use the query string and skip the mid when
>>>> it
>>>>>>>>>>>> shouldn't.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Does that make sense? The only correct solution, is to
>>>>>>>>> somehow
>>>>>>>>>>>> prevent
>>>>>>>>>>>>>> different DSes having the same origin, and tell tenants
>>>>>>>> they
>>>>>>>>> must
>>>>>>>>>>>> use
>>>>>>>>>>>>>> CNAMEs if they need.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> This isn't a bug in Traffic Control. ATS fundamentally
>>>>>>>>> doesn't
>>>>>>>>>>>> support
>>>>>>>>>>>>>> multiple remap rules with the same parent FQDN with
>>>>>>>> different
>>>>>>>>>>>>>> configurations. Hence, Traffic Control needs to prohibit
>>>>>>>>> that.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> On Tue, Dec 18, 2018 at 12:24 PM Jeremy Mitchell <
>>>>>>>>>>>> mitchell852@gmail.com>
>>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> brennan,
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> can you give an example of what parent.config looks like
>>>>>>>>> when 2
>>>>>>>>>>>> ds's share
>>>>>>>>>>>>>>> an origin and have different a different topology?
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> jeremy
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> On Tue, Dec 18, 2018 at 11:39 AM Fieck, Brennan <
>>>>>>>>>>>> Brennan_Fieck@comcast.com
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> To be clear, the "Warning" I'm talking about would
>>>>>>>>> happen at
>>>>>>>>>>>> startup, but
>>>>>>>>>>>>>>>> I'd like a UI-only constraint to come with that to
>>>>>>>>> disallow
>>>>>>>>>>>> using the API
>>>>>>>>>>>>>>>> to bind the same origin to multiple Delivery Services
>>>>>>>>> with
>>>>>>>>>>>> varying
>>>>>>>>>>>>>>>> topography requirements. It wouldn't change the
>>>>>>>> existing
>>>>>>>>>>> data,
>>>>>>>>>>>> but
>>>>>>>>>>>>>>> prevent
>>>>>>>>>>>>>>>> users from creating more bad data.
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> "warning" doesn't really sufficiently describe that, my
>>>>>>>>> bad.
>>>>>>>>>>>>>>>> ________________________________________
>>>>>>>>>>>>>>>> From: Fieck, Brennan <Br...@comcast.com>
>>>>>>>>>>>>>>>> Sent: Tuesday, December 18, 2018 11:24 AM
>>>>>>>>>>>>>>>> To: dev@trafficcontrol.apache.org
>>>>>>>>>>>>>>>> Subject: Re: [EXTERNAL] Re: Origins assigned to Multipe
>>>>>>>>>>>> Delivery Services
>>>>>>>>>>>>>>>> produces indeterminate parent.config
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> Well the cost of fixing this bug is a constraint on the
>>>>>>>>> data.
>>>>>>>>>>>> Unless we
>>>>>>>>>>>>>>>> make it a UI-only constraint - which I'm personally
>>>>>>>>> against -
>>>>>>>>>>>> there must
>>>>>>>>>>>>>>> be
>>>>>>>>>>>>>>>> some point in the future where ATC cannot reasonably be
>>>>>>>>>>>> expected to work
>>>>>>>>>>>>>>>> with data that violates that constraint. The question
>>>>>>>> is
>>>>>>>>> when
>>>>>>>>>>>> that should
>>>>>>>>>>>>>>>> occur, which should likely happen at a minor version
>>>>>>>>> release.
>>>>>>>>>>>> Minor not
>>>>>>>>>>>>>>>> major because it doesn't involve a change in data
>>>>>>>>> structures,
>>>>>>>>>>>> merely
>>>>>>>>>>>>>>>> relationships between them - in my opinion that's a
>>>>>>>> minor
>>>>>>>>>>>> version change
>>>>>>>>>>>>>>>> but that's definitely up for debate. With several
>>>>>>>> release
>>>>>>>>>>>> candidates for
>>>>>>>>>>>>>>>> 3.0.0 that _doesn't_ include this restriction already
>>>>>>>> in
>>>>>>>>> the
>>>>>>>>>>>> wild, I
>>>>>>>>>>>>>>>> wouldn't recommend putting it in there. That means to
>>>>>>>>> fix the
>>>>>>>>>>>> bug as soon
>>>>>>>>>>>>>>>> as possible it should go in 3.1.0 which should be the
>>>>>>>>> target
>>>>>>>>>>>> of "master"
>>>>>>>>>>>>>>>> after the 3.0.0 release is cut from it.
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> So I'd recommend immediately implementing the
>>>>>>>> constraint
>>>>>>>>> in
>>>>>>>>>>>> master with a
>>>>>>>>>>>>>>>> refusal to upgrade with bad data, and backport a
>>>>>>>> warning
>>>>>>>>>>> about
>>>>>>>>>>>> the future
>>>>>>>>>>>>>>>> behavior into 3.0.0 or as part of a 3.0.1 provided we
>>>>>>>> had
>>>>>>>>>>> more
>>>>>>>>>>>> changes
>>>>>>>>>>>>>>> that
>>>>>>>>>>>>>>>> would warrant a micro version bump.
>>>>>>>>>>>>>>>> ________________________________________
>>>>>>>>>>>>>>>> From: Gray, Jonathan <Jo...@comcast.com>
>>>>>>>>>>>>>>>> Sent: Tuesday, December 18, 2018 9:34 AM
>>>>>>>>>>>>>>>> To: dev@trafficcontrol.apache.org
>>>>>>>>>>>>>>>> Subject: Re: [EXTERNAL] Re: Origins assigned to Multipe
>>>>>>>>>>>> Delivery Services
>>>>>>>>>>>>>>>> produces indeterminate parent.config
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> -1 Holding an ATC upgrade hostage to data cleanup seems
>>>>>>>>> like
>>>>>>>>>>> a
>>>>>>>>>>>> bad idea.
>>>>>>>>>>>>>>>> The issue isn't great, but it's also not new. We
>>>>>>>> should
>>>>>>>>>>> allow
>>>>>>>>>>>> teams to
>>>>>>>>>>>>>>> fix
>>>>>>>>>>>>>>>> their data at their normal paces if it doesn't create
>>>>>>>>>>>> significant
>>>>>>>>>>>>>>> overhead
>>>>>>>>>>>>>>>> or an inherant blocker for new functionality or
>>>>>>>>> correction of
>>>>>>>>>>>> other major
>>>>>>>>>>>>>>>> problems imho.
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> Jonathan G
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> On 12/18/18, 9:28 AM, "Fieck, Brennan" <
>>>>>>>>>>>> Brennan_Fieck@comcast.com>
>>>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> Another option is we could detect collisions at
>>>>>>>>> startup
>>>>>>>>>>>> and simply
>>>>>>>>>>>>>>>> refuse to continue with the upgrade until the data is
>>>>>>>>> fixed.
>>>>>>>>>>>> That would
>>>>>>>>>>>>>>>> allow people using the now-unsupported data format to
>>>>>>>>>>> continue
>>>>>>>>>>>> to use
>>>>>>>>>>>>>>> their
>>>>>>>>>>>>>>>> old versions of Traffic Ops without wrecking their
>>>>>>>>> database,
>>>>>>>>>>>> but also
>>>>>>>>>>>>>>>> provide an incentive to clean up the data.
>>>>>>>>>>>>>>>> ________________________________________
>>>>>>>>>>>>>>>> From: Gray, Jonathan <Jo...@comcast.com>
>>>>>>>>>>>>>>>> Sent: Tuesday, December 18, 2018 5:12 AM
>>>>>>>>>>>>>>>> To: dev@trafficcontrol.apache.org
>>>>>>>>>>>>>>>> Subject: Re: [EXTERNAL] Re: Origins assigned to
>>>>>>>>> Multipe
>>>>>>>>>>>> Delivery
>>>>>>>>>>>>>>>> Services produces indeterminate parent.config
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> I'm generally a fan of constrain your data in your
>>>>>>>>>>>> database, but not
>>>>>>>>>>>>>>>> necessarily exclusively. I see this as a one-way
>>>>>>>>>>>> cleanup/conversion so
>>>>>>>>>>>>>>> it
>>>>>>>>>>>>>>>> doesn't need to be configurable; otherwise you have to
>>>>>>>>> ask
>>>>>>>>>>> the
>>>>>>>>>>>> question
>>>>>>>>>>>>>>>> what happens if someone turns it off. That said,
>>>>>>>>> something
>>>>>>>>>>> in
>>>>>>>>>>>> the UI
>>>>>>>>>>>>>>> layer
>>>>>>>>>>>>>>>> would be nice to prevent spending significant
>>>>>>>> quantities
>>>>>>>>> of
>>>>>>>>>>>> time
>>>>>>>>>>>>>>> building a
>>>>>>>>>>>>>>>> complex DS only to have it fail to post for reasons
>>>>>>>> that
>>>>>>>>>>> could
>>>>>>>>>>>> have been
>>>>>>>>>>>>>>>> known earlier.
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> The way my brain works in this case:
>>>>>>>>>>>>>>>> If !unique_constraint_exists_query()
>>>>>>>>>>>>>>>> If has_duplicates_query()
>>>>>>>>>>>>>>>> show_warning()
>>>>>>>>>>>>>>>> else
>>>>>>>>>>>>>>>> add_unique_constraint()
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> to which the API and UI configuration could also
>>>>>>>>> make use
>>>>>>>>>>>> of
>>>>>>>>>>>>>>>> unique_constraint_exists_query() to drive additional
>>>>>>>>> layer
>>>>>>>>>>>> constraints if
>>>>>>>>>>>>>>>> desired.
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> Jonathan G
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> On 12/17/18, 1:11 PM, "Rawlin Peters" <
>>>>>>>>>>>> rawlin.peters@gmail.com>
>>>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> That is an interesting idea...detect at TO
>>>>>>>>> startup
>>>>>>>>>>>> whether or not
>>>>>>>>>>>>>>>> there are duplicate origins and operate in a
>>>>>>>>> "prevent
>>>>>>>>>>>> duplicate
>>>>>>>>>>>>>>>> origins" state if no duplicates are found or
>>>>>>>>> "prevent
>>>>>>>>>>>> conflicting
>>>>>>>>>>>>>>>> DS
>>>>>>>>>>>>>>>> topologies" state if duplicates are found? So
>>>>>>>>> once
>>>>>>>>>>>> operators have
>>>>>>>>>>>>>>>> replaced all the duplicate origins with CNAMEs,
>>>>>>>>> TO
>>>>>>>>>>> will
>>>>>>>>>>>>>>> essentially
>>>>>>>>>>>>>>>> operate in a "prohibit all duplicate origins"
>>>>>>>>> state.
>>>>>>>>>>>> That would
>>>>>>>>>>>>>>>> probably make for a simpler transition, but I'd
>>>>>>>>> want
>>>>>>>>>>>> to remove
>>>>>>>>>>>>>>> that
>>>>>>>>>>>>>>>> logic in a following release that strictly
>>>>>>>>> prohibits
>>>>>>>>>>>> duplicate
>>>>>>>>>>>>>>>> origins
>>>>>>>>>>>>>>>> (assuming that the community agrees we should
>>>>>>>>>>> prohibit
>>>>>>>>>>>> duplicate
>>>>>>>>>>>>>>>> origins altogether).
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> As for DB constraints vs UI, I was thinking
>>>>>>>> those
>>>>>>>>>>>> DS-type
>>>>>>>>>>>>>>>> constraints
>>>>>>>>>>>>>>>> I pointed out would live in the API. It would
>>>>>>>>>>>> basically be added
>>>>>>>>>>>>>>>> validation in the deliveryservices POST/PUT
>>>>>>>>> endpoint
>>>>>>>>>>>> that checks
>>>>>>>>>>>>>>>> the
>>>>>>>>>>>>>>>> DB for existing DSes that conflict with the
>>>>>>>>> requested
>>>>>>>>>>>> DS.
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> - Rawlin
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> On Mon, Dec 17, 2018 at 12:35 PM Gray, Jonathan
>>>>>>>>>>>>>>>> <Jo...@comcast.com> wrote:
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> These kinds of conditions should be
>>>>>>>> detectable
>>>>>>>>>>> with a
>>>>>>>>>>>>>>>> sufficiently advanced SQL query. Is it possible to add
>>>>>>>>> the
>>>>>>>>>>>> constraint if
>>>>>>>>>>>>>>>> it passes and emit a warning during TO startup
>>>>>>>> otherwise?
>>>>>>>>>>>> That would let
>>>>>>>>>>>>>>>> you know the condition exists at startup but not
>>>>>>>> getting
>>>>>>>>> in
>>>>>>>>>>>> your way and
>>>>>>>>>>>>>>>> keep you out of trouble once you've cleaned up. We
>>>>>>>> made
>>>>>>>>> a
>>>>>>>>>>>> mistake early
>>>>>>>>>>>>>>>> on, but this would acknowledge it was bad and encourage
>>>>>>>>> it to
>>>>>>>>>>>> be fixed at
>>>>>>>>>>>>>>>> the speed of operations teams. Also this puts the
>>>>>>>>> constraint
>>>>>>>>>>>> in the
>>>>>>>>>>>>>>>> database rather than the UI which is really where the
>>>>>>>>>>>> contention is for
>>>>>>>>>>>>>>>> usability.
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> Jonathan G
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> On 12/17/18, 11:38 AM, "Rawlin Peters" <
>>>>>>>>>>>>>>> rawlin.peters@gmail.com>
>>>>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> We occasionally discuss this issue but
>>>>>>>>> haven't
>>>>>>>>>>>> tackled it
>>>>>>>>>>>>>>>> yet. I think
>>>>>>>>>>>>>>>>> the main issue is just that duplicate
>>>>>>>>> origins
>>>>>>>>>>>> have been
>>>>>>>>>>>>>>>> allowed since
>>>>>>>>>>>>>>>>> the beginning, and now everyone's Traffic
>>>>>>>>> Ops
>>>>>>>>>>>> could be
>>>>>>>>>>>>>>>> littered with
>>>>>>>>>>>>>>>>> duplicate origins. Also, depending on the
>>>>>>>>>>> config
>>>>>>>>>>>> of the
>>>>>>>>>>>>>>>> duplicate
>>>>>>>>>>>>>>>>> delivery services, the origins might not
>>>>>>>>> be in
>>>>>>>>>>>> conflict at
>>>>>>>>>>>>>>>> all (if
>>>>>>>>>>>>>>>>> they don't have different topology
>>>>>>>>>>> constraints).
>>>>>>>>>>>> I would
>>>>>>>>>>>>>>>> love for us
>>>>>>>>>>>>>>>>> to just add a uniqueness constraint, but
>>>>>>>>> there
>>>>>>>>>>>> would need
>>>>>>>>>>>>>>> to
>>>>>>>>>>>>>>>> be a fair
>>>>>>>>>>>>>>>>> amount of warning to the community before
>>>>>>>>> doing
>>>>>>>>>>>> so and
>>>>>>>>>>>>>>> might
>>>>>>>>>>>>>>>>> invalidate a significant amount of valid
>>>>>>>>> use
>>>>>>>>>>>> cases.
>>>>>>>>>>>>>>>> Operators would
>>>>>>>>>>>>>>>>> need time to make DNS CNAME records for
>>>>>>>> the
>>>>>>>>>>>> duplicate
>>>>>>>>>>>>>>>> origins and
>>>>>>>>>>>>>>>>> update their DSes to use the different
>>>>>>>>> CNAMEs.
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> I think as a good first step to
>>>>>>>>> eliminating the
>>>>>>>>>>>> use of
>>>>>>>>>>>>>>>> duplicate
>>>>>>>>>>>>>>>>> origins altogether, we should identify
>>>>>>>>> which
>>>>>>>>>>>> "topology
>>>>>>>>>>>>>>>> constraints"
>>>>>>>>>>>>>>>>> actually cause conflicting config when
>>>>>>>> used
>>>>>>>>>>> with
>>>>>>>>>>>> duplicate
>>>>>>>>>>>>>>>> origins and
>>>>>>>>>>>>>>>>> prevent creating DSes with duplicate
>>>>>>>>> origins
>>>>>>>>>>> _if
>>>>>>>>>>>> it would
>>>>>>>>>>>>>>>> cause a
>>>>>>>>>>>>>>>>> conflict with an existing DS that uses
>>>>>>>> the
>>>>>>>>> same
>>>>>>>>>>>> origin_.
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> For instance, I believe an HTTP and
>>>>>>>>> DNS-type DS
>>>>>>>>>>>> can live
>>>>>>>>>>>>>>>> happily
>>>>>>>>>>>>>>>>> side-by-side using the same origin
>>>>>>>>> (probably
>>>>>>>>>>>> need different
>>>>>>>>>>>>>>>>> routing_names?), but scenarios like HTTP
>>>>>>>>> and
>>>>>>>>>>>> HTTP_LIVE, or
>>>>>>>>>>>>>>>> DNS and
>>>>>>>>>>>>>>>>> HTTP_NO_CACHE sharing the same origin
>>>>>>>> will
>>>>>>>>>>> cause
>>>>>>>>>>>> conflicts
>>>>>>>>>>>>>>>> for sure.
>>>>>>>>>>>>>>>>> So maybe we can start by making sure the
>>>>>>>> DS
>>>>>>>>>>>> types "match"
>>>>>>>>>>>>>>>> when using
>>>>>>>>>>>>>>>>> the same origin:
>>>>>>>>>>>>>>>>> HTTP + DNS: possibly good, if they have
>>>>>>>>>>>> different routing
>>>>>>>>>>>>>>>> names?
>>>>>>>>>>>>>>>>> HTTP_LIVE + HTTP_LIVE_NATNL: bad
>>>>>>>>>>>>>>>>> HTTP_NO_CACHE + [any other type]: bad
>>>>>>>>>>>>>>>>> HTTP_LIVE + HTTP: bad
>>>>>>>>>>>>>>>>> etc.
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> There are most likely other conflict
>>>>>>>>> scenarios
>>>>>>>>>>>> that don't
>>>>>>>>>>>>>>>> involve the
>>>>>>>>>>>>>>>>> DS types, but I think this would be a
>>>>>>>> good
>>>>>>>>>>>> start. In the
>>>>>>>>>>>>>>>> future with
>>>>>>>>>>>>>>>>> Delivery Service Topologies (aka Flexible
>>>>>>>>>>>> Cachegroups aka
>>>>>>>>>>>>>>>> Bring Your
>>>>>>>>>>>>>>>>> Own Topology), we might be able to
>>>>>>>> prohibit
>>>>>>>>>>>> assigning a DS
>>>>>>>>>>>>>>>> to a
>>>>>>>>>>>>>>>>> Topology if the DS's origin is already
>>>>>>>>> used by
>>>>>>>>>>>> another DS
>>>>>>>>>>>>>>> in
>>>>>>>>>>>>>>>> a
>>>>>>>>>>>>>>>>> different Topology.
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> - Rawlin
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> On Mon, Dec 17, 2018 at 10:52 AM Fieck,
>>>>>>>>> Brennan
>>>>>>>>>>>>>>>>> <Br...@comcast.com> wrote:
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> As some of you may be aware,
>>>>>>>>> `parent.config`
>>>>>>>>>>>> files
>>>>>>>>>>>>>>>> generated by Traffic Ops can vary wildly when an origin
>>>>>>>>> is
>>>>>>>>>>>> assigned to
>>>>>>>>>>>>>>>> multiple Delivery Services. This results in undefined
>>>>>>>>>>>> behavior. I'm told
>>>>>>>>>>>>>>>> that the conflict only happens when two Delivery
>>>>>>>> Services
>>>>>>>>>>> with
>>>>>>>>>>>> different
>>>>>>>>>>>>>>>> "topology requirements" use the same origin, whatever
>>>>>>>>> that
>>>>>>>>>>>> means (content
>>>>>>>>>>>>>>>> routing type?). Regardless, the issue should be
>>>>>>>>> addressed.
>>>>>>>>>>> The
>>>>>>>>>>>> obvious
>>>>>>>>>>>>>>>> solution is to put in place a database constraint that
>>>>>>>>>>>> prevents an origin
>>>>>>>>>>>>>>>> from being assigned to more that one Delivery Service
>>>>>>>>> with
>>>>>>>>>>> API
>>>>>>>>>>>> checks in
>>>>>>>>>>>>>>>> place that would provide helpful error messages when an
>>>>>>>>>>>> attempt is made
>>>>>>>>>>>>>>> to
>>>>>>>>>>>>>>>> violate the constraint. However, would that mess with
>>>>>>>>> things
>>>>>>>>>>>> like
>>>>>>>>>>>>>>>> Multi-Site Origin? Or is it just not viable for some
>>>>>>>>> other
>>>>>>>>>>>> reason? If it
>>>>>>>>>>>>>>> is
>>>>>>>>>>>>>>>> a good solution, I'm prepared to work on a fix that
>>>>>>>>> utilizes
>>>>>>>>>>>> it.
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>
>>>>>>>>
>>>>>>>
>>>>>>>
>>>>
>>>>
>>>>
>>>
>