You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@trafficcontrol.apache.org by Oren Shemesh <or...@qwilt.com> on 2017/06/01 07:14:47 UTC
Re: Drop Server -> Delivery Service assignments?
Assuming that this suggestion implies that a DS can be assigned to a cache
set, I believe it is a good, scalable way to handle the issue of server
assignments.
Cache sets are not mutually exclusive, to you can create different sets
according to different needs, and expose only the sets (Not the individual
caches) to the people assigning DSes to cache sets.
e.g. you can have east-coast/west-coast sets, together with having
all/large/medium/small sets, together with having a set for 'caches that
sit in peering points of type X'.
+1 !
On Wed, May 31, 2017 at 12:34 AM, Durfey, Ryan <Ry...@comcast.com>
wrote:
> I want to lobby for a grouping layer for cache assignments. Just like we
> want to group users and services into Tenant groups I think it makes sense
> to group caches into “Cache Sets” (I am avoiding the words “cache group”).
> Assigning individual caches to services is messy and prone to errors
> especially when you add new caches to the network. You have to figure out
> which services it should be assigned and you will end up with really
> inconsistent assignments. “Cache Sets” wouldn’t need to be mutually
> exclusive. Below are some example use cases. This way when a cache gets
> added to the network, it goes into the “All” set and is auto-assigned
> consistently to all services using this set.
>
>
>
> Delivery Service >> “Cache Set” >> Servers
>
>
>
> Examples Cache Sets
>
> - All (default)
> - Small Soak (1-4 test caches)
> - Med Soak 1 cache in each site
> - Large Soak 2 caches in each site
> - Site Set (all caches in a site)
>
>
>
>
>
> *Ryan Durfey* M | 303-524-5099 <(303)%20524-5099>
>
> CDN Support (24x7): 866-405-2993 <(866)%20405-2993> or
> cdn_support@comcast.com
>
>
>
>
>
> *From: *Dave Neuman <ne...@apache.org>
> *Reply-To: *"users@trafficcontrol.incubator.apache.org" <
> users@trafficcontrol.incubator.apache.org>
> *Date: *Thursday, May 25, 2017 at 7:55 AM
> *To: *"users@trafficcontrol.incubator.apache.org" <users@trafficcontrol.
> incubator.apache.org>
> *Subject: *Re: Drop Server -> Delivery Service assignments?
>
>
>
> Sure, the clone server DS assignment should be in the API too, all of the
> UI functionality should be...
>
> I figured the action of assigning a server to a DS (whether it is one or
> all) would be its own API, we shouldn't tightly couple functionality like
> that.
>
>
>
> On Thu, May 25, 2017 at 5:43 AM, Nir Sopher <ni...@qwilt.com> wrote:
>
> Hi Dave,
>
>
>
> As far as I can see, the "clone server's DS assignment" flow is supported
> only in the UI and does not have a dedicate API endpoint. It should
> probably be added to support the suggested flow.
>
>
>
> May I suggest that in a similar manner, the API we will have a separate
> endpoint assigning all servers to the DS - so the CRUD of adding a DS would
> not be the one to actually write to the "DS/Server" table?
>
> The UI can just hide the separation and call both APIs when adding a DS.
>
>
>
> I believe this will leave the CRUD clearer and will not create a situation
> that the order of 2 CRUD operations on unrelated tables effects the system
> behavior: Take the below sequences for example, after sequence "A", "DS1"
> is deployed on "Server1". This is not the case after sequence "B"
>
>
>
> Sequence A:
>
> POST api/1.2/delivery-service => creating DS1
>
> POST api/1.2/server => creating Server1
>
>
>
> Sequence B:
>
> POST api/1.2/server => creating Server1
>
> POST api/1.2/delivery-service => creating DS1
>
>
>
> Nir
>
>
>
> On Wed, May 24, 2017 at 11:43 PM, Shmulik Asafi <sh...@qwilt.com>
> wrote:
>
> There are still 3 phases, even for the first version of the DS: (1) The DS
> owner defines the DS, (2) the DS owner asks for it to be deployed, (3) the
> CDN operator deploys it.
>
>
>
> So the automatic assignment itself is part of (3)
>
>
>
>
>
> On 24 May 2017 11:37 p.m., "Dave Neuman" <ne...@apache.org> wrote:
>
> This is all part of Step 1. This is the first time a Delivery service is
> created so there are no other versions of it...
>
>
>
> On Wed, May 24, 2017 at 2:33 PM, Shmulik Asafi <sh...@qwilt.com> wrote:
>
> Hi,
>
> Referencing back to Nir's slides in the summit on Self Service, there are
> three main phases to the flow relevant here:
>
> 1. The DS-owner user defines/modifies the DS (using configuration
> versions)
> 2. The DS-owner user declares which DS versions she wants to be
> deployed
> 3. The CDN-operator user/robot configures what DS versions to assign
> to which servers
>
> If we decide to automatically assign DS to all caches by default, in the
> Self Service flow it is part of phase 3.
>
>
>
> If we want to give the DS-owner an opt-out from automatic assignment to
> all caches; it should be a property of whatever tables are used to manage
> phase 2.
>
>
>
> In my view, an immediate implementation of this logic should not affect
> the DS table (which in Self Service becomes associated with "phase 1");
> only the assignment table (which in Self Service becomes associated with
> "phase 3").
>
>
>
> As long as the boundaries are mentally clear and the code is written in
> such a way that respects these future boundaries, I dont believe there's
> much risk in features like this to Self Service.
>
> Hope that made sense
>
> Nir - would you agree?
>
>
>
> On Wed, May 24, 2017 at 10:56 PM, Dave Neuman <ne...@apache.org> wrote:
>
> In the system today when you create a server you have the ability to clone
> the delivery service assignments from another cache or manually update the
> delivery service -> server mapping by going into individual DSs and
> assigning the cache to them. I don't think that needs to change with what
> Jeremy is proposing.
>
> I also don't think we should put a special value in the
> deliveryservice_server table that means "all caches". If you want to assign
> all caches to a DS put all of them in the table. If you add a new cache
> and want your DSs on there, clone the mappings from an existing cache.
>
>
>
>
>
> On Wed, May 24, 2017 at 1:43 PM, Nir Sopher <ni...@qwilt.com> wrote:
>
> Hi Jeremy,
>
>
>
> I’m ok with automatically assigning all edges to a new DS. The aspect of
> "self-service" can be dealt with in a different manner.
>
> However, how would you solve the problem when a new server is created? How
> would you know which DS it should be automatically be assigned?
>
> The model has no configuration marking the DS as one that should be
> deployed on all servers.
>
> My suggestion was to model it with "undef" value in the DS/Servers table.
>
> DS | Server
>
> ------+--------
>
> ds1 | server1 => Allow the CDN owner to deploy the ds on a specific
> cache
>
> ------+--------
>
> ds2 | *undef *=> Allow the CDN owner to deploy the ds on all caches
>
>
>
> This basically means that every place reading the DS/Server table, will
> have to use a wrapper, unfolding a the "server=undef" raw, to a raw per
> relevant server in the CDN.
>
> I'm not sure how much effort it requires.
>
>
>
> Nir
>
>
>
> On Wed, May 24, 2017 at 5:21 PM, Jeremy Mitchell <mi...@gmail.com>
> wrote:
>
> Nir,
>
>
>
> Thinking about self-service, do you envision the "DS owner" creates the DS
> and the "CDN owner" assigns caches to the DS? This seems like an
> unnecessary step to me and one that negates the idea of "self service".
>
>
>
> As the "DS owner", I want to create a DS and by default, employ ALL caches
> in my cdn. To be honest, in 99% of cases I really don't want to be bothered
> with assigning caches. I want the full power of the CDN behind my DS.
>
>
>
> So, I'm suggesting that when a new DS is created thru the API, ALL edge
> caches in the cdn are assigned to that DS.....and then, the CDN owner (or
> maybe DS owner) can remove caches from that DS if they need to.
>
>
>
> Jeremy
>
>
>
> On Tue, May 23, 2017 at 1:08 AM, Nir Sopher <ni...@qwilt.com> wrote:
>
> @Jeremy
>
>
>
> The issue I was trying to point at, is that the "DS-Server" table is the
> one to provide the separation between the DS owner domain and the CDN owner
> domain (tenancy wise).
>
> As I see it, when considering multi tenancy, the DS has its owning tenant
> which is in charge of the specific record in the DS table, and specifically
> on adding the DS. The CDN is usually owned by another tenant that controls
> the deployment - currently by adding the ds/servers record to the DS/Server
> table.
>
>
>
> If the DS/Server table is dismissed or automatically written as a result
> of a ds addition, it removes the "server assignment" step and the CDN owner
> loses the control over the deployment decision.
>
>
>
> Before I open a discussion about how to solve this new tenancy issue, I
> would like to verify it is a real issue and there is not something I'm
> missing there.
>
> What do you think?
>
>
>
> Nir
>
>
>
> On May 22, 2017 6:13 PM, "Jeremy Mitchell" <mi...@gmail.com> wrote:
>
> @Nir - maybe your proposal is something to consider more long-term but for
> the short-term i'm wondering if one of the following can / should be
> implemented:
>
>
>
> A) creating a DS thru the API (POST /api/*/deliveryservices) can have a
> default effect of automatically assigning all edges to the new DS (via the deliveryservice_server
> table)
>
>
>
> or
>
>
>
> B) the deliveryservice_server table simple goes away and it becomes
> implied that a DS employs all edge caches
>
>
>
> "A" would probably be the easiest to be honest.
>
>
>
> jeremy
>
>
>
> On Sat, May 20, 2017 at 12:34 PM, Nir Sopher <ni...@qwilt.com> wrote:
>
> +1 for having a configuration allowing to apply the DS to the entire CDN.
>
>
>
> Note that adding a "deploy by default" bool field to the DS itself may
> cause tenancy issues, as I believe this field should be in the control of
> the "CDN owner" tenant, and not of the "DS owner".
>
>
>
> What about keeping the ds_server table, and allow "undef" for the server
> value - meaning "all servers in the CDN"?
>
> In the future the table may grow and have additional fields to white-list
> servers by.
>
> For example:
>
>
>
> DS | Server | cache-group | profile
>
> ------+--------+-------------+--------
>
> ds1 | s1 | null | null => ds to be applied on s1
>
> ------+--------+-------------+--------
>
> ds2 | null | null | null => ds to be applied on all
> servers (in ds CDN)
>
> ------+--------+-------------+--------
>
> ds3 | null | group1 | null => ds to be applied on all
> servers in cache group "group1"
>
> ------+--------+-------------+--------
>
> ds4 | null | group1 | ATS-6.2 => ds to be applied on all
> servers in cache group "group1", with profile "ATS-6.2"
>
>
>
>
>
> During CR config snapshot and configuration pull (or in advance), the
> required calculation would be done, to provide the list of relevant DSs per
> server.
>
>
>
> Also note that moving in this direction is backward compatible as well as
> supports other non trivial extensions one may want in the future: Patterns
> (wildcards/regexs) support, as well as black list, using a "policy table"
> mechanism.
>
>
>
> In the below example "ds5" is to be deployed on all caches, except for
> caches with "ATS-5" profile
>
>
>
> DS | Server | cache-group | profile | deploy-decision
>
> ------+--------+-------------+---------+----------------
>
> ds5 | null | null | ATS-5 | false
>
> ------+--------+-------------+---------+----------------
>
> ds5 | null | null | null | true
>
> ------+--------+-------------+---------+----------------
>
>
>
>
>
> Nir
>
>
>
> On Sat, May 20, 2017 at 1:42 AM, Jeremy Mitchell <mi...@gmail.com>
> wrote:
>
> @dave - i'm not sure if you're suggesting:
>
>
>
> A) on DS create, by default create an entry in the deliveryservice_server
> table for each cdn edge cache. so if there are 200 cdn edge caches, you end
> up with 200 entries in the deliveryservice_server table for that DS.
>
> B) completely kill the deliveryservice_server table and it becomes implied
> that a DS uses ALL cdn edge caches (the way mids work today. mids don't
> have to be explicitly assigned to a DS)
>
>
>
> "A" would allow you to control the scope of caches by using initial
> dispersion OR control the scope of caches by simply removing the ones you
> don't want to receive traffic.
>
> "B" would only allow you to control the scope of caches by using initial
> dispersion (or so i think).
>
>
>
> jeremy
>
>
>
>
>
>
>
> On Fri, May 19, 2017 at 4:25 PM, Dave Neuman <ne...@apache.org> wrote:
>
> +1, I think assigning DSs to all caches by default is fine. Then we can
> use initial dispersion to make sure traffic is spread between an
> appropriate number of caches.
>
>
>
> On Fri, May 19, 2017 at 4:09 PM, Jeremy Mitchell <mi...@gmail.com>
> wrote:
>
> Bringing up an old email regarding DS to cache assignments.
>
>
>
> It sounds like we want to preserve DS/cache assignments (the
> deliveryservice_server table) so you can assign a subset of cdn edge caches
> to a DS if you want. However, what if we make the assignment of ALL cdn
> edge caches to a new DS the default?
>
>
>
> So when you create a DS for the Foo cdn thru the API, your new DS is
> automagically assigned all the edge caches in the Foo cdn...
>
>
>
> (at that point you could always remove caches from a DS)
>
>
>
> ...or the create DS API could have a boolean flag added to it. for
> example, assignAllCaches: true|false. if you pass in true, all caches are
> assigned to the DS, if false, none are.
>
>
>
> thoughts? I'm trying to think of self-service and i'm guessing most people
> will not want to assign caches to a DS...
>
>
>
> jeremy
>
>
>
> On Fri, Jan 6, 2017 at 11:46 AM, Jeremy Mitchell <mi...@gmail.com>
> wrote:
>
> gotcha. makes sense.
>
>
>
> On Fri, Jan 6, 2017 at 9:40 AM, David Neuman <da...@gmail.com>
> wrote:
>
> I believe Eric had a use case where he wanted to assign certain delivery
> services to certain caches.
>
> If we just use initial dispersion and max DNS answers then we don't have
> the ability to control which servers are used, just how many.
>
>
>
>
>
>
>
> On Fri, Jan 6, 2017 at 9:23 AM, Jeremy Mitchell <mi...@gmail.com>
> wrote:
>
> So what I heard was that potentially we move in this direction:
>
>
>
> 1. No need to explicitly assign edge caches to a delivery service. a
> delivery service (by default) would employ ALL edge caches of the cdn in
> which the ds belongs to.
>
> 2. Ability to override this default behavior and explicitly assign
> specific edge caches to a delivery service (this is what we do today)
>
>
>
> But earlier Mark said "so the same functionality [as #2] can essentially
> be reached with assigning all caches, and setting the config params
> properly [initial dispersion and max dns answers]"
>
>
>
> so why not ditch #2 in favor of using initial dispersion and max dns
> answers to control cache assignments?
>
>
>
> it would be nice imo to get rid of the ds_server table. i'm always in
> favor of reducing complexity... :)
>
>
>
> jeremy
>
>
>
>
>
>
>
>
>
>
>
> On Thu, Jan 5, 2017 at 9:08 AM, Eric Friedrich (efriedri) <
> efriedri@cisco.com> wrote:
>
> Yes - I think its a reasonable default to have a delivery service assigned
> to all caches as long as we still keep the ability to assign some DS’s to
> just a subset of caches.
>
>
>
> —Eric
>
>
>
> On Jan 4, 2017, at 4:56 PM, Mark Torluemke <mt...@apache.org> wrote:
>
>
>
> Good feedback, thanks. I think having _zero_ caches assigned to a delivery
> service could be added fairly easily, and could function as a good default
> to the assumption of _all_ caches being assigned. Would you agree with
> that?
>
>
>
> Thanks,
>
> Mark
>
>
>
> On Wed, Jan 4, 2017 at 1:55 PM, Eric Friedrich (efriedri) <
> efriedri@cisco.com> wrote:
>
> Yeah,
>
> We have several some use cases where where cache or CG level allocation
> would be helpful.
>
>
>
> 1) Assigning new or trial delivery services to a subset of caches
> (considered beta cache nodes). This is helpful in trying out new DS configs
> before rolling them out to production.
>
>
>
> 2) Assigning DSs to caches can be a form of quota or resource reservation
> on the delivery service. Providers of content to the CDN can receive tiered
> service by either sharing caches or using dedicated caches.
>
>
>
> —Eric
>
>
>
>
>
> On Jan 4, 2017, at 2:04 PM, Mark Torluemke <mt...@apache.org> wrote:
>
>
>
> Today, there isn't an actual link from Cache Group to DS -- I think what
> you are referring to is all done in the app layer to make a nice summary in
> the UI to allow operators to very quickly assign all caches in a cache
> group.
>
> Do you use that "feature" to assign caches from some cache groups, and not
> others?
>
>
>
> On Tue, Jan 3, 2017 at 9:50 AM, Eric Friedrich (efriedri) <
> efriedri@cisco.com> wrote:
>
> Would we keep the Cache Group -> DS assignment?
>
> —Eric
>
>
> > On Jan 3, 2017, at 9:22 AM, Mark Torluemke <mt...@apache.org>
> wrote:
> >
> > Seeking community opinion on this.
> >
> > Should we drop the server -> delivery service assignments completely?
> >
> > When the project began, selectively assigning caches was the only way to
> selectively route traffic within a cache group -- let's say you have a
> DNS-routed service with a relatively large library size, you might not want
> assign all caches in a cache group to that service for fear you would
> 'pollute' those caches. Or, if you were turning up a 'test' DS of which you
> wanted to limit exposure on, you might apply the same logic. In recent
> releases, Traffic Control supports "initial dispersion" for _both_ DNS and
> HTTP-routed services, and "max DNS answers" for DNS-routed services, so the
> same functionality can essentially be reached with assigning all caches,
> and setting the config params properly.
> >
> > The motivation for this is to reduce the size & complexity of the
> Traffic Router's config (aka CRConfig), as this config scales mostly
> linearly with the number of caches and the number of delivery services in
> your CDN.
> >
> > Another option might be to not assign caches as the default, and
> individual cache assignments is available, but used only when necessary.
> >
> > Other thoughts? Clarifications needed?
> >
> > Thanks,
> > Mark
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
> --
>
> *Shmulik Asafi*
> Qwilt | Work: +972-72-2221692 <+972%2072-222-1692>| Mobile:
> +972-54-6581595 <+972%2054-658-1595>| shmulika@qwilt.com <yo...@qwilt.com>
>
>
>
>
>
>
>
--
*Oren Shemesh*
Qwilt | Work: +972-72-2221637| Mobile: +972-50-2281168 | orens@qwilt.com
<yo...@qwilt.com>
Re:
Posted by Robert Butts <ro...@apache.org>.
Erm, sorry for the (no subject), using the lists.apache.org website failed.
I replied to the real thread, please reply there, not here.
On Wed, Jan 2, 2019 at 11:06 AM Robert O Butts <ro...@apache.org> wrote:
> Reviving this, because I noticed a few things, when working on the DS
> Snapshots PR:
>
> (1) I noticed the DeliveryService-Servers query is _by far_ the slowest
> query we have, building the CRConfig. For a given machine and a given large
> CDN, selecting all DS data takes about ~10ms; the DSS query takes ~200ms.
> This is to trivially select the data, with indexes. If you ever need a
> nontrivial query with nontrivial joins, it quickly becomes seconds.
>
> (2) Writing code, I accidentally generated a CRConfig without
> DeliveryService-Servers. It was ~0.5mb (uncompressed). Compared to ~10mb
> for the same CDN with DSSes.
>
> These sizes and query times are because the DSSes are a cross product of
> DS and Servers.
>
> I think we need to do this, and not only in the interface, but implement
> it such that we (1) don't actually store the DSS mappings, for DSes and
> Servers which are "default-assigned", and (2) don't put them in the
> CRConfig. In terms of performance, I strongly suspect it will be faster for
> the Router to compute them, than the time it takes it to receive all that
> data over the network.
>
> We keep talking about "breaking up the huge CRConfig" - it looks to me
> like removing DSSes will make it so small we don't _need_ to break it up
> anytime soon.
>
> To @mtorluemke 's original point, doing this will significantly improve
> scalability, completely aside from the Self Service and interface benefits.
>
> It sounds like there was consensus here. We just need to do it. I'm
> suggesting, with the evidence above, this really ought to be a higher
> priority. The evidence suggests fixing this at the data level will
> significantly improve the scalability of Traffic Control.
>
> On 2017/06/01 07:14:47, Oren Shemesh <or...@qwilt.com> wrote:
> > Assuming that this suggestion implies that a DS can be assigned to a
> cache
> > set, I believe it is a good, scalable way to handle the issue of server
> > assignments.
> > Cache sets are not mutually exclusive, to you can create different sets
> > according to different needs, and expose only the sets (Not the
> individual
> > caches) to the people assigning DSes to cache sets.
> > e.g. you can have east-coast/west-coast sets, together with having
> > all/large/medium/small sets, together with having a set for 'caches that
> > sit in peering points of type X'.
> >
> > +1 !
> >
> > On Wed, May 31, 2017 at 12:34 AM, Durfey, Ryan <Ry...@comcast.com>
> > wrote:
> >
> > > I want to lobby for a grouping layer for cache assignments. Just like
> we
> > > want to group users and services into Tenant groups I think it makes
> sense
> > > to group caches into “Cache Sets” (I am avoiding the words “cache
> group”).
> > > Assigning individual caches to services is messy and prone to errors
> > > especially when you add new caches to the network. You have to figure
> out
> > > which services it should be assigned and you will end up with really
> > > inconsistent assignments. “Cache Sets” wouldn’t need to be mutually
> > > exclusive. Below are some example use cases. This way when a cache
> gets
> > > added to the network, it goes into the “All” set and is auto-assigned
> > > consistently to all services using this set.
> > >
> > >
> > >
> > > Delivery Service >> “Cache Set” >> Servers
> > >
> > >
> > >
> > > Examples Cache Sets
> > >
> > > - All (default)
> > > - Small Soak (1-4 test caches)
> > > - Med Soak 1 cache in each site
> > > - Large Soak 2 caches in each site
> > > - Site Set (all caches in a site)
> > >
> > >
> > >
> > >
> > >
> > > *Ryan Durfey* M | 303-524-5099 <(303)%20524-5099>
> > >
> > > CDN Support (24x7): 866-405-2993 <(866)%20405-2993> or
> > > cdn_support@comcast.com
> > >
> > >
> > >
> > >
> > >
> > > *From: *Dave Neuman <ne...@apache.org>
> > > *Reply-To: *"users@trafficcontrol.incubator.apache.org" <
> > > users@trafficcontrol.incubator.apache.org>
> > > *Date: *Thursday, May 25, 2017 at 7:55 AM
> > > *To: *"users@trafficcontrol.incubator.apache.org"
> <users@trafficcontrol.
> > > incubator.apache.org>
> > > *Subject: *Re: Drop Server -> Delivery Service assignments?
> > >
> > >
> > >
> > > Sure, the clone server DS assignment should be in the API too, all of
> the
> > > UI functionality should be...
> > >
> > > I figured the action of assigning a server to a DS (whether it is one
> or
> > > all) would be its own API, we shouldn't tightly couple functionality
> like
> > > that.
> > >
> > >
> > >
> > > On Thu, May 25, 2017 at 5:43 AM, Nir Sopher <ni...@qwilt.com> wrote:
> > >
> > > Hi Dave,
> > >
> > >
> > >
> > > As far as I can see, the "clone server's DS assignment" flow is
> supported
> > > only in the UI and does not have a dedicate API endpoint. It should
> > > probably be added to support the suggested flow.
> > >
> > >
> > >
> > > May I suggest that in a similar manner, the API we will have a separate
> > > endpoint assigning all servers to the DS - so the CRUD of adding a DS
> would
> > > not be the one to actually write to the "DS/Server" table?
> > >
> > > The UI can just hide the separation and call both APIs when adding a
> DS.
> > >
> > >
> > >
> > > I believe this will leave the CRUD clearer and will not create a
> situation
> > > that the order of 2 CRUD operations on unrelated tables effects the
> system
> > > behavior: Take the below sequences for example, after sequence "A",
> "DS1"
> > > is deployed on "Server1". This is not the case after sequence "B"
> > >
> > >
> > >
> > > Sequence A:
> > >
> > > POST api/1.2/delivery-service => creating DS1
> > >
> > > POST api/1.2/server => creating Server1
> > >
> > >
> > >
> > > Sequence B:
> > >
> > > POST api/1.2/server => creating Server1
> > >
> > > POST api/1.2/delivery-service => creating DS1
> > >
> > >
> > >
> > > Nir
> > >
> > >
> > >
> > > On Wed, May 24, 2017 at 11:43 PM, Shmulik Asafi <sh...@qwilt.com>
> > > wrote:
> > >
> > > There are still 3 phases, even for the first version of the DS: (1)
> The DS
> > > owner defines the DS, (2) the DS owner asks for it to be deployed, (3)
> the
> > > CDN operator deploys it.
> > >
> > >
> > >
> > > So the automatic assignment itself is part of (3)
> > >
> > >
> > >
> > >
> > >
> > > On 24 May 2017 11:37 p.m., "Dave Neuman" <ne...@apache.org> wrote:
> > >
> > > This is all part of Step 1. This is the first time a Delivery service
> is
> > > created so there are no other versions of it...
> > >
> > >
> > >
> > > On Wed, May 24, 2017 at 2:33 PM, Shmulik Asafi <sh...@qwilt.com>
> wrote:
> > >
> > > Hi,
> > >
> > > Referencing back to Nir's slides in the summit on Self Service, there
> are
> > > three main phases to the flow relevant here:
> > >
> > > 1. The DS-owner user defines/modifies the DS (using configuration
> > > versions)
> > > 2. The DS-owner user declares which DS versions she wants to be
> > > deployed
> > > 3. The CDN-operator user/robot configures what DS versions to assign
> > > to which servers
> > >
> > > If we decide to automatically assign DS to all caches by default, in
> the
> > > Self Service flow it is part of phase 3.
> > >
> > >
> > >
> > > If we want to give the DS-owner an opt-out from automatic assignment to
> > > all caches; it should be a property of whatever tables are used to
> manage
> > > phase 2.
> > >
> > >
> > >
> > > In my view, an immediate implementation of this logic should not affect
> > > the DS table (which in Self Service becomes associated with "phase 1");
> > > only the assignment table (which in Self Service becomes associated
> with
> > > "phase 3").
> > >
> > >
> > >
> > > As long as the boundaries are mentally clear and the code is written in
> > > such a way that respects these future boundaries, I dont believe
> there's
> > > much risk in features like this to Self Service.
> > >
> > > Hope that made sense
> > >
> > > Nir - would you agree?
> > >
> > >
> > >
> > > On Wed, May 24, 2017 at 10:56 PM, Dave Neuman <ne...@apache.org>
> wrote:
> > >
> > > In the system today when you create a server you have the ability to
> clone
> > > the delivery service assignments from another cache or manually update
> the
> > > delivery service -> server mapping by going into individual DSs and
> > > assigning the cache to them. I don't think that needs to change with
> what
> > > Jeremy is proposing.
> > >
> > > I also don't think we should put a special value in the
> > > deliveryservice_server table that means "all caches". If you want to
> assign
> > > all caches to a DS put all of them in the table. If you add a new
> cache
> > > and want your DSs on there, clone the mappings from an existing cache.
> > >
> > >
> > >
> > >
> > >
> > > On Wed, May 24, 2017 at 1:43 PM, Nir Sopher <ni...@qwilt.com> wrote:
> > >
> > > Hi Jeremy,
> > >
> > >
> > >
> > > I’m ok with automatically assigning all edges to a new DS. The aspect
> of
> > > "self-service" can be dealt with in a different manner.
> > >
> > > However, how would you solve the problem when a new server is created?
> How
> > > would you know which DS it should be automatically be assigned?
> > >
> > > The model has no configuration marking the DS as one that should be
> > > deployed on all servers.
> > >
> > > My suggestion was to model it with "undef" value in the DS/Servers
> table.
> > >
> > > DS | Server
> > >
> > > ------+--------
> > >
> > > ds1 | server1 => Allow the CDN owner to deploy the ds on a specific
> > > cache
> > >
> > > ------+--------
> > >
> > > ds2 | *undef *=> Allow the CDN owner to deploy the ds on all caches
> > >
> > >
> > >
> > > This basically means that every place reading the DS/Server table, will
> > > have to use a wrapper, unfolding a the "server=undef" raw, to a raw per
> > > relevant server in the CDN.
> > >
> > > I'm not sure how much effort it requires.
> > >
> > >
> > >
> > > Nir
> > >
> > >
> > >
> > > On Wed, May 24, 2017 at 5:21 PM, Jeremy Mitchell <
> mitchell852@gmail.com>
> > > wrote:
> > >
> > > Nir,
> > >
> > >
> > >
> > > Thinking about self-service, do you envision the "DS owner" creates
> the DS
> > > and the "CDN owner" assigns caches to the DS? This seems like an
> > > unnecessary step to me and one that negates the idea of "self service".
> > >
> > >
> > >
> > > As the "DS owner", I want to create a DS and by default, employ ALL
> caches
> > > in my cdn. To be honest, in 99% of cases I really don't want to be
> bothered
> > > with assigning caches. I want the full power of the CDN behind my DS.
> > >
> > >
> > >
> > > So, I'm suggesting that when a new DS is created thru the API, ALL edge
> > > caches in the cdn are assigned to that DS.....and then, the CDN owner
> (or
> > > maybe DS owner) can remove caches from that DS if they need to.
> > >
> > >
> > >
> > > Jeremy
> > >
> > >
> > >
> > > On Tue, May 23, 2017 at 1:08 AM, Nir Sopher <ni...@qwilt.com> wrote:
> > >
> > > @Jeremy
> > >
> > >
> > >
> > > The issue I was trying to point at, is that the "DS-Server" table is
> the
> > > one to provide the separation between the DS owner domain and the CDN
> owner
> > > domain (tenancy wise).
> > >
> > > As I see it, when considering multi tenancy, the DS has its owning
> tenant
> > > which is in charge of the specific record in the DS table, and
> specifically
> > > on adding the DS. The CDN is usually owned by another tenant that
> controls
> > > the deployment - currently by adding the ds/servers record to the
> DS/Server
> > > table.
> > >
> > >
> > >
> > > If the DS/Server table is dismissed or automatically written as a
> result
> > > of a ds addition, it removes the "server assignment" step and the CDN
> owner
> > > loses the control over the deployment decision.
> > >
> > >
> > >
> > > Before I open a discussion about how to solve this new tenancy issue, I
> > > would like to verify it is a real issue and there is not something I'm
> > > missing there.
> > >
> > > What do you think?
> > >
> > >
> > >
> > > Nir
> > >
> > >
> > >
> > > On May 22, 2017 6:13 PM, "Jeremy Mitchell" <mi...@gmail.com>
> wrote:
> > >
> > > @Nir - maybe your proposal is something to consider more long-term but
> for
> > > the short-term i'm wondering if one of the following can / should be
> > > implemented:
> > >
> > >
> > >
> > > A) creating a DS thru the API (POST /api/*/deliveryservices) can have a
> > > default effect of automatically assigning all edges to the new DS (via
> the deliveryservice_server
> > > table)
> > >
> > >
> > >
> > > or
> > >
> > >
> > >
> > > B) the deliveryservice_server table simple goes away and it becomes
> > > implied that a DS employs all edge caches
> > >
> > >
> > >
> > > "A" would probably be the easiest to be honest.
> > >
> > >
> > >
> > > jeremy
> > >
> > >
> > >
> > > On Sat, May 20, 2017 at 12:34 PM, Nir Sopher <ni...@qwilt.com> wrote:
> > >
> > > +1 for having a configuration allowing to apply the DS to the entire
> CDN.
> > >
> > >
> > >
> > > Note that adding a "deploy by default" bool field to the DS itself may
> > > cause tenancy issues, as I believe this field should be in the control
> of
> > > the "CDN owner" tenant, and not of the "DS owner".
> > >
> > >
> > >
> > > What about keeping the ds_server table, and allow "undef" for the
> server
> > > value - meaning "all servers in the CDN"?
> > >
> > > In the future the table may grow and have additional fields to
> white-list
> > > servers by.
> > >
> > > For example:
> > >
> > >
> > >
> > > DS | Server | cache-group | profile
> > >
> > > ------+--------+-------------+--------
> > >
> > > ds1 | s1 | null | null => ds to be applied on s1
> > >
> > > ------+--------+-------------+--------
> > >
> > > ds2 | null | null | null => ds to be applied on all
> > > servers (in ds CDN)
> > >
> > > ------+--------+-------------+--------
> > >
> > > ds3 | null | group1 | null => ds to be applied on all
> > > servers in cache group "group1"
> > >
> > > ------+--------+-------------+--------
> > >
> > > ds4 | null | group1 | ATS-6.2 => ds to be applied on all
> > > servers in cache group "group1", with profile "ATS-6.2"
> > >
> > >
> > >
> > >
> > >
> > > During CR config snapshot and configuration pull (or in advance), the
> > > required calculation would be done, to provide the list of relevant
> DSs per
> > > server.
> > >
> > >
> > >
> > > Also note that moving in this direction is backward compatible as well
> as
> > > supports other non trivial extensions one may want in the future:
> Patterns
> > > (wildcards/regexs) support, as well as black list, using a "policy
> table"
> > > mechanism.
> > >
> > >
> > >
> > > In the below example "ds5" is to be deployed on all caches, except for
> > > caches with "ATS-5" profile
> > >
> > >
> > >
> > > DS | Server | cache-group | profile | deploy-decision
> > >
> > > ------+--------+-------------+---------+----------------
> > >
> > > ds5 | null | null | ATS-5 | false
> > >
> > > ------+--------+-------------+---------+----------------
> > >
> > > ds5 | null | null | null | true
> > >
> > > ------+--------+-------------+---------+----------------
> > >
> > >
> > >
> > >
> > >
> > > Nir
> > >
> > >
> > >
> > > On Sat, May 20, 2017 at 1:42 AM, Jeremy Mitchell <
> mitchell852@gmail.com>
> > > wrote:
> > >
> > > @dave - i'm not sure if you're suggesting:
> > >
> > >
> > >
> > > A) on DS create, by default create an entry in the
> deliveryservice_server
> > > table for each cdn edge cache. so if there are 200 cdn edge caches,
> you end
> > > up with 200 entries in the deliveryservice_server table for that DS.
> > >
> > > B) completely kill the deliveryservice_server table and it becomes
> implied
> > > that a DS uses ALL cdn edge caches (the way mids work today. mids don't
> > > have to be explicitly assigned to a DS)
> > >
> > >
> > >
> > > "A" would allow you to control the scope of caches by using initial
> > > dispersion OR control the scope of caches by simply removing the ones
> you
> > > don't want to receive traffic.
> > >
> > > "B" would only allow you to control the scope of caches by using
> initial
> > > dispersion (or so i think).
> > >
> > >
> > >
> > > jeremy
> > >
> > >
> > >
> > >
> > >
> > >
> > >
> > > On Fri, May 19, 2017 at 4:25 PM, Dave Neuman <ne...@apache.org>
> wrote:
> > >
> > > +1, I think assigning DSs to all caches by default is fine. Then we
> can
> > > use initial dispersion to make sure traffic is spread between an
> > > appropriate number of caches.
> > >
> > >
> > >
> > > On Fri, May 19, 2017 at 4:09 PM, Jeremy Mitchell <
> mitchell852@gmail.com>
> > > wrote:
> > >
> > > Bringing up an old email regarding DS to cache assignments.
> > >
> > >
> > >
> > > It sounds like we want to preserve DS/cache assignments (the
> > > deliveryservice_server table) so you can assign a subset of cdn edge
> caches
> > > to a DS if you want. However, what if we make the assignment of ALL cdn
> > > edge caches to a new DS the default?
> > >
> > >
> > >
> > > So when you create a DS for the Foo cdn thru the API, your new DS is
> > > automagically assigned all the edge caches in the Foo cdn...
> > >
> > >
> > >
> > > (at that point you could always remove caches from a DS)
> > >
> > >
> > >
> > > ...or the create DS API could have a boolean flag added to it. for
> > > example, assignAllCaches: true|false. if you pass in true, all caches
> are
> > > assigned to the DS, if false, none are.
> > >
> > >
> > >
> > > thoughts? I'm trying to think of self-service and i'm guessing most
> people
> > > will not want to assign caches to a DS...
> > >
> > >
> > >
> > > jeremy
> > >
> > >
> > >
> > > On Fri, Jan 6, 2017 at 11:46 AM, Jeremy Mitchell <
> mitchell852@gmail.com>
> > > wrote:
> > >
> > > gotcha. makes sense.
> > >
> > >
> > >
> > > On Fri, Jan 6, 2017 at 9:40 AM, David Neuman <david.neuman64@gmail.com
> >
> > > wrote:
> > >
> > > I believe Eric had a use case where he wanted to assign certain
> delivery
> > > services to certain caches.
> > >
> > > If we just use initial dispersion and max DNS answers then we don't
> have
> > > the ability to control which servers are used, just how many.
> > >
> > >
> > >
> > >
> > >
> > >
> > >
> > > On Fri, Jan 6, 2017 at 9:23 AM, Jeremy Mitchell <mitchell852@gmail.com
> >
> > > wrote:
> > >
> > > So what I heard was that potentially we move in this direction:
> > >
> > >
> > >
> > > 1. No need to explicitly assign edge caches to a delivery service. a
> > > delivery service (by default) would employ ALL edge caches of the cdn
> in
> > > which the ds belongs to.
> > >
> > > 2. Ability to override this default behavior and explicitly assign
> > > specific edge caches to a delivery service (this is what we do today)
> > >
> > >
> > >
> > > But earlier Mark said "so the same functionality [as #2] can
> essentially
> > > be reached with assigning all caches, and setting the config params
> > > properly [initial dispersion and max dns answers]"
> > >
> > >
> > >
> > > so why not ditch #2 in favor of using initial dispersion and max dns
> > > answers to control cache assignments?
> > >
> > >
> > >
> > > it would be nice imo to get rid of the ds_server table. i'm always in
> > > favor of reducing complexity... :)
> > >
> > >
> > >
> > > jeremy
> > >
> > >
> > >
> > >
> > >
> > >
> > >
> > >
> > >
> > >
> > >
> > > On Thu, Jan 5, 2017 at 9:08 AM, Eric Friedrich (efriedri) <
> > > efriedri@cisco.com> wrote:
> > >
> > > Yes - I think its a reasonable default to have a delivery service
> assigned
> > > to all caches as long as we still keep the ability to assign some
> DS’s to
> > > just a subset of caches.
> > >
> > >
> > >
> > > —Eric
> > >
> > >
> > >
> > > On Jan 4, 2017, at 4:56 PM, Mark Torluemke <mt...@apache.org>
> wrote:
> > >
> > >
> > >
> > > Good feedback, thanks. I think having _zero_ caches assigned to a
> delivery
> > > service could be added fairly easily, and could function as a good
> default
> > > to the assumption of _all_ caches being assigned. Would you agree with
> > > that?
> > >
> > >
> > >
> > > Thanks,
> > >
> > > Mark
> > >
> > >
> > >
> > > On Wed, Jan 4, 2017 at 1:55 PM, Eric Friedrich (efriedri) <
> > > efriedri@cisco.com> wrote:
> > >
> > > Yeah,
> > >
> > > We have several some use cases where where cache or CG level
> allocation
> > > would be helpful.
> > >
> > >
> > >
> > > 1) Assigning new or trial delivery services to a subset of caches
> > > (considered beta cache nodes). This is helpful in trying out new DS
> configs
> > > before rolling them out to production.
> > >
> > >
> > >
> > > 2) Assigning DSs to caches can be a form of quota or resource
> reservation
> > > on the delivery service. Providers of content to the CDN can receive
> tiered
> > > service by either sharing caches or using dedicated caches.
> > >
> > >
> > >
> > > —Eric
> > >
> > >
> > >
> > >
> > >
> > > On Jan 4, 2017, at 2:04 PM, Mark Torluemke <mt...@apache.org>
> wrote:
> > >
> > >
> > >
> > > Today, there isn't an actual link from Cache Group to DS -- I think
> what
> > > you are referring to is all done in the app layer to make a nice
> summary in
> > > the UI to allow operators to very quickly assign all caches in a cache
> > > group.
> > >
> > > Do you use that "feature" to assign caches from some cache groups, and
> not
> > > others?
> > >
> > >
> > >
> > > On Tue, Jan 3, 2017 at 9:50 AM, Eric Friedrich (efriedri) <
> > > efriedri@cisco.com> wrote:
> > >
> > > Would we keep the Cache Group -> DS assignment?
> > >
> > > —Eric
> > >
> > >
> > > > On Jan 3, 2017, at 9:22 AM, Mark Torluemke <mt...@apache.org>
> > > wrote:
> > > >
> > > > Seeking community opinion on this.
> > > >
> > > > Should we drop the server -> delivery service assignments completely?
> > > >
> > > > When the project began, selectively assigning caches was the only
> way to
> > > selectively route traffic within a cache group -- let's say you have a
> > > DNS-routed service with a relatively large library size, you might not
> want
> > > assign all caches in a cache group to that service for fear you would
> > > 'pollute' those caches. Or, if you were turning up a 'test' DS of
> which you
> > > wanted to limit exposure on, you might apply the same logic. In recent
> > > releases, Traffic Control supports "initial dispersion" for _both_ DNS
> and
> > > HTTP-routed services, and "max DNS answers" for DNS-routed services,
> so the
> > > same functionality can essentially be reached with assigning all
> caches,
> > > and setting the config params properly.
> > > >
> > > > The motivation for this is to reduce the size & complexity of the
> > > Traffic Router's config (aka CRConfig), as this config scales mostly
> > > linearly with the number of caches and the number of delivery services
> in
> > > your CDN.
> > > >
> > > > Another option might be to not assign caches as the default, and
> > > individual cache assignments is available, but used only when
> necessary.
> > > >
> > > > Other thoughts? Clarifications needed?
> > > >
> > > > Thanks,
> > > > Mark
> > >
> > >
> > >
> > >
> > >
> > >
> > >
> > >
> > >
> > >
> > >
> > >
> > >
> > >
> > >
> > >
> > >
> > >
> > >
> > >
> > >
> > >
> > >
> > >
> > >
> > >
> > >
> > >
> > >
> > >
> > >
> > >
> > >
> > >
> > >
> > > --
> > >
> > > *Shmulik Asafi*
> > > Qwilt | Work: +972-72-2221692 <+972%2072-222-1692>| Mobile:
> > > +972-54-6581595 <+972%2054-658-1595>| shmulika@qwilt.com <
> yoav@qwilt.com>
> > >
> > >
> > >
> > >
> > >
> > >
> > >
> >
> >
> >
> > --
> >
> > *Oren Shemesh*
> > Qwilt | Work: +972-72-2221637| Mobile: +972-50-2281168 | orens@qwilt.com
> > <yo...@qwilt.com>
> >
>
Posted by Robert O Butts <ro...@apache.org>.
Reviving this, because I noticed a few things, when working on the DS Snapshots PR:
(1) I noticed the DeliveryService-Servers query is _by far_ the slowest query we have, building the CRConfig. For a given machine and a given large CDN, selecting all DS data takes about ~10ms; the DSS query takes ~200ms. This is to trivially select the data, with indexes. If you ever need a nontrivial query with nontrivial joins, it quickly becomes seconds.
(2) Writing code, I accidentally generated a CRConfig without DeliveryService-Servers. It was ~0.5mb (uncompressed). Compared to ~10mb for the same CDN with DSSes.
These sizes and query times are because the DSSes are a cross product of DS and Servers.
I think we need to do this, and not only in the interface, but implement it such that we (1) don't actually store the DSS mappings, for DSes and Servers which are "default-assigned", and (2) don't put them in the CRConfig. In terms of performance, I strongly suspect it will be faster for the Router to compute them, than the time it takes it to receive all that data over the network.
We keep talking about "breaking up the huge CRConfig" - it looks to me like removing DSSes will make it so small we don't _need_ to break it up anytime soon.
To @mtorluemke 's original point, doing this will significantly improve scalability, completely aside from the Self Service and interface benefits.
It sounds like there was consensus here. We just need to do it. I'm suggesting, with the evidence above, this really ought to be a higher priority. The evidence suggests fixing this at the data level will significantly improve the scalability of Traffic Control.
On 2017/06/01 07:14:47, Oren Shemesh <or...@qwilt.com> wrote:
> Assuming that this suggestion implies that a DS can be assigned to a cache
> set, I believe it is a good, scalable way to handle the issue of server
> assignments.
> Cache sets are not mutually exclusive, to you can create different sets
> according to different needs, and expose only the sets (Not the individual
> caches) to the people assigning DSes to cache sets.
> e.g. you can have east-coast/west-coast sets, together with having
> all/large/medium/small sets, together with having a set for 'caches that
> sit in peering points of type X'.
>
> +1 !
>
> On Wed, May 31, 2017 at 12:34 AM, Durfey, Ryan <Ry...@comcast.com>
> wrote:
>
> > I want to lobby for a grouping layer for cache assignments. Just like we
> > want to group users and services into Tenant groups I think it makes sense
> > to group caches into “Cache Sets” (I am avoiding the words “cache group”).
> > Assigning individual caches to services is messy and prone to errors
> > especially when you add new caches to the network. You have to figure out
> > which services it should be assigned and you will end up with really
> > inconsistent assignments. “Cache Sets” wouldn’t need to be mutually
> > exclusive. Below are some example use cases. This way when a cache gets
> > added to the network, it goes into the “All” set and is auto-assigned
> > consistently to all services using this set.
> >
> >
> >
> > Delivery Service >> “Cache Set” >> Servers
> >
> >
> >
> > Examples Cache Sets
> >
> > - All (default)
> > - Small Soak (1-4 test caches)
> > - Med Soak 1 cache in each site
> > - Large Soak 2 caches in each site
> > - Site Set (all caches in a site)
> >
> >
> >
> >
> >
> > *Ryan Durfey* M | 303-524-5099 <(303)%20524-5099>
> >
> > CDN Support (24x7): 866-405-2993 <(866)%20405-2993> or
> > cdn_support@comcast.com
> >
> >
> >
> >
> >
> > *From: *Dave Neuman <ne...@apache.org>
> > *Reply-To: *"users@trafficcontrol.incubator.apache.org" <
> > users@trafficcontrol.incubator.apache.org>
> > *Date: *Thursday, May 25, 2017 at 7:55 AM
> > *To: *"users@trafficcontrol.incubator.apache.org" <users@trafficcontrol.
> > incubator.apache.org>
> > *Subject: *Re: Drop Server -> Delivery Service assignments?
> >
> >
> >
> > Sure, the clone server DS assignment should be in the API too, all of the
> > UI functionality should be...
> >
> > I figured the action of assigning a server to a DS (whether it is one or
> > all) would be its own API, we shouldn't tightly couple functionality like
> > that.
> >
> >
> >
> > On Thu, May 25, 2017 at 5:43 AM, Nir Sopher <ni...@qwilt.com> wrote:
> >
> > Hi Dave,
> >
> >
> >
> > As far as I can see, the "clone server's DS assignment" flow is supported
> > only in the UI and does not have a dedicate API endpoint. It should
> > probably be added to support the suggested flow.
> >
> >
> >
> > May I suggest that in a similar manner, the API we will have a separate
> > endpoint assigning all servers to the DS - so the CRUD of adding a DS would
> > not be the one to actually write to the "DS/Server" table?
> >
> > The UI can just hide the separation and call both APIs when adding a DS.
> >
> >
> >
> > I believe this will leave the CRUD clearer and will not create a situation
> > that the order of 2 CRUD operations on unrelated tables effects the system
> > behavior: Take the below sequences for example, after sequence "A", "DS1"
> > is deployed on "Server1". This is not the case after sequence "B"
> >
> >
> >
> > Sequence A:
> >
> > POST api/1.2/delivery-service => creating DS1
> >
> > POST api/1.2/server => creating Server1
> >
> >
> >
> > Sequence B:
> >
> > POST api/1.2/server => creating Server1
> >
> > POST api/1.2/delivery-service => creating DS1
> >
> >
> >
> > Nir
> >
> >
> >
> > On Wed, May 24, 2017 at 11:43 PM, Shmulik Asafi <sh...@qwilt.com>
> > wrote:
> >
> > There are still 3 phases, even for the first version of the DS: (1) The DS
> > owner defines the DS, (2) the DS owner asks for it to be deployed, (3) the
> > CDN operator deploys it.
> >
> >
> >
> > So the automatic assignment itself is part of (3)
> >
> >
> >
> >
> >
> > On 24 May 2017 11:37 p.m., "Dave Neuman" <ne...@apache.org> wrote:
> >
> > This is all part of Step 1. This is the first time a Delivery service is
> > created so there are no other versions of it...
> >
> >
> >
> > On Wed, May 24, 2017 at 2:33 PM, Shmulik Asafi <sh...@qwilt.com> wrote:
> >
> > Hi,
> >
> > Referencing back to Nir's slides in the summit on Self Service, there are
> > three main phases to the flow relevant here:
> >
> > 1. The DS-owner user defines/modifies the DS (using configuration
> > versions)
> > 2. The DS-owner user declares which DS versions she wants to be
> > deployed
> > 3. The CDN-operator user/robot configures what DS versions to assign
> > to which servers
> >
> > If we decide to automatically assign DS to all caches by default, in the
> > Self Service flow it is part of phase 3.
> >
> >
> >
> > If we want to give the DS-owner an opt-out from automatic assignment to
> > all caches; it should be a property of whatever tables are used to manage
> > phase 2.
> >
> >
> >
> > In my view, an immediate implementation of this logic should not affect
> > the DS table (which in Self Service becomes associated with "phase 1");
> > only the assignment table (which in Self Service becomes associated with
> > "phase 3").
> >
> >
> >
> > As long as the boundaries are mentally clear and the code is written in
> > such a way that respects these future boundaries, I dont believe there's
> > much risk in features like this to Self Service.
> >
> > Hope that made sense
> >
> > Nir - would you agree?
> >
> >
> >
> > On Wed, May 24, 2017 at 10:56 PM, Dave Neuman <ne...@apache.org> wrote:
> >
> > In the system today when you create a server you have the ability to clone
> > the delivery service assignments from another cache or manually update the
> > delivery service -> server mapping by going into individual DSs and
> > assigning the cache to them. I don't think that needs to change with what
> > Jeremy is proposing.
> >
> > I also don't think we should put a special value in the
> > deliveryservice_server table that means "all caches". If you want to assign
> > all caches to a DS put all of them in the table. If you add a new cache
> > and want your DSs on there, clone the mappings from an existing cache.
> >
> >
> >
> >
> >
> > On Wed, May 24, 2017 at 1:43 PM, Nir Sopher <ni...@qwilt.com> wrote:
> >
> > Hi Jeremy,
> >
> >
> >
> > I’m ok with automatically assigning all edges to a new DS. The aspect of
> > "self-service" can be dealt with in a different manner.
> >
> > However, how would you solve the problem when a new server is created? How
> > would you know which DS it should be automatically be assigned?
> >
> > The model has no configuration marking the DS as one that should be
> > deployed on all servers.
> >
> > My suggestion was to model it with "undef" value in the DS/Servers table.
> >
> > DS | Server
> >
> > ------+--------
> >
> > ds1 | server1 => Allow the CDN owner to deploy the ds on a specific
> > cache
> >
> > ------+--------
> >
> > ds2 | *undef *=> Allow the CDN owner to deploy the ds on all caches
> >
> >
> >
> > This basically means that every place reading the DS/Server table, will
> > have to use a wrapper, unfolding a the "server=undef" raw, to a raw per
> > relevant server in the CDN.
> >
> > I'm not sure how much effort it requires.
> >
> >
> >
> > Nir
> >
> >
> >
> > On Wed, May 24, 2017 at 5:21 PM, Jeremy Mitchell <mi...@gmail.com>
> > wrote:
> >
> > Nir,
> >
> >
> >
> > Thinking about self-service, do you envision the "DS owner" creates the DS
> > and the "CDN owner" assigns caches to the DS? This seems like an
> > unnecessary step to me and one that negates the idea of "self service".
> >
> >
> >
> > As the "DS owner", I want to create a DS and by default, employ ALL caches
> > in my cdn. To be honest, in 99% of cases I really don't want to be bothered
> > with assigning caches. I want the full power of the CDN behind my DS.
> >
> >
> >
> > So, I'm suggesting that when a new DS is created thru the API, ALL edge
> > caches in the cdn are assigned to that DS.....and then, the CDN owner (or
> > maybe DS owner) can remove caches from that DS if they need to.
> >
> >
> >
> > Jeremy
> >
> >
> >
> > On Tue, May 23, 2017 at 1:08 AM, Nir Sopher <ni...@qwilt.com> wrote:
> >
> > @Jeremy
> >
> >
> >
> > The issue I was trying to point at, is that the "DS-Server" table is the
> > one to provide the separation between the DS owner domain and the CDN owner
> > domain (tenancy wise).
> >
> > As I see it, when considering multi tenancy, the DS has its owning tenant
> > which is in charge of the specific record in the DS table, and specifically
> > on adding the DS. The CDN is usually owned by another tenant that controls
> > the deployment - currently by adding the ds/servers record to the DS/Server
> > table.
> >
> >
> >
> > If the DS/Server table is dismissed or automatically written as a result
> > of a ds addition, it removes the "server assignment" step and the CDN owner
> > loses the control over the deployment decision.
> >
> >
> >
> > Before I open a discussion about how to solve this new tenancy issue, I
> > would like to verify it is a real issue and there is not something I'm
> > missing there.
> >
> > What do you think?
> >
> >
> >
> > Nir
> >
> >
> >
> > On May 22, 2017 6:13 PM, "Jeremy Mitchell" <mi...@gmail.com> wrote:
> >
> > @Nir - maybe your proposal is something to consider more long-term but for
> > the short-term i'm wondering if one of the following can / should be
> > implemented:
> >
> >
> >
> > A) creating a DS thru the API (POST /api/*/deliveryservices) can have a
> > default effect of automatically assigning all edges to the new DS (via the deliveryservice_server
> > table)
> >
> >
> >
> > or
> >
> >
> >
> > B) the deliveryservice_server table simple goes away and it becomes
> > implied that a DS employs all edge caches
> >
> >
> >
> > "A" would probably be the easiest to be honest.
> >
> >
> >
> > jeremy
> >
> >
> >
> > On Sat, May 20, 2017 at 12:34 PM, Nir Sopher <ni...@qwilt.com> wrote:
> >
> > +1 for having a configuration allowing to apply the DS to the entire CDN.
> >
> >
> >
> > Note that adding a "deploy by default" bool field to the DS itself may
> > cause tenancy issues, as I believe this field should be in the control of
> > the "CDN owner" tenant, and not of the "DS owner".
> >
> >
> >
> > What about keeping the ds_server table, and allow "undef" for the server
> > value - meaning "all servers in the CDN"?
> >
> > In the future the table may grow and have additional fields to white-list
> > servers by.
> >
> > For example:
> >
> >
> >
> > DS | Server | cache-group | profile
> >
> > ------+--------+-------------+--------
> >
> > ds1 | s1 | null | null => ds to be applied on s1
> >
> > ------+--------+-------------+--------
> >
> > ds2 | null | null | null => ds to be applied on all
> > servers (in ds CDN)
> >
> > ------+--------+-------------+--------
> >
> > ds3 | null | group1 | null => ds to be applied on all
> > servers in cache group "group1"
> >
> > ------+--------+-------------+--------
> >
> > ds4 | null | group1 | ATS-6.2 => ds to be applied on all
> > servers in cache group "group1", with profile "ATS-6.2"
> >
> >
> >
> >
> >
> > During CR config snapshot and configuration pull (or in advance), the
> > required calculation would be done, to provide the list of relevant DSs per
> > server.
> >
> >
> >
> > Also note that moving in this direction is backward compatible as well as
> > supports other non trivial extensions one may want in the future: Patterns
> > (wildcards/regexs) support, as well as black list, using a "policy table"
> > mechanism.
> >
> >
> >
> > In the below example "ds5" is to be deployed on all caches, except for
> > caches with "ATS-5" profile
> >
> >
> >
> > DS | Server | cache-group | profile | deploy-decision
> >
> > ------+--------+-------------+---------+----------------
> >
> > ds5 | null | null | ATS-5 | false
> >
> > ------+--------+-------------+---------+----------------
> >
> > ds5 | null | null | null | true
> >
> > ------+--------+-------------+---------+----------------
> >
> >
> >
> >
> >
> > Nir
> >
> >
> >
> > On Sat, May 20, 2017 at 1:42 AM, Jeremy Mitchell <mi...@gmail.com>
> > wrote:
> >
> > @dave - i'm not sure if you're suggesting:
> >
> >
> >
> > A) on DS create, by default create an entry in the deliveryservice_server
> > table for each cdn edge cache. so if there are 200 cdn edge caches, you end
> > up with 200 entries in the deliveryservice_server table for that DS.
> >
> > B) completely kill the deliveryservice_server table and it becomes implied
> > that a DS uses ALL cdn edge caches (the way mids work today. mids don't
> > have to be explicitly assigned to a DS)
> >
> >
> >
> > "A" would allow you to control the scope of caches by using initial
> > dispersion OR control the scope of caches by simply removing the ones you
> > don't want to receive traffic.
> >
> > "B" would only allow you to control the scope of caches by using initial
> > dispersion (or so i think).
> >
> >
> >
> > jeremy
> >
> >
> >
> >
> >
> >
> >
> > On Fri, May 19, 2017 at 4:25 PM, Dave Neuman <ne...@apache.org> wrote:
> >
> > +1, I think assigning DSs to all caches by default is fine. Then we can
> > use initial dispersion to make sure traffic is spread between an
> > appropriate number of caches.
> >
> >
> >
> > On Fri, May 19, 2017 at 4:09 PM, Jeremy Mitchell <mi...@gmail.com>
> > wrote:
> >
> > Bringing up an old email regarding DS to cache assignments.
> >
> >
> >
> > It sounds like we want to preserve DS/cache assignments (the
> > deliveryservice_server table) so you can assign a subset of cdn edge caches
> > to a DS if you want. However, what if we make the assignment of ALL cdn
> > edge caches to a new DS the default?
> >
> >
> >
> > So when you create a DS for the Foo cdn thru the API, your new DS is
> > automagically assigned all the edge caches in the Foo cdn...
> >
> >
> >
> > (at that point you could always remove caches from a DS)
> >
> >
> >
> > ...or the create DS API could have a boolean flag added to it. for
> > example, assignAllCaches: true|false. if you pass in true, all caches are
> > assigned to the DS, if false, none are.
> >
> >
> >
> > thoughts? I'm trying to think of self-service and i'm guessing most people
> > will not want to assign caches to a DS...
> >
> >
> >
> > jeremy
> >
> >
> >
> > On Fri, Jan 6, 2017 at 11:46 AM, Jeremy Mitchell <mi...@gmail.com>
> > wrote:
> >
> > gotcha. makes sense.
> >
> >
> >
> > On Fri, Jan 6, 2017 at 9:40 AM, David Neuman <da...@gmail.com>
> > wrote:
> >
> > I believe Eric had a use case where he wanted to assign certain delivery
> > services to certain caches.
> >
> > If we just use initial dispersion and max DNS answers then we don't have
> > the ability to control which servers are used, just how many.
> >
> >
> >
> >
> >
> >
> >
> > On Fri, Jan 6, 2017 at 9:23 AM, Jeremy Mitchell <mi...@gmail.com>
> > wrote:
> >
> > So what I heard was that potentially we move in this direction:
> >
> >
> >
> > 1. No need to explicitly assign edge caches to a delivery service. a
> > delivery service (by default) would employ ALL edge caches of the cdn in
> > which the ds belongs to.
> >
> > 2. Ability to override this default behavior and explicitly assign
> > specific edge caches to a delivery service (this is what we do today)
> >
> >
> >
> > But earlier Mark said "so the same functionality [as #2] can essentially
> > be reached with assigning all caches, and setting the config params
> > properly [initial dispersion and max dns answers]"
> >
> >
> >
> > so why not ditch #2 in favor of using initial dispersion and max dns
> > answers to control cache assignments?
> >
> >
> >
> > it would be nice imo to get rid of the ds_server table. i'm always in
> > favor of reducing complexity... :)
> >
> >
> >
> > jeremy
> >
> >
> >
> >
> >
> >
> >
> >
> >
> >
> >
> > On Thu, Jan 5, 2017 at 9:08 AM, Eric Friedrich (efriedri) <
> > efriedri@cisco.com> wrote:
> >
> > Yes - I think its a reasonable default to have a delivery service assigned
> > to all caches as long as we still keep the ability to assign some DS’s to
> > just a subset of caches.
> >
> >
> >
> > —Eric
> >
> >
> >
> > On Jan 4, 2017, at 4:56 PM, Mark Torluemke <mt...@apache.org> wrote:
> >
> >
> >
> > Good feedback, thanks. I think having _zero_ caches assigned to a delivery
> > service could be added fairly easily, and could function as a good default
> > to the assumption of _all_ caches being assigned. Would you agree with
> > that?
> >
> >
> >
> > Thanks,
> >
> > Mark
> >
> >
> >
> > On Wed, Jan 4, 2017 at 1:55 PM, Eric Friedrich (efriedri) <
> > efriedri@cisco.com> wrote:
> >
> > Yeah,
> >
> > We have several some use cases where where cache or CG level allocation
> > would be helpful.
> >
> >
> >
> > 1) Assigning new or trial delivery services to a subset of caches
> > (considered beta cache nodes). This is helpful in trying out new DS configs
> > before rolling them out to production.
> >
> >
> >
> > 2) Assigning DSs to caches can be a form of quota or resource reservation
> > on the delivery service. Providers of content to the CDN can receive tiered
> > service by either sharing caches or using dedicated caches.
> >
> >
> >
> > —Eric
> >
> >
> >
> >
> >
> > On Jan 4, 2017, at 2:04 PM, Mark Torluemke <mt...@apache.org> wrote:
> >
> >
> >
> > Today, there isn't an actual link from Cache Group to DS -- I think what
> > you are referring to is all done in the app layer to make a nice summary in
> > the UI to allow operators to very quickly assign all caches in a cache
> > group.
> >
> > Do you use that "feature" to assign caches from some cache groups, and not
> > others?
> >
> >
> >
> > On Tue, Jan 3, 2017 at 9:50 AM, Eric Friedrich (efriedri) <
> > efriedri@cisco.com> wrote:
> >
> > Would we keep the Cache Group -> DS assignment?
> >
> > —Eric
> >
> >
> > > On Jan 3, 2017, at 9:22 AM, Mark Torluemke <mt...@apache.org>
> > wrote:
> > >
> > > Seeking community opinion on this.
> > >
> > > Should we drop the server -> delivery service assignments completely?
> > >
> > > When the project began, selectively assigning caches was the only way to
> > selectively route traffic within a cache group -- let's say you have a
> > DNS-routed service with a relatively large library size, you might not want
> > assign all caches in a cache group to that service for fear you would
> > 'pollute' those caches. Or, if you were turning up a 'test' DS of which you
> > wanted to limit exposure on, you might apply the same logic. In recent
> > releases, Traffic Control supports "initial dispersion" for _both_ DNS and
> > HTTP-routed services, and "max DNS answers" for DNS-routed services, so the
> > same functionality can essentially be reached with assigning all caches,
> > and setting the config params properly.
> > >
> > > The motivation for this is to reduce the size & complexity of the
> > Traffic Router's config (aka CRConfig), as this config scales mostly
> > linearly with the number of caches and the number of delivery services in
> > your CDN.
> > >
> > > Another option might be to not assign caches as the default, and
> > individual cache assignments is available, but used only when necessary.
> > >
> > > Other thoughts? Clarifications needed?
> > >
> > > Thanks,
> > > Mark
> >
> >
> >
> >
> >
> >
> >
> >
> >
> >
> >
> >
> >
> >
> >
> >
> >
> >
> >
> >
> >
> >
> >
> >
> >
> >
> >
> >
> >
> >
> >
> >
> >
> >
> >
> > --
> >
> > *Shmulik Asafi*
> > Qwilt | Work: +972-72-2221692 <+972%2072-222-1692>| Mobile:
> > +972-54-6581595 <+972%2054-658-1595>| shmulika@qwilt.com <yo...@qwilt.com>
> >
> >
> >
> >
> >
> >
> >
>
>
>
> --
>
> *Oren Shemesh*
> Qwilt | Work: +972-72-2221637| Mobile: +972-50-2281168 | orens@qwilt.com
> <yo...@qwilt.com>
>