You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@trafficcontrol.apache.org by "Gelinas, Derek" <De...@comcast.com> on 2017/07/25 19:13:31 UTC
Delivery Service based config generation and Cache Manager
At the summit, there was some talk about changing the manner in which we generate configuration files. The early stages of this idea had me creating large CDN definition files, but in the course of our discussion it became clear that we would be better served by creating delivery service configuration files instead. This would shift us from a server-generated implementation, as we have now, to generating the configuration files for the caches locally. The data for this would come from a new API that would provide the delivery service definitions in json format.
What I’m envisioning is creating delivery service “snapshots” which are saved to the database as json objects. These snapshots would have the full range of information specific to the delivery service, including the new DS profiles. The database would store up to five of these objects per DS, and one DS object would be set to “active” through the UI or API.
In this way, we could create multiple versions of a delivery service, or safely modify the definition currently “live” (but not necessarily active) in the database without changing the configuration in the field. Configuration would only be changed when the DS was saved and then that saved version was set to become active. In the reverse manner, existing saved delivery services could be restored to the live DB for modification.
By divorcing the “live” db from the active configuration we prevent the possibility of accidental edits affecting the field, or edits-in-progress from being sent out prematurely when one person is working on a delivery service and another is queueing updates.
Once set, it would be this active delivery service definition that would be provided to the rest of traffic ops for any delivery service operations. For config file generation, new API endpoints would be created that do the following:
- List the delivery services and the active versions of each assigned to the specific server.
- Provide the json object from the database when requested - I’m thinking that the endpoint would send the current active by default, or a specific version if specified.
These definitions would be absurdly cacheable - we would not need to worry about sending stale data because each new version would have a completely different name - and so could be generated once and sent to thousands of caches with greatly reduced load on traffic ops. The load would consist of the initial creation of the json object, and the minimal serving of that object, so this would still result in greatly reduced load on the traffic ops host(s) even without the use of caching. Because of this, the new cache management service could check with traffic ops multiple times per minute for updates. Once a delivery service was changed, the new json would be downloaded and configs generated on the cache itself.
Other benefits of the use of a cache manager service rather than the ORT script include:
- Decreased load from logins - once the cache has logged in, it could use the cookie from the previous session and only re-login when that cookie has expired. we could also explore the use of certificates or keys instead, and eliminate logins altogether.
- Multiple checks per minute rather than every X minutes - faster checks, more agile CDN.
- Service could provide regular status updates to traffic ops, giving us the ability to keep an eye out for drastic shifts in i/o, unwanted behavior, problems with the ATS service, etc. This leads to building a traffic ops that can adapt itself on the fly to changing conditions and adjust accordingly.
- Queue commands to run on the host from traffic ops. ATS restarts, system reboots, all manner of things could be triggered and scheduled right from traffic ops.
Thoughts?
Derek
Re: Delivery Service based config generation and Cache Manager
Posted by Nir Sopher <ni...@qwilt.com>.
Hi,
What would you say about the following refinement, merging the ideas of
configuration versioning, and a DS individual JSON:
- The configuration of the different delivery-services is propagated to
the caches using separate JSON files - a JSON per DS
- The different versions are kept in the DB as raw records in the DS
table.
- Whenever the ort script are connecting to traffic ops, it:
1. Requests the list of DS configuration "file identifier"s to be
pulled - each file identifier represent the configuration related to a
single DS.
2. For each such identifier, if the JSON associated with the
identifier was not pulled before, pulls it
Note: The "to be pulled" JSON is generated from the active version
configuration during this process.
3. Removes JSONs associated with identifies that were previously
pulled but are no longer in the list.
4. Uses the gathered files to adjust the ATS configuration
- A "file identifier" changes on the below circumstances:
1. A delivery-service "active version" change
2. A format change (e.g. due to a new added field, a bug in the
previous JSON creation, a newly introduced pattern support, etc.)
In this paradigm:
1. Every DS configuration is pulled only once by each server
2. Cache may reduce TO work in file generation (due to multiple servers
pulling the same configuration)
3. We do not save JSON products in the DB.
Nir
On Fri, Jul 28, 2017 at 5:51 PM, Dewayne Richardson <de...@gmail.com>
wrote:
> When: Read · Fri, Jul 28.
> <https://timyo.com/?utm_source=expectationheader&utm_medium=email>
> [image: Timyo expectation line]
> If Postgres performance becomes a concern, we can make ORT send it's
> "Update" to a different endpoint than the Read, we can also take advantage
> of Postgres HA to have dedicated Postgres Replicas that will allow us to
> scale even more.
>
> -Dew
>
> On Thu, Jul 27, 2017 at 8:28 AM, Robert Butts <ro...@gmail.com>
> wrote:
>
> > As long as we have the proper indexes, database performance shouldn't be
> an
> > issue. Postgres is capable of handling, and I've personally worked with,
> > databases several orders of magnitude larger than ours. Now, if you don't
> > have the indexes, querying will absolutely be slow. But as long as we add
> > indexes to the columns we query on, it won't be an issue.
> >
> > I'm likewise not concerned about size. Our current production database is
> > 162mb. But I'm not opposed to writing a truncate utility, if we think
> it's
> > necessary. The queries are simple, it should be easy enough to write a
> > function and a GUI button. I would oppose it being used unless absolutely
> > necessary though; history is invaluable.
> >
> > On Thu, Jul 27, 2017 at 8:15 AM, Gelinas, Derek <
> Derek_Gelinas@comcast.com
> > >
> > wrote:
> >
> > > I'm down with this! But I'm worried about database performance, for
> one,
> > > and table size. I think we need to have a utility for removing older
> > > entries if we are to go this route.
> > >
> > > On Jul 27, 2017, at 10:12 AM, Robert Butts <robert.o.butts@gmail.com<
> > > mailto:robert.o.butts@gmail.com>> wrote:
> > >
> > > Can I propose an adjustment?
> > >
> > > If we add a timestamp to every table, we can generate the JSON
> > on-the-fly.
> > > Then, the snapshot becomes a timestamp field, `snapshot_time`, and all
> > the
> > > data `select` queries add `where timestamp <= snapshot_time limit 1`.
> > > Instead of updating rows, we only ever insert new rows with new
> > timestamps.
> > > This gives us snapshots back to eternity, and if a snapshot ever
> breaks,
> > > rolling back is as simple as updating the `snapshot_time`. Our data is
> so
> > > tiny, space is almost certainly not a problem, but if it is, truncating
> > is
> > > as easy as `delete where count > X and timestamp < Y`.
> > >
> > > That gives us all the benefits of your plan, plus the benefits of
> > > relational data, type safety, more powerful querying, etc. And it
> > shouldn't
> > > be much more work to implement: add timestamp columns, snapshotting
> > updates
> > > the snapshot field, and getting the config simply runs what the
> snapshot
> > > otherwise would to create the JSON. If generation performance is an
> issue
> > > (it may be in Perl, probably not in Go), we can always cache the latest
> > > snapshot in memory, and only regenerate it when the `snapshot_time`
> > > changes.
> > >
> > > On Wed, Jul 26, 2017 at 9:08 AM, Gelinas, Derek <
> > Derek_Gelinas@comcast.com
> > > <ma...@comcast.com>>
> > > wrote:
> > >
> > > That’s not a terrible idea. Fewer changes to the code that way for
> sure,
> > > really just the DS interface page.
> > >
> > > On Jul 26, 2017, at 10:30 AM, Nir Sopher <nirs@qwilt.com<mailto:nirs@
> > > qwilt.com>> wrote:
> > >
> > > Hi Derek,
> > >
> > > As discussed in the summit, we also see significant value in
> > >
> > > 1. DS Deployment Granularity - using DS individual config files.
> > > 2. Delivery Service Configuration Versioning (DSCV) - separating the
> > > "provisioning" from the "deployment".
> > > 3. Improving the roll-out procedure, joining the capabilities #1 & #2
> > >
> > > We are on the same page with these needs:)
> > >
> > > However, as I see it, these are #1 & #2 are 2 separate features, each
> has
> > > different requirements.
> > > For example, for DSCV, I would suggest to manage the versions as
> > > standard
> > > rows in the Delivery-Service table, side by side with the "hot" DS
> > > configuration.
> > > This will allow the existing code (with minor adjustments) to properly
> > > work
> > > on these rows.
> > > Furthermore, it also allows you to simply "restore" the DS "hot"
> > > configuration to a specified revision.
> > > It is also more resilient to DS table schema updates.
> > >
> > > I'll soon share, on another thread, a link to a "DSCV functional spec"
> I
> > > was working on. It extends the presentation
> > > <https://cwiki.apache.org/confluence/download/attachments/69407844/TC%
> > > 20Summit%20-%20Spring%202017%20-%20Self-Service.pptx?
> > > version=1&modificationDate=1495451091000&api=v2>
> > > we
> > > had in the summit.
> > > I would appreciate any inputs to this spec.
> > >
> > > Nir
> > >
> > > On Tue, Jul 25, 2017 at 10:13 PM, Gelinas, Derek <
> > > Derek_Gelinas@comcast.com<ma...@comcast.com>>
> > > wrote:
> > >
> > > At the summit, there was some talk about changing the manner in which
> we
> > > generate configuration files. The early stages of this idea had me
> > > creating large CDN definition files, but in the course of our
> > > discussion it
> > > became clear that we would be better served by creating delivery
> service
> > > configuration files instead. This would shift us from a
> > > server-generated
> > > implementation, as we have now, to generating the configuration files
> > > for
> > > the caches locally. The data for this would come from a new API that
> > > would
> > > provide the delivery service definitions in json format.
> > >
> > > What I’m envisioning is creating delivery service “snapshots” which are
> > > saved to the database as json objects. These snapshots would have the
> > > full
> > > range of information specific to the delivery service, including the
> > > new DS
> > > profiles. The database would store up to five of these objects per DS,
> > > and
> > > one DS object would be set to “active” through the UI or API.
> > >
> > > In this way, we could create multiple versions of a delivery service,
> or
> > > safely modify the definition currently “live” (but not necessarily
> > > active)
> > > in the database without changing the configuration in the field.
> > > Configuration would only be changed when the DS was saved and then that
> > > saved version was set to become active. In the reverse manner,
> existing
> > > saved delivery services could be restored to the live DB for
> > > modification.
> > >
> > > By divorcing the “live” db from the active configuration we prevent the
> > > possibility of accidental edits affecting the field, or
> > > edits-in-progress
> > > from being sent out prematurely when one person is working on a
> delivery
> > > service and another is queueing updates.
> > >
> > > Once set, it would be this active delivery service definition that
> would
> > > be provided to the rest of traffic ops for any delivery service
> > > operations. For config file generation, new API endpoints would be
> > > created
> > > that do the following:
> > >
> > > - List the delivery services and the active versions of each assigned
> to
> > > the specific server.
> > > - Provide the json object from the database when requested - I’m
> > > thinking
> > > that the endpoint would send the current active by default, or a
> > > specific
> > > version if specified.
> > >
> > > These definitions would be absurdly cacheable - we would not need to
> > > worry
> > > about sending stale data because each new version would have a
> > > completely
> > > different name - and so could be generated once and sent to thousands
> of
> > > caches with greatly reduced load on traffic ops. The load would
> > > consist of
> > > the initial creation of the json object, and the minimal serving of
> that
> > > object, so this would still result in greatly reduced load on the
> > > traffic
> > > ops host(s) even without the use of caching. Because of this, the new
> > > cache management service could check with traffic ops multiple times
> per
> > > minute for updates. Once a delivery service was changed, the new json
> > > would be downloaded and configs generated on the cache itself.
> > >
> > > Other benefits of the use of a cache manager service rather than the
> ORT
> > > script include:
> > >
> > > - Decreased load from logins - once the cache has logged in, it could
> > > use
> > > the cookie from the previous session and only re-login when that cookie
> > > has
> > > expired. we could also explore the use of certificates or keys
> instead,
> > > and eliminate logins altogether.
> > > - Multiple checks per minute rather than every X minutes - faster
> > > checks,
> > > more agile CDN.
> > > - Service could provide regular status updates to traffic ops, giving
> us
> > > the ability to keep an eye out for drastic shifts in i/o, unwanted
> > > behavior, problems with the ATS service, etc. This leads to building a
> > > traffic ops that can adapt itself on the fly to changing conditions and
> > > adjust accordingly.
> > > - Queue commands to run on the host from traffic ops. ATS restarts,
> > > system reboots, all manner of things could be triggered and scheduled
> > > right
> > > from traffic ops.
> > >
> > > Thoughts?
> > >
> > > Derek
> > >
> > >
> > >
> >
>
Re: Delivery Service based config generation and Cache Manager
Posted by Dewayne Richardson <de...@gmail.com>.
When: Read · Fri, Jul 28.
<https://timyo.com/?utm_source=expectationheader&utm_medium=email>
[image: Timyo expectation line]
If Postgres performance becomes a concern, we can make ORT send it's
"Update" to a different endpoint than the Read, we can also take advantage
of Postgres HA to have dedicated Postgres Replicas that will allow us to
scale even more.
-Dew
On Thu, Jul 27, 2017 at 8:28 AM, Robert Butts <ro...@gmail.com>
wrote:
> As long as we have the proper indexes, database performance shouldn't be an
> issue. Postgres is capable of handling, and I've personally worked with,
> databases several orders of magnitude larger than ours. Now, if you don't
> have the indexes, querying will absolutely be slow. But as long as we add
> indexes to the columns we query on, it won't be an issue.
>
> I'm likewise not concerned about size. Our current production database is
> 162mb. But I'm not opposed to writing a truncate utility, if we think it's
> necessary. The queries are simple, it should be easy enough to write a
> function and a GUI button. I would oppose it being used unless absolutely
> necessary though; history is invaluable.
>
> On Thu, Jul 27, 2017 at 8:15 AM, Gelinas, Derek <Derek_Gelinas@comcast.com
> >
> wrote:
>
> > I'm down with this! But I'm worried about database performance, for one,
> > and table size. I think we need to have a utility for removing older
> > entries if we are to go this route.
> >
> > On Jul 27, 2017, at 10:12 AM, Robert Butts <robert.o.butts@gmail.com<
> > mailto:robert.o.butts@gmail.com>> wrote:
> >
> > Can I propose an adjustment?
> >
> > If we add a timestamp to every table, we can generate the JSON
> on-the-fly.
> > Then, the snapshot becomes a timestamp field, `snapshot_time`, and all
> the
> > data `select` queries add `where timestamp <= snapshot_time limit 1`.
> > Instead of updating rows, we only ever insert new rows with new
> timestamps.
> > This gives us snapshots back to eternity, and if a snapshot ever breaks,
> > rolling back is as simple as updating the `snapshot_time`. Our data is so
> > tiny, space is almost certainly not a problem, but if it is, truncating
> is
> > as easy as `delete where count > X and timestamp < Y`.
> >
> > That gives us all the benefits of your plan, plus the benefits of
> > relational data, type safety, more powerful querying, etc. And it
> shouldn't
> > be much more work to implement: add timestamp columns, snapshotting
> updates
> > the snapshot field, and getting the config simply runs what the snapshot
> > otherwise would to create the JSON. If generation performance is an issue
> > (it may be in Perl, probably not in Go), we can always cache the latest
> > snapshot in memory, and only regenerate it when the `snapshot_time`
> > changes.
> >
> > On Wed, Jul 26, 2017 at 9:08 AM, Gelinas, Derek <
> Derek_Gelinas@comcast.com
> > <ma...@comcast.com>>
> > wrote:
> >
> > That’s not a terrible idea. Fewer changes to the code that way for sure,
> > really just the DS interface page.
> >
> > On Jul 26, 2017, at 10:30 AM, Nir Sopher <nirs@qwilt.com<mailto:nirs@
> > qwilt.com>> wrote:
> >
> > Hi Derek,
> >
> > As discussed in the summit, we also see significant value in
> >
> > 1. DS Deployment Granularity - using DS individual config files.
> > 2. Delivery Service Configuration Versioning (DSCV) - separating the
> > "provisioning" from the "deployment".
> > 3. Improving the roll-out procedure, joining the capabilities #1 & #2
> >
> > We are on the same page with these needs:)
> >
> > However, as I see it, these are #1 & #2 are 2 separate features, each has
> > different requirements.
> > For example, for DSCV, I would suggest to manage the versions as
> > standard
> > rows in the Delivery-Service table, side by side with the "hot" DS
> > configuration.
> > This will allow the existing code (with minor adjustments) to properly
> > work
> > on these rows.
> > Furthermore, it also allows you to simply "restore" the DS "hot"
> > configuration to a specified revision.
> > It is also more resilient to DS table schema updates.
> >
> > I'll soon share, on another thread, a link to a "DSCV functional spec" I
> > was working on. It extends the presentation
> > <https://cwiki.apache.org/confluence/download/attachments/69407844/TC%
> > 20Summit%20-%20Spring%202017%20-%20Self-Service.pptx?
> > version=1&modificationDate=1495451091000&api=v2>
> > we
> > had in the summit.
> > I would appreciate any inputs to this spec.
> >
> > Nir
> >
> > On Tue, Jul 25, 2017 at 10:13 PM, Gelinas, Derek <
> > Derek_Gelinas@comcast.com<ma...@comcast.com>>
> > wrote:
> >
> > At the summit, there was some talk about changing the manner in which we
> > generate configuration files. The early stages of this idea had me
> > creating large CDN definition files, but in the course of our
> > discussion it
> > became clear that we would be better served by creating delivery service
> > configuration files instead. This would shift us from a
> > server-generated
> > implementation, as we have now, to generating the configuration files
> > for
> > the caches locally. The data for this would come from a new API that
> > would
> > provide the delivery service definitions in json format.
> >
> > What I’m envisioning is creating delivery service “snapshots” which are
> > saved to the database as json objects. These snapshots would have the
> > full
> > range of information specific to the delivery service, including the
> > new DS
> > profiles. The database would store up to five of these objects per DS,
> > and
> > one DS object would be set to “active” through the UI or API.
> >
> > In this way, we could create multiple versions of a delivery service, or
> > safely modify the definition currently “live” (but not necessarily
> > active)
> > in the database without changing the configuration in the field.
> > Configuration would only be changed when the DS was saved and then that
> > saved version was set to become active. In the reverse manner, existing
> > saved delivery services could be restored to the live DB for
> > modification.
> >
> > By divorcing the “live” db from the active configuration we prevent the
> > possibility of accidental edits affecting the field, or
> > edits-in-progress
> > from being sent out prematurely when one person is working on a delivery
> > service and another is queueing updates.
> >
> > Once set, it would be this active delivery service definition that would
> > be provided to the rest of traffic ops for any delivery service
> > operations. For config file generation, new API endpoints would be
> > created
> > that do the following:
> >
> > - List the delivery services and the active versions of each assigned to
> > the specific server.
> > - Provide the json object from the database when requested - I’m
> > thinking
> > that the endpoint would send the current active by default, or a
> > specific
> > version if specified.
> >
> > These definitions would be absurdly cacheable - we would not need to
> > worry
> > about sending stale data because each new version would have a
> > completely
> > different name - and so could be generated once and sent to thousands of
> > caches with greatly reduced load on traffic ops. The load would
> > consist of
> > the initial creation of the json object, and the minimal serving of that
> > object, so this would still result in greatly reduced load on the
> > traffic
> > ops host(s) even without the use of caching. Because of this, the new
> > cache management service could check with traffic ops multiple times per
> > minute for updates. Once a delivery service was changed, the new json
> > would be downloaded and configs generated on the cache itself.
> >
> > Other benefits of the use of a cache manager service rather than the ORT
> > script include:
> >
> > - Decreased load from logins - once the cache has logged in, it could
> > use
> > the cookie from the previous session and only re-login when that cookie
> > has
> > expired. we could also explore the use of certificates or keys instead,
> > and eliminate logins altogether.
> > - Multiple checks per minute rather than every X minutes - faster
> > checks,
> > more agile CDN.
> > - Service could provide regular status updates to traffic ops, giving us
> > the ability to keep an eye out for drastic shifts in i/o, unwanted
> > behavior, problems with the ATS service, etc. This leads to building a
> > traffic ops that can adapt itself on the fly to changing conditions and
> > adjust accordingly.
> > - Queue commands to run on the host from traffic ops. ATS restarts,
> > system reboots, all manner of things could be triggered and scheduled
> > right
> > from traffic ops.
> >
> > Thoughts?
> >
> > Derek
> >
> >
> >
>
Re: Delivery Service based config generation and Cache Manager
Posted by Robert Butts <ro...@gmail.com>.
As long as we have the proper indexes, database performance shouldn't be an
issue. Postgres is capable of handling, and I've personally worked with,
databases several orders of magnitude larger than ours. Now, if you don't
have the indexes, querying will absolutely be slow. But as long as we add
indexes to the columns we query on, it won't be an issue.
I'm likewise not concerned about size. Our current production database is
162mb. But I'm not opposed to writing a truncate utility, if we think it's
necessary. The queries are simple, it should be easy enough to write a
function and a GUI button. I would oppose it being used unless absolutely
necessary though; history is invaluable.
On Thu, Jul 27, 2017 at 8:15 AM, Gelinas, Derek <De...@comcast.com>
wrote:
> I'm down with this! But I'm worried about database performance, for one,
> and table size. I think we need to have a utility for removing older
> entries if we are to go this route.
>
> On Jul 27, 2017, at 10:12 AM, Robert Butts <robert.o.butts@gmail.com<
> mailto:robert.o.butts@gmail.com>> wrote:
>
> Can I propose an adjustment?
>
> If we add a timestamp to every table, we can generate the JSON on-the-fly.
> Then, the snapshot becomes a timestamp field, `snapshot_time`, and all the
> data `select` queries add `where timestamp <= snapshot_time limit 1`.
> Instead of updating rows, we only ever insert new rows with new timestamps.
> This gives us snapshots back to eternity, and if a snapshot ever breaks,
> rolling back is as simple as updating the `snapshot_time`. Our data is so
> tiny, space is almost certainly not a problem, but if it is, truncating is
> as easy as `delete where count > X and timestamp < Y`.
>
> That gives us all the benefits of your plan, plus the benefits of
> relational data, type safety, more powerful querying, etc. And it shouldn't
> be much more work to implement: add timestamp columns, snapshotting updates
> the snapshot field, and getting the config simply runs what the snapshot
> otherwise would to create the JSON. If generation performance is an issue
> (it may be in Perl, probably not in Go), we can always cache the latest
> snapshot in memory, and only regenerate it when the `snapshot_time`
> changes.
>
> On Wed, Jul 26, 2017 at 9:08 AM, Gelinas, Derek <Derek_Gelinas@comcast.com
> <ma...@comcast.com>>
> wrote:
>
> That’s not a terrible idea. Fewer changes to the code that way for sure,
> really just the DS interface page.
>
> On Jul 26, 2017, at 10:30 AM, Nir Sopher <nirs@qwilt.com<mailto:nirs@
> qwilt.com>> wrote:
>
> Hi Derek,
>
> As discussed in the summit, we also see significant value in
>
> 1. DS Deployment Granularity - using DS individual config files.
> 2. Delivery Service Configuration Versioning (DSCV) - separating the
> "provisioning" from the "deployment".
> 3. Improving the roll-out procedure, joining the capabilities #1 & #2
>
> We are on the same page with these needs:)
>
> However, as I see it, these are #1 & #2 are 2 separate features, each has
> different requirements.
> For example, for DSCV, I would suggest to manage the versions as
> standard
> rows in the Delivery-Service table, side by side with the "hot" DS
> configuration.
> This will allow the existing code (with minor adjustments) to properly
> work
> on these rows.
> Furthermore, it also allows you to simply "restore" the DS "hot"
> configuration to a specified revision.
> It is also more resilient to DS table schema updates.
>
> I'll soon share, on another thread, a link to a "DSCV functional spec" I
> was working on. It extends the presentation
> <https://cwiki.apache.org/confluence/download/attachments/69407844/TC%
> 20Summit%20-%20Spring%202017%20-%20Self-Service.pptx?
> version=1&modificationDate=1495451091000&api=v2>
> we
> had in the summit.
> I would appreciate any inputs to this spec.
>
> Nir
>
> On Tue, Jul 25, 2017 at 10:13 PM, Gelinas, Derek <
> Derek_Gelinas@comcast.com<ma...@comcast.com>>
> wrote:
>
> At the summit, there was some talk about changing the manner in which we
> generate configuration files. The early stages of this idea had me
> creating large CDN definition files, but in the course of our
> discussion it
> became clear that we would be better served by creating delivery service
> configuration files instead. This would shift us from a
> server-generated
> implementation, as we have now, to generating the configuration files
> for
> the caches locally. The data for this would come from a new API that
> would
> provide the delivery service definitions in json format.
>
> What I’m envisioning is creating delivery service “snapshots” which are
> saved to the database as json objects. These snapshots would have the
> full
> range of information specific to the delivery service, including the
> new DS
> profiles. The database would store up to five of these objects per DS,
> and
> one DS object would be set to “active” through the UI or API.
>
> In this way, we could create multiple versions of a delivery service, or
> safely modify the definition currently “live” (but not necessarily
> active)
> in the database without changing the configuration in the field.
> Configuration would only be changed when the DS was saved and then that
> saved version was set to become active. In the reverse manner, existing
> saved delivery services could be restored to the live DB for
> modification.
>
> By divorcing the “live” db from the active configuration we prevent the
> possibility of accidental edits affecting the field, or
> edits-in-progress
> from being sent out prematurely when one person is working on a delivery
> service and another is queueing updates.
>
> Once set, it would be this active delivery service definition that would
> be provided to the rest of traffic ops for any delivery service
> operations. For config file generation, new API endpoints would be
> created
> that do the following:
>
> - List the delivery services and the active versions of each assigned to
> the specific server.
> - Provide the json object from the database when requested - I’m
> thinking
> that the endpoint would send the current active by default, or a
> specific
> version if specified.
>
> These definitions would be absurdly cacheable - we would not need to
> worry
> about sending stale data because each new version would have a
> completely
> different name - and so could be generated once and sent to thousands of
> caches with greatly reduced load on traffic ops. The load would
> consist of
> the initial creation of the json object, and the minimal serving of that
> object, so this would still result in greatly reduced load on the
> traffic
> ops host(s) even without the use of caching. Because of this, the new
> cache management service could check with traffic ops multiple times per
> minute for updates. Once a delivery service was changed, the new json
> would be downloaded and configs generated on the cache itself.
>
> Other benefits of the use of a cache manager service rather than the ORT
> script include:
>
> - Decreased load from logins - once the cache has logged in, it could
> use
> the cookie from the previous session and only re-login when that cookie
> has
> expired. we could also explore the use of certificates or keys instead,
> and eliminate logins altogether.
> - Multiple checks per minute rather than every X minutes - faster
> checks,
> more agile CDN.
> - Service could provide regular status updates to traffic ops, giving us
> the ability to keep an eye out for drastic shifts in i/o, unwanted
> behavior, problems with the ATS service, etc. This leads to building a
> traffic ops that can adapt itself on the fly to changing conditions and
> adjust accordingly.
> - Queue commands to run on the host from traffic ops. ATS restarts,
> system reboots, all manner of things could be triggered and scheduled
> right
> from traffic ops.
>
> Thoughts?
>
> Derek
>
>
>
Re: Delivery Service based config generation and Cache Manager
Posted by "Gelinas, Derek" <De...@comcast.com>.
I'm down with this! But I'm worried about database performance, for one, and table size. I think we need to have a utility for removing older entries if we are to go this route.
On Jul 27, 2017, at 10:12 AM, Robert Butts <ro...@gmail.com>> wrote:
Can I propose an adjustment?
If we add a timestamp to every table, we can generate the JSON on-the-fly.
Then, the snapshot becomes a timestamp field, `snapshot_time`, and all the
data `select` queries add `where timestamp <= snapshot_time limit 1`.
Instead of updating rows, we only ever insert new rows with new timestamps.
This gives us snapshots back to eternity, and if a snapshot ever breaks,
rolling back is as simple as updating the `snapshot_time`. Our data is so
tiny, space is almost certainly not a problem, but if it is, truncating is
as easy as `delete where count > X and timestamp < Y`.
That gives us all the benefits of your plan, plus the benefits of
relational data, type safety, more powerful querying, etc. And it shouldn't
be much more work to implement: add timestamp columns, snapshotting updates
the snapshot field, and getting the config simply runs what the snapshot
otherwise would to create the JSON. If generation performance is an issue
(it may be in Perl, probably not in Go), we can always cache the latest
snapshot in memory, and only regenerate it when the `snapshot_time` changes.
On Wed, Jul 26, 2017 at 9:08 AM, Gelinas, Derek <De...@comcast.com>>
wrote:
That’s not a terrible idea. Fewer changes to the code that way for sure,
really just the DS interface page.
On Jul 26, 2017, at 10:30 AM, Nir Sopher <ni...@qwilt.com>> wrote:
Hi Derek,
As discussed in the summit, we also see significant value in
1. DS Deployment Granularity - using DS individual config files.
2. Delivery Service Configuration Versioning (DSCV) - separating the
"provisioning" from the "deployment".
3. Improving the roll-out procedure, joining the capabilities #1 & #2
We are on the same page with these needs:)
However, as I see it, these are #1 & #2 are 2 separate features, each has
different requirements.
For example, for DSCV, I would suggest to manage the versions as
standard
rows in the Delivery-Service table, side by side with the "hot" DS
configuration.
This will allow the existing code (with minor adjustments) to properly
work
on these rows.
Furthermore, it also allows you to simply "restore" the DS "hot"
configuration to a specified revision.
It is also more resilient to DS table schema updates.
I'll soon share, on another thread, a link to a "DSCV functional spec" I
was working on. It extends the presentation
<https://cwiki.apache.org/confluence/download/attachments/69407844/TC%
20Summit%20-%20Spring%202017%20-%20Self-Service.pptx?
version=1&modificationDate=1495451091000&api=v2>
we
had in the summit.
I would appreciate any inputs to this spec.
Nir
On Tue, Jul 25, 2017 at 10:13 PM, Gelinas, Derek <
Derek_Gelinas@comcast.com<ma...@comcast.com>>
wrote:
At the summit, there was some talk about changing the manner in which we
generate configuration files. The early stages of this idea had me
creating large CDN definition files, but in the course of our
discussion it
became clear that we would be better served by creating delivery service
configuration files instead. This would shift us from a
server-generated
implementation, as we have now, to generating the configuration files
for
the caches locally. The data for this would come from a new API that
would
provide the delivery service definitions in json format.
What I’m envisioning is creating delivery service “snapshots” which are
saved to the database as json objects. These snapshots would have the
full
range of information specific to the delivery service, including the
new DS
profiles. The database would store up to five of these objects per DS,
and
one DS object would be set to “active” through the UI or API.
In this way, we could create multiple versions of a delivery service, or
safely modify the definition currently “live” (but not necessarily
active)
in the database without changing the configuration in the field.
Configuration would only be changed when the DS was saved and then that
saved version was set to become active. In the reverse manner, existing
saved delivery services could be restored to the live DB for
modification.
By divorcing the “live” db from the active configuration we prevent the
possibility of accidental edits affecting the field, or
edits-in-progress
from being sent out prematurely when one person is working on a delivery
service and another is queueing updates.
Once set, it would be this active delivery service definition that would
be provided to the rest of traffic ops for any delivery service
operations. For config file generation, new API endpoints would be
created
that do the following:
- List the delivery services and the active versions of each assigned to
the specific server.
- Provide the json object from the database when requested - I’m
thinking
that the endpoint would send the current active by default, or a
specific
version if specified.
These definitions would be absurdly cacheable - we would not need to
worry
about sending stale data because each new version would have a
completely
different name - and so could be generated once and sent to thousands of
caches with greatly reduced load on traffic ops. The load would
consist of
the initial creation of the json object, and the minimal serving of that
object, so this would still result in greatly reduced load on the
traffic
ops host(s) even without the use of caching. Because of this, the new
cache management service could check with traffic ops multiple times per
minute for updates. Once a delivery service was changed, the new json
would be downloaded and configs generated on the cache itself.
Other benefits of the use of a cache manager service rather than the ORT
script include:
- Decreased load from logins - once the cache has logged in, it could
use
the cookie from the previous session and only re-login when that cookie
has
expired. we could also explore the use of certificates or keys instead,
and eliminate logins altogether.
- Multiple checks per minute rather than every X minutes - faster
checks,
more agile CDN.
- Service could provide regular status updates to traffic ops, giving us
the ability to keep an eye out for drastic shifts in i/o, unwanted
behavior, problems with the ATS service, etc. This leads to building a
traffic ops that can adapt itself on the fly to changing conditions and
adjust accordingly.
- Queue commands to run on the host from traffic ops. ATS restarts,
system reboots, all manner of things could be triggered and scheduled
right
from traffic ops.
Thoughts?
Derek
Re: Delivery Service based config generation and Cache Manager
Posted by Robert Butts <ro...@gmail.com>.
Can I propose an adjustment?
If we add a timestamp to every table, we can generate the JSON on-the-fly.
Then, the snapshot becomes a timestamp field, `snapshot_time`, and all the
data `select` queries add `where timestamp <= snapshot_time limit 1`.
Instead of updating rows, we only ever insert new rows with new timestamps.
This gives us snapshots back to eternity, and if a snapshot ever breaks,
rolling back is as simple as updating the `snapshot_time`. Our data is so
tiny, space is almost certainly not a problem, but if it is, truncating is
as easy as `delete where count > X and timestamp < Y`.
That gives us all the benefits of your plan, plus the benefits of
relational data, type safety, more powerful querying, etc. And it shouldn't
be much more work to implement: add timestamp columns, snapshotting updates
the snapshot field, and getting the config simply runs what the snapshot
otherwise would to create the JSON. If generation performance is an issue
(it may be in Perl, probably not in Go), we can always cache the latest
snapshot in memory, and only regenerate it when the `snapshot_time` changes.
On Wed, Jul 26, 2017 at 9:08 AM, Gelinas, Derek <De...@comcast.com>
wrote:
> That’s not a terrible idea. Fewer changes to the code that way for sure,
> really just the DS interface page.
>
> > On Jul 26, 2017, at 10:30 AM, Nir Sopher <ni...@qwilt.com> wrote:
> >
> > Hi Derek,
> >
> > As discussed in the summit, we also see significant value in
> >
> > 1. DS Deployment Granularity - using DS individual config files.
> > 2. Delivery Service Configuration Versioning (DSCV) - separating the
> > "provisioning" from the "deployment".
> > 3. Improving the roll-out procedure, joining the capabilities #1 & #2
> >
> > We are on the same page with these needs:)
> >
> > However, as I see it, these are #1 & #2 are 2 separate features, each has
> > different requirements.
> > For example, for DSCV, I would suggest to manage the versions as
> standard
> > rows in the Delivery-Service table, side by side with the "hot" DS
> > configuration.
> > This will allow the existing code (with minor adjustments) to properly
> work
> > on these rows.
> > Furthermore, it also allows you to simply "restore" the DS "hot"
> > configuration to a specified revision.
> > It is also more resilient to DS table schema updates.
> >
> > I'll soon share, on another thread, a link to a "DSCV functional spec" I
> > was working on. It extends the presentation
> > <https://cwiki.apache.org/confluence/download/attachments/69407844/TC%
> 20Summit%20-%20Spring%202017%20-%20Self-Service.pptx?
> version=1&modificationDate=1495451091000&api=v2>
> > we
> > had in the summit.
> > I would appreciate any inputs to this spec.
> >
> > Nir
> >
> > On Tue, Jul 25, 2017 at 10:13 PM, Gelinas, Derek <
> Derek_Gelinas@comcast.com>
> > wrote:
> >
> >> At the summit, there was some talk about changing the manner in which we
> >> generate configuration files. The early stages of this idea had me
> >> creating large CDN definition files, but in the course of our
> discussion it
> >> became clear that we would be better served by creating delivery service
> >> configuration files instead. This would shift us from a
> server-generated
> >> implementation, as we have now, to generating the configuration files
> for
> >> the caches locally. The data for this would come from a new API that
> would
> >> provide the delivery service definitions in json format.
> >>
> >> What I’m envisioning is creating delivery service “snapshots” which are
> >> saved to the database as json objects. These snapshots would have the
> full
> >> range of information specific to the delivery service, including the
> new DS
> >> profiles. The database would store up to five of these objects per DS,
> and
> >> one DS object would be set to “active” through the UI or API.
> >>
> >> In this way, we could create multiple versions of a delivery service, or
> >> safely modify the definition currently “live” (but not necessarily
> active)
> >> in the database without changing the configuration in the field.
> >> Configuration would only be changed when the DS was saved and then that
> >> saved version was set to become active. In the reverse manner, existing
> >> saved delivery services could be restored to the live DB for
> modification.
> >>
> >> By divorcing the “live” db from the active configuration we prevent the
> >> possibility of accidental edits affecting the field, or
> edits-in-progress
> >> from being sent out prematurely when one person is working on a delivery
> >> service and another is queueing updates.
> >>
> >> Once set, it would be this active delivery service definition that would
> >> be provided to the rest of traffic ops for any delivery service
> >> operations. For config file generation, new API endpoints would be
> created
> >> that do the following:
> >>
> >> - List the delivery services and the active versions of each assigned to
> >> the specific server.
> >> - Provide the json object from the database when requested - I’m
> thinking
> >> that the endpoint would send the current active by default, or a
> specific
> >> version if specified.
> >>
> >> These definitions would be absurdly cacheable - we would not need to
> worry
> >> about sending stale data because each new version would have a
> completely
> >> different name - and so could be generated once and sent to thousands of
> >> caches with greatly reduced load on traffic ops. The load would
> consist of
> >> the initial creation of the json object, and the minimal serving of that
> >> object, so this would still result in greatly reduced load on the
> traffic
> >> ops host(s) even without the use of caching. Because of this, the new
> >> cache management service could check with traffic ops multiple times per
> >> minute for updates. Once a delivery service was changed, the new json
> >> would be downloaded and configs generated on the cache itself.
> >>
> >> Other benefits of the use of a cache manager service rather than the ORT
> >> script include:
> >>
> >> - Decreased load from logins - once the cache has logged in, it could
> use
> >> the cookie from the previous session and only re-login when that cookie
> has
> >> expired. we could also explore the use of certificates or keys instead,
> >> and eliminate logins altogether.
> >> - Multiple checks per minute rather than every X minutes - faster
> checks,
> >> more agile CDN.
> >> - Service could provide regular status updates to traffic ops, giving us
> >> the ability to keep an eye out for drastic shifts in i/o, unwanted
> >> behavior, problems with the ATS service, etc. This leads to building a
> >> traffic ops that can adapt itself on the fly to changing conditions and
> >> adjust accordingly.
> >> - Queue commands to run on the host from traffic ops. ATS restarts,
> >> system reboots, all manner of things could be triggered and scheduled
> right
> >> from traffic ops.
> >>
> >> Thoughts?
> >>
> >> Derek
>
>
Re: Delivery Service based config generation and Cache Manager
Posted by "Gelinas, Derek" <De...@comcast.com>.
That’s not a terrible idea. Fewer changes to the code that way for sure, really just the DS interface page.
> On Jul 26, 2017, at 10:30 AM, Nir Sopher <ni...@qwilt.com> wrote:
>
> Hi Derek,
>
> As discussed in the summit, we also see significant value in
>
> 1. DS Deployment Granularity - using DS individual config files.
> 2. Delivery Service Configuration Versioning (DSCV) - separating the
> "provisioning" from the "deployment".
> 3. Improving the roll-out procedure, joining the capabilities #1 & #2
>
> We are on the same page with these needs:)
>
> However, as I see it, these are #1 & #2 are 2 separate features, each has
> different requirements.
> For example, for DSCV, I would suggest to manage the versions as standard
> rows in the Delivery-Service table, side by side with the "hot" DS
> configuration.
> This will allow the existing code (with minor adjustments) to properly work
> on these rows.
> Furthermore, it also allows you to simply "restore" the DS "hot"
> configuration to a specified revision.
> It is also more resilient to DS table schema updates.
>
> I'll soon share, on another thread, a link to a "DSCV functional spec" I
> was working on. It extends the presentation
> <https://cwiki.apache.org/confluence/download/attachments/69407844/TC%20Summit%20-%20Spring%202017%20-%20Self-Service.pptx?version=1&modificationDate=1495451091000&api=v2>
> we
> had in the summit.
> I would appreciate any inputs to this spec.
>
> Nir
>
> On Tue, Jul 25, 2017 at 10:13 PM, Gelinas, Derek <De...@comcast.com>
> wrote:
>
>> At the summit, there was some talk about changing the manner in which we
>> generate configuration files. The early stages of this idea had me
>> creating large CDN definition files, but in the course of our discussion it
>> became clear that we would be better served by creating delivery service
>> configuration files instead. This would shift us from a server-generated
>> implementation, as we have now, to generating the configuration files for
>> the caches locally. The data for this would come from a new API that would
>> provide the delivery service definitions in json format.
>>
>> What I’m envisioning is creating delivery service “snapshots” which are
>> saved to the database as json objects. These snapshots would have the full
>> range of information specific to the delivery service, including the new DS
>> profiles. The database would store up to five of these objects per DS, and
>> one DS object would be set to “active” through the UI or API.
>>
>> In this way, we could create multiple versions of a delivery service, or
>> safely modify the definition currently “live” (but not necessarily active)
>> in the database without changing the configuration in the field.
>> Configuration would only be changed when the DS was saved and then that
>> saved version was set to become active. In the reverse manner, existing
>> saved delivery services could be restored to the live DB for modification.
>>
>> By divorcing the “live” db from the active configuration we prevent the
>> possibility of accidental edits affecting the field, or edits-in-progress
>> from being sent out prematurely when one person is working on a delivery
>> service and another is queueing updates.
>>
>> Once set, it would be this active delivery service definition that would
>> be provided to the rest of traffic ops for any delivery service
>> operations. For config file generation, new API endpoints would be created
>> that do the following:
>>
>> - List the delivery services and the active versions of each assigned to
>> the specific server.
>> - Provide the json object from the database when requested - I’m thinking
>> that the endpoint would send the current active by default, or a specific
>> version if specified.
>>
>> These definitions would be absurdly cacheable - we would not need to worry
>> about sending stale data because each new version would have a completely
>> different name - and so could be generated once and sent to thousands of
>> caches with greatly reduced load on traffic ops. The load would consist of
>> the initial creation of the json object, and the minimal serving of that
>> object, so this would still result in greatly reduced load on the traffic
>> ops host(s) even without the use of caching. Because of this, the new
>> cache management service could check with traffic ops multiple times per
>> minute for updates. Once a delivery service was changed, the new json
>> would be downloaded and configs generated on the cache itself.
>>
>> Other benefits of the use of a cache manager service rather than the ORT
>> script include:
>>
>> - Decreased load from logins - once the cache has logged in, it could use
>> the cookie from the previous session and only re-login when that cookie has
>> expired. we could also explore the use of certificates or keys instead,
>> and eliminate logins altogether.
>> - Multiple checks per minute rather than every X minutes - faster checks,
>> more agile CDN.
>> - Service could provide regular status updates to traffic ops, giving us
>> the ability to keep an eye out for drastic shifts in i/o, unwanted
>> behavior, problems with the ATS service, etc. This leads to building a
>> traffic ops that can adapt itself on the fly to changing conditions and
>> adjust accordingly.
>> - Queue commands to run on the host from traffic ops. ATS restarts,
>> system reboots, all manner of things could be triggered and scheduled right
>> from traffic ops.
>>
>> Thoughts?
>>
>> Derek
Re: Delivery Service based config generation and Cache Manager
Posted by Nir Sopher <ni...@qwilt.com>.
Hi Derek,
As discussed in the summit, we also see significant value in
1. DS Deployment Granularity - using DS individual config files.
2. Delivery Service Configuration Versioning (DSCV) - separating the
"provisioning" from the "deployment".
3. Improving the roll-out procedure, joining the capabilities #1 & #2
We are on the same page with these needs:)
However, as I see it, these are #1 & #2 are 2 separate features, each has
different requirements.
For example, for DSCV, I would suggest to manage the versions as standard
rows in the Delivery-Service table, side by side with the "hot" DS
configuration.
This will allow the existing code (with minor adjustments) to properly work
on these rows.
Furthermore, it also allows you to simply "restore" the DS "hot"
configuration to a specified revision.
It is also more resilient to DS table schema updates.
I'll soon share, on another thread, a link to a "DSCV functional spec" I
was working on. It extends the presentation
<https://cwiki.apache.org/confluence/download/attachments/69407844/TC%20Summit%20-%20Spring%202017%20-%20Self-Service.pptx?version=1&modificationDate=1495451091000&api=v2>
we
had in the summit.
I would appreciate any inputs to this spec.
Nir
On Tue, Jul 25, 2017 at 10:13 PM, Gelinas, Derek <De...@comcast.com>
wrote:
> At the summit, there was some talk about changing the manner in which we
> generate configuration files. The early stages of this idea had me
> creating large CDN definition files, but in the course of our discussion it
> became clear that we would be better served by creating delivery service
> configuration files instead. This would shift us from a server-generated
> implementation, as we have now, to generating the configuration files for
> the caches locally. The data for this would come from a new API that would
> provide the delivery service definitions in json format.
>
> What I’m envisioning is creating delivery service “snapshots” which are
> saved to the database as json objects. These snapshots would have the full
> range of information specific to the delivery service, including the new DS
> profiles. The database would store up to five of these objects per DS, and
> one DS object would be set to “active” through the UI or API.
>
> In this way, we could create multiple versions of a delivery service, or
> safely modify the definition currently “live” (but not necessarily active)
> in the database without changing the configuration in the field.
> Configuration would only be changed when the DS was saved and then that
> saved version was set to become active. In the reverse manner, existing
> saved delivery services could be restored to the live DB for modification.
>
> By divorcing the “live” db from the active configuration we prevent the
> possibility of accidental edits affecting the field, or edits-in-progress
> from being sent out prematurely when one person is working on a delivery
> service and another is queueing updates.
>
> Once set, it would be this active delivery service definition that would
> be provided to the rest of traffic ops for any delivery service
> operations. For config file generation, new API endpoints would be created
> that do the following:
>
> - List the delivery services and the active versions of each assigned to
> the specific server.
> - Provide the json object from the database when requested - I’m thinking
> that the endpoint would send the current active by default, or a specific
> version if specified.
>
> These definitions would be absurdly cacheable - we would not need to worry
> about sending stale data because each new version would have a completely
> different name - and so could be generated once and sent to thousands of
> caches with greatly reduced load on traffic ops. The load would consist of
> the initial creation of the json object, and the minimal serving of that
> object, so this would still result in greatly reduced load on the traffic
> ops host(s) even without the use of caching. Because of this, the new
> cache management service could check with traffic ops multiple times per
> minute for updates. Once a delivery service was changed, the new json
> would be downloaded and configs generated on the cache itself.
>
> Other benefits of the use of a cache manager service rather than the ORT
> script include:
>
> - Decreased load from logins - once the cache has logged in, it could use
> the cookie from the previous session and only re-login when that cookie has
> expired. we could also explore the use of certificates or keys instead,
> and eliminate logins altogether.
> - Multiple checks per minute rather than every X minutes - faster checks,
> more agile CDN.
> - Service could provide regular status updates to traffic ops, giving us
> the ability to keep an eye out for drastic shifts in i/o, unwanted
> behavior, problems with the ATS service, etc. This leads to building a
> traffic ops that can adapt itself on the fly to changing conditions and
> adjust accordingly.
> - Queue commands to run on the host from traffic ops. ATS restarts,
> system reboots, all manner of things could be triggered and scheduled right
> from traffic ops.
>
> Thoughts?
>
> Derek