You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@trafficcontrol.apache.org by "Durfey, Ryan" <Ry...@comcast.com> on 2017/04/27 16:19:31 UTC
Configuration Management - Future State Design for Self Service
As we move into Self-Service discussions, Configuration Management needs to be discussed in greater depth. I wanted to get the conversation started and get feedback from everyone on what the future state should look like and how we get there from our current state.
TO 2.1 Changes In Progress
Several changes are underway with TO 2.1 that will make a significant impact to current challenges. Derek G., keep me honest.
1. Invalidation is being decoupled from configuration updates
* Allows for invalidations on 1 min CRON jobs which should take approximately 3 min total to apply to mid and edge tier in succession.
* Allows configs to be applied to mid tier and edge tier simultaneously reducing time by half.
2. Reduction in the overall number of config files generated per configuration update
* Allows configs to be applied on tighter than the current 15 min CRON intervals, though not sure how rapidly yet. Will require testing.
This still leaves us with a monolithic configuration management process which contains all Server and Service configs. This presents some challenges to individual Self-Service.
Future State (v3?) Ideas
1. Separate out Server configs from Service configs so they can be:
* Tested separately
* Applied separately
* Logged & reported separately
* Reverted separately
* Non-blocking when issues are encountered.
2. Separate out individual Service configs for same reasons as above.
3. Allow for instant push of new configs, no CRON wait time.
4. Allow for Service builds and changes to be staged for initial testing prior to production roll out.
5. Allow for rollback of config changes in both staging and production environments.
6. Log all Service changes so that a Tenant User can pull back a history of all changes related to their services through the API.
Ryan Durfey
Sr. Product Manager - CDN | Comcast Technology Solutions
1899 Wynkoop Ste. 550 | Denver, CO 8020
M | 303-524-5099
ryan_durfey@comcast.com<ma...@comcast.com>
24x7 CDN Support: 866-405-2993 or cdn_support@comcast.com<ma...@comcast.com>
Re: Configuration Management - Future State Design for Self Service
Posted by Nir Sopher <ni...@qwilt.com>.
Hi all,
First of all +1 for the described efforts. The delivery service deployment
time is an issue that bothers us as well.
I personally believe that the decoupling of servers config and individual
service configs is an important step towards "self-service", where
ultimately the flow of operations on an individual DS is completely
decoupled from of other DSs. Beyond the values already listed, I also see
the value of improved troubleshooting.
As I see it, much of the value provided via the configuration decoupling,
can be achieved working with "delivery-service configuration versioning".
It allows testing of individual DSs, DS level apply granularity, better
auditing and troubleshooting, and even a "no CRON wait" rollback.
Furthermore, engineering wise, one may consider the "delivery-service
configuration versioning" as a building blocks on the way to services
decoupling and efficient deployment. It allows the system to simply test
what is deployed on which cache, and the cache to identify changes on a DS
granularity.
I'll open a separate thread on this feature, and we will probably discuss
it on the summit.
Nir
On Apr 28, 2017 5:43 PM, "Durfey, Ryan" <Ry...@comcast.com> wrote:
> Great feedback Eric. My response is below.
>
> Ryan Durfey M | 303-524-5099
> 24x7 CDN Support: 866-405-2993 or cdn_support@comcast.com
>
>
> On 4/28/17, 8:18 AM, "Eric Friedrich (efriedri)" <ef...@cisco.com>
> wrote:
>
>
> > On Apr 27, 2017, at 12:19 PM, Durfey, Ryan <Ry...@comcast.com>
> wrote:
> >
> > As we move into Self-Service discussions, Configuration Management
> needs to be discussed in greater depth. I wanted to get the conversation
> started and get feedback from everyone on what the future state should look
> like and how we get there from our current state.
> >
> > TO 2.1 Changes In Progress
> > Several changes are underway with TO 2.1 that will make a
> significant impact to current challenges. Derek G., keep me honest.
> >
> > 1. Invalidation is being decoupled from configuration updates
> > * Allows for invalidations on 1 min CRON jobs which should
> take approximately 3 min total to apply to mid and edge tier in succession.
> > * Allows configs to be applied to mid tier and edge tier
> simultaneously reducing time by half.
> > 2. Reduction in the overall number of config files generated per
> configuration update
> > * Allows configs to be applied on tighter than the current 15
> min CRON intervals, though not sure how rapidly yet. Will require testing.
> > This still leaves us with a monolithic configuration management
> process which contains all Server and Service configs. This presents some
> challenges to individual Self-Service.
> >
> > Future State (v3?) Ideas
> >
> > 1. Separate out Server configs from Service configs so they can be:
> > * Tested separately
> > * Applied separately
> > * Logged & reported separately
> > * Reverted separately
> > * Non-blocking when issues are encountered.
> > 2. Separate out individual Service configs for same reasons as
> above.
> > 3. Allow for instant push of new configs, no CRON wait time.
> EF> We have to be careful with Pushes. Depending on how its
> implemented, this may open up a new API (and a new attack surface) on
> publicly accessible caches.
>
> Any reason to believe a 1 minute cronjob would not be fast enough?
>
> RD: I think instant push is the ideal especially if you need to roll
> something back that is broken. But the reality is that a 1 min cronjob
> would be pretty close and I wouldn’t have issues with that. I think we
> attempt to articulate the ideal situation and any time we can get an 80/20
> solution that gets us close with significantly less headaches we are very
> happy with that.
>
>
> > 4. Allow for Service builds and changes to be staged for initial
> testing prior to production roll out.
> > 5. Allow for rollback of config changes in both staging and
> production environments.
> > 6. Log all Service changes so that a Tenant User can pull back a
> history of all changes related to their services through the API.
> >
> >
> > Ryan Durfey
> > Sr. Product Manager - CDN | Comcast Technology Solutions
> > 1899 Wynkoop Ste. 550 | Denver, CO 8020
> > M | 303-524-5099 <(303)%20524-5099>
> > ryan_durfey@comcast.com<ma...@comcast.com>
> > 24x7 CDN Support: 866-405-2993 or cdn_support@comcast.com<mailto:
> cdn_support@comcast.com>
> >
>
>
>
>
>
Re: Configuration Management - Future State Design for Self Service
Posted by "Durfey, Ryan" <Ry...@comcast.com>.
Great feedback Eric. My response is below.
Ryan Durfey M | 303-524-5099
24x7 CDN Support: 866-405-2993 or cdn_support@comcast.com
On 4/28/17, 8:18 AM, "Eric Friedrich (efriedri)" <ef...@cisco.com> wrote:
> On Apr 27, 2017, at 12:19 PM, Durfey, Ryan <Ry...@comcast.com> wrote:
>
> As we move into Self-Service discussions, Configuration Management needs to be discussed in greater depth. I wanted to get the conversation started and get feedback from everyone on what the future state should look like and how we get there from our current state.
>
> TO 2.1 Changes In Progress
> Several changes are underway with TO 2.1 that will make a significant impact to current challenges. Derek G., keep me honest.
>
> 1. Invalidation is being decoupled from configuration updates
> * Allows for invalidations on 1 min CRON jobs which should take approximately 3 min total to apply to mid and edge tier in succession.
> * Allows configs to be applied to mid tier and edge tier simultaneously reducing time by half.
> 2. Reduction in the overall number of config files generated per configuration update
> * Allows configs to be applied on tighter than the current 15 min CRON intervals, though not sure how rapidly yet. Will require testing.
> This still leaves us with a monolithic configuration management process which contains all Server and Service configs. This presents some challenges to individual Self-Service.
>
> Future State (v3?) Ideas
>
> 1. Separate out Server configs from Service configs so they can be:
> * Tested separately
> * Applied separately
> * Logged & reported separately
> * Reverted separately
> * Non-blocking when issues are encountered.
> 2. Separate out individual Service configs for same reasons as above.
> 3. Allow for instant push of new configs, no CRON wait time.
EF> We have to be careful with Pushes. Depending on how its implemented, this may open up a new API (and a new attack surface) on publicly accessible caches.
Any reason to believe a 1 minute cronjob would not be fast enough?
RD: I think instant push is the ideal especially if you need to roll something back that is broken. But the reality is that a 1 min cronjob would be pretty close and I wouldn’t have issues with that. I think we attempt to articulate the ideal situation and any time we can get an 80/20 solution that gets us close with significantly less headaches we are very happy with that.
> 4. Allow for Service builds and changes to be staged for initial testing prior to production roll out.
> 5. Allow for rollback of config changes in both staging and production environments.
> 6. Log all Service changes so that a Tenant User can pull back a history of all changes related to their services through the API.
>
>
> Ryan Durfey
> Sr. Product Manager - CDN | Comcast Technology Solutions
> 1899 Wynkoop Ste. 550 | Denver, CO 8020
> M | 303-524-5099
> ryan_durfey@comcast.com<ma...@comcast.com>
> 24x7 CDN Support: 866-405-2993 or cdn_support@comcast.com<ma...@comcast.com>
>
Re: Configuration Management - Future State Design for Self Service
Posted by "Eric Friedrich (efriedri)" <ef...@cisco.com>.
> On Apr 27, 2017, at 12:19 PM, Durfey, Ryan <Ry...@comcast.com> wrote:
>
> As we move into Self-Service discussions, Configuration Management needs to be discussed in greater depth. I wanted to get the conversation started and get feedback from everyone on what the future state should look like and how we get there from our current state.
>
> TO 2.1 Changes In Progress
> Several changes are underway with TO 2.1 that will make a significant impact to current challenges. Derek G., keep me honest.
>
> 1. Invalidation is being decoupled from configuration updates
> * Allows for invalidations on 1 min CRON jobs which should take approximately 3 min total to apply to mid and edge tier in succession.
> * Allows configs to be applied to mid tier and edge tier simultaneously reducing time by half.
> 2. Reduction in the overall number of config files generated per configuration update
> * Allows configs to be applied on tighter than the current 15 min CRON intervals, though not sure how rapidly yet. Will require testing.
> This still leaves us with a monolithic configuration management process which contains all Server and Service configs. This presents some challenges to individual Self-Service.
>
> Future State (v3?) Ideas
>
> 1. Separate out Server configs from Service configs so they can be:
> * Tested separately
> * Applied separately
> * Logged & reported separately
> * Reverted separately
> * Non-blocking when issues are encountered.
> 2. Separate out individual Service configs for same reasons as above.
> 3. Allow for instant push of new configs, no CRON wait time.
EF> We have to be careful with Pushes. Depending on how its implemented, this may open up a new API (and a new attack surface) on publicly accessible caches.
Any reason to believe a 1 minute cronjob would not be fast enough?
> 4. Allow for Service builds and changes to be staged for initial testing prior to production roll out.
> 5. Allow for rollback of config changes in both staging and production environments.
> 6. Log all Service changes so that a Tenant User can pull back a history of all changes related to their services through the API.
>
>
> Ryan Durfey
> Sr. Product Manager - CDN | Comcast Technology Solutions
> 1899 Wynkoop Ste. 550 | Denver, CO 8020
> M | 303-524-5099
> ryan_durfey@comcast.com<ma...@comcast.com>
> 24x7 CDN Support: 866-405-2993 or cdn_support@comcast.com<ma...@comcast.com>
>