Re: Traffic Ops Configuration Management - Proposed Webex Meeting Tue 8/29/17 at 2 pm EST

["Durfey, Ryan" <Ry...@comcast.com>]

Going to shift my proposed date to Tue 8/29 at 2pm per feedback from the Qwilt team.
R

From: Ryan Durfey <Ry...@cable.comcast.com>
Date: Tuesday, August 15, 2017 at 7:42 AM
To: "dev@trafficcontrol.incubator.apache.org" <de...@trafficcontrol.incubator.apache.org>
Subject: Traffic Ops Configuration Management - Proposed Webex Meeting Friday 8/25/17 at 11 am EST

All,

The email discussion on configuration management has gone cold and we are no closer to hashing out a final direction on several key issues.  I am planning to host a 90 minute webex for the open source community to organize and debate the key issues.

Please let me know if you have feedback on the date/time so I can adjust if needed.

 Tentative Proposal – Webex on Configuration Management

  1.  Date: Tuesday 8/29/17
  2.  Time:  2 PM EST / 7 PM GMT
  3.  Length: 90 min
  4.  Agenda: Additional Discussion Section:  https://cwiki.apache.org/confluence/display/TC/Configuration+Management

Please feel free to respond if you see something that needs to be added.

Ryan Durfey
Sr. Product Manager - CDN | Comcast Technology Solutions
1899 Wynkoop Ste. 550 | Denver, CO 80202
M | 303-524-5099
ryan_durfey@comcast.com<ma...@comcast.com>
CDN Support (24x7): 866-405-2993 or cdn_support@comcast.com<ma...@comcast.com>

Re: Traffic Ops Configuration Management - Proposed Webex Meeting Tue 8/29/17 at 2 pm EST

["Mills, Matthew" <Ma...@comcast.com>]

Hello all,

Following up on Ryan’s thread to provide a proposal for this discussion of config management, I tried to not make it a giant wall of text… but I’m not sure I succeeded. I wanted to also provide a proposed database schema but after a series of unfortunate events it no longer exists. Feel free to reach out on Traffic Control slack (or here) with any questions (or complaints). I also realize that this is as a whole is suggesting a fairly drastic change to how Traffic Control works today, and I think most of what is described can be implemented in a fairly incremental manner.

Thanks,
Matt.


The path to self-service (as I see it):

1. Lua Rules Engine
2. Incremental Go replacement for ORT that implements instant (<5s) reconfig and uses generic APIs (Not ATS specific)
3. UI for Configuring Lua rules engine
4. Generic DS parameters
5. Client side Commit (OR) Get rid of things that need atomic changes



1. Lua rules engine (More detail on what this is below)
    Lua rules engine is a mechanism to define (some) delivery service configuration
    in a generic way that can be "ported" to non-ATS caching proxies (with some work).
    It is also intended to provide:
    - A mechanism to easily and quickly iterate new DS-specific functionality in Lua (I'd like to return 403s for any URL ending in .m3u8 on Tuesdays)
    - Isolation between delivery services (No more DS specific configuration breaking entire server)
    - A less confusing mechanism for self service users to add dynamic functionality themselves (Most users don't/won't understand regexes, header rewrite)
    - More accessible debug mechanism (secret header/param that enables ATS debugging for that transaction)
    - A functional unit that can be versioned apart from the entire DS (not sure if good idea?)
    - Can be used standalone (without any components described below) as a normal lua plugin.
    - "Simple mode" that hides more complicated config / provides normal stuff people are interested in
    - Would be theoretically possible to provide config dynamically (Redis, memcache, etc) which wouldn't require an ATS reload

2. Go incremental replacement for ORT
    I think this makes sense to talk about in this discussion because I think it should be written
    alongside the Go Traffic Ops API replacement. While I think it's possible to add the functionality
    Lua rules engine would need to ORT, I think ORT uses data in the wrong way. Traffic Ops APIs should provide
    the most generic data (read: non-ATS specific) possible the actual turning that data into config should happen
    on the server so that all TO functionality is abstracted away from the server and anyone can write
    an implementation for their caching proxy of preference without knowing all the internals of TO.

    - Traffic Ops events out any server or DS configuration that has changed when it changes
      Possible ways to implement: JMS (Kafka?), Reworking existing changelog table, New "notifications" table?
      (Up to implementor, my preference is enhancing changelog)
    - Instant (< 5s) changes
      Possible ways to implement: If-Modified-Since polling, HTTP Long poll, Websocket (+1), JMS
    - Modules/Packages/Plugins/Whatever "subscribe" to the event types they are interested
    - Once a subscribed event is received module is triggered to execute its reconfiguration (Using provided data or by going and getting data from TO)
    - Modules would be specific to 1. Type of Cache (ATS, Nginx, Varnish, Bob's discount cache software) 2. Cache-specific "config section" (remap.config, plugin.config)
    - First module would probably be one that gets notified of "Queued updates" being checked, and executes ORT, providing near-instant ORT as step 1.
    - As API routes are rebuilt in go TO API, they can be implemented here and removed/disabled in ORT (IE: implementation is entirely incremental)

3. UI for configuring lua rules engine
    - A way for self-service users to see/add/modify rules
    - I am not a UI developer, but in my mind, I see a list of rules that you can "expand" to see the conditions and actions, with all 3 having + boxes to add more
    - I think there would be an option toggle that toggles a DS between current and rules engine mode (which would get rid of most of the text boxes in DS config)
    - In my opinion, this component should be entirely data driven, with all the defined condition types and action types defined by the administrator so that adding an action is as simple as writing and deploying some lua, then adding that action to the config so users can use it.

4. Some way to do generic parameters on DSes WITH data integrity (for things that don’t fit into Rules)
    - I don't have anything for this, but I think it needs to exist...
    - Probably needs to be data driven (Can be configured by an admin-type user at run time)
    - -1 on DS profile/parameters, they're FAR too generic and confusing, feels like it's become very difficult to maintain and understand (IMO, one of the big hurdles for new users)
    - But maybe it could be an enhancement on top of the existing profiles/parameters?
    - Ability to make fields required/non-deletable fields
    - Description and help text accessible in UI
    - Has to have validation of input data (Input must match regex?, some way to plugin "validation" api for more complex types)
    - Generically implemented HTML/UI templating for input types would be awesome.

5a. Client side commit (OR)
    Queued updates and snapshots need to go away or dramatically change for Self-service to ever work.
    - Changes made in the UI are "held" in local storage until the user hits a "commit" button
    - All pending changes are "saved" at that point
    - User can see all changes as a group before they happen

5b. Let’s get rid of things that need to happen in an atomic block of changes
    - I don't understand the technical challenges here, so not going to comment further.





----------------------------------------




What is this lua rules engine thing:
    Lua rules engine is a proof of concept I've been working on for a different way of doing DS config in TO that really has two pieces:
    1. Data model that allows you to define DS specific functionality in a generic way
    2. Lua code that runs on your caching proxy of choice that implements rule functionality

Details about Lua rules engine:
 - Delivery services have Rules
 - Rules have conditions and actions
 - If all conditions evaluate to true, actions execute
 - Conditions and actions can both define their target (client_request, server_request, server_response, client_response)
 - Rules define their name, and hook point (This one is a bit ATS specific... might need to be reworked)
 - Conditions define their type, target, arguments, operator, and value
 - Actions define their type, target, and arguments
 - Condition types implemented on the Proof of Concept:
    - is_internal_request
    - uri
    - url
    - header
    - incoming_port
    - scheme
  - Action types implemented on the Proof of Concept:
    - http_redirect
    - set_header
    - remove_header
    - set_response
    - set_uri
    - set_uri_args
    - set_uri_params
    - set_url_host
    - set_url_scheme
    - set_status
  - Operators implemented and planned on PoC:
    - ==
    - !=
    - regex (Not implemented, needs https://github.com/portl4t/ts-lua/blob/master/src/ts_lua_regex.c or alternate Lua module)
    - store (Put value into variable for future variable replacement)
    - starts_with
    - ends_with
    - contains



Example Config (pulled from proof of concept config, actual implementation doesn't need to look like this):
 {
    name = 'test_set_header-client_response',
    hook_point = 'client_response',
    conditions =
    {
        {
            type = 'uri',
            target = 'client_request',
            args = {},
            operator = '==',
            value = '/test_set_header-client_response',
        },
    },
    actions =
    {
        {
            type = 'set_header',
            target = 'client_response',
            args = {name='X-test-header', value='THIS IS ONLY A TEST!'},
        },
    }
},