[GitHub] [trafficcontrol] zrhoffman commented on a change in pull request #6358: Traffic Router Ultimate Test Harness blueprint

[GitBox <gi...@apache.org>]

zrhoffman commented on a change in pull request #6358:
URL: https://github.com/apache/trafficcontrol/pull/6358#discussion_r777690754



##########
File path: blueprints/tr-load-tests.md
##########
@@ -0,0 +1,296 @@
+<!--
+Licensed to the Apache Software Foundation (ASF) under one
+or more contributor license agreements.  See the NOTICE file
+distributed with this work for additional information
+regarding copyright ownership.  The ASF licenses this file
+to you under the Apache License, Version 2.0 (the
+"License"); you may not use this file except in compliance
+with the License.  You may obtain a copy of the License at
+
+    http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing,
+software distributed under the License is distributed on an
+"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+KIND, either express or implied.  See the License for the
+specific language governing permissions and limitations
+under the License.
+-->
+# Traffic Router Ultimate Test Harness <!-- a concise title for the new feature this blueprint will describe -->
+
+## Problem Description
+<!--
+*What* is being asked for?
+*Why* is this necessary?
+*How* will this be used?
+-->
+As the entrypoint of an Apache Traffic Control CDN, Traffic Router is the most exposed, most critical component, and it must be able to route traffic quickly and at a very high rate. Although Traffic Router has met this requirement in the past, if Traffic Router undergoes no performance tests as it evolves, there is no guarantee that its performance will not decline with time.
+
+Important times to run performance tests:
+
+* When traffic volume to your CDN changes significantly
+* When making harware changes to the server hosting the Traffic Router instance
+* When upgrading Traffic Router to a new Apache Traffic Control version
+* When developing or maintaining a Traffic Router feature that could have an impact on any aspect of Traffic Router performance, including before, during, and after pull request review
+* When a commit that modifies Traffic Router is pushed to a GitHub branch, for all Apache Traffic Control branches
+
+A load test for Delivery Services exists in the project at [`/test/router`](https://github.com/apache/trafficcontrol/tree/RELEASE-6.0.1/test/router), but
+- It has not been maintained over time, currently does not work
+- Only tests HTTP Delivery Services
+- Does not support testing Coverage Zone Maps
+- Does not use the TO Client Library for requests to Traffic Ops
+- Prompts only for the number of requests to make to Delivery Services, not a length of time to run the test
+- Does not fail if some minimum threshold of requests per second is not met
+- Is not configurable in other ways, such as length of paths generated for HTTP requests to Delivery Service, which client IP address to use
+
+## Proposed Change
+<!--
+*How* will this be implemented (at a high level)?
+-->
+The Traffic Router Ultimate Test Harness will include an end-to-end performance test suite verify that the features of Traffic Router meet expected performance thresholds, as well as additional end-to-end tests of other Traffic Router features.
+
+The TR Ultimate Test Harness may extend [`/test/router`](https://github.com/apache/trafficcontrol/tree/RELEASE-6.0.1/test/router) where possible, but it should not limit itself for that secondary goal.
+
+### Traffic Portal Impact
+<!--
+*How* will this impact Traffic Portal?
+What new UI changes will be required?
+Will entirely new pages/views be necessary?
+Will a new field be added to an existing form?
+How will the user interact with the new UI changes?
+-->
+No Traffic Portal impact is anticipated.
+
+### Traffic Ops Impact
+<!--
+*How* will this impact Traffic Ops (at a high level)?
+-->
+The Operations-level TR Ultimate Test Harness user will update the status of any Traffic Router that is currently undergoing load testing to `"OFFLINE"` in Traffic Ops. It should also update the status of that Traffic Router to its previous status once the load testing is complete, regardless of whether the test failed or succeeded.
+
+#### REST API Impact
+<!--
+*How* will this impact the Traffic Ops REST API?
+
+What new endpoints will be required?
+How will existing endpoints be changed?
+What will the requests and responses look like?
+What fields are required or optional?
+What are the defaults for optional fields?
+What are the validation constraints?
+-->
+No Traffic Ops REST API impact is anticipated.
+
+#### Client Impact
+<!--
+*How* will this impact Traffic Ops REST API clients (Go, Python, Java)?
+
+If new endpoints are required, will corresponding client methods be added?
+-->
+Clients importing the `github.com/apache/trafficcontrol/lib/go-tc` package will optionally be able to import a constant for `X-MM-Client-IP`, a request header Traffic Router to specify to Traffic Router the IP address to use to geolocate that client:  
+https://github.com/apache/trafficcontrol/blob/1ed2964d16618aeebef142b01a538336a44d07dd/traffic_router/core/src/main/java/org/apache/traffic_control/traffic_router/core/request/HTTPRequest.java#L29
+
+Additionally, a struct used to unmarshall a Coverage Zone File could be placed in `lib/go-tc`.
+
+#### Data Model / Database Impact
+<!--
+*How* will this impact the Traffic Ops data model?
+*How* will this impact the Traffic Ops database schema?
+
+What changes to the lib/go-tc structs will be required?
+What new tables and columns will be required?
+How will existing tables and columns be changed?
+What are the column data types and modifiers?
+What are the FK references and constraints?
+-->
+No Data Model impact is anticipated.
+
+### Cache Config Impact
+<!--
+*How* will this impact ORT?
+-->
+No Cache Config impact is anticipated.
+
+### Traffic Monitor Impact
+<!--
+*How* will this impact Traffic Monitor?
+
+Will new profile parameters be required?
+-->
+No Traffic Monitor impact is anticipated.
+
+### Traffic Router Impact
+<!--
+*How* will this impact Traffic Router?
+
+Will new profile parameters be required?
+How will the CRConfig be changed?
+How will changes in Traffic Ops data be reflected in the CRConfig?
+Will Traffic Router remain backwards-compatible with old CRConfigs?
+Will old Traffic Routers remain forwards-compatible with new CRConfigs?
+-->
+The addition of the TR Ultimate Test Harness themselves will not change Traffic Router functionality in any way. For visibility, however, the TR Ultimate Test Harness should reside in a directory within the `traffic_router` directory. This will be the first time since 545929f7cc that Golang sources will exist in the `traffic_router` directory, so any assumption that all sources within the `traffic_router` directory directly impact Traffic Router's ability to compile should be abandoned.
+
+The TR Ultimate Test Harness should not be included in the Traffic Router RPM, as it is meant to be run on a host separate from Traffic Routers.
+
+### Traffic Stats Impact
+<!--
+*How* will this impact Traffic Stats?
+-->
+No Traffic Stats impact is anticipated.
+
+### Traffic Vault Impact
+<!--
+*How* will this impact Traffic Vault?
+
+Will there be any new data stored in or removed from Riak?
+Will there be any changes to the Riak requests and responses?
+-->
+No Traffic Vault impact is anticipated.
+
+### Documentation Impact
+<!--
+*How* will this impact the documentation?
+
+What new documentation will be required?
+What existing documentation will need to be updated?
+-->
+Instructions for using the Traffic Router Ultimate Test Harness should be added to the documentation. This should include:
+- Small rationale for inclusion of TR Ultimate Test Harness
+- Setup instructions
+    * The permissions that a user running the Traffic Router Test Harness should have:
+        - CDN snapshots
+        - CDN information
+        - information about Traffic Router-type Servers in those CDNs
+        - Type information
+        - Delivery Service information
+    - Documentation of each CLI option
+    - Example commands
+
+### Testing Impact
+<!--
+*How* will this impact testing?
+
+What is the high-level test plan?
+How should this be tested?
+Can this be tested within the existing test frameworks?
+How should the existing frameworks be enhanced in order to test this properly?
+-->
+#### Load Tests
+The Router Ultimate Test Harness should include a load test for HTTP-routed Delivery Services and for DNS-routed Delivery Services
+
+##### Load Test Options

Review comment:
       > Are these going to be CLI options?
   
   No
   
   > Will there be a config file?
   
   Config file structure clarified in c0bd076c46
   
   > If we extend the test harness with more than just load testing, would each different type of test have its own config options and/or file?
   
   Each test can specify its own options, but options applying to more than one test can go under `"all"`.




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@trafficcontrol.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org