You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@trafficcontrol.apache.org by GitBox <gi...@apache.org> on 2021/11/18 14:03:43 UTC
[GitHub] [trafficcontrol] zrhoffman opened a new pull request #6358: Traffic Router Ultimate Test Harness blueprint
zrhoffman opened a new pull request #6358:
URL: https://github.com/apache/trafficcontrol/pull/6358
This Pull Request adds a blueprint for the Traffic Router Ultimate Test Harness.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@trafficcontrol.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [trafficcontrol] zrhoffman commented on a change in pull request #6358: Traffic Router Ultimate Test Harness blueprint
Posted by GitBox <gi...@apache.org>.
zrhoffman commented on a change in pull request #6358:
URL: https://github.com/apache/trafficcontrol/pull/6358#discussion_r767052901
##########
File path: blueprints/tr-load-tests.md
##########
@@ -0,0 +1,296 @@
+<!--
+Licensed to the Apache Software Foundation (ASF) under one
+or more contributor license agreements. See the NOTICE file
+distributed with this work for additional information
+regarding copyright ownership. The ASF licenses this file
+to you under the Apache License, Version 2.0 (the
+"License"); you may not use this file except in compliance
+with the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing,
+software distributed under the License is distributed on an
+"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+KIND, either express or implied. See the License for the
+specific language governing permissions and limitations
+under the License.
+-->
+# Traffic Router Ultimate Test Harness <!-- a concise title for the new feature this blueprint will describe -->
+
+## Problem Description
+<!--
+*What* is being asked for?
+*Why* is this necessary?
+*How* will this be used?
+-->
+As the entrypoint of an Apache Traffic Control CDN, Traffic Router is the most exposed, most critical component, and it must be able to route traffic quickly and at a very high rate. Although Traffic Router has met this requirement in the past, if Traffic Router undergoes no performance tests as it evolves, there is no guarantee that its performance will not decline with time.
+
+Important times to run performance tests:
+
+* When traffic volume to your CDN changes significantly
+* When making harware changes to the server hosting the Traffic Router instance
+* When upgrading Traffic Router to a new Apache Traffic Control version
+* When developing or maintaining a Traffic Router feature that could have an impact on any aspect of Traffic Router performance, including before, during, and after pull request review
+* When a commit that modifies Traffic Router is pushed to a GitHub branch, for all Apache Traffic Control branches
+
+A load test for Delivery Services exists in the project at [`/test/router`](https://github.com/apache/trafficcontrol/tree/RELEASE-6.0.1/test/router), but
+- It has not been maintained over time, currently does not work
+- Only tests HTTP Delivery Services
+- Does not support testing Coverage Zone Maps
+- Does not use the TO Client Library for requests to Traffic Ops
+- Prompts only for the number of requests to make to Delivery Services, not a length of time to run the test
+- Does not fail if some minimum threshold of requests per second is not met
+- Is not configurable in other ways, such as length of paths generated for HTTP requests to Delivery Service, which client IP address to use
+
+## Proposed Change
+<!--
+*How* will this be implemented (at a high level)?
+-->
+The Traffic Router Ultimate Test Harness will include an end-to-end performance test suite verify that the features of Traffic Router meet expected performance thresholds, as well as additional end-to-end tests of other Traffic Router features.
+
+The TR Ultimate Test Harness may extend [`/test/router`](https://github.com/apache/trafficcontrol/tree/RELEASE-6.0.1/test/router) where possible, but it should not limit itself for that secondary goal.
+
+### Traffic Portal Impact
+<!--
+*How* will this impact Traffic Portal?
+What new UI changes will be required?
+Will entirely new pages/views be necessary?
+Will a new field be added to an existing form?
+How will the user interact with the new UI changes?
+-->
+No Traffic Portal impact is anticipated.
+
+### Traffic Ops Impact
+<!--
+*How* will this impact Traffic Ops (at a high level)?
+-->
+The Operations-level TR Ultimate Test Harness user will update the status of any Traffic Router that is currently undergoing load testing to `"OFFLINE"` in Traffic Ops. It should also update the status of that Traffic Router to its previous status once the load testing is complete, regardless of whether the test failed or succeeded.
Review comment:
> Also, I'm not positive, but I _think_ setting the TR under test to OFFLINE would affect its responses in that it would exclude itself from any responses.
You're right, setting the TR status to OFFLINE would mean needing a separate ONLINE TR. To avoid that complexity, mention of setting the status to OFFLINE is removed in 9d06008767.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@trafficcontrol.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [trafficcontrol] ocket8888 commented on a change in pull request #6358: Traffic Router Ultimate Test Harness blueprint
Posted by GitBox <gi...@apache.org>.
ocket8888 commented on a change in pull request #6358:
URL: https://github.com/apache/trafficcontrol/pull/6358#discussion_r760517239
##########
File path: blueprints/tr-load-tests.md
##########
@@ -102,7 +109,7 @@ How will existing tables and columns be changed?
What are the column data types and modifiers?
What are the FK references and constraints?
-->
-No Data Model impact is anticipated.
+A database migration should be added to seed a new Operations-level user for use with the TR Ultimate Test Harness.
Review comment:
As users cannot be deleted through the API and any valid installation of ATC follows the setup process that runs `postinstall` which will create a default user, no supported environments have no users, and as of #6324 a Role will always exist that has all Permissions. So why do all new installs/upgrades of ATC need to create a new user just for running tests?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@trafficcontrol.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [trafficcontrol] zrhoffman commented on pull request #6358: Traffic Router Ultimate Test Harness blueprint
Posted by GitBox <gi...@apache.org>.
zrhoffman commented on pull request #6358:
URL: https://github.com/apache/trafficcontrol/pull/6358#issuecomment-973178357
> @zrhoffman - do you envision the TRUTH :) will someday run via a GitHub Action triggered by any code changes to TR and will fail if those changes negatively impact TR performance?
Yes, although only if we can get that GHA to consistently meet a chosen threshold. @shamrickus added a similar GHA for Traffic Monitor in #5950, but because the performance of GitHub Actions runners is so variable, it only passed 14 of 20 times in the last 20 commits to the [`master`](https://github.com/apache/trafficcontrol/commits/master) branch.
So if we can make a TRUTH GHA consistent, then yes. Otherwise, a GHA would have to wait.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@trafficcontrol.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [trafficcontrol] zrhoffman commented on a change in pull request #6358: Traffic Router Ultimate Test Harness blueprint
Posted by GitBox <gi...@apache.org>.
zrhoffman commented on a change in pull request #6358:
URL: https://github.com/apache/trafficcontrol/pull/6358#discussion_r765318774
##########
File path: blueprints/tr-load-tests.md
##########
@@ -0,0 +1,296 @@
+<!--
+Licensed to the Apache Software Foundation (ASF) under one
+or more contributor license agreements. See the NOTICE file
+distributed with this work for additional information
+regarding copyright ownership. The ASF licenses this file
+to you under the Apache License, Version 2.0 (the
+"License"); you may not use this file except in compliance
+with the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing,
+software distributed under the License is distributed on an
+"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+KIND, either express or implied. See the License for the
+specific language governing permissions and limitations
+under the License.
+-->
+# Traffic Router Ultimate Test Harness <!-- a concise title for the new feature this blueprint will describe -->
+
+## Problem Description
+<!--
+*What* is being asked for?
+*Why* is this necessary?
+*How* will this be used?
+-->
+As the entrypoint of an Apache Traffic Control CDN, Traffic Router is the most exposed, most critical component, and it must be able to route traffic quickly and at a very high rate. Although Traffic Router has met this requirement in the past, if Traffic Router undergoes no performance tests as it evolves, there is no guarantee that its performance will not decline with time.
+
+Important times to run performance tests:
+
+* When traffic volume to your CDN changes significantly
+* When making harware changes to the server hosting the Traffic Router instance
+* When upgrading Traffic Router to a new Apache Traffic Control version
+* When developing or maintaining a Traffic Router feature that could have an impact on any aspect of Traffic Router performance, including before, during, and after pull request review
+* When a commit that modifies Traffic Router is pushed to a GitHub branch, for all Apache Traffic Control branches
+
+A load test for Delivery Services exists in the project at [`/test/router`](https://github.com/apache/trafficcontrol/tree/RELEASE-6.0.1/test/router), but
+- It has not been maintained over time, currently does not work
+- Only tests HTTP Delivery Services
+- Does not support testing Coverage Zone Maps
+- Does not use the TO Client Library for requests to Traffic Ops
+- Prompts only for the number of requests to make to Delivery Services, not a length of time to run the test
+- Does not fail if some minimum threshold of requests per second is not met
+- Is not configurable in other ways, such as length of paths generated for HTTP requests to Delivery Service, which client IP address to use
+
+## Proposed Change
+<!--
+*How* will this be implemented (at a high level)?
+-->
+The Traffic Router Ultimate Test Harness will include an end-to-end performance test suite verify that the features of Traffic Router meet expected performance thresholds, as well as additional end-to-end tests of other Traffic Router features.
+
+The TR Ultimate Test Harness may extend [`/test/router`](https://github.com/apache/trafficcontrol/tree/RELEASE-6.0.1/test/router) where possible, but it should not limit itself for that secondary goal.
+
+### Traffic Portal Impact
+<!--
+*How* will this impact Traffic Portal?
+What new UI changes will be required?
+Will entirely new pages/views be necessary?
+Will a new field be added to an existing form?
+How will the user interact with the new UI changes?
+-->
+No Traffic Portal impact is anticipated.
+
+### Traffic Ops Impact
+<!--
+*How* will this impact Traffic Ops (at a high level)?
+-->
+The Operations-level TR Ultimate Test Harness user will update the status of any Traffic Router that is currently undergoing load testing to `"OFFLINE"` in Traffic Ops. It should also update the status of that Traffic Router to its previous status once the load testing is complete, regardless of whether the test failed or succeeded.
+
+The Operations-level TR Ultimate Test Harness user will also be used to get
+* CDN snapshots
+* CDN information
+* information about Traffic Router-type Servers in those CDNs
+* Type information
+* Delivery Service information
+
+#### REST API Impact
+<!--
+*How* will this impact the Traffic Ops REST API?
+
+What new endpoints will be required?
+How will existing endpoints be changed?
+What will the requests and responses look like?
+What fields are required or optional?
+What are the defaults for optional fields?
+What are the validation constraints?
+-->
+No Traffic Ops REST API impact is anticipated.
+
+#### Client Impact
+<!--
+*How* will this impact Traffic Ops REST API clients (Go, Python, Java)?
+
+If new endpoints are required, will corresponding client methods be added?
+-->
+Clients importing the `github.com/apache/trafficcontrol/lib/go-tc` package will optionally be able to import a constant for `X-MM-Client-IP`, a request header Traffic Router to specify to Traffic Router the IP address to use to geolocate that client:
+https://github.com/apache/trafficcontrol/blob/1ed2964d16618aeebef142b01a538336a44d07dd/traffic_router/core/src/main/java/org/apache/traffic_control/traffic_router/core/request/HTTPRequest.java#L29
+
+Additionally, a struct used to unmarshall a Coverage Zone File could be placed in `lib/go-tc`.
+
+#### Data Model / Database Impact
+<!--
+*How* will this impact the Traffic Ops data model?
+*How* will this impact the Traffic Ops database schema?
+
+What changes to the lib/go-tc structs will be required?
+What new tables and columns will be required?
+How will existing tables and columns be changed?
+What are the column data types and modifiers?
+What are the FK references and constraints?
+-->
+A database migration should be added to seed a new Operations-level user for use with the TR Ultimate Test Harness.
+
+### Cache Config Impact
+<!--
+*How* will this impact ORT?
+-->
+No Cache Config impact is anticipated.
+
+### Traffic Monitor Impact
+<!--
+*How* will this impact Traffic Monitor?
+
+Will new profile parameters be required?
+-->
+No Traffic Monitor impact is anticipated.
+
+### Traffic Router Impact
+<!--
+*How* will this impact Traffic Router?
+
+Will new profile parameters be required?
+How will the CRConfig be changed?
+How will changes in Traffic Ops data be reflected in the CRConfig?
+Will Traffic Router remain backwards-compatible with old CRConfigs?
+Will old Traffic Routers remain forwards-compatible with new CRConfigs?
+-->
+The addition of the TR Ultimate Test Harness themselves will not change Traffic Router functionality in any way. For visibility, however, the TR Ultimate Test Harness should reside in a directory within the `traffic_router` directory. This will be the first time since 545929f7cc that Golang sources will exist in the `traffic_router` directory, so any assumption that all sources within the `traffic_router` directory directly impact Traffic Router's ability to compile should be abandoned.
+
+The TR Ultimate Test Harness should not be included in the Traffic Router RPM, as it is meant to be run on a host separate from Traffic Routers.
+
+### Traffic Stats Impact
+<!--
+*How* will this impact Traffic Stats?
+-->
+No Traffic Stats impact is anticipated.
+
+### Traffic Vault Impact
+<!--
+*How* will this impact Traffic Vault?
+
+Will there be any new data stored in or removed from Riak?
+Will there be any changes to the Riak requests and responses?
+-->
+No Traffic Vault impact is anticipated.
+
+### Documentation Impact
+<!--
+*How* will this impact the documentation?
+
+What new documentation will be required?
+What existing documentation will need to be updated?
+-->
+Instructions for using the Traffic Router Ultimate Test Harness should be added to the documentation. This should include:
+- Small rationale for inclusion of TR Ultimate Test Harness
+- Setup instructions
+- Documentation of each CLI option
+- Example commands
+
+### Testing Impact
+<!--
+*How* will this impact testing?
+
+What is the high-level test plan?
+How should this be tested?
+Can this be tested within the existing test frameworks?
+How should the existing frameworks be enhanced in order to test this properly?
+-->
+#### Load Tests
+The Router Ultimate Test Harness should include a load test for HTTP-routed Delivery Services and for DNS-routed Delivery Services
Review comment:
> Do we care about STEERING
Testing STEERING specifically should not be necessary, but we should at least test HTTP Delivery Services with no Location header (`trred=false`). Added in 7d5fec0e1
> /federated Delivery Services?
Federations are already mentioned:
https://github.com/apache/trafficcontrol/blob/f202723c5c2475f51c7558c7a83b80ecf6ee9774/blueprints/tr-load-tests.md#L201
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@trafficcontrol.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [trafficcontrol] ocket8888 commented on a change in pull request #6358: Traffic Router Ultimate Test Harness blueprint
Posted by GitBox <gi...@apache.org>.
ocket8888 commented on a change in pull request #6358:
URL: https://github.com/apache/trafficcontrol/pull/6358#discussion_r760513629
##########
File path: blueprints/tr-load-tests.md
##########
@@ -0,0 +1,296 @@
+<!--
+Licensed to the Apache Software Foundation (ASF) under one
+or more contributor license agreements. See the NOTICE file
+distributed with this work for additional information
+regarding copyright ownership. The ASF licenses this file
+to you under the Apache License, Version 2.0 (the
+"License"); you may not use this file except in compliance
+with the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing,
+software distributed under the License is distributed on an
+"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+KIND, either express or implied. See the License for the
+specific language governing permissions and limitations
+under the License.
+-->
+# Traffic Router Ultimate Test Harness <!-- a concise title for the new feature this blueprint will describe -->
+
+## Problem Description
+<!--
+*What* is being asked for?
+*Why* is this necessary?
+*How* will this be used?
+-->
+As the entrypoint of an Apache Traffic Control CDN, Traffic Router is the most exposed, most critical component, and it must be able to route traffic quickly and at a very high rate. Although Traffic Router has met this requirement in the past, if Traffic Router undergoes no performance tests as it evolves, there is no guarantee that its performance will not decline with time.
+
+Important times to run performance tests:
+
+* When traffic volume to your CDN changes significantly
+* When making harware changes to the server hosting the Traffic Router instance
+* When upgrading Traffic Router to a new Apache Traffic Control version
+* When developing or maintaining a Traffic Router feature that could have an impact on any aspect of Traffic Router performance, including before, during, and after pull request review
+* When a commit that modifies Traffic Router is pushed to a GitHub branch, for all Apache Traffic Control branches
+
+A load test for Delivery Services exists in the project at [`/test/router`](https://github.com/apache/trafficcontrol/tree/RELEASE-6.0.1/test/router), but
+- It has not been maintained over time, currently does not work
+- Only tests HTTP Delivery Services
+- Does not support testing Coverage Zone Maps
+- Does not use the TO Client Library for requests to Traffic Ops
+- Prompts only for the number of requests to make to Delivery Services, not a length of time to run the test
+- Does not fail if some minimum threshold of requests per second is not met
+- Is not configurable in other ways, such as length of paths generated for HTTP requests to Delivery Service, which client IP address to use
+
+## Proposed Change
+<!--
+*How* will this be implemented (at a high level)?
+-->
+The Traffic Router Ultimate Test Harness will include an end-to-end performance test suite verify that the features of Traffic Router meet expected performance thresholds, as well as additional end-to-end tests of other Traffic Router features.
+
+The TR Ultimate Test Harness may extend [`/test/router`](https://github.com/apache/trafficcontrol/tree/RELEASE-6.0.1/test/router) where possible, but it should not limit itself for that secondary goal.
+
+### Traffic Portal Impact
+<!--
+*How* will this impact Traffic Portal?
+What new UI changes will be required?
+Will entirely new pages/views be necessary?
+Will a new field be added to an existing form?
+How will the user interact with the new UI changes?
+-->
+No Traffic Portal impact is anticipated.
+
+### Traffic Ops Impact
+<!--
+*How* will this impact Traffic Ops (at a high level)?
+-->
+The Operations-level TR Ultimate Test Harness user will update the status of any Traffic Router that is currently undergoing load testing to `"OFFLINE"` in Traffic Ops. It should also update the status of that Traffic Router to its previous status once the load testing is complete, regardless of whether the test failed or succeeded.
+
+The Operations-level TR Ultimate Test Harness user will also be used to get
+* CDN snapshots
+* CDN information
+* information about Traffic Router-type Servers in those CDNs
+* Type information
+* Delivery Service information
+
+#### REST API Impact
+<!--
+*How* will this impact the Traffic Ops REST API?
+
+What new endpoints will be required?
+How will existing endpoints be changed?
+What will the requests and responses look like?
+What fields are required or optional?
+What are the defaults for optional fields?
+What are the validation constraints?
+-->
+No Traffic Ops REST API impact is anticipated.
+
+#### Client Impact
+<!--
+*How* will this impact Traffic Ops REST API clients (Go, Python, Java)?
+
+If new endpoints are required, will corresponding client methods be added?
+-->
+Clients importing the `github.com/apache/trafficcontrol/lib/go-tc` package will optionally be able to import a constant for `X-MM-Client-IP`, a request header Traffic Router to specify to Traffic Router the IP address to use to geolocate that client:
+https://github.com/apache/trafficcontrol/blob/1ed2964d16618aeebef142b01a538336a44d07dd/traffic_router/core/src/main/java/org/apache/traffic_control/traffic_router/core/request/HTTPRequest.java#L29
+
+Additionally, a struct used to unmarshall a Coverage Zone File could be placed in `lib/go-tc`.
+
+#### Data Model / Database Impact
+<!--
+*How* will this impact the Traffic Ops data model?
+*How* will this impact the Traffic Ops database schema?
+
+What changes to the lib/go-tc structs will be required?
+What new tables and columns will be required?
+How will existing tables and columns be changed?
+What are the column data types and modifiers?
+What are the FK references and constraints?
+-->
+A database migration should be added to seed a new Operations-level user for use with the TR Ultimate Test Harness.
+
+### Cache Config Impact
+<!--
+*How* will this impact ORT?
+-->
+No Cache Config impact is anticipated.
+
+### Traffic Monitor Impact
+<!--
+*How* will this impact Traffic Monitor?
+
+Will new profile parameters be required?
+-->
+No Traffic Monitor impact is anticipated.
+
+### Traffic Router Impact
+<!--
+*How* will this impact Traffic Router?
+
+Will new profile parameters be required?
+How will the CRConfig be changed?
+How will changes in Traffic Ops data be reflected in the CRConfig?
+Will Traffic Router remain backwards-compatible with old CRConfigs?
+Will old Traffic Routers remain forwards-compatible with new CRConfigs?
+-->
+The addition of the TR Ultimate Test Harness themselves will not change Traffic Router functionality in any way. For visibility, however, the TR Ultimate Test Harness should reside in a directory within the `traffic_router` directory. This will be the first time since 545929f7cc that Golang sources will exist in the `traffic_router` directory, so any assumption that all sources within the `traffic_router` directory directly impact Traffic Router's ability to compile should be abandoned.
+
+The TR Ultimate Test Harness should not be included in the Traffic Router RPM, as it is meant to be run on a host separate from Traffic Routers.
+
+### Traffic Stats Impact
+<!--
+*How* will this impact Traffic Stats?
+-->
+No Traffic Stats impact is anticipated.
+
+### Traffic Vault Impact
+<!--
+*How* will this impact Traffic Vault?
+
+Will there be any new data stored in or removed from Riak?
+Will there be any changes to the Riak requests and responses?
+-->
+No Traffic Vault impact is anticipated.
+
+### Documentation Impact
+<!--
+*How* will this impact the documentation?
+
+What new documentation will be required?
+What existing documentation will need to be updated?
+-->
+Instructions for using the Traffic Router Ultimate Test Harness should be added to the documentation. This should include:
+- Small rationale for inclusion of TR Ultimate Test Harness
+- Setup instructions
+- Documentation of each CLI option
+- Example commands
+
+### Testing Impact
+<!--
+*How* will this impact testing?
+
+What is the high-level test plan?
+How should this be tested?
+Can this be tested within the existing test frameworks?
+How should the existing frameworks be enhanced in order to test this properly?
+-->
+#### Load Tests
+The Router Ultimate Test Harness should include a load test for HTTP-routed Delivery Services and for DNS-routed Delivery Services
Review comment:
Do we care about STEERING/federated Delivery Services?
##########
File path: blueprints/tr-load-tests.md
##########
@@ -0,0 +1,296 @@
+<!--
+Licensed to the Apache Software Foundation (ASF) under one
+or more contributor license agreements. See the NOTICE file
+distributed with this work for additional information
+regarding copyright ownership. The ASF licenses this file
+to you under the Apache License, Version 2.0 (the
+"License"); you may not use this file except in compliance
+with the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing,
+software distributed under the License is distributed on an
+"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+KIND, either express or implied. See the License for the
+specific language governing permissions and limitations
+under the License.
+-->
+# Traffic Router Ultimate Test Harness <!-- a concise title for the new feature this blueprint will describe -->
+
+## Problem Description
+<!--
+*What* is being asked for?
+*Why* is this necessary?
+*How* will this be used?
+-->
+As the entrypoint of an Apache Traffic Control CDN, Traffic Router is the most exposed, most critical component, and it must be able to route traffic quickly and at a very high rate. Although Traffic Router has met this requirement in the past, if Traffic Router undergoes no performance tests as it evolves, there is no guarantee that its performance will not decline with time.
+
+Important times to run performance tests:
+
+* When traffic volume to your CDN changes significantly
+* When making harware changes to the server hosting the Traffic Router instance
+* When upgrading Traffic Router to a new Apache Traffic Control version
+* When developing or maintaining a Traffic Router feature that could have an impact on any aspect of Traffic Router performance, including before, during, and after pull request review
+* When a commit that modifies Traffic Router is pushed to a GitHub branch, for all Apache Traffic Control branches
+
+A load test for Delivery Services exists in the project at [`/test/router`](https://github.com/apache/trafficcontrol/tree/RELEASE-6.0.1/test/router), but
+- It has not been maintained over time, currently does not work
+- Only tests HTTP Delivery Services
+- Does not support testing Coverage Zone Maps
+- Does not use the TO Client Library for requests to Traffic Ops
+- Prompts only for the number of requests to make to Delivery Services, not a length of time to run the test
+- Does not fail if some minimum threshold of requests per second is not met
+- Is not configurable in other ways, such as length of paths generated for HTTP requests to Delivery Service, which client IP address to use
+
+## Proposed Change
+<!--
+*How* will this be implemented (at a high level)?
+-->
+The Traffic Router Ultimate Test Harness will include an end-to-end performance test suite verify that the features of Traffic Router meet expected performance thresholds, as well as additional end-to-end tests of other Traffic Router features.
+
+The TR Ultimate Test Harness may extend [`/test/router`](https://github.com/apache/trafficcontrol/tree/RELEASE-6.0.1/test/router) where possible, but it should not limit itself for that secondary goal.
+
+### Traffic Portal Impact
+<!--
+*How* will this impact Traffic Portal?
+What new UI changes will be required?
+Will entirely new pages/views be necessary?
+Will a new field be added to an existing form?
+How will the user interact with the new UI changes?
+-->
+No Traffic Portal impact is anticipated.
+
+### Traffic Ops Impact
+<!--
+*How* will this impact Traffic Ops (at a high level)?
+-->
+The Operations-level TR Ultimate Test Harness user will update the status of any Traffic Router that is currently undergoing load testing to `"OFFLINE"` in Traffic Ops. It should also update the status of that Traffic Router to its previous status once the load testing is complete, regardless of whether the test failed or succeeded.
+
+The Operations-level TR Ultimate Test Harness user will also be used to get
+* CDN snapshots
+* CDN information
+* information about Traffic Router-type Servers in those CDNs
+* Type information
+* Delivery Service information
Review comment:
This sounds more like a step in the testing setup than actual changes to Traffic Ops
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@trafficcontrol.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [trafficcontrol] ocket8888 commented on a change in pull request #6358: Traffic Router Ultimate Test Harness blueprint
Posted by GitBox <gi...@apache.org>.
ocket8888 commented on a change in pull request #6358:
URL: https://github.com/apache/trafficcontrol/pull/6358#discussion_r760509317
##########
File path: blueprints/tr-load-tests.md
##########
@@ -102,7 +109,7 @@ How will existing tables and columns be changed?
What are the column data types and modifiers?
What are the FK references and constraints?
-->
-No Data Model impact is anticipated.
+A database migration should be added to seed a new Operations-level user for use with the TR Ultimate Test Harness.
Review comment:
Do all install environments really need to support running the test harness? Surely we only want to run it against testing environments? What new Permission would this user need that existing users don't have?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@trafficcontrol.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [trafficcontrol] rawlinp commented on a change in pull request #6358: Traffic Router Ultimate Test Harness blueprint
Posted by GitBox <gi...@apache.org>.
rawlinp commented on a change in pull request #6358:
URL: https://github.com/apache/trafficcontrol/pull/6358#discussion_r765334389
##########
File path: blueprints/tr-load-tests.md
##########
@@ -0,0 +1,296 @@
+<!--
+Licensed to the Apache Software Foundation (ASF) under one
+or more contributor license agreements. See the NOTICE file
+distributed with this work for additional information
+regarding copyright ownership. The ASF licenses this file
+to you under the Apache License, Version 2.0 (the
+"License"); you may not use this file except in compliance
+with the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing,
+software distributed under the License is distributed on an
+"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+KIND, either express or implied. See the License for the
+specific language governing permissions and limitations
+under the License.
+-->
+# Traffic Router Ultimate Test Harness <!-- a concise title for the new feature this blueprint will describe -->
+
+## Problem Description
+<!--
+*What* is being asked for?
+*Why* is this necessary?
+*How* will this be used?
+-->
+As the entrypoint of an Apache Traffic Control CDN, Traffic Router is the most exposed, most critical component, and it must be able to route traffic quickly and at a very high rate. Although Traffic Router has met this requirement in the past, if Traffic Router undergoes no performance tests as it evolves, there is no guarantee that its performance will not decline with time.
+
+Important times to run performance tests:
+
+* When traffic volume to your CDN changes significantly
+* When making harware changes to the server hosting the Traffic Router instance
+* When upgrading Traffic Router to a new Apache Traffic Control version
+* When developing or maintaining a Traffic Router feature that could have an impact on any aspect of Traffic Router performance, including before, during, and after pull request review
+* When a commit that modifies Traffic Router is pushed to a GitHub branch, for all Apache Traffic Control branches
+
+A load test for Delivery Services exists in the project at [`/test/router`](https://github.com/apache/trafficcontrol/tree/RELEASE-6.0.1/test/router), but
+- It has not been maintained over time, currently does not work
+- Only tests HTTP Delivery Services
+- Does not support testing Coverage Zone Maps
+- Does not use the TO Client Library for requests to Traffic Ops
+- Prompts only for the number of requests to make to Delivery Services, not a length of time to run the test
+- Does not fail if some minimum threshold of requests per second is not met
+- Is not configurable in other ways, such as length of paths generated for HTTP requests to Delivery Service, which client IP address to use
+
+## Proposed Change
+<!--
+*How* will this be implemented (at a high level)?
+-->
+The Traffic Router Ultimate Test Harness will include an end-to-end performance test suite verify that the features of Traffic Router meet expected performance thresholds, as well as additional end-to-end tests of other Traffic Router features.
+
+The TR Ultimate Test Harness may extend [`/test/router`](https://github.com/apache/trafficcontrol/tree/RELEASE-6.0.1/test/router) where possible, but it should not limit itself for that secondary goal.
+
+### Traffic Portal Impact
+<!--
+*How* will this impact Traffic Portal?
+What new UI changes will be required?
+Will entirely new pages/views be necessary?
+Will a new field be added to an existing form?
+How will the user interact with the new UI changes?
+-->
+No Traffic Portal impact is anticipated.
+
+### Traffic Ops Impact
+<!--
+*How* will this impact Traffic Ops (at a high level)?
+-->
+The Operations-level TR Ultimate Test Harness user will update the status of any Traffic Router that is currently undergoing load testing to `"OFFLINE"` in Traffic Ops. It should also update the status of that Traffic Router to its previous status once the load testing is complete, regardless of whether the test failed or succeeded.
Review comment:
There seems to be a minor discrepancy between this line and L260, which states that an operator may want to set the TR to OFFLINE manually. Which line is correct? Also, I'm not positive, but I _think_ setting the TR under test to OFFLINE would affect its responses in that it would exclude itself from any responses.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@trafficcontrol.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [trafficcontrol] zrhoffman commented on a change in pull request #6358: Traffic Router Ultimate Test Harness blueprint
Posted by GitBox <gi...@apache.org>.
zrhoffman commented on a change in pull request #6358:
URL: https://github.com/apache/trafficcontrol/pull/6358#discussion_r765318774
##########
File path: blueprints/tr-load-tests.md
##########
@@ -0,0 +1,296 @@
+<!--
+Licensed to the Apache Software Foundation (ASF) under one
+or more contributor license agreements. See the NOTICE file
+distributed with this work for additional information
+regarding copyright ownership. The ASF licenses this file
+to you under the Apache License, Version 2.0 (the
+"License"); you may not use this file except in compliance
+with the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing,
+software distributed under the License is distributed on an
+"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+KIND, either express or implied. See the License for the
+specific language governing permissions and limitations
+under the License.
+-->
+# Traffic Router Ultimate Test Harness <!-- a concise title for the new feature this blueprint will describe -->
+
+## Problem Description
+<!--
+*What* is being asked for?
+*Why* is this necessary?
+*How* will this be used?
+-->
+As the entrypoint of an Apache Traffic Control CDN, Traffic Router is the most exposed, most critical component, and it must be able to route traffic quickly and at a very high rate. Although Traffic Router has met this requirement in the past, if Traffic Router undergoes no performance tests as it evolves, there is no guarantee that its performance will not decline with time.
+
+Important times to run performance tests:
+
+* When traffic volume to your CDN changes significantly
+* When making harware changes to the server hosting the Traffic Router instance
+* When upgrading Traffic Router to a new Apache Traffic Control version
+* When developing or maintaining a Traffic Router feature that could have an impact on any aspect of Traffic Router performance, including before, during, and after pull request review
+* When a commit that modifies Traffic Router is pushed to a GitHub branch, for all Apache Traffic Control branches
+
+A load test for Delivery Services exists in the project at [`/test/router`](https://github.com/apache/trafficcontrol/tree/RELEASE-6.0.1/test/router), but
+- It has not been maintained over time, currently does not work
+- Only tests HTTP Delivery Services
+- Does not support testing Coverage Zone Maps
+- Does not use the TO Client Library for requests to Traffic Ops
+- Prompts only for the number of requests to make to Delivery Services, not a length of time to run the test
+- Does not fail if some minimum threshold of requests per second is not met
+- Is not configurable in other ways, such as length of paths generated for HTTP requests to Delivery Service, which client IP address to use
+
+## Proposed Change
+<!--
+*How* will this be implemented (at a high level)?
+-->
+The Traffic Router Ultimate Test Harness will include an end-to-end performance test suite verify that the features of Traffic Router meet expected performance thresholds, as well as additional end-to-end tests of other Traffic Router features.
+
+The TR Ultimate Test Harness may extend [`/test/router`](https://github.com/apache/trafficcontrol/tree/RELEASE-6.0.1/test/router) where possible, but it should not limit itself for that secondary goal.
+
+### Traffic Portal Impact
+<!--
+*How* will this impact Traffic Portal?
+What new UI changes will be required?
+Will entirely new pages/views be necessary?
+Will a new field be added to an existing form?
+How will the user interact with the new UI changes?
+-->
+No Traffic Portal impact is anticipated.
+
+### Traffic Ops Impact
+<!--
+*How* will this impact Traffic Ops (at a high level)?
+-->
+The Operations-level TR Ultimate Test Harness user will update the status of any Traffic Router that is currently undergoing load testing to `"OFFLINE"` in Traffic Ops. It should also update the status of that Traffic Router to its previous status once the load testing is complete, regardless of whether the test failed or succeeded.
+
+The Operations-level TR Ultimate Test Harness user will also be used to get
+* CDN snapshots
+* CDN information
+* information about Traffic Router-type Servers in those CDNs
+* Type information
+* Delivery Service information
+
+#### REST API Impact
+<!--
+*How* will this impact the Traffic Ops REST API?
+
+What new endpoints will be required?
+How will existing endpoints be changed?
+What will the requests and responses look like?
+What fields are required or optional?
+What are the defaults for optional fields?
+What are the validation constraints?
+-->
+No Traffic Ops REST API impact is anticipated.
+
+#### Client Impact
+<!--
+*How* will this impact Traffic Ops REST API clients (Go, Python, Java)?
+
+If new endpoints are required, will corresponding client methods be added?
+-->
+Clients importing the `github.com/apache/trafficcontrol/lib/go-tc` package will optionally be able to import a constant for `X-MM-Client-IP`, a request header Traffic Router to specify to Traffic Router the IP address to use to geolocate that client:
+https://github.com/apache/trafficcontrol/blob/1ed2964d16618aeebef142b01a538336a44d07dd/traffic_router/core/src/main/java/org/apache/traffic_control/traffic_router/core/request/HTTPRequest.java#L29
+
+Additionally, a struct used to unmarshall a Coverage Zone File could be placed in `lib/go-tc`.
+
+#### Data Model / Database Impact
+<!--
+*How* will this impact the Traffic Ops data model?
+*How* will this impact the Traffic Ops database schema?
+
+What changes to the lib/go-tc structs will be required?
+What new tables and columns will be required?
+How will existing tables and columns be changed?
+What are the column data types and modifiers?
+What are the FK references and constraints?
+-->
+A database migration should be added to seed a new Operations-level user for use with the TR Ultimate Test Harness.
+
+### Cache Config Impact
+<!--
+*How* will this impact ORT?
+-->
+No Cache Config impact is anticipated.
+
+### Traffic Monitor Impact
+<!--
+*How* will this impact Traffic Monitor?
+
+Will new profile parameters be required?
+-->
+No Traffic Monitor impact is anticipated.
+
+### Traffic Router Impact
+<!--
+*How* will this impact Traffic Router?
+
+Will new profile parameters be required?
+How will the CRConfig be changed?
+How will changes in Traffic Ops data be reflected in the CRConfig?
+Will Traffic Router remain backwards-compatible with old CRConfigs?
+Will old Traffic Routers remain forwards-compatible with new CRConfigs?
+-->
+The addition of the TR Ultimate Test Harness themselves will not change Traffic Router functionality in any way. For visibility, however, the TR Ultimate Test Harness should reside in a directory within the `traffic_router` directory. This will be the first time since 545929f7cc that Golang sources will exist in the `traffic_router` directory, so any assumption that all sources within the `traffic_router` directory directly impact Traffic Router's ability to compile should be abandoned.
+
+The TR Ultimate Test Harness should not be included in the Traffic Router RPM, as it is meant to be run on a host separate from Traffic Routers.
+
+### Traffic Stats Impact
+<!--
+*How* will this impact Traffic Stats?
+-->
+No Traffic Stats impact is anticipated.
+
+### Traffic Vault Impact
+<!--
+*How* will this impact Traffic Vault?
+
+Will there be any new data stored in or removed from Riak?
+Will there be any changes to the Riak requests and responses?
+-->
+No Traffic Vault impact is anticipated.
+
+### Documentation Impact
+<!--
+*How* will this impact the documentation?
+
+What new documentation will be required?
+What existing documentation will need to be updated?
+-->
+Instructions for using the Traffic Router Ultimate Test Harness should be added to the documentation. This should include:
+- Small rationale for inclusion of TR Ultimate Test Harness
+- Setup instructions
+- Documentation of each CLI option
+- Example commands
+
+### Testing Impact
+<!--
+*How* will this impact testing?
+
+What is the high-level test plan?
+How should this be tested?
+Can this be tested within the existing test frameworks?
+How should the existing frameworks be enhanced in order to test this properly?
+-->
+#### Load Tests
+The Router Ultimate Test Harness should include a load test for HTTP-routed Delivery Services and for DNS-routed Delivery Services
Review comment:
> Do we care about STEERING
Testing STEERING specifically should not be necessary, but we should at least test HTTP Delivery Services with no Location header (`trred=false`). Added in 7d5fec0e1f
> /federated Delivery Services?
Federations are already mentioned:
https://github.com/apache/trafficcontrol/blob/f202723c5c2475f51c7558c7a83b80ecf6ee9774/blueprints/tr-load-tests.md#L201
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@trafficcontrol.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [trafficcontrol] limited commented on pull request #6358: Traffic Router Ultimate Test Harness blueprint
Posted by GitBox <gi...@apache.org>.
limited commented on pull request #6358:
URL: https://github.com/apache/trafficcontrol/pull/6358#issuecomment-983830825
Hey @zrhoffman -
I fully support improving performance testing of TR. I've used both `wrk` and `flamethrower` before to decent success. Configuring those tools is definitely a challenge, especially across large configurations. This looks like it will simplify that barrier for many others
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@trafficcontrol.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [trafficcontrol] zrhoffman commented on pull request #6358: Traffic Router Ultimate Test Harness blueprint
Posted by GitBox <gi...@apache.org>.
zrhoffman commented on pull request #6358:
URL: https://github.com/apache/trafficcontrol/pull/6358#issuecomment-973302010
> @zrhoffman - do you envision the TRUTH :) will someday run via a GitHub Action triggered by any code changes to TR and will fail if those changes negatively impact TR performance?
*Automation Impact* section added in 4d0e5594c2.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@trafficcontrol.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [trafficcontrol] zrhoffman commented on a change in pull request #6358: Traffic Router Ultimate Test Harness blueprint
Posted by GitBox <gi...@apache.org>.
zrhoffman commented on a change in pull request #6358:
URL: https://github.com/apache/trafficcontrol/pull/6358#discussion_r767052901
##########
File path: blueprints/tr-load-tests.md
##########
@@ -0,0 +1,296 @@
+<!--
+Licensed to the Apache Software Foundation (ASF) under one
+or more contributor license agreements. See the NOTICE file
+distributed with this work for additional information
+regarding copyright ownership. The ASF licenses this file
+to you under the Apache License, Version 2.0 (the
+"License"); you may not use this file except in compliance
+with the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing,
+software distributed under the License is distributed on an
+"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+KIND, either express or implied. See the License for the
+specific language governing permissions and limitations
+under the License.
+-->
+# Traffic Router Ultimate Test Harness <!-- a concise title for the new feature this blueprint will describe -->
+
+## Problem Description
+<!--
+*What* is being asked for?
+*Why* is this necessary?
+*How* will this be used?
+-->
+As the entrypoint of an Apache Traffic Control CDN, Traffic Router is the most exposed, most critical component, and it must be able to route traffic quickly and at a very high rate. Although Traffic Router has met this requirement in the past, if Traffic Router undergoes no performance tests as it evolves, there is no guarantee that its performance will not decline with time.
+
+Important times to run performance tests:
+
+* When traffic volume to your CDN changes significantly
+* When making harware changes to the server hosting the Traffic Router instance
+* When upgrading Traffic Router to a new Apache Traffic Control version
+* When developing or maintaining a Traffic Router feature that could have an impact on any aspect of Traffic Router performance, including before, during, and after pull request review
+* When a commit that modifies Traffic Router is pushed to a GitHub branch, for all Apache Traffic Control branches
+
+A load test for Delivery Services exists in the project at [`/test/router`](https://github.com/apache/trafficcontrol/tree/RELEASE-6.0.1/test/router), but
+- It has not been maintained over time, currently does not work
+- Only tests HTTP Delivery Services
+- Does not support testing Coverage Zone Maps
+- Does not use the TO Client Library for requests to Traffic Ops
+- Prompts only for the number of requests to make to Delivery Services, not a length of time to run the test
+- Does not fail if some minimum threshold of requests per second is not met
+- Is not configurable in other ways, such as length of paths generated for HTTP requests to Delivery Service, which client IP address to use
+
+## Proposed Change
+<!--
+*How* will this be implemented (at a high level)?
+-->
+The Traffic Router Ultimate Test Harness will include an end-to-end performance test suite verify that the features of Traffic Router meet expected performance thresholds, as well as additional end-to-end tests of other Traffic Router features.
+
+The TR Ultimate Test Harness may extend [`/test/router`](https://github.com/apache/trafficcontrol/tree/RELEASE-6.0.1/test/router) where possible, but it should not limit itself for that secondary goal.
+
+### Traffic Portal Impact
+<!--
+*How* will this impact Traffic Portal?
+What new UI changes will be required?
+Will entirely new pages/views be necessary?
+Will a new field be added to an existing form?
+How will the user interact with the new UI changes?
+-->
+No Traffic Portal impact is anticipated.
+
+### Traffic Ops Impact
+<!--
+*How* will this impact Traffic Ops (at a high level)?
+-->
+The Operations-level TR Ultimate Test Harness user will update the status of any Traffic Router that is currently undergoing load testing to `"OFFLINE"` in Traffic Ops. It should also update the status of that Traffic Router to its previous status once the load testing is complete, regardless of whether the test failed or succeeded.
Review comment:
> Also, I'm not positive, but I _think_ setting the TR under test to OFFLINE would affect its responses in that it would exclude itself from any responses.
You're right, setting the TR status to OFFLINE would mean needing a separate ONLINE TR. To avoid that complexity, mention of setting the status to OFFLINE is removed in 9d06008767.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@trafficcontrol.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [trafficcontrol] rawlinp commented on a change in pull request #6358: Traffic Router Ultimate Test Harness blueprint
Posted by GitBox <gi...@apache.org>.
rawlinp commented on a change in pull request #6358:
URL: https://github.com/apache/trafficcontrol/pull/6358#discussion_r765335721
##########
File path: blueprints/tr-load-tests.md
##########
@@ -0,0 +1,296 @@
+<!--
+Licensed to the Apache Software Foundation (ASF) under one
+or more contributor license agreements. See the NOTICE file
+distributed with this work for additional information
+regarding copyright ownership. The ASF licenses this file
+to you under the Apache License, Version 2.0 (the
+"License"); you may not use this file except in compliance
+with the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing,
+software distributed under the License is distributed on an
+"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+KIND, either express or implied. See the License for the
+specific language governing permissions and limitations
+under the License.
+-->
+# Traffic Router Ultimate Test Harness <!-- a concise title for the new feature this blueprint will describe -->
+
+## Problem Description
+<!--
+*What* is being asked for?
+*Why* is this necessary?
+*How* will this be used?
+-->
+As the entrypoint of an Apache Traffic Control CDN, Traffic Router is the most exposed, most critical component, and it must be able to route traffic quickly and at a very high rate. Although Traffic Router has met this requirement in the past, if Traffic Router undergoes no performance tests as it evolves, there is no guarantee that its performance will not decline with time.
+
+Important times to run performance tests:
+
+* When traffic volume to your CDN changes significantly
+* When making harware changes to the server hosting the Traffic Router instance
+* When upgrading Traffic Router to a new Apache Traffic Control version
+* When developing or maintaining a Traffic Router feature that could have an impact on any aspect of Traffic Router performance, including before, during, and after pull request review
+* When a commit that modifies Traffic Router is pushed to a GitHub branch, for all Apache Traffic Control branches
+
+A load test for Delivery Services exists in the project at [`/test/router`](https://github.com/apache/trafficcontrol/tree/RELEASE-6.0.1/test/router), but
+- It has not been maintained over time, currently does not work
+- Only tests HTTP Delivery Services
+- Does not support testing Coverage Zone Maps
+- Does not use the TO Client Library for requests to Traffic Ops
+- Prompts only for the number of requests to make to Delivery Services, not a length of time to run the test
+- Does not fail if some minimum threshold of requests per second is not met
+- Is not configurable in other ways, such as length of paths generated for HTTP requests to Delivery Service, which client IP address to use
+
+## Proposed Change
+<!--
+*How* will this be implemented (at a high level)?
+-->
+The Traffic Router Ultimate Test Harness will include an end-to-end performance test suite verify that the features of Traffic Router meet expected performance thresholds, as well as additional end-to-end tests of other Traffic Router features.
+
+The TR Ultimate Test Harness may extend [`/test/router`](https://github.com/apache/trafficcontrol/tree/RELEASE-6.0.1/test/router) where possible, but it should not limit itself for that secondary goal.
+
+### Traffic Portal Impact
+<!--
+*How* will this impact Traffic Portal?
+What new UI changes will be required?
+Will entirely new pages/views be necessary?
+Will a new field be added to an existing form?
+How will the user interact with the new UI changes?
+-->
+No Traffic Portal impact is anticipated.
+
+### Traffic Ops Impact
+<!--
+*How* will this impact Traffic Ops (at a high level)?
+-->
+The Operations-level TR Ultimate Test Harness user will update the status of any Traffic Router that is currently undergoing load testing to `"OFFLINE"` in Traffic Ops. It should also update the status of that Traffic Router to its previous status once the load testing is complete, regardless of whether the test failed or succeeded.
+
+#### REST API Impact
+<!--
+*How* will this impact the Traffic Ops REST API?
+
+What new endpoints will be required?
+How will existing endpoints be changed?
+What will the requests and responses look like?
+What fields are required or optional?
+What are the defaults for optional fields?
+What are the validation constraints?
+-->
+No Traffic Ops REST API impact is anticipated.
+
+#### Client Impact
+<!--
+*How* will this impact Traffic Ops REST API clients (Go, Python, Java)?
+
+If new endpoints are required, will corresponding client methods be added?
+-->
+Clients importing the `github.com/apache/trafficcontrol/lib/go-tc` package will optionally be able to import a constant for `X-MM-Client-IP`, a request header Traffic Router to specify to Traffic Router the IP address to use to geolocate that client:
+https://github.com/apache/trafficcontrol/blob/1ed2964d16618aeebef142b01a538336a44d07dd/traffic_router/core/src/main/java/org/apache/traffic_control/traffic_router/core/request/HTTPRequest.java#L29
+
+Additionally, a struct used to unmarshall a Coverage Zone File could be placed in `lib/go-tc`.
+
+#### Data Model / Database Impact
+<!--
+*How* will this impact the Traffic Ops data model?
+*How* will this impact the Traffic Ops database schema?
+
+What changes to the lib/go-tc structs will be required?
+What new tables and columns will be required?
+How will existing tables and columns be changed?
+What are the column data types and modifiers?
+What are the FK references and constraints?
+-->
+No Data Model impact is anticipated.
+
+### Cache Config Impact
+<!--
+*How* will this impact ORT?
+-->
+No Cache Config impact is anticipated.
+
+### Traffic Monitor Impact
+<!--
+*How* will this impact Traffic Monitor?
+
+Will new profile parameters be required?
+-->
+No Traffic Monitor impact is anticipated.
+
+### Traffic Router Impact
+<!--
+*How* will this impact Traffic Router?
+
+Will new profile parameters be required?
+How will the CRConfig be changed?
+How will changes in Traffic Ops data be reflected in the CRConfig?
+Will Traffic Router remain backwards-compatible with old CRConfigs?
+Will old Traffic Routers remain forwards-compatible with new CRConfigs?
+-->
+The addition of the TR Ultimate Test Harness themselves will not change Traffic Router functionality in any way. For visibility, however, the TR Ultimate Test Harness should reside in a directory within the `traffic_router` directory. This will be the first time since 545929f7cc that Golang sources will exist in the `traffic_router` directory, so any assumption that all sources within the `traffic_router` directory directly impact Traffic Router's ability to compile should be abandoned.
+
+The TR Ultimate Test Harness should not be included in the Traffic Router RPM, as it is meant to be run on a host separate from Traffic Routers.
+
+### Traffic Stats Impact
+<!--
+*How* will this impact Traffic Stats?
+-->
+No Traffic Stats impact is anticipated.
+
+### Traffic Vault Impact
+<!--
+*How* will this impact Traffic Vault?
+
+Will there be any new data stored in or removed from Riak?
+Will there be any changes to the Riak requests and responses?
+-->
+No Traffic Vault impact is anticipated.
+
+### Documentation Impact
+<!--
+*How* will this impact the documentation?
+
+What new documentation will be required?
+What existing documentation will need to be updated?
+-->
+Instructions for using the Traffic Router Ultimate Test Harness should be added to the documentation. This should include:
+- Small rationale for inclusion of TR Ultimate Test Harness
+- Setup instructions
+ * The permissions that a user running the Traffic Router Test Harness should have:
+ - CDN snapshots
+ - CDN information
+ - information about Traffic Router-type Servers in those CDNs
+ - Type information
+ - Delivery Service information
+ - Documentation of each CLI option
+ - Example commands
+
+### Testing Impact
+<!--
+*How* will this impact testing?
+
+What is the high-level test plan?
+How should this be tested?
+Can this be tested within the existing test frameworks?
+How should the existing frameworks be enhanced in order to test this properly?
+-->
+#### Load Tests
+The Router Ultimate Test Harness should include a load test for HTTP-routed Delivery Services and for DNS-routed Delivery Services
+
+##### Load Test Options
Review comment:
Are these going to be CLI options? Will there be a config file? Some of these options seem more generic (e.g. CDN name), while some of them are load-test specific (e.g. req/s threshold). If we extend the test harness with more than just load testing, would each different type of test have its own config options and/or file?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@trafficcontrol.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [trafficcontrol] mitchell852 commented on pull request #6358: Traffic Router Ultimate Test Harness blueprint
Posted by GitBox <gi...@apache.org>.
mitchell852 commented on pull request #6358:
URL: https://github.com/apache/trafficcontrol/pull/6358#issuecomment-973172649
@zrhoffman - do you envision the TRUTH :) will someday run via a GitHub Action triggered by any code changes to TR and will fail if those changes negatively impact TR performance?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@trafficcontrol.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [trafficcontrol] zrhoffman commented on a change in pull request #6358: Traffic Router Ultimate Test Harness blueprint
Posted by GitBox <gi...@apache.org>.
zrhoffman commented on a change in pull request #6358:
URL: https://github.com/apache/trafficcontrol/pull/6358#discussion_r765313558
##########
File path: blueprints/tr-load-tests.md
##########
@@ -102,7 +109,7 @@ How will existing tables and columns be changed?
What are the column data types and modifiers?
What are the FK references and constraints?
-->
-No Data Model impact is anticipated.
+A database migration should be added to seed a new Operations-level user for use with the TR Ultimate Test Harness.
Review comment:
Good points, removed mention of the *user creation* migration section in f202723c5c.
##########
File path: blueprints/tr-load-tests.md
##########
@@ -0,0 +1,296 @@
+<!--
+Licensed to the Apache Software Foundation (ASF) under one
+or more contributor license agreements. See the NOTICE file
+distributed with this work for additional information
+regarding copyright ownership. The ASF licenses this file
+to you under the Apache License, Version 2.0 (the
+"License"); you may not use this file except in compliance
+with the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing,
+software distributed under the License is distributed on an
+"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+KIND, either express or implied. See the License for the
+specific language governing permissions and limitations
+under the License.
+-->
+# Traffic Router Ultimate Test Harness <!-- a concise title for the new feature this blueprint will describe -->
+
+## Problem Description
+<!--
+*What* is being asked for?
+*Why* is this necessary?
+*How* will this be used?
+-->
+As the entrypoint of an Apache Traffic Control CDN, Traffic Router is the most exposed, most critical component, and it must be able to route traffic quickly and at a very high rate. Although Traffic Router has met this requirement in the past, if Traffic Router undergoes no performance tests as it evolves, there is no guarantee that its performance will not decline with time.
+
+Important times to run performance tests:
+
+* When traffic volume to your CDN changes significantly
+* When making harware changes to the server hosting the Traffic Router instance
+* When upgrading Traffic Router to a new Apache Traffic Control version
+* When developing or maintaining a Traffic Router feature that could have an impact on any aspect of Traffic Router performance, including before, during, and after pull request review
+* When a commit that modifies Traffic Router is pushed to a GitHub branch, for all Apache Traffic Control branches
+
+A load test for Delivery Services exists in the project at [`/test/router`](https://github.com/apache/trafficcontrol/tree/RELEASE-6.0.1/test/router), but
+- It has not been maintained over time, currently does not work
+- Only tests HTTP Delivery Services
+- Does not support testing Coverage Zone Maps
+- Does not use the TO Client Library for requests to Traffic Ops
+- Prompts only for the number of requests to make to Delivery Services, not a length of time to run the test
+- Does not fail if some minimum threshold of requests per second is not met
+- Is not configurable in other ways, such as length of paths generated for HTTP requests to Delivery Service, which client IP address to use
+
+## Proposed Change
+<!--
+*How* will this be implemented (at a high level)?
+-->
+The Traffic Router Ultimate Test Harness will include an end-to-end performance test suite verify that the features of Traffic Router meet expected performance thresholds, as well as additional end-to-end tests of other Traffic Router features.
+
+The TR Ultimate Test Harness may extend [`/test/router`](https://github.com/apache/trafficcontrol/tree/RELEASE-6.0.1/test/router) where possible, but it should not limit itself for that secondary goal.
+
+### Traffic Portal Impact
+<!--
+*How* will this impact Traffic Portal?
+What new UI changes will be required?
+Will entirely new pages/views be necessary?
+Will a new field be added to an existing form?
+How will the user interact with the new UI changes?
+-->
+No Traffic Portal impact is anticipated.
+
+### Traffic Ops Impact
+<!--
+*How* will this impact Traffic Ops (at a high level)?
+-->
+The Operations-level TR Ultimate Test Harness user will update the status of any Traffic Router that is currently undergoing load testing to `"OFFLINE"` in Traffic Ops. It should also update the status of that Traffic Router to its previous status once the load testing is complete, regardless of whether the test failed or succeeded.
+
+The Operations-level TR Ultimate Test Harness user will also be used to get
+* CDN snapshots
+* CDN information
+* information about Traffic Router-type Servers in those CDNs
+* Type information
+* Delivery Service information
Review comment:
True, moved to the *documentation* section in 8e5b1f14d4.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@trafficcontrol.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [trafficcontrol] zrhoffman commented on a change in pull request #6358: Traffic Router Ultimate Test Harness blueprint
Posted by GitBox <gi...@apache.org>.
zrhoffman commented on a change in pull request #6358:
URL: https://github.com/apache/trafficcontrol/pull/6358#discussion_r777690754
##########
File path: blueprints/tr-load-tests.md
##########
@@ -0,0 +1,296 @@
+<!--
+Licensed to the Apache Software Foundation (ASF) under one
+or more contributor license agreements. See the NOTICE file
+distributed with this work for additional information
+regarding copyright ownership. The ASF licenses this file
+to you under the Apache License, Version 2.0 (the
+"License"); you may not use this file except in compliance
+with the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing,
+software distributed under the License is distributed on an
+"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+KIND, either express or implied. See the License for the
+specific language governing permissions and limitations
+under the License.
+-->
+# Traffic Router Ultimate Test Harness <!-- a concise title for the new feature this blueprint will describe -->
+
+## Problem Description
+<!--
+*What* is being asked for?
+*Why* is this necessary?
+*How* will this be used?
+-->
+As the entrypoint of an Apache Traffic Control CDN, Traffic Router is the most exposed, most critical component, and it must be able to route traffic quickly and at a very high rate. Although Traffic Router has met this requirement in the past, if Traffic Router undergoes no performance tests as it evolves, there is no guarantee that its performance will not decline with time.
+
+Important times to run performance tests:
+
+* When traffic volume to your CDN changes significantly
+* When making harware changes to the server hosting the Traffic Router instance
+* When upgrading Traffic Router to a new Apache Traffic Control version
+* When developing or maintaining a Traffic Router feature that could have an impact on any aspect of Traffic Router performance, including before, during, and after pull request review
+* When a commit that modifies Traffic Router is pushed to a GitHub branch, for all Apache Traffic Control branches
+
+A load test for Delivery Services exists in the project at [`/test/router`](https://github.com/apache/trafficcontrol/tree/RELEASE-6.0.1/test/router), but
+- It has not been maintained over time, currently does not work
+- Only tests HTTP Delivery Services
+- Does not support testing Coverage Zone Maps
+- Does not use the TO Client Library for requests to Traffic Ops
+- Prompts only for the number of requests to make to Delivery Services, not a length of time to run the test
+- Does not fail if some minimum threshold of requests per second is not met
+- Is not configurable in other ways, such as length of paths generated for HTTP requests to Delivery Service, which client IP address to use
+
+## Proposed Change
+<!--
+*How* will this be implemented (at a high level)?
+-->
+The Traffic Router Ultimate Test Harness will include an end-to-end performance test suite verify that the features of Traffic Router meet expected performance thresholds, as well as additional end-to-end tests of other Traffic Router features.
+
+The TR Ultimate Test Harness may extend [`/test/router`](https://github.com/apache/trafficcontrol/tree/RELEASE-6.0.1/test/router) where possible, but it should not limit itself for that secondary goal.
+
+### Traffic Portal Impact
+<!--
+*How* will this impact Traffic Portal?
+What new UI changes will be required?
+Will entirely new pages/views be necessary?
+Will a new field be added to an existing form?
+How will the user interact with the new UI changes?
+-->
+No Traffic Portal impact is anticipated.
+
+### Traffic Ops Impact
+<!--
+*How* will this impact Traffic Ops (at a high level)?
+-->
+The Operations-level TR Ultimate Test Harness user will update the status of any Traffic Router that is currently undergoing load testing to `"OFFLINE"` in Traffic Ops. It should also update the status of that Traffic Router to its previous status once the load testing is complete, regardless of whether the test failed or succeeded.
+
+#### REST API Impact
+<!--
+*How* will this impact the Traffic Ops REST API?
+
+What new endpoints will be required?
+How will existing endpoints be changed?
+What will the requests and responses look like?
+What fields are required or optional?
+What are the defaults for optional fields?
+What are the validation constraints?
+-->
+No Traffic Ops REST API impact is anticipated.
+
+#### Client Impact
+<!--
+*How* will this impact Traffic Ops REST API clients (Go, Python, Java)?
+
+If new endpoints are required, will corresponding client methods be added?
+-->
+Clients importing the `github.com/apache/trafficcontrol/lib/go-tc` package will optionally be able to import a constant for `X-MM-Client-IP`, a request header Traffic Router to specify to Traffic Router the IP address to use to geolocate that client:
+https://github.com/apache/trafficcontrol/blob/1ed2964d16618aeebef142b01a538336a44d07dd/traffic_router/core/src/main/java/org/apache/traffic_control/traffic_router/core/request/HTTPRequest.java#L29
+
+Additionally, a struct used to unmarshall a Coverage Zone File could be placed in `lib/go-tc`.
+
+#### Data Model / Database Impact
+<!--
+*How* will this impact the Traffic Ops data model?
+*How* will this impact the Traffic Ops database schema?
+
+What changes to the lib/go-tc structs will be required?
+What new tables and columns will be required?
+How will existing tables and columns be changed?
+What are the column data types and modifiers?
+What are the FK references and constraints?
+-->
+No Data Model impact is anticipated.
+
+### Cache Config Impact
+<!--
+*How* will this impact ORT?
+-->
+No Cache Config impact is anticipated.
+
+### Traffic Monitor Impact
+<!--
+*How* will this impact Traffic Monitor?
+
+Will new profile parameters be required?
+-->
+No Traffic Monitor impact is anticipated.
+
+### Traffic Router Impact
+<!--
+*How* will this impact Traffic Router?
+
+Will new profile parameters be required?
+How will the CRConfig be changed?
+How will changes in Traffic Ops data be reflected in the CRConfig?
+Will Traffic Router remain backwards-compatible with old CRConfigs?
+Will old Traffic Routers remain forwards-compatible with new CRConfigs?
+-->
+The addition of the TR Ultimate Test Harness themselves will not change Traffic Router functionality in any way. For visibility, however, the TR Ultimate Test Harness should reside in a directory within the `traffic_router` directory. This will be the first time since 545929f7cc that Golang sources will exist in the `traffic_router` directory, so any assumption that all sources within the `traffic_router` directory directly impact Traffic Router's ability to compile should be abandoned.
+
+The TR Ultimate Test Harness should not be included in the Traffic Router RPM, as it is meant to be run on a host separate from Traffic Routers.
+
+### Traffic Stats Impact
+<!--
+*How* will this impact Traffic Stats?
+-->
+No Traffic Stats impact is anticipated.
+
+### Traffic Vault Impact
+<!--
+*How* will this impact Traffic Vault?
+
+Will there be any new data stored in or removed from Riak?
+Will there be any changes to the Riak requests and responses?
+-->
+No Traffic Vault impact is anticipated.
+
+### Documentation Impact
+<!--
+*How* will this impact the documentation?
+
+What new documentation will be required?
+What existing documentation will need to be updated?
+-->
+Instructions for using the Traffic Router Ultimate Test Harness should be added to the documentation. This should include:
+- Small rationale for inclusion of TR Ultimate Test Harness
+- Setup instructions
+ * The permissions that a user running the Traffic Router Test Harness should have:
+ - CDN snapshots
+ - CDN information
+ - information about Traffic Router-type Servers in those CDNs
+ - Type information
+ - Delivery Service information
+ - Documentation of each CLI option
+ - Example commands
+
+### Testing Impact
+<!--
+*How* will this impact testing?
+
+What is the high-level test plan?
+How should this be tested?
+Can this be tested within the existing test frameworks?
+How should the existing frameworks be enhanced in order to test this properly?
+-->
+#### Load Tests
+The Router Ultimate Test Harness should include a load test for HTTP-routed Delivery Services and for DNS-routed Delivery Services
+
+##### Load Test Options
Review comment:
> Are these going to be CLI options?
No
> Will there be a config file?
Config file structure clarified in c0bd076c46
> If we extend the test harness with more than just load testing, would each different type of test have its own config options and/or file?
Each test can specify its own options, but options applying to more than one test can go under `"all"`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@trafficcontrol.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [trafficcontrol] mitchell852 merged pull request #6358: Traffic Router Ultimate Test Harness blueprint
Posted by GitBox <gi...@apache.org>.
mitchell852 merged pull request #6358:
URL: https://github.com/apache/trafficcontrol/pull/6358
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@trafficcontrol.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org