You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@trafficcontrol.apache.org by GitBox <gi...@apache.org> on 2020/11/16 20:31:43 UTC
[GitHub] [trafficcontrol] zrhoffman opened a new pull request #5283: Remove unnecessary CDN in a Box waits
zrhoffman opened a new pull request #5283:
URL: https://github.com/apache/trafficcontrol/pull/5283
<!--
************ STOP!! ************
If this Pull Request is intended to fix a security vulnerability, DO NOT submit it! Instead, contact
the Apache Software Foundation Security Team at security@trafficcontrol.apache.org and follow the
guidelines at https://www.apache.org/security/ regarding vulnerability disclosure.
-->
## What does this PR (Pull Request) do?
<!-- Explain the changes you made here. If this fixes an Issue, identify it by
replacing the text in the checkbox item with the Issue number e.g.
- [x] This PR fixes #9001 OR is not related to any Issue
^ This will automatically close Issue number 9001 when the Pull Request is
merged (The '#' is important).
Be sure you check the box properly, see the "The following criteria are ALL
met by this PR" section for details.
-->
* [x] This PR is not related to any Issue <!-- You can check for an issue here: https://github.com/apache/trafficcontrol/issues -->
* This PR makes CDN in a Box start up faster.
- Lets you skip things waiting for the enroller initial data load to complete. This saves about 24 seconds.
- Lets you set `DIG_IP_RETRY` from `set-to-ips-from-dns.sh` to any value, letting you optionally skip this step . When set to 0, this saves 30 seconds per missing service. So, in the case of the Traffic Ops API tests, where the `trafficportal`, `trafficmonitor`, and `trafficrouter` are not included, this saves 90 seconds.
- Traffic Vault now waits to enroll itself until after Riak finishes starting up. Saves about 24 seconds (the amount of time the enroller initial data load takes).
Being able to skip the enroller initial data load is a bugfix for the Traffic Ops API tests. Right now, if you start the `integration` service (from `docker-compose.traffic-ops-test.yml`) when starting all of the other CDN in a Box services, the Traffic Ops API v1 tests fail (but v2 and v3 succeed) at the end of `TestCacheGroupsDeliveryServices` due to data conflicts:
```
types_test.go:96: ---- DeleteTestTypes ----
types_test.go:108: cannot DELETE Type by name: 400 Bad Request[400] - Error requesting Traffic Ops https://trafficops.infra.ciab.test:443/api/1.3/types/135 {"alerts":[{"text":"cannot delete type because it is being used by a cachegroup","level":"error"}]}
- {[]}
types_test.go:117: expected Type name: ORG_LOC to be deleted
types_test.go:108: cannot DELETE Type by name: 400 Bad Request[400] - Error requesting Traffic Ops https://trafficops.infra.ciab.test:443/api/1.3/types/136 {"alerts":[{"text":"cannot delete type because it is being used by a cachegroup","level":"error"}]}
- {[]}
types_test.go:117: expected Type name: MID_LOC to be deleted
types_test.go:108: cannot DELETE Type by name: 400 Bad Request[400] - Error requesting Traffic Ops https://trafficops.infra.ciab.test:443/api/1.3/types/145 {"alerts":[{"text":"cannot delete type because it is being used by a cachegroup","level":"error"}]}
- {[]}
types_test.go:117: expected Type name: EDGE_LOC to be deleted
--- FAIL: TestCacheGroupsDeliveryServices (8.69s)
```
## Which Traffic Control components are affected by this PR?
<!-- Please delete all components from this list that are NOT affected by this
Pull Request. Also, feel free to add the name of a tool or script that is
affected but not on the list.
Additionally, if this Pull Request does NOT affect documentation, please
explain why documentation is not required. -->
- CDN in a Box
## What is the best way to verify this PR?
<!-- Please include here ALL the steps necessary to test your Pull Request. If
it includes tests (and most should), outline here the steps needed to run the
tests. If not, lay out the manual testing procedure and please explain why
tests are unnecessary for this Pull Request. -->
* Make sure the Traffic Ops API tests pass when `dig` retries and enroller initial data loading is disabled:
```
export DIG_IP_RETRY=0 LOAD_TRAFFIC_OPS_DATA=false;
docker-compose up --no-deps -d edge dns db smtp trafficops-perl trafficops trafficvault;
docker-compose -f docker-compose.traffic-ops-test.yml -f docker-compose.yml up integration;
```
* Make sure CDN in a Box still works when started normally.
## If this is a bug fix, what versions of Traffic Control are affected?
<!-- If this PR fixes a bug, please list here all of the affected versions - to
the best of your knowledge. It's also pretty helpful to include a commit hash
of where 'master' is at the time this PR is opened (if it affects master),
because what 'master' means will change over time. For example, if this PR
fixes a bug that's present in master (at commit hash '1df853c8'), in v4.0.0,
and in the current 4.0.1 Release candidate (e.g. RC1), then this list would
look like:
- master (1df853c8)
- 4.0.0
- 4.0.1 (RC1)
If you don't know what other versions might have this bug, AND don't know how
to find the commit hash of 'master', then feel free to leave this section
blank (or, preferably, delete it entirely).
-->
For the Traffic Ops API tests:
- master (f6a8056513)
- 5.0.0 (RC1)
## The following criteria are ALL met by this PR
<!-- Check the boxes to signify that the associated statement is true. To
"check a box", replace the space inside of the square brackets with an 'x'.
e.g.
- [ x] <- Wrong
- [x ] <- Wrong
- [] <- Wrong
- [*] <- Wrong
- [x] <- Correct!
-->
- [x] This PR fixes tests
- [x] This PR includes documentation
- [x] An update to CHANGELOG.md is not necessary
- [x] This PR includes any and all required license headers
- [x] This PR **DOES NOT FIX A SERIOUS SECURITY VULNERABILITY** (see [the Apache Software Foundation's security guidelines](https://www.apache.org/security/) for details)
<!--
Licensed to the Apache Software Foundation (ASF) under one
or more contributor license agreements. See the NOTICE file
distributed with this work for additional information
regarding copyright ownership. The ASF licenses this file
to you under the Apache License, Version 2.0 (the
"License"); you may not use this file except in compliance
with the License. You may obtain a copy of the License at
http://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing,
software distributed under the License is distributed on an
"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
KIND, either express or implied. See the License for the
specific language governing permissions and limitations
under the License.
-->
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [trafficcontrol] rawlinp commented on a change in pull request #5283: Remove unnecessary CDN in a Box waits
Posted by GitBox <gi...@apache.org>.
rawlinp commented on a change in pull request #5283:
URL: https://github.com/apache/trafficcontrol/pull/5283#discussion_r525372832
##########
File path: infrastructure/cdn-in-a-box/variables.env
##########
@@ -122,3 +122,5 @@ AUTO_SNAPQUEUE_ENABLED=true
AUTO_SNAPQUEUE_SERVERS=trafficops,trafficops-perl,trafficmonitor,trafficrouter,trafficvault,edge,mid-01,mid-02
AUTO_SNAPQUEUE_POLL_INTERVAL=2
AUTO_SNAPQUEUE_ACTION_WAIT=2
+LOAD_TRAFFIC_OPS_DATA=${LOAD_TRAFFIC_OPS_DATA:-true}
Review comment:
Looking at https://docs.docker.com/compose/env-file/ it doesn't seem to indicate that variable interpolation can be done in a .env file. It seems like it only accepts `VAR=VAL`. Maybe you have that variable set in your environment already?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [trafficcontrol] zrhoffman commented on a change in pull request #5283: Remove unnecessary CDN in a Box waits
Posted by GitBox <gi...@apache.org>.
zrhoffman commented on a change in pull request #5283:
URL: https://github.com/apache/trafficcontrol/pull/5283#discussion_r525406812
##########
File path: infrastructure/cdn-in-a-box/variables.env
##########
@@ -122,3 +122,5 @@ AUTO_SNAPQUEUE_ENABLED=true
AUTO_SNAPQUEUE_SERVERS=trafficops,trafficops-perl,trafficmonitor,trafficrouter,trafficvault,edge,mid-01,mid-02
AUTO_SNAPQUEUE_POLL_INTERVAL=2
AUTO_SNAPQUEUE_ACTION_WAIT=2
+LOAD_TRAFFIC_OPS_DATA=${LOAD_TRAFFIC_OPS_DATA:-true}
Review comment:
From my testing, parameter expansion only works in .env files in docker/compose@6d2658ea65 and up (stable docker-compose versions 1.26.0 and up), when docker-compose started using `python-dotenv` to process .env files.
Specifying those environment variables in `docker-compose.yml` in 4b7d9a0a4e.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [trafficcontrol] zrhoffman commented on a change in pull request #5283: Remove unnecessary CDN in a Box waits
Posted by GitBox <gi...@apache.org>.
zrhoffman commented on a change in pull request #5283:
URL: https://github.com/apache/trafficcontrol/pull/5283#discussion_r525406812
##########
File path: infrastructure/cdn-in-a-box/variables.env
##########
@@ -122,3 +122,5 @@ AUTO_SNAPQUEUE_ENABLED=true
AUTO_SNAPQUEUE_SERVERS=trafficops,trafficops-perl,trafficmonitor,trafficrouter,trafficvault,edge,mid-01,mid-02
AUTO_SNAPQUEUE_POLL_INTERVAL=2
AUTO_SNAPQUEUE_ACTION_WAIT=2
+LOAD_TRAFFIC_OPS_DATA=${LOAD_TRAFFIC_OPS_DATA:-true}
Review comment:
From my testing, parameter expansion only works in .env files in docker/compose@6d2658ea65 and up (stable docker-compose versions 1.26.0 and up), when docker-compose started using `python-dotenv` to process .env files.
Specifying those environment variables in `docker-compose.yml` in e992a16041.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [trafficcontrol] rawlinp merged pull request #5283: Remove unnecessary CDN in a Box waits
Posted by GitBox <gi...@apache.org>.
rawlinp merged pull request #5283:
URL: https://github.com/apache/trafficcontrol/pull/5283
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [trafficcontrol] rawlinp commented on a change in pull request #5283: Remove unnecessary CDN in a Box waits
Posted by GitBox <gi...@apache.org>.
rawlinp commented on a change in pull request #5283:
URL: https://github.com/apache/trafficcontrol/pull/5283#discussion_r525301043
##########
File path: infrastructure/cdn-in-a-box/variables.env
##########
@@ -122,3 +122,5 @@ AUTO_SNAPQUEUE_ENABLED=true
AUTO_SNAPQUEUE_SERVERS=trafficops,trafficops-perl,trafficmonitor,trafficrouter,trafficvault,edge,mid-01,mid-02
AUTO_SNAPQUEUE_POLL_INTERVAL=2
AUTO_SNAPQUEUE_ACTION_WAIT=2
+LOAD_TRAFFIC_OPS_DATA=${LOAD_TRAFFIC_OPS_DATA:-true}
Review comment:
These don't appear to be evaluating as if this were a bash script. This variable is getting set to literal `'${LOAD_TRAFFIC_OPS_DATA:-true}'`
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [trafficcontrol] rawlinp commented on a change in pull request #5283: Remove unnecessary CDN in a Box waits
Posted by GitBox <gi...@apache.org>.
rawlinp commented on a change in pull request #5283:
URL: https://github.com/apache/trafficcontrol/pull/5283#discussion_r525369241
##########
File path: infrastructure/cdn-in-a-box/variables.env
##########
@@ -122,3 +122,5 @@ AUTO_SNAPQUEUE_ENABLED=true
AUTO_SNAPQUEUE_SERVERS=trafficops,trafficops-perl,trafficmonitor,trafficrouter,trafficvault,edge,mid-01,mid-02
AUTO_SNAPQUEUE_POLL_INTERVAL=2
AUTO_SNAPQUEUE_ACTION_WAIT=2
+LOAD_TRAFFIC_OPS_DATA=${LOAD_TRAFFIC_OPS_DATA:-true}
Review comment:
I added an echo statement to the trafficops-init.sh script:
```
echo "LOAD_TRAFFIC_OPS_DATA = $LOAD_TRAFFIC_OPS_DATA. Skipping loading of traffic ops data."
```
output:
```
+ echo 'LOAD_TRAFFIC_OPS_DATA = ${LOAD_TRAFFIC_OPS_DATA:-true}. Skipping loading of traffic ops data.'
LOAD_TRAFFIC_OPS_DATA = ${LOAD_TRAFFIC_OPS_DATA:-true}. Skipping loading of traffic ops data.
```
When I run that command you pasted, I don't see any output. Maybe a difference in docker-compose versions?
```
$ docker-compose version
docker-compose version 1.18.0, build 8dd22a9
docker-py version: 2.6.1
CPython version: 3.6.8
OpenSSL version: OpenSSL 1.0.2k-fips 26 Jan 2017
$ docker version
Client:
Version: 18.06.3-ce
API version: 1.38
Go version: go1.10.3
Git commit: d7080c1
Built: Wed Feb 20 02:26:51 2019
OS/Arch: linux/amd64
Experimental: false
Server:
Engine:
Version: 18.06.3-ce
API version: 1.38 (minimum version 1.12)
Go version: go1.10.3
Git commit: d7080c1
Built: Wed Feb 20 02:28:17 2019
OS/Arch: linux/amd64
Experimental: false
$ cat /etc/centos-release
CentOS Linux release 7.4.1708 (Core)
```
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [trafficcontrol] zrhoffman commented on a change in pull request #5283: Remove unnecessary CDN in a Box waits
Posted by GitBox <gi...@apache.org>.
zrhoffman commented on a change in pull request #5283:
URL: https://github.com/apache/trafficcontrol/pull/5283#discussion_r525339935
##########
File path: infrastructure/cdn-in-a-box/variables.env
##########
@@ -122,3 +122,5 @@ AUTO_SNAPQUEUE_ENABLED=true
AUTO_SNAPQUEUE_SERVERS=trafficops,trafficops-perl,trafficmonitor,trafficrouter,trafficvault,edge,mid-01,mid-02
AUTO_SNAPQUEUE_POLL_INTERVAL=2
AUTO_SNAPQUEUE_ACTION_WAIT=2
+LOAD_TRAFFIC_OPS_DATA=${LOAD_TRAFFIC_OPS_DATA:-true}
Review comment:
Where are you seeing `LOAD_TRAFFIC_OPS_DATA` being set to literal `${LOAD_TRAFFIC_OPS_DATA:-true}`? I tested on macOS with Docker Desktop and on Linux and was unable to reproduce. Testcase:
```shell
docker-compose run --rm edge sh -c 'echo $LOAD_TRAFFIC_OPS_DATA'
```
echos `true` for me.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [trafficcontrol] zrhoffman commented on a change in pull request #5283: Remove unnecessary CDN in a Box waits
Posted by GitBox <gi...@apache.org>.
zrhoffman commented on a change in pull request #5283:
URL: https://github.com/apache/trafficcontrol/pull/5283#discussion_r525339935
##########
File path: infrastructure/cdn-in-a-box/variables.env
##########
@@ -122,3 +122,5 @@ AUTO_SNAPQUEUE_ENABLED=true
AUTO_SNAPQUEUE_SERVERS=trafficops,trafficops-perl,trafficmonitor,trafficrouter,trafficvault,edge,mid-01,mid-02
AUTO_SNAPQUEUE_POLL_INTERVAL=2
AUTO_SNAPQUEUE_ACTION_WAIT=2
+LOAD_TRAFFIC_OPS_DATA=${LOAD_TRAFFIC_OPS_DATA:-true}
Review comment:
Where are you seeing `LOAD_TRAFFIC_OPS_DATA` being set to literal `${LOAD_TRAFFIC_OPS_DATA:-true}`? I tested on macOS with Docker Desktop and on Linux and was unable to reproduce. Testcase:
```shell
docker-compose run --rm edge sh -c 'echo $LOAD_TRAFFIC_OPS_DATA'
```
echoes `true` for me.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org