You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@trafficcontrol.apache.org by GitBox <gi...@apache.org> on 2021/05/06 16:31:20 UTC
[GitHub] [trafficcontrol] shamrickus opened a new pull request #5817: Traffic Monitor Integration Test Framework
shamrickus opened a new pull request #5817:
URL: https://github.com/apache/trafficcontrol/pull/5817
<!--
************ STOP!! ************
If this Pull Request is intended to fix a security vulnerability, DO NOT submit it! Instead, contact
the Apache Software Foundation Security Team at security@trafficcontrol.apache.org and follow the
guidelines at https://www.apache.org/security/ regarding vulnerability disclosure.
-->
## What does this PR (Pull Request) do?
<!-- Explain the changes you made here. If this fixes an Issue, identify it by
replacing the text in the checkbox item with the Issue number e.g.
- [x] This PR fixes #9001 OR is not related to any Issue
^ This will automatically close Issue number 9001 when the Pull Request is
merged (The '#' is important).
Be sure you check the box properly, see the "The following criteria are ALL
met by this PR" section for details.
-->
- [x] This PR is not related to any Issue <!-- You can check for an issue here: https://github.com/apache/trafficcontrol/issues -->
It is based off of #3707 and includes fixes for dual homing changes as well as addressing the feedback posted on the previous PR.
Also adds a TM client as well for use in the tests.
## Which Traffic Control components are affected by this PR?
<!-- Please delete all components from this list that are NOT affected by this
Pull Request. Also, feel free to add the name of a tool or script that is
affected but not on the list.
Additionally, if this Pull Request does NOT affect documentation, please
explain why documentation is not required. -->
- Traffic Monitor
- CI tests
## What is the best way to verify this PR?
Run the integration tests by following the instructions found in the `tests/_integration/README.md`.
Should also verify that @zrhoffman's comments on the previous PR have been addressed.
<!-- Please include here ALL the steps necessary to test your Pull Request. If
it includes tests (and most should), outline here the steps needed to run the
tests. If not, lay out the manual testing procedure and please explain why
tests are unnecessary for this Pull Request. -->
## If this is a bug fix, what versions of Traffic Control are affected?
<!-- If this PR fixes a bug, please list here all of the affected versions - to
the best of your knowledge. It's also pretty helpful to include a commit hash
of where 'master' is at the time this PR is opened (if it affects master),
because what 'master' means will change over time. For example, if this PR
fixes a bug that's present in master (at commit hash '1df853c8'), in v4.0.0,
and in the current 4.0.1 Release candidate (e.g. RC1), then this list would
look like:
- master (1df853c8)
- 4.0.0
- 4.0.1 (RC1)
If you don't know what other versions might have this bug, AND don't know how
to find the commit hash of 'master', then feel free to leave this section
blank (or, preferably, delete it entirely).
-->
## The following criteria are ALL met by this PR
<!-- Check the boxes to signify that the associated statement is true. To
"check a box", replace the space inside of the square brackets with an 'x'.
e.g.
- [ x] <- Wrong
- [x ] <- Wrong
- [] <- Wrong
- [*] <- Wrong
- [x] <- Correct!
-->
- [x] This PR includes tests OR I have explained why tests are unnecessary
- [x] This PR includes documentation OR I have explained why documentation is unnecessary
- [x] This PR includes an update to CHANGELOG.md OR such an update is not necessary
- [x] This PR includes any and all required license headers
- [x] This PR **DOES NOT FIX A SERIOUS SECURITY VULNERABILITY** (see [the Apache Software Foundation's security guidelines](https://www.apache.org/security/) for details)
<!-- If you would like to include any additional information on the PR for
potential reviewers please put it here.
Some examples of this would be:
- Before and after screenshots/gifs of the Traffic Portal if it is affected
- Links to other dependent Pull Requests
- References to relevant context (e.g. new/updates to dependent libraries,
mailing list records, blueprints)
Feel free to leave this section blank (or, preferably, delete it entirely).
-->
<!--
Licensed to the Apache Software Foundation (ASF) under one
or more contributor license agreements. See the NOTICE file
distributed with this work for additional information
regarding copyright ownership. The ASF licenses this file
to you under the Apache License, Version 2.0 (the
"License"); you may not use this file except in compliance
with the License. You may obtain a copy of the License at
http://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing,
software distributed under the License is distributed on an
"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
KIND, either express or implied. See the License for the
specific language governing permissions and limitations
under the License.
-->
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [trafficcontrol] zrhoffman commented on a change in pull request #5817: Traffic Monitor Integration Test Framework
Posted by GitBox <gi...@apache.org>.
zrhoffman commented on a change in pull request #5817:
URL: https://github.com/apache/trafficcontrol/pull/5817#discussion_r634347622
##########
File path: traffic_monitor/tmclient/tmclient.go
##########
@@ -23,6 +23,7 @@ import (
"encoding/json"
"errors"
"fmt"
+ "github.com/apache/trafficcontrol/lib/go-log"
Review comment:
Another *apache/trafficcontrol* that should be grouped separately from standard library imports.
##########
File path: traffic_monitor/tests/_integration/kbps_test.go
##########
@@ -16,6 +16,7 @@ package _integration
import (
"fmt"
+ "github.com/apache/trafficcontrol/lib/go-log"
Review comment:
apache/trafficcontrol imports should be grouped separately from standard library imports.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [trafficcontrol] zrhoffman commented on a change in pull request #5817: Traffic Monitor Integration Test Framework
Posted by GitBox <gi...@apache.org>.
zrhoffman commented on a change in pull request #5817:
URL: https://github.com/apache/trafficcontrol/pull/5817#discussion_r636973883
##########
File path: traffic_monitor/Dockerfile
##########
@@ -0,0 +1,32 @@
+# Licensed to the Apache Software Foundation (ASF) under one
Review comment:
Not in the Dockerfile, you can't, but in `docker-compose.yml`, the build context can be `../..`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [trafficcontrol] shamrickus commented on a change in pull request #5817: Traffic Monitor Integration Test Framework
Posted by GitBox <gi...@apache.org>.
shamrickus commented on a change in pull request #5817:
URL: https://github.com/apache/trafficcontrol/pull/5817#discussion_r638194666
##########
File path: traffic_monitor/tests/_integration/client_test.go
##########
@@ -0,0 +1,117 @@
+package _integration
+
+/*
+ Licensed under the Apache License, Version 2.0 (the "License");
+ you may not use this file except in compliance with the License.
+ You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+ Unless required by applicable law or agreed to in writing, software
+ distributed under the License is distributed on an "AS IS" BASIS,
+ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ See the License for the specific language governing permissions and
+ limitations under the License.
+*/
+
+import (
+ "testing"
+)
+
+func TestClient(t *testing.T) {
Review comment:
Same issue as above, should be good
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [trafficcontrol] shamrickus commented on a change in pull request #5817: Traffic Monitor Integration Test Framework
Posted by GitBox <gi...@apache.org>.
shamrickus commented on a change in pull request #5817:
URL: https://github.com/apache/trafficcontrol/pull/5817#discussion_r636914274
##########
File path: traffic_monitor/tests/_integration/Dockerfile_run.sh
##########
@@ -0,0 +1,166 @@
+#!/usr/bin/env bash
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+# The following environment variables must be set (ordinarily by `docker run -e` arguments):
+# TO_URI
+# TO_USER
+# TO_PASS
+# CDN
+# Check that env vars are set
+envvars=(TESTTO_URI TESTTO_PORT TESTCACHES_URI TESTCACHES_PORT_START TM_URI)
+for v in $envvars; do
+ if [[ -z ${!v} ]]; then
+ echo "$v is unset"
+ exit 1
+ fi
+done
+
+TO_API_VERSION=4.0
+CFG_FILE=/traffic-monitor-integration-test.cfg
+
+start() {
+ printf "DEBUG traffic_monitor_integration starting\n"
+
+ exec /traffic_monitor_integration_test -test.v -cfg $CFG_FILE
+}
+
+init() {
+ wait_for_endpoint "${TESTTO_URI}/api/${TO_API_VERSION}/servers"
+ wait_for_endpoint "${TESTCACHES_URI}:${TESTCACHES_PORT_START}/_astats"
+ wait_for_endpoint "${TM_URI}"
+ TESTCACHES_ADDRESS=$(ping testcaches -4 -c 1 | head -n 1 | grep -Eo '[0-9]+.[0-9]+.[0-9]+.[0-9]+')
+ TESTCACHES_GATEWAY=$(echo $TESTCACHES_ADDRESS | sed "s/\([0-9]\+.[0-9]\+.[0-9]\+.\)[0-9]/\11/")
+
+ jq "(.. | .address?) |= \"$TESTCACHES_ADDRESS\" | (.. | .gateway?) |= \"$TESTCACHES_GATEWAY\"" \
+ /monitoring.json >/monitoring.json.tmp && mv /monitoring.json.tmp /monitoring.json
+
+ curl -Lvsk ${TESTTO_URI}/api/${TO_API_VERSION}/cdns/fake/snapshot -X POST -d "@/snapshot.json"
+
+ curl -Lvsk ${TESTTO_URI}/api/${TO_API_VERSION}/cdns/fake/configs/monitoring -X POST -d '@/monitoring.json'
+
+ curl -Lvsk ${TESTTO_URI}/api/${TO_API_VERSION}/servers -X POST -d '
+[
+ {
+ "cachegroup": "foo",
+ "cachegroupId": 0,
+ "cdnId": 1,
+ "cdnName": "fake",
+ "deliveryServices": null,
+ "fqdn": "trafficmonitor.traffic-monitor-integration.test",
+ "guid": "foo",
+ "hostName": "trafficmonitor",
+ "httpsPort": null,
+ "id": 1,
+ "iloIpAddress": null,
+ "iloIpGateway": null,
+ "iloIpNetmask": null,
+ "iloPassword": null,
+ "iloUsername": null,
+ "interfaceMtu": null,
+ "interfaceName": "bond0",
+ "ip6Address": null,
+ "ip6Gateway": null,
+ "interfaces": [
+ {
+ "ipAddresses": [
+ {
+ "address": "4.0.16.239.6",
+ "gateway": "4.0.16.239.1",
+ "serviceAddress": true
+ },
+ {
+ "address": "fc01:9400:1000:8::6",
+ "gateway": "fc01:9400:1000:8::1",
+ "serviceAddress": true
+ }
+ ],
+ "maxBandwidth": null,
+ "monitor": true,
+ "mtu": 1500,
+ "name": "eth0"
+ }
+ ],
+ "ipGateway": "4.0.0.0.1",
+ "ipNetmask": "255.255.255.0",
+ "lastUpdated": "2019",
+ "mgmtIpAddress": null,
+ "mgmtIpGateway": null,
+ "mgmtIpNetmask": null,
+ "offlineReason": "none",
+ "physLocation": "",
+ "physLocationId": 0,
+ "profile": "Monitor0",
+ "profileDesc": "nodesc",
+ "profileId": 0,
+ "rack": "",
+ "revalPending": false,
+ "routerHostName": "",
+ "routerPortName": "",
+ "status": "REPORTED",
+ "statusId": 0,
+ "tcpPort": 80,
+ "type": "RASCAL",
+ "typeId": 0,
+ "updPending": false,
+ "xmppId": "",
+ "xmppPasswd": ""
+ }
+]
+'
+
+ cat >$CFG_FILE <<-EOF
+{
+ "trafficMonitor": {
+ "url": "$TM_URI"
+ },
+ "default": {
+ "session": {
+ "timeoutInSecs": 30
+ },
+ "log": {
+ "debug": "stdout",
+ "event": "stdout",
+ "info": "stdout",
+ "error": "stdout",
+ "warning": "stdout"
+ }
+ }
+}
+EOF
+
+ echo "INITIALIZED=1" >>/etc/environment
+}
+
+function wait_for_endpoint() {
+ try=0
+ while curl -Lvsk "$1" 2>&1 1>/dev/null | grep -Eq '< HTTP/[0-9]\.?[0-9]* 200' && [[ $try -lt 5 ]]; do
+ if [[ $try -gt 0 ]]; then
Review comment:
lol :facepalm:
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [trafficcontrol] zrhoffman merged pull request #5817: Traffic Monitor Integration Test Framework
Posted by GitBox <gi...@apache.org>.
zrhoffman merged pull request #5817:
URL: https://github.com/apache/trafficcontrol/pull/5817
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [trafficcontrol] zrhoffman commented on a change in pull request #5817: Traffic Monitor Integration Test Framework
Posted by GitBox <gi...@apache.org>.
zrhoffman commented on a change in pull request #5817:
URL: https://github.com/apache/trafficcontrol/pull/5817#discussion_r633879792
##########
File path: traffic_monitor/tools/testto/Dockerfile_run.sh
##########
@@ -0,0 +1,40 @@
+#!/usr/bin/env bash
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+# envvars=( PORT )
+# for v in $envvars
+# do
+# if [[ -z $$v ]]; then echo "$v is unset"; exit 1; fi
+# done
+
+start() {
+ ARGS=
+ if [[ -n "${PORT}" ]]; then
+ ARGS="$ARGS -port ${PORT}"
+ fi
+ testto ${ARGS}
+}
+
+init() {
+
Review comment:
Nit: This blank line serves no purpose
##########
File path: traffic_monitor/tests/_integration/traffic_monitor_test.go
##########
@@ -0,0 +1,97 @@
+package _integration
+
+/*
+
+ Licensed under the Apache License, Version 2.0 (the "License");
+ you may not use this file except in compliance with the License.
+ You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+ Unless required by applicable law or agreed to in writing, software
+ distributed under the License is distributed on an "AS IS" BASIS,
+ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ See the License for the specific language governing permissions and
+ limitations under the License.
+*/
+
+import (
+ // "database/sql"
Review comment:
Unused import is commented out
##########
File path: traffic_monitor/tmclient/tmclient.go
##########
@@ -0,0 +1,260 @@
+package tmclient
+
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+import (
+ "encoding/json"
+ "errors"
+ "fmt"
+ "io/ioutil"
+ "net/http"
+ "strconv"
+ "strings"
+ "time"
+
+ "github.com/apache/trafficcontrol/lib/go-tc"
+ "github.com/apache/trafficcontrol/traffic_monitor/datareq"
+ "github.com/apache/trafficcontrol/traffic_monitor/dsdata"
+ "github.com/apache/trafficcontrol/traffic_monitor/handler"
+ "github.com/apache/trafficcontrol/traffic_monitor/towrap"
+)
+
+type TMClient struct {
+ url string
+ timeout time.Duration
+}
+
+func New(url string, timeout time.Duration) *TMClient {
+ return &TMClient{url: strings.TrimSuffix(url, "/"), timeout: timeout}
+}
+
+func (c *TMClient) CacheCount() (int, error) { return c.getInt("/api/cache-count") }
+
+func (c *TMClient) CacheAvailableCount() (int, error) { return c.getInt("/api/cache-available-count") }
+
+func (c *TMClient) CacheDownCount() (int, error) { return c.getInt("/api/cache-down-count") }
+
+func (c *TMClient) Version() (string, error) { return c.getStr("/api/version") }
+
+func (c *TMClient) TrafficOpsURI() (string, error) { return c.getStr("/api/traffic-ops-uri") }
+
+func (c *TMClient) BandwidthKBPS() (float64, error) { return c.getFloat("/api/bandwidth-kbps") }
+
+func (c *TMClient) BandwidthCapacityKBPS() (float64, error) {
+ return c.getFloat("/api/bandwidth-capacity-kbps")
+}
+
+func (c *TMClient) CacheStatuses() (map[tc.CacheName]datareq.CacheStatus, error) {
+ path := "/api/cache-statuses"
+ obj := map[tc.CacheName]datareq.CacheStatus{}
+ if err := c.GetJSON(path, &obj); err != nil {
+ return nil, err // GetJSON adds context
+ }
+ return obj, nil
+}
+
+func (c *TMClient) MonitorConfig() (tc.TrafficMonitorConfigMap, error) {
+ path := "/api/monitor-config"
+ obj := tc.TrafficMonitorConfigMap{}
+ if err := c.GetJSON(path, &obj); err != nil {
+ return tc.TrafficMonitorConfigMap{}, err // GetJSON adds context
+ }
+ return obj, nil
+}
+
+func (c *TMClient) CRConfigHistory() ([]towrap.CRConfigStat, error) {
+ path := "/api/crconfig-history"
+ obj := []towrap.CRConfigStat{}
+ if err := c.GetJSON(path, &obj); err != nil {
+ return nil, err // GetJSON adds context
+ }
+ return obj, nil
+}
+
+func (c *TMClient) EventLog() (datareq.JSONEvents, error) {
+ path := "/publish/EventLog"
+ obj := datareq.JSONEvents{}
+ if err := c.GetJSON(path, &obj); err != nil {
+ return datareq.JSONEvents{}, err // GetJSON adds context
+ }
+ return obj, nil
+}
+
+func (c *TMClient) CacheStatsNew() (tc.Stats, error) {
+ path := "/publish/CacheStats"
+ obj := tc.Stats{}
+ if err := c.GetJSON(path, &obj); err != nil {
+ return tc.Stats{}, err // GetJSON adds context
+ }
+ return obj, nil
+}
+
+func (c *TMClient) CacheStats() (tc.LegacyStats, error) {
+ path := "/publish/CacheStats"
+ obj := tc.LegacyStats{}
+ if err := c.GetJSON(path, &obj); err != nil {
+ return tc.LegacyStats{}, err // GetJSON adds context
+ }
+ return obj, nil
+}
+
+func (c *TMClient) DSStats() (dsdata.Stats, error) {
+ path := "/publish/DsStats"
+ obj := dsdata.Stats{}
+ if err := c.GetJSON(path, &obj); err != nil {
+ return dsdata.Stats{}, err // GetJSON adds context
+ }
+ return obj, nil
+}
+
+func (c *TMClient) CRStates(raw bool) (tc.CRStates, error) {
+ path := "/publish/CrStates"
+ if raw {
+ path += "?raw"
+ }
+ obj := tc.CRStates{}
+ if err := c.GetJSON(path, &obj); err != nil {
+ return tc.CRStates{}, err // GetJSON adds context
+ }
+ return obj, nil
+}
+
+func (c *TMClient) CRConfig() (tc.CRConfig, error) {
+ path := "/publish/CrConfig"
+ obj := tc.CRConfig{}
+ if err := c.GetJSON(path, &obj); err != nil {
+ return tc.CRConfig{}, err // GetJSON adds context
+ }
+ return obj, nil
+}
+
+// CRConfigBytes returns the raw bytes of the Monitor's CRConfig.
+//
+// If you need a deserialized object, use TMClient.CRConfig() instead.
+//
+// This function exists because the Monitor very intentionally serves the CRConfig bytes as
+// published by Traffic Ops, without deserializing or reserializing it.
+//
+// This can be useful to check for serialization or versioning issues, in case the Go object
+// is missing values sent by Traffic Ops, or has other serialization issues.
+//
+//
+func (c *TMClient) CRConfigBytes() ([]byte, error) { return c.getBytes("publish/CrConfig") }
+
+func (c *TMClient) PeerStates() (datareq.APIPeerStates, error) {
+ path := "/publish/PeerStates"
+ obj := datareq.APIPeerStates{}
+ if err := c.GetJSON(path, &obj); err != nil {
+ return datareq.APIPeerStates{}, err // GetJSON adds context
+ }
+ return obj, nil
+}
+
+func (c *TMClient) Stats() (datareq.Stats, error) {
+ path := "/publish/Stats"
+ obj := datareq.Stats{}
+ if err := c.GetJSON(path, &obj); err != nil {
+ return datareq.Stats{}, err // GetJSON adds context
+ }
+ return obj, nil
+}
+
+func (c *TMClient) StatSummary() (datareq.StatSummary, error) {
+ path := "/publish/StatSummary"
+ obj := datareq.StatSummary{}
+ if err := c.GetJSON(path, &obj); err != nil {
+ return datareq.StatSummary{}, err // GetJSON adds context
+ }
+ return obj, nil
+}
+
+func (c *TMClient) ConfigDoc() (handler.OpsConfig, error) {
+ path := "/publish/ConfigDoc"
+ obj := handler.OpsConfig{}
+ if err := c.GetJSON(path, &obj); err != nil {
+ return handler.OpsConfig{}, err // GetJSON adds context
+ }
+ return obj, nil
+}
+
+func (c *TMClient) getBytes(path string) ([]byte, error) {
+ url := c.url + path
+ httpClient := http.Client{Timeout: c.timeout}
+ resp, err := httpClient.Get(url)
+ if err != nil {
+ return nil, errors.New("getting from '" + url + "': " + err.Error())
+ }
+ defer resp.Body.Close()
Review comment:
Should defer `log.Close()` to handle the potential error.
##########
File path: traffic_monitor/tools/testcaches/fakesrvrdata/run.go
##########
@@ -72,44 +72,82 @@ func Run(s FakeServerData, remapIncrements map[string]BytesPerSec) (Ths, error)
}
ths := NewThs()
ths.Set(&s)
+
go run(ths, remapIncrements)
return ths, nil
}
+type IncrementChanT struct {
+ RemapName string
+ BytesPerSec BytesPerSec
+}
+
// run starts a goroutine incrementing the FakeServerData's values according to the remapIncrements. Never returns.
func run(srvrThs Ths, remapIncrements map[string]BytesPerSec) {
tickSecs := uint64(1) // adjustable for performance (i.e. a higher number is less CPU work)
+
+ ticker := time.NewTicker(time.Second * time.Duration(tickSecs))
+
for {
- time.Sleep(time.Second * time.Duration(tickSecs))
- srvr := srvrThs.Get()
- newRemaps := copyRemaps(srvr.ATS.Remaps)
- for remap, increments := range remapIncrements {
- srvrRemap := newRemaps[remap]
- if increments.Min.InBytes != increments.Min.InBytes {
- i := uint64(rand.Int63n(int64((increments.Max.InBytes-increments.Min.InBytes)*tickSecs))) + (increments.Min.InBytes * tickSecs)
- srvrRemap.InBytes += i
- srvr.System.ProcNetDev.RcvBytes += i
- }
- if increments.Min.OutBytes != increments.Max.OutBytes {
- i := uint64(rand.Int63n(int64((increments.Max.OutBytes-increments.Min.OutBytes)*tickSecs))) + (increments.Min.OutBytes * tickSecs)
- srvrRemap.OutBytes += i
- srvr.System.ProcNetDev.SndBytes += i
- }
- if increments.Min.Status2xx != increments.Max.Status2xx {
- srvrRemap.Status2xx += uint64(rand.Int63n(int64((increments.Max.Status2xx-increments.Min.Status2xx)*tickSecs))) + (increments.Min.Status2xx * tickSecs)
- }
- if increments.Min.Status3xx != increments.Max.Status3xx {
- srvrRemap.Status3xx += uint64(rand.Int63n(int64((increments.Max.Status3xx-increments.Min.Status3xx)*tickSecs))) + (increments.Min.Status3xx * tickSecs)
- }
- if increments.Min.Status4xx != increments.Max.Status4xx {
- srvrRemap.Status4xx += uint64(rand.Int63n(int64((increments.Max.Status4xx-increments.Min.Status4xx)*tickSecs))) + (increments.Min.Status4xx * tickSecs)
- }
- if increments.Min.Status5xx != increments.Max.Status5xx {
- srvrRemap.Status5xx += uint64(rand.Int63n(int64((increments.Max.Status5xx-increments.Min.Status5xx)*tickSecs))) + (increments.Min.Status5xx * tickSecs)
+ select {
+ case srvrThs.GetIncrementsChan <- remapIncrements:
+ case newIncrement := <-srvrThs.IncrementChan:
+ remapIncrements[newIncrement.RemapName] = newIncrement.BytesPerSec
+ case <-ticker.C:
+ srvr := srvrThs.Get()
+ newRemaps := copyRemaps(srvr.ATS.Remaps)
+ for remap, increments := range remapIncrements {
+ srvrRemap := newRemaps[remap]
+
+ addInBytes := increments.Min.InBytes * tickSecs
+ if increments.Min.InBytes != increments.Max.InBytes {
+ addInBytes += uint64(rand.Int63n(int64((increments.Max.InBytes - increments.Min.InBytes) * tickSecs)))
+ }
+ srvrRemap.InBytes += addInBytes
+ srvr.System.ProcNetDev.RcvBytes += addInBytes
+
+ addOutBytes := increments.Min.OutBytes * tickSecs
+ if increments.Min.OutBytes != increments.Max.OutBytes {
+ addOutBytes += uint64(rand.Int63n(int64((increments.Max.OutBytes - increments.Min.OutBytes) * tickSecs)))
+ }
+ srvrRemap.OutBytes += addOutBytes
+ srvr.System.ProcNetDev.SndBytes += addOutBytes
+
+ srvrRemap.Status2xx += increments.Min.Status2xx * tickSecs
+ if increments.Min.Status2xx != increments.Max.Status2xx {
+ srvrRemap.Status2xx += uint64(rand.Int63n(int64((increments.Max.Status2xx - increments.Min.Status2xx) * tickSecs)))
+ }
+
+ srvrRemap.Status3xx += increments.Min.Status3xx * tickSecs
+ if increments.Min.Status3xx != increments.Max.Status3xx {
+ srvrRemap.Status3xx += uint64(rand.Int63n(int64((increments.Max.Status3xx - increments.Min.Status3xx) * tickSecs)))
+ }
+
+ srvrRemap.Status4xx += increments.Min.Status4xx * tickSecs
+ if increments.Min.Status4xx != increments.Max.Status4xx {
+ srvrRemap.Status4xx += uint64(rand.Int63n(int64((increments.Max.Status4xx - increments.Min.Status4xx) * tickSecs)))
+ }
+
+ srvrRemap.Status5xx += increments.Min.Status5xx * tickSecs
+ if increments.Min.Status5xx != increments.Max.Status5xx {
+ srvrRemap.Status5xx += uint64(rand.Int63n(int64((increments.Max.Status5xx - increments.Min.Status5xx) * tickSecs)))
+ }
+
+ newRemaps[remap] = srvrRemap
}
- newRemaps[remap] = srvrRemap
+ srvr.ATS.Remaps = newRemaps
+ srvrThs.Set(srvr)
}
- srvr.ATS.Remaps = newRemaps
- srvrThs.Set(srvr)
+ }
+}
+
+// tryReceiveIncrement asynchronously tries to recieve from incrementChan if it has a value.
+// Returns the value if it was read, or returns false if no value was waiting.
+func tryReceiveIncrement(incrementChan <-chan IncrementChanT) (IncrementChanT, bool) {
+ select {
+ case inc := <-incrementChan:
+ return inc, true
+ default:
+ return IncrementChanT{}, false
}
}
Review comment:
`tryReceiveIncrement()` is unused, do we need it?
##########
File path: traffic_monitor/Dockerfile
##########
@@ -0,0 +1,37 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+# This is a very simple Dockerfile.
+# All it does is install and start the Traffic Monitor, given a Traffic Ops to point it to.
+# It doesn't do any of the complex things the Dockerfiles in infrastructure/docker or infrastructure/cdn-in-a-box do, like inserting itself into Traffic Ops.
+# It is designed for a very simple use case, where the complex orchestration of other Traffic Control components is done elsewhere (or manually).
+
+# Example Build and Run:
+# docker build --build-arg RPM=traffic_monitor.rpm --tag traffic_monitor:latest .
+#
+# docker run --detach --name tm --hostname tm --net=tmi --env TO_URI=http://to.invalid:3000 --env TO_USER=user --env TO_PASS=pass traffic_monitor:latest
Review comment:
These instructions should use `docker-compose` or be removed, too
##########
File path: traffic_monitor/tools/testto/Dockerfile_run.sh
##########
@@ -0,0 +1,40 @@
+#!/usr/bin/env bash
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+# envvars=( PORT )
+# for v in $envvars
+# do
+# if [[ -z $$v ]]; then echo "$v is unset"; exit 1; fi
+# done
Review comment:
Can these commented lines be removed?
##########
File path: traffic_monitor/health/event.go
##########
@@ -33,6 +35,15 @@ func (t Time) MarshalJSON() ([]byte, error) {
return []byte(fmt.Sprintf("%d", time.Time(t).Unix())), nil
}
+func (t *Time) UnmarshalJSON(data []byte) error {
+ i, err := strconv.ParseInt(string(data), 10, 64)
+ if err != nil {
+ return errors.New("health.Time (" + string(data) + ") must be a unix epoch integer: " + err.Error())
+ }
+ *t = Time(time.Unix(i, 0))
Review comment:
Can this be a more descriptive variable name than `i`? If someone puts 2+2 together and sees that `i` stands for `int`, its purpose is still unclear.
##########
File path: traffic_monitor/tests/_integration/Dockerfile
##########
@@ -0,0 +1,38 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+# This is a very simple Dockerfile.
+# All it does is install and start the Traffic Monitor, given a Traffic Ops to point it to.
+# It doesn't do any of the complex things the Dockerfiles in infrastructure/docker or infrastructure/cdn-in-a-box do, like inserting itself into Traffic Ops.
+# It is designed for a very simple use case, where the complex orchestration of other Traffic Control components is done elsewhere (or manually).
+
+# Example Build and Run:
+# docker build --build-arg RPM=traffic_monitor.rpm --tag traffic_monitor:latest .
+#
+# docker run --detach --name tm --hostname tm --net=tmi --env TO_URI=http://to.invalid:3000 --env TO_USER=user --env TO_PASS=pass traffic_monitor:latest
Review comment:
Instructions should use `docker-compose` or be removed
##########
File path: traffic_monitor/tests/_integration/Dockerfile_run.sh
##########
@@ -0,0 +1,166 @@
+#!/usr/bin/env bash
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+# The following environment variables must be set (ordinarily by `docker run -e` arguments):
+# TO_URI
+# TO_USER
+# TO_PASS
+# CDN
+# Check that env vars are set
+envvars=(TESTTO_URI TESTTO_PORT TESTCACHES_URI TESTCACHES_PORT_START TM_URI)
+for v in $envvars; do
+ if [[ -z ${!v} ]]; then
+ echo "$v is unset"
+ exit 1
+ fi
+done
+
+TO_API_VERSION=4.0
+CFG_FILE=/traffic-monitor-integration-test.cfg
+
+start() {
+ printf "DEBUG traffic_monitor_integration starting\n"
+
+ exec /traffic_monitor_integration_test -test.v -cfg $CFG_FILE
+}
+
+init() {
+ wait_for_endpoint "${TESTTO_URI}/api/${TO_API_VERSION}/servers"
+ wait_for_endpoint "${TESTCACHES_URI}:${TESTCACHES_PORT_START}/_astats"
+ wait_for_endpoint "${TM_URI}"
+ TESTCACHES_ADDRESS=$(ping testcaches -4 -c 1 | head -n 1 | grep -Eo '[0-9]+.[0-9]+.[0-9]+.[0-9]+')
+ TESTCACHES_GATEWAY=$(echo $TESTCACHES_ADDRESS | sed "s/\([0-9]\+.[0-9]\+.[0-9]\+.\)[0-9]/\11/")
+
+ jq "(.. | .address?) |= \"$TESTCACHES_ADDRESS\" | (.. | .gateway?) |= \"$TESTCACHES_GATEWAY\"" \
+ /monitoring.json >/monitoring.json.tmp && mv /monitoring.json.tmp /monitoring.json
+
+ curl -Lvsk ${TESTTO_URI}/api/${TO_API_VERSION}/cdns/fake/snapshot -X POST -d "@/snapshot.json"
+
+ curl -Lvsk ${TESTTO_URI}/api/${TO_API_VERSION}/cdns/fake/configs/monitoring -X POST -d '@/monitoring.json'
+
+ curl -Lvsk ${TESTTO_URI}/api/${TO_API_VERSION}/servers -X POST -d '
+[
+ {
+ "cachegroup": "foo",
+ "cachegroupId": 0,
+ "cdnId": 1,
+ "cdnName": "fake",
+ "deliveryServices": null,
+ "fqdn": "trafficmonitor.traffic-monitor-integration.test",
+ "guid": "foo",
+ "hostName": "trafficmonitor",
+ "httpsPort": null,
+ "id": 1,
+ "iloIpAddress": null,
+ "iloIpGateway": null,
+ "iloIpNetmask": null,
+ "iloPassword": null,
+ "iloUsername": null,
+ "interfaceMtu": null,
+ "interfaceName": "bond0",
+ "ip6Address": null,
+ "ip6Gateway": null,
+ "interfaces": [
+ {
+ "ipAddresses": [
+ {
+ "address": "4.0.16.239.6",
+ "gateway": "4.0.16.239.1",
+ "serviceAddress": true
+ },
+ {
+ "address": "fc01:9400:1000:8::6",
+ "gateway": "fc01:9400:1000:8::1",
+ "serviceAddress": true
+ }
+ ],
+ "maxBandwidth": null,
+ "monitor": true,
+ "mtu": 1500,
+ "name": "eth0"
+ }
+ ],
+ "ipGateway": "4.0.0.0.1",
+ "ipNetmask": "255.255.255.0",
+ "lastUpdated": "2019",
+ "mgmtIpAddress": null,
+ "mgmtIpGateway": null,
+ "mgmtIpNetmask": null,
+ "offlineReason": "none",
+ "physLocation": "",
+ "physLocationId": 0,
+ "profile": "Monitor0",
+ "profileDesc": "nodesc",
+ "profileId": 0,
+ "rack": "",
+ "revalPending": false,
+ "routerHostName": "",
+ "routerPortName": "",
+ "status": "REPORTED",
+ "statusId": 0,
+ "tcpPort": 80,
+ "type": "RASCAL",
+ "typeId": 0,
+ "updPending": false,
+ "xmppId": "",
+ "xmppPasswd": ""
+ }
+]
+'
+
+ cat >$CFG_FILE <<-EOF
+{
+ "trafficMonitor": {
+ "url": "$TM_URI"
+ },
+ "default": {
+ "session": {
+ "timeoutInSecs": 30
+ },
+ "log": {
+ "debug": "stdout",
+ "event": "stdout",
+ "info": "stdout",
+ "error": "stdout",
+ "warning": "stdout"
+ }
+ }
+}
+EOF
+
+ echo "INITIALIZED=1" >>/etc/environment
+}
+
+function wait_for_endpoint() {
+ try=0
+ while curl -Lvsk "$1" 2>&1 1> /dev/null | grep -E '< HTTP/[0-9]\.?[0-9]* 200' >/dev/null && [[ $try -lt 5 ]]; do
Review comment:
Can do `grep -q` instead of `grep >/dev/null`
##########
File path: traffic_monitor/tests/_integration/kbps_test.go
##########
@@ -0,0 +1,74 @@
+package _integration
+
+/*
+ Licensed under the Apache License, Version 2.0 (the "License");
+ you may not use this file except in compliance with the License.
+ You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+ Unless required by applicable law or agreed to in writing, software
+ distributed under the License is distributed on an "AS IS" BASIS,
+ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ See the License for the specific language governing permissions and
+ limitations under the License.
+*/
+
+import (
+ "fmt"
+ "net/http"
+ "testing"
+ "time"
+
+ "github.com/apache/trafficcontrol/lib/go-tc"
+)
+
+func TestKBPS(t *testing.T) {
+ crc, err := TMClient.CRConfig()
+ if err != nil {
+ t.Fatalf("client CRConfig error expected nil, actual %v\n", err)
+ }
+
+ if len(crc.ContentServers) == 0 {
+ t.Fatalf("Monitor CRConfig has no servers, cannot test KBPS")
+ }
+
+ serverName := ""
+ server := tc.CRConfigTrafficOpsServer{}
+ for crcServerName, crcServer := range crc.ContentServers {
+ server = crcServer
+ serverName = crcServerName
+ break
+ }
+ if server.Ip == nil {
+ t.Fatalf("Monitor CRConfig server '" + serverName + "' has no Ip, cannot test KBPS")
+ }
+ if server.Port == nil {
+ t.Fatalf("Monitor CRConfig server '" + serverName + "' has no Port, cannot test KBPS")
+ }
+
+ const bytesPerKilobit = 125
+
+ expectedKbps := 10000
+
+ httpClient := http.Client{Timeout: time.Duration(Config.Default.Session.TimeoutInSecs) * time.Second}
+
+ kbps10 := bytesPerKilobit * expectedKbps
+ uri := fmt.Sprintf(`http://%v:%v/cmd/setstat?remap=num1.example.net&stat=out_bytes&min=%v&max=%v`, *server.Ip, *server.Port, kbps10, kbps10)
+ resp, err := httpClient.Get(uri)
+ if err != nil {
+ t.Fatalf("Error posting fake cache command '" + uri + "': " + err.Error())
+ }
+ resp.Body.Close()
Review comment:
Unhandled error. For easy error handling, this can use `log.Close()`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [trafficcontrol] shamrickus commented on a change in pull request #5817: Traffic Monitor Integration Test Framework
Posted by GitBox <gi...@apache.org>.
shamrickus commented on a change in pull request #5817:
URL: https://github.com/apache/trafficcontrol/pull/5817#discussion_r636918534
##########
File path: traffic_monitor/Dockerfile
##########
@@ -0,0 +1,32 @@
+# Licensed to the Apache Software Foundation (ASF) under one
Review comment:
Can't reference parent dirs in docker (tests/_integration -> ../../tools/test*). Or at least in so far as I'm aware.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [trafficcontrol] zrhoffman commented on a change in pull request #5817: Traffic Monitor Integration Test Framework
Posted by GitBox <gi...@apache.org>.
zrhoffman commented on a change in pull request #5817:
URL: https://github.com/apache/trafficcontrol/pull/5817#discussion_r633646195
##########
File path: traffic_monitor/tools/testcaches/README.md
##########
@@ -30,3 +30,68 @@ Each port is a unique fake server, with distinct incrementing stats.
When run with no parameters, it defaults to ports 40000-40999 and 1000 remaps.
Stats are served at the regular ATS `stats_over_http` endpoint, `_astats`. For example, if it's serving on port 40000, it can be reached via `curl http://localhost:40000/_astats`. It also respects the `?application=system` query parameter, and will serve only system stats (the Monitor "health check" [as opposed to the "stat check"]). For example, `curl http://localhost:40000/_astats?application=system`.
+
+## Commands
+
+The `testcaches` app accepts a number of commands, which manipulate the data it serves. These command are all available via HTTP requests.
+
+Each HTTP request is made to a fake cache at a specific port. Thus, you can modify data served by each fake cache independently.
+
+The commands are:
+
+### `/cmd/setstat`
+
+Sets how much a stat increments by every interval (currently, an interval is hard-coded to 1 second). Accepts a min and max, and will increment by a random number between them. The min may equal the max, if a constant increment is desired.
+
+Query Parameters:
+`remap` - the remap rule to set
+`stat` - the stat to set
+`min` - the minimum number to increment by
+`max` - the minimum to increment by
+
+Example:
+`curl -Lvsk http://localhost:4242/cmd/setstat?remap=num1.example.net&stat=out_bytes&min=10&max=25`
+
+### `setsystem`
+
+Sets system stats to constant values. Multiple stats may be set with a single request.
+
+Query Parameters:
+`loadavg1m` - the 1m loadavg in the `system` object.
+`loadavg5m` - the 5m loadavg in the `system` object.
+`loadavg10m` - the 10m loadavg in the `system` object.
+`speed` - the network interface speed in the `system` object. This number is in kilobits. I.e. 20000 means 20Gbps.
+
+Example:
+`curl -sk 'http://localhost:4242/cmd/setsystem?loadavg1m=10.1&loadavg5m=27.92&loadavg10m=3.4&speed=20000' `
+
+### `setdelay`
+
+Sets the delay for serving all _astats requests to this fake cache. Accepts a minimum and maximum, which may be qual, and delays the request by a random interval between them. When a delay is set, the server immediately accepts client requests, reads headers and sets up the connection, and then delays writing out the body.
+
+Query Parameters:
+`min` - the minimum delay time, in milliseconds
+`max` - the maximum delay time, in milliseconds
+
+Example:
+`curl -Lvsk 'http://localhost:4242/cmd/setdelay?min=200&max=600'`
+
+## Docker
+
+Build environment variables: none
+
+Run environment variables:
+- `NUM_PORTS` - app `numPorts` argument
+- `NUM_REMAPS` - app `numRemaps` argument
+- `PORT_START` - app `portStart` argument
+
+### Building
+
+`docker build . --tag testcaches`
+
+### Running
+
+```
+docker network create tmi
+docker run --detach --network tmi --name testcaches --hostname testcaches --publish 30000-30099:30000-30099 --env NUM_PORTS=100 --env NUM_REMAPS=100 --env PORT_START=30000 testcaches
Review comment:
IMO having a non-docker-compose command when a `docker-compose.yml` exists anyway is redundant and confusing. Better to just remove it if using it is covered elsewhere.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [trafficcontrol] shamrickus commented on a change in pull request #5817: Traffic Monitor Integration Test Framework
Posted by GitBox <gi...@apache.org>.
shamrickus commented on a change in pull request #5817:
URL: https://github.com/apache/trafficcontrol/pull/5817#discussion_r631203839
##########
File path: traffic_monitor/tools/testcaches/README.md
##########
@@ -30,3 +30,68 @@ Each port is a unique fake server, with distinct incrementing stats.
When run with no parameters, it defaults to ports 40000-40999 and 1000 remaps.
Stats are served at the regular ATS `stats_over_http` endpoint, `_astats`. For example, if it's serving on port 40000, it can be reached via `curl http://localhost:40000/_astats`. It also respects the `?application=system` query parameter, and will serve only system stats (the Monitor "health check" [as opposed to the "stat check"]). For example, `curl http://localhost:40000/_astats?application=system`.
+
+## Commands
+
+The `testcaches` app accepts a number of commands, which manipulate the data it serves. These command are all available via HTTP requests.
+
+Each HTTP request is made to a fake cache at a specific port. Thus, you can modify data served by each fake cache independently.
+
+The commands are:
+
+### `/cmd/setstat`
+
+Sets how much a stat increments by every interval (currently, an interval is hard-coded to 1 second). Accepts a min and max, and will increment by a random number between them. The min may equal the max, if a constant increment is desired.
+
+Query Parameters:
+`remap` - the remap rule to set
+`stat` - the stat to set
+`min` - the minimum number to increment by
+`max` - the minimum to increment by
+
+Example:
+`curl -Lvsk http://localhost:4242/cmd/setstat?remap=num1.example.net&stat=out_bytes&min=10&max=25`
+
+### `setsystem`
+
+Sets system stats to constant values. Multiple stats may be set with a single request.
+
+Query Parameters:
+`loadavg1m` - the 1m loadavg in the `system` object.
+`loadavg5m` - the 5m loadavg in the `system` object.
+`loadavg10m` - the 10m loadavg in the `system` object.
+`speed` - the network interface speed in the `system` object. This number is in kilobits. I.e. 20000 means 20Gbps.
+
+Example:
+`curl -sk 'http://localhost:4242/cmd/setsystem?loadavg1m=10.1&loadavg5m=27.92&loadavg10m=3.4&speed=20000' `
+
+### `setdelay`
+
+Sets the delay for serving all _astats requests to this fake cache. Accepts a minimum and maximum, which may be qual, and delays the request by a random interval between them. When a delay is set, the server immediately accepts client requests, reads headers and sets up the connection, and then delays writing out the body.
+
+Query Parameters:
+`min` - the minimum delay time, in milliseconds
+`max` - the maximum delay time, in milliseconds
+
+Example:
+`curl -Lvsk 'http://localhost:4242/cmd/setdelay?min=200&max=600'`
+
+## Docker
+
+Build environment variables: none
+
+Run environment variables:
+- `NUM_PORTS` - app `numPorts` argument
+- `NUM_REMAPS` - app `numRemaps` argument
+- `PORT_START` - app `portStart` argument
+
+### Building
+
+`docker build . --tag testcaches`
+
+### Running
+
+```
+docker network create tmi
+docker run --detach --network tmi --name testcaches --hostname testcaches --publish 30000-30099:30000-30099 --env NUM_PORTS=100 --env NUM_REMAPS=100 --env PORT_START=30000 testcaches
Review comment:
For the purposes of running the integration tests, the `tests/integration/docker-compose.yml` orchestrates this. This command is just to run testcaches stand alone.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [trafficcontrol] shamrickus commented on pull request #5817: Traffic Monitor Integration Test Framework
Posted by GitBox <gi...@apache.org>.
shamrickus commented on pull request #5817:
URL: https://github.com/apache/trafficcontrol/pull/5817#issuecomment-839973182
> A lot of the TM integration test Docker environment seems to be copied from an older version of CDN in a Box.
> I can understand only wanting to build and use the pieces that are relevant for the TM integration tests, but using CDN in a Box where it sits instead of copying it does not require starting everything. For example, neither the TO integration tests nor the TP integration tests require starting all of the CDN in a Box services:
>
> https://github.com/apache/trafficcontrol/blob/55dbb7eacd399d3ccef9d4eb34bfaeb0c5e2e7ad/infrastructure/cdn-in-a-box/docker-compose.traffic-ops-test.yml#L23-L24
>
> https://github.com/apache/trafficcontrol/blob/55dbb7eacd399d3ccef9d4eb34bfaeb0c5e2e7ad/infrastructure/cdn-in-a-box/docker-compose.traffic-portal-test.yml#L23-L24
I agree it'd be better for that, but I was going off of what Rob already did. I figured it would be more valuable to get this merged and working, and then later down the road it can be moved to CiaB (testcaches would also need to be ported). That also has the added benefit of building and running this currently only takes about as long as it does to build just TM.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [trafficcontrol] zrhoffman commented on a change in pull request #5817: Traffic Monitor Integration Test Framework
Posted by GitBox <gi...@apache.org>.
zrhoffman commented on a change in pull request #5817:
URL: https://github.com/apache/trafficcontrol/pull/5817#discussion_r633646195
##########
File path: traffic_monitor/tools/testcaches/README.md
##########
@@ -30,3 +30,68 @@ Each port is a unique fake server, with distinct incrementing stats.
When run with no parameters, it defaults to ports 40000-40999 and 1000 remaps.
Stats are served at the regular ATS `stats_over_http` endpoint, `_astats`. For example, if it's serving on port 40000, it can be reached via `curl http://localhost:40000/_astats`. It also respects the `?application=system` query parameter, and will serve only system stats (the Monitor "health check" [as opposed to the "stat check"]). For example, `curl http://localhost:40000/_astats?application=system`.
+
+## Commands
+
+The `testcaches` app accepts a number of commands, which manipulate the data it serves. These command are all available via HTTP requests.
+
+Each HTTP request is made to a fake cache at a specific port. Thus, you can modify data served by each fake cache independently.
+
+The commands are:
+
+### `/cmd/setstat`
+
+Sets how much a stat increments by every interval (currently, an interval is hard-coded to 1 second). Accepts a min and max, and will increment by a random number between them. The min may equal the max, if a constant increment is desired.
+
+Query Parameters:
+`remap` - the remap rule to set
+`stat` - the stat to set
+`min` - the minimum number to increment by
+`max` - the minimum to increment by
+
+Example:
+`curl -Lvsk http://localhost:4242/cmd/setstat?remap=num1.example.net&stat=out_bytes&min=10&max=25`
+
+### `setsystem`
+
+Sets system stats to constant values. Multiple stats may be set with a single request.
+
+Query Parameters:
+`loadavg1m` - the 1m loadavg in the `system` object.
+`loadavg5m` - the 5m loadavg in the `system` object.
+`loadavg10m` - the 10m loadavg in the `system` object.
+`speed` - the network interface speed in the `system` object. This number is in kilobits. I.e. 20000 means 20Gbps.
+
+Example:
+`curl -sk 'http://localhost:4242/cmd/setsystem?loadavg1m=10.1&loadavg5m=27.92&loadavg10m=3.4&speed=20000' `
+
+### `setdelay`
+
+Sets the delay for serving all _astats requests to this fake cache. Accepts a minimum and maximum, which may be qual, and delays the request by a random interval between them. When a delay is set, the server immediately accepts client requests, reads headers and sets up the connection, and then delays writing out the body.
+
+Query Parameters:
+`min` - the minimum delay time, in milliseconds
+`max` - the maximum delay time, in milliseconds
+
+Example:
+`curl -Lvsk 'http://localhost:4242/cmd/setdelay?min=200&max=600'`
+
+## Docker
+
+Build environment variables: none
+
+Run environment variables:
+- `NUM_PORTS` - app `numPorts` argument
+- `NUM_REMAPS` - app `numRemaps` argument
+- `PORT_START` - app `portStart` argument
+
+### Building
+
+`docker build . --tag testcaches`
+
+### Running
+
+```
+docker network create tmi
+docker run --detach --network tmi --name testcaches --hostname testcaches --publish 30000-30099:30000-30099 --env NUM_PORTS=100 --env NUM_REMAPS=100 --env PORT_START=30000 testcaches
Review comment:
IMO having a non-docker-compose command when a `docker-compose.yml` exists anyway is redundant and confusing. Better to just remove it if using it is covered elsewhere.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [trafficcontrol] zrhoffman commented on pull request #5817: Traffic Monitor Integration Test Framework
Posted by GitBox <gi...@apache.org>.
zrhoffman commented on pull request #5817:
URL: https://github.com/apache/trafficcontrol/pull/5817#issuecomment-842645859
> > A lot of the TM integration test Docker environment seems to be copied from an older version of CDN in a Box.
>
> For the record, it is not. It's based on #3707 whose compose was written from scratch. I believe the Dockerfiles may have been based on the independent dockerfiles in infrastructure/docker (which I also originally wrote), but neither they nor the docker-compose.yml are derivative works of the CDN-in-a-Box.
Fair enough, we don't really group things based on `infrastructure/docker` like we do CDN in a Box, and it's not a ton of code duplication.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [trafficcontrol] shamrickus commented on a change in pull request #5817: Traffic Monitor Integration Test Framework
Posted by GitBox <gi...@apache.org>.
shamrickus commented on a change in pull request #5817:
URL: https://github.com/apache/trafficcontrol/pull/5817#discussion_r638194149
##########
File path: traffic_monitor/tests/_integration/traffic_monitor_test.go
##########
@@ -0,0 +1,96 @@
+package _integration
+
+/*
+
+ Licensed under the Apache License, Version 2.0 (the "License");
+ you may not use this file except in compliance with the License.
+ You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+ Unless required by applicable law or agreed to in writing, software
+ distributed under the License is distributed on an "AS IS" BASIS,
+ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ See the License for the specific language governing permissions and
+ limitations under the License.
+*/
+
+import (
+ "flag"
+ "fmt"
+ "net/http"
+ "os"
+ "strings"
+ "testing"
+ "time"
+
+ "github.com/apache/trafficcontrol/lib/go-log"
+ "github.com/apache/trafficcontrol/traffic_monitor/tests/_integration/config"
+ "github.com/apache/trafficcontrol/traffic_monitor/tmclient"
+)
+
+var Config config.Config
+var TMClient *tmclient.TMClient
+
+func TestMain(m *testing.M) {
+ var err error
+ configFileName := flag.String("cfg", "traffic-monitor-test.conf", "The config file path")
+ flag.Parse()
+
+ if Config, err = config.LoadConfig(*configFileName); err != nil {
+ fmt.Printf("Error Loading Config %v %v\n", Config, err)
+ os.Exit(1)
+ }
+
+ if err = log.InitCfg(Config); err != nil {
+ fmt.Printf("Error initializing loggers: %v\n", err)
+ os.Exit(1)
+ }
+
+ log.Infof(`Using Config values:
+ TM Config File: %s
+ TM URL: %s
+ TM Session Timeout: %d\n`,
+ *configFileName, Config.TrafficMonitor.URL, Config.Default.Session.TimeoutInSecs)
+
+ tmReqTimeout := time.Second * time.Duration(Config.Default.Session.TimeoutInSecs)
+
+ monitorWaitSpan := 30 * time.Second // TODO make configurable?
+
+ if !WaitForMonitor(Config.TrafficMonitor.URL, monitorWaitSpan) {
+ fmt.Printf("\nError communicating with Monitor '%v' - didn't return a 200 OK in %v\n",
+ Config.TrafficMonitor.URL, monitorWaitSpan)
+ os.Exit(1)
+ }
Review comment:
Should be fixed (wasn't defering closes on the instances change to `log.Close`)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [trafficcontrol] zrhoffman commented on a change in pull request #5817: Traffic Monitor Integration Test Framework
Posted by GitBox <gi...@apache.org>.
zrhoffman commented on a change in pull request #5817:
URL: https://github.com/apache/trafficcontrol/pull/5817#discussion_r627815017
##########
File path: traffic_monitor/tests/integration/docker-compose.yml
##########
@@ -0,0 +1,102 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+#
+# To run the integration test:
+# 1. build Traffic Monitor and then copy the RPM to this directory as traffic_monitor.rpm.
+# 2. build traffic_monitor/tools/testcaches and place the binary in this directory.
+# 3. build traffic_monitor/tools/testto and place the binary in this directory.
+# 4. Run this compose
+#
+# Commands are, from the root trafficcontrol directory:
+# ./pkg -v traffic_monitor_build
+# cp ./dist/traffic_monitor*.rpm ./traffic_monitor/
+# (cd ./traffic_monitor/tools/testto && go build)
+# (cd ./traffic_monitor/tools/testcaches && go build)
+# cd ./traffic_monitor
+# docker-compose up
+#
+# To run the integration tests, your current working directory must be trafficcontrol/traffic_monitor.
+# This is because Docker doesn't allow accessing files outside the current "context," and the integration tests need
Review comment:
> and the integration tests need
Rephrase?
##########
File path: traffic_monitor/tests/integration/traffic_monitor_test.go
##########
@@ -0,0 +1,125 @@
+package integration
+
+/*
+
+ Licensed under the Apache License, Version 2.0 (the "License");
+ you may not use this file except in compliance with the License.
+ You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+ Unless required by applicable law or agreed to in writing, software
+ distributed under the License is distributed on an "AS IS" BASIS,
+ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ See the License for the specific language governing permissions and
+ limitations under the License.
+*/
+
+import (
+ // "database/sql"
+ "flag"
+ "fmt"
+ "net/http"
+ "os"
+ "strings"
+ "testing"
+ "time"
+
+ "github.com/apache/trafficcontrol/lib/go-log"
+ "github.com/apache/trafficcontrol/traffic_monitor/tests/integration/config"
+ "github.com/apache/trafficcontrol/traffic_monitor/tmclient"
+)
+
+var Config config.Config
+var TMClient *tmclient.TMClient
+
+func TestMain(m *testing.M) {
+ var err error
+ configFileName := flag.String("cfg", "traffic-monitor-test.conf", "The config file path")
+ flag.Parse()
+
+ if Config, err = config.LoadConfig(*configFileName); err != nil {
+ fmt.Printf("Error Loading Config %v %v\n", Config, err)
+ os.Exit(1)
+ }
+
+ if err = log.InitCfg(Config); err != nil {
+ fmt.Printf("Error initializing loggers: %v\n", err)
+ os.Exit(1)
+ }
+
+ log.Infof(`Using Config values:
+ TM Config File: %s
+ TM URL: %s
+ TM Session Timeout: %ds
+`, *configFileName, Config.TrafficMonitor.URL, Config.Default.Session.TimeoutInSecs)
+
+ // //Load the test data
+ // LoadFixtures(*tcFixturesFileName)
+
+ // var db *sql.DB
+ // db, err = OpenConnection()
+ // if err != nil {
+ // fmt.Printf("\nError opening connection to %s - %s, %v\n", Config.TrafficOps.URL, Config.TrafficOpsDB.User, err)
+ // os.Exit(1)
+ // }
+ // defer db.Close()
+
+ // err = Teardown(db)
+ // if err != nil {
+ // fmt.Printf("\nError tearingdown data %s - %s, %v\n", Config.TrafficOps.URL, Config.TrafficOpsDB.User, err)
+ // os.Exit(1)
+ // }
+
+ // err = SetupTestData(db)
+ // if err != nil {
+ // fmt.Printf("\nError setting up data %s - %s, %v\n", Config.TrafficOps.URL, Config.TrafficOpsDB.User, err)
+ // os.Exit(1)
+ // }
+
+ tmReqTimeout := time.Second * time.Duration(Config.Default.Session.TimeoutInSecs)
+
+ // err = SetupSession(toReqTimeout, Config.TrafficOps.URL, Config.TrafficOps.Users.Admin, Config.TrafficOps.UserPassword)
+ // if err != nil {
+ // fmt.Printf("\nError creating session to %s - %s, %v\n", Config.TrafficOps.URL, Config.TrafficOpsDB.User, err)
+ // os.Exit(1)
+ // }
Review comment:
Can the commented code be removed?
##########
File path: traffic_monitor/tests/integration/traffic_monitor_test.go
##########
@@ -0,0 +1,125 @@
+package integration
+
+/*
+
+ Licensed under the Apache License, Version 2.0 (the "License");
+ you may not use this file except in compliance with the License.
+ You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+ Unless required by applicable law or agreed to in writing, software
+ distributed under the License is distributed on an "AS IS" BASIS,
+ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ See the License for the specific language governing permissions and
+ limitations under the License.
+*/
+
+import (
+ // "database/sql"
Review comment:
Unused import is commented out
##########
File path: traffic_monitor/tools/testcaches/Dockerfile_run.sh
##########
@@ -0,0 +1,47 @@
+#!/usr/bin/env bash
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+
+# envvars=( PORT )
+# for v in $envvars
+# do
+# if [[ -z $$v ]]; then echo "$v is unset"; exit 1; fi
+# done
Review comment:
Can this commented code be removed?
##########
File path: traffic_monitor/tests/integration/Dockerfile
##########
@@ -0,0 +1,48 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+# This is a very simple Dockerfile.
+# All it does is install and start the Traffic Monitor, given a Traffic Ops to point it to.
+# It doesn't do any of the complex things the Dockerfiles in infrastructure/docker or infrastructure/cdn-in-a-box do, like inserting itself into Traffic Ops.
+# It is designed for a very simple use case, where the complex orchestration of other Traffic Control components is done elsewhere (or manually).
+
+# Example Build and Run:
+# docker build --build-arg RPM=traffic_monitor.rpm --tag traffic_monitor:latest .
+#
+# docker run --detach --name tm --hostname tm --net=tmi --env TO_URI=http://to.invalid:3000 --env TO_USER=user --env TO_PASS=pass traffic_monitor:latest
+
+FROM centos/systemd
Review comment:
Since the [`centos/systemd`](https://hub.docker.com/r/centos/systemd/tags) image was last updated in December 2018, this should use `centos:8` (preferred, since RPMs build for CentOS 8 by default) or `centos:7`.
##########
File path: traffic_monitor/tests/integration/Dockerfile_run.sh
##########
@@ -0,0 +1,394 @@
+#!/usr/bin/env bash
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+# The following environment variables must be set (ordinarily by `docker run -e` arguments):
+# TO_URI
+# TO_USER
+# TO_PASS
+# CDN
+
+# Check that env vars are set
+envvars=( TESTTO_URI TESTTO_PORT TESTCACHES_URI TESTCACHES_PORT_START TM_URI )
+for v in $envvars
+do
+ if [[ -z $$v ]]; then echo "$v is unset"; exit 1; fi
+done
+
+
+CFG_FILE=/traffic-monitor-integration-test.cfg
+
+start() {
+ printf "DEBUG traffic_monitor_integration starting\n"
+
+ exec /traffic_monitor_integration_test -test.v -cfg $CFG_FILE
+}
+
+init() {
+ wait_for_to
+
+ curl -Lvsk ${TESTTO_URI}/api/1.2/cdns/fake/snapshot -X POST -d '
+{
+ "config": {
+ "api.cache-control.max_age": "30",
+ "consistent.dns.routing": "true",
+ "coveragezone.polling.interval": "30",
+ "coveragezone.polling.url": "30",
+ "dnssec.dynamic.response.expiration": "60",
+ "dnssec.enabled": "false",
+ "domain_name": "monitor-integration.test",
+ "federationmapping.polling.interval": "60",
+ "federationmapping.polling.url": "foo",
+ "geolocation.polling.interval": "30",
+ "geolocation.polling.url": "foo",
+ "keystore.maintenance.interval": "30",
+ "neustar.polling.interval": "30",
+ "neustar.polling.url": "foo",
+ "soa": {
+
+ },
+ "dnssec.inception": "0",
+ "ttls": {
+ "admin": "30",
+ "expire": "30",
+ "minimum": "30",
+ "refresh": "30",
+ "retry": "30"
+ },
+ "weight": "1",
+ "zonemanager.cache.maintenance.interval": "30",
+ "zonemanager.threadpool.scale": "1"
+ },
+ "contentServers": {
+ "server0": {
+ "cacheGroup": "cg0",
+ "profile": "Edge0",
+ "fqdn": "server0.monitor-integration.test",
+ "hashCount": 1,
+ "hashId": "server0",
+ "httpsPort" : null,
+ "ip": "testcaches",
+ "ip6": null,
+ "locationId": "",
+ "port" : 30000,
+ "status": "REPORTED",
+ "type": "EDGE",
+ "deliveryServices": {"ds0":["ds0.monitor-integration.test"]},
+ "routingDisabled": 0
+ },
+ "server1": {
+ "cacheGroup": "cg0",
+ "profile": "Edge0",
+ "fqdn": "server1.monitor-integration.test",
+ "hashCount": 1,
+ "hashId": "server1",
+ "httpsPort" : null,
+ "ip": "testcaches",
+ "ip6": null,
+ "locationId": "",
+ "port" : 30001,
+ "status": "REPORTED",
+ "type": "EDGE",
+ "deliveryServices": {"ds0":["ds0.monitor-integration.test"]},
+ "routingDisabled": 0
+ }
+ },
+ "deliveryServices": {
+ "ds0": {
+ "anonymousBlockingEnabled": false,
+ "consistentHashQueryParams": [],
+ "consistentHashRegex": "",
+ "coverageZoneOnly": false,
+ "dispersion": {
+ "limit": 1,
+ "shuffled": false
+ },
+ "domains": ["ds0.monitor-integration.test"],
+ "geolocationProvider": null,
+ "matchsets": [
+ {
+ "protocol": "HTTP",
+ "matchlist": [
+ {
+ "regex": "\\.*ds0\\.*",
+ "match-type": "regex"
+ }
+ ]
+ }
+ ],
+ "missLocation": {"lat": 0, "lon": 0},
+ "protocol": {
+ "acceptHttp": true,
+ "acceptHttps": false,
+ "redirectToHttps": false
+ },
+ "regionalGeoBlocking": "false",
+ "responseHeaders": {},
+ "requestHeaders": [],
+ "soa": {
+ "admin": "60",
+ "expire": "60",
+ "minimum": "60",
+ "refresh": "60",
+ "retry": "60"
+ },
+ "sslEnabled": false,
+ "ttl": 60,
+ "ttls": {
+ "A": "60",
+ "AAAA": "60",
+ "DNSKEY": "60",
+ "DS": "60",
+ "NS": "60",
+ "SOA": "60"
+ },
+ "maxDnsIpsForLocation": 3,
+ "ip6RoutingEnabled": false,
+ "routingName": "ccr",
+ "bypassDestination": null,
+ "deepCachingType": null,
+ "geoEnabled": false,
+ "geoLimitRedirectURL": null,
+ "staticDnsEntries": []
+ }
+ },
+ "edgeLocations": {
+ "cg0": {"latitude":0, "longitude":0}
+ },
+ "trafficRouterLocations": {
+ "tr0": {"latitude":0, "longitude":0}
+ },
+ "monitors": {
+ "trafficmonitor": {
+ "fqdn": "trafficmonitor.monitor-integration.test",
+ "httpsPort": null,
+ "ip": "trafficmonitor",
+ "ip6": null,
+ "location": "cg0",
+ "port": 80,
+ "profile": "Monitor0",
+ "status": "REPORTED"
+ }
+ },
+ "stats": {
+ "CDN_name": "fake",
+ "date": 1561000000,
+ "tm_host": "testto",
+ "tm_path": "/fake",
+ "tm_user": "fake",
+ "tm_version": "integrationtest/0.fake"
+ }
+}
+'
+
+ curl -Lvsk ${TESTTO_URI}/api/1.2/cdns/fake/configs/monitoring.json -X POST -d '
+{
+ "trafficServers": [
+ {
+ "profile": "Edge0",
+ "ip": "testcaches",
+ "status": "REPORTED",
+ "cacheGroup": "cg0",
+ "ip6": null,
+ "port": 30000,
+ "httpsPort": null,
+ "hostName": "server0",
+ "fqdn": "server0.monitor-integration.test",
+ "interfaceName": "bond0",
+ "type": "EDGE",
+ "hashId": "server0",
+ "deliveryServices": {"ds0":["ds0.monitor-integration.test"]}
+ },
+ {
+ "profile": "Edge0",
+ "ip": "testcaches",
+ "status": "REPORTED",
+ "cacheGroup": "cg0",
+ "ip6": null,
+ "port": 30001,
+ "httpsPort": null,
+ "hostName": "server1",
+ "fqdn": "server1.monitor-integration.test",
+ "interfaceName": "bond0",
+ "type": "EDGE",
+ "hashId": "server1",
+ "deliveryServices": {"ds0":["ds0.monitor-integration.test"]}
+ }
+ ],
+ "cacheGroups": [
+ {
+ "cg0": {
+ "name": "cg0",
+ "coordinates": {"latitude": 0, "longitude": 0}
+ }
+ }
+ ],
+ "config": {
+ "peers.polling.interval": 30,
+ "health.polling.interval": 2000,
+ "heartbeat.polling.interval": 2000,
+ "tm.polling.interval": 30
+ },
+ "trafficMonitors": [
+ {
+ "port": 80,
+ "ip6": "",
+ "ip": "trafficmonitor",
+ "hostName": "trafficmonitor",
+ "fqdn": "trafficmonitor.traffic-monitor-integration.test",
+ "profile": "Monitor0",
+ "location": "cg0",
+ "status": "REPORTED"
+ }
+ ],
+ "deliveryServices": [
+ {
+ "xmlId": "ds0",
+ "TotalTpsThreshold": 1000000,
+ "status": "Available",
+ "TotalKbpsThreshold": 10000000
+ }
+ ],
+ "profiles": [
+ {
+ "parameters": {
+ "health.connection.timeout": 10,
+ "health.polling.url": "http://${hostname}/_astats?application=plugin.remap",
+ "health.polling.format": "",
+ "health.polling.type": "",
+ "history.count": 0,
+ "MinFreeKbps": 20000,
+ "health_threshold": {}
+ },
+ "name": "Edge0",
+ "type": "EDGE"
+ },
+ {
+ "parameters": {
+ "health.connection.timeout": 10,
+ "health.polling.url": "",
+ "health.polling.format": "",
+ "health.polling.type": "",
+ "history.count": 5,
+ "MinFreeKbps": 20000,
+ "health_threshold": {}
+ },
+ "name": "Monitor0",
+ "type": "RASCAL"
+ }
+ ]
+}
+'
+
+ curl -Lvsk ${TESTTO_URI}/api/1.2/servers -X POST -d '
Review comment:
Use newer API version than v1
##########
File path: traffic_monitor/tests/integration/Dockerfile_run.sh
##########
@@ -0,0 +1,394 @@
+#!/usr/bin/env bash
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+# The following environment variables must be set (ordinarily by `docker run -e` arguments):
+# TO_URI
+# TO_USER
+# TO_PASS
+# CDN
+
+# Check that env vars are set
+envvars=( TESTTO_URI TESTTO_PORT TESTCACHES_URI TESTCACHES_PORT_START TM_URI )
+for v in $envvars
+do
+ if [[ -z $$v ]]; then echo "$v is unset"; exit 1; fi
+done
+
+
+CFG_FILE=/traffic-monitor-integration-test.cfg
+
+start() {
+ printf "DEBUG traffic_monitor_integration starting\n"
+
+ exec /traffic_monitor_integration_test -test.v -cfg $CFG_FILE
+}
+
+init() {
+ wait_for_to
+
+ curl -Lvsk ${TESTTO_URI}/api/1.2/cdns/fake/snapshot -X POST -d '
+{
+ "config": {
+ "api.cache-control.max_age": "30",
+ "consistent.dns.routing": "true",
+ "coveragezone.polling.interval": "30",
+ "coveragezone.polling.url": "30",
+ "dnssec.dynamic.response.expiration": "60",
+ "dnssec.enabled": "false",
+ "domain_name": "monitor-integration.test",
+ "federationmapping.polling.interval": "60",
+ "federationmapping.polling.url": "foo",
+ "geolocation.polling.interval": "30",
+ "geolocation.polling.url": "foo",
+ "keystore.maintenance.interval": "30",
+ "neustar.polling.interval": "30",
+ "neustar.polling.url": "foo",
+ "soa": {
+
+ },
+ "dnssec.inception": "0",
+ "ttls": {
+ "admin": "30",
+ "expire": "30",
+ "minimum": "30",
+ "refresh": "30",
+ "retry": "30"
+ },
+ "weight": "1",
+ "zonemanager.cache.maintenance.interval": "30",
+ "zonemanager.threadpool.scale": "1"
+ },
+ "contentServers": {
+ "server0": {
+ "cacheGroup": "cg0",
+ "profile": "Edge0",
+ "fqdn": "server0.monitor-integration.test",
+ "hashCount": 1,
+ "hashId": "server0",
+ "httpsPort" : null,
+ "ip": "testcaches",
+ "ip6": null,
+ "locationId": "",
+ "port" : 30000,
+ "status": "REPORTED",
+ "type": "EDGE",
+ "deliveryServices": {"ds0":["ds0.monitor-integration.test"]},
+ "routingDisabled": 0
+ },
+ "server1": {
+ "cacheGroup": "cg0",
+ "profile": "Edge0",
+ "fqdn": "server1.monitor-integration.test",
+ "hashCount": 1,
+ "hashId": "server1",
+ "httpsPort" : null,
+ "ip": "testcaches",
+ "ip6": null,
+ "locationId": "",
+ "port" : 30001,
+ "status": "REPORTED",
+ "type": "EDGE",
+ "deliveryServices": {"ds0":["ds0.monitor-integration.test"]},
+ "routingDisabled": 0
+ }
+ },
+ "deliveryServices": {
+ "ds0": {
+ "anonymousBlockingEnabled": false,
+ "consistentHashQueryParams": [],
+ "consistentHashRegex": "",
+ "coverageZoneOnly": false,
+ "dispersion": {
+ "limit": 1,
+ "shuffled": false
+ },
+ "domains": ["ds0.monitor-integration.test"],
+ "geolocationProvider": null,
+ "matchsets": [
+ {
+ "protocol": "HTTP",
+ "matchlist": [
+ {
+ "regex": "\\.*ds0\\.*",
+ "match-type": "regex"
+ }
+ ]
+ }
+ ],
+ "missLocation": {"lat": 0, "lon": 0},
+ "protocol": {
+ "acceptHttp": true,
+ "acceptHttps": false,
+ "redirectToHttps": false
+ },
+ "regionalGeoBlocking": "false",
+ "responseHeaders": {},
+ "requestHeaders": [],
+ "soa": {
+ "admin": "60",
+ "expire": "60",
+ "minimum": "60",
+ "refresh": "60",
+ "retry": "60"
+ },
+ "sslEnabled": false,
+ "ttl": 60,
+ "ttls": {
+ "A": "60",
+ "AAAA": "60",
+ "DNSKEY": "60",
+ "DS": "60",
+ "NS": "60",
+ "SOA": "60"
+ },
+ "maxDnsIpsForLocation": 3,
+ "ip6RoutingEnabled": false,
+ "routingName": "ccr",
+ "bypassDestination": null,
+ "deepCachingType": null,
+ "geoEnabled": false,
+ "geoLimitRedirectURL": null,
+ "staticDnsEntries": []
+ }
+ },
+ "edgeLocations": {
+ "cg0": {"latitude":0, "longitude":0}
+ },
+ "trafficRouterLocations": {
+ "tr0": {"latitude":0, "longitude":0}
+ },
+ "monitors": {
+ "trafficmonitor": {
+ "fqdn": "trafficmonitor.monitor-integration.test",
+ "httpsPort": null,
+ "ip": "trafficmonitor",
+ "ip6": null,
+ "location": "cg0",
+ "port": 80,
+ "profile": "Monitor0",
+ "status": "REPORTED"
+ }
+ },
+ "stats": {
+ "CDN_name": "fake",
+ "date": 1561000000,
+ "tm_host": "testto",
+ "tm_path": "/fake",
+ "tm_user": "fake",
+ "tm_version": "integrationtest/0.fake"
+ }
+}
+'
+
+ curl -Lvsk ${TESTTO_URI}/api/1.2/cdns/fake/configs/monitoring.json -X POST -d '
+{
+ "trafficServers": [
+ {
+ "profile": "Edge0",
+ "ip": "testcaches",
+ "status": "REPORTED",
+ "cacheGroup": "cg0",
+ "ip6": null,
+ "port": 30000,
+ "httpsPort": null,
+ "hostName": "server0",
+ "fqdn": "server0.monitor-integration.test",
+ "interfaceName": "bond0",
+ "type": "EDGE",
+ "hashId": "server0",
+ "deliveryServices": {"ds0":["ds0.monitor-integration.test"]}
+ },
+ {
+ "profile": "Edge0",
+ "ip": "testcaches",
+ "status": "REPORTED",
+ "cacheGroup": "cg0",
+ "ip6": null,
+ "port": 30001,
+ "httpsPort": null,
+ "hostName": "server1",
+ "fqdn": "server1.monitor-integration.test",
+ "interfaceName": "bond0",
+ "type": "EDGE",
+ "hashId": "server1",
+ "deliveryServices": {"ds0":["ds0.monitor-integration.test"]}
+ }
+ ],
+ "cacheGroups": [
+ {
+ "cg0": {
+ "name": "cg0",
+ "coordinates": {"latitude": 0, "longitude": 0}
+ }
+ }
+ ],
+ "config": {
+ "peers.polling.interval": 30,
+ "health.polling.interval": 2000,
+ "heartbeat.polling.interval": 2000,
+ "tm.polling.interval": 30
+ },
+ "trafficMonitors": [
+ {
+ "port": 80,
+ "ip6": "",
+ "ip": "trafficmonitor",
+ "hostName": "trafficmonitor",
+ "fqdn": "trafficmonitor.traffic-monitor-integration.test",
+ "profile": "Monitor0",
+ "location": "cg0",
+ "status": "REPORTED"
+ }
+ ],
+ "deliveryServices": [
+ {
+ "xmlId": "ds0",
+ "TotalTpsThreshold": 1000000,
+ "status": "Available",
+ "TotalKbpsThreshold": 10000000
+ }
+ ],
+ "profiles": [
+ {
+ "parameters": {
+ "health.connection.timeout": 10,
+ "health.polling.url": "http://${hostname}/_astats?application=plugin.remap",
+ "health.polling.format": "",
+ "health.polling.type": "",
+ "history.count": 0,
+ "MinFreeKbps": 20000,
+ "health_threshold": {}
+ },
+ "name": "Edge0",
+ "type": "EDGE"
+ },
+ {
+ "parameters": {
+ "health.connection.timeout": 10,
+ "health.polling.url": "",
+ "health.polling.format": "",
+ "health.polling.type": "",
+ "history.count": 5,
+ "MinFreeKbps": 20000,
+ "health_threshold": {}
+ },
+ "name": "Monitor0",
+ "type": "RASCAL"
+ }
+ ]
+}
+'
+
+ curl -Lvsk ${TESTTO_URI}/api/1.2/servers -X POST -d '
+[
+ {
+ "cachegroup": "foo",
+ "cachegroupId": 0,
+ "cdnId": 1,
+ "cdnName": "fake",
+ "deliveryServices": null,
+ "fqdn": "trafficmonitor.traffic-monitor-integration.test",
+ "guid": "foo",
+ "hostName": "trafficmonitor",
+ "httpsPort": null,
+ "id": 1,
+ "iloIpAddress": null,
+ "iloIpGateway": null,
+ "iloIpNetmask": null,
+ "iloPassword": null,
+ "iloUsername": null,
+ "interfaceMtu": null,
+ "interfaceName": "bond0",
+ "ip6Address": null,
+ "ip6Gateway": null,
+ "ipAddress": "trafficmonitor",
+ "ipGateway": "192.0.0.1",
+ "ipNetmask": "255.255.255.0",
+ "lastUpdated": "2019",
+ "mgmtIpAddress": null,
+ "mgmtIpGateway": null,
+ "mgmtIpNetmask": null,
+ "offlineReason": "none",
+ "physLocation": "",
+ "physLocationId": 0,
+ "profile": "Monitor0",
+ "profileDesc": "nodesc",
+ "profileId": 0,
+ "rack": "",
+ "revalPending": false,
+ "routerHostName": "",
+ "routerPortName": "",
+ "status": "REPORTED",
+ "statusId": 0,
+ "tcpPort": 80,
+ "type": "RASCAL",
+ "typeId": 0,
+ "updPending": false,
+ "xmppId": "",
+ "xmppPasswd": ""
+ }
+]
+'
+
+ # DEBUG
+ printf "\n\ntestto:\n"
+ curl -Lk ${TESTTO_URI}/api/1.2/cdns/foo/snapshot | head -5
+ printf "\n\ntestcaches:\n"
+ curl -Lk ${TESTCACHES_URI}:${TESTCACHES_PORT_START}/_astats | head -5
+ printf "\n\ntraffic_monitor:\n"
+ curl -Lk http://trafficmonitor | head -5
+
+
+ cat > $CFG_FILE <<- EOF
+{
+ "trafficMonitor": {
+ "url": "$TM_URI"
+ },
+ "default": {
+ "session": {
+ "timeoutInSecs": 30
+ },
+ "log": {
+ "debug": "stdout",
+ "event": "stdout",
+ "info": "stdout",
+ "error": "stdout",
+ "warning": "stdout"
+ }
+ }
+}
+EOF
+
+ echo "INITIALIZED=1" >> /etc/environment
+}
+
+wait_for_to() {
+ while true; do
+ curl -Lvsk ${TESTTO_URI}/api/1.2/servers 2>&1 1> /dev/null | grep -E '< HTTP/[0-9]\.?[0-9]* 200'
Review comment:
This should use a newer TO API version.
##########
File path: traffic_monitor/tests/integration/Dockerfile_run.sh
##########
@@ -0,0 +1,394 @@
+#!/usr/bin/env bash
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+# The following environment variables must be set (ordinarily by `docker run -e` arguments):
+# TO_URI
+# TO_USER
+# TO_PASS
+# CDN
+
+# Check that env vars are set
+envvars=( TESTTO_URI TESTTO_PORT TESTCACHES_URI TESTCACHES_PORT_START TM_URI )
+for v in $envvars
+do
+ if [[ -z $$v ]]; then echo "$v is unset"; exit 1; fi
Review comment:
Should be `${!v}`, not `$$v`
##########
File path: traffic_monitor/tests/integration/client_test.go
##########
@@ -0,0 +1,112 @@
+package integration
+
+/*
+
Review comment:
This newline can be removed
##########
File path: traffic_monitor/tests/integration/Dockerfile_run.sh
##########
@@ -0,0 +1,394 @@
+#!/usr/bin/env bash
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+# The following environment variables must be set (ordinarily by `docker run -e` arguments):
+# TO_URI
+# TO_USER
+# TO_PASS
+# CDN
+
+# Check that env vars are set
+envvars=( TESTTO_URI TESTTO_PORT TESTCACHES_URI TESTCACHES_PORT_START TM_URI )
+for v in $envvars
+do
+ if [[ -z $$v ]]; then echo "$v is unset"; exit 1; fi
+done
+
+
+CFG_FILE=/traffic-monitor-integration-test.cfg
+
+start() {
+ printf "DEBUG traffic_monitor_integration starting\n"
+
+ exec /traffic_monitor_integration_test -test.v -cfg $CFG_FILE
+}
+
+init() {
+ wait_for_to
+
+ curl -Lvsk ${TESTTO_URI}/api/1.2/cdns/fake/snapshot -X POST -d '
Review comment:
Should not use API v1
##########
File path: traffic_monitor/tests/integration/kbps_test.go
##########
@@ -0,0 +1,75 @@
+package integration
+
+/*
+
Review comment:
Unnecessary newline
##########
File path: traffic_monitor/tools/testto/Dockerfile
##########
@@ -0,0 +1,29 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+FROM centos/systemd
Review comment:
The [`centos/systemd`](https://hub.docker.com/r/centos/systemd/tags) image was last updated in December 2018. Use `centos:8` (preferred, since RPMs build for CentOS 8 by default) or `centos:7`.
##########
File path: traffic_monitor/Dockerfile
##########
@@ -0,0 +1,41 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+# This is a very simple Dockerfile.
+# All it does is install and start the Traffic Monitor, given a Traffic Ops to point it to.
+# It doesn't do any of the complex things the Dockerfiles in infrastructure/docker or infrastructure/cdn-in-a-box do, like inserting itself into Traffic Ops.
+# It is designed for a very simple use case, where the complex orchestration of other Traffic Control components is done elsewhere (or manually).
+
+# Example Build and Run:
+# docker build --build-arg RPM=traffic_monitor.rpm --tag traffic_monitor:latest .
+#
+# docker run --detach --name tm --hostname tm --net=tmi --env TO_URI=http://to.invalid:3000 --env TO_USER=user --env TO_PASS=pass traffic_monitor:latest
+
+FROM centos/systemd
Review comment:
As mentioned above, the [`centos/systemd`](https://hub.docker.com/r/centos/systemd/tags) image was last updated in December 2018. Use `centos:8` (preferred, since RPMs build for CentOS 8 by default) or `centos:7`.
##########
File path: traffic_monitor/tools/testcaches/Dockerfile
##########
@@ -0,0 +1,29 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+FROM centos/systemd
Review comment:
[The `centos/systemd` image](https://hub.docker.com/r/centos/systemd/tags) was last updated in December 2018. Use `centos:8` (preferred, since RPMs build for CentOS 8 by default) or `centos:7`.
##########
File path: traffic_monitor/tools/testcaches/Dockerfile_run.sh
##########
@@ -0,0 +1,47 @@
+#!/usr/bin/env bash
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+
+# envvars=( PORT )
+# for v in $envvars
+# do
+# if [[ -z $$v ]]; then echo "$v is unset"; exit 1; fi
+# done
+
+start() {
+ ARGS=
+ if [[ -n "${NUM_PORTS}" ]]; then
+ ARGS="$ARGS -numPorts ${NUM_PORTS}"
+ fi
+ if [[ -n "${NUM_REMAPS}" ]]; then
+ ARGS="$ARGS -numRemaps ${NUM_REMAPS}"
+ fi
+ if [[ -n "${PORT_START}" ]]; then
+ ARGS="$ARGS -portStart ${PORT_START}"
+ fi
+ testcaches ${ARGS}
+}
+
+init() {
+
Review comment:
Nit: This blank line serves no purpose
##########
File path: traffic_monitor/tools/testcaches/README.md
##########
@@ -30,3 +30,68 @@ Each port is a unique fake server, with distinct incrementing stats.
When run with no parameters, it defaults to ports 40000-40999 and 1000 remaps.
Stats are served at the regular ATS `stats_over_http` endpoint, `_astats`. For example, if it's serving on port 40000, it can be reached via `curl http://localhost:40000/_astats`. It also respects the `?application=system` query parameter, and will serve only system stats (the Monitor "health check" [as opposed to the "stat check"]). For example, `curl http://localhost:40000/_astats?application=system`.
+
+## Commands
+
+The `testcaches` app accepts a number of commands, which manipulate the data it serves. These command are all available via HTTP requests.
+
+Each HTTP request is made to a fake cache at a specific port. Thus, you can modify data served by each fake cache independently.
+
+The commands are:
+
+### `/cmd/setstat`
+
+Sets how much a stat increments by every interval (currently, an interval is hard-coded to 1 second). Accepts a min and max, and will increment by a random number between them. The min may equal the max, if a constant increment is desired.
+
+Query Parameters:
+`remap` - the remap rule to set
+`stat` - the stat to set
+`min` - the minimum number to increment by
+`max` - the minimum to increment by
+
+Example:
+`curl -Lvsk http://localhost:4242/cmd/setstat?remap=num1.example.net&stat=out_bytes&min=10&max=25`
+
+### `setsystem`
+
+Sets system stats to constant values. Multiple stats may be set with a single request.
+
+Query Parameters:
+`loadavg1m` - the 1m loadavg in the `system` object.
+`loadavg5m` - the 5m loadavg in the `system` object.
+`loadavg10m` - the 10m loadavg in the `system` object.
+`speed` - the network interface speed in the `system` object. This number is in kilobits. I.e. 20000 means 20Gbps.
+
+Example:
+`curl -sk 'http://localhost:4242/cmd/setsystem?loadavg1m=10.1&loadavg5m=27.92&loadavg10m=3.4&speed=20000' `
+
+### `setdelay`
+
+Sets the delay for serving all _astats requests to this fake cache. Accepts a minimum and maximum, which may be qual, and delays the request by a random interval between them. When a delay is set, the server immediately accepts client requests, reads headers and sets up the connection, and then delays writing out the body.
+
+Query Parameters:
+`min` - the minimum delay time, in milliseconds
+`max` - the maximum delay time, in milliseconds
+
+Example:
+`curl -Lvsk 'http://localhost:4242/cmd/setdelay?min=200&max=600'`
+
+## Docker
+
+Build environment variables: none
+
+Run environment variables:
+- `NUM_PORTS` - app `numPorts` argument
+- `NUM_REMAPS` - app `numRemaps` argument
+- `PORT_START` - app `portStart` argument
+
+### Building
+
+`docker build . --tag testcaches`
+
+### Running
+
+```
+docker network create tmi
+docker run --detach --network tmi --name testcaches --hostname testcaches --publish 30000-30099:30000-30099 --env NUM_PORTS=100 --env NUM_REMAPS=100 --env PORT_START=30000 testcaches
Review comment:
`tools/testcaches` should have a `docker-compose.yml` so that the user doesn't need to manually create networks, manually set hostnames, keep track of long commands like this, etc.
##########
File path: traffic_monitor/Dockerfile
##########
@@ -0,0 +1,41 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+# This is a very simple Dockerfile.
+# All it does is install and start the Traffic Monitor, given a Traffic Ops to point it to.
+# It doesn't do any of the complex things the Dockerfiles in infrastructure/docker or infrastructure/cdn-in-a-box do, like inserting itself into Traffic Ops.
+# It is designed for a very simple use case, where the complex orchestration of other Traffic Control components is done elsewhere (or manually).
+
+# Example Build and Run:
+# docker build --build-arg RPM=traffic_monitor.rpm --tag traffic_monitor:latest .
+#
+# docker run --detach --name tm --hostname tm --net=tmi --env TO_URI=http://to.invalid:3000 --env TO_USER=user --env TO_PASS=pass traffic_monitor:latest
+
+FROM centos/systemd
+MAINTAINER dev@trafficcontrol.apache.org
+
+ARG RPM=traffic_monitor.rpm
+ADD $RPM /
+
+RUN yum install -y initscripts
+RUN yum install -y /$(basename $RPM)
+RUN rm /$(basename $RPM)
+
+RUN curl -L jq https://github.com/stedolan/jq/releases/download/jq-1.5/jq-linux64 > /usr/sbin/jq && chmod +x /usr/sbin/jq
Review comment:
[`jq` 1.5](https://github.com/stedolan/jq/releases/tag/jq-1.5) was released in 2015. Also, we should install `jq` from EPEL, not manually downloading a binary into the PATH.
##########
File path: traffic_monitor/tests/integration/Dockerfile_run.sh
##########
@@ -0,0 +1,394 @@
+#!/usr/bin/env bash
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+# The following environment variables must be set (ordinarily by `docker run -e` arguments):
+# TO_URI
+# TO_USER
+# TO_PASS
+# CDN
+
+# Check that env vars are set
+envvars=( TESTTO_URI TESTTO_PORT TESTCACHES_URI TESTCACHES_PORT_START TM_URI )
+for v in $envvars
+do
+ if [[ -z $$v ]]; then echo "$v is unset"; exit 1; fi
+done
+
+
+CFG_FILE=/traffic-monitor-integration-test.cfg
+
+start() {
+ printf "DEBUG traffic_monitor_integration starting\n"
+
+ exec /traffic_monitor_integration_test -test.v -cfg $CFG_FILE
+}
+
+init() {
+ wait_for_to
+
+ curl -Lvsk ${TESTTO_URI}/api/1.2/cdns/fake/snapshot -X POST -d '
+{
+ "config": {
+ "api.cache-control.max_age": "30",
+ "consistent.dns.routing": "true",
+ "coveragezone.polling.interval": "30",
+ "coveragezone.polling.url": "30",
+ "dnssec.dynamic.response.expiration": "60",
+ "dnssec.enabled": "false",
+ "domain_name": "monitor-integration.test",
+ "federationmapping.polling.interval": "60",
+ "federationmapping.polling.url": "foo",
+ "geolocation.polling.interval": "30",
+ "geolocation.polling.url": "foo",
+ "keystore.maintenance.interval": "30",
+ "neustar.polling.interval": "30",
+ "neustar.polling.url": "foo",
+ "soa": {
+
+ },
+ "dnssec.inception": "0",
+ "ttls": {
+ "admin": "30",
+ "expire": "30",
+ "minimum": "30",
+ "refresh": "30",
+ "retry": "30"
+ },
+ "weight": "1",
+ "zonemanager.cache.maintenance.interval": "30",
+ "zonemanager.threadpool.scale": "1"
+ },
+ "contentServers": {
+ "server0": {
+ "cacheGroup": "cg0",
+ "profile": "Edge0",
+ "fqdn": "server0.monitor-integration.test",
+ "hashCount": 1,
+ "hashId": "server0",
+ "httpsPort" : null,
+ "ip": "testcaches",
+ "ip6": null,
+ "locationId": "",
+ "port" : 30000,
+ "status": "REPORTED",
+ "type": "EDGE",
+ "deliveryServices": {"ds0":["ds0.monitor-integration.test"]},
+ "routingDisabled": 0
+ },
+ "server1": {
+ "cacheGroup": "cg0",
+ "profile": "Edge0",
+ "fqdn": "server1.monitor-integration.test",
+ "hashCount": 1,
+ "hashId": "server1",
+ "httpsPort" : null,
+ "ip": "testcaches",
+ "ip6": null,
+ "locationId": "",
+ "port" : 30001,
+ "status": "REPORTED",
+ "type": "EDGE",
+ "deliveryServices": {"ds0":["ds0.monitor-integration.test"]},
+ "routingDisabled": 0
+ }
+ },
+ "deliveryServices": {
+ "ds0": {
+ "anonymousBlockingEnabled": false,
+ "consistentHashQueryParams": [],
+ "consistentHashRegex": "",
+ "coverageZoneOnly": false,
+ "dispersion": {
+ "limit": 1,
+ "shuffled": false
+ },
+ "domains": ["ds0.monitor-integration.test"],
+ "geolocationProvider": null,
+ "matchsets": [
+ {
+ "protocol": "HTTP",
+ "matchlist": [
+ {
+ "regex": "\\.*ds0\\.*",
+ "match-type": "regex"
+ }
+ ]
+ }
+ ],
+ "missLocation": {"lat": 0, "lon": 0},
+ "protocol": {
+ "acceptHttp": true,
+ "acceptHttps": false,
+ "redirectToHttps": false
+ },
+ "regionalGeoBlocking": "false",
+ "responseHeaders": {},
+ "requestHeaders": [],
+ "soa": {
+ "admin": "60",
+ "expire": "60",
+ "minimum": "60",
+ "refresh": "60",
+ "retry": "60"
+ },
+ "sslEnabled": false,
+ "ttl": 60,
+ "ttls": {
+ "A": "60",
+ "AAAA": "60",
+ "DNSKEY": "60",
+ "DS": "60",
+ "NS": "60",
+ "SOA": "60"
+ },
+ "maxDnsIpsForLocation": 3,
+ "ip6RoutingEnabled": false,
+ "routingName": "ccr",
+ "bypassDestination": null,
+ "deepCachingType": null,
+ "geoEnabled": false,
+ "geoLimitRedirectURL": null,
+ "staticDnsEntries": []
+ }
+ },
+ "edgeLocations": {
+ "cg0": {"latitude":0, "longitude":0}
+ },
+ "trafficRouterLocations": {
+ "tr0": {"latitude":0, "longitude":0}
+ },
+ "monitors": {
+ "trafficmonitor": {
+ "fqdn": "trafficmonitor.monitor-integration.test",
+ "httpsPort": null,
+ "ip": "trafficmonitor",
+ "ip6": null,
+ "location": "cg0",
+ "port": 80,
+ "profile": "Monitor0",
+ "status": "REPORTED"
+ }
+ },
+ "stats": {
+ "CDN_name": "fake",
+ "date": 1561000000,
+ "tm_host": "testto",
+ "tm_path": "/fake",
+ "tm_user": "fake",
+ "tm_version": "integrationtest/0.fake"
+ }
+}
+'
+
+ curl -Lvsk ${TESTTO_URI}/api/1.2/cdns/fake/configs/monitoring.json -X POST -d '
Review comment:
If you want to stick to the TO API version that TM uses, you could use API v2, but API v1 is too old.
##########
File path: traffic_monitor/tests/integration/Dockerfile
##########
@@ -0,0 +1,48 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+# This is a very simple Dockerfile.
+# All it does is install and start the Traffic Monitor, given a Traffic Ops to point it to.
+# It doesn't do any of the complex things the Dockerfiles in infrastructure/docker or infrastructure/cdn-in-a-box do, like inserting itself into Traffic Ops.
+# It is designed for a very simple use case, where the complex orchestration of other Traffic Control components is done elsewhere (or manually).
+
+# Example Build and Run:
+# docker build --build-arg RPM=traffic_monitor.rpm --tag traffic_monitor:latest .
+#
+# docker run --detach --name tm --hostname tm --net=tmi --env TO_URI=http://to.invalid:3000 --env TO_USER=user --env TO_PASS=pass traffic_monitor:latest
+
+FROM centos/systemd
+MAINTAINER dev@trafficcontrol.apache.org
+
+# ARG RPM=traffic_monitor.rpm
+# ADD $RPM /
+
+RUN yum install -y initscripts
+RUN yum install -y epel-release
+RUN yum install -y golang
+# RUN yum install -y /$(basename $RPM)
+# RUN rm /$(basename $RPM)
+
+COPY traffic_monitor_integration_test /
+
+# TODO build go test here, instead of requiring it manually be built beforehand
+# See /infrastructure/cdn-in-a-box/traffic_ops_integration_test/Dockerfile
+
+RUN curl -L jq https://github.com/stedolan/jq/releases/download/jq-1.5/jq-linux64 > /usr/sbin/jq && chmod +x /usr/sbin/jq
Review comment:
Should install `jq` from EPEL
##########
File path: traffic_monitor/tests/integration/Dockerfile
##########
@@ -0,0 +1,48 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+# This is a very simple Dockerfile.
+# All it does is install and start the Traffic Monitor, given a Traffic Ops to point it to.
+# It doesn't do any of the complex things the Dockerfiles in infrastructure/docker or infrastructure/cdn-in-a-box do, like inserting itself into Traffic Ops.
+# It is designed for a very simple use case, where the complex orchestration of other Traffic Control components is done elsewhere (or manually).
+
+# Example Build and Run:
+# docker build --build-arg RPM=traffic_monitor.rpm --tag traffic_monitor:latest .
+#
+# docker run --detach --name tm --hostname tm --net=tmi --env TO_URI=http://to.invalid:3000 --env TO_USER=user --env TO_PASS=pass traffic_monitor:latest
+
+FROM centos/systemd
+MAINTAINER dev@trafficcontrol.apache.org
+
+# ARG RPM=traffic_monitor.rpm
+# ADD $RPM /
Review comment:
Can these commented lines be removed?
##########
File path: traffic_monitor/tests/integration/Dockerfile_run.sh
##########
@@ -0,0 +1,394 @@
+#!/usr/bin/env bash
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+# The following environment variables must be set (ordinarily by `docker run -e` arguments):
+# TO_URI
+# TO_USER
+# TO_PASS
+# CDN
+
+# Check that env vars are set
+envvars=( TESTTO_URI TESTTO_PORT TESTCACHES_URI TESTCACHES_PORT_START TM_URI )
+for v in $envvars
+do
+ if [[ -z $$v ]]; then echo "$v is unset"; exit 1; fi
+done
+
+
+CFG_FILE=/traffic-monitor-integration-test.cfg
+
+start() {
+ printf "DEBUG traffic_monitor_integration starting\n"
+
+ exec /traffic_monitor_integration_test -test.v -cfg $CFG_FILE
+}
+
+init() {
+ wait_for_to
+
+ curl -Lvsk ${TESTTO_URI}/api/1.2/cdns/fake/snapshot -X POST -d '
+{
+ "config": {
+ "api.cache-control.max_age": "30",
+ "consistent.dns.routing": "true",
+ "coveragezone.polling.interval": "30",
+ "coveragezone.polling.url": "30",
+ "dnssec.dynamic.response.expiration": "60",
+ "dnssec.enabled": "false",
+ "domain_name": "monitor-integration.test",
+ "federationmapping.polling.interval": "60",
+ "federationmapping.polling.url": "foo",
+ "geolocation.polling.interval": "30",
+ "geolocation.polling.url": "foo",
+ "keystore.maintenance.interval": "30",
+ "neustar.polling.interval": "30",
+ "neustar.polling.url": "foo",
+ "soa": {
+
+ },
+ "dnssec.inception": "0",
+ "ttls": {
+ "admin": "30",
+ "expire": "30",
+ "minimum": "30",
+ "refresh": "30",
+ "retry": "30"
+ },
+ "weight": "1",
+ "zonemanager.cache.maintenance.interval": "30",
+ "zonemanager.threadpool.scale": "1"
+ },
+ "contentServers": {
+ "server0": {
+ "cacheGroup": "cg0",
+ "profile": "Edge0",
+ "fqdn": "server0.monitor-integration.test",
+ "hashCount": 1,
+ "hashId": "server0",
+ "httpsPort" : null,
+ "ip": "testcaches",
+ "ip6": null,
+ "locationId": "",
+ "port" : 30000,
+ "status": "REPORTED",
+ "type": "EDGE",
+ "deliveryServices": {"ds0":["ds0.monitor-integration.test"]},
+ "routingDisabled": 0
+ },
+ "server1": {
+ "cacheGroup": "cg0",
+ "profile": "Edge0",
+ "fqdn": "server1.monitor-integration.test",
+ "hashCount": 1,
+ "hashId": "server1",
+ "httpsPort" : null,
+ "ip": "testcaches",
+ "ip6": null,
+ "locationId": "",
+ "port" : 30001,
+ "status": "REPORTED",
+ "type": "EDGE",
+ "deliveryServices": {"ds0":["ds0.monitor-integration.test"]},
+ "routingDisabled": 0
+ }
+ },
+ "deliveryServices": {
+ "ds0": {
+ "anonymousBlockingEnabled": false,
+ "consistentHashQueryParams": [],
+ "consistentHashRegex": "",
+ "coverageZoneOnly": false,
+ "dispersion": {
+ "limit": 1,
+ "shuffled": false
+ },
+ "domains": ["ds0.monitor-integration.test"],
+ "geolocationProvider": null,
+ "matchsets": [
+ {
+ "protocol": "HTTP",
+ "matchlist": [
+ {
+ "regex": "\\.*ds0\\.*",
+ "match-type": "regex"
+ }
+ ]
+ }
+ ],
+ "missLocation": {"lat": 0, "lon": 0},
+ "protocol": {
+ "acceptHttp": true,
+ "acceptHttps": false,
+ "redirectToHttps": false
+ },
+ "regionalGeoBlocking": "false",
+ "responseHeaders": {},
+ "requestHeaders": [],
+ "soa": {
+ "admin": "60",
+ "expire": "60",
+ "minimum": "60",
+ "refresh": "60",
+ "retry": "60"
+ },
+ "sslEnabled": false,
+ "ttl": 60,
+ "ttls": {
+ "A": "60",
+ "AAAA": "60",
+ "DNSKEY": "60",
+ "DS": "60",
+ "NS": "60",
+ "SOA": "60"
+ },
+ "maxDnsIpsForLocation": 3,
+ "ip6RoutingEnabled": false,
+ "routingName": "ccr",
+ "bypassDestination": null,
+ "deepCachingType": null,
+ "geoEnabled": false,
+ "geoLimitRedirectURL": null,
+ "staticDnsEntries": []
+ }
+ },
+ "edgeLocations": {
+ "cg0": {"latitude":0, "longitude":0}
+ },
+ "trafficRouterLocations": {
+ "tr0": {"latitude":0, "longitude":0}
+ },
+ "monitors": {
+ "trafficmonitor": {
+ "fqdn": "trafficmonitor.monitor-integration.test",
+ "httpsPort": null,
+ "ip": "trafficmonitor",
+ "ip6": null,
+ "location": "cg0",
+ "port": 80,
+ "profile": "Monitor0",
+ "status": "REPORTED"
+ }
+ },
+ "stats": {
+ "CDN_name": "fake",
+ "date": 1561000000,
+ "tm_host": "testto",
+ "tm_path": "/fake",
+ "tm_user": "fake",
+ "tm_version": "integrationtest/0.fake"
+ }
+}
+'
+
+ curl -Lvsk ${TESTTO_URI}/api/1.2/cdns/fake/configs/monitoring.json -X POST -d '
+{
+ "trafficServers": [
+ {
+ "profile": "Edge0",
+ "ip": "testcaches",
+ "status": "REPORTED",
+ "cacheGroup": "cg0",
+ "ip6": null,
+ "port": 30000,
+ "httpsPort": null,
+ "hostName": "server0",
+ "fqdn": "server0.monitor-integration.test",
+ "interfaceName": "bond0",
+ "type": "EDGE",
+ "hashId": "server0",
+ "deliveryServices": {"ds0":["ds0.monitor-integration.test"]}
+ },
+ {
+ "profile": "Edge0",
+ "ip": "testcaches",
+ "status": "REPORTED",
+ "cacheGroup": "cg0",
+ "ip6": null,
+ "port": 30001,
+ "httpsPort": null,
+ "hostName": "server1",
+ "fqdn": "server1.monitor-integration.test",
+ "interfaceName": "bond0",
+ "type": "EDGE",
+ "hashId": "server1",
+ "deliveryServices": {"ds0":["ds0.monitor-integration.test"]}
+ }
+ ],
+ "cacheGroups": [
+ {
+ "cg0": {
+ "name": "cg0",
+ "coordinates": {"latitude": 0, "longitude": 0}
+ }
+ }
+ ],
+ "config": {
+ "peers.polling.interval": 30,
+ "health.polling.interval": 2000,
+ "heartbeat.polling.interval": 2000,
+ "tm.polling.interval": 30
+ },
+ "trafficMonitors": [
+ {
+ "port": 80,
+ "ip6": "",
+ "ip": "trafficmonitor",
+ "hostName": "trafficmonitor",
+ "fqdn": "trafficmonitor.traffic-monitor-integration.test",
+ "profile": "Monitor0",
+ "location": "cg0",
+ "status": "REPORTED"
+ }
+ ],
+ "deliveryServices": [
+ {
+ "xmlId": "ds0",
+ "TotalTpsThreshold": 1000000,
+ "status": "Available",
+ "TotalKbpsThreshold": 10000000
+ }
+ ],
+ "profiles": [
+ {
+ "parameters": {
+ "health.connection.timeout": 10,
+ "health.polling.url": "http://${hostname}/_astats?application=plugin.remap",
+ "health.polling.format": "",
+ "health.polling.type": "",
+ "history.count": 0,
+ "MinFreeKbps": 20000,
+ "health_threshold": {}
+ },
+ "name": "Edge0",
+ "type": "EDGE"
+ },
+ {
+ "parameters": {
+ "health.connection.timeout": 10,
+ "health.polling.url": "",
+ "health.polling.format": "",
+ "health.polling.type": "",
+ "history.count": 5,
+ "MinFreeKbps": 20000,
+ "health_threshold": {}
+ },
+ "name": "Monitor0",
+ "type": "RASCAL"
+ }
+ ]
+}
+'
+
+ curl -Lvsk ${TESTTO_URI}/api/1.2/servers -X POST -d '
+[
+ {
+ "cachegroup": "foo",
+ "cachegroupId": 0,
+ "cdnId": 1,
+ "cdnName": "fake",
+ "deliveryServices": null,
+ "fqdn": "trafficmonitor.traffic-monitor-integration.test",
+ "guid": "foo",
+ "hostName": "trafficmonitor",
+ "httpsPort": null,
+ "id": 1,
+ "iloIpAddress": null,
+ "iloIpGateway": null,
+ "iloIpNetmask": null,
+ "iloPassword": null,
+ "iloUsername": null,
+ "interfaceMtu": null,
+ "interfaceName": "bond0",
+ "ip6Address": null,
+ "ip6Gateway": null,
+ "ipAddress": "trafficmonitor",
+ "ipGateway": "192.0.0.1",
+ "ipNetmask": "255.255.255.0",
+ "lastUpdated": "2019",
+ "mgmtIpAddress": null,
+ "mgmtIpGateway": null,
+ "mgmtIpNetmask": null,
+ "offlineReason": "none",
+ "physLocation": "",
+ "physLocationId": 0,
+ "profile": "Monitor0",
+ "profileDesc": "nodesc",
+ "profileId": 0,
+ "rack": "",
+ "revalPending": false,
+ "routerHostName": "",
+ "routerPortName": "",
+ "status": "REPORTED",
+ "statusId": 0,
+ "tcpPort": 80,
+ "type": "RASCAL",
+ "typeId": 0,
+ "updPending": false,
+ "xmppId": "",
+ "xmppPasswd": ""
+ }
+]
+'
+
+ # DEBUG
+ printf "\n\ntestto:\n"
+ curl -Lk ${TESTTO_URI}/api/1.2/cdns/foo/snapshot | head -5
Review comment:
API v1 is deprecated
##########
File path: traffic_monitor/Dockerfile_run.sh
##########
@@ -0,0 +1,81 @@
+#!/usr/bin/env bash
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+# The following environment variables must be set (ordinarily by `docker run -e` arguments):
+# TO_URI
+# TO_USER
+# TO_PASS
+# CDN
+
+# Check that env vars are set
+envvars=( TO_URI TO_USER TO_PASS CDN PORT )
+for v in $envvars
+do
+ if [[ -z $$v ]]; then echo "$v is unset"; exit 1; fi
Review comment:
`$$v` expands to `*process ID*v`. For parameter indirection, it can be `${!v}`.
##########
File path: traffic_monitor/tests/integration/Dockerfile
##########
@@ -0,0 +1,48 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+# This is a very simple Dockerfile.
+# All it does is install and start the Traffic Monitor, given a Traffic Ops to point it to.
+# It doesn't do any of the complex things the Dockerfiles in infrastructure/docker or infrastructure/cdn-in-a-box do, like inserting itself into Traffic Ops.
+# It is designed for a very simple use case, where the complex orchestration of other Traffic Control components is done elsewhere (or manually).
+
+# Example Build and Run:
+# docker build --build-arg RPM=traffic_monitor.rpm --tag traffic_monitor:latest .
+#
+# docker run --detach --name tm --hostname tm --net=tmi --env TO_URI=http://to.invalid:3000 --env TO_USER=user --env TO_PASS=pass traffic_monitor:latest
+
+FROM centos/systemd
+MAINTAINER dev@trafficcontrol.apache.org
+
+# ARG RPM=traffic_monitor.rpm
+# ADD $RPM /
+
+RUN yum install -y initscripts
+RUN yum install -y epel-release
+RUN yum install -y golang
+# RUN yum install -y /$(basename $RPM)
+# RUN rm /$(basename $RPM)
Review comment:
Are these commented lines needed?
##########
File path: traffic_monitor/tests/integration/config/config.go
##########
@@ -0,0 +1,157 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package config
Review comment:
Package declaration should come before Apache license
##########
File path: traffic_monitor/tests/integration/Dockerfile
##########
@@ -0,0 +1,48 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+# This is a very simple Dockerfile.
+# All it does is install and start the Traffic Monitor, given a Traffic Ops to point it to.
+# It doesn't do any of the complex things the Dockerfiles in infrastructure/docker or infrastructure/cdn-in-a-box do, like inserting itself into Traffic Ops.
+# It is designed for a very simple use case, where the complex orchestration of other Traffic Control components is done elsewhere (or manually).
+
+# Example Build and Run:
+# docker build --build-arg RPM=traffic_monitor.rpm --tag traffic_monitor:latest .
+#
+# docker run --detach --name tm --hostname tm --net=tmi --env TO_URI=http://to.invalid:3000 --env TO_USER=user --env TO_PASS=pass traffic_monitor:latest
+
+FROM centos/systemd
+MAINTAINER dev@trafficcontrol.apache.org
+
+# ARG RPM=traffic_monitor.rpm
+# ADD $RPM /
+
+RUN yum install -y initscripts
+RUN yum install -y epel-release
+RUN yum install -y golang
Review comment:
These `RUN` instructions can be consolidated into 1.
##########
File path: traffic_monitor/tests/integration/Dockerfile_run.sh
##########
@@ -0,0 +1,394 @@
+#!/usr/bin/env bash
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+# The following environment variables must be set (ordinarily by `docker run -e` arguments):
+# TO_URI
+# TO_USER
+# TO_PASS
+# CDN
+
+# Check that env vars are set
+envvars=( TESTTO_URI TESTTO_PORT TESTCACHES_URI TESTCACHES_PORT_START TM_URI )
+for v in $envvars
+do
+ if [[ -z $$v ]]; then echo "$v is unset"; exit 1; fi
+done
+
+
+CFG_FILE=/traffic-monitor-integration-test.cfg
+
+start() {
+ printf "DEBUG traffic_monitor_integration starting\n"
+
+ exec /traffic_monitor_integration_test -test.v -cfg $CFG_FILE
+}
+
+init() {
+ wait_for_to
+
+ curl -Lvsk ${TESTTO_URI}/api/1.2/cdns/fake/snapshot -X POST -d '
+{
+ "config": {
+ "api.cache-control.max_age": "30",
+ "consistent.dns.routing": "true",
+ "coveragezone.polling.interval": "30",
+ "coveragezone.polling.url": "30",
+ "dnssec.dynamic.response.expiration": "60",
+ "dnssec.enabled": "false",
+ "domain_name": "monitor-integration.test",
+ "federationmapping.polling.interval": "60",
+ "federationmapping.polling.url": "foo",
+ "geolocation.polling.interval": "30",
+ "geolocation.polling.url": "foo",
+ "keystore.maintenance.interval": "30",
+ "neustar.polling.interval": "30",
+ "neustar.polling.url": "foo",
+ "soa": {
+
+ },
+ "dnssec.inception": "0",
+ "ttls": {
+ "admin": "30",
+ "expire": "30",
+ "minimum": "30",
+ "refresh": "30",
+ "retry": "30"
+ },
+ "weight": "1",
+ "zonemanager.cache.maintenance.interval": "30",
+ "zonemanager.threadpool.scale": "1"
+ },
+ "contentServers": {
+ "server0": {
+ "cacheGroup": "cg0",
+ "profile": "Edge0",
+ "fqdn": "server0.monitor-integration.test",
+ "hashCount": 1,
+ "hashId": "server0",
+ "httpsPort" : null,
+ "ip": "testcaches",
+ "ip6": null,
+ "locationId": "",
+ "port" : 30000,
+ "status": "REPORTED",
+ "type": "EDGE",
+ "deliveryServices": {"ds0":["ds0.monitor-integration.test"]},
+ "routingDisabled": 0
+ },
+ "server1": {
+ "cacheGroup": "cg0",
+ "profile": "Edge0",
+ "fqdn": "server1.monitor-integration.test",
+ "hashCount": 1,
+ "hashId": "server1",
+ "httpsPort" : null,
+ "ip": "testcaches",
+ "ip6": null,
+ "locationId": "",
+ "port" : 30001,
+ "status": "REPORTED",
+ "type": "EDGE",
+ "deliveryServices": {"ds0":["ds0.monitor-integration.test"]},
+ "routingDisabled": 0
+ }
+ },
+ "deliveryServices": {
+ "ds0": {
+ "anonymousBlockingEnabled": false,
+ "consistentHashQueryParams": [],
+ "consistentHashRegex": "",
+ "coverageZoneOnly": false,
+ "dispersion": {
+ "limit": 1,
+ "shuffled": false
+ },
+ "domains": ["ds0.monitor-integration.test"],
+ "geolocationProvider": null,
+ "matchsets": [
+ {
+ "protocol": "HTTP",
+ "matchlist": [
+ {
+ "regex": "\\.*ds0\\.*",
+ "match-type": "regex"
+ }
+ ]
+ }
+ ],
+ "missLocation": {"lat": 0, "lon": 0},
+ "protocol": {
+ "acceptHttp": true,
+ "acceptHttps": false,
+ "redirectToHttps": false
+ },
+ "regionalGeoBlocking": "false",
+ "responseHeaders": {},
+ "requestHeaders": [],
+ "soa": {
+ "admin": "60",
+ "expire": "60",
+ "minimum": "60",
+ "refresh": "60",
+ "retry": "60"
+ },
+ "sslEnabled": false,
+ "ttl": 60,
+ "ttls": {
+ "A": "60",
+ "AAAA": "60",
+ "DNSKEY": "60",
+ "DS": "60",
+ "NS": "60",
+ "SOA": "60"
+ },
+ "maxDnsIpsForLocation": 3,
+ "ip6RoutingEnabled": false,
+ "routingName": "ccr",
+ "bypassDestination": null,
+ "deepCachingType": null,
+ "geoEnabled": false,
+ "geoLimitRedirectURL": null,
+ "staticDnsEntries": []
+ }
+ },
+ "edgeLocations": {
+ "cg0": {"latitude":0, "longitude":0}
+ },
+ "trafficRouterLocations": {
+ "tr0": {"latitude":0, "longitude":0}
+ },
+ "monitors": {
+ "trafficmonitor": {
+ "fqdn": "trafficmonitor.monitor-integration.test",
+ "httpsPort": null,
+ "ip": "trafficmonitor",
+ "ip6": null,
+ "location": "cg0",
+ "port": 80,
+ "profile": "Monitor0",
+ "status": "REPORTED"
+ }
+ },
+ "stats": {
+ "CDN_name": "fake",
+ "date": 1561000000,
+ "tm_host": "testto",
+ "tm_path": "/fake",
+ "tm_user": "fake",
+ "tm_version": "integrationtest/0.fake"
+ }
+}
+'
+
+ curl -Lvsk ${TESTTO_URI}/api/1.2/cdns/fake/configs/monitoring.json -X POST -d '
+{
+ "trafficServers": [
+ {
+ "profile": "Edge0",
+ "ip": "testcaches",
+ "status": "REPORTED",
+ "cacheGroup": "cg0",
+ "ip6": null,
+ "port": 30000,
+ "httpsPort": null,
+ "hostName": "server0",
+ "fqdn": "server0.monitor-integration.test",
+ "interfaceName": "bond0",
+ "type": "EDGE",
+ "hashId": "server0",
+ "deliveryServices": {"ds0":["ds0.monitor-integration.test"]}
+ },
+ {
+ "profile": "Edge0",
+ "ip": "testcaches",
+ "status": "REPORTED",
+ "cacheGroup": "cg0",
+ "ip6": null,
+ "port": 30001,
+ "httpsPort": null,
+ "hostName": "server1",
+ "fqdn": "server1.monitor-integration.test",
+ "interfaceName": "bond0",
+ "type": "EDGE",
+ "hashId": "server1",
+ "deliveryServices": {"ds0":["ds0.monitor-integration.test"]}
+ }
+ ],
+ "cacheGroups": [
+ {
+ "cg0": {
+ "name": "cg0",
+ "coordinates": {"latitude": 0, "longitude": 0}
+ }
+ }
+ ],
+ "config": {
+ "peers.polling.interval": 30,
+ "health.polling.interval": 2000,
+ "heartbeat.polling.interval": 2000,
+ "tm.polling.interval": 30
+ },
+ "trafficMonitors": [
+ {
+ "port": 80,
+ "ip6": "",
+ "ip": "trafficmonitor",
+ "hostName": "trafficmonitor",
+ "fqdn": "trafficmonitor.traffic-monitor-integration.test",
+ "profile": "Monitor0",
+ "location": "cg0",
+ "status": "REPORTED"
+ }
+ ],
+ "deliveryServices": [
+ {
+ "xmlId": "ds0",
+ "TotalTpsThreshold": 1000000,
+ "status": "Available",
+ "TotalKbpsThreshold": 10000000
+ }
+ ],
+ "profiles": [
+ {
+ "parameters": {
+ "health.connection.timeout": 10,
+ "health.polling.url": "http://${hostname}/_astats?application=plugin.remap",
+ "health.polling.format": "",
+ "health.polling.type": "",
+ "history.count": 0,
+ "MinFreeKbps": 20000,
+ "health_threshold": {}
+ },
+ "name": "Edge0",
+ "type": "EDGE"
+ },
+ {
+ "parameters": {
+ "health.connection.timeout": 10,
+ "health.polling.url": "",
+ "health.polling.format": "",
+ "health.polling.type": "",
+ "history.count": 5,
+ "MinFreeKbps": 20000,
+ "health_threshold": {}
+ },
+ "name": "Monitor0",
+ "type": "RASCAL"
+ }
+ ]
+}
+'
+
+ curl -Lvsk ${TESTTO_URI}/api/1.2/servers -X POST -d '
+[
+ {
+ "cachegroup": "foo",
+ "cachegroupId": 0,
+ "cdnId": 1,
+ "cdnName": "fake",
+ "deliveryServices": null,
+ "fqdn": "trafficmonitor.traffic-monitor-integration.test",
+ "guid": "foo",
+ "hostName": "trafficmonitor",
+ "httpsPort": null,
+ "id": 1,
+ "iloIpAddress": null,
+ "iloIpGateway": null,
+ "iloIpNetmask": null,
+ "iloPassword": null,
+ "iloUsername": null,
+ "interfaceMtu": null,
+ "interfaceName": "bond0",
+ "ip6Address": null,
+ "ip6Gateway": null,
+ "ipAddress": "trafficmonitor",
+ "ipGateway": "192.0.0.1",
+ "ipNetmask": "255.255.255.0",
+ "lastUpdated": "2019",
+ "mgmtIpAddress": null,
+ "mgmtIpGateway": null,
+ "mgmtIpNetmask": null,
+ "offlineReason": "none",
+ "physLocation": "",
+ "physLocationId": 0,
+ "profile": "Monitor0",
+ "profileDesc": "nodesc",
+ "profileId": 0,
+ "rack": "",
+ "revalPending": false,
+ "routerHostName": "",
+ "routerPortName": "",
+ "status": "REPORTED",
+ "statusId": 0,
+ "tcpPort": 80,
+ "type": "RASCAL",
+ "typeId": 0,
+ "updPending": false,
+ "xmppId": "",
+ "xmppPasswd": ""
+ }
+]
+'
+
+ # DEBUG
+ printf "\n\ntestto:\n"
+ curl -Lk ${TESTTO_URI}/api/1.2/cdns/foo/snapshot | head -5
+ printf "\n\ntestcaches:\n"
+ curl -Lk ${TESTCACHES_URI}:${TESTCACHES_PORT_START}/_astats | head -5
+ printf "\n\ntraffic_monitor:\n"
+ curl -Lk http://trafficmonitor | head -5
+
+
+ cat > $CFG_FILE <<- EOF
+{
+ "trafficMonitor": {
+ "url": "$TM_URI"
+ },
+ "default": {
+ "session": {
+ "timeoutInSecs": 30
+ },
+ "log": {
+ "debug": "stdout",
+ "event": "stdout",
+ "info": "stdout",
+ "error": "stdout",
+ "warning": "stdout"
+ }
+ }
+}
+EOF
+
+ echo "INITIALIZED=1" >> /etc/environment
+}
+
+wait_for_to() {
+ while true; do
+ curl -Lvsk ${TESTTO_URI}/api/1.2/servers 2>&1 1> /dev/null | grep -E '< HTTP/[0-9]\.?[0-9]* 200'
+ RC=$?;
+ if [[ $RC -eq 0 ]] ; then
Review comment:
This can be
```shell
if curl -Lvsk ${TESTTO_URI}/api/1.2/servers 2>&1 1> /dev/null | grep -E '< HTTP/[0-9]\.?[0-9]* 200'; then
```
##########
File path: traffic_monitor/Dockerfile
##########
@@ -0,0 +1,41 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+# This is a very simple Dockerfile.
+# All it does is install and start the Traffic Monitor, given a Traffic Ops to point it to.
+# It doesn't do any of the complex things the Dockerfiles in infrastructure/docker or infrastructure/cdn-in-a-box do, like inserting itself into Traffic Ops.
+# It is designed for a very simple use case, where the complex orchestration of other Traffic Control components is done elsewhere (or manually).
+
+# Example Build and Run:
+# docker build --build-arg RPM=traffic_monitor.rpm --tag traffic_monitor:latest .
+#
+# docker run --detach --name tm --hostname tm --net=tmi --env TO_URI=http://to.invalid:3000 --env TO_USER=user --env TO_PASS=pass traffic_monitor:latest
+
+FROM centos/systemd
+MAINTAINER dev@trafficcontrol.apache.org
+
+ARG RPM=traffic_monitor.rpm
+ADD $RPM /
+
+RUN yum install -y initscripts
+RUN yum install -y /$(basename $RPM)
+RUN rm /$(basename $RPM)
Review comment:
This can all be a single `RUN` instruction
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [trafficcontrol] rob05c commented on pull request #5817: Traffic Monitor Integration Test Framework
Posted by GitBox <gi...@apache.org>.
rob05c commented on pull request #5817:
URL: https://github.com/apache/trafficcontrol/pull/5817#issuecomment-835431435
> A lot of the TM integration test Docker environment seems to be copied from an older version of CDN in a Box.
For the record, it is not. It's based on https://github.com/apache/trafficcontrol/pull/3707 whose compose was written from scratch. I believe the Dockerfiles may have been based on the independent dockerfiles in infrastructure/docker (which I also originally wrote), but neither they nor the docker-compose.yml are derivative works of the CDN-in-a-Box.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [trafficcontrol] zrhoffman commented on a change in pull request #5817: Traffic Monitor Integration Test Framework
Posted by GitBox <gi...@apache.org>.
zrhoffman commented on a change in pull request #5817:
URL: https://github.com/apache/trafficcontrol/pull/5817#discussion_r636636329
##########
File path: traffic_monitor/build_tests.sh
##########
@@ -0,0 +1,37 @@
+#!/bin/bash
+
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+export CGO_ENABLED=0
+export GOOS=linux
+
+if [ ! -f "traffic_monitor.rpm" ]; then
+ echo "Unable to find traffic_monitor.rpm."
+ echo "Run './pkg traffic_monitor_build' and copy TM rpm to this directory."
+ exit 1
+fi
+
+cd tools/testto
+rm testto
+go build
+cd - > /dev/null
+
+cd tools/testcaches
+rm testcaches
+go build
+cd - > /dev/null
Review comment:
The `testcaches` binary should be built as a step in `/traffic_monitor/tools/testcaches/Dockerfile`.
##########
File path: traffic_monitor/tests/_integration/Dockerfile
##########
@@ -0,0 +1,33 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+# This is a very simple Dockerfile.
+# All it does is install and start the Traffic Monitor, given a Traffic Ops to point it to.
+# It doesn't do any of the complex things the Dockerfiles in infrastructure/docker or infrastructure/cdn-in-a-box do, like inserting itself into Traffic Ops.
+# It is designed for a very simple use case, where the complex orchestration of other Traffic Control components is done elsewhere (or manually).
+
+FROM centos:8
+MAINTAINER dev@trafficcontrol.apache.org
+
+RUN yum install -y initscripts epel-release golang glibc jq
+
+COPY traffic_monitor_integration_test /
+
+COPY Dockerfile_run.sh /
+COPY monitoring.json /
+COPY snapshot.json /
Review comment:
This can all be a single `COPY` instruction to save layers
##########
File path: traffic_monitor/tests/_integration/README.md
##########
@@ -0,0 +1,13 @@
+# Traffic Monitor Integration Test Framework
+
+## Building
+
+From the `trafficcontrol` run `./pkg -v traffic_monitor_build` and copy the traffic_monitor rpm to the `traffic_monitor` directory.
+
+From the `trafficcontrol/traffic_monitor` directory run `build_tests.sh`
+
+## Running
+
+From the `trafficcontrol/traffic_monitor` directory run:
+
+`sudo docker-compose -p tmi --project-directory . -f tests/_integration/docker-compose.yml run tmintegrationtest`
Review comment:
`-p tmi` is unnecessary
##########
File path: traffic_monitor/build_tests.sh
##########
@@ -0,0 +1,37 @@
+#!/bin/bash
+
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+export CGO_ENABLED=0
+export GOOS=linux
+
+if [ ! -f "traffic_monitor.rpm" ]; then
+ echo "Unable to find traffic_monitor.rpm."
+ echo "Run './pkg traffic_monitor_build' and copy TM rpm to this directory."
+ exit 1
+fi
+
+cd tools/testto
+rm testto
+go build
+cd - > /dev/null
Review comment:
The `testto` binary should be built as a step in `/traffic_monitor/tools/testto/Dockerfile`.
##########
File path: traffic_monitor/tests/_integration/client_test.go
##########
@@ -0,0 +1,117 @@
+package _integration
+
+/*
+ Licensed under the Apache License, Version 2.0 (the "License");
+ you may not use this file except in compliance with the License.
+ You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+ Unless required by applicable law or agreed to in writing, software
+ distributed under the License is distributed on an "AS IS" BASIS,
+ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ See the License for the specific language governing permissions and
+ limitations under the License.
+*/
+
+import (
+ "testing"
+)
+
+func TestClient(t *testing.T) {
Review comment:
After increasing `monitorWaitSpan` to `90`, `_integration.TestClient()` fails for me:
```
* We are completely uploaded and fine
< HTTP/1.1 204 No Content
< Date: Fri, 21 May 2021 05:41:57 GMT
<
* Connection #0 to host testto left intact
DEBUG traffic_monitor_integration starting
=== RUN TestClient
client_test.go:23: client CacheCount error expected nil, actual reading body from 'http://trafficmonitor/api/cache-count': http: read on closed response body
client_test.go:29: client CacheAvailableCount error expected nil, actual reading body from 'http://trafficmonitor/api/cache-available-count': http: read on closed response body
client_test.go:35: client CacheDownCount error expected nil, actual reading body from 'http://trafficmonitor/api/cache-down-count': http: read on closed response body
client_test.go:41: client Version error expected nil, actual reading body from 'http://trafficmonitor/api/version': http: read on closed response body
client_test.go:47: client TrafficOpsURI error expected nil, actual reading body from 'http://trafficmonitor/api/traffic-ops-uri': http: read on closed response body
client_test.go:53: client BandwidthKBPS error expected nil, actual reading body from 'http://trafficmonitor/api/bandwidth-kbps': http: read on closed response body
client_test.go:59: client BandwidthCapacityKBPS error expected nil, actual reading body from 'http://trafficmonitor/api/bandwidth-capacity-kbps': http: read on closed response body
client_test.go:65: client CacheStatuses error expected nil, actual reading body from 'http://trafficmonitor/api/cache-statuses': http: read on closed response body
client_test.go:71: client MonitorConfig error expected nil, actual reading body from 'http://trafficmonitor/api/monitor-config': http: read on closed response body
client_test.go:77: client CRConfigHistory error expected nil, actual reading body from 'http://trafficmonitor/api/crconfig-history': http: read on closed response body
client_test.go:83: client EventLog error expected nil, actual reading body from 'http://trafficmonitor/publish/EventLog': http: read on closed response body
client_test.go:89: client CacheStats error expected nil, actual reading body from 'http://trafficmonitor/publish/CacheStats': http: read on closed response body
client_test.go:95: client CacheStatsNew error expected nil, actual reading body from 'http://trafficmonitor/publish/CacheStats': http: read on closed response body
client_test.go:101: client DSStats error expected nil, actual reading body from 'http://trafficmonitor/publish/DsStats': http: read on closed response body
client_test.go:107: client CRStates error expected nil, actual reading body from 'http://trafficmonitor/publish/CrStates': http: read on closed response body
client_test.go:113: client CRConfig error expected nil, actual reading body from 'http://trafficmonitor/publish/CrConfig': http: read on closed response body
--- FAIL: TestClient (0.07s)
=== RUN TestKBPS
kbps_test.go:30: client CRConfig error expected nil, actual reading body from 'http://trafficmonitor/publish/CrConfig': http: read on closed response body
--- FAIL: TestKBPS (0.00s)
FAIL
ERROR: 1
```
##########
File path: traffic_monitor/tests/_integration/Dockerfile_run.sh
##########
@@ -0,0 +1,166 @@
+#!/usr/bin/env bash
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+# The following environment variables must be set (ordinarily by `docker run -e` arguments):
+# TO_URI
+# TO_USER
+# TO_PASS
+# CDN
+# Check that env vars are set
+envvars=(TESTTO_URI TESTTO_PORT TESTCACHES_URI TESTCACHES_PORT_START TM_URI)
+for v in $envvars; do
+ if [[ -z ${!v} ]]; then
+ echo "$v is unset"
+ exit 1
+ fi
+done
+
+TO_API_VERSION=4.0
+CFG_FILE=/traffic-monitor-integration-test.cfg
+
+start() {
+ printf "DEBUG traffic_monitor_integration starting\n"
+
+ exec /traffic_monitor_integration_test -test.v -cfg $CFG_FILE
+}
+
+init() {
+ wait_for_endpoint "${TESTTO_URI}/api/${TO_API_VERSION}/servers"
+ wait_for_endpoint "${TESTCACHES_URI}:${TESTCACHES_PORT_START}/_astats"
+ wait_for_endpoint "${TM_URI}"
+ TESTCACHES_ADDRESS=$(ping testcaches -4 -c 1 | head -n 1 | grep -Eo '[0-9]+.[0-9]+.[0-9]+.[0-9]+')
+ TESTCACHES_GATEWAY=$(echo $TESTCACHES_ADDRESS | sed "s/\([0-9]\+.[0-9]\+.[0-9]\+.\)[0-9]/\11/")
+
+ jq "(.. | .address?) |= \"$TESTCACHES_ADDRESS\" | (.. | .gateway?) |= \"$TESTCACHES_GATEWAY\"" \
+ /monitoring.json >/monitoring.json.tmp && mv /monitoring.json.tmp /monitoring.json
+
+ curl -Lvsk ${TESTTO_URI}/api/${TO_API_VERSION}/cdns/fake/snapshot -X POST -d "@/snapshot.json"
+
+ curl -Lvsk ${TESTTO_URI}/api/${TO_API_VERSION}/cdns/fake/configs/monitoring -X POST -d '@/monitoring.json'
+
+ curl -Lvsk ${TESTTO_URI}/api/${TO_API_VERSION}/servers -X POST -d '
+[
+ {
+ "cachegroup": "foo",
+ "cachegroupId": 0,
+ "cdnId": 1,
+ "cdnName": "fake",
+ "deliveryServices": null,
+ "fqdn": "trafficmonitor.traffic-monitor-integration.test",
+ "guid": "foo",
+ "hostName": "trafficmonitor",
+ "httpsPort": null,
+ "id": 1,
+ "iloIpAddress": null,
+ "iloIpGateway": null,
+ "iloIpNetmask": null,
+ "iloPassword": null,
+ "iloUsername": null,
+ "interfaceMtu": null,
+ "interfaceName": "bond0",
+ "ip6Address": null,
+ "ip6Gateway": null,
+ "interfaces": [
+ {
+ "ipAddresses": [
+ {
+ "address": "4.0.16.239.6",
+ "gateway": "4.0.16.239.1",
+ "serviceAddress": true
+ },
+ {
+ "address": "fc01:9400:1000:8::6",
+ "gateway": "fc01:9400:1000:8::1",
+ "serviceAddress": true
+ }
+ ],
+ "maxBandwidth": null,
+ "monitor": true,
+ "mtu": 1500,
+ "name": "eth0"
+ }
+ ],
+ "ipGateway": "4.0.0.0.1",
+ "ipNetmask": "255.255.255.0",
+ "lastUpdated": "2019",
+ "mgmtIpAddress": null,
+ "mgmtIpGateway": null,
+ "mgmtIpNetmask": null,
+ "offlineReason": "none",
+ "physLocation": "",
+ "physLocationId": 0,
+ "profile": "Monitor0",
+ "profileDesc": "nodesc",
+ "profileId": 0,
+ "rack": "",
+ "revalPending": false,
+ "routerHostName": "",
+ "routerPortName": "",
+ "status": "REPORTED",
+ "statusId": 0,
+ "tcpPort": 80,
+ "type": "RASCAL",
+ "typeId": 0,
+ "updPending": false,
+ "xmppId": "",
+ "xmppPasswd": ""
+ }
+]
+'
+
+ cat >$CFG_FILE <<-EOF
+{
+ "trafficMonitor": {
+ "url": "$TM_URI"
+ },
+ "default": {
+ "session": {
+ "timeoutInSecs": 30
+ },
+ "log": {
+ "debug": "stdout",
+ "event": "stdout",
+ "info": "stdout",
+ "error": "stdout",
+ "warning": "stdout"
+ }
+ }
+}
+EOF
+
+ echo "INITIALIZED=1" >>/etc/environment
+}
+
+function wait_for_endpoint() {
+ try=0
+ while curl -Lvsk "$1" 2>&1 1>/dev/null | grep -Eq '< HTTP/[0-9]\.?[0-9]* 200' && [[ $try -lt 5 ]]; do
+ if [[ $try -gt 0 ]]; then
Review comment:
`if [[ $try -gt 0 ]]` will make it branch straight to `echo "Unable to get $1"` and exit. `if [[ $try -lt 5 ]]` should work, although the loop condition would mean the `if` condition is always true.
##########
File path: traffic_monitor/build_tests.sh
##########
@@ -0,0 +1,37 @@
+#!/bin/bash
+
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+export CGO_ENABLED=0
+export GOOS=linux
+
+if [ ! -f "traffic_monitor.rpm" ]; then
+ echo "Unable to find traffic_monitor.rpm."
+ echo "Run './pkg traffic_monitor_build' and copy TM rpm to this directory."
+ exit 1
+fi
+
+cd tools/testto
+rm testto
+go build
+cd - > /dev/null
+
+cd tools/testcaches
+rm testcaches
+go build
+cd - > /dev/null
+
+cd tests/_integration
+rm traffic_monitor_integration_test
+go test -c -o traffic_monitor_integration_test
+cd - > /dev/null
Review comment:
The `traffic_monitor_integration_test` binary should be built as a step in `/traffic_monitor/tests/_integration/Dockerfile`.
##########
File path: traffic_monitor/Dockerfile_run.sh
##########
@@ -0,0 +1,81 @@
+#!/usr/bin/env bash
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+# The following environment variables must be set (ordinarily by `docker run -e` arguments):
+# TO_URI
+# TO_USER
+# TO_PASS
+# CDN
+
+# Check that env vars are set
+envvars=( TO_URI TO_USER TO_PASS CDN PORT )
+for v in $envvars
Review comment:
`$envvars` here needs to be `${envvars[@]}`, otherwise `v` will only iterate over the first element.
##########
File path: traffic_monitor/tests/_integration/README.md
##########
@@ -0,0 +1,13 @@
+# Traffic Monitor Integration Test Framework
+
+## Building
+
+From the `trafficcontrol` run `./pkg -v traffic_monitor_build` and copy the traffic_monitor rpm to the `traffic_monitor` directory.
+
+From the `trafficcontrol/traffic_monitor` directory run `build_tests.sh`
+
+## Running
+
+From the `trafficcontrol/traffic_monitor` directory run:
+
+`sudo docker-compose -p tmi --project-directory . -f tests/_integration/docker-compose.yml run tmintegrationtest`
Review comment:
No need to include `sudo`
##########
File path: traffic_monitor/variables.env
##########
@@ -0,0 +1,34 @@
+# Licensed under the Apache License, Version 2.0 (the "License");
Review comment:
`variables.env` is for use by the TM integration tests specifically and should be in the `/traffic_monitor/tests/_integration` directory
##########
File path: traffic_monitor/tests/_integration/traffic_monitor_test.go
##########
@@ -0,0 +1,96 @@
+package _integration
+
+/*
+
+ Licensed under the Apache License, Version 2.0 (the "License");
+ you may not use this file except in compliance with the License.
+ You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+ Unless required by applicable law or agreed to in writing, software
+ distributed under the License is distributed on an "AS IS" BASIS,
+ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ See the License for the specific language governing permissions and
+ limitations under the License.
+*/
+
+import (
+ "flag"
+ "fmt"
+ "net/http"
+ "os"
+ "strings"
+ "testing"
+ "time"
+
+ "github.com/apache/trafficcontrol/lib/go-log"
+ "github.com/apache/trafficcontrol/traffic_monitor/tests/_integration/config"
+ "github.com/apache/trafficcontrol/traffic_monitor/tmclient"
+)
+
+var Config config.Config
+var TMClient *tmclient.TMClient
+
+func TestMain(m *testing.M) {
+ var err error
+ configFileName := flag.String("cfg", "traffic-monitor-test.conf", "The config file path")
+ flag.Parse()
+
+ if Config, err = config.LoadConfig(*configFileName); err != nil {
+ fmt.Printf("Error Loading Config %v %v\n", Config, err)
+ os.Exit(1)
+ }
+
+ if err = log.InitCfg(Config); err != nil {
+ fmt.Printf("Error initializing loggers: %v\n", err)
+ os.Exit(1)
+ }
+
+ log.Infof(`Using Config values:
+ TM Config File: %s
+ TM URL: %s
+ TM Session Timeout: %d\n`,
+ *configFileName, Config.TrafficMonitor.URL, Config.Default.Session.TimeoutInSecs)
+
+ tmReqTimeout := time.Second * time.Duration(Config.Default.Session.TimeoutInSecs)
+
+ monitorWaitSpan := 30 * time.Second // TODO make configurable?
+
+ if !WaitForMonitor(Config.TrafficMonitor.URL, monitorWaitSpan) {
+ fmt.Printf("\nError communicating with Monitor '%v' - didn't return a 200 OK in %v\n",
+ Config.TrafficMonitor.URL, monitorWaitSpan)
+ os.Exit(1)
+ }
Review comment:
Without increasing `monitorWaitSpan`, it times out for me:
```
DEBUG traffic_monitor_integration starting
Error communicating with Monitor 'http://trafficmonitor' - didn't return a 200 OK in 30s
ERROR: 1
```
##########
File path: traffic_monitor/tests/_integration/README.md
##########
@@ -0,0 +1,13 @@
+# Traffic Monitor Integration Test Framework
+
+## Building
+
+From the `trafficcontrol` run `./pkg -v traffic_monitor_build` and copy the traffic_monitor rpm to the `traffic_monitor` directory.
+
+From the `trafficcontrol/traffic_monitor` directory run `build_tests.sh`
+
+## Running
+
+From the `trafficcontrol/traffic_monitor` directory run:
+
+`sudo docker-compose -p tmi --project-directory . -f tests/_integration/docker-compose.yml run tmintegrationtest`
Review comment:
The build contexts in `docker-compose.yml` should be rewritten so the user does not need to specify either `--project-directory .` or `-f tests/_integration/docker-compose.yml`. A user should be able to run the TM integration tests with just
```shell
docker-compose run tmintegrationtest
```
##########
File path: traffic_monitor/Dockerfile
##########
@@ -0,0 +1,32 @@
+# Licensed to the Apache Software Foundation (ASF) under one
Review comment:
Any reason this Dockerfile is in the main `traffic_monitor` directory?
##########
File path: traffic_monitor/build_tests.sh
##########
@@ -0,0 +1,37 @@
+#!/bin/bash
+
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+export CGO_ENABLED=0
+export GOOS=linux
+
+if [ ! -f "traffic_monitor.rpm" ]; then
+ echo "Unable to find traffic_monitor.rpm."
+ echo "Run './pkg traffic_monitor_build' and copy TM rpm to this directory."
+ exit 1
Review comment:
Can building and placing `traffic_monitor.rpm` be scripted?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org