You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@trafficcontrol.apache.org by zr...@apache.org on 2023/04/04 15:27:32 UTC
[trafficcontrol] branch master updated: added anycast traffic routing ability, and lastpoll time for v6 (#7302)
This is an automated email from the ASF dual-hosted git repository.
zrhoffman pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/trafficcontrol.git
The following commit(s) were added to refs/heads/master by this push:
new 0e2b42f924 added anycast traffic routing ability, and lastpoll time for v6 (#7302)
0e2b42f924 is described below
commit 0e2b42f9243800308266da1169e5e9ea5e00fd68
Author: serDrem <se...@gmail.com>
AuthorDate: Tue Apr 4 09:27:24 2023 -0600
added anycast traffic routing ability, and lastpoll time for v6 (#7302)
* added anycast traffic routing ability, and lastpoll time for v6
updated docs
refactor datareq/crstate.go
updated changelog
update results with hostname from via header, if exists
* documentation and minor regex, and logic updates
* reverting change
* performance and other refactor
* putting includes in order
* renaming ServerPartners to SameIpServers
* fixed documentation grammar and formatting
* fixed documentation formatting
* redability, naming scheme, and commonality improvements, added test case
* removing datareq/crstate_test.go
* update test cases, removed previously intoroduced bug, and added test accessor
* added enums for threshold messages and an option to turn off anycast adjustments by default
* updated docs and made types easier to use
* optimizations and safery for crStates
---------
Co-authored-by: Dremin, Sergey <Se...@comcast.com>
---
CHANGELOG.md | 1 +
docs/source/admin/traffic_monitor.rst | 1 +
docs/source/glossary.rst | 5 ++
docs/source/overview/traffic_monitor.rst | 3 +
lib/go-rfc/http.go | 1 +
lib/go-tc/crstates.go | 1 +
traffic_monitor/cache/astats.go | 18 +++++
traffic_monitor/cache/astats_csv.go | 3 +
traffic_monitor/cache/cache.go | 8 ++
traffic_monitor/cache/data.go | 4 +
traffic_monitor/cache/stats_over_http.go | 14 +++-
traffic_monitor/datareq/crstate.go | 77 +++++++++++++++---
traffic_monitor/datareq/crstate_test.go | 135 +++++++++++++++++++++++++++++++
traffic_monitor/datareq/datareq.go | 17 +++-
traffic_monitor/health/cache.go | 32 ++++++--
traffic_monitor/manager/manager.go | 2 +
traffic_monitor/manager/statecombiner.go | 2 +-
traffic_monitor/todata/todata.go | 56 +++++++++++++
traffic_monitor/todata/todata_test.go | 68 ++++++++++++++++
19 files changed, 426 insertions(+), 22 deletions(-)
diff --git a/CHANGELOG.md b/CHANGELOG.md
index e02c85e7e5..fcac2c78fb 100644
--- a/CHANGELOG.md
+++ b/CHANGELOG.md
@@ -5,6 +5,7 @@ The format is based on [Keep a Changelog](http://keepachangelog.com/en/1.0.0/).
## [unreleased]
### Added
+- [#7290](https://github.com/apache/trafficcontrol/pull/7302) *Traffic Monitor* Update TM results with hostname from via header, syncronize health on caches with same service address
- [#7291](https://github.com/apache/trafficcontrol/pull/7291) *Traffic Ops* Extended Layered Profile feature to aggregate parameters for all server profiles.
- [#7314](https://github.com/apache/trafficcontrol/pull/7314) *Traffic Portal* Added capability feature to Delivery Service Form (HTTP, DNS).
- [#7295](https://github.com/apache/trafficcontrol/pull/7295) *Traffic Portal* Added description and priority order for Layered Profile on server form.
diff --git a/docs/source/admin/traffic_monitor.rst b/docs/source/admin/traffic_monitor.rst
index cd370c35e9..33ca244a45 100644
--- a/docs/source/admin/traffic_monitor.rst
+++ b/docs/source/admin/traffic_monitor.rst
@@ -79,6 +79,7 @@ traffic_monitor.cfg
- ``health.polling.interval``
- ``peers.polling.interval``
- ``heartbeat.polling.interval``
+- ``tm.sameipservers.control`` - When set to true, performs an AND operation on the availability statuses of servers with same ip. Any unavailable server(s) with same ip as other server(s) will cause the other server(s) to be set to unavailable.
Upon receiving this configuration, Traffic Monitor begins polling :term:`cache server` s. Once every :term:`cache server` has been polled, :ref:`health-proto` state is available via RESTful JSON endpoints and a web browser UI.
diff --git a/docs/source/glossary.rst b/docs/source/glossary.rst
index c7bd96a450..3d0607c6e2 100644
--- a/docs/source/glossary.rst
+++ b/docs/source/glossary.rst
@@ -41,6 +41,11 @@ Glossary
- :term:`forward proxy`: Used by Traffic Control for Mid-tier :dfn:`cache servers`.
- transparent proxy: These are not used by Traffic Control. If you are interested you can learn more about transparent proxies on `wikipedia <http://en.wikipedia.org/wiki/Proxy_server#Transparent_proxy>`_.
+ anycast group
+ Anycast group
+ Anycast Group
+ A group of caching HTTP proxy servers that have the same service address that is equal cost multi path routed at the last hop
+
Cache Group
Cache Groups
A group of caching HTTP proxy servers that together create a combined larger cache using consistent hashing. Traffic Router treats all servers in a :dfn:`Cache Group` as though they are in the same geographic location, though they are in fact only in the same general area. A :dfn:`Cache Group` has one single set of geographical coordinates even if the :term:`cache servers` that make up the :dfn:`Cache Group` are actually in :term:`Physical Locations`. The :term:`cache servers` in a :d [...]
diff --git a/docs/source/overview/traffic_monitor.rst b/docs/source/overview/traffic_monitor.rst
index 90311e6f84..aaff86bf25 100644
--- a/docs/source/overview/traffic_monitor.rst
+++ b/docs/source/overview/traffic_monitor.rst
@@ -56,3 +56,6 @@ Protocol Engagement
-------------------
Short polling intervals of both the :term:`cache servers` and Traffic Monitor peers help to reduce customer impact of outages. It is not uncommon for a :term:`cache server` to be marked unavailable by Traffic Monitor - in fact, it is business as usual for many CDNs. Should a widely requested video asset cause a single :term:`cache server` to get close to its interface capacity, the Health Protocol will "kick in," and Traffic Monitor marks the :term:`cache server` as unavailable. New clie [...]
+Handling Cache Servers with Same Service IP
+-------------------------------------------
+Traffic Monitor is able to direct Traffic Router to route traffic to groups of :term:`cache servers` with the same service IP address, but not to an individual :term:`cache server` within that group. To monitor :term:`cache servers` within that group, HTTP :mailheader:`Keep-Alive` header is omitted with the ``health.polling.keepalive`` and ``stat.polling.keepalive`` Traffic Monitor :term:`profile` :term:`parameters` so that the Traffic Monitor source port changes for those connections an [...]
\ No newline at end of file
diff --git a/lib/go-rfc/http.go b/lib/go-rfc/http.go
index db452a259b..ffa2876e9a 100644
--- a/lib/go-rfc/http.go
+++ b/lib/go-rfc/http.go
@@ -43,6 +43,7 @@ const (
Age = "Age" // RFC7234§5.1
Location = "Location" // RFC7231§7.1.2
Authorization = "Authorization" // RFC7235§4.2
+ Via = "Via" // RFC3261§8.1.1.7
Cookie = "Cookie" // RFC7873
)
diff --git a/lib/go-tc/crstates.go b/lib/go-tc/crstates.go
index a7d99bffbc..4154b9f363 100644
--- a/lib/go-tc/crstates.go
+++ b/lib/go-tc/crstates.go
@@ -44,6 +44,7 @@ type IsAvailable struct {
DirectlyPolled bool `json:"-"`
Status string `json:"status"`
LastPoll time.Time `json:"lastPoll"`
+ LastPollV6 time.Time `json:"lastPollV6"`
}
// NewCRStates creates a new CR states object, initializing pointer members.
diff --git a/traffic_monitor/cache/astats.go b/traffic_monitor/cache/astats.go
index 00981b3c10..b684c16b24 100644
--- a/traffic_monitor/cache/astats.go
+++ b/traffic_monitor/cache/astats.go
@@ -33,17 +33,21 @@ import (
"errors"
"fmt"
"io"
+ "regexp"
"strings"
"github.com/apache/trafficcontrol/lib/go-log"
+ "github.com/apache/trafficcontrol/lib/go-rfc"
"github.com/apache/trafficcontrol/traffic_monitor/dsdata"
"github.com/apache/trafficcontrol/traffic_monitor/poller"
"github.com/apache/trafficcontrol/traffic_monitor/todata"
+
jsoniter "github.com/json-iterator/go"
)
func init() {
registerDecoder("astats", astatsParse, astatsPrecompute)
+ hostnameRegex = regexp.MustCompile(`(?:http|https)/\d+\.\d+ ([A-Za-z0-9\-]{0,61})`)
}
// AstatsSystem represents fixed system stats returned from the
@@ -116,8 +120,22 @@ func astatsParse(cacheName string, rdr io.Reader, pollCTX interface{}) (Statisti
astats.Ats["system.proc.loadavg"] = astats.System.ProcLoadavg
astats.Ats["system.proc.net.dev"] = astats.System.ProcNetDev
+ via := ctx.HTTPHeader.Get(rfc.Via)
+ if via != "" {
+ viaRegexSubmatch := hostnameRegex.FindStringSubmatch(via)
+ if len(viaRegexSubmatch) > 0 {
+ astats.Ats[rfc.Via] = viaRegexSubmatch[1]
+ }
+ }
return stats, astats.Ats, nil
} else if ctype == "text/csv" {
+ via := ctx.HTTPHeader.Get(rfc.Via)
+ if via != "" {
+ viaRegexSubmatch := hostnameRegex.FindStringSubmatch(via)
+ if len(viaRegexSubmatch) > 0 {
+ cacheName = viaRegexSubmatch[1]
+ }
+ }
return astatsCsvParseCsv(cacheName, rdr)
} else {
return stats, nil, fmt.Errorf("stats Content-Type (%s) can not be parsed by astats", ctype)
diff --git a/traffic_monitor/cache/astats_csv.go b/traffic_monitor/cache/astats_csv.go
index d36495db52..10317a82dd 100644
--- a/traffic_monitor/cache/astats_csv.go
+++ b/traffic_monitor/cache/astats_csv.go
@@ -28,6 +28,7 @@ import (
"strings"
"github.com/apache/trafficcontrol/lib/go-log"
+ "github.com/apache/trafficcontrol/lib/go-rfc"
)
type astatsDataCsv struct {
@@ -128,5 +129,7 @@ func astatsCsvParseCsv(cacheName string, data io.Reader) (Statistics, map[string
return stats, nil, fmt.Errorf("cache '%s' had no interfaces", cacheName)
}
+ statMap[rfc.Via] = cacheName
+
return stats, statMap, nil
}
diff --git a/traffic_monitor/cache/cache.go b/traffic_monitor/cache/cache.go
index 211b390718..339c4a5f5c 100644
--- a/traffic_monitor/cache/cache.go
+++ b/traffic_monitor/cache/cache.go
@@ -20,10 +20,13 @@ package cache
*/
import (
+ "fmt"
"io"
+ "regexp"
"time"
"github.com/apache/trafficcontrol/lib/go-log"
+ "github.com/apache/trafficcontrol/lib/go-rfc"
"github.com/apache/trafficcontrol/lib/go-tc"
"github.com/apache/trafficcontrol/traffic_monitor/todata"
)
@@ -34,6 +37,8 @@ type Handler struct {
ToData *todata.TODataThreadsafe
}
+var hostnameRegex *regexp.Regexp
+
func (h Handler) ResultChan() <-chan Result {
return h.resultChan
}
@@ -301,6 +306,9 @@ func (handler Handler) Handle(id string, rdr io.Reader, format string, reqTime t
handler.resultChan <- result
return
}
+ if value, ok := miscStats[rfc.Via]; ok {
+ result.ID = fmt.Sprintf("%v", value)
+ }
result.Statistics = stats
result.Miscellaneous = miscStats
diff --git a/traffic_monitor/cache/data.go b/traffic_monitor/cache/data.go
index 0659e06905..3afd06e153 100644
--- a/traffic_monitor/cache/data.go
+++ b/traffic_monitor/cache/data.go
@@ -64,6 +64,10 @@ type AvailableStatus struct {
UnavailableStat string
// Poller is the name of the poller which set this availability status.
Poller string
+ // Time of last poll
+ LastPoll time.Time
+ // Time of v6 last poll
+ LastPollV6 time.Time
}
// CacheAvailableStatuses is the available status of each cache.
diff --git a/traffic_monitor/cache/stats_over_http.go b/traffic_monitor/cache/stats_over_http.go
index 7926d42f62..3e1498d242 100644
--- a/traffic_monitor/cache/stats_over_http.go
+++ b/traffic_monitor/cache/stats_over_http.go
@@ -25,10 +25,12 @@ import (
"fmt"
"io"
"math"
+ "regexp"
"strconv"
"strings"
"github.com/apache/trafficcontrol/lib/go-log"
+ "github.com/apache/trafficcontrol/lib/go-rfc"
"github.com/apache/trafficcontrol/traffic_monitor/poller"
"github.com/apache/trafficcontrol/traffic_monitor/todata"
@@ -52,8 +54,8 @@ import (
const LOADAVG_SHIFT = 65536
func init() {
- // AddStatsType("stats_over_http", statsParse, statsPrecompute)
registerDecoder("stats_over_http", statsOverHTTPParse, statsOverHTTPPrecompute)
+ hostnameRegex = regexp.MustCompile(`(?:http|https)/\d+\.\d+ ([A-Za-z0-9\-]{0,61})`)
}
type stats_over_httpData struct {
@@ -72,6 +74,14 @@ func statsOverHTTPParse(cacheName string, data io.Reader, pollCTX interface{}) (
ctx := pollCTX.(*poller.HTTPPollCtx)
+ via := ctx.HTTPHeader.Get(rfc.Via)
+ if via != "" {
+ viaRegexSubmatch := hostnameRegex.FindStringSubmatch(via)
+ if len(viaRegexSubmatch) > 0 {
+ cacheName = viaRegexSubmatch[1]
+ }
+ }
+
ctype := ctx.HTTPHeader.Get("Content-Type")
if ctype == "text/json" || ctype == "text/javascript" || ctype == "application/json" || ctype == "" {
@@ -95,6 +105,8 @@ func statsOverHTTPParse(cacheName string, data io.Reader, pollCTX interface{}) (
statMap := sohData.Global
+ statMap[rfc.Via] = cacheName
+
if stats.Loadavg, err = parseLoadAvg(statMap); err != nil {
return stats, nil, fmt.Errorf("Error parsing loadavg for cache '%s': %v", cacheName, err)
}
diff --git a/traffic_monitor/datareq/crstate.go b/traffic_monitor/datareq/crstate.go
index da1ef42c78..61122a4a93 100644
--- a/traffic_monitor/datareq/crstate.go
+++ b/traffic_monitor/datareq/crstate.go
@@ -23,10 +23,14 @@ import (
"fmt"
"net/http"
"net/url"
+ "strings"
"github.com/apache/trafficcontrol/lib/go-log"
"github.com/apache/trafficcontrol/lib/go-tc"
+ "github.com/apache/trafficcontrol/traffic_monitor/health"
"github.com/apache/trafficcontrol/traffic_monitor/peer"
+ "github.com/apache/trafficcontrol/traffic_monitor/threadsafe"
+ "github.com/apache/trafficcontrol/traffic_monitor/todata"
)
func srvTRState(
@@ -35,11 +39,13 @@ func srvTRState(
combinedStates peer.CRStatesThreadsafe,
peerStates peer.CRStatesPeersThreadsafe,
distributedPollingEnabled bool,
+ toData todata.TODataThreadsafe,
+ monitorConfig threadsafe.TrafficMonitorConfigMap,
) ([]byte, int, error) {
_, raw := params["raw"] // peer polling case
_, local := params["local"] // distributed peer polling case
if raw {
- data, err := srvTRStateSelf(localStates, distributedPollingEnabled)
+ data, err := srvTRStateData(localStates, distributedPollingEnabled, toData, monitorConfig)
return data, http.StatusOK, err
}
@@ -58,19 +64,11 @@ func srvTRState(
}
}
- data, err := srvTRStateDerived(combinedStates, local && distributedPollingEnabled)
+ data, err := srvTRStateData(combinedStates, local && distributedPollingEnabled, toData, monitorConfig)
return data, http.StatusOK, err
}
-func srvTRStateDerived(combinedStates peer.CRStatesThreadsafe, directlyPolledOnly bool) ([]byte, error) {
- if !directlyPolledOnly {
- return tc.CRStatesMarshall(combinedStates.Get())
- }
- unfiltered := combinedStates.Get()
- return tc.CRStatesMarshall(filterDirectlyPolledCaches(unfiltered))
-}
-
func filterDirectlyPolledCaches(crstates tc.CRStates) tc.CRStates {
filtered := tc.CRStates{
Caches: make(map[tc.CacheName]tc.IsAvailable),
@@ -84,10 +82,63 @@ func filterDirectlyPolledCaches(crstates tc.CRStates) tc.CRStates {
return filtered
}
-func srvTRStateSelf(localStates peer.CRStatesThreadsafe, directlyPolledOnly bool) ([]byte, error) {
+func srvTRStateData(localStates peer.CRStatesThreadsafe, directlyPolledOnly bool, toData todata.TODataThreadsafe, monitorConfig threadsafe.TrafficMonitorConfigMap) ([]byte, error) {
+ if val, ok := monitorConfig.Get().Config["tm.sameipservers.control"]; ok && val.(string) == "true" {
+ localStatesC := updateStatusSameIpServers(localStates, toData)
+ if !directlyPolledOnly {
+ return tc.CRStatesMarshall(localStatesC)
+ }
+ return tc.CRStatesMarshall(filterDirectlyPolledCaches(localStatesC))
+ }
if !directlyPolledOnly {
return tc.CRStatesMarshall(localStates.Get())
}
- unfiltered := localStates.Get()
- return tc.CRStatesMarshall(filterDirectlyPolledCaches(unfiltered))
+ return tc.CRStatesMarshall(filterDirectlyPolledCaches(localStates.Get()))
+}
+
+func updateStatusSameIpServers(localStates peer.CRStatesThreadsafe, toData todata.TODataThreadsafe) tc.CRStates {
+ localStatesC := localStates.Get()
+ toDataC := toData.Get()
+
+ for cache, _ := range localStatesC.Caches {
+ if _, ok := toDataC.SameIpServers[cache]; ok {
+ // all servers with same ip must be available if they are in reported state
+ allAvailableV4 := true
+ allAvailableV6 := true
+ allIsAvailable := true
+ for partner, _ := range toDataC.SameIpServers[cache] {
+ if partnerState, ok := localStatesC.Caches[partner]; ok {
+ // a partner host is reported but is marked down for exceeding a threshold
+ // this host also needs to be marked down to divert all traffic for their
+ // common ip
+ if strings.Contains(partnerState.Status, string(tc.CacheStatusReported)) &&
+ strings.Contains(partnerState.Status, health.TooHigh.String()) {
+ if !partnerState.Ipv4Available {
+ allAvailableV4 = false
+ }
+ if !partnerState.Ipv6Available {
+ allAvailableV6 = false
+ }
+ if !partnerState.IsAvailable {
+ allIsAvailable = false
+ }
+ if !allAvailableV4 && !allAvailableV6 && !allIsAvailable {
+ break
+ }
+ }
+ }
+ }
+ newIsAvailable := tc.IsAvailable{}
+ newIsAvailable.DirectlyPolled = localStatesC.Caches[cache].DirectlyPolled
+ newIsAvailable.Status = localStatesC.Caches[cache].Status
+ newIsAvailable.LastPoll = localStatesC.Caches[cache].LastPoll
+ newIsAvailable.LastPollV6 = localStatesC.Caches[cache].LastPollV6
+ newIsAvailable.IsAvailable = localStatesC.Caches[cache].IsAvailable && allIsAvailable
+ newIsAvailable.Ipv4Available = localStatesC.Caches[cache].Ipv4Available && allAvailableV4
+ newIsAvailable.Ipv6Available = localStatesC.Caches[cache].Ipv6Available && allAvailableV6
+
+ localStatesC.Caches[cache] = newIsAvailable
+ }
+ }
+ return localStatesC
}
diff --git a/traffic_monitor/datareq/crstate_test.go b/traffic_monitor/datareq/crstate_test.go
new file mode 100644
index 0000000000..1373774ff5
--- /dev/null
+++ b/traffic_monitor/datareq/crstate_test.go
@@ -0,0 +1,135 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package datareq
+
+import (
+ "github.com/apache/trafficcontrol/lib/go-tc"
+ "github.com/apache/trafficcontrol/traffic_monitor/peer"
+ "github.com/apache/trafficcontrol/traffic_monitor/todata"
+ "github.com/apache/trafficcontrol/traffic_ops/traffic_ops_golang/test"
+ "testing"
+)
+
+func setMockTOData(tod *todata.TOData) {
+ numCaches := 100
+ numDSes := 100
+ numCacheDSes := numDSes / 3
+ numCGs := 20
+
+ types := []tc.CacheType{tc.CacheTypeEdge, tc.CacheTypeEdge, tc.CacheTypeEdge, tc.CacheTypeEdge, tc.CacheTypeEdge, tc.CacheTypeMid}
+
+ caches := []tc.CacheName{}
+ for i := 0; i < numCaches; i++ {
+ caches = append(caches, tc.CacheName(test.RandStr()))
+ }
+
+ dses := []tc.DeliveryServiceName{}
+ for i := 0; i < numDSes; i++ {
+ dses = append(dses, tc.DeliveryServiceName(test.RandStr()))
+ }
+
+ cgs := []tc.CacheGroupName{}
+ for i := 0; i < numCGs; i++ {
+ cgs = append(cgs, tc.CacheGroupName(test.RandStr()))
+ }
+
+ serverDSes := map[tc.CacheName][]tc.DeliveryServiceName{}
+ for _, ca := range caches {
+ for i := 0; i < numCacheDSes; i++ {
+ serverDSes[ca] = append(serverDSes[ca], dses[test.RandIntn(len(dses))])
+ }
+ }
+
+ dsServers := map[tc.DeliveryServiceName][]tc.CacheName{}
+ for server, dses := range serverDSes {
+ for _, ds := range dses {
+ dsServers[ds] = append(dsServers[ds], server)
+ }
+ }
+
+ serverCGs := map[tc.CacheName]tc.CacheGroupName{}
+ for _, cache := range caches {
+ serverCGs[cache] = cgs[test.RandIntn(len(cgs))]
+ }
+
+ serverTypes := map[tc.CacheName]tc.CacheType{}
+ for _, cache := range caches {
+ serverTypes[cache] = types[test.RandIntn(len(types))]
+ }
+
+ tod.DeliveryServiceServers = dsServers
+ tod.ServerDeliveryServices = serverDSes
+ tod.ServerTypes = serverTypes
+ tod.ServerCachegroups = serverCGs
+}
+
+func TestUpdateStatusSameIpServers(t *testing.T) {
+ toDataTS := todata.NewThreadsafe()
+ toData := todata.New()
+ setMockTOData(toData)
+
+ toData.SameIpServers = map[tc.CacheName]map[tc.CacheName]bool{}
+ toData.SameIpServers["server1_ip1_up"] = map[tc.CacheName]bool{}
+ toData.SameIpServers["server1_ip1_up"]["server2_ip1_down"] = true
+ toData.SameIpServers["server2_ip1_down"] = map[tc.CacheName]bool{}
+ toData.SameIpServers["server2_ip1_down"]["server1_ip1_up"] = true
+
+ toData.SameIpServers["server3_ip3_up"] = map[tc.CacheName]bool{}
+ toData.SameIpServers["server3_ip3_up"]["server4_ip3_up"] = true
+ toData.SameIpServers["server4_ip3_up"] = map[tc.CacheName]bool{}
+ toData.SameIpServers["server4_ip3_up"]["server3_ip3_up"] = true
+
+ localStates := peer.NewCRStatesThreadsafe()
+ localStates.AddCache("server1_ip1_up", tc.IsAvailable{IsAvailable: true, Ipv4Available: true, Ipv6Available: true, Status: string(tc.CacheStatusReported)})
+ localStates.AddCache("server2_ip1_down", tc.IsAvailable{IsAvailable: false, Ipv4Available: false, Ipv6Available: false, Status: string(tc.CacheStatusReported) + "too high"})
+ localStates.AddCache("server3_ip3_up", tc.IsAvailable{IsAvailable: true, Ipv4Available: true, Ipv6Available: true, Status: string(tc.CacheStatusReported)})
+ localStates.AddCache("server4_ip3_up", tc.IsAvailable{IsAvailable: true, Ipv4Available: true, Ipv6Available: true, Status: string(tc.CacheStatusReported)})
+ localStates.AddCache("server5_ip5_up", tc.IsAvailable{IsAvailable: true, Ipv4Available: true, Ipv6Available: true, Status: string(tc.CacheStatusReported)})
+
+ toDataTS.SetForTest(*toData)
+
+ localStatesC := updateStatusSameIpServers(localStates, toDataTS)
+
+ if localStatesC.Caches["server1_ip1_up"].IsAvailable == true ||
+ localStatesC.Caches["server1_ip1_up"].Ipv4Available == true ||
+ localStatesC.Caches["server1_ip1_up"].Ipv6Available == true {
+ t.Error("expected server1_ip1_up to be false for IsAvailable Ipv4Available Ipv6Available")
+ }
+ if localStatesC.Caches["server2_ip1_down"].IsAvailable != false ||
+ localStatesC.Caches["server2_ip1_down"].Ipv4Available != false ||
+ localStatesC.Caches["server2_ip1_down"].Ipv6Available != false {
+ t.Error("expected server2_ip1_up to be false for IsAvailable Ipv4Available Ipv6Available")
+ }
+ if localStatesC.Caches["server3_ip3_up"].IsAvailable != true ||
+ localStatesC.Caches["server3_ip3_up"].Ipv4Available != true ||
+ localStatesC.Caches["server3_ip3_up"].Ipv6Available != true {
+ t.Error("expected server3_ip3_up to be true for IsAvailable Ipv4Available Ipv6Available")
+ }
+ if localStatesC.Caches["server4_ip3_up"].IsAvailable != true ||
+ localStatesC.Caches["server4_ip3_up"].Ipv4Available != true ||
+ localStatesC.Caches["server4_ip3_up"].Ipv6Available != true {
+ t.Error("expected server4_ip3_up to be true for IsAvailable Ipv4Available Ipv6Available")
+ }
+ if localStatesC.Caches["server5_ip5_up"].IsAvailable != true ||
+ localStatesC.Caches["server5_ip5_up"].Ipv4Available != true ||
+ localStatesC.Caches["server5_ip5_up"].Ipv6Available != true {
+ t.Error("expected server5_ip5_up to be true for IsAvailable Ipv4Available Ipv6Available")
+ }
+}
diff --git a/traffic_monitor/datareq/datareq.go b/traffic_monitor/datareq/datareq.go
index 42244d65af..b7ba5f1cd7 100644
--- a/traffic_monitor/datareq/datareq.go
+++ b/traffic_monitor/datareq/datareq.go
@@ -79,12 +79,27 @@ func MakeDispatchMap(
}
}
+ // if any server has partners listed in toData.SameIpServers, this is an anycast enabled server
+ // tm can not control traffic to individual anycast enabled servers, since all of them have the same
+ // service IP
+ // It can poll all of them, but will only detect not available status on the last server that becomes
+ // unavailable. Therefore if any of the servers in the anycast group becomes unavailable, that means
+ // that the TM can not get to ANY of the anycast group servers and all of them must be marked unavailable
+ // TM will detect server load and bandwidth limits on all the servers that are responding to polls, and
+ // are serving traffic
+ // If TM detects server load issues, it can only control that by diverting traffic to all the servers in the
+ // anycast group
+ // If TM detects server bandwidth limits, it must divert traffic to all the servers that are responding and not
+ // responding in the anycast server group. The ones that are responding are in ECMP, and have the same bandwidth,
+ // so bandwidth to all must be decreased. The ones that are not responding, can appear to be in reported state,
+ // and must be marked unavailable to control traffic to the remaining servers
+
dispatchMap := map[string]http.HandlerFunc{
"/publish/CrConfig": wrap(WrapAgeErr(errorCount, func() ([]byte, time.Time, error) {
return srvTRConfig(opsConfig, toSession)
}, rfc.ApplicationJSON)),
"/publish/CrStates": wrap(WrapParams(func(params url.Values, path string) ([]byte, int) {
- bytes, statusCode, err := srvTRState(params, localStates, combinedStates, peerStates, distributedPollingEnabled)
+ bytes, statusCode, err := srvTRState(params, localStates, combinedStates, peerStates, distributedPollingEnabled, toData, monitorConfig)
return WrapErrStatusCode(errorCount, path, bytes, statusCode, err)
}, rfc.ApplicationJSON)),
"/publish/CacheStatsNew": wrap(WrapParams(func(params url.Values, path string) ([]byte, int) {
diff --git a/traffic_monitor/health/cache.go b/traffic_monitor/health/cache.go
index 750317025f..94489884db 100644
--- a/traffic_monitor/health/cache.go
+++ b/traffic_monitor/health/cache.go
@@ -46,6 +46,18 @@ const AvailableStr = "available"
// available to serve traffic.
const UnavailableStr = "unavailable"
+type Threshold string
+
+const (
+ NotEqual Threshold = "not equal"
+ TooLow Threshold = "too low"
+ TooHigh Threshold = "too high"
+)
+
+func (t Threshold) String() string {
+ return string(t)
+}
+
// GetVitals Gets the vitals to decide health on in the right format
func GetVitals(newResult *cache.Result, prevResult *cache.Result, mc *tc.TrafficMonitorConfigMap) {
if newResult.Error != nil {
@@ -356,13 +368,21 @@ func CalcAvailability(
availStatus.UnavailableStat = aggUnavailableStat
}
+ if result.UsingIPv4 {
+ availStatus.LastPoll = result.Time
+ availStatus.LastPollV6 = lastStatus.LastPollV6
+ } else {
+ availStatus.LastPoll = lastStatus.LastPoll
+ availStatus.LastPollV6 = result.Time
+ }
localStates.SetCache(tc.CacheName(result.ID), tc.IsAvailable{
IsAvailable: availStatus.ProcessedAvailable,
Ipv4Available: availStatus.Available.IPv4,
Ipv6Available: availStatus.Available.IPv6,
DirectlyPolled: true, // we know this cache was directly polled because otherwise we wouldn't have a cache.Result for it
Status: availStatus.Why,
- LastPoll: result.Time,
+ LastPoll: availStatus.LastPoll,
+ LastPollV6: availStatus.LastPollV6,
})
if available, ok := localStates.GetCache(tc.CacheName(result.ID)); !ok || available.IsAvailable != lastStatus.ProcessedAvailable {
@@ -396,15 +416,15 @@ func CalcAvailability(
func exceedsThresholdMsg(stat string, threshold tc.HealthThreshold, val float64) string {
switch threshold.Comparator {
case "=":
- return fmt.Sprintf("%s not equal (%.2f != %.2f)", stat, val, threshold.Val)
+ return fmt.Sprintf("%s %s (%.2f == %.2f)", stat, NotEqual, val, threshold.Val)
case ">":
- return fmt.Sprintf("%s too low (%.2f < %.2f)", stat, val, threshold.Val)
+ return fmt.Sprintf("%s %s (%.2f < %.2f)", stat, TooLow, val, threshold.Val)
case "<":
- return fmt.Sprintf("%s too high (%.2f > %.2f)", stat, val, threshold.Val)
+ return fmt.Sprintf("%s %s (%.2f > %.2f)", stat, TooHigh, val, threshold.Val)
case ">=":
- return fmt.Sprintf("%s too low (%.2f <= %.2f)", stat, val, threshold.Val)
+ return fmt.Sprintf("%s %s(%.2f <= %.2f)", stat, TooLow, val, threshold.Val)
case "<=":
- return fmt.Sprintf("%s too high (%.2f >= %.2f)", stat, val, threshold.Val)
+ return fmt.Sprintf("%s %s (%.2f >= %.2f)", stat, TooHigh, val, threshold.Val)
default:
return fmt.Sprintf("ERROR: Invalid Threshold: %+v", threshold)
}
diff --git a/traffic_monitor/manager/manager.go b/traffic_monitor/manager/manager.go
index 2911d52b04..2ff954ad84 100644
--- a/traffic_monitor/manager/manager.go
+++ b/traffic_monitor/manager/manager.go
@@ -51,7 +51,9 @@ func Start(opsConfigFile string, cfg config.Config, appData config.StaticAppData
toData := todata.NewThreadsafe()
+ // makes results chan
cacheHealthHandler := cache.NewHandler()
+ // passes results chan to poller
cacheHealthPoller := poller.NewCache(true, cacheHealthHandler, cfg, appData)
cacheStatHandler := cache.NewPrecomputeHandler(toData)
cacheStatPoller := poller.NewCache(false, cacheStatHandler, cfg, appData)
diff --git a/traffic_monitor/manager/statecombiner.go b/traffic_monitor/manager/statecombiner.go
index 3fdf4f0bc8..4a4fe465bd 100644
--- a/traffic_monitor/manager/statecombiner.go
+++ b/traffic_monitor/manager/statecombiner.go
@@ -128,7 +128,7 @@ func combineCacheState(
IPv6Available: ipv6Available})
}
- combinedStates.AddCache(cacheName, tc.IsAvailable{IsAvailable: available, Ipv4Available: ipv4Available, Ipv6Available: ipv6Available, DirectlyPolled: localCacheState.DirectlyPolled, Status: localCacheState.Status, LastPoll: localCacheState.LastPoll})
+ combinedStates.AddCache(cacheName, tc.IsAvailable{IsAvailable: available, Ipv4Available: ipv4Available, Ipv6Available: ipv6Available, DirectlyPolled: localCacheState.DirectlyPolled, Status: localCacheState.Status, LastPoll: localCacheState.LastPoll, LastPollV6: localCacheState.LastPollV6})
}
func combineDSState(
diff --git a/traffic_monitor/todata/todata.go b/traffic_monitor/todata/todata.go
index 8991e5b2d2..dce3aaf14c 100644
--- a/traffic_monitor/todata/todata.go
+++ b/traffic_monitor/todata/todata.go
@@ -78,6 +78,7 @@ type TOData struct {
ServerCachegroups map[tc.CacheName]tc.CacheGroupName
ServerDeliveryServices map[tc.CacheName][]tc.DeliveryServiceName
ServerTypes map[tc.CacheName]tc.CacheType
+ SameIpServers map[tc.CacheName]map[tc.CacheName]bool
}
// New returns a new empty TOData object, initializing pointer members.
@@ -89,6 +90,7 @@ func New() *TOData {
DeliveryServiceTypes: map[tc.DeliveryServiceName]tc.DSTypeCategory{},
DeliveryServiceRegexes: NewRegexes(),
ServerCachegroups: map[tc.CacheName]tc.CacheGroupName{},
+ SameIpServers: map[tc.CacheName]map[tc.CacheName]bool{},
}
}
@@ -118,6 +120,12 @@ func (d TODataThreadsafe) set(newTOData TOData) {
d.m.Unlock()
}
+func (d TODataThreadsafe) SetForTest(newTOData TOData) {
+ d.m.Lock()
+ *d.toData = newTOData
+ d.m.Unlock()
+}
+
// CRConfig is the CrConfig data needed by TOData. Note this is not all data in the CRConfig.
// TODO change strings to type?
type CRConfig struct {
@@ -179,6 +187,12 @@ func (d TODataThreadsafe) Update(to towrap.TrafficOpsSessionThreadsafe, cdn stri
return fmt.Errorf("getting server types from monitoring config: %v", err)
}
+ if val, ok := mc.Config["tm.sameipservers.control"]; ok && val.(string) == "true" {
+ newTOData.SameIpServers = getSameIPServers(mc)
+ } else {
+ newTOData.SameIpServers = make(map[tc.CacheName]map[tc.CacheName]bool)
+ }
+
d.set(newTOData)
return nil
}
@@ -342,6 +356,48 @@ func getServerTypes(mc tc.TrafficMonitorConfigMap) (map[tc.CacheName]tc.CacheTyp
return serverTypes, nil
}
+// getSameIPServers gets the caches that have the same VIP
+func getSameIPServers(mc tc.TrafficMonitorConfigMap) map[tc.CacheName]map[tc.CacheName]bool {
+ sameIPServers := map[tc.CacheName]map[tc.CacheName]bool{}
+
+ // get service addresses
+ serviceAddress := map[string][]string{}
+ for server, serverData := range mc.TrafficServer {
+ for _, intf := range serverData.Interfaces {
+ for _, addr := range intf.IPAddresses {
+ if addr.ServiceAddress {
+ if _, ok := serviceAddress[addr.Address]; !ok {
+ serviceAddress[addr.Address] = []string{}
+ }
+ serviceAddress[addr.Address] = append(serviceAddress[addr.Address], server)
+ }
+ }
+ }
+ }
+
+ for server, serverData := range mc.TrafficServer {
+ for _, intf := range serverData.Interfaces {
+ for _, addr := range intf.IPAddresses {
+ if addr.ServiceAddress {
+ // if service addresses belongs to more than one server
+ if len(serviceAddress[addr.Address]) > 1 {
+ if _, ok := sameIPServers[tc.CacheName(server)]; !ok {
+ sameIPServers[tc.CacheName(server)] = map[tc.CacheName]bool{}
+ }
+ for _, partner := range serviceAddress[addr.Address] {
+ if partner != server {
+ sameIPServers[tc.CacheName(server)][tc.CacheName(partner)] = true
+ }
+ }
+ }
+ }
+ }
+ }
+ }
+
+ return sameIPServers
+}
+
// canUseMonitorConfig returns true if we can prefer monitor config data to crconfig data.
func canUseMonitorConfig(mc tc.TrafficMonitorConfigMap) bool {
for _, dsData := range mc.DeliveryService {
diff --git a/traffic_monitor/todata/todata_test.go b/traffic_monitor/todata/todata_test.go
index 43e354316a..c222d7dac8 100644
--- a/traffic_monitor/todata/todata_test.go
+++ b/traffic_monitor/todata/todata_test.go
@@ -142,3 +142,71 @@ func TestGetDeliveryServiceServersWithNonTopologyBasedDeliveryService(t *testing
t.Fatalf("getDeliveryServiceServers with non-topology-based delivery service expected: %+v actual: %+v", expectedNonTopologiesTOData, nonTopologiesTOData)
}
}
+
+func TestGetSameIPServers(t *testing.T) {
+ mc := tc.TrafficMonitorConfigMap{TrafficServer: make(map[string]tc.TrafficServer)}
+ mc.TrafficServer["server1_ip1"] = tc.TrafficServer{
+ Interfaces: []tc.ServerInterfaceInfo{
+ {
+ IPAddresses: []tc.ServerIPAddress{
+ {Address: "10.0.0.1", ServiceAddress: true},
+ },
+ },
+ },
+ }
+ mc.TrafficServer["server2_ip1"] = tc.TrafficServer{
+ Interfaces: []tc.ServerInterfaceInfo{
+ {
+ IPAddresses: []tc.ServerIPAddress{
+ {Address: "10.0.0.1", ServiceAddress: true},
+ },
+ },
+ },
+ }
+ mc.TrafficServer["server4_ip1_no_service"] = tc.TrafficServer{
+ Interfaces: []tc.ServerInterfaceInfo{
+ {
+ IPAddresses: []tc.ServerIPAddress{
+ {Address: "10.0.0.1"},
+ },
+ },
+ },
+ }
+ mc.TrafficServer["server3_ip3"] = tc.TrafficServer{
+ Interfaces: []tc.ServerInterfaceInfo{
+ {
+ IPAddresses: []tc.ServerIPAddress{
+ {Address: "10.0.0.3", ServiceAddress: true},
+ },
+ },
+ },
+ }
+ sameIpServers := getSameIPServers(mc)
+ if _, ok := sameIpServers[("server1_ip1")]; !ok {
+ t.Fatal("getSameIPServers expected to find server1_ip1")
+ }
+ if _, ok := sameIpServers["server1_ip1"]["server2_ip1"]; !ok {
+ t.Fatal("getSameIPServers expected to find server1_ip1 to have same ip as server2_ip1")
+ }
+ if _, ok := sameIpServers["server2_ip1"]; !ok {
+ t.Fatal("getSameIPServers expected to find server2_ip1")
+ }
+ if _, ok := sameIpServers["server1_ip1"]["server2_ip1"]; !ok {
+ t.Fatal("getSameIPServers expected to find server2_ip1 to have same ip as server1_ip1")
+ }
+ if _, ok := sameIpServers["server1_ip1"]["server4_ip1_no_service"]; ok {
+ t.Fatal("getSameIPServers expected to find server1_ip1 not to have same ip as server4_ip1_no_service")
+ }
+ if _, ok := sameIpServers["server3_ip3"]; ok {
+ t.Fatal("getSameIPServers expected to not find server3_ip3")
+ }
+
+ expectedSameIpServers := map[string]map[string]bool{}
+ expectedSameIpServers["server1_ip1"] = map[string]bool{"server2_ip1": true}
+ expectedSameIpServers["server2_ip1"] = map[string]bool{"server1_ip1": true}
+
+ if !reflect.DeepEqual(expectedSameIpServers, sameIpServers) {
+
+ }
+
+}