You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@trafficcontrol.apache.org by Nir Sopher <ni...@qwilt.com> on 2017/01/12 09:20:14 UTC
Question regarding TC integration
Hello,
One of our team members (Adan Alper, CCed) is currently trying to connect a
non ATS cache to Traffic Control.
In order to be able to do this, we tried to figure out what is the bare
minimum information that needs to be sent by the cache to TC in order for
it to consider the cache as active.
Currently we found the following:
1. Monitoring file (_astat) is read from the traffic monitor ones
every few seconds. In this file we must provide the following data:
a. availableBandwidthInKbps
b. loadavg
2. Configuration pull – The cache should pull configuration from his
queue in traffic OPS once every 15 minutes and suppose to signal traffic
ops once it was read and applied.
Our questions are:
1. Are these the only information pieces (bare minimum) that we need
to provide or are there any more?
2. Regarding the configuration read signaling. what exactly does that
cache needs to send to traffic ops for it to acknowledge that the
configuration was read and applied? Is this a must for basic functionality?
Thanks,
Nir
Re: Question regarding TC integration
Posted by Nir Sopher <ni...@qwilt.com>.
Thank you Eric,
Indeed, it seems that these 2 fields derive the following list of fields
from astat: "system.proc.loadavg", "system.proc.net.dev",
"system.inf.speed".
Nir
On Thu, Jan 12, 2017 at 2:49 PM, Eric Friedrich (efriedri) <
efriedri@cisco.com> wrote:
> Hey Nir, Adan-
> I don’t think #2 configuration pull is actually required. The cache must
> be configured in Traffic Ops, so Traffic Monitor can learn its hostname and
> capacity. Other than that it should just be meeting the astats criteria.
>
> I think you might need interface speed and actual used bandwidth in the
> astats (rather than available bandwidth in Kbps). I can’t remember if the
> availableBandwidth is computed in astats or in TM.
>
> —Eric
>
>
> > On Jan 12, 2017, at 4:20 AM, Nir Sopher <ni...@qwilt.com> wrote:
> >
> > Hello,
> >
> >
> >
> > One of our team members (Adan Alper, CCed) is currently trying to
> connect a
> > non ATS cache to Traffic Control.
> >
> > In order to be able to do this, we tried to figure out what is the bare
> > minimum information that needs to be sent by the cache to TC in order for
> > it to consider the cache as active.
> >
> > Currently we found the following:
> >
> > 1. Monitoring file (_astat) is read from the traffic monitor ones
> > every few seconds. In this file we must provide the following data:
> >
> > a. availableBandwidthInKbps
> >
> > b. loadavg
> >
> > 2. Configuration pull – The cache should pull configuration from
> his
> > queue in traffic OPS once every 15 minutes and suppose to signal traffic
> > ops once it was read and applied.
> >
> >
> >
> > Our questions are:
> >
> > 1. Are these the only information pieces (bare minimum) that we
> need
> > to provide or are there any more?
> >
> > 2. Regarding the configuration read signaling. what exactly does
> that
> > cache needs to send to traffic ops for it to acknowledge that the
> > configuration was read and applied? Is this a must for basic
> functionality?
> >
> >
> >
> > Thanks,
> >
> > Nir
>
>
Re: Question regarding TC integration
Posted by Nir Sopher <ni...@qwilt.com>.
Thank you both for the answers and elaboration.
It helps:)
Nir
On Thu, Jan 12, 2017 at 6:40 PM, Robert Butts <ro...@gmail.com>
wrote:
> Ok, I don't know how much you know, so, sorry if I'm telling you things you
> already know, but here goes.
>
> > currently trying to connect a non ATS cache to Traffic Control.
>
> So, Traffic Control involves configuring caches, polling those caches to
> determine if they're 'healthy', and routing requests to them.
>
> > what is the bare minimum information that needs to be sent by the cache
> to TC in order for it to consider the cache as active.
>
> I guess there are multiple levels of 'integrating' the cache with the CDN.
>
> For the "absolute minimum", you could simply enter the cache as a Server in
> Traffic Ops, and set it as Online (and snapshot CRConfig). That will make
> Traffic Router send it requests, with no regard for whether it's actually
> 'healthy' (you'd never want to do this in a production system).
>
> One step beyond that, is setting it Reported, and making Traffic Monitor
> capable of polling it. Which is what it sounds like you're trying to do.
>
> The polling URL doesn't actually have to be "/_astats", you can put
> anything in the cache profile's `health.polling.url` parameter, though it
> does expect the param to contain `${hostname}` and `${interface_name}`. The
> JSON returned from the cache (or a proxy representing it) is pretty
> inflexible, though. That JSON needs to be of the form ```
> { "ats": {
> "plugin.remap_stats.delivery-service-one-fqdn.in_bytes": 42000,
> "plugin.remap_stats.delivery-service-one-fqdn.out_bytes": 42001,
> "plugin.remap_stats.delivery-service-two-fqdn.in_bytes": 42002,
> ⋮
> }
> "system": {
> "inf.name": "eth0",
> "inf.speed": 10000,
> "proc.net.dev": "bond0: 12345 6789012314 0 0 0 0
> 0
> 1234 12345678 12345678 0 0 0 0 0 0",
> "proc.loadavg": "1.1 1.2 1.3 1/816 13630"
> }
>
> ```
>
> If `system` isn't obvious, `proc.net.dev` is the value of Linux
> /proc/net/dev for that interface, `proc.loadavg` is /proc/loadavg, `
> inf.name`
> is the name of the network interface the cache will be serving content on,
> and `inf.speed` is the interface speed of the cache's network card, e.g.
> from `ethtool eth0 | grep -i speed`.
>
> At minimum, there needs to be a `plugin.remap_stats.` key, for every
> delivery service assigned to that cache, with `.in_bytes` and `.out_bytes`.
> If keys also exist for .status_2xx, .status_3xx, .status_4xx, .status_5xx
> Traffic Monitor will aggregate and serve those stats, but I _think_ it will
> work without them. Traffic Monitor will also aggregate and serve other
> stats in the "ats" JSON key, if you want them for Traffic Stats or other
> analytics, but they shouldn't be necessary for the monitoring itself.
>
> Then, Traffic Monitor should mark that cache as healthy, based on the
> cache's profile's `health.threshold.loadavg` and
> `health.threshold.queryTime` parameters, and Traffic Router will get the
> Monitor's health data and route requests to that cache accordingly.
>
> That's all I can think, but I'm almost certainly missing something. Making
> a non-ATS cache work with Traffic Control won't be an easy task. We'd
> definitely like Traffic Control to be generic and work with any cache, but
> presently, everything has been designed very ATS-specific. But we'd
> certainly welcome pull requests toward making it generic!
>
>
> Once you have health monitoring and routing working, you'll want to make
> Traffic Control able to configure the cache. You probably don't want to
> worry about that too much until monitoring works, but just the high level
> overview:
>
> Right now, Traffic Ops generates config files for each cache, and we use a
> script (
> https://github.com/apache/incubator-trafficcontrol/blob/
> master/traffic_ops/bin/traffic_ops_ort.pl)
> on a cron job on each cache, to pull updates and new config files from
> Traffic Ops. So, you'll need to write that cron-job-script for the cache
> you're using, and write a script/app/service to generate the configs. I'd
> encourage you to use the Traffic Ops Client (
> https://github.com/apache/incubator-trafficcontrol/tree/
> master/traffic_ops/client)
> for the config generator. If you're comfortable with Go, it's the easiest
> way to work with the Traffic Ops API.
>
> You'll also have to generate the config files yourself. There's a good
> chance not all the data you need to generate those configs is available in
> the Traffic Ops API. In which case, you'll have to make Pull Requests
> adding those APIs. I did a prototype standalone service/app config
> generator last month (
> https://github.com/rob05c/incubator-trafficcontrol/tree/
> atsconfig/traffic_ops/experimental/ats_config),
> and the big APIs I found missing were Jobs, and Delivery Service Regexes.
> Some of the data, like DS regexes, you can get from the CRConfig
> (/CRConfig-Snapshots/cdn-name/CRConfig.json). Though the CRConfig endpoint
> is huge and slow, adding the API endpoints you need would be better.
>
> > what exactly does that cache needs to send to traffic ops for it to
> acknowledge that the configuration was read and applied?
>
> Once your script has updated the cache's config, you can notify Traffic Ops
> (which will clear the clock icon in the Health -> Server Checks grid) by
> writing to `https://traffic-ops.example.net/update/cache-hostname`. See
> https://github.com/apache/incubator-trafficcontrol/blob/
> master/traffic_ops/bin/traffic_ops_ort.pl#L654
>
> Again, sorry if I'm telling you things you already know. Hope that helps.
> Did I miss anything?
>
>
> On Thu, Jan 12, 2017 at 6:23 AM, Jan van Doorn <jv...@knutsel.com> wrote:
>
> > Don’t forget about remap stats. If you want Traffic Router to enforce max
> > Gbps or TPS by DS, and (more likely) have Traffic Stats show per delivery
> > service stats, you’ll need to send remap stats. I think we should
> consider
> > making those more generic, as they appear under the ats key now.
> >
> > It’s also possible to add your own keys, and have Traffic Monitor act on
> > those by putting corresponding parameters in the profile, but I don’t
> think
> > you need that.
> >
> > Rgds,
> > JvD
> >
> >
> > > On Jan 12, 2017, at 5:49 AM, Eric Friedrich (efriedri) <
> > efriedri@cisco.com> wrote:
> > >
> > > Hey Nir, Adan-
> > > I don’t think #2 configuration pull is actually required. The cache
> > must be configured in Traffic Ops, so Traffic Monitor can learn its
> > hostname and capacity. Other than that it should just be meeting the
> astats
> > criteria.
> > >
> > > I think you might need interface speed and actual used bandwidth in the
> > astats (rather than available bandwidth in Kbps). I can’t remember if the
> > availableBandwidth is computed in astats or in TM.
> > >
> > > —Eric
> > >
> > >
> > >> On Jan 12, 2017, at 4:20 AM, Nir Sopher <ni...@qwilt.com> wrote:
> > >>
> > >> Hello,
> > >>
> > >>
> > >>
> > >> One of our team members (Adan Alper, CCed) is currently trying to
> > connect a
> > >> non ATS cache to Traffic Control.
> > >>
> > >> In order to be able to do this, we tried to figure out what is the
> bare
> > >> minimum information that needs to be sent by the cache to TC in order
> > for
> > >> it to consider the cache as active.
> > >>
> > >> Currently we found the following:
> > >>
> > >> 1. Monitoring file (_astat) is read from the traffic monitor
> ones
> > >> every few seconds. In this file we must provide the following data:
> > >>
> > >> a. availableBandwidthInKbps
> > >>
> > >> b. loadavg
> > >>
> > >> 2. Configuration pull – The cache should pull configuration from
> > his
> > >> queue in traffic OPS once every 15 minutes and suppose to signal
> traffic
> > >> ops once it was read and applied.
> > >>
> > >>
> > >>
> > >> Our questions are:
> > >>
> > >> 1. Are these the only information pieces (bare minimum) that we
> > need
> > >> to provide or are there any more?
> > >>
> > >> 2. Regarding the configuration read signaling. what exactly does
> > that
> > >> cache needs to send to traffic ops for it to acknowledge that the
> > >> configuration was read and applied? Is this a must for basic
> > functionality?
> > >>
> > >>
> > >>
> > >> Thanks,
> > >>
> > >> Nir
> > >
> >
> >
>
Re: Question regarding TC integration
Posted by Robert Butts <ro...@gmail.com>.
Ok, I don't know how much you know, so, sorry if I'm telling you things you
already know, but here goes.
> currently trying to connect a non ATS cache to Traffic Control.
So, Traffic Control involves configuring caches, polling those caches to
determine if they're 'healthy', and routing requests to them.
> what is the bare minimum information that needs to be sent by the cache
to TC in order for it to consider the cache as active.
I guess there are multiple levels of 'integrating' the cache with the CDN.
For the "absolute minimum", you could simply enter the cache as a Server in
Traffic Ops, and set it as Online (and snapshot CRConfig). That will make
Traffic Router send it requests, with no regard for whether it's actually
'healthy' (you'd never want to do this in a production system).
One step beyond that, is setting it Reported, and making Traffic Monitor
capable of polling it. Which is what it sounds like you're trying to do.
The polling URL doesn't actually have to be "/_astats", you can put
anything in the cache profile's `health.polling.url` parameter, though it
does expect the param to contain `${hostname}` and `${interface_name}`. The
JSON returned from the cache (or a proxy representing it) is pretty
inflexible, though. That JSON needs to be of the form ```
{ "ats": {
"plugin.remap_stats.delivery-service-one-fqdn.in_bytes": 42000,
"plugin.remap_stats.delivery-service-one-fqdn.out_bytes": 42001,
"plugin.remap_stats.delivery-service-two-fqdn.in_bytes": 42002,
⋮
}
"system": {
"inf.name": "eth0",
"inf.speed": 10000,
"proc.net.dev": "bond0: 12345 6789012314 0 0 0 0 0
1234 12345678 12345678 0 0 0 0 0 0",
"proc.loadavg": "1.1 1.2 1.3 1/816 13630"
}
```
If `system` isn't obvious, `proc.net.dev` is the value of Linux
/proc/net/dev for that interface, `proc.loadavg` is /proc/loadavg, `inf.name`
is the name of the network interface the cache will be serving content on,
and `inf.speed` is the interface speed of the cache's network card, e.g.
from `ethtool eth0 | grep -i speed`.
At minimum, there needs to be a `plugin.remap_stats.` key, for every
delivery service assigned to that cache, with `.in_bytes` and `.out_bytes`.
If keys also exist for .status_2xx, .status_3xx, .status_4xx, .status_5xx
Traffic Monitor will aggregate and serve those stats, but I _think_ it will
work without them. Traffic Monitor will also aggregate and serve other
stats in the "ats" JSON key, if you want them for Traffic Stats or other
analytics, but they shouldn't be necessary for the monitoring itself.
Then, Traffic Monitor should mark that cache as healthy, based on the
cache's profile's `health.threshold.loadavg` and
`health.threshold.queryTime` parameters, and Traffic Router will get the
Monitor's health data and route requests to that cache accordingly.
That's all I can think, but I'm almost certainly missing something. Making
a non-ATS cache work with Traffic Control won't be an easy task. We'd
definitely like Traffic Control to be generic and work with any cache, but
presently, everything has been designed very ATS-specific. But we'd
certainly welcome pull requests toward making it generic!
Once you have health monitoring and routing working, you'll want to make
Traffic Control able to configure the cache. You probably don't want to
worry about that too much until monitoring works, but just the high level
overview:
Right now, Traffic Ops generates config files for each cache, and we use a
script (
https://github.com/apache/incubator-trafficcontrol/blob/master/traffic_ops/bin/traffic_ops_ort.pl)
on a cron job on each cache, to pull updates and new config files from
Traffic Ops. So, you'll need to write that cron-job-script for the cache
you're using, and write a script/app/service to generate the configs. I'd
encourage you to use the Traffic Ops Client (
https://github.com/apache/incubator-trafficcontrol/tree/master/traffic_ops/client)
for the config generator. If you're comfortable with Go, it's the easiest
way to work with the Traffic Ops API.
You'll also have to generate the config files yourself. There's a good
chance not all the data you need to generate those configs is available in
the Traffic Ops API. In which case, you'll have to make Pull Requests
adding those APIs. I did a prototype standalone service/app config
generator last month (
https://github.com/rob05c/incubator-trafficcontrol/tree/atsconfig/traffic_ops/experimental/ats_config),
and the big APIs I found missing were Jobs, and Delivery Service Regexes.
Some of the data, like DS regexes, you can get from the CRConfig
(/CRConfig-Snapshots/cdn-name/CRConfig.json). Though the CRConfig endpoint
is huge and slow, adding the API endpoints you need would be better.
> what exactly does that cache needs to send to traffic ops for it to
acknowledge that the configuration was read and applied?
Once your script has updated the cache's config, you can notify Traffic Ops
(which will clear the clock icon in the Health -> Server Checks grid) by
writing to `https://traffic-ops.example.net/update/cache-hostname`. See
https://github.com/apache/incubator-trafficcontrol/blob/master/traffic_ops/bin/traffic_ops_ort.pl#L654
Again, sorry if I'm telling you things you already know. Hope that helps.
Did I miss anything?
On Thu, Jan 12, 2017 at 6:23 AM, Jan van Doorn <jv...@knutsel.com> wrote:
> Don’t forget about remap stats. If you want Traffic Router to enforce max
> Gbps or TPS by DS, and (more likely) have Traffic Stats show per delivery
> service stats, you’ll need to send remap stats. I think we should consider
> making those more generic, as they appear under the ats key now.
>
> It’s also possible to add your own keys, and have Traffic Monitor act on
> those by putting corresponding parameters in the profile, but I don’t think
> you need that.
>
> Rgds,
> JvD
>
>
> > On Jan 12, 2017, at 5:49 AM, Eric Friedrich (efriedri) <
> efriedri@cisco.com> wrote:
> >
> > Hey Nir, Adan-
> > I don’t think #2 configuration pull is actually required. The cache
> must be configured in Traffic Ops, so Traffic Monitor can learn its
> hostname and capacity. Other than that it should just be meeting the astats
> criteria.
> >
> > I think you might need interface speed and actual used bandwidth in the
> astats (rather than available bandwidth in Kbps). I can’t remember if the
> availableBandwidth is computed in astats or in TM.
> >
> > —Eric
> >
> >
> >> On Jan 12, 2017, at 4:20 AM, Nir Sopher <ni...@qwilt.com> wrote:
> >>
> >> Hello,
> >>
> >>
> >>
> >> One of our team members (Adan Alper, CCed) is currently trying to
> connect a
> >> non ATS cache to Traffic Control.
> >>
> >> In order to be able to do this, we tried to figure out what is the bare
> >> minimum information that needs to be sent by the cache to TC in order
> for
> >> it to consider the cache as active.
> >>
> >> Currently we found the following:
> >>
> >> 1. Monitoring file (_astat) is read from the traffic monitor ones
> >> every few seconds. In this file we must provide the following data:
> >>
> >> a. availableBandwidthInKbps
> >>
> >> b. loadavg
> >>
> >> 2. Configuration pull – The cache should pull configuration from
> his
> >> queue in traffic OPS once every 15 minutes and suppose to signal traffic
> >> ops once it was read and applied.
> >>
> >>
> >>
> >> Our questions are:
> >>
> >> 1. Are these the only information pieces (bare minimum) that we
> need
> >> to provide or are there any more?
> >>
> >> 2. Regarding the configuration read signaling. what exactly does
> that
> >> cache needs to send to traffic ops for it to acknowledge that the
> >> configuration was read and applied? Is this a must for basic
> functionality?
> >>
> >>
> >>
> >> Thanks,
> >>
> >> Nir
> >
>
>
Re: Question regarding TC integration
Posted by Jan van Doorn <jv...@knutsel.com>.
Don’t forget about remap stats. If you want Traffic Router to enforce max Gbps or TPS by DS, and (more likely) have Traffic Stats show per delivery service stats, you’ll need to send remap stats. I think we should consider making those more generic, as they appear under the ats key now.
It’s also possible to add your own keys, and have Traffic Monitor act on those by putting corresponding parameters in the profile, but I don’t think you need that.
Rgds,
JvD
> On Jan 12, 2017, at 5:49 AM, Eric Friedrich (efriedri) <ef...@cisco.com> wrote:
>
> Hey Nir, Adan-
> I don’t think #2 configuration pull is actually required. The cache must be configured in Traffic Ops, so Traffic Monitor can learn its hostname and capacity. Other than that it should just be meeting the astats criteria.
>
> I think you might need interface speed and actual used bandwidth in the astats (rather than available bandwidth in Kbps). I can’t remember if the availableBandwidth is computed in astats or in TM.
>
> —Eric
>
>
>> On Jan 12, 2017, at 4:20 AM, Nir Sopher <ni...@qwilt.com> wrote:
>>
>> Hello,
>>
>>
>>
>> One of our team members (Adan Alper, CCed) is currently trying to connect a
>> non ATS cache to Traffic Control.
>>
>> In order to be able to do this, we tried to figure out what is the bare
>> minimum information that needs to be sent by the cache to TC in order for
>> it to consider the cache as active.
>>
>> Currently we found the following:
>>
>> 1. Monitoring file (_astat) is read from the traffic monitor ones
>> every few seconds. In this file we must provide the following data:
>>
>> a. availableBandwidthInKbps
>>
>> b. loadavg
>>
>> 2. Configuration pull – The cache should pull configuration from his
>> queue in traffic OPS once every 15 minutes and suppose to signal traffic
>> ops once it was read and applied.
>>
>>
>>
>> Our questions are:
>>
>> 1. Are these the only information pieces (bare minimum) that we need
>> to provide or are there any more?
>>
>> 2. Regarding the configuration read signaling. what exactly does that
>> cache needs to send to traffic ops for it to acknowledge that the
>> configuration was read and applied? Is this a must for basic functionality?
>>
>>
>>
>> Thanks,
>>
>> Nir
>
Re: Question regarding TC integration
Posted by "Eric Friedrich (efriedri)" <ef...@cisco.com>.
Hey Nir, Adan-
I don’t think #2 configuration pull is actually required. The cache must be configured in Traffic Ops, so Traffic Monitor can learn its hostname and capacity. Other than that it should just be meeting the astats criteria.
I think you might need interface speed and actual used bandwidth in the astats (rather than available bandwidth in Kbps). I can’t remember if the availableBandwidth is computed in astats or in TM.
—Eric
> On Jan 12, 2017, at 4:20 AM, Nir Sopher <ni...@qwilt.com> wrote:
>
> Hello,
>
>
>
> One of our team members (Adan Alper, CCed) is currently trying to connect a
> non ATS cache to Traffic Control.
>
> In order to be able to do this, we tried to figure out what is the bare
> minimum information that needs to be sent by the cache to TC in order for
> it to consider the cache as active.
>
> Currently we found the following:
>
> 1. Monitoring file (_astat) is read from the traffic monitor ones
> every few seconds. In this file we must provide the following data:
>
> a. availableBandwidthInKbps
>
> b. loadavg
>
> 2. Configuration pull – The cache should pull configuration from his
> queue in traffic OPS once every 15 minutes and suppose to signal traffic
> ops once it was read and applied.
>
>
>
> Our questions are:
>
> 1. Are these the only information pieces (bare minimum) that we need
> to provide or are there any more?
>
> 2. Regarding the configuration read signaling. what exactly does that
> cache needs to send to traffic ops for it to acknowledge that the
> configuration was read and applied? Is this a must for basic functionality?
>
>
>
> Thanks,
>
> Nir