Unexpected behavior with parent routing

[Nick Dunkin <Ni...@vecima.com>]

Hi,

I wanted to raise an interesting issue we discovered last night during some troubleshooting.  This is Traffic Server version 7.1.4.

We were investigating unexpectedly long response times on cache hits.

A cache miss was taking approximately 10ms in client response time, but subsequent cache hits, for the same content, were taking a constant, and repeatable, 60ms to return.

After many hours of sanity checking, and configuration checking we discovered the culprit.

We are using parent routing, and our single parent rule used “dest_domain=.”

If we replace the dest_domain wildcard rule, with a url_regex rule, the issue is resolved and our cache hits become expectedly fast.

So our assumption here is that the 60 ms latency is down to some DNS related query, because we were using dest_domain, but if that’s the reason why is parent routing even in play during a cache hit, and even a cache mem hit?

We discovered the issue on a production system, and we are going to produce a minimal repeatable test case in the lab.  When we have done that I can post again if necessary.

Can anybody help us understand what may be going on here?

Many thanks,

Nick

Nick Dunkin
Director, Software Engineering
Manager – Architecture and New Product Introduction
o:  +1 678.258.4071
e: nick.dunkin@vecima.com<ma...@vecima.com>

[cidimage001.png@01D6CC8C.6FC5A580]

Re: [E] Re: Unexpected behavior with parent routing

[Jeremy Payne <jp...@gmail.com>]

Right.. I tested Duke's master branch a few month's back and didnt see
the 'random' 60ms ttms that I see in 7.1.4.


On Mon, Mar 15, 2021 at 10:24 AM Alan Carroll
<so...@verizonmedia.com.invalid> wrote:
>
> I would expect the 60ms ttms is due to the event loop timing - that because
> of the DNS callout, there is a miss which then waits for the event loop to
> timeout and try again, which takes 60ms on a lightly loaded system. DNS is
> done even for cache hits but that's not something I've ever understood. I
> expect for ATS 10 that will be different.

Re: [E] Re: Unexpected behavior with parent routing

[Alan Carroll <so...@verizonmedia.com.INVALID>]

I would expect the 60ms ttms is due to the event loop timing - that because
of the DNS callout, there is a miss which then waits for the event loop to
timeout and try again, which takes 60ms on a lightly loaded system. DNS is
done even for cache hits but that's not something I've ever understood. I
expect for ATS 10 that will be different.

Re: Unexpected behavior with parent routing

[Jeremy Payne <jp...@gmail.com>]

re: 60ms ttms

take a look at the below.

https://github.com/apache/trafficserver/pull/6234

https://github.com/duke8253/trafficserver/commit/159ae2510f0e2dab0907bdf2aec1ce74da0b3653

On Wed, Mar 10, 2021 at 9:07 AM Nick Dunkin <Ni...@vecima.com> wrote:

> Hi,
>
>
>
> I wanted to raise an interesting issue we discovered last night during
> some troubleshooting.  This is Traffic Server version 7.1.4.
>
>
>
> We were investigating unexpectedly long response times on cache hits.
>
>
>
> A cache miss was taking approximately *10ms* in client response time, but
> subsequent cache hits, for the same content, were taking a constant, and
> repeatable, *60ms* to return.
>
>
>
> After many hours of sanity checking, and configuration checking we
> discovered the culprit.
>
>
>
> We are using parent routing, and our single parent rule used “
> *dest_domain=.”*
>
>
>
> If we replace the *dest_domain* wildcard rule, with a *url_regex* rule,
> the issue is resolved and our cache hits become expectedly fast.
>
>
>
> So our assumption here is that the 60 ms latency is down to some DNS
> related query, because we were using dest_domain, but if that’s the reason
> why is parent routing even in play during a cache hit, and even a cache mem
> hit?
>
>
>
> We discovered the issue on a production system, and we are going to
> produce a minimal repeatable test case in the lab.  When we have done that
> I can post again if necessary.
>
>
>
> Can anybody help us understand what may be going on here?
>
>
>
> Many thanks,
>
>
>
> Nick
>
>
>
> *Nick Dunkin*
>
> Director, Software Engineering
>
> Manager – Architecture and New Product Introduction
>
> *o: * *+1 678.258.4071*
>
> *e:* nick.dunkin@vecima.com
>
>
>
> [image: cidimage001.png@01D6CC8C.6FC5A580]
>