You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@trafficserver.apache.org by Jean Baptiste Favre <we...@jbfavre.org> on 2014/02/04 08:39:37 UTC
Cache almost full. What next ?
Hello,
I configured TrafficServer, in virtual machines, to use raw disks as
cache. Currently 2 disks of 10G each.
TrafficServer now reports 80% of cache used.
Basically, I see 2 solutions:
- Add more disk space (best one because Trafficservers are behind a
Load-balancer)
- Add more servers (which won't solve the per server disk usage)
Regarding first solution, I wonder how raw disk cache exactly works, in
terms of repartition ?
Adding new disk, will trafficserver:
- invalidate part of the cache (sort of sharding) ?
- somehow rebalance data ?
- add new data after existing ones ?
Regards,
Jean Baptiste
Re: Cache almost full. What next ?
Posted by Jean Baptiste Favre <we...@jbfavre.org>.
Hello,
On 04/02/2014 16:09, Alan M. Carroll wrote:
> Tuesday, February 4, 2014, 1:39:37 AM, you wrote:
>
>> Hello,
>> I configured TrafficServer, in virtual machines, to use raw disks as
>> cache. Currently 2 disks of 10G each.
>> TrafficServer now reports 80% of cache used.
>
>> Basically, I see 2 solutions:
>> - Add more disk space (best one because Trafficservers are behind a
>> Load-balancer)
>> - Add more servers (which won't solve the per server disk usage)
>
> I would recommend taking the third option, just let it run. It is completely normal and expected for the Traffic Server cache to become full. All of the deployments I know of treat this as the normal state. Traffic Server will evict old content from the cache as needed.
Problem is, TrafficServer is used to cache API calls.
And API usage is limited by daily quota, so I definitely need to add
more capacity :)
>> Regarding first solution, I wonder how raw disk cache exactly works, in
>> terms of repartition ?
>
>> Adding new disk, will trafficserver:
>> - invalidate part of the cache (sort of sharding) ?
>> - somehow rebalance data ?
>> - add new data after existing ones ?
>
> Some of the cache will be invalidated, none of it will be migrated between physical disks.
I've added new disk and see that cache has been partially invalidated.
Nothing crazy, but still.
Anyway, thanks for the answer, I now understand TrafficServer cache
more clearly :)
Regards,
Jean Baptiste
Re: Cache almost full. What next ?
Posted by Phil Sorber <so...@apache.org>.
On Wed, Feb 5, 2014 at 11:10 AM, Miles Libbey <ml...@apache.org> wrote:
> Wednesday, February 5, 2014, 10:32:09 AM, you wrote:
> >> On Tue, Feb 4, 2014 at 6:10 PM, John Plevyak <jp...@gmail.com>wrote:
> >>
> >> You can pin stuff in the cache for a given time. There is also a bit
> evacuation feature which should keep fresh content in the cache of it is
> accessed with any frequency.
> >>
> >> Is this second feature you are talking about the thing that evacuates
> content that is currently being served, or is there something else?
> >
> > Something else. See `proxy.config.cache.hit_evacuate_percent`. I wonder
> if that's documented...need to check.
>
>
> https://trafficserver.readthedocs.org/en/latest/reference/configuration/records.config.en.html#proxy-config-cache-hit-evacuate-percent
Nice. I wasn't aware of this before. Thanks.
>
>
> > What this does is when a cache item is hit, if it is within that percent
> of the stripe size of the write cursor, it is marked for evacuation even if
> it is not being accessed at the time the write cursor reaches it.
>
Re: Cache almost full. What next ?
Posted by Phil Sorber <so...@apache.org>.
On Wed, Feb 5, 2014 at 11:10 AM, Miles Libbey <ml...@apache.org> wrote:
> Wednesday, February 5, 2014, 10:32:09 AM, you wrote:
> >> On Tue, Feb 4, 2014 at 6:10 PM, John Plevyak <jp...@gmail.com>wrote:
> >>
> >> You can pin stuff in the cache for a given time. There is also a bit
> evacuation feature which should keep fresh content in the cache of it is
> accessed with any frequency.
> >>
> >> Is this second feature you are talking about the thing that evacuates
> content that is currently being served, or is there something else?
> >
> > Something else. See `proxy.config.cache.hit_evacuate_percent`. I wonder
> if that's documented...need to check.
>
>
> https://trafficserver.readthedocs.org/en/latest/reference/configuration/records.config.en.html#proxy-config-cache-hit-evacuate-percent
Nice. I wasn't aware of this before. Thanks.
>
>
> > What this does is when a cache item is hit, if it is within that percent
> of the stripe size of the write cursor, it is marked for evacuation even if
> it is not being accessed at the time the write cursor reaches it.
>
Re: Cache almost full. What next ?
Posted by Miles Libbey <ml...@apache.org>.
Wednesday, February 5, 2014, 10:32:09 AM, you wrote:
>> On Tue, Feb 4, 2014 at 6:10 PM, John Plevyak <jp...@gmail.com>wrote:
>>
>> You can pin stuff in the cache for a given time. There is also a bit evacuation feature which should keep fresh content in the cache of it is accessed with any frequency.
>>
>> Is this second feature you are talking about the thing that evacuates content that is currently being served, or is there something else?
>
> Something else. See `proxy.config.cache.hit_evacuate_percent`. I wonder if that's documented...need to check.
https://trafficserver.readthedocs.org/en/latest/reference/configuration/records.config.en.html#proxy-config-cache-hit-evacuate-percent
> What this does is when a cache item is hit, if it is within that percent of the stripe size of the write cursor, it is marked for evacuation even if it is not being accessed at the time the write cursor reaches it.
Re: Cache almost full. What next ?
Posted by "Alan M. Carroll" <am...@network-geographics.com>.
Wednesday, February 5, 2014, 10:32:09 AM, you wrote:
> On Tue, Feb 4, 2014 at 6:10 PM, John Plevyak <jp...@gmail.com>wrote:
>
> You can pin stuff in the cache for a given time. There is also a bit evacuation feature which should keep fresh content in the cache of it is accessed with any frequency.
>
> Is this second feature you are talking about the thing that evacuates content that is currently being served, or is there something else?
Something else. See `proxy.config.cache.hit_evacuate_percent`. I wonder if that's documented...need to check.
What this does is when a cache item is hit, if it is within that percent of the stripe size of the write cursor, it is marked for evacuation even if it is not being accessed at the time the write cursor reaches it.
Re: Cache almost full. What next ?
Posted by Phil Sorber <so...@apache.org>.
On Tue, Feb 4, 2014 at 6:10 PM, John Plevyak <jp...@gmail.com> wrote:
> You can pin stuff in the cache for a given time. There is also a bit
> evacuation feature which should keep fresh content in the cache of it is
> accessed with any frequency.
>
Is this second feature you are talking about the thing that evacuates
content that is currently being served, or is there something else?
> On Feb 4, 2014 10:21 AM, "Alan M. Carroll" <am...@network-geographics.com>
> wrote:
>
>> For anyone who doesn't mind getting their brain dirty, there is some
>> documentation on how the cache works here[1]. A key point to keep in mind
>> is that the TS cache is a circular buffer, so "full" is a rather vague
>> concept for it. That is, non-stale content can be evicted even if the cache
>> is not technically full.
>>
>> [1]
>> https://trafficserver.readthedocs.org/en/latest/arch/cache/cache.en.html
>>
>> Tuesday, February 4, 2014, 10:30:36 AM, you wrote:
>>
>> > A related question I have is what would be the best way to determine
>> > if your cache misses are a consequence of otherwise fresh content
>> > being evicted from the cache?
>>
>> That's quite difficult currently. My clients are not overly pleased with
>> that situation either and I am currently working for them on improving the
>> cache API to make this much easier. Hopefully I will be able to present
>> some progress on this at ApacheCon/TS Summit.
>>
>> Because of the way the cache works, it's not really feasible for it to
>> actually know if it is evicting non-stale content. One thing you could do
>> is track cache write bytes and compare that to the total cache size, which
>> should give you a rough estimate of how often the cache wraps. It might be
>> reasonable to put in a specific statistic that counts such wraps.
>>
>> It would be possible to detect an eviction on a cache miss, somewhat
>> reliably. I don't know if that is tracked specifically though.
>>
>>
Re: Cache almost full. What next ?
Posted by John Plevyak <jp...@gmail.com>.
You can pin stuff in the cache for a given time. There is also a bit
evacuation feature which should keep fresh content in the cache of it is
accessed with any frequency.
On Feb 4, 2014 10:21 AM, "Alan M. Carroll" <am...@network-geographics.com>
wrote:
> For anyone who doesn't mind getting their brain dirty, there is some
> documentation on how the cache works here[1]. A key point to keep in mind
> is that the TS cache is a circular buffer, so "full" is a rather vague
> concept for it. That is, non-stale content can be evicted even if the cache
> is not technically full.
>
> [1]
> https://trafficserver.readthedocs.org/en/latest/arch/cache/cache.en.html
>
> Tuesday, February 4, 2014, 10:30:36 AM, you wrote:
>
> > A related question I have is what would be the best way to determine
> > if your cache misses are a consequence of otherwise fresh content
> > being evicted from the cache?
>
> That's quite difficult currently. My clients are not overly pleased with
> that situation either and I am currently working for them on improving the
> cache API to make this much easier. Hopefully I will be able to present
> some progress on this at ApacheCon/TS Summit.
>
> Because of the way the cache works, it's not really feasible for it to
> actually know if it is evicting non-stale content. One thing you could do
> is track cache write bytes and compare that to the total cache size, which
> should give you a rough estimate of how often the cache wraps. It might be
> reasonable to put in a specific statistic that counts such wraps.
>
> It would be possible to detect an eviction on a cache miss, somewhat
> reliably. I don't know if that is tracked specifically though.
>
>
Re: Cache almost full. What next ?
Posted by Mark Harrison <ma...@omniti.com>.
That document clears a few things up - If I understand correctly now,
evacuation is the mechanism used to prevent an object from being
overwritten/evicted
(https://trafficserver.readthedocs.org/en/latest/arch/cache/cache-arch.en.html#evacuation-mechanics
and the following section). Does that mean that TS-1508 is still valid
or is it that if I'm seeing 0 for that stat that nothing is being
evacuated?
I wonder also if it might be possible to just have whatever process
writes to the cache simply skip over any items marked for evacuation
or that are pinned, which would eliminate the copy.
Thanks for the info on counting the cache wraps, that makes sense and
looks like it would give a decent estimate, which is what I'm looking
for.
On Tue, Feb 4, 2014 at 1:20 PM, Alan M. Carroll
<am...@network-geographics.com> wrote:
> For anyone who doesn't mind getting their brain dirty, there is some documentation on how the cache works here[1]. A key point to keep in mind is that the TS cache is a circular buffer, so "full" is a rather vague concept for it. That is, non-stale content can be evicted even if the cache is not technically full.
>
> [1] https://trafficserver.readthedocs.org/en/latest/arch/cache/cache.en.html
>
> Tuesday, February 4, 2014, 10:30:36 AM, you wrote:
>
>> A related question I have is what would be the best way to determine
>> if your cache misses are a consequence of otherwise fresh content
>> being evicted from the cache?
>
> That's quite difficult currently. My clients are not overly pleased with that situation either and I am currently working for them on improving the cache API to make this much easier. Hopefully I will be able to present some progress on this at ApacheCon/TS Summit.
>
> Because of the way the cache works, it's not really feasible for it to actually know if it is evicting non-stale content. One thing you could do is track cache write bytes and compare that to the total cache size, which should give you a rough estimate of how often the cache wraps. It might be reasonable to put in a specific statistic that counts such wraps.
>
> It would be possible to detect an eviction on a cache miss, somewhat reliably. I don't know if that is tracked specifically though.
>
--
Mark Harrison
Lead Site Reliability Engineer
OmniTI
Re: Cache almost full. What next ?
Posted by "Alan M. Carroll" <am...@network-geographics.com>.
For anyone who doesn't mind getting their brain dirty, there is some documentation on how the cache works here[1]. A key point to keep in mind is that the TS cache is a circular buffer, so "full" is a rather vague concept for it. That is, non-stale content can be evicted even if the cache is not technically full.
[1] https://trafficserver.readthedocs.org/en/latest/arch/cache/cache.en.html
Tuesday, February 4, 2014, 10:30:36 AM, you wrote:
> A related question I have is what would be the best way to determine
> if your cache misses are a consequence of otherwise fresh content
> being evicted from the cache?
That's quite difficult currently. My clients are not overly pleased with that situation either and I am currently working for them on improving the cache API to make this much easier. Hopefully I will be able to present some progress on this at ApacheCon/TS Summit.
Because of the way the cache works, it's not really feasible for it to actually know if it is evicting non-stale content. One thing you could do is track cache write bytes and compare that to the total cache size, which should give you a rough estimate of how often the cache wraps. It might be reasonable to put in a specific statistic that counts such wraps.
It would be possible to detect an eviction on a cache miss, somewhat reliably. I don't know if that is tracked specifically though.
Re: Cache almost full. What next ?
Posted by Mark Harrison <ma...@omniti.com>.
> I would recommend taking the third option, just let it run. It is completely normal and expected for the Traffic Server cache to become full. All of the deployments I know of treat this as the normal state. Traffic Server will evict old content from the cache as needed.
A related question I have is what would be the best way to determine
if your cache misses are a consequence of otherwise fresh content
being evicted from the cache? In other words, how to determine if your
disk cache size really is too small for your workload, without having
to try increasing the size and seeing if it improves things.
Currently the only way I can think of is just to monitor cold cache
misses and make some assumptions/guesses about how much new content
your requests take up vs content that was there but has been evicted.
There appear to be metrics for cache evictions (e.g.
proxy.process.cache.gc_bytes_evacuated) but
https://issues.apache.org/jira/browse/TS-1508 suggests that they will
never go above 0.
Thanks,
Mark
Re: Cache almost full. What next ?
Posted by "Alan M. Carroll" <am...@network-geographics.com>.
Tuesday, February 4, 2014, 1:39:37 AM, you wrote:
> Hello,
> I configured TrafficServer, in virtual machines, to use raw disks as
> cache. Currently 2 disks of 10G each.
> TrafficServer now reports 80% of cache used.
> Basically, I see 2 solutions:
> - Add more disk space (best one because Trafficservers are behind a
> Load-balancer)
> - Add more servers (which won't solve the per server disk usage)
I would recommend taking the third option, just let it run. It is completely normal and expected for the Traffic Server cache to become full. All of the deployments I know of treat this as the normal state. Traffic Server will evict old content from the cache as needed.
> Regarding first solution, I wonder how raw disk cache exactly works, in
> terms of repartition ?
> Adding new disk, will trafficserver:
> - invalidate part of the cache (sort of sharding) ?
> - somehow rebalance data ?
> - add new data after existing ones ?
Some of the cache will be invalidated, none of it will be migrated between physical disks.