Doubts about cache behavior

[Acácio Centeno <ac...@azion.com>]

Hello,

We've being performing some cache tests and noticed two issues that we
would like to check with the community:

1. The interim cache is limited to the same size as the RAM cache, no
matter how big the partition allocated to it is.

2. Whenever a volume gets full and the write head begins writing to the
volume's start again, the RAM cache gets unused for a while, and requests
that could be served from RAM get served from the storage. Notice that we
considered the possibility that the objects that we _thought_ were in the
RAM had been replaced, but even requesting the same set of objects a few
times would not make them be served from the RAM.

Are these statements correct? We came to them based on observations made
during the tests fully described on [1]. We hope there's enough information
on the document to reproduce the tests.

We're available should someone need more info about our test scenarios.
Feel free to contact me or Daniel Biazus at daniel.biazus@azion.com .

[1]
https://drive.google.com/file/d/0B_pQbXm7hKZ4X2VSbXQxN1BOaEU/view?usp=sharing

Acácio Centeno
Software Engineering
Azion Technologies
Porto Alegre, Brasil +55 51 3012 3005 | +55 51 8118 9947
Miami, USA +1 305 704 8816

Quaisquer informações contidas neste e-mail e anexos podem ser
confidenciais e privilegiadas, protegidas por sigilo legal. Qualquer forma
de utilização deste documento depende de autorização do emissor, sujeito as
penalidades cabíveis.

Any information in this e-mail and attachments may be confidential and
privileged, protected by legal confidentiality. The use of this document
require authorization by the issuer, subject to penalties.

Re: Doubts about cache behavior

[Acácio Centeno <ac...@azion.com>]

Hello Allan,

I was hacking a bash script to help explain the issue [1] but I realized it
was not related to the volume getting full. Actually the "problem" was that
the Index was getting full. On the first test scenario that happened almost
when the storage got full, but on the scenario built by the script, that
happens with about 41% of storage usage. On [2] I explain what the script
does, and why I think the problem was the index. But it's actually not an
issue, it's just that some objects have to be evicted so others can be
saved. Why the RAM is so little used on this situation is still not very
clear, but I suppose it's probably due to the high hash collision rate that
the situation generates.

Thanks for the help!

[1]
https://drive.google.com/file/d/0B_pQbXm7hKZ4R0ZfOU5sS0Y5NGc/view?usp=sharing
[2]
https://drive.google.com/file/d/0B_pQbXm7hKZ4RmV0WHdoMWNtUjA/view?usp=sharing



Acácio Centeno
Software Engineering
Azion Technologies
Porto Alegre, Brasil +55 51 3012 3005 | +55 51 8118 9947
Miami, USA +1 305 704 8816

Quaisquer informações contidas neste e-mail e anexos podem ser
confidenciais e privilegiadas, protegidas por sigilo legal. Qualquer forma
de utilização deste documento depende de autorização do emissor, sujeito as
penalidades cabíveis.

Any information in this e-mail and attachments may be confidential and
privileged, protected by legal confidentiality. The use of this document
require authorization by the issuer, subject to penalties.


2015-02-05 12:59 GMT-02:00 Alan M. Carroll <am...@apache.org>:

> Tuesday, February 3, 2015, 2:45:03 PM, you wrote:
>
> > Hello,
>
> > We've being performing some cache tests and noticed two issues that we
> > would like to check with the community:
>
> > 1. The interim cache is limited to the same size as the RAM cache, no
> > matter how big the partition allocated to it is.
>
> Unfortunately the developers of the interim cache don't really work on
> Traffic Server anymore and I'm not sure anyone left understands how it
> works.
>
> > 2. Whenever a volume gets full and the write head begins writing to the
> > volume's start again, the RAM cache gets unused for a while, and requests
> > that could be served from RAM get served from the storage.
>
> That seems very odd. There shouldn't be any interaction between stripe
> wraps and use of the ram cache, especially since the ram cache is filled
> from reads which should be happening from all parts of the stripe. When you
> say it is unused, is this for objects just written, or do no objects seem
> to end up in the ram cache?
>
>

Re: Doubts about cache behavior

[Yongming Zhao <mi...@gmail.com>]

we are still work on the ATS, hehe. Weijin and me are working on our new startup, just lack of time to support codes changed recently, you guys made so many changes :D

will take a look at this issue.


> 在 2015年2月5日，下午10:59，Alan M. Carroll <am...@apache.org> 写道：
> 
> Tuesday, February 3, 2015, 2:45:03 PM, you wrote:
> 
>> Hello,
> 
>> We've being performing some cache tests and noticed two issues that we
>> would like to check with the community:
> 
>> 1. The interim cache is limited to the same size as the RAM cache, no
>> matter how big the partition allocated to it is.
> 
> Unfortunately the developers of the interim cache don't really work on Traffic Server anymore and I'm not sure anyone left understands how it works.
> 
>> 2. Whenever a volume gets full and the write head begins writing to the
>> volume's start again, the RAM cache gets unused for a while, and requests
>> that could be served from RAM get served from the storage.
> 
> That seems very odd. There shouldn't be any interaction between stripe wraps and use of the ram cache, especially since the ram cache is filled from reads which should be happening from all parts of the stripe. When you say it is unused, is this for objects just written, or do no objects seem to end up in the ram cache?
> 

- Yongming Zhao 赵永明

Re: Doubts about cache behavior

["Alan M. Carroll" <am...@apache.org>]

Tuesday, February 3, 2015, 2:45:03 PM, you wrote:

> Hello,

> We've being performing some cache tests and noticed two issues that we
> would like to check with the community:

> 1. The interim cache is limited to the same size as the RAM cache, no
> matter how big the partition allocated to it is.

Unfortunately the developers of the interim cache don't really work on Traffic Server anymore and I'm not sure anyone left understands how it works.

> 2. Whenever a volume gets full and the write head begins writing to the
> volume's start again, the RAM cache gets unused for a while, and requests
> that could be served from RAM get served from the storage.

That seems very odd. There shouldn't be any interaction between stripe wraps and use of the ram cache, especially since the ram cache is filled from reads which should be happening from all parts of the stripe. When you say it is unused, is this for objects just written, or do no objects seem to end up in the ram cache?

Re: Doubts about cache behavior

[Yongming Zhao <mi...@gmail.com>]

some point of view:
1, the interim cache is the cache for the data stored in storage.config defined devices, same as ram cache, that means if on object is removed(overwritten) from the storage devices, it is removed from both interim-cache and ram-cache, that is very critical ensure procedure for data consistency.
2, the object maybe promote to ram-cache and interim-cache, that depends on your testing case, that means, if you batch load some uniqued files with your test tool, that will not help LRU to get the replacement works like real worlds.
3, clfus will try to cache the small files instead of the big files, which may introduce more complex in your situation.
4, if you want to test the feature function, I’d advice that you make your test urls really small, and limit the ram cache to only cache a few size limited objects, and limit the test requests to see how the object is stored in the ram interim and storage.
5, jtest is a good random cache testing tool for you to generate some testing on storage and cache
6, the interim cache is working just like the storage devices, same as a log based journal file system.
7, there is a proxy.process.cache.interim.read.success stat to count all interim cache hits



> 在 2015年2月4日，上午4:45，Acácio Centeno <ac...@azion.com> 写道：
> 
> Hello,
> 
> We've being performing some cache tests and noticed two issues that we
> would like to check with the community:
> 
> 1. The interim cache is limited to the same size as the RAM cache, no
> matter how big the partition allocated to it is.
> 
> 2. Whenever a volume gets full and the write head begins writing to the
> volume's start again, the RAM cache gets unused for a while, and requests
> that could be served from RAM get served from the storage. Notice that we
> considered the possibility that the objects that we _thought_ were in the
> RAM had been replaced, but even requesting the same set of objects a few
> times would not make them be served from the RAM.
> 
> Are these statements correct? We came to them based on observations made
> during the tests fully described on [1]. We hope there's enough information
> on the document to reproduce the tests.
> 
> We're available should someone need more info about our test scenarios.
> Feel free to contact me or Daniel Biazus at daniel.biazus@azion.com .
> 
> [1]
> https://drive.google.com/file/d/0B_pQbXm7hKZ4X2VSbXQxN1BOaEU/view?usp=sharing
> 
> Acácio Centeno
> Software Engineering
> Azion Technologies
> Porto Alegre, Brasil +55 51 3012 3005 | +55 51 8118 9947
> Miami, USA +1 305 704 8816
> 
> Quaisquer informações contidas neste e-mail e anexos podem ser
> confidenciais e privilegiadas, protegidas por sigilo legal. Qualquer forma
> de utilização deste documento depende de autorização do emissor, sujeito as
> penalidades cabíveis.
> 
> Any information in this e-mail and attachments may be confidential and
> privileged, protected by legal confidentiality. The use of this document
> require authorization by the issuer, subject to penalties.

- Yongming Zhao 赵永明