RE: generating hash from packet content

[Rasim Saltuk Alakuş <ra...@turksat.com.tr>]

Hi All,

Thanks for your kind replies. 

Looks like it is not feasible/possible to remove URL hash map solution completely.. However, storage optimization was another topic in our mind, which content hash can save the day. We are thinking this can be a nice feature if implemented. If anybody decide to implement please let us know.


regards
Saltuk







________________________________________
From: Rasim Saltuk Alakuş
Sent: Wednesday, August 27, 2014 7:17 PM
To: dev@trafficserver.apache.org; users@trafficserver.apache.org
Subject: generating hash from packet content

Hi All,

ATS uses URL hash for cache storage. And CacheUrl plugin adds some more flexibility in URL hashing strategy.

We think of creating hash based on packet content and use it as the hash while storing and retrieving from cache This looks a better solution, so that URI changes won't hurt caching system. One immediate benefit for example if you cache YouTube , each request for same video can have different URL and CacheUrl plugin does not always provide a good solution. Also maintaining site based hash filters looks not an elegant solution.

Is there any previous or active work for implementing content based hashing? What kind of problems and constrains you may guess. Is there any volunteer to implement this feature together with us?

Kind regards
Saltuk Alakuş

RE: generating hash from packet content

[Luca Rea <lu...@contactlab.com>]

Ok, what about using a second ATS (parent of the first one) with a small storage as buffering system? 

Re: generating hash from packet content

["Alan M. Carroll" <am...@network-geographics.com>]

Luca,

Monday, September 1, 2014, 1:49:13 PM, you wrote:

> You can also choose to store it in cache and delete dupe in a second moment.

You could, but once you've written to cache, you have advanced the write cursor. Deleting the duplicate later has no effect except changing a directory entry. To have any benefit, you need to detect it as a duplicate before writing it to disk.

Re: generating hash from packet content

["Alan M. Carroll" <am...@network-geographics.com>]

Tuesday, September 2, 2014, 3:04:55 AM, you wrote:

> How does ATS manage the expired objecs? If an object has expired/removed , I assume allocated space for it can later be used by other objects.

ATS doesn't have allocated space in the cache. It's a circular buffer. All objects, including expired ones, are eventually overwritten. There is no other space management.

> PS: I am very new to ATS, and the link you provided is quite low level. So above is my basic expectation at best from a file system like structure.

Well, there isn't any higher level to the ATS cache. That low level description is effectively complete description. There is a write cursor, and ATS writes to cache at that location, advancing (circularly) as it writes. The directory keeps track of what has been written but data is always written at the write cursor.

RE: generating hash from packet content

[Rasim Saltuk Alakuş <ra...@turksat.com.tr>]

Hi Allan,

How does ATS manage the expired objecs? If an object has expired/removed , I assume allocated space for it can later be used by other objects. 

Total size is pre-allocated for ATS, but using the allocated space effectively is still an optimization. So I assume removing the duplicated data will do it.

If my understanding is correct, I like to implement this; Write all cachable content to disk when they arrive and a background low priority process will check for dublicates and remove them. Do you think is this possible?

PS: I am very new to ATS, and the link you provided is quite low level. So above is my basic expectation at best from a file system like structure.

Kind regards
Saltuk Alakuş 







-----Original Message-----
From: Alan M. Carroll [mailto:amc@network-geographics.com] 
Sent: Tuesday, September 02, 2014 1:53 AM
To: dev@trafficserver.apache.org
Subject: Re: generating hash from packet content

Luca,

Monday, September 1, 2014, 1:49:13 PM, you wrote:

> You can also choose to store it in cache and delete dupe in a second moment.

You could, but once you've written to cache, you have advanced the write cursor. Deleting the duplicate later has no effect except changing a directory entry. To have any benefit, you need to detect it as a duplicate before writing it to disk.

RE: generating hash from packet content

[Luca Rea <lu...@contactlab.com>]

You can also choose to store it in cache and delete dupe in a second moment.

Re: generating hash from packet content

[Leif Hedstrom <zw...@apache.org>]

I assume the idea would be to not write to the cache until you have decided it is not a dupe? Which would imply, buffering it somewhere which does not move the write header forward.

-- Leif 

> On Sep 1, 2014, at 9:55 AM, Luca Rea <lu...@contactlab.com> wrote:
> 
> Can it help to keep only the useful cache and limit the recycle process?

RE: generating hash from packet content

[Luca Rea <lu...@contactlab.com>]

Can it help to keep only the useful cache and limit the recycle process?

Re: generating hash from packet content

["Alan M. Carroll" <am...@network-geographics.com>]

Rasim,

Monday, September 1, 2014, 10:22:06 AM, you wrote:

> Looks like it is not feasible/possible to remove URL hash map solution completely.. However, storage optimization was another topic in our mind, which content hash can save the day. We are thinking this can be a nice feature if implemented. If anybody decide to implement please let us know.

How would it optimize storage? ATS uses all of its assigned storage at all times. Because of how the cache works, removing duplicates would have no effect on ATS's use of cache storage. Please see this link for some introductory detail

https://docs.trafficserver.apache.org/en/latest/arch/cache/cache-arch.en.html#storage-layout