You are viewing a plain text version of this content. The canonical link for it is here.
Posted to olio-dev@incubator.apache.org by Akara Sucharitakul <Ak...@Sun.COM> on 2009/01/28 17:18:11 UTC

Thumbnail caching proposal

As we run into filesystem bottlenecks, it makes sense to reduce the 
filesystem access by caching thumbnail images in memcached. The 
thumbnails are about 6-7 KB in size. We want to ensure there's a hot set 
of thumbnails being cached in memcached but not that every image gets 
cached. So here's my first draft on how to achieve that:

1. All thumbnails that are accessed by the client and driver are cached 
in memcached with a timeout of 90 seconds. Of course, we can still 
adjust this. Any access to the thumbnails will check the cache at first.

2. In order to keep the hot set low, we will modify our tag search query 
to order by event date with the most recent upcoming events first. As a 
result, the set of events searched by each tag are more deterministic 
and will not often change. Since the tag search only shows 10 events per 
page, the thumbnails for these top ten events for each tag will be held 
in memcached.

The mathematical model for this is a little complicated, though. Since 
we're dealing with non-uniform distribution of tags and a cap of 10 
upcoming events on top of that. But please let me know whether you agree 
with this conceptually.

Thanks,
-Akara

Re: Thumbnail caching proposal

Posted by Shanti Subramanyam - PAE <Sh...@Sun.COM>.
Wouldn't it make more sense to shift the caching to an image server that 
directly serves all the files in the filestore ?
This will also reduce the load on the php application significantly. 
 From Will's and Amanda's notes, it appears that they are deploying the 
rails app in this manner - using nginx to serve the files and forwarding 
all other requests to the thins in the back.

Ideally, we would like a distributed filestore - otherwise,sooner or 
later we're going to run into a bottleneck on the image server as well. 
But at least this is a good first step to take.

Comments ?

Shanti

On 01/28/09 09:18 AM, Akara Sucharitakul wrote:
> As we run into filesystem bottlenecks, it makes sense to reduce the 
> filesystem access by caching thumbnail images in memcached. The 
> thumbnails are about 6-7 KB in size. We want to ensure there's a hot set 
> of thumbnails being cached in memcached but not that every image gets 
> cached. So here's my first draft on how to achieve that:
> 
> 1. All thumbnails that are accessed by the client and driver are cached 
> in memcached with a timeout of 90 seconds. Of course, we can still 
> adjust this. Any access to the thumbnails will check the cache at first.
> 
> 2. In order to keep the hot set low, we will modify our tag search query 
> to order by event date with the most recent upcoming events first. As a 
> result, the set of events searched by each tag are more deterministic 
> and will not often change. Since the tag search only shows 10 events per 
> page, the thumbnails for these top ten events for each tag will be held 
> in memcached.
> 
> The mathematical model for this is a little complicated, though. Since 
> we're dealing with non-uniform distribution of tags and a cap of 10 
> upcoming events on top of that. But please let me know whether you agree 
> with this conceptually.
> 
> Thanks,
> -Akara