[users@httpd] Question n best practices to layout file system for a large number of fils

[Ali Naddaf <al...@naddaf.org>]

Hello everyone.

I am planning on using Apache to serve a large number of files that can 
easily grow over time (to hundreds of thousands). My use case is around 
client applications (controlled by me as well) that given a file name, 
they want to download that file. One easy, and probably not very wise in 
terms of performance, is to put all the files in one directory and 
expose that  them through apache. As a side note, I am using certain 
hashing in the file names that avoids name clashing so that approach is 
safe. However, I imagine if I break that to multiple directories, I can 
get a better performance and that is what I like to learn from the 
experts whether that is a correct assumption, and if so, how deep I 
should consider creating the directory hierarchy and what factors play 
into my decision. I also understand that the file system used on the 
server (say Ext2 vs Ext3 vs Reiser, ...) can play a role here.

In my particular case, filenames are all hashes (like 
"0067df7874d05b5a22346a2c2a4a14e18cdbc721") with fixed width so 
"distributing" the files based on their names into multiple nested 
directories is an easy task (e.g. the file that I mentioned can go into 
"00/67/df" directory) and since the client is also mine,I can easily 
instruct the client to follow the same pattern when looking for a file.

If anyone has an insight into this type of problem and is willing to 
share his suggestions and ideas, I would greatly appreciate that.

Many thanks,
Ali.



---------------------------------------------------------------------
The official User-To-User support forum of the Apache HTTP Server Project.
See <URL:http://httpd.apache.org/userslist.html> for more info.
To unsubscribe, e-mail: users-unsubscribe@httpd.apache.org
   "   from the digest: users-digest-unsubscribe@httpd.apache.org
For additional commands, e-mail: users-help@httpd.apache.org

Re: [users@httpd] Question n best practices to layout file system for a large number of fils

[Peter Schober <pe...@univie.ac.at>]

* Ali Naddaf <al...@naddaf.org> [2009-04-09 18:19]:
> I also understand that the file system used on the server (say Ext2
> vs Ext3 vs Reiser, ...) can play a role here.

I'd suggest finding out how many files you can comfortably handle in a
single directory for the given filesystem before performance starts to
degrade (e.g. per the filesystems' docs). Estimate how many files you
would possibly want to store, at a maximum, and then you can calculate
how many levels of directories you'll need.

Unless .htaccess files will also live there I doubt this has a lot to
do with httpd.

cheers,
-peter

---------------------------------------------------------------------
The official User-To-User support forum of the Apache HTTP Server Project.
See <URL:http://httpd.apache.org/userslist.html> for more info.
To unsubscribe, e-mail: users-unsubscribe@httpd.apache.org
   "   from the digest: users-digest-unsubscribe@httpd.apache.org
For additional commands, e-mail: users-help@httpd.apache.org