On OLIO-51 : Increasing the size of the data store

[Shanti Subramanyam <Sh...@Sun.COM>]

We have successfully tested the PHP workload with the data size being 
set to 100x the number of concurrent users. This very nicely increases 
the size of the filestore (about 670GB for a scale of 8000) as well as 
the number of rows in the database (e.g 600,000 users for a scale of 
6000). However we still need to work on padding the rows and/or adding 
additional unused tables to get a respectable sized db. The db at 6000 
users is only about 3.5 GB - this will just not do. Will is supposed to 
put out a proposal on this. Any progress Will ?

In trying to load for a large scale, I did run into issues with the 
dbloader not being able to load more than 6000 users. After this point, 
we run out of heap space and I ended up with core dumps when trying to 
use a 64bit JVM. Akara is working on a fix to this issue so that the 
loader does not indiscriminately use memory.

I didn't run into any issues with the file loader - in fact, it is still 
pretty fast. But Akara being never satisfied :-) is working on speeding 
this up as well along with adding the directory heirarchy to the filestore.

At this point, I would like to propose that we check in the changes to 
increase the scale factor to 100. This is a one-line change in 
com.sun.web20.util.ScaleFactors.java to set USERS_RATIO to 100 instead 
of 10. If no one objects, I'll make the change to both the php and rails 
versions.

Although this will not close OLIO-51, it is a first step towards it.

Shanti