Derby as a dacapo benchmark

[Robin Garner <ro...@anu.edu.au>]

We are looking for input into a derby benchmark to the next release of the
dacapo benchmarks (http://www.dacapobench.org/).

We currently have a workload ("hsqldb") which runs JDBCbench over HSQLDB. 
Our plan is to replace HSQLDB with derby.  However, when analyzing derby in
this setting, I noticed some problems:

1. JDBCbench runs the same workload 4 times, using autocommit vs
transactions, and prepared statements vs direct execution.  The problem is
that when several thousand queries are executed using direct SQL, differing
only in parameter values, the benchmark spends most of its time compiling
the queries to bytecode, JITing the bytecode, and eventually has to do
class-unloading to get rid of them.
2. JDBCBench is based on TPC-B, which is a pretty simplistic workload.

Now on one level, problem 1 is nice, since it gives JVM researchers a
concrete reason to support class-unloading.  But I can imagine a derby
developer saying to a JVM developer 'don't waste your time speeding that up,
any sane application developer would use a prepared statement'.

So, if the next release of DaCapo were to include a derby benchmark, would
people object to JDBCbench, and if not, what better clean, DB-focussed, open
source, workloads for derby can you point to?

Some important background:

1. The dacapo benchmark will *not* compare derby performance against any
other database.

2. Derby will most likely be a (secondary) part of other future benchmarks
that require a database, but do not focus on it (eg DayTrader).

3. The aim of the dacapo benchmarks is to give Java VM researchers (and
hopefully computer architects) standard workloads they can evaluate against.  
While standardization necessitates some contrivance, we want these workloads
to be as realistic as we can.  Our goal is for improvements against these
standard workloads to be reflected in improvements in real-world
applications.

regards,

Robin Garner
Dept. of Computer Science
Australian National University
http://cs.anu.edu.au/people/Robin.Garner/
-- 
View this message in context: http://www.nabble.com/Derby-as-a-dacapo-benchmark-tf4749895.html#a13581991
Sent from the Apache Derby Developers mailing list archive at Nabble.com.

Re: Derby as a dacapo benchmark

[Daniel John Debrunner <dj...@apache.org>]

Robin Garner wrote:
> We are looking for input into a derby benchmark to the next release of the
> dacapo benchmarks (http://www.dacapobench.org/).

> So, if the next release of DaCapo were to include a derby benchmark, would
> people object to JDBCbench, and if not, what better clean, DB-focussed, open
> source, workloads for derby can you point to?

Derby has a Java TPC-C based setup, more a collection of tools to allow 
a benchmark to be run than an out of the box benchmark.

Some details are in:

https://issues.apache.org/jira/browse/DERBY-1987

Dan.