You are viewing a plain text version of this content. The canonical link for it is here.
Posted to derby-dev@db.apache.org by Jan Hlavatý <hl...@code.cz> on 2004/09/04 13:17:07 UTC
Re: java1.4.2 "rws" mode - log write performance
As this seems crucial for log performance,
I made a test to see for myself what's really the best way to write a log.
It tests various forced sequential write methods, with growing and preallocated file.
Using direct buffers doesn't seem to have any effect over plain RandomAccessFile.write(byte[]).
Following results are from my Athlon XP 1700+ Windows XP Professional machine,
with IDE disk that can dish out unsynced writes at 40MB/s, running JDK1.5 RC.
It proves Mike's point about "rws" being best on preallocated files on Windows.
That is, it can write 10MB/s in 4228 byte chunks, that is about 2490 chunks per sec.
So if you want to get 10000 transactions/sec, you will have to combine logs and
write them in one chunk. I think that's exactly what Derby does. So we're all good ;)
It also shows that sync() / force() are thing to avoid at all costs on Windows ;)
Can anyone run the test on another OS? I've attached the source file for the tests.
(warning: 10000 records test ran for 30minutes ;-) You can decrease the chunk count)
---------------------------------------------
1. System info:
---------------------------------------------
OS Platform: Windows XP/x86/5.1
Java spec: Java Platform API Specification version 1.5 from Sun Microsystems Inc.
Java VM: Java HotSpot(TM) Client VM version 1.5.0-rc-b63 from Sun Microsystems Inc.
Java home: C:\java\jdk1.5.0\jre
Test file: C:\Documents and Settings\hlavac\iotest\test.bin
Test file exists, deleting...
---------------------------------------------------------------------
2. Testing growing file using RandomAccessFile "rw" + .getFD().sync()
---------------------------------------------------------------------
Chunk size: 4228 bytes
Count: 10000
Total time: 283527 ms
Writes per second: 35.27 writes/s
Time per chunk: 28.353 ms
Write bandwidth: 144.0KB/s
---------------------------------------------------------------------------
3. Testing preallocated file using RandomAccessFile "rw" + .getFD().sync()
---------------------------------------------------------------------------
Chunk size: 4228 bytes
Count: 10000
Total time: 276988 ms
Writes per second: 36.103 writes/s
Time per chunk: 27.699 ms
Write bandwidth: 148.0KB/s
---------------------------------------------------------------------
4. Testing growing file using RandomAccessFile "rws"
---------------------------------------------------------------------
Chunk size: 4228 bytes
Count: 10000
Total time: 9173 ms
Writes per second: 1090.156 writes/s
Time per chunk: 0.917 ms
Write bandwidth: 4.0MB/s
---------------------------------------------------------------------
5. Testing preallocated file using RandomAccessFile "rws"
---------------------------------------------------------------------
Chunk size: 4228 bytes
Count: 10000
Total time: 4016 ms
Writes per second: 2490.04 writes/s
Time per chunk: 0.402 ms
Write bandwidth: 10.0MB/s
---------------------------------------------------------------------
6. Testing growing file using RandomAccessFile "rwd"
---------------------------------------------------------------------
Chunk size: 4228 bytes
Count: 10000
Total time: 10264 ms
Writes per second: 974.279 writes/s
Time per chunk: 1.026 ms
Write bandwidth: 3.0MB/s
---------------------------------------------------------------------
7. Testing preallocated file using RandomAccessFile "rwd"
---------------------------------------------------------------------
Chunk size: 4228 bytes
Count: 10000
Total time: 5598 ms
Writes per second: 1786.352 writes/s
Time per chunk: 0.56 ms
Write bandwidth: 7.0MB/s
------------------------------------------------------------------------------
8. Testing growing file using direct buffer + FileChannel "rw" + force(true)
------------------------------------------------------------------------------
Chunk size: 4228 bytes
Count: 10000
Total time: 281445 ms
Writes per second: 35.531 writes/s
Time per chunk: 28.144 ms
Write bandwidth: 144.0KB/s
-----------------------------------------------------------------------------------
9. Testing preallocated file using direct buffer + FileChannel "rw" + force(true)
-----------------------------------------------------------------------------------
Chunk size: 4228 bytes
Count: 10000
Total time: 276638 ms
Writes per second: 36.148 writes/s
Time per chunk: 27.664 ms
Write bandwidth: 148.0KB/s
------------------------------------------------------------------------------
10. Testing growing file using direct buffer + FileChannel "rw" + force(false)
------------------------------------------------------------------------------
Chunk size: 4228 bytes
Count: 10000
Total time: 283908 ms
Writes per second: 35.223 writes/s
Time per chunk: 28.391 ms
Write bandwidth: 144.0KB/s
------------------------------------------------------------------------------
11. Testing preallocated file using direct buffer + FileChannel "rw" + force(false)
------------------------------------------------------------------------------
Chunk size: 4228 bytes
Count: 10000
Total time: 277960 ms
Writes per second: 35.976 writes/s
Time per chunk: 27.796 ms
Write bandwidth: 144.0KB/s
------------------------------------------------------------------------------
12. Testing growing file using direct buffer + FileChannel "rws"
------------------------------------------------------------------------------
Chunk size: 4228 bytes
Count: 10000
Total time: 8733 ms
Writes per second: 1145.082 writes/s
Time per chunk: 0.873 ms
Write bandwidth: 4.0MB/s
------------------------------------------------------------------------------
13. Testing preallocated file using direct buffer + FileChannel "rws"
------------------------------------------------------------------------------
Chunk size: 4228 bytes
Count: 10000
Total time: 3805 ms
Writes per second: 2628.121 writes/s
Time per chunk: 0.381 ms
Write bandwidth: 10.0MB/s
------------------------------------------------------------------------------
14. Testing growing file using direct buffer + FileChannel "rwd"
------------------------------------------------------------------------------
Chunk size: 4228 bytes
Count: 10000
Total time: 10595 ms
Writes per second: 943.841 writes/s
Time per chunk: 1.06 ms
Write bandwidth: 3.0MB/s
------------------------------------------------------------------------------
15. Testing preallocated file using direct buffer + FileChannel "rwd"
------------------------------------------------------------------------------
Chunk size: 4228 bytes
Count: 10000
Total time: 5508 ms
Writes per second: 1815.541 writes/s
Time per chunk: 0.551 ms
Write bandwidth: 7.0MB/s
Re: java1.4.2 "rws" mode - log write performance - Linux numbers
Posted by Davide Savazzi <ds...@apache.org>.
On Saturday 04 September 2004 15:07, Jan Hlavatý wrote:
> I've ran the test on my crappy Linux based Celeron 800 firewall machine.
> It has RedHat linux 9.0 on single old IDE disk formatted with ext3
> filesystem.
> I'd really like to see what numbers I'd get with different filesystems,
> like reiser4 - ext3 has a reputation of being a bit slow...
My Linux box is a PIII 667, with a single 80GB IDE disk formatted with
reiser3. It runs Gentoo Linux 1.4.
I've attached the results...
It seems that with a preallocated file all the methods have the same (good)
performance.
Cheers,
--
Davide Savazzi
Re: java1.4.2 "rws" mode - log write performance - Linux numbers
Posted by Jan Hlavatý <hl...@code.cz>.
-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1
I've ran the test on my crappy Linux based Celeron 800 firewall machine.
It has RedHat linux 9.0 on single old IDE disk formatted with ext3 filesystem.
On linux, preallocated files do seem to help a lot too - theyre about 3 times faster
than growing files. Interesting thing here is that "rw"+sync() is as fasr (slow) as "rws"/"rwd"
on growing files here. "rws" on preallocated file seems to be best here, like on Windows.
I'd really like to see what numbers I'd get with different filesystems, like reiser4 -
ext3 has a reputation of being a bit slow...
Here are the numbers:
- ---------------------------------------------
1. System info:
- ---------------------------------------------
OS Platform: Linux/i386/2.4.20-20.9
Java spec: Java Platform API Specification version 1.5 from Sun Microsystems Inc.
Java VM: Java HotSpot(TM) Client VM version 1.5.0-rc-b63 from Sun Microsystems Inc.
Java home: /usr/java/jdk1.5.0/jre
Test file: /data/iotest/test.bin
Test file exists, deleting...
- ---------------------------------------------------------------------
2. Testing growing file using RandomAccessFile "rw" + .getFD().sync()
- ---------------------------------------------------------------------
Chunk size: 4228 bytes
Count: 10000
Total time: 117320 ms
Writes per second: 85.237 writes/s
Time per chunk: 11.732 ms
Write bandwidth: 350.0KB/s
- ---------------------------------------------------------------------------
3. Testing preallocated file using RandomAccessFile "rw" + .getFD().sync()
- ---------------------------------------------------------------------------
Chunk size: 4228 bytes
Count: 10000
Total time: 162398 ms
Writes per second: 61.577 writes/s
Time per chunk: 16.24 ms
Write bandwidth: 251.0KB/s
- ---------------------------------------------------------------------
4. Testing growing file using RandomAccessFile "rws"
- ---------------------------------------------------------------------
Chunk size: 4228 bytes
Count: 10000
Total time: 118010 ms
Writes per second: 84.739 writes/s
Time per chunk: 11.801 ms
Write bandwidth: 346.0KB/s
- ---------------------------------------------------------------------
5. Testing preallocated file using RandomAccessFile "rws"
- ---------------------------------------------------------------------
Chunk size: 4228 bytes
Count: 10000
Total time: 38827 ms
Writes per second: 257.553 writes/s
Time per chunk: 3.883 ms
Write bandwidth: 1.0MB/s
- ---------------------------------------------------------------------
6. Testing growing file using RandomAccessFile "rwd"
- ---------------------------------------------------------------------
Chunk size: 4228 bytes
Count: 10000
Total time: 117842 ms
Writes per second: 84.859 writes/s
Time per chunk: 11.784 ms
Write bandwidth: 346.0KB/s
- ---------------------------------------------------------------------
7. Testing preallocated file using RandomAccessFile "rwd"
- ---------------------------------------------------------------------
Chunk size: 4228 bytes
Count: 10000
Total time: 22387 ms
Writes per second: 446.688 writes/s
Time per chunk: 2.239 ms
Write bandwidth: 1.0MB/s
- ------------------------------------------------------------------------------
8. Testing growing file using direct buffer + FileChannel "rw" + force(true)
- ------------------------------------------------------------------------------
Chunk size: 4228 bytes
Count: 10000
Total time: 116817 ms
Writes per second: 85.604 writes/s
Time per chunk: 11.682 ms
Write bandwidth: 350.0KB/s
- -----------------------------------------------------------------------------------
9. Testing preallocated file using direct buffer + FileChannel "rw" + force(true)
- -----------------------------------------------------------------------------------
Chunk size: 4228 bytes
Count: 10000
Total time: 84007 ms
Writes per second: 119.038 writes/s
Time per chunk: 8.401 ms
Write bandwidth: 491.0KB/s
- ------------------------------------------------------------------------------
10. Testing growing file using direct buffer + FileChannel "rw" + force(false)
- ------------------------------------------------------------------------------
Chunk size: 4228 bytes
Count: 10000
Total time: 116254 ms
Writes per second: 86.019 writes/s
Time per chunk: 11.625 ms
Write bandwidth: 355.0KB/s
- ------------------------------------------------------------------------------
11. Testing preallocated file using direct buffer + FileChannel "rw" + force(false)
- ------------------------------------------------------------------------------
Chunk size: 4228 bytes
Count: 10000
Total time: 83028 ms
Writes per second: 120.441 writes/s
Time per chunk: 8.303 ms
Write bandwidth: 495.0KB/s
- ------------------------------------------------------------------------------
12. Testing growing file using direct buffer + FileChannel "rws"
- ------------------------------------------------------------------------------
Chunk size: 4228 bytes
Count: 10000
Total time: 117226 ms
Writes per second: 85.305 writes/s
Time per chunk: 11.723 ms
Write bandwidth: 350.0KB/s
- ------------------------------------------------------------------------------
13. Testing preallocated file using direct buffer + FileChannel "rws"
- ------------------------------------------------------------------------------
Chunk size: 4228 bytes
Count: 10000
Total time: 20279 ms
Writes per second: 493.121 writes/s
Time per chunk: 2.028 ms
Write bandwidth: 1.0MB/s
- ------------------------------------------------------------------------------
14. Testing growing file using direct buffer + FileChannel "rwd"
- ------------------------------------------------------------------------------
Chunk size: 4228 bytes
Count: 10000
Total time: 117511 ms
Writes per second: 85.098 writes/s
Time per chunk: 11.751 ms
Write bandwidth: 350.0KB/s
- ------------------------------------------------------------------------------
15. Testing preallocated file using direct buffer + FileChannel "rwd"
- ------------------------------------------------------------------------------
Chunk size: 4228 bytes
Count: 10000
Total time: 20724 ms
Writes per second: 482.532 writes/s
Time per chunk: 2.072 ms
Write bandwidth: 1.0MB/s
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.2.4 (MingW32)
Comment: Using GnuPG with Thunderbird - http://enigmail.mozdev.org
iQEVAwUBQTm+D3FDePgyse5HAQI3mAf/ekh37zW8MUH3roZbP0P8rSEDxrtJSAzl
LTY4BjXrP2hfWbz25L/Ki2eiXRyKpgXsXl2gRzQG7CVpFOza7i70WWXYVdp+cXB6
SYP3QyTjPiXp7Nv6Qx8kv77gEPHIIOY2wS/yyJmfOLEg2jnJ4GQOFHXFuaOAGcc0
oJLAtlqCU7vpQYpqaIRstU3lKhfKKdxdi6E/c1oHq168n7+3KELCpd5slaFfm5U1
rO8AwapTq2Al8jZ8TeCyf8rjkIv2kirbMmQldbUJ9yt6t/R6qUjkEdbT+K08Ql06
5IQxDvjPM/+rFgiu+9c3PNVUD0y4VbRWeyTNIJ/M/ZQlTxNL7L0RRw==
=ITMV
-----END PGP SIGNATURE-----