You are viewing a plain text version of this content. The canonical link for it is here.
Posted to derby-dev@db.apache.org by Anders Morken <an...@stud.ntnu.no> on 2006/05/24 09:37:29 UTC
Re: [jira] Created: (DERBY-801) Allow parallel access to data files.
Øystein Grøvlen (JIRA):
> Allow parallel access to data files.
> ------------------------------------
>
> Key: DERBY-801
> URL: http://issues.apache.org/jira/browse/DERBY-801
> Project: Derby
> Type: Improvement
> Components: Performance, Store
> Versions: 10.0.2.0, 10.0.2.1, 10.1.1.0, 10.1.1.1, 10.1.1.2, 10.1.2.0, 10.1.2.1
> Environment: Any > Reporter: Øystein Grøvlen
>
>
> Derby currently serializes accesses to a data file. For example, the
> implementation of RAFContainer.readPage is as follows:
>
> synchronized (this) { // 'this' is a FileContainer, i.e. a file object
> fileData.seek(pageOffset); // fileData is a RandomAccessFile
> fileData.readFully(pageData, 0, pageSize);
> }
>
> I have experiemented with a patch where I have introduced several file
> descriptors (RandomAccessFile objects) per RAFContainer. These are
> used for reading. The principle is that when all readers are busy, a
> readPage request will create a new reader. (There is a maximum number
> of readers.) With this patch, throughput was improved by 50% on
> linux. For more discussion on this, see
>
> http://www.nabble.com/Derby-I-O-issues-during-checkpointing-t473523.html
>
> The challenge with the suggested approach is to make a mechanism to
> limit the number of open file descpriptors. Mike Matrigali has
> suggested to use the existing CacheManager infrastructure for this
> purpose. For a discussion on that, see:
>
> http://www.nabble.com/new-uses-for-basic-services-cache---looking-for-advice-t756863.html
I've played around a bit with a different approach - using the
FileChannel class from Java 1.4's new IO API. I've written a class
RAFContainer4 which extends RAFContainer and overrides the readPage and
writePage methods of that class to use read/write(ByteBuffer buf, long
postition) in FileChannel to access the container's file, without
synchronizing on the FileContainer during the read and write calls.
With a bit of hackery in BaseDataFileFactory#newContainerObject() this
class is then used instead of the regular RAFContainer on creation of
new RAFContainer objects when Derby runs in a 1.4+ JVM.
This approach gives the JVM and OS the opportunity to issue multiple
file operations concurrently, although we have no guarantees that this
will actually happen. This is JVM/OS dependent, but stracing the Sun
1.4.2_09 VM on Linux 2.6 shows that the VM now uses pread64()/pwrite64()
system calls instead of seek(), read() and write(). pread and pwrite
have similar semantics to the FileChannel#read/write(ByteBuffer buf,
long position) methods, and do not alter the file's seek() position, and
are supposed to be thread safe.
Of course only people running Derby on 1.4+ JVMs will have the
opportunity to benefit from this approach. As support for 1.3 is to be
deprecated this might not be much of an issue?
But anyway, I would like to see if this hack of mine actually works. I
see mentions of a "TPC-B like benchmark" in the threads Øystein links to
above, and wonder if that is something Sun internal, or if it's a
publicly available benchmark implementation that I can get my grubby
little paws on and try out this patch with? =)
Thanks,
--
Anders Morken
My opinions may have changed, but not the fact that I am right!
Re: [jira] Created: (DERBY-801) Allow parallel access to data files.
Posted by Anders Morken <an...@stud.ntnu.no>.
Oystein Grovlen - Sun Norway:
> >Of course only people running Derby on 1.4+ JVMs will have the
> >opportunity to benefit from this approach. As support for 1.3 is to be
> >deprecated this might not be much of an issue?
>
> If this means that 1.3 still works, but the old way, I think this is
> acceptable.
It does. You can think of RAFContainer4 as an overlay on RAFContainer
- it only comes into play on 1.4+ JVMs: "if( JVMInfo.JDK_ID >=
JVMInfo.J2SE_14)".
The current code is quite cowardly, and falls back to the old
version if something goes wrong. =)
Thanks for the description of the benchmark, if I'm bored I might smash
together something based on it, but as I'm supposed to be preparing for
the exams next week instead of coding now that'll probably have to wait.
I've procrastinated enough as it is. ;)
I've put the patch in Jira (DERBY-801) after a little cleanup and
running storeall. Caveat: It needs some more work to be worthy of
consideration for the trunk, but at least it passes storeall. =)
Anyone interested is welcome to beat the c!!p out of it, maybe we can
turn it into something useful together? =)
Thanks,
--
Anders Morken
My opinions may have changed, but not the fact that I am right!
Re: [jira] Created: (DERBY-801) Allow parallel access to data files.
Posted by Oystein Grovlen - Sun Norway <Oy...@Sun.COM>.
Anders Morken wrote:
> I've played around a bit with a different approach - using the
> FileChannel class from Java 1.4's new IO API. I've written a class
> RAFContainer4 which extends RAFContainer and overrides the readPage and
> writePage methods of that class to use read/write(ByteBuffer buf, long
> postition) in FileChannel to access the container's file, without
> synchronizing on the FileContainer during the read and write calls.
>
> With a bit of hackery in BaseDataFileFactory#newContainerObject() this
> class is then used instead of the regular RAFContainer on creation of
> new RAFContainer objects when Derby runs in a 1.4+ JVM.
>
> This approach gives the JVM and OS the opportunity to issue multiple
> file operations concurrently, although we have no guarantees that this
> will actually happen. This is JVM/OS dependent, but stracing the Sun
> 1.4.2_09 VM on Linux 2.6 shows that the VM now uses pread64()/pwrite64()
> system calls instead of seek(), read() and write(). pread and pwrite
> have similar semantics to the FileChannel#read/write(ByteBuffer buf,
> long position) methods, and do not alter the file's seek() position, and
> are supposed to be thread safe.
Great, Anders. This looks like a promising idea.
>
> Of course only people running Derby on 1.4+ JVMs will have the
> opportunity to benefit from this approach. As support for 1.3 is to be
> deprecated this might not be much of an issue?
If this means that 1.3 still works, but the old way, I think this is
acceptable.
> But anyway, I would like to see if this hack of mine actually works. I
> see mentions of a "TPC-B like benchmark" in the threads Øystein links to
> above, and wonder if that is something Sun internal, or if it's a
> publicly available benchmark implementation that I can get my grubby
> little paws on and try out this patch with? =)
The actual code is something we have developed internally here, and I am
not sure we will have time to make it available any time soon. If you
make a patch of your changes, I should be able to test this next week.
If you want to try this out yourself, I think you should be able to make
a sufficient test client quickly. (However, it will take some time to
create the large database). What I used was:
1. A database much larger than physical memory on computer. I think I
had around 17 GB of data including indexes.
2. A large page cache. I used 500MB on a computer with 2GB RAM.
3. Log device on separate disk. (I.e., you need a computer with 2 disks.)
4. I used TPC-B like transactions, but I would accept that any load
where transactions access records in a large table by primary key should
work. Make sure to try to avoid frequent lock conflicts or deadlocks.
(E.g., two random accesses to the same table within a transaction is
dead-lock prone)
5. Multi-threaded application where all threads ran the same type of
transaction back-to-back. (I had 20 threads). Our application prints
throughput per thread and total throughput for every 10 second interval
and an average at the end.
6. Run for at least 30 mins to allow for several checkpoints to happen
during your run. (I ran for 1 hour).
A short description of our TPC-B like app:
4 tables:
branch(bid int, bbal int, junk char(92), primary key(bid))
teller(tid int, bid, int , tbal int, junk char(88), primary key(tid))
account(aid int, bid int, abal int, junk char(88), primary key(aid))
history(aid int, tid int, bid int, delta int, tstamp timestamp, primary
key(tstamp,aid,delta))
All primary keys are numbered from 0 to n-1, where n are the number of
rows in the table. *bal columns are initially 0.
teller: bid=tid/10, account:bid=aid/100000
I had 1000 branches, 10000 tellers and 100 million accounts. history
table is initially empty
A transaction has 5 statements:
update account set abal = abal+? where aid=? and bid=?
insert into history values (?, ?, ?, ?, CURRENT_TIMESTAMP)
update teller set tbal = tbal+? where tid=? and bid=?
update branch set bbal = bbal+? where bid=?
select abal from account where aid=y
In TPC-B there are some rules about selecting teller and account where a
certain percentage of transactions will use a teller from a different
branch than the branch of the account, but I do not think that will
matter here. I suggest by random determining a balance change (we use a
number between -1 million and 1 million), and a random aid and determine
tid and bid based on the aid. (I.e., tid=aid/10000, bid=tid/100000).
--
Øystein
Re: [jira] Created: (DERBY-801) Allow parallel access to data files.
Posted by Mike Matrigali <mi...@sbcglobal.net>.
I have not reviewed the code, but your description of the approach and
the results posted later indicate this is a good path for this problem.
I would not worry about pre-1.4 jvm's not getting the benefit. As long
as the system still works then the separate module appoach is consistent
with other parts of the system.
In addition to showing that the fix helps the dual i/o case, it would
be nice to compare the pre and post performance of single user I/O just
to insure the new interfaces are not significantly slower than the old.
Anders Morken wrote:
> Øystein Grøvlen (JIRA):
>
>>Allow parallel access to data files.
>>------------------------------------
>>
>> Key: DERBY-801
>> URL: http://issues.apache.org/jira/browse/DERBY-801
>> Project: Derby
>> Type: Improvement
>> Components: Performance, Store
>> Versions: 10.0.2.0, 10.0.2.1, 10.1.1.0, 10.1.1.1, 10.1.1.2, 10.1.2.0, 10.1.2.1
>> Environment: Any > Reporter: Øystein Grøvlen
>>
>>
>>Derby currently serializes accesses to a data file. For example, the
>>implementation of RAFContainer.readPage is as follows:
>>
>> synchronized (this) { // 'this' is a FileContainer, i.e. a file object
>> fileData.seek(pageOffset); // fileData is a RandomAccessFile
>> fileData.readFully(pageData, 0, pageSize);
>> }
>>
>>I have experiemented with a patch where I have introduced several file
>>descriptors (RandomAccessFile objects) per RAFContainer. These are
>>used for reading. The principle is that when all readers are busy, a
>>readPage request will create a new reader. (There is a maximum number
>>of readers.) With this patch, throughput was improved by 50% on
>>linux. For more discussion on this, see
>>
>>http://www.nabble.com/Derby-I-O-issues-during-checkpointing-t473523.html
>>
>>The challenge with the suggested approach is to make a mechanism to
>>limit the number of open file descpriptors. Mike Matrigali has
>>suggested to use the existing CacheManager infrastructure for this
>>purpose. For a discussion on that, see:
>>
>>http://www.nabble.com/new-uses-for-basic-services-cache---looking-for-advice-t756863.html
>
>
> I've played around a bit with a different approach - using the
> FileChannel class from Java 1.4's new IO API. I've written a class
> RAFContainer4 which extends RAFContainer and overrides the readPage and
> writePage methods of that class to use read/write(ByteBuffer buf, long
> postition) in FileChannel to access the container's file, without
> synchronizing on the FileContainer during the read and write calls.
>
> With a bit of hackery in BaseDataFileFactory#newContainerObject() this
> class is then used instead of the regular RAFContainer on creation of
> new RAFContainer objects when Derby runs in a 1.4+ JVM.
>
> This approach gives the JVM and OS the opportunity to issue multiple
> file operations concurrently, although we have no guarantees that this
> will actually happen. This is JVM/OS dependent, but stracing the Sun
> 1.4.2_09 VM on Linux 2.6 shows that the VM now uses pread64()/pwrite64()
> system calls instead of seek(), read() and write(). pread and pwrite
> have similar semantics to the FileChannel#read/write(ByteBuffer buf,
> long position) methods, and do not alter the file's seek() position, and
> are supposed to be thread safe.
>
> Of course only people running Derby on 1.4+ JVMs will have the
> opportunity to benefit from this approach. As support for 1.3 is to be
> deprecated this might not be much of an issue?
>
> But anyway, I would like to see if this hack of mine actually works. I
> see mentions of a "TPC-B like benchmark" in the threads Øystein links to
> above, and wonder if that is something Sun internal, or if it's a
> publicly available benchmark implementation that I can get my grubby
> little paws on and try out this patch with? =)
>
> Thanks,