You are viewing a plain text version of this content. The canonical link for it is here.
Posted to derby-dev@db.apache.org by "Anders Morken (JIRA)" <ji...@apache.org> on 2006/11/15 16:56:42 UTC
[jira] Resolved: (DERBY-801) Allow parallel access to data files.
[ http://issues.apache.org/jira/browse/DERBY-801?page=all ]
Anders Morken resolved DERBY-801.
---------------------------------
Resolution: Fixed
OK, I'm marking this one as resolved, then. I've logged DERBY-2086 to remind us to look into some kind of resource pooling mechanism for encryption buffers and similar. I guess this isn't the only place where a pooling mechanism could be useful, so I intentionally worded it in a generic fashion.
Regarding enhancing the StorageFactory interfaces to support general parallel access, I'm not sure I understand Suresh's intention completely. Implementing support for concurrent IO is specific to the different container types - and it may not even be feasible for compressed containers that require on decompression for seeks. Suresh, maybe you could explain it in a new Jira issue? ;-)
As I considered earlier, we could enhance the StorageRandomAccessFile interface to add getChannel() support, but even that would apply only to random access files backed by real files that do support getChannel - and we'd have to throw in some JVM-dependent trickery as well, since getChannel() in RandomAccessFile only exists in Java 1.4+. (What's the timeframe on deprecating 1.3 support? I guess it could be a while? =)
Anyway, thanks to everyone who helped me on this one. =)
> Allow parallel access to data files.
> ------------------------------------
>
> Key: DERBY-801
> URL: http://issues.apache.org/jira/browse/DERBY-801
> Project: Derby
> Issue Type: Improvement
> Components: Performance, Store
> Affects Versions: 10.0.2.0, 10.0.2.1, 10.1.1.0, 10.1.2.1
> Environment: Any
> Reporter: Øystein Grøvlen
> Assigned To: Anders Morken
> Fix For: 10.3.0.0
>
> Attachments: DERBY-801-6.patch, DERBY-801-7.patch, DERBY-801-v2.patch, DERBY-801-v3.patch, DERBY-801-v4.patch, DERBY-801-v5.patch, NIO-RAFContainer-v1.patch
>
>
> Derby currently serializes accesses to a data file. For example, the
> implementation of RAFContainer.readPage is as follows:
> synchronized (this) { // 'this' is a FileContainer, i.e. a file object
> fileData.seek(pageOffset); // fileData is a RandomAccessFile
> fileData.readFully(pageData, 0, pageSize);
> }
> I have experiemented with a patch where I have introduced several file
> descriptors (RandomAccessFile objects) per RAFContainer. These are
> used for reading. The principle is that when all readers are busy, a
> readPage request will create a new reader. (There is a maximum number
> of readers.) With this patch, throughput was improved by 50% on
> linux. For more discussion on this, see
> http://www.nabble.com/Derby-I-O-issues-during-checkpointing-t473523.html
> The challenge with the suggested approach is to make a mechanism to
> limit the number of open file descpriptors. Mike Matrigali has
> suggested to use the existing CacheManager infrastructure for this
> purpose. For a discussion on that, see:
> http://www.nabble.com/new-uses-for-basic-services-cache---looking-for-advice-t756863.html
--
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators: http://issues.apache.org/jira/secure/Administrators.jspa
-
For more information on JIRA, see: http://www.atlassian.com/software/jira
Re: [jira] Resolved: (DERBY-801) Allow parallel access to data files.
Posted by Knut Anders Hatlen <Kn...@Sun.COM>.
Mike Matrigali <mi...@sbcglobal.net> writes:
> One can assume jdk 1.4 or higher in the trunk, it was agreed to
> deprecate jdk1.3 starting with then next release (likely to be
> called 10.3). Of course, do not backport any such change to 10.2
> or before.
How does this affect J2ME? Are most of the jdk 1.4 features supported,
like FileChannel for this particular issue?
--
Knut Anders
Re: [jira] Resolved: (DERBY-801) Allow parallel access to data files.
Posted by Mike Matrigali <mi...@sbcglobal.net>.
One can assume jdk 1.4 or higher in the trunk, it was agreed to
deprecate jdk1.3 starting with then next release (likely to be
called 10.3). Of course, do not backport any such change to 10.2
or before.
Anders Morken (JIRA) wrote:
> [ http://issues.apache.org/jira/browse/DERBY-801?page=all ]
>
> Anders Morken resolved DERBY-801.
> ---------------------------------
>
> Resolution: Fixed
>
> OK, I'm marking this one as resolved, then. I've logged DERBY-2086 to remind us to look into some kind of resource pooling mechanism for encryption buffers and similar. I guess this isn't the only place where a pooling mechanism could be useful, so I intentionally worded it in a generic fashion.
>
> Regarding enhancing the StorageFactory interfaces to support general parallel access, I'm not sure I understand Suresh's intention completely. Implementing support for concurrent IO is specific to the different container types - and it may not even be feasible for compressed containers that require on decompression for seeks. Suresh, maybe you could explain it in a new Jira issue? ;-)
>
> As I considered earlier, we could enhance the StorageRandomAccessFile interface to add getChannel() support, but even that would apply only to random access files backed by real files that do support getChannel - and we'd have to throw in some JVM-dependent trickery as well, since getChannel() in RandomAccessFile only exists in Java 1.4+. (What's the timeframe on deprecating 1.3 support? I guess it could be a while? =)
>
> Anyway, thanks to everyone who helped me on this one. =)
>
>
>>Allow parallel access to data files.
>>------------------------------------
>>
>> Key: DERBY-801
>> URL: http://issues.apache.org/jira/browse/DERBY-801
>> Project: Derby
>> Issue Type: Improvement
>> Components: Performance, Store
>> Affects Versions: 10.0.2.0, 10.0.2.1, 10.1.1.0, 10.1.2.1
>> Environment: Any
>> Reporter: Øystein Grøvlen
>> Assigned To: Anders Morken
>> Fix For: 10.3.0.0
>>
>> Attachments: DERBY-801-6.patch, DERBY-801-7.patch, DERBY-801-v2.patch, DERBY-801-v3.patch, DERBY-801-v4.patch, DERBY-801-v5.patch, NIO-RAFContainer-v1.patch
>>
>>
>>Derby currently serializes accesses to a data file. For example, the
>>implementation of RAFContainer.readPage is as follows:
>> synchronized (this) { // 'this' is a FileContainer, i.e. a file object
>> fileData.seek(pageOffset); // fileData is a RandomAccessFile
>> fileData.readFully(pageData, 0, pageSize);
>> }
>>I have experiemented with a patch where I have introduced several file
>>descriptors (RandomAccessFile objects) per RAFContainer. These are
>>used for reading. The principle is that when all readers are busy, a
>>readPage request will create a new reader. (There is a maximum number
>>of readers.) With this patch, throughput was improved by 50% on
>>linux. For more discussion on this, see
>>http://www.nabble.com/Derby-I-O-issues-during-checkpointing-t473523.html
>>The challenge with the suggested approach is to make a mechanism to
>>limit the number of open file descpriptors. Mike Matrigali has
>>suggested to use the existing CacheManager infrastructure for this
>>purpose. For a discussion on that, see:
>>http://www.nabble.com/new-uses-for-basic-services-cache---looking-for-advice-t756863.html
>
>
Re: [jira] Resolved: (DERBY-801) Allow parallel access to data files.
Posted by Suresh Thalamati <su...@gmail.com>.
Anders Morken (JIRA) wrote:
> [ http://issues.apache.org/jira/browse/DERBY-801?page=all ]
>
> Anders Morken resolved DERBY-801.
> ---------------------------------
>
> Resolution: Fixed
>
> OK, I'm marking this one as resolved, then. I've logged DERBY-2086 to remind us to look into some kind of
> resource pooling mechanism for encryption buffers and similar.
> I guess this isn't the only place where a pooling mechanism could be useful,
> so I intentionally worded it in a generic fashion.
>
> Regarding enhancing the StorageFactory interfaces to support general parallel access, I'm not sure I
> understand Suresh's intention completely. Implementing support for concurrent IO is specific to the
> different container types - and it may not even be feasible for compressed containers that require on
> decompression for seeks. Suresh, maybe you could explain it in a new Jira issue? ;-)
>
> As I considered earlier, we could enhance the StorageRandomAccessFile interface to add
> getChannel() support, but even that would apply only to random access files backed by real files
> that do support getChannel - and we'd have to throw in some JVM-dependent trickery as well,
> since getChannel() in RandomAccessFile only exists in Java 1.4+.
> (What's the timeframe on deprecating 1.3 support? I guess it could be a while? =)
>
Not sure when I made such a generic statement about storage factory
parallel access support, sorry if I had confused you. One thing I
wanted to work is if some one replaces the storage factory with
another implementation, new readPage/WritePage you added gets executed
instead of the old ones that allowed only single read/write of a
container file at a time.
I think we both are talking the same thing , As you say all that might
be needed is to add getChannel() method to the StorageRandomAccessFile
Interface and use it it in the RafContainer4.java instead of checking
for fileData is an instanceof java.io.RandomAccessFile.
Thanks
-suresh