You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@directory.apache.org by Emmanuel Lécharny <el...@gmail.com> on 2012/03/16 10:26:12 UTC
Question about the Log file
Hi,
AFAICS, the og file contains a buffer which stores UserLogRecord until
we write them on disk. This allows the Log system to be fast, and also
allow a dedicated thread to flush the data on disk regularly.
So far, so good.
But I'm wondering if it would'nt be simpler to use a MemoryMappedFile
instead, as the Log file size is fixed. We will then let the underlying
OS to deal with flushes, not needing a dedicated thread to handle it.
Also MemoryMappedFile are faster than RandomFile, as it's working on
pages, which are managed by the OS (their size is depending on the OS
tuning). Last, not least, we won't need to dedicate a 4Mb buffer to
store temporary userRecords, as MemoryMappedFiles aren't using the JVM
memory.
The only modification would be to check that the file is not full, and
if so, we will just have to create a new one.
Concurrent access to the file are also protected by the
MemoryMappedFile, we don't need to do anything but to declare the kind
of access we want on the portion of the file we wanyt to access.
thoughts ?
--
Regards,
Cordialement,
Emmanuel Lécharny
www.iktek.com
Re: Question about the Log file
Posted by Selcuk AYA <ay...@gmail.com>.
On Fri, Mar 16, 2012 at 1:01 PM, Emmanuel Lécharny <el...@gmail.com> wrote:
> Le 3/16/12 7:07 PM, Selcuk AYA a écrit :
>
>> On Fri, Mar 16, 2012 at 2:26 AM, Emmanuel Lécharny<el...@gmail.com>
>> wrote:
>>>
>>> Hi,
>>>
>>> AFAICS, the og file contains a buffer which stores UserLogRecord until we
>>> write them on disk. This allows the Log system to be fast, and also allow
>>> a
>>> dedicated thread to flush the data on disk regularly.
>>>
>>> So far, so good.
>>>
>>> But I'm wondering if it would'nt be simpler to use a MemoryMappedFile
>>> instead, as the Log file size is fixed. We will then let the underlying
>>> OS
>>> to deal with flushes, not needing a dedicated thread to handle it. Also
>>> MemoryMappedFile are faster than RandomFile, as it's working on pages,
>>> which
>>> are managed by the OS (their size is depending on the OS tuning). Last,
>>> not
>>> least, we won't need to dedicate a 4Mb buffer to store temporary
>>> userRecords, as MemoryMappedFiles aren't using the JVM memory.
>>
>> the current implementation is flexible to work with any underlying
>> file system rather than being tied to a single implementation.
>
> MemoryMappedFile in Java are not OS dependent. It's a feature that has been
> added in Java 4, and it works on top of any FS.
>
>> Currently it is a random access file system. But memory mapped file
>> implementation should work as well. I am also thinking of using HDFS
>> files in the future.A couple of notes about some concerns:
>>
>> *I dont see 4MB being a big concern. This size can be tuned. It can
>> also be made zero if the underlying implementation is good to deal
>> with writes.
>> *I dont think a dedicated background thread is a problem either. If we
>> want, we can make user threads to do the log sync as they log the
>> records.
>>
>> Right now this is very low priority given the things we need to
>> implement to get the txns to work.
>
> Sure ! Just wanted to mention that, AFAIU, the current log with the buffer
> could probably be switched to a MMF with some extra benefits in the future.
>
> There are some area that need to be checked, like the file locks and how we
> should manage them.
>
> I'll create a JIRA mentionning that this is an area which may need to be
> investigated later, as it's not urgent.
>
> We discussed a lot with Alex about using MMF instead of other kind of file
> IO especially if we are to rewrite JDBM at some point.
>
> One more question, relative to the LogBuffer : I see it's flushed in two
> cases :
> - when the buffer is full (obviously !)
> - if a thread is calling the Log.sync() method.
>
> My understanding is that there will be a thread somewhere which will call
> this Log.sync() method periodically, right ?
that is true, background thread will do syncing. If I remember I
added Log.sync() for some testing as per your request but now it is
being called from several places in the modification code
path(probably to make some tests pass). This is part of the cleanup I
need to do later.
>
> One last thing : we will need to expose the configurable size so that it can
> be changed. ALl our configuration is stored in the cn=config branch, I will
> try to draft something on the wiki to gather all the configurable elements
> plus the suggested AT/OC needed to manipulate those elements.
>
> I'll be mostly offline this week-end, because I don't have any credit on my
> 3G phone, and also forgot to bring my phone charger. I'll be back on monday
> afternoon.
>
> Have a good week-end !
>
>
> --
> Regards,
> Cordialement,
> Emmanuel Lécharny
> www.iktek.com
>
Re: Question about the Log file
Posted by Emmanuel Lécharny <el...@gmail.com>.
Le 3/16/12 7:07 PM, Selcuk AYA a écrit :
> On Fri, Mar 16, 2012 at 2:26 AM, Emmanuel Lécharny<el...@gmail.com> wrote:
>> Hi,
>>
>> AFAICS, the og file contains a buffer which stores UserLogRecord until we
>> write them on disk. This allows the Log system to be fast, and also allow a
>> dedicated thread to flush the data on disk regularly.
>>
>> So far, so good.
>>
>> But I'm wondering if it would'nt be simpler to use a MemoryMappedFile
>> instead, as the Log file size is fixed. We will then let the underlying OS
>> to deal with flushes, not needing a dedicated thread to handle it. Also
>> MemoryMappedFile are faster than RandomFile, as it's working on pages, which
>> are managed by the OS (their size is depending on the OS tuning). Last, not
>> least, we won't need to dedicate a 4Mb buffer to store temporary
>> userRecords, as MemoryMappedFiles aren't using the JVM memory.
> the current implementation is flexible to work with any underlying
> file system rather than being tied to a single implementation.
MemoryMappedFile in Java are not OS dependent. It's a feature that has
been added in Java 4, and it works on top of any FS.
> Currently it is a random access file system. But memory mapped file
> implementation should work as well. I am also thinking of using HDFS
> files in the future.A couple of notes about some concerns:
>
> *I dont see 4MB being a big concern. This size can be tuned. It can
> also be made zero if the underlying implementation is good to deal
> with writes.
> *I dont think a dedicated background thread is a problem either. If we
> want, we can make user threads to do the log sync as they log the
> records.
>
> Right now this is very low priority given the things we need to
> implement to get the txns to work.
Sure ! Just wanted to mention that, AFAIU, the current log with the
buffer could probably be switched to a MMF with some extra benefits in
the future.
There are some area that need to be checked, like the file locks and how
we should manage them.
I'll create a JIRA mentionning that this is an area which may need to be
investigated later, as it's not urgent.
We discussed a lot with Alex about using MMF instead of other kind of
file IO especially if we are to rewrite JDBM at some point.
One more question, relative to the LogBuffer : I see it's flushed in two
cases :
- when the buffer is full (obviously !)
- if a thread is calling the Log.sync() method.
My understanding is that there will be a thread somewhere which will
call this Log.sync() method periodically, right ?
One last thing : we will need to expose the configurable size so that it
can be changed. ALl our configuration is stored in the cn=config branch,
I will try to draft something on the wiki to gather all the configurable
elements plus the suggested AT/OC needed to manipulate those elements.
I'll be mostly offline this week-end, because I don't have any credit on
my 3G phone, and also forgot to bring my phone charger. I'll be back on
monday afternoon.
Have a good week-end !
--
Regards,
Cordialement,
Emmanuel Lécharny
www.iktek.com
Re: Question about the Log file
Posted by Alex Karasulu <ak...@apache.org>.
On Fri, Mar 16, 2012 at 8:07 PM, Selcuk AYA <ay...@gmail.com> wrote:
> On Fri, Mar 16, 2012 at 2:26 AM, Emmanuel Lécharny <el...@gmail.com>
> wrote:
> > Hi,
> >
> > AFAICS, the og file contains a buffer which stores UserLogRecord until we
> > write them on disk. This allows the Log system to be fast, and also
> allow a
> > dedicated thread to flush the data on disk regularly.
> >
> > So far, so good.
> >
> > But I'm wondering if it would'nt be simpler to use a MemoryMappedFile
> > instead, as the Log file size is fixed. We will then let the underlying
> OS
> > to deal with flushes, not needing a dedicated thread to handle it. Also
> > MemoryMappedFile are faster than RandomFile, as it's working on pages,
> which
> > are managed by the OS (their size is depending on the OS tuning). Last,
> not
> > least, we won't need to dedicate a 4Mb buffer to store temporary
> > userRecords, as MemoryMappedFiles aren't using the JVM memory.
>
> the current implementation is flexible to work with any underlying
> file system rather than being tied to a single implementation.
> Currently it is a random access file system. But memory mapped file
> implementation should work as well. I am also thinking of using HDFS
> files in the future.A couple of notes about some concerns:
>
> *I dont see 4MB being a big concern. This size can be tuned. It can
> also be made zero if the underlying implementation is good to deal
> with writes.
> *I dont think a dedicated background thread is a problem either. If we
> want, we can make user threads to do the log sync as they log the
> records.
>
> Right now this is very low priority given the things we need to
> implement to get the txns to work.
>
>
I agree that this is lower priority than getting the branch working
correctly. This is an optimization IMO. I'd like to get things working and
then get some metrics on performance. Then we can start looking at
optimizations.
It's certainly worth mentioning, noting and discussing. Maybe we should put
these notes into JIRA so we can force ourselves to get back to asking these
questions. I'm always thinking about what performance gains we can get from
using memory mapped files but never had the chance to try it out. I'd love
to put it to the test when we get the chance after getting a full
implementation completed.
--
Best Regards,
-- Alex
Re: Question about the Log file
Posted by Selcuk AYA <ay...@gmail.com>.
On Fri, Mar 16, 2012 at 2:26 AM, Emmanuel Lécharny <el...@gmail.com> wrote:
> Hi,
>
> AFAICS, the og file contains a buffer which stores UserLogRecord until we
> write them on disk. This allows the Log system to be fast, and also allow a
> dedicated thread to flush the data on disk regularly.
>
> So far, so good.
>
> But I'm wondering if it would'nt be simpler to use a MemoryMappedFile
> instead, as the Log file size is fixed. We will then let the underlying OS
> to deal with flushes, not needing a dedicated thread to handle it. Also
> MemoryMappedFile are faster than RandomFile, as it's working on pages, which
> are managed by the OS (their size is depending on the OS tuning). Last, not
> least, we won't need to dedicate a 4Mb buffer to store temporary
> userRecords, as MemoryMappedFiles aren't using the JVM memory.
the current implementation is flexible to work with any underlying
file system rather than being tied to a single implementation.
Currently it is a random access file system. But memory mapped file
implementation should work as well. I am also thinking of using HDFS
files in the future.A couple of notes about some concerns:
*I dont see 4MB being a big concern. This size can be tuned. It can
also be made zero if the underlying implementation is good to deal
with writes.
*I dont think a dedicated background thread is a problem either. If we
want, we can make user threads to do the log sync as they log the
records.
Right now this is very low priority given the things we need to
implement to get the txns to work.
>
> The only modification would be to check that the file is not full, and if
> so, we will just have to create a new one.
>
> Concurrent access to the file are also protected by the MemoryMappedFile, we
> don't need to do anything but to declare the kind of access we want on the
> portion of the file we wanyt to access.
>
> thoughts ?
>
> --
> Regards,
> Cordialement,
> Emmanuel Lécharny
> www.iktek.com
>
thanks
Selcuk