You are viewing a plain text version of this content. The canonical link for it is here.
Posted to solr-user@lucene.apache.org by Rallavagu <ra...@gmail.com> on 2015/10/27 01:47:29 UTC
Solr hard commit
All,
Are memory mapped files (mmap) flushed to disk during "hard commit"? If
yes, should we disable OS level (Linux for example) memory mapped flush?
I am referring to following for mmap files for Lucene/Solr
http://blog.thetaphi.de/2012/07/use-lucenes-mmapdirectory-on-64bit.html
Linux level flush
http://www.cyberciti.biz/faq/linux-stop-flushing-of-mmaped-pages-to-disk/
Solr's hard and soft commit
https://lucidworks.com/blog/2013/08/23/understanding-transaction-logs-softcommit-and-commit-in-sorlcloud/
Thanks in advance.
Re: Solr hard commit
Posted by Alessandro Benedetti <ab...@apache.org>.
Just thinking loud, but Mapping is a OS feature.
For example , if a recovery is happening ( and it involves less than 100
docs of difference), the Tlog will be read to reproduce the data.
This will cause the Tlog to be accessed and ideally memory mapped.
Am I correct ?
Cheers
On 28 October 2015 at 01:56, Erick Erickson <er...@gmail.com> wrote:
> yep. Although I won't guarantee that the tlog won't be MMapped
> even if this is turned off.
>
> On Tue, Oct 27, 2015 at 4:21 PM, Rallavagu <ra...@gmail.com> wrote:
> > Is it related to this config?
> >
> > <!-- If true, IndexReaders will be opened/reopened from the IndexWriter
> > instead of from the Directory. Hosts in a master/slave setup
> > should have this set to false while those in a SolrCloud
> > cluster need to be set to true. Default: true
> > -->
> > <!--
> > <nrtMode>true</nrtMode>
> >
> > this will be true by default.
> >
> > On 10/27/15 1:24 PM, Erick Erickson wrote:
> >>
> >> Hmm, the tlog. AFAIK, that's because of
> >> "real time get" (Yonik/Mark/Whoever, please
> >> correct if wrong).
> >>
> >> This is a feature where when fetching a document as
> >> the result of a search it insures that you get the most
> >> recent version whether it's been committed or not.....
> >>
> >> The tlog isn't relevant for searching however.
> >>
> >> Best,
> >> Erick
> >>
> >> On Tue, Oct 27, 2015 at 8:56 AM, Rallavagu <ra...@gmail.com> wrote:
> >>>
> >>>
> >>>
> >>> On 10/27/15 8:43 AM, Erick Erickson wrote:
> >>>>
> >>>>
> >>>> bq: So, the updated file(s) on the disk automatically read into memory
> >>>> as they are Memory mapped?
> >>>
> >>>
> >>> Yes.
> >>>>
> >>>>
> >>>>
> >>>> Not quite sure why you care, curiosity or is there something you're
> >>>> trying to accomplish?
> >>>
> >>>
> >>> This is out of curiosity. So, I can get better understanding of Solr's
> >>> memory usage (heap & mmap).
> >>>
> >>>>
> >>>> The contents of the index's segment files are read into virtual memory
> >>>> by MMapDirectory as needed to satisfy queries. Which is the point of
> >>>> autowarming BTW.
> >>>
> >>>
> >>>
> >>> Ok. But, I have noticed that even "tlog" files are memory mapped
> (output
> >>> from "lsof") in addition to all other files under "data" directory.
> >>>
> >>>>
> >>>> commit in the following is either hard commit with openSearcher=true
> >>>> or soft commit.
> >>>
> >>>
> >>>
> >>> Hard commit is setup with openSearcher=false and softCommit is setup
> for
> >>> every 2 min.
> >>>
> >>>>
> >>>> Segments that have been created (closed actually) after the last
> >>>> commit are _not_ read at all until the next searcher is opened via
> >>>> another commit. Nothing is done with these new segments before the new
> >>>> searcher is opened which you control with your commit strategy.
> >>>
> >>>
> >>>
> >>> I see. Thanks for the insight.
> >>>
> >>>>
> >>>> Best,
> >>>> Erick
> >>>>
> >>>> On Mon, Oct 26, 2015 at 9:07 PM, Rallavagu <ra...@gmail.com>
> wrote:
> >>>>>
> >>>>>
> >>>>> Erick, Thanks for clarification. I was under impression that
> >>>>> MMapDirectory
> >>>>> is being used for both read/write operations. Now, I see how it is
> >>>>> being
> >>>>> used. Essentially, it only reads from MMapDirectory and writes
> directly
> >>>>> to
> >>>>> disk. So, the updated file(s) on the disk automatically read into
> >>>>> memory
> >>>>> as
> >>>>> they are Memory mapped?
> >>>>>
> >>>>> On 10/26/15 8:43 PM, Erick Erickson wrote:
> >>>>>>
> >>>>>>
> >>>>>>
> >>>>>> You're really looking at this backwards. The MMapDirectory stuff is
> >>>>>> for Solr (Lucene, really) _reading_ data from closed segment files.
> >>>>>>
> >>>>>> When indexing, there are internal memory structures that are flushed
> >>>>>> to disk on commit, but these have nothing to do with MMapDirectory.
> >>>>>>
> >>>>>> So the question is really moot ;)
> >>>>>>
> >>>>>> Best,
> >>>>>> Erick
> >>>>>>
> >>>>>> On Mon, Oct 26, 2015 at 5:47 PM, Rallavagu <ra...@gmail.com>
> >>>>>> wrote:
> >>>>>>>
> >>>>>>>
> >>>>>>>
> >>>>>>> All,
> >>>>>>>
> >>>>>>> Are memory mapped files (mmap) flushed to disk during "hard
> commit"?
> >>>>>>> If
> >>>>>>> yes,
> >>>>>>> should we disable OS level (Linux for example) memory mapped flush?
> >>>>>>>
> >>>>>>> I am referring to following for mmap files for Lucene/Solr
> >>>>>>>
> >>>>>>>
> >>>>>>>
> http://blog.thetaphi.de/2012/07/use-lucenes-mmapdirectory-on-64bit.html
> >>>>>>>
> >>>>>>> Linux level flush
> >>>>>>>
> >>>>>>>
> >>>>>>>
> >>>>>>>
> http://www.cyberciti.biz/faq/linux-stop-flushing-of-mmaped-pages-to-disk/
> >>>>>>>
> >>>>>>> Solr's hard and soft commit
> >>>>>>>
> >>>>>>>
> >>>>>>>
> >>>>>>>
> >>>>>>>
> https://lucidworks.com/blog/2013/08/23/understanding-transaction-logs-softcommit-and-commit-in-sorlcloud/
> >>>>>>>
> >>>>>>> Thanks in advance.
>
--
--------------------------
Benedetti Alessandro
Visiting card : http://about.me/alessandro_benedetti
"Tyger, tyger burning bright
In the forests of the night,
What immortal hand or eye
Could frame thy fearful symmetry?"
William Blake - Songs of Experience -1794 England
Re: Solr hard commit
Posted by Erick Erickson <er...@gmail.com>.
yep. Although I won't guarantee that the tlog won't be MMapped
even if this is turned off.
On Tue, Oct 27, 2015 at 4:21 PM, Rallavagu <ra...@gmail.com> wrote:
> Is it related to this config?
>
> <!-- If true, IndexReaders will be opened/reopened from the IndexWriter
> instead of from the Directory. Hosts in a master/slave setup
> should have this set to false while those in a SolrCloud
> cluster need to be set to true. Default: true
> -->
> <!--
> <nrtMode>true</nrtMode>
>
> this will be true by default.
>
> On 10/27/15 1:24 PM, Erick Erickson wrote:
>>
>> Hmm, the tlog. AFAIK, that's because of
>> "real time get" (Yonik/Mark/Whoever, please
>> correct if wrong).
>>
>> This is a feature where when fetching a document as
>> the result of a search it insures that you get the most
>> recent version whether it's been committed or not.....
>>
>> The tlog isn't relevant for searching however.
>>
>> Best,
>> Erick
>>
>> On Tue, Oct 27, 2015 at 8:56 AM, Rallavagu <ra...@gmail.com> wrote:
>>>
>>>
>>>
>>> On 10/27/15 8:43 AM, Erick Erickson wrote:
>>>>
>>>>
>>>> bq: So, the updated file(s) on the disk automatically read into memory
>>>> as they are Memory mapped?
>>>
>>>
>>> Yes.
>>>>
>>>>
>>>>
>>>> Not quite sure why you care, curiosity or is there something you're
>>>> trying to accomplish?
>>>
>>>
>>> This is out of curiosity. So, I can get better understanding of Solr's
>>> memory usage (heap & mmap).
>>>
>>>>
>>>> The contents of the index's segment files are read into virtual memory
>>>> by MMapDirectory as needed to satisfy queries. Which is the point of
>>>> autowarming BTW.
>>>
>>>
>>>
>>> Ok. But, I have noticed that even "tlog" files are memory mapped (output
>>> from "lsof") in addition to all other files under "data" directory.
>>>
>>>>
>>>> commit in the following is either hard commit with openSearcher=true
>>>> or soft commit.
>>>
>>>
>>>
>>> Hard commit is setup with openSearcher=false and softCommit is setup for
>>> every 2 min.
>>>
>>>>
>>>> Segments that have been created (closed actually) after the last
>>>> commit are _not_ read at all until the next searcher is opened via
>>>> another commit. Nothing is done with these new segments before the new
>>>> searcher is opened which you control with your commit strategy.
>>>
>>>
>>>
>>> I see. Thanks for the insight.
>>>
>>>>
>>>> Best,
>>>> Erick
>>>>
>>>> On Mon, Oct 26, 2015 at 9:07 PM, Rallavagu <ra...@gmail.com> wrote:
>>>>>
>>>>>
>>>>> Erick, Thanks for clarification. I was under impression that
>>>>> MMapDirectory
>>>>> is being used for both read/write operations. Now, I see how it is
>>>>> being
>>>>> used. Essentially, it only reads from MMapDirectory and writes directly
>>>>> to
>>>>> disk. So, the updated file(s) on the disk automatically read into
>>>>> memory
>>>>> as
>>>>> they are Memory mapped?
>>>>>
>>>>> On 10/26/15 8:43 PM, Erick Erickson wrote:
>>>>>>
>>>>>>
>>>>>>
>>>>>> You're really looking at this backwards. The MMapDirectory stuff is
>>>>>> for Solr (Lucene, really) _reading_ data from closed segment files.
>>>>>>
>>>>>> When indexing, there are internal memory structures that are flushed
>>>>>> to disk on commit, but these have nothing to do with MMapDirectory.
>>>>>>
>>>>>> So the question is really moot ;)
>>>>>>
>>>>>> Best,
>>>>>> Erick
>>>>>>
>>>>>> On Mon, Oct 26, 2015 at 5:47 PM, Rallavagu <ra...@gmail.com>
>>>>>> wrote:
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> All,
>>>>>>>
>>>>>>> Are memory mapped files (mmap) flushed to disk during "hard commit"?
>>>>>>> If
>>>>>>> yes,
>>>>>>> should we disable OS level (Linux for example) memory mapped flush?
>>>>>>>
>>>>>>> I am referring to following for mmap files for Lucene/Solr
>>>>>>>
>>>>>>>
>>>>>>> http://blog.thetaphi.de/2012/07/use-lucenes-mmapdirectory-on-64bit.html
>>>>>>>
>>>>>>> Linux level flush
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> http://www.cyberciti.biz/faq/linux-stop-flushing-of-mmaped-pages-to-disk/
>>>>>>>
>>>>>>> Solr's hard and soft commit
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> https://lucidworks.com/blog/2013/08/23/understanding-transaction-logs-softcommit-and-commit-in-sorlcloud/
>>>>>>>
>>>>>>> Thanks in advance.
Re: Solr hard commit
Posted by Rallavagu <ra...@gmail.com>.
Is it related to this config?
<!-- If true, IndexReaders will be opened/reopened from the IndexWriter
instead of from the Directory. Hosts in a master/slave setup
should have this set to false while those in a SolrCloud
cluster need to be set to true. Default: true
-->
<!--
<nrtMode>true</nrtMode>
this will be true by default.
On 10/27/15 1:24 PM, Erick Erickson wrote:
> Hmm, the tlog. AFAIK, that's because of
> "real time get" (Yonik/Mark/Whoever, please
> correct if wrong).
>
> This is a feature where when fetching a document as
> the result of a search it insures that you get the most
> recent version whether it's been committed or not.....
>
> The tlog isn't relevant for searching however.
>
> Best,
> Erick
>
> On Tue, Oct 27, 2015 at 8:56 AM, Rallavagu <ra...@gmail.com> wrote:
>>
>>
>> On 10/27/15 8:43 AM, Erick Erickson wrote:
>>>
>>> bq: So, the updated file(s) on the disk automatically read into memory
>>> as they are Memory mapped?
>>
>> Yes.
>>>
>>>
>>> Not quite sure why you care, curiosity or is there something you're
>>> trying to accomplish?
>>
>> This is out of curiosity. So, I can get better understanding of Solr's
>> memory usage (heap & mmap).
>>
>>>
>>> The contents of the index's segment files are read into virtual memory
>>> by MMapDirectory as needed to satisfy queries. Which is the point of
>>> autowarming BTW.
>>
>>
>> Ok. But, I have noticed that even "tlog" files are memory mapped (output
>> from "lsof") in addition to all other files under "data" directory.
>>
>>>
>>> commit in the following is either hard commit with openSearcher=true
>>> or soft commit.
>>
>>
>> Hard commit is setup with openSearcher=false and softCommit is setup for
>> every 2 min.
>>
>>>
>>> Segments that have been created (closed actually) after the last
>>> commit are _not_ read at all until the next searcher is opened via
>>> another commit. Nothing is done with these new segments before the new
>>> searcher is opened which you control with your commit strategy.
>>
>>
>> I see. Thanks for the insight.
>>
>>>
>>> Best,
>>> Erick
>>>
>>> On Mon, Oct 26, 2015 at 9:07 PM, Rallavagu <ra...@gmail.com> wrote:
>>>>
>>>> Erick, Thanks for clarification. I was under impression that
>>>> MMapDirectory
>>>> is being used for both read/write operations. Now, I see how it is being
>>>> used. Essentially, it only reads from MMapDirectory and writes directly
>>>> to
>>>> disk. So, the updated file(s) on the disk automatically read into memory
>>>> as
>>>> they are Memory mapped?
>>>>
>>>> On 10/26/15 8:43 PM, Erick Erickson wrote:
>>>>>
>>>>>
>>>>> You're really looking at this backwards. The MMapDirectory stuff is
>>>>> for Solr (Lucene, really) _reading_ data from closed segment files.
>>>>>
>>>>> When indexing, there are internal memory structures that are flushed
>>>>> to disk on commit, but these have nothing to do with MMapDirectory.
>>>>>
>>>>> So the question is really moot ;)
>>>>>
>>>>> Best,
>>>>> Erick
>>>>>
>>>>> On Mon, Oct 26, 2015 at 5:47 PM, Rallavagu <ra...@gmail.com> wrote:
>>>>>>
>>>>>>
>>>>>> All,
>>>>>>
>>>>>> Are memory mapped files (mmap) flushed to disk during "hard commit"? If
>>>>>> yes,
>>>>>> should we disable OS level (Linux for example) memory mapped flush?
>>>>>>
>>>>>> I am referring to following for mmap files for Lucene/Solr
>>>>>>
>>>>>> http://blog.thetaphi.de/2012/07/use-lucenes-mmapdirectory-on-64bit.html
>>>>>>
>>>>>> Linux level flush
>>>>>>
>>>>>>
>>>>>> http://www.cyberciti.biz/faq/linux-stop-flushing-of-mmaped-pages-to-disk/
>>>>>>
>>>>>> Solr's hard and soft commit
>>>>>>
>>>>>>
>>>>>>
>>>>>> https://lucidworks.com/blog/2013/08/23/understanding-transaction-logs-softcommit-and-commit-in-sorlcloud/
>>>>>>
>>>>>> Thanks in advance.
Re: Solr hard commit
Posted by Erick Erickson <er...@gmail.com>.
Hmm, the tlog. AFAIK, that's because of
"real time get" (Yonik/Mark/Whoever, please
correct if wrong).
This is a feature where when fetching a document as
the result of a search it insures that you get the most
recent version whether it's been committed or not.....
The tlog isn't relevant for searching however.
Best,
Erick
On Tue, Oct 27, 2015 at 8:56 AM, Rallavagu <ra...@gmail.com> wrote:
>
>
> On 10/27/15 8:43 AM, Erick Erickson wrote:
>>
>> bq: So, the updated file(s) on the disk automatically read into memory
>> as they are Memory mapped?
>
> Yes.
>>
>>
>> Not quite sure why you care, curiosity or is there something you're
>> trying to accomplish?
>
> This is out of curiosity. So, I can get better understanding of Solr's
> memory usage (heap & mmap).
>
>>
>> The contents of the index's segment files are read into virtual memory
>> by MMapDirectory as needed to satisfy queries. Which is the point of
>> autowarming BTW.
>
>
> Ok. But, I have noticed that even "tlog" files are memory mapped (output
> from "lsof") in addition to all other files under "data" directory.
>
>>
>> commit in the following is either hard commit with openSearcher=true
>> or soft commit.
>
>
> Hard commit is setup with openSearcher=false and softCommit is setup for
> every 2 min.
>
>>
>> Segments that have been created (closed actually) after the last
>> commit are _not_ read at all until the next searcher is opened via
>> another commit. Nothing is done with these new segments before the new
>> searcher is opened which you control with your commit strategy.
>
>
> I see. Thanks for the insight.
>
>>
>> Best,
>> Erick
>>
>> On Mon, Oct 26, 2015 at 9:07 PM, Rallavagu <ra...@gmail.com> wrote:
>>>
>>> Erick, Thanks for clarification. I was under impression that
>>> MMapDirectory
>>> is being used for both read/write operations. Now, I see how it is being
>>> used. Essentially, it only reads from MMapDirectory and writes directly
>>> to
>>> disk. So, the updated file(s) on the disk automatically read into memory
>>> as
>>> they are Memory mapped?
>>>
>>> On 10/26/15 8:43 PM, Erick Erickson wrote:
>>>>
>>>>
>>>> You're really looking at this backwards. The MMapDirectory stuff is
>>>> for Solr (Lucene, really) _reading_ data from closed segment files.
>>>>
>>>> When indexing, there are internal memory structures that are flushed
>>>> to disk on commit, but these have nothing to do with MMapDirectory.
>>>>
>>>> So the question is really moot ;)
>>>>
>>>> Best,
>>>> Erick
>>>>
>>>> On Mon, Oct 26, 2015 at 5:47 PM, Rallavagu <ra...@gmail.com> wrote:
>>>>>
>>>>>
>>>>> All,
>>>>>
>>>>> Are memory mapped files (mmap) flushed to disk during "hard commit"? If
>>>>> yes,
>>>>> should we disable OS level (Linux for example) memory mapped flush?
>>>>>
>>>>> I am referring to following for mmap files for Lucene/Solr
>>>>>
>>>>> http://blog.thetaphi.de/2012/07/use-lucenes-mmapdirectory-on-64bit.html
>>>>>
>>>>> Linux level flush
>>>>>
>>>>>
>>>>> http://www.cyberciti.biz/faq/linux-stop-flushing-of-mmaped-pages-to-disk/
>>>>>
>>>>> Solr's hard and soft commit
>>>>>
>>>>>
>>>>>
>>>>> https://lucidworks.com/blog/2013/08/23/understanding-transaction-logs-softcommit-and-commit-in-sorlcloud/
>>>>>
>>>>> Thanks in advance.
Re: Solr hard commit
Posted by Rallavagu <ra...@gmail.com>.
On 10/27/15 8:43 AM, Erick Erickson wrote:
> bq: So, the updated file(s) on the disk automatically read into memory
> as they are Memory mapped?
Yes.
>
> Not quite sure why you care, curiosity or is there something you're
> trying to accomplish?
This is out of curiosity. So, I can get better understanding of Solr's
memory usage (heap & mmap).
>
> The contents of the index's segment files are read into virtual memory
> by MMapDirectory as needed to satisfy queries. Which is the point of
> autowarming BTW.
Ok. But, I have noticed that even "tlog" files are memory mapped (output
from "lsof") in addition to all other files under "data" directory.
>
> commit in the following is either hard commit with openSearcher=true
> or soft commit.
Hard commit is setup with openSearcher=false and softCommit is setup for
every 2 min.
>
> Segments that have been created (closed actually) after the last
> commit are _not_ read at all until the next searcher is opened via
> another commit. Nothing is done with these new segments before the new
> searcher is opened which you control with your commit strategy.
I see. Thanks for the insight.
>
> Best,
> Erick
>
> On Mon, Oct 26, 2015 at 9:07 PM, Rallavagu <ra...@gmail.com> wrote:
>> Erick, Thanks for clarification. I was under impression that MMapDirectory
>> is being used for both read/write operations. Now, I see how it is being
>> used. Essentially, it only reads from MMapDirectory and writes directly to
>> disk. So, the updated file(s) on the disk automatically read into memory as
>> they are Memory mapped?
>>
>> On 10/26/15 8:43 PM, Erick Erickson wrote:
>>>
>>> You're really looking at this backwards. The MMapDirectory stuff is
>>> for Solr (Lucene, really) _reading_ data from closed segment files.
>>>
>>> When indexing, there are internal memory structures that are flushed
>>> to disk on commit, but these have nothing to do with MMapDirectory.
>>>
>>> So the question is really moot ;)
>>>
>>> Best,
>>> Erick
>>>
>>> On Mon, Oct 26, 2015 at 5:47 PM, Rallavagu <ra...@gmail.com> wrote:
>>>>
>>>> All,
>>>>
>>>> Are memory mapped files (mmap) flushed to disk during "hard commit"? If
>>>> yes,
>>>> should we disable OS level (Linux for example) memory mapped flush?
>>>>
>>>> I am referring to following for mmap files for Lucene/Solr
>>>>
>>>> http://blog.thetaphi.de/2012/07/use-lucenes-mmapdirectory-on-64bit.html
>>>>
>>>> Linux level flush
>>>>
>>>> http://www.cyberciti.biz/faq/linux-stop-flushing-of-mmaped-pages-to-disk/
>>>>
>>>> Solr's hard and soft commit
>>>>
>>>>
>>>> https://lucidworks.com/blog/2013/08/23/understanding-transaction-logs-softcommit-and-commit-in-sorlcloud/
>>>>
>>>> Thanks in advance.
Re: Solr hard commit
Posted by Erick Erickson <er...@gmail.com>.
bq: So, the updated file(s) on the disk automatically read into memory
as they are Memory mapped?
Not quite sure why you care, curiosity or is there something you're
trying to accomplish?
The contents of the index's segment files are read into virtual memory
by MMapDirectory as needed to satisfy queries. Which is the point of
autowarming BTW.
commit in the following is either hard commit with openSearcher=true
or soft commit.
Segments that have been created (closed actually) after the last
commit are _not_ read at all until the next searcher is opened via
another commit. Nothing is done with these new segments before the new
searcher is opened which you control with your commit strategy.
Best,
Erick
On Mon, Oct 26, 2015 at 9:07 PM, Rallavagu <ra...@gmail.com> wrote:
> Erick, Thanks for clarification. I was under impression that MMapDirectory
> is being used for both read/write operations. Now, I see how it is being
> used. Essentially, it only reads from MMapDirectory and writes directly to
> disk. So, the updated file(s) on the disk automatically read into memory as
> they are Memory mapped?
>
> On 10/26/15 8:43 PM, Erick Erickson wrote:
>>
>> You're really looking at this backwards. The MMapDirectory stuff is
>> for Solr (Lucene, really) _reading_ data from closed segment files.
>>
>> When indexing, there are internal memory structures that are flushed
>> to disk on commit, but these have nothing to do with MMapDirectory.
>>
>> So the question is really moot ;)
>>
>> Best,
>> Erick
>>
>> On Mon, Oct 26, 2015 at 5:47 PM, Rallavagu <ra...@gmail.com> wrote:
>>>
>>> All,
>>>
>>> Are memory mapped files (mmap) flushed to disk during "hard commit"? If
>>> yes,
>>> should we disable OS level (Linux for example) memory mapped flush?
>>>
>>> I am referring to following for mmap files for Lucene/Solr
>>>
>>> http://blog.thetaphi.de/2012/07/use-lucenes-mmapdirectory-on-64bit.html
>>>
>>> Linux level flush
>>>
>>> http://www.cyberciti.biz/faq/linux-stop-flushing-of-mmaped-pages-to-disk/
>>>
>>> Solr's hard and soft commit
>>>
>>>
>>> https://lucidworks.com/blog/2013/08/23/understanding-transaction-logs-softcommit-and-commit-in-sorlcloud/
>>>
>>> Thanks in advance.
Re: Solr hard commit
Posted by Rallavagu <ra...@gmail.com>.
Erick, Thanks for clarification. I was under impression that
MMapDirectory is being used for both read/write operations. Now, I see
how it is being used. Essentially, it only reads from MMapDirectory and
writes directly to disk. So, the updated file(s) on the disk
automatically read into memory as they are Memory mapped?
On 10/26/15 8:43 PM, Erick Erickson wrote:
> You're really looking at this backwards. The MMapDirectory stuff is
> for Solr (Lucene, really) _reading_ data from closed segment files.
>
> When indexing, there are internal memory structures that are flushed
> to disk on commit, but these have nothing to do with MMapDirectory.
>
> So the question is really moot ;)
>
> Best,
> Erick
>
> On Mon, Oct 26, 2015 at 5:47 PM, Rallavagu <ra...@gmail.com> wrote:
>> All,
>>
>> Are memory mapped files (mmap) flushed to disk during "hard commit"? If yes,
>> should we disable OS level (Linux for example) memory mapped flush?
>>
>> I am referring to following for mmap files for Lucene/Solr
>>
>> http://blog.thetaphi.de/2012/07/use-lucenes-mmapdirectory-on-64bit.html
>>
>> Linux level flush
>>
>> http://www.cyberciti.biz/faq/linux-stop-flushing-of-mmaped-pages-to-disk/
>>
>> Solr's hard and soft commit
>>
>> https://lucidworks.com/blog/2013/08/23/understanding-transaction-logs-softcommit-and-commit-in-sorlcloud/
>>
>> Thanks in advance.
Re: Solr hard commit
Posted by Erick Erickson <er...@gmail.com>.
You're really looking at this backwards. The MMapDirectory stuff is
for Solr (Lucene, really) _reading_ data from closed segment files.
When indexing, there are internal memory structures that are flushed
to disk on commit, but these have nothing to do with MMapDirectory.
So the question is really moot ;)
Best,
Erick
On Mon, Oct 26, 2015 at 5:47 PM, Rallavagu <ra...@gmail.com> wrote:
> All,
>
> Are memory mapped files (mmap) flushed to disk during "hard commit"? If yes,
> should we disable OS level (Linux for example) memory mapped flush?
>
> I am referring to following for mmap files for Lucene/Solr
>
> http://blog.thetaphi.de/2012/07/use-lucenes-mmapdirectory-on-64bit.html
>
> Linux level flush
>
> http://www.cyberciti.biz/faq/linux-stop-flushing-of-mmaped-pages-to-disk/
>
> Solr's hard and soft commit
>
> https://lucidworks.com/blog/2013/08/23/understanding-transaction-logs-softcommit-and-commit-in-sorlcloud/
>
> Thanks in advance.