You are viewing a plain text version of this content. The canonical link for it is here.
Posted to solr-user@lucene.apache.org by RadhaJayalakshmi <rl...@inautix.co.in> on 2014/03/10 14:14:09 UTC
The way Autocommit works in solr - Wierd
Hi,
Brief Description of my application:
We have a java program which reads a flat file, and adds document to solr
using cloudsolrserver.
And we index for every 1000 documents(bulk indexing).
And the Autocommit setting of my application is:
<autoCommit>
<maxDocs>100000</maxDocs>
<openSearcher>false</openSearcher>
</autoCommit>
So after every 100,000 documents are indexed, engine should perform a
HardCommit/AutoCommit. But still the OpenSearcher will be false.
Once the file is fully read, we are issuing a commit() from the
CloudSolrServer class. So this by default opens a new Searcher.
Also, from the Log, i can see that three times, Autocommit is happenning.
and Only with the last/final Autocommit, opensearcher is set to true.
So, till now all looks fine and working as expected.
But one strange issue i observed during the course of indexing.
Now, as per the documentation, the data that is being indexed should first
get written into tlog. When the Autocommit is performed, the data will be
flushed to disk.
So only at three times, there should have been size difference in the /index
folder. All the time only the size of the /tlog folder should have been
changing
But actually happened is, all the time, i see the size of the /index folder
getting increased in parallel to the size of the /tlog folder.
Actually it is increasing to certain limit and coming down. Again increasing
and coming down to a point.
So Now the bigger doubt is have is, during hard commit, is the data getting
written into both /index or /tlog folder??
I am using solr 4.5.1.
Some one please clear me how the hardcommit works. I am asumming the
following sequence:
1. Reads the data and writes to tlog
2. During hardcommit, flushes the data from tlog to index. If openSearcher
is false, should not open a new searcher
3. In the end, once all the datas are indexed, it should open a new
searcher.
If not please explain me..
Thanks in Advance
Radha
--
View this message in context: http://lucene.472066.n3.nabble.com/The-way-Autocommit-works-in-solr-Wierd-tp4122558.html
Sent from the Solr - User mailing list archive at Nabble.com.
Re: The way Autocommit works in solr - Wierd
Posted by Erick Erickson <er...@gmail.com>.
You have to separate out a couple of things.
First, data gets written to segments _without_
the segment getting closed and _before_ you
commit. What happens is that when
ramBufferSizeMB in solrconfig.xml is exceeded,
its contents are flushed to the currently-opened
segment. The segment is _not_ closed, so
the fact that you see index size varying on disk
apart from commits is not surprising at all.
Second, segments are merged. During merging,
a complete copy is made of the segments being
merged. Only when the merge is successful are
the old segments deleted (well, and the searcher
holding them open is closed). This happens as
a background process.
All of which is a way of saying I think you're
being puzzled by the extra stuff going on and
can safely ignore it all...
Best,
Erick
On Mon, Mar 10, 2014 at 9:57 AM, Furkan KAMACI <fu...@gmail.com> wrote:
> Hi;
>
> Did you read here:
> http://searchhub.org/2013/08/23/understanding-transaction-logs-softcommit-and-commit-in-sorlcloud/
>
> Thanks;
> Furkan KAMACI
>
>
> 2014-03-10 15:14 GMT+02:00 RadhaJayalakshmi <rlakshminarayanan@inautix.co.in
>>:
>
>> Hi,
>>
>> Brief Description of my application:
>> We have a java program which reads a flat file, and adds document to solr
>> using cloudsolrserver.
>> And we index for every 1000 documents(bulk indexing).
>>
>> And the Autocommit setting of my application is:
>> <autoCommit>
>> <maxDocs>100000</maxDocs>
>> <openSearcher>false</openSearcher>
>> </autoCommit>
>>
>> So after every 100,000 documents are indexed, engine should perform a
>> HardCommit/AutoCommit. But still the OpenSearcher will be false.
>> Once the file is fully read, we are issuing a commit() from the
>> CloudSolrServer class. So this by default opens a new Searcher.
>>
>> Also, from the Log, i can see that three times, Autocommit is happenning.
>> and Only with the last/final Autocommit, opensearcher is set to true.
>>
>> So, till now all looks fine and working as expected.
>>
>> But one strange issue i observed during the course of indexing.
>> Now, as per the documentation, the data that is being indexed should first
>> get written into tlog. When the Autocommit is performed, the data will be
>> flushed to disk.
>> So only at three times, there should have been size difference in the
>> /index
>> folder. All the time only the size of the /tlog folder should have been
>> changing
>>
>> But actually happened is, all the time, i see the size of the /index folder
>> getting increased in parallel to the size of the /tlog folder.
>> Actually it is increasing to certain limit and coming down. Again
>> increasing
>> and coming down to a point.
>>
>> So Now the bigger doubt is have is, during hard commit, is the data getting
>> written into both /index or /tlog folder??
>>
>> I am using solr 4.5.1.
>>
>> Some one please clear me how the hardcommit works. I am asumming the
>> following sequence:
>> 1. Reads the data and writes to tlog
>> 2. During hardcommit, flushes the data from tlog to index. If openSearcher
>> is false, should not open a new searcher
>> 3. In the end, once all the datas are indexed, it should open a new
>> searcher.
>>
>> If not please explain me..
>>
>> Thanks in Advance
>> Radha
>>
>>
>>
>>
>>
>>
>>
>>
>> --
>> View this message in context:
>> http://lucene.472066.n3.nabble.com/The-way-Autocommit-works-in-solr-Wierd-tp4122558.html
>> Sent from the Solr - User mailing list archive at Nabble.com.
>>
Re: The way Autocommit works in solr - Wierd
Posted by Furkan KAMACI <fu...@gmail.com>.
Hi;
Did you read here:
http://searchhub.org/2013/08/23/understanding-transaction-logs-softcommit-and-commit-in-sorlcloud/
Thanks;
Furkan KAMACI
2014-03-10 15:14 GMT+02:00 RadhaJayalakshmi <rlakshminarayanan@inautix.co.in
>:
> Hi,
>
> Brief Description of my application:
> We have a java program which reads a flat file, and adds document to solr
> using cloudsolrserver.
> And we index for every 1000 documents(bulk indexing).
>
> And the Autocommit setting of my application is:
> <autoCommit>
> <maxDocs>100000</maxDocs>
> <openSearcher>false</openSearcher>
> </autoCommit>
>
> So after every 100,000 documents are indexed, engine should perform a
> HardCommit/AutoCommit. But still the OpenSearcher will be false.
> Once the file is fully read, we are issuing a commit() from the
> CloudSolrServer class. So this by default opens a new Searcher.
>
> Also, from the Log, i can see that three times, Autocommit is happenning.
> and Only with the last/final Autocommit, opensearcher is set to true.
>
> So, till now all looks fine and working as expected.
>
> But one strange issue i observed during the course of indexing.
> Now, as per the documentation, the data that is being indexed should first
> get written into tlog. When the Autocommit is performed, the data will be
> flushed to disk.
> So only at three times, there should have been size difference in the
> /index
> folder. All the time only the size of the /tlog folder should have been
> changing
>
> But actually happened is, all the time, i see the size of the /index folder
> getting increased in parallel to the size of the /tlog folder.
> Actually it is increasing to certain limit and coming down. Again
> increasing
> and coming down to a point.
>
> So Now the bigger doubt is have is, during hard commit, is the data getting
> written into both /index or /tlog folder??
>
> I am using solr 4.5.1.
>
> Some one please clear me how the hardcommit works. I am asumming the
> following sequence:
> 1. Reads the data and writes to tlog
> 2. During hardcommit, flushes the data from tlog to index. If openSearcher
> is false, should not open a new searcher
> 3. In the end, once all the datas are indexed, it should open a new
> searcher.
>
> If not please explain me..
>
> Thanks in Advance
> Radha
>
>
>
>
>
>
>
>
> --
> View this message in context:
> http://lucene.472066.n3.nabble.com/The-way-Autocommit-works-in-solr-Wierd-tp4122558.html
> Sent from the Solr - User mailing list archive at Nabble.com.
>