You are viewing a plain text version of this content. The canonical link for it is here.
Posted to solr-user@lucene.apache.org by Timothy Potter <th...@gmail.com> on 2013/02/24 18:23:18 UTC
A few operations questions about the tlog (UpdateLog)
I'm wondering if it makes sense to have the tlog on a separate disk
from the index, ie. something like:
data
|__index -> /disk1/index
|__tlog -> /disk2/tlog
Also, are large documents compressed in the tlog?
Thanks.
Tim
Re: A few operations questions about the tlog (UpdateLog)
Posted by Mark Miller <ma...@gmail.com>.
To add:
Current best practice is to do a hard commit with openSearcher=false every minute or so. AutoCommit is great for this. It shouldn't affect your overall indexing performance and it will constrain the transaction log.
- Mark
On Feb 28, 2013, at 8:35 PM, Erick Erickson <er...@gmail.com> wrote:
> A new tlog gets created and the current one is closed on a hard commit
> (openSearcher=true or false doesn't matter). The old one will be kept
> around for a bit, I suspect if you'd done it a third time the first one
> would go away. For all I know, the code might read if (tlog docs > 100) and
> you may be hitting an off-by-one situation.
>
> But you're right. If you follow older practices and _never_ commit until
> the end of a really long indexing process, you'll see these logs grow huge.
> Even worse, anytime they're replayed they'll then re-index a lot of
> documents.
>
> So I'd try the experiment over again with 110 docs.... But I can say
> they're getting purged on my machine quite regularly with some stress
> testing I'm doing.
>
> Best,
> Erick
>
>
> On Thu, Feb 28, 2013 at 7:59 AM, adityab <ad...@yahoo.com> wrote:
>
>> whats the life cycle of a tlog file. Is it purged after commit (even with
>> soft commit) ?
>> I posted 100 docs to solr (standalone) did hard commit. Observed a new tlog
>> file is created.
>> re-posted the same 100 docs and did hard commit. Observed a new tlog file
>> is
>> created. Old one still exists.
>>
>> When do they get purged. Concern is we have at least 20K docs published
>> every 2hrs so need to understand if its safe to put them in a different
>> location where we can have a script to purge old files at regular interval.
>>
>> thanks
>>
>>
>>
>>
>> --
>> View this message in context:
>> http://lucene.472066.n3.nabble.com/A-few-operations-questions-about-the-tlog-UpdateLog-tp4042560p4043618.html
>> Sent from the Solr - User mailing list archive at Nabble.com.
>>
Re: A few operations questions about the tlog (UpdateLog)
Posted by Erick Erickson <er...@gmail.com>.
A new tlog gets created and the current one is closed on a hard commit
(openSearcher=true or false doesn't matter). The old one will be kept
around for a bit, I suspect if you'd done it a third time the first one
would go away. For all I know, the code might read if (tlog docs > 100) and
you may be hitting an off-by-one situation.
But you're right. If you follow older practices and _never_ commit until
the end of a really long indexing process, you'll see these logs grow huge.
Even worse, anytime they're replayed they'll then re-index a lot of
documents.
So I'd try the experiment over again with 110 docs.... But I can say
they're getting purged on my machine quite regularly with some stress
testing I'm doing.
Best,
Erick
On Thu, Feb 28, 2013 at 7:59 AM, adityab <ad...@yahoo.com> wrote:
> whats the life cycle of a tlog file. Is it purged after commit (even with
> soft commit) ?
> I posted 100 docs to solr (standalone) did hard commit. Observed a new tlog
> file is created.
> re-posted the same 100 docs and did hard commit. Observed a new tlog file
> is
> created. Old one still exists.
>
> When do they get purged. Concern is we have at least 20K docs published
> every 2hrs so need to understand if its safe to put them in a different
> location where we can have a script to purge old files at regular interval.
>
> thanks
>
>
>
>
> --
> View this message in context:
> http://lucene.472066.n3.nabble.com/A-few-operations-questions-about-the-tlog-UpdateLog-tp4042560p4043618.html
> Sent from the Solr - User mailing list archive at Nabble.com.
>
Re: A few operations questions about the tlog (UpdateLog)
Posted by adityab <ad...@yahoo.com>.
whats the life cycle of a tlog file. Is it purged after commit (even with
soft commit) ?
I posted 100 docs to solr (standalone) did hard commit. Observed a new tlog
file is created.
re-posted the same 100 docs and did hard commit. Observed a new tlog file is
created. Old one still exists.
When do they get purged. Concern is we have at least 20K docs published
every 2hrs so need to understand if its safe to put them in a different
location where we can have a script to purge old files at regular interval.
thanks
--
View this message in context: http://lucene.472066.n3.nabble.com/A-few-operations-questions-about-the-tlog-UpdateLog-tp4042560p4043618.html
Sent from the Solr - User mailing list archive at Nabble.com.
Re: A few operations questions about the tlog (UpdateLog)
Posted by Erick Erickson <er...@gmail.com>.
Sure, putting the tlog on a different disk can reduce disk I/O contention,
of course you don't need to bother unless you can demonstrate that your
Solr app is I/O bound. If it's not, you won't see much benefit....
Don't know about the compression. Note that tlogs are only guaranteed (at
present) to contain 100 docs so we're probably not talking about huge
numbers here. If you're seeing very large tlogs, you probably should
re-visit your commit strategy.
Best
Erick
On Sun, Feb 24, 2013 at 12:23 PM, Timothy Potter <th...@gmail.com>wrote:
> I'm wondering if it makes sense to have the tlog on a separate disk
> from the index, ie. something like:
>
> data
> |__index -> /disk1/index
> |__tlog -> /disk2/tlog
>
> Also, are large documents compressed in the tlog?
>
> Thanks.
> Tim
>