You are viewing a plain text version of this content. The canonical link for it is here.
Posted to solr-user@lucene.apache.org by Arkadi Colson <ar...@smartbit.be> on 2013/03/13 09:02:04 UTC
commit
Hi
I'm filling our solr database with about 5mil docs. All docs are in some
kind of queue which are processed by 5 simultaneous workers. What is the
best way to do commits is such a situation? If I say to let every worker
do a commit after 100 docs there will be 5 commits in a short period. Or
should I use the autocommit option for this?
Thx!
Arkadi
Re: commit
Posted by Timothy Potter <th...@gmail.com>.
collection -> Plugins / Stats -> CORE -> searcher
On Wed, Mar 13, 2013 at 4:53 AM, Arkadi Colson <ar...@smartbit.be> wrote:
> Sorry I'm quite new to solr but where exactly in the admin interface can I
> find how long it takes to warm the index?
>
> Arkadi
>
>
> On 03/13/2013 11:19 AM, Upayavira wrote:
>
>> It depends whether you are using soft commits - that changes things a
>> lot.
>>
>> If you aren't, then you should look in the admin interface, and see how
>> long it takes to warm your index, and commit at least less frequently
>> than that (commit more often, and you'll have concurrent warming
>> searchers which will use up a lot of your memory).
>>
>> If you are, then the commit frequency becomes less important. You could
>> use soft commits between 1s and 15s, and hard commits maybe every 15s to
>> 1min. Those seem to me to be reasonable values.
>>
>> Upayavira
>>
>> On Wed, Mar 13, 2013, at 09:19 AM, Arkadi Colson wrote:
>>
>>> What would be a good value for maxTime or maxDocs knowing that we insert
>>> about 10 docs/sec? Will it be a problem that we only use maxDocs = 10000
>>> because it's not searchable yet...
>>>
>>> On 03/13/2013 10:00 AM, Upayavira wrote:
>>>
>>>> Auto commit would seem a good idea, as you don't want your independent
>>>> worker threads issuing overlapping commits. There's also commtWithin
>>>> that achieves the same thing.
>>>>
>>>> Upayavira
>>>>
>>>> On Wed, Mar 13, 2013, at 08:02 AM, Arkadi Colson wrote:
>>>>
>>>>> Hi
>>>>>
>>>>> I'm filling our solr database with about 5mil docs. All docs are in
>>>>> some
>>>>> kind of queue which are processed by 5 simultaneous workers. What is
>>>>> the
>>>>> best way to do commits is such a situation? If I say to let every
>>>>> worker
>>>>> do a commit after 100 docs there will be 5 commits in a short period.
>>>>> Or
>>>>> should I use the autocommit option for this?
>>>>>
>>>>> Thx!
>>>>>
>>>>> Arkadi
>>>>>
>>>>
>>
>
Re: commit
Posted by Arkadi Colson <ar...@smartbit.be>.
Sorry I'm quite new to solr but where exactly in the admin interface can
I find how long it takes to warm the index?
Arkadi
On 03/13/2013 11:19 AM, Upayavira wrote:
> It depends whether you are using soft commits - that changes things a
> lot.
>
> If you aren't, then you should look in the admin interface, and see how
> long it takes to warm your index, and commit at least less frequently
> than that (commit more often, and you'll have concurrent warming
> searchers which will use up a lot of your memory).
>
> If you are, then the commit frequency becomes less important. You could
> use soft commits between 1s and 15s, and hard commits maybe every 15s to
> 1min. Those seem to me to be reasonable values.
>
> Upayavira
>
> On Wed, Mar 13, 2013, at 09:19 AM, Arkadi Colson wrote:
>> What would be a good value for maxTime or maxDocs knowing that we insert
>> about 10 docs/sec? Will it be a problem that we only use maxDocs = 10000
>> because it's not searchable yet...
>>
>> On 03/13/2013 10:00 AM, Upayavira wrote:
>>> Auto commit would seem a good idea, as you don't want your independent
>>> worker threads issuing overlapping commits. There's also commtWithin
>>> that achieves the same thing.
>>>
>>> Upayavira
>>>
>>> On Wed, Mar 13, 2013, at 08:02 AM, Arkadi Colson wrote:
>>>> Hi
>>>>
>>>> I'm filling our solr database with about 5mil docs. All docs are in some
>>>> kind of queue which are processed by 5 simultaneous workers. What is the
>>>> best way to do commits is such a situation? If I say to let every worker
>>>> do a commit after 100 docs there will be 5 commits in a short period. Or
>>>> should I use the autocommit option for this?
>>>>
>>>> Thx!
>>>>
>>>> Arkadi
>
Re: commit
Posted by Upayavira <uv...@odoko.co.uk>.
It depends whether you are using soft commits - that changes things a
lot.
If you aren't, then you should look in the admin interface, and see how
long it takes to warm your index, and commit at least less frequently
than that (commit more often, and you'll have concurrent warming
searchers which will use up a lot of your memory).
If you are, then the commit frequency becomes less important. You could
use soft commits between 1s and 15s, and hard commits maybe every 15s to
1min. Those seem to me to be reasonable values.
Upayavira
On Wed, Mar 13, 2013, at 09:19 AM, Arkadi Colson wrote:
> What would be a good value for maxTime or maxDocs knowing that we insert
> about 10 docs/sec? Will it be a problem that we only use maxDocs = 10000
> because it's not searchable yet...
>
> On 03/13/2013 10:00 AM, Upayavira wrote:
> > Auto commit would seem a good idea, as you don't want your independent
> > worker threads issuing overlapping commits. There's also commtWithin
> > that achieves the same thing.
> >
> > Upayavira
> >
> > On Wed, Mar 13, 2013, at 08:02 AM, Arkadi Colson wrote:
> >> Hi
> >>
> >> I'm filling our solr database with about 5mil docs. All docs are in some
> >> kind of queue which are processed by 5 simultaneous workers. What is the
> >> best way to do commits is such a situation? If I say to let every worker
> >> do a commit after 100 docs there will be 5 commits in a short period. Or
> >> should I use the autocommit option for this?
> >>
> >> Thx!
> >>
> >> Arkadi
> >
>
Re: commit
Posted by Arkadi Colson <ar...@smartbit.be>.
What would be a good value for maxTime or maxDocs knowing that we insert
about 10 docs/sec? Will it be a problem that we only use maxDocs = 10000
because it's not searchable yet...
On 03/13/2013 10:00 AM, Upayavira wrote:
> Auto commit would seem a good idea, as you don't want your independent
> worker threads issuing overlapping commits. There's also commtWithin
> that achieves the same thing.
>
> Upayavira
>
> On Wed, Mar 13, 2013, at 08:02 AM, Arkadi Colson wrote:
>> Hi
>>
>> I'm filling our solr database with about 5mil docs. All docs are in some
>> kind of queue which are processed by 5 simultaneous workers. What is the
>> best way to do commits is such a situation? If I say to let every worker
>> do a commit after 100 docs there will be 5 commits in a short period. Or
>> should I use the autocommit option for this?
>>
>> Thx!
>>
>> Arkadi
>
Re: commit
Posted by Upayavira <uv...@odoko.co.uk>.
Auto commit would seem a good idea, as you don't want your independent
worker threads issuing overlapping commits. There's also commtWithin
that achieves the same thing.
Upayavira
On Wed, Mar 13, 2013, at 08:02 AM, Arkadi Colson wrote:
> Hi
>
> I'm filling our solr database with about 5mil docs. All docs are in some
> kind of queue which are processed by 5 simultaneous workers. What is the
> best way to do commits is such a situation? If I say to let every worker
> do a commit after 100 docs there will be 5 commits in a short period. Or
> should I use the autocommit option for this?
>
> Thx!
>
> Arkadi