You are viewing a plain text version of this content. The canonical link for it is here.
Posted to solr-user@lucene.apache.org by Frederico Azeiteiro <Fr...@cision.com> on 2010/03/11 10:46:19 UTC
Cant commit on 125 GB index
Hi,
I'm having timeouts commiting on a 125 GB index with about 22000000
docs.
I'm inserting new docs every 5m and commiting after that.
I would like to try the autocommit option and see if I can get better
results. I need the docs indexed available for searching in about 10
minutes after the insert.
I was thinking of using something like
<autoCommit>
<maxDocs>5000</maxDocs>
<maxTime>86000</maxTime>
</autoCommit>
I update about 4000 docs every 15m.
Can you share your thoughts on this config?
Do you think this will solve my commits timeout problem?
Thanks,
Frederico
Re: Cant commit on 125 GB index
Posted by Lance Norskog <go...@gmail.com>.
Commit actions are in the jetty log. I don't have a script to pull
them out in a spread-sheet-able form, but that would be useful.
On 3/13/10, Frederico Azeiteiro <Fr...@cision.com> wrote:
> Yes, the http request is timing out even when using values of 10m.
>
> Normally the commit takes about 10s. I did an optimize (it took 6h) and it
> looks good for now...
>
> 59m? well i didn't wait that long, i restarted the solr instance and tried
> again.
>
> I'll try to use autocommit on a near future.
>
> Using autocommit how can i check how many commits are happening at the
> moment, when they started to? Is there a way to control and konw what is
> happening behind the scenes in "real time"?
>
> I'm using solr 1.4 with jetty.
>
> ________________________________
>
> De: Lance Norskog [mailto:goksron@gmail.com]
> Enviada: sáb 13-03-2010 23:31
> Para: solr-user@lucene.apache.org
> Assunto: Re: Cant commit on 125 GB index
>
>
>
> What is timing out? The external HTTP request? Commit times are a
> sawtooth and slowly increase. My record is 59 minutes, but I was doing
> benchmarking.
>
> On Thu, Mar 11, 2010 at 1:46 AM, Frederico Azeiteiro
> <Fr...@cision.com> wrote:
>> Hi,
>>
>> I'm having timeouts commiting on a 125 GB index with about 22000000
>> docs.
>>
>>
>>
>> I'm inserting new docs every 5m and commiting after that.
>>
>>
>>
>> I would like to try the autocommit option and see if I can get better
>> results. I need the docs indexed available for searching in about 10
>> minutes after the insert.
>>
>>
>>
>> I was thinking of using something like
>>
>>
>>
>> <autoCommit>
>>
>> <maxDocs>5000</maxDocs>
>>
>> <maxTime>86000</maxTime>
>>
>> </autoCommit>
>>
>>
>>
>> I update about 4000 docs every 15m.
>>
>>
>>
>> Can you share your thoughts on this config?
>>
>> Do you think this will solve my commits timeout problem?
>>
>>
>>
>> Thanks,
>>
>> Frederico
>>
>>
>
>
>
> --
> Lance Norskog
> goksron@gmail.com
>
>
>
--
Lance Norskog
goksron@gmail.com
RE: Cant commit on 125 GB index
Posted by Frederico Azeiteiro <Fr...@cision.com>.
Yes, the http request is timing out even when using values of 10m.
Normally the commit takes about 10s. I did an optimize (it took 6h) and it looks good for now...
59m? well i didn't wait that long, i restarted the solr instance and tried again.
I'll try to use autocommit on a near future.
Using autocommit how can i check how many commits are happening at the moment, when they started to? Is there a way to control and konw what is happening behind the scenes in "real time"?
I'm using solr 1.4 with jetty.
________________________________
De: Lance Norskog [mailto:goksron@gmail.com]
Enviada: sáb 13-03-2010 23:31
Para: solr-user@lucene.apache.org
Assunto: Re: Cant commit on 125 GB index
What is timing out? The external HTTP request? Commit times are a
sawtooth and slowly increase. My record is 59 minutes, but I was doing
benchmarking.
On Thu, Mar 11, 2010 at 1:46 AM, Frederico Azeiteiro
<Fr...@cision.com> wrote:
> Hi,
>
> I'm having timeouts commiting on a 125 GB index with about 22000000
> docs.
>
>
>
> I'm inserting new docs every 5m and commiting after that.
>
>
>
> I would like to try the autocommit option and see if I can get better
> results. I need the docs indexed available for searching in about 10
> minutes after the insert.
>
>
>
> I was thinking of using something like
>
>
>
> <autoCommit>
>
> <maxDocs>5000</maxDocs>
>
> <maxTime>86000</maxTime>
>
> </autoCommit>
>
>
>
> I update about 4000 docs every 15m.
>
>
>
> Can you share your thoughts on this config?
>
> Do you think this will solve my commits timeout problem?
>
>
>
> Thanks,
>
> Frederico
>
>
--
Lance Norskog
goksron@gmail.com
Re: Cant commit on 125 GB index
Posted by Lance Norskog <go...@gmail.com>.
What is timing out? The external HTTP request? Commit times are a
sawtooth and slowly increase. My record is 59 minutes, but I was doing
benchmarking.
On Thu, Mar 11, 2010 at 1:46 AM, Frederico Azeiteiro
<Fr...@cision.com> wrote:
> Hi,
>
> I'm having timeouts commiting on a 125 GB index with about 22000000
> docs.
>
>
>
> I'm inserting new docs every 5m and commiting after that.
>
>
>
> I would like to try the autocommit option and see if I can get better
> results. I need the docs indexed available for searching in about 10
> minutes after the insert.
>
>
>
> I was thinking of using something like
>
>
>
> <autoCommit>
>
> <maxDocs>5000</maxDocs>
>
> <maxTime>86000</maxTime>
>
> </autoCommit>
>
>
>
> I update about 4000 docs every 15m.
>
>
>
> Can you share your thoughts on this config?
>
> Do you think this will solve my commits timeout problem?
>
>
>
> Thanks,
>
> Frederico
>
>
--
Lance Norskog
goksron@gmail.com