You are viewing a plain text version of this content. The canonical link for it is here.
Posted to solr-user@lucene.apache.org by sunnyfr <jo...@gmail.com> on 2009/02/05 11:41:28 UTC
commit very long ?? solr
Hi,
Sorry but I don't know where is the problem ??
Don't you think it's a bit long ???
2009/02/05 11:30:01 started by root
2009/02/05 11:30:01 command: /data/solr/book/bin/commit
2009/02/05 11:37:03 ended (elapsed time: 422 sec)
<str name="status">idle</str>
<str name="importResponse"/>
−
<lst name="statusMessages">
<str name="Total Requests made to DataSource">85926</str>
<str name="Total Rows Fetched">43347</str>
<str name="Total Documents Skipped">0</str>
<str name="Delta Dump started">2009-02-05 11:28:19</str>
<str name="Identifying Delta">2009-02-05 11:28:19</str>
<str name="Deltas Obtained">2009-02-05 11:28:50</str>
<str name="Building documents">2009-02-05 11:28:50</str>
<str name="Total Changed Documents">12275</str>
−
<str name="">
Indexing completed. Added/Updated: 12275 documents. Deleted 0 documents.
</str>
<str name="Committed">2009-02-05 11:37:03</str>
<str name="Optimized">2009-02-05 11:37:03</str>
<str name="Time taken ">0:8:43.711</str>
Once he added/updated docs its stays a long time busy ...
delta import was started around 11:28:30
Thanks a lot for your help,
Sunny
--
View this message in context: http://www.nabble.com/commit-very-long----solr-tp21848973p21848973.html
Sent from the Solr - User mailing list archive at Nabble.com.
Re: commit very long ?? solr
Posted by sunnyfr <jo...@gmail.com>.
Batch committing is always a better option than committing for each
document. An optimize automatically commits. Note that you may not need to
optimize very frequently. For a lot cases, optimizing once per day works
fine.
Yes but commit once per day won"t show updated datas until the next day, If
I need to show them almost straight away how can I do ? How can I check
which my commit is that long ?
elapsed time: 337 sec
-- Regards - Sunny
--
View this message in context: http://www.nabble.com/commit-very-long----solr-tp21848973p21979442.html
Sent from the Solr - User mailing list archive at Nabble.com.
Re: commit very long ?? solr
Posted by Shalin Shekhar Mangar <sh...@gmail.com>.
On Thu, Feb 5, 2009 at 7:07 PM, Gert Brinkmann <g1...@netcologne.de> wrote:
> sunnyfr wrote:
> > Yes the average is 12 docs seconde updated.
>
> In our case with indexing normal web-pages on a normal workstation we
> have about 10 docs per second (updating + committing). This feels quite
> long. But if this is normal... ok.
>
> > I actually reduce warmup and cache, it works fine now, I will see if it
> > impact a lot or not the request time.
>
> Is a warmup and cache refill done on every commit? Might it be possible
> to do this on optimization only and not on every commit?
Yes autowarming, if enabled, is done on each commit. It is not possible to
do it only for optimize because the caches contain the lucene's document ids
which change on commits.
>
>
> Or does optimize automatically trigger a commit? In this case I could
> turn off committing all 100 documents and only do an optimize at the end
> of indexing.
>
Batch committing is always a better option than committing for each
document. An optimize automatically commits. Note that you may not need to
optimize very frequently. For a lot cases, optimizing once per day works
fine.
--
Regards,
Shalin Shekhar Mangar.
Re: commit very long ?? solr
Posted by Gert Brinkmann <g1...@netcologne.de>.
sunnyfr wrote:
> Yes the average is 12 docs seconde updated.
In our case with indexing normal web-pages on a normal workstation we
have about 10 docs per second (updating + committing). This feels quite
long. But if this is normal... ok.
> I actually reduce warmup and cache, it works fine now, I will see if it
> impact a lot or not the request time.
Is a warmup and cache refill done on every commit? Might it be possible
to do this on optimization only and not on every commit?
Or does optimize automatically trigger a commit? In this case I could
turn off committing all 100 documents and only do an optimize at the end
of indexing.
Thanks,
Gert
Re: commit very long ?? solr
Posted by sunnyfr <jo...@gmail.com>.
Yes the average is 12 docs seconde updated.
I've 8,5M documents and I try to update every 5mn so I guess I've no choice
with 8G of ram to have almost null warmup and cache. My data folder is about
5.8G.
What would you reckon ?
I actually reduce warmup and cache, it works fine now, I will see if it
impact a lot or not the request time.
Thanks Shalin,
Shalin Shekhar Mangar wrote:
>
> 12275 documents in 422 seconds = 29 docs/second. How fast do you want it
> to
> complete?
>
> How much time do the queries take to create a document? We don't know the
> size of the documents?
>
> On Thu, Feb 5, 2009 at 4:11 PM, sunnyfr <jo...@gmail.com> wrote:
>
>>
>> Hi,
>>
>> Sorry but I don't know where is the problem ??
>> Don't you think it's a bit long ???
>>
>> 2009/02/05 11:30:01 started by root
>> 2009/02/05 11:30:01 command: /data/solr/book/bin/commit
>> 2009/02/05 11:37:03 ended (elapsed time: 422 sec)
>>
>> <str name="status">idle</str>
>> <str name="importResponse"/>
>> -
>> <lst name="statusMessages">
>> <str name="Total Requests made to DataSource">85926</str>
>> <str name="Total Rows Fetched">43347</str>
>> <str name="Total Documents Skipped">0</str>
>> <str name="Delta Dump started">2009-02-05 11:28:19</str>
>> <str name="Identifying Delta">2009-02-05 11:28:19</str>
>> <str name="Deltas Obtained">2009-02-05 11:28:50</str>
>> <str name="Building documents">2009-02-05 11:28:50</str>
>> <str name="Total Changed Documents">12275</str>
>> -
>> <str name="">
>> Indexing completed. Added/Updated: 12275 documents. Deleted 0 documents.
>> </str>
>> <str name="Committed">2009-02-05 11:37:03</str>
>> <str name="Optimized">2009-02-05 11:37:03</str>
>> <str name="Time taken ">0:8:43.711</str>
>>
>> Once he added/updated docs its stays a long time busy ...
>> delta import was started around 11:28:30
>>
>> Thanks a lot for your help,
>> Sunny
>>
>> --
>> View this message in context:
>> http://www.nabble.com/commit-very-long----solr-tp21848973p21848973.html
>> Sent from the Solr - User mailing list archive at Nabble.com.
>>
>>
>
>
> --
> Regards,
> Shalin Shekhar Mangar.
>
>
--
View this message in context: http://www.nabble.com/commit-very-long----solr-tp21848973p21851246.html
Sent from the Solr - User mailing list archive at Nabble.com.
Re: commit very long ?? solr
Posted by Shalin Shekhar Mangar <sh...@gmail.com>.
12275 documents in 422 seconds = 29 docs/second. How fast do you want it to
complete?
How much time do the queries take to create a document? We don't know the
size of the documents?
On Thu, Feb 5, 2009 at 4:11 PM, sunnyfr <jo...@gmail.com> wrote:
>
> Hi,
>
> Sorry but I don't know where is the problem ??
> Don't you think it's a bit long ???
>
> 2009/02/05 11:30:01 started by root
> 2009/02/05 11:30:01 command: /data/solr/book/bin/commit
> 2009/02/05 11:37:03 ended (elapsed time: 422 sec)
>
> <str name="status">idle</str>
> <str name="importResponse"/>
> -
> <lst name="statusMessages">
> <str name="Total Requests made to DataSource">85926</str>
> <str name="Total Rows Fetched">43347</str>
> <str name="Total Documents Skipped">0</str>
> <str name="Delta Dump started">2009-02-05 11:28:19</str>
> <str name="Identifying Delta">2009-02-05 11:28:19</str>
> <str name="Deltas Obtained">2009-02-05 11:28:50</str>
> <str name="Building documents">2009-02-05 11:28:50</str>
> <str name="Total Changed Documents">12275</str>
> -
> <str name="">
> Indexing completed. Added/Updated: 12275 documents. Deleted 0 documents.
> </str>
> <str name="Committed">2009-02-05 11:37:03</str>
> <str name="Optimized">2009-02-05 11:37:03</str>
> <str name="Time taken ">0:8:43.711</str>
>
> Once he added/updated docs its stays a long time busy ...
> delta import was started around 11:28:30
>
> Thanks a lot for your help,
> Sunny
>
> --
> View this message in context:
> http://www.nabble.com/commit-very-long----solr-tp21848973p21848973.html
> Sent from the Solr - User mailing list archive at Nabble.com.
>
>
--
Regards,
Shalin Shekhar Mangar.