You are viewing a plain text version of this content. The canonical link for it is here.
Posted to solr-user@lucene.apache.org by darul <da...@gmail.com> on 2011/10/22 18:47:13 UTC
Re: questions about autocommit & committing documents
Old entry but I try to configure auto commit.
I am still not sure to understand how Solr handles the commit process.
Does Solr really wait for 10000 documents before send a commit ?
I was thinking, it will use maxTime and then commit a number of documents
less than 10000.
Could you please correct this following scenario:
- 20 documents are added.
- After value of maxTime is reached, the 20 documents are committed because
less than 10000 ?
- 20000 documents are added.
- After value of maxTime is reached, only the first 10000 documents are
committed. The next 10000 will on next iteration of commit phase.
Is it the right way to understand both maxTime and maxDocs parameters ?
Thanks,
> - If I enable <autoCommit> and set <maxDocs> at 10000, does it mean that
> my new documents won't be avalable for searching until 10,000 new
> documents have been added?
>
Yes, that's correct. However, you can do a commit explicitly, if you want to
do so.
--
View this message in context: http://lucene.472066.n3.nabble.com/questions-about-autocommit-committing-documents-tp1582487p3443838.html
Sent from the Solr - User mailing list archive at Nabble.com.
Re: questions about autocommit & committing documents
Posted by darul <da...@gmail.com>.
While sending documents with SolrJ Http API...at the end, I am never sure
documents are indexed.
I would like to store them somewhere and resend them in case commit has
failed.
If commit occurred every 10 minutes for example, and 100 documents are
waiting to be commit, server crash or stop..this 100 documents won't be
indexed later because my business logic won't send them again...
Then I would like create a Job (cron) which look into a table or somewhere
for documents which may not have been indexed due a problem occurred during
commit process.
--
View this message in context: http://lucene.472066.n3.nabble.com/questions-about-autocommit-committing-documents-tp1582487p3459089.html
Sent from the Solr - User mailing list archive at Nabble.com.
Re: questions about autocommit & committing documents
Posted by Erick Erickson <er...@gmail.com>.
Not sure what you mean by a callback, can you clarify? You don't get
anything except the return from the add call as far as I know...
Best
Erick
On Tue, Oct 25, 2011 at 4:15 AM, darul <da...@gmail.com> wrote:
> I was not sure thank you.
>
> --
> View this message in context: http://lucene.472066.n3.nabble.com/questions-about-autocommit-committing-documents-tp1582487p3450794.html
> Sent from the Solr - User mailing list archive at Nabble.com.
>
Re: questions about autocommit & committing documents
Posted by darul <da...@gmail.com>.
I was not sure thank you.
--
View this message in context: http://lucene.472066.n3.nabble.com/questions-about-autocommit-committing-documents-tp1582487p3450794.html
Sent from the Solr - User mailing list archive at Nabble.com.
Re: questions about autocommit & committing documents
Posted by Mark Miller <ma...@gmail.com>.
It's not 'mandatory', but it makes no sense to keep it. Even without autocommit, committing after every doc add is horribly inefficient.
On Oct 25, 2011, at 9:45 AM, darul wrote:
> Well until now I was using SolrJ API to commit() (for each document added...)
> changes but wonder in case of a production deployment it was not a best
> solution to use AutoCommit feature instead.
>
> With AutoCommit parameters, is it mandatory to remove commit() instruction
> called on CommonsHttpSolrServer
>
> try
> {
> getServer().addBean(o);
> getServer().commit(); => to remove ?
> ...}
>
> I just have another questions, I was looking all over the threads but not
> found any solutions yet about how to get a CallbackHandler with all
> documents commited. Is there a way simple way to achieve this ?
>
> Thanks again Erick.
>
>
>
> --
> View this message in context: http://lucene.472066.n3.nabble.com/questions-about-autocommit-committing-documents-tp1582487p3450739.html
> Sent from the Solr - User mailing list archive at Nabble.com.
- Mark Miller
lucidimagination.com
Re: questions about autocommit & committing documents
Posted by darul <da...@gmail.com>.
Well until now I was using SolrJ API to commit() (for each document added...)
changes but wonder in case of a production deployment it was not a best
solution to use AutoCommit feature instead.
With AutoCommit parameters, is it mandatory to remove commit() instruction
called on CommonsHttpSolrServer
try
{
getServer().addBean(o);
getServer().commit(); => to remove ?
...}
I just have another questions, I was looking all over the threads but not
found any solutions yet about how to get a CallbackHandler with all
documents commited. Is there a way simple way to achieve this ?
Thanks again Erick.
--
View this message in context: http://lucene.472066.n3.nabble.com/questions-about-autocommit-committing-documents-tp1582487p3450739.html
Sent from the Solr - User mailing list archive at Nabble.com.
Re: questions about autocommit & committing documents
Posted by Erick Erickson <er...@gmail.com>.
A full commit of all pending documents is performed whenever
the first trigger is reached.
So, maxdocs = 1000. Max time=1 minute.
Index a packet with 999 docs. Index another packet with
50 documents immediately after. One commit of 1049 documents
happens....
Index a packet of 999 docs. Do nothing for a minute. One commit of
999 docs happens because of maxtime...
But I have to ask, "why do you care"? What high level problem
are you trying to handle?
Best
Erick
On Sun, Oct 23, 2011 at 3:03 PM, darul <da...@gmail.com> wrote:
> May someone explain me different use case when both or only one AutoCommit
> parameters is filled ?
>
> I really need to understand it.
>
> For example with these configurations :
>
> <autoCommit>
> <maxDocs>10000</maxDocs>
> </autoCommit>
>
> or
>
> <autoCommit>
> <maxTime>1000</maxTime>
> </autoCommit>
>
> or
>
> <autoCommit>
> <maxDocs>10000</maxDocs>
> <maxTime>1000</maxTime>
> </autoCommit>
>
> Thanks to everyone
>
> --
> View this message in context: http://lucene.472066.n3.nabble.com/questions-about-autocommit-committing-documents-tp1582487p3445607.html
> Sent from the Solr - User mailing list archive at Nabble.com.
>
Re: questions about autocommit & committing documents
Posted by darul <da...@gmail.com>.
May someone explain me different use case when both or only one AutoCommit
parameters is filled ?
I really need to understand it.
For example with these configurations :
<autoCommit>
<maxDocs>10000</maxDocs>
</autoCommit>
or
<autoCommit>
<maxTime>1000</maxTime>
</autoCommit>
or
<autoCommit>
<maxDocs>10000</maxDocs>
<maxTime>1000</maxTime>
</autoCommit>
Thanks to everyone
--
View this message in context: http://lucene.472066.n3.nabble.com/questions-about-autocommit-committing-documents-tp1582487p3445607.html
Sent from the Solr - User mailing list archive at Nabble.com.